Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo

Transcript

Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_04: having been in the trenches, understanding which parts worked and which parts didn't work very well.
• 0:06
  SPEAKER_04: Because yeah, you get first hand experience and you really use that in designing your solution.
• 0:12
  SPEAKER_03: And I think also the advantage is that you are decoupling what the users do right from what the platform is doing.
• 0:19
  SPEAKER_03: So then you can change the platform all you want.
• 0:22
  SPEAKER_03: You only need to keep the interface working.
• 0:24
  SPEAKER_03: And that's easy because we control the interface.
• 0:32
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:42
  SPEAKER_00: Here Airflow community we've got exciting news.
• 0:45
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:48
  SPEAKER_00: In just seven minutes you can help shape the future of Airflow and grab a free Airflow 3 fundamentals or DAG authoring certification worth$150 each.
• 0:58
  SPEAKER_00: Plus you can enter to win an exclusive workshop with Mark Lumberti.
• 1:02
  SPEAKER_00: Don't miss out.
• 1:04
  SPEAKER_00: The survey link is in the show notes and closes November 21st hello everyone welcome to the Data Flowcast.
• 1:12
  SPEAKER_02: I'm your host Mark Lumati and today I'm joined by Oscar Ligart, lead and engineer at Vinted and Rodrigo Loredo lead analytics engineer.
• 1:22
  SPEAKER_02: I'm super excited to have you both here for this episode.
• 1:25
  SPEAKER_02: But the first question is how you are doing today?
• 1:28
  SPEAKER_04: Great.
• 1:28
  SPEAKER_04: Yeah all is well excited to be on this episode excited to talk a bit about the work we've been doing and make it known to the world and hopefully be in the position to answer some questions.
• 1:39
  SPEAKER_02: Yeah no that's going to be amazing.
• 1:40
  SPEAKER_02: I'm so happy that we are finally making this episode.
• 1:43
  SPEAKER_02: So today we are talking about how Oskar and Rodrigo coordinate data workflows across multiple teams by building a code generation system that I'm super excited about actually that lets users configure pipelines through YAML files while completely abstracting away the complexity of writing Airflow DAX which is pretty awesome.
• 2:04
  SPEAKER_02: But first thing first I would like to start with a very brief question.
• 2:07
  SPEAKER_02: Can you tell us more about who you are what is your background and what you do at Vinted?
• 2:12
  SPEAKER_03: Yeah of course I can start.
• 2:14
  SPEAKER_03: So as you said I'm a lead analytics engineer and I've been working in this airflow project for a while but before I was actually in the trenches doing the pipelines and writing the SQL code and actually being a user of the solutions that we have.
• 2:32
  SPEAKER_03: So when I joined the platform team I team up with Oscar to try to actually do something better for the people because we saw that different types of teams had different knowledge and this was an opportunity to do something that actually boosted how we use Airflow at Pintel.
• 2:51
  SPEAKER_03: So now I'm more on the dark side as we said and yeah trying to be the voice of the analytics engineers and trying to bring the features that they use into the data platform to make that work awesome as I want to say.
• 3:06
  SPEAKER_03: So yeah that's me and my experience has been in data engineering analytics engineering and also some data science with ML things but a little bit of a different background and now more focused on this part of infrastructure mostly.
• 3:23
  SPEAKER_02: Do you miss missing in in being in the tranchée?
• 3:26
  SPEAKER_02: Uh no okay what about you Oscar?
• 3:32
  SPEAKER_04: Yeah it's different trenches that we're now and I was actually very happy to have Rodrigo on board having been in the trenches understanding which parts worked and which parts didn't work very well because yeah you get first hand experience and you really can use that in designing your solution.
• 3:50
  SPEAKER_04: But for me personally I have a background in AI which is a very exciting field to be a part of to have learned the basics and I transitioned into data engineering so I really in that sense jumped into the trenches.
• 4:04
  SPEAKER_04: I've been at Vintage since the beginning in the platform team for about one and a half years of which I think of yeah last year I've been spending on this project which I really thoroughly enjoyed and I think the result of it is something to be proud of if I do say so myself.
• 4:17
  SPEAKER_02: That's awesome.
• 4:18
  SPEAKER_02: Thank you both I guess obviously as a lead data engineer and the lead analytics engineer you are both managing data workflows across many different teams analytics engineers, ML engineers, data scientists what does that organizational landscape look like?
• 4:33
  SPEAKER_02: And what challenges were you facing when teams were working independently with their own airflow setups because obviously you know when you have many teams like that usually they are all working in silos.
• 4:43
  SPEAKER_04: Can you tell us more about the challenges that you faced so Vinted is Europe's largest secondhand marketplace and we're active in over 20 countries and we're at the core a data driven company.
• 4:59
  SPEAKER_04: So with millions of transactions per week you can imagine there's quite some data flowing through these pipelines.
• 5:05
  SPEAKER_04: We used to have a monolithic approach where teams would be in a single location working on a single repository.
• 5:12
  SPEAKER_04: But in order to stay ahead of the growth we decided to decentralize the tooling we then gave them was DBT we did pipelines with Vertexai for any ML jobs.
• 5:24
  SPEAKER_04: We had Spark jobs and containers that people could build and with the decentralization of teams came challenges as you said like how do we handle dependencies as you can imagine when you have a monolithic repository it's quite easy to see what's where how to use things how to grab things see if they match or not and when you start decentralizing the teams this becomes more tricky to do.
• 5:51
  SPEAKER_04: But not only that what we also find is that when people have their own setups is that people start designing their own solutions.
• 5:58
  SPEAKER_02: Right.
• 5:59
  SPEAKER_04: And this can be seen as an advantage because people can work in a decentralized way in a fast manner.
• 6:07
  SPEAKER_04: But for a centralized platform team this is the nightmare so to speak yeah in the sense that we still have to maintain this platform and with these differing solutions coming to play it's very difficult to have an understanding of the total landscape and also be able to maintain it upgrade it keep it clean keep it scalable you name to avoid being in that nightmare.
• 6:29
  SPEAKER_02: What did you decide to do?
• 6:30
  SPEAKER_02: Because I already know a little bit and actually you mentioned that at the beginning but can you tell us more about what is the solution that you found and in which way it helps you to have that solution today.
• 6:41
  SPEAKER_04: So what we did as you mentioned previously was that we built a generator which acts as an abstraction layer between the user and airflow.
• 6:51
  SPEAKER_04: So what is this abstraction layer we allow users to configure through YAML what their pipeline should look like which is then fed to a generator which uses a predetermined amount of building blocks to create a pipeline.
• 7:06
  SPEAKER_04: And what this enables us is that we nudge users to create pipelines that sort of as little pieces in a puzzle fit together.
• 7:17
  SPEAKER_04: So instead of having isolated silos each in a specific shape we sort of mold it to ensure that if you're building that grant puzzle that things fit together.
• 7:29
  SPEAKER_04: With these limited amounts of building blocks we can ensure that these dependency problems that we mentioned before can be handled because we can make contracts on how teams provide input and output we can make contracts on how we handle dependencies how we ensure that whenever something upstream fails that people are notified correctly or that all the downstream processes wait until they're actually getting the green light from their upstream dependencies and so on.
• 8:00
  SPEAKER_02: So this abstraction seems to be I would say than just for coding blocks you just mentioned that for example it helps in terms of dependencies between the teams for example that the asset of one team is ready.
• 8:12
  SPEAKER_02: So I guess just before diving into the technical details can you tell me more why first did you decide to go with YAML instead of JSON for example because you know there is always this battle between JSON and YAML and which one we should go with.
• 8:25
  SPEAKER_02: Can you tell me more about why did you go with YAML and from a user perspective like what I should do as a user if I start using your abstraction that's a really good question.
• 8:35
  SPEAKER_03: I think that it was because we were highly influenced by DBT right where you have a lot of YAML things and it's something that people know very well.
• 8:46
  SPEAKER_03: And for our data engineers we have also Kubernetes deployments that they also work with YAML so it was like a reasonable choice to go there.
• 8:57
  SPEAKER_03: The solution is nothing revolutionary I think in this case what we thought at the beginning is like simple it's good.
• 9:03
  SPEAKER_03: Like it should be boring.
• 9:04
  SPEAKER_03: It shouldn't be something very fancy okay because what people actually need to do is to connect different nodes in a massive graph of the whole organization right so what we have is that each team have their repositories and in their repositories where they write this code that they want to orchestrate they will have one file that is our scheduler file.
• 9:29
  SPEAKER_03: Also the name is very boring we had a discussion about the name and we said yeah let's put scheduler it's good we're not in sales I think yeah that's right I agree.
• 9:39
  SPEAKER_03: So there you have general alerting and ownership settings in which channel we should alert who should we tag etc and do you use dbt or not?
• 9:48
  SPEAKER_03: That's a simple question Boolean question.
• 9:51
  SPEAKER_03: If you use dbt of course we expand the graph that dbt generates like Cosmos but with our little tricks and then for other tasks like the one that Oscar mentioned for example vertex AI things we have declarative list of tasks.
• 10:09
  SPEAKER_03: So you just make a list of what you want to do and we connect those tasks with each other and with dbt by conditions.
• 10:18
  SPEAKER_03: So what are you waiting for?
• 10:20
  SPEAKER_03: And then we connect things downstream with assets.
• 10:24
  SPEAKER_03: So if your task produces something that someone else will be able to use we just put it there as okay this task generates A B and C.
• 10:34
  SPEAKER_03: And the idea is that when we deploy this code generation we not only deploy airflow Python code we also write to what we call our asset registry that specifies each DAG what building blocks for other teams and in this case I'm saying building blocks as data assets so tables or data they are generating and when they promise that data will be ready.
• 11:05
  SPEAKER_03: So an SLO for the people and then as Oscar said this allows us to generate this abstract graph so it's a DAG in the sense of the graph theory and then we translate it to airflow DAGs and to actual code.
• 11:25
  SPEAKER_03: And we did this also because we want to keep the different layers independent from each other and to be able to translate this massive graph in different ways if the time comes and we want to do something different or we need to do an upgrade or whatever.
• 11:46
  SPEAKER_02: This is super interesting.
• 11:46
  SPEAKER_02: So I guess just to summarize and let me know if I'm wrong at any point so as a user what I'm going to do is I'm not going to write any Python code.
• 11:53
  SPEAKER_02: Let's take a very simple example I have a DAG with one task.
• 11:57
  SPEAKER_02: Okay.
• 11:57
  SPEAKER_02: And I want to use let's say the Python operator then in that case I'm not going to write any Python code.
• 12:02
  SPEAKER_02: I will have a YAML file like a configuration file and then I have some fields that I will know I can use in that YML file to describe what my pipeline is going to do.
• 12:14
  SPEAKER_02: And for example this task like the task that uses the Python operator it will be like I will mention the type of the task that I want and what it produces and is that correct?
• 12:24
  SPEAKER_03: Yes exactly you have like kind of task so for example yeah Kubernetes task and then you specify an image you specify a CMD command and etc and then the powerful thing is that you can later connect this simple YAML configuration by just declaring okay I need global customers data.
• 12:50
  SPEAKER_03: Then your task will know where to look for that data in some other place for example right in some other DAC.
• 12:58
  SPEAKER_03: And that's connected through what we call the asset registry.
• 13:02
  SPEAKER_02: Okay so this is thanks to the asset registry.
• 13:05
  SPEAKER_02: Okay.
• 13:05
  SPEAKER_02: And so I guess like when I describe the task then I need to configure what this task produces as an asset.
• 13:12
  SPEAKER_02: Is that correct?
• 13:12
  SPEAKER_02: That's how you know that's how you make the links and that's register the assets in your asset registry.
• 13:17
  SPEAKER_02: Okay.
• 13:18
  SPEAKER_04: I think what's interesting here as well is what we do like Rodrigo also mentioned with these dbt settings first the bullying question we answer is whether people are making use of dbt and when they do then we dive into the lineage created by dbt and through this way of setting up the configuration people can actually insert in their lineage these specific tasks.
• 13:40
  SPEAKER_04: So we don't have a list of tasks that will run one after the other we go top to bottom we actually have this setup where we have to say what are you waiting for and this could be different assets outside of the team is something that they could be waiting for or a specific model within deep within their DBT projects.
• 13:58
  SPEAKER_01: So they have a lot of flexibility in inserting these tasks in the correct spot at their lines this episode is brought to you by Astronomer creators of Astro the leading fully managed airflow service designed for the modern data stack.
• 14:16
  SPEAKER_02: Whether you're orchestrating large language models implementing a data mesh architecture or streamlining complex workflows Astro empowers your team to focus on innovation not infrastructure to see how you can maximize airflow's potential within your data ecosystem visit www.astronomer.io If we focus a little bit more on the assets part if I have my DAG and it needs to wait for an asset that comes from another DAG using your solution I guess you rely a lot on sensors to make sure that the dependencies are met.
• 14:50
  SPEAKER_02: Can you tell me more about that?
• 14:52
  SPEAKER_02: Like how do you coordinate the different tasks together using sensors and so on?
• 14:56
  SPEAKER_02: I imagine it's a pretty complex solution.
• 14:59
  SPEAKER_03: It's complex but not complicated.
• 15:01
  SPEAKER_03: The idea is that we don't hack airflow in any way we are not forking airflow and making our own things we are using airflow custom behaviors so things that are already available in the tool.
• 15:16
  SPEAKER_03: I'm gonna also add something to your question is like why are you not using data sets or in airflow three the new assets right and the limitation that we found there is that those are very useful if you are waiting between DAX but we have very big DAX where the dependencies can also be in different parts of a DAX in different tasks.
• 15:39
  SPEAKER_03: And if you look at from the DAG perspective there might be some circular dependencies between teams.
• 15:45
  SPEAKER_03: Okay.
• 15:46
  SPEAKER_03: But inside if you look at the tasks there are no circular dependencies.
• 15:51
  SPEAKER_03: Of course if we wanted to use events or the data sets or the assets we could split these bigger DAGs into smaller DAGs but then that split in our case look very artificial and people should look into three or four DAGs to understand how they are connected and like all those things.
• 16:14
  SPEAKER_03: So then what we wanted to do is the same data sets or assets do but with tasks.
• 16:21
  SPEAKER_03: And of course the clear candidate for that are sensors that's quite straightforward to do and for the sensors you need to define a DAG where a task is running and a timeout and we get that from the asset registry.
• 16:37
  SPEAKER_03: But I think that the magic comes from something that we do in the tasks that are being censored that are the tasks that other DAGs are waiting for that is that we have a callback function that automatically clears the sensors that are waiting for that task.
• 16:58
  SPEAKER_03: If everything goes well you don't need that because everything goes well the sensor works and yeah you're happy but if one team has something that fails and your sensor fails in another part then when you fix your thing you need to tell the other person hey I fixed my thing can you rerun and then you need to start this communication in this decentralized way that it's very annoying if you have like 15 people saying like hey are you done have you fixed it whatever.
• 17:27
  SPEAKER_03: But then what we have here is that with that callback function when you fix your things if someone else is waiting for that and the sensor is still in their timeout you don't even know but when the sensor failed and you know that someone is in fault when you fix it you automatically re-trigger everything.
• 17:48
  SPEAKER_03: So in this way we are connecting all the tags as if there were one massive one but in little blocks.
• 17:55
  SPEAKER_03: And this is also quite cool because of the alerting that we can do with that because the owner of the task can know that their thing failed but then the sensor person so the sensor person can know like hey something failed in your pipeline but it's this other person's fault or responsibility let's say I see so you can go find them you can help them to fix the problem you can bring them yeah and then when they fix their system everything resumes automatically and and everything it's cleared up and yeah in a sense it works as the data sets or assets but at the task level okay and I think that's the powerful thing of it.
• 18:40
  SPEAKER_02: I love the solution because you know in a way it seems that you definitely had some constraints in terms of how you can write create your pipelines but also based on your explanations it feels also that you add some flexibility.
• 18:54
  SPEAKER_02: The fact that you go at the task level for creating the dependencies and so on it actually adds some flexibility as well.
• 19:00
  SPEAKER_02: Honestly that's the first time that I hear that.
• 19:03
  SPEAKER_02: So speaking of constraints obviously like I'm sure that the solution is great but you know what it is right like for some engineers they don't like that much the constraints for example let's say I want to add a task that it is not in the provided list that you have of tasks that I can use like how does it work then if I want to add something that it is not in the system yet how does it work it's a good question and definitely a relatable one.
• 19:28
  SPEAKER_04: The way we try to mitigate it is through that possibility to create a container.
• 19:35
  SPEAKER_04: So a team can create a container with their custom solution or custom code that they can then schedule.
• 19:42
  SPEAKER_04: So we have a full CI CD system that can create this container it ends up in the correct registry and then they can configure this as a task block through the kind Kubernetes in order to ensure that this is still feasible.
• 19:55
  SPEAKER_04: So in practice we have this solution but we definitely find people still feel constrained.
• 20:02
  SPEAKER_04: I think one of the difficulties that this center setup introduces is that we have to set specific fixed schedules.
• 20:11
  SPEAKER_04: So we cannot say that everyone is free to set up their cron because then through the behavior of a sensor it would be very difficult to know which interval to pull because it's like Rodrigo said it's one connected tag but we still want to connect it on the right intervals.
• 20:31
  SPEAKER_04: We can't just say pick any and if you find a green block somewhere go ahead actually that might be a cool feature.
• 20:38
  SPEAKER_04: So with that in mind we had one of the constraints set up to be you can only select a predefined set of schedules say daily hourly weekly and users at first there was a bit of backlash but luckily we had one of the soldiers in our team so he could keep everyone at bay but it also introduced a new way of thinking for most so do you really need this everyone's in a blue moon when it's the year of the dragon kind of cron setup or can you remodel your solution to be fitting to an hourly pipeline so that whatever your solution produces can also be used according to specific expectations and rules by other teams without needing to set up 17 meetings to understand how to connect the two together.
• 21:31
  SPEAKER_04: So in terms of falling back to your question these constraints depending on the persona you are dealing with it can be taken in different ways but we see them as a very positive thing.
• 21:42
  SPEAKER_02: Actually I can relate a lot on what you said about do you truly need that?
• 21:46
  SPEAKER_02: Because it's always easy to add things but it's much harder to remove those things.
• 21:50
  SPEAKER_02: So it's nice to think twice about okay do I truly need that?
• 21:55
  SPEAKER_02: And if so then let's figure out the right way to do it to add that.
• 21:58
  SPEAKER_02: But you know we might Constraints, but I'm sure that when we have this layer of abstraction, it can help a lot in terms of migration.
• 22:07
  SPEAKER_02: Like whenever you need to upgrade Airflow, for example, can you tell me more about that?
• 22:12
  SPEAKER_02: Actually, does that solution help in terms of migrating your DAGs or migrating from Airflow 2.10 to 3, for example?
• 22:21
  SPEAKER_02: Like something that you find out?
• 22:23
  SPEAKER_04: This helps massively.
• 22:24
  SPEAKER_04: Okay.
• 22:25
  SPEAKER_04: So we used to have periodical upgrades, I think back in the time of 2.6 to 2.7, and so on, where we didn't have this generator, we didn't have the centralized solution of a limited amount of building blocks.
• 22:38
  SPEAKER_04: And what we found in said setups is that people are creating their own solutions and they might have different dependencies, and some might be better maintainers than others.
• 22:50
  SPEAKER_04: And then you have to put them all in one bag, and then from one day to the other, say, okay, we're gonna run this upgrade.
• 22:56
  SPEAKER_04: And you have to sift through.
• 22:57
  SPEAKER_04: Python is notorious for this, all the dependency issues you might have for any breaking chains that might be introduced along the way.
• 23:05
  SPEAKER_04: So we had to go team by team to understand, okay, can we change this?
• 23:09
  SPEAKER_04: We don't know where it's used, downstream or upstream, things weren't connected properly.
• 23:13
  SPEAKER_04: So we were interviewing all of the time, and then there was a big script of steps to follow for running such a migration.
• 23:20
  SPEAKER_04: I think a lot of platforms as data engineers will be familiar with this problem.
• 23:25
  SPEAKER_04: I don't think it will be new to many people.
• 23:27
  SPEAKER_04: But in the case of this generator, what we do through this abstraction is that we force people to create their pipelines based on a limited amount of building blocks.
• 23:38
  SPEAKER_04: I think the one we would use the majority of is GKE bot operator, where people run containers.
• 23:45
  SPEAKER_04: And imagine you're only so you start out where you have hundreds of operators, you have to have hundreds of cases that you have to fix, and then you boil it down to we are running two or three operators.
• 23:56
  SPEAKER_04: Right now we have to go through those operators, fix the braking changes, if any.
• 24:01
  SPEAKER_04: And the best part is we know how they connect to one another, right?
• 24:05
  SPEAKER_04: We know through the sensor setup how things are orchestrated, how things are connected.
• 24:09
  SPEAKER_04: So we don't have to go around and ask this question anymore.
• 24:12
  SPEAKER_04: So it's a massive win in terms of maintenance and the ability to upgrade, but also I would even extend it to how we scale the system.
• 24:22
  SPEAKER_04: If you have the exact expectation of what's going to be running or what kind of tasks are going to be running, you can really design your solution around that.
• 24:31
  SPEAKER_04: And we don't have to account for people introducing unexpected building blocks that might be very resource intensive or hungry, and we have to change our system to fit those needs.
• 24:44
  SPEAKER_04: So for us, it's a massive win in that sense.
• 24:47
  SPEAKER_04: And with the generator setup, we provide the flexibility that our users need.
• 24:52
  SPEAKER_04: So I say win all across the board.
• 24:55
  SPEAKER_03: And I think also the advantage is that you are decoupling what the users do from what the platform is doing.
• 25:04
  SPEAKER_03: So then you can change the platform all you want, you only need to keep the interface working, and that's easy because we control the interface, right?
• 25:12
  SPEAKER_03: So then people don't need to change anything or almost anything, and we can just change that connection and make it work, and that's very powerful.
• 25:24
  SPEAKER_03: And I think people is very happy with that because they cannot believe, okay, I only need to put DVT truth and it works.
• 25:31
  SPEAKER_03: We have happy users as well, believe it.
• 25:33
  SPEAKER_03: We're happy because you also give time back to them to focus in what is their area of expertise, right?
• 25:41
  SPEAKER_03: Like why they don't want to debug airflow problems because they are not expert in airflow, they want to debug vertex AI problems.
• 25:49
  SPEAKER_03: So, okay, you just don't need to debug them anymore.
• 25:52
  SPEAKER_03: So they're quite happy with that, I think.
• 25:55
  SPEAKER_02: That's what I was about to say.
• 25:56
  SPEAKER_02: Obviously, that as you said, right?
• 25:57
  SPEAKER_02: If I'm a data engineer, if I'm a data analyst, if I'm a ML engineer, whatever, like I have different expertise, and maybe I don't want to spend a ton of time on debugging Airflow and so on.
• 26:06
  SPEAKER_02: I actually want to focus on things that matter a lot, and your solution seems to do that.
• 26:11
  SPEAKER_02: And also as a data engineer, but I'm sure that it is the case for other engineers as well.
• 26:15
  SPEAKER_02: But I don't like being stuck in Airflow 2 just because I have so many customizations, so many things that I built on top of Airflow.
• 26:23
  SPEAKER_02: So actually, your solution provides also a great way to migrate from different Airflow versions without having to go through a ton of fixes just because, again, too many things built on top of Airflow right away.
• 26:35
  SPEAKER_02: Yeah, I'm sure that definitely like people love your solution at your company.
• 26:40
  SPEAKER_02: I'm sure they do.
• 26:41
  SPEAKER_02: I guess one very last technical question: do you rely a lot on container operators, like the Kubernetes pod operator?
• 26:48
  SPEAKER_02: Is it something that you actually rely a lot on, or do you use the classic Python operator or things like that?
• 26:53
  SPEAKER_04: Basically, all we use are the container operators.
• 26:56
  SPEAKER_04: We are connecting to Kubernetes and we run everything in the containers.
• 27:00
  SPEAKER_04: I also find it's an interesting question because for me, the power of airflow is especially very strong when you get started.
• 27:08
  SPEAKER_04: Because it's such a huge open source community, the chances of there already having been someone that created an operator that may fit to your platform or your use case is incredibly big.
• 27:20
  SPEAKER_04: So I think getting started is a massive win, and you can really set up your pipelines from scratch in a quick manner.
• 27:29
  SPEAKER_04: However, when you scale to a larger company, I think this strong suit of airflow can turn into a challenge because of the reasons mentioned.
• 27:39
  SPEAKER_04: Maintaining a system where teams start grabbing operators out of the thin air, out of the abundance of operators that are there.
• 27:47
  SPEAKER_04: But the only one I would still allow is the smooth operator.
• 27:50
  SPEAKER_04: For listeners that are not familiar with this one, look it up.
• 27:53
  SPEAKER_04: It's amazing.
• 27:54
  SPEAKER_04: Put it in your pipelines, have a laugh.
• 27:56
  SPEAKER_02: It's basically it brings AGI in your pipelines.
• 27:58
  SPEAKER_04: Exactly.
• 27:59
  SPEAKER_04: Bring a bit of flair to your pipelines.
• 28:01
  SPEAKER_04: It's important.
• 28:02
  SPEAKER_04: We have to enjoy our work.
• 28:03
  SPEAKER_02: I agree.
• 28:04
  SPEAKER_02: That's amazing.
• 28:05
  SPEAKER_02: Look, guys, I could spend like hours with you.
• 28:07
  SPEAKER_02: It's truly amazing.
• 28:08
  SPEAKER_02: But I just would like to mention one last thing.
• 28:10
  SPEAKER_02: This is just for the audience, actually, for the listeners.
• 28:13
  SPEAKER_02: This is just a taste of Oscar and Rodrigo's solution that they have implemented in their company.
• 28:18
  SPEAKER_02: If you want to learn more about that, you will learn more about that in the Airflow Summit.
• 28:22
  SPEAKER_02: Again, I think that's a really solid solution.
• 28:25
  SPEAKER_02: I can't wait to be at the Airflow Summit as well to learn more about the implementation of it.
• 28:29
  SPEAKER_02: But can you tell to the audience what is the best way for Fults to get in touch with you if they would like to learn more about your solution, if they want to just speak with you and so on?
• 28:39
  SPEAKER_03: Yeah, definitely.
• 28:40
  SPEAKER_03: I think the best way, at least for me, it will be to look it up in LinkedIn.
• 28:46
  SPEAKER_03: I'm there.
• 28:47
  SPEAKER_03: I have an abandoned soapstack that I write some stuff once in a while when I want to rant about something.
• 28:54
  SPEAKER_03: But LinkedIn definitely.
• 28:56
  SPEAKER_04: Same goes for me.
• 28:58
  SPEAKER_04: Just type the name in LinkedIn and feel free to connect, and I'll happily have a discussion and dive a bit deeper into the solution or be criticized even.
• 29:06
  SPEAKER_04: Also, awesome.
• 29:07
  SPEAKER_02: Guys, again, thank you so much for joining.
• 29:09
  SPEAKER_02: It was a wonderful pleasure.
• 29:10
  SPEAKER_02: Who knows?
• 29:10
  SPEAKER_02: Maybe I will see you again for another episode.
• 29:12
  SPEAKER_02: Thank you so much.
• 29:13
  SPEAKER_04: We love that.
• 29:14
  SPEAKER_04: Thank you so much.
• 29:17
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 29:24
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 29:28
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five-star review.