Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari

Transcript

How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: Can call like that creation or HTTP calls.
• 0:03
  SPEAKER_03: Basically, what this provides the end users is like a flexibility of a UI tool to just drag and drop the different task or operators, specify the dependency, like you can create a graph out of it, and specify the parameter.
• 0:16
  SPEAKER_03: If you want to just run the HQL query on hype, you need to specify VMQ and the query.
• 0:22
  SPEAKER_03: And you don't need to write any code, right?
• 0:24
  SPEAKER_03: It's like a local solution for creating a pipeline.
• 0:32
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:42
  SPEAKER_00: Airflow users, we have exciting news.
• 0:46
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:49
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow.
• 0:52
  SPEAKER_00: And as a thank you for filling it out, you'll get a free Airflow Fundamentals or DAG Authoring certification.
• 0:58
  SPEAKER_00: Those are worth$150 each.
• 1:01
  SPEAKER_00: Plus, you'll also be entered to win an exclusive workshop with Mark Lumberti.
• 1:05
  SPEAKER_00: So make sure you don't miss out.
• 1:07
  SPEAKER_00: The survey link is in the show notes and it closes December 20th.
• 1:12
  SPEAKER_00: Hi everyone.
• 1:13
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:15
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Sumit Nashwari and Shobit Shah, who are two staff software engineers at Uber.
• 1:23
  SPEAKER_00: Sumit, Shobit, how are you guys doing?
• 1:26
  SPEAKER_03: Hey, connect yeah, great.
• 1:27
  SPEAKER_03: Thanks for having us on this podcast.
• 1:29
  SPEAKER_03: Yeah, doing good.
• 1:31
  SPEAKER_00: Yeah, great.
• 1:32
  SPEAKER_00: Wonderful to have you here.
• 1:33
  SPEAKER_00: Remind me where in India you both are.
• 1:35
  SPEAKER_00: You're obviously together in the office.
• 1:37
  SPEAKER_03: Yeah, yeah.
• 1:39
  SPEAKER_03: We both are in my Uber office right now.
• 1:43
  SPEAKER_00: Okay, that's great.
• 1:43
  SPEAKER_00: Does Uber have a large office there?
• 1:46
  SPEAKER_03: Yeah, pretty good.
• 1:47
  SPEAKER_03: Yeah, pretty disendious.
• 1:49
  SPEAKER_00: All right.
• 1:50
  SPEAKER_00: Well, let's go ahead and dive in.
• 1:52
  SPEAKER_00: So for today's podcast for our listeners, we're going to be discussing the evolution of Airflow at Uber, which is a really rich story.
• 1:58
  SPEAKER_00: I'm excited to talk to you both about it.
• 2:00
  SPEAKER_00: So let's start with some background.
• 2:02
  SPEAKER_00: I'm sure everybody is familiar with Uber and what Uber does, but can you describe what your team does within Uber?
• 2:10
  SPEAKER_03: So basically, our team is data workflow management solution at Uber.
• 2:15
  SPEAKER_03: So we are a part of the platform engineering team, which manages all the toolings for the different teams here.
• 2:21
  SPEAKER_03: We provide like airflow like setup and those service for all the teams who want to create a workflow orchestration pipelines and tags.
• 2:29
  SPEAKER_03: Yeah, I think we are involved in feature development migrations of Python versions.
• 2:34
  SPEAKER_03: Recently we have heard that Uber is going from OneTime Stack to Cloud.
• 2:38
  SPEAKER_03: So we are helping with the data cloud migration.
• 2:41
  SPEAKER_03: And I believe this is like one of the large supports of Airflow folk.
• 2:45
  SPEAKER_03: We have about 200,000 active pipelines running with 45,000 Python files and 1 million task executions a day.
• 2:55
  SPEAKER_00: And that's all managed by your team.
• 2:57
  SPEAKER_00: So your team manages 200,000 pipelines.
• 3:00
  SPEAKER_00: Wow.
• 3:02
  SPEAKER_03: Like there are different teams that owns these pipelines.
• 3:11
  SPEAKER_00: Got it.
• 3:11
  SPEAKER_00: Got it.
• 3:11
  SPEAKER_00: That is a large Airflow instance or Airflow-like instance, as we'll dive into for sure.
• 3:17
  SPEAKER_00: And I'd imagine you mentioned so you're working on a on-prem to cloud migration.
• 3:22
  SPEAKER_00: That's a big project for a company like Uber.
• 3:25
  SPEAKER_03: Yeah, I'm doing span across multiple years.
• 3:28
  SPEAKER_00: Yeah, very cool.
• 3:30
  SPEAKER_00: Um Great.
• 3:31
  SPEAKER_00: So with that background, you recently spoke at the Airflow Summit about your evolution from implementing Airflow at Uber originally with open source Airflow to you just alluded to a customized solution that's built on top of Airflow.
• 3:46
  SPEAKER_00: So let's dig into that.
• 3:47
  SPEAKER_00: Can you share some details about that evolution and what your current solution looks like?
• 3:52
  SPEAKER_02: Yeah, sure.
• 3:53
  SPEAKER_02: As you might have listened to our talk in Airflow summit, it's a multi-year journey actually.
• 3:59
  SPEAKER_02: And all it started around 2016 when Uber decided to say use Airflow as the data map process.
• 4:06
  SPEAKER_02: There were multiple systems at the time, but we actually decided to double down on Airflow.
• 4:11
  SPEAKER_02: And then the very first instance was the Chilean Airflow instance, and the old DSL was Air Producer to write the tags about it.
• 4:19
  SPEAKER_02: So in that in the whole journey from 2016-17 to today, there were a lot of pivotal points, I would call them, which makes our solution different from the airflow because we wanted different directions.
• 4:30
  SPEAKER_02: And those pivotal points I would I would start with the first one, which was around the serialization.
• 4:36
  SPEAKER_02: So if you remember around that time, 2016 17, I think Airflow 1.6 or 1.7 was out.
• 4:42
  SPEAKER_02: And all the services in the MM server scheduled the worker, everybody would need the pipeline tech file, and they would parse it at the runtime.
• 4:51
  SPEAKER_02: And it's a little messy, right?
• 4:53
  SPEAKER_02: It's not very secure.
• 4:54
  SPEAKER_02: It is like very error prone because a bit code could actually hamper or can create down types, the other services.
• 5:01
  SPEAKER_02: So the very first feature point was around spitting out the selection part into different services and making changes into the web server and scheduler to not pass the Python find again and again and just use the metadata from the database and just work on from it actually.
• 5:17
  SPEAKER_02: So I think this is something or very similar to what came later in Airflow, I think around 2020 time.
• 5:23
  SPEAKER_02: It's very similar to that.
• 5:24
  SPEAKER_02: But yeah, our schema might be different than what Airflow has.
• 5:28
  SPEAKER_02: But Aero Concept is the same.
• 5:30
  SPEAKER_02: The another big point, I think, is around like the getting a new scheduler service out of the existing Airflow scheduler.
• 5:38
  SPEAKER_02: So if you remember, there was a lot of issues with the Airflow scheduler back in time.
• 5:42
  SPEAKER_02: Like it was not in the HA service.
• 5:45
  SPEAKER_02: It used to go high, right?
• 5:47
  SPEAKER_02: And then somebody go and need to restart the scheduler to work it again and things like that.
• 5:52
  SPEAKER_02: So it was actually redetailed into Java.
• 5:55
  SPEAKER_02: I think the reason why Java because at the time was only encouraging Go and Java services.
• 6:00
  SPEAKER_02: So that is what I think really or uh thing that is the reason to add into Java.
• 6:05
  SPEAKER_02: Otherwise, it would have been into Python itself.
• 6:07
  SPEAKER_02: But yeah, that it was actually uh split out into three different services.
• 6:11
  SPEAKER_02: So the first service was to just keep track of a number of scheduler nodes and assign the pipeline IDs in an HTML so that each scheduler node would actually look at one of the pattern IDs that are assigned to it.
• 6:23
  SPEAKER_02: The second service is very similar to what F schedular is.
• 6:27
  SPEAKER_02: So it's just letting all the brain about it, like creating new tasks, new pipeline drugs, and creating that the task themselves.
• 6:34
  SPEAKER_02: And the third service was just to mate the poll, the quotas, and the Redis connections, basically, like how you push the data into Redis so that the downstream series executors can actually pick it up.
• 6:47
  SPEAKER_02: So yeah, that was uh very big change overall.
• 6:50
  SPEAKER_02: And the third one, I believe, is around the disaster recovery.
• 6:54
  SPEAKER_02: So I think back in 2021, Uber decided to not dual compute the data sets.
• 7:01
  SPEAKER_02: So just a bit story there.
• 7:03
  SPEAKER_02: So Uber used to have still have data centers and it used to dual compute all the data on primary and secondary DC.
• 7:10
  SPEAKER_02: So that in the event of a disaster, all the systems can run out of the secondary and they would be just as functional as they are on primary.
• 7:18
  SPEAKER_02: But in 2021, it was decided that maybe we do not really need to dual compute the data, and we can just you know have the syncing mechanism from primary to secondary.
• 7:28
  SPEAKER_02: So in the event of some issues, event of a disaster, we can actually switch to secondary and then resume the computation on the secondary.
• 7:36
  SPEAKER_02: So the secondary is running a competent computation now, roughly 50% of the primary.
• 7:41
  SPEAKER_02: So our system is also very unique.
• 7:43
  SPEAKER_02: It is not Active MLP, it is not active PC like that because we are running 200k pipelines in primary, we'll be running around 50-60k pipelines in the secondary as well all the time.
• 7:54
  SPEAKER_02: But in the event of a disaster, we would be switching the flags.
• 7:58
  SPEAKER_02: And then in the secondary would be running around close to 200k pipelines again, right?
• 8:02
  SPEAKER_02: So it's a mix of active PC system.
• 8:04
  SPEAKER_02: So a lot of customizations and lot of code we have to build on top of say Floor Piper to support this use case.
• 8:14
  SPEAKER_02: And I think that is another big digression from the existing FLO world because so far I'm not aware about any subsystem out there.
• 8:23
  SPEAKER_02: Yeah, some more things.
• 8:24
  SPEAKER_03: So I think, yeah, I think I think SOP can need more.
• 8:27
  SPEAKER_03: Yeah, yeah.
• 8:27
  SPEAKER_03: So I can talk about the marriage pipelines and the backwing pipelines and about some data migration.
• 8:33
  SPEAKER_03: Yeah, Mikulak, do you have some even?
• 8:37
  SPEAKER_00: Sure.
• 8:37
  SPEAKER_00: I guess it just to sum up, so you guys started diverging or building off of Open Source Airflow pretty early on.
• 8:45
  SPEAKER_00: I think you mentioned 1.6, 1.7.
• 8:47
  SPEAKER_00: That was quite some time ago.
• 8:49
  SPEAKER_00: Um, I know you mentioned the scheduler was not as reliable than it is as it is today uh in open source airflow, which is certainly is a problem I remember myself.
• 8:57
  SPEAKER_00: I think I started on 1.9, maybe when I was running Airflow.
• 9:02
  SPEAKER_00: And yeah, remember the days of having to manually restart, which you fortunately don't have to do anymore.
• 9:08
  SPEAKER_00: But so that totally makes sense.
• 9:10
  SPEAKER_00: So you were just a little bit ahead of the game with implementing your own HA system.
• 9:16
  SPEAKER_02: Yeah.
• 9:17
  SPEAKER_02: Yeah.
• 9:17
  SPEAKER_00: Yeah.
• 9:18
  SPEAKER_02: I think that's a I think 1.9, maybe it is better, but if you work in 1.6 and 1.7, I think the problem was much more prominent there.
• 9:25
  unknown: Yeah.
• 9:26
  SPEAKER_02: So I think if you are running at a scale, I think those solutions had to be built into it, right?
• 9:31
  SPEAKER_02: Uh some yeah.
• 9:32
  SPEAKER_02: So I think that was the whole reason of.
• 9:34
  SPEAKER_03: Yeah, I think we started with that 1.6 version and a lot of issues.
• 9:39
  SPEAKER_03: Yeah.
• 9:40
  SPEAKER_00: Yeah.
• 9:40
  SPEAKER_00: Yeah.
• 9:41
  SPEAKER_03: That makes sense.
• 9:43
  SPEAKER_03: Yeah.
• 9:43
  SPEAKER_03: So I will uh add a few more features that we have built custom on top of this piper.
• 9:48
  SPEAKER_03: So one of my favorite features is about the managed findings, or you can call like that creation over HTTP calls.
• 9:55
  SPEAKER_03: Basically, what this provides the end users is like a flexibility of a UI tool to just drag and drop the different task or operators, specify the dependency like you can create a graph out of it, and specify the parameter.
• 10:08
  SPEAKER_03: If you want to just run the HQ inquiry on the high, you need to specify VMQ and the query.
• 10:14
  SPEAKER_03: And you don't need to write any code, right?
• 10:16
  SPEAKER_03: It's like a local uh solution for creating a pipeline.
• 10:19
  SPEAKER_03: And we'll just go into a bit more detail on how this things work.
• 10:23
  SPEAKER_03: So basically, what we do is like for any operator when you're trying to create a diagonal pipeline, the tasks and operators, if those are night standard, supported by internal stack, it's relatively very simple.
• 10:35
  SPEAKER_03: You just need to uh pass in the queries and some information.
• 10:39
  SPEAKER_03: The only thing that is different that needs to have at the time of task running is the query and some configuration parameters.
• 10:46
  SPEAKER_03: So, what we do is like we have defined a standardized format in JSON about the contract that will be passed by the UI, or users can also call the API directly or HTTP.
• 10:57
  SPEAKER_03: So they can tell that this are my tasks that I'm going to run.
• 11:00
  SPEAKER_03: This is the configuration you should run it with at this given schedule, and this is the dependency.
• 11:05
  SPEAKER_03: So, what we do is like when we receive the request on the web server, we will just save it in the database and then we will launch a synchronous only one realization of that particular data.
• 11:15
  SPEAKER_03: So and on the UI, we just return responses uh 200k.
• 11:19
  SPEAKER_03: And in background, what we do is we will try to see like we have our own custom driver that will leave the entry from the database about this configuration.
• 11:26
  SPEAKER_03: It will check okay, this is the task and this is the configuration.
• 11:30
  SPEAKER_03: So it will generate the back on the fly based on the configuration and return the pipeline object.
• 11:36
  SPEAKER_03: So this makes it like very easy to for the users who are not familiar with coding, machine learning scientists, and data scientists.
• 11:45
  SPEAKER_03: So not just from the UI line, we have a lot of services that have integrated with us that are using this API to create pipelines on the fly as when they need, and then they deactivate the pipelines.
• 11:57
  SPEAKER_03: So roughly, I think about 50% of pipelines today are created through this API compared to the traditional code-based approach.
• 12:06
  SPEAKER_03: Yeah, this was like one of my favorite features that is different in Piper compared to Atlow.
• 12:11
  SPEAKER_03: Another similar feature is allowed with backfills.
• 12:13
  SPEAKER_03: I think I still going to have this backfilling support in the new version of IFLO3 from the UI.
• 12:20
  SPEAKER_03: So yeah, before that, you have some questions or anything.
• 12:34
  SPEAKER_00: Did you implement that as an Airflow plugin, the drag and drop interface, or is it something else?
• 12:40
  SPEAKER_03: Well, it's like we have a custom UI interface tooling that is used for creating a real nine uh workload as well, like we have already now and uh free jobs.
• 12:49
  SPEAKER_03: So it's like a common UI for creating us in the engine, and that is actually easy to overall that particular engine.
• 12:55
  unknown: Okay.
• 12:56
  SPEAKER_03: This is not like in the card for learning team.
• 13:02
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 13:11
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 13:23
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:35
  SPEAKER_00: I think having a drag and drop extension seems to be something that's divisive in the open source airflow project.
• 13:42
  SPEAKER_00: But it's often like pipelines and Python is the core tenant of Airflow.
• 13:46
  SPEAKER_00: And so there are some folks who are very purist and against adding that to open source Airflow.
• 13:50
  SPEAKER_00: And usually the answer I hear from those people is well, Airflow is super flexible.
• 13:54
  SPEAKER_00: So if that's something you need for your company, you can just build it, which is exactly what you've done.
• 13:59
  SPEAKER_00: But that's pretty cool.
• 14:00
  SPEAKER_00: But for like a pretty huge scale that you're talking about, you said half of the pipelines are submitted in this way.
• 14:08
  SPEAKER_00: So a hundred thousand or so.
• 14:11
  SPEAKER_03: Yeah, exactly.
• 14:12
  SPEAKER_03: And this makes the management of pipelines also easier.
• 14:15
  SPEAKER_03: Because there is no code from the user side.
• 14:17
  SPEAKER_03: It's just like the task name or the operator name and the configuration, right?
• 14:21
  SPEAKER_03: So let's say if you want to do migration of Python 2 to Python 3 or version upgrade, so we just have to take care of the driver that we own as a team instead of going to customers and tell them that fix this thing was uh compatibly in the new way.
• 14:35
  SPEAKER_00: Yeah, interesting.
• 14:37
  SPEAKER_02: I think just to add in the similar so I think though Apro as the open system community may not is supportive of writing this in drop UI, but uh a lot of companies are using like something similar, maybe like having a Yaml Spy.
• 14:48
  SPEAKER_02: You write your query there, it will create a make out of it, or because there are common use cases come across in each of these companies, right?
• 14:56
  SPEAKER_02: And they will try to find out the common solution which can work for the majority part of their use base, like user base.
• 15:02
  SPEAKER_02: So I think I just seen something similar into multiple companies.
• 15:06
  SPEAKER_00: Yeah, definitely config-based DAG generation is very popular.
• 15:11
  SPEAKER_00: In fact, recently, Astronomer took over the open source DAG factory package and are updating it.
• 15:17
  SPEAKER_00: That's something we've seen a ton of use with.
• 15:19
  SPEAKER_00: So there's definitely value in, as you said, democratizing airflow usage or whatever Airflow-esque system and opening it up to folks who don't necessarily have Python experience or even just experience writing pipelines in that way.
• 15:35
  SPEAKER_02: So in Uber, also we have something similar as well.
• 15:38
  SPEAKER_02: Like we call it one detail framework, where basically you mention the source table in the destination table and you defy itself, whether it's a tafka source or high destination or thief and third and it will automatically figure out the complete date.
• 15:50
  SPEAKER_02: It will create the tables, it will create messaging data sets, and it will do all the things and it then it will run the quality test as well on top of it.
• 15:59
  SPEAKER_02: So it's like complete solution.
• 16:01
  SPEAKER_02: And there are more than like a couple of thousand pipelines are actually of that sort as well.
• 16:05
  SPEAKER_00: Yeah.
• 16:06
  SPEAKER_00: Cool.
• 16:07
  SPEAKER_00: And with your interface, are users able to actually manage their pipelines from there?
• 16:14
  SPEAKER_00: So that's they're never actually going to like the Airflow UI.
• 16:18
  SPEAKER_00: Once they've written them, they're able to see the run status and manage scheduling and all of that from the UI that you've built as well.
• 16:26
  SPEAKER_03: Yeah, so based on the APIs, like the whole UI was based on React with like we have the variables connections and everything that looks similar to Airflow, but like with the whole modern UI design.
• 16:36
  SPEAKER_03: Definitely, I think we have no version now is much advanced, but from where we find the UI that we have is very beautiful.
• 16:44
  SPEAKER_02: Yeah, if you compare to pre-air flow to two.
• 16:48
  SPEAKER_00: 1.6 might have been uh that was a lower bar for sure than the current one.
• 16:53
  SPEAKER_00: Yeah.
• 16:54
  SPEAKER_00: Yeah.
• 16:55
  SPEAKER_00: Cool.
• 16:55
  SPEAKER_00: So Shobit, you mentioned also about backfill functionality, and you touched on this a little bit, but this is something that is being expanded on quite a lot for Airflow 3, which should come out springtime ish of next year.
• 17:08
  SPEAKER_00: That's frequently, I would say, something that's on the top of the request list for the Airflow community, especially in the survey.
• 17:15
  SPEAKER_00: Lots and lots of companies need to be able to do backfills.
• 17:19
  SPEAKER_00: So, what is the solution that you guys have built?
• 17:21
  SPEAKER_03: Yeah, I think this is very similar to the pipeline generation over HTTP calls itself.
• 17:27
  SPEAKER_03: So we have a backfill API.
• 17:30
  SPEAKER_03: You can either call the API if you're solving presentated data service, or you can do it from the UI as well.
• 17:35
  SPEAKER_03: So the way backfills work at Uber is that you need to have a parent pipeline ID.
• 17:40
  SPEAKER_03: For which pipeline you need to backfill.
• 17:42
  SPEAKER_03: So you specify the parent pipeline ID and you specify the start date and end date.
• 17:46
  SPEAKER_03: Like if it's like result and date one year before, you specify that I want to backfill data from this time to this time.
• 17:52
  SPEAKER_03: And you can specify maybe a different code for the Anvil pipeline.
• 17:56
  SPEAKER_03: Similarly, like the HTTP request is BC the request, and then all the ones that seems table.
• 18:01
  SPEAKER_03: So the signalization driver that takes care of the backfill, it what it does is like it will check that this is a parent pipeline ID and it will just pick up the driver path of the fine location of the parent pipeline and copy the exact same data of the signalized format in the new pipeline, which is backfill pipeline.
• 18:18
  SPEAKER_03: So it's created as a new new pipeline.
• 18:21
  SPEAKER_03: And the only difference is the start date and end date is different for this backfill.
• 18:26
  SPEAKER_03: And so on the UI has a different pipeline, like SPI name as well.
• 18:31
  SPEAKER_03: So you think that okay, this is my pipeline run, and there is like pipeline running this for backfill that runs on older date.
• 18:38
  SPEAKER_03: And the way like we have some internal modified scheduler internally.
• 18:43
  SPEAKER_03: So we have this additional functionality of auto-backframe.
• 18:47
  SPEAKER_03: So it's like a parameter in the property that we set for the backfill pipeline.
• 18:51
  SPEAKER_03: So what happens when the scheduler finds out that this pipeline ID has this property set, it will not wait for the next date.
• 18:58
  SPEAKER_03: If the date is historical from the current date, it will just keep on scheduling until it catches up to the end date all you currently.
• 19:04
  SPEAKER_03: So that means if my pipeline is running LCS, it will continue to run from the expert to data after one finishes, the next one will start and so on.
• 19:13
  SPEAKER_03: So once you cover it, it will start and then complete at the ending and then it will get autoboiled.
• 19:19
  SPEAKER_03: So yeah, this is like something we get for back mail.
• 19:22
  SPEAKER_03: We still have some limitations, I think.
• 19:24
  SPEAKER_03: Like we attended the talk at Airflow.
• 19:27
  SPEAKER_03: And in fact, like a lot of our customers have also been asking, like, let's say if your pipeline is having like multiple tasks and you only want to backfill like by running a few tasks.
• 19:36
  SPEAKER_03: So we don't have a functionality like that.
• 19:37
  SPEAKER_03: Like it will run the complete pipeline from top to bottom.
• 19:41
  SPEAKER_03: And but I think AppNO is having this so looking forward on how Airflow is doing that.
• 19:46
  SPEAKER_02: Yeah, I think another limitation is that I think in Airflow, if you run a by it won't actually clean up the gaps in the same pipeline.
• 19:54
  SPEAKER_02: So if you go to the UI, you'll be able to see the boxes, like say which one left, right now, red or green, whatever.
• 19:59
  SPEAKER_02: But I think the one limitation we have it here is that we created new pipeline.
• 20:03
  SPEAKER_02: Yeah, so logically the gaps are fit up, but on the UI will look like you have to go to different pipelines and see some holes are filled in a different pipeline and the extra holes on the big pipeline.
• 20:14
  SPEAKER_02: But yeah, like even like other challenges, I think this one essentially suits us.
• 20:18
  SPEAKER_02: Yeah.
• 20:19
  SPEAKER_00: Yeah.
• 20:20
  SPEAKER_00: Interesting.
• 20:21
  SPEAKER_00: Do you I guess related to backfills?
• 20:24
  SPEAKER_00: This is also a question I have more broadly.
• 20:27
  SPEAKER_00: But given that you have identified some limitations and open source Airflow is working on adding additional functionality on this feature.
• 20:37
  SPEAKER_00: Do you ever evaluate with newer versions of Airflow whether you want to pull in functionality?
• 20:43
  SPEAKER_00: Is that possible in your setup?
• 20:45
  SPEAKER_00: Or have you completely diverged since when you initially made that decision?
• 20:51
  SPEAKER_03: Yeah, so I think we have completely diverged, but every year we do this evaluation about comparison of the feature sex in the latest version of Airflow.
• 20:58
  SPEAKER_03: What is the limitation of Squid and what are the extracts in there to do?
• 21:02
  SPEAKER_03: So we can talk more about like features that we are trying to bring in on like the feature man.
• 21:08
  SPEAKER_02: I think there are a lot of things.
• 21:09
  SPEAKER_02: For example, right now we are not executors, so we want to say more into Kubernetes.
• 21:13
  SPEAKER_02: So that's extra work for us.
• 21:14
  SPEAKER_02: Similarly, we are right now going to async operators.
• 21:17
  SPEAKER_02: So that is again uh work for us, right?
• 21:19
  SPEAKER_02: So we have evaluating, I think we are right now in the middle of the evaluation where we would like to go back to Airflow in some time.
• 21:26
  SPEAKER_02: And I think a lot of features which actually airflow 3 brings to the table, it makes us easy for to make a decision, actually, right?
• 21:33
  SPEAKER_02: A lot of features we have built in-house or are looking to build in-house.
• 21:37
  SPEAKER_02: I think those are coming in the approval 3 as well.
• 21:40
  SPEAKER_02: So we actually trying our best to say all back to AppNo.
• 21:44
  SPEAKER_02: And then whatever the customizations we had to done on top of, say, on Piper.
• 21:49
  SPEAKER_02: Right now, we may have to do it in Python.
• 21:51
  SPEAKER_02: And in that case, if we do it, we will do it directly to open source, right?
• 21:54
  SPEAKER_02: We'll not do it in-house anymore.
• 21:57
  unknown: Yeah.
• 21:57
  SPEAKER_02: For all the existing features as well as feature features as well.
• 22:00
  SPEAKER_03: Yeah.
• 22:00
  SPEAKER_03: Yeah, I think one of the main reasons like we got diverse is in the initial time I was not this good as it is currently.
• 22:07
  SPEAKER_03: And at that time we wanted to build and scale very fast.
• 22:11
  SPEAKER_03: So having this internal post gives us the flexibility to do constant leases and development and bring in the features for the users before going with Google autour where index and releases and all those things.
• 22:28
  SPEAKER_03: And looking forward to the L.
• 22:32
  SPEAKER_00: Yeah.
• 22:33
  SPEAKER_00: I think everybody is, myself included.
• 22:35
  SPEAKER_00: This is a really cool story.
• 22:37
  SPEAKER_00: I think it kind of perfectly highlights the beauty of open source, right?
• 22:41
  SPEAKER_00: And that's what it's designed to do is allow you to build on top of it, especially a tool like Airflow that's meant to be super flexible and extensible.
• 22:49
  SPEAKER_00: And then it's still there, so you can pull from it as the project evolves when you need to use it for inspiration.
• 22:57
  SPEAKER_03: Yeah.
• 22:59
  SPEAKER_00: Cool.
• 22:59
  SPEAKER_00: That speaking of Airflow 3, I know we just have a couple more minutes here.
• 23:04
  SPEAKER_00: And I always like to end with a standard final question.
• 23:06
  SPEAKER_00: So you don't this doesn't have to be about Airflow 3, but just in general, what would you most like to see from development of the Airflow project in the near future?
• 23:16
  SPEAKER_03: Yeah, so one of the features that I'm very excited about that's coming in is about the RAD versioning.
• 23:22
  SPEAKER_03: So I attended out there in the Air Force Summit.
• 23:25
  SPEAKER_03: So this is very similar to what we were planning to build in this half at Uber.
• 23:31
  SPEAKER_03: So basically, we need to provide a way for the users to do a to have a rollback trigger on the pipeline.
• 23:38
  SPEAKER_03: Because of some errors or any trigger condition, we want to roll back a pipeline to a previous version.
• 23:43
  SPEAKER_03: So for that, right now, the way it works is everything is deployed on top of the git repo, right?
• 23:48
  SPEAKER_03: Like we have a whole single bundle of the all the pipelines together, and it's not possible to revert single pipeline unless you make a git revert and land it on top.
• 23:57
  SPEAKER_03: There's no UI functionality, like you disconnect the version and it will automatically deploy it.
• 24:02
  SPEAKER_03: So this is very similar to what we have been trying, and I'm looking forward to this particular feature specifically.
• 24:10
  SPEAKER_02: Yeah, I think uh another some other another features we are really particularly interested on in the task is TK.
• 24:18
  SPEAKER_02: So I I couldn't attain the S task in the survey.
• 24:21
  SPEAKER_02: But I think the just what I know is that user will have not really necessary to write their task in Python and they can actually write it to different languages, and that's actually a big thing for us because Uber is mainly Java and Go top, basically, right?
• 24:36
  SPEAKER_02: So Python is was never never the first class candidate for CDC as Uber.
• 24:41
  SPEAKER_02: So it actually opens up a lot of possibilities to actually onboard new teams who are not willing to write into Python, right?
• 24:49
  SPEAKER_02: Or they the tasks are a little different than what a standard Python or FROPAS would look like.
• 24:55
  SPEAKER_02: So I think that is the one big feature we're looking at.
• 24:58
  SPEAKER_02: Another, I think, cool feature about that is not connecting to the database to update the state and all those things.
• 25:04
  SPEAKER_02: So that is also I think it's very really good security feature actually, and also protocol scalability ideas, actually, right?
• 25:11
  SPEAKER_02: Because we do not write into the MySQL instance and then MySQL post resistance every now and then.
• 25:18
  SPEAKER_02: So I think this is another big topic we are looking at.
• 25:22
  SPEAKER_02: Apart from that, like some standard thing like the new UI, new capabilities, like the drivers, support, and those are, of course, we are we interested in.
• 25:32
  SPEAKER_03: Yeah.
• 25:33
  SPEAKER_03: I believe what Stuber mentioned in relation to that about the task SDK.
• 25:36
  SPEAKER_03: Since the connections will all be moving to the web server instead of all executors creating a database connection, I believe it should also help with better scaling.
• 25:44
  SPEAKER_03: That will give you the concern that we have that employees won't be able to screen as our solution.
• 25:50
  SPEAKER_03: Uh hopefully, yeah, you can reevaluate and um maybe move on to airflow.
• 25:56
  SPEAKER_02: I I believe that there were some uh talks uh in the open source community about the security of connections as well itself, right?
• 26:03
  SPEAKER_02: I believe that in Airflow right now anybody can use anybody's connection.
• 26:07
  SPEAKER_02: So somehow we have actually guarded around that in Ruber, and I think that is another reason that we could not say migrate to Airflow so far, right?
• 26:15
  SPEAKER_02: Where because anybody can use anybody's connections.
• 26:17
  SPEAKER_02: And I think we see it as well, right?
• 26:20
  SPEAKER_02: So you see there are some things.
• 26:21
  SPEAKER_02: I don't know whether it's coming in airflow 3 or not, but in future version of Airflow, if we can have some things around it, like where the connections are actually tied up with the with the users or the take authors or things like that, I think it's a really, really good to have feature overall.
• 26:38
  SPEAKER_00: Yeah, that totally makes sense.
• 26:39
  SPEAKER_00: I think I'm not entirely sure that maybe part of the multi-tenancy AIP that allows multiple teams to work within the same Airflow instance.
• 26:49
  SPEAKER_00: So I I don't know if that's coming in Airflow 3 or not, or what that would entail.
• 26:53
  SPEAKER_00: I think everything else that you've touched on are the big exciting features within Airflow 3.
• 26:58
  SPEAKER_00: So certainly DAG versioning has been very, very widely requested and is highly anticipated, as well as the task SDK I'm really excited about as well.
• 27:08
  SPEAKER_00: That we did a podcast recently with Vikram Koka, who's been you know involved in the Airflow 3 planning.
• 27:13
  SPEAKER_00: He referred to it as the kind of run anywhere in any language principle.
• 27:18
  SPEAKER_00: So both that scalability aspect and being able to have workers in different clouds outside of Airflow, as and as you said, I'm pretty sure Go is one of the first languages they're implementing.
• 27:28
  SPEAKER_00: So hopefully that'll be useful for your team.
• 27:31
  SPEAKER_02: So yeah, this felt great.
• 27:35
  SPEAKER_00: Cool.
• 27:35
  SPEAKER_00: I know we are about out of time.
• 27:37
  SPEAKER_00: So thanks so much to both of you for coming on and sharing your story.
• 27:40
  SPEAKER_00: It's been really interesting.
• 27:42
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 27:47
  SPEAKER_02: I think via LinkedIn, I think yeah, we should be able to find out on LinkedIn.
• 27:52
  SPEAKER_02: And yeah, connect to us, message us.
• 27:54
  SPEAKER_02: We should be able to compliment.
• 27:55
  SPEAKER_03: I think we'll talk about this kind of things.
• 27:57
  SPEAKER_02: And if you have some if something similar is going on, we'll definitely be able to talk about it.
• 28:03
  SPEAKER_02: Yeah, I see our experience of building some customized solution on top of airflow.
• 28:07
  SPEAKER_00: Yeah.
• 28:08
  SPEAKER_00: Okay.
• 28:08
  SPEAKER_00: Great.
• 28:09
  SPEAKER_00: Well, that's a really valuable experience to share.
• 28:11
  SPEAKER_00: So I'm sure the community appreciates that a lot.
• 28:14
  SPEAKER_00: Thank you both again for coming on, and I hope you have a great rest of your day.
• 28:18
  SPEAKER_03: Yeah, yeah.
• 28:19
  SPEAKER_03: Thank you.
• 28:23
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 28:30
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 28:34
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.