Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma

Transcript

AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: So even though the same DAG is there, but due to different task concurrency, we were able to control at what time what should run.
• 0:07
  SPEAKER_01: And that's how, like, if a GPU hungry job is running at that time, we will run some other CPU jobs.
• 0:14
  SPEAKER_01: And that's where we also came across like one more optimization that we did.
• 0:19
  SPEAKER_01: Initially, we the way our HPC works is like you have to submit the job that goes into the queue.
• 0:25
  SPEAKER_01: It's like any other HPC or high-performance computing cluster, right?
• 0:29
  SPEAKER_01: You submit the job and then it goes to the queue with everyone else, and then it runs.
• 0:47
  SPEAKER_02: Hi everyone.
• 0:48
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:50
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Sergesh Sharma, who is a supervisor of ADAS MLOps at Ford Motorco.
• 0:58
  SPEAKER_02: Sergesh, how are you doing today?
• 1:00
  SPEAKER_01: I'm good.
• 1:01
  SPEAKER_01: Thanks for having me.
• 1:02
  SPEAKER_02: And yeah, it's great to have you on.
• 1:04
  SPEAKER_02: Have you been having a good summer?
• 1:06
  SPEAKER_01: Yeah.
• 1:08
  SPEAKER_02: Okay.
• 1:11
  SPEAKER_02: Yeah, yeah.
• 1:12
  SPEAKER_02: Just about.
• 1:12
  SPEAKER_02: I know a lot of the kids are going back to school and it feels like summer is winding down, which is always a little bittersweet, but it's nice to go back to a more normal schedule, I think, for a lot of folks.
• 1:25
  SPEAKER_02: Well, let's go ahead and dive in.
• 1:27
  SPEAKER_02: So for our audience for today's podcast, we're going to discuss how Ford uses Airflow to help power some of their autonomous driving systems and the scaling challenges and solutions that they've encountered when doing so.
• 1:39
  SPEAKER_02: Um so sure Desh, let's start by talking about your use case and what your team does.
• 1:44
  SPEAKER_02: I think everybody in our audience, I'm sure, is familiar with Ford, but they may not be familiar with the advanced driver assistance systems or ADAS, correct?
• 1:53
  SPEAKER_02: Specifically.
• 1:54
  SPEAKER_02: So can you talk a little bit about what those are?
• 1:56
  SPEAKER_01: Yeah, sure.
• 1:57
  SPEAKER_01: So generally, when you think about it as you think about like autonomous vehicle, level five no driver.
• 2:03
  SPEAKER_01: But in companies like Ford, GM and all this OEM, ADAS also means like advanced driver assistance system, which consists of a pre-collision assist or blind spot, and even self-driving feature, like which is called as Blue Groose for us.
• 2:21
  SPEAKER_01: And if you see ADAS has actually five features, five level from zero to five.
• 2:27
  SPEAKER_01: Zero means no automation, and five means full automation.
• 2:31
  SPEAKER_01: Right now we are at level two, and many companies are trying to launch my team.
• 2:36
  SPEAKER_01: It's a part of a larger umbrella of DataOps team.
• 2:39
  SPEAKER_01: And we work on creating data pipeline and MLOps pipeline, apart from some of the machine learning stuff that is used for validating this feature and also helping finding out if there is some quality issue or some drive purpose or new product development.
• 2:57
  SPEAKER_02: Yeah, got it.
• 2:58
  SPEAKER_02: That makes sense.
• 2:58
  SPEAKER_02: And so to summarize, that's I think a lot of features that are part of modern driving.
• 3:05
  SPEAKER_02: So people might not always realize that they're even using them, that they require a lot of data.
• 3:10
  SPEAKER_02: So you touched on this a little bit already, but maybe dig in a little bit more to how data powers these systems and how your team works on that more generally.
• 3:20
  SPEAKER_01: Yeah.
• 3:20
  SPEAKER_01: So right now I have seen some posts from LinkedIn and they were saying the most engineering marvel for anyone is like Ubehikle, because there are so many systems which come together, right?
• 3:32
  SPEAKER_01: And all these companies, like big companies like us, we are known as OEM, and we buy a lot of ECUs or components from other systems, from other vendors, this kind of big companies, right?
• 3:44
  SPEAKER_01: Chip producer and also software producer, there are many in the market.
• 3:50
  SPEAKER_01: Now, what happens when we buy this system?
• 3:52
  SPEAKER_01: We need to ensure it integrates well.
• 3:55
  SPEAKER_01: So everything that is working there, it requires some sensor fusion.
• 3:59
  SPEAKER_01: We need to check this sensor.
• 4:00
  SPEAKER_01: Apart from that, there is vehicle safety.
• 4:03
  SPEAKER_01: So to start with, how data play in vehicle safety, right?
• 4:06
  SPEAKER_01: So before launching any vehicle, we have to drive our vehicle for around a certain amount of mile, which can vary.
• 4:14
  SPEAKER_01: And we need to re-simulate and see like all the sensors are working properly, it is properly calibrated, we are getting the correct output, we are consuming the correct input, and it is performing as well.
• 4:26
  SPEAKER_01: It is expected to do.
• 4:27
  SPEAKER_01: Also, the safety rating for safety rating, we need to do that.
• 4:30
  SPEAKER_01: So, my team, what it does is to process all these logs.
• 4:34
  SPEAKER_01: And just to give you a little glimpse into our infrastructure, we have a huge on-prem as well as cloud presence.
• 4:40
  SPEAKER_01: And that's why our processing can be on-prem and can could be on cloud.
• 4:45
  SPEAKER_01: Apart from that, now more and more development is like becoming more like a software development.
• 4:51
  SPEAKER_01: And when it is becoming a software development, you start getting like the customer data, you get into the privacy part of it.
• 4:57
  SPEAKER_01: And that's where it mirrors very much close to what you do in like insurance or financial domain.
• 5:04
  SPEAKER_01: And then you need to start collecting that data using all that condition.
• 5:08
  SPEAKER_01: Apart from that, the biggest challenge that comes here is that it's not all structured data because it's a dash, there is a vision component to it.
• 5:18
  SPEAKER_01: So it's a lot of images and video processing, apart from log processing from different sensors, and it goes into a realm of a bit of robotics, and that's where a lot of data processing happens.
• 5:31
  SPEAKER_02: Yeah, that's interesting.
• 5:32
  SPEAKER_02: That sounds really complicated.
• 5:34
  SPEAKER_02: I know, especially anytime you have to deal with spatial data like that.
• 5:37
  SPEAKER_02: It adds an extra layer of complexity.
• 5:40
  SPEAKER_02: Let's talk a little bit about Airflow.
• 5:42
  SPEAKER_02: So, Airflow is one of the tools that you use as part of your solution for managing all of this data.
• 5:47
  SPEAKER_02: Tell me about how that works on your team and how did you originally choose Airflow and how has your usage of it grown over time?
• 5:55
  SPEAKER_01: We were doing a lot of on-prem development.
• 5:57
  SPEAKER_01: And all that on-prem development was used through schedule through con or some of the scheduler that we have, native scheduler.
• 6:06
  SPEAKER_01: But with the start, we start looking into something which can help us schedule it on our HPC cluster.
• 6:14
  SPEAKER_01: And that's where we came across Airflow.
• 6:16
  SPEAKER_01: So we started with a small deployment and now we have scaled it to like multiple deployments.
• 6:21
  SPEAKER_01: And we are using it to run our workload not only on-prem, but we need something which helps us to run the same processes on GCP.
• 6:30
  SPEAKER_01: And what we found there, like Airflow, if we use either the open source version or the managed instance like Astronomer, we can get a very clear view of what is running where, and we can support all the data pipeline through one view.
• 6:47
  SPEAKER_01: Apart from that, we also found out that it is easier to repeat some processes using Airflow, whereas initially it was like everyone was writing some con job or something, and scheduling was not that clear.
• 7:00
  SPEAKER_01: So this helped us to make use of computation better, and we can chain our processes properly.
• 7:06
  SPEAKER_01: Chain in the sense like there is some pre-processing initially, everyone was running that.
• 7:11
  SPEAKER_01: Now, having a centralized airflow version, we can run it once and make it available for the downstream team to use it for their use cases.
• 7:20
  SPEAKER_02: Yeah, got it.
• 7:21
  SPEAKER_02: That makes sense.
• 7:22
  SPEAKER_02: So moving off of cron a couple of years ago, I mean, you certainly have a lot more visibility with something like Airflow, but as you mentioned, also a lot more reusability when you're working within like a code structure that people can build off of and pull things in from standardized functions and stuff as opposed to just cron.
• 7:39
  SPEAKER_02: I think that makes a ton of sense.
• 7:41
  SPEAKER_02: You mentioned that you started with just a single airflow instance, kind of trialing it out, which I think is where a lot of teams start, but now you have multiple airflow instances.
• 7:50
  SPEAKER_02: I'm sure you're dealing with data on a pretty massive scale.
• 7:54
  SPEAKER_02: Kind of how have you found that experience of scaling Airflow out to meet your needs as you've rolled it out more thoroughly within your team?
• 8:03
  SPEAKER_01: The good thing about uh Airflow, it's quite modular, right?
• 8:06
  SPEAKER_01: And you can start breaking it down.
• 8:08
  SPEAKER_01: You can start with one container and then you can take the web server out, you can take the scheduler out, then you can use your own Cloud SQL.
• 8:16
  SPEAKER_01: So that way we started with one container, and then like on the open source version, we start uh federating it.
• 8:23
  SPEAKER_01: Like we went with Cloud SQL, then we had our own web server, and that way we are able to scale it.
• 8:29
  SPEAKER_01: Just to give you some idea about uh the scale we are uh dealing with, right?
• 8:34
  SPEAKER_01: So we drive around uh 1500 miles per two weeks, and one mile is around like five to six GB of data.
• 8:44
  SPEAKER_02: Oh my gosh.
• 8:45
  SPEAKER_01: And currently we are processing around one petabyte of data per week.
• 8:51
  SPEAKER_01: And now this one petabyte data is not very simple processing, right?
• 8:55
  SPEAKER_01: You just read it at some component and you are done.
• 8:59
  SPEAKER_01: It requires a lot of CPU, in some cases GPU, and that's where like some of the distributed computing comes in, which is like Ray or something.
• 9:07
  SPEAKER_01: But again, this is not one log, these are multiple logs.
• 9:11
  SPEAKER_01: So once we write one Airflow component, right now we are processing like 300 logs in parallel, which is like running 300 instances of same DAG in parallel.
• 9:21
  SPEAKER_01: So we are hitting the limit, and that's where we went ahead.
• 9:24
  SPEAKER_01: Like we removed the web server, we removed this scheduler.
• 9:28
  SPEAKER_01: When I say remove, isolate and make it as a standalone component, created the Cloud SQL so that our database can provide better scalability.
• 9:38
  SPEAKER_01: Apart from that, we have used the one thing that we have realized is like having a good observability tool.
• 9:44
  SPEAKER_01: And we had our like what is the CPU usage and everything connected through our Grafana dashboard.
• 9:51
  SPEAKER_01: And that's where we were able to find out the right size of what should be our airflow footprint, and that's how we are going to we are able to iterate.
• 10:00
  SPEAKER_01: Apart from that, once we got into this kind of setup, then we realized that we also need to have some software engineering best practices.
• 10:09
  SPEAKER_01: So there also we went to like Astronomer, which provided us a little more flexibility there.
• 10:15
  SPEAKER_01: So the generally our production workload, we run it on Astronomer and similar processes we are using for our dev workflow, which is on our HPC server, which will process most of our logs.
• 10:28
  SPEAKER_02: Got it.
• 10:29
  SPEAKER_02: That makes sense.
• 10:30
  SPEAKER_02: And so when you say best practices, I want to dig in on that a little bit more because I think this is a topic that we see is super, super popular.
• 10:37
  SPEAKER_02: I think everybody is always wondering how to use Airflow in the best way given how flexible it is.
• 10:43
  SPEAKER_02: So when you say best practices for like your data engineers, is that for writing DAGs in a way that's going to allow Airflow to scale in the way that you need it to?
• 10:54
  SPEAKER_02: Or is that kind of outside of the scalability question?
• 10:58
  SPEAKER_01: It's both.
• 11:00
  SPEAKER_01: One, you need to start writing good code, which is easily readable and it scales well.
• 11:06
  SPEAKER_01: Like we came to know about the deferred operator, for example, right?
• 11:09
  SPEAKER_01: We had long-running processes and it was causing a lot of issues with our pod.
• 11:14
  SPEAKER_01: So we start using deferred operator.
• 11:16
  SPEAKER_01: So it only comes in like when you run the pod, and it doesn't continuously have the pod there.
• 11:22
  SPEAKER_01: The other part is like this is our team have mechanical engineers, embedded engineers, and all kinds of talent.
• 11:29
  SPEAKER_01: They might not be the data engineer.
• 11:32
  SPEAKER_01: And now, when they write sometime, there is a possibility it can blow up.
• 11:37
  SPEAKER_01: And that's where having a different instance is and ensuring they have a seamless onboarding process.
• 11:43
  SPEAKER_01: Apart from that, there is some nuances when you move from open source to astronomer.
• 11:48
  SPEAKER_01: And what are the best practices to change that so you can run it seamlessly?
• 11:53
  SPEAKER_01: Then how you manage, like for a team, do you have an instance or you should have one instance, centralized instance for everyone?
• 12:01
  SPEAKER_01: So those are the some of the best practices that we try.
• 12:04
  SPEAKER_01: So whenever a new team is starting, we give them their own instance so that they can play with that and also deploy their DAX as they want.
• 12:12
  SPEAKER_01: And once they are ready, we bring them into when they have some reusable component and they are ready to use Astronomer in a or Airflow in a more mature way.
• 12:20
  SPEAKER_01: We bring them into a centralized location so that that code can be reused by everyone.
• 12:26
  SPEAKER_01: Apart from that, for some of the DAX, we are also bringing in unit test cases.
• 12:31
  SPEAKER_01: So when they merge for the centralized, at least for centralized, we use uh unit test cases.
• 12:36
  SPEAKER_01: So whenever they are merging, we run that and we ensure that it is not corrupting our whole build.
• 12:43
  SPEAKER_01: So it's in a way like bringing it more closer to software engineering practices and trying to create a repo which is easily explorable, and you can find out all the use cases that you need.
• 12:58
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 13:07
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 13:19
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:28
  SPEAKER_02: As you've gone through this journey from building airflow out from like a proof of concept internally to what is it sounds like, you know, a pretty standardized tool in terms of how you're using it internally.
• 13:42
  SPEAKER_02: You've got a lot of infrastructure built around it to protect it and make sure that it's useful for your teams and also is going to support all of the data that you need to have on it.
• 13:51
  SPEAKER_02: Have there been any other interesting challenges with that or kind of solutions that you've implemented that you want to share?
• 13:58
  SPEAKER_01: Our onboarding is really cool, like where we are like asking them to use GitHub issues.
• 14:04
  SPEAKER_01: And using that, we create a kind of an Azure AD group and we are able to create kind of a astronomer airflow instance for a team.
• 14:15
  SPEAKER_01: That was one.
• 14:17
  SPEAKER_02: So you have when you have a new person start, you have them submit a GitHub issue, and that triggers something to automatically provision them with the infrastructure that they need to start working with Airflow?
• 14:29
  SPEAKER_01: Yes.
• 14:30
  SPEAKER_01: That's what we are trying.
• 14:32
  SPEAKER_01: We have like where we add user to the FIM group, and that way they can get onboarded.
• 14:38
  SPEAKER_02: Interesting.
• 14:39
  SPEAKER_02: What made you choose GitHub issues for that?
• 14:42
  SPEAKER_01: With GitHub issue, the good thing is it's visible to everyone.
• 14:45
  SPEAKER_01: You can use Zera ticket, but with Zera ticket, the thing is it goes off once your sprint closes.
• 14:51
  SPEAKER_01: It's not searchable.
• 14:52
  SPEAKER_01: With GitHub issue, the thing is it keeps uh you can easily use it to use it for all the requests.
• 14:59
  SPEAKER_01: And it's always there.
• 15:00
  SPEAKER_01: You can search it anytime, right?
• 15:02
  SPEAKER_01: Apart from that, it comes with GitHub action, which is really easy to you can write in any language the validation script, and you can validate the input really easily.
• 15:12
  SPEAKER_01: And it's easy to integrate with CICD compared to any other tools that we have.
• 15:18
  SPEAKER_01: And at Ford, right now we are like very big on using GitHub discussion, GitHub issues, okay, and GitHub around tooling.
• 15:26
  SPEAKER_01: So that's why that is another best practices which I might not have touched on.
• 15:30
  SPEAKER_01: Like generally, any tool we adopt, we try to integrate with the existing enterprise-level tools.
• 15:36
  SPEAKER_01: And that's where like GitHub Cloud was one of our tools, and that's why we are using that.
• 15:41
  SPEAKER_02: Yeah, that makes sense.
• 15:42
  SPEAKER_02: I'm sure that helps with the silos internally for a company that's as large as Ford is, right?
• 15:48
  SPEAKER_01: Yes.
• 15:48
  SPEAKER_01: And it's really easy then to like search for it.
• 15:51
  SPEAKER_01: If somebody has done that, you search on GitHub and you can find that.
• 15:55
  SPEAKER_01: The other interesting part that we on the challenges we faced was running a single deck for 300 times, a concurrent run.
• 16:03
  SPEAKER_01: And that's where my team came to know about the different parameters that are there.
• 16:07
  SPEAKER_01: Sometimes, like you have a concurrent deck run and you have a task run.
• 16:10
  SPEAKER_01: And there could be some kind of a contradiction between the two and how to find the correct value so that you can run that.
• 16:16
  SPEAKER_01: Apart from that, you also need to see you are not overwhelming the system because when you start doing it, you have too many.
• 16:22
  SPEAKER_01: And if your system is not able to run that many, your queue will get saturated.
• 16:27
  SPEAKER_01: And there can be an instance where the whole airflow can come down.
• 16:32
  SPEAKER_01: So that journey was really great where it took us two to three months to find out what should be the correct parameter for each of airflow native parameters so that we can use it for running at that massive scale.
• 16:44
  SPEAKER_02: Yeah, that makes sense.
• 16:45
  SPEAKER_02: 300 concurrent runs of the same DAG is pretty high, I would think, especially I assume that DAG has quite a few tasks in it as well.
• 16:53
  SPEAKER_02: That's interesting.
• 16:54
  SPEAKER_02: Did you go through kind of the process of designing the DAGs themselves to support that level of concurrency?
• 17:02
  SPEAKER_02: So obviously, within Airflow, there's a lot of parameters that you can tune for, as you said, like how many tasks or how many DAGs you can run at the same time to make sure that Airflow is set up in a way that supports what you need it.
• 17:15
  SPEAKER_02: But I think we also see a lot of people go through the process of trying to design whether or not they're creating the same pipeline that is running that many times versus like multiple separate pipelines that are running at the same time.
• 17:31
  SPEAKER_02: Or did you guys go through that process of just design work for how to architect it?
• 17:37
  SPEAKER_01: Yes, because this the pipeline that I am talking about, it has CPU plus GPU usages.
• 17:43
  SPEAKER_01: So we try to schedule it in a way that if stage one is running, you can kick off stage two.
• 17:51
  SPEAKER_01: So there is a proper pipeline that can run.
• 17:53
  SPEAKER_01: So even though the same DAG is there, but due to different task concurrency, we were able to control at what time what should run.
• 18:01
  SPEAKER_01: And that's how, like, if a GPU hungry job is running at that time, we will run some other CPU jobs.
• 18:08
  SPEAKER_01: And that's where we also came across like one more optimization that we did.
• 18:13
  SPEAKER_01: Initially, we the way our HPC works is like you have to submit the job that goes into the queue.
• 18:19
  SPEAKER_01: It's like any other uh HPC or high performance computing cluster, right?
• 18:24
  SPEAKER_01: You submit the job and then it goes to the queue with everyone else, and then it runs.
• 18:29
  SPEAKER_01: That is for GPUs.
• 18:30
  SPEAKER_01: Now, what we figured out there, Airflow has Kubernetes pod operator.
• 18:34
  SPEAKER_01: So, why we should submit it to the queue?
• 18:36
  SPEAKER_01: We will try to run all the CPU jobs on the Kubernetes itself instead of sending it to the HPC cluster.
• 18:44
  SPEAKER_01: So those are some of the optimization we learned on like over our one-year journey.
• 18:50
  SPEAKER_02: Yeah, yeah, that's really interesting.
• 18:52
  SPEAKER_02: That's really cool to hear about.
• 18:54
  SPEAKER_02: Thanks for taking everybody through that.
• 18:56
  SPEAKER_02: You said before, I think your use case is fascinating just given how big it is.
• 19:12
  SPEAKER_01: Yeah, we had a discussion previously, like we ran into the issue and we have the astronomer also at our side.
• 19:18
  SPEAKER_01: And when we tell, when we discussed about the scale, they were really surprised like what we are able to do with the like open source and what we are trying to do with the astronomer.
• 19:28
  SPEAKER_01: Definitely it's one of the biggest clusters, uh, one of the biggest installation we are doing for a use case, not for everything, but for a use case, like one petabyte of data processing every week.
• 19:40
  SPEAKER_01: It's huge.
• 19:42
  SPEAKER_02: Yeah, that's definitely a lot.
• 19:43
  SPEAKER_02: I'm also, I think, pretty impressed just hearing what you were able to do with the open source project.
• 19:49
  SPEAKER_02: It's great that it's so flexible in that way and customizable.
• 19:52
  SPEAKER_02: But yeah, you guys definitely put it through the stress testing there.
• 19:56
  SPEAKER_02: Yeah, well, that's really cool.
• 19:57
  SPEAKER_02: Sergish, I always like to end with a standard final question.
• 20:01
  SPEAKER_02: And that is what would you most like to see from development of the open source airflow project in the near future?
• 20:08
  SPEAKER_01: Well, I have several wish lists.
• 20:10
  SPEAKER_02: Okay.
• 20:11
  SPEAKER_01: And it touches a lot many areas, right?
• 20:14
  SPEAKER_01: The first one, which is like from the community, I guess, is for streaming.
• 20:20
  SPEAKER_01: Okay.
• 20:20
  SPEAKER_01: Start supporting streaming.
• 20:22
  SPEAKER_01: That's the first one.
• 20:22
  SPEAKER_02: That's a big ask.
• 20:24
  SPEAKER_01: Yeah.
• 20:25
  SPEAKER_01: Everyone wants that, right?
• 20:27
  SPEAKER_01: Because just think about it.
• 20:28
  SPEAKER_01: We are right now working on a data collection platform.
• 20:32
  SPEAKER_01: And as soon as there is some event and we are able to process and create some alert, it will really help us to increase the quality and see how our tools are performing.
• 20:43
  SPEAKER_01: The second one I would like is something, see more use of Airflow registry or like astronomer registry that you have.
• 20:51
  SPEAKER_01: Similarly, you have Docker registry, you have everything, right?
• 20:54
  SPEAKER_01: Why can't companies start having those kinds of registry, which becomes really easy for them to share open source thing?
• 21:01
  SPEAKER_01: And having that kind of development will also make uh life easy.
• 21:05
  SPEAKER_01: Like you already have created it, you have scaled it.
• 21:09
  SPEAKER_01: Now just go and use it at the enterprise level.
• 21:12
  SPEAKER_01: So if somebody has done that once, you don't need to worry about then.
• 21:15
  SPEAKER_01: You only need to worry about like the edge cases or the exceptional scenario.
• 21:21
  SPEAKER_02: One follow-up given so the astronomer registry as it currently exists, have you seen that?
• 21:26
  SPEAKER_02: Were you referring to like an expansion of the existing astronomer registry or something separate?
• 21:33
  SPEAKER_01: I am looking more into like you have Pi Fi, right, for Python, where companies can create their own artifactory.
• 21:40
  SPEAKER_01: So if we are able to do that as an extension where company can have their own private astronomer registry or your flow registry, that will help everyone.
• 21:51
  SPEAKER_01: Okay.
• 21:52
  SPEAKER_01: Because this is something it has a lot of good code, but when anyone starts, they start writing their own, right?
• 21:59
  SPEAKER_01: Whereas I think They should start from there and then try to write the custom code because somebody has already done a lot of hard work.
• 22:05
  SPEAKER_01: It is like from the community.
• 22:08
  SPEAKER_01: And also each company have their own system and would they interact differently.
• 22:13
  SPEAKER_01: So that will help us to scale.
• 22:15
  SPEAKER_01: That is my second wish list.
• 22:18
  SPEAKER_02: Got it.
• 22:19
  SPEAKER_02: That makes sense.
• 22:20
  SPEAKER_02: So the existing astronomer registry is more about the building blocks of airflow.
• 22:26
  SPEAKER_02: So it has some extra information on using operators and things like that.
• 22:29
  SPEAKER_02: There is an option for community members to contribute example pipelines, but it doesn't go as far as providing a private option for companies to work from internally.
• 22:40
  SPEAKER_02: Yeah, that's certainly interesting feedback.
• 22:42
  SPEAKER_02: I uh my team at Astronomer does a lot of work on the registry.
• 22:45
  SPEAKER_02: So we like to hear feedback like that.
• 22:47
  SPEAKER_02: That's useful.
• 22:48
  SPEAKER_02: You mentioned you had one more.
• 22:50
  SPEAKER_01: I got one more, but I just realized I have other one also.
• 22:54
  SPEAKER_01: Oh, sure.
• 22:55
  SPEAKER_01: Second last one would be like adding more LLMs development.
• 22:59
  SPEAKER_01: You already have one astro, which is for at least for airflow kind of thing, is much better than generic one because it points to the knowledge base that we need.
• 23:09
  SPEAKER_01: And that was one of the reasons like we were able to do a lot many things because it gave us a pointed answer whenever we asked the question.
• 23:16
  SPEAKER_01: And it helped my team to scale up on airflow development.
• 23:20
  SPEAKER_01: So we would like to see can it be enhanced to increase the developer experience?
• 23:26
  SPEAKER_01: Can it be used?
• 23:27
  SPEAKER_01: Can you have some kind of an agent return which could help us to write some kind of foot DAX?
• 23:32
  SPEAKER_01: That is one which like seeing more integration of LLMs.
• 23:37
  SPEAKER_01: Not to execute, but to more from the developer experience, right?
• 23:41
  SPEAKER_01: We already are using astronomer for LLM ops and other things.
• 23:45
  SPEAKER_01: But what about the developer experience?
• 23:47
  SPEAKER_01: Like you have GitHub Copilot chat, right?
• 23:49
  SPEAKER_01: Similarly, like something for astronomer, especially or airflow that can be helpful be helpful.
• 23:58
  SPEAKER_01: And my last one is like DAG Factory.
• 24:02
  SPEAKER_01: I would like to see it more mature because we are seeing it and DAG Factory, I see DAC Factory and the registry that we talk about, right?
• 24:11
  SPEAKER_01: They can work side by side.
• 24:13
  SPEAKER_01: If you have like an operator and you have a kind of uh way where you can write a YML to uh stitch together your use case, that can help in writing DAC faster.
• 24:25
  SPEAKER_01: And the other use case I had for we want to do A-B testing.
• 24:29
  SPEAKER_01: In our side, we do a lot of uh algorithm.
• 24:31
  SPEAKER_01: My end user does a lot of algorithm development where they want to run the same kind of a DAG with different algorithms.
• 24:38
  SPEAKER_01: So if I have YML-based things, I can write like with the different Docker image or something, and it can be scheduled, and I can run it for two days, and I my end user can just see the result after that.
• 24:49
  SPEAKER_01: Hey, this is which uh algorithm is performing better instead of writing that, scheduling that, and doing all this thing.
• 24:56
  SPEAKER_01: So I would like to see like DAG factory to enhance from development to deployment and have a little more integration with astronomer registry.
• 25:06
  SPEAKER_02: Okay, got it.
• 25:07
  SPEAKER_02: That's very helpful.
• 25:08
  SPEAKER_02: Yeah, so the DAG factory, the open source package astronomer recently took over.
• 25:13
  SPEAKER_02: And I do know that it's on our we've got eyeballs on it in terms of feature development.
• 25:19
  SPEAKER_02: That's a really common request from the community.
• 25:21
  SPEAKER_02: So it's great to hear that reinforced from a team like yours.
• 25:25
  SPEAKER_02: That's a great, great list of wishes.
• 25:27
  SPEAKER_02: Some big stuff for Airflow, but I think a lot of it really will be tackled in the future, especially with Airflow 3.
• 25:32
  SPEAKER_02: Yeah, that's great.
• 25:33
  SPEAKER_02: Sergesh, it's been great to have you on.
• 25:35
  SPEAKER_02: Just to close us out, what's the best way for folks to get in touch with you?
• 25:39
  SPEAKER_01: Best way is LinkedIn.
• 25:40
  SPEAKER_02: Okay.
• 25:41
  SPEAKER_01: I'm active there, so uh you can send friend requests or add me.
• 25:46
  SPEAKER_02: Okay.
• 25:46
  SPEAKER_02: Perfect.
• 25:47
  SPEAKER_02: Thanks so much.
• 25:48
  SPEAKER_02: It's been great having you on.
• 25:49
  SPEAKER_01: Yeah, we are.
• 25:50
  SPEAKER_00: Thank you.
• 25:52
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 25:59
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 26:03
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.