Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall

Transcript

Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Spark uses DAG as well, data and acrylic graphs as well.
• 0:03
  SPEAKER_02: So I have been working on Spark, Apache Spark, like for a decade.
• 0:07
  SPEAKER_02: So I saw the DAG.
• 0:09
  SPEAKER_02: Oh, this is a familiar term.
• 0:10
  SPEAKER_02: Okay.
• 0:11
  SPEAKER_02: I know how DAG works.
• 0:13
  SPEAKER_02: So it was like a common term used.
• 0:16
  SPEAKER_02: And it was like, oh, I know this one.
• 0:18
  SPEAKER_02: And Python is cool.
• 0:20
  SPEAKER_02: Operators, I think Airflow is much more organized and much more rich in terms of off-the-shelf operators and etc.
• 0:28
  SPEAKER_02: All of that.
• 0:29
  SPEAKER_02: It was great.
• 0:30
  SPEAKER_02: It took one a year, but when I said in-house tolls, it was all about scaling up.
• 0:40
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:50
  SPEAKER_03: Airflow users, we have exciting news.
• 0:54
  SPEAKER_03: The annual Apache Airflow survey is live.
• 0:57
  SPEAKER_03: In just seven minutes, you can help shape the future of Airflow.
• 1:00
  SPEAKER_03: And as a thank you for filling it out, you'll get a free Airflow Fundamentals or DAG Authoring certification.
• 1:06
  SPEAKER_03: Those are worth$150 each.
• 1:09
  SPEAKER_03: Plus, you'll also be entered to win an exclusive workshop with Mark Lumberti.
• 1:13
  SPEAKER_03: So make sure you don't miss out.
• 1:15
  SPEAKER_03: The survey link is in the show notes and it closes December 20th.
• 1:20
  SPEAKER_01: Hello everyone.
• 1:21
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:23
  SPEAKER_01: I'm your host, Mark Lomberti, and today I'm joined by Vasanta Kosuri, Data MLOps engineer at Ford Motor Company.
• 1:31
  SPEAKER_01: How are you doing?
• 1:33
  SPEAKER_02: I'm doing pretty good, great, Mark.
• 1:35
  SPEAKER_02: It's an honor to talk to you.
• 1:37
  SPEAKER_02: I might have mentioned it last time when we spoke for the very first time.
• 1:41
  SPEAKER_02: Huge fan of your LinkedIn posts.
• 1:44
  SPEAKER_02: Have been following, talking to you in person, face to face, is a dream come true.
• 1:49
  SPEAKER_02: Thank you so much for having me here today.
• 1:51
  SPEAKER_01: Thank you so much.
• 1:52
  SPEAKER_01: And I'm glad that we're gonna that I'm gonna meet you during the Airflow Summit, uh, which is coming very soon.
• 1:57
  SPEAKER_01: Yes.
• 1:58
  SPEAKER_02: I am so excited to meet you in person at San Francisco Summit this year.
• 2:02
  SPEAKER_02: At the same time, pretty excited to meet all of your team at Astronomer.
• 2:07
  SPEAKER_01: Awesome.
• 2:08
  SPEAKER_01: Okay.
• 2:09
  SPEAKER_01: Before starting, I would just would like to give the primary takeaways for the people that are listening to this episode.
• 2:15
  SPEAKER_01: So basically, at the end of this episode, you will have a better idea of what Ford Motor Company does with Airflow.
• 2:22
  SPEAKER_01: You will see different use cases and more a little bit about the like the technical details, how things work at Ford Motor Company with Airflow, and we will maybe dive more into the details of some questions.
• 2:36
  SPEAKER_01: Without further ado, let's get started.
• 2:39
  SPEAKER_01: To give everyone a lay of the land, can you start by talking a little bit about your role at Ford Motor Company?
• 2:46
  SPEAKER_02: Yeah, for sure.
• 2:48
  SPEAKER_02: So I have been with Ford for the last seven and a half years, joined uh GDIA, which is Global Data Insights and Analytics Department, 2016 uh fall.
• 3:00
  SPEAKER_02: So I worked in investment banking for seven years before moving to Michigan to join Ford Motor Company in this data department.
• 3:09
  SPEAKER_02: So I joined as a big data engineer, specialized in Spark development.
• 3:14
  SPEAKER_02: At the time, it was like one year into data lake concept and where all the Ford motor company data sources started landing into Hadoop Data Lake.
• 3:26
  SPEAKER_02: So I worked on a in-house data quality and data discovering in-house too.
• 3:34
  SPEAKER_02: That's how I started my journey at Ford.
• 3:36
  SPEAKER_02: And last year, fall, I joined EDAS team.
• 3:41
  SPEAKER_02: ADAS is advanced driving Assist systems team.
• 3:44
  SPEAKER_02: That's where I am right now.
• 3:46
  SPEAKER_02: Now I got more responsibilities.
• 3:48
  SPEAKER_02: I have been data operations uh for almost a decade now.
• 3:53
  SPEAKER_02: Now I got into machine learning operations as well.
• 3:57
  SPEAKER_02: So started wearing a new hat of machine learning operations here.
• 4:01
  SPEAKER_02: Yep, thanks.
• 4:02
  SPEAKER_02: I mentioned Hadoop Data League, so you could ask like why there is GCP.
• 4:07
  SPEAKER_02: So everybody started going into cloud probably a decade ago slowly, but Ford started moving all the data into cloud, which is Google Cloud Platform, three years ago.
• 4:20
  SPEAKER_02: That's when I started migrating the two in-house tools from Hadoop, Spark environment to GCP, Dataproc, etc.
• 4:28
  SPEAKER_02: And also in ed as Teams, we are using GCP as our cloud environment very heavily.
• 4:36
  SPEAKER_02: I'm I first responsibility in my role is I help teams to get onboarded onto GCP projects.
• 4:44
  SPEAKER_02: So here, yeah, mostly like GCP infrastructure.
• 4:48
  SPEAKER_02: You need to know Terraform to create or destroy any infrastructure into GCP.
• 4:53
  SPEAKER_02: So I did a GCP certification, it was like data engineering certification.
• 4:58
  SPEAKER_02: It was like last year, yes.
• 5:00
  SPEAKER_02: So it changed my path a little bit.
• 5:02
  SPEAKER_02: It has been exciting since then.
• 5:04
  SPEAKER_02: So yeah, I helped teams to get their infrastructure into GCP project, also onboarding, access permissions, as well as working with teams on various services from GCP, like BigQuery, Dataflow, Data Proc, Cloud Run, etc.
• 5:22
  SPEAKER_02: It's and also we closely, Sadish and I very closely work with Google team as well.
• 5:28
  SPEAKER_02: So they are amazing guests on our team.
• 5:30
  SPEAKER_02: So whenever we see a new tool, or for instance, we saw a new LLM tool that was like a due at AI, oh, we want to onvoard onto LLM.
• 5:39
  SPEAKER_02: So we are like technology enthusiasts at the same time, like AIML enthusiasts.
• 5:46
  SPEAKER_02: So yeah, basically onboarding new tools or new services, that was that's where I was going.
• 5:51
  SPEAKER_02: And then so Airflow is a Airflow plays a bigger role for orchestrating any pipelines across edas teams.
• 6:01
  SPEAKER_02: So my journey started with Airflow two years ago when we migrated on-prem data leak jobs to GCP.
• 6:11
  SPEAKER_02: So there was a yeah, go ahead, please.
• 6:13
  SPEAKER_01: Yeah, so actually I have a question before running forward.
• 6:16
  SPEAKER_01: Did you like what were what were you using in the bank for the orchestration?
• 6:21
  SPEAKER_01: Did you have an orchestrator at the time or was it like all manual?
• 6:26
  SPEAKER_02: Oh, so that's a good question.
• 6:28
  SPEAKER_02: Now you're bringing me a decade back into time, but yeah, so autosys, I worked uh very heavy on ETL.
• 6:36
  SPEAKER_02: Also, we I worked on Informatica, which was like ETL.
• 6:41
  SPEAKER_02: So it was like in the off-the-shelf ETL tools, like scheduler tools like autosys and informatica scheduler, etc.
• 6:49
  SPEAKER_02: Also, file system was a huge thing, so we used cron jabs as well there.
• 6:55
  SPEAKER_02: But like in from in Informatica, you could just schedule the ETL uh workflow.
• 6:59
  SPEAKER_02: Also, we used Autosys for integrating the pipelines with other systems like file systems as well as database as well.
• 7:09
  SPEAKER_02: Like TR data.
• 7:10
  SPEAKER_01: Yeah, if I and so you discovered airflow when you moved to Ford.
• 7:15
  SPEAKER_02: Yes, exactly.
• 7:16
  SPEAKER_02: That's where I was going.
• 7:17
  SPEAKER_02: Composer is not enabled at Ford.
• 7:19
  SPEAKER_02: When someone said Airflow, it was Astronomer Airflow because Ford already onboarded onto Astronomer.
• 7:26
  SPEAKER_02: I could see that a lot of teams started working on Astronomer Airflow itself.
• 7:30
  SPEAKER_02: So that was like two years ago.
• 7:32
  SPEAKER_02: So it was I I like coding.
• 7:36
  SPEAKER_02: I even when you could ask me what did I use, which scheduler did I use when I was working on DataLake based on Claudera Hadoop Data Lake.
• 7:46
  SPEAKER_02: So it was Uzi.
• 7:48
  SPEAKER_01: Wow.
• 7:48
  SPEAKER_01: Yes.
• 7:50
  SPEAKER_01: Okay, you did all of them.
• 7:51
  SPEAKER_01: You did all of them.
• 7:52
  SPEAKER_02: Yeah.
• 7:53
  SPEAKER_02: Uzi was interesting.
• 7:55
  SPEAKER_02: I liked Uzi.
• 7:56
  SPEAKER_02: I was more of creating workflow using code person, not go to GUI.
• 8:01
  SPEAKER_02: I did not use GUI a lot compared to the workflow code.
• 8:05
  SPEAKER_02: So I was okay to write my any job or workflow using the code.
• 8:11
  SPEAKER_02: So that kind of did help when I might when I moved to Airflow.
• 8:14
  SPEAKER_02: I'm like, oh, this is Python.
• 8:15
  SPEAKER_02: Okay, Python is even better compared to the other language.
• 8:19
  SPEAKER_02: I think it was like it was not exactly.
• 8:22
  SPEAKER_01: XML, I think.
• 8:23
  SPEAKER_01: No, was it XML?
• 8:25
  SPEAKER_02: It's not, it's it was XML or some other language.
• 8:30
  SPEAKER_02: I need to recall.
• 8:30
  SPEAKER_01: I think it was XML for Uzi, I think, but it was like so much.
• 8:34
  SPEAKER_01: It was a long time ago.
• 8:36
  SPEAKER_01: I don't remember.
• 8:37
  SPEAKER_02: Okay, now I um I need to look out, probably I will have to look up Uzi.
• 8:42
  SPEAKER_02: What is Uzi?
• 8:44
  SPEAKER_01: I think it was XML.
• 8:50
  SPEAKER_02: Yeah.
• 8:50
  SPEAKER_02: Okay.
• 8:51
  SPEAKER_02: Yeah, right.
• 8:52
  SPEAKER_02: Yeah, there you go.
• 8:53
  SPEAKER_02: Yep, yep.
• 8:54
  SPEAKER_02: Yep.
• 8:54
  SPEAKER_02: I was okay writing XML.
• 8:56
  SPEAKER_02: Python is even better.
• 8:57
  SPEAKER_02: Okay, write our DAGs into Python.
• 9:00
  SPEAKER_02: I like the DAG term.
• 9:02
  SPEAKER_02: Again, I think I'm going a lot into the background here, but I just wanted to share my journey.
• 9:07
  SPEAKER_01: It's very interesting.
• 9:07
  SPEAKER_01: Actually, it's very interesting.
• 9:09
  SPEAKER_02: DAG was like Spark uses DAG as well, data and acrylic graphs as well.
• 9:14
  SPEAKER_02: So I have been working on Spark, Apache Spark, like for a decade.
• 9:18
  SPEAKER_02: So I saw the DAG.
• 9:19
  SPEAKER_02: Oh, this is a familiar term.
• 9:20
  SPEAKER_02: Okay.
• 9:21
  SPEAKER_02: I know how DAG works.
• 9:23
  SPEAKER_02: So it was like a common term used.
• 9:26
  SPEAKER_02: And it was like, oh, I know this one.
• 9:28
  SPEAKER_02: And Python is cool.
• 9:30
  SPEAKER_02: Operators, I think Airflow is much more organized and much more rich in terms of off-the-shelf operators and etc.
• 9:39
  SPEAKER_02: All of that.
• 9:40
  SPEAKER_02: It was great.
• 9:40
  SPEAKER_02: It took only a year, but it but when I said in-house tolls, it was all about scaling up.
• 9:46
  SPEAKER_02: So if we if you create one DAG or two DAGs, we were like repeating the pattern and like parameterizing all of that, using the variables, etc.
• 9:54
  SPEAKER_02: So yeah, go ahead, please.
• 9:56
  SPEAKER_01: And did you feel that because it's very interesting because there are many, many people, unfortunately for them right now, that are using like I would say old orchestrators, like Autosys, uh Informatica, uh, even Uzi maybe?
• 10:10
  SPEAKER_01: Um, and I'm curious, did you feel that it was hard to move from those orchestrators to Airflow?
• 10:16
  SPEAKER_01: Like, how was it for you to move to Airflow?
• 10:19
  SPEAKER_01: Did you were you able to onboard on Airflow pretty easily, or did you have some struggle?
• 10:25
  SPEAKER_01: And just tell me more about that, the transition.
• 10:29
  SPEAKER_02: Yeah, a little bit struggled with how to bundle the tasks.
• 10:33
  SPEAKER_02: So I am the I'm a person like who always tries to correlate what I have learned and what I'm gonna learn.
• 10:39
  SPEAKER_02: So in tech in terms of technology, because in the beginning of my career, I started working on mainframes, green screens when I was in banking, right?
• 10:48
  SPEAKER_02: So now we are like it's we are using LLMs and all of that.
• 10:52
  SPEAKER_02: Throughout my journey, I always try to correlate based on the primitive concepts.
• 10:56
  SPEAKER_02: So here, what was Uzi coordinator, and what is here, it's it would be like a schedule.
• 11:02
  SPEAKER_02: I think only struggle I faced was like task groups and the syntax before 2.4.
• 11:10
  SPEAKER_02: And also, I had to understand.
• 11:14
  SPEAKER_02: Also, another thing that kind of bothered me was like schedule, interval, and schedule.
• 11:19
  SPEAKER_02: I was like, oh, what is the difference between schedule interval and schedule?
• 11:23
  SPEAKER_02: And I started using schedule interval, and jobs were not kicking off when they were specifically like weekly or monthly.
• 11:30
  SPEAKER_02: I was like, daily jobs are fine.
• 11:32
  SPEAKER_02: What is going on with weekly and monthly?
• 11:35
  SPEAKER_02: Then I was like, oh, there is something else called like schedule.
• 11:38
  SPEAKER_02: I had to I had to learn how airflow looks at schedule, like timetables, all of that concept.
• 11:44
  SPEAKER_02: So, okay, this is a little bit harder, it is supposed to be.
• 11:48
  SPEAKER_02: That's what I felt like.
• 11:49
  SPEAKER_02: I'm being honest with you here.
• 11:50
  SPEAKER_02: So then I was like, oh, schedule can be used.
• 11:53
  SPEAKER_02: That is much more easier because I love cron.
• 11:56
  SPEAKER_02: So let's go with cron there.
• 11:58
  SPEAKER_02: So I started using that kind of change in my life, and we had to upgrade to Airflow 2.3.2 to start using that schedule there, little by little.
• 12:07
  SPEAKER_02: Like there were like small details I had to learn, I had to sit and research.
• 12:11
  SPEAKER_02: But I got to say it is Airflow has a great community, and I'm so excited to see them, meet with them, talk to them at some mate.
• 12:20
  SPEAKER_01: So, would you say if, for example, I'm a radar engineer and right now I have to use auto seas or informatica, uh, would you say that hey, you should go to Airflow?
• 12:29
  SPEAKER_01: It's gonna be much easier for you ultimately.
• 12:33
  SPEAKER_02: Yeah, I would say yes.
• 12:35
  SPEAKER_02: Why?
• 12:36
  SPEAKER_02: Because everybody loves Python.
• 12:38
  SPEAKER_02: One thing.
• 12:39
  SPEAKER_02: A second thing is like Airflow can that is something we will go into the details when we go to when we go further up in our conversation.
• 12:48
  SPEAKER_02: Airflow enables joining heterogeneous uh systems.
• 12:53
  SPEAKER_02: It's oh, you could connect to remote on-prem system, you could connect to step one, you run your job on cloud, TCP.
• 13:01
  SPEAKER_02: You can step, you can run your step two or task two on AWS or go to Azure or go to database, go to file system, go to on-prem.
• 13:11
  SPEAKER_02: That is, it's stitching all of them together within the same DAG is magical to me.
• 13:16
  SPEAKER_02: So I would definitely recommend it to anybody.
• 13:20
  SPEAKER_01: I think you are touching a very important point is that as Airflow is an open source project, um, you you won't feel that you are limited in any way, right?
• 13:29
  SPEAKER_01: Like ultimately, if you want to add your own operator or your own task to interact with whatever you want, you can do it, right?
• 13:36
  SPEAKER_01: Uh and I think this is what you are touching on right now is like the flexibility that you have when you create a DAG, a data pipeline.
• 13:43
  SPEAKER_01: And so yeah, I agree.
• 13:45
  unknown: I agree.
• 13:45
  SPEAKER_01: And to circle back on the on what you said for scheduling, that reminds me when I started to use the Airflow like maybe five years ago.
• 13:53
  SPEAKER_01: At that time, we there wasn't the notion of data interval, it was just the execution date.
• 13:59
  SPEAKER_01: But the execution date wasn't really the date at which your DAG was executed, so it was just very confusing, but hopefully now it's much better.
• 14:08
  SPEAKER_01: But yeah, I can feel also the pain that you have at that time.
• 14:12
  SPEAKER_02: So it was not just me, thank you for uh sharing your experience.
• 14:17
  SPEAKER_02: Yeah.
• 14:18
  SPEAKER_02: So we have like uh awesome internal astronomer a flow web expaces there where we share our experience, where we help each other.
• 14:27
  SPEAKER_02: So I at least like help like two to other two to three other teams with the same thing.
• 14:32
  SPEAKER_02: Oh, you could do schedule, go with the cron, and like you don't have to worry about uh uh your jobs not kicking off on the schedule.
• 14:40
  SPEAKER_02: But schedule with cron works amazingly.
• 14:43
  SPEAKER_02: I love it.
• 14:43
  SPEAKER_02: That's what I use currently.
• 14:45
  SPEAKER_02: That's awesome.
• 14:46
  SPEAKER_01: So tell me tell me more about your like how Airflow helps you today with Fall.
• 14:51
  SPEAKER_02: Yeah, so going back to like my role, I think I have a couple of more hats that I'm wearing from day to day here at Ford.
• 14:58
  SPEAKER_02: I help teams to make to understand, oh, we have different operators here, and you could use this operator.
• 15:05
  SPEAKER_02: Also, you touched upon a very good point.
• 15:08
  SPEAKER_02: You could make your own operator based on the base operator, and this is this is the this is you could write your custom code and operator.
• 15:18
  SPEAKER_02: And then so I help teams to onboard onto Airflow.
• 15:21
  SPEAKER_02: Also, I could look at their DAGs, and oh, you could do this, you could do that, and also optimizing their DAGs, all of that.
• 15:28
  SPEAKER_02: So that's something I help with, and then Astronomer, I am working on both the flavors of Airflow here, Astronomer Airflow and Open Source Airflow.
• 15:39
  SPEAKER_02: So when I came to ADAS, then I'm I realized like, oh, people are running Airflow instances from using open source using Kubernetes, on-prem Kubernetes.
• 15:51
  SPEAKER_02: So we have a huge open on-prem server where we have a huge Kubernetes cluster.
• 15:58
  SPEAKER_02: So here in EDAS teams, I I came to EDAS like last year, like I said, then I was like, oh, this is not astronomer, but this is Airflow.
• 16:07
  SPEAKER_02: So what is this?
• 16:08
  SPEAKER_02: Then I started learning, oh, you could spin up your own Airflow instance using Kubernetes.
• 16:15
  SPEAKER_02: So as simple as that.
• 16:16
  SPEAKER_02: That's what uh a couple of teams are doing.
• 16:18
  SPEAKER_02: They have been doing it for I think a couple of years by now.
• 16:22
  SPEAKER_02: Then I did help uh a team, uh very large-scale offline algorithm team.
• 16:28
  SPEAKER_02: So they run computer vision algorithms and their pipelines, their data pipelines are so huge.
• 16:33
  SPEAKER_02: I'm talking about the scale.
• 16:35
  SPEAKER_02: So I did uh spin up an open source airflow instance for them.
• 16:40
  SPEAKER_02: And but the challenge there was like scale up.
• 16:42
  SPEAKER_02: We needed to scale up the defaults.
• 16:45
  SPEAKER_02: We talked about it when we spoke for the first time.
• 16:48
  SPEAKER_02: So default is like 16.
• 16:50
  SPEAKER_02: Maximum DAG instances that come up with the Airflow instances is like 16, but we had to go to hundreds to to cope up with their uh amount of data that they need to process on a daily basis.
• 17:02
  SPEAKER_02: So I am ready to create a support for the team.
• 17:04
  SPEAKER_02: I just recently I upgraded their airflow to 2.9.1, which is really cool.
• 17:10
  SPEAKER_02: If you look at you know, a lot of different differences like on GUI.
• 17:13
  SPEAKER_02: It's like you could collapse like a lot of logs, like free crossing logs or something like that.
• 17:18
  SPEAKER_02: That is really cool.
• 17:19
  SPEAKER_02: I love the histograms on the task duration and all of that.
• 17:24
  SPEAKER_02: So yeah, I'm getting the taste of both.
• 17:26
  SPEAKER_02: I'm spoiled by Astronomer Airflow at the same time.
• 17:28
  SPEAKER_02: Oh, okay.
• 17:29
  SPEAKER_02: I need to look at all the low-level details on open source Airflow instance as well.
• 17:34
  SPEAKER_02: But we are evangelizing the teams like to go to Astronomer Airflow to say like how easy it is to migrate, because Astronomer runtimes packages, Airflow versions into like runtimes.
• 17:46
  SPEAKER_02: And we, like I said, we have awesome internal for the Astronomer support team.
• 17:51
  SPEAKER_02: So basically, we need to keep running our load on open source Airflow.
• 17:56
  SPEAKER_02: At the same time, I'm like, oh, teams, you could slowly start migrating your Airflow DAGs to Astronomer.
• 18:03
  SPEAKER_02: Then the most frequent question I face is, oh, what is the difference?
• 18:07
  SPEAKER_02: Should I change my DAG?
• 18:09
  SPEAKER_02: Yeah, probably yes, because when when you can spin up your Airflow instance using Kubernetes, you can directly mount your file systems to uh the Kubernetes, but we cannot do it on the Astronomer Airport, a remote instance.
• 18:24
  SPEAKER_02: You need to set up service accounts, GCP.
• 18:27
  SPEAKER_02: It's a little bit different for the first time when we start with, but DAG should be uh DAG doesn't have to be changed drastically, but you need to tweak a little bit.
• 18:38
  SPEAKER_02: That's how I'm helping the teams here at NS.
• 18:41
  SPEAKER_02: Yeah, go ahead, please, Mark.
• 18:43
  SPEAKER_01: Yeah, yeah.
• 18:43
  SPEAKER_01: No, I I think I think that's a good point that you touch on.
• 18:46
  SPEAKER_01: Is honestly, like I've been using Airflow again like for many years now.
• 18:50
  SPEAKER_01: But what I what I can tell is definitely using Airflow with Astronomer is much easier, especially when you when you run at scale and when you uh need to run Airflow in a in a production environment in a company, not in local for making experiments.
• 19:04
  SPEAKER_01: Honestly, going with astronomer is the easiest way.
• 19:06
  SPEAKER_01: That's what I keep saying to everybody because that's the truth.
• 19:10
  SPEAKER_01: And you just do it again.
• 19:11
  SPEAKER_01: Yes, you can run your Airflow instance with Kubernetes and everything, but ultimately, if you have issues and you will have some issues, things like that, having astronomer support, having astronomer features and so on ultimately will help you.
• 19:25
  SPEAKER_01: I totally agree with you there.
• 19:27
  SPEAKER_01: And so I feel that you are more on the like DevOps side than on the data engineering side.
• 19:34
  SPEAKER_01: Is that correct?
• 19:35
  SPEAKER_01: Would you say that you still write data pipelines or you are more on the the like the back-end side of it?
• 19:41
  SPEAKER_02: Funny you ask that.
• 19:43
  SPEAKER_02: I started wearing DevOps uh hats only after coming to ed as Teams because I that emphasizes a lot of DevOps work or effort that goes into maintaining open source Airflow instance.
• 19:58
  SPEAKER_02: I am also responsible for data pipelines.
• 20:02
  SPEAKER_02: I I'm a data engineer, I'm still a data engineer here.
• 20:05
  SPEAKER_02: I have more responsibilities as part of my current role.
• 20:08
  SPEAKER_02: I'm working on data flow to process the data that is landing into our GCS, Google Cloud Storage, and then process it in like a distributed computing fashion.
• 20:19
  SPEAKER_02: If you ask me, distributed computing is my passion and which I love, which I have been loving for the last one decade.
• 20:27
  SPEAKER_02: So I love the scale.
• 20:28
  SPEAKER_02: I love uh developing scalable and flexible systems.
• 20:32
  SPEAKER_02: So I'm working on a data flow.
• 20:35
  SPEAKER_02: It's not specific to data flow, but we are processing the data, unstructured data using data flow.
• 20:41
  SPEAKER_02: So I'm developing on that as well.
• 20:42
  SPEAKER_02: At the same time, Sajesh and I, and we have a couple of more team members.
• 20:47
  SPEAKER_02: Our team is called like part of the bigger data ops team.
• 20:50
  SPEAKER_02: So we are responsible for integrating this data engineering pipeline with data science pipeline.
• 20:58
  SPEAKER_02: So I keep mentioning computer vision algorithms.
• 21:02
  SPEAKER_02: Our data, the nature, the nature of our data is unstructured data at the moment.
• 21:06
  SPEAKER_02: Um, it's like images and uh video data that is coming from cards as part of diagnostics uh data.
• 21:13
  SPEAKER_02: We need to churn the pro, we need to cut the data and then store the data into cloud storage.
• 21:20
  SPEAKER_02: But we are only storing, I think I'm going to I'm going to details at this moment, but I will save it for later.
• 21:27
  SPEAKER_02: What I was saying is, yes, I'm also a data engineer.
• 21:30
  SPEAKER_02: I'm working on data engineering pipeline, I'm coding at the same time.
• 21:34
  SPEAKER_02: I work on Terraform to create the infrastructure and write the pipelines, data engineering pipelines to process the data and process the data into DCS as well as BigQuery.
• 21:44
  SPEAKER_02: At the same time, work with the data science team, our sister team, to help them to set up a hybrid pipeline.
• 21:54
  SPEAKER_02: I'm gonna talk about hybrid pipeline in a little bit, but hybrid pipeline is on prem and GCP.
• 22:00
  SPEAKER_02: To run the inference as well as training, all of that.
• 22:03
  SPEAKER_02: So we are also working on that.
• 22:06
  SPEAKER_02: At the same time, we are we where another hat is like integrated role.
• 22:10
  SPEAKER_02: So we work with vendors.
• 22:11
  SPEAKER_02: Like currently, we are working with like other data science vendors, like how to visualize the data.
• 22:18
  SPEAKER_02: Okay, you run inference, but how do you look at your output, modal output, and then how do you make it better?
• 22:26
  SPEAKER_02: How do you keep on enhancing it?
• 22:28
  SPEAKER_02: So we are also working on that part.
• 22:30
  SPEAKER_02: So that's another hadby where like we onboard the new tools onto the teams.
• 22:35
  SPEAKER_02: But like we have two main use cases that I'm currently working on.
• 22:38
  SPEAKER_02: One I'm not gonna reveal it at this moment, but I will wait for that later.
• 22:42
  SPEAKER_02: We are working on two edas features that are like that are running on vehicles, like our Ford fleet.
• 22:52
  SPEAKER_02: So we are helping with that as well.
• 22:54
  SPEAKER_02: So we are working on data engineering as well as data science pipeline.
• 22:57
  SPEAKER_02: So we have like, yeah, and then okay, we are generating petabytes of data.
• 23:03
  SPEAKER_02: You need to maintain the data, so you need to archive the data.
• 23:06
  SPEAKER_02: How do we maintain the retention and archive the data, which is very important when it comes to customer data as well as lots of test data we generate on a weekly basis?
• 23:18
  SPEAKER_02: Funny you ask, yeah, okay.
• 23:20
  SPEAKER_02: I'm like data, I'm wearing like data engineer hat now, uh data ops hat.
• 23:24
  SPEAKER_02: Oh, I just moved like four petabytes of data from on-prem to cloud over last month using Google Cloud Utilities.
• 23:31
  SPEAKER_02: And you may ask, how did we orchestrate it using uh airflow?
• 23:35
  SPEAKER_02: So I think it all ends with Airflow in a way for us.
• 23:39
  SPEAKER_02: So yeah, I think that that is some most of what I am responsible at the moment.
• 23:45
  SPEAKER_02: I love working on all of these different areas and I love helping other teams with all of this.
• 23:51
  SPEAKER_01: And good, you definitely wear a lot of hats.
• 23:55
  SPEAKER_01: So congrats on that because it must be it must be difficult.
• 23:58
  SPEAKER_01: But on the infrastructure side, because it's interesting, most of the time, I think since we started, at least with my episodes, so far I've been talking with data engineers, I would say, mostly.
• 24:10
  SPEAKER_01: And you are a data engineer too, but you also have this uh hat on the on the infrastructure side on Airflow.
• 24:16
  SPEAKER_01: And so I'm a little bit wondering what is there anything that you for example, is there any like configuration setting or anything you can think of that that made a a change in your infrastruc in your airflow infrastructure, or something that you changed or tweaked to improve the performance, for example, or something like that.
• 24:36
  SPEAKER_01: Is there any maybe best practices or recommendations that you could make on this side?
• 24:41
  SPEAKER_02: Are you talking about the airflow in specific airflow?
• 24:44
  SPEAKER_02: Okay, necessity is the mother of invention.
• 24:48
  SPEAKER_02: Remember, I mentioned one use case where we have to run like hundreds of diagrams on a daily basis.
• 24:57
  SPEAKER_02: So first property I looked up is like parallelism on airflow.
• 25:03
  SPEAKER_02: So that's that kind of opened a door of a lot of possibilities to me.
• 25:09
  SPEAKER_02: Okay, we need to scale up our airflow instance.
• 25:12
  SPEAKER_02: It's like almost like production scale, right?
• 25:14
  SPEAKER_02: Okay, we need to run like hundreds of hundreds of DAG runs, thousands of task runs on a daily basis, and it's only keep on growing.
• 25:22
  SPEAKER_02: It we cannot expect that to go down.
• 25:24
  SPEAKER_02: So we need to find a sustainable way of tuning the properties.
• 25:28
  SPEAKER_02: So I started with airflow core parallelism.
• 25:32
  SPEAKER_02: So I did it was not just enough, right?
• 25:35
  SPEAKER_02: When you did that, it's like there are so many other properties that are helping parallelism to work or like other tuning.
• 25:45
  SPEAKER_02: I I have a list actually.
• 25:46
  SPEAKER_02: Uh that's what I have on airflow instance, like open source airflow instance.
• 25:53
  SPEAKER_02: Because yeah, so that I can just talk about all that.
• 25:56
  SPEAKER_02: Oh, okay, yeah.
• 25:57
  SPEAKER_02: Parallelism, I made it to I'm I changed it from 16 to 48 and then 48 to 96 or like in stuff.
• 26:06
  SPEAKER_02: So I ended up right now.
• 26:08
  SPEAKER_02: We have 350 is our parallelism.
• 26:10
  SPEAKER_02: That means you can run 350 DAG runs across the airflow instance, okay.
• 26:16
  SPEAKER_02: And then how many tasks are there for every DAG?
• 26:20
  SPEAKER_02: So I made like task runs.
• 26:22
  SPEAKER_02: Airflow core maximum active task runs per DAG is like 300.
• 26:26
  SPEAKER_02: Okay, okay, but it's still slow.
• 26:29
  SPEAKER_02: Yeah, let's see what this max map length is.
• 26:33
  SPEAKER_02: Okay, let's make it 12,000.
• 26:35
  SPEAKER_02: I really have to give uh give shout out to Airflow community here.
• 26:41
  SPEAKER_02: I love the documentation, those properties, what every property does there, how is it related to others?
• 26:47
  SPEAKER_02: What is the default value?
• 26:49
  SPEAKER_02: Oh, you make it zero, that means like there is no limit.
• 26:52
  SPEAKER_02: So I like that.
• 26:52
  SPEAKER_02: Anyway, going back to the list I have here: active runs, maximum active runs per DAG.
• 26:58
  SPEAKER_02: I had to understand, oh, what is the parallelism and what is the maximum active runs per DAG?
• 27:03
  SPEAKER_02: Okay, parallelism is like the instance deployment level, and others can be controlled within the DAG.
• 27:10
  SPEAKER_02: So that is also cool.
• 27:12
  SPEAKER_02: And then Kubernetes.
• 27:14
  SPEAKER_02: So I might have mentioned that we are spinning up Airflow, open source airflow instance using Kubernetes, on-prem Kubernetes here.
• 27:21
  SPEAKER_02: So very important thing is I started learning Kubernetes only after I started taking care of this open source airflow instance.
• 27:30
  SPEAKER_02: So that has been a deep learning curve for me in terms of Kubernetes.
• 27:34
  SPEAKER_02: But then I understand, oh, every yeah, but going back to the properties, scheduler minimum file process interval.
• 27:41
  SPEAKER_02: Oh, we could tune that too.
• 27:42
  SPEAKER_02: And then a parsing cleanup interval, which is also important because scheduler runs loops.
• 27:48
  SPEAKER_02: So we need to clean up after every loop.
• 27:51
  SPEAKER_02: That is also important, okay?
• 27:52
  SPEAKER_02: And then DAG back import timeout.
• 27:56
  SPEAKER_02: Okay, that is also important.
• 27:58
  SPEAKER_02: But I all it all started with parallelism.
• 28:00
  SPEAKER_02: I needed to scale up, that was necessary.
• 28:02
  SPEAKER_02: So I started learning every property that could uh help us tuning.
• 28:07
  SPEAKER_02: But I always like, yeah, look at look if you want to optimize, I think it it always there should be always a symptom, right?
• 28:16
  SPEAKER_02: What do we want to optimize?
• 28:18
  SPEAKER_02: Do you want to run like more instances that is like parallelism, right?
• 28:22
  SPEAKER_02: You're scaling up.
• 28:23
  SPEAKER_02: If it is not scaling up, if you see some issues, I think it should be specific to the issues and symptoms.
• 28:29
  SPEAKER_02: What for instance?
• 28:30
  SPEAKER_02: I can give you an example.
• 28:31
  SPEAKER_02: Okay, I could see that a lot of parts.
• 28:34
  SPEAKER_02: There is a uh there is this property, Airflow Kubernetes property, delete worker parts and failure.
• 28:40
  SPEAKER_02: Okay, you just upgrade it to some other upgraded Airflow version, and you're getting all the task failures.
• 28:47
  SPEAKER_02: There are like thousands of tasks failing because you're you have a use scale.
• 28:52
  SPEAKER_02: And then there is always this thing like admin team, right?
• 28:56
  SPEAKER_02: Okay, our on-prem admin team.
• 28:58
  SPEAKER_02: Why there are so many thousands of failed parts sitting there in your Kubernetes namespace?
• 29:03
  SPEAKER_02: Oh, that can be controlled with one single property, just make it false or true.
• 29:08
  SPEAKER_02: It's like a debug mode.
• 29:09
  SPEAKER_02: Whenever I upgrade the version, I just put it like true true so that I can see out-of-memory errors in part or but you could just turn it off once it is stable.
• 29:19
  SPEAKER_02: So that was really useful.
• 29:21
  SPEAKER_02: Um and then, okay, so this is cool.
• 29:26
  SPEAKER_02: Okay, when you are scaling up on Airflow instance, but your backend database should be tuned uh accordingly as well.
• 29:34
  SPEAKER_02: Like back, I'm talking about like pool size.
• 29:37
  SPEAKER_02: Okay, pool size and then pool connections on the SQL side.
• 29:42
  SPEAKER_01: But I'm like, this is super interesting, and I'm very curious on one thing.
• 29:46
  SPEAKER_01: Like, how when you like when you changed those parameters, those uh configuration settings, how did you know that you were moving towards the right direction?
• 29:57
  SPEAKER_01: Did you were you looking at a dashboard some metrics?
• 30:01
  SPEAKER_01: Like, how did you know that you are improving that you are improving the performances of your reference?
• 30:06
  SPEAKER_02: Okay, that's a pretty good question.
• 30:08
  SPEAKER_02: Observability plays a major role here.
• 30:11
  SPEAKER_02: First objective is are our tasks running on the track?
• 30:16
  SPEAKER_02: Are they getting completed within the average time?
• 30:19
  SPEAKER_02: So that is one thing.
• 30:20
  SPEAKER_02: Okay, we are we are good with our objective of running DAX and tasks on the time, and then look at Kubernetes cluster because that's where we are spinning up the open source airflow.
• 30:32
  SPEAKER_02: There is a Grafana dashboard that was enabled by our on-prem Kubernetes team.
• 30:38
  SPEAKER_02: So I look at pod usage.
• 30:40
  SPEAKER_02: Oh, is it using all the memory that we allocated?
• 30:44
  SPEAKER_02: Are we using resources correctly?
• 30:47
  SPEAKER_02: And all um so we're gonna talk about this our Airflow journey evolution submit as well.
• 30:56
  SPEAKER_02: I but there was so much learning on the road, right?
• 31:00
  SPEAKER_02: Okay, you started uh doing this, but oh, you were running, tasks of running fun, but we didn't even know there was a Grafana dashboard or Kibana, which is like elastic where we could see the errors.
• 31:12
  SPEAKER_02: We have learned all of that when we started seeing errors or some kind of weird behavior in the airflow.
• 31:20
  SPEAKER_02: So I always like start with airflow, I always monitor it.
• 31:23
  SPEAKER_02: I'm like, hey, I'm a huge fan of like monitoring, even though it's running fine.
• 31:28
  SPEAKER_02: If I have a minute, I just go and monitor them.
• 31:30
  SPEAKER_02: Oh, are they fine?
• 31:31
  SPEAKER_02: Are the tasks scheduled?
• 31:33
  SPEAKER_02: Are they ta are the tasks in schedule stayed longer than usual?
• 31:37
  SPEAKER_02: That is also like a red flag, right?
• 31:39
  SPEAKER_02: So I always look out for red flags, and it could be anything.
• 31:43
  SPEAKER_02: It could be on-prem cluster, it could be back-end database crashing, it could be anything else, like uh probably I need to repair the indices on back-end uh database.
• 31:53
  SPEAKER_02: Like you said, it's a really it's a huge maintenance project when you're talking about open source airflow.
• 32:01
  SPEAKER_02: But on the other hand, Airflow, Astronomer Airflow, I don't know how to do much.
• 32:05
  SPEAKER_02: Okay, DAX are running fine, tasks are running fine.
• 32:08
  SPEAKER_02: Again, many thanks to our awesome internal astronomer forward support team who closely work with your team.
• 32:14
  SPEAKER_02: It's I love the logs there, like matrix page on the Airflow astronomer dashboard.
• 32:20
  SPEAKER_02: So that is really helpful.
• 32:21
  SPEAKER_02: It shows the health of the cluster and all the health of all the parts and all of that, and the metrics of the usage of the resources as well.
• 32:29
  SPEAKER_02: Um, also, internal team is always behind us, like, oh, are you are you using all your resources?
• 32:35
  SPEAKER_02: Because astronomer has like astronomer units in terms of resources when you allocate them.
• 32:40
  SPEAKER_02: They do great.
• 32:41
  SPEAKER_02: And also, on the other hand, when we are talking about astronomer, so many benefits, right?
• 32:46
  SPEAKER_02: Okay, oh, you're you could use celery, which when you're running like thousands of instances, so they told me the recommendation.
• 32:54
  SPEAKER_02: Yeah, like I said, I learned a lot from astronomer discussions as well.
• 32:58
  SPEAKER_02: I try to apply them on Airflow, but it's a little bit different because you're doing everything on your own, like bringing up the demons, you're creating deployments, Kubernetes deployments, and oh, okay.
• 33:10
  SPEAKER_02: It took a couple of months for me to get to optimize the part template configuration.
• 33:16
  SPEAKER_02: Because part template is applied across all the tasks.
• 33:20
  SPEAKER_01: Yes.
• 33:20
  SPEAKER_02: So if you have a small task, you have a big task, how do you get to your good number?
• 33:24
  SPEAKER_02: So that was a really good experience as well.
• 33:27
  SPEAKER_02: Interesting as well.
• 33:28
  SPEAKER_01: Yeah, it's a lot of trial and URLs and experimentations for sure.
• 33:32
  SPEAKER_01: And monitor, yes.
• 33:33
  SPEAKER_02: Yeah, thanks to observability tools that are enabled by our admin team.
• 33:37
  SPEAKER_01: That is Yeah, that's super helpful.
• 33:39
  SPEAKER_01: And I have uh maybe one last question for you.
• 33:41
  SPEAKER_01: Is there anything that you have achieved or put in place uh with Airflow that you are really proud of?
• 33:46
  SPEAKER_01: Anything that you can think of?
• 33:47
  SPEAKER_02: I think I'm really proud of the scale at which we are running.
• 33:50
  SPEAKER_02: I talked about parallelism, like hundreds and thousands of hundreds of diagrams and thousands of task runs, and getting the good number of part templates that is liked by our on-prem Kubernetes team just to make sure everything is fine.
• 34:04
  SPEAKER_02: But I'm really proud of learning Kubernetes in our in my Airflow journey.
• 34:10
  SPEAKER_02: And uh when this Airflow, open source airflow instance was spun up.
• 34:15
  SPEAKER_02: It was like local executor.
• 34:17
  SPEAKER_02: It was local executor, everything was running within Airflow pod, like web server, scheduler, even all the tasks.
• 34:25
  SPEAKER_02: Then I was like, we could do Kubernetes executor.
• 34:28
  SPEAKER_02: So I had to learn all the basics of Kubernetes and all of that.
• 34:32
  SPEAKER_02: I just said it, I'm gonna say it again, part templates, it's good, but coming to that number, enable it, and parameterize it for all our three environments like dev, QA, and prod.
• 34:44
  SPEAKER_02: So basically, all I have to do is so it is like Docker-based, right?
• 34:49
  SPEAKER_02: Because I keep saying Kubernetes, we have our Airflow Docker image using the versions, our Ford robotics, or other dependencies that we really need in terms of this.
• 35:00
  SPEAKER_02: So basically, we have our own Docker image and then parametrizing all of that, learning config maps.
• 35:06
  SPEAKER_02: Oh, you could just uh have different config maps for different environments, and you could have the central Docker image again.
• 35:13
  SPEAKER_02: Uh I am very grateful to my supervisor, Sarjay.
• 35:17
  SPEAKER_02: She always comes up with like new ideas to make it uh continuously better.
• 35:22
  SPEAKER_02: I think I'm really proud of scale that we are running and scale we will run on Astronomer as well.
• 35:29
  SPEAKER_02: But then it's a different ballgame when you go to Astronomer because Astronomer uses GKE, Google Kubernetes engine, while we are spinning up other open source airflow instance using on-prem Kubernetes.
• 35:41
  SPEAKER_02: There is a huge difference.
• 35:42
  SPEAKER_02: You know, like two different teams taking care of uh two different flavors of Kubernetes here.
• 35:47
  SPEAKER_02: Also, GKE is a bit expensive compared to on-prem Kubernetes.
• 35:52
  SPEAKER_02: I actually have to bring it up.
• 35:54
  SPEAKER_02: So thanks to the Astronomer team for telling me about deferable operators.
• 35:59
  SPEAKER_01: Yeah, yep.
• 36:00
  SPEAKER_02: So that was another game changer.
• 36:03
  SPEAKER_02: I have a huge son of colors, it always comes up purple.
• 36:06
  SPEAKER_02: Oh, when you're deferring the operator.
• 36:08
  SPEAKER_02: So that's where I had to work with other four teams, who is like a tools team.
• 36:13
  SPEAKER_02: They make they made a custom operator for our hybrid environment because we run a lot of on-prem jobs.
• 36:21
  SPEAKER_02: At the same time, we have to run cloud jobs.
• 36:24
  SPEAKER_02: Test data should be processed, test vehicle data should be processed on-prem, and customer data should go through our cloud, which is like our data factory, which is data governed in terms of privacy and all that good stuff.
• 36:37
  SPEAKER_02: Data governance team takes care of it.
• 36:38
  SPEAKER_02: What I was saying is to do both, to have a true hybrid and modern architecture, yeah.
• 36:46
  SPEAKER_02: Astronomer Airflow did enable us to do.
• 36:49
  SPEAKER_01: And you actually brought a very good point on the defrayable operators.
• 36:53
  SPEAKER_01: This is something that uh some people don't know about, but it's not only for sensors, it's just like for many operators.
• 37:01
  SPEAKER_01: You can use the defrayable version, and ultimately they will be more efficient and they will save you uh some money as well.
• 37:09
  SPEAKER_01: So that's why, like, whenever you have the defrayable operator for your operator, use most of the time you should use with the defrayable operator instead of the classic one.
• 37:19
  SPEAKER_01: You will it will be more efficient.
• 37:20
  SPEAKER_02: Yeah, that was my learning, and I like it.
• 37:22
  SPEAKER_02: I it was a little learning curve.
• 37:24
  SPEAKER_02: I thought, okay, how do you do that?
• 37:26
  SPEAKER_02: And you have to, we had to write our own sensor there.
• 37:29
  SPEAKER_02: You you nailed it there, right?
• 37:31
  SPEAKER_02: So one of our tools, Ford Tools team, did uh write that for me.
• 37:35
  SPEAKER_02: I tested it, and that was great.
• 37:38
  SPEAKER_02: And we could see that parts are not holding, parts are not yelled as long as the task is running.
• 37:44
  SPEAKER_02: They get released nicely, and then they get allocated based on the the duration that we mentioned in the deferable operator.
• 37:52
  SPEAKER_02: Yeah, that was great too.
• 37:53
  SPEAKER_01: Yeah, yeah, I can stress uh this out enough.
• 37:55
  SPEAKER_01: Just people that are listening to this episode, if you don't know about deferrable operators, take a look at them.
• 38:00
  SPEAKER_01: They will you will love them.
• 38:01
  SPEAKER_01: Awesome! Wow, that's already time.
• 38:04
  SPEAKER_01: Vasanta, that was amazing to have you in this episode.
• 38:08
  SPEAKER_01: Uh so thank you so much for joining.
• 38:11
  SPEAKER_01: But what is the best way for folks to get in touch with you?
• 38:14
  SPEAKER_02: LinkedIn message would be a good uh yeah, a channel.
• 38:18
  SPEAKER_01: Okay, okay, that's awesome.
• 38:20
  SPEAKER_01: Again, it was a wonderful pleasure to have you, and I hope to see you soon, maybe for another episode.
• 38:25
  SPEAKER_01: Who knows?
• 38:26
  SPEAKER_02: Thank you so much, Mark.
• 38:27
  SPEAKER_02: It's an honor.
• 38:28
  SPEAKER_02: Thank you so much for having me.
• 38:30
  SPEAKER_01: Thank you.
• 38:30
  SPEAKER_01: Have a good one.
• 38:31
  SPEAKER_02: You too.
• 38:32
  SPEAKER_02: Bye.
• 38:35
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 38:41
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 38:45
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five-star review.