Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Airflow in Action: Powering Instacart's Complex Ecosystem

Transcript

Airflow in Action: Powering Instacart's Complex Ecosystem · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: I think one of the things Airflow has already addressed is this ability to run tasks as remote executors.
• 0:05
  SPEAKER_01: I would want to see like more support around more native integration with cloud providers.
• 0:09
  SPEAKER_01: For example, with AWS, we built our own ECS operator, which wasn't available at that time.
• 0:14
  SPEAKER_01: But on the community side, if like certain operators are easily accessible to us, that would be great.
• 0:18
  SPEAKER_01: And then I think I'm pretty excited to play with Airflow 3 when it comes out.
• 0:22
  SPEAKER_01: I think there are a lot of things there.
• 0:24
  SPEAKER_01: The UI refresh.
• 0:25
  SPEAKER_01: I think it was time when we needed to change the UI a little bit, make it more modern.
• 0:35
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:45
  SPEAKER_02: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:50
  SPEAKER_02: Join the Astronomer Roadshow for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:58
  SPEAKER_02: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:06
  SPEAKER_02: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:11
  SPEAKER_02: Register now.
• 1:12
  SPEAKER_02: The link is in the show notes.
• 1:16
  SPEAKER_02: Hi everyone.
• 1:17
  SPEAKER_02: Welcome to the Data Flowcast.
• 1:19
  SPEAKER_02: I'm your host, Kenton Damas, and today I'm joined by Anand Argawal, who is a staff software engineer at Instacart.
• 1:26
  SPEAKER_02: Anand, how are you doing today?
• 1:28
  SPEAKER_01: I'm doing good.
• 1:28
  SPEAKER_02: That's great.
• 1:29
  SPEAKER_02: Remind me where you're located.
• 1:31
  SPEAKER_01: So I live in Santa Clara.
• 1:32
  SPEAKER_01: Right.
• 1:33
  SPEAKER_01: I'm in Instacart's office in San Francisco.
• 1:35
  SPEAKER_02: Okay.
• 1:36
  SPEAKER_02: That's great.
• 1:36
  SPEAKER_02: Do you go into the office regularly?
• 1:38
  SPEAKER_01: Not much.
• 1:39
  SPEAKER_01: We are a remote first company.
• 1:42
  SPEAKER_01: So yeah, whenever there's like some certain meetings or certain things that I need to be there.
• 1:47
  SPEAKER_02: Okay.
• 1:48
  SPEAKER_02: Got it.
• 1:48
  SPEAKER_02: Yeah.
• 1:48
  SPEAKER_02: Astronomer is the same.
• 1:50
  SPEAKER_02: We have our office in New York, which is where our headquarters is.
• 1:52
  SPEAKER_02: So I make it there, I don't know, maybe a couple of times a year or so.
• 1:56
  SPEAKER_02: But it's always nice to see people in person when you're mostly remote.
• 1:59
  SPEAKER_01: Yeah, yeah.
• 1:59
  SPEAKER_01: Especially when you have you're working on big design decisions and you're doing some planning.
• 2:03
  SPEAKER_01: It's nice to be there face to face.
• 2:06
  SPEAKER_01: Just get your creative juices flowing.
• 2:08
  SPEAKER_02: Yeah, absolutely.
• 2:09
  SPEAKER_02: Super helpful with for that.
• 2:11
  SPEAKER_02: Appreciate you taking the time of your in-office time to join us today.
• 2:15
  SPEAKER_02: So let's go ahead and dive in.
• 2:17
  SPEAKER_02: For today's podcast, we're going to discuss Instacart's airflow journey.
• 2:21
  SPEAKER_02: Love this topic.
• 2:22
  SPEAKER_02: And to start off, so Instacart, I'm guessing it's probably familiar to many of our listeners, especially our US-based listeners.
• 2:30
  SPEAKER_02: But just in case, can you describe for everybody what your company does?
• 2:34
  SPEAKER_01: Yeah.
• 2:34
  SPEAKER_01: So Instacart is a multi-sided marketplace.
• 2:37
  SPEAKER_01: So most popular is the Instacart app, which allows you to order your groceries online from your favorite retailers.
• 2:44
  SPEAKER_01: It could be like small baskets through something like CVS or 7-Eleven, or it could be like your weekly or your bi-weekly shopping trips to places like Costco.
• 2:54
  SPEAKER_01: But when I say it's a multi-sided marketplace, on one side of the marketplace is all the different retailers we have.
• 3:00
  SPEAKER_01: So we have from Walmart, Sprouts, to non-grocery retailers like Ulta Beauty or Sephora and a bunch of others.
• 3:08
  SPEAKER_01: And we have different kinds of offerings we have for our retail partners.
• 3:11
  SPEAKER_01: The other side is consumers like you and me, who wants to order from the Instacart marketplace.
• 3:16
  SPEAKER_01: Then on the other side is the shoppers who actually go into the store and actually do the shopping.
• 3:21
  SPEAKER_01: And then we also have the fourth side of the marketplace, which is our ads business, which is all the different CPG partners, be it Coke, Proctor and Gamble, PepsiCo, who want to advertise on the Instacart marketplace and get their revenue going from there.
• 3:36
  SPEAKER_02: Yeah, totally makes sense.
• 3:37
  SPEAKER_02: And I imagine your business has grown quite a lot over the past, I'm gonna call it five years now, which is crazy, but there's been such a huge increase in online shopping in this manner.
• 3:48
  SPEAKER_01: Yeah, absolutely.
• 3:49
  SPEAKER_01: Like I think COVID was a big driver to the growth just because of the nature of the world we were living in at that time.
• 3:54
  SPEAKER_01: Instacart became like a pretty important product.
• 3:57
  SPEAKER_01: Our shopper community really engaged and helped delivered all those autos to our customers.
• 4:02
  SPEAKER_01: And then it's been like it's been like a pretty good growth journey so far, not only in terms of the different products, we have a lot more retailers on our platform.
• 4:10
  SPEAKER_01: And then on the technical side, our systems have grown.
• 4:13
  SPEAKER_01: The amount of data that we process is relatively high.
• 4:16
  SPEAKER_01: And that also calls for like more maturity, more scalability and reliability in all the engineering uh systems that power uh that traffic on the business side.
• 4:25
  SPEAKER_02: Yeah, absolutely.
• 4:26
  SPEAKER_02: And so speaking about the engineering side, what does your team do for the business?
• 4:31
  SPEAKER_01: So I work on the data infrastructure team.
• 4:33
  SPEAKER_01: So my team deals with everything data related at Instacart, whether it is systems that allows you to access or visualize data, to systems that allow you to visualize and access data through different BI tooling or data or tools that allow allows you to do self-serve for data, um, and then to tools that allows you to ingest data from the point of source, which could be the Instacard app, it could be third parties, to the point where it is consumed by, let's say, someone on the business side or someone on the product side to actually drive a lot of decision.
• 5:04
  SPEAKER_01: And Instacard being an internet-first marketplace, and we work with a lot of data.
• 5:09
  SPEAKER_01: And then data also drives a lot of the decision making that goes behind any product that we want to invest in, what to prioritize, what to deprioritize, and so on, identifying the market trends.
• 5:19
  SPEAKER_01: So data plays a very critical role at Instacard.
• 5:21
  SPEAKER_01: And I'm a staff software engineer on the group that kind of builds all the tooling, builds all the systems that moves data through different parts.
• 5:29
  SPEAKER_02: Yeah, that's awesome.
• 5:30
  SPEAKER_02: And that's such a huge scope.
• 5:31
  SPEAKER_02: So, as you said, data is very key to your business and a huge part of your product, as well as your business operations and powering all of your decision making.
• 5:41
  SPEAKER_02: So, related to probably managing, orchestrating all of that data, I understand Instacart started with Airflow back in 2019, which as we've just touched on, was a much different time, both for the Airflow project and of course for your company.
• 5:54
  SPEAKER_02: So, can you maybe describe what your Airflow usage looked like back then in the beginning?
• 6:00
  SPEAKER_01: Yeah, so so I joined the company um around 2000, end of 2020, so right along after the second wave of the pandemic, but I have a fairly good idea of what our journey looked like.
• 6:10
  SPEAKER_01: So before COVID, we had like different teams or different organizations who deployed their own airflow clusters for their relatively smallish orchestration needs.
• 6:20
  SPEAKER_01: It was purely used both as an orchestrator and then also to move data like through data pipelines from one system to the other, do some kind of processing in between.
• 6:31
  SPEAKER_01: We didn't have very modern AI tools or very modern data transformation tools.
• 6:36
  SPEAKER_01: So a lot of the work was actually done on the Airflow workers itself.
• 6:40
  SPEAKER_01: And each org had their own smaller clusters that they had set up.
• 6:44
  SPEAKER_01: They had expertise within their respective teams who managed those clusters.
• 6:48
  SPEAKER_01: And there was no homogeneity in how we deployed Airflow or homogeneity in terms of expertise around Airflow.
• 6:55
  SPEAKER_01: Each team had different patterns through which they wrote their DAGs.
• 6:58
  SPEAKER_01: They had different like source and destinations where they were sending and moving the data from, and just completely different ways to understand data and to how to transform it.
• 7:07
  SPEAKER_01: Some teams were comfortable doing the transformation in Python or some bespoke language.
• 7:12
  SPEAKER_01: Some teams wanted more advanced tooling through TensorFlow or libraries that you get in Python or R.
• 7:18
  SPEAKER_01: So we had like very different views of how Airflow was being used across different orgs.
• 7:24
  SPEAKER_01: And then that was the status quo at that point.
• 7:26
  SPEAKER_01: And then also Airflow was a relatively smaller project compared to today, both in terms of the majority of the product itself and the features it supports.
• 7:34
  SPEAKER_01: So we had to implement it certain wrappers on top of it just to make it work more seamless with our infrastructure and our internal libraries.
• 7:42
  SPEAKER_01: But yeah, I think the main point to call out would be just that there was no homogeneity in how it was deployed.
• 7:47
  SPEAKER_01: And just everyone had a different understanding of airflow and orchestration and transformations driven through Airflow.
• 7:54
  SPEAKER_02: Yeah, that fractured nature can be really tough.
• 7:58
  SPEAKER_02: So as you encountered that kind of throughout your Airflow usage, how did you change to adapt to that?
• 8:03
  SPEAKER_02: What did you do to make that problem better?
• 8:05
  SPEAKER_01: Yeah, so I would say really in 2020, 2021 was when I got involved more closely with Airflow.
• 8:12
  SPEAKER_01: We were running into a lot of scalability issues around Airflow, um, especially the way we had deployed it on our infrastructure.
• 8:18
  SPEAKER_01: We didn't have like great monitoring, great um tooling to just understand the state of the world at that point.
• 8:24
  SPEAKER_01: And then teams were running into frequent interrupts.
• 8:27
  SPEAKER_01: Some devs wanted to do some testing on it, whereas we still wanted to have a high SLA just because we were running production workloads there as well.
• 8:34
  SPEAKER_01: So when I got involved, the the part of the reason was we wanted to set up a new transformation system for all things data movement at Instacart through DBT.
• 8:44
  SPEAKER_01: And then teams wanted to build like new clusters in order to kind of power new use cases so that they don't affect like some of the central clusters that were at that point.
• 8:54
  SPEAKER_01: And then when I got about deploying a cluster, it took a lot of time because Airflow is a complex project.
• 8:59
  SPEAKER_01: You need a database, you need some kind of a broker, you have multiple services, you have a front end there as well, you want to secure it because actual data transformation is happening.
• 9:08
  SPEAKER_01: It could, the data could be finance and non-finance related, could contain PII and everything.
• 9:13
  SPEAKER_01: So it really started by taking like a holistic approach to what different teams had done.
• 9:19
  SPEAKER_01: And then it was right around 2021, 22, I believe, when Airflow 2 came out, which was like a pretty big difference over Airflow one, both in terms of the architecture and then in terms of the complexity of the project itself.
• 9:32
  SPEAKER_01: So we started off by building like a standardized approach and taking an approach where, hey, this is a standard template of deploying Airflow and running Airflow.
• 9:41
  SPEAKER_01: And that's what we're going to ask everyone to do from this point onwards.
• 9:44
  SPEAKER_01: Obviously, you can't go back and modify legacy clusters.
• 9:47
  SPEAKER_01: So we ended up kind of uh creating modules that allow you to easily deploy the Airflow cluster itself.
• 9:53
  SPEAKER_01: So something that could take uh certain engineers months, we brought it down to weeks and then we brought it down to days.
• 9:59
  SPEAKER_01: Um, and then we uh along the way, we established best practices on what kind of workloads have to run uh in what manner.
• 10:06
  SPEAKER_01: Do you want to run all the airflow logic on the workers itself?
• 10:10
  SPEAKER_01: So you do you want like really beefy workers?
• 10:13
  SPEAKER_01: Do you want like multiple smallish workers?
• 10:16
  SPEAKER_01: How do you do proper resource allocation and pool size allotment?
• 10:20
  SPEAKER_01: And then at the same time, we had a lot of these use cases where different teams wanted to run their own way of running the code or processing the data.
• 10:28
  SPEAKER_01: Some wanted to do it in Ruby, some wanted to do it in R, some wanted to do it in like Python, and they had their own scripts.
• 10:35
  SPEAKER_01: So we had to make their own scripts work because rewriting a whole lot of that would have been a lot of work.
• 10:40
  SPEAKER_01: And then when you're trying to establish standards at a large company with a large number of people, rewriting code is something that obviously gets deprioritized, unless it's like super critical.
• 10:50
  SPEAKER_01: So you wanted to give them a way to easily run their own, their current scripts, their current code.
• 10:55
  SPEAKER_01: And that's where we started investing a lot into this concept of remote executors.
• 10:59
  SPEAKER_01: So basically, Airflow just acts as like a purely scheduling and orchestration layer, and then the actual work actually happens on some executor somewhere.
• 11:07
  SPEAKER_01: And then we started building out all these patterns and kind of the standardization process started through that.
• 11:15
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:24
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 11:36
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 11:44
  SPEAKER_02: So your standardization is very much on the airflow infrastructure side of things and instituting patterns there that allow it to be both easier to adopt and I'd imagine democratize it quite a bit because setting up a complex airflow instance that supports a lot of pipelines is not a skill set that every data engineer or every software engineer is going to have.
• 12:05
  SPEAKER_02: So I'd imagine that you've made it much easier internally for Airflow to be used more broadly.
• 12:10
  SPEAKER_01: Yeah, I mean, we had we have we still have a lot of, we had a very different persona of users, a very variety of persona of users who use Airflow.
• 12:19
  SPEAKER_01: So on one side, you have software engineers who are, they can take any challenge you throw at them, right?
• 12:24
  SPEAKER_01: That's part of their day-to-day job.
• 12:25
  SPEAKER_01: You have infrastructure engineers, cloud engineers, then you have data engineers who have like very focused work.
• 12:31
  SPEAKER_01: They want a platform where they just run their code and they build their pipelines.
• 12:35
  SPEAKER_01: And then you also have some analysts who are who have a different job profile.
• 12:39
  SPEAKER_01: So they would expect a cluster that is there for them, managed by some team or some third-party provider.
• 12:44
  SPEAKER_01: And their job is mostly to focus on writing the pipelines and writing the SQL transformations themselves.
• 12:49
  SPEAKER_01: So you have to cater to all of them.
• 12:50
  SPEAKER_01: And we and we made a decision back then that we would self-host Airflow because we felt like we had sufficient expertise within the company.
• 12:57
  SPEAKER_01: And then we would be able to manage it and integrate it much better with the internal tooling we had developed.
• 13:03
  SPEAKER_01: So tooling could be like how we deploy databases, what kind of monitoring do we have on databases, our internal logging and metric libraries and so on.
• 13:11
  SPEAKER_01: So it really started by like standardizing the infrastructure layer.
• 13:15
  SPEAKER_01: So like how you deploy the cluster, and then it moved on to standardizing different patterns on how you write the actual Airflow DAGs.
• 13:23
  SPEAKER_01: So we have a subset of DAGs today which run on the workers themselves.
• 13:27
  SPEAKER_01: They do like data transformations through DVD.
• 13:30
  SPEAKER_01: Then we have some subsets of DAG which are more on that remote executor pattern where Airflow is just acting as an orchestrator and a scheduler.
• 13:39
  SPEAKER_01: Teams, they they bring in what we call as tasks.
• 13:42
  SPEAKER_01: They bring in their task in their own language, however they want to run it.
• 13:47
  SPEAKER_01: And then we simply plug it into Airflow and we orchestrate it.
• 13:50
  SPEAKER_01: And sometimes it's a little bit more like in between, where we have defined custom DAG factories where teams just write a YAML interface through the UI or just like really simple YAML interfaces.
• 14:01
  SPEAKER_01: And then we transform it into an Airflow DAG, and then we run it like in different formats.
• 14:06
  SPEAKER_01: So we have a wide variety of use cases from Snowflake transformation, data bricks to delta movement, general data pipelines outside of our DPD solutions, and then other data pipelines where we're moving data from Google Sheet to Snowflake, let's say, or capturing the incidents, the data that we happen, the Jira data, so that you can do some analytics on it and so on.
• 14:29
  SPEAKER_02: Yeah.
• 14:30
  SPEAKER_02: I want to double-click on that remote execution piece because this is very timely considering Airflow 3, which right now is about to come out as we're recording when this gets released.
• 14:40
  SPEAKER_02: It may have actually already been released.
• 14:43
  SPEAKER_02: Remote execution is one of the big features coming in Airflow 3.
• 14:46
  SPEAKER_02: So this concept of an executor that can manage tasks that are being executed on workers that are not connected to your Airflow instance might be in an entirely different cloud or something like that.
• 14:57
  SPEAKER_02: Tell me more about how you implemented that and what that looks like at Instacart.
• 15:02
  SPEAKER_01: Yeah, so uh I think yes, very timely.
• 15:05
  SPEAKER_01: I was at the Airflow Summit last year.
• 15:06
  SPEAKER_01: I spoke at that, and that was like the highlight there.
• 15:09
  SPEAKER_01: So I think the the way they are positioning it is running a task anywhere in any language at any time.
• 15:15
  SPEAKER_01: And I think that's really powerful given just like the polyglot ecosystem of systems and programming languages that we have today.
• 15:23
  SPEAKER_01: Some languages are more suited for like analytical and data processing related tasks.
• 15:28
  SPEAKER_01: Some are low latency, so you obviously want to write different kinds of systems in them.
• 15:32
  SPEAKER_01: And the way we actually went about implementing something like this is at Instacart, we are in we are a big AWS shop.
• 15:39
  SPEAKER_01: And the way we deploy our services, we deploy it as ECS containers.
• 15:44
  SPEAKER_01: So Amazon has its own way of packaging containerized applications and running them through what they call as ECS.
• 15:51
  SPEAKER_01: And then we basically run a piece of code as an ECS task within the bounds of certain internal wrappers that we have developed.
• 15:58
  SPEAKER_01: So what teams do is they let's say very simply write a Python script.
• 16:02
  SPEAKER_01: There's a certain way to invoke that Python script.
• 16:05
  SPEAKER_01: And then we have Airflow call the task, call that particular Python script through that, through those set of parameters, and that gets executed on ECS directly.
• 16:15
  SPEAKER_01: And then while it's on executing on ECS, you can specify granular controls as to how much CPU to give it, how much memory to give it.
• 16:24
  SPEAKER_01: And then since it's just a script, the script can be in any language.
• 16:27
  SPEAKER_01: It could be Python, it could be Golang, it could be Ruby, it could be R.
• 16:30
  SPEAKER_01: And then Airflow is, and one of the other advantages with Airflow 2 is this service that they called, that they included as part of Airflow 2 called Trigger.
• 16:40
  SPEAKER_01: And this concept of deferable operators.
• 16:42
  SPEAKER_01: So deferable operators really helped us a lot here because even with given since I mentioned that we have a lot of different DAG patterns, some DAGs are running on the workers itself, it allows us to scale the cluster pretty well.
• 16:54
  SPEAKER_01: Whereas in this case, like we have a bunch of tasks who are just trying to poke at the easiest task.
• 16:59
  SPEAKER_01: Is it done?
• 17:00
  SPEAKER_01: What's the status there?
• 17:02
  SPEAKER_01: It's not taking up the entire worker slot at that point.
• 17:05
  SPEAKER_01: So that's advantageous because you are not compromising your other DAGs who want to run on the worker.
• 17:10
  SPEAKER_01: And you have a set you have offloaded the work to like a separate service who's managing all of that.
• 17:14
  SPEAKER_01: So it really, and then I think Airflow being open source, and then this wide support of operators and plugins that Airflow has helps there as well.
• 17:23
  SPEAKER_01: So we were able to use a leverage a lot of that to build our own solutions.
• 17:27
  SPEAKER_01: And it's really great to see that feedback from the community has moved into Airflow 3, where this is becoming like a marquee feature for the for Airflow project itself.
• 17:36
  SPEAKER_02: Yeah, absolutely.
• 17:37
  SPEAKER_02: Definitely community feedback is a huge part in developing these features, especially when it's coming from huge users that, you know, really large companies that have big flow instances and have learned a lot, like you're describing, Instacart has.
• 17:51
  SPEAKER_02: Yeah, that's really interesting.
• 17:52
  SPEAKER_02: I think task isolation, which was the underlying architectural change that is allowing for remote execution in Airflow 3, was something that has definitely come up in prior Airflow surveys and been requested by a lot of users.
• 18:04
  SPEAKER_02: That's really cool to hear about your implementation and how you were ahead of the open source curve in that regard.
• 18:10
  SPEAKER_02: I'm curious, speaking of Airflow 3, you talked about how a lot of these, the standardization that you've implemented was around the same timeframe of upgrading to Airflow 2, kind of end of 2020, beginning of 2021.
• 18:24
  SPEAKER_02: How do you treat upgrading within this framework?
• 18:26
  SPEAKER_02: Is that something that you standardize?
• 18:28
  SPEAKER_02: Do you stay on recent versions?
• 18:30
  SPEAKER_01: Yeah.
• 18:30
  SPEAKER_01: So within our, so like right now at Instacart, we are moving towards a central cluster, which like the which um my team manages.
• 18:38
  SPEAKER_01: And then everyone, we are we are working with every different different airflow, that smaller airflow that we have in the company to onboard to our airflow so that we establish certain patterns, certain standards, and then different teams can use it as use us as like a platform and we maintain the central cluster.
• 18:54
  SPEAKER_02: Um sorry to interrupt you.
• 18:56
  SPEAKER_02: By central cluster, do you mean that like a single airflow instance that everybody is working out of?
• 19:01
  SPEAKER_01: Yeah, a single airflow instance.
• 19:03
  SPEAKER_01: We are still okay if teams want to spin up like a new cluster if there are specific needs.
• 19:09
  SPEAKER_01: For example, sometimes for like finance pipelines, you may want to run it in like a separate cluster.
• 19:14
  SPEAKER_01: Some teams have protected data that they work with that they don't want to be within the the, like they want it completely isolated.
• 19:21
  SPEAKER_01: So we have those cases here and there, but more or less we want a central cluster across the company that everyone uses, a single instance, basically.
• 19:29
  SPEAKER_02: Interesting.
• 19:30
  SPEAKER_02: That must be a pretty large airflow instance once you, or at least it will be once you get everybody on it.
• 19:35
  SPEAKER_01: Yeah, we are, I would say halfway there.
• 19:38
  SPEAKER_01: And then I think by the end of this year, we should be in a pretty good place where after the end of this year, we might have like very few clusters left.
• 19:45
  SPEAKER_01: Today we run close to 3,000 tags on our cluster.
• 19:49
  SPEAKER_01: And we run, I believe, close to a million tasks per month.
• 19:52
  SPEAKER_01: If I got pretty recently.
• 19:54
  SPEAKER_01: So, yeah, so we do run a fairly large cluster, which is still a smallish team managing it.
• 19:59
  SPEAKER_01: And that's like one of the lessons we learned.
• 20:01
  SPEAKER_01: If you build things through standards, if, for example, our entire cluster configuration is in Terraform.
• 20:06
  SPEAKER_01: So at Instacart, we are like big uh proponents of infrastructure as code.
• 20:10
  SPEAKER_01: It gives us like a lot of uh visibility on what the configuration is running.
• 20:14
  SPEAKER_01: And then each change goes through like a process that we have developed.
• 20:18
  SPEAKER_01: And then we use a lot of internal um tooling around.
• 20:22
  SPEAKER_01: So whenever we need to deploy a database, for example, at Instacart, there's a separate Terraform module for it.
• 20:27
  SPEAKER_01: It imposes a lot of standards, which our storage infra team can operate that database much efficiently.
• 20:33
  SPEAKER_01: So we use a lot of core components like that.
• 20:35
  SPEAKER_01: Plus, we have a lot of internal libraries around metrics and logging and just incident management in general that allows us to seamlessly scale out the cluster.
• 20:44
  SPEAKER_01: And bringing in all those best practices, those pre-built solutions that we had already developed allowed us to scale this cluster pretty, pretty easily.
• 20:53
  SPEAKER_01: And then in terms of upgrades, when we do upgrades, we have a we have a cadence of upgrading every six months.
• 20:58
  SPEAKER_01: Sometimes it goes into nine months, depends on just where we are with different projects that we are working on.
• 21:05
  SPEAKER_01: So right now we are on two seven, right, latest.
• 21:09
  SPEAKER_01: And in edge two this year, we plan to go to 210.
• 21:12
  SPEAKER_01: So that would be just before, or whatever is the last cutoff release before Airflow 3.
• 21:17
  SPEAKER_01: So we do plan to stay on the latest and greatest and then upgrade subsequently.
• 21:22
  SPEAKER_01: And then since I mentioned we do things in Terraform, we use a lot of our libraries around, let's say, database upgrade.
• 21:29
  SPEAKER_01: We have a pretty robust ecosystem around database upgrades.
• 21:32
  SPEAKER_01: We are able to leverage all of the learnings from Airflow, and then Airflow upgrades become pretty seamless because a big big part of the upgrade is just the upgrading of the DAG and the database.
• 21:42
  SPEAKER_01: And then we have, because of the standards we maintained and some other tooling that we have built, it makes us easy for us to go through that upgrade once a year or twice a year.
• 21:52
  SPEAKER_02: Yeah, yeah, that's great.
• 21:54
  SPEAKER_02: Thanks for going through that story.
• 21:55
  SPEAKER_02: I think that was so interesting to hear.
• 21:57
  SPEAKER_02: And I love the concept of being able to add so much stability and democratization to your Airflow use internally by implementing this standardized infrastructure around it.
• 22:08
  SPEAKER_02: My team spends a lot of time talking to the community about best practices.
• 22:12
  SPEAKER_02: And we're typically not talking on the infrastructure side, more on best practices for the DAG authors, but it's equally as important.
• 22:19
  SPEAKER_02: So it's great to hear that as a company you've really focused in on that.
• 22:22
  SPEAKER_02: And it seems like it's been really successful for you.
• 22:25
  SPEAKER_01: Yeah, it has.
• 22:26
  SPEAKER_01: And I think one of the things, like infrastructure aside, like on the actual like end users who are more concerned with the DAGs, I think we have also kind of limited the amount of patterns and dependencies that we have.
• 22:37
  SPEAKER_01: So that also makes a problem a little bit simpler.
• 22:40
  SPEAKER_01: And so far, we've been able to model things into these different patterns we have.
• 22:44
  SPEAKER_01: And if there is like a new pattern that we have to support, for example, Databricks or ClickHouse or some other database technology, then we take it with the premise that yes, there are there are other pipelines running on the central cluster.
• 22:56
  SPEAKER_01: So uptime, SLA is like a very high priority for us.
• 23:00
  SPEAKER_01: We always know with Python dependency management is like a big thing.
• 23:04
  SPEAKER_01: So we are very cognizant of whenever we introduce new dependencies, how does it affect upgrades and so on.
• 23:10
  SPEAKER_01: And then we try to limit uh the amount of the amount of DAGs we onboard and we try to fit it in some kind of uh reproducible pattern so that it helps other teams as they onboard as well.
• 23:21
  SPEAKER_02: Yeah, yeah, totally makes sense.
• 23:24
  SPEAKER_02: That's great.
• 23:24
  SPEAKER_02: I think that's a good place to wrap up.
• 23:26
  SPEAKER_02: So I always like to end with a standard final question, again, in the spirit of looking forward.
• 23:31
  SPEAKER_02: And that's what would you most like to see from the Airflow project in the near future?
• 23:35
  SPEAKER_01: I think one of the things Airflow has already addressed is this ability to run tasks as remote executors.
• 23:41
  SPEAKER_01: I would want to see like more support around more native integration with cloud providers.
• 23:46
  SPEAKER_01: For example, with AWS, we built our own ECS operator, which wasn't available at that time.
• 23:51
  SPEAKER_01: But on the community side, if like certain operators are easily accessible to us, that would be great.
• 23:56
  SPEAKER_01: And then just I think I'm pretty excited to play with Airflow 3 when it comes out.
• 24:00
  SPEAKER_01: I think there are a lot of things there.
• 24:02
  SPEAKER_01: The the UI refresh, I think it it was time when we needed to change the UI a little bit, make it more modern.
• 24:08
  SPEAKER_01: So I think I'm looking forward to that.
• 24:10
  SPEAKER_01: And also some of the security upgrades.
• 24:11
  SPEAKER_01: I think security is becoming a big and a big part of any running like any large-scale project, especially when you have such a, we have your you work at a large company, different personas, different people have access to different kinds of information.
• 24:25
  SPEAKER_01: I'm also interested in the security aspect to see how Airflow 3 improves that.
• 24:29
  SPEAKER_02: Yeah, those are all great things.
• 24:30
  SPEAKER_02: Well, we are extremely excited for Airflow 3 as well.
• 24:34
  SPEAKER_02: Anand, thank you so much for joining us today.
• 24:36
  SPEAKER_02: It's been really great speaking with you.
• 24:38
  SPEAKER_02: What's the best way for folks to get in touch with you?
• 24:40
  SPEAKER_01: The best way would be LinkedIn.
• 24:42
  SPEAKER_01: So I think in the podcast, if we could drop a link to my LinkedIn profile, that's the best way to reach me.
• 24:48
  SPEAKER_02: Yeah, we will do so for sure in the show notes.
• 24:50
  SPEAKER_02: Well, thanks again, Anand.
• 24:51
  SPEAKER_02: This has been great.
• 24:52
  SPEAKER_01: Thank you.
• 24:56
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 25:02
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 25:06
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.