Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya

Transcript

Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: We have made DAG factories for every team and developers can just invoke those DAG factories and make sure that they're just sticking to their task of thinking about the architectural decisions in the pipelines and how to bring the data, how best to transform it.
• 0:13
  SPEAKER_01: No need to worry about the power team DAGs and labels and all of that.
• 0:17
  SPEAKER_01: And we just make use of the DAG factory, applied labels itself to basically reuse those labels and put them on the Kubernetes pods so as to enable the team-based tracking and cost attribution.
• 0:34
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 1:01
  SPEAKER_02: Plus, you can enter to win an exclusive workshop with Mark Lumberti.
• 1:05
  SPEAKER_02: Don't miss out.
• 1:06
  SPEAKER_02: The survey link is in the show notes and closes November 21st.
• 1:13
  SPEAKER_02: Hi everyone.
• 1:14
  SPEAKER_02: Welcome to the Data Flowcast.
• 1:16
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Kunal Bhajrachaya, who is a senior manager of data platform engineering at PepsiCo.
• 1:25
  SPEAKER_02: Kunal, how are you doing?
• 1:26
  SPEAKER_01: I'm doing good.
• 1:27
  SPEAKER_01: Thank you.
• 1:27
  SPEAKER_01: How are you doing?
• 1:28
  SPEAKER_02: I'm good.
• 1:29
  SPEAKER_02: I'm good.
• 1:30
  SPEAKER_02: Enjoying the last little bit of summer here in the US.
• 1:34
  SPEAKER_02: So yeah, it's good.
• 1:36
  SPEAKER_01: We are happy to see summer's off.
• 1:38
  SPEAKER_01: So we are officially into the monsoon season.
• 1:41
  SPEAKER_02: Okay.
• 1:42
  SPEAKER_02: All right.
• 1:42
  SPEAKER_02: So it's already started raining there.
• 1:45
  SPEAKER_02: Yep.
• 1:45
  SPEAKER_02: Yep.
• 1:46
  SPEAKER_02: Sure, that's a nice break.
• 1:48
  unknown: Yeah.
• 1:48
  SPEAKER_00: Definitely.
• 1:49
  SPEAKER_02: Okay.
• 1:50
  SPEAKER_02: Let's go ahead and dive in.
• 1:51
  SPEAKER_02: So for today's podcast, we're going to discuss how Pepsi's data platform team manages airflow for their internal users.
• 1:58
  SPEAKER_02: I think for this one, our listeners are probably already familiar with what PepsiCo does.
• 2:04
  SPEAKER_02: But maybe let's start with some background on what your team does and how it serves your broader company goals.
• 2:10
  SPEAKER_01: Hello, everyone.
• 2:11
  SPEAKER_01: I'm Kunal Bhattucharya.
• 2:13
  SPEAKER_01: I'm a senior manager of the Data Platform Engineering at PepsiCo Digital Commerce.
• 2:18
  SPEAKER_01: Currently, I work as an individual contributor within the architecture and security group of the data platforms team.
• 2:26
  SPEAKER_01: As part of the data platforms team, our responsibilities are spread across the board.
• 2:31
  SPEAKER_01: We are responsible for designing, maintaining, and extending the capabilities of our data platform so that we can enable the data engineering, business intelligence, and the data science teams to ingest, clean, and transform data as per the business requirements, and then generate meaningful and actionable insights via the dashboards for various stakeholders like finance marketing or drive competitive advantage via data science models.
• 2:58
  SPEAKER_01: Aside from extending the capabilities of our data platform, we are also responsible for enhancing the developer experience, basically driving developer delight by introducing new capabilities, mostly which include enabling the latest advancements in the tech and the data engineering space.
• 3:16
  SPEAKER_01: Some examples would include Rust-based Python dependency management, then the separation of compute and storage with technologies like iceberg.
• 3:24
  SPEAKER_01: All of this while keeping the lights on.
• 3:27
  SPEAKER_01: So just generally making sure our platform is stable, no systems are down, and also making sure that the security of our platform is retained.
• 3:36
  SPEAKER_02: That's a big job that your team is filling, lots of really important stuff in there.
• 3:42
  SPEAKER_02: We'd love to dig in a little bit more to the data platform.
• 3:46
  SPEAKER_02: So tell me a little bit about how you've set that up.
• 3:49
  SPEAKER_02: And I assume Airflow fits in there somewhere.
• 3:51
  SPEAKER_02: So maybe we can talk about that as well.
• 3:53
  SPEAKER_01: Yep, absolutely.
• 3:54
  SPEAKER_01: Airflow is one of the spines of our platform.
• 3:58
  SPEAKER_01: So just to go from the start, so we have Snowflake as our primary cloud data warehouse hosted on AWS and GR.
• 4:06
  SPEAKER_01: So we follow that up with open source DBT as the primary tool for transformations and business logic application.
• 4:13
  SPEAKER_01: And then we have Airflow, which is the primary orchestration tool for all of our data transformation pipelines, which we have hosted on a distributed containerized cluster that's orchestrated by Kubernetes.
• 4:25
  SPEAKER_01: And finally, we also make use of grid expectations and Monte Carlo data for our data governance and insights.
• 4:33
  SPEAKER_02: Interesting.
• 4:33
  SPEAKER_02: And then Airflow is, I liked the term you use, the spine that runs through it all and ties it all together.
• 4:39
  SPEAKER_01: Yes, Airflow is actually the spine, making sure that all of our pipelines are getting all the memory that they need and running on schedule.
• 4:48
  SPEAKER_01: We also have Python scripts integrated with our ABT and our Snowflake jobs.
• 4:53
  SPEAKER_01: So all of this is basically held together by Airflow.
• 4:57
  SPEAKER_01: So it's a very core component of our system.
• 4:59
  SPEAKER_02: Okay, great.
• 5:00
  SPEAKER_02: And do you have Airflow deployed on Kubernetes or something like that?
• 5:03
  SPEAKER_02: Or how does it work?
• 5:05
  SPEAKER_01: Yes.
• 5:06
  SPEAKER_01: So we have deployed the open source edition of Airflow on Kubernetes orchestrated containerized environment.
• 5:12
  SPEAKER_01: We are making use of the official Airflow Helm chart to basically pull in the Airflow image and then we are deploying it on the Kubernetes environment.
• 5:21
  SPEAKER_02: Okay.
• 5:21
  SPEAKER_01: So every DAGs get their own resources.
• 5:24
  SPEAKER_02: Okay, got it.
• 5:25
  SPEAKER_02: That's great.
• 5:26
  SPEAKER_02: And you mentioned a little bit at the beginning that one of the things that your team does is help democratize airflow usage so you make it easier for developers across Zco or across different teams to write DAGs or I imagine just interact with your data platform.
• 5:44
  SPEAKER_02: What sort of solutions there have you implemented around airflow to help make that easier for folks?
• 5:50
  SPEAKER_01: Yeah, so I'd like to call out a couple of approaches that we have taken there.
• 5:54
  SPEAKER_01: So the first thing is making use of the official Airflow Helm chart to develop our own local developer platform, very similar to Astronomer's, I think Astro is the platform, if I'm not wrong.
• 6:06
  SPEAKER_02: Yeah, Astro.
• 6:06
  SPEAKER_01: So very similar to that kind of a platform.
• 6:08
  SPEAKER_01: So we are basically pulling the latest official Airflow Helm chart, and we are enabling the developers to spin up their own isolated developer environments on their local systems.
• 6:18
  SPEAKER_01: What this has done is basically provided every developer with an isolated, decentralized environment for them to experiment, play around with, and before they can actually think about promoting these to higher environments where there are more dependencies involved with other teams.
• 6:33
  SPEAKER_01: At the same time, we've also tried to keep the security paramount by allowing them to mount the credentials and all kinds of secrets and keys as Kubernetes secrets.
• 6:43
  SPEAKER_01: The secret management part is kept as closely mimicking the higher environments as possible.
• 6:48
  SPEAKER_01: That's one application we have done there.
• 6:51
  SPEAKER_01: And we've also enabled another version.
• 6:54
  SPEAKER_01: We call it the pre-staging.
• 6:56
  SPEAKER_01: So in case this local developer platform is not available because of some network policy changes on the local systems, or something has gone wrong with the local environment, and their developers are not able to spin it up, or they just want an additional environment to test their changes on.
• 7:12
  SPEAKER_01: We also have a PR environment that's basically making use of the GitSync feature of Airflow.
• 7:18
  SPEAKER_01: So that's an additional testing playground that we have enabled the developers.
• 7:21
  SPEAKER_01: So this PR environment is closely mirroring the design decisions of the higher environments, like staging and prod.
• 7:28
  SPEAKER_01: So they get a very close resemblance to how their DAGs would function on the higher environments.
• 7:34
  SPEAKER_01: And finally, I just like to mention the DAG factory setup that we have.
• 7:38
  SPEAKER_01: So we don't want our developers to get boiled down with things like boilerplate code and what labels to apply and all of that.
• 7:45
  SPEAKER_01: So developers hate doing that.
• 7:46
  SPEAKER_01: I've become the data engineering side, and we just want to work with the business logic.
• 7:51
  SPEAKER_01: So to enable that, what we have done is we have made DAG factories for every team, and developers can just invoke those DAG factories and make sure that they're just sticking to their task of thinking about the architectural decisions and the pipelines and how to bring the data, how best to transform it.
• 8:06
  SPEAKER_01: No need to worry about the part team DAGs and labels and all of that.
• 8:09
  SPEAKER_01: And we just make use of the DAG factory, applied labels itself to basically reuse those labels and put them on the Kubernetes pods so as to enable the team-based tracking and cost attribution.
• 8:23
  SPEAKER_02: So that sounds like you've taken care of a lot of the things that I often think of as the most tricky for developers to work with Airflow.
• 8:29
  SPEAKER_02: Like local development is often really difficult to set up and managing the connections to the external systems as well as just learning how all of the operators work.
• 8:39
  SPEAKER_02: So you're maintaining all of that for people so they can skip over all of that difficulty and just get to, as you said, the point where they're implementing business logic, which sounds great.
• 8:49
  SPEAKER_01: Yeah, that's our continuous goal to make sure in a distributor setup, it can be a little tricky if you have to take care of the entire thing.
• 8:56
  SPEAKER_01: We don't have to worry about the business side of things.
• 8:58
  SPEAKER_01: So we take care of the underlying system architecture while there are dedicated teams that are not having to think about what's happening under the hood.
• 9:06
  SPEAKER_01: So I think that makes it easy for everyone.
• 9:08
  SPEAKER_02: And with the DAG factor, you've implemented, I didn't catch if you said how many kind of patterns do you provide to people that they can just plug into?
• 9:16
  SPEAKER_02: Like, is everybody using the same set of operators or doing the same types of things in their DAGs?
• 9:22
  SPEAKER_02: Or how often do you have to update that because a new use case has come in or someone wants to do something different?
• 9:28
  SPEAKER_01: We get requirements to update the basic layer pretty regularly.
• 9:32
  SPEAKER_01: That's expected when you're consistently extending the capabilities, not just technologically, but also from the business side of things.
• 9:38
  SPEAKER_01: We are constantly onboarding newer sets of data, so newer data sources and new requirements.
• 9:43
  SPEAKER_01: When you say what kind of capabilities, how are we enabling the team?
• 9:47
  SPEAKER_01: So basically, every team has their own respective DAG factory that they can make use of.
• 9:51
  SPEAKER_01: That's not a restriction, that's more of a capacity that we have enabled them.
• 9:55
  SPEAKER_01: So the moment that they're going and working with that DAG factory, it's implied that any DAGs that they develop with those DAG factories are going to come with a certain set of labels and memory resources.
• 10:08
  SPEAKER_01: They don't have to worry about that.
• 10:10
  SPEAKER_01: And we continuously fine-tune these based on what kind of resource replication are we seeing on our visualization end.
• 10:17
  SPEAKER_01: So if you're seeing that a particular team has DAGs that are very memory intensive, so we can just bump up the resources for just that team, keeping the other DAGs intact.
• 10:26
  SPEAKER_01: So we don't over-provision or under provision, and teams are not basically fighting over the same slice of the pile.
• 10:34
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 10:44
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:56
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astonomer.io.
• 11:04
  SPEAKER_02: So less about saying you can only use this set of operators that we've put into a DAG factory and more about managing the resource utilization so that people don't have to worry about like how the DAGs are actually running.
• 11:16
  SPEAKER_01: We've kept the operators usable for everyone.
• 11:19
  SPEAKER_01: We don't want to restrict teams based on the operators they can use because I think that's more closely related to making use of the functionalities of Airflow.
• 11:28
  SPEAKER_01: But the DAG factory essentially takes care of the DAG documentation, the DAG labeling, the resource parameters for the Kubernetes ports.
• 11:36
  SPEAKER_01: So those are the kinds of things that we are managing with the DAG factory.
• 11:39
  SPEAKER_01: So any changes to the operators come in separately.
• 11:42
  SPEAKER_01: They can be taken up by the respective data engineering teams or by us, depending on how the requirement is basically put forward.
• 11:49
  SPEAKER_01: But these are the things that the developers don't even have to think about.
• 11:52
  SPEAKER_01: This is just done.
• 11:53
  SPEAKER_02: Got it.
• 11:54
  SPEAKER_02: Got it.
• 11:54
  SPEAKER_02: Yeah, that sounds like it would be very helpful.
• 11:57
  SPEAKER_02: And speaking of resource utilization, so you mentioned earlier that you're running Airflow on Kubernetes.
• 12:05
  SPEAKER_02: I'm guessing that for a company of your size that you're managing Airflow at a pretty large scale.
• 12:11
  SPEAKER_02: Can you talk about that a little bit and whether there's anything that you've done to optimize your Airflow setup?
• 12:17
  SPEAKER_01: So this optimization of the Airflow setup and not letting the Kubernetes costs spiral out of control was one of the primary reasons why we actually went with this DAG factory setup.
• 12:27
  SPEAKER_01: So when we have downstream visualizations for all of the pods on things like Datadog, we can easily see per team what is the memory utilization look like for all of the DAGs for that particular team.
• 12:40
  SPEAKER_01: Continuous insight into those kinds of matrix actually helps us effectively scale up and down our pods adequately without hampering the performance of the DAGs, or even in certain cases, finding out the problems before they strike us, right?
• 12:53
  SPEAKER_01: If you're seeing that DAGs are continuously taking up like close to 60-70% of the memory, that's a good time to start looking into whether we should bump up those parameters.
• 13:02
  SPEAKER_01: I think the labeling and making effective use of the pod mutation hooks that Airflow provides has been very critical to democratizing the entire usage of our platform across the entire team while also keeping close watch over the nitty-gritties of the system engineering.
• 13:19
  SPEAKER_01: I'd like to think.
• 13:20
  SPEAKER_01: So this has been helpful to keep the costs in check.
• 13:23
  SPEAKER_01: The other side I'd like to also mention is with the GitSync PR environment that I talked about and the local development environment, with these being throwaway environments, we have developers that do not have to spend too much of time clashing with other teams at the nascent stage of their development, while we also don't have to spend extra dollars to basically host those kinds of development environments because dev environments are typically throwaway environments.
• 13:46
  SPEAKER_01: People just go there, experiment it out.
• 13:49
  SPEAKER_02: Yeah, that makes total sense from a cost optimization standpoint.
• 13:53
  SPEAKER_02: And what about from like a performance optimization standpoint?
• 13:56
  SPEAKER_02: Is there anything that you've had to do with Airflow on that side to make sure that it keeps working for everybody?
• 14:03
  SPEAKER_01: So from a performance optimization standpoint, what we do is first of all, we separated out the DBT.
• 14:10
  SPEAKER_01: So basically, Airflow takes care not just of ingesting data in our system, but also of the transformation layer.
• 14:16
  SPEAKER_01: So when you think about the transformation layer, the DBT comes into the picture as well.
• 14:20
  SPEAKER_01: All of the DBT models and seeds and macros that are there, those are also getting pulled into our Airflow pods via operators that we have.
• 14:30
  SPEAKER_01: We are kind of extending the operators that Apache Airflow currently provides to basically pull in our custom DBT images and then running the DAGs on top of them, right?
• 14:39
  SPEAKER_01: So one of the main things to keep optimize the performance is we have the transformation layer tends to be very heavy because a lot of data has been ingested at the time.
• 14:49
  SPEAKER_01: And the transformations typically in a company of our size, those are very costly, both in size and in the volume.
• 14:56
  SPEAKER_01: Right.
• 14:56
  SPEAKER_01: So what we have done is we have actually isolated out the DBT engine, and we are basically working with the data engineering and the BI teams to say, hey, you guys have complete ownership of these respective DBT engines.
• 15:08
  SPEAKER_01: You are the team one DBTA engine, you are the team two DBT engine, and you manage these environments completely.
• 15:14
  SPEAKER_01: You manage the models that are there.
• 15:17
  SPEAKER_01: So at the time when we are pulling those in, so we are basically the team A is not interfering with the memory allocated to the team B pods, right?
• 15:28
  SPEAKER_01: So that way what we have done is we have again separated out.
• 15:30
  SPEAKER_01: So I think just to get this down to a nutshell, isolation of the pods, isolation of the running execution environments has been important to basically keep the performance smooth.
• 15:42
  SPEAKER_02: Yeah, yeah, definitely.
• 15:43
  SPEAKER_02: It does, it does make sense.
• 15:45
  SPEAKER_02: And I think especially I've seen this a lot with companies who are using particularly DBT and Airflow.
• 15:51
  SPEAKER_02: I think there's a lot of performance optimization that happens to use those two together.
• 15:55
  SPEAKER_02: My team has been working on some Cosmos resources recently on optimization for folks who are using that as the tie together between DBT and Airflow.
• 16:03
  SPEAKER_02: So yeah, I think it totally makes sense.
• 16:05
  SPEAKER_02: And it sounds like you've implemented that in a way that works pretty well and is pretty stable.
• 16:10
  SPEAKER_02: Yeah, cool.
• 16:11
  SPEAKER_02: Well, thank you so much for walking through all of that.
• 16:14
  SPEAKER_02: This has been really interesting.
• 16:15
  SPEAKER_02: I want to jump to my standard final question for everybody, which is what would you most like to see from the Airflow project in the near future?
• 16:22
  SPEAKER_01: So, a fair call out before I get into that question.
• 16:25
  SPEAKER_01: We are still exploring Airflow 3.
• 16:27
  SPEAKER_01: So I apologize in ahead of time if any of the points I state have already been solved in Airflow 3.
• 16:33
  SPEAKER_01: Two things come to mind.
• 16:34
  SPEAKER_01: One would be a better way to assign grants to new roles.
• 16:38
  SPEAKER_01: So if I'm speaking strictly of Airflow less than three, this is a kind of a tedious process where we have to basically select the grants on the UI manually and then based on the string keyword matches.
• 16:48
  SPEAKER_01: And if you think about a scenario where there are thousands of DAGs and I want to give access to a particular team to just 25 DAGs, that means I have to manually invoke those 25 DAGs and get them.
• 17:00
  SPEAKER_02: It's a bit of a manual process.
• 17:01
  SPEAKER_02: Yeah.
• 17:02
  SPEAKER_01: Yep, that's a manual process.
• 17:03
  SPEAKER_01: And if there's a check boxes, that would be probably easier, but having to type those down and then search from the list is something that we are looking forward to.
• 17:12
  SPEAKER_01: I'm sure the FRO team will get to that sometime.
• 17:14
  SPEAKER_01: And the other thing would be improvement in the database model.
• 17:18
  SPEAKER_01: We have seen situations where our database size ballooned up to a very significant amount because of how the database has been modeled, specifically the case where human-defined strings like DAG IDs are part of the primary key.
• 17:32
  SPEAKER_01: So those grow up to 32 bits in size, and then they take up a lot of space.
• 17:37
  SPEAKER_01: And then we have to manually go ahead and clean up the database to get everything back in order.
• 17:41
  SPEAKER_01: So these are the two things that we have felt would be great to get some solution in the near future.
• 17:46
  SPEAKER_02: That's really interesting.
• 17:47
  SPEAKER_02: I actually couldn't tell you off the top of my head what has changed in terms of the database modeling in Airflow 3.
• 17:53
  SPEAKER_02: I know it's a little different, obviously with the major change, but I don't know whether it would solve the issue you're getting at.
• 17:59
  SPEAKER_02: So something to look into.
• 18:00
  SPEAKER_02: And if not, I think definitely uh we see requests all the time for folks looking for best practices on how to manage their database, especially over time as it grows.
• 18:10
  SPEAKER_02: So I think that's a common request or point of stickiness for folks using Airflow at larger scales.
• 18:16
  SPEAKER_02: Yeah.
• 18:17
  SPEAKER_02: Okay, great.
• 18:18
  SPEAKER_02: Well, thank you so much, Kunal.
• 18:19
  SPEAKER_02: It's been really nice speaking with you.
• 18:21
  SPEAKER_02: What is the best way for folks to get in touch with you?
• 18:23
  SPEAKER_01: I would say LinkedIn.
• 18:24
  SPEAKER_01: I'm mostly active over there.
• 18:26
  SPEAKER_01: So just hit me up on LinkedIn.
• 18:28
  SPEAKER_01: If you type that in with Papsico, I think you'll get me.
• 18:30
  SPEAKER_02: Okay.
• 18:31
  SPEAKER_02: We'll throw it in the show notes as well.
• 18:33
  SPEAKER_02: So people will have an easy link to it.
• 18:35
  SPEAKER_02: Okay, well, Kunal, thank you again.
• 18:36
  SPEAKER_02: This has been really interesting.
• 18:38
  SPEAKER_01: Thank you so much for inviting me to Data Flowcast.
• 18:40
  SPEAKER_01: It's been a great pleasure talking to you.
• 18:43
  SPEAKER_02: Great.
• 18:46
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 18:52
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 18:56
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.