Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla

Transcript

The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: It's really super critical to be always walking along the side of innovation and being part of communities where you're having discussions about where the landscape and the changes in technology are heading.
• 0:15
  SPEAKER_02: So a big part of my role is to work on a lot of and a lot of areas of innovation.
• 0:23
  SPEAKER_02: Working with technology teams external allows me to have conversations where maybe those discussions are not taking place internally.
• 0:47
  SPEAKER_01: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:52
  SPEAKER_01: Join the Astronomer Road Show for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 1:00
  SPEAKER_01: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:09
  SPEAKER_01: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:14
  SPEAKER_01: Register now.
• 1:15
  SPEAKER_01: The link is in the show notes.
• 1:18
  SPEAKER_01: Hi everyone.
• 1:19
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:21
  SPEAKER_01: I'm your host, Kenton Danis, and today I'm joined by Anu Pabla, who is a principal engineer at the ODP Corporation supporting the VAR Logistics Division.
• 1:31
  SPEAKER_01: Anu, how are you?
• 1:33
  SPEAKER_02: I am doing great.
• 1:34
  SPEAKER_02: And Kenton, thank you.
• 1:36
  SPEAKER_01: Thank you for having me on.
• 1:38
  SPEAKER_02: I appreciate it.
• 1:39
  SPEAKER_01: Yeah, we're really excited to have you here.
• 1:41
  SPEAKER_01: We have a very exciting topic or set of topics for today's podcast that I think will be very interesting to our listeners.
• 1:49
  SPEAKER_01: We're going to talk about why data teams should choose Airflow as an orchestrator, relevant for I think everybody who has tuned into our show before and also its relevance to AI implementations, a hot topic of the day.
• 2:03
  SPEAKER_01: So yeah, very excited on me to have you here.
• 2:06
  SPEAKER_01: Maybe just start us off with some background.
• 2:08
  SPEAKER_01: Can you tell me a little bit more about you and the company you work for?
• 2:12
  SPEAKER_01: What do they do?
• 2:14
  SPEAKER_02: Absolutely.
• 2:15
  SPEAKER_02: So I'm currently serving as a principal engineer for Baya Logistics.
• 2:21
  SPEAKER_02: And it's uh it's a division that provides end-to-end e-commerce fulfillment services.
• 2:28
  SPEAKER_02: So namely the orders for receiving, packaging, shipping, and returns.
• 2:34
  SPEAKER_02: So all of that, and then that's taking place, all those operations are done under 40 fulfillment centers across the United States.
• 2:42
  SPEAKER_02: So I work with a team and leadership supporting all of that through various technology solutions.
• 2:50
  SPEAKER_02: And it's all so the dominant business area is logistics.
• 2:54
  SPEAKER_02: And so that's exactly what we're supporting through technology.
• 2:59
  SPEAKER_01: That sounds great.
• 3:00
  SPEAKER_01: Something that is a very complicated use case.
• 3:02
  SPEAKER_01: I have worked in supply chain analytics in one of my past lives, so not exactly the same, but I think a similar level of just lots of different levels of complexity there and a lot of data to manage, as it turns out.
• 3:17
  SPEAKER_02: It certainly is.
• 3:18
  SPEAKER_02: And with logistics, the use cases are really interesting when it comes to real-time tracking of packages and getting that information back out immediately to your customers.
• 3:30
  SPEAKER_02: So yeah, a lot of interesting challenges there.
• 3:32
  SPEAKER_01: Yeah, that's really cool.
• 3:34
  SPEAKER_01: And tell me a little bit more about what your team does to help out with all that.
• 3:42
  SPEAKER_02: All right.
• 3:43
  SPEAKER_02: Okay, so let's talk about what I pretty much do first, which is I pretty much monitor the technology landscapes, uh, working with the engineers internal to the organization, and then also collaborating with technologists external to the organization.
• 4:01
  SPEAKER_02: And so with the company and the space that we're in with technology and driving the business, it's really super critical to be always walking along the side of innovation and being part of communities where you're having discussions about where the landscape and the changes in technology are heading.
• 4:25
  SPEAKER_02: So a big part of my role is to work on a lot of and a lot of areas of innovation.
• 4:33
  SPEAKER_02: Working with technology teams external allows me to have conversations where maybe those discussions are not taking place internally, and then to bring back that education and knowledge to the organization, and then being able to help not just engineers but businesses, the business division make the right, make the right selection of technologies, frameworks that it's overwhelming to understand what is what is something to consider as a possible solution.
• 5:05
  SPEAKER_02: Education is a big part of my role too.
• 5:08
  SPEAKER_02: I'm such a strong advocate for younger engineers navigating.
• 5:13
  SPEAKER_02: And it's really right now, so it's such an overwhelming tech landscape.
• 5:18
  SPEAKER_02: And I think when you're able to come in and you've got younger engineers coming into technology, providing them guidance and direction to help them understand what they can take on as a new framework, engaging with the new changing landscape of AI, how to walk into this, it's it provides a certain confidence level to move forward.
• 5:41
  SPEAKER_02: And then I think like it's just that mindset of uh when you are educating like just your business and your engineering team, you have success.
• 5:51
  SPEAKER_02: And so you have this mindset of the business ends up taking care of itself because you're you're growing with the technical changes that are taking place in the world.
• 6:02
  SPEAKER_01: Yeah, it makes me think of that.
• 6:04
  SPEAKER_01: I'm not gonna get the attribution right in the moment here, but that diagram that I'm sure everybody has seen about the ML AI kind of data landscape, and it's just like a full slide of logos that are tiny and there's hundreds of them.
• 6:18
  SPEAKER_01: And just to illustrate that the landscape is really complex and there's a lot of different tools.
• 6:23
  SPEAKER_01: And as you're discussing, I think knowing what the right tool is to use for your use case and for your company, and even for you personally, with your background as an engineer, that's all really complicated to figure out and really important.
• 6:39
  SPEAKER_01: So I think that's really cool that you're sitting at that level of helping with that educational aspect of it.
• 6:47
  SPEAKER_01: Yeah.
• 6:47
  SPEAKER_02: Yeah, I think it's it's like what you've just described, the technology stacks, the different types of libraries and frameworks that are emerging right now.
• 6:59
  SPEAKER_02: I think just even for engineers, it's like where do you drive direction and focus?
• 7:04
  SPEAKER_02: And it's really, well, somebody has to go out and innovate and solution and say, oh, this is really good for using in such and such use case.
• 7:13
  SPEAKER_02: So I get to enjoy that part of my role.
• 7:17
  SPEAKER_01: Yeah, I love that.
• 7:19
  SPEAKER_01: And you've touched on this a little bit working with folks who are newer to their careers or new to engineering.
• 7:26
  SPEAKER_01: And I understand you also have a history of working with engineering teams at startups and helping provide that leadership for platform engineering or software development teams for what kind of tools they should be using and how they can implement an effective platform.
• 7:42
  SPEAKER_01: Is there anything else you'd like to share there more generally that you think is relevant to this discussion?
• 7:50
  SPEAKER_02: Yeah, I think with the experiences that I've actually gone through working with organizational challenges when it comes to data and just the period of time working with Airflow itself as well, it's been it's so wonderful to be able to now have this perspective of where we came from building pipelines where the dominant pattern was extract, transform, and load.
• 8:20
  SPEAKER_02: I'm pretty sure every engineer is well aware that's what we do as data engineers, because for a business to even be able to do anything meaningful with data, the preparation and the work that data engineers do, sometimes that's not highlighted.
• 8:40
  SPEAKER_02: And so I've gone from this era of using orchestration engines like Airflow to do those typical types of DAGs implementations.
• 8:50
  SPEAKER_02: And now it's like I get to experience that paradigm shifting into what I feel like is a pass, ingest, and vectorize.
• 9:02
  SPEAKER_02: So we've taken ETL and now we're moving with this AI space.
• 9:09
  SPEAKER_02: And I think the common theme here is that in both in the shift as we go from ETL to pass ingest vectorize, is what I term it data, you've got to have meaningful data.
• 9:24
  SPEAKER_02: And so with the fact that you have to go through this process regardless of the era or the technology we're serving, you have an orchestration engine like Airflow come in.
• 9:39
  SPEAKER_02: And it's so wonderful to see how it's able to take in the frameworks that are currently out on the market that a lot of developers go in and start working with immediately, like the Langchain, the Lama Indexes, the Langsmith.
• 9:55
  SPEAKER_02: And so intuitively you would lean into that.
• 9:58
  SPEAKER_02: But then the patterns that are emerging from those types of use cases are how do you orchestrate all of that?
• 10:06
  SPEAKER_02: How do you orchestrate the LLM execution, the retrieval, the observability of how the LLM behaves?
• 10:13
  SPEAKER_02: And so this is where I think there's this beautiful marriage of an open source technology like Airflow, leaning straight into AI and being able to take those types of different use cases and be able to orchestrate that, making development efforts so much more easier for engineers.
• 10:34
  SPEAKER_02: I mean, it you truly have to go through the hands-on development of building out one of these pipelines where you spend weeks just cleaning data.
• 10:46
  SPEAKER_02: And then if there is a process in your step that's that that doesn't work out so well, there's a failure in your pipeline.
• 10:53
  SPEAKER_02: But now you've got something like Airflow that you can come in, break down these types of tasks, use the libraries, and because like Airflow is a natural Pythonic library, it leans into a lot of the AI libraries that I've just mentioned, which are all built in Python too.
• 11:13
  SPEAKER_02: And so it's just such a wonderful marriage to see that and to see how over time this shift has occurred, where still the end goal is clean data that's needed.
• 11:26
  SPEAKER_02: Whatever you're building, AI analytics, a bot, being able to do real-time AI inference, all of that comes.
• 11:36
  SPEAKER_02: How do you orchestrate your pipeline?
• 11:39
  SPEAKER_02: And how do you go through the process, which is quite different from the ETL to the passenger specterize?
• 11:45
  SPEAKER_02: And how do you do that?
• 11:46
  SPEAKER_02: And it's the constant is Airflow being able to do and manage all of those use cases.
• 11:54
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:04
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:15
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:24
  SPEAKER_01: It makes me think a lot of my own story.
• 12:27
  SPEAKER_01: And when I was in grad school studying data science, so this is long before LLMs were what they are today or in a usable state.
• 12:37
  SPEAKER_01: There was no Gen AI story at the time.
• 12:41
  SPEAKER_01: We still had this concept of when doing what now I would consider more like traditional machine learning, so modeling of data and trying to use those results, of this concept of like garbage in, garbage out, of that if you don't have the right data in the right state at the right time, you just can't do anything.
• 13:02
  SPEAKER_01: And since then, this is years and years ago, like the landscape has changed so much, but that story has really not.
• 13:08
  SPEAKER_01: And I think I'm probably not the only one where, you know, after I graduated, my first job was consulting in data science, but I really almost immediately ended up in data engineering, which was a happy thing for me.
• 13:21
  SPEAKER_01: I ended up using Airflow almost immediately.
• 13:23
  SPEAKER_01: And then that led me to this job at astronomer.
• 13:26
  SPEAKER_01: But I think the reason for that is that a lot of the work is just in that data preparation because you can't be doing even a traditional, just simple like logistic regression on data if you don't have the right data, much less implementing something a lot more complex, like an AI bot or whatever it is on the end state for companies.
• 13:46
  SPEAKER_01: So it is that is a really interesting constant that has stayed through that whole evolution.
• 13:53
  SPEAKER_01: Tell me a little bit more about Airflow.
• 13:56
  SPEAKER_01: So you've touched a lot on why Airflow is good for this use case.
• 14:01
  SPEAKER_01: You've seen a ton of teams and organizations build data and engineering platforms from scratch and try and adapt to this rapidly changing landscape.
• 14:11
  SPEAKER_01: Anything else you want to say on why teams choose Airflow?
• 14:15
  SPEAKER_01: You've touched a little bit on its flexibility and kind of the concept of everything being written in Python, which I think are huge, but expand on that a little more for me.
• 14:26
  SPEAKER_02: Where do I start?
• 14:27
  SPEAKER_02: So I think I've had the front row seat watching all of these orchestrations and the evolution of the orchestration movements.
• 14:35
  SPEAKER_02: I've seen so many waves come through where there's this promise to simplify pipelines.
• 14:43
  SPEAKER_02: And also it's not just embracing an orchestration engine.
• 14:48
  SPEAKER_02: You've also got to consider the cost of operation for running these engines as well.
• 14:53
  SPEAKER_02: That's a big factor.
• 14:55
  SPEAKER_02: As engineers, we just lean into oh, that technology is hot, I'm gonna go with that.
• 15:00
  SPEAKER_02: But also the cost of compute has to be considered as well.
• 15:03
  SPEAKER_02: That's a that's a it's a balance for an engineer to think like that.
• 15:08
  SPEAKER_02: But I think with all that I've seen and experienced, and I don't want to go into the names or the orchestration engines I've had to play with, but Airflow came out to be the front leader.
• 15:20
  SPEAKER_02: It's uh it's open source, it's backed by the community.
• 15:22
  SPEAKER_02: Look at the GitHub star ratings.
• 15:24
  SPEAKER_02: There's nothing more that I need to say.
• 15:26
  SPEAKER_02: There's so much support for the technology out there in the community, and it's not just something that is emerging out of a sort of a solution that's baked up by a community of devs and then not supported by a community and then just licensed off and music.
• 15:47
  SPEAKER_02: This is this is an Apache foundation solution.
• 15:50
  SPEAKER_02: And so that's one of the reasons me personally, I back it up.
• 15:55
  SPEAKER_02: But then I'd have to walk in and say, in comparison to the other orchestration engines, Airflow, it's just the right balance between the flexibility and the scalability and the observability.
• 16:09
  SPEAKER_02: So, as I was just talking about the paradigm shift into working with uh the AI use cases, I can immediately construct a DAG that's gonna go out and it's gonna connect to my pane cones, my quadrants.
• 16:29
  SPEAKER_02: I can immediately have that connectivity available in a DAG.
• 16:33
  SPEAKER_02: I can immediately start working on calling multiple different types of LLMs in an entire DAG and then also having the ability to do the observability.
• 16:46
  SPEAKER_02: So Airflow allows for the lot, the work that needs to be done by engineers when you look at other orchestration engines is the retries, the alerting, the failure points.
• 16:59
  SPEAKER_02: And so building in this ecosystem, you're developing a solution that works with your traditional ETL, now your AI use cases, and it's just emerging, and it's it's almost like a self-healing ecosystem where you can run your pipes.
• 17:14
  SPEAKER_02: And should, of course, pipes break, you have the ability to go in there and immediately isolate where the problems are.
• 17:22
  SPEAKER_02: It's an amazing, you know, and I think that's one of the elegant things about that I appreciate about Airflow is some of the other platforms don't have that UI interface, so that you could have a non-technical person come in and say, Oh, okay, I'm not exactly sure like what is wrong with this pipeline, but with the interface provided by Airflow, you can have a non-technical person see, oh, there's a failure with an API call to an external party.
• 17:54
  SPEAKER_02: Somebody go deal with that.
• 17:56
  SPEAKER_02: So a lot of other solutions don't allow for that.
• 17:59
  SPEAKER_02: So I think that's definitely a strength from the Airflow side.
• 18:02
  SPEAKER_02: And I think the connectivity to all the data sources, the movement with the AI SDKs that are coming out that work with Airflow, I think that's going to be a huge plush.
• 18:17
  SPEAKER_02: And then you've got just this overall idea of building these AI running pipelines.
• 18:26
  SPEAKER_02: It just, it's such a natural fit and a natural solution to go with as you as we start to build more towards the AI landscape.
• 18:37
  SPEAKER_01: Yeah, absolutely.
• 18:38
  SPEAKER_01: I love the way that you're framing this of a lot of the things that have made Airflow really popular even before AI use cases were big, what I would consider the core tenets of Airflow, of being Pythonic, having this inner of the vast array of integrations with other tools and making that really easy so that Airflow sits at the center and it's this plug and play where when something new comes up, it's fine.
• 19:03
  SPEAKER_01: You just adjust and now you're connecting to this vector database instead of this other vector database.
• 19:08
  SPEAKER_01: I think is why we've seen Airflow be so popular and is continuing to grow with AI use cases in particular.
• 19:18
  SPEAKER_01: And yeah, I think it's really exciting to see with such a broad community all of the things that pop up around that.
• 19:23
  SPEAKER_01: You mentioned AI SDKs.
• 19:25
  SPEAKER_01: Astronomer has built our own AI SDK that is open source.
• 19:29
  SPEAKER_01: So anybody can go and download that and use that with their Airflow pipelines.
• 19:33
  SPEAKER_01: We at Astronomer will be talking a lot more about that kind of in the coming months.
• 19:37
  SPEAKER_01: But there's certainly no shortage of solutions there popping up within that Airflow ecosystem to use it for kind of whatever the use case might be.
• 19:47
  SPEAKER_02: That's awesome.
• 19:48
  SPEAKER_02: Yeah, looking forward to that.
• 19:49
  SPEAKER_01: Yeah.
• 19:50
  SPEAKER_01: I think that's a great segue into my final question for folks, which is always what would you most like to see from the Airflow project in the near future?
• 20:00
  SPEAKER_01: Okay.
• 20:01
  SPEAKER_02: So I think with the experiences that I'm going through right now, which is I have to get involved with building out a lot of different types of pipelines.
• 20:14
  SPEAKER_02: And in the different use cases that I'm looking at, especially with the focus on AI, I would love to see Airflow evolve into the first class citizen for AI native workloads.
• 20:28
  SPEAKER_02: So some of the challenges that I deal with, I can see like an AI orchestration layer that sort of understands like an LLM task pattern straight out of the box.
• 20:40
  SPEAKER_02: So I can give you an example.
• 20:41
  SPEAKER_02: In many use cases, we have this semantic being able to take a semantic search to understand the intent of what a user is asking for before you execute the rest of a flow.
• 20:54
  SPEAKER_02: So taking something like that and then being able to create a solution out of the box where a lot of that coding is massed.
• 21:04
  SPEAKER_02: And I know a lot of the libraries are doing that right now.
• 21:07
  SPEAKER_02: One of my favorites is definitely the Llama Index.
• 21:10
  SPEAKER_02: It's amazing for all of the solutions I need to build out, like the agent workflows.
• 21:16
  SPEAKER_02: And then just that word alone, agent workflow.
• 21:20
  SPEAKER_02: The word workflow should lean into the fact that something could be built out in the astronomer AI workspace where you can give engineers a sort of a plug-and-play experience of, oh, you want to talk to five LLMs at once just to see what kind of response is returned for a single prompt because every LLM is going to behave differently.
• 21:42
  SPEAKER_02: And so that kind of capability, I think, is going to supercharge development efforts for the AI data engineering space.
• 21:53
  SPEAKER_01: Yeah, absolutely.
• 21:54
  SPEAKER_01: I love that answer and very timely.
• 21:56
  SPEAKER_01: There was literally just yesterday at the time of recording, not the time of publishing, but there was a conversation on the Airflow dev list about adding functionality for kind of human in the loop use cases.
• 22:09
  SPEAKER_01: So where people need to be interacting with or intervening in or whatever it may be, the workflows in some way to kind of manage the interaction with LLM.
• 22:19
  SPEAKER_01: So I think that's very top of mind for a lot of folks in the Airflow community.
• 22:25
  SPEAKER_01: It's certainly top of mind for Astronomer and our teams that are working on developing the open source project and within Astros, our managed service on adding more functionality around AI use cases.
• 22:36
  SPEAKER_01: But yeah, I think that's a great answer.
• 22:39
  SPEAKER_01: Okay, Anu, it's been so nice speaking with you today.
• 22:43
  SPEAKER_01: What is the best way for folks to get in touch with you?
• 22:46
  SPEAKER_01: You can reach me by my LinkedIn.
• 22:48
  SPEAKER_01: Okay, perfect.
• 22:49
  SPEAKER_01: We will throw that in the show notes.
• 22:51
  SPEAKER_01: Thank you again for coming on.
• 22:52
  SPEAKER_01: This has been a really interesting conversation.
• 22:54
  SPEAKER_01: And yeah, it's been great to have you.
• 22:56
  SPEAKER_01: Thank you.
• 22:57
  SPEAKER_01: I appreciate the time.
• 22:58
  SPEAKER_01: Yeah.
• 22:59
  SPEAKER_01: Thanks, Anu.
• 23:03
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:09
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 23:13
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.