Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Modern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi

Transcript

Modern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: I am a big fan of Kubernetes pod operator.
• 0:02
  SPEAKER_01: I believe in decoupling everything.
• 0:04
  SPEAKER_01: But there is a trade-off because if you want to use all the providers and all the connections and hooks that Airflow provides, you absolutely have to have some amalgamation.
• 0:14
  SPEAKER_01: So that's a trade-off that depends on every other use case.
• 0:17
  SPEAKER_01: But highly recommend if you have a core business logic that does not use anything airflow related, it's better to Dockeriz it and have it separately.
• 0:25
  SPEAKER_01: Those are the two DAG pipeline and design principles level things.
• 0:48
  SPEAKER_02: Hi everyone.
• 0:49
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:51
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Bavani Ravi, who is an independent software consultant and Apache Airflow champion.
• 0:58
  SPEAKER_02: Bivani, it's great to have you on.
• 1:00
  SPEAKER_02: How are you?
• 1:00
  SPEAKER_01: I'm super excited, Kenton.
• 1:02
  SPEAKER_01: Thank you for having me here.
• 1:03
  SPEAKER_01: It's been a long view, I guess.
• 1:06
  SPEAKER_02: We have, I think, known each other for quite a while, or at least been in the same space for quite a while.
• 1:11
  SPEAKER_02: So it's really fun to get to have this conversation.
• 1:14
  SPEAKER_02: I think for our listeners, we'll give a short overview for today's podcast.
• 1:18
  SPEAKER_02: We're going to discuss Airflow best practices, which is an ever-popular topic.
• 1:23
  SPEAKER_02: But before we dive into that, I want to start with some background on you.
• 1:26
  SPEAKER_02: So tell me what is your current role and where does Airflow fit into that?
• 1:30
  SPEAKER_01: So, like you said, I'm an independent software consultant, a freelancer, if you might say.
• 1:35
  SPEAKER_01: I recently started something called as Flow Experts, basically helping companies adopt better data pipelining and integration practices.
• 1:43
  SPEAKER_01: And now everybody also asks for AI.
• 1:45
  SPEAKER_01: So that's also in the bag.
• 1:46
  SPEAKER_01: A mix of all of this.
• 1:48
  SPEAKER_01: And Airflow is the bread and butter of what I do, I can say.
• 1:52
  SPEAKER_01: So I advocate for open source.
• 1:55
  SPEAKER_01: So all my clients come to me and say, I want to adopt Airflow, and can you help me with it?
• 2:00
  SPEAKER_01: That's pretty much the service I do.
• 2:03
  SPEAKER_02: Okay, I love this.
• 2:04
  SPEAKER_02: I've probably told this story on the podcast before, but before I came to Astronomer, I worked in consulting as a data engineering consultant.
• 2:11
  SPEAKER_02: And that was how I first learned Airflow was helping a company implement it.
• 2:14
  SPEAKER_02: So it's really cool that you've gotten to continue to do that for so many years.
• 2:19
  SPEAKER_02: And speaking of, I understand you've been working with Airflow since about 2018.
• 2:23
  SPEAKER_02: Is that right?
• 2:24
  SPEAKER_02: Yeah.
• 2:25
  SPEAKER_02: So pretty early days for the Airflow project and quite some time since has passed since then.
• 2:32
  SPEAKER_02: How have you observed the ecosystem evolve over those years?
• 2:36
  SPEAKER_01: Oh, I had to start right from the first, right?
• 2:38
  SPEAKER_01: 2018 uh was when I first started.
• 2:41
  SPEAKER_01: And around that time, it was again 1.9 and 1.10.
• 2:45
  SPEAKER_01: That was the version I started on with.
• 2:48
  SPEAKER_01: And I was not a data engineer by craft or anything.
• 2:50
  SPEAKER_01: Until then, I was a software engineer.
• 2:52
  SPEAKER_01: I was five years experience then.
• 2:54
  SPEAKER_01: I was thrown into this Airflow project and I had to pick up everything from scratch, like doing my mistakes.
• 2:59
  SPEAKER_01: Is Kubernetes the right thing to do?
• 3:00
  SPEAKER_01: Or is Docker the right thing to do?
• 3:02
  SPEAKER_01: Or should we put it in ECT and call it a day?
• 3:05
  SPEAKER_01: So these are the questions that we still get today, right?
• 3:08
  SPEAKER_01: People who are adopting Airflow for the first time, we answer these questions even today.
• 3:12
  SPEAKER_01: Of course, there are like a lot more questions to answer.
• 3:14
  SPEAKER_01: GCP or AWS or MWA or Astronomer, which one should I use?
• 3:19
  SPEAKER_01: All these questions come together.
• 3:21
  SPEAKER_01: But as an open source project, to keep it sustaining, to keep it alive, I know how much of back-end work it has we are putting in for this project, right?
• 3:30
  SPEAKER_01: That's what's the most exciting thing about Airflow is.
• 3:33
  SPEAKER_01: So the evolution I have seen is in all directions, right?
• 3:37
  SPEAKER_01: If you're using Airflow locally, previously we used to open four terminals and we run four commands in four terminals.
• 3:43
  SPEAKER_01: It started like that, if you remember.
• 3:45
  SPEAKER_01: And then we now have Airflow standalone.
• 3:48
  SPEAKER_01: One command which sets you up with everything.
• 3:50
  SPEAKER_01: And we also have to run migrations separately, which was also one of those things.
• 3:54
  SPEAKER_01: So now we don't have any of those chaos.
• 3:56
  SPEAKER_01: It's just one command to get you started with Airflow, Airflow standalone.
• 4:00
  SPEAKER_01: The second thing is Helm shards.
• 4:02
  SPEAKER_01: When I started, there was no official Helm shots in Airflow.
• 4:05
  SPEAKER_01: I think we got it from Bitnami.
• 4:07
  SPEAKER_01: Now we have an official Helm shots which is being supported, and we have a separate support channel also for that, if I'm not wrong.
• 4:14
  SPEAKER_01: And the adoption by Google, Amazon, and latest thing is Azure has adopted it with Fabric.
• 4:19
  SPEAKER_01: So there has been a lot of questions around that also.
• 4:22
  SPEAKER_01: So now if you're using Airflow, you're not just using it, you have all these companies backing it up.
• 4:27
  SPEAKER_01: So that comes with a lot of support.
• 4:30
  SPEAKER_01: And then on the features, where do I even start?
• 4:33
  SPEAKER_01: With 3.2, we covered all bases.
• 4:37
  SPEAKER_01: We have task STKs, which makes it purely Pythonic.
• 4:40
  SPEAKER_01: We have assets as first-class citizens.
• 4:43
  SPEAKER_01: Now we can also support event-based mechanisms.
• 4:45
  SPEAKER_01: We have edge executors, which means you can run it in your coffee machine.
• 4:49
  SPEAKER_01: I saw that use case also.
• 4:50
  SPEAKER_01: Recently, someone was running all the tasks in coffee machines.
• 4:54
  SPEAKER_01: And we have DAG versioning, which we have been asking for so long.
• 4:58
  SPEAKER_01: DAG bundles has been my favorite because now you don't have to put everything in one repository and wonder whether I should build it in Docker or use a Git sync.
• 5:07
  SPEAKER_01: And human in the loop is just opening doors for so many opportunities.
• 5:11
  SPEAKER_01: Now you can have a whole application running inside your Airflow environment.
• 5:17
  SPEAKER_01: It's becoming like Django in some level.
• 5:19
  SPEAKER_01: If you're a contributor, you have Birbreze CLI.
• 5:22
  SPEAKER_01: So it has grown in almost all directions, I can say.
• 5:25
  SPEAKER_01: That's a long answer, but there is so much we have covered in the last seven years.
• 5:30
  SPEAKER_02: Yeah, certainly the project has come a really long way, and we couldn't even begin to touch on every single improvement or new feature that has happened.
• 5:38
  SPEAKER_02: But yeah, it's been really exciting to see.
• 5:41
  SPEAKER_02: And certainly from an astronomer's perspective, we've contributed heavily to the open source project, and we're really excited with where it's at now compared to where it was in 2018.
• 5:51
  SPEAKER_02: I'm curious if helping implement Airflow for all of these different customers, does this evolution make you more confident in recommending it?
• 6:01
  SPEAKER_01: Oh, absolutely.
• 6:02
  SPEAKER_01: Before we go to that, on the growth of Airflow, there is a running joke I run in almost every Airflow talk.
• 6:07
  SPEAKER_01: We have 12 months in a year, but we release one version almost every month in that year.
• 6:12
  SPEAKER_01: Which means we are releasing 13 versions in a year.
• 6:15
  SPEAKER_01: And companies come to me and be like, there is a new update.
• 6:18
  SPEAKER_01: Should I take it or not?
• 6:19
  SPEAKER_01: And it's justified that we cannot adopt to everything that's happening, but it's justified that to keep the project sustainable and adopt to the changes of the environment, we have to do that.
• 6:30
  SPEAKER_01: So I think as an open source project, why adopt Airflow is that it's the one of the most sustainable and fast growing and feature-heavy in terms of where it is now.
• 6:40
  SPEAKER_01: So that's one of those reasons.
• 6:42
  SPEAKER_02: Definitely.
• 6:43
  SPEAKER_02: I have a slide that we use in our new release webinars that we do to tell people about.
• 6:48
  SPEAKER_02: We only do them for minor releases, so we don't do the patch releases, or we'd be way too busy.
• 6:54
  SPEAKER_02: But the slide has all of the releases, including the patch ones, on it over the past couple of years.
• 7:00
  SPEAKER_02: And every time I think I make some comment to the team about how hard it is to keep that slide up to date because there's just so many of them, and it's like a very finicky thing to fit all of them in there and put them all on every month or so, as you said.
• 7:13
  SPEAKER_02: Yeah, it's definitely the project is moving forward very quickly.
• 7:16
  SPEAKER_02: That's a big boon and a band in board census.
• 7:20
  SPEAKER_02: Exactly.
• 7:20
  SPEAKER_02: And so given that you said you're pretty confident in recommending Airflow, why do you typically tell companies to use Airflow over like some other orchestrator that might also be moving forward at a good clip or kept up to date?
• 7:34
  SPEAKER_01: I have been an Airflow efficient auto.
• 7:36
  SPEAKER_01: Like I have fought with people to choose Airflow over everything else in the market.
• 7:40
  SPEAKER_01: Of course, I do not have expertise in everything else that's in the market, right?
• 7:44
  SPEAKER_01: Be prefect, Daxter.
• 7:45
  SPEAKER_01: But why I recommend Airflow is I believe that this is one of the truly open source community out there.
• 7:51
  SPEAKER_01: And like I said, 2017, 18, when I started, there was nothing in the ecosystem to support me.
• 7:58
  SPEAKER_01: There was only one thing that was Airflow support channel or the Airflow Slack channel.
• 8:02
  SPEAKER_01: Any issues that you have, you go there, you ask people.
• 8:05
  SPEAKER_01: I think now we are around 30k, or maybe I'm wrong.
• 8:08
  SPEAKER_01: There are about 30k members in the Airflow channels.
• 8:11
  SPEAKER_02: It's over 60,000 now in the Airflow Slack.
• 8:14
  SPEAKER_02: Yeah.
• 8:15
  SPEAKER_01: 60.
• 8:16
  SPEAKER_01: So that's crazy amount of support you get from uh open community, right?
• 8:20
  SPEAKER_01: That's not even paid or packed by a company per se.
• 8:24
  SPEAKER_01: And the provider ecosystem that we come with, anytime you want a provider included or any app that you want to integrate, there is already something built inside Airflow.
• 8:33
  SPEAKER_01: And in fact, there has been times when companies come to me, they want to use Airflow instead of Airbite because of the connectors and the providers that it comes with.
• 8:41
  SPEAKER_01: So these are some of the reasons.
• 8:43
  SPEAKER_01: Don't get fooled by the old Airflow logs and blogs and other stuff.
• 8:47
  SPEAKER_01: We have evolved so much as an ecosystem and as a community.
• 8:50
  SPEAKER_01: So there is definitely a lot of hope that Airflow is the best thing.
• 9:06
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 9:18
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 9:26
  SPEAKER_02: Still lots of content floating around out there talking about issues with Airflow that then when you read them were relevant in version 2.2 from three years ago.
• 9:36
  SPEAKER_02: So yeah, definitely a good thing to be on the lookout for.
• 9:40
  SPEAKER_02: Moving into best practices a little bit, because this is a really popular topic and I think something that people struggle with a lot and look for advice on.
• 9:49
  SPEAKER_02: From your perspective, having worked with many companies that use Airflow, where do you think that people get tripped up the most?
• 9:55
  SPEAKER_01: This is going to be a long answer.
• 9:56
  SPEAKER_01: If I do the long answer, I have an Airflow Summit talk on the same thing, but I will touch on a few points, right?
• 10:03
  SPEAKER_01: These are the general guidelines I give to all my clients.
• 10:06
  SPEAKER_01: Don't start with the pipeline code directly.
• 10:09
  SPEAKER_01: Do a pipeline design process.
• 10:11
  SPEAKER_01: Why do we suggest a pipeline design process?
• 10:13
  SPEAKER_01: Is you can capture so many ouch moments, if I can say so many of the glitches beforehand.
• 10:19
  SPEAKER_01: So, what does this design phase involve?
• 10:21
  SPEAKER_01: Is what are the tasks that we are going to run?
• 10:23
  SPEAKER_01: What is this big workflow and how can we split them into tasks?
• 10:27
  SPEAKER_01: It might feel like, okay, I can just write Python and call it a day, but doing this exercise will nail down every little piece of information.
• 10:35
  SPEAKER_01: And in this little blocks of tasks that you are designing, you can go back and say, okay, if this fails, how the rest of the pipeline is going to get affected?
• 10:43
  SPEAKER_01: Okay, if this fails, do I run it again or should I uh intimate someone, send a Slack notification or something?
• 10:49
  SPEAKER_01: So this design process is something I recommend to all my clients.
• 10:52
  SPEAKER_01: Don't just hurry into writing code, do this first.
• 10:55
  SPEAKER_01: The second place where I see people get tripped off is where do I put my business logic, right?
• 11:00
  SPEAKER_01: Should I put it along with Airflow among the DAGs and put it in a separate folder like scripts, or should I put it in a Docker container and call it separately using like a Kubernetes pod operator or something?
• 11:11
  SPEAKER_01: I usually am a big fan of Kubernetes pod operator.
• 11:14
  SPEAKER_01: I believe in decoupling everything, but there is a trade-off because if you want to use all the providers and all the connections and hooks that Airflow provides, you absolutely have to have some amalgamation.
• 11:26
  SPEAKER_01: So that's a trade-off that depends on every other use case.
• 11:29
  SPEAKER_01: But highly recommend if you have a core business logic that does not use anything Airflow related, it's better to Dockeriz it and have it separately.
• 11:37
  SPEAKER_01: Those are the two DAG pipeline and design principles level things.
• 11:42
  SPEAKER_01: And the on the infrastructure thing, it's again a huge topic because every company needs a different infrastructure for them.
• 11:49
  SPEAKER_01: There was one client last week who was like, I do everything inside fabric, I cannot do anything outside of it.
• 11:55
  SPEAKER_01: Now Azure has fabric in it and Airflow as a part of it.
• 11:59
  SPEAKER_01: We cannot do anything outside.
• 12:01
  SPEAKER_01: So that has a very specific solution that you have to design for.
• 12:04
  SPEAKER_01: You cannot use PubSub, you cannot put Postgres, like there are very specific things.
• 12:09
  SPEAKER_01: So knowing what's right for you.
• 12:10
  SPEAKER_01: Maybe at the start, EC2 is the best way to go.
• 12:13
  SPEAKER_01: You only have five pipelines that you want to run, and you know that it's going to be five and not 50 the next month.
• 12:19
  SPEAKER_01: So that is a trade-off that you need to, you know, work around.
• 12:23
  SPEAKER_01: Again, the talk has a very nice matrix that I have pitched of how do you choose which one to deploy and where to deploy, whether you should use Kubernetes or EC2.
• 12:32
  SPEAKER_01: All of this thing is covered there.
• 12:34
  SPEAKER_02: Yeah, that's great.
• 12:35
  SPEAKER_02: We'll definitely link to your talk in the show notes so folks can go watch that entire thing if that would be helpful for them.
• 12:41
  SPEAKER_02: And yeah, I agree the DAG design up front or pipeline design is really important if you just jump into it blindly.
• 12:48
  SPEAKER_02: It usually ends up not where you thought it was going to at the beginning.
• 12:52
  SPEAKER_02: It's really hard to scale that way.
• 12:53
  SPEAKER_01: There has been cases where we're like, oh, we designed this pipeline, it's going to process only 40 MB of files.
• 12:59
  SPEAKER_01: Nothing more is going to happen.
• 13:00
  SPEAKER_01: And two months later, the pipeline breaks, customer calls, and then no, like, what happened?
• 13:05
  SPEAKER_01: No, client suddenly uploaded 40 GB of data, and we don't have the intra to support it.
• 13:10
  SPEAKER_01: These are the kind of things that can happen on live.
• 13:13
  SPEAKER_01: So production is always a tricky business.
• 13:16
  SPEAKER_01: Definitely.
• 13:17
  SPEAKER_02: Cool.
• 13:17
  SPEAKER_02: And then with all of that, looking ahead a little bit, where do you think Airflow fits into the future of data engineering?
• 13:24
  SPEAKER_02: Especially, you touched on this a little bit before, but with the rise of cloud native and AI-driven workflows.
• 13:30
  SPEAKER_01: Oh, I see it almost everywhere in the era of like we are evolving in almost all directions.
• 13:35
  SPEAKER_01: Plug-in is one of those I was saying like.
• 13:37
  SPEAKER_01: So plugins with human in the loop, you can literally build a web app inside Airflow and call it a day.
• 13:43
  SPEAKER_01: All your business logic and everything can work inside that.
• 13:46
  SPEAKER_01: Again, human in the loop has become my favorite.
• 13:48
  SPEAKER_01: You can use it for data annotation.
• 13:49
  SPEAKER_01: We already do a lot of data quality checks.
• 13:52
  SPEAKER_01: So annotation as a part of it.
• 13:54
  SPEAKER_01: And then AgenTick loops, human in the loop, plus plugins.
• 13:58
  SPEAKER_01: You're opening like a huge world out there.
• 14:01
  SPEAKER_01: And I've seen a lot of support for inf like model training and model inferenced pipelines as well.
• 14:08
  SPEAKER_01: That has also becoming prominent.
• 14:10
  SPEAKER_01: So I think it's like you're getting a whole toolbox where you want to use it and which tool you want to use it.
• 14:16
  SPEAKER_01: You have all the options in Airflow now.
• 14:18
  SPEAKER_02: Yeah.
• 14:18
  SPEAKER_02: So you think Airflow will be used even more than it is now with these new workflows.
• 14:23
  SPEAKER_02: Yeah.
• 14:24
  SPEAKER_02: I agree.
• 14:24
  SPEAKER_02: I love that you brought up the plugins and humor for in running applications within Airflow.
• 14:30
  SPEAKER_02: That's something that my team has done a lot of, mostly for fun.
• 14:32
  SPEAKER_02: So they're playing a lot of video games in Airflow right now, which is obviously just highlighting the functionality in a fun way that folks will find interesting.
• 14:41
  SPEAKER_02: But it does, I think, yeah, open up a door for a lot of really cool use cases.
• 14:45
  SPEAKER_02: And I think inference execution is another one that's in that category of you can now have Airflow be the back end for a front-end application.
• 14:53
  SPEAKER_02: And a lot of times those are LLM driven.
• 14:56
  SPEAKER_02: And yeah, that's something that you really couldn't do even a couple of years ago within Airflow, at least not very easily.
• 15:02
  SPEAKER_02: Yeah.
• 15:02
  SPEAKER_02: All right.
• 15:03
  SPEAKER_02: Thank you for going through that.
• 15:04
  SPEAKER_02: I think this is really insightful.
• 15:07
  SPEAKER_02: And again, I will point folks to your talk for more on that because, like I said, I think best practices are always something that people are looking to learn more about, no matter where they are in their Airflow journey.
• 15:17
  SPEAKER_02: With that, I always like to end with a standard final question.
• 15:20
  SPEAKER_02: And that is what would you most like to see from development of the Airflow project in the near future?
• 15:25
  SPEAKER_01: So I had something on my mind, but I have two things now.
• 15:29
  SPEAKER_01: For the longest time since 2018, the most asked feature, even today, is multi-tenancy.
• 15:34
  SPEAKER_01: If I'm not mentioning it, the Airflow community will come behind me.
• 15:37
  SPEAKER_01: It's definitely the number one.
• 15:39
  SPEAKER_01: But today I was thinking, as I'm seeing the MCP directories and everything, what would it look like to have an Airflow plugin directory?
• 15:47
  SPEAKER_01: What all our ecosystem can build for that, right?
• 15:50
  SPEAKER_01: There is so many use cases I can think of, and I would love to see like a plugin directory where we can install a share that knowledge also.
• 15:58
  SPEAKER_02: That would be really cool.
• 15:59
  SPEAKER_02: I've heard that brought up at an astronomer recently as well.
• 16:02
  SPEAKER_02: People saying, hey, this would be really useful now that there is this really powerful interface for creating them.
• 16:08
  SPEAKER_02: But I think not everybody knows where to start.
• 16:10
  SPEAKER_02: And it would be helpful to have a framework for more standard implementations.
• 16:15
  SPEAKER_02: That would definitely be very cool.
• 16:16
  SPEAKER_01: That would be lovely.
• 16:18
  SPEAKER_01: Seriously.
• 16:18
  SPEAKER_01: A framework for Airflow plugin would be lovely.
• 16:21
  SPEAKER_02: I would love that.
• 16:22
  SPEAKER_02: Cool.
• 16:23
  SPEAKER_02: Well, Bhavani, thank you so much for coming on.
• 16:25
  SPEAKER_02: It's been really great speaking with you.
• 16:27
  SPEAKER_02: I'll close out by asking what is the best way for folks to get in touch with you?
• 16:31
  SPEAKER_02: Maybe if they want you to implement Airflow for them.
• 16:33
  SPEAKER_01: LinkedIn is the best way.
• 16:35
  SPEAKER_01: I think it's non-ignore it in life.
• 16:37
  SPEAKER_01: Yes.
• 16:38
  SPEAKER_02: Yes, that is the truth.
• 16:39
  SPEAKER_02: Okay, well, we'll definitely link your LinkedIn in the show notes as well.
• 16:43
  SPEAKER_02: Well, thank you again for coming on.
• 16:44
  SPEAKER_02: It's been really great speaking with you.
• 16:46
  SPEAKER_01: Pleasure is all mine.
• 16:48
  SPEAKER_01: I'm so glad.
• 16:51
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 16:58
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 17:02
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star reveal.