Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham

Transcript

Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: So the other feature which I'm really excited about as part of 3.0 is really the task execution evidence.
• 0:06
  SPEAKER_03: And the reason for it is because it's such a foundational architectural change for us.
• 0:11
  SPEAKER_03: Splitting up Airflow into literally a fun server model.
• 0:15
  SPEAKER_03: We're really breaking it apart so that the worker code, which is especially the user code running on the workers, are actually separated from the rest of Airflow, i.e., the scheduler, the processors, and everything else, completely separate, which gives us a lot of flexibility going forward.
• 0:38
  SPEAKER_02: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:48
  SPEAKER_01: Hey Airflow community, Airflow Theory is here and we are taking it on the road.
• 0:53
  SPEAKER_01: Join the Astronomer Roadshow for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 1:01
  SPEAKER_01: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:09
  SPEAKER_01: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:14
  SPEAKER_01: Register now.
• 1:15
  SPEAKER_01: The link is in the show notes.
• 1:18
  SPEAKER_01: Hi everyone.
• 1:19
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:21
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm very excited to be joined by Vikram Koka, who is a Chief Strategy Officer at Astronomer, as well as Jed Cunningham, who is a principal software engineer at Astronomer.
• 1:33
  SPEAKER_01: And you both are also Airflow PMC members.
• 1:36
  SPEAKER_01: Which is wonderful.
• 1:37
  SPEAKER_01: How are you both doing today?
• 1:39
  SPEAKER_04: Very well, thank you.
• 1:41
  SPEAKER_04: Yeah, doing well.
• 1:42
  SPEAKER_04: A little tired, but I'm here.
• 1:44
  SPEAKER_04: Let's do it.
• 1:45
  SPEAKER_01: You're here.
• 1:46
  SPEAKER_01: Yeah, lots going on both at Astronomer and in everybody's outside of work lives, but all good stuff.
• 1:53
  SPEAKER_01: Okay, great.
• 1:54
  SPEAKER_01: We have a ton to cover today, so let's go ahead and dive in.
• 1:57
  SPEAKER_01: For today's podcast, we're going to discuss the recent Airflow 3.0 release, which is a huge milestone in the Airflow project that we are all thrilled has just come out.
• 2:07
  SPEAKER_01: And I always like to start with some background.
• 2:09
  SPEAKER_01: So Airflow 3 is the first major release for the project in about four and a half years or so.
• 2:16
  SPEAKER_01: I think we all vividly remember the 2.0 release as if it was not quite so long ago, but it was a bit of time.
• 2:24
  SPEAKER_01: But obviously, minor releases in the meantime have brought lots of big new features to the project.
• 2:28
  SPEAKER_01: There were huge changes between the 2.0 release and the 2.10 release, which is what we were on right before Airflow 3.
• 2:36
  SPEAKER_01: So tell me a little bit about the decision to have a major release right now.
• 2:40
  SPEAKER_01: What brought that on?
• 2:42
  SPEAKER_03: So there was a lot of discussions, which we actually had as part of the community.
• 2:47
  SPEAKER_03: We also saw a lot of new Airflow improvement proposals, what we call AIPs, getting submitted into the project as well.
• 2:55
  SPEAKER_03: A lot of them around advanced capabilities like hybrid execution, remote execution, different kinds of execution mechanisms being requested.
• 3:06
  SPEAKER_03: And one of the key things which we saw was the need for data and the need for data orchestration was especially growing because of the AI explosion of actually people needing more and more AI applications and generative AI applications.
• 3:23
  SPEAKER_03: And data is the foundation for AI applications.
• 3:27
  SPEAKER_03: And as a result of that, that sheer pervasiveness of needing data in a variety of different forms, in a variety of different applications, was actually growing.
• 3:36
  SPEAKER_03: And one thing, as these new airflow improvement proposals kept coming up, we realized we just needed a more modern foundation or a more up-to-date foundation to be able to build on top of towards beaning all these particular requirements.
• 3:51
  SPEAKER_03: And so as part of that, it was a significant jump.
• 3:56
  SPEAKER_03: It was a significant sort of conversations towards saying, hey, is this actually time to get towards an updated technical architectural foundation as the basis towards meeting all these new Airflow improvement proposals that we would actually like to be able to address, and a lot of which we've addressed now as part of Airflow 3.0 as well.
• 4:15
  SPEAKER_03: And that's really where the genesis of Airflow 3 came up.
• 4:21
  SPEAKER_01: Yeah, that's great.
• 4:23
  SPEAKER_01: Totally makes sense.
• 4:24
  SPEAKER_01: We have seen so many changes in the data landscape in those last four years.
• 4:28
  SPEAKER_01: We've talked about them a lot on this podcast, especially how they've impacted a lot of the Airflow users.
• 4:34
  SPEAKER_01: So really great to see the project evolving along with that.
• 4:38
  SPEAKER_01: And I think we've seen that reflected when the use cases that people use Airflow for and just the general user base has grown so much since we were talking about 2.0.
• 4:48
  SPEAKER_01: I honestly can't even do the math in my head right now of what that like time X times percentage is.
• 4:54
  SPEAKER_01: But it's a really exciting time.
• 4:56
  SPEAKER_01: And yeah, love that Airflow has kept kept in step with everything that's happening.
• 5:02
  SPEAKER_01: Yeah.
• 5:02
  SPEAKER_04: It also gives us the opportunity to rid ourselves of some of the baggage that we've been carrying along.
• 5:07
  SPEAKER_04: Like I think of subdags as an example.
• 5:09
  SPEAKER_04: It's never been suggested to use in Airflow 2, even so every AIP that touches core Airflow, the scheduler, has to deal with supporting subdags still.
• 5:20
  SPEAKER_04: And so just being able to like clean some of that out gives us a better like foundation for the future of having a more stable, simpler to maintain core airflow.
• 5:28
  SPEAKER_01: Yeah, a great opportunity for tech debt.
• 5:31
  SPEAKER_01: And I'm also thinking of the change to switch the catch up default to false.
• 5:37
  SPEAKER_01: This was one that my team helped work on.
• 5:39
  SPEAKER_01: It was we work with new airflow users a lot, and that's always one of the worst gotchas.
• 5:43
  SPEAKER_01: And it's one of those tiny little things that maybe the decision to make it how it was is no longer relevant.
• 5:49
  SPEAKER_01: It was made many years ago, but because it is a breaking change, you can't make that change in a minor bump.
• 5:56
  SPEAKER_01: So even though those minor versions have included pretty groundbreaking new features, sometimes there are things that you just can't fix without a major release.
• 6:04
  SPEAKER_03: Actually, that's a great point, Kenton.
• 6:06
  SPEAKER_03: It is we've tried really hard to follow strict semantic versionings since R4 2.0.
• 6:14
  SPEAKER_03: And the positive was that yes, we could introduce new features, but the negative was we couldn't really take stuff away until we actually could and did have the opportunity to do a major release.
• 6:25
  SPEAKER_03: And 3.0 was absolutely the opportunity for us to clear some of that tech debt, some of the old deprecated features which we otherwise couldn't remove.
• 6:34
  SPEAKER_00: Yeah, yeah.
• 6:35
  SPEAKER_00: Maybe some of sorry, go ahead.
• 6:38
  SPEAKER_03: Sorry, one of the things which we were which we had been talking about a lot during the course of 3.0 was how much code was actually removed from the project.
• 6:47
  SPEAKER_03: And I'm sure Jed can read off far more details than I can, but there are so many areas where hey, this many thousands of lines of code got removed was actually a cause of celebration within the team and the community working on 3.0 as well.
• 7:02
  SPEAKER_01: I was just about to say the same thing.
• 7:04
  SPEAKER_01: I feel like those are maybe the most satisfying PRs as the ones that just delete, you know, hundreds or thousands of lines of code.
• 7:13
  SPEAKER_01: It's very exciting.
• 7:16
  SPEAKER_01: Yeah.
• 7:16
  SPEAKER_01: Awesome.
• 7:17
  SPEAKER_01: On that topic, so we've just talked a little bit about stuff that's being removed, which is great for the project, but there's also a lot of new stuff to talk about in this release.
• 7:27
  SPEAKER_01: There's way more than we can cover in the 25 minutes that we're going to be here in terms of new features.
• 7:32
  SPEAKER_01: We will, of course, have plenty of resources for folks to learn all about those.
• 7:36
  SPEAKER_01: But let's talk a little bit just high level about the major themes of this release in terms of what's new.
• 7:45
  SPEAKER_03: So I always like to start with like two major themes, honestly, which is one is the first, which is frankly ease of use.
• 7:53
  SPEAKER_03: And that's really making existing use cases, existing user scenarios easier to use as part of our flow three.
• 8:01
  SPEAKER_03: And the second one really is about one of my favorite themes, is being able to run tasks anywhere at any time in any language.
• 8:10
  SPEAKER_03: Now, each of those are pretty broad themes, and there's a lot of details in that front.
• 8:14
  SPEAKER_03: I'll let Jed talk about the first one, which is very near and dear to his heart within that aspect, because there's so many things inside of just making it easier to use for existing use cases, which we work on as part of 3.0.
• 8:31
  SPEAKER_04: Yeah, and I guess I want to just call out that we, as we were talking about tech dead earlier, right?
• 8:37
  SPEAKER_04: Like we took the opportunity to make sure that we're using like like new tooling that makes sense.
• 8:42
  SPEAKER_04: So for example, like the whole new the whole UI now is rewritten with React and the REST API has been moved over to Fast API, and it's all async endpoints behind it.
• 8:50
  SPEAKER_04: A lot of work went into like auto-generating the like the HTT calls that the UI actually makes out to the back end so that front-end developers don't have to recreate that wheel every time.
• 9:00
  SPEAKER_04: And so yeah, there's a lot of like nice features that like may not even be like exposed to the users directly, but that will pay benefits longer term with flexibility and things that we're able to do longer term in Airflow 3.x.
• 9:15
  SPEAKER_03: Yeah, indeed.
• 9:17
  SPEAKER_03: And just like part of that is also like in a stronger security profile, which we could do as part of this, which is not something we tend to talk about as much because it's often a feature.
• 9:27
  SPEAKER_03: But a lot of the stuff which we actually work on really enables a much stronger security posture with the Airflow 3.0 deployments.
• 9:36
  SPEAKER_03: The other thing which we don't explicitly talk about really is like even the how Airflow can be deployed and integrated within the broader data ecosystem is one of the things that really excites me a lot.
• 9:49
  SPEAKER_03: And some of that is really based on all of these APIs, but also the ability to trigger DAGs based on or have DAGs run based on data showing up in other sources.
• 10:00
  SPEAKER_03: And that's actually really critical because of the different ways now you can actually integrate Airflow within basically your own data ecosystem rather than necessarily having to trigger stuff on a schedule from within Airflow, which then pulls stuff from external ecosystems.
• 10:16
  SPEAKER_03: So I think Airflow actually plays better as a result of the changes with the broader ecosystem and more variety of interactions are now feasible as part of that.
• 10:28
  SPEAKER_01: Yeah, absolutely.
• 10:29
  SPEAKER_01: I love that you touch even briefly on the security aspect, because I think, as you said, some of those underlying architectural changes, they're not as visible.
• 10:38
  SPEAKER_01: And so they're not the sexy new feature that you get a lot of screenshots of and things like that, but they're no less important for the project, especially one that's used by so many huge companies with such a scale of data and really important and often sensitive use cases.
• 10:54
  SPEAKER_01: I was just earlier today speaking with somebody about kind of the big architectural changes between Airflow One and Airflow two, which end up, you know, new use cases that those enabled and making Airflow Enterprise ready.
• 11:06
  SPEAKER_01: And it's great to see how it's continued to evolve in that direction.
• 11:10
  SPEAKER_01: Again, so we touched on the three key themes outside of the security stuff, of course, the ease of use and the run and the how the architectural changes have enabled this ability to run anywhere at any time.
• 11:21
  SPEAKER_01: Those are going to be huge, kind of in the day-to-day of developers.
• 11:24
  SPEAKER_01: And again, we're going to have lots of resources on all of these features, specifically on how to adopt them.
• 11:30
  SPEAKER_01: I think a podcast is probably not most people's first choice for learning how to adopt the feature.
• 11:36
  SPEAKER_01: Or if they are, I'd love to chat with them how they're able to do that.
• 11:40
  SPEAKER_01: But again, given our limited time, I am curious to hear from you both since you were both heavily involved in the planning and development of this release for, you know, I'm sure like the better part of the last year, if not longer.
• 11:54
  SPEAKER_01: What are you most excited about?
• 11:56
  SPEAKER_01: Which feature do you think you really want people to know about and go run to get started with?
• 12:04
  SPEAKER_04: Um yeah, I guess I'll go first.
• 12:06
  SPEAKER_04: Near and dear to my heart is DAG versioning.
• 12:08
  SPEAKER_04: And so if you look back, that's the most asked for feature over a number of years in the Airflow survey.
• 12:13
  SPEAKER_04: I probably should have looked up exactly what years and had a more concrete answer there.
• 12:17
  SPEAKER_04: But uh but yeah, like the problem is you go and you ask people what does DAG versioning mean?
• 12:22
  SPEAKER_04: And it really you get different answers, right?
• 12:25
  SPEAKER_04: Like the problems that you face will sway what how you answer that question.
• 12:30
  SPEAKER_04: And so if we look at it, like what has Airflow done with in this space before?
• 12:35
  SPEAKER_04: We could have assumed that DAGs don't change in meaningful ways, but of course they do.
• 12:40
  SPEAKER_04: That's the real world, and DAGs reflect some of that messiness.
• 12:43
  SPEAKER_04: And Airflow has never tracked what a DAG looked like historically.
• 12:47
  SPEAKER_04: If you remove a task, it's gone in the UI for most purposes, even for old runs that had that task.
• 12:54
  SPEAKER_04: And then, yeah, like similarly on the execution side, like you Airflow would always run on the latest version of the code, right?
• 13:03
  SPEAKER_04: So if you had a long-running DAG and you changed your DAG code halfway through, the tests that are already run in the beginning ran on one version, and then all the tests that ran after the change are now on a different version of code, you're at the mercy of whether those are compatible enough that Airflow doesn't blow up, and sometimes it will.
• 13:20
  SPEAKER_04: So when I went to when I sat down and was like trying to figure out how do we solve this holistically, I really wanted to tackle two aspects of it, right?
• 13:29
  SPEAKER_04: So there's history of the DAG, how do we keep track of that?
• 13:32
  SPEAKER_04: How do we show users what really ran at DAG run last week?
• 13:37
  SPEAKER_04: And then also, how do we give people control and the ability to run a whole DAG run on the same version of code?
• 13:44
  SPEAKER_04: So you don't have to deal with the you know vaguities of, oh, my DAG code's updating halfway through my DAG run.
• 13:50
  SPEAKER_04: And so that the first one is relatively easy, right?
• 13:54
  SPEAKER_04: Okay, yeah, we have serialized DAGs already.
• 13:55
  SPEAKER_04: We've had them since Airflow 2.
• 13:57
  SPEAKER_04: Now we just version them.
• 13:59
  SPEAKER_04: And now it's available for the UI to show you what that DAG looked like.
• 14:03
  SPEAKER_04: The more fundamental change though is when we get over to the execution side.
• 14:06
  SPEAKER_04: And so we've had to introduce something called it's a we call them bundles.
• 14:10
  SPEAKER_04: Basically, it's a mechanism for you to be able to get your whole DAG directory essentially at a specific version.
• 14:19
  SPEAKER_04: And what that lets Airflow do is you can the scheduler basically can say, Hey, when I when you go start this task, you need to be running on this version of the code.
• 14:30
  SPEAKER_04: And concurrently, on the same worker, you could have another task from a different DAG run of the same DAG that's using a different version of that code.
• 14:40
  SPEAKER_04: Fundamentally, though, that means that Airflow has to be able to go out and get your DAGs.
• 14:45
  SPEAKER_04: They can't just magically appear like we've assumed always up to this point.
• 14:49
  SPEAKER_04: So that's a 180-degree shift on who's responsible for getting DAGs into Airflow.
• 14:55
  SPEAKER_04: Now Airflow is able to go out and get them themselves.
• 14:57
  SPEAKER_04: So day zero, Airflow, the community airflow supports bundle.
• 15:03
  SPEAKER_04: And so now you'll be able to configure your DAGs to come out of a Git repository, and Airflow will be able to go out and get them and run them on its own.
• 15:12
  SPEAKER_04: And I guess one cool feature about this too is that there's now just not one DAG folder.
• 15:18
  SPEAKER_04: You could have a half dozen Git repos that you could pull DAGs out of.
• 15:23
  SPEAKER_04: And so you get a little bit of like flexibility from that change as well.
• 15:27
  SPEAKER_04: And yeah, there's a lot of, I think, especially on the UI side, there's a lot of room to still improve the interface and be like make better use of the information that we do now track.
• 15:38
  SPEAKER_04: So yeah, I'm really excited to see what DAG versioning from a UI perspective looks like in 3.2 or whatever.
• 15:45
  SPEAKER_04: And then similarly, I'm really curious to see what uh bundle types the community decides to add and support.
• 15:52
  SPEAKER_04: Today we have Git, but I'd be shocked if we don't have a half dozen here soon.
• 16:00
  SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:09
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:21
  SPEAKER_02: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:29
  SPEAKER_01: In my pre-release events with the community that we've done, we had already had so many questions about oh, will you support the system in the system?
• 16:38
  SPEAKER_01: So, yes, I think, as you said, so much room to expand on this in the future.
• 16:42
  SPEAKER_01: I am also so excited for this feature.
• 16:45
  SPEAKER_01: I think it's not when every release that you have a feature that I feel like we can confidently say every single Airflow user will use that this truly impacts everybody and will be such a big kind of quality of life difference.
• 16:59
  SPEAKER_01: And as you said, I love that it was very community driven.
• 17:02
  SPEAKER_01: This has been asked for by so many people for so long, and I think there's already a ton of excitement about it.
• 17:09
  SPEAKER_03: I would give a huge shout out to Jed and the rest of the team for um working on this.
• 17:14
  SPEAKER_03: We actually tried to we actually took a stab at even trying to build this as part of 2.0.
• 17:20
  SPEAKER_03: I don't know how many people remember that.
• 17:23
  SPEAKER_03: And we started working on it and was gonna go, all right, this is really big, and it's getting bigger every week as we look at it.
• 17:32
  SPEAKER_03: And that's when we said, all right, this is too big to try to do within 2.0, along with everything else we had already committed in 2.0.
• 17:39
  SPEAKER_03: And when Jet really dived in and really opened up all the details around this, it was phenomenal to see how much depth he had thought through the problem, even by Airflow Summit.
• 17:50
  SPEAKER_03: And that's one of the reasons I was very confident on this particular feature, making it in Airflow 3.
• 17:56
  SPEAKER_03: This is like kind of way back last year, and but it was a huge level of depth of thinking and analysis.
• 18:02
  SPEAKER_03: So huge shovel on that one.
• 18:04
  SPEAKER_03: It's actually hard to match any other feature with that model.
• 18:07
  SPEAKER_03: That is awesome.
• 18:09
  SPEAKER_01: Yeah, absolutely.
• 18:10
  SPEAKER_01: I like I said, so impactful, and Jed, you alluded to also something that I think sounds simple on the surface, and then it turns out can be interpreted in so many different ways.
• 18:22
  SPEAKER_01: One of those things that everybody wants, but nobody quite knows what it is.
• 18:25
  SPEAKER_01: So certainly not easy from a development perspective.
• 18:28
  SPEAKER_01: And I know like the whole community is going to be very thankful for that.
• 18:32
  SPEAKER_01: I did pose this question to both of you.
• 18:34
  SPEAKER_01: So, Vikram, I know that's a hard thing to follow, the headlining feature of the release, but I am going to ask you to do it.
• 18:40
  SPEAKER_01: So, what is the feature that you're most excited about?
• 18:43
  SPEAKER_03: So, the other feature which I'm really excited about as part of 3.0 is really the task execution interface.
• 18:50
  SPEAKER_03: And the reason for it is because it's such a foundational architectural change for Airflow.
• 18:56
  SPEAKER_03: That in we're really splitting up Airflow into literally a fine server model.
• 19:02
  SPEAKER_03: We're really breaking it apart so that the worker code, which is especially the user code running on the workers, are actually separated from the rest of Airflow, i.e., the scheduler, the processors, and everything else, completely separate, which gives us a lot of flexibility going forward.
• 19:20
  SPEAKER_03: And that separation is really what we're calling the cask execution interface.
• 19:24
  SPEAKER_03: Whereas now the workers and the cask themselves communicate using this particular protocol, which is language independent, which can be remote, and they communicate with an API server which runs as part of the web server, which is the server side of this particular protocol.
• 19:41
  SPEAKER_03: And then the client side of this particular protocol is what we're calling the task SDK.
• 19:45
  SPEAKER_03: The first incarnation of this task SDK is a Python-based Task SDK on which all the workers actually use this particular and run this particular task SDK-based tasks, which use a very separate set of uh integrations, dependencies, all completely separate from the servers.
• 20:06
  SPEAKER_03: Now, this separation, which is a very significant architectural change within Airflow, enables us to do a lot of different things.
• 20:16
  SPEAKER_03: And one of the key things is because it's now going through an API, because the CAS no longer needs to directly contact the Airflow metadata base for accessing information such as connections and XCOM data and everything else, it enables us to be able to distribute this across data centers, across clouds, across very different network boundaries, and also gives us tremendously greater scalability.
• 20:43
  SPEAKER_03: And as a result of this, Airflow can now support true multi-cloud deployments and truly support hybrid-edge execution.
• 20:52
  SPEAKER_03: And really, you can actually have paths running on a your local data center, which are CPU bound and are like actually leap CPUs.
• 20:59
  SPEAKER_03: You can have other attachments run on a different computing cluster, which is based on GPUs, which is on a completely different environment, and have these kind of interact completely seamlessly through one single managed airflow infrastructure.
• 21:13
  SPEAKER_03: You can also then support local data centers wherein people say, hey, look, this data can't leave this particular location for security reasons, i.e., we've got first class support now for data sovereignty.
• 21:27
  SPEAKER_03: And uh this separation is also the foundation for them being able to build additional language support because this execution interface is language agnostic.
• 21:38
  SPEAKER_03: And therefore, even though we're starting with a Python SDK with just overleased, we also actually have initial incarnations of a Golang SDK.
• 21:48
  SPEAKER_03: And I've heard a lot of people in the community immediately wanting to contribute actually based on Rust or Java or Scala or other languages.
• 22:00
  SPEAKER_03: Coming up.
• 22:00
  SPEAKER_03: And I think that's really exciting to see how we can really evolve Airflow to be truly distributed from a language perspective or multiple different application integration frameworks and really be extremely enterprise ready across multi-cloud deployments as well and completely seamlessly.
• 22:22
  SPEAKER_03: So that's it to me, it's a foundational feature, really.
• 22:25
  SPEAKER_03: And that's so we're going to see initial implications, our initial capabilities built on top of it, which we've seen with Airflow 3.0.
• 22:34
  SPEAKER_03: But I'm really excited to see how all this gets leveraged over 3.1, 2, 3, and like all the other integrations which come up over the next few releases as well.
• 22:45
  SPEAKER_01: Yeah, absolutely.
• 22:46
  SPEAKER_01: Another really big, exciting change that lays so much groundwork, as you said, for kind of future development.
• 22:53
  SPEAKER_04: Yeah.
• 22:54
  SPEAKER_04: What one thing I'll just add there is I think you mentioned it briefly, but I want to just really call it out.
• 22:59
  SPEAKER_04: Another really cool thing, and actually one that I'm really selfishly looking forward to, is it makes it very clear what the public interface of Airflow is, right?
• 23:06
  SPEAKER_04: Like historically, it's like there's a document on the website you have to go read and understand, and no one does.
• 23:11
  SPEAKER_04: But now it's is it in the task SDK?
• 23:14
  SPEAKER_04: Yes or no?
• 23:18
  SPEAKER_01: I love that.
• 23:19
  SPEAKER_01: That's huge.
• 23:20
  SPEAKER_01: Amazing.
• 23:21
  SPEAKER_01: I think those were two very good features.
• 23:23
  SPEAKER_01: I know, like I said, we could probably talk for hours and hours about everything coming up in the release.
• 23:28
  SPEAKER_01: We will, by we astronomer, will have lots of other events coming up where we are going to talk in more depth about these features as well as others.
• 23:37
  SPEAKER_01: But I know you both have lots of stuff to get back to, so I don't want to keep you all day.
• 23:41
  SPEAKER_01: I typically always end with a standard final question, which is what would you most like to see from the Airflow project in the near future?
• 23:48
  SPEAKER_01: Given our topic today, that feels a little bit like asking someone who just finished a marathon when they're going to do their next marathon.
• 23:55
  SPEAKER_01: So I'm going to change it a little bit and say, what would you want the Airflow community to know about the future of the Airflow project?
• 24:05
  SPEAKER_03: From my perspective, I really want them to know that this is, in at least my pers my view, that this is the foundation.
• 24:15
  SPEAKER_03: Right.
• 24:15
  SPEAKER_03: Airflow 3 is not the end.
• 24:17
  SPEAKER_03: Airflow 3 is the beginning of the next phase of development on top of Airflow.
• 24:23
  SPEAKER_03: And so I really want from the community thinking about, hey, we've got this amazing foundation which has just been laid with Airflow 3.0.
• 24:32
  SPEAKER_03: Here's all the new stuff we would now like to see coming from the project.
• 24:36
  SPEAKER_03: And I want to see people contribute to that particular project.
• 24:40
  SPEAKER_03: And with respect to, I alluded to language support earlier.
• 24:54
  SPEAKER_03: Start contributing in your particular languages which are of most interest to you, and add the support Airflow.
• 25:00
  SPEAKER_03: Would love to have contributors coming in from a variety of different languages, which maybe we haven't thought about, but is really critical for data use cases which are upcoming.
• 25:12
  SPEAKER_03: That's my shameless plug for additional contributors.
• 25:16
  SPEAKER_03: There you go.
• 25:20
  SPEAKER_04: No, I think like I I have a very similar feel, right?
• 25:24
  SPEAKER_04: It's like we've really set the foundation here to have really awesome future minor releases.
• 25:30
  SPEAKER_04: And it's going to be really cool to see what people come in and contribute.
• 25:33
  SPEAKER_04: Just the amount of activity that we've had in the community the last like year.
• 25:37
  SPEAKER_04: It's just it's crazy how active it's been.
• 25:39
  SPEAKER_04: And yeah, really curious to see what collectively we're able to do with some of these past decisions off our backs.
• 25:47
  SPEAKER_01: For sure.
• 25:47
  SPEAKER_01: I love that.
• 25:48
  SPEAKER_01: Just getting started.
• 25:49
  SPEAKER_01: That is a great place to close out.
• 25:52
  SPEAKER_01: So I'm going to say it has been really wonderful having both of you on.
• 25:56
  SPEAKER_01: And also a huge thank you from myself and the whole Airflow community for all of the work that you have done on this release.
• 26:04
  SPEAKER_01: I know obviously there's been lots of others involved too, and it has been a huge amount of work and it's a really big accomplishment for the project.
• 26:12
  SPEAKER_01: So we're all very thankful and very excited for it.
• 26:16
  SPEAKER_03: Thank you, Kenton.
• 26:17
  SPEAKER_03: It's I would actually say it's been amazing, as Jed said, the community contributions.
• 26:22
  SPEAKER_03: We have consistently had over a hundred people contributing from around the world every single month during the course of the project.
• 26:29
  SPEAKER_03: And so this wouldn't be the this wouldn't have happened without all of them making their contributions and lending their time, energy, and passion into this release.
• 26:38
  SPEAKER_04: Succinctly, I run the script to nominate the PR of the month.
• 26:41
  SPEAKER_04: There was a hard-coded limit of 500 PRs for core airflow.
• 26:45
  SPEAKER_04: We blew past that.
• 26:46
  SPEAKER_04: I had to go update that number as like such a cool like quantitative example where like just the activity's been insane.
• 26:53
  SPEAKER_01: I love that.
• 26:54
  SPEAKER_01: What a fun community kind of record to have broken the PR of the month script.
• 27:00
  SPEAKER_01: That is great.
• 27:01
  SPEAKER_01: Okay.
• 27:02
  SPEAKER_01: Thank you again both so much.
• 27:03
  SPEAKER_01: It's been really wonderful having you on, and we appreciate you doing the podcast with us.
• 27:09
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 27:16
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 27:20
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.