Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Inside Airflow 3: Redefining Data Engineering with Vikram Koka

Transcript

Inside Airflow 3: Redefining Data Engineering with Vikram Koka · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Airflow dives can now run automatically when the incoming data just shows up in a bucket in a blob storage or an update to a database table.
• 0:09
  SPEAKER_02: Data-driven scheduling originally introduced in Airflow a couple of years ago was limited to data handling within Airflow or within an Airflow deployment.
• 0:18
  SPEAKER_02: This now goes beyond that to enable dives to be clicked from data showing up from external data systems.
• 0:36
  SPEAKER_00: Hi everyone, welcome to the Data Flowcast.
• 0:39
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Vikram Koka, who's the Chief Strategy Officer at Astronomer.
• 0:46
  SPEAKER_00: Vikram, how are you doing?
• 0:48
  SPEAKER_02: I'm doing well.
• 0:49
  SPEAKER_02: Thank you, Kenton.
• 0:50
  SPEAKER_02: Thank you for having me.
• 0:51
  SPEAKER_00: Yeah, it's great to have you on.
• 0:52
  SPEAKER_00: You and I have worked together for a long time.
• 0:55
  SPEAKER_00: It's always fun to have old colleagues on the podcast.
• 0:57
  SPEAKER_00: You just got back from London, is that correct?
• 1:00
  SPEAKER_02: That's right.
• 1:01
  SPEAKER_00: Okay.
• 1:01
  SPEAKER_00: How was your trip?
• 1:02
  SPEAKER_00: How are you feeling post-return?
• 1:05
  SPEAKER_02: It was fun.
• 1:06
  SPEAKER_02: It was uh a fun trip, short.
• 1:08
  SPEAKER_02: Was only there for about a couple of days.
• 1:10
  SPEAKER_02: So was there at the uh forum talking about Airflow 3.
• 1:15
  SPEAKER_02: Got back, inevitable jet lag and various things, and now getting back into the flow of things have worked.
• 1:20
  SPEAKER_00: Yeah, that's great.
• 1:22
  SPEAKER_00: That's a quick turnaround from the West Coast of the US.
• 1:26
  SPEAKER_00: But great that we were able to go to another event and have you speaking about Airflow 3.
• 1:31
  SPEAKER_02: Yeah, it was fun.
• 1:32
  SPEAKER_02: It is, I really enjoy talking about this.
• 1:34
  SPEAKER_02: I enjoy talking to people in the community, have answering questions.
• 1:37
  SPEAKER_02: There's a lot of interest and a lot of follow-up questions, which I just felt just didn't have enough time to answer all the bell, and a lot of people wanting to engage and participate in the process as well, which is uh very fun and encouraging to see.
• 1:52
  SPEAKER_00: Yeah, that's great.
• 1:53
  SPEAKER_00: I love hearing so much excitement about Airflow 3, which I guess leads us into our main topic today.
• 1:58
  SPEAKER_00: So we can go ahead and dive into it.
• 2:00
  SPEAKER_00: For our listeners for today's podcast, we, as you might have guessed, are going to discuss the very highly anticipated Airflow 3 release, which is currently in development.
• 2:09
  SPEAKER_00: And Vikram is a great person to have on to talk us through and give us some more details about that.
• 2:15
  SPEAKER_00: So, Vikram, I want to start with a little bit of background first before we get into Airflow 3 specifically, especially about your an astronomer's role in open source airflow.
• 2:24
  SPEAKER_00: Many folks might not be aware of that.
• 2:27
  SPEAKER_00: So, can you talk about how you originally got involved with Airflow and how you became a committer and how you work on Airflow at Astronomer?
• 2:36
  SPEAKER_02: Sure, happen to.
• 2:37
  SPEAKER_02: I've been working with Astronomer for a little bit over five years now, and the majority of that time has been medium-range driving team, including the open source teamwork than Apache Airflow.
• 2:48
  SPEAKER_02: I started working with Airflow in 2019, and my personal contributions to Airflow have been primarily on the architecture front, beginning with Scheduler HA, which we released as part of Airflow 2.0.
• 3:00
  SPEAKER_02: Before 2.0, the biggest challenge for Airflow was that the scheduler was a single point of failure.
• 3:07
  SPEAKER_02: And there was a lot of people complaining about that.
• 3:09
  SPEAKER_02: And when I started working on Airflow, my focus was on, because of my background and distributed systems, my focus was on how do we actually make this caterer not only more reliable, but a lot more performant.
• 3:24
  SPEAKER_02: And I had a very strong passion towards making this an active model rather than an active passive model in the interests of future scalability.
• 3:33
  SPEAKER_02: I managed to convince the community to go along with my desire to make this active and then working with Ash and Capsule, who did the actual implementation of SkillDorHA as part of the Airflow 2.0 release, and being able to make that a reality.
• 3:48
  SPEAKER_02: As part of the 2.0 release, I took on the project and program management role for the release to Shepherd it during its development and uh towards GA.
• 3:58
  SPEAKER_02: And after we released 2.0, I was invited to become an Airflow committer and I accepted.
• 4:04
  SPEAKER_02: And since then I've continued working on Airflow, again, primarily in the architecture design proposals for features such as data-driven scheduling, dynamic task mapping, setup and teardown, and now Airflow 3.
• 4:18
  SPEAKER_00: Yeah, that's amazing.
• 4:19
  SPEAKER_00: I also started using Airflow back in 2019, and at that point was a data engineer myself and was implementing it.
• 4:27
  SPEAKER_00: I remember the days when you had to manually restart your single scheduler because it would stop working.
• 4:33
  SPEAKER_00: That was never fun.
• 4:34
  SPEAKER_00: I started at Astronomer a few months before Airflow 2 came out.
• 4:37
  SPEAKER_00: And I think the HA scheduler was one of the biggest, most exciting things in that major release.
• 4:42
  SPEAKER_00: Although that contribution I know is very much appreciated by the community.
• 4:46
  SPEAKER_00: I almost never hear about people having to restart their scheduler anymore.
• 4:50
  SPEAKER_00: So that was a great success.
• 4:52
  SPEAKER_00: And you mentioned so a lot of this work has been at Astronomer.
• 4:56
  SPEAKER_00: For those who don't know, Astronomer is invested in open source airflow.
• 5:00
  SPEAKER_00: We have an engineering team internally that focus exclusively on contributing to the project and as you said, help drive a lot of their releases, including this upcoming major one, along with other members of the community.
• 5:14
  SPEAKER_02: Absolutely.
• 5:15
  SPEAKER_02: It is something which we'll actually focus on.
• 5:18
  SPEAKER_02: And I would actually stress that it is not just driving the features or contributing to key features, the releases, but it's also really helping to grow the Airflow community.
• 5:27
  SPEAKER_02: Right?
• 5:27
  SPEAKER_02: There's a huge part of what we do is in enabling new members of the community to actually contribute to Airflow.
• 5:35
  SPEAKER_02: So we take a definite focus towards creating effectively what we call good first issues, which enable people to contribute, and then making sure that when they actually create their PRs, guiding them through their PR contribution process so that their PRs eventually get merged into the project, and then that that the somebody who's spending their own time to work and continue on this project has a fulfilling experience to actually be part of this particular community.
• 6:03
  SPEAKER_02: And that's, I think, one of the key things which we spend a lot of time, and honestly, Kenton, you and your team spent a lot of time on that as well, which is a huge part of growing the Airflow community.
• 6:13
  SPEAKER_00: Yeah.
• 6:14
  SPEAKER_00: Yeah, that's very true.
• 6:15
  SPEAKER_00: That's something that my team really loves working on is getting more contributors.
• 6:19
  SPEAKER_00: I know Airflow just passed the 3,000 contributor mark a couple of months ago, which is very exciting.
• 6:25
  SPEAKER_00: So making that process easier and lowering the barrier to entry is very important, especially as we near more major releases, which brings me to my next question.
• 6:35
  SPEAKER_00: So we just talked a little bit about the Airflow 2 release, which happened almost exactly four years ago, coming up in another couple of weeks here.
• 6:43
  SPEAKER_02: December 17th, 2020.
• 6:46
  SPEAKER_00: Okay, that date is probably burned in your brain very specifically.
• 6:50
  SPEAKER_00: Yeah.
• 6:51
  SPEAKER_00: I just remembered mid-December.
• 6:53
  SPEAKER_00: That's almost four years ago.
• 6:55
  SPEAKER_00: The project has progressed a ton since then.
• 6:58
  SPEAKER_00: There's a a lot of new features that have come out in the interim and that have really enabled a huge increase in the amount of use cases that you can use Airflow for.
• 7:07
  SPEAKER_00: Tell me a little bit about what the logic was behind wanting to have a new major release now.
• 7:14
  SPEAKER_00: So we're talking about Airflow 3 coming out sometime early to mid-next year timeframe.
• 7:21
  SPEAKER_02: You're absolutely right.
• 7:22
  SPEAKER_02: Airflow has grown tremendously since Tutor came out.
• 7:26
  SPEAKER_02: I remember, I think roughly November of 2020, the number of downloads were about 460,000 downloads per month.
• 7:36
  SPEAKER_02: I could be off by a little bit, but that's roughly what I remember.
• 7:39
  SPEAKER_02: And now the most recent month, we had about 30 million or a little bit over 30 million downloads a month, which is honestly really encouraging.
• 7:47
  SPEAKER_02: One of the key things from a pattern standpoint, Airflow has grown widely since 2.0.
• 7:53
  SPEAKER_02: Data engineers started downloading Airflow.
• 7:56
  SPEAKER_02: They started writing pipelines in Python, then deployed them into production to solve the problems.
• 8:01
  SPEAKER_02: Over time, these data pipelines grew in number and became mission critical.
• 8:07
  SPEAKER_02: And as enterprises understood the criticality of these data pipelines, which had almost grown without them even really noticing about it, then the size of the teams maintaining and the extending these pipelines also increased.
• 8:22
  SPEAKER_02: There's also been a dramatic increase in the need for data and the awareness of the importance of data, especially fueled by the AI then.
• 8:32
  SPEAKER_02: And one of the key things we realized is that some of the operational patterns suitable for viral individual adoption of Airflow are not necessarily suitable for large production deployments, spanning data engineering teams with access to data of varied security and privacy requirements.
• 8:54
  SPEAKER_02: This led to us really working through hey, what do we need to do towards making this level of change happen to meet the diversity between viral adoption and large production deployments?
• 9:09
  SPEAKER_02: And that led towards the thinking around significant architectural updates needed to Airflow, which needed a large major release, which led to Airflow 3.
• 9:21
  SPEAKER_00: Right.
• 9:21
  SPEAKER_00: Yeah, that makes sense.
• 9:23
  SPEAKER_00: So you're thinking both about it in terms of obviously the structural changes, especially if you're removing code or making any breaking changes, require that major version.
• 9:32
  SPEAKER_00: But that's being precipitated by this sort of sea change over the last four years.
• 9:38
  SPEAKER_00: Now Airflow is being used and its importance and the use cases that businesses are relying on it for.
• 9:44
  SPEAKER_02: Exactly right.
• 9:45
  SPEAKER_02: It's the diversity of use cases and deployment patterns and the criticality of the spread across the enterprise rather than single data engineers, you probably started in your early days of working with Airflow, and then small data engineering teams, and then many small data engineering teams now becoming like in a pervasive standard.
• 10:08
  SPEAKER_02: And a lot of the changes now are towards how to make sure that Airflow continues to become the pervasive standard for data orchestration across enterprises in their entirety.
• 10:21
  SPEAKER_00: Yeah, yeah, totally.
• 10:23
  SPEAKER_00: So you mentioned a little bit how right now there's a difference between the patterns that allow somebody who's not familiar with Airflow to get started with it compared to what's actually going to be conducive towards using Airflow at a large scale and across like really large teams or many teams within an organization.
• 10:42
  SPEAKER_00: Yes, potentially that leads me into my next question, which Airflow 3 is already in development.
• 10:49
  SPEAKER_00: And so there are already plans for what's going to be in it that are well underway.
• 10:54
  SPEAKER_00: What do you think is most important for the Airflow users out there to be aware of that is coming?
• 11:02
  SPEAKER_02: Let me answer that question in two parts.
• 11:05
  SPEAKER_02: For an existing user Airflow, somebody who has already been using Airflow, who's actually happy with Airflow.
• 11:14
  SPEAKER_02: Our first goal with Airflow 3 is to make it easier to use for existing use cases and patterns.
• 11:21
  SPEAKER_02: So we want to make sure that we hear the community, we hear the people who are already using Airflow, and to respond to what they feel is important, what they voted for based on filling in the surveys.
• 11:33
  SPEAKER_02: And for the people who are on the call listening to the call, please do take time to fill out the next Airflow survey.
• 11:40
  SPEAKER_02: It's going to be coming out in a few weeks.
• 12:13
  SPEAKER_02: There's going to be a bunch of features based on that.
• 12:16
  SPEAKER_02: The second theme is a dramatic expansion to Airflow for new use cases and deployment patterns.
• 12:24
  SPEAKER_02: Specifically, they want you to be able to run tasks anywhere at any time using any language.
• 12:33
  SPEAKER_02: Now I know that's a pretty broad statement.
• 12:39
  SPEAKER_02: So the first thing meaning running tasks anywhere.
• 12:43
  SPEAKER_02: Airflow 3 will now support multi-cloud deployments, including those where the Airflow server components can be running on one cloud, and the Airflow workers can be running on a completely different cloud.
• 12:57
  SPEAKER_02: Including an enterprise private cloud, also on edge deployments.
• 13:02
  SPEAKER_02: The reason for this is we want to enable integration with those enterprise applications, which can continue to be isolated to meet security or privacy needs.
• 13:44
  SPEAKER_02: I mentioned data-driven scheduling.
• 13:46
  SPEAKER_02: And now I'm really excited to talk about a dramatic expansion with full event-driven scheduling based on the data asset framework, which is the next generation of data sets.
• 13:57
  SPEAKER_02: Airflow dives can now run automatically when incoming data just shows up in a bucket, in a blob storage or an update to a database table.
• 14:07
  SPEAKER_02: Data-driven scheduling originally introduced in Airflow a couple of years ago, was limited to data handling within Airflow or within an Airflow deployment.
• 14:15
  SPEAKER_02: This now goes beyond that to enable dives to be clicked from data showing up from external data systems.
• 14:22
  SPEAKER_02: And it's a foundation for a far more complete event-based integration.
• 14:29
  SPEAKER_02: And as part of the temporal element of making things better with Airflow, we're also introducing a notion of data partitioning that enables targeted and efficient processing of large data sets.
• 14:42
  SPEAKER_02: And in addition to this, we're also adding the ability to do ad hoc scheduling in Airflow so that DAX can run independent of any data interval.
• 14:53
  SPEAKER_02: Now, this is a critical feature needed to support Gen AI inference execution and is also relevant for hyperparameter tuning, MAI models, and a lot, and has been requested by multiple AI users in different ways and forms.
• 15:09
  SPEAKER_02: And is one of the key things which we're going to be doing as part of Airflow 3.
• 15:13
  SPEAKER_02: And as a result of these changes, Airflow 3 will now support the full range of DAG invocations.
• 15:20
  SPEAKER_02: I schedule.
• 15:25
  SPEAKER_02: Now the third thing I made a comment about is run in any language.
• 15:29
  SPEAKER_02: And with Airflow 3, you can run fast in any language, not just Python.
• 15:36
  SPEAKER_02: As we've seen more and more software teams starting to build data applications, we want them to be able to use Airflow in their language of choice.
• 15:45
  SPEAKER_02: I we want to meet them where they are, not required that they have to write their applications in a new language.
• 15:52
  SPEAKER_02: And therefore, to summarize the theme, which the second big theme of Airflow 3 really is to be able to run paths anywhere at any time in any language.
• 16:06
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:15
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:27
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:36
  SPEAKER_00: There's a lot to unpack and a lot to be really excited about.
• 16:40
  SPEAKER_00: I'm gonna work, I think, backwards from the way you just said it.
• 16:44
  SPEAKER_00: I mean the way that I remember it.
• 16:45
  SPEAKER_00: Run in any language is interesting to me.
• 16:48
  SPEAKER_00: I, as you said before, do a lot of work with the Airflow community for teaching people how to use Airflow, especially new Airflow users.
• 16:55
  SPEAKER_00: Um, oftentimes, data engineers have some familiarity with Python, but we also encounter lots of people who do not, or as you said, they're not coming from a place where their current pipelines are written in Python.
• 17:09
  SPEAKER_00: Airflow right now does have some ability to run tasks that are written in other code languages.
• 17:16
  SPEAKER_00: So if it's a SQL script or something, there might be operators there, or even up to if you wanted to run, say, some JavaScript thing, you could do it with a Kubernetes pod operator or maybe a bash operator, but it would take quite a bit of setup on the part of the user to get an environment in which you could run that task.
• 17:36
  SPEAKER_00: And then obviously the pipelines themselves, the DAGs themselves are defined in Python.
• 17:41
  SPEAKER_00: When you're saying run in any language, are you talking more about like more native support for kind of the first piece that I was talking about, just the tasks themselves that might be written in another language or the whole DAG logic itself, or both?
• 17:57
  SPEAKER_02: So to start with, it's the task, which is primarily the biggest amount of code which most people write.
• 18:04
  SPEAKER_02: And that is the reason why the Qt port operator has become pretty popular.
• 18:09
  SPEAKER_02: And originally, honestly, uh way back when I always considered the Qt port operator to be something like an escape hatch.
• 18:16
  SPEAKER_02: You do that if there is no other option, right?
• 18:20
  SPEAKER_02: And what we really want with the native language integration is so that you can actually have excoms with that data.
• 18:27
  SPEAKER_02: You can actually say, hey, what's going on?
• 18:29
  SPEAKER_02: It it's like a full-fledged Pythonic kind of model, but it doesn't have to be in Python, right?
• 18:34
  SPEAKER_02: So you can actually get connections, you can access secrets, you can access environment variables, you can pull an XCOM data for the from prior task for this particular task to run.
• 18:45
  SPEAKER_02: You can then push XTOP data for like subsequent tasks to be able to take the data you produced in this task.
• 18:52
  SPEAKER_02: And this task can be in any language you want.
• 18:55
  SPEAKER_02: It could be in Go, it could be in, as you said, JavaScript, it could be in any different language you want.
• 19:00
  SPEAKER_02: And it would be native to have the same level of interaction, abstraction, and integration like a Python task.
• 19:07
  SPEAKER_02: And because we've seen very complex tasks, and an airflow task is unusual in a way that it is a complete end-to-end task, right?
• 19:18
  SPEAKER_02: Like an airflow operator.
• 19:20
  SPEAKER_02: If I'm getting my data from Salesforce, for example, I want to put this into stuff like Databrights or Redshift or BigFit, whatever, right?
• 19:29
  SPEAKER_02: It's an entire operation, and that is encapsulated within a single task.
• 19:35
  SPEAKER_02: So Airflow tasks tend to be fairly complex.
• 19:38
  SPEAKER_02: And that's the reason why we wanted to make this available in the languages of choice.
• 19:42
  SPEAKER_02: For example, you have an enterprise application which is written in Java, and you know how to interact with that in Java.
• 19:48
  SPEAKER_02: You've got the classes, you've got the interfaces, you know exactly how to do that.
• 19:52
  SPEAKER_02: And that makes it so much easier than how to say, now I need to expose a different public API, which can be accessed using Python and all the challenges towards having to make all that happen rather than just natively integrating with application.
• 20:07
  SPEAKER_02: At this point in time, we're really focused on tasks.
• 20:11
  SPEAKER_02: Based on what people come up with and feedback and things like that, we could expand this to the DACs themselves.
• 20:18
  SPEAKER_02: But the first level of integration which we're providing is the task level.
• 20:23
  SPEAKER_02: Initial feedback has been, hey, that's the right place to be.
• 20:26
  SPEAKER_02: And then let's see how this goes.
• 20:27
  SPEAKER_02: Airflow DAGs can actually be multilingual as part of this change.
• 20:32
  SPEAKER_02: So in a single DAG, you can have Python tasks, you can have SQL tasks, you can actually have Golang tasks and all of these levels.
• 20:40
  SPEAKER_02: So that is the reason why we believe that this is the right place to be at this point in time, and of course, very interested in hearing feedback from the community about saying, hey, is this necessary but not sufficient?
• 20:54
  SPEAKER_02: This is necessary and sufficient.
• 20:56
  SPEAKER_02: Love to hear more.
• 20:58
  SPEAKER_00: Yeah, that's really cool.
• 20:59
  SPEAKER_00: I'm gonna go back to my past life pre-astronomer.
• 21:02
  SPEAKER_00: I have spent months of my life rewriting tasks into Python so that they could be run in Airflow.
• 21:08
  SPEAKER_00: I'm continually always thinking back to that time about how much easier my life would have been if I had been using it now as opposed to, I want to say I was maybe on 1.9 at the time, early 2019.
• 21:20
  SPEAKER_00: It was still pretty great.
• 21:21
  SPEAKER_00: It was much better than the tool we transitioned off of.
• 21:24
  SPEAKER_00: But I can see where, especially for, as you said, new users to Airflow who are coming onto it, that will lower the barrier to entry quite a lot if you don't even have to think about what language the tasks are currently written in.
• 21:37
  SPEAKER_02: Exactly.
• 21:38
  SPEAKER_02: I'd love to get feedback during this particular process.
• 21:42
  SPEAKER_02: Uh, there's an entire set of ecosystems and potentially operators we'd need to build for these different languages.
• 21:49
  SPEAKER_02: But again, that's the reason for wanting to have conversations with the community.
• 21:53
  SPEAKER_02: Clearly, the first one would be the equaler, the Python operator.
• 21:57
  SPEAKER_02: I hit rid of the Go operator.
• 22:00
  SPEAKER_02: Right.
• 22:00
  SPEAKER_02: And then you can start to say, hey, what interactions do I want?
• 22:04
  SPEAKER_02: And how do I connect to self and all that kind of stuff?
• 22:07
  SPEAKER_02: I'd love to learn more about what the community would want over and beyond that beginning.
• 22:11
  SPEAKER_02: But we're really excited about this.
• 22:13
  SPEAKER_00: Yeah, that's great.
• 22:15
  SPEAKER_00: We I feel like we could do an entire podcast about each one of these, and hopefully we will when the time comes and they come out.
• 22:22
  SPEAKER_00: I do want to just touch a little bit more on the data sets aspect of it.
• 22:28
  SPEAKER_00: So that run anytime piece, because this is something I even the data sets feature was really impactful and has been expanded a lot in even recent versions of Airflow, as recent as 210, which just came out a couple of months ago.
• 22:41
  SPEAKER_00: I still see a lot of the kind of misrepresentation of Airflow in the broader data engineering community as not an event-driven tool that it's only used for kind of time-based batch schedules.
• 22:58
  SPEAKER_00: Obviously, datasets solves a lot of this for a lot of use cases, but you're looking at going much further than they currently do in 3.0, which, as you said, will open up some very specific use cases like for those Gen AI cases, but are often relevant for anybody even using ETL cases.
• 23:19
  SPEAKER_00: So maybe for just a minute here, to talk a little bit more about kind of the importance behind that and what that will unlock for users.
• 23:26
  SPEAKER_02: Absolutely.
• 23:27
  SPEAKER_02: It is data sets with all those features, which even when we originally introduced it in Airflow, there was a very long AIP.
• 23:38
  SPEAKER_02: And we said there was a let's get the core thing out first.
• 23:42
  SPEAKER_02: And then there was a lot of future work really defined at that point, and saying we'll get to that once we see what the initial adoption is, if we get the right shape and so on and so forth.
• 23:51
  SPEAKER_02: And after a couple of releases, including evolution, including in 210, we definitely feel like we're poised for the next big jump in data sets.
• 23:59
  SPEAKER_02: So there is a fundamental change happening, which is the introduction of data assets, which is the higher level of abstraction on top of data sets, towards being able to incorporate not just physical data sets, i.e., files and tables for the ECON, but also abstract data sets such as models and others.
• 24:18
  SPEAKER_02: And data assets, which is a new framework, which is going to subsume data sets, also the ability to define effectively processing, for lack of a better phrase, along with data assets.
• 24:30
  SPEAKER_02: So the really the event-driven scheduling is based on the integration of data assets with the combination of triggers or sensors.
• 24:39
  SPEAKER_02: So you could say, hey, this is my data asset.
• 24:42
  SPEAKER_02: It's actually going to get produced by somebody else.
• 24:44
  SPEAKER_02: And when that production happens, I this lines, uh, let's trigger everything else based on that.
• 24:51
  SPEAKER_02: And that's really the foundation for event-driven scheduling towards being able to leverage what we already have, take it up to the next level of abstraction, towards being able to pull stuff.
• 25:02
  SPEAKER_02: And then that's going to come up so that you can actually pull data from external systems.
• 25:06
  SPEAKER_02: When external systems produce data, Airflow automatically can trigger its tags based on this external event coming up, which absolutely leads towards very standard micro-batching capabilities, very simple triggering.
• 25:21
  SPEAKER_02: You don't have to think about it, it automatically actually happens.
• 25:24
  SPEAKER_02: And also leads to other capabilities which we can add over time of a full-fledged event bus integration, pub sub integration for all of these.
• 25:34
  SPEAKER_02: We don't feel like Airflow should have a pub sub integrated under it.
• 25:37
  SPEAKER_02: That would be that's actually a different system.
• 25:40
  SPEAKER_02: But this enables Airflow services and Airflow users to be able to integrate those two concepts very seamlessly for an entire event framework or like a data ecosystem, which they may actually have.
• 25:53
  SPEAKER_02: So that's the reason why, again, we could dive into this a lot more and say, you know, hey, how does it trigger subsequent events?
• 25:59
  SPEAKER_02: And but it enables that full-fledged integration from an end-to-end standpoint.
• 26:04
  SPEAKER_02: So data being produced by anywhere can trigger an airflow pipeline immediately.
• 26:10
  SPEAKER_00: Yeah, that's great.
• 26:11
  SPEAKER_00: And this is very exciting.
• 26:13
  SPEAKER_00: We just have a couple of minutes left, and I do really want to touch on my next question, which is there's a ton of work going into making all of these changes that you've described.
• 26:23
  SPEAKER_00: These are huge improvements to Airflow.
• 26:26
  SPEAKER_00: Very briefly, can you talk a little bit about the process that Airflow PMC and the community is going through for making this release happen?
• 26:34
  SPEAKER_00: And then maybe touch on as well if community members do have the feedback that you're saying you guys want or want to contribute.
• 26:40
  SPEAKER_00: How can they do that?
• 26:41
  SPEAKER_02: Sure.
• 26:42
  SPEAKER_02: We started very active discussions about Airflow 3 in April.
• 26:46
  SPEAKER_02: We completed the definition phase in the first week of August.
• 26:49
  SPEAKER_02: This involved lots of discussions, definition of over 15 Airflow improvement proposals, comments and voting LBs, and a lot more.
• 26:58
  SPEAKER_02: All of these were conducted by the Airflow Dev mailing list.
• 27:02
  SPEAKER_02: And that's in the spirit of the Apache Software Foundation, using the Airflow Wiki as a source of truth for all information.
• 27:08
  SPEAKER_02: So all the proposals are on the wiki.
• 27:11
  SPEAKER_02: We've also been having bi-weekly dev calls when everybody is welcome to join and voice their input because you need some live interaction as a conversation rather than only working async.
• 27:22
  SPEAKER_02: It's been a completely open process, and I personally feel privileged to be part of this wonderful collaborative community.
• 27:29
  SPEAKER_02: We welcome feedback from everybody, and we welcome contributions as much as the feedback as well.
• 27:37
  SPEAKER_02: Anybody from the community should subscribe to the dev mailing list or follow along on Slack, on the Airflow Slack.
• 27:43
  SPEAKER_02: There's also a specific channel inside that for Airflow3 Dev Channel.
• 27:48
  SPEAKER_02: And additionally, they should look at the IPs on the AirflowWiki and they can leave comments or ask questions on the mailing list.
• 27:55
  SPEAKER_00: Yeah, that's great.
• 27:56
  SPEAKER_00: And we can definitely throw a link to the dev list and to the Airflow 3 call link in the show notes here so that folks have that available.
• 28:04
  SPEAKER_00: I'll make one final plug for the Airflow survey just because it's going to be coming out.
• 28:08
  SPEAKER_00: It might have already been released by the time this episode goes live.
• 28:12
  SPEAKER_00: And a lot of the features that you touched on, especially DAG versioning, came to mind.
• 28:17
  SPEAKER_00: We actually just took it out of the what would you like to see an airflow survey question because it's already in development.
• 28:24
  SPEAKER_00: As you said, that feedback is taken very seriously and is the impetus for a lot of the development that happens, like we were talking about today.
• 28:32
  SPEAKER_00: I know we're over time, as I said.
• 28:34
  SPEAKER_00: I think we could talk for three more hours easily about this, but we'll leave it there for now.
• 28:39
  SPEAKER_00: We will definitely have more episodes and lots more content from Astronomer about Airflow 3 in the months leading up and obviously after the release.
• 28:47
  SPEAKER_00: So more to come there.
• 28:49
  SPEAKER_00: Vikram, it was really great to have you on.
• 28:51
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 28:54
  SPEAKER_02: Thank you for having me, Kenton.
• 28:56
  SPEAKER_02: I'd love to hear from folks.
• 28:58
  SPEAKER_02: I'm reachable on LinkedIn as Vikram Koker and over email at Vikrametastomer.io.
• 29:04
  SPEAKER_00: Okay.
• 29:04
  SPEAKER_00: Thanks again, Vikram.
• 29:06
  SPEAKER_00: We really appreciate your time.
• 29:07
  SPEAKER_00: And hopefully we will have you on the podcast again soon to talk more about Airflow3.
• 29:12
  SPEAKER_02: Thanks, Kenton.
• 29:13
  SPEAKER_02: Plus for chatting with you as always.
• 29:15
  SPEAKER_00: Airflow users, we have exciting news.
• 29:18
  SPEAKER_00: The annual Apache Airflow survey is live.
• 29:21
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow.
• 29:25
  SPEAKER_00: And as a thank you for filling it out, you'll get a free Airflow Fundamentals or DAG Authoring certification.
• 29:30
  SPEAKER_00: Those are worth$150 each.
• 29:33
  SPEAKER_00: Plus, you'll also be entered to win an exclusive workshop with Mark Lumberti.
• 29:37
  SPEAKER_00: So make sure you don't miss out.
• 29:39
  SPEAKER_00: The survey link is in the show notes and it closes December 20th.
• 29:47
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 29:54
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 29:57
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.