Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Introducing Airflow 3.2

Transcript

Introducing Airflow 3.2 · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Well in Airflow 3, we are not talking about having async function within your Python operator.
• 0:05
  SPEAKER_02: We are really saying that now you can put async next to your Python operator.
• 0:10
  SPEAKER_02: So it's actually the task, the airflow task that becomes asynchronous.
• 0:19
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:28
  SPEAKER_00: Hi everyone.
• 0:29
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:31
  SPEAKER_00: I'm your host, Kenton Danis.
• 0:33
  SPEAKER_00: And today I'm joined by our other host, Mark Lumberti.
• 0:37
  SPEAKER_00: Mark, how are you?
• 0:39
  SPEAKER_02: Super excited to be in this uh in this episode.
• 0:42
  SPEAKER_02: Yeah.
• 0:43
  SPEAKER_02: A lot to talk about.
• 0:44
  SPEAKER_00: Yes, we do have a lot to talk about.
• 0:46
  SPEAKER_00: Well, listeners, you get both of our hosts today because we have a special episode to talk about what is new in the Airflow 3.2 release, which I think as of publishing, this probably the release probably should have come out in the last week or so, depending.
• 1:03
  SPEAKER_00: And yeah, we're excited to dive into it.
• 1:06
  SPEAKER_00: Uh Airflow minor releases happen every, you know, three to six months or so.
• 1:12
  SPEAKER_00: And they always bring a lot of updates for the project.
• 1:14
  SPEAKER_00: So uh we want to make sure that everybody is aware of what's new in this one.
• 1:19
  SPEAKER_00: So with that, Mark, I guess before we dive into any specifics, I think this Airflow 3.2 in my mind is still notable because it's continuing along with the momentum of Airflow 3, of course, which was the new major release about a year ago, uh, a huge milestone for the project.
• 1:38
  SPEAKER_00: And we hope that everybody is already using Airflow 3 or at least aware of what's in it and working towards upgrading.
• 1:44
  SPEAKER_00: But maybe for those who have not upgraded yet, we can spend a minute talking about, you know, why the major release last year, why it came out in the first place and why it was important for the project.
• 1:57
  SPEAKER_02: Yeah, no, absolutely.
• 1:58
  SPEAKER_02: I think uh I mean definitely Airflow 3 was uh was a huge release.
• 2:01
  SPEAKER_02: I mean, I don't even remember when Airflow 2 was released, but it was a few years ago for sure.
• 2:07
  SPEAKER_00: 2020, December of 2020, at the very end of the year.
• 2:11
  SPEAKER_00: I do remember very clearly.
• 2:15
  SPEAKER_02: So, like four years later, um you actually get a new a new Airflow release.
• 2:20
  SPEAKER_02: So, yeah, no, Airflow 3 uh was was huge for for many reasons.
• 2:24
  SPEAKER_02: I think the first one is definitely the architecture, all the the changes.
• 2:28
  SPEAKER_02: So basically now you have you know your components, like the workers actually do not communicate with your metadatabase directly anymore.
• 2:36
  SPEAKER_02: And this is important for a few reasons, but uh one is that now you can have your Airflow instance running in one cluster and then your workers running in another cluster, like somewhere else, completely separated from your Airflow instance.
• 2:50
  SPEAKER_02: And so this is great for security reasons, uh for you know uh compliance reasons and so on.
• 2:56
  SPEAKER_02: So that gives a lot of flexibility.
• 2:58
  SPEAKER_02: Also, you have this API server, which is you know still like the like the it serves the user interface, but also it's actually the the interface between the workers and the rest of your airflow instance.
• 3:11
  SPEAKER_02: So you have all uh those archite those new architectural uh um uh choices that are major.
• 3:18
  SPEAKER_02: And then the second one is DAG versioning.
• 3:20
  SPEAKER_02: I think DAG versioning was huge.
• 3:22
  SPEAKER_02: I mean, I I I remember how many times I got people asking about DAG versioning, like when it's gonna be released.
• 3:31
  SPEAKER_02: That that was almost like a running joke at some point.
• 3:34
  SPEAKER_02: Uh but now we finally have DAG versioning in Airflow 3 and also the backfilling mechanism.
• 3:39
  SPEAKER_02: Now you can backfill your DAGs directly from the Airflow UI.
• 3:42
  SPEAKER_02: Uh, if you remember in Airflow 2, you had to use the Airflow CLI and it was a little bit cunky.
• 3:47
  SPEAKER_02: Um but now with with Airflow 3, the backfilling process is actually really part of the of the scheduler, and and you know, it you can run it from the Airflow UI, you don't have to go through the CLI anymore.
• 3:59
  SPEAKER_02: Even you can even use the API, I believe, uh, for running your backfill jobs.
• 4:04
  SPEAKER_02: So a lot of good features like that.
• 4:06
  SPEAKER_02: Uh, and you know, there are many that I'm forgetting right now, but just for those, I will definitely uh upgrade my my Airflow version to Airflow 3.
• 4:16
  SPEAKER_00: Definitely those are very useful.
• 4:17
  SPEAKER_00: And I think some of the ones we've seen with the highest adoption so far, including among our customer base.
• 4:23
  SPEAKER_00: And I like that you talked about the architectural changes because I do think that was a big reason for the major version release and also just helping get Airflow in a spot where it can evolve more quickly to support new types of workflows that we see uh gaining popularity.
• 4:40
  SPEAKER_00: So AI workflows, managing those with Airflow, MLOps, things like that.
• 4:44
  SPEAKER_00: I also you mentioned DAG versioning and all of the requests we got for it.
• 4:49
  SPEAKER_00: I think it was always the number one requested feature uh in the survey and even just kind of out in the wild in our discussions with the community.
• 4:59
  SPEAKER_00: So plug for filling out the survey because it's important to determine what comes in the project.
• 5:04
  SPEAKER_00: I think also relevant, uh, I hope I'm not pulling making this up, but I think that one of the new features in 3.2, which kind of builds on some of these changes that came in Airflow 3, is asset partitions, which I think was also a very highly requested feature within some previous surveys.
• 5:24
  SPEAKER_00: So maybe we can talk about that one first.
• 5:26
  SPEAKER_00: I think it's one of the bigger updates to Airflow in version 3.2 and expanding on this asset and data awareness functionality that I think existed to some degree in Airflow 2, but really got a big kind of boost forward in Airflow 3.
• 5:41
  SPEAKER_00: So, yeah, what what do you think of this feature so far?
• 5:44
  SPEAKER_02: Yeah, I think um, so yeah, I mean that's that's uh that's actually a very good point.
• 5:49
  SPEAKER_02: Um, well, just for uh, you know, just for for um as a quick reminder for everybody, uh remember that now in Airflow 3 you have the assets which were the data sets uh in Airflow 2.
• 6:00
  SPEAKER_02: So it's just the name that it is different.
• 6:01
  SPEAKER_02: But to your point now, Airflow is becoming more and more asset aware.
• 6:06
  SPEAKER_02: And part of that is the new uh asset partitioning mechanism that you will get in 3.2.
• 6:12
  SPEAKER_02: And so, you know, like so far, before 3.2, uh you were able to schedule your DAGs on assets.
• 6:19
  SPEAKER_02: So typically you have one DAG or actually one task that produces an asset and that triggers a downstream DAG that it is scheduled on that specific asset.
• 6:28
  SPEAKER_02: But it was, you know, it was just like that asset is materialized and then it triggers the downstream DAG.
• 6:34
  SPEAKER_02: But with asset partitioning, it's a little bit more granular.
• 6:37
  SPEAKER_02: Uh actually that adds uh granularity to your assets because you can attach a partition key to your asset.
• 6:44
  SPEAKER_02: So, what does it mean?
• 6:46
  SPEAKER_02: It means that again, let's say you have like sales data, okay, and uh you want to trigger a downstream DAG, not just like for the day, but like let's say you get uh some, you have a DAG that produces an asset at uh 2 p.m.
• 7:02
  SPEAKER_02: And then you have another DAG that produces the same asset but for different sales data on 2 p.m.
• 7:08
  SPEAKER_02: as well.
• 7:08
  SPEAKER_02: You can attach a partition key, like on on actually on the um like an hourly partition key, so that like you will trigger the downstream DAGs on that hourly partition and not on the on the entire asset.
• 7:23
  SPEAKER_02: Okay, if that makes sense.
• 7:25
  SPEAKER_02: So you can clearly go through, you know, like you don't trigger your DAG on just the uh entire asset, you trigger your DAG on the partitions within that asset.
• 7:34
  SPEAKER_02: So it can be uh hourly, it can be uh I think like your yearly as well.
• 7:38
  SPEAKER_02: You have multiple partitions like that that you can define in 3.2.
• 7:42
  SPEAKER_02: And so again, that gives a lot of uh flexibility that you don't have until now in 3.2.
• 7:48
  SPEAKER_02: So you can have like hourly batches, daily stuff, and actually something that I forgot to mention you can you can even have uh composite keys, right?
• 7:56
  SPEAKER_02: So you can have like based on time, but you can also add some, I don't know, like you can have the time and then the region.
• 8:04
  SPEAKER_02: Okay, so for example, if you have in your asset sales data but in different regions like uh Europe, USA, and so on, you can even add that as a partition key for your asset, right?
• 8:15
  SPEAKER_02: Like you can add you and then a pipe and then uh the time, for example.
• 8:20
  SPEAKER_00: Yeah, I think this is so exciting.
• 8:23
  SPEAKER_00: And to your point about this, both helps you with scheduling your pipeline so that they're uh just more responsive to the data that you're actually processing, but also you can see these assets or the partition keys in the airflow UI.
• 8:40
  SPEAKER_00: So as the user, you can go in and understand like which DAG run ran for like actually which subset of your data without digging into the specific tasks.
• 8:49
  SPEAKER_00: So yeah, definitely more granular, as you said.
• 8:52
  SPEAKER_02: Yeah, it it's really about like you actually, you know, it's really about uh carrying much of like uh which slice of the of the asset was updated, you know, instead of the entire asset, you truly care about which slice of the asset was uh was updated.
• 9:07
  SPEAKER_02: And this is what asset partitioning allows you to do now.
• 9:10
  SPEAKER_00: Yeah, I think this will be huge for users because at least the this was true for me when I worked as a data engineer.
• 9:16
  SPEAKER_00: Maybe it was true for you too.
• 9:17
  SPEAKER_00: I I feel like everybody is partitioning their data in terms of how it's processed, like with batch workflows, there's just always going to be some amount of looking at this sort of window, or maybe it's not by time, but by some other characteristic, like you said, region or something like that.
• 9:36
  SPEAKER_00: So like everybody has to do this already, and now it's just a little bit easier in Airflow.
• 9:42
  SPEAKER_02: Yeah, and you will be, I think you will be able to.
• 9:44
  SPEAKER_02: I didn't try that yet, but I think you will be able to backfill your DAG still, even with asset partitioning.
• 9:49
  SPEAKER_02: I mean, I'm pretty sure that you will be able to.
• 9:50
  SPEAKER_00: Yeah, definitely will be helpful for maintaining items, and you know, not trying to reprocess a bunch of data that you don't need to reprocess if you do have to do that.
• 10:00
  SPEAKER_00: Yeah.
• 10:01
  SPEAKER_00: Yeah.
• 10:01
  SPEAKER_00: So that's very exciting.
• 10:02
  SPEAKER_00: Well, we I I should have mentioned at the start, we'll have resources uh to help with adoption of all of these new features.
• 10:09
  SPEAKER_00: So we'll link some of these in the show notes if anybody wants to learn how to implement them.
• 10:13
  SPEAKER_00: I think over audio is not the best way for us to talk about the code implementation.
• 10:17
  SPEAKER_00: So we'll leave that to some of the written content and link to it.
• 10:21
  SPEAKER_00: Uh, but yeah, very excited about this next step forward for assets and data awareness in general.
• 10:27
  SPEAKER_00: Uh, and it's also not the end of the story for this feature.
• 10:32
  SPEAKER_00: So there's even, I think there are at least a couple of other AIPs planned for 3.3 and beyond.
• 10:38
  SPEAKER_00: So uh Airflow is just gonna get better and better with regards to that data awareness piece.
• 10:44
  SPEAKER_00: Changing direction a little bit, another really big feature, not related to assets and data awareness, uh, that I'm really excited about in 3.2 is the new async Python operator functionality.
• 10:58
  SPEAKER_00: Have you had a chance to try this yet?
• 11:01
  SPEAKER_02: Yeah, and actually that's a that's a really cool feature.
• 11:04
  SPEAKER_02: Um that's a really cool feature.
• 11:06
  SPEAKER_02: Like so some of you may may be uh a little bit like, well, I was able to use async before uh in Airflow 2.
• 11:12
  SPEAKER_02: So what exact what does that mean?
• 11:14
  SPEAKER_02: All right.
• 11:15
  SPEAKER_02: Well, in Airflow 3, we are not talking about having uh you know an async function within your Python operator.
• 11:21
  SPEAKER_02: We are really saying that now you can put async next to your Python operator.
• 11:26
  SPEAKER_02: Because so it's actually the task, the Airflow task that becomes uh asynchronous.
• 11:31
  SPEAKER_02: And you know, so before like diving into that feature, I think it's important to uh talk a little bit about why we have this feature in 3.2, um, because you can also think about deferable operators.
• 11:43
  SPEAKER_02: And actually, deferable operators are super useful.
• 11:45
  SPEAKER_02: So if you don't know about them very quickly, deferable operators are useful because they don't waste resources when, for example, you have your operator that is waiting for something.
• 11:56
  SPEAKER_02: Okay, so typically, if you have the Spark submit job operator, something like that, okay, you submit your Spark job and then your operator is waiting for this job to complete.
• 12:06
  SPEAKER_02: While the operator is waiting, you are taking a worker slot and you have a limited number of worker slots.
• 12:11
  SPEAKER_02: So typically you have a limited number of tasks that you can run in your reference stance, uh, just for simplicity.
• 12:18
  SPEAKER_02: And the the idea is if you use the deferable version of this operator, then you are not using a worker slot anymore and you are not wasting resources waiting for something anymore.
• 12:28
  SPEAKER_02: Okay, so this is more uh this is great actually.
• 12:32
  SPEAKER_02: And that scales pretty well until some point, right?
• 12:35
  SPEAKER_02: So typically, you know, like behind the scene deferable operators work using uh triggers and the trigger.
• 12:42
  SPEAKER_02: I won't dive into the details here, but if you have many, many um deferable operators that can put a little bit of pressure on your database.
• 12:51
  SPEAKER_02: Because remember that the data that a deferable operator returns will be stored in the Airflow meta database by default.
• 13:00
  SPEAKER_02: Okay, you have no way actually to use the XCOM backend.
• 13:02
  SPEAKER_02: So typically, if you use a deferable operator and you fetch a lot of data from it, then again, that can be an issue because the data will be stored in the uh database.
• 13:13
  SPEAKER_02: Also, keep in mind that you know when your operator is deferred, that is like uh, you know, it goes through uh the worker and then the scheduler and then the trigger.
• 13:22
  SPEAKER_02: I mean, this is a wall that involves a round trip, that it is not that long, but if you have again many deferable operators, it can be.
• 13:30
  SPEAKER_02: So the point here is that deferable operators are great, but the async operators might be better.
• 13:36
  SPEAKER_02: So the idea of the async operator is actually to solve those issues, okay?
• 13:41
  SPEAKER_02: Because now you don't have to go through this round trip anymore.
• 13:45
  SPEAKER_02: Okay, you don't have to go through the worker first and then the scheduler and then the trigger and then the scheduler back and so on.
• 13:50
  SPEAKER_02: Okay, so that saves time, especially again if you have like thousands of tasks that are that were using the deferable operator.
• 13:59
  SPEAKER_02: Uh I mean a deferable operator.
• 14:01
  SPEAKER_02: And also that does not put any pressure on your database because now with the async operator, you can actually store data well within a different backend.
• 14:11
  SPEAKER_02: Okay, you don't have to use the airflow metadata base for that.
• 14:14
  SPEAKER_02: And last but not least, uh keep in mind that many operators uh didn't have their deferred version, that their deferable version.
• 14:23
  SPEAKER_02: And so in some cases, you had to use the traditional operators.
• 14:28
  SPEAKER_02: And again, with the async Python operator, now you can run asynchronous operations and you can do whatever you want because the Python operator is just Python, right?
• 14:39
  SPEAKER_02: Like within the Python operator, you can do whatever you want, uh as long as it is in Python.
• 14:44
  SPEAKER_02: So you will save a lot of time, you will save a lot of resources, and just to be clear, uh like it's not gonna be, you know, it's not gonna make a single execution faster, but you will be able to have multiple uh executions within the same loop.
• 14:59
  SPEAKER_02: Okay, so that's the idea here.
• 15:01
  SPEAKER_02: Uh you are still you are gonna use a worker using the async operator, but you will save a lot of IO operations and uh performances using the async operator, the async Python operator over the deferable operators.
• 15:14
  SPEAKER_02: And again, this is true truly if you are if you have a lot of um uh deferable operators today.
• 15:23
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 15:32
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 15:44
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 15:52
  SPEAKER_00: Yeah, I think that's a great description.
• 15:54
  SPEAKER_00: I would add a couple of things.
• 15:56
  SPEAKER_00: One is that we know uh from the data that we've seen and from what the community has said in previous surveys that the Python operator is by far the most used operator in Airflow.
• 16:07
  SPEAKER_00: So it's great if there are, you know, specialized kind of operators for what you're doing, um, things like the SQL execute query operator, FiveTrain operator, et cetera.
• 16:19
  SPEAKER_00: But not everybody uses those.
• 16:21
  SPEAKER_00: Lots of people just write their own Python functions.
• 16:23
  SPEAKER_00: So this is the first time that those folks have access to kind of better asynchronous functionality.
• 16:30
  SPEAKER_00: I think you described really well like the performance issues of using a lot of deferable operators at the same time.
• 16:37
  SPEAKER_00: And I think there's still definitely a place for those.
• 16:41
  SPEAKER_00: Like if you have, if you are working with a provider where maybe I'll choose the five trend operator as an example where like there's a very well-built-out provider package for the service that you're using.
• 16:52
  SPEAKER_00: Uh, you have these long-running jobs and you don't care about the extra overhead of you know the communication loop between the scheduler and the worker and the triggerer, then you can still keep using those and that functionality is still there.
• 17:04
  SPEAKER_00: Um, and it works really well for those folks.
• 17:07
  SPEAKER_00: But if you've got something where it's a lot more high throughput, you want to write your own custom Python, this new async functionality will be really helpful.
• 17:16
  SPEAKER_00: I think also I saw in some of the uh like testing metrics for this feature that were thrown into some of the PRs uh to the open source repo, that the performance improvements are pretty staggering.
• 17:29
  SPEAKER_00: Like these are taking functions that might have taken hours to run before and now take a couple minutes.
• 17:36
  SPEAKER_00: So if that's what you're doing, if you're hitting, you know, a database a whole bunch of times or an API a bunch of times or downloading many, many, many files uh and you want to do that in your own Python, this will be a huge help.
• 17:48
  SPEAKER_02: Yeah, actually, that's a very good point.
• 17:50
  SPEAKER_02: Like it's it's you you will see a huge improvement, especially if you are processing a lot of you know small files, like you know, like small tasks, actually, short-lived tasks.
• 18:00
  SPEAKER_02: You will see a huge improvement if you use the async Python operator.
• 18:03
  SPEAKER_02: Uh but the deferable operator has still a lot of, I mean, you know, it's still important to use it.
• 18:08
  SPEAKER_02: Uh, because like typically what you do within the async Python operator, uh, you will lose a little bit of observability, right?
• 18:15
  SPEAKER_02: Because, for example, all the concurrent things tasks that you will do within the async operator, you won't be able to see that on the airflow UI, right?
• 18:23
  SPEAKER_02: Whereas like when you use the deferable operator, you see this operator as a task.
• 18:28
  SPEAKER_02: And so again, like if you have a thousand of tasks, you will see that on the airflow UI.
• 18:33
  SPEAKER_02: Whereas if you are doing a thousand operations within your async uh async uh Python operator, you won't see that on the Airflow UI.
• 18:41
  SPEAKER_02: So, you know, they are definitely it's not meant to replace deferable operators at all.
• 18:46
  SPEAKER_00: Yeah, just another tool in the toolbox.
• 18:49
  SPEAKER_00: Well, we are definitely excited about that.
• 18:52
  SPEAKER_00: Okay, so those two really big features coming in Airflow 3.2.
• 18:56
  SPEAKER_00: I think that it's also worth noting that while not part of the release itself, there are a couple of other exciting things happening in the project kind of alongside.
• 19:05
  SPEAKER_00: Um, one is the launch of the new provider registry.
• 19:08
  SPEAKER_00: We're very excited about this.
• 19:10
  SPEAKER_00: Anything you want to add on that?
• 19:12
  SPEAKER_02: No, I think it's great.
• 19:14
  SPEAKER_02: Honestly, I think it's great.
• 19:15
  SPEAKER_02: Uh, you know, like the the idea of this um registry is to provide like a centralized system where you can discover all the airflow providers and their capabilities.
• 19:25
  SPEAKER_02: So maybe you don't know that.
• 19:26
  SPEAKER_02: I mean, I'm pretty sure that you know, and when I say you know, I'm not talking about you, uh, but you know, like there are like so many um airflow providers, so many tools that you can interact with, and it can be overwhelming when you start with the airflow.
• 19:40
  SPEAKER_02: And so, again, like the idea of the registry is for you to have a single place where you can uh discover all those providers, uh, how you can use again, like um, you know, operators, hooks, connections, and so on for those different uh um providers.
• 19:56
  SPEAKER_02: So you will save a lot of time uh thanks to that, and uh that will be accessible from the uh I think the Airflow website.
• 20:02
  SPEAKER_00: Yeah, it will be on the Airflow website.
• 20:04
  SPEAKER_00: Again, we'll throw a link to that in the show notes.
• 20:07
  SPEAKER_00: I'll also just add for those who may be familiar with astronomers registry that we have provided in the past.
• 20:13
  SPEAKER_00: Uh, this is the next iteration of that.
• 20:16
  SPEAKER_00: Uh so yeah, we're really excited to have this be on the open source Airflow website and have, you know, a good one-stop shop for everybody to go.
• 20:25
  SPEAKER_00: I think this will also, you know, make it easier for folks to contribute new providers and get the information about them out there.
• 20:32
  SPEAKER_00: So uh really cool step forward for the project.
• 20:36
  SPEAKER_00: The other thing that is coming that is new that I think we won't spend a ton of time on in this episode, but we will definitely talk about in future episodes is the new common AI provider.
• 20:46
  SPEAKER_00: And because this is a provider, it's not officially part of the 3.2 release, uh, but it is coming within the same general time frame, at least experimentally at first.
• 20:56
  SPEAKER_00: Uh, but this is designed to provide a lot of functionality to make it easier to work with LLMs from Airflow, which is of course a huge topic and something that a lot of teams are grappling with right now.
• 21:08
  SPEAKER_00: So we're pretty excited about yeah, the new functionality.
• 21:12
  SPEAKER_02: Oh, yeah.
• 21:13
  SPEAKER_02: Yeah, yeah.
• 21:13
  SPEAKER_02: Uh do you want me to talk a little bit about the this provider?
• 21:17
  SPEAKER_02: No, yeah.
• 21:18
  SPEAKER_02: No, I think it's a I think it's uh it's a really interesting one.
• 21:21
  SPEAKER_02: Um it's the common AI uh provider.
• 21:24
  SPEAKER_02: So again, like the idea is to help you know anybody to integrate AI within their workflows uh with Airflow.
• 21:32
  SPEAKER_02: So maybe some of you are familiar with the uh AI SDK, but the idea of this provider is to actually provide uh some decorators that you can use, like the at LLM, LLM branch, uh, agent, and so on.
• 21:45
  SPEAKER_02: So a couple of decorators like that that ultimately will help you to use LLMs within your workflows uh and make uh some smart decisions uh with them.
• 21:54
  SPEAKER_02: So I'm very excited about this uh this provider as well.
• 21:57
  SPEAKER_00: Yeah, definitely.
• 21:58
  SPEAKER_00: Like I said, I believe.
• 22:00
  SPEAKER_00: Leave experimental for now, but there will definitely be more to come in this space.
• 22:05
  SPEAKER_00: The development's happening really quickly.
• 22:06
  SPEAKER_00: So keep an eye out for more info there.
• 22:10
  SPEAKER_00: Okay.
• 22:11
  SPEAKER_00: And then of course, this being a minor release, there's always a very long list of other features and improvements that are coming in every minor release 3.2 is no different.
• 22:22
  SPEAKER_00: We can't talk about all of them here.
• 22:23
  SPEAKER_00: We are almost out of time.
• 22:25
  SPEAKER_00: But just to call out a couple of them, Mark, of the quote, smaller features.
• 22:29
  SPEAKER_00: I'm quoting because some features are really important to different teams.
• 22:34
  SPEAKER_00: So they may not be smaller to you, but outside of the kind of headlining ones that we've talked about, what are you most excited about in this release?
• 22:41
  SPEAKER_02: Yeah, I think like um, I mean, we definitely talked about the the most uh I would say we we we already talked about the the features that I'm actually the most excited about.
• 22:52
  SPEAKER_02: But uh one that I find interesting is is now you will see the um inlets and outlets uh relationships within the asset UI.
• 23:02
  SPEAKER_02: So I like maybe you don't know that, but if you are using assets actually, the outlets parameter you should already know it because that's how you can tell that a task updates a given asset with the outlets parameter.
• 23:15
  SPEAKER_02: But the inlets parameter it's is much less used actually.
• 23:20
  SPEAKER_02: And the purpose of the inlets parameter is to say that this task is using, is consuming a given asset.
• 23:27
  SPEAKER_02: Okay.
• 23:28
  SPEAKER_02: And now within the asset UI, you will be able to see if you use the inlets and outlets parameters, you will be able to see that on the asset UI.
• 23:37
  SPEAKER_02: You will be able to see that this task is consuming this asset and produces this asset.
• 23:41
  SPEAKER_02: So it gives a better picture of actually what's going on with your assets through your tasks within the Airflow UI.
• 23:49
  SPEAKER_02: So I think this one is great.
• 23:51
  SPEAKER_02: And the other one that uh I found interesting as well is the um um optional marker that you get with DAG versioning now.
• 23:59
  SPEAKER_02: So actually you will see like a bar or something directly on the Airflow UI.
• 24:04
  SPEAKER_02: And for example, if you change the schedule of your DAGs, uh you will be able to see that you got a modification, like you got a change, or if you change a task, you will you will see that directly from the grid view where you have your different diagrams, you will see a bar indicating that after this bar, this is like DAG version two, and so on.
• 24:23
  SPEAKER_02: Okay, so you will see a clear distinction between uh your different diagrams from the grid view.
• 24:29
  SPEAKER_02: Uh so I think this is again uh very useful for you know observability and uh so you don't have to actually click on the diagram to see which DAG version was used.
• 24:39
  SPEAKER_02: So I think this this is great.
• 24:40
  SPEAKER_02: Yeah.
• 24:41
  SPEAKER_02: But again, I believe this is an optional marker.
• 24:43
  SPEAKER_00: So right, right.
• 24:45
  SPEAKER_00: Those are both great ones to call out.
• 24:46
  SPEAKER_00: I think for me personally, my favorite is the update to the retry exponential back off parameter.
• 24:53
  SPEAKER_00: So you can now provide an actual number to that as a multiplier.
• 24:57
  SPEAKER_00: Before I think you could just set it to true and it would use a 2.x multiplier.
• 25:02
  SPEAKER_00: So every time you retry, you would wait twice as much time.
• 25:05
  SPEAKER_00: Now you have more granular control over that.
• 25:07
  SPEAKER_00: So if you want to wait a lot longer or if you want to do, I don't know if it'll take like a decimal or not.
• 25:15
  SPEAKER_00: I'm not sure.
• 25:16
  SPEAKER_00: So uh something to look into.
• 25:18
  SPEAKER_00: But anyways, you have more control over your retry behavior with that parameter.
• 25:23
  SPEAKER_00: So again, small change, but it could potentially be very helpful for teams who are using that.
• 25:28
  SPEAKER_00: Okay, cool.
• 25:29
  SPEAKER_00: Well, I know we are about out of time.
• 25:31
  SPEAKER_00: So thanks, Mark, so much for going through that.
• 25:34
  SPEAKER_00: Again, we will include links to a bunch of other resources in the show notes here, including our webinar while we're where we will cover all of this stuff in more in more depth and you know, give some uh demos so you can actually see it in action in the code and all of that good stuff.
• 25:49
  SPEAKER_00: Uh, but yeah, we're really excited about the 3.2 release and we can't wait to hear what everybody uses it for.
• 25:56
  SPEAKER_02: Yep, it's gonna be exciting.
• 25:58
  SPEAKER_02: Thank you so much, Cajun, for having me.
• 25:59
  SPEAKER_00: Yeah, thanks, Mark.
• 26:02
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 26:09
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 26:13
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.