Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / What's New in Apache Airflow® 3.3

Transcript

What's New in Apache Airflow® 3.3 · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:08
  SPEAKER_01: Let's get into it.
• 0:12
  SPEAKER_02: Okay, hi everyone.
• 0:13
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:15
  SPEAKER_02: I'm your host, Kenton Dannis, and today we have another special episode where I'm joined by our other host, uh Mark Lamberti, who is our education content lead at astronomer.
• 0:24
  SPEAKER_02: We're here today to talk about the new Airflow 3.3 release.
• 0:28
  SPEAKER_02: Mark, how are you doing?
• 0:31
  SPEAKER_00: Well, very good.
• 0:31
  SPEAKER_00: Thank you so much, uh Kayton, for having me in this episode.
• 0:34
  SPEAKER_00: I'm just sweating a lot right now because it's super hot in France, but uh I will try my best for this episode.
• 0:40
  SPEAKER_02: In the middle of the European heat wave, which I hope by the time 3.3 comes out and we're airing this episode that that's over and you've gotten some relief.
• 0:49
  SPEAKER_02: Uh and all of our friends in Europe are a little bit cooler, but yeah.
• 0:53
  SPEAKER_00: I hope so.
• 0:54
  SPEAKER_00: I hope so.
• 0:55
  SPEAKER_00: Otherwise, I might be dead before before it happens.
• 0:59
  SPEAKER_02: Well, we hope not.
• 1:00
  SPEAKER_02: There's so much cool stuff in the air in the airflow project right now, you wouldn't want to miss that.
• 1:05
  SPEAKER_00: That's true.
• 1:07
  SPEAKER_02: Yeah.
• 1:07
  SPEAKER_02: All right.
• 1:08
  SPEAKER_02: Well, we'll go ahead and dive in.
• 1:10
  SPEAKER_02: Um, so like I said, for today's podcast, we're gonna talk about the new Airflow 3.3 release that has just come out.
• 1:17
  SPEAKER_02: Um we'll talk about high-level kind of some of the stuff that's in it, and then provide some additional resources for people to learn more.
• 1:25
  SPEAKER_02: And digging right in, you know, as with most minor Airflow releases, 3.3 has a ton of great stuff in it, has some new features, uh, improvements, bug fixes, all that good stuff that everyone will be interested in.
• 1:38
  SPEAKER_02: I want to start by talking about some of the biggest new features that have come in this release.
• 1:44
  SPEAKER_02: And uh one of the big updates was the addition of this new task state store, also the asset state store.
• 1:51
  SPEAKER_02: So this was AIP 103, in case anybody is following the AIPs.
• 1:56
  SPEAKER_02: Uh and this, you know, high level allows tasks to persist some amount of state information.
• 2:02
  SPEAKER_02: So I guess first, what do you think about this feature?
• 2:06
  SPEAKER_00: Yeah, I think uh I think actually this is a really, really great feature.
• 2:10
  SPEAKER_00: I'm very excited about 3.3 b mostly because of that feature, honestly.
• 2:14
  SPEAKER_00: I mean, there are other features that are great, but this one is amazing.
• 2:17
  SPEAKER_00: Um, so basically, like the you know, the idea is imagine that, for example, you are running a long-running task.
• 2:25
  SPEAKER_00: You know, it might be like a spark job.
• 2:27
  SPEAKER_00: This is actually pretty common, but it might be something else.
• 2:29
  SPEAKER_00: Just think about a long-running task, and then uh for some reason your worker dies.
• 2:34
  SPEAKER_00: Bad, right?
• 2:36
  SPEAKER_00: Um, but then like because of that, you might still have your um you know having your uh your uh job, your spark job that keeps running, but just Airflow is not aware of it anymore because the worker died.
• 2:50
  SPEAKER_00: Uh and so if you retry the task, you will end up with two Spark jobs.
• 2:56
  SPEAKER_00: Uh and you don't want that, obviously.
• 2:58
  SPEAKER_00: And so the idea of the task store, one of the ideas is for you to be able to actually reattach this job to your Airflow worker so that you kind of resume where you left.
• 3:10
  SPEAKER_00: Uh so this is pretty amazing.
• 3:12
  SPEAKER_00: Um and so the idea is, as you said, uh you have this task store that uh by default will be the Airflow metadata base behind the scene.
• 3:20
  SPEAKER_00: Uh and so you are just gonna persist data that will stay there even when you retry the task.
• 3:28
  SPEAKER_00: So basically uh it is coped to your task instance.
• 3:32
  SPEAKER_00: And so if you retry the task, uh the data will still be there.
• 3:36
  SPEAKER_00: But as soon as you run a new task instance of your task, then the data is not, you know, it's it's it's something different.
• 3:44
  SPEAKER_00: But that's that's the idea of the task store is that you can purchase data across retries uh so that you can, for example, reattach uh you know jobs that were running, and for some reason you got an issue with your workers.
• 3:56
  SPEAKER_00: So this is for the task store.
• 3:58
  SPEAKER_00: Um for the asset store, it's a little bit different, but also pretty amazing.
• 4:04
  SPEAKER_00: So now what you're gonna do is instead of having your uh your data uh you know um scoped to the task instance, it will be scoped to the asset, to an asset.
• 4:17
  SPEAKER_00: And why this is interesting is because well now you can do watermarking.
• 4:24
  SPEAKER_00: So for example, if you let's say you read some files in an L3 bucket uh until uh until a certain date.
• 4:33
  SPEAKER_00: Well, when you uh want to read uh you know, when you run the same task again and you want to read the rest of the files in that bucket or just new files that just arrived in that bucket, you don't want to read all the files again, right?
• 4:45
  SPEAKER_00: You just want to read from the latest date at which you well read the the the previous files.
• 4:52
  SPEAKER_00: And that's exactly what you can do now.
• 4:54
  SPEAKER_00: What you're gonna do, for example, is that you're gonna persist that date in the asset store, so as an asset, uh yeah, so that you can just come back where you left.
• 5:03
  SPEAKER_00: And again, you uh the idea here is that you won't have to uh read everything again, you will just read where you left.
• 5:10
  SPEAKER_00: And obviously that applies to databases and so on.
• 5:13
  SPEAKER_00: So that's pretty amazing.
• 5:14
  SPEAKER_00: So that's what I would say.
• 5:15
  SPEAKER_00: It's I mean that's why I love this uh this this new feature, like the task store and the asset store, is because to me they are just amazing, they solve real issues uh that you have in in the data engineering world.
• 5:26
  SPEAKER_00: And also like for AI agents.
• 5:28
  SPEAKER_00: Yeah, if you have to, you know, um, instead of uh resume like like uh I would say like retrying all the actions that your agent took, now you can just come back where you where your agent uh left.
• 5:40
  SPEAKER_00: So pretty amazing.
• 5:42
  SPEAKER_02: Yeah, this is so cool.
• 5:44
  SPEAKER_02: Very exciting new feature, and that's a great description of it.
• 5:47
  SPEAKER_02: I love the two examples you gave because one of the early questions that I've encountered when talking about this feature as it's been in development has been, you know, we've always talked on our team about a best practice with Innerflow being to design your DAG so that your tasks are atomic, so that they each do kind of one unit of work.
• 6:07
  SPEAKER_02: And one of the reasons that we say that is so that if something has to retry, you're not, you know, redoing a bunch of work that has already happened.
• 6:16
  SPEAKER_02: Uh so I have gotten the question like, is that still a best practice?
• 6:20
  SPEAKER_02: Can you just cram all of your logic into one task now?
• 6:24
  SPEAKER_02: And I think you actually gave the examples where it does make sense to use this feature in this way.
• 6:28
  SPEAKER_02: So I don't think that you should just cram all of your task logic into a single task anymore, but it does make a lot of sense in cases where Airflow is, you know, communicating with some external service where, as you said, like the Spark job might still be running and you don't want to submit a duplicate, or the Agentic Workflows is another great example where you might have an agent that does multiple steps worth of work that you can't reasonably split into different tasks, but you still want to maintain some sort of caching so that you're not redoing the whole thing if something gets interrupted.
• 7:02
  SPEAKER_02: So uh yeah, I think in those use cases, this is very, very useful.
• 7:08
  SPEAKER_00: Yeah, it is, it is.
• 7:09
  SPEAKER_00: And and also, you know, like some people may think, well, why not using XCOMs?
• 7:12
  SPEAKER_00: But uh keep in mind that your XCOMs get reinitialized, you know, like destroyed after every task we try.
• 7:19
  SPEAKER_00: And so that's why, for example, this didn't work.
• 7:22
  SPEAKER_00: Um, and that's why also we have the task store now.
• 7:25
  SPEAKER_00: Um, so no, I'm I'm pretty excited about this.
• 7:27
  SPEAKER_00: It solves uh a couple of uh issues uh for sure.
• 7:30
  SPEAKER_00: And it's very easy to actually use, uh right.
• 7:33
  SPEAKER_00: Like I mean it it comes out of the box because again it uses the the Airflow metadatabase, and then to access the task store or the asset store, you just have to you know import like the um the task store from the context of your task, and then you can access it and you can get the value that you want, you can save the value that you want.
• 7:49
  SPEAKER_00: It's very easy, pretty much like XCOMS.
• 7:52
  SPEAKER_00: Um, so it's it's really cool.
• 7:54
  SPEAKER_02: Yeah, we will have a guide published with some examples of how you can use this in your code.
• 7:59
  SPEAKER_02: And yeah, like you said, it's it's very straightforward.
• 8:02
  SPEAKER_02: I think the asset store as well is really interesting.
• 8:06
  SPEAKER_02: And I guess a question for you because I haven't tried this yet, I assume that means that you can also use this, you know, when you're triggering other DAGs downstream based on an asset or an asset partition, maybe that if you had only processed some of those, that they would the relevant downstream DAGs would be triggered, but the ones that got interrupted would not.
• 8:28
  SPEAKER_00: If I'm describing this well enough for you to answer, I no, I I mean look, I didn't try that, but yeah, I guess I guess it will work that way.
• 8:35
  SPEAKER_00: But I I didn't try that yet with um with asset uh partitioning, but uh but yeah, I mean again, like very excited about it.
• 8:44
  SPEAKER_00: I think the the guide will be very um interesting for sure.
• 8:47
  SPEAKER_00: I highly recommend people to look at it because uh you will have a ton of examples of how this feature is great.
• 8:52
  SPEAKER_02: Yeah, definitely.
• 8:54
  SPEAKER_02: Yeah, well, and I've been kind of talking high level about th the 3.3 release in this context of you know, a big theme with the new features has been making failure management easier.
• 9:07
  SPEAKER_02: And that if we kind of think about it, like a lot of data engineering or maybe any engineering work actually is really just about managing failures because if it wasn't, if nothing ever failed, like you could write all of the messy code that you wanted and just ship it off, and it would it would always be fine.
• 9:24
  SPEAKER_02: Uh so but that's obviously not what happens in production.
• 9:27
  SPEAKER_02: So this is a big piece of that.
• 9:29
  SPEAKER_02: Another new feature that is part of this sort of making failure management easier is pluggable retries, uh, which I'm really excited about.
• 9:39
  SPEAKER_02: Have you had a chance to try this out yet?
• 9:41
  SPEAKER_00: Yeah, yeah, actually I did.
• 9:43
  SPEAKER_00: And uh also this this is one of the features that I'm very excited about for Airflow 3.3.
• 9:48
  SPEAKER_00: Um you know, like imagine that you have a task that, for example, fails for um a network issue.
• 9:56
  SPEAKER_00: Okay.
• 9:56
  SPEAKER_00: Well, in that case, it max it makes sense to retry the task because uh maybe at some point the network issue will go away and then it will work.
• 10:05
  SPEAKER_00: Um but now imagine that you are getting uh an issue because of a permanent uh you know, like a like a like a forbidden, like a page that is forbidden, you know, for or three ER or something like that.
• 10:17
  SPEAKER_00: Well, you can just keep retrying the task, that won't change anything, right?
• 10:20
  SPEAKER_00: I mean you will ski you will keep eating the issue and you will keep wasting resources and time and so on just because of that.
• 10:26
  SPEAKER_00: So it's a kind of uh a little bit dumb in some way.
• 10:30
  SPEAKER_00: I mean, not dumb, but could be smarter.
• 10:33
  SPEAKER_00: Um and that's exactly the idea of the retry policy.
• 10:36
  SPEAKER_00: Uh I mean there are also uh that sort of uh other things, but the idea of a retry policy is that now you can attach a retry policy to your task.
• 10:45
  SPEAKER_00: So what does that mean?
• 10:46
  SPEAKER_00: Well it means that um you're gonna basically you're gonna uh define a rule like how you want to retry your task based on the exception that you get.
• 10:57
  SPEAKER_00: Okay, so for example, I'm getting like a HTTP error classic.
• 11:02
  SPEAKER_00: So I'm gonna put that as an exception in my retry rule object, which is a new object that you can import uh in Airflow.
• 11:10
  SPEAKER_00: And then once I've defined the exception, I'm gonna define the action that I want to make.
• 11:15
  SPEAKER_00: So either I want to retry the task or I just want to stop the task.
• 11:20
  SPEAKER_00: Again, if I'm hitting like a forbidden error, then I might want to stop the task right away.
• 11:25
  SPEAKER_00: Okay, I don't want to retry the task.
• 11:27
  SPEAKER_00: And the default is retry.
• 11:29
  SPEAKER_00: Okay, so you have the option, like you have two options here, either you retry or you stop the task.
• 11:33
  SPEAKER_00: And then you can also change the retry delay if you want.
• 11:36
  SPEAKER_00: Okay, for example, if you are getting like a network issue, maybe you want to wait a little bit more.
• 11:42
  SPEAKER_00: Okay, maybe you want to wait uh I don't know, five minutes instead of three.
• 11:47
  SPEAKER_00: And again, you can define that in the retry rule.
• 11:50
  SPEAKER_00: So you know, according to the exception you will get, you can define how you want to retry the task, if you want to retry it, I don't how long you want to wait before each retry, and also you can attach like the reason of the retry.
• 12:06
  SPEAKER_00: Okay, so you have like a lot of flexibility now in how you want to retry the tasks than you um used to have.
• 12:13
  SPEAKER_00: Now, one thing that I have to say about it is if that's not enough for you, you can even subclass the um the I don't remember exactly the the class name to be honest.
• 12:24
  SPEAKER_00: I think it's a retry rule or retry policy.
• 12:27
  SPEAKER_00: But you can you can subclass this and then you can uh implement the logic that you want.
• 12:33
  SPEAKER_00: You know, if you want a more complex logic, like uh I don't know, I I want to um I don't know, I want to add some cleaning or I want or the retry mechanism should be more complex than just retrying the task, whatever you want, you can do that.
• 12:46
  SPEAKER_00: Okay, you you have like all the freedom that you want for retrying your task.
• 12:50
  SPEAKER_00: And I know that this has been like a very common issue.
• 12:53
  SPEAKER_00: You know, people were saying, hey, uh I've just retried the task, but now I want to change the retry, the retry number, for example, to retry the task more than three times.
• 13:03
  SPEAKER_00: Now I want four for some reason and I want to do that dynamically.
• 13:06
  SPEAKER_00: People used to change like the meta database, like you know, change the data in the meta database.
• 13:10
  SPEAKER_00: That that obviously doesn't work anymore in Airflow 3 and was already hacky in Airflow 2.
• 13:15
  SPEAKER_00: But now with the retry policy, you have like all of that all of that uh freedom.
• 13:19
  SPEAKER_00: So all of that to say, I'm very, very excited by uh by this feature as well.
• 13:23
  SPEAKER_00: I think uh after the task store and the asset store, this one is definitely um in my top list.
• 13:29
  SPEAKER_02: Yeah, I think this will be so impactful.
• 13:31
  SPEAKER_02: And as you said, it makes retries within Airflow a lot smarter or less sort of brute force was the term that I was using before of like you can just sort of keep retrying this thing until you say not to, but it's not going to do any other logic than that.
• 13:47
  SPEAKER_02: So so much additional flexibility here.
• 13:50
  SPEAKER_02: I think this will be really helpful for our team.
• 13:53
  SPEAKER_02: Like, we have at least one pipeline I know of that just periodically has like a transient failure that there's nothing to do anything about other than it just happens from time to time and we can't really fix it.
• 14:06
  SPEAKER_02: It's not actually a problem, but retrying in that case does make sense because it will go away if we just retry it, but we also probably don't want to retry like a minute later.
• 14:17
  SPEAKER_02: We might want to wait like half an hour or something like that.
• 14:20
  SPEAKER_02: Um, we also have other pipelines where I know of like the pipelines that help us produce this podcast.
• 14:26
  SPEAKER_02: Like if there's a failure, there is nothing that retrying will help with.
• 14:30
  SPEAKER_02: Like that is something where I have to step in and fix something.
• 14:34
  SPEAKER_02: And so there's just no point if it's you know an error from the LLM that's coming back or something like that.
• 14:40
  SPEAKER_02: So uh so yeah, I think this will be really helpful for us and probably for a lot of teams as well.
• 14:46
  SPEAKER_00: Yeah, no, no, this is this is really cool.
• 14:48
  SPEAKER_00: This is uh this is really cool.
• 14:50
  SPEAKER_00: To your point, there are so many use cases in which we try policies will be useful.
• 14:54
  SPEAKER_00: Um so I can't wait to see like how people uh you know what uh what policies they will uh they will define.
• 14:59
  SPEAKER_00: I think it's gonna be uh an interesting one for sure.
• 15:03
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 15:12
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 15:16
  SPEAKER_01: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 15:22
  SPEAKER_01: Check it out at astronomer.io Yeah, I think I think it's great.
• 15:29
  SPEAKER_02: Hopefully we'll alleviate some of the frustration with, I don't know, do you get sort of attached to like random errors, you know, that you see come up a lot where I'm like, oh, it's that error again.
• 15:39
  SPEAKER_02: Um like an emotional response to like one particular exception.
• 15:43
  SPEAKER_02: And so it feels at least useful to be able to say, like, okay, for this exception, we're gonna do something different.
• 15:48
  SPEAKER_00: Yeah, yeah, yeah.
• 15:49
  SPEAKER_00: No, absolutely.
• 15:50
  SPEAKER_00: Absolutely.
• 15:51
  SPEAKER_02: Yeah, awesome.
• 15:52
  SPEAKER_02: Well, I think it is also worth discussing, kind of on a bit of a different topic from these last two features that we just covered.
• 15:59
  SPEAKER_02: Uh, there have been some updates to the asset partitions feature.
• 16:04
  SPEAKER_02: Uh so this, of course, was first released in Airflow 3.2, you know, a couple months ago.
• 16:09
  SPEAKER_02: Um, and it has gotten, like I said, some updates for this release.
• 16:14
  SPEAKER_02: So, what's new with asset partitions that you think users should be aware of?
• 16:18
  SPEAKER_00: Yeah, that's a that's a good question.
• 16:20
  SPEAKER_00: Um, so about asset partitioning, I think like the the new um the new update that has been uh that's gonna be shipped with um Airflow 3.3 is what we call segment-based uh partitioning.
• 16:33
  SPEAKER_00: Um so concretely, like imagine that you have uh you know that you are partitioning a table by regions, for example.
• 16:42
  SPEAKER_00: Uh so you have like EU, APAC, US.
• 16:46
  SPEAKER_00: Um and so what you can do now is you can actually like run a DAG for every uh partition, and then you can have a downstream DAG that will run as soon as all of those partitions have been processed.
• 17:05
  SPEAKER_00: Okay, and so this is like you know, like you take all of your partitions and then they will be all uh processed under a new asset that could be all regions in that downstream DAG.
• 17:17
  SPEAKER_00: So this is something that now you can do uh with uh segment-based uh partitioning.
• 17:22
  SPEAKER_00: Um so that's one thing, and I think the other the other um the other update I think that has been made, but this has to be re verified honestly, uh, is that now you can also have for every partition again like US EU uh region, you can actually trigger like one DAC per uh per asset.
• 17:46
  SPEAKER_00: Um which I think was not possible uh in uh in 3.2.
• 17:50
  SPEAKER_00: You have like this fan out uh um way of uh of uh of uh processing your uh your assets, your asset partitions.
• 18:00
  SPEAKER_00: Uh so I think like those two those two uh updates are definitely uh interesting and and worth the time to dig in.
• 18:06
  SPEAKER_02: Yeah, definitely.
• 18:07
  SPEAKER_02: I think the new mappers, I think that's what it's called mappers, the fan out mapper, um, as you said, which allows you to then sort of take an asset and split it into a bunch of different pieces.
• 18:18
  SPEAKER_02: And then as you said, you can use those to schedule downstream DAGs.
• 18:22
  SPEAKER_02: So if you have a downstream DAG that says, okay, this one is only dealing with APAC data, but this one is dealing with EU data, you can schedule that way.
• 18:30
  SPEAKER_02: I think you can also go the other way.
• 18:32
  SPEAKER_02: So there's yes, I'm gonna get the names wrong.
• 18:34
  SPEAKER_02: So anybody listening, definitely check our guide before you like write code based on what I'm saying here, but it's like a fan in.
• 18:42
  SPEAKER_02: I don't think it's actually called fan in.
• 18:43
  SPEAKER_02: I think I'm getting that wrong.
• 18:45
  SPEAKER_00: It's the roll-up, roll-up mapper.
• 18:47
  SPEAKER_00: Oh, roll-up, thank you.
• 18:48
  SPEAKER_00: Rollup mapper, yes.
• 18:49
  SPEAKER_02: That makes more sense than fan in.
• 18:51
  SPEAKER_00: It's fun out, it's fun out and roll-up mapper.
• 18:53
  SPEAKER_02: Roll up, okay, yeah.
• 18:55
  SPEAKER_02: Um yeah, so that uh, as I said, allows you to go the other way.
• 18:59
  SPEAKER_02: So if you need to have a DAG that's scheduled when all of the partitions are done or something like that, um, you can do that.
• 19:06
  SPEAKER_02: And of course, there's other pieces involved so that you can map them to cleanly, you know, especially if this is like daytime based, um, cleanly to like an hour or a day or something like that, so that it's not attached to like the full timestamp of the DAG and stuff like that.
• 19:23
  SPEAKER_00: Yeah, and as you said, right?
• 19:24
  SPEAKER_00: I mean, uh uh before it was um it was uh only time-based partitions, and now you have like um segment-based uh partitioning.
• 19:33
  SPEAKER_00: So that that's the the the update in uh in 3.3.
• 19:37
  SPEAKER_00: So you have like I would say like if you want to use uh asset partition, you have to be aware of those mappers because they that's how you can basically use uh asset partitioning.
• 19:46
  SPEAKER_00: Um and uh and yeah, you will enjoy that.
• 19:50
  SPEAKER_02: Yeah, definitely.
• 19:52
  SPEAKER_02: We have uh, like I said, in our documentation, some examples of how you can combine those mappers that we've talked about in the code um to make sense of them.
• 20:00
  SPEAKER_02: I think it can look a little, I don't know, intimidating maybe uh when you're first working with this feature, because there are just a lot of concepts and it can take some, it's one of those things where I feel like I have to like draw it out on a piece of paper.
• 20:15
  SPEAKER_02: Like what okay, how how are we breaking this up?
• 20:18
  SPEAKER_02: Um, but once you have like a good understanding of all of those, it it is actually pretty straightforward.
• 20:23
  SPEAKER_02: Um and I think using some of the code examples will help folks implement that.
• 20:27
  SPEAKER_00: Yeah, yeah, no, absolutely.
• 20:28
  SPEAKER_00: I think you you definitely need some code examples to uh actually grasp how asset partitioning works because you know it can be simple, but it can be also quite complex.
• 20:38
  SPEAKER_00: Um so you know, either way you have to find uh and the guys are high for that.
• 20:42
  SPEAKER_02: So yeah, definitely.
• 20:43
  SPEAKER_02: Well, I do think the updates to this feature, I as with most things in Airflow, you know, they're they're all sort of having ongoing development.
• 20:52
  SPEAKER_02: So uh it's definitely top of mind, I would say, for lots of the people working on the Airflow project, you know, trying to take user feedback from the initial iteration of a feature and making sure that they're making updates to it uh to make it more usable or work for more use cases or whatever it is.
• 21:11
  SPEAKER_02: And so that's that's the case here of again this thing that was released in the last minor version is still being developed to uh kind of fully bake it so that again it's useful to the most number of people possible.
• 21:22
  SPEAKER_02: So very cool to see those updates in the project.
• 21:25
  SPEAKER_00: Completely agree.
• 21:26
  SPEAKER_00: Agree.
• 21:27
  SPEAKER_02: Yeah.
• 21:28
  SPEAKER_02: And then so as I mentioned at the beginning, of course, minor releases always have so much stuff in them.
• 21:34
  SPEAKER_02: We'll direct everybody to the release notes because all of it is relevant to at least someone.
• 21:39
  SPEAKER_02: Um we can't go through all of it in our podcast today, but I'm curious of the remaining, we'll say quote, smaller features, because again, they might not be smaller to whoever's using them.
• 21:49
  SPEAKER_02: But of the stuff that we're not covering more in depth in this episode, what what are you most excited about?
• 21:55
  SPEAKER_00: Well, interestingly, interestingly, I don't think it's uh it's a small feature, but in Airflow 3.3.
• 22:00
  SPEAKER_00: I believe you will be able to run tasks in Java and Go.
• 22:04
  SPEAKER_00: So I think those, you know, for those who I believe mostly like for Java, I think like there are quite many people that obviously interact with Java every day.
• 22:15
  SPEAKER_00: And the fact that now you will be able to run like you know Java using Airflow directly without having like.
• 22:21
  SPEAKER_00: I remember that someone told me, hey, we are we are using like a custom operator that runs uh jar.
• 22:28
  SPEAKER_00: Uh I I they are doing like some crazy stuff to run uh Java code.
• 22:32
  SPEAKER_00: Well, guess what?
• 22:33
  SPEAKER_00: Now you won't have to do that anymore.
• 22:34
  SPEAKER_00: I have no idea how mature is the feature, to be honest.
• 22:38
  SPEAKER_00: Uh you can expect that there will be some issues, uh, but I think it's quite exciting to see that yeah, Airflow is gonna be uh like a multilinguage um orchestrator at some point.
• 22:49
  SPEAKER_00: I mean, everything will be still in Python, but you will be able to run tasks in different languages, which I think is quite amazing.
• 22:56
  SPEAKER_02: Yeah, yeah, it's really cool.
• 22:58
  SPEAKER_02: I, as you said, not actually a smaller feature, and I think it'll be super impactful for teams who are moving from another orchestrator where things are more written in another language.
• 23:09
  SPEAKER_02: I know Java that's really common, especially with legacy orchestrators.
• 23:12
  SPEAKER_02: If you're exporting jobs that are were built with like a drag and drop or something, oftentimes that ends up in Java.
• 23:18
  SPEAKER_02: Uh, and you can now just run that uh without having to convert everything to Python.
• 23:24
  SPEAKER_02: So yeah, I do think I'm also gonna get this wrong.
• 23:27
  SPEAKER_02: So check the documentation.
• 23:28
  SPEAKER_02: But of the Java and Go ones, I think one of them is still experimental, at least.
• 23:33
  SPEAKER_02: Maybe both of them.
• 23:34
  SPEAKER_02: I can't remember, but yeah, keep it keep that in mind.
• 23:37
  SPEAKER_02: The the UI um or the user interface might change a little bit, but yeah, uh, yeah, we're excited to see what people are able to build with this.
• 23:46
  SPEAKER_00: Same no, no, very exciting.
• 23:48
  SPEAKER_02: Yeah, that's a great answer.
• 23:50
  SPEAKER_02: My when I was just looking through the release notes, I think one of my uh favorites that stood out to me was you can now with DAG versioning uh choose in the UI if you rerun um a past DAG run, you can choose whether you want to use the most recent DAG version or an older DAG version.
• 24:11
  SPEAKER_02: Okay.
• 24:11
  SPEAKER_02: So I think before this, if you were using like a local DAG bundle, you would use whatever the version was at that time um that it ran, even if you were on a newer version.
• 24:22
  SPEAKER_02: And with the um uh no, sorry, I said that wrong.
• 24:26
  SPEAKER_02: With the local DAG bundle, you'll use the most recent version of the code.
• 24:30
  SPEAKER_02: With a version DAG bundle, you'd use the version at the time that you ran it.
• 24:34
  SPEAKER_02: Uh so you could do either or, but now you just have the option.
• 24:37
  SPEAKER_02: Um, and I'm I'm pretty sure this is only if you're using a version DAG bundle that that's possible, but uh just a small piece of kind of additional choice on the part of the user.
• 24:47
  SPEAKER_02: So I can see where this will be useful in a case where like you've shipped new code and that has caused a failure, and so you want to just quickly rerun with the older version to get things back on track and then go fix whatever has gone wrong.
• 24:59
  SPEAKER_00: So yeah, no, uh yeah, it's good it's gonna be great.
• 25:02
  SPEAKER_00: And also like for people that don't don't know about git um DAG bundles, I highly recommend you to look at that feature.
• 25:08
  SPEAKER_00: Yeah.
• 25:08
  SPEAKER_00: I won't say I won't say anything else, just look at them.
• 25:12
  SPEAKER_02: Yeah, definitely.
• 25:13
  SPEAKER_02: And then I guess very quickly, you know, for anybody who is listening who might not already be upgraded to Airflow 3, because we know there are still users in the community who are on Airflow 2 and they're interested in the features in 3.3 uh or any of the other minor versions in the 3.x series, uh, what is your advice to them for going through that upgrade process?
• 25:36
  SPEAKER_00: Yeah, so I think we have a, I don't think, I'm sure, we have a great ebook for that uh to upgrade from Airflow 2 to Airflow 3.
• 25:42
  SPEAKER_00: I highly recommend you to uh take a look at it.
• 25:44
  SPEAKER_00: And also we have auto, uh, which if you want to go with the AI way of uh upgrading your DAGs from Airflow 2 to Airflow 3 and follow best practices and so on, I think it definitely worth the shot um because it's uh it's gonna save you a lot of time.
• 25:58
  SPEAKER_02: Yeah, absolutely.
• 25:59
  SPEAKER_02: It can take over a lot of the kind of manual changes, you know, especially minor changes in your code that you have to make.
• 26:06
  SPEAKER_02: Uh it's yeah, extremely helpful.
• 26:09
  SPEAKER_00: No nobody wants to do my migrations.
• 26:12
  SPEAKER_02: So no, talk about like undifferentiated heavy lifting, especially if you have a lot of DAGs.
• 26:18
  SPEAKER_02: That's not a fun process to go through.
• 26:20
  SPEAKER_02: So it's uh it's great to offload that to some of your AI coworkers, those that have been trained specifically in Airflow.
• 26:28
  SPEAKER_00: That's right.
• 26:28
  SPEAKER_00: Yeah, cool.
• 26:29
  SPEAKER_02: Well, we will throw links to all of those things in the show notes.
• 26:32
  SPEAKER_02: I know this has been just a very high-level overview of some of the new stuff in 3.3.
• 26:37
  SPEAKER_02: We will be having a webinar uh that we will also link in the show notes if folks want to register where we'll go through all of this in a little bit more detail and show, you know, some code examples and a demo and stuff like that.
• 26:49
  SPEAKER_02: So a little more tangible than you know an audio version of a podcast.
• 26:53
  SPEAKER_02: So that's a great follow-up resource.
• 26:56
  SPEAKER_02: But yeah, we're really excited about the new release.
• 26:58
  SPEAKER_02: There's always so much stuff going on in the Airflow project, and the momentum is still there.
• 27:02
  SPEAKER_02: So it's great to be able to talk about another one.
• 27:05
  SPEAKER_00: Yep, can't wait for you.
• 27:06
  SPEAKER_00: Thank you so much, Kitan, for having me.
• 27:08
  SPEAKER_00: Was a pleasure.
• 27:09
  SPEAKER_02: Yeah, thanks, Mark, for going through all this.
• 27:11
  SPEAKER_01: Thank you so much.
• 27:13
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 27:16
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 27:19
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 27:24
  SPEAKER_01: And leave us a five-star review.