Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Road to Airflow 2.0

Transcript

The Road to Airflow 2.0 · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: Hi there and welcome to this very step service Airflow five mass property by astronomer.
• 0:05
  SPEAKER_00: This week is the airflow 2.0 roadmap with airflow manager, and astronomer platform engineer airflow and the airflow process 1016 response to the API product.
• 0:37
  SPEAKER_00: Before we dive in, I wanted to mention that our team at Astronomer deploys the next generation of Astronomer Cloud, our fully managed Airflow as a service product.
• 0:45
  SPEAKER_00: We have some awesome new bells and whistles that make Airflow more observable, secure, and user-friendly.
• 0:50
  SPEAKER_00: Feel free to get in touch if you'd like a Taurus.
• 0:53
  SPEAKER_00: Now, without further ado, let's dive into our conversation with Ash.
• 1:04
  SPEAKER_00: No, you know, you've been kind of doing the meetup circuit and all that stuff, and you gave a talk last night as well with the London Airflow meetup.
• 1:12
  SPEAKER_00: Do you want to just kind of quickly give a high-level summary of what the chat you gave last night was about and what we're going to be talking about today?
• 1:19
  SPEAKER_01: Yeah, sure.
• 1:20
  SPEAKER_01: So last night it was um kind of a a warm-up act.
• 1:24
  SPEAKER_01: It was just a nice short 10-minute or so talk.
• 1:26
  SPEAKER_01: Um just some new features in Airflow 1, 10, 4, and 5.
• 1:30
  SPEAKER_01: They both came out in quick succession in um start and end of August.
• 1:34
  SPEAKER_01: And then I kind of went into a little bit about what we're thinking for a roadmap for Airflow 2.0.
• 1:39
  SPEAKER_01: We're kind of excited to get going and kind of fix some of the longer standing annoyances and and niggles on Airflow and uh kind of put it out there yesterday and go like here's what we're thinking, this is what we want to work on.
• 1:54
  SPEAKER_01: Um so yeah, I can kind of run run you through that and uh flash out a few things in more detail.
• 1:59
  SPEAKER_00: Yeah, that's awesome.
• 2:00
  SPEAKER_00: I mean, this list of features, this the this list of high-level features, at least on the confluence page, looks very interactive.
• 2:06
  SPEAKER_00: Um so maybe maybe that's a good place to start.
• 2:09
  SPEAKER_00: Um kind of high-level um, what direction do you think the project is going in in 2.0?
• 2:15
  SPEAKER_00: What key areas are are gonna be fixed, and then kind of maybe a little bit later we can talk about how how they might be fixed.
• 2:22
  SPEAKER_01: Yeah, sure.
• 2:23
  SPEAKER_01: So um to kind of quickly run run through it, there are um six items on our on our roadmap that we think are good fit for our roadmap.
• 2:35
  SPEAKER_01: Um just kind of listing through them, and then we can go kind of walk through them all.
• 2:39
  SPEAKER_01: So adding support to the Kubernetes executed to run via um native, native.
• 2:47
  SPEAKER_01: Um, we want to improve the web server performance.
• 2:51
  SPEAKER_01: Um we should have a real-time UI so that you know, no more having to refresh to see the state of your tasks change.
• 2:58
  SPEAKER_01: Um improve the performance and reliability of the scheduler.
• 3:03
  SPEAKER_01: Um it's kind of the core of what Airflow is, and it's occasionally got some issues which we want to fix.
• 3:10
  SPEAKER_01: Um we want to extend and finish the API and kind of bring it out of experimental into fully supported and used internally by internally by Airflow.
• 3:22
  SPEAKER_01: Um and then the kind of final thing we want to do is a production-ready Docker image and um Helm chart that's kind of under the Apache name.
• 3:32
  SPEAKER_01: Um, just so that kind of Dockers, you know, how many people run Airflow and they kind of we want to go like this is how we we do it.
• 3:38
  SPEAKER_01: If you do this, it's it's recommended and supported.
• 3:43
  SPEAKER_02: Cool.
• 3:43
  SPEAKER_02: Yeah.
• 3:44
  SPEAKER_02: Um I think there's a lot of cool things there that'll definitely like uh um for some of these.
• 3:49
  SPEAKER_02: I know like for the Docker image and scheduler performance, people kind of hack their own solutions too.
• 3:54
  SPEAKER_02: Um, like so there's the puckle image a lot of users use, and then for scheduler performance, you see things like automatic restarts and other such things.
• 4:01
  SPEAKER_02: Uh so I'm like uh pretty hopeful this will bring uh kind of another layer of standardization to how people use Airflow.
• 4:07
  SPEAKER_02: Uh just because for some of those core things there'll be like some uh community backing behind.
• 4:12
  SPEAKER_01: Yeah, absolutely.
• 4:14
  SPEAKER_01: Like, you know, I I I use the puckle container as well.
• 4:16
  SPEAKER_01: It's it's great.
• 4:17
  SPEAKER_01: There's nothing wrong with it other than the fact it's it's run by one guy and he doesn't have time to be authorized the changes, which is you know, I've been there, I know exactly.
• 4:28
  SPEAKER_01: It's like not a complaint against him, but it's just kind of let's let's bring it all under the Apache Airflow umbrella and you know, Docker pull Apache slash Airflow is kind of what what we're what we're aiming for.
• 4:39
  SPEAKER_02: Yeah.
• 4:41
  SPEAKER_02: So let's kind of uh run through these step by step.
• 4:44
  SPEAKER_02: Uh and I'd love to kind of get your thoughts on each of them and how uh how it affects current Airflow users and uh even people who might want to start using it after some of these features are released.
• 4:53
  SPEAKER_02: Uh going from the top there, um Knative executor.
• 4:57
  SPEAKER_02: Um you want to kind of give a high-level summary on what on what that's gonna look like?
• 5:02
  SPEAKER_02: I know Daniel's doing some of the work, but I would love to hear it from you.
• 5:05
  SPEAKER_01: Yeah, so um Daniel Imberman, one of our kind of colleagues and fellow commaker on Airflow, gave a talk uh last week in Seattle about this.
• 5:12
  SPEAKER_01: Recording should be available soon, which we've covered more detail.
• 5:15
  SPEAKER_01: But yeah, K-native is a product that's kind of you know part of the Kubernetes offering, it's functions as a service on Kubernetes, that's kind of loosely what it is.
• 5:30
  SPEAKER_01: Um our goal in trying to uh integrate that into Airflow is to kind of fix some of the drawbacks of using the Kubernetes executor.
• 5:41
  SPEAKER_01: So if you use the Kubernetes executor, um you every time you run a new task, it spins up a pod, um, which is great for isolation, but does have some drawbacks when it comes to kind of like speed of execution, just because you pay that that pod startup time and um every time you run a task, which is kind of even worse if the image is on it, you know, if it gets scheduled onto a node that doesn't yet have that image.
• 6:06
  SPEAKER_01: Um if you kind of compare that with the celery executor where where it runs four or sixteen or whatever tasks at the same time in the same container, um, that's kind of a lot quicker to get things going, uh, but you don't have the isolation.
• 6:24
  SPEAKER_01: So the K-native executor is kind of like the best of both worlds, and we're trying to work out the exact details of kind of what it will look, what it will look like, how you how you enable it, what the setup will be, but we're kind of hoping it will be a per task or per dag or config level setting so that you can kind of pick and choose between them what makes sense for your task.
• 6:45
  SPEAKER_01: So basically it's kind of like trying to add more power and flexibility to the existing Kubernetes executor offering in Airflow.
• 6:54
  SPEAKER_01: Um so that's gonna kind of be super interesting for for just speed and things like that in Airflow.
• 6:58
  SPEAKER_01: So that's yeah, looking forward to seeing that one.
• 7:03
  SPEAKER_02: Yeah, I think uh that task start time will be huge for users.
• 7:07
  SPEAKER_02: Um and also it'll tell a really nice auto-scaling story too.
• 7:10
  SPEAKER_02: Um kind of scale things up and then scale down afterwards without the without waiting as much.
• 7:16
  SPEAKER_01: Yeah, that that's that's the other kind of the point I've got to mention with the kind of celery, you have to scale the workers manually yourself.
• 7:22
  SPEAKER_01: Um, whereas the whole point of Knative is it will scale workers down to zero when FO isn't doing anything, um, which is yeah, you can kind of configure like you know, timeouts and pre-warming, all that kind of stuff.
• 7:33
  SPEAKER_01: But yeah, not having to manually scale workers or have them sitting around doing nothing will be yeah, a huge benefit.
• 7:40
  SPEAKER_00: Yeah.
• 7:42
  SPEAKER_00: Cool.
• 7:43
  SPEAKER_00: Just so why don't we just kind of work our way down the list then?
• 7:46
  SPEAKER_00: Uh we can kind of move on to the the number two, um, improving web server performance and what that will entail, um, kind of what work is being done on that right now.
• 7:54
  SPEAKER_01: Yeah, so so this is kind of mainly um if you have a large deployment with hundreds to thousands of DAGs, um actually starting up the web server can be slow, like it can take five or ten minutes sometimes, which is it's just silly.
• 8:13
  SPEAKER_01: Um that's simply because um when Airflow starts the web server up, uh it forks off, I think the default's four, but it forks off a number of um web server worker processes to kind of serve requests.
• 8:27
  SPEAKER_01: Each one of those goes and loads all the DAG files sequentially.
• 8:33
  SPEAKER_01: Um so you've kind of got a lot of duplicated effort, and they're all thrashing to access the same files on disk and repeat the effort.
• 8:39
  SPEAKER_01: Um and then coupled with that, Airflow also restarts the workers one at a time every 30 seconds.
• 8:44
  SPEAKER_01: So you're kind of your web servers will spend a lot of time in this model just parsing the DAG files, um, which is not something they they they need to do, or something not something they should be doing.
• 8:55
  SPEAKER_01: Um so we've already started started this work, started prototyping it and kind of testing it.
• 9:00
  SPEAKER_01: Um it's it's part of AIP24, so Airflow Improvement Proposal 24.
• 9:06
  SPEAKER_01: Um, and as part of this change, we're making it so that the web server doesn't have to access the DAG files on disk at all.
• 9:16
  SPEAKER_01: Everything it needs to display and operate on the DAGs is now coming from um a database field, database row, um, which once we get it kind of working and and and work out the kinks, uh, should massively help this thing, help kind of this particular workload.
• 9:35
  SPEAKER_01: And there were a number of kind of other bugs in the past where um the trigger DAG endpoint from the API would also do something similar of it would load, it would create a DAG bag with every DAG in it to then go and trigger one specific DAG.
• 9:50
  SPEAKER_01: Um and just kind of like on this new model, you won't have to think about that.
• 9:55
  SPEAKER_01: You won't have to think about how to fix that problem, work around that problem when it implementing the API and new features.
• 10:00
  SPEAKER_01: So it'll kind of make things nicer going forward.
• 10:04
  SPEAKER_01: Um, the way we've so we started working on this one.
• 10:07
  SPEAKER_01: This is myself and um Catchrill, the fellow PMC member, uh, along with um a person from Google based out of Seattle.
• 10:15
  SPEAKER_01: We're kind of working on this together.
• 10:16
  SPEAKER_01: Um, the way we started doing this is to store a JSON blob of the representation of the DAG, and that contains everything that the web server will need to do its job, with the one exception of um rendering templates.
• 10:35
  SPEAKER_01: Because of some limitations or ways those currently work, we've decided that if you visit the rendered template tab, it will still go to load that one file from disk.
• 10:45
  SPEAKER_01: Um, but it should be much better than everything else.
• 10:47
  SPEAKER_01: And so that's you know, 90 plus percent of requests won't need to read files and disks.
• 10:52
  SPEAKER_01: Web server startup gets quicker, um, git should have a whole load of like web server out-of-sync issues too, because the web server will be loading the same version that the scheduler kind of puts in the database.
• 11:04
  SPEAKER_01: So it's it's kind of it fixes a whole load of bugs all at once and kind of heads us down a nice path where we can make the kind of improves and bugs in the scheduler too.
• 11:16
  SPEAKER_02: So that uh serialization for the DAGs, you're gonna start with web server, but can that functionality be kind of uh because you're if you can read the DAGs from one spot where they don't have to be read from disk, can that be kind of pushed out other components as well?
• 11:28
  SPEAKER_02: Like eventually is a scheduler gonna maybe read DAGs from the database after the first time it's parsed them?
• 11:35
  SPEAKER_01: Yeah, absolutely.
• 11:36
  SPEAKER_01: We we kind of we're purposefully excluding the scheduler from any changes as part of this to kind of make the change smaller, but yes, absolutely.
• 11:45
  SPEAKER_01: Um in the future, we kind of there's like multiple ways you could do it.
• 11:49
  SPEAKER_01: We could have a component that runs along the side or in the scheduler that's just responsible for uh reading the DAGs and updating the database.
• 12:00
  SPEAKER_01: Um you could we could also at this point start accepting DAGs.
• 12:06
  SPEAKER_01: You could like post a JSON blob to an endpoint and go, like, here's a new DAG, please start running it, which opens up some kind of super interesting uh possibilities for dynamic and generated DAGs that kind of weren't possible before.
• 12:19
  SPEAKER_01: Um we'd have to work out how that would work for the on the executing side, but it kind of yeah, it opens up a whole load of possibilities kind of for better ways of supporting dynamic DAGs, um, which a number of people use.
• 12:35
  SPEAKER_01: So yeah, it's kind of nice to fix that too.
• 12:38
  SPEAKER_02: Yeah.
• 12:39
  SPEAKER_02: Uh so going a little tension to that, but number three is enhanced real-time UI.
• 12:44
  SPEAKER_02: Um, how does that play into some of the web server changes?
• 12:48
  SPEAKER_01: Um yeah, I mean basically it it this is sort of just like having to refresh to see your tasks makes for a sub-par experience, both from like initial user and demo and just uh and then also from like experienced users who are operating it.
• 13:05
  SPEAKER_01: It's kind of like, you know, it's like, oh, it should be done by now refresh.
• 13:08
  SPEAKER_01: Oh no, it's not.
• 13:08
  SPEAKER_01: And it's just like we should be able to just push, you know, server center events, push, long pole, whatever.
• 13:14
  SPEAKER_01: We haven't decided the technology.
• 13:16
  SPEAKER_01: But if you're on a DAG page, you should see the tasks kind of come and go and update and finish automatically.
• 13:22
  SPEAKER_01: Um, it's not a big user change or not a big like functional change, but it kind of just makes it a lot nicer to use Airflow, makes it a of a more polished experience and just kind of yeah, more in line with what people expect from a web app from 2019.
• 13:39
  SPEAKER_00: Definitely.
• 13:40
  SPEAKER_00: Aside from the like auto refresh, like that you could do with any sort of like any sort of framework, right?
• 13:45
  SPEAKER_00: Like just making the UI generally stateful and also like I feel like it could just be modernized a whole lot as well.
• 13:52
  SPEAKER_00: Um, has anybody kind of taken ownership of that at this point, or is it still kind of up for grabs?
• 13:58
  SPEAKER_01: Um, so kind of Rai, our head of product um at Astronomer, has um he's kind of when he has time, is leading the push on that.
• 14:07
  SPEAKER_01: He wants to fix it.
• 14:09
  SPEAKER_01: And yeah, it's kind of some questions about do we do it as React or not?
• 14:13
  SPEAKER_01: Because then you know, React helps, I mean React or View or any other frameworks, they kind of do a lot of this for you, but there's also then the learning curve.
• 14:22
  SPEAKER_01: Um don't know the answer, but it's probably it will it will probably be we're going to use React or something, and we'll just have to have a here's how you do React in Airflow document to help committers.
• 14:35
  SPEAKER_01: Because yeah, there's kind of there's a learning curve for you know, the the traditional contributor to Airflow will be someone in the data data space, so you know, data scientist or data engineer who kind of they're not gonna be most likely not gonna be super savvy front-end developers.
• 14:51
  SPEAKER_01: So kind of React will be a whoa, what's all this?
• 14:53
  SPEAKER_01: So we just kind of need to do something to make sure we don't alienate some of our possible contributors.
• 15:00
  SPEAKER_00: Definitely, that's a good point.
• 15:02
  SPEAKER_00: Yeah, I'm excited, I'm excited for that push.
• 15:03
  SPEAKER_00: I mean, I think it'll be a really interesting uh kind of change to the project.
• 15:06
  SPEAKER_00: So I mean, I know number four is is improved scheduler performance, but I think maybe it makes more sense to talk about number five right now following the UI, because uh obviously we probably can't, or maybe we can, maybe I'm wrong in my assumption here, but we can't really build rebuild the UI until the API is solid.
• 15:21
  SPEAKER_00: Um I'm saying that or pretty much, yeah.
• 15:28
  SPEAKER_01: Um there are some things which are kind of there are endpoints that currently exist, but they are protected behind CSRF, CSRF tokens.
• 15:39
  SPEAKER_01: So, you know, kind of they're fine for the web server to call because it can get a CRS CRSF token.
• 15:44
  SPEAKER_01: Um but not suitable for you know scripting API, scripting use, you know, kind of you know, integration with other things.
• 15:53
  SPEAKER_01: Um but yeah, basically the API, anything that the web server does, or anything that the CLI does, should be available over an API so that you can integrate it into whatever else you want.
• 16:09
  SPEAKER_01: Um yeah, every operation should be API, it should be part of the API.
• 16:14
  SPEAKER_01: The web server should use the the API internally to do things.
• 16:17
  SPEAKER_01: Um we'll probably also split out the API into a optional second or extra component.
• 16:25
  SPEAKER_01: So, like you have the web server, you might have API server.
• 16:29
  SPEAKER_01: Um, and then that would just serve the API API request so that you can scale that up and down um as you like.
• 16:35
  SPEAKER_01: But yeah, it's kind of like still by default, it will kind of run in a web server.
• 16:40
  SPEAKER_01: Um but if you want to scale your deployment, it will you can just like turn off a config setting and run the run it as a separate component.
• 16:49
  SPEAKER_02: Yeah, I also think that it'll kind of open up Airflow to a lot more use cases there, um, because it'll give more people a chance to uh interact with like dad functionality, maybe if not like directly using it through Airflow.
• 17:02
  SPEAKER_02: Uh, do you think it'll open a door for that at all?
• 17:05
  SPEAKER_01: Yeah, definitely.
• 17:05
  SPEAKER_01: I kind of like a a fully featured and documented API will kind of open up a whole lot of uses that we haven't thought about.
• 17:13
  SPEAKER_01: Um and yeah, that's kind of I'm kind of excited to see what people will build once we start once we have a fully featured API that isn't marked as experimental.
• 17:23
  SPEAKER_01: Um because yeah, it's kind of it's been experimental for years.
• 17:28
  SPEAKER_01: And basically I'm sure some people have looked at it and went, oh maybe, maybe I shouldn't build on this API.
• 17:33
  SPEAKER_01: Um, which is yeah.
• 17:36
  SPEAKER_00: Ash, do you have any do you have any examples of things that you'd like to expose in kind of the the finalized API that aren't currently exposed by the experimental API, or maybe that just need to be locked down?
• 17:48
  SPEAKER_01: Not off the top of my head, but kind of that there's there's lots of operations that the CLI does.
• 17:54
  SPEAKER_01: So you know, airflow list tags or airflow trigger DAGs, kind of like that's a bad example because that was there.
• 18:00
  SPEAKER_01: But anything that the AI, so anything that you can do over the CLI should be done possible via the API.
• 18:07
  SPEAKER_01: That way you can use a local CLI against a remote Airflow server.
• 18:12
  SPEAKER_01: Um kind of stick auth in it and handle that.
• 18:15
  SPEAKER_01: But yeah, then it would be nice to kind of list tasks, all that kind of stuff, and have that work over the over the API too, so that it's kind of local or remote, you know, is just kind of the same and you get the same experience, just slightly slower as you pay a network request.
• 18:32
  SPEAKER_02: I think uh as we kind of open up all these new use cases of the API, I think point number four comes into a little more spotlight with improved scheduler performance.
• 18:41
  SPEAKER_01: Um yeah, scheduler performance and definitely reliability.
• 18:45
  SPEAKER_01: Um kind of like that there's it's kind of two sides of the same coin, um, which is yeah, I'm sure we've kind of we've all hit this, which is kind of like, why is the scheduler taking so long to schedule a dummy task?
• 19:00
  SPEAKER_01: Um, you know, sometimes it kind of sits there 15 seconds between between two tasks, not running anything, and it's like this this shouldn't happen.
• 19:09
  SPEAKER_01: This should be snappy and you know, kind of a dummy task, you should just see it bang, bang, bang, all the way to go through, and it should you kind of like you should blink and it should be done kind of kind of time frame because it doesn't change that often.
• 19:23
  SPEAKER_01: So it should just kind of run quicker.
• 19:25
  SPEAKER_01: Um on the flip side, kind of you know the there are still occasional issues where the scheduler will just not schedule tasks, um, which is absolutely terrible for a job scheduler to be in a state of where it's not doing its its one job.
• 19:42
  SPEAKER_01: So we need to fix that and make sure that it's kind of we we we hammer out those bugs, but also um if it's not scheduling it for some reason, we should make that more more visible to to users.
• 19:58
  SPEAKER_01: Like, you know, there are kind of all sorts of these weird which this is kind of a crossover between scheduler and just revamping the um the UI, but it's kind of like there's all sorts of cases of why isn't Airfill running this task?
• 20:10
  SPEAKER_01: And as like, oh, it's because the DAG runs already marked as success, so this is a new task, so I'm not gonna run it.
• 20:15
  SPEAKER_01: And we should just like make that more obvious, or this task date is this this DAG, this this task is earlier than the start date, so I'm not gonna run it because of that.
• 20:23
  SPEAKER_01: And like there are they they're there sometimes, but not always, so it's just kind of making it more obvious to users what the structure is doing, or importantly why it's not doing something.
• 20:37
  SPEAKER_02: It's kind of uh bringing down the learning curve a little bit.
• 20:39
  SPEAKER_02: Like right now, you have to learn those things painfully.
• 20:42
  SPEAKER_02: Like after two hours of looking at logs, you're like, oh, like this is why this is scheduling a course, but you still kind of have to suffer for a few hours as you do that or days or whatever.
• 20:52
  SPEAKER_01: Exactly, yeah.
• 20:54
  SPEAKER_02: Just kind of making it nicer to operate and use effort.
• 20:56
  SPEAKER_02: Yeah.
• 20:57
  SPEAKER_02: Um, I think point six kind of uh wraps it all together with a production Docker image.
• 21:01
  SPEAKER_02: Uh, we touched on it a little bit, but I think there's some other committers that are doing some cool work there with Breeze and such.
• 21:07
  SPEAKER_02: Um so yeah, kind of what does a production Docker image mean to you and how does it play into the work that uh is going on with Breeze and some other cool projects there?
• 21:16
  SPEAKER_01: Yeah, so kind of uh production ready Docker image to me means one I would be personally happy running in production.
• 21:22
  SPEAKER_01: Um so that kind of includes things like well-documented, tested, um small layer sizes as small as practical an image, kind of no extraneous things, so like no node and npm installed in the container because it's not needed at one time.
• 21:40
  SPEAKER_01: Um correct use of layers so that if you already have 1104 installed and you pull 1105, it should only be you know kind of the minimum changes necessary, um just to kind of reduce image size and and pull speed, things like that.
• 21:57
  SPEAKER_01: Um and yeah, kind of.
• 22:00
  SPEAKER_01: Yarrick, one of the committers who did a whole load of of work on kind of the Airflow Breeze product, as he called it, was you know kind of making development on Airflow a breeze, which was a kind of lovely pun.
• 22:11
  SPEAKER_01: He did a lot of work kind of in a multi-stage build for the for the CI images.
• 22:15
  SPEAKER_01: And um yeah, he's gonna be working with um with Daniel, Daniel Memberman, to kind of take the same approach but apply it to production Docker images.
• 22:26
  SPEAKER_01: So yeah, kind of that's gonna be a nice nice thing when we get it.
• 22:31
  SPEAKER_01: Um and then kind of there's a there's a baseline for if you're running Docker, you use this one and everyone's kind of off the same and decent instructions on how to extend it to here's how you should add your bags, here's how you should add extra Python modules, things like that.
• 22:45
  SPEAKER_01: Um, and then yeah, we'll we'll couple that with a Helm chart because Kubernetes is equally as popular, particularly in our space.
• 22:53
  SPEAKER_01: So we're kind of there's already a Helm chart that's been contributed under the the official kind of Helm chart repository, which I think is based off an old version of the astronomer image.
• 23:06
  SPEAKER_01: But we've kind of we've added a whole load of more of features and kind of tweaked that over the last year or two.
• 23:13
  SPEAKER_01: So we'll update that.
• 23:16
  SPEAKER_01: Um either put a Helm chart inside the Airflow repo or update the one in Helm kind of as as appropriate.
• 23:26
  SPEAKER_01: But yeah, we'll we'll kind of add that, add and extend features to the public Helm chart so that everyone can get you know kind of nice, fully featured, configurable Helm chart to run Airflow in Kubernetes.
• 23:40
  SPEAKER_02: Yeah, that'll be great.
• 23:42
  SPEAKER_02: And I think that'll uh more people will start running Airflow on Kubernetes if it's easier to run Airflow on Kubernetes.
• 23:47
  SPEAKER_02: Right now it's still uh it's not trivial getting it up there.
• 23:51
  SPEAKER_01: Yeah, and we can kind of walk them through the decisions about you know namespaces and limits and quotas and service accounts and all this kind of other stuff.
• 24:00
  SPEAKER_01: Yeah.
• 24:01
  unknown: Yeah.
• 24:03
  SPEAKER_00: Go ahead, go ahead.
• 24:06
  SPEAKER_00: I don't want to derail too far, but I was just gonna bring up I saw I saw a hilarious picture of you on Twitter from last night with a bat massive slide that just said please migrate off Python 2.
• 24:15
  SPEAKER_01: Yeah, um, please migrate away from Python 2.
• 24:18
  SPEAKER_01: Like seriously, do it like now.
• 24:20
  SPEAKER_01: Um Python 2 is officially unsupported by Python as of January 1st, 2020.
• 24:28
  SPEAKER_01: Um so yes, the big the big change here, Airflow 2.0.
• 24:33
  SPEAKER_01: So Airflow 2.0 is going to not work on Python 2.
• 24:38
  SPEAKER_01: Um if you are still on Python 2 and are using Airflow, you've basically got let's call it six months um before you have to upgrade.
• 24:48
  SPEAKER_01: Like, you know, we're not gonna pull the old releases, they'll still stop still work.
• 24:52
  SPEAKER_01: But um yeah, the the next major version of Airflow, which kind of rough ballpark early next year, so kind of you know, let's say end of Q1 next year, that kind of time frame, um will not work on Python 2.
• 25:09
  SPEAKER_01: Um if you still need some Python 2, kind of like for some of your tasks or jobs, um use something like the Python virtual end operator.
• 25:20
  SPEAKER_01: But for your main DAGs themselves, use Python 3.
• 25:25
  SPEAKER_01: Um that's probably a good way of doing it.
• 25:27
  SPEAKER_01: So you can swap your core over to Python 3 and then slowly port away the the use cases or keep them under Python 2 virtual end operators for if you actually can't.
• 25:36
  SPEAKER_01: But yeah, move away from Python 2, please.
• 25:40
  SPEAKER_01: Um, because that makes our life as makers easier.
• 25:44
  SPEAKER_00: That's cool.
• 25:44
  SPEAKER_00: I'm getting I'm getting flashbacks right now.
• 25:46
  SPEAKER_00: I didn't, I mean I we had I hadn't asked I wasn't actually planning on asking you about timeline for all this stuff because I know uh that's that's an engineer's worst uh nightmare.
• 25:54
  SPEAKER_00: But um yeah, it's funny because I remember last year I feel like in December like we were targeting for V1 of Kubernetes executor support by like mid-Q1, end of Q1.
• 26:04
  SPEAKER_00: So now it's like moving on to the Knative.
• 26:06
  SPEAKER_00: It's just like really cool to see that progress year over year.
• 26:10
  SPEAKER_01: Yeah, like it's kind of like no no promises on this time frame, but I I'm kind of driving to get FO2.0 out from pretty much a selfish point of view, which is I spend a lot of my time backporting fixes and having to manually re add Python 2 support.
• 26:28
  SPEAKER_01: So as soon as I cannot have to do that, my job as a release manager gets easier.
• 26:32
  SPEAKER_01: So these I think are some decent features that we need to get in to call call the next release airflow 2.0.
• 26:39
  SPEAKER_01: Um, and yeah, that's kind of you know a very rough back of the envelope estimate of how long it'll take us to do those features.
• 26:47
  SPEAKER_01: Um kind of I kind of mentioned this on the mailing list when I announced this kind of or talked about this roadmap, um, which is we at Astronomer are committing to do these features, and this is the kind of sort of time frame we think we can do them.
• 27:05
  SPEAKER_01: People are welcome to suggest other things, but you have to step up and do the work.
• 27:09
  SPEAKER_01: Um, this is our priority, we think it's good for airflow, but we're not gonna do other things unless someone makes a very good case as to why they're more important to Airflow than the things we've got.
• 27:22
  SPEAKER_01: So, yeah.
• 27:23
  SPEAKER_01: Ideas welcome, but yeah, they come with a well volunteers.
• 27:29
  SPEAKER_02: Well, that's on that here.
• 27:32
  SPEAKER_02: Um, how are you seeing the kind of the pace of the project right now?
• 27:36
  SPEAKER_02: Um, are you seeing like uh because you're still doing a lot of work on the uh being general release manager, or are you seeing pace of commits increase, decrease?
• 27:44
  SPEAKER_02: Like what does the general like state of the airflow community look like to you right now?
• 27:50
  SPEAKER_01: As a kind of general gut feel, we're getting we're getting better at reviewing PRs in approaching a timely fashion.
• 28:00
  SPEAKER_01: Um it's still far from where I want it to be, but I think we're getting better at it.
• 28:04
  SPEAKER_01: Um we're kind of getting new new features and our integrations are kind of getting getting tighter.
• 28:11
  SPEAKER_01: So the the folks at Polybear have done a lot of work integrating with GCP.
• 28:16
  SPEAKER_01: Um to kind of like you know, basically any GCP service has an operator and airflow.
• 28:21
  SPEAKER_01: So that's kind of exciting.
• 28:23
  SPEAKER_01: And yeah, like I definitely think that kind of the community is getting more engaged, more mature.
• 28:31
  SPEAKER_01: Um yeah, definitely I feel we're we're we're we're doing better as a project than we were two years ago.
• 28:39
  SPEAKER_01: We've got more committers, we've got more people reviewing PRs, we've got more people submitting PRs, is kind of what my my gut tells me about that.
• 28:47
  SPEAKER_01: Um I haven't looked at the data, so we may find out if we actually go to the data that it's completely the opposite, but that's what it feels like to me.
• 28:54
  SPEAKER_02: Well, if you're doing a lot of the reviews, I feel like you guys are probably a uh good thing to follow in in this case.
• 29:00
  SPEAKER_00: Yeah.
• 29:02
  SPEAKER_00: Yeah, cool.
• 29:03
  SPEAKER_00: Cool guys.
• 29:04
  SPEAKER_00: Well, Ash, I don't have anything else for you.
• 29:05
  SPEAKER_00: I don't know if you want to dive into anything else about the about about 2.0 or uh kind of anything else you want to talk about here, but um, you have the floor, if so.
• 29:15
  SPEAKER_01: Um no, I think I think that covers it.
• 29:18
  SPEAKER_01: Kind of um I guess kind of one last point, which is um kind of already in progress and mostly done, is as part of Airflow 2.0, we've renamed a whole load of modules and kind of moved them around.
• 29:34
  SPEAKER_01: Um there should be compatibility kind of you know compatibility shims in that, but essentially we've got rid of contrib.
• 29:42
  SPEAKER_01: So, you know, sometimes you had to kind of work out was this operator in contrib or not.
• 29:46
  SPEAKER_01: Um and we've kind of all rationalized that, and if we've just decided that contrib doesn't make sense under an Apache world, it was kind of when Airflow was still under Airbnb, it was contrib were things that contributed, and anything not under contrib was kind of officially supported by Airbnb.
• 30:05
  SPEAKER_01: But now it's under Apache, everything's contributed or nothing's contributed and it's all supported.
• 30:09
  SPEAKER_01: So it's kind of we don't need contrib anymore.
• 30:13
  SPEAKER_01: So let's let's remove that and yeah, for a release or two, we should have um import shims, and you'll kind of get a deprecation warning when you when you import it to tell you please update your code to use this new this new um new name.
• 30:29
  SPEAKER_01: Um and the other thing is we've reorganized the commands on the CLI to be kind of grouped.
• 30:42
  SPEAKER_01: Um so whereas before you would have airflow trigger dag and airflow list dag, it's kind of now airflow dags trigger, airflow dags list.
• 30:51
  SPEAKER_01: Um so everything's kind of scoped from from biggest to smallest and grouped.
• 30:57
  SPEAKER_01: Um when you when Floyd 2.0 finally comes out and you kind of your your your muscle memory runs a command and it goes like subcommand not found, that's why.
• 31:07
  SPEAKER_01: And yeah, I just kind of have to relearn a few things, but it makes it easier and nicer kind of to work out what's going on.
• 31:14
  SPEAKER_01: Um yeah, particularly with things like connections, so FO connections, which would have dash L to list, dash A to add, and dash D to delete, and then kind of the the help around all different arguments were a little bit confusing.
• 31:28
  SPEAKER_01: Now it's three separate subcommands, so the help can be a little bit clearer, and yeah, it just kind of tidies things up from that point of view.
• 31:35
  SPEAKER_02: Yeah, that's huge.
• 31:36
  SPEAKER_02: The uh I always have to like uh like paste at one spot and just like fill it in because all the get me all the flags right and such.
• 31:42
  SPEAKER_02: So I think that'll be a huge usability thing.
• 31:46
  SPEAKER_01: Yeah, it makes it a lot easier to kind of oh I want to operate on a DAG, therefore I look under the DAG.
• 31:51
  SPEAKER_00: Yeah.
• 31:53
  SPEAKER_01: So yeah.
• 31:54
  SPEAKER_01: Not a big thing, but again, kind of make makes it just nicer to use and feels more like what we expect from what I expect from The Liz these days.
• 32:04
  SPEAKER_01: So yeah, it's good.
• 32:08
  SPEAKER_00: Thanks to Ash for taking the time this week.
• 32:10
  SPEAKER_00: It was a pleasure chatting with him, and we're increasingly excited for the features to come in Airflow 2.0.
• 32:16
  SPEAKER_00: Thanks for listening, and please don't hesitate to shoot me an email at pete at astronomer.io if you have any podcast feedback or would like to hear us cover any specific topics in our upcoming episodes.
• 32:25
  SPEAKER_00: Until next time.