Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Airflow 1.10 Release

Transcript

Airflow 1.10 Release · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:01
  SPEAKER_02: Hi there, and welcome to episode 8 of the Airflow Podcast.
• 0:04
  SPEAKER_02: This week, we're excited to dive into the new Airflow 1.10 release with the release manager himself, Ashberlynn Taylor.
• 0:11
  SPEAKER_02: We'll discuss the release itself, what it's like to be a release manager on an open source project, Airflow's bid to graduate from incubating status, and the next phase of Airflow's project development.
• 0:22
  SPEAKER_02: This podcast is brought to you by Astronomer.
• 0:24
  SPEAKER_02: We at Astronomer are determined to make Airflow easier to run and manage.
• 0:29
  SPEAKER_02: Our offerings include both SaaS and self-hosted platforms that leverage the power of Kubernetes to make Airflow easy to run and scale, support for all things Airflow, and Airflow training that we call Space Camp.
• 0:41
  SPEAKER_02: We're also hiring platform engineers.
• 0:43
  SPEAKER_02: If you're interested in making it easy to run, scale, and monitor airflow on Kubernetes, drop us an email at humans at astronomer.io.
• 0:51
  SPEAKER_02: I'd also like to quickly mention that Astronomer recently released V.7 of our platform.
• 0:57
  SPEAKER_02: This release is our biggest yet.
• 0:59
  SPEAKER_02: Now you can choose your executor and scale your worker count, worker size, scheduler CPU, and web server CPU directly through our UI.
• 1:07
  SPEAKER_02: We also added a slew of Grafana dashboards that allow you to monitor information regarding persistent storage, Prometheus, the Docker Registry, Fluent D, and Airflow container state.
• 1:18
  SPEAKER_02: Finally, we rolled out initial Airflow 1.10 support in our platform, so it's only fitting that we dove into the release with Ash this week.
• 1:27
  SPEAKER_02: Without further ado, let's dive into this week's interview with Ash Berlin Taylor.
• 1:35
  SPEAKER_02: Alright.
• 1:36
  SPEAKER_02: Hey Ash, how's it going?
• 1:38
  SPEAKER_02: Hey, good.
• 1:39
  SPEAKER_02: Great.
• 1:39
  SPEAKER_02: Thanks for thanks for coming on today.
• 1:41
  SPEAKER_02: We're really excited to chat.
• 1:42
  SPEAKER_02: Um it's been we you know we've we've wanted to chat with you for a long time.
• 1:46
  SPEAKER_02: It was it was kind of tough tying down an interview.
• 1:48
  SPEAKER_02: I know you got a busy schedule.
• 1:49
  SPEAKER_02: You you mentioned that you recently had a kid, yeah.
• 1:51
  SPEAKER_00: Yeah, nine month old.
• 1:52
  SPEAKER_00: It's um it's been good, but tiring.
• 1:55
  SPEAKER_02: Cool.
• 1:56
  SPEAKER_02: Well, congrats on that and uh kind of congrats on balancing that with the recent with the recent airflow release as well.
• 2:02
  SPEAKER_02: That must have been quite a lot that you had on your plate.
• 2:05
  SPEAKER_02: Yeah.
• 2:06
  SPEAKER_00: It was a difficult month and a bit to to get it all done and kind of push everything through.
• 2:12
  SPEAKER_00: Yes, it it was a challenge.
• 2:14
  SPEAKER_00: It's uh glad to have done it.
• 2:16
  SPEAKER_02: So so uh why don't we just why don't we just dive right in here?
• 2:19
  SPEAKER_02: You mind just telling us a little bit about yourself, uh kind of how you got involved with Airflow and what you're up to right now?
• 2:24
  SPEAKER_00: Yeah, sure.
• 2:25
  SPEAKER_00: So um my background is I'm I'm the infrastructure person, you know, what some people might call DevOps, but I try not to.
• 2:32
  SPEAKER_00: Um so yeah, kind of lots of AWS automation, Terraform, Docker, and Kubernetes and that kind of stuff.
• 2:38
  SPEAKER_00: And um about two years ago I started a job and they went, well, you know AWS, so great, come in and deploy Airflow and AWS.
• 2:45
  SPEAKER_00: So I went, okay, that's that's fine, I can deploy software.
• 2:48
  SPEAKER_00: And then one thing turned into the next, and I ended up, I think for 1.9 release, there were eight release candidates, and I think six of them were because I kept finding bugs and then simple fixes, and then I think eventually the committers just got bored of having to you know review and merge my pull quests and went, ah, just just commit it yourself.
• 3:09
  SPEAKER_00: Um so I ended up as a committer of Airflow.
• 3:12
  SPEAKER_00: Um the start of this year, I kind of joined the project officially as a committer, and um yeah, then decided that it was only fair after putting bulk air through through all that hassle of uh 1.9 and and a bit on 1.10 that I should do the next release.
• 3:30
  SPEAKER_00: So I I ended up doing being the release manager for Flo 1.10.1, which uh we released a few weeks ago now.
• 3:38
  SPEAKER_02: That's awesome.
• 3:39
  SPEAKER_02: So so what percentage of your time would you say that you were spending on that release?
• 3:44
  SPEAKER_00: It was a good proportion of my what little spare time I had.
• 3:51
  SPEAKER_00: Um kind of and eking out an hour here and there at lunch and one or two hours every evening when I wasn't, you know, putting my child to bed or getting up to feed her halfway through the evening.
• 4:02
  SPEAKER_00: Um it was yeah, it was pretty full on, and it was it was a lot of a lot of work just to kind of chase things through and make sure you get the fixes we need and get people to test it, and then you know, helping debug problems and things like that.
• 4:16
  SPEAKER_00: So it was it was pretty awe-consuming for November.
• 4:21
  SPEAKER_02: Sure.
• 4:21
  SPEAKER_02: Wait, so so what do you do, just winding back a little bit, what do you do when you're not working on Airflow?
• 4:26
  SPEAKER_00: Um when I'm not working on Airflow, I hack on other things.
• 4:34
  SPEAKER_00: I I yeah, I like spending my time.
• 4:37
  SPEAKER_00: What little time I yeah, spare time is a little bit more precious now than it used to be, but um hack on things, try and yeah, just I noodle around on the computer.
• 4:46
  SPEAKER_00: That's that's what I do, is what I enjoy doing.
• 4:48
  SPEAKER_00: Um, whatever takes my fancy of an evening or of a week.
• 4:51
  SPEAKER_01: Yeah, I love that answer.
• 4:54
  SPEAKER_01: Um yeah, so kind of just diving into the release a little more.
• 4:58
  SPEAKER_01: Um kind of uh would love to talk about some of your pride points for the 1.10 release, right?
• 5:03
  SPEAKER_01: Like uh was there anything in particular that you're just super pumped is out there um and how you think how you think that affects the future of the project?
• 5:10
  SPEAKER_00: Um one of the big things in 110.1 was um so we tried to keep point releases to mostly just fixes.
• 5:20
  SPEAKER_00: Um but obviously you know people go like, oh, I've got this thing and it's just it's ready to be merged.
• 5:24
  SPEAKER_00: One of the big ones actually was um AWS contributed a whole suite of operators for their SageMaker thing.
• 5:31
  SPEAKER_00: So it's their kind of hosted machine learning.
• 5:33
  SPEAKER_00: They contributed a whole load of um operators and hooks and stuff.
• 5:38
  SPEAKER_00: So there's kind of I haven't dug into SageMaker that much yet.
• 5:41
  SPEAKER_00: It's kind of as with everything else AWS, there's just far more things coming out faster than you can learn them all.
• 5:47
  SPEAKER_00: But it's um it's an interesting one to think, so it's it's a kind of yeah, hosted machine learning kind of model generation thing.
• 5:53
  SPEAKER_00: And yeah, they've tied it into Airflow.
• 5:54
  SPEAKER_00: They AWS kind of took a look at it and went, Airflow is the Airflow is what people are going to use, so let's tie this into Airflow.
• 6:02
  SPEAKER_00: Um, so that's kind of super exciting that that AWS goes, yeah, Airflow is the way forward.
• 6:06
  SPEAKER_00: Um, particularly as they sort of have their own thing, like AWS Glue and Batch, and they've got a few other things.
• 6:11
  SPEAKER_00: It's it's kind of it's good that yeah, it's it's a good new sort of project that AWS have decided that it's worth them spending the time to build the operators and get them all merged and accepted.
• 6:22
  SPEAKER_00: So yeah, that was that was super exciting.
• 6:25
  SPEAKER_01: Very cool.
• 6:26
  SPEAKER_02: Um so just uh do you have any do you have a little bit of a kind of background or maybe a little bit more information on on what it was like to be a release manager and whether or not you'd like to do it again in the future?
• 6:41
  SPEAKER_00: So being a release manager is a little bit like it's the same as being an Airflow committer, but more so.
• 6:49
  SPEAKER_00: So our job as committers is mostly not to write our own code.
• 6:54
  SPEAKER_00: We'll do the occasional bug fixes, but most of what we do, most of the time we all spend on Airflow, is reviewing other people's PRs, giving them feedback, giving them getting them through to kind of accept it and merging it.
• 7:08
  SPEAKER_00: Um release management's the same, but it was also that particular release, I think there were two different penalty, you know, dependencies or dependencies of dependencies that um end up causing test failures for two or three days straight.
• 7:24
  SPEAKER_00: So at first it was dig into oh, the release branch is failing again, but we haven't actually merged anything, so what's going on?
• 7:30
  SPEAKER_00: Dig thing that, dig that.
• 7:32
  SPEAKER_00: And then yeah, so like the release manager is it's getting the fixes you need.
• 7:39
  SPEAKER_00: Um, because there are a couple of bugs that we needed to fix, getting those in, making sure they worked, um, working with the community, so um being active on Slack a lot, kind of helping people out, particularly as it's like I had a feeling that there might well be some issues in 110.0 that hadn't yet been reported that need fixing in 1101.
• 8:00
  SPEAKER_00: So it's kind of helping people out in Slack and being active and making sure that you know I kept an eye out for problems to fix to get in the release.
• 8:09
  SPEAKER_00: And then, yeah, a lot of a lot of merging and cherry picking to masters moved on a bit and uh our br the release branch is not so kind of having to get that and just general project management and chasing and making trying to get people to get their fixes in.
• 8:26
  SPEAKER_00: Um and then a little bit of strong army to get it through.
• 8:29
  SPEAKER_00: Um the final the final vote um particularly.
• 8:34
  SPEAKER_00: So Redis oh yeah got released.
• 8:39
  SPEAKER_00: I think it was like two days into our three-day vote in the final thing, and that broke celery, and it's like, well, yeah, but celery's not a core dependency and we're fixing some other things, and yeah, so it was a little bit of strong arming of going, I agree that's a problem, but can we release anyway, please?
• 8:57
  SPEAKER_00: Um especially as we're currently we're still in incubation for that release.
• 9:03
  SPEAKER_00: The next one, hopefully, we won't be.
• 9:04
  SPEAKER_00: But yeah, for that release, we are we're still in incubation.
• 9:06
  SPEAKER_00: So we had another 72-hour vote after our one passed.
• 9:10
  SPEAKER_00: So it was kind of like, well, let's just get this one out and we'll get the next one out quickly.
• 9:14
  SPEAKER_00: Um yeah, it's a little bit a little bit of strong arming there to get it through, but um I think it's better for it.
• 9:19
  SPEAKER_01: Yeah, super funny because uh my literal day today has been uh figuring out that uh Redis thing for our uh we want to rope in 1101 into our platform, so know that feeling.
• 9:28
  SPEAKER_00: Um Yeah, I was hoping I was hoping that um that Redis, oh sorry, that Celery themselves, um, you know, that it's it's Celery's transport library, which is called Combu.
• 9:37
  SPEAKER_00: There's a they've got a fix.
• 9:38
  SPEAKER_00: I submitted the fix.
• 9:39
  SPEAKER_00: Someone took my fix and did it properly, but you know, I submitted the fix and I was hoping they would they would release it quickly, but they're uh they're still sat on it, so they haven't fixed that yet either.
• 9:49
  SPEAKER_00: Um but yeah.
• 9:50
  SPEAKER_00: Yeah.
• 9:51
  SPEAKER_01: Um so jumping into the open source community a little bit more, um, I'm a pretty young dev and Pete's uh our newest React guy.
• 9:57
  SPEAKER_01: Um what like high-level advice would you have for like uh kind of like not super experienced devs that want to get more involved in open source projects, Airflow or just anything?
• 10:07
  SPEAKER_00: Yep.
• 10:08
  SPEAKER_00: Um there are two ways, two main ways to get involved in an open source project that are really useful.
• 10:15
  SPEAKER_00: One, which is the easiest one, fix the things that bug you.
• 10:20
  SPEAKER_00: If you find a problem, fix it.
• 10:22
  SPEAKER_00: If you've find, you know, like this this feature is close, but it doesn't quite do what I want.
• 10:28
  SPEAKER_00: Open a pull request, fix that.
• 10:29
  SPEAKER_00: Try keep your pull requests small, because they're easy to review.
• 10:32
  SPEAKER_00: So don't don't like build up loads of features.
• 10:34
  SPEAKER_00: Do a little bit at a time, do one small thing, change, it's a lot easier to review.
• 10:38
  SPEAKER_00: The other thing that's just massively, massively undervalued by undervalued is the wrong word, but underappreciated docs.
• 10:50
  SPEAKER_00: If you come along and submit a pull request that just add docs, you will be loved by the king by the committers.
• 10:56
  SPEAKER_00: Um it it it's super valuable.
• 10:59
  SPEAKER_00: If you know, even if you find if you find something that's confusing and then you you get some help in a you know support channel, Slack, uh, an email or whatever.
• 11:06
  SPEAKER_00: And if you then take that and turn it into a PR that improves the doc so that the next person doesn't have to struggle with that and ask the same question again, you'll be loved.
• 11:16
  SPEAKER_00: Um so yeah, write docs um and fix for both of you.
• 11:20
  SPEAKER_00: Um yeah, that that's the that's the two bits of advice I've got.
• 11:25
  SPEAKER_02: Um well those those definitely ring true for me personally, just for some context.
• 11:29
  SPEAKER_02: I I have a bit of a technical background in that I studied life sciences in college, but none in software.
• 11:33
  SPEAKER_02: So, you know, just kind of add a necessity at Astronomer, I've had to pick up a lot of things like fixing our internal docs, um, you know, fixing things that we don't have somebody to fix that need to be fixed, just like things like that.
• 11:46
  SPEAKER_02: So and also to be to be completely frank, like the idea of contributing uh back to the open source community at some point is really exciting to me.
• 11:53
  SPEAKER_02: And that I admit that that might be a product of you know my work environment and that I work for a company that is heavily involved in open source.
• 11:59
  SPEAKER_02: Yeah, um, but that's kind of my end goal, and I think it's super, super exciting.
• 12:02
  SPEAKER_02: So I definitely recognize that the community has a huge influence on folks that just want to get involved in software engineering, um, because it you know it gives you an end goal, really.
• 12:11
  SPEAKER_00: Yeah.
• 12:12
  SPEAKER_00: Yeah, absolutely.
• 12:13
  SPEAKER_00: No, I mean I've I mean I've been doing software development in one form or another for let's say prof professionally for 15 years.
• 12:23
  SPEAKER_00: Um and I started in Perl where everything was open source, you know, your your company bit, your company stuff, but like you built on so many open source libraries and keep frameworks and ORMs and stuff that just was all open source.
• 12:36
  SPEAKER_00: So yeah, I kind of got involved that way, and then a long time ago stopped doing Perl and just kind of missed the open source stuff a bit.
• 12:44
  SPEAKER_00: It's just nice to be involved, and you know, it's being involved, being a committer on a project is a is a slightly thankless task.
• 12:55
  SPEAKER_00: Um, you don't get paid to do it, and if you let it, it will eat all your time.
• 13:01
  SPEAKER_00: Um so you have you have to strike a balance, otherwise you get burnt out as a committer and that doesn't help anyone.
• 13:07
  SPEAKER_00: Sure.
• 13:08
  SPEAKER_02: Well, I want to say first and foremost that we and the entire community hugely appreciates your dedication to the project and the time that you've put in.
• 13:15
  SPEAKER_02: Because you know, while it is thankless you get there in in terms of you know you're putting a lot of time in and not getting paid for it, uh we really do appreciate the amount of work that you put in.
• 13:24
  SPEAKER_00: Thank you very much.
• 13:26
  SPEAKER_02: So, I mean, I can I kind of love to dive into the more dive into the incubation piece a little bit more.
• 13:31
  SPEAKER_02: Um, you know, Airflow has really hit this exciting point now where it's almost finally out of the incubating phase.
• 13:36
  SPEAKER_02: Um, first off, do you think it's ready to come out of the incubating phase?
• 13:40
  SPEAKER_02: Um and if and when it does, what do you think is the next milestone for Airflow?
• 13:45
  SPEAKER_00: Yeah, we're we're absolutely ready to be out of the incubation.
• 13:48
  SPEAKER_00: I think the last three releases have just been getting smoother and smoother.
• 13:54
  SPEAKER_00: Um the the licensing issues are we know what we're doing with that now.
• 13:58
  SPEAKER_00: Um so yeah, it's kind of you know, the Apache Software Foundation, the ASF, they they have licensing laws and a legal team to help you out with that, and you know, everyone loves to help.
• 14:09
  SPEAKER_00: And we've got that now and it's all sorted.
• 14:10
  SPEAKER_00: And yeah, the last point release went off without a hitch.
• 14:14
  SPEAKER_00: Um minimal police candidates, so we're getting it better at testing.
• 14:19
  SPEAKER_00: Um what leaving incubation and graduating to a top-level project means for airflow.
• 14:27
  SPEAKER_00: Um, well, the primary one is we get to get rid of that really annoying incubating tag up on our docs.
• 14:32
  SPEAKER_00: Um but what it actually means in practical terms, um releases just get a bit easier.
• 14:39
  SPEAKER_00: Um we we we don't have the extra vote, which this time happened pretty quickly, but previous times has taken one to two weeks to get enough votes just to get people to test it.
• 14:51
  SPEAKER_00: So we can release a bit quicker, um so that our releases are smaller.
• 14:57
  SPEAKER_00: Um the smaller the release, the less painful it is, the better we'll be like that.
• 15:00
  SPEAKER_00: So, yeah, that's something I'd like to see us do is release more often.
• 15:03
  SPEAKER_00: Um the next milestone, the big one, so it'll be it will be Airflow 2.0.
• 15:12
  SPEAKER_00: Um, what that will mean in practice is we'll kind of use it as an opportunity to get a bit of a clean slate, get rid of some deprecation warnings, remove some old code, remove some old ways of doing things.
• 15:24
  SPEAKER_00: Um one of the big ones is the whole web server will be changed.
• 15:29
  SPEAKER_00: So if you run an Airflow web server, it's currently just a simple Flask app with some you know minimal permissions model, things like that.
• 15:37
  SPEAKER_00: Um in 110, uh behind a config flag, there was a way to enable the experimental web server, often called RBAC, so role-based access control, um, which is built on a project called Flask App Builder, which kind of gives you a bit more scaffolding.
• 15:54
  SPEAKER_00: It means we have to do less ourselves.
• 15:55
  SPEAKER_00: We can have, and with that, we can have a much more granular permissions model.
• 16:00
  SPEAKER_00: Um so the Airflow web server itself will be a lot more powerful.
• 16:03
  SPEAKER_00: You can finally uh I'm not gonna say finally because I don't know this how how complete it is yet, but we're hoping to have um DAG level permissions.
• 16:12
  SPEAKER_00: So you can grant users permissions to view this one DAG, or you can view all the DAGs, but the DAGs for your team, you can clear them, you can restart them, you can view the logs and things like that.
• 16:22
  SPEAKER_00: So more granular permissions uh of uh on a per DAG level.
• 16:27
  SPEAKER_00: That's kind of cool.
• 16:29
  SPEAKER_00: Um also included um improvements to the scheduler.
• 16:36
  SPEAKER_00: So um Airbnb are still using Airflow, they're still contributing FedExes, which is great.
• 16:42
  SPEAKER_00: Um someone, I hope it's Airbnb, otherwise I've done someone a great disservice.
• 16:47
  SPEAKER_00: Um Kevin Yang um is the uh the committer name.
• 16:50
  SPEAKER_00: I'm fairly sure he's Airbnb.
• 16:51
  SPEAKER_00: If not, sorry.
• 16:53
  SPEAKER_00: Um but yeah, no, he's submitted a whole load of two, three really big PRs that have done wonders to improve the performance of um the scheduler.
• 17:05
  SPEAKER_00: So Airflow have something crazy like 30,000 tasks a day that get scheduled.
• 17:10
  SPEAKER_00: Um and yeah, they they he's they've submitted a whole load of PRs that have done great guns to just improve the performance of the scheduler, which is um which is really great.
• 17:22
  SPEAKER_00: So yeah, looking forward to that.
• 17:24
  SPEAKER_00: Um and then yeah, just generally a bit of a tidy up, a bit trying to get a bit of a handle on our tests and possibly our dependency versioning scheme to work out what we do about how tightly we pin or not the our the Python modules we depend upon.
• 17:45
  SPEAKER_00: Um it's caused some problems in the past.
• 17:48
  SPEAKER_00: The downside to pinning them too tightly is if you need to use a different version in your environment for a task or something, then it gets more difficult.
• 17:57
  SPEAKER_00: So we need to strike the right balance screen there.
• 18:00
  SPEAKER_00: Um yeah, and then kind of whatever else people think from the community contribute.
• 18:07
  SPEAKER_00: So um send us your pull requests.
• 18:10
  SPEAKER_02: Sure.
• 18:11
  SPEAKER_02: We're excited to see what's next.
• 18:13
  SPEAKER_02: I I I have a I have a little bit of a naive question, um, just kind of about coming out of incubating, and this could be my lack of knowledge, so apologies in advance if that's the case.
• 18:21
  SPEAKER_02: But do you think kind of pulling pulling off that incubating tab leads to a greater adoption at the enterprise level?
• 18:28
  SPEAKER_02: Like, do you think there are enterprises right now that are hesitant to adopt Airflow because it's an incubating project?
• 18:33
  SPEAKER_02: Does that label deter people from using it?
• 18:36
  SPEAKER_00: I don't think it does.
• 18:37
  SPEAKER_00: Okay.
• 18:38
  SPEAKER_00: Given the the companies that have publicly stated they're using it on our on our README, I think it's something over 400 companies now, and there's there's big companies in there.
• 18:49
  SPEAKER_00: Um and then there are banks and hedge funds and governments using it as well.
• 18:57
  SPEAKER_00: I don't think incubating is is deterring anyone.
• 19:00
  SPEAKER_00: Maybe a few, but there's still plenty who go like, no, it's it's it's been around for four years, I think.
• 19:07
  SPEAKER_00: 2014, 2015?
• 19:09
  SPEAKER_00: Yeah, yeah.
• 19:10
  SPEAKER_00: Something like that.
• 19:10
  SPEAKER_00: It's been around a while.
• 19:11
  SPEAKER_00: I I think it's it helps that it had been around a while before it was an Apache project.
• 19:16
  SPEAKER_00: So people have confidence in it.
• 19:17
  SPEAKER_02: It definitely it definitely helps when some of those larger institutions declare their loyalty to it.
• 19:21
  SPEAKER_02: I mean, like we talked to get we we talked to uh we talked to Dan Eberman over at Bloomberg, and you know, there's such heavy users over there, and um, you know, some other financial institutions with super sensitive data or something.
• 19:31
  SPEAKER_01: I feel like it's pretty battle-tested at this point.
• 19:32
  SPEAKER_01: Yeah, yeah.
• 19:34
  SPEAKER_01: Um, so I think uh one really interesting question we'd love to get your perspective on.
• 19:38
  SPEAKER_01: Uh you kind of alluded to this a bit before, but uh if you could kind of snap your fingers and have one feature built out or one thing like uh fixed on the project right now, uh where where would you put that and why?
• 19:52
  SPEAKER_00: Operability.
• 19:53
  SPEAKER_00: Operating Airflow is still not as easy as it should be.
• 19:56
  SPEAKER_01: Yeah.
• 19:57
  SPEAKER_00: Um knowing what Airflow is doing, knowing that it's healthy, knowing that it's running.
• 20:02
  SPEAKER_00: The worst one, a task is queued and it's really hard to work out why it's not getting picked up and run.
• 20:07
  SPEAKER_00: Yeah.
• 20:08
  SPEAKER_00: If that's that will be the features making Airflow easy to operate.
• 20:12
  SPEAKER_00: Um this kind of clearly comes from my background.
• 20:15
  SPEAKER_00: I I'm you know, sysadmin, web ops, DevOps, infrastructure, whatever.
• 20:20
  SPEAKER_00: I you know, I run software for professionally, and that's that's what I would add.
• 20:25
  SPEAKER_00: That's the thing that is highest up my list that I want to fix.
• 20:29
  SPEAKER_00: I don't know how to fix it yet.
• 20:30
  SPEAKER_00: I don't know what it looks like, but I know what I want to know.
• 20:34
  SPEAKER_00: Um, that would be the one thing I would add if someone went, we'll pay you for a year to do this.
• 20:38
  SPEAKER_00: I would, I would, I would that's what I would be.
• 20:40
  SPEAKER_00: I'll just start working on that.
• 20:42
  SPEAKER_01: Yeah.
• 20:43
  SPEAKER_01: Um, how about how do you think the uh adoption of uh like kind of like right now, I think everyone's uh standardizing infrastructure on Kubernetes.
• 20:51
  SPEAKER_01: Um do you think kind of the rise of Kubernetes adoption will make a problem like that a little easier?
• 20:55
  SPEAKER_01: Just because uh you can actually think of a one-size-fits-all solution.
• 21:02
  SPEAKER_00: No, I think it's to my mind it's sort of an author or you know, it's an orthogonal issue.
• 21:07
  SPEAKER_00: It's Kubernetes will make it easier to know.
• 21:13
  SPEAKER_00: Yeah, no, I don't it's all that they're orthogonal because a lot of the problem are kind of it's Airflow's scheduling logic that's some of the opaqueness.
• 21:22
  SPEAKER_00: So like Airflow, yeah, Kubernetes health checks might help with with some things of yeah, kind of this bit isn't working and you know your your web server's not running or your scheduler's not running.
• 21:33
  SPEAKER_00: Um but there are still plenty of people who are not on Kubernetes as well and won't be for years, if ever.
• 21:43
  SPEAKER_00: Um some people, you know, some people are still on prem, some people just don't like Kubernetes, and I I can't blame them.
• 21:52
  SPEAKER_00: It's it's a monstrously large piece of software um that also has its warts and operational operational issues.
• 22:00
  SPEAKER_00: And can bite you.
• 22:02
  SPEAKER_00: So um Kubernetes might help a bit, but uh, but I'd like to fix it in a way that works if you're still not using Kubernetes.
• 22:11
  SPEAKER_01: Yeah, super interesting.
• 22:13
  SPEAKER_02: Um sorry, I just completely lost my train of thought here.
• 22:18
  SPEAKER_02: So do you have you seen it run effectively outside of Kubernetes?
• 22:22
  SPEAKER_02: Like, what's the most effective like kind of structure that you've seen outside of using Kubernetes?
• 22:26
  SPEAKER_02: Like most effective deployment method?
• 22:28
  SPEAKER_02: Because we, I mean, I don't know if you know like kind of astronomer's history a little bit, but we I don't know, Viraj, you can probably speak better to it than I can, but we started out running running it on Mesos just as like a big monolithic airflow cluster, and that just did not work for us for reasons that Viraj can talk about.
• 22:44
  SPEAKER_01: Um big part of it was also just like uh re- internal resources, right?
• 22:49
  SPEAKER_01: Like we didn't have a full off DevOps team to help maintain that.
• 22:52
  SPEAKER_01: Uh we've liked Kubernetes a lot more, just uh because it's not as much of a um you don't need as many resources looking at one thing.
• 22:59
  SPEAKER_01: But uh yeah, I think it all depends on the org, right?
• 23:02
  SPEAKER_01: Absolutely.
• 23:03
  SPEAKER_01: The team you have.
• 23:08
  SPEAKER_00: So I mean the current place we've we've actually got a fairly small cluster, but we've got because of the way the the the you know multinational group is structured, we have multiple small clusters.
• 23:22
  SPEAKER_00: You know, we kind of have one which is just a master and a worker, um, another one which is about five nodes and stuff, and the deployment kind of also depends on what your workload is.
• 23:36
  SPEAKER_00: If most of your workload is triggering EMR jobs and you spin up EMR clusters or farm things off to EMR, Airflow doesn't need much because it's it's just making network requests and small ones at that.
• 23:48
  SPEAKER_00: Um I would be interested to to find out what Airbnb deploy, because I know they have a very sizable cluster.
• 23:55
  SPEAKER_00: Um and yeah, kind of like that.
• 23:58
  SPEAKER_00: Um generally anything that kind of so long as your deployments are repeatable, so build AMIs or use Chef or build RPMs or whatever it is, so long as you've got something like that that's repeatable.
• 24:10
  SPEAKER_00: Um and if you get the deployment pipeline down for getting your DAGs out to all your all your nodes at the same time, then they all kind of once you've got to that, they're all at the same level and they all they all work just as well.
• 24:26
  SPEAKER_00: Um so yeah, cool.
• 24:29
  SPEAKER_02: Cool.
• 24:31
  SPEAKER_02: No, that's not that's good.
• 24:32
  SPEAKER_02: That's good.
• 24:33
  SPEAKER_02: Uh so how do you kind of we talked about this a little bit before, but how do you see your involvement with the project changing going forward?
• 24:42
  SPEAKER_02: And I know that might be tough to say right now, given, of course, the nine-month-old kid and changing priorities and things like that, but I I would definitely still like to be involved with Airflow.
• 24:56
  SPEAKER_00: Um to some extent it depends on so I'm a I'm a I'm a freelance contractor, an independent consultant kind of thing.
• 25:05
  SPEAKER_00: Um so to some extent that depends upon my what my next gig is.
• 25:13
  SPEAKER_00: Um but I definitely think I will set I will still be involved in Airflow to some extent or the other.
• 25:20
  SPEAKER_00: Um if I'm no longer using it day to day, it might get a little a little bit harder to keep up with what's going on, but I definitely want to want to still be involved.
• 25:29
  SPEAKER_00: Um and yeah, and I really like Airflow.
• 25:30
  SPEAKER_00: It's it's now, you know, I've permanently got the Slack channel open and just kind of keep an eye on what's going on and what problems that people have, and you know, point them out that yeah, this having this issue here, no, no one's submitted a fix yet.
• 25:45
  SPEAKER_00: Maybe you could um open a pull request and be the hero.
• 25:49
  SPEAKER_00: But um yeah, it's just like reviewing pull requests every now and then as they kind of come in, it's I can kind of spend five or ten minutes a morning without feeling guilty and do that.
• 26:02
  SPEAKER_00: So yeah, I'm gonna stick involved, but it may it may be slightly less involved, slightly less all-consuming, which is again probably a good thing.
• 26:10
  SPEAKER_00: Um going forward.
• 26:12
  SPEAKER_01: Awesome.
• 26:12
  SPEAKER_01: Yeah, sweet.
• 26:13
  SPEAKER_01: Do you have any other you have anything else?
• 26:15
  SPEAKER_01: No, I think uh I think that was just the bulk of what we wanted to ask.
• 26:18
  SPEAKER_02: Um anything else specific that you want to talk about or plug?
• 26:23
  SPEAKER_00: Um yeah, I mean I guess there's a there's a couple of things, which is um finding yeah, trying to so the whole thing that went on with um the Node.js module, I can't even remember which one it was, but the one that got backdoored and um very recently and ended up stealing Bitcoin wallet private keys um kind of led me and a few other people, yeah, some people on my Twitter stream to kind of wonder and think about like is there a better way of funding open source?
• 27:04
  SPEAKER_00: Um and yeah, so trying to work out if it's worth trying to offer as a single, you know, one-man band, offer paid support for Airflow.
• 27:17
  SPEAKER_00: Um so yeah, kind of if if people, probably people in Europe, because that's where I'm based, so time time zones would make it difficult to do it for the states, but yeah, if people in Europe are interested in that, get in touch with me and um we'll see if we can make something work and then I get to work on Airflow more when I get to make it better.
• 27:38
  SPEAKER_00: Um because yeah, you know, if I can pay my costs and then have more time to do airflow work, then I can make Airflow better for everyone.
• 27:48
  SPEAKER_00: So that would be uh an interesting thing.
• 27:51
  SPEAKER_00: Um and yeah, just the other thing, just get involved.
• 27:55
  SPEAKER_00: If you want to add an operator for something, please, that would be great.
• 28:00
  SPEAKER_00: Just as long as it's got tests, we'll take it.
• 28:02
  SPEAKER_00: Um it's got tests, it's got some minimal documentation, and it's useful for even one more person than yourself.
• 28:09
  SPEAKER_00: Open a PR, and um we'd love to well we'd welcome the contributions.
• 28:15
  SPEAKER_02: Awesome, cool, yeah.
• 28:17
  SPEAKER_02: Sweet.
• 28:17
  SPEAKER_02: Well, Ashwell, thanks, thanks for coming on.
• 28:19
  SPEAKER_02: It's great to meet you.
• 28:20
  SPEAKER_02: Um again, Viraj and I were we're really pumped about this conversation, and I mean it's it's always so kind of exciting to hear uh the perspective of somebody that's like kind of been so intimately involved in the project of late.
• 28:31
  SPEAKER_02: Um so yeah, just thank you for giving us your time.
• 28:35
  SPEAKER_02: My pleasure.
• 28:40
  SPEAKER_02: Thanks to Ash for coming on.
• 28:42
  SPEAKER_02: Being the release manager is unquestionably a bit of a thankless job that requires a ton of donated time and intellectual energy, so we hugely appreciate the dedication to the project.
• 28:51
  SPEAKER_02: Thanks for listening, and please don't hesitate to shoot me an email at pete at astronomer.io if you have any podcast feedback or would like to hear us cover any specific topics in our next episode.