Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Airflow on Kubernetes

Transcript

Airflow on Kubernetes · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:02
  SPEAKER_05: Hi there, and welcome to episode six of the Airflow Podcast, produced by Astronomer.
• 0:07
  SPEAKER_05: In the first five episodes of this series, we've begun a deep investigation into Apache Airflow and have examined its origins, use cases, best practices, competitors, and pain points.
• 0:16
  SPEAKER_05: We're really excited to release this week's episode on running Airflow on Kubernetes.
• 0:20
  SPEAKER_05: It's a piece that we've been looking forward to diving into for a long time, and I really think you'll enjoy the interviews in this episode.
• 0:26
  SPEAKER_05: For those of you that don't know, Kubernetes, also known as K8S, is originally an internal project from Google that is now maintained by the Cloud Native Computing Foundation.
• 0:36
  SPEAKER_05: With Kubernetes, you can cluster together groups of hosts running Linux containers and easily manage those clusters.
• 0:42
  SPEAKER_05: It works with a range of different container tools, but it's designed with Docker in mind as a first-class citizen.
• 0:47
  SPEAKER_05: Based on our experience here at Astronomer, running Airflow on Kubernetes is the best way to achieve a scalable, highly available, and safe deployment.
• 0:55
  SPEAKER_05: This is due to the fact that Kubernetes has really strong community support, managed resources, is fault tolerant, uses resources efficiently, and has natural modularity.
• 1:06
  SPEAKER_05: This week we'll speak with two people, Daniel Emberman, senior software engineer at Bloomberg, and Greg Neijl, CTO here at Astronomer.
• 1:13
  SPEAKER_05: For those of you that don't know, Daniel has done a ton of work on the Kubernetes executor for Airflow, so we linked up with him to discuss what inspired that project and where he sees it going.
• 1:21
  SPEAKER_05: Hope you enjoy.
• 1:29
  SPEAKER_05: Awesome.
• 1:30
  SPEAKER_05: So do you mind just starting us off by giving us a little bit of an introduction for yourself?
• 1:33
  SPEAKER_05: Tell us about your background, how you ended up at Bloomberg.
• 1:36
  SPEAKER_03: Absolutely.
• 1:37
  SPEAKER_03: So uh my name is Daniel Emberman.
• 1:40
  SPEAKER_03: Uh and my background is I actually kind of start out mostly in the uh big data field.
• 1:46
  SPEAKER_03: So I've I've been working with Hadoop and like big data technologies for at least like six, seven years.
• 1:52
  SPEAKER_03: Um and basically what kind of got me to Bloomberg was I when I when I started my current job search, I really kind of decided that like I wanted this specific job to kind of be like my dojo.
• 2:04
  SPEAKER_03: Like I wanted to find a place where basically there were really interesting data problems, but also a large number of engineers specifically skilled in big data technologies.
• 2:14
  SPEAKER_03: Uh so when it came to Bloomberg, like we're talking obscenely large data sets that have extremely high veil high availability and also just um low latency, like you know, people are doing derivatives trading, like they you can't afford for anything to go down, you can't afford for anything to not be like like millisecond SLAs.
• 2:35
  SPEAKER_03: So it was really just kind of uh pretty net.
• 2:39
  SPEAKER_03: Um so that's so that's how I ended up at Bloomberg.
• 2:42
  SPEAKER_03: And basically what we're working on right now is that we kind of came in and we kind of realized that Kubernetes was a really great way to kind of take a lot of people off of legacy systems and into more open source technologies.
• 2:55
  SPEAKER_03: So we were kind of building out this um data science platform.
• 2:59
  SPEAKER_03: It's it's similar to uh an open source project called Kubeflow, where basically users can uh, you know, if you have a data scientist, rather than we wanted to lower the barriers to using a lot of open source technologies, be it like Airflow, Spark, TensorFlow.
• 3:13
  SPEAKER_03: So now you can just kind of request, like, hey, I want a Spark job.
• 3:17
  SPEAKER_03: And then rather than building it yourself, it just automatically appears in our cluster, you can do our work and then tear it down.
• 3:24
  SPEAKER_05: Yeah, it's funny that you mentioned that.
• 3:25
  SPEAKER_05: We've talked to a ton of different people on this podcast, and I feel like a lot of them fall into the bucket of we started on Hadoop, and my job is to get us off of Hadoop.
• 3:34
  SPEAKER_03: So I wouldn't I wouldn't say it's about getting off of Hadoop necessarily, but it's more just like, you know, Kubernetes is a much easier framework to work with than something like Yarn, for example.
• 3:46
  SPEAKER_05: Yeah, that's really cool.
• 3:47
  SPEAKER_05: Um it's it's really cool that you guys are doing it at such a tremendous scale.
• 3:49
  SPEAKER_05: I mean, we've talked to so many folks that are using all this open source technology, and you know, I mean, scaling tends to be a common issue.
• 3:56
  SPEAKER_05: Um we'll definitely we'll definitely dive into that in a little bit.
• 3:59
  SPEAKER_05: Uh but before we go there, do you mind just talking generally uh about like your data flow at Bloomberg, kind of like what what you're doing with all of it, where it's going, what tools you're using?
• 4:06
  SPEAKER_05: I know you just touched on it a little bit, but yeah.
• 4:09
  SPEAKER_03: So as far as as far as our data flow goes, um it's it's hard to give a single answer to that because there's just such a massively varied number of use cases.
• 4:21
  SPEAKER_03: Um like there, like when it comes to like how data from Bloomberg is being used, you have everything from like the like like streaming use cases like derivatives, all the way up to like these like really gnarly uh quant kind of use cases where people are like backtesting on like years worth of financial data.
• 4:39
  SPEAKER_03: Um so but what I what I can say as far as like um the the use cases I've personally seen is that you end up having there's a there's a lot of really cool stuff going on in terms of like uh like I guess the I guess the best way I could put it is that um teams at Bloomberg are really, really good at picking the best technologies for their use case.
• 5:04
  SPEAKER_03: So when it comes to streaming, there's been a lot of usage of things like Kafka and Flink.
• 5:09
  SPEAKER_03: When it comes to large-scale batch operations, I would say that Apache Spark has definitely taken on a lot of it.
• 5:15
  SPEAKER_03: Are you are you kind of more interested in like the actual like use cases themselves or the technologies we're using behind them?
• 5:21
  SPEAKER_03: Just to kind of clarify.
• 5:22
  SPEAKER_04: Um a little of both.
• 5:24
  SPEAKER_04: Like the tools hand in hand, right?
• 5:26
  SPEAKER_04: You can't talk about technology without talking about the use case because you're choosing the best tool for the job, right?
• 5:32
  SPEAKER_03: Yeah, so I mean, like a lot, there's a lot of text analysis.
• 5:35
  SPEAKER_03: So um like there are fun.
• 5:37
  SPEAKER_03: So just to kind of uh as a refresh, just to kind of like let people know, since um it's not as it's not a commonly known thing, surprisingly, is that Bloomberg is a like the the news corporation within Bloomberg is a very small percentage of what we actually do.
• 5:52
  SPEAKER_03: The main thing we sell is what's called the Bloomberg Terminal.
• 5:55
  SPEAKER_03: And the way that the Bloomberg Terminal works is that any every single like large like financial company and every single hedge fund, all these companies use this thing that it basically has access to any piece of data you could possibly need to make a financial decision.
• 6:09
  SPEAKER_03: Uh we we keep track of every single uh oil tanker on Earth, and we basically use water display water displacement to predict how much oil is in each tanker.
• 6:18
  SPEAKER_03: We have the supply chain function that you can see every single person that sells to a company like Apple and every single company that Apple sells to, so you can kind of figure out where what where people are sourcing their like glass screens and stuff like that.
• 6:33
  SPEAKER_03: So the that's why it's so hard to have a single answer for Bloomberg is that like we have like thousands of data sources taking in like terabytes of data every single day.
• 6:44
  SPEAKER_03: Um actually, one of my one of my fate one of my favorite kind of examples of that is that um Bloomberg actually has a full-time beekeeper on staff, where this beekeeper will basically keep track of bee populations because that's really important for things like like um crop futures and commodities.
• 7:02
  SPEAKER_03: Um so when it's so so when it comes to like asking like use cases for Bloomberg, our use cases are literally everything.
• 7:11
  SPEAKER_04: I'm I'm waiting for the blog post that's like Kafka for beekeeping or something.
• 7:15
  SPEAKER_05: We've been muting our audio while you're talking just to get better quality, but I wanted to flip my camera on to show you that we were that we cracked up when you started talking about the beekeepers.
• 7:23
  SPEAKER_03: That's awesome.
• 7:25
  SPEAKER_03: Like we we uh so and like that's like one of my favorite parts about working here is that like you know, I used to work in marketing data, and it didn't really get me out of bed in the morning to help Procter and Gamble sell more pampers, but when you're basically working on data that like moves the entire worldwide financial industry, it's it's such a fascinating way to work.
• 7:46
  SPEAKER_03: For sure.
• 7:46
  SPEAKER_05: Yeah.
• 7:47
  SPEAKER_05: We Procter and Gamble, we're in Cincinnati, we're Cincinnati-based, and PNG is our our kind of next door, our next door neighbor.
• 7:53
  SPEAKER_05: So we might have to cut that one out, but no, I'm just kidding.
• 7:57
  SPEAKER_05: All right, so so kind of how does Kubernetes play into that processing at scale then?
• 8:01
  SPEAKER_03: So Kubernetes is just an unbelievably powerful framework when it comes to kind of taking these old, like taking these behemoth projects and moving them into like individual microservices.
• 8:12
  SPEAKER_03: The way I like to describe Kubernetes is the way that Docker abstracted away the single machine.
• 8:18
  SPEAKER_03: So now you don't have to you have virtual ports instead of actual ports and stuff like that.
• 8:21
  SPEAKER_03: Kubernetes abstracts away the cluster.
• 8:24
  SPEAKER_03: So now you don't need to know what machines your Maria DB lives on.
• 8:29
  SPEAKER_03: You can just have a MariaDB service, and then you could have that service act as a load function and then just abstract the whole thing away.
• 8:39
  SPEAKER_04: Um so kind of uh taking a step to the side here, uh I think you mentioned this a little bit earlier, but what made you guys start using Airflow?
• 8:47
  SPEAKER_04: Just kind of as a SparkTop coordinator or just that whole process?
• 8:52
  SPEAKER_04: Sorry, supplement words.
• 8:54
  SPEAKER_03: No, no, right.
• 8:55
  SPEAKER_03: So our main use case is for uh machine learning and data scientists.
• 8:58
  SPEAKER_03: So we wanted to allow these data scientists to basically have access to like if they had a complex workflow.
• 9:04
  SPEAKER_03: So for example, if you're taking in data from multiple, uh let me mute my Slack, sorry about that.
• 9:10
  SPEAKER_03: Um if you're if you're taking in like a complex workflow where like you might need to like access a Kafka cluster for one section and then like like HTFS for another, um we wanted to give that flexibility where you can kind of run these pipelines and whatever configuration you want, that configuration is very easy to use.
• 9:30
  SPEAKER_03: And also very importantly, um the ability to just kind of write it out as Python rather than worrying about like complex YAMLs or like XML in the case of Uzi, it it works a lot better for people for people in data science who come from Python backgrounds.
• 9:46
  SPEAKER_04: Yeah, we've kind of heard the same thing where data scientists love Python, and uh from the infrastructure side, I heard someone describe it as like the lowest common denominator, as in like easiest to spread to everyone.
• 9:58
  SPEAKER_04: And that's kind of helped the adoption of Airflow for data engineering as well as data science.
• 10:03
  SPEAKER_03: Oh, absolutely.
• 10:05
  SPEAKER_05: So then what what really inspired you to start the work on the Kube's executor?
• 10:08
  SPEAKER_05: Like what were the problems that you were facing internally that that led to kind of um you diving into that issue?
• 10:14
  SPEAKER_05: So this is sorry, before we start, maybe you should explain what exactly the Kube's executor is.
• 10:19
  SPEAKER_03: Absolutely.
• 10:20
  SPEAKER_03: So I I mentioned a little bit about like the the the Spark, the Spark on Kubernetes where there's you have this dynamic allocation, and it's a very similar philosophy where basically one kind of long-standing issue that I've seen with a lot of people who use airflow is you end up wasting a lot of resources with these static clusters.
• 10:37
  SPEAKER_03: So you if you're running, if you're running like the salary executor, you end up having a lot of Airflow machines that aren't doing anything.
• 10:43
  SPEAKER_03: And so that can get really expensive on your AWS bill.
• 10:46
  SPEAKER_03: So we wanted to basically allow people, especially since Kubernetes makes it so easy to like launch and destroy things, why not just have it that you can scale up your cluster as much as you need, run your like run your large DAGs, and then the second you don't need it anymore, it shrinks down to a single pod.
• 11:05
  SPEAKER_04: Yeah, that sounds like sounds like the dream, right?
• 11:07
  SPEAKER_04: Only pay for what you use at all times, never any idle resources, and also no manual scaling, yeah, no adding workers by hand.
• 11:14
  SPEAKER_03: Exactly.
• 11:14
  SPEAKER_03: Uh and so as far as why we got into it, this is kind of just a cool philosophy that Bloomers taken on, which is anytime we start using a new open source technology, the first thing we ask is, how can we contribute to it?
• 11:27
  SPEAKER_03: And what we find is that by doing open source contributions, you end up really getting to know the community, you end up getting really valuable resources.
• 11:36
  SPEAKER_03: So through Airflow and Kubernetes and Spark on Kubernetes, we now have we we now have regular contact with like the actual Kubernetes team.
• 11:44
  SPEAKER_03: And that has that has paid off so well in terms of like if we run into a Kubernetes problem, we can now hit, we can now contact these people who are world Kubernetes experts and get their opinions on how to solve these things.
• 11:59
  SPEAKER_03: Uh you're a still music.
• 12:00
  SPEAKER_05: Yeah, for sure.
• 12:01
  SPEAKER_05: And definitely kind of a good example where being a good open source community member and like a faithful citizen pays off, right, at a high level.
• 12:08
  SPEAKER_05: Like you get that access to the team.
• 12:09
  SPEAKER_05: We're huge fans of open source at astronomer, of course.
• 12:12
  SPEAKER_05: So it's always good, always good to hear that um people are adding back.
• 12:16
  SPEAKER_05: So then kind of going back to the Ku's executor a little bit.
• 12:19
  SPEAKER_05: Um so what what what kind of aspect of of writing it and starting it off?
• 12:24
  SPEAKER_05: I know it's not finished yet, um, but did you what what aspect of it was the most challenging to kind of dive into, technically speaking?
• 12:31
  SPEAKER_03: The hardest the hardest part by far was just kind of making it wasn't so much like the large scale effort.
• 12:42
  SPEAKER_03: It was there were two things that were kind of complic complicated with it.
• 12:45
  SPEAKER_03: One was basically writing in a way that like didn't involve any okay.
• 12:54
  SPEAKER_03: So the two things I would say was number one, robustness.
• 12:58
  SPEAKER_03: Like, how do you there's so many unknown unknowns?
• 13:00
  SPEAKER_03: You have to re like it's it's a it was a pretty large effort.
• 13:04
  SPEAKER_03: So to do that in a way that doesn't cause like issues, like we didn't want to end up having people like having their like daily runs not working or run into running into issues in the wild.
• 13:14
  SPEAKER_03: So there was just a lot of kind of testing to like find these like kind of like small small issues before we felt like comfortable releasing it.
• 13:23
  SPEAKER_03: So like um Seth Edwards, who uh just kind of put a plug in, Seth and I are gonna actually be speaking at Pi Bay in August.
• 13:31
  SPEAKER_03: Uh he he and his company PubNub were basically able to um put put the Kubernetes executor into production, and through that they were able to find a lot of these kind of like in the wild bugs that gave us a lot more confidence when we did this initial release that people weren't going to end up like losing their workflows or anything as they run the Airflow Kubernetes executor.
• 13:56
  SPEAKER_04: Yeah, very cool.
• 13:57
  SPEAKER_04: Yeah.
• 13:58
  SPEAKER_04: Um so Taylor can probably speak a little more about this than I can, but we ran a Mazels executor for a while.
• 14:04
  SPEAKER_04: Um we got that working pretty good, not exactly 100%.
• 14:08
  SPEAKER_04: Um and we still had the issue of the idle resource.
• 14:11
  SPEAKER_04: So hopefully the Kubernetes executor kind of pushes over the edge and gets us to where we need to go.
• 14:17
  SPEAKER_03: So like the two things that we really wanted to do in terms of benefit were not only the dynamic allocation, but also high availability, where we basically took advantage of something called a resource version in Kubernetes.
• 14:28
  SPEAKER_03: And using this resource version, we basically made the entire, like the entire airflow system stateless.
• 14:36
  SPEAKER_03: So that that to us was a big concern, which was that in the using that we didn't want to risk a situation where like an airflow cluster goes down and then you lose information.
• 14:45
  SPEAKER_03: So what ends up happening is like when every time a thing happens in Kubernetes, like a pod gets launched, a pod gets modified, a pod gets deleted, all of that gets stored in the cluster temporarily.
• 14:57
  SPEAKER_03: So we're able to take that information to recreate state if your Airflow scheduler goes down.
• 15:03
  SPEAKER_05: Yeah.
• 15:04
  SPEAKER_05: That's kind of a good that's kind of a good segue into the.
• 15:07
  SPEAKER_05: Oh no, Taylor, go for it.
• 15:09
  SPEAKER_01: I was just gonna say, can you dive into that point a little more?
• 15:12
  SPEAKER_01: Sounds uh pretty interesting.
• 15:13
  SPEAKER_01: And especially I'm kind of thinking like, yes, we do a lot of stateless stuff in Kubernetes.
• 15:19
  SPEAKER_01: Um and then I know also Airflow sort of says similarly the best practice for your DAGs is to write idempotent DAGs, but like in practice, we've had a lot of difficulty in terms of we can't impose on our customers that all their DAGs are idempotent.
• 15:32
  SPEAKER_01: So I'm just curious if you guys have done anything like that kind of in conjunction with just staying consistent with statelessness.
• 15:41
  SPEAKER_03: So I mean, as far as like itempotence within the DAGs themselves, like you can only you can only push best practices.
• 15:49
  SPEAKER_03: You can't the it would be very hard to actually enforce what people are doing inside the code of like what they're running.
• 15:56
  SPEAKER_03: Um but what we were more focused on was just making sure that like you know, because Airflow, Airflow uses a lot of like cues that are in memory, and so if those if the Airflow scheduler crashes, then those cues end up being emptied.
• 16:11
  SPEAKER_03: So we wanted to make sure there was an easy way to repopulate those cues with up-to-date data.
• 16:15
  SPEAKER_03: So for example, if you're run, if if Airflow launches, if Airflow ends up like launching your um launching a pod, like launching a task, and then the scheduler dies, and then the task completes while uh while the Airflow cluster is down and then comes back up, we wanted to make sure that the Airflow, the the database in the back end knows to actually report that task as like passed or failed, rather than still believing that it's running.
• 16:41
  SPEAKER_03: So then what ends up happening is that we have a different table in called resource version in the in the Postgres instance where uh it will know the most recent resource version and then it will read from the Kubernetes using the Kubernetes API, it will read to see what's happened since that last reported checkpoint, and then pull, like basically update the database as needed with that information.
• 17:06
  SPEAKER_01: So do you dump that in memory queue somewhere?
• 17:10
  SPEAKER_03: Um so what ends up happening is uh we don't we don't dump the in-memory queue.
• 17:17
  SPEAKER_03: What we do is uh instead of what we do is that like every single time that the pod is like every single time the pod is launched, like so we have this thing called we the Kubernetes executor has a thing called a watcher, which is uh part of the Kubernetes client where uh this watcher will basically be reading the Kubernetes event queue.
• 17:41
  SPEAKER_03: So when the like when the pod is launched, that's when we consider the pot like the task as running.
• 17:47
  SPEAKER_03: Therefore, we'll receive that information from the from the watcher and then basically be able to like pass that information to the Postgres instance as well as take that resource version and pass that resource version to Postgres as well.
• 18:05
  SPEAKER_01: I I might make sense.
• 18:07
  SPEAKER_01: It's so it's definitely a little different than the uh Mesos executor setup.
• 18:10
  SPEAKER_01: I don't know if you've looked into the Mesos executor at all.
• 18:13
  SPEAKER_03: So what's actually really funny is that when I when I first started with the Kubernetes executor, the the dynamic allocation system was how I assumed that the Mesos executor worked.
• 18:24
  SPEAKER_03: So I like I I like I lost about like three days trying to figure out how the Mesos executor was doing dynamic allocation before I figured out that it actually wasn't.
• 18:34
  SPEAKER_03: And it was just like you were having a like a static Airflow cluster in Mesos.
• 18:41
  SPEAKER_04: Yeah.
• 18:42
  SPEAKER_04: So uh as you've kind of worked through all this, just kind of staring back at Airflow itself, what has been your like biggest hair pull-out moments about Airflow?
• 18:50
  SPEAKER_04: Like what is uh Airflow is just like driven you crazy?
• 18:58
  SPEAKER_03: Well, I can say what well to start out to start out with the like just kind of keeping things positive, I can say that the Airflow community has been absolutely incredible through the entire process.
• 19:10
  SPEAKER_03: Like Ruin, like have been who are Airflow committers, have been really, really like positive influences.
• 19:19
  SPEAKER_03: They've been extremely helpful.
• 19:20
  SPEAKER_03: Um, I've also been able like the Kubernetes community as well.
• 19:23
  SPEAKER_03: Uh so just to kind of give a shout-out, there's a Kubernetes community called Sig Big Data that their big focus is basically bringing big data technologies into Kubernetes.
• 19:34
  SPEAKER_03: And so they led Spark on Kubernetes and it they helped with a lot with the airflow.
• 19:38
  SPEAKER_03: They're also doing HTFS.
• 19:40
  SPEAKER_03: Uh so those were kind of like the things where like any issues we ran into would really basically um there was a lot of resources to address them with.
• 19:50
  SPEAKER_03: In terms of hair pullout moments, I think that the one thing that I would consider a hair pull-out moment that kind of is something that's being actively worked on is it it's not super straightforward to set up an Airflow testing environment.
• 20:10
  SPEAKER_03: Um I think that once once the Kubernetes executor kind of settles down, that that would be kind of like a next interest would be how to simplify deployment of like just being able to like spin up an Airflow cluster from nothing and being able to like test custom code so that it would also kind of like lower the barrier to entry for new people who are interested in working within Airflow.
• 20:36
  SPEAKER_04: Yeah.
• 20:37
  SPEAKER_04: Um, that was my next question.
• 20:38
  SPEAKER_04: What would have been the best part about it?
• 20:40
  SPEAKER_04: And the community is definitely something that uh it's been a huge plus for us, right?
• 20:43
  SPEAKER_04: Like as we've decided to make Airflow a bigger and bigger part of our company.
• 20:47
  SPEAKER_04: Uh it's been great just to have this like huge community of really positive, helpful people out there.
• 20:52
  SPEAKER_03: I would say that the Airflow community and the Kubernetes community have two been two of the most like positive, like welcoming open source communities I've I've had the pleasure of working with.
• 21:02
  SPEAKER_03: Like they like it's re it's really awesome to see that like they really actively like there are certain there are certain open source projects where like you'll have like a single company kind of like running the show, and those ones end up being a lot harder because it's harder to like kind of get into those communities without kind of considering the business goals of the the main company.
• 21:24
  SPEAKER_03: And I it's been great with these up with these two projects in particular.
• 21:29
  SPEAKER_04: Cool.
• 21:30
  SPEAKER_04: Um, so you mentioned it a little bit with the testing and deploying environment.
• 21:34
  SPEAKER_04: But uh, what are some features that if you could just snap your fingers and be done 100% you would want to get done?
• 21:40
  SPEAKER_03: For for Airflow?
• 21:41
  SPEAKER_04: Yep, Airflow and even cubes.
• 21:44
  SPEAKER_03: Um so you know it's it's it's a it's hard to say what specific features I would want in terms of like airflow core, but what I can say is that there's some uh really like there's there's some features I'm really, really excited.
• 22:00
  SPEAKER_03: in terms of like the like the next steps for the Kubernetes executor that like like once those go out, I think the it'll kind of really increase the ability to like do like like a single Kubernetes executor for like a very large scale cluster and multi-tenancy and stuff like that.
• 22:15
  SPEAKER_05: So what are can you talk about some of those features?
• 22:17
  SPEAKER_03: Yeah, absolutely.
• 22:18
  SPEAKER_03: So I mean uh one thing to kind of mention is that like when when we started the the the kind of like the version that's coming out in Airflow 1.10, uh Airflow didn't have RBAC.
• 22:29
  SPEAKER_03: So because Airflow didn't have RBAC, there was no the the only security we could really kind of guarantee would be like Kubernetes has this concept of a namespace which is a form of isolation where you like your username is attached to your namespace.
• 22:43
  SPEAKER_03: So right now the the main form of security would be launch airflow in your namespace and then you can uh you'll you'll be able to ensure that no one else is like doing anything on your airflow like with your airflow cluster.
• 22:56
  SPEAKER_03: But what's really cool is that with this kind of new version um that's been there's a there's a that's that's kind of in the works right now is that you'd have a single like kind of a single privilege like it's called a custom controller where basically you can request from Kubernetes to build a thing for you.
• 23:17
  SPEAKER_03: So what ends up happening is that you'd have this single custom controller that you could tie to your airflow RBAC and then what ends up happening is rather than having airflow launch a single pod per task, you could actually make a request of like give for this DAG, give me an Airflow cluster of like six workers just to complete this DAG and then that way you won't have like the overhead of like every single task having a pod.
• 23:44
  SPEAKER_03: And I think that that will highly increase like the kind of flexibility and scalability where um you can really just define per DAG and then yeah that I think that's going to be a really really cool feature.
• 23:57
  SPEAKER_05: Yeah it does sound super cool.
• 23:58
  SPEAKER_05: We're actually talking to Joy Gao who did all the RBAC work um on that PR tomorrow so we're we're pumped about that conversation as well.
• 24:05
  SPEAKER_05: Awesome so I mean so those are some good strengths and some some some kind of bright things to look forward to with respect to the Kube's executor.
• 24:12
  SPEAKER_05: Can you talk about some of the shortcomings of it in its current state?
• 24:15
  SPEAKER_03: Yeah so um as I as I'd mentioned like one definite like kind of shortcoming is that uh when it comes like launching a new pod for every single task um that works out it works really well when it comes to like very like wide DAGs.
• 24:36
  SPEAKER_03: So if you are running a lot of things in parallel it kind of works out nicely to kind of like launch like a one at a time.
• 24:43
  SPEAKER_03: But it does have the overhead of having like relaunch every single time.
• 24:47
  SPEAKER_03: So um I would say but like I said that was kind of working with the the features we had at the time and also just kind of like we wanted to make sure that there was no dependencies on like any like go go laying or anything like that.
• 25:02
  SPEAKER_03: So we wanted to kind of get the get the airflow community into Kubernetes with as like low of a barrier as possible and then kind of start bringing in a lot some more like cute best practices later.
• 25:17
  SPEAKER_03: As far as other any other shortcomings I think that we're gonna I I'm gonna have to wait to see how people are using it and what what feedback we're getting and that that's where I really want to kind of like be able to iterate based on use cases.
• 25:34
  SPEAKER_05: Yeah for sure that totally makes sense so then kind of I'm curious to hear your thoughts on the greater container orchestration landscape.
• 25:43
  SPEAKER_05: What why why do you think Kubernetes is the best option compared to other options like Mesos or Docker Swarm?
• 25:47
  SPEAKER_05: I mean we talked a little bit about Mesos before but um yeah I would just love to hear your thoughts on the future of of that space.
• 25:53
  SPEAKER_03: Abs yeah absolutely and I I definitely would agree that I think that like so Kubernetes is still a a pretty young project.
• 26:02
  SPEAKER_03: The two main things holding it back right now is just like uh it there's some work that needs to be done in terms of like scaling to larger clusters.
• 26:11
  SPEAKER_03: So right now like like Mesos can handle a much larger cluster than like the Kubernetes can.
• 26:16
  SPEAKER_03: And also uh Kubernetes needs to kind of work on work on their resource allocation.
• 26:20
  SPEAKER_03: So right now Kubernetes is a greedy scheduler but they kind of need like smarter ways to like handle the scheduling.
• 26:27
  SPEAKER_03: But in terms of just for its age and how quickly it's amassed just a massive community I just see the tr the trajectory of Kubernetes is just really like I I I haven't seen anything like it since like Spark with in terms of just building this massive and varied community.
• 26:46
  SPEAKER_03: I'd also say that Kubernetes kind of won the won the battle of easy usability like the the API is very simple.
• 26:53
  SPEAKER_03: They there's a lot of because it's been so good at building this like very like cross company ecosystem there's been a lot of multiple efforts for it on each step to kind of like build these abstraction layers on top of it.
• 27:08
  SPEAKER_03: So now you have things like kubeflow coming out to simplify it for data science you have um all these other companies coming out to handle everything from like from like mesh networking and yeah basically basically it's just that um Kubernetes I think is just Kubernetes just has such a like a a vast ecosystem of multiple companies but more importantly it's just it understands that the like the like the low level systems are not what the users are supposed to be interacting with.
• 27:42
  SPEAKER_03: And that's actually a big part of what we wanted to do with Airflow was abstract away a lot of the complexity of kind of like working directly with the Kubernetes API.
• 27:52
  SPEAKER_04: So you mentioned those abstractions um I'm just getting my feet wet with kubes recently um and I found those really power powerful as like understanding what's going on under the hood without actually having to like deconstruct anything like hide them from the end user.
• 28:06
  SPEAKER_03: Sorry Viraja um uh you you cut out there for a second uh sorry I was saying uh I'm just getting started with cubes um and you mentioned those abstractions and I definitely think that's been a huge part of why it's growing so quickly so that uh as you just said the user doesn't have to interact with the things under the hood but can still like get things working to the way they want to there's so there's this there's um there's this a really awesome thought leader in the Kubernetes community his name is uh Kelsey Hightower and he gave the uh he gave one of the keynote speaks speeches at uh kubeflow Austin a couple months back and his big point of his cute of his speech was basically that the Kubernetes API and like the YAML files or anything like that, those are not supposed to be the ways that the user should interact with Kubernetes.
• 28:55
  SPEAKER_03: Like the goal of the Kubernetes community should be to make sure that those users never see those like low level low level issues.
• 29:03
  SPEAKER_03: And I think that the airflow the airflow and Kubernetes effort as a whole is a really good kind of like one one facet of the realization of that vision that I think it's it's cool to see a lot of things going on in that community to do that abstraction.
• 29:20
  SPEAKER_05: Yeah yeah for sure sweet so Taylor do you have anything else that you want to ask or do you Raj that was a list of questions I had running down them all.
• 29:31
  SPEAKER_01: I do kind of have some more points about Kubernetes.
• 29:34
  SPEAKER_01: It was interesting that you brought up the Kelsey Hightower point.
• 29:38
  SPEAKER_01: He actually came to Cincinnati a few months ago as well there's a very big uh retailer in Cincinnati that is going full GCP and is embracing Kubernetes a little more and um we're not directly involved with that project but we're involved in like a related project at one of their other companies uh but anyway they brought Kelsey Hightower to town and he gave a nice meetup talk and he explained the point like you said of you know this is how you configure Kubernetes today.
• 30:05
  SPEAKER_01: We have these YAML files this isn't our ideal API but we'd rather have explicitness here and then we'll build layers on it later and make it easier to use.
• 30:13
  SPEAKER_01: But I I agree with you as well like coming from the DCOS or Mesos versus Kubernetes background um it's just easier to start using Kubernetes and the the concepts you know pods services all this are a little more delineated and uh it's like very clear cut layers um with airflow specifically in the cube executor I'm kind of wondering how do you think about a dynamic resource allocation for tasks like um for instance something that's hard for us today is let's say I have a DAG and my DAG wants to spin up uh a few tasks that needs like let's say tons of memory and a few tasks that need like some memory uh but they need a ton of CPU or maybe um it's like an hourly task and uh on a typical hour let's say it needs 128 megs of memory but then there's like one hour per day where it gets a big burst of data and needs like gigabytes or more have you guys handled sort of dynamic allocation around that kind of stuff?
• 31:16
  SPEAKER_03: That that's actually thank you for bringing that up that's actually a point I should I should have mentioned is another really big benefit of the Kubernetes executor is that on a per task basis you can actually define how much resources you want in that individual task.
• 31:31
  SPEAKER_03: So we we we added an we added an extra column to uh to the the the task like definition called executor underscore config and so if you define the executor config you can actually give it a map and say like give me this much memory give me this much CPU power and then that way you know if you're doing one task that's like this that's working with scikit learn and another task that's that's just like launching a spark job you don't have you can differentiate how much how much uh how much power you want to give to each of those tasks.
• 32:05
  SPEAKER_01: Nice and and do you have anything for the use case or maybe just your thoughts on how you would address this if you have a task where it needs a certain amount of resources most of the time but then every once in a while it spikes like the way we kind of address it today is uh we notice when things run out of memory and then we just bump on it for that task really high but it kind of you know the trade-off right is now we're saying every time it launches uh allocate for two gigs of memory even if you only need 256.
• 32:37
  SPEAKER_01: Do you have like flexible limits or anything like that?
• 32:40
  SPEAKER_01: Have you thought about that or is that like a problem you've experienced?
• 32:43
  SPEAKER_03: So it's not a it's not a problem we've experienced because we're mostly working in the case of like data science where like the only the only long running task would be something like model inference and if you're doing model inference then at that point you can kind of scale horizontally rather than vertically.
• 33:02
  SPEAKER_03: So in terms of in terms of being able to kind of like have like changes in resources in the middle of tasks that's that's not something we've addressed yet.
• 33:16
  SPEAKER_03: It would be interesting to see if there's something in the I think there might be if there's something in the like Kubernetes pod API where you could actually like define something in the YAML to kind of say like when it reaches this percent threshold like resize the pod.
• 33:30
  SPEAKER_03: I I can't answer that on the I I I wouldn't know the answer that on the fly though unfortunately.
• 33:36
  SPEAKER_01: Yeah yeah no it's uh kind of just thinking uh high level like that I think even conceptual level is super helpful.
• 33:43
  SPEAKER_03: We haven't really dug into the technical details of that yet either it's it's it's interesting because it's also like when you kind of think of like the like uh when you think of airflow tasks like you usually think of something that like wouldn't run so long that it would have to worry about those like peaks and valleys but uh I guess it I would love to kind of like at some other point dive more into that use case to see if there's something we could do about that.
• 34:07
  SPEAKER_01: Yeah I I think it's interesting just to um kind of circle back to what you said a minute ago what we've seen in our airflow users so far is um some people are running these long running tasks like that on airflow directly and then other people are running almost nothing on Airflow talking more like your Spark use case where you uh you spin up a Spark cluster or you have like a long running Spark cluster and you just dispatch jobs to the Spark cluster but as far as like like airflow all the execution there is like very thin is that it seems like what you guys use is more that second style like thin airflow and dispatch to clusters.
• 34:44
  SPEAKER_01: Is that right?
• 34:46
  SPEAKER_03: I would say that yeah I would say that the the majority of the use cases I see is to kind of keep airflow itself pretty lightweight.
• 34:55
  SPEAKER_03: Just because like especially if you have like if you have things like already set up like you know when when you're submitting a Spark job you all you need is to kind of like have those resources set and then Spark and Kubernetes or even just like the static Spark cluster will take care of it.
• 35:14
  SPEAKER_03: And you know I I even if you if you talk to Max if you talk to Maxime about it like he'll even say like I I never intended airflow to be like like a computation engine.
• 35:25
  SPEAKER_01: I've always intended it to be an orchestration engine but that's not to say like that's not to say it's definitely not like a possibility to kind of use these things like actually never mind sweet Peter Barrage do you guys have anything else?
• 35:51
  SPEAKER_05: I think you have anything Brage no I think uh that was great.
• 35:55
  SPEAKER_03: Yeah this was an awesome conversation thanks for coming on yeah you have anything else you want to plug or talk about uh so as far as plugging uh once again just want to say please come to Pibay uh myself and Seth Edwards are going to be giving a pretty awesome talk with some demos and discussion of like how to run the Kubernetes executor in production.
• 36:11
  SPEAKER_03: When and where is that so let's see Pi Bay 2018 Pi Bay 2018 is going to be um August 16th through 17th in San Francisco California uh maybe we'll make the trip out for it yeah that'd be fun please please do uh also uh Bloomberg is always hiring we we got off we have a beautiful office in a skyscraper in San Francisco so you can see the entire bay we also have a lot of positions open in New York and uh it's a really awesome company that gave me this amazing opportunity to like hopefully make a pretty cool impact in an open source project.
• 36:56
  SPEAKER_05: Thanks so much to Daniel for coming on it was truly a pleasure talking with him and we learned a ton from our conversation up next we have Greg Nyheisel astronomer CTO Greg plans on picking up some of the work on the Coob's Executor going forward and has some really cool ideas with respect to the future of the project before we dive in just a quick disclaimer just by nature of being the astronomer CTO, Greg does talk a bit about our product in this conversation.
• 37:22
  SPEAKER_05: We really want this podcast to be a community resource and not an astronomer sales pitch so we did our best to avoid a tone that's too salesy.
• 37:29
  SPEAKER_05: Regardless hope you can enjoy all right so to talk a little bit more about why Airflow and Kubernetes are a really good fit we're gonna have a quick conversation with our CTO and astronomer Greg Nijheisel.
• 37:44
  SPEAKER_05: Viraj is just going to ask him some questions so can we just do some quick introductions so everybody gets their voices kind of recognized here.
• 37:50
  SPEAKER_04: Yeah um I'm Viraj data engineer here at Astronomer.
• 37:54
  SPEAKER_04: And I'm Greg, CTO here at Astronomer.
• 37:57
  SPEAKER_04: How are you doing today Greg?
• 37:59
  SPEAKER_04: Doing just fine cool um so before we jump into like why Airflow on Coops you want to talk a little bit about what we were doing before with the Airflow and Mesos setup just like a high level view on that.
• 38:11
  SPEAKER_00: Yeah sure so um pretty much I mean it's it's the same kind of objective with the two I mean really on Mesos um we just operated a large Mesos cluster um and you can uh deploy many or you know one or many uh Airflow clusters to that Mesos uh cluster each scheduler would register as a Mesos framework um upon starting up and then pretty much you know as as the scheduler had tasks to do and they came in through the uh the Airflow queue the internal queue they would just get matched up with Mesos resource software and then delegate and then launched as containers.
• 38:55
  SPEAKER_00: So the Airflow run command uh would actually just get executed and inside a container same the same airflow container that runs the scheduler on the web server um and the task will you know run to completion and the pod or the um the container would die and you know life goes on.
• 39:12
  SPEAKER_00: But um you know it wasn't it we it wasn't really fully built I would say so I mean maybe that's what you're getting at but um it didn't handle a lot of things it didn't handle uh failures very well it didn't handle um some things in Meso's land you know around like reconciliation um like if frameworks go offline and come back up and tasks we're still executing like what happens um those things are kind of um not really the answer to the Mazo's schedulers so those are things we're looking to improve on um this time around.
• 39:42
  SPEAKER_04: Yeah so uh kind of going off that like how does the whole how does Kubernetes help with problems right like how is how are Kubernetes and airflow a very natural fit?
• 39:52
  SPEAKER_00: Yeah um I mean very much the same way I mean the thing that we're trying to do is just you know as opposed like if we put it in the context of like the celery executor um say you know the celery executor is just you know it's uh it's like it's a standard queue um where you have workers out there a pool of workers and they pull tasks off a cube like a Redis or a rabbit and then they they run the they run they do a subprocess so they fork off and they run um you know up to X number of concurrent tasks and then as they finish you know the the worker the main process kind of pulls more tasks off but really you know you're just kind of adding a different layer there's like another layer of um complexity in there and the fact that um you know one you're over provisioned so that you know a very you know Airflow is a very batch oriented um system so you know jobs come and go in waves typically so midnight is a is a pretty typical job a pretty typical time for uh tasks to launch or like every hour so you know in between those nights those nightly runs or those hourly runs you know if you have cellary workers running they're pretty much just sitting there idle taking up resources on your cluster and the fact that you actually have like another process in the way.
• 41:07
  SPEAKER_00: So you have a you know a worker process so like in the world of Airflow you actually have the airflow worker process which forks and has a cellary worker process and that cellary worker process has children processes that are actually executing the tasks.
• 41:22
  SPEAKER_00: Well that's a that's a couple layers more than you actually need.
• 41:25
  SPEAKER_00: So if you can just tell the scheduler can just delegate you know airflow runs just like it does locally but just delegate those out to the cluster just run a prop run this process on the cluster in a sandbox that's a little that's a little simpler.
• 41:39
  SPEAKER_00: Like um it sounds a little different because celery is pretty much easy to work with you know but you know in a in a in the Kubernetes world just seems like you know let's just let's just run one process out on the cluster, let it finish, free those resources back up without hogging them for idle workers.
• 41:57
  SPEAKER_00: Yeah.
• 41:58
  SPEAKER_04: So it sounds like the kubed executor just like takes care of idle takes care of idle resources but also just removes an entire layer of complexity because you get a Redis and your cellular workers as well then right?
• 42:08
  SPEAKER_00: Right exactly so I mean it's it's a I think it's a more it's a simple it's a simpler metal model.
• 42:13
  SPEAKER_00: It's really just an extension of like the local executor just instead of running them just instead of you know running the shelling out the airflow run command on the look on the same box as the scheduler we're just running those things in pods without without the Redis or the rabbit middle layer and then the cell room worker queue.
• 42:30
  SPEAKER_00: Yeah Kubernetes just kind of handled that for us.
• 42:33
  SPEAKER_00: We can just kick jobs over to it.
• 42:36
  SPEAKER_04: And it just kind of works yeah so kind of uh just stepping back from the kube's executed itself a little bit and more on airflow and Kubernetes uh like we love the questions around here in like uh hey how do how do I make my high flow my airflow like highly available you know like how do I make it scale up pretty easily how do I make it so that if something dies like I don't lose anything.
• 42:54
  SPEAKER_04: Do you want to get a little bit on how Kubernetes solve those problems just like natively in Kubernetes?
• 43:00
  SPEAKER_00: Yeah I mean I think there's a couple things you know there's some of those are you know native Kubernetes constructs and there's a couple things that we get uh from the way that the Kubernetes executor is actually written um it actually works pretty great but um as far as keeping things highly available um delivering a high availability system on Kubernetes um you know there's like there's a couple components that you want to keep highly available uh there's the web server and the scheduler right uh everything other than that is just pods executing tasks so two main components that you want to keep alive um and keep available the web server is a pretty simple component because it's just an HTTP server it's like the traditional you know the first use case for you know um you know especially on Kubernetes you know before staple sets and things were around the simplest use case is just like launching a web app and um you know putting a load balancer in front of those things and uh you know um proxying requests back to you know any number of instances so first thing We do is just we just we can just launch a web server with multiple replicas and put it behind um you know an ingress on Kubernetes, so like Nginx.
• 44:08
  SPEAKER_00: So in our case, we use the Nginx ingress controller, so um as requests come into Nginx, um they can just be forwarded to any number of replicas of the web server.
• 44:18
  SPEAKER_00: So if you want to run a nice, highly available system, throw three back there.
• 44:22
  SPEAKER_00: If one goes down, there's always two more sitting there accepting requests.
• 44:26
  SPEAKER_00: Kubernetes will take care of any failed pods and just start up, start a new one.
• 44:29
  SPEAKER_00: So you always have X number of pods running, um, and then Kubernetes takes care of restarting failed pods.
• 44:36
  SPEAKER_00: Um so that that kind of checks the boxes for the web server.
• 44:41
  SPEAKER_00: Uh pretty simple, pretty standard use case.
• 44:44
  SPEAKER_00: Scheduler is a little different, um, just because the the uh the the way that just by nature of how the Airflow scheduler works, uh it's only meant for one instance to be running at a time, so there is no, you know, there is no idea of running like three, you know, like three schedulers with two on a failover.
• 45:03
  SPEAKER_00: If one fails, um, you know, kick another one on.
• 45:06
  SPEAKER_00: Um there's that clairvoyant, the package from those uh the clairvoyant folks that kind of does some of this, but I think we get we get a um, you know, Kubernetes gets us these things for free out of the box.
• 45:18
  SPEAKER_00: So a couple things that we're doing, um, I think is uh we you know if you want to the most the most critical point of of Airflow is assuming is trying to get your tasks to run.
• 45:31
  SPEAKER_00: You know, you want your tasks to run when they're scheduled, um, and you don't want them to fail, obviously.
• 45:38
  SPEAKER_00: So um what we do is we put a a Kubernetes uh resource type called a um a pod disruption budget on it.
• 45:47
  SPEAKER_00: So what that does is keeps the scheduler alive in the event of um kind of like unplanned uh issues in Kubernetes.
• 45:57
  SPEAKER_00: So you know, um say a node goes down, say you have a three-node cluster, a node goes down, um just loses network connectivity or something, you know, it just gets blown off the face of the Earth.
• 46:07
  SPEAKER_00: Um well Kubernetes will realize that that pod or that that node is now gone, and any pods that were scheduled on that uh node need to be rescheduled to the remaining two nodes.
• 46:17
  SPEAKER_00: Um like in the event that the the cluster actually has um was actually pretty fully utilized across those three nodes, there might not be room on those two nodes to fit you know all the pods from that third node.
• 46:30
  SPEAKER_00: So it will be Kubernetes will be forced to evict pods.
• 46:34
  SPEAKER_00: So some things will have to go.
• 46:36
  SPEAKER_00: Um and it kind of uses these pod disruption budgets um to kind of decide you know rules for like who stays and who goes.
• 46:44
  SPEAKER_00: So we put one on the on the scheduler just to say, hey, we want one scheduler running, and only one, um, at any given time, um, but we want it to stay up in the event of some unforeseen disaster, like a node going down.
• 46:57
  SPEAKER_00: So um we we get some reasonable assurances that that pod will stay alive.
• 47:02
  SPEAKER_00: Um so it's like a best effort, you know, it's not like foolproof.
• 47:05
  SPEAKER_00: Um, you know, obviously you can lose all your nodes and that doesn't matter anymore.
• 47:10
  SPEAKER_00: But yeah.
• 47:11
  SPEAKER_00: Um so we you know that's what that's what we can do there.
• 47:14
  SPEAKER_00: Um one other thing, and we're actually we're kind of looking at this right now, um, is putting a better health check on the scheduler.
• 47:22
  SPEAKER_00: So um so in Kubernetes and most um systems like this, you know, if if if a pod fails a health check, you know, x amount of times over some interval, you know, Kubernetes will say, okay, this thing is clearly not healthy.
• 47:37
  SPEAKER_00: Let's try moving it or let's try rebooting it, uh, maybe put it somewhere else, put it on a different node, see what happens.
• 47:42
  SPEAKER_00: Um, you know, it can kind of end up anywhere.
• 47:45
  SPEAKER_00: But um a better health check.
• 47:48
  SPEAKER_00: Uh right now we don't, I'm not sure that we actually have a health check other than doing like a PS, like executing a PS command and saying, hey, yeah, the scheduler process is running, um, which is that doesn't really get us anything because if the scheduler process died, since it is PID1 within the container, the container would die, and then Kubernetes would restart it anyway.
• 48:08
  SPEAKER_00: So it's not, it's kind of like a a dummy, it's kind of a dummy health check.
• 48:13
  SPEAKER_00: So we're looking into a better one.
• 48:15
  SPEAKER_00: Um I think what we're what we're kind of we were talking with Max Maxine the other day, and I think um what we may end up doing is writing a script, either a bash script or a Python script that actually kind of maybe actually uses the Airflow library and just pings through to the database, to the to the Airflow metadatabase and actually checks um checks the last time that the scheduler heartbeated.
• 48:38
  SPEAKER_00: So the scheduler actually keeps track of its last heartbeat.
• 48:41
  SPEAKER_00: Um and it does that at like a critical point in its like scheduling loop.
• 48:47
  SPEAKER_00: So if we can determine that, yeah, that that last heartbeat happened within like some some number of time or some some interval of time, we can reasonably assume that the scheduler is healthy.
• 48:59
  SPEAKER_00: If it slides out of that window, we can assume that you know the scheduler kind of died or it got hung up.
• 49:04
  SPEAKER_00: We've seen some cases in the um in the past where the actual scheduler, main process, is running, but one of its forked processes can actually get hung up and not terminate and then kind of hold the loop up, so it pretty much just blocks the whole loop.
• 49:19
  SPEAKER_00: And that's kind of the situation where we want to protect against.
• 49:21
  SPEAKER_00: Um, is this kind of things where unexpected things um kind of within the loop itself kind of hold things could possibly hold things up.
• 49:30
  SPEAKER_00: So we just want to check for that.
• 49:31
  SPEAKER_00: Um and if if we do detect a failure or some downtime, just restart it, let it try again.
• 49:38
  SPEAKER_00: Um so those are kind of like the two things to kind of keep the scheduler up.
• 49:43
  SPEAKER_00: Um the last thing that we do, um we actually get this out of the just the way that the Kubernetes executor for Airflow was written.
• 49:51
  SPEAKER_00: Um so when it when it boots up, when the when the when the uh the executor boots up for the first time, it's gonna connect to something called the Kubernetes uh watch API.
• 50:03
  SPEAKER_00: So the watch API is pretty much an event string.
• 50:05
  SPEAKER_00: So once it connects, you know, Kubernetes will start forwarding events as things happen within the uh within the cluster.
• 50:13
  SPEAKER_00: So what we subscribe to is what the executor actually subscribes to is like it watches for events in the pods.
• 50:20
  SPEAKER_00: So when a pod finishes, it'll get either an event that says, yep, the pod ran and it failed, or the pod ran and it succeeded.
• 50:28
  SPEAKER_00: Um which actually is a reflection of how the actual task ran.
• 50:31
  SPEAKER_00: So if the Airflow task fails, the container fails.
• 50:35
  SPEAKER_00: Um and that's pretty much how it's able to launch pods, just kind of fire pods off when it's time to run tasks, and then it just subscribes to the event API or the event uh the watch API to get events back to say, hey, that pod you launched, that actually uh succeeded.
• 50:50
  SPEAKER_00: So uh go ahead and tell Airflow, you know, mark it off in the database that you know this task succeeded.
• 50:55
  SPEAKER_00: If the container fails, marks it as a failed task and you know, we'll probably retry it again, depending on the retry policies, to find an Airflow.
• 51:02
  SPEAKER_00: Um sounds like you get a like free free graceful shutdowns for that, right?
• 51:07
  SPEAKER_00: Yeah, yeah, that's actually a big one.
• 51:09
  SPEAKER_00: Um so right now, you know, with the Celery executor, we have the issue of you know, we're running immutable Docker containers on our cluster.
• 51:18
  SPEAKER_00: So whenever you want to push new code, that's really a new Docker tag in the system for us.
• 51:24
  SPEAKER_00: And uh we actually go through and we reboot your your web server, your scheduler, um, and your act and your salary workers with uh with the new image tag.
• 51:34
  SPEAKER_00: So with the celery workers, like the best thing we can do is define a graceful shutdown period.
• 51:39
  SPEAKER_00: So when Kubernetes you know sends a sig term to PID1 in the worker, Celery will take care of it itself and say, you know, stop, I'm gonna stop accepting tasks off the queue, and I'm gonna let my my currently running tasks drain and then I'll restart.
• 51:56
  SPEAKER_00: You can define a hard limit on how long you wait before Kubernetes doesn't forceful shutdown.
• 52:01
  SPEAKER_00: So it's kind of not a perfect solution because you can just set that to something very high, and then a task could get locked up and then run forever, and the thing would never restart.
• 52:12
  SPEAKER_00: Um, or you're gonna hit the limit and you're gonna forcefully reboot in the middle of a long-running task.
• 52:20
  SPEAKER_00: You know, neither of those situations is uh great.
• 52:23
  SPEAKER_00: So the Kubernetes executor actually gets us around that entirely by um you know at when the scheduler's up and running, it's just gonna fire tasks as it comes.
• 52:34
  SPEAKER_00: Um when we do a deployment, say there's a new tag available, and we want to restart the web server and the scheduler.
• 52:40
  SPEAKER_00: Um we don't we have no workers to restart, uh, so but but there could be tasks running.
• 52:44
  SPEAKER_00: So there could be pods out there running uh tasks that are long running that could be lasting you know an hour or so.
• 52:50
  SPEAKER_00: Um we don't wait, we don't have to wait or anything.
• 52:52
  SPEAKER_00: We don't have to define a grace period for anything to restart.
• 52:55
  SPEAKER_00: We can just go ahead and boom, immediately turn over the web server and the scheduler to the new tag.
• 53:00
  SPEAKER_00: Um Kubernetes will keep the web server up, it'll you know bring up a new deployment or a new replica set and kind of start moving pods from the old replica set to the new one, so you experience you know, no from the outside perspective, you see no downtime from the web server.
• 53:14
  SPEAKER_04: Yeah.
• 53:14
  SPEAKER_00: And the scheduler gets rebooted in you know under 10 seconds, probably eight seconds.
• 53:20
  SPEAKER_04: So for the record, this is exactly why Greg is CTO.
• 53:23
  SPEAKER_04: Pete and I pulled him aside five minutes ago, and he's pre-blogged this off the top of his hand.
• 53:28
  SPEAKER_00: Been in the weeds too long.
• 53:31
  SPEAKER_00: But the good yeah, the good thing about that is that um pods that were running and executing, you know, they just continue to run.
• 53:38
  SPEAKER_00: Um they run to either a failure or uh you know a succeed, and then um, as the new scheduler booted up, it's gonna resubscribe to that event API where it left off, so it actually keeps track um of where the last time it checkpointed in the event stream, so it can resubscribe exactly where it left off.
• 53:56
  SPEAKER_00: And then any events that happened while the scheduler was down, um you know, the scheduler gets notified of those and they can go ahead and say, hey, like you know, these five tasks actually completed while the scheduler was rebooting in that in that eight-second time period.
• 54:10
  SPEAKER_00: Um go ahead and like uh clear that with the with the Airflow metadata base and mark those tasks as succeeded or failed.
• 54:17
  SPEAKER_00: Yeah.
• 54:17
  SPEAKER_00: Um and it just continues as normal from there.
• 54:20
  SPEAKER_00: So it's pretty cool stuff.
• 54:22
  SPEAKER_06: Yeah.
• 54:23
  SPEAKER_00: Pretty sound work to be done.
• 54:27
  SPEAKER_05: So yeah.
• 54:28
  SPEAKER_05: Oh well, Greg, thanks for linking up with us in chat.
• 54:30
  SPEAKER_05: And the patterns come together.
• 54:32
  SPEAKER_05: Yeah, last minute pulling in.
• 54:34
  SPEAKER_05: Greg was actually in Singapore last week, and so is Virage.
• 54:37
  SPEAKER_05: I mean, these guys are exhausted on a 12-hour time difference right now, and I'm kind of just forcing them forcing them to come in and do this podcast interview.
• 54:45
  SPEAKER_05: So thanks to these guys for for helping out with this.
• 54:57
  SPEAKER_05: We hope you enjoyed these conversations as much as we did.
• 55:01
  SPEAKER_05: Thanks for tuning in, and as always, feel free to reach out to me at p at astronomer.io with any podcast feedback, or if you'd like to be considerate for an interview.