Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Best Practices

Transcript

Best Practices · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: Hi there, and welcome to episode three of the Airflow podcast produced by Astronomer.
• 0:05
  SPEAKER_00: Over the past six months, we've taken a complete deep dive into the data engineering space and have heard from some extremely interesting people about their use of Apache Airflow.
• 0:14
  SPEAKER_00: We're extremely excited to keep the ball rolling on our mission to solidify airflow as the gold standard for ETL.
• 0:20
  SPEAKER_00: If you didn't catch the last episode, my name is Pete DeJoy, and I'm a product special steered astronomer.
• 0:25
  SPEAKER_00: I conducted these interviews with Viraj Perek, one of astronomers' data engineers.
• 0:29
  SPEAKER_00: While you'll likely hear from me occasionally, you'll probably hear more of Viraj's voice acting interview questions, as he's the real Airflow expert.
• 0:37
  SPEAKER_00: For reference, he said something like this.
• 0:40
  SPEAKER_03: Hi, I'm Viraj.
• 0:42
  SPEAKER_03: I like The New York Next, Long Walk on the Beach, and talking about data infrastructure.
• 0:47
  SPEAKER_00: This week, we'll be discussing Airflow best practices.
• 0:50
  SPEAKER_00: You'll hear from pieces of our conversations with Alan Crooke, President Insight and Data Manager at Tails.com, Chris Ricomini, principal software engineer at WiiPay, and Boca De Bruin, CTO of Advanced Analytics Technology at IMG.
• 1:03
  SPEAKER_00: Just a quick note before we dive in, we received some feedback on last week's episode that was kind of hard to follow with all the fragments of conversations and relatively abrupt transitions.
• 1:12
  SPEAKER_00: We hugely appreciate the feedback, and I'm really going to try to include longer portions of interviews going forward to improve the fluidity of the episodes.
• 1:18
  SPEAKER_00: As I might have mentioned earlier, we're podcasters by hobby and data nerds by trade.
• 1:22
  SPEAKER_00: So thanks for bearing with us as we work to make this podcast great.
• 1:25
  SPEAKER_00: Without further ado, let's dive into episode three of the Airflow podcast, Best Practices.
• 1:30
  SPEAKER_00: We'll begin with a quick rundown of how Alan Krukshnak approaches Airflow DevOps and best practices at Tails.com.
• 1:35
  SPEAKER_00: Hope you enjoy.
• 1:54
  SPEAKER_05: So I guess let's tackle it in two hards.
• 1:59
  SPEAKER_05: So there's the the DevOps side of Airflow itself as a core application.
• 2:04
  SPEAKER_05: So we run our Airflow deployment entirely Dockerized.
• 2:10
  SPEAKER_05: So it's all Docker containers.
• 2:11
  SPEAKER_05: And then we have so we have a separate container for the scheduler, we have a separate container for the web app, we have a separate container for each of the workers.
• 2:18
  SPEAKER_05: And each of those containers is running on Amazon ECS, so Elastic Container Servers.
• 2:23
  SPEAKER_05: So I think with them rolling out tools like Fargate, so which I guess is any kind of managed container running service where you don't have to worry about the servers, I think we'll probably take advantage of that.
• 2:38
  SPEAKER_05: We run all of our infrastructure in Europe.
• 2:40
  SPEAKER_05: I mean we're we're a UK-based company and Fargate isn't available yet.
• 2:45
  SPEAKER_05: But for now, we run it on ECS.
• 2:47
  SPEAKER_05: And I think there beyond, I guess, basic uh continuous deployment practices.
• 2:56
  SPEAKER_05: So you know, we commit to a Git repo and that will automatically push out new versions of each of the containers.
• 3:01
  SPEAKER_05: I don't think we're doing anything particularly exciting there.
• 3:05
  SPEAKER_05: Um the DevOps side around um actually writing the operators and um writing pipelines, I actually were were super light, almost to the point.
• 3:21
  SPEAKER_05: I mean, we're given we're a relatively small team, you kind of do your own DevOps.
• 3:26
  SPEAKER_05: Um if you if you're gonna be in a particular area of the code base often, make it better for the next person that's around there and do do things that are that are nice there.
• 3:34
  SPEAKER_05: I think there's a there's a space in the future for us to have dedicated DevOps people, but at a at the scale of a data team where we have you know five people, we don't we don't have a DevOps guy.
• 3:44
  SPEAKER_05: Everyone's a DevOps guy.
• 3:46
  SPEAKER_03: Yeah.
• 3:47
  SPEAKER_03: I can I hear that.
• 3:48
  SPEAKER_03: Yeah, cool.
• 3:50
  SPEAKER_03: Um so kind of uh going along that same line, um, do you have any like internal set of best practices that you use or kind of standard conventions that you've you've implemented through your team just for like documentation and standardization's sake?
• 4:03
  SPEAKER_05: Um yeah, and I I think a lot of it comes down to uh I guess there are two there are two areas that I found super helpful.
• 4:12
  SPEAKER_05: One is around naming conventions.
• 4:14
  SPEAKER_05: Um I think Airflow is is really flexible, but you it um it gives you lots of different places that you can give things names.
• 4:22
  SPEAKER_05: Um so there's there's the DAG object, there is the name of the DAG, the name property of the DAG object, there is the file name of the file name of the file that has the DAG in it, and then you have that all for the task as well.
• 4:37
  SPEAKER_05: So each task instant has an object which has a name, it also has a name property, uh, and it may also have various dependent file file structures.
• 4:47
  SPEAKER_05: Um so I guess we've we've tried to adopt a pretty strict naming convention where the name the file name that the DAG lives in is the same as the DAG object, is the same as the name of the DAG.
• 4:59
  SPEAKER_05: Um, that task names are the same as the task objects, like I guess, label.
• 5:05
  SPEAKER_05: Um, and that where they have external template files, so whether they're HTML files or SQL files, they either live in a folder that is the same name as the DAG name, or that they are at least prefixed with the name of the DAG.
• 5:21
  SPEAKER_05: So that it's really obvious which templates apply to which DAG, which DAG has which tasks in it, um and and and so on and so forth.
• 5:30
  SPEAKER_05: I think the other bit of I guess general practice, and I think this is more um uh software best practice, is is just as far as possible, try not to repeat yourself.
• 5:41
  SPEAKER_05: If you find yourself writing the same thing over and over again, uh you know, build it into the operator um or or write some function to do that rather than just copying and pasting the same code everywhere.
• 5:53
  SPEAKER_05: I think what when when we first started using Airflow, because um because it's almost so easy to set up new tasks by just copying and pasting the previous one, I think until it becomes a pain point, you actually realize that you're unwittingly writing a lot of duplicated code.
• 6:14
  SPEAKER_05: So for example, you have a DAG that's got five tasks in it, likelihood is the owner property of all those tasks is the same, the DAG property of all of those tasks is the same.
• 6:23
  SPEAKER_05: The uh there's probably a whole heap of other properties of each of those tasks that are the same.
• 6:28
  SPEAKER_05: But because it's so easy to just copy and paste that code between them all, you just copy and paste it all rather than rather than either writing some general properties or um abstracting a little more.
• 6:38
  SPEAKER_05: And I think as you know our Airflow deployments got bigger, and I think we I guess for scale, we've probably got around 30 DAGs running um regularly, um, and they've got uh schedules anywhere from every 15 minutes to every week.
• 6:59
  SPEAKER_05: Um I think as we've got to that scale, and it starts to be at the point where you forget the code you wrote you know a year ago, and that's become more important for us and just standardization.
• 7:09
  SPEAKER_05: But I think that's that's general, that's good software development practice more generally rather thanks so much to Alan for coming on.
• 7:18
  SPEAKER_00: We had an awesome chat with him.
• 7:19
  SPEAKER_00: He's an incredibly smart dude, and I'm sure we'll be releasing the rest of our interview at some point soon because it was a really, really great one.
• 7:26
  SPEAKER_00: Up next, we have a more extended cut of the Chris Riccomini interview that you heard a little portion of last week.
• 7:32
  SPEAKER_00: He'll be discussing how he defines Airflow, some of the design principles that he uses when pushing up new releases, and some of his personal best practices.
• 7:40
  SPEAKER_00: It's a really great chat and really excited for you guys to hear it.
• 7:46
  SPEAKER_03: Cool.
• 7:47
  SPEAKER_03: Um, so I think another cool part about this is uh we tend to ask all our guests the same question and see how they respond differently to it.
• 7:55
  SPEAKER_03: And the question is uh define airflow for me.
• 7:58
  SPEAKER_03: So if I was just a random person asked, like, hey, what's Airflow do?
• 8:02
  SPEAKER_03: Uh, how would you answer that?
• 8:04
  SPEAKER_02: Okay.
• 8:04
  SPEAKER_02: Uh I'm thinking.
• 8:15
  SPEAKER_02: My inclination is to say it's a it's a job scheduler, but I think that gets pretty mixed in with like yarn and grid schedulers and stuff.
• 8:21
  SPEAKER_02: So I would say um it's a workflow executor that orchestrates um and executes uh tasks, where tasks can be basically any logic you want, from you know random programmatic logic to interacting with databases to interacting with web services.
• 8:41
  SPEAKER_02: Um and it's its primary goal is really to um you know measure and maintain the life cycle of these tasks to deploy to from deployment all the way through to you know either failure or completion, um, to allocate them on specific machines, um and to I guess provide the the necessary uh metrics and monitoring for it.
• 9:03
  SPEAKER_02: So how is the task performing?
• 9:05
  SPEAKER_02: Is it failed?
• 9:05
  SPEAKER_02: How many times is it failed?
• 9:06
  SPEAKER_02: How long is it taking?
• 9:07
  SPEAKER_02: What do the logs say?
• 9:09
  SPEAKER_02: That kind of stuff.
• 9:10
  SPEAKER_02: So it definitely has overlap with like other grid schedulers.
• 9:12
  SPEAKER_02: If you look at something like, again, Yarn, where you know it's it's responsible for putting tasks on machines and you know, ostensibly letting you see um the logs for the tasks that are running and uh whether they're you know failed or still running.
• 9:28
  SPEAKER_02: Um where it differs is it's very ETL and scheduler driven, or I guess time driven, um in that it uh it's meant for like recurring tasks.
• 9:40
  SPEAKER_02: So, for example, if you want to suck data out of a database and load it into uh a data warehouse, you might want to do that every hour.
• 9:46
  SPEAKER_02: Um so you do this on a recurring schedule.
• 9:48
  SPEAKER_02: This is something that most generic grid schedulers don't have.
• 9:53
  SPEAKER_03: Yeah.
• 9:54
  SPEAKER_03: Um that was a really good answer, really in-depth.
• 9:56
  SPEAKER_03: Um we had one of the person say cron on crack.
• 9:59
  SPEAKER_03: So yeah, that that is a uh a snappier, more marketable, I guess.
• 10:06
  SPEAKER_03: Yeah, though I'm not sure our marketing team would love us saying cron on crack on a port paper or something.
• 10:12
  SPEAKER_03: But representing the wrong way.
• 10:14
  SPEAKER_03: Yeah, right?
• 10:15
  SPEAKER_03: Cool.
• 10:16
  SPEAKER_03: Um so kind of just going right along those lines.
• 10:18
  SPEAKER_03: Um can you talk to me a little bit about uh you can stay as high level as you want, but some of the specific tasks you have DAGs running for and kind of the technical side and the use case and the business side of those?
• 10:30
  SPEAKER_02: Yeah.
• 10:31
  SPEAKER_02: Um so let me start with the business side first, or maybe the um the use case side rather than the implementations of it.
• 10:40
  SPEAKER_02: Um we have a number of different tasks, but they break or DAGs rather, but they break down into a few buckets.
• 10:47
  SPEAKER_02: One of them is ETL.
• 10:49
  SPEAKER_02: So we need to get data out of our OLTP uh production databases and get it into various systems downstream.
• 10:56
  SPEAKER_02: Um so we we use it for uh for that.
• 10:59
  SPEAKER_02: And as part of as part of that, we um we have a bunch of data quality check DAGs as well that validate that the data in the destination database matches what's in the source database.
• 11:10
  SPEAKER_02: So you don't have missing rows, and you don't have, you know, perhaps you you might miss an update, and so the field value in the source is different than in the destination database.
• 11:19
  SPEAKER_02: Um the so that's one category, sort of ETL.
• 11:23
  SPEAKER_02: The the second category is about report generation and materialization.
• 11:26
  SPEAKER_02: So this is essentially once we have the data in our data warehouse, we want to generate reports based off of the data.
• 11:33
  SPEAKER_02: So this could be everything from like financial reconciliation stuff to uh use it, usage of our product, to um uh even in some cases, like uh not really monitoring, but um looking at logs in our data warehouse and seeing what's going on uh there.
• 11:52
  SPEAKER_02: Um so that's a second use case.
• 11:55
  SPEAKER_02: Um the third use case is really around machine learning.
• 11:59
  SPEAKER_02: So this is uh we want to train models, we want to um uh basically explore the data, do feature uh engineering for the for the models, uh, and so on.
• 12:11
  SPEAKER_02: So our data scientists um interact with Airflow a bit um to do the training.
• 12:16
  SPEAKER_02: And so essentially how it works is they start off, they're fiddling around, maybe they're doing ad hoc queries, eventually they get to a point where uh they train a model and it looks good.
• 12:26
  SPEAKER_02: Um, we need to productize the model so that it um you know it runs always, not just one time and not just on somebody's laptop.
• 12:34
  SPEAKER_02: And so for that, we uh we run the um the DAGs in Airflow.
• 12:40
  SPEAKER_02: And that one in particular, it's really nice to have Airflow as a Python, you know, the DAGs as Python and the tasks as Python, because a lot of our data scientists spend time in Python, so it's pretty natural for them to interact with Airflow.
• 12:52
  SPEAKER_02: Um that's the third uh large bucket.
• 12:56
  SPEAKER_02: Um the fourth bucket is something that we're just getting into, and this is more uh to get back to your your cron on crack mention.
• 13:05
  SPEAKER_02: Um we we were uh heavily uh cron-based a few years ago.
• 13:09
  SPEAKER_02: Um payments by nature, a lot of it is asynchronous processing after the you know credit card swipe happens.
• 13:15
  SPEAKER_02: And so um uh we had a bunch of stuff that was running in cron, and uh you know, you bump up against all the problems that Airflow tends to solve, like retrying and alerting and all that kind of stuff, scheduling and not allowing overlapping jobs.
• 13:30
  SPEAKER_02: Um these are all problems that uh I think anyone who uses cron at quote unquote at scale uh can relate to.
• 13:36
  SPEAKER_02: Um and so we are migrating a lot of our cron jobs out of um cron and into Airflow.
• 13:43
  SPEAKER_02: And the model there is is much more of um uh the task kicks off basically a call to a web service, and the web service then triggers some logic versus where it was before where like cron would would execute um the logic itself.
• 14:01
  SPEAKER_02: Um and I can get into the implementation details about that, but those are the four use cases, right?
• 14:08
  SPEAKER_02: So um cron, machine learning, reporting, and uh ETL.
• 14:14
  SPEAKER_02: Um now you wanted me to go into implementation a little bit, or did you want to uh do something else?
• 14:20
  SPEAKER_03: Oh, let's uh let's get into implementation.
• 14:24
  SPEAKER_02: Yeah, so implementation on the um on the ETL side is pretty, I would say pretty normal.
• 14:31
  SPEAKER_02: Um it's you know, select star from data over the last 15 minutes, do that every 15 minutes, load it into the destination database.
• 14:39
  SPEAKER_02: We've increasingly been moving away from uh Airflow for the core ETL pipeline in favor of uh streaming solutions, but we still use it for some uh things.
• 14:49
  SPEAKER_02: And namely, uh some of our um some of our MySQL databases are still using Airflow, um, where we we take the data from MySQL and load it into uh our data warehouse, um which in our case is Google Cloud BigQuery.
• 15:03
  SPEAKER_02: Um But uh we the second part that I mentioned I think is a little less common is to run these data quality checks, and we do two kinds of checks.
• 15:11
  SPEAKER_02: One is just a row level count.
• 15:13
  SPEAKER_02: So how many rows are in this table, how many rows are in that table?
• 15:16
  SPEAKER_02: If they match, things are good.
• 15:18
  SPEAKER_02: Um, there's some slight nuance to it.
• 15:20
  SPEAKER_02: Um, for example, if you have hard deletes in your tables, it gets a little wonky.
• 15:24
  SPEAKER_02: Um in general, um, that's one data quality check we do that's sort of uh easy and relatively cheap to do.
• 15:34
  SPEAKER_02: The second one is more expensive but more thorough, and that is where we will actually do row by row, column by column checks.
• 15:41
  SPEAKER_02: And we do this less frequently, um sort of on a you know, um maybe a weekly basis, uh, would be uh more the the schedule by which we would do this, is we actually take you know a sliver of a table and compare it to the exact same sliver of the table in the destination database and validate that every single row and column is identical.
• 16:01
  SPEAKER_02: And that that basically lets us sleep at night because it knows it means that we know that as of you know within a week, that we would know if there was any uh bad data in the system at all.
• 16:11
  SPEAKER_02: And and that's important when you're doing things like you know, generating board reports or generating reports that you would want to send your customers, or generating reports that you know, maybe for uh financial purposes, like all that kind of stuff, you really have to be able to trust your data.
• 16:26
  SPEAKER_02: Um so doing the data quality checks is something that gets us there.
• 16:29
  SPEAKER_02: Now, scalability-wise, some of this stuff may not uh be feasible for everybody or for all tables, but um it's definitely worth doing in cases where you can do it.
• 16:39
  SPEAKER_02: Um so that's the ETL implementation part of it.
• 16:43
  SPEAKER_02: The um reporting part uh I guess breaks down into two areas.
• 16:48
  SPEAKER_02: One are these sort of materialized reports that just get made into CSV files, and those get implemented uh emailed rather using like an Airflow email operator, say once a day, and they just get blasted out to whoever wants to consume them.
• 17:04
  SPEAKER_02: This is something that I think um specifically some of our non-engineering teams like because they tend to consume things in like you know, CSV files or Excel spreadsheet kind of thing, and um getting just a report every day with the stuff they need in a in an email attachment um makes them happy.
• 17:20
  SPEAKER_02: Um, the second path in this um use case bucket is that we have our business analyst folks generating um these materialized uh data warehouses for various um teams.
• 17:33
  SPEAKER_02: So, for example, you might want a risk data warehouse, you might want a product data warehouse or a sales data warehouse.
• 17:38
  SPEAKER_02: So they do that.
• 17:38
  SPEAKER_02: So they take all of our um the data that we load into our data warehouse and they you know join, filter, munge uh the stuff into something that's more consumable for the various teams or more relevant to the various teams.
• 17:50
  SPEAKER_02: We we tend to follow kind of a uh extract load transform pattern rather than extract transform load pattern, um, which means that my team, the data infrastructure team, pretty much just takes the data out and loads it into uh the downstream systems.
• 18:03
  SPEAKER_02: It doesn't do a lot of transformation as it's loading.
• 18:07
  SPEAKER_02: Um, so the the um analysts, the business analyst folks tend to do some of the transformation to make the data a little more consumable to non-engineers.
• 18:16
  SPEAKER_02: Um and then some of that stuff gets fed into our visualization tier where we you know can make lovely little line charts and pie charts and so on.
• 18:26
  SPEAKER_02: Um now, machine learning-wise, um, the implementation is um it's basically a series of uh very large and sophisticated DAGs that do um some feature engineering and then uh a bunch of model uh training and then uh some validation as well to make sure that things are performing reasonably well against the holdout set.
• 18:53
  SPEAKER_02: Um and then the last part of that is pushing the data from uh our data environment into the production environment where where it can be used by the modeling system.
• 19:03
  SPEAKER_02: Um for more details on this, like we we do have blog posts.
• 19:06
  SPEAKER_02: Basically, everything I'm talking about, we have blog posts on.
• 19:08
  SPEAKER_02: So I'm I'm gonna kind of leave it at a relatively high level, but we have like pretty detailed blog posts about um how that stuff works.
• 19:14
  SPEAKER_02: But the general development flow is, as I mentioned before, uh data scientists um playing around, maybe playing with a new library or uh trying to add a new feature to an existing model, uh, trying a new model, um, initially starts locally on their laptop.
• 19:32
  SPEAKER_02: Um and one of the things that we do that that I think is really pretty cool is um with with Google Cloud Airflow integration, you can actually spin up Airflow locally on your laptop and run it locally.
• 19:45
  SPEAKER_02: Um, but you can authenticate uh as yourself, which means uh in Google Cloud, which means you have uh Google Cloud access to uh the data that you need to do the model training and so on.
• 19:57
  SPEAKER_02: Um so they do a bit of that.
• 19:59
  SPEAKER_02: Um sorry, I'm rambling.
• 20:03
  SPEAKER_04: No, you're fine, you're fine.
• 20:04
  SPEAKER_02: That's exactly what we want it.
• 20:06
  SPEAKER_02: Uh where was I going with this?
• 20:09
  SPEAKER_02: Um I'm gonna validate here.
• 20:14
  SPEAKER_02: I was talking about the the development lifecycle for for model training and stuff.
• 20:18
  SPEAKER_02: Um anyway, so they they do some uh training either locally or uh in a Google Cloud project on some higher-end machines, maybe with GPUs.
• 20:29
  SPEAKER_02: Um eventually they um they take the stuff that they've written and uh make it into an Airflow DAG, and that's what actually gets run by us on the Airflow uh systems that we have.
• 20:41
  SPEAKER_02: Um the last one was cron, which I mentioned, and this one is a little bit um I'd say not experimental, but uh it's our our newest use case.
• 20:53
  SPEAKER_02: Um I think you basically have two options when it comes to cron replacement.
• 20:57
  SPEAKER_02: One is to take the logic that cron was triggering uh locally and run it locally in an airflow um task or series of tasks, right?
• 21:07
  SPEAKER_02: So this would be, for example, if you had a um if you had a cron script that was say querying a database and then uh updating a bunch of rows based on some custom logic, um you could you could have a task that was that would do that um in Airflow.
• 21:28
  SPEAKER_02: And the disadvantage to that approach is that you're basically doing all the logic locally on the task instance that is uh running the logic, um, which is sort of a bit of an Airflow anti pattern.
• 21:44
  SPEAKER_02: You want in general to have as much um logic being executed outside of the Airflow worker itself.
• 21:49
  SPEAKER_02: Um so, for example, if you were querying a database uh, or rather, if you were doing some logic, you would you would want another machine to do the logic.
• 21:57
  SPEAKER_02: Um so the way we implemented it was not to just Directly port the scripts that we had in cron and copy them in as tasks in Airflow, but actually to have Airflow kind of be the scheduler, but be the um have this have a web service or series of web services do the execution.
• 22:13
  SPEAKER_02: So in our model, what that means is the Airflow uh DAGs kick off and the tasks uh essentially issue a curl request or an HTTP request to a web service, um, and that web service is responsible for executing uh the logic that needs to get executed.
• 22:30
  SPEAKER_02: Um, the trade-off here is like if it if it takes a long period of time for the web service to do the work, you end up having to manage the connection between Airflow and the web service over a long period of time.
• 22:40
  SPEAKER_02: If there's network blips, you have to reconnect.
• 22:42
  SPEAKER_02: So you end up having to build kind of like job execution state into the web service.
• 22:47
  SPEAKER_02: Um, so that's the the downside.
• 22:48
  SPEAKER_02: But the upside is that the tasks in Airflow become very dumb and and therefore very uh very easy to operate because they're not using any CPU, they're not using any disk, they're not um you know using any memory, they're just kind of sitting there triggering the execution of the logic.
• 23:03
  SPEAKER_03: Yeah, it sounds like keep that from pretty item potent too, which sounds nice.
• 23:08
  SPEAKER_03: Sounds nice.
• 23:08
  SPEAKER_02: Yeah, yeah.
• 23:09
  SPEAKER_02: That again, that's sort of like uh Airflow golden rule.
• 23:13
  SPEAKER_02: You should be able to run the tasks in DAGs over and over again without breaking things.
• 23:17
  SPEAKER_03: Yeah, yeah.
• 23:18
  SPEAKER_03: Um so kind of going right along that golden rule thing.
• 23:21
  SPEAKER_03: Um, do you have a list of like internal best practices you try to push when you're developing DAGs or writing workflows or anything?
• 23:28
  SPEAKER_02: Yes.
• 23:29
  SPEAKER_02: Um we have uh off the top of my head, I don't remember them all.
• 23:33
  SPEAKER_02: But initially, when we started running Airflow, I was very um uh guarded about what we would allow people to do because we were running a multi-tenant airflow, basically many teams, many different DAGs, all in one machine.
• 23:49
  SPEAKER_02: Um and on top of that, we weren't using the salary executor, which allows you to have some level of isolation, at least at the machine level.
• 23:56
  SPEAKER_02: Um and so what that would that means is one poorly behaving DAG would uh could could potentially you know bring down the whole machine.
• 24:05
  SPEAKER_02: Now, even if we were using something like salary executor, uh at the time that we were first spinning it up, there were still enough bugs in the scheduler itself to where you you could bring down the scheduler with a poorly performing DAG, even if you were using Celery Executor.
• 24:19
  SPEAKER_02: So uh we or we I I banned a number of features.
• 24:24
  SPEAKER_02: For example, subdags was a big one.
• 24:26
  SPEAKER_02: I was just like, we're not supporting subdags.
• 24:28
  SPEAKER_02: This this thing is uh not really working the way uh we we want it to right now.
• 24:32
  SPEAKER_03: Yeah, we did the same thing.
• 24:34
  SPEAKER_03: Subdags are just kind of the bane of everyone's existence around here.
• 24:38
  SPEAKER_02: Yeah, and I think I I'm not I think there's like some faction of people that use them and like them, and some faction that that really don't.
• 24:44
  SPEAKER_02: Um so I've I definitely fall in the faction that doesn't like them and um we we don't use them much.
• 24:49
  SPEAKER_02: We opt for many smaller DAGs rather than fewer larger DAGs with subdags in them.
• 24:55
  SPEAKER_02: So we have like you know hundreds and hundreds of DAGs, but they tend to be on the small side.
• 24:58
  SPEAKER_02: The machine learning ones tend to be, excuse me, the ones that uh are a little bit larger.
• 25:02
  SPEAKER_02: But um anyway, uh other stuff that we that we banned.
• 25:08
  SPEAKER_02: Starting a schedule.
• 25:10
  SPEAKER_02: Uh oh, sorry, give me a minute here.
• 25:12
  SPEAKER_02: You're fine.
• 25:14
  SPEAKER_02: I just had this meeting schedule it thing schedule itself.
• 25:18
  SPEAKER_02: We've got this conference system and it like started its own meeting.
• 25:20
  SPEAKER_02: Um, okay.
• 25:21
  SPEAKER_02: Um so sub DAGs is one.
• 25:24
  SPEAKER_02: Um we implemented some other rules, uh not really preventing people from doing stuff, but we um we require that every DAG have an owner, that all DAGs have um you know an email address associated with it where uh emails can get sent when failures occur.
• 25:41
  SPEAKER_02: Um and we validate all that stuff via CI script that runs in our CI environment.
• 25:47
  SPEAKER_02: So anytime a DAG gets, or rather a pull request gets sent in, we run this um script and check whether uh all the DAGs that are in the repo are conforming and have an owner and uh at least one email that we know of, that we understand, right?
• 26:04
  SPEAKER_02: So it can't just be some random person, it has to be like a team's email address so that when that person leaves the company, the emails don't just go into the ether anymore.
• 26:11
  SPEAKER_02: Um, so we do a little bit of um uh validation prior to commit.
• 26:18
  SPEAKER_02: The other thing we do prior to commit is just sort of basic flake eight pep validation where you make sure that you know your spaces and colons are all nice and looking good.
• 26:26
  SPEAKER_02: And then lastly, we um we make sure that all DAGs are able to be imported by Airflow.
• 26:35
  SPEAKER_02: So what that means is not necessarily a guarantee that your DAG is going to run, but that at least your your DAG isn't going to fail to import due to a missing requirement or uh some funky typo or anything like that.
• 26:52
  SPEAKER_02: And so it gives us some measure of stability and understanding that at least you're not likely to break anyone else, even if your DAG doesn't really behave as expected.
• 27:02
  SPEAKER_02: It's at least going to be imported by Airflow.
• 27:04
  SPEAKER_02: The DAG parser is gonna be able to parse it and give some you know meaningful information about it.
• 27:10
  SPEAKER_02: So those are kind of the three things we do uh on the continuous integration, uh pre-commit side of things.
• 27:18
  SPEAKER_02: Um other features that we don't really use, we're not heavy users of variables right now.
• 27:25
  SPEAKER_02: Um not for any particular reason, it's just we've gotten by without them.
• 27:31
  SPEAKER_02: Um I'm sure there's other stuff.
• 27:34
  SPEAKER_02: I actually have like a vouch out list.
• 27:36
  SPEAKER_02: If you want to give me a minute, I can pull it up.
• 27:39
  SPEAKER_02: Please do, yeah.
• 27:41
  SPEAKER_03: Let me uh let me pull up the list here.
• 27:45
  SPEAKER_03: My weekly we have a weekly meeting every Friday over uh do's and don'ts for Airflow.
• 27:50
  SPEAKER_03: It's almost like a style guide meeting, and we're just looking for more stuff to throw into that because uh like there's a lot of great documentation on there, but there isn't so much as to like what's the best way to do things or what's the way you should avoid.
• 28:02
  SPEAKER_03: So we're trying to like produce that ourselves.
• 28:04
  SPEAKER_02: I think that uh the it is your happiness with airflow to a large extent is going to depend upon your ability to know what you should and shouldn't run.
• 28:14
  SPEAKER_02: Because you can run Airflow extremely stably with a certain set of features, or you can have a wildly unstable experience with other set of features, right?
• 28:24
  SPEAKER_02: Um so the stuff that we ban uh more or less outright are sub DAGs, uh timeouts using timeouts, DAG timeouts, um, sensors, um, and XCOM uh data larger than 48 KB, uh, because again, all the XCOM stuff is stored in the database.
• 28:44
  SPEAKER_02: The sensor banning is due to uh starvation.
• 28:47
  SPEAKER_02: You can basically have sensors that just sit there and take all the slots in a given queue.
• 28:52
  SPEAKER_02: Um, and they don't have in Airflow the concept.
• 28:54
  SPEAKER_02: Sensor is not a first-class thing, it's just an operator that blocks, and so it's not really implemented properly, uh so you can get starvation there.
• 29:02
  SPEAKER_02: Um, we also don't allow the backfill command uh in Airflow, which is via the CLI you can trigger these backfills, which kind of spin up uh a like scheduler uh in and of itself that's running independent of the main scheduler.
• 29:19
  SPEAKER_02: And there were just a ton of problems with that initially.
• 29:21
  SPEAKER_02: I believe there's still like one minor race condition uh with running multiple schedulers, but it's very small because bulky when fixed a bunch of stuff in the scheduler area.
• 29:30
  SPEAKER_02: I think it was bulky.
• 29:31
  SPEAKER_02: Um, but we we just don't allow it.
• 29:33
  SPEAKER_02: The other thing in general that we don't allow, mostly for security reasons, is we don't is is pretty common in a lot of other uh companies, especially startup type companies, to allow people to like SSH in or run airflow commands locally that interact with the Airflow cluster remotely.
• 29:48
  SPEAKER_02: Uh, we just flat out don't allow that.
• 29:50
  SPEAKER_02: Like you commit your DAGs to a repository, that repository is then deployed to an airflow that you don't have command line access to where it runs, right?
• 30:00
  SPEAKER_02: And so all the CLI stuff, um again, like backfill or trigger dag and that kind of stuff, just is not allowed here.
• 30:09
  SPEAKER_03: Yeah, don't ask me at the time I put a 2015 and 20 instead of 2017 on a backfill.
• 30:15
  SPEAKER_03: Came back like two hours later, like why isn't it stunned yet?
• 30:17
  SPEAKER_02: Oh, wait, it's going back two years instead of uh we handle backfills uh via the start date usually.
• 30:24
  SPEAKER_02: So, for example, if you want to um load a new table using um uh an Airflow DAG, um and that table's first row is in 2015, we will just set the start date in 2015.
• 30:38
  SPEAKER_02: And um we have catch up enabled in our Airflow CFT, which basically means that Airflow will see the start date is 2015 and that no tasks have been or DAGs have been run for it.
• 30:46
  SPEAKER_02: And so it will start going through one by one running them all, um, which effectively does what backfill does, except it does it on the main scheduler.
• 30:54
  SPEAKER_03: Oh, that's super interesting.
• 30:56
  SPEAKER_03: Oh yeah.
• 30:58
  SPEAKER_03: Let me let me make a note of that.
• 31:01
  SPEAKER_02: Yeah, that's how we do all of our uh ETL backfilling, is just set the start date.
• 31:05
  SPEAKER_02: We basically like look at the minimum modify time on uh the row on the table, rather, and then we set that as the start date.
• 31:13
  SPEAKER_02: And because we're actually chunking you know day by day in the task or or hour by hour or whatever it is, it'll start at the point of that first row and it'll just go forward from there.
• 31:27
  SPEAKER_00: As I'm sure many of you know, Chris is a pretty huge contributor to the Airflow community and to the open source project.
• 31:32
  SPEAKER_00: So we're really excited to get a chance to speak with him, and we had we were really happy with how the conversation turned out.
• 31:38
  SPEAKER_00: So up next, we did something similar and we brought in a longer cut of a little snippet that you heard last week from Bulky De Bruin at IMG.
• 31:45
  SPEAKER_00: He's also a huge contributor to the project, and we really thought that we should share a conversation with the community because he had a lot of valuable insights.
• 31:51
  SPEAKER_00: So up next, you'll hear a conversation ranging from discussions of an Airflow book to best practices to some of his personal kind of feature requests or things that he would like to see built out in Airflow going forward.
• 32:04
  SPEAKER_00: So hope you enjoy this one.
• 32:09
  SPEAKER_03: Uh what are some of the core airflow design principles or core airflow features that you really like about it, right?
• 32:14
  SPEAKER_03: Like what are some of the way like the way it handles tasks or something?
• 32:18
  SPEAKER_03: Like what about it makes you really like it and makes you want to use it more, if anything?
• 32:22
  SPEAKER_01: Um what we uh now one thing comes to mind is obviously uh the web interface, although it's starting to get a little bit dated, but uh I know the guys from WePay are pretty busy, uh uh busy with uh with moving that to uh a different setup.
• 32:38
  SPEAKER_01: Uh but having the overview of what's going on now and what happened in the past and what's going on in the future is something that's not very available within other products.
• 32:49
  SPEAKER_01: Um, the other thing is that having the ability to do the backfilling, so changing basically changing history, um, is also something that is very convenient because you know while you're creating your products, uh you need to versionize your data, but your data also needs to change to the to the data uh or and it needs to change to the new demands that you're getting in.
• 33:14
  SPEAKER_01: Uh, but you need you might even need both.
• 33:17
  SPEAKER_01: So you need to go back into history, do redo all those calculations, but might not want to do the full set every time.
• 33:24
  SPEAKER_01: So those backfills really help out uh to just do the particular what parts that you actually need to do, so you can add the information while not you know doing it from the entirely from the beginning with all the tasks involved right away, because that you know solves a lot of um capacity issues uh if you would need to do that then every time again.
• 33:44
  SPEAKER_01: Um the required item potency is actually also in uh uh is also um in or is actually very useful because it does allow you to go back in time and go forward, and for auditing, this is actually also interesting because if you need to recalculate your models every time fully from the beginning, um, and you know, do all the data control from the beginning, it comes becomes pretty hard to have a kind of auditing trail in place, and this is important uh for us as well to uh to do that.
• 34:16
  SPEAKER_01: It's not perfect, but it's a lot better than you know doing that calculation every time again.
• 34:22
  SPEAKER_01: Um the other things we do like is obviously the retryability of tasks.
• 34:28
  SPEAKER_01: The the data is unreliable, the data is unreliable, and the systems that it's that that they're on are unreliable, that allows you to do retries is uh is uh and have those on-failure callbacks and these kind of things are are also very interesting and very useful.
• 34:48
  SPEAKER_01: Um because the um because the uh the the the other systems that are in place that they just fail and they just stop.
• 34:56
  SPEAKER_01: And you'll be called as you if you're an operations guy, you'll be called up in the middle of the night and you have to restart your job, and then it then it doesn't fail anymore.
• 35:03
  SPEAKER_01: Well, that kind of sucks if you can have that programmatically done.
• 35:07
  SPEAKER_01: That's actually uh uh a lot better.
• 35:10
  SPEAKER_01: Um, and then finally, I guess not having that it's that's being written in a programming language, and you can use it in a programming language, uh, in this case Python obviously makes it easier to test and also much more flexible because you can use uh NiFi or something else, but you'll be just clicking uh clicking it.
• 35:31
  SPEAKER_01: And uh if if that connector doesn't work for you at that moment, well uh that's tough luck because you cannot continue.
• 35:38
  SPEAKER_01: And there and data, and especially with the kind of source systems that we are getting, uh it's definitely not standardized, so you'll need to do uh changes every time.
• 35:51
  SPEAKER_01: Yeah, for sure.
• 35:53
  SPEAKER_03: Um and then kind of going right along that, um, as you've uh how how long have you guys been using Airflow?
• 36:00
  SPEAKER_03: And kind of how has the setup for it changed since?
• 36:06
  SPEAKER_01: So say again, you want to know how much how long have we been using Airflow?
• 36:10
  SPEAKER_03: And the second question was like how has your use of it changed since you started using it?
• 36:14
  SPEAKER_03: Most people we talk to start using it for something like a very, very specific thing, and then it branches out into a much more general tool set rather than something to solve a specific problem.
• 36:24
  SPEAKER_01: Um we started using it about two years ago, I think, uh, when we needed a a workflow, uh a workflow manager, um, and we didn't want to get involved with Uzi.
• 36:36
  SPEAKER_01: Um, and other parts of this bank were using uh were using Chrome instead, and we thought, ah, that doesn't really work for us.
• 36:44
  SPEAKER_01: Um so we started looking at Luigi first, and then one of the guys uh in the team said, Oh, why don't you look at Airflow?
• 36:50
  SPEAKER_01: And then okay, we got the and I personally got involved with Airflow uh because of that remark.
• 36:56
  SPEAKER_01: Uh, and then we just started putting it to use.
• 36:59
  SPEAKER_01: Um the uh what has changed?
• 37:02
  SPEAKER_01: Um I think we're on the verge of the bigger change.
• 37:05
  SPEAKER_01: So for managing the the different clusters, I think that's the big change.
• 37:09
  SPEAKER_01: For now, it's just building the decks uh and and having the you know running the products that we have.
• 37:16
  SPEAKER_01: Uh I consider that pretty standard basically.
• 37:20
  SPEAKER_01: Um, and did that does that um we're not let's say we're not at the size of an Airbnb yet, or uh maybe even WePay uh is also uh is also more advanced in these areas just because they have those data lakes basically ready.
• 37:38
  SPEAKER_01: So they just build a new deck and you get a new output.
• 37:40
  SPEAKER_01: Uh for us, it's more like okay, uh filling those data lakes in uh and and having that data available and then go from there.
• 37:47
  SPEAKER_01: And we're just getting at that point that it makes sense to go beyond that uh and having uh sort of, for example, something like superset on top of that, uh, where it becomes more flexible.
• 38:00
  SPEAKER_03: Um and then as you've kind of expanded use for it and put it into different problems, uh kind of have you had any issues with scaling up or just giving it more load?
• 38:12
  SPEAKER_01: Um not really.
• 38:15
  SPEAKER_01: Uh what we do see on the operation side is well, obviously I've been involved with Airflow pretty intimately, so I uh I know quite a lot of things about how the scheduler works uh and uh what the um kind of automatically know what kind of best practices there are there.
• 38:36
  SPEAKER_01: And I do know that uh for newcomers to Airflow, this is kind of uh an area where we're lacking in documentation.
• 38:46
  SPEAKER_01: And uh, I think somebody wrote a or wrote something somewhere else and said, you know, you can actually make money by writing a book on airflow operations.
• 38:53
  SPEAKER_01: And I did I do think that is a bit true because the kind of questions that we get is like, hey, my uh my my Airflow scheduling is not working and the tasks are not running anymore, and then we'll find out that somebody just gave it uh maybe two or three gigs of memory and started running long-running tasks with with you know where the processing happens even on the local machine.
• 39:14
  SPEAKER_01: Well, well, that doesn't really work.
• 39:16
  SPEAKER_01: Uh these kind of and it's it's these kind of things come in, and we didn't have those issues.
• 39:23
  SPEAKER_01: Um but I do think that uh this could be in documentation where uh where it is a little bit easier to start from, uh and those assumptions are you know straightened out a bit quicker.
• 39:39
  SPEAKER_01: So those growing pains that other people have seen, um especially because we're just gradually growing, we're not exploding in the amount of DAX right away.
• 39:49
  SPEAKER_01: Um we're and therefore we are able to tackle some of the issues that we're that are coming in a little bit easier with the capacity uh that we have.
• 39:58
  SPEAKER_01: Um we'll just create a patch for it uh when we encounter it, and we're not being hurt right away.
• 40:04
  SPEAKER_01: Um, so um um the experience has been pretty good, I guess.
• 40:10
  SPEAKER_01: That's the conclusion I would say.
• 40:13
  SPEAKER_03: Yeah, um, it's funny you mentioned writing a book for Airflow for more like documentation best practices.
• 40:19
  SPEAKER_03: Um, we've gotten that from a few other people we've talked to where all of their remarks are essentially like, I think what I'm doing is right, but I wish there was like a defined answer.
• 40:27
  SPEAKER_03: Like, should I be using a subdag for this?
• 40:29
  SPEAKER_03: Should I be using a sensor versus two DAGs or whatever the heck?
• 40:32
  SPEAKER_03: Like everyone wants like confirmation that what they're doing is correct.
• 40:37
  SPEAKER_01: Uh I think I think a couple of best practices there would be interesting because there's a lot of ways to a lot of that's a strength of Airflow as well, to have a lot of ways to solve um uh to solve particular problems.
• 40:48
  SPEAKER_01: There's not one way to solve a particular issue, there's multiple ways to solve it.
• 40:53
  SPEAKER_01: Um, but you probably want to have some guidance how other people are solving it and uh why they made certain choices uh and why they didn't make certain other choices in this case.
• 41:03
  SPEAKER_01: I think that would really help.
• 41:05
  SPEAKER_01: Um the the treasure trove is indeed uh quite often on the mailing list or and on the older mailing list on the Google Groups one as well, um, and also sometimes by just asking.
• 41:17
  SPEAKER_01: Um, but yeah, I think you're right.
• 41:19
  SPEAKER_01: You know, having that or airflow operations book, I think, is something uh maybe if I find a time somewhere or somebody wants to co-write it, I'll deliver a couple of uh uh paragraphs or uh chapters.
• 41:31
  SPEAKER_01: Um sure.
• 41:32
  SPEAKER_01: But it's also uh we uh we need to run uh this group, we need we're growing it, finding a time uh to write a book in the meantime is uh well a bit tough.
• 41:42
  SPEAKER_03: Yeah, right.
• 41:43
  SPEAKER_03: Yeah, um it's funny.
• 41:45
  SPEAKER_03: So when we were talking about this podcast internally, our uh our CEO Rai wanted to kind of like have this.
• 41:51
  SPEAKER_03: So his argument was that the book Agile Development, Agile Development on Rails, really helped the Ruby on Rails movement.
• 41:58
  SPEAKER_03: And he he's kind of like, this can kind of be that, but for Airflow, and if it goes well, we can like talk about writing a book or something.
• 42:03
  SPEAKER_03: So super interesting that about that.
• 42:06
  SPEAKER_01: Yeah, yeah, I think the podcasts are actually nice.
• 42:08
  SPEAKER_01: I listened to the podcast you guys had with Max.
• 42:10
  SPEAKER_01: Um, but I do think you'd like to kind of you know be able to have an example that you can look up.
• 42:16
  SPEAKER_01: Uh and that's a bit different from doing that from a podcast, because I would talk if I would talk code now, that would be a bit awkward, I guess.
• 42:22
  SPEAKER_03: Yeah, for sure.
• 42:24
  SPEAKER_03: Yeah, maybe, maybe if not a full book, then just like some added documentations with some more dynamic examples.
• 42:30
  SPEAKER_03: Like I think on our side, one of the coolest parts about Airflow is how you can uh dynamically configure DAGs and dynamically generate DAGs.
• 42:38
  SPEAKER_03: And most of the documentation out there just doesn't even talk about that.
• 42:41
  SPEAKER_03: It's like a little too basic for some of the heavier use cases.
• 42:45
  SPEAKER_01: That is absolutely true.
• 42:47
  SPEAKER_01: So, but that's uh um, yeah, uh and there have been talks about it.
• 42:51
  SPEAKER_01: There've been meetups in uh in uh in SF uh uh around this as well in the Bay Area, um, that where people talked about it and how to approach these kind of things.
• 43:00
  SPEAKER_01: Uh WiiPay definitely did that.
• 43:02
  SPEAKER_01: I think there was uh a Laura Lawrence, I think her name is, she talked about it as well.
• 43:07
  SPEAKER_01: Um, and also the different testing strategies.
• 43:10
  SPEAKER_01: How do you test your DAGs?
• 43:12
  SPEAKER_01: Um is uh is uh is something that should be in this book, uh, but has been talked about online as well.
• 43:18
  SPEAKER_01: But there's you know that's only they're provided with generic examples with a lot with a large uh uh with a large uh um they'll give a big picture of how they generate what they generated, uh, but I think uh an airflow course, so it's also a nice addition to it.
• 43:36
  SPEAKER_01: I think you can make money with that too.
• 43:39
  SPEAKER_03: Yeah, I'll start teaching something at Udemy or something, like ten dollars a course or something.
• 43:43
  SPEAKER_01: Uh we have a consultancy organization uh in uh in uh the Netherlands now called GoDaddy Driven.
• 43:48
  SPEAKER_01: One of the uh guys there just became a committer for Airflow, and I think they'll probably do something like that because they're a big user of Airflow, they put it in kind of in every kind of company they uh counter because.
• 44:00
  SPEAKER_01: Because there's uh nothing else really on the market that you know makes it easy for them to uh to uh to do their own companies.
• 44:07
  SPEAKER_01: They might actually do a uh uh a course in the future, at least here Oh yeah?
• 44:12
  SPEAKER_03: What's that company's name?
• 44:13
  SPEAKER_03: I'd love to reach out to them one more time.
• 44:17
  SPEAKER_03: It's called Go Data Driven.
• 44:19
  SPEAKER_03: Go data driven.
• 44:20
  SPEAKER_03: Yeah, all right, super interesting.
• 44:22
  SPEAKER_03: We'll reach out to them for sure.
• 44:24
  SPEAKER_03: Um cool.
• 44:26
  SPEAKER_03: So, in kind of the spirit of best practices and testing procedures, um, do you have any internal best practices that you use when you're trying to write DAGs or any like how do you handle testing a DAG versus deploying a DAG?
• 44:38
  SPEAKER_01: Um, tests are executed in our our CI pipeline.
• 44:42
  SPEAKER_01: That's cool.
• 44:43
  SPEAKER_01: Um, to be honest, uh I'm not writing the DAGs currently myself.
• 44:47
  SPEAKER_01: That's what happens in a team, so I don't have a kernel look on what they do it.
• 44:51
  SPEAKER_01: I know they're integrated within the within the pipelines.
• 44:54
  SPEAKER_01: Um so I to be honest, I can't share much about it.
• 44:58
  SPEAKER_01: We do test them, that's the only thing I can say.
• 45:00
  SPEAKER_01: And there's test coverage provided with them, and uh uh, but sometimes they still get stuck in in production sometimes as well.
• 45:06
  SPEAKER_01: Uh we're uh we have upgraded to the 1.90 uh RC1 uh uh uh pre-release, so release candidate in this case.
• 45:14
  SPEAKER_01: Um we encountered some issues with the SSH operators, we didn't find them during testing.
• 45:20
  SPEAKER_01: Um, so uh that's sometimes you get those things.
• 45:24
  SPEAKER_03: Yeah, we're still uh I think we're on 1.82, and it's been relatively stable so far.
• 45:29
  SPEAKER_03: Um that being said, I'll probably break now that I've jinxed it.
• 45:32
  SPEAKER_01: But that's it's a serious watch it for 1.9.0.
• 45:36
  SPEAKER_01: That's a that's a scary release because it's a dot O release.
• 45:40
  SPEAKER_03: Yeah, and I think they're they're dropping a bunch of things in 2.0, correct, right?
• 45:44
  SPEAKER_03: Like direct imports and some other small things like that.
• 45:46
  SPEAKER_01: Ah, yeah, we're saying that, right?
• 45:48
  SPEAKER_01: I mean, but we can go from 1.9 to 1.10.
• 45:50
  SPEAKER_01: Nobody's saying that we go from 1.9 to 2.0.
• 45:54
  SPEAKER_01: So we'll find out if that's really the case.
• 45:56
  SPEAKER_01: I do think some of those deprecated features do make sense to remove them and just you know, uh trying to push people to a more sensible uh way of doing robot things.
• 46:08
  SPEAKER_03: Yeah, that makes sense.
• 46:11
  SPEAKER_03: Um I'm trying to see if I have anything else in my list of questions here.
• 46:14
  SPEAKER_03: Give me one second.
• 46:23
  SPEAKER_03: Yeah, so uh kind of talking about the future of the project.
• 46:27
  SPEAKER_03: Um if you can kind of snap your fingers and have one feature fully built out or one bug fully uh solved, uh kind of where would you want that to be?
• 46:39
  SPEAKER_01: I think one of the most ideal things I would say, um and some of our let's say some are already being worked on.
• 46:47
  SPEAKER_01: So the Kubernetes part is somebody's already we've been work, uh been working on it.
• 46:51
  SPEAKER_01: So the guy from Bloomberg is doing that.
• 46:53
  SPEAKER_01: Um so if that's there, I wouldn't.
• 46:55
  SPEAKER_01: I want it, uh I'm not uh it's not going to be my choice for now, but I if I would be able to uh disallow tasks to access the Airflow database, I would be pretty happy.
• 47:09
  SPEAKER_01: Um, because this is uh I would consider this a bit of a security issue.
• 47:14
  SPEAKER_01: Uh you and you you get people that arbitrarily write those DACs basically, and uh and those operators, and they can access uh the connection database, for example, so or the connection table.
• 47:28
  SPEAKER_01: And so they can pick up any kind of connection they'd like.
• 47:32
  SPEAKER_01: Um, and that's also uh to locations where I would not like to have this DAC basically accessing these kind of connection details.
• 47:41
  SPEAKER_01: The only way to solve that is to kind of have an API in between that uh that either that you know when the task is being executed, it gets the connection information with that task uh already being supplied when it's being sent out to the uh to the to the worker.
• 47:59
  SPEAKER_01: Uh, this requires probably something like registration of connections because otherwise the the uh the executor doesn't know which uh connections to send with it.
• 48:11
  SPEAKER_01: Um the other uh option uh and the other and the API but the other option here is that it picks it up from the API, but that will require a lot of questions to the API up front, and that's maybe not what you want because that will put another component with Airflow there, which is uh and Airflow already consists of quite a lot of components.
• 48:31
  SPEAKER_01: Um but uh and but what would allow this API if you have this API in place, you could actually have um independent upgrades from each other because if you upgrade Scheduler to a new version but it still allows the task to uh to access the old version of the API, you can asynchronously so or with uh with a kind of a uh uh rollover, I think it is, uh, upgrade your workers and your scheduler.
• 49:05
  SPEAKER_01: Now, basically, if you want to do it properly, you have to bring the scheduler down in every worker and the web server down and then upgrade and then turn them on again because otherwise, you know, the the schema from the database might not make sense anymore.
• 49:19
  SPEAKER_01: Having an API in between that allows versioning of this uh will uh have more uh uh is is easier uh for upgrades and and uh stability and availability in the future.
• 49:31
  SPEAKER_01: If I can and security as well.
• 49:33
  SPEAKER_01: So if I can uh do this, then that's this would be in place.
• 49:37
  SPEAKER_01: Um if I put a second one there, uh I would say uh a scheduler that will be event-driven.
• 49:43
  SPEAKER_01: Um so it will start it will be snowballing the uh the task.
• 49:48
  SPEAKER_01: The task can actually notify downstream tasks uh if that they're finished or have failed, and uh those downstream card tasks can then decide by themselves whether they need to be in acute state or not.
• 50:00
  SPEAKER_01: And nowadays the scheduler is pulling all of these tasks every time.
• 50:03
  SPEAKER_01: So if you have a lot of tasks and a lot of DAX, the scheduler will take more time or just new needs to be.
• 50:10
  SPEAKER_03: That would have huge performance gains as well, right?
• 50:12
  SPEAKER_03: Because then you don't have to queue up things that you know are gonna execute.
• 50:16
  SPEAKER_01: Exactly.
• 50:17
  SPEAKER_01: So and I already had uh one uh uh I was emailing with Max about it, but in the during somewhere between 1.6 and 1.7 I already had a uh a proof of concept of that one, but then the the multi-processing of the the multi-the DAC multi-processor basically kicked in and it didn't fit anymore, so it didn't continue it at that time, and it's a still big effort to get it up uh to get it stable.
• 50:41
  SPEAKER_01: And the multi-processor uh for the DAX is already solving quite a lot of these issues, obviously.
• 50:47
  SPEAKER_03: Yeah.
• 50:48
  SPEAKER_03: Um it's interesting you mentioned the API too, because uh a lot of the other people we've talked to have kind of said that if there was almost like an Airflow API, you could abstract that use cases to more non-technical people on your team because you wouldn't have to directly interact with the console itself.
• 51:04
  SPEAKER_03: Um, it would just give the platform more end users because you could wrap more things around that.
• 51:10
  SPEAKER_01: Yep, true, absolutely true.
• 51:12
  SPEAKER_01: Um I think it will, yeah, the the API, there's some of the API parts are already in place, uh, but I'm not entirely sure if it's the right approach yet.
• 51:22
  SPEAKER_01: I mean, there are it it's basically in the web server currently, uh, but there is some um so the design of the API is something that still needs to be considered.
• 51:35
  SPEAKER_01: And should it with be with the with the web server?
• 51:38
  SPEAKER_01: Probably not, I guess.
• 51:39
  SPEAKER_01: I mean, I think it should be a separate component, uh, especially because the web server itself is uh and the UI is going to be replaced in the future.
• 51:48
  SPEAKER_01: Um, and I think it just makes more sense to have it separately because you know you know you have got an independent budget, you can uh more microservices like uh to replace them.
• 51:57
  SPEAKER_01: Um and also um well, if I would change the third thing, is that the constant polling of both the web server and the scheduler on the DAGs itself, um, and with you know, even if they haven't changed, so they're parsing the the DAX the whole time and it's not again not even driven, um, then that would be my third option to change that because that will just reduce so much load from these uh areas and also a lot of you know disk load, I guess.
• 52:28
  SPEAKER_01: Uh, and uh then the web server would immediately know whether it's a new DAG or not, instead of you know again pulling uh again polling the file system for it.
• 52:39
  SPEAKER_03: Yeah.
• 52:40
  SPEAKER_03: Um I had to find out the hard way that every all top level code on a DAG gets executed every heartbeat and not every DAG run.
• 52:47
  SPEAKER_03: If you don't put something in the DAG, and I was like, What this is like like I think I was like I had like a 10 element list or something, it was going crazy.
• 52:54
  SPEAKER_01: I'm like, yeah, yeah, yeah, yeah.
• 52:56
  SPEAKER_01: Yeah, it's code and it's being run every time.
• 52:58
  SPEAKER_01: That's absolutely true.
• 53:00
  SPEAKER_01: It's uh and I we're thinking the serialization part of the DAG is actually a bit hard, um, because that's it it seems simple from my from my uh from a first perspective, you know, just serialize the DAG and you know have some serializer there and keep that in memory.
• 53:15
  SPEAKER_01: Um but there some of the cases are actually pretty complex, and uh, and uh I what I understood from the Airbnb guys, if they would serialize some of them that would be multiple gigabytes in memory, uh, because it just then pulls in all the dependencies uh right away, and that makes it sometimes a little bit tough uh to do it.
• 53:35
  SPEAKER_01: I don't know what the right approach is yet.
• 53:37
  SPEAKER_01: Um, I do think that kind of wireframe DAC needs to be in place and that you only read the full DAC when it's really needed.
• 53:44
  SPEAKER_01: I think the dependencies on um for the scheduler, for example, the scheduler only cares about you know, uh does the task need to be scheduled or doesn't, and does it meet its dependency or not?
• 53:55
  SPEAKER_01: Uh but the dependency management of that, so the check of the dependencies, actually happens with a little a bit of executing of the code, uh, and this is where it gets a little bit more tough.
• 54:05
  SPEAKER_03: Yeah, we had a huge issue with how the pickling industrialization was working.
• 54:09
  SPEAKER_03: Um, our CTO Greg eventually figured out a way around it.
• 54:12
  SPEAKER_03: I don't really remember, but that's definitely something that uh we could be doing that.
• 54:18
  SPEAKER_03: Do you guys pickle the DAGs?
• 54:19
  SPEAKER_03: We don't pickle DAGs anymore.
• 54:21
  SPEAKER_03: Um I don't know the exact reason behind it, but I can have Greg follow up with why exactly that is because it was a pretty it was a pretty good reason as to why because the way they were pickled before was breaking a bunch of our other stuff.
• 54:33
  SPEAKER_01: Yeah, yeah, absolutely.
• 54:34
  SPEAKER_01: So it's it's uh I would say we need to deprecate the pickling uh entirely or replace the pickling module with a different one.
• 54:42
  SPEAKER_01: There's a I think there's another uh it's called something something pickle in this case, somebody mentioned it.
• 54:47
  SPEAKER_01: Uh that could be an option.
• 54:49
  SPEAKER_01: The other one is called marshmallow to pickle things, uh, and it's also widely used.
• 54:54
  SPEAKER_01: Um, but sometimes I also think is that you know if you have an API, why do you pickle, why do you pickle DAGs because you want to supply them to the workers uh across the across the uh across the network?
• 55:06
  SPEAKER_01: Well, the DAGs in itself are often not that big to send around.
• 55:10
  SPEAKER_01: This is just a synchronization problem, right?
• 55:11
  SPEAKER_01: So it's it's either have an NFS share in place or do a git sync.
• 55:15
  SPEAKER_01: But the other thing is you could also have an API call in place.
• 55:18
  SPEAKER_01: Uh, do I have the latest DAG for this one?
• 55:20
  SPEAKER_01: Verify the checksum.
• 55:22
  SPEAKER_01: If it isn't, you know, copy it over and then uh and then start running it.
• 55:25
  SPEAKER_01: If the new one comes in, you know, make sure that you and you can even send letters to the executor so that even the uh the to the API to check it is not required.
• 55:34
  SPEAKER_01: If you know if it's if it's the same one, if the same uh hash for it, then uh then it's actually quite easy.
• 55:40
  SPEAKER_01: Uh and then it's uh then that that issue is solved.
• 55:43
  SPEAKER_01: I had some discussion with Max around it.
• 55:45
  SPEAKER_01: He was really adamant about having uh the uh the the git sync, the git time machine thing in place.
• 55:52
  SPEAKER_01: Um uh but both mine and his yet.
• 55:58
  SPEAKER_01: So the the git time machine, I think we've talked about it for over a year uh now, and uh and the uh the API call I didn't get around to implement it properly either.
• 56:07
  SPEAKER_03: Yeah, I think last we chatted with Max about pickling is uh he's trying to stay away from pickling as well because it's caused him some issues.
• 56:14
  SPEAKER_01: Yeah, yeah, it's it's uh it's yeah.
• 56:18
  SPEAKER_01: The standard pickling in the Python is not really it's not very flexible and has a lot of issues, and people do really complex things in DEX.
• 56:27
  SPEAKER_01: Uh especially if you start adding methods and uh function calls to any uh or any lambda kind of thing, then it gets uh it gets difficult.
• 56:36
  SPEAKER_03: Yeah, the lambdas break the pickle as always.
• 56:38
  SPEAKER_01: Yeah, yeah, absolutely.
• 56:40
  SPEAKER_03: Yeah, cool.
• 56:41
  SPEAKER_03: So uh I think those are all the questions I had lined up.
• 56:44
  SPEAKER_03: Um are there is there anything cool that you're doing with Airflow that you want to give a quick plug to or anything along those lines?
• 56:50
  SPEAKER_01: Um well I if we do get the the the HA uh multi-data center airflow up and running, I'm definitely gonna scream it out to the world, I guess.
• 57:00
  SPEAKER_01: But that will take us some time, even the the the multidata center uh clusters are not in place yet, so that's kind of difficult uh to do it with.
• 57:08
  SPEAKER_01: And in the meantime, we need to hire people to actually take airflow in this case also further for us because uh 35 people, sure, uh uh, but that's you know, it's not an and not enough engineers to actually improve airflow for our use cases yet.
• 57:23
  SPEAKER_01: Um, I'm doing it in my spare time, to be honest, quite a lot of times.
• 57:28
  SPEAKER_01: Um and that you know, is is in this case it won't be enough for us anymore as a company where we need to pick it up a little bit more structurally.
• 57:36
  SPEAKER_01: Um, and so we'll do that uh most likely coming year, uh, where we have the budgets and uh and the people, hopefully as well eventually.
• 57:44
  SPEAKER_01: So uh if anyone is listening to this podcast, yes, we're hiring engineers that like to work on airflow.
• 57:52
  SPEAKER_00: Anyhow, we had that conversation with Boke quite a while ago about the Airflow Udemy course or some sort of guided development program.
• 57:58
  SPEAKER_00: And we then launched our space cam program, a guided development course just for Airflow, completely by Astronomer's Airflow experts.
• 58:05
  SPEAKER_00: I'll link to that in the episode description in case anyone's interested in checking it out.
• 58:09
  SPEAKER_00: Furthermore, I also wanted to mention that astronomer launched our managed airflow offering on product on the last week.
• 58:14
  SPEAKER_00: So feel free to head over and show us the level if you feel some comment.
• 58:17
  SPEAKER_00: Thanks so much for tuning in again.
• 58:19
  SPEAKER_00: And as I mentioned, we're working to get better in production and making this content as accessible as possible.
• 58:23
  SPEAKER_00: So please don't hesitate to send us any feedback you want.
• 58:26
  SPEAKER_00: You can find my email address and podcast description.
• 58:28
  SPEAKER_00: Open to you for the next episode.