Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Origins of Airflow

Transcript

The Origins of Airflow · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: Hi there, and welcome to the Airflow podcast produced by Astronomer.
• 0:04
  SPEAKER_01: Over the past six months, we've taken a complete deep dive into the data engineering space and I've heard from some extremely interesting people about their use of Apache Airflow.
• 0:11
  SPEAKER_01: We're extremely excited to get this season kicked off and can't wait to share some of our interviews with you.
• 0:16
  SPEAKER_01: If you didn't catch our teaser, here's a quick summary of what this season will cover.
• 0:20
  SPEAKER_01: We'll explore Airflow, including the concepts and motivations behind its creation, how specific data teams are using it in cutting edgeways, and best practices.
• 0:28
  SPEAKER_01: You'll hear from key contributors to the open source project, players in the larger data engineering space, and even some of our own Airflow gurus, Here at Astronomer.
• 0:37
  SPEAKER_01: It's our hope that this podcast will solidify Airflow's status as a gold standard for ETL.
• 0:43
  SPEAKER_01: My name is Pete DeJoy.
• 0:44
  SPEAKER_01: I'm a product specialist here at astronomer, and I conducted these interviews with Viraj Bereck, one of astronomers' data engineers.
• 0:50
  SPEAKER_01: While you'll hear from me occasionally, most of the interview questions and dialogue will be coming from Virage, as he's a real Airflow expert and can get really, really granular with the technology.
• 1:00
  SPEAKER_01: Without further ado, let's dive into episode one: The Origins of Airflow.
• 1:05
  SPEAKER_01: On today's episode, we'll be chatting with Airflow creator Maxime Bochemin to explore the origins of Airflow, including the motivations behind its creation and what problems it was designed to solve.
• 1:14
  SPEAKER_01: Hope you enjoy.
• 1:20
  SPEAKER_01: Hey Maxime, we're we're really excited to speak with you today.
• 1:23
  SPEAKER_01: As I'm sure you're aware, Arcumid Astronomer works really hard with Airflow, and we're huge fans of all the stuff that you've created with um with both Airflow and Superset.
• 1:30
  SPEAKER_01: So let's just dive right in.
• 1:32
  SPEAKER_01: You mind starting off by giving us a little bit of a background about yourself?
• 1:36
  SPEAKER_00: Right.
• 1:37
  SPEAKER_00: So I've been in the data space for a very long time.
• 1:40
  SPEAKER_00: So I started uh my career doing a little bit of web development, and then I um I proceeded to become involved in data warehousing projects and business intelligence.
• 1:51
  SPEAKER_00: So at the time, uh you know, data engineering uh and the data work uh was done in quite a different uh set of tools and different way.
• 2:01
  SPEAKER_00: Um so for like a decade, I was uh I was kind of an old school uh ETL engineer, uh doing a lot of um OLAP and business intelligence uh type solution uh at a company called Ubisoft that's still very active.
• 2:17
  SPEAKER_00: So they uh they make video games.
• 2:19
  SPEAKER_00: Um and um I think around 2007 is when I 2006 or seven is when I went to Yahoo, and that's where I was first exposed with um with big data.
• 2:30
  SPEAKER_00: And I was kind of I was there while Hadoop was uh starting to shape up, uh doing some analytics there.
• 2:38
  SPEAKER_00: I guess that's where where the world started to transform a little bit into uh you know data engineering and and modern analytics, yeah, and some way MapReduce stuff back then, right?
• 2:49
  SPEAKER_00: Yeah, MapReduce and you know distributed file systems, and so so that stuff started taking on.
• 2:55
  SPEAKER_00: I went back to Ubisoft for a little while, did a little bit of like uh you know game analytics, which is similar to web analytics in a lot of ways.
• 3:02
  SPEAKER_00: It's like uh gamer actions and time, a little bit like uh user actions and time uh on typical web analytics.
• 3:09
  SPEAKER_00: Um and then uh later on I joined Facebook where I did a lot of like more uh modern data engineering, so a lot of building, like writing script to build pipelines dynamically, uh building frameworks that compute things like uh engagement and growth accounting metrics.
• 3:25
  SPEAKER_00: Um, and I started participating in and improving the tools there.
• 3:28
  SPEAKER_00: So they uh there's a tool at Facebook called Data Swarm, um, that's very much like the uh some of the inspiration for for Airflow, along with my more traditional background.
• 3:39
  SPEAKER_00: But so I started um using Data Swarm heavily and contributing to it internally.
• 3:44
  SPEAKER_00: Um and by the time I left to join Airbnb, uh that's about three years ago in 2014, uh, I was looking for similar tools outside the context of Facebook.
• 3:56
  SPEAKER_00: Um and I pitched uh to the team there, the data infrastructure team at Airbnb, um, like starting to work on a project for modern, you know, like a modern scheduler to do uh modern ETL uh in a very dynamic way.
• 4:11
  SPEAKER_00: And that was the the early days and the beginning of Airflow.
• 4:15
  SPEAKER_00: Um and soon soon after that, after a year, uh a year or two working on Airflow, I started um to think about uh the visualization aspect of things.
• 4:25
  SPEAKER_00: And uh for a hackathon project, I I started um what ultimately became Apache Superset after a couple name changes.
• 4:34
  SPEAKER_00: So originally that was Panoramics and Caravel, um, and now it's superset.
• 4:38
  SPEAKER_00: But the idea um back then was to um was to uh try to recreate a little bit what existed at Facebook with something called Scuba.
• 4:48
  SPEAKER_00: There's a there's a paper on Scuba at Facebook, but essentially uh Scuba is a fast uh in-memory database that we can scan billions of rows and you know uh gigabyte scale kind of database where you can scan billions of rows in less than a second.
• 5:05
  SPEAKER_00: So druid.io is very similar to that, but had no UI at a time.
• 5:10
  SPEAKER_00: So um as my hackathon project, I decided to start building a front end for Druid IO so that we could have some real-time uh analytics and very fast scans um you know at Airbnb.
• 5:22
  SPEAKER_00: So that's kind of where I'm at.
• 5:24
  SPEAKER_00: So more recently, so I just left uh Airbnb to join Lyft, and now I think I'm gonna be working on uh geospatial visualization for a little while and just data infrastructure and and tooling uh moving forward.
• 5:38
  SPEAKER_02: Yeah, super interesting.
• 5:41
  SPEAKER_01: Just a quick note that voice that you just heard is Virage, the astronomer data engineer that I mentioned earlier.
• 5:47
  SPEAKER_02: We've been we were looking at superset for a little bit too, too.
• 5:50
  SPEAKER_02: So uh I think uh it's really interesting.
• 5:52
  SPEAKER_02: You kind of marked on this a little earlier, but just kind of distill it down.
• 5:55
  SPEAKER_02: Um could you explain to me and like what at its core Airflow is, just from the creator's perspective?
• 6:02
  SPEAKER_02: Like what was the original problem it was designed to solve?
• 6:05
  SPEAKER_00: Right.
• 6:05
  SPEAKER_00: So so Airflow is it's it's really a workflow orchestrator, um, but that was the original vision and it became much more than that.
• 6:15
  SPEAKER_00: Um so so it's somewhere in between a workflow orchestrator and and a big data platform, right?
• 6:23
  SPEAKER_00: And and recently we've been thinking about like, do we which direction do we want to grow the project towards?
• 6:29
  SPEAKER_00: Do we want to you know um be more humble and kind of just get the core part of Airflow working very well, or do you want to expand and you know do more things uh with with the project?
• 6:41
  SPEAKER_00: But I think the the main problem that it was solving at one is just orchestration.
• 6:46
  SPEAKER_00: So just the problem of having um, you know, you have large teams and even even small teams, um, people that will generate a lot of batch processes that need to run on a schedule that typically have very intricate uh dependencies.
• 7:01
  SPEAKER_00: So you have these these complex graphs of dependencies of jobs that need to run every day in a very specific way.
• 7:09
  SPEAKER_00: Um and there's the challenge of operating all this machinery every day.
• 7:14
  SPEAKER_00: So if you're using something like cron or something, something basic, it's really easy to get tangled and spend a lot of time trying to figure out where your log files are or what failed and why and who hones what.
• 7:26
  SPEAKER_00: So Airflow really helps, you know, orchestrating all of your batch processes.
• 7:32
  SPEAKER_00: Um and there's there's a limitation there.
• 7:34
  SPEAKER_00: I think if we we're talking about what Airflow is not, Airflow is not a streaming uh, you know, we we're not playing on the on this on the streaming um side of things.
• 7:45
  SPEAKER_00: So it's really uh an orchestrator for batch processes.
• 7:48
  SPEAKER_00: So that's the core of it.
• 7:50
  SPEAKER_00: Then you know there's a lot of connectors or you know, what we call operators and hooks in the Airflow world.
• 7:56
  SPEAKER_00: So I think that's definitely in scope to grow um the glue essentially that allows Airflow to talk uh and to abstract out all of these external systems, whether it's you know Hive Pig, MySQL, so database engines, as as well as you know, potentially things like Salesforce and external APIs, right?
• 8:18
  SPEAKER_00: So that people can easily, easily in their environment connect to different systems and talk to them and orchestrate all their jobs.
• 8:28
  SPEAKER_02: Yeah.
• 8:29
  SPEAKER_02: Um, I think that you definitely hit a lot about the hooks and operators thing.
• 8:32
  SPEAKER_02: And kind of as that library grows out, I think use cases of Airflow will kind of grow up naturally because it will connect and talk to more things, as you were saying.
• 8:39
  SPEAKER_00: Yeah, so it's been uh a challenge too to think about like should we package those in in the main Airflow package or should those live uh more externally so that they can evolve faster?
• 8:49
  SPEAKER_00: Um and you know, I think if it was if it was me, I would probably move a lot of these things out of the core package and perhaps have a core Airflow, you know, Python package along with external packages.
• 9:02
  SPEAKER_00: But a challenge there is that a lot of operators are kind of cross hooks.
• 9:06
  SPEAKER_00: So you might have a Hive to MySQL operator or Postgres to uh Postgres to S3 operator, for instance.
• 9:14
  SPEAKER_00: And and these are hard to package because then you have an S3 package, a Hive package, and there's a lot of operators that uh kind of cross over.
• 9:23
  SPEAKER_02: Yeah, for sure.
• 9:24
  SPEAKER_00: So so eventually we might refactor all the operators and and move them out so that they can evolve on a different schedule than the Apache release, right?
• 9:34
  SPEAKER_00: So so we can release uh the core of Airflow at a different pace than we can release say the operators.
• 9:40
  SPEAKER_02: Yeah.
• 9:41
  SPEAKER_02: Um I think that's a really interesting idea, especially because I think it would push the use cases for it further while still giving you time to put a lot of tension detail in the core, like the heavy lifting part.
• 9:52
  SPEAKER_02: Yeah.
• 9:53
  SPEAKER_00: Right.
• 9:53
  SPEAKER_00: It it doesn't require the same level of rigor, too.
• 9:56
  SPEAKER_00: So um, I mean, not that we don't need to be rigorous with with the operators, but it seems a lot easier to evolve the operators uh than it is to say evolve the skill evolve the scheduler or other things.
• 10:06
  SPEAKER_00: Um another thing I did not mention too um early on in the struggle of like whether Airflow is a data platform or just a scheduler.
• 10:15
  SPEAKER_00: Um so early on, you know, there's a there's a data profiling tab on the web UI that does a it's it's kind of a very humble take on what Superset does.
• 10:25
  SPEAKER_00: So, you know, allowing people to run SQL queries and see results, as well as like a charting, a small charting library to create line charts and and things like that.
• 10:34
  SPEAKER_00: So I think this has become it's it has been become more clear over time that this is out of scope for Airflow, that you know, data profiling, and there's other tools that do that well, namely superset, and that we want to you know keep maybe refocus on the core and probably in future releases at some point we we might uh take that out.
• 10:56
  SPEAKER_02: Yeah.
• 10:57
  SPEAKER_02: Um I think I have a question about that a little later, so we're gonna get back to it.
• 11:00
  SPEAKER_02: But sticking with Airflow design stuff, um so from my perspective, I am not the most senior dev in the world, but I really like Airflow because how easy is to develop on it, right?
• 11:09
  SPEAKER_02: Like everything's inheritance, and you can just it's very self-defining what a hook does versus what an operator does.
• 11:15
  SPEAKER_02: So can you talk to me about kind of some of the design principles that you follow to get that way?
• 11:19
  SPEAKER_02: Like what made you want to break things out into hooks and operators?
• 11:22
  SPEAKER_02: Um, and just what made you want to decide on DAGs as the uh as the way you model dependencies, like any anything along those lines?
• 11:30
  SPEAKER_00: Yeah, a lot of the design was informed by um in relation to to the Data Swarm design.
• 11:36
  SPEAKER_00: And and some of it is about like making it, making what's what was great in Data Swarm the same and what was not so great um different.
• 11:44
  SPEAKER_00: So one decision around like having one gigantic DAG.
• 11:49
  SPEAKER_00: So looking at Data Swarm is just one big DAG.
• 11:53
  SPEAKER_00: So if you want to visualize different parts of that big DAG, you have different entry points, but uh, but essentially uh they think of it as just like one big logical entity.
• 12:03
  SPEAKER_00: Um and for Airflow, I wanted to have this logical uh breakdown of having different DAG objects and to have that object per se.
• 12:12
  SPEAKER_00: And perhaps later on, you know, so we have the external DAG dependencies to connect DAGs together.
• 12:18
  SPEAKER_00: But to me, it was important to at least allow for boxing different different uh logical areas of the the more bit the bigger conceptual DAG into smaller chunks.
• 12:29
  SPEAKER_02: Was that for the sake of uh item potency a little bit or just for like overall readability?
• 12:34
  SPEAKER_00: More about like just organizing your work and like creating logical entities so that we can we have different entry points in the UI and entry points in the code, as opposed to just having like one big folder and one big you know namespace for all the tasks.
• 12:49
  SPEAKER_00: So it's almost like namespacing for the tasks, is the is the way I was thinking about that.
• 12:53
  SPEAKER_00: Um about about hooks and operators.
• 12:57
  SPEAKER_00: So I think Data Swarm had this notion of operators, but not much of a notion of a hook.
• 13:02
  SPEAKER_00: So building operators, it became kind of clear that we need we needed an abstraction for around external systems.
• 13:10
  SPEAKER_00: So hooks is is really what it is, right?
• 13:12
  SPEAKER_00: The MySQL hook, the hive hook.
• 13:14
  SPEAKER_00: Um so it's a place where we put information as to how we connect to um to external systems and to have uh you know put all all most of the logic around methods on how to interact with those systems.
• 13:28
  SPEAKER_00: So things like I want to get a set of rows or I want to load a set of rows into the system.
• 13:34
  SPEAKER_00: Uh so then most of the logic is in the hooks, and then the operators are very simple.
• 13:38
  SPEAKER_00: So it's just composition of of uh hook operations, right?
• 13:43
  SPEAKER_00: So you can think of the operators as very high-level, simple, you know, get some get some records from a system, uh, apply a transformation and put the put those records into a different system.
• 13:56
  SPEAKER_00: Uh for instance, and yeah, most of the logic is inside the hooks themselves.
• 14:02
  SPEAKER_00: So that was the high level.
• 14:03
  SPEAKER_00: And then there's like there's other abstractions in the system, and there's perhaps a need for for uh even more abstractions, but there's the executor as well that I I could get into.
• 14:14
  SPEAKER_02: Definitely drive right in, yeah.
• 14:16
  SPEAKER_00: Yeah, so so on the executor side, um, so as I started the project, it was clear that I needed some sort of just sequential or local executor just for testing and just to um to be able to run you know tasks in parallel and sequence as I was uh just writing my prototype and and proof of concept.
• 14:37
  SPEAKER_00: But then it was also clear that we were gonna need at some point some distributed uh way of running tasks.
• 14:44
  SPEAKER_00: Uh so that's when you know I thought of originally in the first uh prototype, the first version of Airflow, we had sequential executor for testing, local executor for uh maybe for parallel and for unambitious deployments, and then Celery for a slightly more distributed you know orchestration with multi- uh a cluster of Airflow machines.
• 15:05
  SPEAKER_00: And then I just knew too that other companies were gonna have like their own schedulers or their own um work uh you know systems that are essentially that allow for people to run arbitrary tasks.
• 15:17
  SPEAKER_00: So things like uh Kubernetes now, which did not necessarily exist at the time, but you know, most large companies have a um some sort of execution um right where you say, can you run this jar now or every day, right?
• 15:36
  SPEAKER_00: And these systems are usually don't have much orchestration or much uh knowledge of dependencies, but they're uh they they typically have the features of like give me a workload, perhaps a container, and I'll run that for you on a certain cluster of machines, and I'll ensure that you get the resources you're asking for and not more.
• 15:59
  SPEAKER_00: And um and yeah, so so I knew that you know different companies were gonna have different ways of running tasks.
• 16:06
  SPEAKER_00: So I was like, oh, this is a great place for an abstraction and a few implementation of this abstraction.
• 16:11
  SPEAKER_00: And now we have the community uh started working on a Kubernetes operators, uh, executor.
• 16:16
  SPEAKER_00: So people at uh Bloomberg, I believe, are driving it, and people at Airbnb are on board.
• 16:22
  SPEAKER_00: I know people at Lyft here using uh Airflow are thinking it's probably the ultimate, the best way to eventually run Airflow job is to run this workload inside um Kubernetes.
• 16:33
  SPEAKER_02: Yeah, um that was actually my next question a little bit.
• 16:36
  SPEAKER_02: So can you talk to me about kind of the ideal stack around Airflow?
• 16:40
  SPEAKER_02: Like you have your Airflow jobs, but in the ideal world, from what your perspective, what does orchestration look like?
• 16:46
  SPEAKER_02: What does logging look like?
• 16:47
  SPEAKER_02: Like where does all the metadata get stored?
• 16:49
  SPEAKER_02: Like if you could pick your Airflow environment, what is the ideal Airflow environment?
• 16:54
  SPEAKER_00: Yeah, so I think I I think there's not really ideal, or there might be like more or less mature workflow environment.
• 17:00
  SPEAKER_00: I can talk about more mature environments, but I think the the vision from the get-go was if you don't have a lot of infrastructure or you don't want a lot of infra infrastructure just yet, you should be able to run Airflow on your laptop.
• 17:13
  SPEAKER_00: Um, if you grow into running it on the on a on a VM, on the cloud, that's fine too.
• 17:19
  SPEAKER_00: And the day that you want to run it on four machines, that's fine.
• 17:23
  SPEAKER_00: Uh so you so you should have like the prop, it should be able to adapt and work on uh whatever infrastructure you have.
• 17:31
  SPEAKER_00: So I would say a mature airflow environment would today would probably use the salary executor.
• 17:38
  SPEAKER_00: And it's like unless like you have a very unambitious uh kind of take on this, like you probably should not use the local executor for much, uh, right?
• 17:50
  SPEAKER_00: Because the local executor runs um airflows jobs in a subprocess inside the scheduler process.
• 17:58
  SPEAKER_00: And that seems like um, you know, if you need to restart a scheduler for whatever reason or upgrade the scheduler, you have to like kill the tasks, uh, which is really not ideal.
• 18:07
  SPEAKER_00: So uh ultimately, uh from an executor perspective, I would say uh you can evolve into either using the salary executor with uh resource containment.
• 18:18
  SPEAKER_00: So there's a feature called uh C groups that uh Paul Yang wrote, I believe, originally, uh, that's that's about like uh provisioning or enforcing that airflow workers are allocating only a certain amount of memory or CPU resources to individual tasks.
• 18:36
  SPEAKER_00: So that prevents uh your data scientists from running you know massively parallel R jobs on your workers and ultimately you know killing or or uh creating problems on your on your airflow workers.
• 18:48
  SPEAKER_00: Um so definitely like C group containment is somewhat important.
• 18:53
  SPEAKER_00: Um, and then there's you know Kubernetes uh that's on the horizon for probably the the best way to run airflow jobs in the future.
• 19:02
  SPEAKER_00: I think containers are are pretty important too, and they're becoming kind of part of the of how people think about things and it solves problems.
• 19:11
  SPEAKER_00: Like I want to run certain jobs with uh specific sets of versions of libraries, and my colleague here wants to use a different version of Scikit Learn, and we can't agree on which version we should run.
• 19:23
  SPEAKER_00: And ultimately, the worker can only have one set of libraries.
• 19:26
  SPEAKER_00: So the container really you know solves this problem of like give me a task, give me a um the guarantee is I'll run your task inside your container, and then there's less um crossover and less need for consensus around uh configuration, which is a great thing.
• 19:43
  SPEAKER_00: Yeah, so that so that's one part of the setup.
• 19:47
  SPEAKER_00: Um I think you know, do we need elasticity?
• 19:51
  SPEAKER_00: I think there's there's no there's no right way to do it too.
• 19:54
  SPEAKER_00: I think a pattern we've seen emerge lately that thing is pretty cool is having Airflow provision it's the resources it needs, uh right.
• 20:04
  SPEAKER_00: So so people on on AWS like will uh run a job where the the first task of the airflow DAG will be to spin off, say, a Spark cluster, then they'll do a certain set of tasks within that Spark cluster that will be provisioned just for that DAG.
• 20:21
  SPEAKER_00: At the end of that DAG, they'll destroy the resources, uh, you know, and and just kind of flush it out.
• 20:26
  SPEAKER_00: And it solves problem around multi-tenancy on Spark.
• 20:30
  SPEAKER_00: And then maybe from a cost perspective, you're just doing your compute, you know, uh you're just using the resources that you need during the time that you need them.
• 20:40
  SPEAKER_00: If you need more resources for a specific DAG, you can just change the configuration element and say, I want uh I want 32 machines instead of 16 machines and not affect anyone else provisioning or other DAGs that might be using whatever resources they need, right?
• 20:57
  SPEAKER_00: So I think decoupling uh storage and and compute is a great thing.
• 21:01
  SPEAKER_00: And then like using elastic using airflow to provision and and be um to order the elasticity that it needs is is a cool pattern too.
• 21:10
  SPEAKER_02: Yeah, that's uh that's pretty similar to the setup we have.
• 21:13
  SPEAKER_02: Um that's you're what you're saying is literally we were dealing with this error the other day, and it was around like killing task instances that were queued but never actually execute, like um almost exactly what we were saying.
• 21:24
  SPEAKER_02: Okay.
• 21:25
  SPEAKER_02: Yeah.
• 21:25
  SPEAKER_02: So speaking of like just kind of ideal stuff, let's say that uh how do you see Airflow and Superset playing together down the road?
• 21:32
  SPEAKER_02: Now that you said a bunch of your time is gonna be spent on Superset.
• 21:36
  SPEAKER_00: Um yeah, so so I guess what joins it is the way that people work.
• 21:42
  SPEAKER_00: And um what we see or what we'd see a lot at Airbnb do uh people doing is you know, people write their own little pipelines that prepare data in a specific way to do a uh reporting in a certain subject area, a certain topic, and people might write their Their DAC that prepares a certain amount of data and then uh use superset to go and visualize some of that.
• 22:07
  SPEAKER_00: But um the the pattern, I think these tools are fairly decoupled, you know.
• 22:12
  SPEAKER_00: While uh data engineers do need access to the visualization tool to visualize the data that they output or even to monitor operational data, right?
• 22:23
  SPEAKER_00: As to say monitor your airflow cluster and make sure everything is running fine.
• 22:29
  SPEAKER_00: Uh so superset can be used to say monitor the airflow cluster or to consume the output of a data pipeline.
• 22:37
  SPEAKER_00: If if there's going to be much more integration there, um one cool feature we we had at Airbnb was uh you know an airflow um operator called like a superset cache refresh, so that when you would um your pro your pipeline would be done processing the amount uh the data for a dashboard, then it could trigger a cache refresh on the superset side.
• 23:02
  SPEAKER_00: So it would warm up the cache so that the dashboards uh would would reload the new data and load very fast uh with the latest data.
• 23:12
  SPEAKER_00: Right.
• 23:13
  SPEAKER_00: So so I'm not I'm not sure.
• 23:15
  SPEAKER_00: I think I see these tools as not having like too much too much overlap, though the people who are Airflow user uh most likely are superset users because they're they work with data.
• 23:27
  SPEAKER_00: And in some cases, you know, some of the superset dashboard require um a specific pipeline to prepare the data for the for the dashboard too.
• 23:36
  SPEAKER_02: Yeah.
• 23:37
  SPEAKER_02: And I also think that a superset is great for like almost integrity checks because when you're moving a bunch of data, you want to see if it's the right data you need or if anything's gone wrong, and visualizing it through a really like coupled tool makes it a lot easier.
• 23:50
  SPEAKER_00: Right.
• 23:51
  SPEAKER_00: So whether there would be uh integrations in that space, right, or like things that would bring the two together seems it seems logical, but I haven't really thought of a very, very specific way.
• 24:03
  SPEAKER_00: Um one thing I was thinking about more immediately, uh now I'm deploying superset at Lyft, and I was thinking that I will probably write a small um uh dashboard to monitor the the airflow health, right?
• 24:16
  SPEAKER_00: Like looking at how many tasks are we running, how many tasks are running right now, um you know, like who are the top users and what are the DAGs that consume the most resources.
• 24:27
  SPEAKER_00: Uh so it's it seems like pretty natural to go and create that dashboard in Superset to monitor you know the operations of Airflow or any any other systems for that for that matter.
• 24:38
  SPEAKER_02: Yeah, we uh we talked to someone a few days ago that's sending events from the Airflow Postgres to Keenio as a way of logging his DAG runs.
• 24:46
  SPEAKER_02: I think there's a bunch of interesting ideas on how people can approach that.
• 24:50
  SPEAKER_00: Right, right, right.
• 24:50
  SPEAKER_00: And at Airbnb we definitely add um and they they still have uh you know dashboards monitoring airflow and workloads and you know uh on superset for specific to airflow.
• 25:04
  SPEAKER_00: Yeah.
• 25:05
  SPEAKER_02: Yeah.
• 25:05
  SPEAKER_02: So kind of uh stepping back a little bit, um I'm sure that as you were building Airflow and looking at different like solutions for the problem you try to solve, you looked at a bunch of the other uh open source task schedules out there.
• 25:17
  SPEAKER_02: So kind of can you kind of say like what elements from if you still remember at all, like what elements from like Luigi or Uzi you wanted to take versus which ones you wanted to kind of forget about?
• 25:27
  SPEAKER_01: So I'm right.
• 25:28
  SPEAKER_00: Um so I did I did do some homework and uh like as to like looking at what was out there.
• 25:34
  SPEAKER_00: Um one thing I really wanted was to uh create pipelines dynamically, right?
• 25:39
  SPEAKER_00: So Airflow is a DSL where you can really easily add you know programming logic or or for loops and say, you know, read a config file or or read you know from a database, say a list of um experiments running in my A-B testing framework.
• 25:54
  SPEAKER_00: And then for each experiment, go and build a set of tasks uh based on that configuration parameter.
• 26:01
  SPEAKER_00: So that that was a hard requirement uh that I wanted coming coming from Data Swarm.
• 26:08
  SPEAKER_00: And I was I looked into um Uzi, Ascabon, Luigi, and and then I came from um from drag and drop tools.
• 26:16
  SPEAKER_00: So I know very well, you know, data stage, informatica, um, ab initial, and and similar tools in that space.
• 26:23
  SPEAKER_00: And I definitely did not want uh drag and drop, right?
• 26:26
  SPEAKER_00: I wanted to write code uh that defined workflows, put that code in the source control, um, collaborate with people on it and use uh programming logic to uh in some cases to define tasks and and DAGs.
• 26:42
  SPEAKER_00: Um looking at Uzi, so there so Uzi had just like an extremely uh bad reputation.
• 26:48
  SPEAKER_00: So talking with people that that used it in the past at Yahoo, it's really hard to find people that that um that you know have had good experiences with Uzi.
• 26:57
  SPEAKER_00: So that uh, you know, and just looking at the config language all in um in XML, I thought that was just uh not not a great fit uh from from the get-go.
• 27:08
  SPEAKER_00: So I did not like play with it very much.
• 27:10
  SPEAKER_00: And I and I know um Uzi got better over time.
• 27:14
  SPEAKER_00: So like my assessment back then might have been kind of a lagging um indicator on on where it really was at the time or where it is now.
• 27:23
  SPEAKER_00: So uh it might be great now.
• 27:25
  SPEAKER_00: I I just haven't really looked at it very much.
• 27:27
  SPEAKER_00: Uh on the Luigi side, I think that was the the closer to what I was looking for.
• 27:32
  SPEAKER_00: And I thought it was a pretty interesting project.
• 27:35
  SPEAKER_00: Um, I thought there was a lot of good stuff.
• 27:37
  SPEAKER_00: There's no um scheduler baked in, right?
• 27:40
  SPEAKER_00: So the the way um it's executed at uh uh Spotify, I believe, is like it's you know, they use other means of like scheduling their tasks.
• 27:48
  SPEAKER_00: And um, for me, it was somewhat important to have um like for the workflow engine to manage its own metadata as opposed to say looking for whether the existence of a file in HDFS is how uh Luigi operates to figure out whether it needs to uh run a task or not.
• 28:06
  SPEAKER_00: And there's some some potential like scalability issues there.
• 28:09
  SPEAKER_00: Uh the though that was not necessarily a blocker, um, but like one blocker for me with Luigi was the fact that to create a Luigi task, you need to derive a task class as opposed to instantiating an object.
• 28:24
  SPEAKER_00: So in Airflow, if you want to create a task, you want to create a hive task, you instantiate a hive operator, right?
• 28:31
  SPEAKER_00: And you get a task out of it.
• 28:32
  SPEAKER_00: So it's very natural to like say in a loop or in a condition to say um please create this this task programmatically, and you you get an object, you put in an array or whatever you want to do with that task.
• 28:45
  SPEAKER_00: With uh Luigi, you need to derive a class, which means uh it's not very natural to do that, or it's like static code, right?
• 28:54
  SPEAKER_00: You probably should not write a for loop that create tasks because that then you have to get into meta programming.
• 29:02
  SPEAKER_00: And clearly, um, like Luigi was written with the idea in mind that you would statically define pipeline, where Airflow was written knowing that we would dynamically um generate pipelines.
• 29:14
  SPEAKER_00: So that was that was like a big premise uh that didn't fit there and that was missing.
• 29:19
  SPEAKER_00: Uh similarly with with Askabon, it was also static configs.
• 29:24
  SPEAKER_00: Um and you know, I was working with people at Airbnb out of LinkedIn that were like, uh just like we don't like, trust me, we don't want to use Askabon.
• 29:34
  SPEAKER_00: So so and then I was driven with like I wanted to build something.
• 29:37
  SPEAKER_00: Like I'm a builder, I like to build systems, right?
• 29:39
  SPEAKER_00: I'm excited about writing fresh new code.
• 29:41
  SPEAKER_00: So these opportunities don't happen all the time, too.
• 29:44
  SPEAKER_00: Where like, okay, now I have this opportunity to start a new project, write fresh code.
• 29:48
  SPEAKER_00: So I would definitely jump on that.
• 29:49
  SPEAKER_00: That might have skewed my my decision making too.
• 29:53
  SPEAKER_02: Yeah, it makes a ton of sense.
• 29:55
  SPEAKER_02: Yeah.
• 29:56
  SPEAKER_02: Um, so I got one question from my infrastructure team that I think they would kill me if I didn't ask.
• 30:00
  SPEAKER_02: So uh I'm just gonna do it on their behalf.
• 30:03
  SPEAKER_02: Yeah.
• 30:04
  SPEAKER_02: Talk about how you've handled some of the problems with scaling airflow across from one laptop to one server to kind of a cluster.
• 30:10
  SPEAKER_02: And how have you kind of when you were building it or when you were heavy user of it, how'd you kind of wrestle with those?
• 30:16
  SPEAKER_00: Yeah.
• 30:16
  SPEAKER_00: So for the first, I would say, year of Airflow at Airbnb, and you know, so I joined Airbnb, it was uh, I think it was October or November 2014.
• 30:26
  SPEAKER_00: And it was, it took like a month or two before we had like some sort of like DAG in production using Airflow.
• 30:33
  SPEAKER_00: I think within like two or three months, we had some stuff in production, and then I wrote some of the Chef recipes to uh to, and I'm not really good at ops or I don't come from an ops background, but I'm full stack, so I decided to, you know, to or to set up the cluster and make it work.
• 30:50
  SPEAKER_00: And um, I was ultimately on call for a lot of it for the for the first year or so.
• 30:56
  SPEAKER_00: Um I think you know, scaling something is about, I guess from the get-go, knowing that you're gonna have to scale and making the right decisions around that, right?
• 31:05
  SPEAKER_00: Not having any, not be short-sighted on where there might be bottlenecks.
• 31:10
  SPEAKER_00: And then, and then from that point on, it's about identifying bottlenecks and and enlarging them, right?
• 31:16
  SPEAKER_00: So if the bottleneck that day is the number of MySQL connection, the number of connections that MySQL can handle, then you go and you fix that problem.
• 31:26
  SPEAKER_00: And the next day, the issue might be around, I don't know, the Redis message queue doing something or salary um having trouble scaling.
• 31:35
  SPEAKER_00: So it's about as you scale the system um over time, identify the bottleneck and get them out of the way and then find the neck the next bottleneck.
• 31:43
  SPEAKER_00: So that's my not so scientific kind of process to get to that.
• 31:48
  SPEAKER_00: Um, in terms of like operations and you know uh airflow, I'm sure there's a challenge for all the people uh running Airflow everywhere is that a scheduler, an orchestrator will get abused by internal teams, right?
• 32:05
  SPEAKER_00: So um hats had hat off to um all the airflow administrators out there.
• 32:10
  SPEAKER_00: Like you run a system and at your company that data scientists and analysts and data engineers uh will you know try to get the most out of and abuse in all sorts of ways, and that can be a really difficult role.
• 32:26
  SPEAKER_00: Um, at Airbnb, uh very quickly I got I got help.
• 32:31
  SPEAKER_00: I I could not have done done that on my own.
• 32:33
  SPEAKER_00: Uh and you know we we build a team of people, uh Dan and Alex and Paul and Krishna, and you know, there's a large team of people that got involved in different ways.
• 32:44
  SPEAKER_00: Um in a lot of cases, it's people that have incentives that are aligned, right?
• 32:49
  SPEAKER_00: So the big users of Airflow have incentives to keep it running and running well, right?
• 32:55
  SPEAKER_00: So you can ask for help from these engineering teams that are that are using the system to um to help out uh and into managing and scaling it and um you know be and sometimes be on call and be in the trenches, you know, with with the Airflow team and at making it work.
• 33:14
  SPEAKER_00: Yeah.
• 33:15
  SPEAKER_00: And over time, you know, oh I think it got better over time too.
• 33:18
  SPEAKER_00: So the stability and scalability of a system like Airflow with a large community of users, like there's a lot of people finding their own local bottlenecks and and uh fixing them.
• 33:32
  SPEAKER_00: So I think we're in a much better place as we're what three three years into the project now, and uh few hundred at least a hundred companies officially using the project the product and production.
• 33:43
  SPEAKER_00: So we're uh we're far beyond like anytime there's uh an issue that's significant, you can you know for sure that someone's gonna hit it pretty quickly and push a fix for it, which is great.
• 33:58
  SPEAKER_02: That's almost like the vision that like we're building our product around, right?
• 34:01
  SPEAKER_02: Where it's the same, it takes a team to get this thing up and running and managed and all that, and that's kind of what we can do.
• 34:07
  SPEAKER_02: Um so as far as my questions go, I think that was everything I had on my docket.
• 34:12
  SPEAKER_02: Um, I know we're a little over the allowed time slot, so are you still good to stick around?
• 34:15
  SPEAKER_02: Because I think Pete had one or two.
• 34:17
  SPEAKER_00: Yeah, I've got I've got time.
• 34:18
  SPEAKER_00: Um, and we started a little bit late too.
• 34:20
  SPEAKER_00: I'm actually I'm going for uh lunch over at Airbnb today to talk to the Super Set team.
• 34:24
  SPEAKER_00: So so even though I'm at Lyft, I'm gonna be collaborating with with the teams there pretty uh, you know, pretty significantly, you know, as a like maintaining these two projects is still gonna be like a large portion of my my workload.
• 34:39
  SPEAKER_00: Um so yeah.
• 34:40
  SPEAKER_01: Yeah, definitely.
• 34:41
  SPEAKER_01: And we're really glad to hear you'll be maintaining the two projects at your new gig.
• 34:45
  SPEAKER_01: Do you mind talking about that transition a little bit from Airbnb to Lyft?
• 34:48
  SPEAKER_01: Kind of what you're excited about, what sparked it?
• 34:52
  SPEAKER_00: Um yeah, so so you know, change is a good thing.
• 34:56
  SPEAKER_00: And uh the first thing, first thing I want to say is that like Airbnb has been super, super good to me, and it was like really good there.
• 35:03
  SPEAKER_00: So I'm not leaving with any uh you know tension or or any with motives that are coming from a place of anything negative.
• 35:13
  SPEAKER_00: So I just started uh recently talking with people at Lyft uh about collaborating on open source, and that turned into more and more conversation, and they uh they they pull that string until ultimately like uh I was there.
• 35:27
  SPEAKER_00: Um, you know, I think there's uh this this like just huge possibility for impact.
• 35:32
  SPEAKER_00: I think Lyft in terms of data infrastructure is a little bit behind uh compared to say where Airbnb is at this point.
• 35:39
  SPEAKER_00: And you know, I got excited about coming in and helping out and and you know, just a hygiene of change too.
• 35:46
  SPEAKER_00: Like I think it's a very healthy thing for people to uh to be a little bit more nomadic and and change environment and be exposed to a slightly different uh technology stack.
• 35:58
  SPEAKER_00: And I think you you bring a baggage of things that that can uh be very useful, you know, uh coming from a different company.
• 36:07
  SPEAKER_00: So just got kind of excited about it.
• 36:09
  SPEAKER_00: And you know, in Silicon Valley, people move around quite a bit, and you know, it seems like a big change going into it sometimes, or maybe externally, but we're I feel like looking at Silicon Valley is almost as if we're one big company or one big family building the future of the internet, and the change uh is really not that dramatic.
• 36:29
  SPEAKER_00: I'll be working uh much on the same projects with the same people, maybe with a different set of priorities.
• 36:35
  SPEAKER_00: Um, I'm excited to work, I've been meaning to work on like day uh geospatial data visualization for a while.
• 36:41
  SPEAKER_00: Um, and just kind of never made the top of my list at Airbnb.
• 36:45
  SPEAKER_00: And like immediately, that's like my immediate excitement about joining Lyft is I'm gonna be working on integrating uh DECGL into Superset.
• 36:54
  SPEAKER_00: And I think that's that's fun and exciting.
• 36:56
  SPEAKER_00: There's a PR that I sent uh that just went out this this morning to that's kind of the first step in that direction uh that has a good um animated GIF on it, so you can check it out and see what it's gonna look like too.
• 37:10
  SPEAKER_01: Awesome.
• 37:10
  SPEAKER_01: And if you're listening to this and you haven't checked out Superset, you should definitely check it out.
• 37:14
  SPEAKER_01: We use it a lot for some really, really robust internal reporting here at Astronomer.
• 37:18
  SPEAKER_01: We actually use it to um visualize the metadata of our airflow DAGs, which is which is really cool.
• 37:24
  SPEAKER_01: So, Max, we're big fans of your medium post, The Rise of the Data Engineer.
• 37:27
  SPEAKER_01: It's actually something that we reference pretty often internally.
• 37:31
  SPEAKER_01: One of the core philosophies that you present in that post is that ETL is best expressed in code and that drag and drop tools are too rigid for modern data teams.
• 37:38
  SPEAKER_01: Do you mind just diving into that philosophy a little bit?
• 37:40
  SPEAKER_01: What you really mean when you say that ETL is best expressed in code?
• 37:43
  SPEAKER_00: Well, I'll say, yeah, so that um so that was about um, you know, yeah, that the role of the data engineer and how it's different from more traditional roles.
• 37:52
  SPEAKER_00: And you know, in there I talked a little bit about drag and drop tools, but my point around around that is that I believe strongly that um the best abstraction for complex logic is code, right?
• 38:04
  SPEAKER_00: Like everything in software that's complex is expressed with code because code is uh it's modular, it allows for all sorts of logical operations, uh, you can you can build all sorts of uh you know reusable components.
• 38:20
  SPEAKER_00: Um it's easy to collaborate on, it's easy to diff with with technologies like like uh you know like Git and file editors, you can collaborate at scale on code and kind of like make it work in in terms of like knowing exactly where in time you are.
• 38:40
  SPEAKER_00: Uh with drag and drop tools, if you're trying to diff, you know, I know some of these tools do work with source control and theory, but the way they will surface that is visually where you might have like last week's workflow and this week's workflow and put some red tags where things have changed, and that just doesn't work as well.
• 39:00
  SPEAKER_00: Like it's just nice to be able to be to do something like git diff or git checkout um and you know just work with code.
• 39:07
  SPEAKER_00: So I think that's part of it.
• 39:09
  SPEAKER_00: That doesn't mean that there's no value in that uh drag and drop abstractions, though I think uh the abstractions that we need are not uh to to me um and it's probably easier to you know read the article because it's hard to express some of that stuff in words, but um but to to me, like the abstractions of drag and drop, which is exposing uh data transformation primitives like Group eyes and filters and movement through arrows and drag and drop, uh is not necessarily the abstraction that we need to build for ourselves.
• 39:43
  SPEAKER_00: Like the abstraction that we might need to build for ourselves is something like an engagement metric computation framework or growth accounting framework, right?
• 39:53
  SPEAKER_00: And the input and output of that system is probably some config files or sort of some forms uh that are not necessarily drag and drop uh ETL primitives.
• 40:03
  SPEAKER_00: They're more like what it what is it you're trying to do as you compute these engagement metrics or these growth accounting metrics?
• 40:10
  SPEAKER_00: Or if you're trying to do an abstraction around an A B testing framework, uh the input of that system would be something like what is your experiment and what are your metrics and what is what are your different treatments and what is your control and how you want to configure all this.
• 40:26
  SPEAKER_00: That is the abstraction we need to build something uh useful to analysts and uh and data scientists nowadays, right?
• 40:34
  SPEAKER_00: It's a custom computation framework that has a certain set of input and output.
• 40:39
  SPEAKER_00: And those are rarely drag and drop ETL primitives.
• 40:43
  SPEAKER_00: So that's that's kind of my my uh short, not so short, um, you know, collection of thoughts on this.
• 40:51
  SPEAKER_01: Very cool.
• 40:52
  SPEAKER_01: So kind of going off everything that you just said, do you mind talking a little bit about where you think there's still value to be added on top of native airflow and what you think about what Astronomer is doing right now with our managed airflow?
• 41:04
  SPEAKER_00: Yeah, there's there's a lot of value in everything that's gonna serverless nowadays, right?
• 41:08
  SPEAKER_00: Like so it I've been talking to lots of people at lots of data teams around around the valley mostly.
• 41:15
  SPEAKER_00: And uh, we all are building similar things and we all have the same struggle.
• 41:20
  SPEAKER_00: Um, and I'm not sure there's just not enough data infrastructure engineers nowadays to manage all the airflow clusters in the world, right?
• 41:29
  SPEAKER_00: So there's more need for infrastructure than there are people to manage that infrastructure.
• 41:34
  SPEAKER_00: So it does make sense to bring all this infrastructure, I think, under the same uh logical roof, I guess, and have a team of people that really focus on making that infrastructure work for uh for a bundle of companies.
• 41:48
  SPEAKER_00: So glad you guys are taking that on and uh doing that not sometimes it's not super rewarding work to be the on to be the on-caller to be you know the ops people for um a big data um system.
• 42:02
  SPEAKER_02: So yeah, um good stuff.
• 42:05
  SPEAKER_02: Yeah, not always the most rewarding.
• 42:07
  SPEAKER_02: Yeah.
• 42:08
  SPEAKER_02: Um I think that was all the questions either of us had.
• 42:10
  SPEAKER_02: Um, is there anything you want to say about the project or anything before this one's over?
• 42:14
  SPEAKER_00: Well, so so the first thing I want to say is like thank you to like everyone in the community.
• 42:18
  SPEAKER_00: Uh like largely their Airflow project over the past like year or two.
• 42:22
  SPEAKER_00: I've been like overseeing from a pretty high level, and there's a lot of other like awesome committers uh driving the project.
• 42:29
  SPEAKER_00: I don't want to start giving names because I'm gonna forget people and I don't want to make anyone upset, but like you know who they are uh if you're working close to the project.
• 42:37
  SPEAKER_00: Uh, there's a core team of of devs you know working on releases and on making everything uh work very well.
• 42:44
  SPEAKER_00: So people at Airbnb and beyond, uh right.
• 42:48
  SPEAKER_00: So I think Chris is working, Chris Ricardy me is working on the current release, uh, which is the 1.9 release, and that's just a lot of work to package an Apache release of a large uh you know software distribution that's used at a bunch of places.
• 43:04
  SPEAKER_00: Um Bolka, the Brune at uh ing in the Netherland Netherlands has been uh uh also packaged or released um like one or two releases earlier.
• 43:16
  SPEAKER_00: I did one.
• 43:17
  SPEAKER_00: Um, and then you know Dan and Alex and all the people at Airbnb um are doing awesome work.
• 43:22
  SPEAKER_00: So thank you to these guys.
• 43:24
  SPEAKER_00: I think uh about a little bit more about me, like recently I gave a talk that I'm hoping is is online now.
• 43:31
  SPEAKER_00: I haven't looked, uh, but um I was at Crunch um in Budapest and I talked about um it the talk was called um Advanced Data Engineering Patterns with Apache Airflow, and it's talking about these computation frameworks and some of the design ideas behind like building abstractions on top of Airflow.
• 43:53
  SPEAKER_00: Um so so people interested should definitely check out that talk.
• 43:56
  SPEAKER_00: It's a talk that I've given before.
• 43:58
  SPEAKER_00: I'm not sure if it's online anywhere, but you can Google advanced uh advanced data engineering patterns with Apache Airflow.
• 44:05
  SPEAKER_00: You might find the talk, and I think it's gonna be relevant to a lot of people in the community.
• 44:09
  SPEAKER_01: And we'll link that talk right down in the information section of this of this episode, just in case anybody wants to check it out.
• 44:16
  SPEAKER_01: Thank you all for tuning in to episode one of the Airflow podcast.
• 44:18
  SPEAKER_01: And thanks, of course, to Maxine for giving us that awesome interview.
• 44:21
  SPEAKER_01: We had a ton of fun talking to you and really interested to hear what you had to say.
• 44:25
  SPEAKER_01: Look out for another episode of this podcast coming two weeks from now.
• 44:28
  SPEAKER_01: We'll be covering Airflow use cases and different things that people are incorporating into their stack.
• 44:33
  SPEAKER_01: We'll also have on some very, very interesting guests, uh, many of whom Maxine actually entered mentioned in this conversation, other contributors to the project and people that are using it to do some really cool stuff.
• 44:43
  SPEAKER_01: So stay tuned.
• 44:44
  SPEAKER_01: Just drop us any questions or feedback for episodes that you would like to hear in the comments below.
• 44:48
  SPEAKER_01: Uh if you have any questions about airflow in general, you can head over to astronomer.io.
• 44:53
  SPEAKER_01: That's www.astronomer.io.
• 44:56
  SPEAKER_01: And just drop us a question in the web chat on the on the website.
• 44:59
  SPEAKER_01: We have Airflow experts monitoring it pretty regularly that are that are ready to answer any questions anybody has.
• 45:04
  SPEAKER_01: Again, thank you for listening and excited to keep going with this podcast in the coming weeks.