Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Open Source and Airflow at Google

Transcript

Open Source and Airflow at Google · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:01
  SPEAKER_00: Hi there, and welcome to this week's episode of the Airflow Podcast.
• 0:04
  SPEAKER_00: After a brief hiatus, we're back with season two of the show.
• 0:08
  SPEAKER_00: We're excited for this next generation of episodes and look forward to expanding upon our conversational focus a bit.
• 0:13
  SPEAKER_00: Whereas the episodes up to this point have largely been focused on airflow fundamentals and origins, the next bunch will look forward towards the future and hone in on exciting current development on the project and community.
• 0:24
  SPEAKER_00: This podcast is brought to you by Astronomer.
• 0:26
  SPEAKER_00: We at Astronomer are determined to make Airflow easier to run and manage.
• 0:30
  SPEAKER_00: Our offerings include both SaaS and self-hosted platforms that leverage the power of Kubernetes to make Airflow easy to run and scale, support for all things Airflow, and Airflow training that we call Space Camp.
• 0:42
  SPEAKER_00: We're excited to announce that we'll be kicking off season two with an interview with Google Cloud Composer product manager James Malone.
• 0:49
  SPEAKER_00: James and his team have been working on Composer for about a year now, and in this interview we dive into many areas in which they hope to improve the Airflow project.
• 0:57
  SPEAKER_00: Without further ado, let's dive into this week's interview with James.
• 1:06
  SPEAKER_00: Alright, hey James, how's it going today?
• 1:09
  SPEAKER_00: Good thanks.
• 1:10
  SPEAKER_00: Thanks for hopping on with us today.
• 1:11
  SPEAKER_00: We're excited to chat a little bit.
• 1:12
  SPEAKER_00: Um it's been a long time coming.
• 1:14
  SPEAKER_00: Um we've been looking forward to this conversation for a while, so I'm really excited to hear about what you're working on over Google and uh the what the cloud composer team is hacking on these days.
• 1:22
  SPEAKER_00: Definitely.
• 1:22
  SPEAKER_00: Thanks for having me.
• 1:23
  SPEAKER_00: I appreciate it.
• 1:24
  SPEAKER_00: Absolutely.
• 1:24
  SPEAKER_00: So do you mind just kind of getting started here by giving us a little bit of a background on yourself, how you ended up, where you are today, um, what kind of what set of circumstances led to that?
• 1:33
  SPEAKER_03: That's good.
• 1:33
  SPEAKER_03: That's a very I'll give you the the Google version of that, because uh the abridged version, because the the real version is very, very long story.
• 1:40
  SPEAKER_03: Uh it always is, right?
• 1:42
  SPEAKER_02: Like it always is.
• 1:44
  SPEAKER_03: Uh so came to Google to specifically work in cloud uh about four years ago, I actually think today, maybe my Google anniversary.
• 1:53
  SPEAKER_03: I think it is.
• 1:53
  SPEAKER_03: Um, yeah.
• 1:56
  SPEAKER_03: Um it's gone very quickly, actually.
• 1:58
  SPEAKER_03: So uh came to Google about four years ago uh to launch our Managed Spark in Hadoop service.
• 2:03
  SPEAKER_03: Um and at this point, um I am fortunate enough to work with uh other PMs on a team and uh are we look at uh essentially making managed services with open source software.
• 2:14
  SPEAKER_03: Uh it is an exciting space because we get to combine uh sort of the best of ecosystem and software, which is open source, with the best of Google's ability to run uh infrastructure at scale.
• 2:26
  SPEAKER_03: Um PM for for quite a while.
• 2:29
  SPEAKER_03: Um previous lives worked with Disney and Amazon.
• 2:32
  SPEAKER_03: So uh yeah, I'm I'm I I I do what I love and I get a uh do uh uh what I love at Google, so happy to be here.
• 2:40
  SPEAKER_00: Awesome.
• 2:40
  SPEAKER_00: Well that that sounds like uh kind of a great experience thus far, and congrats on the anniversary as well.
• 2:45
  SPEAKER_00: It's exciting stuff.
• 2:46
  SPEAKER_00: Yeah.
• 2:47
  SPEAKER_02: Yeah.
• 2:48
  SPEAKER_02: Yeah.
• 2:48
  SPEAKER_00: So kind of uh you have to be a good idea.
• 2:51
  SPEAKER_02: Yeah, uh you kind of jumped on this a little bit, right?
• 2:54
  SPEAKER_02: But you just mentioned how open source software and Google kind of had this like long history that's uh goes across a lot of different technologies.
• 3:02
  SPEAKER_02: Do you want to just kind of jump on that a little bit?
• 3:04
  SPEAKER_02: Uh kind of before the cloud compute age and kind of what it looks like now?
• 3:08
  SPEAKER_03: Yeah, definitely.
• 3:09
  SPEAKER_03: Um yeah, the the relationship is um not necessarily straightforward.
• 3:13
  SPEAKER_03: So um but it's not bad either.
• 3:16
  SPEAKER_03: It's just it's just complicated.
• 3:18
  SPEAKER_03: So Google um has been innovating, especially in the data space, for quite a long time.
• 3:25
  SPEAKER_03: Um I think you know, given uh many number of years ago, Google was very fond, and still does, but um it's a bit different now.
• 3:34
  SPEAKER_03: But Google is very fond of releasing white papers, um and a whole lot of open source ecosystems sprung up around some of the white papers that uh Google had released, which is really exciting, and I think it's good for the open source community.
• 3:46
  SPEAKER_03: Um strategically was probably not so great for Google because Google is fairly detached from um some of that open source.
• 3:53
  SPEAKER_03: Uh the white papers were out there and the ecosystem sprung up and grew, um, but Google is not necessarily, I think, as involved as we would be today.
• 4:01
  SPEAKER_03: Um that has slowly started to change over time as Google has built out a cloud.
• 4:07
  SPEAKER_03: Um and I think the sort of intersection that Google is at now is trying to both contribute to uh leading an open source software.
• 4:15
  SPEAKER_03: And I think sort of recent examples that many may be familiar with are TensorFlow and Kubernetes, um, but also try and make cloud services around those offerings and provide people sort of the best of open source, but also uh services that run the open source in really unique and interesting ways that provide uh end benefit to customers.
• 4:36
  SPEAKER_03: Uh at this point, from sort of my um my vantage point within Google, we're contributing actively to a number of open source projects.
• 4:44
  SPEAKER_03: Uh Airflow is one of them.
• 4:45
  SPEAKER_03: Um our cloud data proc service also actively contributes to a whole lot of projects within and kind of around the Apache Spark and Hadoop ecosystem.
• 4:54
  SPEAKER_03: Um and we see continued growth and demand for open source uh from customers.
• 4:59
  SPEAKER_03: So it's an area that we continue to grow and uh focus within uh uh Google Cloud.
• 5:05
  SPEAKER_00: Awesome, yeah, yeah.
• 5:06
  SPEAKER_00: You guys have such an illustrious history kind of contributing to open source and um being involved in the community as well.
• 5:12
  SPEAKER_00: So I guess I'm kind of curious like what led to uh you guys kind of bringing in airflow, bringing airflow into the picture?
• 5:17
  SPEAKER_00: What kind of initially brought it into your purview over at Google?
• 5:21
  SPEAKER_03: Yeah, so uh this all started a couple of years ago.
• 5:26
  SPEAKER_03: So uh creating workflows, and I'm gonna use the kind of umbrella term workflows to represent airflow DAGs and all of the things that you can do with uh with Airflow.
• 5:35
  SPEAKER_03: Um it's a bit of an overloaded term and a broad term, but uh just keep things simple.
• 5:39
  SPEAKER_03: That's sort of how I'm gonna run with this.
• 5:41
  SPEAKER_03: Uh workflows are critical when you're operating in a cloud environment.
• 5:45
  SPEAKER_03: I mean, really any environment, but especially in a cloud environment.
• 5:47
  SPEAKER_03: You have a slate of services that you want to use.
• 5:50
  SPEAKER_03: Uh we have a lot of customers that want to tie things together, want to orchestrate things, want to automate things.
• 5:55
  SPEAKER_03: So a number of years ago, uh we took a look at what we might be able, we being Google, might be able to do in this space.
• 6:02
  SPEAKER_03: Uh Google had tested out a few uh workflow solutions to see if they would meet customer needs before we uh settled on Airflow.
• 6:10
  SPEAKER_03: Um, and that was actually before Airflow uh really had gained momentum um within the open source community.
• 6:15
  SPEAKER_03: It was still very, very early days for Airflow at that point.
• 6:18
  SPEAKER_03: Uh we tried a few solutions out.
• 6:20
  SPEAKER_03: Uh people were somewhat happy with them, but there's always the kind of edge problems of people want to create workflows that they feel are open.
• 6:28
  SPEAKER_03: Uh customers wanted to create workflows that could talk to a whole number of services uh inside and outside of Google Cloud.
• 6:35
  SPEAKER_03: Um and in general, this is a space where we thought open source software could make a really big impact.
• 6:40
  SPEAKER_03: Uh we took a look at what was available within the open source community um a number of years ago.
• 6:45
  SPEAKER_03: Airflow uh was kind of gaining momentum at that point as um something that was getting a lot of excitement within the open source community.
• 6:54
  SPEAKER_03: Uh we had a number of customers uh within Google Cloud who had contributed originally to Airflow, so we uh poked around a little bit to uh just get a temperature check and sort of see what their vision for the project was and strengths and weaknesses.
• 7:07
  SPEAKER_03: Um and then we decided to sort of use the expertise that we had developed in um launching other managed open source software um services within Google Cloud to uh launch a managed uh Apache Airflow service.
• 7:22
  SPEAKER_03: Um and that's continued, I think, to be the right approach for us.
• 7:25
  SPEAKER_03: Um, but it was really the um Airflow's focus on um open workflows, so you can take a workflow and run it anywhere, whether it's inside of Google or not.
• 7:35
  SPEAKER_03: Um, and you you absolutely should be able to.
• 7:37
  SPEAKER_03: Uh it was the ecosystem of uh sensors and operators um and hooks that that really was interesting to us.
• 7:44
  SPEAKER_03: Uh and the fact that uh all of your workflows were code it made the workflows fairly approachable uh to a large number of people.
• 7:51
  SPEAKER_03: And I think those were the three key reasons that we chose Airflow to begin with and continue to be strengths of Airflow.
• 7:58
  SPEAKER_02: Gotcha.
• 7:59
  SPEAKER_02: Yeah.
• 8:00
  SPEAKER_02: So I think uh one of the things you mentioned that it's pretty exciting, especially from us, because I think it led into a lot of our decision as to why we kind of uh put so much varsity behind Airflow, is just the community behind it.
• 8:10
  SPEAKER_02: Um talk a little about how the community kind of factored into your decision and uh kind of how that plays with how Google likes to interact with the open source community nowadays for other projects as well.
• 8:19
  SPEAKER_03: Yeah.
• 8:21
  SPEAKER_03: So when we first chose Airflow, the community was definitely much smaller.
• 8:26
  SPEAKER_03: So we're making a strategic bet on Airflow itself.
• 8:29
  SPEAKER_03: Um and we were assuming that Airflow would grow, and we we sort of thought that it with our involvement and our uh contribution and attention to it, we could help grow the Airflow community.
• 8:39
  SPEAKER_03: Um I think in general, when we look at uh, and I'm sort of speaking for my my the area of Google Cloud that I see, um it's not the authoritative view, but it's a kind of data analytics view, I guess.
• 8:52
  SPEAKER_03: Um when we look at open source, we're looking for projects which um have a compelling technical direction, uh which have a growing community uh or an enthusiastic community, uh, which solve kind of net new or real customer problems, um, and which would which we think we can either help contribute to andor run effectively using what resources we have.
• 9:16
  SPEAKER_03: Um at that point, Airflow, I think, ticked a lot of those boxes.
• 9:21
  SPEAKER_03: Um, and that's the general slate of how we look at new uh or continued involvement in open source.
• 9:28
  SPEAKER_03: Um I think from what we've realized over the last number of years running managed uh open source within Google Cloud is there's a set of challenges which uh we definitely see where we can help uh grow the open source community and make open source a bit more effective through contribution.
• 9:46
  SPEAKER_03: Um and um I think we have really tried to focus our contribution efforts on uh boosting open source specifically with an emphasis on how that open source software runs, um, how it can scale, um, especially in sort of a cloud native environment, um, because we see that day in and day out.
• 10:08
  SPEAKER_03: And uh we both want to contribute back to the open source, but we want to make it better because it makes our managed services built on it better, uh, but it also just makes customers using it wherever inside of Google or not better.
• 10:20
  unknown: Yeah.
• 10:21
  SPEAKER_02: I think you hit on a really interesting point right there.
• 10:23
  SPEAKER_02: Like uh when it comes to a lot of these, like uh running some version of commercial open source, um, you kind of have to walk a balance between doing what your customers want and kind of listening to your customers, but also uh kind of taking the communic the community into regard as to like how does the community want to build something and how does that conflict in terms of timeline or uh design or whatever with uh kind of what your customers are asking for.
• 10:47
  SPEAKER_02: Um how does that kind of balance work internally?
• 10:50
  SPEAKER_03: Yeah, that's a that's a fair question.
• 10:53
  SPEAKER_03: So there's a few principles that I think we have are not written down, but uh if there were principles to be set in stone, these are sort of VR principles for how we view open source, which I think shapes the answer to your question.
• 11:06
  SPEAKER_03: Uh one is we absolutely want to contribute back and grow and participate within an open source community.
• 11:13
  SPEAKER_03: Uh two, we never want to fork an open source project.
• 11:17
  SPEAKER_03: We may have a private repository where we have uh a set of maybe diff patches uh or updates which have pending pull requests or things that we need to create Jira's for, uh, but essentially it's just gonna be a differential change set.
• 11:30
  SPEAKER_03: We never want to fork uh uh an open source project where we offer it to a customer.
• 11:35
  SPEAKER_03: Because that's not good for us.
• 11:37
  SPEAKER_03: Uh it creates a lot of work for us.
• 11:39
  SPEAKER_03: It's not good for uh customer because it means that they're inherently using something that is uh proprietary.
• 11:45
  SPEAKER_03: Um and it's not good for open source because it uh splices uh contributions and knowledge away from that project.
• 11:53
  SPEAKER_03: Um within Google, uh Google has, I think, traditionally had to solve really difficult problems.
• 12:01
  SPEAKER_03: Uh there's a lot of data um uh coming at I think Google in general very, very quickly.
• 12:07
  SPEAKER_03: Google has exceeded at developing a set of software and infrastructure to try and tackle those really big problems.
• 12:14
  SPEAKER_03: So uh I think Google has traditionally had less of an emphasis on trying to adapt open source to run inside of Google and more of a focus on developing services to actually just go run Google.
• 12:26
  SPEAKER_03: You know, there's I think the last I heard, there's you know, 400 hours of video added to YouTube um, you know, every minute or so.
• 12:34
  SPEAKER_03: So it's a lot of lot of data.
• 12:35
  SPEAKER_03: So Google's tried to develop the infrastructure and the software to scale out.
• 12:39
  SPEAKER_03: Uh, I think where sort of our area and the composer team is part of this area, where we've really tried to excel is both evangelizing open source uh inside of Google to solve problems and doing that because we are building and running this open source uh using a lot of the special tools that Google has developed to solve problems at scale.
• 13:01
  SPEAKER_03: So to give you a non-Airflow example, uh we have customers that are creating um Presto clusters that are that span over 90,000 cores, which traditionally is a pretty hard operational problem to do, and we have customers that do it um without a blink of an eye.
• 13:15
  SPEAKER_03: Um and that's sort of the union of us trying to improve open source, but also uh Google being able to run open source pretty effectively because of its hardware infrastructure.
• 13:26
  SPEAKER_00: Yeah, that no, no, no, that all definitely makes sense.
• 13:28
  SPEAKER_00: I I kind of want to touch on something that you just said though.
• 13:30
  SPEAKER_00: Um so can you tell talk a little bit about what that internal evangelized evangelization looks like?
• 13:36
  SPEAKER_00: How do you um expose Airflow to your team members, kind of get their thoughts on it before you like kind of decide to make a decision to build a product around it?
• 13:45
  SPEAKER_03: Yeah, uh so there's two questions here, and I'm gonna separate the questions so I can so I can answer them.
• 13:51
  SPEAKER_03: Uh one is sort of how can we evangelize just open source in general um inside of Google to try and build uh momentum behind it?
• 14:01
  SPEAKER_03: And two is getting buy-in to go build against airflow specifically.
• 14:05
  SPEAKER_03: Uh I'm gonna start in reverse here because that's sort of the origin of Composer.
• 14:09
  SPEAKER_03: Yeah, that's perfect.
• 14:11
  SPEAKER_03: Thanks for splitting them up.
• 14:12
  SPEAKER_03: Uh when we when we set out to when we when we set out to build a uh workflow service, uh I think as part of our due diligence, there's the the real question, and this is probably true in abstract and sort of product management, abstract, especially in cloud, should you build it or should you use something that already exists?
• 14:32
  SPEAKER_03: And there's pros and cons to bullet.
• 14:34
  SPEAKER_03: And when we really asked ourselves, should we go build a net new uh cloud native but Google-specific workflow orchestration system, um, does that make sense?
• 14:46
  SPEAKER_03: And then there's the counter question of is there something else out that if we're gonna spend a certain amount of energy and we're gonna go build something, uh, is there something can we sort of double or or multiply our efforts by just participating in something that already exists?
• 15:01
  SPEAKER_03: And that's just I'm just looking from a net resource standpoint.
• 15:04
  SPEAKER_03: I'm not even talking about the you know huge benefits to customers for running open source or just the benefit to everyone else for making open source better and sort of evangelizing open source.
• 15:14
  SPEAKER_03: Uh and I think the one of the big drivers of our involvement sort of in selling Airflow internally was we don't have to go solve all of these problems ourselves.
• 15:26
  SPEAKER_03: And I think there's uh as Google Cloud has grown and as running and creating managed services with open source and Google Cloud has grown, I think there's been a realization uh and an appreciation for the fact that when you run open source really well, you provide a benefit to the customer, no doubt.
• 15:41
  SPEAKER_03: But it also means that you don't have to go solve every problem on your own because you're also utilizing the power of the larger open source community.
• 15:48
  SPEAKER_03: Um at that point it actually became an easy sell, but developing that viewpoint and uh the sort of culture, that was, I think, uh a focus area over the last uh the last few years for us.
• 16:01
  SPEAKER_03: Uh in terms of sort of evangelizing Airflow specifically internally to Google, it's been very interesting because we've actually had more internal Google customers ask about Airflow since they know that we've been involved with Airflow.
• 16:15
  SPEAKER_03: Uh so it's actually, we've not really needed to sell it at all.
• 16:19
  SPEAKER_03: Uh we've actually been sort of inundated with the number of people who have wanted to use Airflow for cloud and non-cloud uses within Google, um, which has been really, really great to see because it's you know both great when you can use something to go build a company and be better, but it's also really neat to see uh Airflow just gain that that mind share and that momentum.
• 16:39
  SPEAKER_00: Yeah, absolutely.
• 16:40
  SPEAKER_00: And that's that's just generally great to hear for the community at large as well, right?
• 16:44
  SPEAKER_00: Like more users means more contributors, which means a better project, and um so on and so forth.
• 16:50
  SPEAKER_00: So do you want to talk a little bit about kind of where you see the future of the project going?
• 16:55
  SPEAKER_00: Um like in terms of what you see, or like where if if you had to predict like what would be an Airflow 2.0 right now, what what would kind of be on your list of of key features?
• 17:06
  SPEAKER_03: Uh that's a good question.
• 17:08
  SPEAKER_03: Um the the quick and dirty answer is sort of just cleaning up some of the code base, which I think is has been on the radar for Airflow too.
• 17:16
  SPEAKER_03: Um so I'm gonna give you the perspective of uh enterprise customers.
• 17:21
  SPEAKER_03: Because we deal with a lot of enterprise customers in cloud, um, and uh a lot of their needs are common but also really important and also actually really difficult to tackle.
• 17:30
  SPEAKER_03: So one of the big ones is role-based access control.
• 17:33
  SPEAKER_03: Uh, one of the weak spots in Airflow today is how role-based access control works or doesn't work, kind of depending on your your viewpoint.
• 17:41
  SPEAKER_03: Um and this is an area where we definitely want to focus on trying to help grow and extend Airflow in the community.
• 17:48
  SPEAKER_03: Uh, but role-based access control for most large enterprise customers is a showstopper.
• 17:52
  SPEAKER_03: If you don't have fine-grained uh workflow or task level control, that becomes a big problem.
• 17:58
  SPEAKER_03: Um so I think extending role-based access control is going to be uh a really big challenge.
• 18:04
  SPEAKER_03: Um I think number two, uh a lot of customers as they start using and growing Airflow look for greater resource efficiency.
• 18:15
  SPEAKER_03: Um and in Airflow today, there's the the Kubernetes executor, um, which is something that we were involved in, something that we continue to sort of think about and how it relates to Composer itself.
• 18:26
  SPEAKER_03: Um but I think additional tools and mechanisms to help people scale Airflow as elastically as possible, um, that's just a net win for Airflow in general, because it lowers the cost and sort of resource overhead of uh of Airflow.
• 18:42
  SPEAKER_03: Uh I think uh number three, something that we've heard from most customers is I just want to go talk to more and more things.
• 18:49
  SPEAKER_03: Uh so extending the operators sensors hooks uh within Airflow is is super critical.
• 18:57
  SPEAKER_02: Yeah.
• 18:58
  SPEAKER_02: So kind of jumping on some of those uh get in the weeds a little bit there.
• 19:02
  SPEAKER_02: Um I think it's probably kind of fascinating, like uh how you get Airflow scale more elastically in the context of your bigger cloud.
• 19:10
  SPEAKER_02: Um kind of like what are some ways you see that growing, right?
• 19:14
  SPEAKER_02: Do you see it being more just integration with Kubernetes, or uh is it gonna be some other uh some other way you can kind of scale out elastically like that?
• 19:22
  SPEAKER_03: Well, I think there's both.
• 19:23
  SPEAKER_03: I mean, there's not necessarily one right answer.
• 19:25
  SPEAKER_03: There might be the potential for multiple answers.
• 19:28
  SPEAKER_03: Um I'm gonna give you the perspective from sort of the composer, the composer standpoint.
• 19:33
  SPEAKER_03: So um there's there's always the potential that one solution might not work for every customer, which means that you have multiple solutions to solve a common problem and people can kind of pick and choose what's right for them.
• 19:43
  SPEAKER_03: And I think in general, that's a really good approach because if you give customers what they want, generally customers are happier um and you actually spend less work trying to force everybody in one um one narrow channel.
• 19:54
  SPEAKER_03: Um I think from the resource elasticity standpoint, um, you know, we've been looking heavily at the Kubernetes executor for uh Composer.
• 20:04
  SPEAKER_03: There's I think pros and cons to moving to that model, but it's something that's definitely on our radar, and that helps solve for some of the resource elasticity.
• 20:13
  SPEAKER_03: Um something else that we've we've really been thinking about is metadata persistence and sort of web server resources and how we can scale those uh as elastically as possible.
• 20:25
  SPEAKER_03: Um I think within Google Cloud specifically, there's other things that we continue to take a look at and how they might be practically usable uh within Composer and Airflow in general.
• 20:35
  SPEAKER_03: Uh so something like Cloud Run is a good example uh where you're sort of able to elastically uh scale out um containers uh very, very quickly at very low cost.
• 20:46
  SPEAKER_03: What might a world look like where we could build um airflow that uses something like Cloud Run?
• 20:51
  SPEAKER_03: So uh I think of in terms of timeline, we're looking at what can we do in the near term?
• 20:56
  SPEAKER_03: Um and I think the sort of Kubernetes executor is much more in that area.
• 20:59
  SPEAKER_03: And then when we look at the longer term, a lot of cloud services are moving towards offering services that are highly ephemeral, extremely low cost, um, but maybe they don't offer the same persistence or durability guarantees as a typical like VM or container service.
• 21:15
  SPEAKER_03: Uh so we're thinking through what the future looks like for us uh in the world where people want to use that.
• 21:20
  SPEAKER_03: I mean potentially allowing people to kind of pick and choose what makes most sense for them.
• 21:25
  SPEAKER_02: Yeah, yeah, I think you hit on a good point there, like uh giving people the option to pick and choose it generally makes most customers the happiest.
• 21:34
  SPEAKER_02: Uh we've seen ourselves like uh we have this Kubernetes executor on our platform, but also Celery, and uh there's like good reasons customers might choose to use one or the other based on kind of their use case at hand.
• 21:47
  SPEAKER_02: Um jumping into kind of uh what you said a little bit before about the web server resources and uh how it's gonna be making Airflow more efficient outside of task execution, uh could you jump into that a bit?
• 21:59
  SPEAKER_02: Uh kind of curious.
• 22:00
  SPEAKER_02: Hear how you all are thinking about that.
• 22:01
  SPEAKER_02: As that's something I know we we've had a lot of internal discussion about.
• 22:07
  SPEAKER_03: Yeah.
• 22:07
  SPEAKER_03: Yeah.
• 22:08
  SPEAKER_03: So there's I think two sides to this, and they're somewhat related.
• 22:11
  SPEAKER_03: Um and and worker like resource effectiveness is one side.
• 22:15
  SPEAKER_03: Um I'm gonna go into a direction that seems like I'm ignoring you and I'm not, or they're related, so I'm gonna sort of willfully go there.
• 22:23
  SPEAKER_03: Um and if I go too far, please pull me back.
• 22:26
  SPEAKER_03: Um but that's also sort of the consistency of the tooling.
• 22:29
  SPEAKER_03: Um and what I mean there is right now in Composer, people interact with the uh Airflow Web UI, which is awesome.
• 22:36
  SPEAKER_03: Um it is um very beneficial.
• 22:40
  SPEAKER_03: Generally, people like it quite a bit.
• 22:42
  SPEAKER_03: Um but there's also the problem that if you offer a cloud service and you link to something like an Airflow Web UI and it changes substantially, uh, you have to ask yourself the question of am I breaking customers by giving them this new version?
• 22:55
  SPEAKER_03: Or are customers going to break themselves uh unknowingly potentially by changing versions where you know options have changed or things have moved around?
• 23:04
  SPEAKER_03: Uh, because very small changes can cause really big impacts for large customers, especially.
• 23:10
  SPEAKER_03: Um we've done a lot of thinking about how we can both try to use what's available in uh open source, but also ensure that um things like the web UI are not going to break customers, but they're also fully accessible, because accessibility is actually a really big um point in how we design and roll out Google products.
• 23:30
  SPEAKER_03: So uh I think that in that context, we've sort of been thinking about what we might be able to do.
• 23:36
  SPEAKER_03: And then the other, going back to the resource efficiency, is just how can we scale resources as efficiently as possible so people don't over or under consume resources.
• 23:45
  SPEAKER_03: Um I think sort of on the technical side, we've been looking a lot at what we can do to help extend the Airflow uh REST API to make it possible to not um take airflow in a direction where we don't want to go and sort of introduce uh proprietary bits or uh duplicate functionality, um, but also build on something that is is open and potentially offer people um sort of like a uh UI or um command line tooling or other sort of interaction mechanisms which which uh we have a bit more fine-grained control over, um, but still totally are open and don't don't block or break the customer in any way.
• 24:28
  SPEAKER_03: Hopefully that's a fair answer to your question.
• 24:30
  SPEAKER_03: Um but it really boils down to the the we've been looking at what we can do in in terms of the the REST API to try and extend and grow that API to make Airflow a bit more adaptable to uh specific use cases.
• 24:43
  SPEAKER_02: Yeah, yeah, for sure.
• 24:44
  SPEAKER_02: I think uh that's also that's a general thing people in the community are pretty excited for.
• 24:48
  SPEAKER_02: Um uh once you start to like something, you want to use it through an API just generally, right?
• 24:53
  SPEAKER_02: Just tie it into other applications.
• 24:56
  SPEAKER_02: Uh and it's not really built for that without an API, but with an API you can really kind of see that ecosystem coming together a lot more.
• 25:03
  SPEAKER_03: Yeah, definitely.
• 25:04
  SPEAKER_03: And if you sort of uh thinking putting myself in the place of a typical cloud customer, uh they love the fact that we have composer and it's built on Airflow, but they also may expect that the UI they're shown to interact with Airflow and Composer looks like the UI they use for things like BigQuery or cloud storage or compute engine.
• 25:22
  SPEAKER_03: Um so the I think the to sort of echo the excitement of the REST API, the the REST API and going back to sort of desired Airflow 2 features is definitely I think something that we're we're actively trying to actually participate in.
• 25:36
  SPEAKER_03: Um I think that that that REST API allows uh Airflow to be adaptable to specific use cases, but also remain quite open so people have portable workflows that are based on a solid open source project.
• 25:52
  SPEAKER_00: Yeah, that definitely make that definitely makes a lot of sense.
• 25:54
  SPEAKER_00: Um I mean if you have the API kind of foundation in place, you can build whatever you want, whatever type of interface you want over it, right?
• 26:02
  SPEAKER_00: Going forward.
• 26:03
  SPEAKER_00: That's cool.
• 26:03
  SPEAKER_00: I kind of want to go back to what you were talking about a little bit earlier.
• 26:06
  SPEAKER_00: Sorry, I've been I've been a little bit quiet while you guys were going in on um the Kubexecutor and all that other stuff, but I kind of want to ask a quick question about kind of the RBAC stuff that you mentioned.
• 26:15
  SPEAKER_00: Um where do you see kind of RBAC speci more where specifically in there for a Airflow project do you see the RBAC functionality being extended to?
• 26:23
  SPEAKER_00: Um I know we kind of already have like the um airflow kind of um let's say team level uh read and write permissions, um DAG level read and write permissions as well.
• 26:35
  SPEAKER_00: And I I heard you mention briefly like task level.
• 26:37
  SPEAKER_00: Um so how granular do you see that getting?
• 26:40
  SPEAKER_03: So uh there's a this is a bit of an onion um and there's multiple layers.
• 26:45
  SPEAKER_03: Uh to answer your direct question, very granular, but uh that I think jumps over some of the other things that have been on our mind.
• 26:53
  SPEAKER_03: So uh if, for example, I have a workflow that is malicious and I want to steal data from the particular airflow environment where my workflow is running.
• 27:04
  SPEAKER_03: Um conceivably that could be possible.
• 27:07
  SPEAKER_03: Umce you give somebody access to an airflow environment, sort of our perspective has been, and what what we generally advise customers is if you give somebody access to an airflow environment or composer environment, uh you generally have to trust that they're not going to be a malicious actor.
• 27:21
  SPEAKER_03: Um, because you don't necessarily get uh isolation of workflows, uh, you also don't necessarily gate somebody off from uh being able to do something that you might not want them to do.
• 27:33
  SPEAKER_03: Um so from our perspective, we've offered customers the ability to add identity access management to environments to allow uh people to grant or deny them access.
• 27:43
  SPEAKER_03: But um from something like compute isolation, environment isolation, uh potentially the ability to like interact with the Airflow metadata database, um, there's a whole bunch of other security considerations that uh we're trying to think through because it's unfortunately it's not as easy as just adding role-based access control.
• 28:06
  SPEAKER_03: Uh there's a lot of additional security uh considerations that we're trying to think through to solve.
• 28:13
  SPEAKER_03: And I think they're all things that absolutely can and should be solved in core Airflow.
• 28:18
  SPEAKER_03: I have very little doubt that they eventually will be solved in Airflow Core.
• 28:23
  SPEAKER_03: Um, but it it becomes a really tricky case because we also don't want to give customers a false sense of security where they think that something is inherently um secure when in reality it's you're relying on the goodwill and trust of people that are creating and authoring DAGs.
• 28:39
  SPEAKER_03: Um so from our perspective, I I think fine-grained access control across all resources makes a lot of sense.
• 28:46
  SPEAKER_03: Um so workflows, tasks, connections, um, potentially things like scheduling.
• 28:52
  SPEAKER_03: Um but on the other hand, there's a whole bunch of added things that probably need to be part of what uh would constitute a super secure uh or sort of like industrial grade um security airflow environment.
• 29:05
  SPEAKER_03: And I think that's this is uh this is a probably one of the weaker points of Airflow, but it's still early days for Airflow.
• 29:12
  SPEAKER_03: Um I think I other open source projects have solved for security.
• 29:16
  SPEAKER_03: Uh, you know, when you look at like the Spark and Hadoop ecosystem, um they it they didn't immediately solve for security, but that came.
• 29:23
  SPEAKER_03: And now there's different security modes.
• 29:25
  SPEAKER_03: Um and I think that the same will probably happen for Airflow as well.
• 29:30
  SPEAKER_02: Yeah.
• 29:31
  SPEAKER_02: Yeah.
• 29:31
  SPEAKER_02: I think a lot of times when you uh especially when you're trying to get started or going fast, you have to kind of balance between security and usability.
• 29:38
  SPEAKER_02: Uh, because sometimes uh it's a little harder to like get started with as a new user end.
• 29:44
  SPEAKER_02: Um just jumping on that a little bit, like uh I think one of the good parts about an Airflow managed service is the idea that you don't have to think about your DAGs after you after what they're written, right?
• 29:57
  SPEAKER_02: They're just gonna run in an environment that you have some high-decured contents in.
• 30:01
  SPEAKER_02: How are you all thinking about kind of uh improving the user experience of actually getting those DAGs there?
• 30:07
  SPEAKER_02: Um I think Yeah, go ahead.
• 30:09
  SPEAKER_02: Yeah.
• 30:09
  SPEAKER_03: Yeah, no, that's a that's a very good question.
• 30:12
  SPEAKER_03: So workflows are code, right?
• 30:14
  SPEAKER_03: Like I have a workflow, it's it's code.
• 30:16
  SPEAKER_03: Uh you know, it it probably makes sense for Airflow to go look at wherever I keep my code um or to use what mechanisms I use to keep my code.
• 30:26
  SPEAKER_03: Um the the lowest hanging fruit on this is right now inside of Composer, people uh throw um or add their DAGs to a cloud storage bucket, and the Airflow environment will, it continually scans that cloud storage bucket for new and updated DAGs and um adds them to Airflow.
• 30:44
  SPEAKER_03: Um from our perspective, it makes a lot of sense to do something like establish a trust relationship with a Git repository.
• 30:51
  SPEAKER_03: And instead, when something is added to the Git repository that I trust to hold all of my workflows, um, I immediately just add them to uh to my Airflow environment.
• 31:01
  SPEAKER_03: Um, you know, obviously to pay based on what settings you have and whatnot.
• 31:05
  SPEAKER_03: Um the the challenge there is establishing that trust relationship and maintaining the trust relationship.
• 31:10
  SPEAKER_03: But um, you know, that I think that something like a git sync is, I think it would help out.
• 31:16
  SPEAKER_03: There's uh I think like two or three things that would really help out airflow developer productivity.
• 31:20
  SPEAKER_03: I think the like integration with a source control system would be probably like number two on the list.
• 31:28
  SPEAKER_02: Who would be number one on the list?
• 31:31
  SPEAKER_03: Yeah, I I guess I did set that one up a little bit.
• 31:34
  SPEAKER_03: Um I think it's it's hard to test workflows end-to-end for a lot of reasons.
• 31:38
  SPEAKER_03: Um if you're calling external services, sometimes those services don't support mock interactions.
• 31:44
  SPEAKER_03: Um so if I develop a workflow, I either have to do it on like a subset of services or a test set of services, or I just kind of have to hope it's gonna work.
• 31:52
  SPEAKER_03: Um I think the ability to test, kind of iteratively test workflows would be a big help to Airflow developers.
• 32:01
  SPEAKER_03: Because right now, from what we've seen, there is an element of I go build a workflow, I run it and make sure it's not broken and kind of go from there.
• 32:11
  SPEAKER_03: Uh that's not terrible, but there's probably a better development process if we could help people do things like um, you know, for example, I'm thinking of Google services in particular here, but it's probably true for a lot of cloud services.
• 32:23
  SPEAKER_03: Uh if I want to go have a workflow that creates a bunch of VMs and runs some queries, it would be ideal if I could mock out those interactions and make sure that they could succeed and not actually go do it.
• 32:33
  SPEAKER_03: That way my testing doesn't uh you know consume quota, cost money, um, sort of all of that.
• 32:39
  SPEAKER_03: So I think the ability to iteratively mock and test out um uh workflows uh would be would probably be number one from what we've seen.
• 32:48
  SPEAKER_02: Yeah.
• 32:50
  SPEAKER_02: Sometimes it feels like uh it's almost less of sometimes it can be less of a function of your code and more of the a function of the test system that you're testing on.
• 32:59
  SPEAKER_02: And that's just inherently a hard thing to really test end-to-end.
• 33:02
  SPEAKER_02: Yeah.
• 33:04
  SPEAKER_00: Cool.
• 33:04
  SPEAKER_00: So I'll uh James, I want to step back a little bit here.
• 33:06
  SPEAKER_00: Um I kind of feel like we kind of jumped jumped over um a question that I wanted to ask a little bit earlier, but kind of kind of personally for you, um what was your experience like before you kind of dove into the world of Airflow and just like general product management around Airflow?
• 33:22
  SPEAKER_00: Um were you with kind of like a similar product in the open source data space or um like what what was your kind of a personal adoption like when picking all this stuff up?
• 33:31
  SPEAKER_03: Yeah, um so Airflow, so launching Composer was um a project I go on after I had launched uh a managed Spark and Hadoop service inside of Google Cloud.
• 33:43
  SPEAKER_03: So um and and sort of my background has been very um open source uh focused in the data processing space.
• 33:50
  SPEAKER_03: So um I already have a strong opinion that open source uh is is awesome for a whole lot of reasons.
• 33:58
  SPEAKER_03: And from wearing my hat as an employee of a cloud vendor, I think there's a very strong relationship that open source can have um with with clouds to sort of partner the growth of an ecosystem with just running stuff at scale really well.
• 34:13
  SPEAKER_03: Um so I think the the wearing that hat, I was already primed to sort of look at new and interesting candidates for uh for open source.
• 34:21
  SPEAKER_03: Um I really started to first notice Airflow.
• 34:25
  SPEAKER_03: Uh Airflow, I mean, back in the day that definitely had a lot of, I don't know, it's fair to say it was a kind of data uh specific endeavor, but it had a lot of ties to the data analytics community.
• 34:38
  SPEAKER_03: Um so kind of wearing the Spark Nadope ecosystem hat, um, we definitely saw a lot of um at least prototyping around with Airflow, which brought it onto um our radar.
• 34:50
  SPEAKER_03: Um I think since then we've seen Airflow used much more outside of data analytics.
• 34:54
  SPEAKER_03: Um we see it used for a whole lot of really interesting use cases now.
• 34:58
  SPEAKER_03: But um that was that was sort of the how it came on our radar.
• 35:02
  SPEAKER_00: Cool, cool.
• 35:03
  SPEAKER_00: So so did do you want to talk briefly about kind of the differences and maybe similarities that you see between um kind of being involved with the Airflow project and kind of being involved with Spark at Hadoop um in your previous previous gig?
• 35:14
  SPEAKER_03: Yeah, uh yeah, no, I'm I'm thankfully still involved.
• 35:18
  SPEAKER_03: Um so my team um is uh also um is responsible for our managed spark at the dupe service.
• 35:24
  SPEAKER_03: So it's sort of we have a lot of our open source um focus within within a team, within our open data analytics team, within Google Cloud.
• 35:32
  SPEAKER_03: So um that still continues, and and thankfully I'm still happy to be to be part of it.
• 35:37
  SPEAKER_03: Um so I think uh and I can you I I apologize, can you reask your question?
• 35:42
  SPEAKER_03: I totally got lost on a tangent.
• 35:44
  SPEAKER_00: Yeah, sure, sure.
• 35:44
  SPEAKER_00: I just want to know, you know, I'm I'm just really kind of curious to know how what similarities and differences you see between the communities and what the similarities and differences are managing the kind of um commercialization of the project as well.
• 35:58
  SPEAKER_03: Yeah, so similarities and differences.
• 36:01
  SPEAKER_03: Um, you know, I think the the similarities are generally open source.
• 36:06
  SPEAKER_03: Uh uh the similarities revolve around a lot of open source.
• 36:09
  SPEAKER_03: So it open source generally is I think stand on a soapbox a little bit.
• 36:15
  SPEAKER_03: It represents a contract with the user that what they're using is not going to break them or lock them in.
• 36:22
  SPEAKER_03: It represents that uh they're probably going to receive some continued benefit performer momentum.
• 36:28
  SPEAKER_03: Um it represents to them that if they want to peek and see how things are running, uh, it's entirely possible to do that.
• 36:35
  SPEAKER_03: Um so I think the the similarities are very much tied around those, where open source is seen as is awesome for a lot of really important reasons.
• 36:44
  SPEAKER_03: Um the big differences are really around the use cases for something like Spark and Hadoop.
• 36:49
  SPEAKER_03: Uh, a lot of it is really very specifically data and analytics focused.
• 36:53
  SPEAKER_03: Uh much to our surprise, what we've seen from Airflow is a lot of the use cases we're seeing.
• 36:59
  SPEAKER_03: We're seeing use cases for CICD and general DevOps.
• 37:03
  SPEAKER_03: And there's definitely data and analytics use cases, but we're also seeing um alerting and paging.
• 37:09
  SPEAKER_03: We're seeing a whole lot of really different use cases that are well outside of the data analytics space, um, which is interesting from our perspective for Composer because it sits within the big data umbrella of Google Cloud.
• 37:21
  SPEAKER_03: Uh, but a lot of uh what we see in terms of how Airflow is being used is is completely outside the realm of big data, which actually is just really great, great to see.
• 37:31
  SPEAKER_03: Because I think it means the project is maturing and growing um its uh its user base.
• 37:36
  SPEAKER_00: Yeah, well, no, that would that that was great.
• 37:38
  SPEAKER_00: That was kind of um a really great kind of high-level summary of of what we were looking for here.
• 37:42
  SPEAKER_00: Um Viraj, do you have anything else?
• 37:44
  SPEAKER_00: I don't think I have anything else on my list over here.
• 37:46
  SPEAKER_02: No, I think I think I'm uh I think I'm all good here.
• 37:49
  SPEAKER_00: James, anything else you want to talk about or plug while we have you here?
• 37:54
  SPEAKER_03: Uh I mean I just want to uh plug the Airflow community.
• 37:57
  SPEAKER_03: I mean, Airflow is an awesome, uh, awesome project.
• 38:00
  SPEAKER_03: Uh we've seen a lot of really interesting use cases and a lot of people be really happy with uh a lot of people have been very happy with their decision to use Airflow.
• 38:08
  SPEAKER_03: So if I do want to plug one thing, it's it's really the the core of the open source project.
• 38:12
  SPEAKER_03: Um and if people are who are listening and they're curious or they are thinking about trying Airflow, I definitely want to encourage them to try it out, uh, get involved um and um and share their thoughts with the Airflow community.
• 38:26
  SPEAKER_00: Awesome.
• 38:26
  SPEAKER_00: Well, that's great.
• 38:27
  SPEAKER_00: Yeah, yeah.
• 38:27
  SPEAKER_00: I mean, uh we we feel the same way.
• 38:29
  SPEAKER_00: So um generally all very excited about you know um the ration of the project and uh how everything's going.
• 38:35
  SPEAKER_00: So thanks so much for the time today.
• 38:36
  SPEAKER_00: It was uh great to meet you and great to chat.
• 38:39
  SPEAKER_01: Likewise, thank you for having me.
• 38:40
  SPEAKER_01: I really appreciate it.
• 38:45
  SPEAKER_00: Thanks to James for coming on the show this week.
• 38:47
  SPEAKER_00: We enjoyed hearing his ideas about the future of Airflow and Composer and look forward to speaking with him again soon.
• 38:53
  SPEAKER_00: Thanks for listening, and please don't hesitate to shoot me an email at pete at astronomer.io if you have any podcast feedback or would like to hear us cover any specific topics in our next episode.
• 39:01
  SPEAKER_00: Until next time.