Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing

Transcript

Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: As a result, we have lift-specific login and left-specific event streaming that we insert into the operator.
• 0:06
  SPEAKER_03: So we build the customized operator on top of the open source operators and add the leaf-specific code inside.
• 0:14
  SPEAKER_03: And we also have control and governance on the operator because we don't want everything to develop like the same operator.
• 0:22
  SPEAKER_03: And that will just result in a lot of duplicate operators doing the same thing.
• 0:27
  SPEAKER_03: So that's something that we don't really don't want to see.
• 0:34
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:45
  SPEAKER_02: Hey Airflow community.
• 1:15
  SPEAKER_01: Hi everyone.
• 1:16
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:18
  SPEAKER_01: I'm your host, Kenton Danis, and today I'm joined by Yoonhao Ching, who is a software engineer at Lyft.
• 1:24
  SPEAKER_01: Uh Yunhao, thanks for being here.
• 1:26
  SPEAKER_01: How are you doing today?
• 1:27
  SPEAKER_01: Good, good.
• 1:27
  SPEAKER_01: How are you?
• 1:28
  SPEAKER_01: Yeah, yeah, I'm great.
• 1:29
  SPEAKER_01: We're recording here on a Friday, a summer Friday, so always a good time.
• 1:34
  SPEAKER_01: But yeah, we appreciate you being on.
• 1:36
  SPEAKER_01: And for our listeners, uh, we'll go ahead and dive in.
• 1:39
  SPEAKER_01: And for today's podcast, we're gonna discuss how Lyft's Airflow team manages airflow for many teams of internal users.
• 1:46
  SPEAKER_01: So yeah, I'm really excited to dig into this.
• 1:50
  SPEAKER_01: And you know how so to start with some background, I think Lyft is a very household name, um, at least here in the US.
• 1:56
  SPEAKER_01: So I'd imagine most people are familiar with the company.
• 2:00
  SPEAKER_01: But just in case, can you maybe describe, you know, at a high level your business model and then probably more importantly, what your team does to support it?
• 2:08
  SPEAKER_03: Yeah, absolutely.
• 2:09
  SPEAKER_03: So yeah, Lyft is a mobile app and we aim to connect drivers and passengers.
• 2:15
  SPEAKER_03: And besides that, we also offer services like bike and scuba rentals and also car rentals in some cities.
• 2:22
  SPEAKER_03: And we are currently mostly in the US market.
• 2:25
  SPEAKER_03: However, we are also expanding into other markets.
• 2:29
  SPEAKER_03: I think we recently acquired a company in the EU, so we are definitely looking into other places in the world as well.
• 2:35
  SPEAKER_03: And what my team does specifically is that we support a large number of ETL data pipelines that supports pricing, marketplace health, fraud detection, driver rider analysis, and a lot of other customer teams.
• 2:51
  SPEAKER_03: And we provide the platform support to ensure that the teams can write efficient and maintainable DAX.
• 2:57
  SPEAKER_03: And for important teams, we also work directly with them, optimizing their DAX for the best performance.
• 3:04
  SPEAKER_01: Okay, yeah, that's very interesting.
• 3:06
  SPEAKER_01: I actually did not know that Lyft did car rental.
• 3:08
  SPEAKER_01: So as I say, household name, but even I learned something new.
• 3:11
  SPEAKER_01: So that's really cool.
• 3:13
  SPEAKER_01: I'm curious if you're able to share when you say so you're managing Airflow as a platform for all these internal teams with their ETL and other use cases.
• 3:23
  SPEAKER_01: What is kind of the scale you're dealing with?
• 3:25
  SPEAKER_01: How many different internal teams or users do you provide the platform for?
• 3:30
  SPEAKER_03: We are dealing with about 30 to 50 user teams on a regular basis.
• 3:35
  SPEAKER_03: And we have a huge amount of ETL pipelines that is on the solder scale.
• 3:41
  SPEAKER_01: Yeah, yeah, that's huge.
• 3:42
  SPEAKER_01: 30 to 50 teams.
• 3:44
  SPEAKER_01: That's a lot of people.
• 3:45
  SPEAKER_01: So it makes sense that they need a really strong platform.
• 3:48
  SPEAKER_01: So digging into that a little bit more, so maintaining an Airflow platform for so many different teams, and obviously that brings a lot of data pipelines that they have to run.
• 3:59
  SPEAKER_01: Tell me a little bit more about how you do that.
• 4:01
  SPEAKER_01: What sort of considerations are you taking into account to make Airflow work at that scale?
• 4:06
  SPEAKER_03: Yeah.
• 4:06
  SPEAKER_03: So we manage all the stuff from dependency management to also customize operator management, documentations, and also suggestions on what's the best way to write a DAC and also optimization there.
• 4:20
  SPEAKER_03: And we also develop and maintain the local DAF development environment for Airflow.
• 4:26
  SPEAKER_03: And we also manage like the Airflow versions for our customers.
• 4:31
  SPEAKER_01: Okay, interesting.
• 4:32
  SPEAKER_01: So you're both managing the Airflow platform itself as well as helping the users implement their pipeline.
• 4:40
  SPEAKER_01: So I want to talk more about both of those aspects.
• 4:42
  SPEAKER_01: Maybe we can start with the Airflow platform itself.
• 4:45
  SPEAKER_01: So you mentioned just running Airflow and managing the versions and things like that.
• 4:49
  SPEAKER_01: How do you do that?
• 4:50
  SPEAKER_01: Are you using a managed Airflow service or what does that look like at Lyft?
• 4:54
  SPEAKER_03: Yeah, we used to self-host all the pods on our Kubernetes service.
• 4:59
  SPEAKER_03: And we recently migrated to um Astral.
• 5:02
  SPEAKER_03: So right now, all our pipeline actually are running on the Astral environment.
• 5:07
  SPEAKER_03: So we can care less about the infrastructure maintenance and spend more time working with customers directly, improving their experience.
• 5:16
  SPEAKER_01: Yeah, that's great.
• 5:17
  SPEAKER_01: Definitely.
• 5:18
  SPEAKER_01: I would imagine running Airflow on Kubernetes yourself for that scale is not easy.
• 5:23
  SPEAKER_01: And that is it.
• 5:24
  SPEAKER_01: Definitely one of the goals of Astro in general is to allow people to focus more on kind of the best practices aspect of things so that they don't have to worry so much about running Airflow.
• 5:35
  SPEAKER_01: But in terms of, you know, kind of maintaining Airflow's infrastructure, going a little bit beyond that, since you don't have to worry so much about the day-to-day maintenance piece of that.
• 5:45
  SPEAKER_01: You mentioned earlier helping teams add features like custom operators and things like that.
• 5:51
  SPEAKER_01: Tell me a little bit more about what that looks like and why you need to, you know, manage that at kind of the platform engineering level.
• 5:59
  SPEAKER_03: So for operators, a lot of the time, what we are doing with the operator is that we are moving data from one data source to another.
• 6:06
  SPEAKER_03: And there are a lot of like the open source version of those operators.
• 6:10
  SPEAKER_03: However, we need to do it in such a way that we kind of record the event.
• 6:15
  SPEAKER_03: And as a result, we have like lift-specific logging and left-specific event streaming that we insert into the operator.
• 6:23
  SPEAKER_03: So we build the customized operator on top of the open source operators and add the lift-specific uh code inside.
• 6:31
  SPEAKER_03: Yeah, and we also have control and governance on the operator because we don't want everything to develop like the same operator.
• 6:39
  SPEAKER_03: And that will just result in a lot of duplicate operators doing the same thing.
• 6:43
  SPEAKER_03: So that's something that we don't really don't want to see.
• 6:46
  SPEAKER_03: And that's why we have governance on the customized operator.
• 6:50
  SPEAKER_01: Yeah, interesting.
• 6:51
  SPEAKER_01: And so when you're taking existing operators that are part of kind of the provider package ecosystem and you're adding customization to them, you said in terms of adding like customized logging.
• 7:03
  SPEAKER_01: What is your reason for doing that?
• 7:05
  SPEAKER_01: How does that help you?
• 7:07
  SPEAKER_03: So there are other teams that want to understand the data flow.
• 7:11
  SPEAKER_03: And we are required to record the event and also the log.
• 7:15
  SPEAKER_03: So it's more of a requirement instead of us needing an extra set of logs for debugging.
• 7:21
  SPEAKER_01: Okay, I've got it.
• 7:22
  SPEAKER_01: So just helps you make everything more reliable, basically.
• 7:25
  SPEAKER_01: Yeah.
• 7:26
  SPEAKER_01: Yeah, cool.
• 7:27
  SPEAKER_01: And then obviously, for you know, implementing custom operators that don't exist in the provider ecosystem, totally makes sense that you're saving teams a lot of time by maintaining kind of the one standardized version of those so that people aren't reinventing the wheel all the time.
• 7:42
  SPEAKER_01: I'm curious what again, just at the scale of that, how many of these operators do you have to maintain?
• 7:47
  SPEAKER_01: Is that a lot of work for your team to keep up this library?
• 7:51
  SPEAKER_03: Yeah, we have thousands of customized operators.
• 7:54
  SPEAKER_03: We are actually trying hard to reduce that amount, and we're also trying hard to consolidate them and also even think of ways to prevent people from easily create a customized operator in the future.
• 8:08
  SPEAKER_01: Yeah, that totally makes sense.
• 8:09
  SPEAKER_01: Thousands is a lot.
• 8:10
  SPEAKER_01: I don't just hundreds, yeah.
• 8:13
  SPEAKER_01: Oh, hundreds.
• 8:14
  SPEAKER_01: Okay.
• 8:14
  SPEAKER_01: Still a lot.
• 8:15
  SPEAKER_01: That's a lot of work to upkeep.
• 8:17
  SPEAKER_01: I think this is something that I've heard about with lots of companies that Astronomer has worked with in the past, is just instituting kind of some guardrails on which operators people can use.
• 8:27
  SPEAKER_01: And it's actually something that we have in the Astro platform in terms of observability, is just insight into which operators are being used and where across different airflow deployments.
• 8:37
  SPEAKER_01: And I think the reason is exactly what you're describing is for some companies the sprawl can get really big and then it just gets more difficult to manage and can be a risk in terms of reliability.
• 8:47
  SPEAKER_01: So we see lots of companies again instituting kind of rules on which operators you can use, which I'd imagine for the developers is sometimes limiting.
• 8:55
  SPEAKER_01: But if you have a great platform team in place that is helping them implement how to use those in their pipelines, I'm sure it works pretty well.
• 9:01
  SPEAKER_01: That's great.
• 9:02
  SPEAKER_01: Okay.
• 9:03
  SPEAKER_01: And then you mentioned a little bit earlier in additioning to in addition to the work of you know managing airflow and kind of all of the building blocks of it for all of these internal teams.
• 9:14
  SPEAKER_01: You're also helping users optimize their workflows.
• 9:17
  SPEAKER_01: So tell me a little bit more about what that looks like.
• 9:20
  SPEAKER_03: Yeah, so we want our customer stack to be in the most efficient and also in the clearest way.
• 9:27
  SPEAKER_03: And sometimes we see bad practices.
• 9:29
  SPEAKER_03: For example, some people will have extensive numbers of tasks in their DAC.
• 9:34
  SPEAKER_03: And sometimes people will set up really complex notifier system, although they could do it in like one line with the native notifier system.
• 9:42
  SPEAKER_03: Yeah, so when we spot any of this, we will work with the customer directly to optimize their DAX and their uh like a DAC repository so that their DAX are easily like maintainable and debuggable in the future.
• 9:59
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 10:08
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:20
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 10:29
  SPEAKER_01: Complex notifiers is an interesting one.
• 10:31
  SPEAKER_01: So I haven't actually seen that myself, but that totally makes sense that there maybe people just aren't aware of the features that are available that are an easier way to implement something.
• 10:41
  SPEAKER_01: So they're kind of writing their own version.
• 10:43
  SPEAKER_01: Is that typically what you're seeing?
• 10:45
  SPEAKER_03: Yeah, so if you use the open source airflow, you can define like fall and then send an email to something.
• 10:52
  SPEAKER_03: But sometimes people are not aware of that.
• 10:54
  SPEAKER_03: So they will set up an extra like Python function to call email and then send an email to their email, and then they will call that Python function when the DAC fail.
• 11:04
  SPEAKER_03: So they can reduce that amount of the code with just one line notifier.
• 11:09
  SPEAKER_03: But I think that not everyone is a Airflow expert and we understand that.
• 11:13
  SPEAKER_01: Right.
• 11:14
  SPEAKER_01: So you're trying to just implement kind of best practice use of existing airflow features again, so people aren't reinventing the wheel, which I imagine saves them a lot of time in terms of development, but also makes it easier to maintain over time.
• 11:26
  SPEAKER_01: So when you have, you know, developers change teams or new folks come on, they're not trying to, you know, do everything by themselves.
• 11:33
  SPEAKER_03: Yeah, that's a good summary.
• 11:34
  SPEAKER_01: Yeah, yeah, great.
• 11:36
  SPEAKER_01: And then I'm also curious, just your mechanism for reviewing all of this code.
• 11:42
  SPEAKER_01: Are you like reviewing PRs that are going through a CICD process before they get deployed?
• 11:48
  SPEAKER_01: I mean, you're managing Airflow for many different internal teams.
• 11:51
  SPEAKER_01: I'd imagine it's a lot of DAG code to be looking at.
• 11:54
  SPEAKER_01: How do you do that?
• 11:56
  SPEAKER_03: Yeah, so usually when we spot a team is always reporting errors to us, their DACs always fail and they report issue with their, for example, notifier system.
• 12:07
  SPEAKER_03: We will understand that there may be something that's not the best way in the team stack, and then we'll take some time to review the team stack and give suggestions based on our observation.
• 12:19
  SPEAKER_03: We kind of act after after we spot something is not right.
• 12:24
  SPEAKER_01: Yeah, gotcha.
• 12:25
  SPEAKER_01: Okay.
• 12:25
  SPEAKER_01: Makes sense.
• 12:26
  SPEAKER_01: So more reactive when problems are happening.
• 12:29
  SPEAKER_01: I'm curious if you have thought about ways of making that more proactive.
• 12:34
  SPEAKER_01: I know that's something an astronomer has kind of had a focus on recently with our observe platform and others, is just this notion of kind of proactive alerting or observability around your pipeline so that you can identify issues before they happen, which may or may not work for you know what you've written in the code specifically, but it is something that I think there's a lot of focus on of kind of helping implement best practices and identifying things like that in the DAG code.
• 13:00
  SPEAKER_01: Is that something that your team is looking into?
• 13:03
  SPEAKER_03: Yeah, that's definitely a goal for my team.
• 13:06
  SPEAKER_03: So as an example, we want to implement on the CICD level that people can't really create customized operator without our approval.
• 13:14
  SPEAKER_03: So that's that's a way in us trying to proactively prevent people from abusing the system.
• 13:21
  SPEAKER_01: Yeah, yeah, totally, totally makes sense.
• 13:24
  SPEAKER_01: Very cool.
• 13:24
  SPEAKER_01: And then I uh we touched a little bit on, you know, just from the Airflow platform side, you're managing the Airflow versions that folks are using in their deployments.
• 13:34
  SPEAKER_01: What is your process for helping teams upgrade, or do you have rules in place on which versions we can use?
• 13:42
  SPEAKER_01: I know not every Airflow version is supported on Astro, and we always have support for, you know, the new versions right away.
• 13:48
  SPEAKER_01: So are you helping teams upgrade to new versions of Airflow?
• 13:52
  SPEAKER_01: What does that look like?
• 13:53
  SPEAKER_03: Yeah, so we have different deployments on Astro for a different team.
• 13:58
  SPEAKER_03: And right now they are on the later versions of Airflow 2, and we do aim to upgrade to Airflow 3, but not like right now.
• 14:08
  SPEAKER_03: Yeah, so to help them, we kind of need to we there are preparation work for us to do to upload upgrade from like two to three.
• 14:16
  SPEAKER_03: So we will be the ones reading the docs to understand what's the difference and maybe creating a small environment to test a couple of the deck for upgrade.
• 14:25
  SPEAKER_03: So the kind of work we do is to do all the proper.
• 14:28
  SPEAKER_03: So when we actually do the like one button upgrade, nothing will be broken.
• 14:34
  SPEAKER_03: And even if anything goes wrong, we understand clearly how to easily move it back to the lower version.
• 14:42
  SPEAKER_03: So we're doing preparation work for the future upgrade.
• 14:46
  SPEAKER_01: Got it.
• 14:46
  SPEAKER_01: That's uh that's great that you're able to help teams manage that.
• 14:49
  SPEAKER_01: And yeah, of course, on Astra, having the ability to roll back to a previous version if something goes wrong is really helpful.
• 14:56
  SPEAKER_01: Looking at you know Airflow 3 as you're kind of digging into what the upgrade process looks like and how you might move teams there.
• 15:02
  SPEAKER_01: Eventually, it's still pretty early days for a lot of companies.
• 15:05
  SPEAKER_01: Upgrading for a new major version is obviously a bigger undertaking than like a minor version change.
• 15:11
  SPEAKER_01: But for Airflow 3, are you familiar with the new release?
• 15:16
  SPEAKER_01: Are there any features that you're looking forward to helping the teams that you work with implement?
• 15:23
  SPEAKER_03: Yeah, one thing I personally am really interested in is the asset-driven scheduling.
• 15:29
  SPEAKER_03: So I know that the dataset is introduced in Airflow 2.
• 15:33
  SPEAKER_03: However, looking at the code and the documents online, I think that the underlying logic is still the event-driven scheduling.
• 15:40
  SPEAKER_03: And I heard that in Airflow 3, the assets-driven um scheduling is more to the like actual data-driven scheduling.
• 15:49
  SPEAKER_03: So I'm really excited to see how that will work out.
• 15:52
  SPEAKER_03: And I personally feel like that's a better way of scheduling DAC, or it could be a better way for some DACs.
• 15:59
  SPEAKER_03: And I see that in the future we will switch from scheduling DAC at a time to scheduling DAC when the upstream dependency is meet.
• 16:09
  SPEAKER_01: Yeah, yeah, absolutely.
• 16:11
  SPEAKER_01: I think I love to hear that assets are such an exciting feature.
• 16:14
  SPEAKER_01: And as you mentioned, that kind of the next evolution of data sets from Airflow 2.
• 16:20
  SPEAKER_01: So it gives you much more flexibility in terms of scheduling.
• 16:23
  SPEAKER_01: I've also already seen folks using assets just for writing their DAGs, kind of a different methodology for writing compared to tasks, but it often results in less code or at least less boilerplate code.
• 16:34
  SPEAKER_01: So for teams like Lyft, as you're describing, where, you know, managing it at such a huge scale, you have to care a lot about the efficiency of what people are doing.
• 16:45
  SPEAKER_01: I think assets will be a very helpful feature.
• 16:47
  SPEAKER_01: And of course, the event-driven scheduling piece of it is only going to grow, you know, in future airflow versions for that pattern of, you know, waiting for something to happen in an external system and then triggering the DAG based on it.
• 16:59
  SPEAKER_01: So I'm glad to hear that you folks are looking into that.
• 17:02
  SPEAKER_01: Cool.
• 17:03
  SPEAKER_01: All right.
• 17:04
  SPEAKER_01: Well, you know, thank you so much for going through all of that.
• 17:06
  SPEAKER_01: Super interesting to hear what the airflow implementation looks like at a big company like Lyft.
• 17:13
  SPEAKER_01: Uh, I always like to end our episodes with a standard final question, and that is what would you most like to see from the Airflow project in the near future?
• 17:21
  SPEAKER_03: There are actually two things I really want to see in Airflow in the future.
• 17:25
  SPEAKER_03: So the first one is I'm really curious how Gen AI can be a part of like Airflow.
• 17:30
  SPEAKER_03: I can imagine in the future, maybe people will just write a prompt and we can use Gen AI to turn that into a DAC.
• 17:38
  SPEAKER_03: So I see the potential of Gen AI both in terms of the um DAC development and also DAC optimization.
• 17:45
  SPEAKER_03: So that's one area I think I really want to see that happening.
• 17:49
  SPEAKER_03: Another one I think will be interesting will be semantic search across DACs.
• 17:54
  SPEAKER_03: So let's say if I am developing a new DAC and I want to search to see if there's any, there's another DAC that's kind of like do similar stuff, and then I can just take that as a simple.
• 18:05
  SPEAKER_03: Yeah, if I can kind of do the context or semantic search across the deck, I think that will be super cool.
• 18:12
  SPEAKER_01: Yeah, yeah, that's a really interesting and both really good examples of how AI can help in the context of airflow.
• 18:19
  SPEAKER_01: I think for to your first point about, you know, using prompts to kind of generate the DAG code itself, that is definitely something that Astronomer uh has our eye on, and we'll have, you know, some exciting updates hopefully soon in this space.
• 18:34
  SPEAKER_01: And yeah, I think the surge, you know, kind of comes along with that, the idea of AI being built into kind of your airflow instance such that it is aware of the context of what is happening in all of your pipelines is helpful for both generating new DAG code and as you said, just figuring out what is in there when you've got hundreds or thousands of these things.
• 18:55
  SPEAKER_01: So yeah, those are both really good answers.
• 18:57
  SPEAKER_01: I love that.
• 18:58
  SPEAKER_01: Okay.
• 18:58
  SPEAKER_01: Well, you know, thank you so much for joining me today.
• 19:01
  SPEAKER_01: It's been really great speaking with you.
• 19:02
  SPEAKER_01: To close, what is the best way for folks to get in touch with you?
• 19:05
  SPEAKER_03: Yeah, you can find me a linking and message me if you have any question.
• 19:10
  SPEAKER_01: Okay, great.
• 19:11
  SPEAKER_01: Well, thank you again.
• 19:12
  SPEAKER_01: Thank you.
• 19:16
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 19:22
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 19:26
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.