Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building self-service Dags with GitHub Actions at Lyft

Transcript

Building self-service Dags with GitHub Actions at Lyft · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:28
  SPEAKER_01: Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
• 0:35
  SPEAKER_01: Now let's get into the show.
• 0:38
  SPEAKER_00: Okay, hi everyone.
• 0:39
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:41
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Ken Obata, who is a senior software engineer at Lyft.
• 0:47
  SPEAKER_00: Ken, how are you doing?
• 0:48
  SPEAKER_02: Yeah, thanks for having me.
• 0:50
  SPEAKER_02: Um it's it's uh really nice to see you.
• 0:53
  SPEAKER_02: Um and yeah, I'm very excited to have this uh discussion with you today.
• 0:58
  SPEAKER_00: Yeah, I'm excited as well.
• 0:59
  SPEAKER_00: Thanks so much for joining.
• 1:01
  SPEAKER_00: Uh for our listeners for today's podcast, we're gonna discuss how Lyft has implemented self-service tags with an event-driven design for GitHub Actions and Airflow.
• 1:11
  SPEAKER_00: Uh I always like to start with some background.
• 1:13
  SPEAKER_00: In this particular case, I don't think we need to do any background for the company, uh, probably pretty well known.
• 1:19
  SPEAKER_00: And I think we'll also make uh our discussion today very interesting to our listeners.
• 1:25
  SPEAKER_00: I know it's always fascinating to get to kind of peek behind the curtain at really large, you know, well-known like household name companies and how they do things.
• 1:33
  SPEAKER_00: Uh, but maybe start by just telling me a little bit about your team at Lyft and what your team does.
• 1:38
  SPEAKER_02: Yeah, sure.
• 1:39
  SPEAKER_02: Um yeah, so I'm from Marketplace and especially from driver organization.
• 1:45
  SPEAKER_02: Um, you know, it's a Right Share company, so we deal with market balance, uh, supply and demand.
• 1:51
  SPEAKER_02: That's the kind of like whole purpose of Marketplace.
• 1:54
  SPEAKER_02: And especially I uh work for drivers.
• 1:58
  SPEAKER_02: Um, especially uh around driver pay, dynamic driver pay is where um yeah, my responsibility uh aligns.
• 2:08
  SPEAKER_00: Okay, got it.
• 2:09
  SPEAKER_00: And how is your team's work different from the data infra team that runs Airflow?
• 2:17
  SPEAKER_02: Yeah, so um so Lift has like two big org um as a big picture.
• 2:23
  SPEAKER_02: We have like one side from uh infrastructure org, and another is the product org, and the product org is where I come from.
• 2:32
  SPEAKER_02: Um we kind of like spread our responsibility um in terms of airflow.
• 2:38
  SPEAKER_02: So infra org is responsible for, as the name suggests, uh they're responsible for you know scaling of airflow, version control, you know, being bridged to astronomer.
• 2:49
  SPEAKER_02: Um and on the other hand, product side uh is more responsible for actually writing the DAGs uh on a daily basis.
• 2:56
  SPEAKER_02: We create core data set and maintaining table metadata.
• 3:01
  SPEAKER_02: Nowadays, uh we you know have we started using more uh crowd code and agents, so maintaining uh you know table definition semantics metadata, these like business context uh is one of our uh priority as well.
• 3:15
  SPEAKER_02: And then of course we orchestrate uh everything, uh you know, queries, uh MLOPs, uh, yeah, things like that.
• 3:24
  SPEAKER_00: Yeah, makes sense.
• 3:25
  SPEAKER_00: I think a pretty common pattern I've seen at larger companies where they end up having more of a platform team that runs kind of the infracide, at least of airflow uh or other data pipeline work.
• 3:36
  SPEAKER_00: And then that's separate from the people who are actually implementing the pipelines themselves.
• 3:40
  SPEAKER_00: So yeah, that's interesting.
• 3:42
  SPEAKER_00: And you now are a software engineer, but I understand you also have a background in data engineering.
• 3:49
  SPEAKER_00: Tell me a little bit about that and how it's led you to your current role.
• 3:52
  SPEAKER_02: Yeah.
• 3:53
  SPEAKER_02: Yeah.
• 3:53
  SPEAKER_02: So um I've been uh working as a data engineer more than eight years now.
• 3:59
  SPEAKER_02: Um I uh work for you know companies um, you know, some are some larger larger companies like KMG and Amazon uh previously um before joining Lyft, and then now uh I joined Lyft as MOBA, mid-size, um uh in terms of you know company size.
• 4:19
  SPEAKER_02: Um yeah, that's that's where I am.
• 4:22
  SPEAKER_00: Okay, got it.
• 4:24
  SPEAKER_00: And with those previous roles and organizations that were different sizes or maybe structured very differently, how did that change the data problems that you ended up solving?
• 4:35
  SPEAKER_02: Yeah, so hopefully um I can uh set up some business scenes for next uh 15 to 20 minutes um to illustrate different uh data challenge uh to other color.
• 4:46
  SPEAKER_02: Um yeah, I can start with a big take like like Amazon, their um, you know, one of uh principles is to um it is the ownership.
• 4:57
  SPEAKER_02: So each team owns their you know uh data set, core data set, and each team communicate with uh data contract or API.
• 5:04
  SPEAKER_02: So so that uh it's a separation of you know ownership.
• 5:08
  SPEAKER_02: The result is kind of having similar data set uh in many places, uh by that's by design.
• 5:17
  SPEAKER_02: Um so this you know solves the separation of ownership uh problem, but then at the same time um experiencing some the bottleneck is the data, data reconciliation is the bottleneck, data reconciliation is the kind of like main problem to solve for.
• 5:35
  SPEAKER_02: And then uh shifting again a little bit uh towards lift.
• 5:39
  SPEAKER_02: So lift is more uh mid-sized company where uh we prioritize more data leanness over ownership.
• 5:49
  SPEAKER_02: Um so there's no like uh typically there's no data duplication or definition duplication uh because we really tried hard effort for trying to have a single source of truth for data set or semantics layer um definition of metrics.
• 6:07
  SPEAKER_02: Uh we're really trying our best to keep our data you know lean and having a single source of definition.
• 6:15
  SPEAKER_02: Uh as a result, um so that that solves the data leaners problem, but on a different aspect where data engineers is the bottleneck, meaning um because of uh data liness, uh many data engineers should work together very closely.
• 6:34
  SPEAKER_02: Uh, even we are from different teams.
• 6:36
  SPEAKER_02: Like we have like one a big repository of EKL repo that people you know work together.
• 6:42
  SPEAKER_02: Uh and then Lyft is a very scientific, heavy company.
• 6:45
  SPEAKER_02: We're dealing with market balance and you know supply and demand.
• 6:49
  SPEAKER_02: So we really closely work with data science and data analysts uh to understand their business, and we uh have to translate their you know science definition into data in in terms of data.
• 7:03
  SPEAKER_02: So yeah, oftentimes these uh communication and data engineers in the loop is the bottleneck.
• 7:11
  SPEAKER_02: So those are yeah, I've been uh two different uh you know size of company and two different uh data problems to solve two different bottleneck.
• 7:20
  SPEAKER_00: Yeah, I think that's an interesting way of describing that and how it results in just different problems that you have to work through in your day-to-day.
• 7:27
  SPEAKER_00: Yeah, cool.
• 7:28
  SPEAKER_00: And tell me how Airflow fits in at Lyft.
• 7:33
  SPEAKER_00: So maybe we can start with like high-level, what role it plays and what sort of workloads your team is orchestrating with it.
• 7:40
  SPEAKER_02: Yeah, yeah, thanks for asking.
• 7:42
  SPEAKER_02: Um yeah, this all it's all connected to how I work at Lyft, but um data engineers closely work with science and analytics to deal with this uh you know, market balancing, dynamic pricing, dynamic pay.
• 7:55
  SPEAKER_02: Uh, and as a result, um we consider Airflow is not just scheduling ETL uh anymore.
• 8:01
  SPEAKER_02: We one of our core uh work is to productionize um, you know, Jupyter notebook written by science.
• 8:09
  SPEAKER_02: We uh kind of like containerze it, uh like a Docker image and then host um uh AWS stage maker, for example, and then we uh airflow is orchestration, uh, you know, doing the ETL uh, of course, but at the same time more uh ML ops as well.
• 8:27
  SPEAKER_02: And some pure um Python computation uh also involved to process large uh config data set.
• 8:34
  SPEAKER_02: Um because this config will be maintained uh very securely by science and analytics to deal with these uh to determine how much you know driver should get paid, how much larger uh pricing should be.
• 8:49
  SPEAKER_02: So a lot of uh different components uh involved in yeah, Airflow acts like uh like a service.
• 8:56
  SPEAKER_00: Yeah, interesting.
• 8:57
  SPEAKER_00: I'd love to double-click a little bit more on what you mean when you say more like a service as opposed to like a scheduled data pipeline system.
• 9:06
  SPEAKER_00: Tell me more about what that looks like in practice.
• 9:08
  SPEAKER_02: Yeah, yeah.
• 9:09
  SPEAKER_02: So yeah, in a national um airflow is like a service rather than ETL.
• 9:14
  SPEAKER_02: And I think I personally think this is where Airflow shines uh because you know Airflow is very flexible to connect, um, integrate with external system.
• 9:25
  SPEAKER_02: Uh like in practice, we connected uh Airflow with other services.
• 9:29
  SPEAKER_02: GitHub is one of them, and Spark as query, uh query layer, and it can connect to any um you know, storage S3 on HMS in our example.
• 9:40
  SPEAKER_02: Uh, and then people slowly started using more agentic uh processing, like um you know task or LLM, people started exploring agent operator uh so that Airflow is no longer just ETL, but it more dynamically behaves based on customer needs.
• 10:01
  SPEAKER_02: Uh so this is what I mean by uh airflow is shifting towards more like a service layer rather than you know static ETL.
• 10:10
  SPEAKER_00: Yeah, yeah, that's interesting.
• 10:11
  SPEAKER_00: And I would imagine at least that's also happening at a pretty large scale.
• 10:15
  SPEAKER_00: I think what you've described sounds like quite a lot of data and use cases that are implemented on Airflow.
• 10:23
  SPEAKER_00: Kind of related to the having it operate like as a service layer or as a platform.
• 10:28
  SPEAKER_00: You mentioned earlier that you have a separate uh data infra team that is responsible for running Airflow.
• 10:35
  SPEAKER_00: I also know that Lyft runs Airflow on Astro, which is Astronomer's Managed Service.
• 10:40
  SPEAKER_02: Tell me a little bit about what that experience has been like in terms of using a managed service and for that sort of scale of a yeah, I think um it's been uh I think more than almost a year now um since we started um you know Panna with Astronomer.
• 10:58
  SPEAKER_02: It's it's been great uh from a user perspective.
• 11:01
  SPEAKER_02: Um and to be clear, I'm not from infra or so.
• 11:05
  SPEAKER_02: I didn't I didn't have to do the uh B3 uh budget upgrade, uh, but uh the shift from V2 to V3 um has been smooth uh thanks to Astronomer uh support and also in addition to our infrarog uh huge effort.
• 11:20
  SPEAKER_02: Uh so that's been very, very smooth.
• 11:23
  SPEAKER_02: Time to yeah, also time to mitigate the incidents is very quick in my yeah, um in my use case.
• 11:31
  SPEAKER_02: So um another thing I like uh with Astronomer is the built-in dashboard, like observability layer, because you know previously we had um you know separate dashboards.
• 11:44
  SPEAKER_02: Typically, companies you know host like separate dashboard uh outside of Afro, like a Grafana dashboard to track the health of the cluster.
• 11:52
  SPEAKER_02: Um with Astro, everything is like one place.
• 11:55
  SPEAKER_02: So I really from user point of view, I really like to see everything is in one place.
• 12:00
  SPEAKER_02: It's easy to you know navigate for each deployment that I care about what's the you know P90 of the failure lay, what are the duration duration runtime uh for each uh for each tag.
• 12:12
  SPEAKER_02: Uh so it's it's a lot easier to navigate from um user perspective.
• 12:17
  SPEAKER_00: Yeah, totally.
• 12:18
  SPEAKER_00: A little more consolidated.
• 12:20
  SPEAKER_00: Makes sense.
• 12:23
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed aircraft platform built for data engineering teams.
• 12:31
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 12:35
  SPEAKER_01: Astro handles the infrastructure upgrades and scaling so you can focus on building.io.
• 12:46
  SPEAKER_00: Okay, great.
• 12:47
  SPEAKER_00: Well, I would love to dig into a particular use case that I know you've spoken a little bit about recently, and that was making DAGs self-service.
• 12:57
  SPEAKER_00: Uh, you've talked a lot about kind of how the team is structured at Lyft and kind of the volume of DAGs that you're dealing with is really high.
• 13:05
  SPEAKER_00: So tell me when you were working on implementing this, what problems were you trying to solve?
• 13:12
  SPEAKER_02: Or yeah, yeah, I can share a little bit of background um in addition to my earlier um um you know explanation of how our organization uh is structured.
• 13:23
  SPEAKER_02: But um, yeah, our product platform uh we own um you know driver pay configurations uh which eventually impacts on dynamic pricing as well.
• 13:34
  SPEAKER_02: So in our case, uh users are uh the science, applied, applied science, and data analysts are responsible for tuning of these conflicts on demand to balance the market balance of supply and demand.
• 13:48
  SPEAKER_02: Okay.
• 13:49
  SPEAKER_02: And this also tied to my uh earlier statement of data engineers uh at the bottleneck.
• 13:57
  SPEAKER_02: So we only had we only have a few data engineers uh in our org, but we support like around 30 immediate uh you know uh users in our org chart, and sometimes um you know users outside of our uh immediate org wants to run this product um for their uh different purposes of experiment.
• 14:19
  SPEAKER_02: Um so that's like one critical challenge we're trying to solve for with you know small number of data engineering teams.
• 14:28
  SPEAKER_02: And a second challenge we're trying to solve is uh IT control, um, often called SOX uh compliance.
• 14:34
  SPEAKER_02: Uh so anything anything that you know driver pay or anything that changes driver's earnings has to be securely tested, maintained, and validated.
• 14:45
  SPEAKER_02: And yeah, so the two problem we're trying to solve is how do we how do we make our uh platform, how do we support users uh with small data engineering team, but at the same time meeting with these IT controls at the same time.
• 15:01
  SPEAKER_00: Okay.
• 15:02
  SPEAKER_00: Yeah, that makes sense.
• 15:04
  SPEAKER_00: And so you've got a lot of different users or different stakeholders who need to be able to interact with Airflow and with DAGs, but don't necessarily have that airflow background to write a new DAG, or you probably don't want that many people writing new DAGs, even if they could.
• 15:20
  SPEAKER_00: That would be difficult to manage.
• 15:22
  SPEAKER_00: And so this is your solution for that, if I'm summing this up correctly.
• 15:26
  SPEAKER_00: Yeah, that's the big activity.
• 15:28
  SPEAKER_00: Okay, cool.
• 15:29
  SPEAKER_00: And then tell me a little bit more about what that looks like in practice and so how the integration between GitHub actions and Airflow works.
• 15:38
  SPEAKER_02: Yeah, um so if I um yeah, walk through walk you through uh how it works is like uh in a natural GitHub action trigger airflow uh beer uh REST API uh in our case and uh the passing context um at the same time when REST API will trigger Airflow.
• 15:58
  SPEAKER_02: You know, there are some contexts that we want to pass, like type things like configs and GitHub uh GitHub context share.
• 16:07
  SPEAKER_02: Uh and then based on different uh user config inputs, airflow will um you know dynamically behave and trigger uh a different number of uh uh validation or simulation of these uh papercy change uh so that uh we know we are comfortable with these changes before we use as much um these PL changes.
• 16:30
  SPEAKER_02: Um yeah, that uh we I think one thing, um one concept we try to make it is bi-directional.
• 16:39
  SPEAKER_02: So GitHub, it's not just one way with GitHub um calling Airflow, but Airflow will also respond to set us back to GitHub.
• 16:48
  SPEAKER_02: Um we really wanted this because um without this integration, uh users can technically match the PL without running Airflow.
• 16:59
  SPEAKER_02: So we really wanted to block the PL until our validation uh is complete for the sake of you know this ID control, the second problem I mentioned earlier.
• 17:11
  SPEAKER_00: Got it.
• 17:12
  SPEAKER_00: Interesting.
• 17:13
  SPEAKER_00: And tell me about how this is triggered.
• 17:16
  SPEAKER_00: Like what does the user do first to kind of make all of this run?
• 17:22
  SPEAKER_00: And what is the design there?
• 17:24
  SPEAKER_02: Yeah, so um we design in a way that users will open a GitHub uh feature branch first, and then they create a um PR that will kind of like uh change the config uh in their driver pick configs.
• 17:40
  SPEAKER_02: And we built a you know GitHub uh action script to uh automatically tell um and trigger airflow.
• 17:50
  SPEAKER_02: Um and then the rest will flow, uh as I mentioned, uh explained earlier.
• 17:55
  SPEAKER_00: Yeah, yeah.
• 17:56
  SPEAKER_00: And so does that trigger between GitHub actions and airflow, is that event driven on the airflow side, or how does that work?
• 18:05
  SPEAKER_02: Yeah, so we made it uh event driven.
• 18:07
  SPEAKER_02: Uh everything is event driven uh because of main reason is because daily cadence doesn't hold anymore.
• 18:14
  SPEAKER_02: These configs are you know updated and maintained on uh uh on demand anytime.
• 18:21
  SPEAKER_02: Also because uh Rift is now a global company.
• 18:25
  SPEAKER_02: Like we have office from Europe and uh East Coast, West Coast, we have office in Mexico, Canada, US.
• 18:32
  SPEAKER_02: So there are a lot of time gaps.
• 18:35
  SPEAKER_02: And again, we have you know few data engineers to support.
• 18:38
  SPEAKER_02: So we uh our solution is made it uh event-driven, um, you know, using uh REST API.
• 18:46
  SPEAKER_02: And we have like two layers of uh DAGs.
• 18:49
  SPEAKER_02: Uh layer like one is orchestration DAGs that takes user uh input uh and then using dynamic task mapping to you know passing context from upstream to downstream uh simulation layer to measure uh is the change we're making comfortable um range as uh output.
• 19:09
  SPEAKER_02: Um yeah, so we made it uh event driven.
• 19:12
  SPEAKER_02: Okay.
• 19:13
  SPEAKER_00: And did you also evaluate potential implementations like with polling sort of going the other way using a sensor or something like that on the airflow side?
• 19:24
  SPEAKER_00: And how did you end up with event driven being the best solution?
• 19:28
  SPEAKER_02: Yeah, thanks for asking.
• 19:29
  SPEAKER_02: I think that's a very fair, fair question.
• 19:31
  SPEAKER_02: We initially also try to um do with polling approach.
• 19:36
  SPEAKER_02: And just to be fair, both polling approach and event driven can both uh broke the margin the PL because uh at the end of the day it's GitHub API, right?
• 19:46
  SPEAKER_02: So even polling approach, they can the DAGs can uh broke the PL using an GitHub API.
• 19:53
  SPEAKER_02: The reason we ended up with event driven is mainly two reasons.
• 19:57
  SPEAKER_02: One is uh engineering perspective, um, and then second is operational perspective.
• 20:04
  SPEAKER_02: For the first point of engineering perspective, it's a lot easier with event driven to passing uh context from upstream to downstream.
• 20:12
  SPEAKER_02: Um you know in our situation, we needed to pass GitHub commit share um and some of the check ID uh from upstream to downstream so that each uh downstream uh simulation layer of DAGs, one DAG is responsible for one uh check ID.
• 20:30
  SPEAKER_02: Um and then it's a lot easier to do it with event driven rather than polling style, which DAGs are kind of like figuring out what I'm responsible for in uh at the end of the um you know simulation workflow.
• 20:45
  SPEAKER_02: So yeah, that's one reason we ended up with event driven.
• 20:49
  SPEAKER_02: Another, I think important um aspect is is this because of um it's uh on demand.
• 20:56
  SPEAKER_02: So from operational perspective, if we go with supporting approach, it's a lot of wasting of resources and compute uh and cost.
• 21:07
  SPEAKER_02: So everything kind of favored event-driven in our uh business scene.
• 21:13
  SPEAKER_00: Yeah, that totally makes sense.
• 21:15
  SPEAKER_00: And was there anything particularly challenging about implementing an event-driven system on Airflow?
• 21:21
  SPEAKER_02: Yes, yes.
• 21:22
  SPEAKER_02: So this is uh I think funny thing I mentioned um at the end of uh Summit talk is it's the um GitHub action integration is more than just hitting an API.
• 21:33
  SPEAKER_02: And the reason is because two issues with this uh integration.
• 21:37
  SPEAKER_02: One is data modeling challenge of GitHub, and another is uh data consistency problem.
• 21:44
  SPEAKER_02: Um for the data modeling uh issues, uh what I mean by that is uh how GitHub API um data modeling is designed is that uh GitHub GitHub has a commit chart, and then they have uh each test.
• 22:00
  SPEAKER_02: ID is tied to like an Umbrella of these commissar.
• 22:06
  SPEAKER_02: So it's like a tight tightly coupling of these test ID up Umbrella to Commission, but once users make a rebase or whenever they make um you know new commit, uh even if it doesn't change the you know actual config, it can kind of break apart these linkage between the two.
• 22:24
  SPEAKER_02: So that was one technical challenge we had in terms of data modeling.
• 22:28
  SPEAKER_02: Another issue is uh data consistency problem.
• 22:32
  SPEAKER_02: Uh you know, our simulation takes hours of hours of very heavy computation.
• 22:38
  SPEAKER_02: Um so after by the time downstream layer completes the few hours of you know simulation, uh users uh during the three three hours window users can do uh anything.
• 22:50
  SPEAKER_02: They can rebase and then um the context upstream pass to downstream um is no longer radius uh commission.
• 23:02
  SPEAKER_02: So uh downstream try to update the status against the uh outdated uh context is kind of like inaccurate.
• 23:10
  SPEAKER_02: Uh so that's uh another challenge we had uh during the time.
• 23:14
  SPEAKER_02: So the solution we ended up is to uh having uh like very similar to how asset state store works um in a metadata db layer.
• 23:24
  SPEAKER_02: We decided to have our own um you know external storage um to track all the history of these contexts, like check ID and context, uh context of GitHub and then timestamp so that uh by the time upstream, uh by the time downstream simulation layer um finishes, they can always fetch the latest, latest commission so that the data is consistency between the upstream and downstream.
• 23:51
  SPEAKER_02: Um I guess the yeah, one reason why we didn't um use the assets they saw in our example is really uh the business use case because we had uh more downstream layer um that like Spark really needed to access this um you know external uh storage of this commit and history of trigger DAG runs.
• 24:13
  SPEAKER_02: So yeah, that's the main reason we ended up with having external metadata, but the concept is very, very similar to um how you know B3.
• 24:26
  SPEAKER_00: Yeah, I was about to ask you mentioned the asset state store.
• 24:29
  SPEAKER_00: This is a quite new feature.
• 24:31
  SPEAKER_00: Are you already running 3.3?
• 24:34
  SPEAKER_02: So I've already explored in my uh Docker Docker Compose to understand how it behaves.
• 24:39
  SPEAKER_02: Um our production uh lift is uh as of this moment, like 3.1.8.
• 24:45
  SPEAKER_02: So we don't we still don't have asset state store, but um yeah, the reason we really didn't wait for 3.3 yet is because of this uh external connectivity um of accessing uh our external uh metadata layer.
• 25:01
  SPEAKER_00: Yeah, that makes sense.
• 25:02
  SPEAKER_00: I'm curious though, in your exploration of 3.3.
• 25:07
  SPEAKER_00: So the asset and task state store was kind of one of the big headlining new features.
• 25:11
  SPEAKER_00: There's also the pluggable retry policy.
• 25:14
  SPEAKER_00: What, if anything, do you think that you would do with those if you were able to upgrade?
• 25:19
  SPEAKER_02: Yeah, thanks for asking.
• 25:20
  SPEAKER_02: Um, this is exactly why I wanted to have this discussion with you today, and thanks for asking.
• 25:26
  SPEAKER_02: I think um there's something I couldn't you know talk through during uh my summit talk, which is like what's future, what's feature work for our uh organization?
• 25:35
  SPEAKER_02: Um I think 3.3 really introduced uh um great opportunity for us.
• 25:42
  SPEAKER_02: Uh tasks they saw in plaggable retribe policy.
• 25:46
  SPEAKER_02: A lot of our, you know, because we use GitHub API, a lot of our failure is transient.
• 25:52
  SPEAKER_02: Um, like API network.
• 25:53
  SPEAKER_02: These are uh really a lot of times it's a very transient issue.
• 25:58
  SPEAKER_02: Um retrible kind of help us define like when to stop and when we should keep going in terms of retry policy and tasks stay stored.
• 26:10
  SPEAKER_02: The reason why I think it's useful is because um it's all connected to how our simulation behaves, but it's it takes like a few hours, more than a few hours, and it's really heavy.
• 26:20
  SPEAKER_02: Sometimes the task um task out of out of memory um error happened from airflow side, uh, but then actually the spark side uh computation is completed.
• 26:32
  SPEAKER_02: So we really don't need to retry another few hours if the um the first try first attempt of uh you know spark is completed.
• 26:42
  SPEAKER_02: So it it both will really help us more efficient in terms of uh how retry uh works.
• 26:50
  SPEAKER_02: So we really appreciate uh having this new future, and I think that that will be uh our our key what's future for us in terms of uh efficiency.
• 27:01
  SPEAKER_00: Yeah, definitely.
• 27:02
  SPEAKER_00: That's great to hear that it would have such a big impact.
• 27:04
  SPEAKER_00: And I can imagine I was sort of chuckling as you said that because I feel like transient failures with the GitHub in particular seem to be almost comically common right now.
• 27:14
  SPEAKER_00: Um and as you said, most of the time they resolve themselves within a few minutes.
• 27:18
  SPEAKER_00: So uh yeah, that's a great use case for those features.
• 27:22
  SPEAKER_00: Well, I think that's a great lead-in to my standard final question for all of my guests, which is what would you most like to see from the Airflow project in the near future?
• 27:32
  SPEAKER_02: Yeah, thanks for asking.
• 27:34
  SPEAKER_02: Um yeah, I can share what I uh actually answered in my um you know uh Airflow Sub 8 for 2026.
• 27:41
  SPEAKER_02: And I think um I'm I'm very excited for how uh the task loop and how we can work with agent uh agent operator.
• 27:50
  SPEAKER_02: Um it's something I would like to see more.
• 27:53
  SPEAKER_02: Um and I think Lyft is very uh good use will be a good use case uh for this because we are solving like optimization problems, and sometimes we have like multiple correct answers in terms of like market balance, uh, because we have like constraints and multiple optimized answers.
• 28:13
  SPEAKER_02: Um where I think Airflow Shines is you know with AgenTech Operator uh itself like standalone is like a black box uh with you know one attempt.
• 28:25
  SPEAKER_02: But with if we have like task group, um you know each attempt is kind of like properly audited with logging and it's a lot easier to navigate like how agent tries to attempt.
• 28:38
  SPEAKER_02: Uh even with like passing you know previous con previous trial and context to next group will be a lot helpful in terms of efficiency.
• 28:47
  SPEAKER_02: So I really appreciate um you know putting effort on this um you know future work with task group and how it'll you know work together with agent operator is uh very exciting um outlook for me.
• 29:02
  SPEAKER_00: Yeah, absolutely.
• 29:03
  SPEAKER_00: I I totally agree.
• 29:05
  SPEAKER_00: I think, and it sounds like we might get task loops relatively soon uh in the next minor release or two.
• 29:12
  SPEAKER_00: So hopefully either later this year or early next year.
• 29:15
  SPEAKER_00: But yeah, that's great.
• 29:17
  SPEAKER_02: Looking forward to it.
• 29:18
  SPEAKER_00: Yeah, yeah, me too.
• 29:20
  SPEAKER_00: Um okay, Ken, this has been really great speaking with you.
• 29:23
  SPEAKER_00: Thanks so much for coming on.
• 29:25
  SPEAKER_00: Um, appreciate all of the insight you've provided.
• 29:27
  SPEAKER_00: Just to close us out, what is the best way for folks to get in touch with you?
• 29:30
  SPEAKER_02: Yeah, I'm always uh open to uh a new connection through uh LinkedIn.
• 29:34
  SPEAKER_02: So feel free to uh reach out to me uh via LinkedIn.
• 29:38
  SPEAKER_00: Okay, great.
• 29:39
  SPEAKER_00: We will throw that in the show notes.
• 29:40
  SPEAKER_00: Well, thank you again for coming on.
• 29:41
  SPEAKER_00: This has been great.
• 29:42
  SPEAKER_01: Thank you so much.
• 29:44
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 29:47
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 29:50
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 29:55
  SPEAKER_01: And leave it a five-star review.