Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Using Airflow for diverse client projects at Accion Labs

Transcript

Using Airflow for diverse client projects at Accion Labs · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: You have an a uh like uh brilliant LLM, but it's uh it's working with the stale or uh noisy data, the out the output will obviously fall apart.
• 0:11
  SPEAKER_00: So uh the POC is really about uh treating AI data pipelines with the same rigor we had apply to any production data pipeline.
• 0:21
  SPEAKER_02: You're listening to the Data Flowcast, the podcast about the Airflow and the world of data and AI around it.
• 0:29
  SPEAKER_02: We are excited to invite you to orchestrate everything.
• 0:36
  SPEAKER_02: Engineering data around and work out aircraft AI workflow on AI for the new Astronomer Certified AI orchestrate and fundamental account code for a free certificate for a$150 value.
• 0:55
  SPEAKER_02: Go to astronomer.link slash data minus flowcast minus OE.
• 1:01
  SPEAKER_02: Link in the show notes.
• 1:02
  SPEAKER_02: Now let's get into the show.
• 1:05
  SPEAKER_01: All right, hi everyone.
• 1:06
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:08
  SPEAKER_01: I'm your host, Kenton Danis, and today I'm joined by Chandan Gauda, who is a data engineer at Axion Labs.
• 1:14
  SPEAKER_01: Chandan, how are you?
• 1:16
  SPEAKER_00: Hey Kiton, so I'm good and uh thanks for having me here.
• 1:21
  SPEAKER_00: I'm really excited to be here, and uh I have been a listener of this show quite a while, and it feels a bit serious to actually be on the other side of the mic.
• 1:32
  SPEAKER_00: Thank you for having me.
• 1:34
  SPEAKER_01: Yeah, that's great.
• 1:35
  SPEAKER_01: Well, I love to hear that, and it's so great to have you on.
• 1:38
  SPEAKER_01: Uh, we'll go ahead and dive into it.
• 1:40
  SPEAKER_01: So, for listeners for today's podcast, we're going to talk about how Axiom Labs uses Airflow across their client projects, uh, including a financial scoring use case and some early work on AI pipeline orchestration.
• 1:52
  SPEAKER_01: So, both some pretty cool topics.
• 1:54
  SPEAKER_01: Uh but before we get into all of that, I want to start with some background.
• 1:58
  SPEAKER_01: Maybe you can tell me about the company you work for.
• 2:00
  SPEAKER_01: What does Axiom Labs do?
• 2:03
  SPEAKER_00: Sure.
• 2:04
  SPEAKER_00: So Axiom Labs is a technology consulting and services firm, and we essentially help businesses to solve the complex IT and data challenges.
• 2:14
  SPEAKER_00: You can think of us like a partner that comes in, understands client pain points, and then design and build the solutions around the modern data and AI technologies.
• 2:26
  SPEAKER_00: So we also work across uh industries like BFSI, healthcare, retail.
• 2:32
  SPEAKER_00: So as we are consulting firm, the problems will vary.
• 2:36
  SPEAKER_00: So which honestly keep things really interesting day by day.
• 2:40
  SPEAKER_00: You get new things to solve.
• 2:43
  SPEAKER_01: Yeah, very cool.
• 2:44
  SPEAKER_01: I used to work in consulting myself, so I relate to the idea of being able to work on different projects and how that keeps things interesting.
• 2:52
  SPEAKER_01: And so tell me a little bit more then about your role and the kinds of projects that you typically work on.
• 3:00
  SPEAKER_00: Yeah.
• 3:01
  SPEAKER_00: So uh earlier, like uh I started as a data engineer.
• 3:05
  SPEAKER_00: So as you know, the technologies will be changing and uh we will be evolving with the market.
• 3:11
  SPEAKER_00: So currently I'm uh uh associated with data itself, but uh quite more uh occupied with uh Gen AI things.
• 3:19
  SPEAKER_00: So you can call me as a Gen AI data engineer for now.
• 3:23
  SPEAKER_00: So my work sits right uh at the intersection of uh like data engineering and AI, where I have to uh build some uh pipelines in uh one week and uh another week I will be working on the RAC-based uh AI agents, like uh developing some internal tools uh for uh our internal purposes.
• 3:44
  SPEAKER_00: And like uh it'll be like uh all together it's like orchestration plus uh AI.
• 3:51
  SPEAKER_00: So I need to make sure that uh data moves reliably and model gets fed with the right inputs, and everything runs on schedule without like uh someone babysitting it.
• 4:05
  SPEAKER_00: So everything needs to be automated.
• 4:06
  SPEAKER_00: So everyone are fascinating, and yeah, I'm also thinking to automate all the things uh which is uh absolutely possible with the AI today.
• 4:16
  SPEAKER_00: So yeah, that's pretty much about my role.
• 4:20
  SPEAKER_01: Yeah, that's great.
• 4:21
  SPEAKER_01: That sounds yeah, very interesting.
• 4:23
  SPEAKER_01: And where does airflow typically fit into that, either with the data engineering piece or maybe the AI engineering piece?
• 4:30
  SPEAKER_01: I guess we'll get a little bit more into that in the uh later in the interview.
• 4:34
  SPEAKER_01: But yeah, high level, uh, where are you typically using Airflow with your client work?
• 4:39
  SPEAKER_00: Airflow is um savior for us in orchestration things, uh with the data thing.
• 4:45
  SPEAKER_00: So uh while we are um uh we are doing the data pipelining at the time, Airflow will uh help us with orchestrating, bringing the data, and um we can uh use the triggers to fetch the data, uh do some transformations and put it back.
• 5:02
  SPEAKER_00: So there uh we use uh Airflow for now.
• 5:06
  SPEAKER_00: We will talk about this uh uh when I specifically explain about the projects.
• 5:12
  SPEAKER_01: Yeah, absolutely.
• 5:13
  SPEAKER_01: And I'm curious, is Airflow something that you are typically recommending, or do your clients come in asking for it?
• 5:21
  SPEAKER_01: Or both, maybe?
• 5:23
  SPEAKER_00: Yeah, I can say like uh it's a mix.
• 5:26
  SPEAKER_00: So honestly, some clients come uh who are already using the airflow and they wanted to scale it up.
• 5:34
  SPEAKER_00: And um there are others who are like just wanted to evaluate the options and we will assess what is their requirements and we will uh suggest them what fits the best.
• 5:45
  SPEAKER_00: But in most cases, once we lay out the options, uh so airflow will uh win, obviously.
• 5:52
  SPEAKER_00: So, especially for the clients who need a fine-grained control over their pipeline logic, and they don't want to be locked in into a single cloud provider's ecosystem.
• 6:03
  SPEAKER_00: So at that time, uh airflow is a savior, so we recommend that.
• 6:08
  SPEAKER_00: Um like uh it's like uh airflow is not a new thing, you know, right?
• 6:14
  SPEAKER_00: So it was uh created by the uh like uh Maxima Bochiman, uh who's uh founder like at Airbnb, and uh it is obviously open source and it has around uh 30 million of uh downloads per month.
• 6:31
  SPEAKER_00: So, which is like um it's easy to convince the clients if they really don't know what is Airflow and uh they wanted to integrate it into their systems.
• 6:41
  SPEAKER_00: So, yeah, if uh we when we show all of these details and we uh show us how it uh solves the pain points, client will be uh obviously happy with the solution we will be implementing with the airflow.
• 6:56
  SPEAKER_01: Yeah, totally makes sense.
• 6:57
  SPEAKER_01: I think the popularity of the project in some ways speaks for itself.
• 7:01
  SPEAKER_01: And so you mentioned having you know the flexibility of not being locked into a certain cloud, uh, as well as you know, preferring having the control and sort of lower overhead compared to maybe a cloud native tool.
• 7:16
  SPEAKER_01: Tell me a little bit more about that, kind of where where do you think Airflow really stands out in that area over other orchestrators?
• 7:23
  SPEAKER_00: Uh like uh we can say like uh if we speak about the AWS def functions or uh a Zoo Data Factory, they are all great, only if they are in the cloud.
• 7:36
  SPEAKER_00: So we cannot uh pull them in locally and we cannot uh do the orchestration locally.
• 7:42
  SPEAKER_00: But um the moment a client says we might move to a different cloud in uh two years, or if we wanted to uh go for the uh other technologies or other clouds, so at that time um uh we will be in trouble.
• 7:58
  SPEAKER_00: So shifting the things from uh let's say from uh Microsoft to AWS, there is a lot of pain points, it requires a huge effort, and um we have to understand the infrastructure and all, but the magic comes here.
• 8:13
  SPEAKER_00: So when we use the airflow airflow, so it's like infrastructure agnostic.
• 8:18
  SPEAKER_00: So you can containerize it and you can deploy it on Kubernetes or even in the VM, it just works.
• 8:25
  SPEAKER_00: We just need to containerize it and we can deploy it anywhere.
• 8:28
  SPEAKER_00: So this gives us flexibility to uh use Airflow, even we are migrating to a different cloud or ecosystems, it'll be easy.
• 8:38
  SPEAKER_01: Yeah, totally makes sense and certainly consistent with what we've heard uh from our own customer base at Astronomer for why people choose Airflow.
• 8:47
  SPEAKER_01: Uh yeah, very cool.
• 8:49
  SPEAKER_01: So I would love to then dig into a couple of the specific use cases that you've implemented with your clients uh that are, you know, heavily use Airflow as the orchestrator.
• 9:00
  SPEAKER_01: Uh one that I know you've worked on recently is a loan eligibility scoring use case.
• 9:06
  SPEAKER_01: So maybe you can start by telling me a little bit more about that.
• 9:08
  SPEAKER_01: What is that project about?
• 9:11
  SPEAKER_00: Sure.
• 9:13
  SPEAKER_00: So uh this one was uh one of the most uh concrete uh Airflow use cases I worked on.
• 9:20
  SPEAKER_00: So uh basically the client was a financial services company, they needed uh score loan applicants for uh eligibility in the real time.
• 9:30
  SPEAKER_00: So the challenge was that um the uh new applicant data was arriving continuously, it's like a streaming data, so it was coming from uh different multiple sources and um the scoring models were uh downstream.
• 9:46
  SPEAKER_00: So we need to manage the trigger.
• 9:49
  SPEAKER_00: So even if we uh fail to uh trigger the uh pipelines within the specified times, we would lose the data.
• 9:58
  SPEAKER_00: So at that time uh we we needed to use the airflow.
• 10:04
  SPEAKER_00: So uh airflow before airflow we were uh using um manually, we were doing it manually.
• 10:11
  SPEAKER_00: So we have to set some cron jobs and uh the it's like a fragile thing, it can break anytime.
• 10:17
  SPEAKER_00: So uh then we uh thought of airflow and we uh got to know about like if we uh use the airflow, how flexible it is, and how convenient it can be.
• 10:29
  SPEAKER_00: So we replace the entire stack with a clean airflow DAG, so that handles the ingestion, transformation, model inference, and also the result delivery completely from end to end.
• 10:42
  SPEAKER_00: So everything is managed by the airflow.
• 10:45
  SPEAKER_00: So uh by this um we we reduced the efforts nearly around 25%.
• 10:52
  SPEAKER_00: So we came from um 100% to 25% efforts reduction reduction.
• 10:58
  SPEAKER_00: So this was a very uh what we can say great change that uh client was very happy with this, and uh from that onwards, we we don't want to get up in the midnight and check whether the pipeline is uh working or not, it has failed, something has broken.
• 11:16
  SPEAKER_00: It is like a pain point for even the developers or supporting engineers.
• 11:20
  SPEAKER_00: So uh Airflow uh helped us in that point, and uh from that onwards, we encourage very minimal uh bricks and uh everything was smooth like a butter.
• 11:32
  SPEAKER_01: Yeah, that's great.
• 11:33
  SPEAKER_01: What a great Airflow success story.
• 11:36
  SPEAKER_01: Uh and then I'm curious.
• 11:37
  SPEAKER_01: Uh so based on the use case, how are you scheduling these pipelines?
• 11:42
  SPEAKER_01: So are you doing more event driven so that when you know eligibility needs to be determined that the pipeline is actually running, or is it scheduled?
• 11:52
  SPEAKER_01: How does that work?
• 11:54
  SPEAKER_00: Great question.
• 11:56
  SPEAKER_00: So uh this is where like uh I can say airflow really shines.
• 12:01
  SPEAKER_00: So we use the uh combination approach.
• 12:05
  SPEAKER_00: The primary trigger was a file sensor based.
• 12:08
  SPEAKER_00: So we have a component like um airflows flow file sensor, so or uh we have uh S3 file sensor pulse.
• 12:19
  SPEAKER_00: So any new data arriving at a specific location, the moment the batch file lands, the sensor task goes green and uh the rest of the DAC kicks off automatically.
• 12:34
  SPEAKER_00: So we also had a scheduled fallback also.
• 12:38
  SPEAKER_00: If there is no new data arrived by some time, so there must be some problem.
• 12:44
  SPEAKER_00: So the sensor could not pick it up.
• 12:46
  SPEAKER_00: So at that time we had to uh manage the backup also.
• 12:50
  SPEAKER_00: So we had a scheduled backup which would run for a certain time, and uh, if there is something in the queue, it could pick up and uh it's like both it's not like purely event-driven or it's like uh not more like uh scheduled one, also it's like uh combination, but uh I can say the sensor-based trigger was handling about uh 90% of our runs, so it was successful.
• 13:17
  SPEAKER_01: Yeah, that's great, and I think testament to the scheduling flexibility of Airflow that you're able to manage even the backup solution pretty easily, it sounds like.
• 13:28
  SPEAKER_01: Yeah, yeah, that's really cool.
• 13:30
  SPEAKER_01: And then so once that DAG or maybe multiple DAGs kick off, what does the end-to-end flow look like within Airflow?
• 13:38
  SPEAKER_01: So once that data arrives all the way to you know, you've got those eligibility scores landing back in the target system.
• 13:44
  SPEAKER_01: What's happening in the middle there?
• 13:47
  SPEAKER_00: It's like a pretty clean pipeline.
• 13:49
  SPEAKER_00: So once you see it laid out, so step one is ingestion.
• 13:54
  SPEAKER_00: So Airflow pulls the raw applicant records from the source system into our data layer, and uh after pulling that, uh step two starts.
• 14:04
  SPEAKER_00: So step two is uh like a validation and transformations, so we check for the nulls, duplicates, and anything that uh requires attention, and we will apply our business rules and join in in reference data, like uh we will be uh sourcing some uh uh third-party data also, so like uh credit bureau scores uh for uh uh passing on to the model for scorings.
• 14:31
  SPEAKER_00: So that things on our way we're doing the step two, and uh step three was uh like um model inference tags.
• 14:38
  SPEAKER_00: So this was uh calling our scoring models and which runs in a separate containerized environment, and it passes back the eligibility scores and the supporting features which we require for uh accessing the candidate.
• 14:53
  SPEAKER_00: So after this has been done, the step four step four was like um uh writing back the scores into the client's loan management system.
• 15:02
  SPEAKER_00: So we have done the scoring, we have done the analysis, and we need to report it back to the client.
• 15:08
  SPEAKER_00: So we have to uh give him in uh database.
• 15:11
  SPEAKER_00: So we had a database where uh we were uh appending the um results.
• 15:16
  SPEAKER_00: So uh throughout all this, Airflow is uh tracking the task states and it has been retrying the failed task automatically, and if it is also sending the alerts if uh again something was breaking.
• 15:31
  SPEAKER_00: So I can say that Airflow sensors are like uh special operators that uh pause the DAG until uh certain condition is met.
• 15:41
  SPEAKER_00: So they were like uh mostly used for event-driven triggers, and uh we also had some um poke interval and uh timeout on sensors towards uh some of the zombies tasks, so that um if a task uh sits in a running state indefinitely, it would cause other tasks to be in the queues.
• 16:02
  SPEAKER_00: So we had uh set some timeouts so that uh if uh something is uh running for infinite time, it we it used to uh poke mode and it used to free up that uh worker load from the checks.
• 16:17
  SPEAKER_00: So yeah, so like this our whole uh flow looks.
• 16:22
  SPEAKER_01: Yeah, yeah, that's great.
• 16:24
  SPEAKER_01: Um, what a cool use case for airflow.
• 16:28
  SPEAKER_02: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 16:37
  SPEAKER_02: From batch pipelines to training models and wiring up agents.
• 16:41
  SPEAKER_02: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 16:47
  SPEAKER_02: Check it out at astronomer.io uh okay.
• 16:53
  SPEAKER_01: Well, changing uh topics a little bit to another use case I know you've implemented, I mentioned at the beginning that we would talk about uh using Airflow for AI orchestration.
• 17:04
  SPEAKER_01: Obviously, a very hot topic right now, uh, that we're seeing more and more of.
• 17:09
  SPEAKER_01: And I understand you've been working on a POC around this.
• 17:12
  SPEAKER_01: Maybe first tell me what is the problem that you're trying to solve here.
• 17:18
  SPEAKER_00: Yeah, this is the one I'm most uh excited to talk about right now.
• 17:22
  SPEAKER_00: So the problem statement is like um as we are building more on uh Jen AI and uh RAG-based application internally and some of them for the clients also.
• 17:35
  SPEAKER_00: So we realize that the data feeding into these models is just as critical as model itself.
• 17:42
  SPEAKER_00: See, uh model is you can think as a baby.
• 17:45
  SPEAKER_00: So you what you teach the baby is what the baby learns.
• 17:49
  SPEAKER_00: So if you feed him uh good data, he will uh speak good and he will be good.
• 17:54
  SPEAKER_00: If you have uh teach him the bad things, he will uh uh learn the bad things and he will uh do the bad things.
• 18:01
  SPEAKER_00: So that we realized while we are doing the POC.
• 18:05
  SPEAKER_00: And you have an uh like uh brilliant LLM, but it's uh it's working with the stale or uh noisy data, the out the output will obviously fall apart.
• 18:16
  SPEAKER_00: So uh the POC is really about uh treating AI data pipelines with the same rigor we had apply to any production data pipeline, and we used Airflow as orchestrator to uh whatever with the data we are getting, we were not directly feeding it to the model, or we even we are not training with the the noisy data or what we get directly.
• 18:42
  SPEAKER_00: We used Airflow, we did some transformations, and then we put that data to the A model so that it works as expected.
• 18:53
  SPEAKER_01: Yeah, that's great.
• 18:54
  SPEAKER_01: I love that you mentioned uh treating these pipelines like you would any other pipeline.
• 18:59
  SPEAKER_01: This is something that my team at Astronomer has been talking about a lot recently, just the importance of orchestration in general for the same reasons that it's important for ETL pipelines when you're dealing with AI pipelines.
• 19:12
  SPEAKER_01: Uh and then so I understand that part of the goal, as you mentioned, is is controlling the data that gets fed into the models.
• 19:21
  SPEAKER_01: Tell me more about what role Airflow plays with that specifically.
• 19:27
  SPEAKER_00: Okay, you can think of it this way.
• 19:30
  SPEAKER_00: So in a rack pipeline, you have a knowledge base that needs to stay fresh.
• 19:36
  SPEAKER_00: So, knowledge base in the sense uh the data, the data like you can um imagine.
• 19:42
  SPEAKER_00: So uh it has to be fresh, and the documents get updated, like um the new ones will be added, or the old ones will get irrelevante, and also there are some uh cases where we need to update the documents.
• 19:58
  SPEAKER_00: So, whatever we do, um like the data keeps changing, and uh the models should be updated uh with the updated data.
• 20:09
  SPEAKER_00: So we are uh we cannot do this uh without like uh what we can say, without orchestration tools.
• 20:16
  SPEAKER_00: Uh we cannot um refresh all the times, we we cannot refresh the knowledge base, and even the refreshing thing is uh very expensive.
• 20:25
  SPEAKER_00: So that too, while you are running the models and you are doing a refresh, it cues huge.
• 20:33
  SPEAKER_00: So with Airflow, we can build a tag that detects the document changes, re-chunks, and re-embeds only what's changed, and it will update the vector store.
• 20:45
  SPEAKER_00: Instead of uh doing a whole refresh or whole uh change of the documents or the knowledge base, it only picks up what is changed and it only re-chunks it and it only re-embeds what is needed.
• 21:01
  SPEAKER_00: Again, we are cutting cost at vector uh embedding models and um like this, uh, we will validate the quality and we will have some threshold set and there flow.
• 21:12
  SPEAKER_00: So if uh the threshold goes below, then obviously we are not going to process the data, we will stop that.
• 21:19
  SPEAKER_00: If it is met, then we are going to uh switch it to the uh we'll be connecting it to the live system.
• 21:26
  SPEAKER_00: So it's totally controlled, auditable, and even you can roll back if something goes wrong.
• 21:32
  SPEAKER_00: That audit right is something you just don't get with ad hoc scripts.
• 21:36
  SPEAKER_00: So that's where again hairflow shines.
• 21:40
  SPEAKER_01: Yeah, absolutely.
• 21:41
  SPEAKER_01: That traceability aspect or the reliability and monitoring is so important.
• 21:46
  SPEAKER_01: Uh so tell me if if this POC goes well, kind of where where do you see this going in the future?
• 21:55
  SPEAKER_00: Yeah, the vision is basically like a full ML op style orchestration layer.
• 22:00
  SPEAKER_00: For uh Gen A workloads.
• 22:02
  SPEAKER_00: Right now we are uh focused on the data freshness problem, keeping like embeddings and uh context current.
• 22:10
  SPEAKER_00: But the next natural step is uh model evaluation pipelines, um, triggering automated evaluations whenever the underlying knowledge base changes, flagging regressions before the hit production, and beyond that, uh, we are looking for a multi-agent orchestration where uh Airflow manages the sequencing of uh different AI agents, and uh each will be responsible for a specific task in the larger workflow.
• 22:37
  SPEAKER_00: It's like uh genuinely exciting territory, and um uh Airflow 2.x introduced that uh task flow API, which uh makes it much more cleaner to pass data between tasks using Xcons.
• 22:52
  SPEAKER_00: So for AI pipelines, this is very powerful, and you can pass the embeddings, retrieved context changes, or model scores between the DAC tasks like uh Python objects, and additionally, the Airflow community has published uh provider packages for lang chain, open AI, and hugging faces, which making a straightforward to trigger model inference or embedding uh generation directly from the airflow Airflow operators rather than uh going to the third parties.
• 23:27
  SPEAKER_01: Yeah, absolutely.
• 23:28
  SPEAKER_01: There's so much great work going on in the Airflow project uh to make it more usable for AI and agent orchestration, the all the providers you mentioned, and now the new common AI provider as well, uh, which makes it very generally easy to interact with any model.
• 23:44
  SPEAKER_01: So that's so cool to hear how you're making use of that already.
• 23:48
  SPEAKER_01: Uh I think that's a good segue into my standard final question uh for all of my guests, which is what would you most like to see from the Airflow project in the near future?
• 23:59
  SPEAKER_00: Yeah, so honestly, um a few things uh come to my mind.
• 24:04
  SPEAKER_00: And uh number one is uh better native support for uh event-driven triggers beyond polling sensors, something like uh true pub sub integration at the core, not just via workarounds, uh like uh the new Airflow 3.0 uh data set aware scheduling moves into that direction, and I'm really watching that closely and uh hoping for that.
• 24:28
  SPEAKER_00: And number two is like um first-class observability of for the AI workloads, things like uh token usage tracking, model latency histograms, and uh embeddings, drift detection built into the airflow UI rather than uh needing to stitch together external monitoring tools.
• 24:49
  SPEAKER_00: So if we get everything under the one UI, so that will be a great thing.
• 24:55
  SPEAKER_00: And third one is like I would love to see a better dynamic DAG generation at scale.
• 25:02
  SPEAKER_00: So the current dynamic task mapping is great, but when you have hundreds of dynamically generated tasks, the UI will get wider.
• 25:13
  SPEAKER_00: So it's like uh more ergonomic grouping and uh filtering at that scale would be like a big quality of uh life improvement.
• 25:23
  SPEAKER_00: So I'm hoping it uh when uh Apache Airflow 3.0, which is in active development, so it will be bringing more changes and uh a fully decoupled uh DAC processor, improved data set uh driven scheduling and uh redesigned uh REST APIs.
• 25:42
  SPEAKER_00: So these will uh help us in uh better uh task execution isolation.
• 25:48
  SPEAKER_00: So again, the data set concept aligns closely with the event-driven data pipelines and uh is a key step towards uh data aware orchestration, exactly what modern A pipelines needed.
• 26:02
  SPEAKER_00: So if we have these all things, I think there is no better choice than Airflow.
• 26:08
  SPEAKER_01: Yeah, that's great.
• 26:10
  SPEAKER_01: Those are all good answers, and I can see where especially the data sets are now assets, they're called in Airflow 3, uh, and the event-driven scheduling would be very useful for both of the use cases that you've talked about in this interview.
• 26:22
  SPEAKER_01: So uh yeah, definitely lots of stuff to look forward to in the project.
• 26:26
  SPEAKER_01: Okay, well, Chandon, it's been so great speaking with you.
• 26:29
  SPEAKER_01: Thank you for sharing uh your experience with Airflow.
• 26:33
  SPEAKER_01: I always love talking to folks who get to work with uh lots of different projects and implementing Airflow because you get to see so many different implementations and uh yeah, the kind of wide variety of things it's useful for.
• 26:46
  SPEAKER_01: Just to close this out, what is the best way for folks to get in touch with you?
• 26:50
  SPEAKER_00: So to get in touch with me, LinkedIn is the best place.
• 26:54
  SPEAKER_00: Just search for uh Chandan Gauda, data engineer at uh Action Labs, and you will find me.
• 27:00
  SPEAKER_00: I'm always happy to connect with the folks who are working on the data pipelines, Gene or Orchestration Challenges.
• 27:06
  SPEAKER_00: Just feel free to drop a message.
• 27:09
  SPEAKER_01: Okay, great.
• 27:10
  SPEAKER_01: Well, we will throw a link to your LinkedIn in the show notes for sure so people can find it.
• 27:14
  SPEAKER_01: Okay, well, thank you again for joining us today.
• 27:17
  SPEAKER_01: It's been really great speaking with you.
• 27:19
  SPEAKER_02: Thank you for listening to the Data Flowcast.
• 27:22
  SPEAKER_02: Check the show notes for links to everything mentioned today.
• 27:26
  SPEAKER_02: If you're enjoying the podcast, subscribe so you never miss an episode.
• 27:31
  SPEAKER_02: And leave us a five star review.