Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building self-healing Airflow pipelines at ATC Drivetrain

Transcript

Building self-healing Airflow pipelines at ATC Drivetrain · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: You're listening to the data flowcast, a data podcast about Airflow and the world of data and AI around it.
• 0:07
  SPEAKER_01: We are excited to invite you to orchestrate everything for a free online conference on September 19th engineering data from the left around and wait air out there.
• 0:28
  SPEAKER_01: Attendees get a discount code for a free certification.
• 0:32
  SPEAKER_01: A$150 value.
• 0:34
  SPEAKER_01: Go to astronomer.link slash data minus flowcast minus OE.
• 0:39
  SPEAKER_01: Link in the show notes.
• 0:41
  SPEAKER_01: Now let's get into the show.
• 0:43
  SPEAKER_02: Hi everyone.
• 0:44
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:46
  SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by Komuda Srinivasa, who is a senior data architect at ATC Drivetrain.
• 0:54
  SPEAKER_02: Komuda, how are you?
• 0:56
  SPEAKER_00: I'm pretty good, Kenton.
• 0:58
  SPEAKER_00: How about you?
• 0:59
  SPEAKER_02: I'm good.
• 1:00
  SPEAKER_02: Yeah, thanks for asking.
• 1:01
  SPEAKER_02: It's really great to have you on.
• 1:03
  SPEAKER_02: I'm excited to dig into it.
• 1:05
  SPEAKER_02: So just for a bit of a teaser for today's podcast, we're going to discuss self-healing airflow platforms and AI-assisted pipeline recovery.
• 1:15
  SPEAKER_02: Very hot topic.
• 1:16
  SPEAKER_02: I think everybody in the industry is thinking about this right now.
• 1:19
  SPEAKER_02: So I'm sure this will be very helpful to our listeners.
• 1:23
  SPEAKER_02: Before we get into all of that, I want to start with a little bit of background.
• 1:26
  SPEAKER_02: So tell me first about the company that you work for.
• 1:29
  SPEAKER_02: What does ATC Drivetrain do?
• 1:32
  SPEAKER_00: Oh, thank you for having me.
• 1:34
  SPEAKER_00: That's the first thing I want to say.
• 1:36
  SPEAKER_00: So I work for ATC Drivetrain.
• 1:39
  SPEAKER_00: ATC Drivetrain is a remanufacturing uh industry.
• 1:42
  SPEAKER_00: So I think you would know about old engines where like they take up the parts and build a new engine.
• 1:49
  SPEAKER_00: That's what we do.
• 1:51
  SPEAKER_00: And we do like the trucks, the cars, the bigger trucks.
• 1:55
  SPEAKER_00: So ATC kind of operates in the automotive remanufacturing, so powertrain products.
• 2:01
  SPEAKER_00: So we have major four major facilities across the US and also Europe.
• 2:06
  SPEAKER_00: Uh this is like a very complex production environment where data comes from manufacturing equipment, short-floor applications, quality systems, and supply chain processes.
• 2:17
  SPEAKER_00: So the whole scenario is like data reliability is extremely important.
• 2:22
  SPEAKER_00: If an F a delay or an incorrect pipeline in the whole ATC dry train can have a big production visibility or quality decisions going bad.
• 2:32
  SPEAKER_00: Yeah, that's where I work for.
• 2:34
  SPEAKER_02: Yeah, very cool.
• 2:36
  SPEAKER_02: I think it's probably the case that a lot of people don't think a lot about where the parts of their car come from.
• 2:46
  SPEAKER_02: But it has a big impact on their day-to-day lives, or at least it does for those of us in the US.
• 2:51
  SPEAKER_02: Uh so yeah, really cool to hear about that and the role that data plays in the manufacturing of those parts.
• 2:59
  SPEAKER_02: I would love to dig more into that.
• 3:01
  SPEAKER_02: So tell me a little bit more about what your team does specifically and then how Airflow fits into that data management.
• 3:08
  SPEAKER_00: Um, so our whole IT team, we connect operational manufacturing data across the whole Shopflow systems.
• 3:14
  SPEAKER_00: So Shopflow is basically where you have like a live manufacturing going on.
• 3:18
  SPEAKER_00: They take off the screws, they do everything, small, small parts, everything.
• 3:22
  SPEAKER_00: So we have like enterprises database, cloud platforms, you know, analytic tools.
• 3:27
  SPEAKER_00: So we use Airflow has orchestration for like ETLs and ELT jobs, and we do data quality checks and production reporting and file processing.
• 3:37
  SPEAKER_00: We operate almost on thousands of DACs and across different schedules and business processes.
• 3:42
  SPEAKER_00: So that's how a whole team works on like manufacturing data, so systems, and enterprises data systems.
• 3:49
  SPEAKER_00: Yeah.
• 3:50
  SPEAKER_02: Okay, great.
• 3:51
  SPEAKER_02: And then it sounds like so Airflow is kind of your central orchestrator within all of that.
• 3:56
  SPEAKER_00: Yeah, uh, so we have all the workflows, like into centralized platform, data platform, which is Airflow.
• 4:02
  SPEAKER_00: Uh it gives us a common way to manage dependencies, scheduling, uh, monitoring, auditability.
• 4:07
  SPEAKER_00: So even a small percentage of failures, everything workload is taken by Airflow.
• 4:13
  SPEAKER_00: So that's our main centralized management uh dependency system.
• 4:16
  SPEAKER_00: Yeah.
• 4:17
  SPEAKER_02: Awesome.
• 4:18
  SPEAKER_02: That's great background.
• 4:20
  SPEAKER_02: And now I want to dig into this self-healing pipeline use case that I mentioned at the front.
• 4:26
  SPEAKER_02: This is something that I keep hearing more and more about, and it's something that Astronomer is uh very interested in as well and is implementing as part of our product.
• 4:36
  SPEAKER_02: Uh so I want to hear about how you've done that.
• 4:38
  SPEAKER_02: And maybe first we can talk about just kind of conceptually what does self-healing mean in the context of your setup?
• 4:47
  SPEAKER_00: So when we really talk about self-healing, it means like let's say you have done a small pipeline.
• 4:52
  SPEAKER_00: Uh, can that pipeline identify a known failure or a condition, whether it's still safe or like it can recover automatically?
• 5:01
  SPEAKER_00: So, for example, you are performing an approved recovery action, you need to validate the results.
• 5:06
  SPEAKER_00: So, continuous processing without you know having a person, like for example, I'm a senior data architect.
• 5:11
  SPEAKER_00: If you have a pipeline which can just self-heal, you have that in this word itself.
• 5:16
  SPEAKER_00: You have a bandage and you just put a bandage and it heals itself, you don't need like you know, kind of extra stitches or something.
• 5:23
  SPEAKER_00: That's what I would call it as self-heal.
• 5:26
  SPEAKER_00: So without intervention of any person, uh a pipeline is identifying a problem and determining if it is like you can recover automatically and having an action plan and validating the results.
• 5:37
  SPEAKER_00: That's the whole part of self-healing.
• 5:40
  SPEAKER_00: It doesn't mean like you know, giving an AI unrestricted control flow.
• 5:44
  SPEAKER_00: So our approach in ATC drivetrain is like you have a deterministic uh automation, you have policies and AI like you know assisted diagnosis.
• 5:54
  SPEAKER_00: For example, we had a transient database connectivity delay.
• 5:59
  SPEAKER_00: So there was like upstream dependency, a temporary, you know, some API went down.
• 6:04
  SPEAKER_00: So what we did was like we had set up a policy saying that if an API goes down or a resource capacity goes down, you can recover by yourself automatically.
• 6:15
  SPEAKER_00: If there's like uh unexpected schema changes, no unexpected schema changes happening.
• 6:20
  SPEAKER_00: So it did check the data quality, security, and business logics which we had set up instead of escalating to a person.
• 6:27
  SPEAKER_00: We did get a mail like, hey, this went down.
• 6:30
  SPEAKER_00: So it we I took it in in, Airflow took it in and just like recovered on itself.
• 6:37
  SPEAKER_00: So it's not about eliminating the engineers from the process, it's just you know, a person is like fixing the same stuff again and again, our recovery work so that Airflow can do it on its own.
• 6:50
  SPEAKER_00: So that's what is self-healing.
• 6:52
  SPEAKER_00: It heals on itself uh instead of putting a burden on the engineer, which is on or concentrating on, which is really genuinely need concentration on, for example, projects or any investigations they need to do.
• 7:04
  SPEAKER_02: Yeah, it's certainly a much better experience for the engineers to you know log on and get a notification that hey, there was an error and now it is solved, and here's how it was solved, or now there's a PR that does solve it that's open for your review, as opposed to just a red square, and then you have to start from nothing to figure it out.
• 7:24
  SPEAKER_02: So yeah, that's very cool.
• 7:27
  SPEAKER_02: And tell me a little bit more about how you've implemented this from an architecture standpoint.
• 7:33
  SPEAKER_02: So, how does the system detect the failure?
• 7:36
  SPEAKER_02: And then what by what mechanism is it doing that recovery?
• 7:41
  SPEAKER_00: So when we talk about architecture, first is like self-healing or anything, right?
• 7:46
  SPEAKER_00: So it should have like we had like five layers, I would say.
• 7:49
  SPEAKER_00: First in first is the observability layer.
• 7:51
  SPEAKER_00: Like even a person is usually is like if there's a failure, what do what happened?
• 7:57
  SPEAKER_00: What are the locks?
• 7:59
  SPEAKER_00: So we had airflow task, which calls back and collects like the failure task, logs, error messages, you know, metadata, and what was the dependency state, history, execution time, you know, infrastructure signals.
• 8:11
  SPEAKER_00: This gives the idea of like to the A4, like, hey, something went wrong.
• 8:15
  SPEAKER_00: I'm observing it, I'm collecting the data.
• 8:18
  SPEAKER_00: So, second is like you have the use huge data, but you need to classify it.
• 8:23
  SPEAKER_00: So the system compares like incident with like you know, what were the failures?
• 8:27
  SPEAKER_00: Let's say we usually use Airflow only for something which has already repeated, happened.
• 8:34
  SPEAKER_00: We are not exposing AI with that whole control, like, hey, this is a new incident, you just don't want to have it to take a decision and it does happen.
• 8:42
  SPEAKER_00: So classification.
• 8:43
  SPEAKER_00: So it goes to the library of known failure patterns, it includes like you know connections, timeouts, you know, upstream data missing or like you know, resource limit.
• 8:52
  SPEAKER_00: So it just compares with the data quality failures or any files, then comes the policy layer.
• 8:57
  SPEAKER_00: We would have set up with a bunch of policies saying, hey, this was a failure, but you need to check all of this.
• 9:03
  SPEAKER_00: If anything is changed, make sure there's automatic recovery permitted, go back to that category, get the confidence level.
• 9:12
  SPEAKER_00: Uh, you had this number of previous attempts.
• 9:14
  SPEAKER_00: If it is not done by you, like you know, approved recovery actions, whatever you go through, and you match each and every policy layer.
• 9:23
  SPEAKER_00: And the fourth comes the recovery.
• 9:25
  SPEAKER_00: So, depending on the pattern, the confidence level, you try to go off back off, maybe re-establish a connection, you know, rerun a task or like a resource management or processing from a safe point, you can do that.
• 9:39
  SPEAKER_00: But if something happens or you're not sure, just you know, send a mail to the engineer or trigger a big alarm or something.
• 9:48
  SPEAKER_00: So that's like the four layers.
• 9:50
  SPEAKER_00: Once you have validated, sorry, you have recovered, you have gone through the policy, you need a validation layer that if the recovery actually has happened and succeeded.
• 9:59
  SPEAKER_00: So you put some tests which goes through like rogue forms, time stamp, data quality rules, or downstream dependencies.
• 10:07
  SPEAKER_00: So if you have that validation done, so the whole thing, the architecture is this five layers.
• 10:14
  SPEAKER_00: So the system stops.
• 10:15
  SPEAKER_00: Let's say you escalate an incident and you did like a recovery, the validation is done that the system has started, then your whole complete architecture depends on that.
• 10:26
  SPEAKER_00: So I would say observe, classify, policy, and recovery and validate.
• 10:32
  SPEAKER_00: So these are the five architecture pieces for a failure and recovery on its own.
• 10:36
  unknown: Yeah.
• 10:37
  SPEAKER_02: Yeah, that makes sense.
• 10:39
  SPEAKER_02: And tell me a little more about the AI piece of the recovery loop.
• 10:43
  SPEAKER_02: So you're using the AI to help figure out how to solve whatever the issue is, and then decide whether it needs to be routed to a human first or not.
• 10:53
  SPEAKER_02: Kind of what has been your experience working with it in that way?
• 10:57
  SPEAKER_02: And has it required a lot of tuning, or do you have any, I guess, built-in mechanism for kind of improvement in that over time?
• 11:06
  SPEAKER_00: Yes, I would say like AI primarily helps with classification and context gathering.
• 11:10
  SPEAKER_00: You know, for example, let's say that AFlow does have a lot of context, but you need AI to help it like you know, classify it, gather it, and recommend limitations.
• 11:20
  SPEAKER_00: So, for example, I have a startup called Resetly.
• 11:22
  SPEAKER_00: So we have like a lot of data, especially from financial data.
• 11:26
  SPEAKER_00: So traditional runes remain effective when there's like you know, repeated failures or something.
• 11:32
  SPEAKER_00: Let's say uh you AI becomes more useful when the error is like less structured, and you need to have like consider all the signals together.
• 11:41
  SPEAKER_00: Let's say you have like a thousand or through two thousand locks, and you need to identify the likely root cost and compare with the failure with historical incidents or you know, recommend the next uproot actions.
• 11:53
  SPEAKER_00: AI is the best one to do that, it should not bypass the operational controls.
• 11:59
  SPEAKER_00: Uh, the AI should can propose that an issue resembles you know a transient failure.
• 12:04
  SPEAKER_00: For example, let's say you have a policy layer, whether a retry is safe or permitted.
• 12:09
  SPEAKER_00: Let's say you you retry it like four times, but is was is it safe to retry again or permitted?
• 12:14
  SPEAKER_00: So it depends on the high confidence or low risk, you know, the action can be automatic.
• 12:18
  SPEAKER_00: So AI comes into picture with with a lot of these cases.
• 12:22
  SPEAKER_00: So AI prepares the diagnosis, recommended steps for the engineer, it gives us the speed, like you know, while maintaining the production governance or AI policies, especially when you're dealing with financial data and also the automatic automated, which is like a live data.
• 12:35
  SPEAKER_00: Both of them have their own risk.
• 12:37
  SPEAKER_00: And AI uh stands behind to help airflow to rely on like some of the decisions, gathering approved actions.
• 12:46
  SPEAKER_00: But most of the things AI does is like it recommends rather than uh just sitting there and you know just taking the own decision.
• 12:54
  SPEAKER_00: That would be the best case of scenario of like AI fit into the recovery loop.
• 12:59
  SPEAKER_00: You should have AI placed in the back set, helping you out with all of the stuff, but you should have your approval steps, even for a recovery or a human involved in that.
• 13:10
  SPEAKER_00: Yeah.
• 13:12
  SPEAKER_02: Yeah, that's really helpful.
• 13:14
  SPEAKER_02: And I love the example that you gave.
• 13:16
  SPEAKER_02: I'm curious if you have another example you can talk about around like a failure that you might have had before you had the self-healing in place and kind of how that's improved now with the new architecture.
• 13:31
  SPEAKER_00: Uh, for example, when Recetly came into picture, like I have a Recetly app, which is like takes into Xpense app basically.
• 13:39
  SPEAKER_00: So we had like a lot of people started using it and it had a failure because it was the currency exchange, just giving an example.
• 13:48
  SPEAKER_00: And A4 Airflow triggered like, hey, this is currency exchange and the rows are not matching up.
• 13:54
  SPEAKER_00: So we had done an AI policy saying, like, let's say you get an in dollars or something, uh, you need to trigger to change it to Euros or pounds if the person is traveling when he is on or something.
• 14:08
  SPEAKER_00: So it triggered, but it did not work.
• 14:11
  SPEAKER_00: So Airflow recommended the steps, AI recommended the steps, which saved us almost downtime of four hours, which would have costed us almost 120k.
• 14:21
  SPEAKER_00: So wow, yeah, so the DAX, all the formulas, you know, there's a lot of stuff which happens.
• 14:28
  SPEAKER_00: We we say, like, hey, it's an expense app, and uh, we have like the point of sales connected to you know, you buy something and which is like 45 dollars, but it was showing like 450 dollars uh 450 euros pounds, which had like transit in it.
• 14:41
  SPEAKER_00: So AI recommended that it happened at 12 a.m.
• 14:44
  SPEAKER_00: So all of us were asleep or something, it would get a big warning that self-healing with the AI saying, like you need to convert this, this number of rows are not matching.
• 14:54
  SPEAKER_00: It saved us a lot of money, especially production time.
• 14:58
  SPEAKER_00: Uh I mean not production, I would say downtime, and also the money-wise.
• 15:02
  SPEAKER_00: Yeah.
• 15:03
  SPEAKER_02: Yeah, that's an amazing success story.
• 15:07
  SPEAKER_02: How often do you feel like it gets the right answer in terms of the recovery?
• 15:12
  SPEAKER_02: Like, what is the success rate overall?
• 15:15
  SPEAKER_00: Uh it depends.
• 15:17
  SPEAKER_00: So when we talk about recovery time or recovery success, it depends on the confidence level.
• 15:22
  SPEAKER_00: For when you're setting up an AI, I'm saying, like, you know, everyone uses AI, there's a downtime, everyone, you know, this suddenly there's a browser which goes down, let's say.
• 15:32
  SPEAKER_00: So Airflow has something called as confidence levels.
• 15:36
  SPEAKER_00: Let's say if your failure falls in between like 10 to 25 percent, the success rate is almost 99%.
• 15:44
  SPEAKER_00: If it is a higher risk, it may go below, you know, kind of 50%.
• 15:49
  SPEAKER_00: Also, it depends on the confidence level, but I would say 95% of the failure modes it fixes, which can be like a small API or any stuff, it just fixes on its own.
• 15:59
  unknown: Yeah.
• 16:02
  SPEAKER_00: Yeah.
• 16:02
  SPEAKER_00: Uh I would say like 95% of the time, small, small errors, it just takes care.
• 16:07
  SPEAKER_02: That's amazing.
• 16:08
  SPEAKER_02: That must have saved your team so much time.
• 16:10
  SPEAKER_01: That's absolutely this episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 16:22
  SPEAKER_01: From batch pipelines to training models and wiring up agents, Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 16:33
  SPEAKER_01: Check it out at astronomer.io.
• 16:38
  SPEAKER_02: Cool.
• 16:39
  SPEAKER_02: Well, I would love to also talk about AI-assisted troubleshooting.
• 16:44
  SPEAKER_02: So maybe in the case where you do need a little more human involvement to figure out what's going on.
• 16:50
  SPEAKER_02: You mentioned that you're running EarthLow at quite a high scale.
• 16:54
  SPEAKER_02: So I think you told me you have thousands of DAGs.
• 16:57
  SPEAKER_02: So when when you're working with that sort of volume, how does AI help someone find the root cause of a failure?
• 17:07
  SPEAKER_00: So when I talk about like a lot of stuff, when so for an engineer, I'm just saying you trigger a failure, you just go into the locks.
• 17:16
  SPEAKER_00: Let's say uh a piece of information doesn't give so much context.
• 17:22
  SPEAKER_00: So when you need a lot of connection and it takes it saves a lot of time.
• 17:26
  SPEAKER_00: Let's say for me as a senior data architect, I sit down with automative the whole table, database, or airflow or something, right?
• 17:34
  SPEAKER_00: So when you have in this transaction locks, which is like thousands of rows, AI gives us the contextual locks.
• 17:42
  SPEAKER_00: Sometimes you don't understand, like you're not an expert in everything.
• 17:46
  SPEAKER_00: AI does give you, like, you know, for example, let's say something like 404.
• 17:50
  SPEAKER_00: I will be like, what's that?
• 17:52
  SPEAKER_00: I'm gonna Google it, I have to spend a lot of time.
• 17:54
  SPEAKER_00: So AI does give you contextual locks information, which gets from a different, you know, it just not one connection like DTSQL01 or a SQL or any of the database.
• 18:05
  SPEAKER_00: So it gathers all the information and gives you like, hey, you may need to check this connection.
• 18:10
  SPEAKER_00: This API may have failed.
• 18:11
  SPEAKER_00: It's triggering a lot of upstreams or downstreams data loss or something.
• 18:17
  SPEAKER_00: So the biggest challenge is not detecting something which has failed.
• 18:21
  SPEAKER_00: So it gives the visibility and also recommends action plans.
• 18:26
  SPEAKER_00: You may not have that in mind, oh, I may do this, it will fix this API.
• 18:30
  SPEAKER_00: So uh let's say 27 task failures is connected to some one unavailable you know source database, which the AI is checking, which I can't check, like sometimes I may forget it.
• 18:40
  SPEAKER_00: So the whole scenario is that AI gives us visibility, observability, recommended actions.
• 18:48
  SPEAKER_00: So even with thousands of lines, connections, and everything, AI sits in the middle to have a recommendation system.
• 18:55
  SPEAKER_00: So that's helps a lot with especially with this big environment.
• 18:59
  SPEAKER_00: And especially airflows, you have like a manual scheduling or any scheduling, AI sits back of the airflow and helps with all the stuff, which is like recovery, self-healing.
• 19:11
  SPEAKER_00: That's the center of it, yeah, I would say.
• 19:14
  SPEAKER_02: Yeah, absolutely.
• 19:15
  SPEAKER_02: At Astronomer, we have built some of this into our own product where we have our data engineering agent, which is called Auto, can investigate failures for you.
• 19:24
  SPEAKER_02: And the whole purpose is that you can just click a button and it will do it, and you don't ever have to open the logs, which I, as you said, I think is so helpful the amount of time that I have spent trying to figure out like what an error message in the log means, or even trying to find like the right part in the logs.
• 19:40
  SPEAKER_02: Like sometimes the thing that generates the error is not really the actual issue, it's like multiple steps up or down, and it's just cascaded.
• 19:49
  SPEAKER_02: And the AI is like super fast at figuring out, like, oh, this is the actual problem.
• 19:54
  SPEAKER_02: Like, this is great.
• 19:56
  SPEAKER_00: Yes, I mean agents, especially nowadays, is like when you try to use agents in a very specific way, it helps you like a lot save a lot of money, I would say, in terms of money, the time, and not having the stressful day for the engineer rather than doing more projects, more productivity.
• 20:12
  SPEAKER_00: And the first thing is no downtime.
• 20:15
  SPEAKER_02: So, yeah, yeah, definitely opens the door to doing a lot more, it's just like net productive work as opposed to, like you said, lots of Googling of errors.
• 20:26
  SPEAKER_00: Yes, that's absolutely true.
• 20:27
  SPEAKER_00: I've seen my uh network protocols, people like especially shutting the door and like going on like, hey, we can use this, you know, autopilot or something.
• 20:36
  SPEAKER_00: If you own the you know AI agents, you don't need to train the you know some other resources or something, but if you're owning the AI agents, it helps you to keep the company conserved information within itself and help you out.
• 20:49
  SPEAKER_00: That's one of the things here.
• 20:51
  SPEAKER_02: Yeah, totally.
• 20:53
  SPEAKER_02: Well, speaking of agents, I on a slightly different topic, still related to AI and Airflow, but I understand that you are also working with AI agents that run through Airflow.
• 21:05
  SPEAKER_02: Another very popular topic.
• 21:07
  SPEAKER_02: Uh, we do this on my team at Astronomer as well.
• 21:10
  SPEAKER_02: But I'm curious to learn more about how you are using Airflow for agent orchestration.
• 21:16
  SPEAKER_00: Um, we do have like four agents running uh in ATC TriTrain.
• 21:20
  SPEAKER_00: I also have two agents running for Recitly actually.
• 21:23
  SPEAKER_00: So we treat um AI agents as governed workflow components rather than independent process running without like any any boundaries.
• 21:32
  SPEAKER_00: So Airflow is responsible, you know, scheduling, state management, retires, timeouts, logging, and everything.
• 21:38
  SPEAKER_00: So for us in ATC TriTrend or in Ricitly, uh, an agent perform a boundary task such as like classifying an incident, summarizing logs, checking metadata, or like you know, recommendation in a plan or validating.
• 21:51
  SPEAKER_00: So, for example, I have in recitly one agent just analyzes the failure because uh recitly needs to be live, especially.
• 22:00
  SPEAKER_00: Especially for point of sale POS systems, so it can't go down all the time.
• 22:04
  SPEAKER_00: And also we have second agent running all the time because you have this historical incidents happening all the time with the item not matching with the bill and giving wrong, you know, the item amount or something.
• 22:18
  SPEAKER_00: So we have like two running there, but in ATC we have like another one running all the time, which is the data quality, because data quality and data is the most heart of all this automotive industry.
• 22:33
  SPEAKER_00: So Airflow coordinates those steps and records those inputs and outputs with the agent with the help of the agent.
• 22:38
  SPEAKER_00: So whole agent behavior is more of like absorbable and reproducible.
• 22:43
  SPEAKER_00: We control those tools.
• 22:44
  SPEAKER_00: So uh it's it acts up on the human approval, uh, and if there's a sensitive step coming through, it stops execution based on the confidence level or a validation fails, then it triggers us.
• 22:58
  SPEAKER_00: So agent is giving us a contextual reasoning, and airflow is giving the operation discipline for the whole um 82 drive train.
• 23:07
  SPEAKER_02: I would say interesting.
• 23:08
  SPEAKER_02: So it's just like a smarter data quality check.
• 23:12
  SPEAKER_02: Yes, it has some reasoning behind it.
• 23:14
  SPEAKER_02: Yeah, that's really cool.
• 23:16
  SPEAKER_02: I haven't actually heard of somebody doing that yet.
• 23:19
  SPEAKER_00: Yep.
• 23:19
  SPEAKER_00: Uh, this was actually a project brought by someone from Europe.
• 23:23
  SPEAKER_00: Actually, he uh like he eliminated like a lot of work of QE engineers.
• 23:30
  SPEAKER_00: So that was like this agent.
• 23:33
  SPEAKER_00: Uh, we may think it's a small data quality agent, but especially working in ATC drivetrend, we have like almost, I would say 17,000 rows of data every hour, like literally flowing.
• 23:44
  SPEAKER_00: So data quality is like it may be a work of like six to seven people.
• 23:48
  SPEAKER_00: So that agent is saving a lot of money any downtime or any triggers or anything.
• 23:53
  SPEAKER_00: Yeah.
• 23:54
  SPEAKER_02: Yeah, absolutely.
• 23:56
  SPEAKER_02: Well, very interesting.
• 23:57
  SPEAKER_02: Uh cool.
• 23:59
  SPEAKER_02: Well, yeah, thank you for going through all of this.
• 24:01
  SPEAKER_02: It's been, like I said, this is a really hot topic within the Airflow community right now, and it's always helpful for uh folks to get to hear how somebody is implementing this stuff.
• 24:14
  SPEAKER_00: I mean, I've heard like everyone talk about this AI agents, you know, something's wrong or something's correct.
• 24:20
  SPEAKER_00: You know, some people say it's saved on their side, some people say like it's out of control.
• 24:25
  SPEAKER_00: It depends on how we use put boundaries and how you use it.
• 24:28
  SPEAKER_00: And even for Airflow, Airflow saves a lot of time for especially for recovery or you know, capabilities.
• 24:34
  SPEAKER_00: It's it's more of like the smaller team, the larger team, how to use it.
• 24:38
  SPEAKER_00: That's all at the end of the day.
• 24:40
  SPEAKER_02: Absolutely.
• 24:41
  SPEAKER_02: All about the implementation, and as you mentioned, very importantly the guardrails in this case.
• 24:46
  SPEAKER_00: Yeah.
• 24:47
  SPEAKER_02: Yeah.
• 24:48
  SPEAKER_02: Cool.
• 24:49
  SPEAKER_02: Well, I think that's a good place to pivot to my standard final question, uh, which is what would you most like to see from development of the Airflow project in the near future?
• 24:59
  SPEAKER_00: Uh, one of the stronger is like Airflow should continue on stronger capabilities around observability, event-driven orchestration.
• 25:07
  SPEAKER_00: And also, I would ask them to more focus on the AI agent governance because for AI agent, it's uh I would like to standard as ways to track prompts, tool calls, you know, model versions, approvals, confidence, maybe more on agent outputs as a part of the task history that would make easier, you know, the for the agent workflows to be easier to audit it, operate safely for AFLO.
• 25:30
  SPEAKER_00: So I would want it to be more flexible, even-driven execution, you know, for dynamic log running workflows.
• 25:37
  SPEAKER_00: And I mean, AFLO is already a powerful orchestrator, like it's a very based on governance and everything.
• 25:44
  SPEAKER_00: But I would have more on like AI assisted workflows to make it more important for especially like enterprise AI operations, which we use.
• 25:52
  SPEAKER_00: So maybe that.
• 25:54
  SPEAKER_02: Yeah, for sure.
• 25:55
  SPEAKER_02: Sounds like some requests for future development in the common AI provider, which I know is moving forward quite quickly right now.
• 26:01
  SPEAKER_02: They keep adding it, it feels like they add new functionality to that every day.
• 26:05
  SPEAKER_00: But yeah, that's true.
• 26:06
  SPEAKER_00: Like it's hard to track, also, you know, like they just do some few releases and it's really hard, but they're doing like very quickly, also.
• 26:14
  SPEAKER_00: Yep.
• 26:14
  SPEAKER_02: Yeah, yeah.
• 26:16
  SPEAKER_02: Cool.
• 26:16
  SPEAKER_02: Well, I love that answer.
• 26:18
  SPEAKER_02: Um, Kamuda, thank you so much for going through that.
• 26:20
  SPEAKER_02: It's been really great speaking with you.
• 26:21
  SPEAKER_02: Just to close this out, what is the best way for folks to get in touch with you?
• 26:25
  SPEAKER_00: Uh, they can reach out to me through LinkedIn, uh, Kumudashrinivas.
• 26:28
  SPEAKER_00: So that's my whole linked it, or they can just shoot me a mail at Kumudashunivas and Gmail.com if you have any questions or you want to reach out for anything else.
• 26:37
  SPEAKER_00: Yeah.
• 26:38
  SPEAKER_02: Great.
• 26:38
  SPEAKER_02: We can throw links to those in the show notes.
• 26:40
  SPEAKER_02: Well, thank you again.
• 26:41
  SPEAKER_02: This has been a great conversation.
• 26:44
  SPEAKER_00: Thank you so much for having me here, and it was a wonderful conversation with you.
• 26:47
  SPEAKER_00: Yeah.
• 26:48
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 26:51
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 26:55
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 27:00
  SPEAKER_01: And leave us a five star review.