Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating local LLMs for sensitive data with Airflow

Transcript

Orchestrating local LLMs for sensitive data with Airflow · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:26
  SPEAKER_01: $150 value.
• 0:28
  SPEAKER_01: Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
• 0:35
  SPEAKER_01: Now let's get into the show.
• 0:38
  SPEAKER_00: Okay, hi everyone.
• 0:39
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:41
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Chiyonk Jane, who is a senior ML engineer at Visa.
• 0:47
  SPEAKER_00: Chiyonk, how are you doing?
• 0:49
  SPEAKER_00: I'm doing good.
• 0:50
  SPEAKER_00: How are you?
• 0:51
  SPEAKER_00: I'm good.
• 0:52
  SPEAKER_00: It's great to have you on.
• 0:53
  SPEAKER_00: I'm excited to dig into it.
• 0:55
  SPEAKER_00: Uh for today's podcast, we're going to discuss orchestrating local LLMs for sensitive data pipelines with Airflow.
• 1:02
  SPEAKER_00: Of course, this is a very popular topic right now.
• 1:06
  SPEAKER_00: Lots of talk within the Airflow community about using Airflow for AI orchestration of various kinds.
• 1:13
  SPEAKER_00: So I'm really excited to get your perspective on that.
• 1:16
  SPEAKER_00: Before we dive into all of the details, maybe let's start with a little bit of background.
• 1:21
  SPEAKER_00: Tell me about yourself and you know how you got into ML engineering and what you do with Airflow and data pipelines.
• 1:29
  SPEAKER_02: Sure.
• 1:29
  SPEAKER_02: So uh right now I am a senior ML engineer working on inference and LLM system in payments.
• 1:36
  SPEAKER_02: A lot of my time goes in uh serving with Onyx runtime.
• 1:40
  SPEAKER_02: Some some of this there is rust in the serving path.
• 1:43
  SPEAKER_02: But lately, a lot of my time and interest has been going in agentic tooling with Landgraph.
• 1:48
  SPEAKER_02: But I exactly didn't come here with a mindset of building pipelines or something around that.
• 1:56
  SPEAKER_02: I came in through modeling, not just uh uh not just infrastructure.
• 2:00
  SPEAKER_02: Um my first job out of school was uh uh NLP, uh NLP engineer in pharma.
• 2:06
  SPEAKER_02: Uh I built a tool with Spacey, which had which had medical terminology dictionary and compared drug side effects.
• 2:13
  SPEAKER_02: Then I went into risk modeling, then I did uh masters in computer science and spent a few years in clinical machine learning.
• 2:22
  SPEAKER_02: Then I led a five time a five uh five people team building re-admission risk model for electronic health system.
• 2:31
  SPEAKER_02: And that's exactly where my uh my mind shifted, and that's exactly the job that turned me into a pipeline person because we pulled data from uh different hospitals, some were running on Epic, some were running on someone running on Cerner, and there were different formats, different delivery time.
• 2:52
  SPEAKER_02: And I realized I was able to write the core model logic in a few months or um five or six weeks, but I spent a lot more time than expected uh with uh airflow dash, data quality, and backsfill.
• 3:08
  SPEAKER_02: And that's where it clicked to me that uh model is just one task, and there let's say there are 40 tasks, model is just one task, and there are 39 others that I need to look into.
• 3:20
  SPEAKER_02: So that's that's how it came here.
• 3:22
  SPEAKER_00: Yeah, that's a great summary, I think, of one of the challenges with AI orchestration is people sort of think about it as the model that you're working with, but that's really just a small piece of the overall picture, and orchestration helps with a lot of that other stuff.
• 3:37
  SPEAKER_00: So, yeah, very cool.
• 3:39
  SPEAKER_00: And tell me a little bit more about your current work.
• 3:42
  SPEAKER_00: So you're a senior ML engineer, you're focused on these LLM systems.
• 3:46
  SPEAKER_00: What does that look like in your day-to-day?
• 3:49
  SPEAKER_02: So, in day-to-day, uh, let's say, like let me divide it into uh few categories.
• 3:55
  SPEAKER_02: So, one one of uh so some some of my time goes in serving, getting to uh getting a model to run fast, uh identically every time, and managing the runtime session, tuning, batching, memory behavior, keeping the tail latency flat.
• 4:12
  SPEAKER_02: That takes a lot of time and thinking.
• 4:14
  SPEAKER_02: Then, very important next part is failure design.
• 4:17
  SPEAKER_02: So, let's say everything is not ideal in this world.
• 4:20
  SPEAKER_02: So, what happened when the model times out?
• 4:22
  SPEAKER_02: Uh what when the tool call fails?
• 4:24
  SPEAKER_02: Uh, what happens when the output parsing is wrong?
• 4:28
  SPEAKER_02: So, building that failure design so we can rely on that's the second part where I spend a lot of time.
• 4:34
  SPEAKER_02: And the third, with the new way the AI is changing, that's agentic tooling.
• 4:39
  SPEAKER_02: Uh, let's say we build a land graph pipeline with multiple MCP tools, uh let observability setup, benchmarking, a lot of this stuff.
• 4:47
  SPEAKER_02: But uh we need to make sure the tools are getting uh called properly in the right order, and we are getting the expected result.
• 4:56
  SPEAKER_02: And then we work with Airflow, Kubernetes, Grafana, all threaded all together.
• 5:02
  SPEAKER_02: Model maybe one-fifth of my work, but a lot of uh a lot of my time goes in uh surveying failure design and agent tooling.
• 5:12
  SPEAKER_00: Okay, got it.
• 5:13
  SPEAKER_00: That makes sense.
• 5:14
  SPEAKER_00: And then you spoke recently at the Airflow Summit about, I think, some of this work on orchestrating local LLMs for sensitive data pipelines.
• 5:24
  SPEAKER_00: So let's talk a little bit more about that aspect of your work, especially around the sensitive data piece.
• 5:31
  SPEAKER_00: Uh, tell me before we talk about the implementation, kind of what is the core problem that you're trying to solve here?
• 5:39
  SPEAKER_02: Sure.
• 5:39
  SPEAKER_02: Uh so let me take an example of it can be any sensitive field, be it government, healthcare, finance.
• 5:47
  SPEAKER_02: Uh let me take an example of a healthcare field.
• 5:52
  SPEAKER_02: So uh let's say we have uh enormous clinical text data, just our summary, nursing node, physician narrative, and there are a lot of other free test text.
• 6:02
  SPEAKER_02: And these these are unstructured records.
• 6:05
  SPEAKER_02: We might have a structured record which gives us the ICD code, but they don't give us the real story behind it.
• 6:11
  SPEAKER_02: The real story comes from the unstructured data.
• 6:14
  SPEAKER_02: Let's say unstructured node says the patient lives alone and they don't have any transportation to follow up to the clinic.
• 6:21
  SPEAKER_02: That's where the real that's where the real knowledge is.
• 6:27
  SPEAKER_02: And we need text turned into a structured uh structured features for a model, uh, be it financial model, healthcare model, uh, there can be notes from there can be notes out of financial data as well.
• 6:40
  SPEAKER_02: So that's the classic extraction problem, and LLMs are generally good at it.
• 6:45
  SPEAKER_02: But the blocker, when it comes to sensitive data, they have PHIs, protected health information, they have uh they have credit card information, or the person who is making that transaction, they have information about that.
• 6:58
  SPEAKER_02: So the first is you cannot make that data leave out of the system.
• 7:03
  SPEAKER_02: So calling a hosted API is off the table at all, not on the cost, not on the quality.
• 7:09
  SPEAKER_02: So here the problem becomes this.
• 7:11
  SPEAKER_02: Uh the problem becomes run an open weight model like Lama 3 on your own GPU over million of nodes, let's say every night on a on a schedule, with enough checking that the downstream clinical uh model is allowed to depend on it output.
• 7:27
  SPEAKER_02: It's a very sensitive field.
• 7:28
  SPEAKER_02: So people should be able to depend on that.
• 7:31
  SPEAKER_02: And the last uh the last clauses where all it works, getting a Lama 3 to summarize one of the nodes nicely is just a Tuesday afternoon.
• 7:39
  SPEAKER_02: Getting it down to 2 million every night, reproducibility with a paper tail is the real system.
• 7:46
  SPEAKER_02: That's exactly what the whole talk is about, and how we can do it in a good engineering way.
• 7:53
  SPEAKER_00: Yeah, that makes sense with that data.
• 7:55
  SPEAKER_00: That with uh anything super sensitive, you can't send it to external systems, and the traceability and kind of data quality aspects become really, really important.
• 8:05
  SPEAKER_00: So then tell me about where Airflow fits into this architecture.
• 8:11
  SPEAKER_02: Sure.
• 8:12
  SPEAKER_02: So Airflow sits on top of it, like it never touches the data, but it orchestrates it.
• 8:18
  SPEAKER_02: So it's one of the most important parts.
• 8:22
  SPEAKER_02: Let's say the heavy lifting is done by GPU ports on Kubernetes that run the model, object storage holds the uh raw nodes, Postgres holds the result, but Airflow is the one which decides what's to what to run, in what order, and what happens if any of this uh service breaks.
• 8:40
  SPEAKER_02: So if I have to divide, let we can divide it into four jobs.
• 8:44
  SPEAKER_02: One is the sensors.
• 8:45
  SPEAKER_02: So we say every site, let's say every site drops files on its own schedule, and some nights there might not be any files.
• 8:53
  SPEAKER_02: So there is a sensor per site waiting on its data, and each site has its own task.
• 8:58
  SPEAKER_02: So if one site is four hours later, only that branch waits.
• 9:03
  SPEAKER_02: So that's what AFLO comes into picture and it decides which branch to wait, which branch to run.
• 9:09
  SPEAKER_02: And when it when I talk about branches, second thought can come about dependencies.
• 9:15
  SPEAKER_02: Validation directly cannot start until uh validation cannot start until inference for that batch finishes.
• 9:21
  SPEAKER_02: So load cannot start until validation passes.
• 9:25
  SPEAKER_02: That ordering itself is the entire safety design and it literally lives inside the DAG instead of somebody's head, and Airflow is the one that's orchestrating it.
• 9:36
  SPEAKER_02: Next is the retries and backfills.
• 9:39
  SPEAKER_02: What if uh the site uh delivers one day late?
• 9:42
  SPEAKER_02: Or uh let's say there was a there was a failure.
• 9:46
  SPEAKER_02: So we need to make sure we don't retry everything.
• 9:50
  SPEAKER_02: Airflow takes care of retrying that specific node or backfilling that specific uh specific data.
• 9:56
  SPEAKER_02: So you don't need to rerun everything again.
• 9:59
  SPEAKER_02: And the last is the run record.
• 10:02
  SPEAKER_02: When DAG runs with with configs, uh, we need to know which config is it, which model version it's producing, and that's the linear start.
• 10:10
  SPEAKER_02: And you get it free because uh Airflow is tracking every run for it.
• 10:15
  SPEAKER_02: So this is where uh Airflow is an integral part of any orchestration that we do, be it uh LLMs or traditional uh ML systems.
• 10:27
  SPEAKER_00: Yeah, that's interesting.
• 10:29
  SPEAKER_00: I think you're describing a lot of the very bread and butter airflow orchestration functionality, and I love how you've talked about how it's so important for this use case.
• 10:37
  SPEAKER_00: That's great.
• 10:39
  SPEAKER_00: I'm wondering if you can give me any more detail on how you're structuring those DAGs.
• 10:44
  SPEAKER_00: So you're managing quite a lot with them from the LLM inference and the validation and those ingestion stages.
• 10:49
  SPEAKER_00: What do those DAGs look like?
• 10:53
  SPEAKER_02: So, yeah, the uh DAGs can have multiple nodes, a lot of connections.
• 10:57
  SPEAKER_02: There can be multiple stages, but if I have to divide it into some core buckets, I can divide it into four stages.
• 11:05
  SPEAKER_02: Stage one, we can say we can ingest and normalize it.
• 11:09
  SPEAKER_02: So one task per group site, uh, we can run all in parallel.
• 11:12
  SPEAKER_02: Each one lands raw nodes in a big object storage, let's say S3, and writes those manifest rows so you know what arrived, when it arrived, and how it arrived.
• 11:24
  SPEAKER_02: Once we have all that data, second can be the inference.
• 11:29
  SPEAKER_02: So all these nodes or the data, it can get grouped in badges and run on GPU, GPU pods, and that can be done through Kubernetes pod of uh operators.
• 11:41
  SPEAKER_02: So the GPU works in isolation from the scheduler.
• 11:43
  SPEAKER_02: That's very important to keep things isolated at their own level.
• 11:48
  SPEAKER_02: And this stage sits in its own air uh airport, sorry, airflow pool uh with no hard concurrency cap.
• 11:55
  SPEAKER_02: So here we need to make sure uh we don't have a lot of tasks and less number of GPUs.
• 12:02
  SPEAKER_02: So that's uh where we need to configure it properly.
• 12:06
  SPEAKER_02: Uh stage three can be validation.
• 12:08
  SPEAKER_02: So uh let's say we have the data, our inference has uh already ran properly.
• 12:12
  SPEAKER_02: Now we need to make sure the output that is being generated goes through a parentic schema, then cross-checks against the structure record, and we can make sure the output is correct.
• 12:23
  SPEAKER_02: And the final one we can talk about that's load.
• 12:27
  SPEAKER_02: So uh we load those all the output into a feature store.
• 12:32
  SPEAKER_02: So a final model that's serving a readmission or fraud detection, whatever, that can read from the data feature store and give a good, well-structured output or a scoring.
• 12:45
  SPEAKER_02: So these those are the four main stages where we are using Airflow and how we are using it.
• 12:53
  SPEAKER_00: Okay, interesting.
• 12:55
  SPEAKER_00: And then I'd also love to dig a little bit deeper into how you're handling failures.
• 13:01
  SPEAKER_00: So you touched on this a little bit earlier uh with using Airflow to manage retries and things like that.
• 13:08
  SPEAKER_00: But tell me in a little bit more detail if something goes wrong, or you mentioned you have a lot of data quality checks built in.
• 13:16
  SPEAKER_00: Uh, when there's also this LLM in the loop that you're managing, how does all of that work?
• 13:22
  SPEAKER_02: Yes.
• 13:22
  SPEAKER_02: So before directly uh diving into the failures, we can divide it into two types of failure.
• 13:29
  SPEAKER_02: One can be the easier one, that's the loud failure.
• 13:32
  SPEAKER_02: The pod gets OOM killed, the GPU is busy, the model timeout.
• 13:36
  SPEAKER_02: That is an ordinary airflow failure.
• 13:39
  SPEAKER_02: We can retry with exponential fallbacks, we can do task isolation as I mentioned.
• 13:44
  SPEAKER_02: Task isolation is very important.
• 13:46
  SPEAKER_02: We can rerun the task and make sure that the things are not overwritten and we are replacing it with the correct output.
• 13:53
  SPEAKER_02: So those are loud failures.
• 13:54
  SPEAKER_02: We can go and check the system and see everything is working fine or if there is a failure.
• 14:00
  SPEAKER_02: The category two, that's the scarier one.
• 14:04
  SPEAKER_02: Because here, let's assume model runs, it returns a clean output, and nothing is wrong.
• 14:10
  SPEAKER_02: For a normal person, everything is green, the graph is green, everything is looking good.
• 14:15
  SPEAKER_02: That's a good picture.
• 14:18
  SPEAKER_02: But here, uh there are a lot of scenarios that can happen.
• 14:22
  SPEAKER_02: Uh, let's say uh the output is not correctly in not in a correct process or not in a correct format.
• 14:31
  SPEAKER_02: So here we have constraint output.
• 14:33
  SPEAKER_02: Uh the model fills in the uh piedenic schema.
• 14:36
  SPEAKER_02: We cannot have a free text, so it does not pass.
• 14:39
  SPEAKER_02: The record the record is injected, but if it's not in a piedentic schema, we do not let it uh go to a next stage.
• 14:47
  SPEAKER_02: So that is how we check this silent failure.
• 14:50
  SPEAKER_02: We have piedantic for this, then we can check against the structure record.
• 14:55
  SPEAKER_02: So let's say if Note is extracting something about a heart failure and the core diagnosis doesn't have anything about it, so we can write some logic to check this kind of issues.
• 15:08
  SPEAKER_02: Then we can do a distribution check.
• 15:11
  SPEAKER_02: Let's say uh for any fraud detection or for any kind of let's say for fraud detection or any kind of anomaly, uh every day it's just 2%, but suddenly we saw it shot up to 8%.
• 15:25
  SPEAKER_02: Everything is on a rosy picture, everything is green, but here you see there is some anomaly.
• 15:30
  SPEAKER_02: So we need to go in manually and check what's going wrong.
• 15:34
  SPEAKER_02: So that is something we need to debug.
• 15:36
  SPEAKER_02: And initially, when you are setting a system, it's in a dev stage and not prod.
• 15:41
  SPEAKER_02: We can have a human in loop who uh checks every X sample and makes sure what we are outputting is correct.
• 15:48
  SPEAKER_02: So the silent failures are more challenging, and these are some of the ways we can handle them, but still we might miss at some stage.
• 15:58
  SPEAKER_02: But currently, this is the most challenging part in the industry.
• 16:02
  SPEAKER_00: Yeah, that makes sense.
• 16:03
  SPEAKER_00: And how do you go about trying to solve those the silent failures, as you've said, with Airflow?
• 16:10
  SPEAKER_00: Like where do you think Airflow fits in there?
• 16:13
  SPEAKER_00: Because I think everybody probably knows how you can use Airflow to manage the loud ones.
• 16:17
  SPEAKER_00: As you say, that's really easy.
• 16:19
  SPEAKER_00: Obviously, harder to build in some logic to manage the softer failures.
• 16:25
  SPEAKER_02: So we we can at each step we know what are the outputs, what are the inputs.
• 16:31
  SPEAKER_02: So we can have some Chrome jobs or Python script as one of the nodes, which uh which after a specific uh uh output, it checks uh after a specific step, it checks if uh the output is correct and it will go through the as I say, it will go through the pyentic validation.
• 16:50
  SPEAKER_02: So that can be one of the nodes in the airflow.
• 16:52
  SPEAKER_02: So putting those check notes in the whole orchestration, that can be how that that's one of the ways to check these silent flows.
• 17:01
  SPEAKER_00: Got it.
• 17:01
  SPEAKER_00: So you're building in data quality checks of a sort into the pipelines.
• 17:06
  SPEAKER_00: Yeah, that makes sense.
• 17:09
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 17:17
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 17:21
  SPEAKER_01: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 17:28
  SPEAKER_01: Check it out at astronomer.io.
• 17:32
  SPEAKER_00: Okay, great.
• 17:34
  SPEAKER_00: And then I guess on a slightly different but related topic, I understand that part of your approach uh to managing these pipelines includes fine-tuning.
• 17:45
  SPEAKER_00: And so I'm curious if you could tell me more about what that looks like and where that fits into your Airflow pipelines.
• 17:54
  SPEAKER_02: Sure.
• 17:55
  SPEAKER_02: So uh as you mentioned, it's a little bit uh away from the core DAC topics, but uh uh let's think it's not something that we need to run every night.
• 18:06
  SPEAKER_02: Fine-tuning is not something we we need every day or every night or every week.
• 18:11
  SPEAKER_02: It is it has his own Airflow DAC.
• 18:14
  SPEAKER_02: It doesn't sit in the core flow and it fires maybe once a month or maybe once in a quarter when we have enough new data piled up so we can fine-tune the already fine-tuned model we have.
• 18:29
  SPEAKER_02: So it pulls the label set, runs a lot of fine-tuning with Hugging Face, PEFT on top of Lama 3, then scores it against the frozen holdout.
• 18:39
  SPEAKER_02: And if it only if it beats what we have, it writes a new adapter version to the model registry.
• 18:46
  SPEAKER_02: So that is something, that is something has to be a totally different Airflow DAC apart from the gore one which we are working.
• 18:55
  SPEAKER_02: And so Nightly Drag uh, and one more thing is the Nightly DAC trains nothing, it just loads the base Lama 3 weight, loads the current adapter, file uh loads the current adapter file next to them and runs runs it basically uh to the old files.
• 19:10
  SPEAKER_02: And the adapter is maybe let's say a few hundred megabytes on the disk.
• 19:15
  SPEAKER_02: So all of the fine tuning cost you uh you run is just about reading one extra file.
• 19:22
  SPEAKER_02: That's it.
• 19:23
  SPEAKER_02: And one more thing where Airflow can fit perfectly here is when we write to these adapters and we keep updating the record, Airflow helps us to maintain different versions with it.
• 19:34
  SPEAKER_02: So six months later down the line, you can look back, row, uh look back at a row in a feature, and we can know exactly which adapter produced it.
• 19:43
  SPEAKER_02: And if we want to go to the specific version, we can use Airflow, run it across along with that adapter and get the same result.
• 19:51
  SPEAKER_02: So, in this sensitive type of field, if there is an audit or something, we'll have a very good trace, and that is because of Airflow, how we are managing it.
• 20:00
  SPEAKER_00: Okay, that's really cool.
• 20:02
  SPEAKER_00: Uh, I have heard of a few other use cases of people using Airflow to manage fine-tuning like this, but not a ton.
• 20:09
  SPEAKER_00: So this is great to hear your perspective on.
• 20:12
  SPEAKER_00: And then tell me more about so when you're fine-tuning the models, are you looking for better performance?
• 20:20
  SPEAKER_00: Are you looking for better, or I should say, performance in terms of accuracy or performance in terms of speed or efficiency, or is it all of the above?
• 20:30
  SPEAKER_02: It's mostly on term uh terms of accuracy.
• 20:34
  SPEAKER_02: It's uh because we have Lama 3, it's a very good model on uh normal English, but when it comes to a specific domain, let's say financial domain or a medical domain, it lacks a little bit because there is a context to because it's a context uh that it's lacking.
• 20:54
  SPEAKER_02: So when we talk about fine-tuning, we take we freeze the core weights and just train on the last layer.
• 21:00
  SPEAKER_02: So, let me give you an example.
• 21:02
  SPEAKER_02: Let's say uh doctors talk in a language, which say PT denies CP, and that means patients say they have no chest pain.
• 21:09
  SPEAKER_02: So there is a very small word that says does not have chest pain.
• 21:14
  SPEAKER_02: And there is some acronyms, PT, CP, all those stuff.
• 21:17
  SPEAKER_02: So those get lost when it's being trained on normal English.
• 21:22
  SPEAKER_02: And when we fine-tune on uh medical data, we can pick all of those and we can have a much more accurate output.
• 21:31
  SPEAKER_00: Got it.
• 21:32
  SPEAKER_00: That makes sense.
• 21:33
  SPEAKER_00: And then I guess zooming back a little bit and looking at this high level, what were the hardest lessons you learned when trying to productionize this and getting it to run at scale?
• 21:46
  SPEAKER_02: So, first is reproducibility.
• 21:49
  SPEAKER_02: So, what happens when we are uh when we are working at a at a lightning speed with all the AI assisted coding and everything?
• 21:58
  SPEAKER_02: We forget how to train.
• 22:00
  SPEAKER_02: Back and how to uh if something fails, or uh how to trace back how we have iterated and how did we come in so far.
• 22:08
  SPEAKER_02: So let's say uh we uh we need to pin those stuff.
• 22:13
  SPEAKER_02: We need to pin, uh, we need to pin the temperature because there can be a lot of variance.
• 22:19
  SPEAKER_02: The feature starts to drift if we don't pin the model version, we don't pin the features, we don't pin the temperature.
• 22:26
  SPEAKER_02: So that is one important thing.
• 22:28
  SPEAKER_02: Make sure every time you are running a model or any orchestration, pin everything versions, uh versions, seed, features, everything.
• 22:38
  SPEAKER_02: Next is about the GPUs.
• 22:41
  SPEAKER_02: So uh a lot of time uh what we did, we scheduled a lot more tasks in parallel than we had GPUs.
• 22:49
  SPEAKER_02: So even if the tasks are in parallel, we don't have enough resources for them to run in parallel.
• 22:56
  SPEAKER_02: So make sure that the amount of infra you have, you parallelize your orchestration based on that.
• 23:03
  SPEAKER_02: You don't over-parallelize or over-complicate it.
• 23:05
  SPEAKER_02: That's the second thing.
• 23:07
  SPEAKER_02: And the third one that matters the most is the traceability.
• 23:12
  SPEAKER_02: How we can look six months back and say how we reach to that particular output.
• 23:18
  SPEAKER_02: So uh here we need to make sure uh that uh we have a proper feature storage, we have a proper isolation, and every service has a very good trace logs.
• 23:29
  SPEAKER_02: And if anything fails, we can isolate it and we can we can isolate it and make sure that uh uh we can isolate it and make sure that uh some specific thing failed here, and that's the reason these results are broken.
• 23:45
  SPEAKER_02: So these are the three things that I think is most important.
• 23:48
  SPEAKER_00: Yeah, I think those are great things for everybody, maybe even using LLMs or not to think about when trying to productionize stuff with Airflow.
• 23:57
  SPEAKER_00: So, yeah, that's great insight there.
• 24:00
  SPEAKER_00: Okay, well, Chang, thank you for going through all of that.
• 24:04
  SPEAKER_00: I again think this is a super interesting use case that a lot of our listeners will benefit from hearing about.
• 24:10
  SPEAKER_00: Uh, in the interest of time, I will wrap us up and I always like to end with a standard final question, and that is what would you most like to see from development of the Airflow project in the near future?
• 24:22
  SPEAKER_02: Yes, so if I have to look into it right now, uh Airflow has been really useful.
• 24:28
  SPEAKER_02: And with Airflow 3 coming and it's already out, it has been really helpful for developer, but few things if I had to uh pinpoint, then we can have the GPU awareness uh in Airflow itself.
• 24:41
  SPEAKER_02: So right now, if our task needs a GPU, you can express it sideways through pools, cues, and naming convention, and every team reinvents it slightly differently.
• 24:51
  SPEAKER_02: So I would like a task to declare that if it needs a GPU for 20 minutes and have the shed and let the scheduler honor it.
• 24:59
  SPEAKER_02: It is the piece that I have hand-rolled on every project I have done.
• 25:03
  SPEAKER_02: So instead of each developer or each team hand holding uh hand-rolling it, we can have a first-class awareness GPU inbuilt in Airflow system.
• 25:13
  SPEAKER_02: And second is treat model as assets basically, because a lot of time, a lot of teams now are modeling and they are using Airflow for that.
• 25:21
  SPEAKER_02: So I know Airflow already understands that a data set is produced by one task and consumed by another.
• 25:28
  SPEAKER_02: Uh and a base model and a LoRa adapter are exactly that shape.
• 25:33
  SPEAKER_02: So if assets covered a model version, uh, then which model version produced this table becomes a question Airflow can answer.
• 25:41
  SPEAKER_02: And a new developer doesn't have to come with their own versioning.
• 25:44
  SPEAKER_02: So I think these two are the main thing treating model as assets and having more first-class GPU awareness.
• 25:50
  SPEAKER_00: Yeah, got it.
• 25:52
  SPEAKER_00: I think those are both great requests.
• 25:54
  SPEAKER_00: Okay, well, again, thank you so much for going through all this.
• 25:57
  SPEAKER_00: I think this has been great.
• 25:59
  SPEAKER_00: To close us out, what is the best way for folks to get in touch with you?
• 26:02
  SPEAKER_02: Uh I think LinkedIn is the best way that they can get in touch with me.
• 26:06
  SPEAKER_02: And uh I can also have my email ID in it, so LinkedIn and email ID.
• 26:10
  SPEAKER_02: Okay.
• 26:11
  SPEAKER_02: And later I'll be I'll be having a talk at Airflow Summit.
• 26:15
  SPEAKER_02: So more than open to seeing people there.
• 26:18
  SPEAKER_00: Yeah, great.
• 26:18
  SPEAKER_00: Well, I think this episode will probably be released after Airflow Summit since we're in very short time for Airflow Summit this year.
• 26:26
  SPEAKER_00: Uh, but hopefully some folks will have seen you there.
• 26:29
  SPEAKER_00: And yeah, we'll throw links to those other things in the show notes.
• 26:32
  SPEAKER_00: So uh Chang, thank you again so much.
• 26:34
  SPEAKER_00: This has been great.
• 26:36
  SPEAKER_00: Yeah, nice, nice meeting you.
• 26:38
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 26:41
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 26:44
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 26:49
  SPEAKER_01: And leave us a five-star review.