Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue

Transcript

The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: There's the first part about fine-tuning just that component on its own.
• 0:04
  SPEAKER_01: Technically, LLMs in general are fairly decent at summarization just for a general passage.
• 0:10
  SPEAKER_01: Our goal is to teach it to specifically learn how to look at airflow airlocks, not just general passage.
• 0:17
  SPEAKER_03: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:27
  SPEAKER_00: Hello, everyone.
• 0:28
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:30
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Julian Leneve, who's the CTO at Astronomer, and David Ju, who's a machine learning engineer at Astronomer.
• 0:39
  SPEAKER_00: How are you both doing?
• 0:40
  SPEAKER_02: Good.
• 0:40
  SPEAKER_02: It's great to be here.
• 0:46
  SPEAKER_02: Yeah, our new headquarters.
• 0:47
  SPEAKER_02: We just opened this office a couple months ago.
• 0:49
  SPEAKER_00: Yeah, that's awesome.
• 0:50
  SPEAKER_00: I have not been out there yet, but I think I just saw I might be going out in June.
• 0:54
  SPEAKER_00: So I'm excited to see it.
• 0:55
  SPEAKER_00: Yeah, everybody seems pretty excited about it.
• 0:58
  SPEAKER_02: Yeah.
• 0:58
  SPEAKER_02: Yeah.
• 0:59
  SPEAKER_00: All right.
• 1:00
  SPEAKER_00: Well, let's go ahead and dive in.
• 1:01
  SPEAKER_00: So for today's podcast, those listening, we're going to be discussing an extremely hot topic right now that we're very excited about.
• 1:08
  SPEAKER_00: That's generative AI.
• 1:09
  SPEAKER_00: And we want to dive into the practical, realistic implications of it as it relates to airflow.
• 1:16
  SPEAKER_00: So this is a topic that you both have a lot of experience with.
• 1:19
  SPEAKER_00: So to start at a high level, maybe talk about what market trends you're seeing with Gen AI right now.
• 1:25
  SPEAKER_00: I think this is something that our listeners will be very interested in at a high level.
• 1:29
  SPEAKER_02: I'll start here.
• 1:30
  SPEAKER_02: First and foremost, it's on everyone's mind, right?
• 1:32
  SPEAKER_02: It's a very hot topic.
• 1:34
  SPEAKER_02: I think everyone has either intrinsic or extrinsic pressure to go do something in the generative AI space.
• 1:40
  SPEAKER_02: I think a lot of folks really view it as a way to go activate their data.
• 1:44
  SPEAKER_02: But if you look at a lot of the customers that we work with, they've done a great job of collecting data, centralizing it, and really starting to use it in a couple meaningful ways.
• 1:53
  SPEAKER_02: And generative AI can really take that and move it to the next levels.
• 1:57
  SPEAKER_02: We see a ton of investment, actually both from academia and industry, which is pretty interesting.
• 2:02
  SPEAKER_02: You have big companies like Meta that are releasing open source models, but then you also have different research institutions spending a ton of time, energy, and money building best in class models.
• 2:13
  SPEAKER_02: There's also been a pretty big shift from commercial models like GPT-4 now to open source models, that those are catching up.
• 2:21
  SPEAKER_02: And that's for a couple of different reasons.
• 2:23
  SPEAKER_02: I think first and foremost is these commercial models are really a black box.
• 2:27
  SPEAKER_02: You're making an API call and you don't really know what's going on under the hood.
• 2:31
  SPEAKER_02: It means that your LLM provider can go change the model without your knowledge.
• 2:36
  SPEAKER_02: Your code won't change, you'll have no visibility into that.
• 2:39
  SPEAKER_02: And if you're going to go build these things in a production-ready setting, that's very important, right?
• 2:46
  SPEAKER_02: You're relying on these things, you want observability into what's going on under the hood.
• 2:50
  SPEAKER_02: But there's also arguments to be made for customizability and cost.
• 2:54
  SPEAKER_02: It's a lot easier to take an open source model and go customize it and choose a deployment method that's right for your budgets than it is to do that same type of thing with a commercial model.
• 3:06
  SPEAKER_00: Yeah, that makes sense.
• 3:07
  SPEAKER_00: I think that trade-off between customizability and something that's ready to go out of the box, we see that all the time, including with Airflow, right?
• 3:15
  SPEAKER_00: Like that's just hard to get the best of both worlds with that.
• 3:20
  SPEAKER_02: Yeah, exactly.
• 3:21
  SPEAKER_02: Exactly.
• 3:22
  SPEAKER_00: Well, with so much going on kind of in the market, as you just described, this is moving in a really fast-paced way.
• 3:27
  SPEAKER_00: And I think it can seem for a lot of people who are trying to implement these technologies is take advantage of the trends you just talked about, that there's a new model kind of coming out every single week.
• 3:36
  SPEAKER_00: You both, you know, are tasked in your jobs with kind of keeping up with these fast-paced advancements.
• 3:42
  SPEAKER_00: What have you been noticing there?
• 3:44
  SPEAKER_01: I guess I can take on this one.
• 3:46
  SPEAKER_01: Yeah.
• 3:46
  SPEAKER_01: So I think the most notably recently, you have the Llama 3 being released by Facebook or Meta, and also within a week or so, Microsoft released their PIS3 model.
• 3:57
  SPEAKER_01: But I think very interestingly is that the model architectures hasn't been really changing much in the past few months, even though the models are getting better.
• 4:05
  SPEAKER_01: One very interesting point that Microsoft raised in their technical report for PIS3 was that they actually used almost the exact same architecture as the Meta's Lama 2 model, the last generation's model architecture, with some small tweaks, such as making it accepting longer context windows and that kind of things.
• 4:22
  SPEAKER_01: But the architecture is the same.
• 4:23
  SPEAKER_01: What they highlighted is that they actually trained on a smaller amount of data as well, but they used higher quality data.
• 4:30
  SPEAKER_01: And they highlighted the fact that higher quality data actually leads to better performance relative to just stuffing everything into the model.
• 4:38
  SPEAKER_01: And very notable result is that Pfizer is actually a much smaller model compared to the smallest model of Lamas 3.
• 4:45
  SPEAKER_01: And they were achieving similar levels of performance, even though technically Lamas 3 had five types amount of data being ingested compared to Microsoft's model.
• 4:55
  SPEAKER_01: So this could be a growing trend where data processing and data quality is becoming more meaningful to machine learning and LMs in general.
• 5:04
  SPEAKER_01: You know, with training models, you basically have a three category of you need to use a lot of heavy compute, that's the compute constraint, you need to have a lot of good data, that's the data constraint.
• 5:14
  SPEAKER_01: And then there's a model architecture.
• 5:16
  SPEAKER_01: But I think we start to see the trend where the model architecture is uh converging to a very similar standard.
• 5:22
  SPEAKER_01: Now you have the other two remaining parts left.
• 5:26
  SPEAKER_00: That's interesting.
• 5:26
  SPEAKER_00: So it's less the models themselves that are changing and more kind of what we often refer to as like the day two operations around them.
• 5:34
  SPEAKER_00: So everything that you need to actually take that model and make use out of it in production.
• 5:39
  SPEAKER_02: There's two ways that you train the model, right?
• 5:41
  SPEAKER_02: There's these big companies that train very big foundational models.
• 5:45
  SPEAKER_02: And then there's other companies, the general industry that takes those foundational models and fine-tunes them with their own data.
• 5:52
  SPEAKER_02: I think we've known for a long time that in the fine-tuning process, you need to give it high quality data to get it very good at your specific very domain use case.
• 6:02
  SPEAKER_02: Um, but what we're starting to see now, like David mentioned in some of these more recent models, is that the data quality is just as important for those foundational models.
• 6:11
  SPEAKER_02: You get equivalent performance with a much smaller model, which means you can run it for cheaper, you can run it on smaller devices.
• 6:18
  SPEAKER_02: I think Microsoft even said that you can go run their new models on a phone, which is really cool.
• 6:24
  SPEAKER_00: Yeah, wow, that's crazy.
• 6:26
  SPEAKER_00: Well, and so how with the focus on kind of data quality and data processing and management, where do you see Airflow fitting in there?
• 6:35
  SPEAKER_02: Yeah, it's a good question.
• 6:36
  SPEAKER_02: I think if you look at it at its core, Airflow is a data pipeline, a data processing framework.
• 6:41
  SPEAKER_02: We see it everywhere.
• 6:42
  SPEAKER_02: I think for someone to go produce a very high-quality data set reliably and consistently, you need very good day two operations.
• 6:51
  SPEAKER_02: You need very good pipeline reliability.
• 6:54
  SPEAKER_02: And you really need to develop things with an orchestration first mindset.
• 6:58
  SPEAKER_02: Otherwise, what happens is you end up going through this big exercise to go collect a ton of data, do it in a very ad hoc manner, maybe go through, you know, very large fine-tuning exercise.
• 7:11
  SPEAKER_02: Then you leave the model for several months until you decide it's time to go through that big exercise again.
• 7:17
  SPEAKER_02: I think what we're starting to learn and realize, both internally and with a lot of our customers, is a lot of the times this is a repeatable process, right?
• 7:26
  SPEAKER_02: You see new data all the time, whether that's from external sources or the output of the models themselves.
• 7:33
  SPEAKER_02: And so you can use this orchestration-first mindset to go build data collection, data quality monitoring, and actually orchestrate the fine-tuning of your models to happen weekly, daily instead of every six months.
• 7:48
  SPEAKER_00: Yeah, interesting.
• 7:50
  SPEAKER_00: I think we talked a little bit earlier about kind of the trade-offs between like a really flexible tool that you can do anything with versus something that's more prescriptive, which are often more black box.
• 8:02
  SPEAKER_00: I talk a lot with the community about how I think Airflow falls way more to that very flexible side, which is great in a lot of cases, but also means that you can do anything with it.
• 8:13
  SPEAKER_00: I think people get a little bit lost.
• 8:15
  SPEAKER_00: How do you see that kind of playing in?
• 8:17
  SPEAKER_00: Like, where does Airflow's flexibility help it be a good tool in this Gen AI space?
• 8:23
  SPEAKER_02: Yeah, I think that's where it's now super adaptable.
• 8:25
  SPEAKER_02: I mean, Airflow was built almost 10 years ago.
• 8:27
  SPEAKER_02: And the fact that today it's one of the most popular data engineering tools there is.
• 8:32
  SPEAKER_02: I think we can attribute a lot of that to how general purpose it is.
• 8:36
  SPEAKER_02: I think what companies tend to do, especially at a certain scale, is they use Airflow as a platform.
• 8:41
  SPEAKER_02: It's not like they take Airflow off the shelf and go use it as is.
• 8:45
  SPEAKER_02: They build custom providers, custom operators, they go tune it to their needs, they build abstractions on top of it.
• 8:52
  SPEAKER_02: Airbnb actually just released their ML platform called Chrono.
• 8:57
  SPEAKER_02: And they use Airflow as the orchestration engine for that.
• 9:01
  SPEAKER_02: That's just one of many examples where you have some idea, some need, and there's an orchestration component to it.
• 9:09
  SPEAKER_02: Ultimately, like a general purpose tool like Airflow can be the foundation for that.
• 9:13
  SPEAKER_02: But what you end up doing in practice is you take it and you go adapt it for your needs.
• 9:18
  SPEAKER_02: If you go view that in the context of generative AI too, I think that's why we see Airflow come up so much in that context, because it is this very general purpose orchestration tool, you can easily go adapt it to the new needs that have come out of some of these generative AI workloads.
• 9:36
  SPEAKER_00: Yeah, that's really interesting.
• 9:40
  SPEAKER_03: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 9:49
  SPEAKER_03: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:01
  SPEAKER_03: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 10:10
  SPEAKER_00: And thinking about kind of a practical example of this.
• 10:14
  SPEAKER_00: So Astronomer obviously does a lot of work on Airflow itself and are heavily invested in, you know, driving the Airflow project forward and making sure that it's a good tool to be used for these use cases like Gen AI.
• 10:27
  SPEAKER_00: At the same time, we also use Airflow a lot internally ourselves.
• 10:30
  SPEAKER_00: When you two are currently working on an awesome project where you're using an LLM to generate summaries of Airflow task failures, maybe tell me a little bit about that project and what you're hoping to accomplish with it.
• 10:42
  SPEAKER_01: Oh, yeah, I can get started on that.
• 10:44
  SPEAKER_01: We basically see that airlogs can be very long and confusing to read.
• 10:49
  SPEAKER_01: You sometimes have air logs that are blobs that are like maybe thousands of lines long, but the key information are scattered here and there or maybe more towards the end.
• 10:58
  SPEAKER_01: So for the users to immediately know what went wrong with their airflow tasks, it's not very simple.
• 11:04
  SPEAKER_01: And it would cost them a fair amount of time to figure out what's going on.
• 11:07
  SPEAKER_01: So we decided to build this using LLM to build a tool to basically summarize and extract these key information from the air logs and tell the user immediately, hey, here's what's wrong and here's probably how you can fix it.
• 11:21
  SPEAKER_01: And this should save airful users a lot of time and also just mentally more don't have to feel like they have to look up things every time.
• 11:30
  SPEAKER_01: Yeah, I think we also have some additional next steps that Julia may be able to mention as well.
• 11:36
  SPEAKER_02: Yeah, if you also look at the scale that we work at, we run tons and tons of pipelines on behalf of our customers.
• 11:43
  SPEAKER_02: And naturally, as a side effect, these pipelines fail every once in a while.
• 11:46
  SPEAKER_02: So we have millions of failures a month on our platform.
• 11:49
  SPEAKER_02: Anything we can do to make it easier for our customers to resolve those issues quickly is something that we'll invest a lot of time and money in.
• 11:57
  SPEAKER_02: And when we saw you know a lot of these advancements in LLMs and the type of unique data that we have access to around Airflow, we thought it was a perfect fit to go say, we have access to probably more Airflow logs than almost anyone in the world.
• 12:12
  SPEAKER_02: We can go fine-tune an LLM that gets very good and very, very cheap to run at scale, specifically to extract error summaries out of these task failure logs.
• 12:24
  SPEAKER_02: We're looking to bake it into our alerting framework too, so that when we tell a customer their pipeline failed, it's not some obscure it failed, you have to go dig into why.
• 12:35
  SPEAKER_02: It's you failed, here's exactly what happened, and then you can go start to resolve it quicker and really reduce that time to resolution metric that we care a lot about.
• 12:46
  SPEAKER_02: And of course, we're using Airflow to do it, right?
• 12:49
  SPEAKER_02: As this LLM runs over time, we get data out of it.
• 12:53
  SPEAKER_02: So there's a very natural fit for Airflow.
• 12:56
  SPEAKER_02: First, in terms of you know how you traditionally use Airflow.
• 13:00
  SPEAKER_02: As we get data out of the system, we want to be able to track and analyze that data and view it over time.
• 13:06
  SPEAKER_02: Performance metrics, cost metrics, the types of things that you'd want to monitor out of any application.
• 13:12
  SPEAKER_02: But we also get user feedback, right?
• 13:14
  SPEAKER_02: Every time we present a user with a log summary, they get to tell us, yes, this is a good summary, great job.
• 13:21
  SPEAKER_02: Or no, this is actually not what I expected.
• 13:24
  SPEAKER_02: Here instead is what I would have expected.
• 13:27
  SPEAKER_02: And we'll collect that data over time.
• 13:29
  SPEAKER_02: And instead of waiting six months to do something with that data and go through another big fine-tuning exercise, we'll just fine-tune the model every week as we understand what it's good and more importantly, not good at so that it gets better and better over time.
• 13:43
  SPEAKER_02: And we expect to see that accuracy metric go up over time, the user feedback metrics go up over time, so that we can continue teaching the model and letting it adapt to new patterns that we see in Airflow.
• 13:57
  SPEAKER_00: Yeah, that's really interesting.
• 13:58
  SPEAKER_00: I think the idea of just making it easier to figure out what's going on, especially in the case of a filler with logs, resonates a lot as somebody who has had to spend a lot of time in their past lives looking at Airflow logs.
• 14:10
  SPEAKER_00: There's a lot of information in there, most of which I think is irrelevant to a lot of users.
• 14:15
  SPEAKER_00: Even just within the open source project, there was a new feature that just came out in Airflow 2.9 a couple of weeks ago that collapses some of the contextual information about what's going on in the scheduler and stuff, even so that as you're viewing them, you can still expand it.
• 14:29
  SPEAKER_00: It's all still there.
• 14:30
  SPEAKER_00: But even I think the need for something like that points to just the complexity of the information that's being presented there.
• 14:36
  SPEAKER_00: So what you're talking about is obviously a much more sophisticated kind of next step of the solution where you're only going to give me as the end user the part that actually matters to me.
• 14:46
  SPEAKER_02: Yeah.
• 14:47
  SPEAKER_02: And I think this is what a lot of businesses and service providers are going through right now.
• 14:51
  SPEAKER_02: A year ago, before LLMs were as ubiquitous, if we sat down and said, hey, we have all this log information, what can we go do with it?
• 15:00
  SPEAKER_02: The answer is not super clear, right?
• 15:02
  SPEAKER_02: Maybe we can go set up some clever systems.
• 15:04
  SPEAKER_02: But now with LLMs, it gets a lot easier to do what I call activating that data, right?
• 15:10
  SPEAKER_02: We have this data, we want to go use it to provide value back to our customers.
• 15:15
  SPEAKER_02: Let's go fine-tune an LLM based off of it.
• 15:17
  SPEAKER_02: I think this is a pattern that we have seen and will continue to see with a lot of our customers and community members.
• 15:24
  SPEAKER_02: Again, they're very good at collecting information, collecting data and centralizing it.
• 15:28
  SPEAKER_02: Today, a lot of those customers and users do reporting on it.
• 15:32
  SPEAKER_02: A lot of them are starting to move into the ML world and data science world where you try to get more insight out of the information, but generative AI and LLM specifically have have really started to accelerate how quickly you can get value out of your data.
• 15:48
  SPEAKER_00: Yeah, that makes a lot of sense.
• 15:50
  SPEAKER_00: Tell me more about how you're reinforcing the performance of the LM model with kind of this feedback from users.
• 15:58
  SPEAKER_00: Like what is the mechanism there for?
• 16:00
  SPEAKER_00: So you're using the model to pull out the relevant information, but it needs to know the relevant information is the error in my Python code that's one line out of 2000 that also has all of the scheduler information and worker information.
• 16:14
  SPEAKER_00: How does that work?
• 16:15
  SPEAKER_01: I guess to answer your question, there's the first part about fine-tuning just that component on its own.
• 16:21
  SPEAKER_01: Technically, LLMs in general are fairly decent at summarization just for a general passage.
• 16:27
  SPEAKER_01: Imagine you put in like a news article and what is the most important information in here, right?
• 16:32
  SPEAKER_01: It's not gonna do a terrible job at it.
• 16:34
  SPEAKER_01: Our goal is to teach it to specifically learn how to look at airflow airlocks, not just general passage, which we found out that actually doesn't do that well in, and also tends to either hallucinate or extract irrelevant information here and there.
• 16:48
  SPEAKER_01: So that's our goal with fine tuning, which is uh make it so that it's concise, is extracting the right information and the key information that's most useful to the user.
• 16:57
  SPEAKER_01: So that goes to the base fine tuning, but also the iterative fine tuning to continually improve that.
• 17:03
  SPEAKER_01: We actually want to do two ways of iterative fine tuning.
• 17:07
  SPEAKER_01: One, we collect information based on user feedback when the user says this is a good answer.
• 17:12
  SPEAKER_01: This is being added later on for the training to do additional training on top.
• 17:17
  SPEAKER_01: We also do prompting with top best answers.
• 17:21
  SPEAKER_01: So sometimes if you look at ChatGPT, if you're lucky enough, sometimes they would actually give you two answers.
• 17:26
  SPEAKER_01: And OpenAI will say, oh, hey, here's two answers.
• 17:30
  SPEAKER_01: Which one do you prefer?
• 17:31
  SPEAKER_01: You can actually ask the model to generate like top-end answers, and it would have a preference learned so that it knows that I think both of these answers are worded very differently, but they're, in my opinion, like from the model perspective, they're also good answers.
• 17:44
  SPEAKER_01: So which one does the human prefer?
• 17:46
  SPEAKER_01: And when the user selects one, we basically now have a paired answer, whereas this answer is an accepted answer and the other answer is a rejected answer.
• 17:55
  SPEAKER_01: This would teach the model to basically get rid of the noise or the parts that humans don't want and what the what humans actually want out of this model.
• 18:04
  SPEAKER_01: Yeah, I think that's the gist of it.
• 18:06
  SPEAKER_01: Yeah.
• 18:07
  SPEAKER_00: When you talk about one of the issues initially, was the model hallucinating answers that are not relevant.
• 18:13
  SPEAKER_00: In this particular case, is that pulling information that simply wasn't in the logs or is incorrect based on what the logs are saying versus pulling out the relevant information in the logs?
• 18:25
  SPEAKER_01: I think it could be both, but one very funny thing we saw was like from the original models is that sometimes the air logs emits when you have a long string in a line, it's sometimes in the air logs it would do dot and it would skip parts of it.
• 18:39
  SPEAKER_01: And somehow the model predicted what's being omitted and it somehow made up words inside and says, oh, this is what went wrong because it's this.
• 18:47
  SPEAKER_01: Even though that could have hallucinated the correct word, we don't want that because the chances of you guessing that correctly is not great.
• 18:54
  SPEAKER_01: So that happens actually fairly common.
• 18:57
  SPEAKER_01: And as you mentioned, there's also parts where the model basically is focusing too much on information that's not relevant.
• 19:03
  SPEAKER_01: For example, error logs sometimes have a lot of warnings.
• 19:06
  SPEAKER_01: And the model would talk about the warnings all the time and not really about the true error that has occurred.
• 19:12
  SPEAKER_00: Interesting.
• 19:13
  SPEAKER_00: That makes sense.
• 19:13
  SPEAKER_00: So definitely a lot of fine-tuning needed for this case.
• 19:17
  SPEAKER_00: I think Julian also mentioned uh a few minutes ago that we're dealing with millions of these task failures on our platforms every month.
• 19:25
  SPEAKER_00: Imagine we have massive amounts of data with which to train this, which is great for like improving the accuracy.
• 19:31
  SPEAKER_00: But imagine it's also a lot to manage from the pipeline perspective and actually productionizing this in a way that we could eventually provide it as part of the product.
• 19:41
  SPEAKER_00: Anything you want to say there about the scale of this project and how Airflow is able to handle that?
• 19:47
  SPEAKER_02: Yeah, that's where fine-tuning is actually really interesting because not only can you get it good at very specific use cases, but you can also take a larger model and kind of distill the knowledge into a smaller model.
• 20:01
  SPEAKER_02: So if we were using GPT-4 or some very expensive model to run, scale would absolutely be a concern here.
• 20:07
  SPEAKER_02: I think we'd have to go raise a new venture round pretty quickly.
• 20:11
  SPEAKER_02: Instead, though, what we've been able to do is say, well, we don't care that this model is bad at generating answers to a user's random question or doing math or all of the things that LLMs can be good at.
• 20:25
  SPEAKER_02: We just care that it's very good at this one specific use case.
• 20:29
  SPEAKER_02: So we were able to take a smaller model and teach it to be very good at summarizing our flow logs.
• 20:35
  SPEAKER_02: Again, as a consequence, it's not great at other things, but it does help a lot with the scalability problem of we don't need the most expensive GPU.
• 20:44
  SPEAKER_02: We can actually fit the model into a smaller GPU.
• 20:46
  SPEAKER_02: Those GPUs are also more generally available, and so we can go scale that up and down as needed.
• 20:52
  SPEAKER_00: That makes sense.
• 20:53
  SPEAKER_00: Anything interesting you've learned about implementing this from the Airflow side?
• 20:57
  SPEAKER_00: I know this is not the only use case with which we use Airflow internally for the applications, but with this one specifically, anything, any fun takeaways?
• 21:08
  SPEAKER_02: Yeah, I think it's one of those things where Airflow might not come to mind at first for a lot of these use cases.
• 21:14
  SPEAKER_02: Kensa to your point earlier, it's this very general purpose orchestration tool, which means it doesn't naturally come to mind for a lot of use cases.
• 21:22
  SPEAKER_02: But when you step back and think about what it takes to build an LLM system and put it in production, naturally you have lots of batch jobs, scheduled jobs, and that's what Airflow's bread and butter is.
• 21:34
  SPEAKER_02: So you use Airflow as the orchestration tool, the orchestration engine.
• 21:38
  SPEAKER_02: And then the beauty of it is it's written in Python.
• 21:42
  SPEAKER_02: A lot of the LLM and ML specific frameworks and utilities are also written in Python.
• 21:47
  SPEAKER_02: So there's a very natural match there.
• 21:50
  SPEAKER_02: And you can go adapt it for your own use case, right?
• 21:52
  SPEAKER_02: In this case, we adapt Airflow to help us a lot with the operations of an LLM system.
• 21:59
  SPEAKER_02: But at the next deployment over at Astronomer, we use it for ETL for our data platform.
• 22:04
  SPEAKER_02: And the next deployment over, like we're building ML systems.
• 22:08
  SPEAKER_02: So it's the beauty of Airflow.
• 22:11
  SPEAKER_00: Always I never get tired of saying when people ask, can I use Airflow for this?
• 22:14
  SPEAKER_00: My answer is always yes.
• 22:16
  SPEAKER_02: Exactly.
• 22:17
  SPEAKER_02: Exactly.
• 22:18
  SPEAKER_00: All right.
• 22:18
  SPEAKER_00: Thank you for going through that.
• 22:20
  SPEAKER_00: That was a super interesting use case.
• 22:22
  SPEAKER_00: I'm very excited to see what comes of this and the next stage.
• 22:25
  SPEAKER_00: And I'm sure everybody on my team is going to be super excited to use this because we do spend a fair amount of time in the Airflow logs ourselves.
• 22:32
  SPEAKER_00: So we'll be keeping tabs on it for sure.
• 22:35
  SPEAKER_00: We're about at time.
• 22:36
  SPEAKER_00: We really appreciate both of you joining us today.
• 22:38
  SPEAKER_00: This has been a great conversation.
• 22:39
  SPEAKER_00: To close out, what is the best way for folks to get in touch with you?
• 22:43
  SPEAKER_02: We're both on LinkedIn.
• 22:44
  SPEAKER_02: So that's an easy way.
• 22:45
  SPEAKER_02: Or if you feel like remembering our email, mine's Julian, J-U-L-I-A-N at astronomer.io.
• 22:52
  SPEAKER_02: And David's is david.xue at astronomer.io.
• 22:59
  SPEAKER_00: That's great.
• 22:59
  SPEAKER_00: I think we'll probably have show notes or something like that that we can throw those in in case anybody's not writing them down while listening live.
• 23:06
  SPEAKER_00: But all right.
• 23:07
  SPEAKER_00: Again, thank you both.
• 23:08
  SPEAKER_00: It's been great having you.
• 23:10
  SPEAKER_00: And yeah, see everybody next time.
• 23:12
  SPEAKER_03: Thank you.
• 23:13
  SPEAKER_03: Bye.
• 23:14
  SPEAKER_03: Thank you.
• 23:16
  SPEAKER_03: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:23
  SPEAKER_03: You'll find links in the show notes to any resources mentioned in today's show.
• 23:26
  SPEAKER_03: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.