Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard

Transcript

How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: It's really nice to be able to look back at all of the old data that a user has had, regenerate what their inference is, regenerate what their worker should be, and just orchestrate that very easily.
• 0:10
  SPEAKER_03: That's something that I think would be much more difficult without like Airflow's orchestration tools.
• 0:16
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 0:25
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 0:37
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
• 0:45
  SPEAKER_00: Hello, everyone.
• 0:46
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:48
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by two great folks from Laurel, Vincent Law, a principal data scientist, and Jim Howard, a principal machine learning engineer.
• 0:57
  SPEAKER_00: How are you both doing today?
• 0:59
  SPEAKER_02: Doing well.
• 0:59
  SPEAKER_02: Thanks for asking.
• 1:00
  SPEAKER_02: It's a pretty lovely day here in San Francisco.
• 1:02
  SPEAKER_00: So and I heard Jim you said earlier you're also having a lovely day over in Minnesota.
• 1:07
  SPEAKER_03: That's right.
• 1:07
  SPEAKER_03: We don't get very often on sunny days, so we'll take it when we get to them.
• 1:10
  SPEAKER_00: Okay.
• 1:11
  SPEAKER_00: So I'm the only one without the lovely day.
• 1:13
  SPEAKER_00: It's very typically rainy here in Seattle.
• 1:16
  SPEAKER_00: We the joke amongst Seattleites is that summer does not start until after the 4th of July.
• 1:21
  SPEAKER_00: And it's holding true to that this year.
• 1:24
  SPEAKER_00: Typically, June is very much not summer here.
• 1:26
  SPEAKER_00: So we're still waiting on it.
• 1:28
  SPEAKER_00: Awesome.
• 1:29
  SPEAKER_00: Let's go ahead and dive in.
• 1:31
  SPEAKER_00: So for our listeners today's podcast, we're going to discuss how Laurel uses Airflow to power machine learning pipelines that are a cornerstone of their product.
• 1:40
  SPEAKER_00: Before we dive into your Airflow implementation, let's talk a bit about what Laurel does.
• 1:45
  SPEAKER_00: So I understand Laurel provides automated timekeeping software for legal and accounting teams.
• 1:50
  SPEAKER_00: Can you tell our listeners more about what that means and how that works in practice?
• 1:54
  SPEAKER_02: We call it the time automation space.
• 1:57
  SPEAKER_02: And we're an AI company.
• 1:58
  SPEAKER_02: And what that means is our customers are knowledge workers, as you say, currently in the legal and accounting space.
• 2:05
  SPEAKER_02: And what Laurel does is it uses AI to provide a ready-to-review timesheet.
• 2:10
  SPEAKER_02: So in terms of our customers, Laurel has partnered with AMLOWA 200 firms and launched into the accounting protocol with Ernst ⁇ Young.
• 2:18
  SPEAKER_02: Ultimately, our mission is to return time to knowledge workers.
• 2:22
  SPEAKER_02: But specifically for time automation, we have capture technology that comes in two parts.
• 2:29
  SPEAKER_02: The first is an application installed on our user's machine.
• 2:33
  SPEAKER_02: And the second is cloud integrations with common applications such as Zoom, Microsoft Teams, Microsoft Outlook, et cetera.
• 2:41
  SPEAKER_02: What you can imagine is this capture technology captures what we call work activities.
• 2:47
  SPEAKER_02: And what these work activities are, for example, with Zoom or Microsoft Teams or Outlook, are essentially the Zoom meetings or your Outlook meetings or your emails.
• 2:57
  SPEAKER_02: What the application that we install on the Windows does is it also captures essentially the active focus window on your computer and the duration of time that you spend on that window.
• 3:11
  SPEAKER_02: So if you spend 30 seconds on Google Chrome doing research and then five minutes writing a draft in Microsoft Word, and then you go back to Google Chrome to go on Westlaw as a lawyer to do research, we're capturing those activities and certain metadata about each of those activities and then how the duration that it is active on the user computer.
• 3:34
  SPEAKER_02: From the captured work activities specifically to AI, there are three fundamental AI problems that we are trying to solve in order to construct a ready timesheet for our users.
• 3:47
  SPEAKER_02: So the first, from an AI perspective, is that it's a clustering.
• 3:51
  SPEAKER_02: So we have to take these raw activities that I just described and group them in a meaningful fashion that is intelligible to the user so that they actually have a sensible time entry to build to their penultimate client.
• 4:05
  SPEAKER_02: The second AI, like classic AI problem that we're trying to solve is prediction.
• 4:08
  SPEAKER_02: We have these raw activities, like you were in Microsoft Outlook, you were in Word, you were writing an email, you were on a Zoom meeting, but we now have to predict what client and project that you want to build for.
• 4:20
  SPEAKER_02: And then the third fundamental AI problem that we're trying to do is summarization.
• 4:26
  SPEAKER_02: We capture all this regular work activity.
• 4:29
  SPEAKER_02: We not only have to group it in some intelligible fashion, and we not only have to try to predict what client you want to build to, but then we also need to write a narrative or summarize the work so that it is human intelligible and often a very strong requirement for billing as well.
• 4:45
  SPEAKER_02: And for that, we're starting to use large language models, which we'll get into later in our talk.
• 4:50
  SPEAKER_00: That sounds really cool.
• 4:51
  SPEAKER_00: And like it would be a massive time savings for anybody using it.
• 4:55
  SPEAKER_00: My random aside for this episode is that way back in a previous life, I used to work for a very small air quality consulting firm.
• 5:03
  SPEAKER_00: And we were constantly working with a whole bunch of different clients at the same time.
• 5:08
  SPEAKER_00: So we had similar to what you're describing, we had to keep track of our time for every single one of them.
• 5:13
  SPEAKER_00: And it was just such a massive pain in the butt to have to go into the timekeeping software every time.
• 5:18
  SPEAKER_00: And it's supposed to be down to 15 minutes, which I think is maybe even longer than like legal teams have.
• 5:23
  SPEAKER_00: But forgetting to do it and then going, uh oh, somebody called me and I switched what I was doing.
• 5:28
  SPEAKER_00: And now I have to like go figure this out in the timekeeping system.
• 5:31
  SPEAKER_00: Having something to automatically do that would be super, super helpful.
• 5:35
  SPEAKER_00: So I would imagine like tons of industries, even beyond where you are now, would eventually be interested in this.
• 5:41
  SPEAKER_02: Yeah, definitely.
• 5:42
  SPEAKER_02: I think we have definitely talked about expanding to the consulting protocol and really just across what the knowledge worker space.
• 5:48
  SPEAKER_00: Yeah.
• 5:48
  SPEAKER_00: So that's awesome.
• 5:50
  SPEAKER_00: So let's talk about how Airflow fits into that picture.
• 5:52
  SPEAKER_00: Where does Airflow fit in your tech stack and what do you use it for?
• 5:57
  SPEAKER_03: The way that we've structured our overall kind of system is similar to a lot of other companies where we have a bunch of different services that are responsible for their core components.
• 6:06
  SPEAKER_03: So we do have uh what we call an enrichment service that does most of our different AI inferences.
• 6:12
  SPEAKER_03: Airflow for this is used at a number of different levels.
• 6:15
  SPEAKER_03: So, you know, the first is the classic data reconciliation.
• 6:19
  SPEAKER_03: We may try and get data from other services, especially data that may not have certain time requirements.
• 6:25
  SPEAKER_03: When required, we also have a very event-driven architecture where we'll do things in real time.
• 6:31
  SPEAKER_03: But it also can help us then with slower batch jobs that may not have some of these same timing requirements.
• 6:37
  SPEAKER_03: As an example, we may generate what a user's total time throughout a month was on a daily basis.
• 6:43
  SPEAKER_03: And this can run as a batch job every evening.
• 6:46
  SPEAKER_03: There's also uh wider overall metrics, performance metrics for our NL systems, what the performance of any models are, we can check if there's a model degradation performance, different changes in data distribution, things like that.
• 6:58
  SPEAKER_03: But then we also use it for a number of other use cases that are more NL specific.
• 7:04
  SPEAKER_03: And this is around feature generation.
• 7:07
  SPEAKER_03: Certainly for some of our features, we generate them in real time.
• 7:10
  SPEAKER_03: But for some of our more computationally expensive features, and especially ones where the inferences don't have such real-time requirements, we can use Airflow for some more expensive feature generation.
• 7:20
  SPEAKER_03: We also use it then for model training, and we can touch on that a little bit more later.
• 7:25
  SPEAKER_03: But we've found that there's some really nice advantages using Airflow to generate very thin, customized user-specific models, and that you get a really nice bump in performance.
• 7:36
  SPEAKER_03: And then we also generate or use Airflow for generating inferences, especially uh more kind of time-consuming, computationally expensive inferences that don't have, again, those same kind of real-time requirements for our users.
• 7:50
  SPEAKER_00: Yeah, that's great.
• 7:50
  SPEAKER_00: So you've got Airflow at all points along the data journey from just the batch management of data all the way into managing your ML models.
• 7:59
  SPEAKER_00: That's right.
• 7:59
  SPEAKER_00: Yeah, that's great.
• 8:01
  SPEAKER_00: So I want to dive in a little bit more on this ML model management because this is a really hot topic right now and one where we see Airflow being used more and more.
• 8:11
  SPEAKER_00: So tell me a little bit more about the role that machine learning plays in your product.
• 8:15
  SPEAKER_00: You've touched on this a little bit already, but maybe you can give the listeners some more details.
• 8:19
  SPEAKER_00: And then how do you use Airflow to manage those models?
• 8:22
  SPEAKER_03: You know, Vincent kind of laid out a few different areas where we do machine learning, and maybe it would be helpful then to go through one specific example.
• 8:29
  SPEAKER_03: One that might be nice is we have this kind of clustering example.
• 8:32
  SPEAKER_03: We call them work groups.
• 8:34
  SPEAKER_03: But the idea is that we take a user who generates typically about 300 of these kind of like atomic actions every single day.
• 8:42
  SPEAKER_03: So when you're a knowledge worker working on your computer, you may have 300 different things.
• 8:48
  SPEAKER_03: You may not realize that that you touched throughout the day.
• 8:50
  SPEAKER_03: You touched a document, you were doing research on a website, you drafted an email, you even read emails, touching PDFs, things like that.
• 8:56
  SPEAKER_03: And we want to distill that down to something that's actually meaningful for our users.
• 9:01
  SPEAKER_03: And so the first thing is to try and reduce this overwhelm down to something that they would actually bill for.
• 9:06
  SPEAKER_03: And what we found is that a typical lawyer ends up billing about seven things a day.
• 9:12
  SPEAKER_03: So this is a huge, basically uh mapping problem of we have these 300 core activities and we want to map them down to seven different items.
• 9:20
  SPEAKER_03: And we try and tackle this via clustering, but not just conventional unsupervised clustering.
• 9:26
  SPEAKER_03: We first have a trained model that attempts to learn the distances between any actions that users have.
• 9:34
  SPEAKER_03: So that distance function ends up going into our clustering model.
• 9:38
  SPEAKER_03: And where we end up, the feature inputs into this can be quite expensive.
• 9:43
  SPEAKER_03: For example, a really meaningful feature would be to look at the embedding space of whatever that action is.
• 9:49
  SPEAKER_03: And so we can use like a large language model or sentence transformers generated embedding.
• 9:54
  SPEAKER_03: Where we end up using Airflow then is at a couple of different levels.
• 9:57
  SPEAKER_03: So the first one is for any new activity that comes in, we generate all of the different features that come in.
• 10:03
  SPEAKER_03: Most of these features are easy enough that we can generate in real time.
• 10:07
  SPEAKER_03: Something like what were the email addresses associated with an email?
• 10:10
  SPEAKER_03: This is not a problem to generate in real time.
• 10:12
  SPEAKER_03: But some of these may be more computationally expensive.
• 10:16
  SPEAKER_03: And so we hold off on that feature generation and we can do this in one big batch job.
• 10:21
  SPEAKER_03: The other place where we use Airflow then, as we were saying, is in the actual then generation of these clusters.
• 10:28
  SPEAKER_03: So we attempt to do them in real time, but we found that we were really computationally limited in real time to display them reasonably to our users.
• 10:35
  SPEAKER_03: So what we have is if you try and look throughout the day as you're generating activities, you can have these clusters generated for you, but they may not look as good as our orchestrated job that goes through the next day and attempts to do everything associated with this clustering activity.
• 10:54
  SPEAKER_03: And a big part of that is because we actually have after the clustering is done, there's another task that attempts to assign a likelihood that this overall cluster is associated with a buildable entry.
• 11:04
  SPEAKER_03: So that's another spot.
• 11:06
  SPEAKER_03: And then the third way that we use Airflow in this is as we were talking about, is generating custom user models.
• 11:12
  SPEAKER_03: So what we found is that that kind of first phase distance model, if we did it at a global level, ends up being reasonably accurate.
• 11:22
  SPEAKER_03: For those in kind of the machine learning, no, we end up getting AUC of, you know, kind of in the 0.7, 0.8 range.
• 11:30
  SPEAKER_03: But if we found that if we were to use Airflow to orchestrate at a user-specific level, models, we can jump that performance quite a bit.
• 11:38
  SPEAKER_03: For every one of our individual users, we retrain every other day a thin model on top of these kind of core-based models, and it ends up giving really nice performance benefits.
• 11:51
  SPEAKER_00: Interesting.
• 11:51
  SPEAKER_00: So you have models for every single user, you must be dealing with a pretty high scale.
• 11:57
  SPEAKER_03: Yeah, I feel as we're growing, we're constantly bumping up what our airflow uses are underneath as we estimate what our overall usage rate is.
• 12:05
  SPEAKER_03: But yes, and actually we are constantly analyzing what the overall kind of DAG computation times are too, and and we keep having to bump that up so that they don't get uh too large and play other tricks to reduce the overall runtime.
• 12:18
  SPEAKER_03: Our next level DAG recently was hitting three to four hours, and we had to break that up a little bit and add more resources to it.
• 12:25
  SPEAKER_00: That makes sense.
• 12:26
  SPEAKER_00: So I have two follow-up questions.
• 12:28
  SPEAKER_00: The first is I think this pattern you're describing of using airflow to manage models and model retraining, especially in that batch sense where you're doing everything on a time-based scale, is getting very much more common and has over the last year.
• 12:44
  SPEAKER_00: We saw that in the Airflow survey folks this year compared to last year, like this huge jump in companies saying that they were using Airflow in this manner.
• 12:52
  SPEAKER_00: What about Airflow do you think makes it ideal for that use case?
• 12:58
  SPEAKER_00: Or what have you found in terms of the way Airflow works or the features or anything like that that particularly well suits it towards what you're doing?
• 13:06
  SPEAKER_03: Yeah, it's a great question.
• 13:08
  SPEAKER_03: I think holistically using an orchestration tool to do model retraining is just really nice, right?
• 13:15
  SPEAKER_03: Because there's the ability to easily, let's see, what are some of the core features?
• 13:20
  SPEAKER_03: One is the ability to easily link it to some sort of logging or alerting platform so that we can specifically understand what users maybe failed.
• 13:28
  SPEAKER_03: The ability to automatically do even down to user-specific retries on some elements of the DAG.
• 13:34
  SPEAKER_03: Actually, the way that we've generated our user model training makes this much easier so that you don't need to rerun the entire DAG through everyone at once again if anything failed.
• 13:44
  SPEAKER_03: And the ability to do backfills is uh something that we've really liked too, because one thing that isn't something that we want, for example, is that if we wanted to update a new model and we wanted to, for example, let me back up.
• 13:59
  SPEAKER_03: A really common interaction that we have for our users is that they may take an entire month where they don't do any of their time, and then they do all of their time at the end of the month.
• 14:08
  SPEAKER_03: So a feature that we really like, the ability to do backfills or the ability to generate new models and generate new data that Airflow allows, is if we're playing with new versions of models or new features or new improvements, it's really nice to be able to look back at all of the old data that a user has had, regenerate what their inferences are, regenerate what their work should be, and just orchestrate that very easily.
• 14:32
  SPEAKER_03: That's something that I think would be much more difficult without like Airflow's orchestration tools.
• 14:37
  SPEAKER_00: Yeah.
• 14:37
  SPEAKER_00: So you're maintaining the model history just as the DAG run history.
• 14:41
  SPEAKER_03: That's right.
• 14:42
  SPEAKER_03: Yes.
• 14:42
  SPEAKER_03: In fact, in our model store, we do have the model version history.
• 14:47
  SPEAKER_03: We maintain even like what the training and test performance was at the time.
• 14:51
  SPEAKER_03: One thing that we are working on, and we're not quite there yet, is like a really nice historic use case to be able to redo model training at exactly what that the input data was at that time.
• 15:04
  SPEAKER_03: And I think that's a really hard problem in general, is like training data as code and maintaining that.
• 15:10
  SPEAKER_00: Yeah, for sure.
• 15:11
  SPEAKER_00: I think that item potency is something that we talk about a lot just in terms of a best practice in how you design your DAGs, which is obviously much easier, I think, with just like a batch ETL trying to maintain that logic within your pipeline as opposed to, as you said, like an ML pipeline where you've got training data.
• 15:28
  SPEAKER_00: So that's really interesting.
• 15:29
  SPEAKER_00: My second follow-up question was on the LLM side.
• 15:32
  SPEAKER_00: So you mentioned those briefly a couple of times.
• 15:34
  SPEAKER_00: This is, I think, another really hot topic right now that a lot of companies are wading into.
• 15:39
  SPEAKER_00: I'm curious if we can dig in a little bit more into how you're looking at LLMs within this model management.
• 15:47
  SPEAKER_00: I think you mentioned using it for cost management.
• 15:50
  SPEAKER_00: Are there anything else you want to say there or any other use cases you're looking into?
• 15:55
  SPEAKER_02: Yeah, I'll touch a little bit on the cost management side.
• 15:59
  SPEAKER_02: And this relates to what Jim was saying earlier about in our use case per user per day, whether you realize it or not, like you're touching, uh, we found in our data that the average lawyer or talk touches maybe 300 things on their computer, right?
• 16:14
  SPEAKER_02: So they're flipping back and forth.
• 16:16
  SPEAKER_02: Jim will say between email or a document or a Zoom meeting, and then maybe during the Zoom meeting, they're also taking notes.
• 16:21
  SPEAKER_02: So they're in a Google Doc or in the Microsoft Word taking notes, whatever the case might be.
• 16:26
  SPEAKER_02: Now, if we're trying to summarize these activities, you can imagine a world where as these activities are coming in and we use an LLM, a large language model, to try to summarize the activities as they're coming in, you can imagine a world where that very quickly becomes cost prohibitive, right?
• 16:41
  SPEAKER_02: Because you have 300 activities as we're streaming activities in.
• 16:46
  SPEAKER_02: And then if you're super naive and continuously hit an LLM, that's very unnecessary.
• 16:51
  SPEAKER_02: And so what we've actually found is actually a great cost management tool, is we collect the activities in real time.
• 16:59
  SPEAKER_02: We actually have a very simple heuristic-based summarizer that is very cheap and very fast to run.
• 17:06
  SPEAKER_02: It's essentially you we cluster the activities together and then we take the longest common substring in various metadata that we have to generate a very quick description.
• 17:16
  SPEAKER_02: And it's not great, but it's also not often not terrible either.
• 17:20
  SPEAKER_02: So that if the user is looking at Laurel in real time, they at least have some kind of description to key them in on what they worked on at that period of time.
• 17:31
  SPEAKER_02: But often those that heuristic-based description is very insufficient for a billing purpose, right?
• 17:40
  SPEAKER_02: But when a lawyer submits a bill, they have to have a very intelligible narrative that fits compliance guidelines and will be accepted by their penultimate client and be accepted for payment.
• 17:52
  SPEAKER_02: So what we do is the way we use Airflow for this is we scheduled a DAG where at 3 a.m.
• 18:00
  SPEAKER_02: at the user's local time zone, usually when they're sleeping, we our DAG actually goes back on yesterday's date and goes through all the work that the users have done and essentially cleans up both the grouping, the intelligent grouping of the work that we do, and also the descriptions associated with those groups.
• 18:21
  SPEAKER_02: And we're actually currently testing a few different both prompts and also different models.
• 18:27
  SPEAKER_02: When we've looked at this ourselves, for example, like GPT 3.5 Turbo is running at something like 50 cents per million tokens, input tokens, and for output tokens, maybe a dollar 50 uh per million tokens, for GPT-4.0 is about 10 times the cost.
• 18:44
  SPEAKER_02: In our case, we're the summaries that we're trying to generate are really only at max 12 words.
• 18:52
  SPEAKER_02: It's it's not we're not trying to generate a whole novel here.
• 18:55
  SPEAKER_02: So one of the things that we're trying to do and we're trying to experiment with is trying these different models, like and can suffice with uh a cheaper model for this task.
• 19:05
  SPEAKER_02: And one of the things that we can do with the prompt, we've implemented a couple different RAG-based approaches too, by essentially retrieving narratives or summaries that the users themselves have written historically to feed into the prompt so that hopefully that the output can encapsulate some of the user-specific tone and style that they generally like to write.
• 19:26
  SPEAKER_02: And what we found is that because by using Airflow to orchestrate a job the next day, we're able to really perform this summarization in a very cost-effective manner.
• 19:38
  SPEAKER_02: We've also, I think we're very excited.
• 19:40
  SPEAKER_02: We we're also currently experimenting inside the Airflow DAG.
• 19:44
  SPEAKER_02: We essentially are experimenting in an AD testing style manner, of which as we test different types of models, like how can we assess the efficacy of the output of each model?
• 19:55
  SPEAKER_02: And that part is it's super exciting work.
• 19:57
  SPEAKER_02: Yeah.
• 19:57
  SPEAKER_00: Yeah, that's great.
• 19:58
  SPEAKER_00: That makes total sense that you would use LLMs in that way for the summarizations.
• 20:03
  SPEAKER_00: I think that we see a lot of use cases popping up around that for anything that needs to be human readable, they're really useful there.
• 20:09
  SPEAKER_00: So that's really cool to see that you're getting some cost savings with implementing them using Airflow in that way.
• 20:15
  SPEAKER_00: I know we are almost at the end of our time, and I always like to end with a standard final question.
• 20:20
  SPEAKER_00: And that's what would you most like to see from development of the Airflow project in the near future?
• 20:25
  SPEAKER_02: I'll start with one and uh one thing that we encountered very recently, actually, is actually I just want to expand on something that Jim mentioned earlier, which is we have these long-running DAGs, sometimes because we're performing clustering, then we're hitting an LLM.
• 20:40
  SPEAKER_02: And one of our shorter term solutions was basically we essentially just split the DAG manually into, I think I think we chose eight different DAGs and to essentially like power try to parallelize it to get the DAG times faster.
• 20:54
  SPEAKER_02: And I feel like that's something that like I wonder if it's possible to really parallelize those jobs automatically and really like more support and tooling around okay, if you have these like long-running DAGs, like how can we like just automatically speed that up if there is parallelization or something else?
• 21:10
  SPEAKER_02: I think that another use case that we've been coming through is especially with the large language models used using OLMs to summarize, as I talked about earlier, as we're testing different prompts and different versions, uh potentially like fine-tuning, right?
• 21:25
  SPEAKER_02: Like more tooling to make that part easier.
• 21:29
  SPEAKER_02: I'd be curious to learn more and figure out like what might be possible in Airflow around that use case, especially since it is so popular.
• 21:35
  SPEAKER_02: And I imagine some of the other companies are doing something similar to what we just described.
• 21:39
  SPEAKER_00: Yeah, for sure.
• 21:40
  SPEAKER_00: Jim, anything you wanted to add to that?
• 21:43
  SPEAKER_03: Yeah, I think I'd love the ability to, because we're on the engineering side, and maybe this is a dangerous thing to add in, but it's nice when we get a lot of requests from our product or sales people or some of our other partners where they would have the ability to generate certain specific results that they want, especially if these come from orchestrated DAGs.
• 22:06
  SPEAKER_03: One way that I thought might be easy is if they just update a tag or do a new tag and then they go through the Airflow UI and they see something run as a nice way.
• 22:14
  SPEAKER_03: So I'd love to be able to get something like tag access in there.
• 22:17
  SPEAKER_03: So they can just say, like, oh, this is the sales demo, please run.
• 22:20
  SPEAKER_03: And it's just done.
• 22:21
  SPEAKER_03: Right now, we end up breaking out different DAGs for some of these individual custom teams or doing some of these things that are a little bit less clean, I think.
• 22:28
  SPEAKER_00: Okay.
• 22:29
  SPEAKER_00: So updating tags in the UI at runtime as opposed to in the DAG code itself.
• 22:35
  SPEAKER_03: Exactly.
• 22:35
  SPEAKER_03: Or so then the DAG code can also have access to those tags and then make custom decisions and things like that.
• 22:41
  SPEAKER_00: Okay.
• 22:41
  SPEAKER_00: That makes sense.
• 22:42
  SPEAKER_00: I think what both of you, those are very different requests, but I think both of them are things that I've heard before, both in terms of kind of customizability for how we say like less technical personas can like interact with Airflow.
• 22:55
  SPEAKER_00: I know that's something that we're thinking about in terms of the Airflow 3 roadmap, which is a ways out, but that's great feedback.
• 23:01
  SPEAKER_00: And I think as well for the managing lots of large, larger tasks or like long-running jobs at this more automatically as opposed to you having to do it proactively right now.
• 23:11
  SPEAKER_00: There's not really a good way to fix what's happening once it's already happening.
• 23:15
  SPEAKER_00: You have to know and set it up ahead of time.
• 23:17
  SPEAKER_00: So that's great feedback for sure.
• 23:19
  SPEAKER_00: Okay.
• 23:19
  SPEAKER_00: Thank you so much to both of you for joining us.
• 23:21
  SPEAKER_00: We really appreciate your time.
• 23:23
  SPEAKER_00: What's the best way for folks to get in touch with you?
• 23:25
  SPEAKER_02: You can reach out to me at vincent at world.ai.
• 23:28
  SPEAKER_00: And I'm at gym at world.ai.
• 23:31
  SPEAKER_00: Great.
• 23:31
  SPEAKER_00: We can put those in the show notes.
• 23:32
  SPEAKER_00: Thank you both again.
• 23:33
  SPEAKER_00: This has been a really fun conversation, and we really appreciate having you on.
• 23:36
  SPEAKER_03: Thank you.
• 23:37
  SPEAKER_03: Appreciate it.
• 23:40
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:46
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 23:50
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.