Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer

Transcript

A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: You obviously can't know the future, but I think if in a year or two years we're gonna look back on this time, other than finding it really funny what we were stressed about, we're gonna see this as one of the defining moments of Airflow.
• 0:13
  SPEAKER_03: Because Airflow 3 brings so many fundamental changes, like a whole architecture change, a completely new UI, and of course also improved features and features people have asked for a long time now.
• 0:26
  SPEAKER_03: We didn't even ask in this survey because we know that if we ask that, everyone answers it, and people probably know which feature I'm talking about.
• 0:39
  SPEAKER_02: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:49
  SPEAKER_00: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:54
  SPEAKER_00: Join the Astronomer Roadshow for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 1:03
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:11
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:16
  SPEAKER_00: Register now.
• 1:17
  SPEAKER_00: The link is in the show notes.
• 1:20
  SPEAKER_01: Hi everyone, welcome to the Data Flowcast.
• 1:23
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm very excited to be joined by Tamara Fingerlin, who is a developer advocate at Astronomer.
• 1:31
  SPEAKER_01: Uh, not every day that you get to interview someone from your own team, so this is very fun.
• 1:36
  SPEAKER_01: And Tamara, how are you doing?
• 1:37
  SPEAKER_03: Great.
• 1:38
  SPEAKER_03: Yeah, not every day that you get interviewed by a manager.
• 1:42
  SPEAKER_03: In public.
• 1:43
  SPEAKER_03: We talk every day, all day about everything on Slack and on Zoom, but yeah, no, I'm really excited for this.
• 1:50
  SPEAKER_01: We don't normally make it available to the general public, as usual.
• 1:54
  SPEAKER_01: So yeah, special day and insight into our working relationship today.
• 1:58
  SPEAKER_01: Yeah, that's great.
• 1:59
  SPEAKER_01: Let's go ahead and dive in.
• 2:01
  SPEAKER_01: So for today's podcast, we're going to be discussing the 2025 State of Airflow report, which just came out recently.
• 2:09
  SPEAKER_01: We're really happy about it.
• 2:10
  SPEAKER_01: And so for those who might not be familiar, let's start with some background.
• 2:14
  SPEAKER_01: So, tomorrow, what is the state of airflow and why is it important for the airflow community?
• 2:20
  SPEAKER_03: Yeah, the state of airflow is a combined report.
• 2:23
  SPEAKER_03: So Astronomer puts out this report every year and combines the anonymized telemetry that we get from our customers on how they use Airflow with results from the Apache Airflow survey.
• 2:34
  SPEAKER_03: And you might have seen the survey around.
• 2:36
  SPEAKER_03: That's really the big part of the report.
• 2:38
  SPEAKER_03: And this survey is extremely important, not just for astronomer.
• 2:42
  SPEAKER_03: We care about it a lot.
• 2:43
  SPEAKER_03: Obviously, we care about Airflow and the report and we write it, but it's extremely important for the Airflow project itself.
• 2:50
  SPEAKER_03: Because this project is really community driven.
• 2:53
  SPEAKER_03: So the development and decisions on what to develop next, which features to prioritize, or sometimes even the idea of a new feature, is all driven by the community and based on community opinion.
• 3:05
  SPEAKER_03: So if you get to ask your users, hey, do you use this feature?
• 3:09
  SPEAKER_03: Which of these executors do you prefer?
• 3:12
  SPEAKER_03: That really gives the project developers insights that they can use to prioritize features, and they are doing that every time.
• 3:19
  SPEAKER_03: So this is really important to the FLO project.
• 3:21
  SPEAKER_03: And yeah, I'm excited to be a part of that.
• 3:24
  SPEAKER_03: And this year, I don't know if people have seen the results who are listening, but we had over 5,000 responses.
• 3:32
  SPEAKER_03: So we had 5,255 people who filled out the survey, which just blows my mind from 116 different countries, like distinct countries all over the world.
• 3:43
  SPEAKER_03: That was one of the first analyses that I ran, was like, we asked for people's cities, and most people responded.
• 3:48
  SPEAKER_03: And then I was like, wow, over 100 countries, like really showed how global airflow is.
• 3:53
  SPEAKER_03: And thanks to everyone who filled out the survey who's listening, it's really important and helpful.
• 3:58
  SPEAKER_03: And if you missed it, you'll get a chance next year.
• 4:00
  SPEAKER_01: Yes, we will continue to do it every year.
• 4:02
  SPEAKER_01: As you mentioned, I think our regular listeners will have heard me talk multiple times about the Airflow survey and how important the results are.
• 4:10
  SPEAKER_01: I'm gonna bury the lead a little bit.
• 4:12
  SPEAKER_01: We'll talk more later in the episode about upcoming 3.0, but especially for these big releases for the Airflow project.
• 4:19
  SPEAKER_01: As you said, like a lot of those features are pulled from the results of the Airflow survey.
• 4:24
  SPEAKER_01: So it's a hugely important asset for the community.
• 4:27
  SPEAKER_01: And yeah, this year we had so many people fill it out, which was really great.
• 4:31
  SPEAKER_01: It's obviously more useful the more people we have respond.
• 4:35
  SPEAKER_01: You mentioned a little bit in there that you did some of the analysis for the survey.
• 4:40
  SPEAKER_01: So maybe you can tell everybody how astronomer is involved in the open source airflow survey and what your role was in this project.
• 4:47
  SPEAKER_03: Astronomer really drives the survey.
• 4:50
  SPEAKER_03: Of course, the questions are community-decided, but after that, astronomer takes the survey and publicizes it, runs it, collects all of the results.
• 4:59
  SPEAKER_03: And this year and last year too, I got to be the one to analyze it.
• 5:03
  SPEAKER_03: It's a really special feeling to be the first person who sees the results and to just get this very large CSV file.
• 5:11
  SPEAKER_03: And then you just dive in.
• 5:12
  SPEAKER_03: And to be honest, the first two days, I just did random analyses on what I found most interesting, like just wanted to know first, and then I started to like, in a structured way, started to go through the results.
• 5:24
  SPEAKER_03: So that was uh a large part of my January, just looking at those over 5,000 responses and see what people want and try to find trends, try to visualize that, and yeah, figure out what the insights are and communicate that back to the Airflow developers and then also to the larger community.
• 5:40
  SPEAKER_03: We did a State of Airflow webinar.
• 5:42
  SPEAKER_03: In case you missed that, I think there's gonna be a link to that somewhere alongside this podcast.
• 5:47
  SPEAKER_03: And yeah, just being in the middle and seeing all of the feedback from developers, and then also being able to give that feedback back to developers and tell them, hey, you voted, you really like this feature, we heard you.
• 6:00
  SPEAKER_03: So that was my role, and then of course, astronomer took all of the results.
• 6:04
  SPEAKER_03: Yeah, it was a whole team effort.
• 6:06
  SPEAKER_03: I want to shout out definitely Bream, who is our community manager, who ran all of the organization around the survey.
• 6:12
  SPEAKER_03: You've seen Mark and Bertie share the survey on LinkedIn, which that's at least 2,000 answers there.
• 6:18
  SPEAKER_03: Instantly.
• 6:19
  SPEAKER_03: Yeah, and then also for my team.
• 6:20
  SPEAKER_03: If you like the infographic that's on the Airflow website that shows some of the results, Aaron was the person making that, and of course, also designing the whole report.
• 6:29
  SPEAKER_03: In the end, it was like a whole team effort here to create the report.
• 6:33
  SPEAKER_03: But yeah, I got to be involved from the start of thinking about which questions to ask to the very end of finalizing the report, and that was a really interesting journey.
• 6:42
  SPEAKER_03: Yeah, that's great.
• 6:43
  SPEAKER_01: I have to say, exploratory data analysis was never something that I really liked myself.
• 6:48
  SPEAKER_01: That extremely open-endedness can be challenging.
• 6:52
  SPEAKER_01: So that's here, you know, in a category that loves doing that.
• 6:55
  SPEAKER_01: Yeah, that's awesome.
• 6:57
  SPEAKER_01: And working with all of that data, what do you think the big takeaways were from the results of the survey?
• 7:03
  SPEAKER_03: There's there's so many.
• 7:04
  SPEAKER_03: Like, how much time do we have today?
• 7:06
  SPEAKER_03: But 15 minutes.
• 7:07
  SPEAKER_03: Okay, so I have to focus.
• 7:08
  SPEAKER_03: Yeah, there were a lot of results.
• 7:10
  SPEAKER_03: I think one that stood out for me was like we asked people how important airflow is to their business.
• 7:16
  SPEAKER_03: And we knew from like previous surveys and from our customers, there are businesses where airflow is really important.
• 7:22
  SPEAKER_03: But we've never asked 5,000 people how important airflow is.
• 7:26
  SPEAKER_03: And that was something that was really interesting to me.
• 7:29
  SPEAKER_03: Over 90% came back and said, yeah, airflow is important to their business.
• 7:34
  SPEAKER_03: And seven 47.9% said that it's very important.
• 7:38
  SPEAKER_03: And the way we phrased the question was, what happens if it breaks?
• 7:42
  SPEAKER_03: If you want to find out the importance of something, like really anything or anyone, ask what happens if that thing doesn't work or if the person doesn't dare.
• 7:49
  SPEAKER_03: And yeah, almost 48% said that's a critical impact.
• 7:53
  SPEAKER_03: There's an impact across the whole business if airflow is down.
• 7:58
  SPEAKER_03: So that was something that was really interesting to see.
• 8:00
  SPEAKER_01: Yeah, I think not surprising probably to us, obviously, an astronomer and a lot of other people that use Airflow, we certainly see that amongst our customer base, the Airflow is really critical, but it is great to see those results from again, such a large population and really varied in terms of location and title and company and all of that.
• 8:23
  SPEAKER_01: Yeah.
• 8:23
  SPEAKER_03: Yeah.
• 8:23
  SPEAKER_03: And I think the second thing that kind of goes hand in hand with that, like if Airflow is important, why are things important in businesses?
• 8:30
  SPEAKER_03: Because in the end of the day, they make money.
• 8:32
  SPEAKER_03: So we had a question that was about revenue generation.
• 8:35
  SPEAKER_03: And this year was the first year that we asked that question.
• 8:38
  SPEAKER_03: So we asked, are you using Airflow for revenue generating or external use cases?
• 8:44
  SPEAKER_03: So, for example, if you are a business that interacts with customers, are you using Airflow to provide data to your customers?
• 8:51
  SPEAKER_03: For example, in a dashboard that would be a customer-facing use case, and that is eventually revenue generating.
• 8:57
  SPEAKER_03: So we asked, not just if people are doing that now, but do they expect to do that more or less in the next year?
• 9:05
  SPEAKER_03: So a future question.
• 9:07
  SPEAKER_03: And we got over 85% who either said they're gonna do it more or even much more, or they said they're not doing it right now, but they're going to start.
• 9:17
  SPEAKER_03: They expect to start in the next year.
• 9:19
  SPEAKER_03: So that was another thing that it makes sense.
• 9:21
  SPEAKER_03: People say, okay, Airflow is important and critical to their business.
• 9:24
  SPEAKER_03: And then they say, Yeah, because it makes money.
• 9:26
  SPEAKER_03: And that thought that people have built whole businesses on top of Airflow, and that you and me with all our content hopefully help make those businesses succeed.
• 9:36
  SPEAKER_03: That was a really nice thought, and that was another like result that stood out.
• 9:41
  SPEAKER_03: And I think the last one of the three big ones in my mind is this was probably the second thing that I checked.
• 9:48
  SPEAKER_03: Like first I looked how many countries and was there anyone else from Switzerland?
• 9:52
  SPEAKER_03: That's those are the things that you check first in a survey when you get the data.
• 9:56
  SPEAKER_03: But after that, I looked at how many people are using Airflow for MLOps and AI.
• 10:02
  SPEAKER_03: Because that's something that's still very new and that we have created a ton of content about over the last year.
• 10:09
  SPEAKER_03: And I got I was really surprised that it's now one of the major use cases.
• 10:15
  SPEAKER_03: MLOps had 21% of all users said that they already use Airflow for MLOps, and 8% said they are using it for Gen AI already.
• 10:24
  SPEAKER_03: Things are moving fast, sometimes almost a little too fast.
• 10:28
  SPEAKER_03: And then I looked at the subset of experienced Airflow users.
• 10:32
  SPEAKER_03: So with all of these analyses, obviously I looked at the pure percentage, but I also started to make subset analyses.
• 10:38
  SPEAKER_03: What is it, what are the results if you only look at big corporations or only look at experienced users?
• 10:44
  SPEAKER_03: And for this question, I looked at users with five or more years of experience.
• 10:49
  SPEAKER_03: And of those, it was 30% who use it for MLOPs.
• 10:54
  SPEAKER_03: So you saw that increase as people mature and as people are more experienced with Airflow, which was a nice validation because if you have a lot of experience in a tool, you know what it's good at and what it's not good at.
• 11:05
  SPEAKER_03: And those people seem to have decided that yes, Airflow is good for MLOps use cases.
• 11:10
  SPEAKER_03: So that was the third thing, I think, where I was like, okay, this is a really great result, and this goes on the first or second page of the report.
• 11:17
  SPEAKER_01: Yeah, that's awesome.
• 11:18
  SPEAKER_01: And I think those two last ones that you said, the growth in using Airflow for uh critical use cases or revenue generating use cases, and this increase in different or I should say, revenue generating business things, and then using Airflow for different use cases totally make sense together, right?
• 11:37
  SPEAKER_01: I think it's indicative of both the strength of the project and how it has grown over time.
• 11:43
  SPEAKER_01: We've seen so many more just users of Airflow and such a bigger community from the 1.x days five-ish years ago to now the functionality and stability of Airflow keeps growing with the open source project, which just enables you to use it for more critical business things.
• 12:02
  SPEAKER_01: And then on top of it, new use cases are coming up all of the time.
• 12:06
  SPEAKER_01: Jenny wasn't really a thing a couple of years ago.
• 12:09
  SPEAKER_01: And so as you expect to integrate Airflow into more of those use cases, you get to use it for more kind of critical things for your business.
• 12:18
  SPEAKER_03: Yeah, Airflow draws you in by being good at ELT and ETL.
• 12:23
  SPEAKER_03: And then suddenly you have that light bulb moment of, wait, anything I can write in Python?
• 12:28
  SPEAKER_03: And then it takes over your life.
• 12:33
  SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:42
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:54
  SPEAKER_02: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:02
  SPEAKER_01: I think this is great timing with talking about what's going on in the Airflow project.
• 13:06
  SPEAKER_01: I've already mentioned this like four times in this episode, but the Airflow 3.0 release is right around the corner.
• 13:13
  SPEAKER_01: So given what you've just mentioned in the survey results and having du dived really deep into that analysis, what are you most excited about with this major release?
• 13:23
  SPEAKER_01: And how do you think that 3.0 is going to impact the community?
• 13:26
  SPEAKER_03: I think you obviously can't know the future, but I think if in a year or two years we're gonna look back on this time, other than finding it really funny what we were stressed about, we're gonna see this as one of the defining moments of Airflow.
• 13:40
  SPEAKER_03: Because Airflow 3 brings so many fundamental changes, like a whole architecture change, a completely new UI, and of course also improved features and features people have asked for a long time now.
• 13:54
  SPEAKER_03: We didn't even ask in this survey because we know that if we ask that, everyone answers it, and people probably know which feature I'm talking about.
• 14:02
  SPEAKER_03: Gonna reveal that in a second.
• 14:04
  SPEAKER_03: But yeah, it's it really answers people's needs and questions and the needs of people who are using Airflow every single day, like you and me.
• 14:13
  SPEAKER_03: So I'm really excited about that.
• 14:15
  SPEAKER_03: And I think the first thing, especially like now that we're testing the beta.
• 14:20
  SPEAKER_03: So this morning I worked with the Airflow beta free release.
• 14:24
  SPEAKER_03: So not ready for production, but actually really far along now.
• 14:27
  SPEAKER_03: And the first thing that you notice is the completely new UI.
• 14:31
  SPEAKER_03: Like you open it, you set it to dark mode if you meet, and then you get started.
• 14:36
  SPEAKER_03: And it's just it's so much nicer to navigate.
• 14:39
  SPEAKER_03: One of my favorite things about the new UI is that if you enter a DAG, you are extremely fast.
• 14:45
  SPEAKER_03: You can see the recently failed tasks and just the arrow lines.
• 14:49
  SPEAKER_03: Like those get surfaced to you, and this is gonna just save me so much time in my day-to-day job that I was super excited when I saw that.
• 14:58
  SPEAKER_03: And yeah, I think alongside the UI, we recently had a webinar even about data sets.
• 15:05
  SPEAKER_03: This is the most adopted feature in Airflow in the recent survey.
• 15:08
  SPEAKER_03: Almost half of Airflow users are using datasets now.
• 15:12
  SPEAKER_03: And data sets are gonna be renamed to asset, but you keep all of the functionality that you have, just a replacement of the word, and then on top of that, there's a whole new way of defining your pipelines with a new decorator called add asset.
• 15:28
  SPEAKER_03: And I'm very excited about this more data-centric way of writing pipelines, not just because it's less code and less typing, but also because I like the idea that you can now shift in your mindset.
• 15:39
  SPEAKER_03: If you have a pipeline where it's better to define it in a task-oriented approach, you can still do that.
• 15:44
  SPEAKER_03: Your DACs will all still run that you have with some minor migration changes, but most of the DAC code will stay the same.
• 15:51
  SPEAKER_03: And then you have this new asset-oriented approach where you define the dick the data that you want.
• 15:57
  SPEAKER_03: And I really like that it's almost like a click in your brain that you can shift your mindset around, and now you can use Airflow for both.
• 16:04
  SPEAKER_03: So I'm very excited about that.
• 16:06
  SPEAKER_03: And another thing that I think is important to mention because it's something that I think most people will not directly experience because you see the UI change and you see the new features.
• 16:19
  SPEAKER_03: But the architecture change is something that's hidden for most end users.
• 16:23
  SPEAKER_03: But this is a huge effort from the Airflow developers.
• 16:25
  SPEAKER_03: I see that every day in the Slack chats when they discuss something about the API, how to best structure that, how to make it more performant, more secure.
• 16:34
  SPEAKER_03: I think that's fascinating.
• 16:35
  SPEAKER_03: I love watching these discussions, and I think that's something that's gonna it's really enabling more features that were just not possible with Airflow 2, like remote execution, and also running tasks in other languages.
• 16:48
  SPEAKER_03: I will soon have an excuse to learn Golang, and I'm very excited for that.
• 16:53
  SPEAKER_03: That's well that's gonna be the experimental first STK for another language.
• 16:58
  SPEAKER_01: Learning all the time about new priorities for my team.
• 17:02
  SPEAKER_03: Yeah, I'm just gonna take a couple days off to learn Golang in June.
• 17:06
  SPEAKER_01: Seems great.
• 17:07
  SPEAKER_01: Those are all really exciting things, and I think to your point, all of them are coming directly from results of either this year's or previous year's survey, mostly previous years, I guess, for 3.0 since development started before we did the 2024 survey.
• 17:22
  SPEAKER_01: But especially like more functionality for data sets, I know has been a top community request for a long time.
• 17:29
  SPEAKER_01: Better debugging capabilities with kind of the UI changes and easier access to the logs and log summaries and things like that has been a big request for a long time.
• 17:40
  SPEAKER_01: So, yeah, really the community is driving the direction of the project, including big releases like 3.0, which is really great.
• 17:47
  SPEAKER_03: Yeah, exactly.
• 17:48
  SPEAKER_03: Like the Airflow survey is your way to make your voice heard, and you can't talk about feature requests and survey and airflow free without mentioning DAG versioning.
• 17:57
  SPEAKER_03: So that's the one feature that has been the most requested of all time in every survey.
• 18:04
  SPEAKER_03: And I tested it last week for the first time.
• 18:07
  SPEAKER_03: I made a little gif and posted it on LinkedIn.
• 18:11
  SPEAKER_03: And yeah, people really love it and are excited for it.
• 18:14
  SPEAKER_03: And it's really cool.
• 18:15
  SPEAKER_03: Like seeing how your DAG has changed over time and just flipping through the versions.
• 18:20
  SPEAKER_03: That's a major like step of Airflow.
• 18:22
  SPEAKER_03: And I'm really excited what people are going to do with it.
• 18:25
  SPEAKER_01: Yeah, DAG versioning is so exciting.
• 18:28
  SPEAKER_01: Definitely one of those features that I think almost every single Airflow user will use.
• 18:32
  SPEAKER_01: I fully expect in next year's Airflow survey that to surpass data sets and the which features do you use?
• 18:38
  SPEAKER_01: Percentages.
• 18:39
  SPEAKER_01: Well, see, we could start a betting pool on that.
• 18:41
  SPEAKER_01: The DevRel team in Astronomer has many betting pools.
• 18:45
  SPEAKER_01: We're here for predicting what's gonna happen in Airflow.
• 18:48
  SPEAKER_01: Yeah, that's awesome.
• 18:49
  SPEAKER_01: We're all very excited for the release that should be coming out very soon when this episode is published, and we'll have lots of resources here for folks to get started with it.
• 19:00
  SPEAKER_01: Somewhat relevant for that since we are almost out of time.
• 19:03
  SPEAKER_01: Uh, I always end with a standard final question.
• 19:06
  SPEAKER_01: It's particularly timely, I think, right now.
• 19:08
  SPEAKER_01: But that is what would you most like to see from development of the Airflow project in the near future?
• 19:14
  SPEAKER_03: So I have a long wish list, of course.
• 19:16
  SPEAKER_01: I will add some color to this that I love asking you this question because I think you get to interact with so many different developers who are using Airflow.
• 19:25
  SPEAKER_01: That's your job, is figuring out what the community needs and helping them adopt Airflow easier and in better ways and for more use cases.
• 19:33
  SPEAKER_01: So I I do feel like your long wish list is the result of kind of compiling a lot of wish lists from most of the users.
• 19:40
  SPEAKER_01: So I'm also very curious to see what you came up with for number one on this list, given that I'm only giving you one or two minutes to answer this question.
• 19:47
  SPEAKER_03: Yeah, it's a list that's a combination of uh things that people tell me that I see in the wild, people say in webinar comments, codes that I see, and then of course also my own learnings because I use Airflow every day.
• 19:59
  SPEAKER_03: So whenever I think, oh, this could be different or this would be interesting, I also add it to the list.
• 20:03
  SPEAKER_03: And of course I love it if other people and me have the same wishes, then I add a little extra line on that and then count it as two.
• 20:10
  SPEAKER_03: I thought about this a while, and I think for now, one of the things that is something that's still quite a way out, so this is not something that's gonna be in 3.0 or even 3.1, but something that I have been thinking about ever since I started to use Airflow is if you come from a data science background or from a background of experimenting like I do, then one of the things that you want to do is explore your data and you really want to see your data and assess certain things about it.
• 20:42
  SPEAKER_03: So usually in the process when you develop a pipeline, you start in a notebook, especially if you come from a data science background, and then you set everything up there and you experiment and you have a lot of plots, you want to see the distribution of your data, you want to see if there are outliers and you visualize that, and you print a lot of heads of pandas tables because you need to know are my features aligned?
• 21:04
  SPEAKER_03: What do my columns look like?
• 21:05
  SPEAKER_03: And I've noticed that often now I actually start with the Aflow DAC because I'm so used to writing Airflow pipelines.
• 21:12
  SPEAKER_03: I have all my connections set up, and then I have my pipeline and I'm like in the middle, actually I would want to visualize something here.
• 21:19
  SPEAKER_03: And I usually then save the plot in the include folder like everyone.
• 21:24
  SPEAKER_03: But I kept thinking, if I could somehow see this in the UI and actually do this on a production deployment in the cloud, for example, if I have a a table and I have certain aggregation metrics, like how many rows per group is in that table for each run of the DAG, and I could display that in the UI and I could quickly check if something goes wrong, I can check that.
• 21:46
  SPEAKER_03: And I'm like, oh, yeah, like three groups only had two entries this time.
• 21:51
  SPEAKER_03: Of course the model failed.
• 21:53
  SPEAKER_03: I think that's a level of insight in your data and data awareness that would be super interesting to have one day.
• 21:58
  SPEAKER_03: And of course, like if I could display a plot like that I have in Mad Plotlib, and that just shows up in the UI or C Bond plot, that would be, I think, for data scientists, because with a visual brain trained to think of data visually, I think that would be an amazing feature.
• 22:14
  SPEAKER_03: So that's something that's on my wish list, and yes, I've already told the relevant developers.
• 22:20
  SPEAKER_01: Nice.
• 22:20
  SPEAKER_01: I love that.
• 22:21
  SPEAKER_01: That is a great request.
• 22:22
  SPEAKER_01: I do think that would be pretty game-changing in terms of a new feature.
• 22:26
  SPEAKER_01: So that's a good one to highlight.
• 22:28
  SPEAKER_01: Okay, well, Tamara, it's been so great talking to you.
• 22:31
  SPEAKER_01: I really appreciate you taking the time to continue to talk about the Airflow survey and state of Airflow.
• 22:37
  SPEAKER_01: We've been covering that quite a lot recently, but it is so impactful, and we really appreciate all of your work on it.
• 22:43
  SPEAKER_01: To close out, what is the best way for folks to get in touch with you?
• 22:46
  SPEAKER_03: So I think the best way is probably on LinkedIn.
• 22:49
  SPEAKER_03: Just DM me on LinkedIn or in the Airflow Slack.
• 22:52
  SPEAKER_03: That's one of the great places to ask Airflow questions, and I'm also in there.
• 22:57
  SPEAKER_03: So, yeah, LinkedIn or the Aflow Slack.
• 22:59
  SPEAKER_01: Awesome.
• 23:00
  SPEAKER_01: Sounds good.
• 23:00
  SPEAKER_01: Thanks again, Tamara.
• 23:02
  SPEAKER_01: Yeah, it's been great having you.
• 23:03
  SPEAKER_03: Yeah, thanks for having me on.
• 23:07
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:14
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 23:18
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five-star review.