Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / From Sensors to Datasets: Enhancing Airflow at Astronomer with Maggie Stark and Marion Azoulai

Transcript

From Sensors to Datasets: Enhancing Airflow at Astronomer with Maggie Stark and Marion Azoulai · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: How do we build this end-to-end visibility in a scalable way?
• 0:04
  SPEAKER_02: So we ended up deciding to build a control DAG, which basically would show us all of our DAGs in our deployment in a functional DAG.
• 0:11
  SPEAKER_02: But how do we build this in an automated, scalable way?
• 0:14
  SPEAKER_02: We were like Airflow has our lineage.
• 0:16
  SPEAKER_02: When we're naturally defining our dependencies in Airflow, we get this rendered DAG dependency graph that shows us how all of our DAGs are interconnected.
• 0:25
  SPEAKER_02: And so we actually leverage that API to create our control DAG.
• 0:33
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:43
  SPEAKER_03: Okay.
• 0:44
  SPEAKER_03: Hi everyone.
• 0:45
  SPEAKER_03: Welcome to the Data Flowcast.
• 0:46
  SPEAKER_03: I'm your host, Kenton Dannis, and today I'm joined by Maggie Stark and Marian Azule, who are both senior data scientists from astronomer.
• 0:55
  SPEAKER_03: I'm smiling a little bit more because my dog is in the background going nuts, who I'm sure you can't actually hear in the podcast.
• 1:01
  SPEAKER_03: A little distracting, but he'll calm down in a minute.
• 1:04
  SPEAKER_03: How are you both doing today?
• 1:05
  SPEAKER_03: Doing great.
• 1:06
  SPEAKER_03: Thanks, Kenton.
• 1:07
  SPEAKER_03: Your pets are relatively quiet, it seems.
• 1:12
  SPEAKER_03: So far, so been.
• 1:13
  SPEAKER_03: No complaints with the weather here.
• 1:14
  SPEAKER_03: It's a little cooler here.
• 1:16
  SPEAKER_03: Yeah.
• 1:16
  SPEAKER_03: Maggie and I are both in Seattle.
• 1:18
  SPEAKER_03: So we're getting a little bit of an early fall.
• 1:21
  SPEAKER_03: I just saw some map of the US and where like summer was going to be longer than normal or like shorter than normal.
• 1:27
  SPEAKER_03: And it was all like longer than normal, except for literally this little blit over Seattle is what's totally ridiculous to us here.
• 1:36
  SPEAKER_01: Yeah, in my case, my first summer in Barcelona, so it's been like scorching hot.
• 1:43
  SPEAKER_01: I'm still trying to get used to it.
• 1:45
  SPEAKER_01: Yeah.
• 1:45
  SPEAKER_01: I forgot you just moved there.
• 1:47
  SPEAKER_01: They have really nice falls though.
• 1:49
  SPEAKER_01: That's the thing.
• 1:49
  SPEAKER_01: Very nice full winter, spring, everything but summer.
• 1:53
  SPEAKER_01: You're almost through it then.
• 1:55
  SPEAKER_03: You're almost through it.
• 1:56
  SPEAKER_03: Hopefully, by the time this comes out, you'll be in a little bit of a better state.
• 2:00
  SPEAKER_03: Yeah.
• 2:01
  SPEAKER_03: All right.
• 2:01
  SPEAKER_03: Let's go ahead and dive in.
• 2:03
  SPEAKER_03: So for our listeners for today's podcast, we're gonna discuss how Astronomers' data team recently re-architected how they use Airflow to improve reliability and efficiency.
• 2:13
  SPEAKER_03: So to start, let's talk about what you guys do and what your team does.
• 2:18
  SPEAKER_01: Sure.
• 2:19
  SPEAKER_01: Our team, so we're part of the data team, Magina, and it functions as a centralized data hub for the entire organization.
• 2:28
  SPEAKER_01: We work with all types of stakeholders within the company.
• 2:32
  SPEAKER_01: So we work basically with all the departments, sales, customer success, finance, HR, products, obviously, all the departments, as well as the leadership team.
• 2:43
  SPEAKER_01: And our core mission revolves around constructing and maintaining a core pipeline both for operational and analytical work.
• 2:52
  SPEAKER_01: And we're primarily using Airflow for that.
• 2:54
  SPEAKER_01: But we provide all types of dashboards, monitoring tools, strategic analysis, and other data products to all of our stakeholders.
• 3:02
  SPEAKER_01: But we are also deeply involved in the product in different ways.
• 3:08
  SPEAKER_01: One of those is that since we are our own internal customer in a sense, we experience the product firsthand.
• 3:16
  SPEAKER_01: And we get, for example, to identify some product gaps that we will relate to the product team.
• 3:23
  SPEAKER_01: We provide also, of course, a lot of analytics on the product, such as feature adoption, product usage, and other things.
• 3:31
  SPEAKER_01: And I guess in some ways we can also inspire the product with new features, with the way that, and I'm sure Maggie will touch upon that in a bit.
• 3:40
  SPEAKER_01: So we can also influence the product by inspiring the product team with some of the features that we have built in our own airflow architecture.
• 3:50
  SPEAKER_01: And of course, we also dock food a lot of the new features that the product team puts together both for Airflow and Astro.
• 3:59
  SPEAKER_01: And I would say that ultimately our role is really to put the data at the center of the company and to make it as data-driven as possible.
• 4:07
  SPEAKER_01: And that's what we do using Airflow, basically.
• 4:10
  SPEAKER_03: That was a great description of how your team sits at this really unique space.
• 4:15
  SPEAKER_03: I think for our listeners who might not be aware, Astronomer both helps manage the open source Airflow project and then also provides a managed Airflow service.
• 4:24
  SPEAKER_03: That's the Astro that you just heard Marion speaking about to our customers.
• 4:28
  SPEAKER_03: So your team is both providing all of the data to the internal teams that Astronomer needs to keep our business running.
• 4:35
  SPEAKER_03: And you're doing that using Airflow and also using our own products.
• 4:39
  SPEAKER_03: So you have maybe the kind of first row seat to our product development and are able to provide a lot of feedback there, which I know is really valuable.
• 4:47
  SPEAKER_01: Actually, we even built embedded dashboards using data that we produce using Airflow.
• 4:55
  SPEAKER_01: We built a dashboard that is embedded in the Astro product for our customers to see their usage and other important data points.
• 5:04
  SPEAKER_03: Yeah, contributing to the product as well, while you're using it.
• 5:08
  SPEAKER_03: Yeah, that's great.
• 5:09
  SPEAKER_03: And you mentioned all of this is built on airflow for your team.
• 5:13
  SPEAKER_03: Can you talk a little bit more about what your airflow usage looks like holistically?
• 5:18
  SPEAKER_01: Yeah, so Airflow is really the backbone of everything we do.
• 5:22
  SPEAKER_01: So we use it for ingesting data, for building ETL pipelines, machine learning pipelines for delivery, feeding into dashboards, sending emails, alerts, reverse ETLing data into some of our source tools, for example.
• 5:38
  SPEAKER_01: And right now we have mainly two deployments, one for production, one for development, which have about a hundred DAGs each.
• 5:47
  SPEAKER_01: And can you remind me, Maggie, how many tasks we're running on a daily basis?
• 5:53
  SPEAKER_02: Roughly at 30,000 tasks in environment.
• 5:57
  SPEAKER_02: So 60,000 total.
• 5:59
  SPEAKER_02: Yeah.
• 5:59
  SPEAKER_03: So you're not just a customer of astronomer while you're supporting the business.
• 6:03
  SPEAKER_03: You're a pretty big one.
• 6:05
  SPEAKER_02: We are our biggest internal customer, but I think someone else has overtaken us.
• 6:09
  SPEAKER_03: Yeah, that's great.
• 6:09
  SPEAKER_03: And as a promise, we wanted to talk today about this kind of re-architecture you recently went through.
• 6:16
  SPEAKER_03: So that was redesigning how you use Airflow and the DAGs within it.
• 6:21
  SPEAKER_03: So can you talk first a little bit about your original design and then what prompted you to make a change at this stage?
• 6:27
  SPEAKER_03: Yeah.
• 6:28
  SPEAKER_02: So when we first were implementing Airflow, we had already been building a lot of analytics and delivering things to other teams.
• 6:36
  SPEAKER_02: And so we had this backlog of stuff that we just needed to operationalize and orchestrate.
• 6:42
  SPEAKER_02: And so the obvious tool is airflow, but we really needed to focus on raining up quickly and making sure that it was easy for people to onboard.
• 6:51
  SPEAKER_02: Um, but we also wanted to have good governance and make sure that we were following best practices.
• 6:57
  SPEAKER_02: So we started with a DAG factory, and the benefits of that was that it abstracted airflow from the user.
• 7:04
  SPEAKER_02: And it also allowed people to stay in the context that they were familiar.
• 7:08
  SPEAKER_02: So they could write in SQL or R or YAML or Jupyter notebooks.
• 7:13
  SPEAKER_02: And then the other thing that we really focused on was making reusable code.
• 7:17
  SPEAKER_02: So we focused on, I know a lot of people do custom operators.
• 7:20
  SPEAKER_02: We absolutely have focused on doing custom task groups.
• 7:23
  SPEAKER_02: And so within that, we have a series of tasks that are all part of common operations.
• 7:29
  SPEAKER_02: So we have a create table task group that creates the table, tests the table, swaps it into production, documents the table, does the primary keys, and that's that.
• 7:39
  SPEAKER_02: So really the user is writing their normal SQL and then with a few lines of configuration, can add their testing and documentation.
• 7:47
  SPEAKER_02: So that was really important to us to one, create like a contract mechanism for production suitability, but also keep us up to date on documentation and testing, because I think that's something that's easy to fall behind if it's not easy and there at the beginning.
• 8:01
  SPEAKER_02: So those kind of things really allowed us to scale up to where we are now, which is like around 30,000 tasks a day.
• 8:09
  SPEAKER_02: But with that task growth also came a lot of sensor growth because all of our DAG dependencies were specified with sensors.
• 8:17
  SPEAKER_02: And we should also clarify we're a team of four managing this.
• 8:20
  SPEAKER_02: I'll let Marion talk about what happened with the sensors.
• 8:23
  SPEAKER_03: Go ahead and talk about the sensors.
• 8:25
  SPEAKER_03: I mean, that all makes sense, Maggie.
• 8:27
  SPEAKER_03: I think we see a lot of teams go down a similar route, especially with the DAG factory, where by that you mean some sort of framework that like automatically generates DAGs for you based on a configuration you provide.
• 8:40
  SPEAKER_03: Yeah, the sensors part is interesting.
• 8:43
  SPEAKER_03: I'd imagine there are a lot of different interdependencies between the DAGs that you're working with.
• 8:48
  SPEAKER_03: So yeah, Marion, tell us a little bit more about what's happening with the sensors there.
• 8:53
  SPEAKER_01: So I guess I was in the salt line to experience the issues that all those sensors brought in because I'm based in Europe.
• 9:02
  SPEAKER_01: So I'm like nine hours ahead of time from many.
• 9:06
  SPEAKER_03: First person who had first responder exactly.
• 9:09
  SPEAKER_01: And so I tend like when I wake up in the morning, I tend to first thing I do, I go to our Astro and Earthflow instances and I look at the deployment.
• 9:18
  SPEAKER_01: And after some time of our initial pipeline working well, as we started building upon it, I started seeing more and more failures.
• 9:27
  SPEAKER_01: And it was clearly sensor related because having a sensor-heavy architecture leads can lead to a lot of problems.
• 9:35
  SPEAKER_01: One of those is that it makes the deployments a bit more uh fragile in a sense, a bit less robust.
• 9:42
  SPEAKER_01: The whole deployment with DAGs, the pipeline with DAGs depending on each other can create, for example, cascading failure.
• 9:50
  SPEAKER_01: So we have one DAG that's gonna fail, it's gonna trigger like all the downstream DAGs to fail.
• 9:55
  SPEAKER_01: Then you hit resource issues.
• 9:57
  SPEAKER_01: So of course you increased a lot parallel tasks, for example.
• 10:01
  SPEAKER_01: Like we had literally like all of our DAGs.
• 10:03
  SPEAKER_01: And remember, we had maybe we didn't have as many DAGs as today, but we had you know several dozens of DAGs, and they all had multiple sensors that they were depending on to be to start.
• 10:14
  SPEAKER_01: And so every hour you had dozens, dozens, and dozens of sensors running at the same time.
• 10:21
  SPEAKER_01: So we started hitting a lot of resource issues, uh sensor timeout issues.
• 10:25
  SPEAKER_01: We had some DAGs that would systematically fail because they could not have complete in time.
• 10:31
  SPEAKER_01: And it would cause all the downstream DAGs to fail for the same reason.
• 10:35
  SPEAKER_01: And also, so we had all those failures, and it was also much harder to investigate the root cause of failure.
• 10:40
  SPEAKER_01: So we had some that were like daily, we knew it was just happening, but then an actual legit failure was harder to track down because it would cause all your pipeline to go red.
• 10:50
  SPEAKER_01: So that was like time consuming and hard to get to the root cause.
• 10:54
  SPEAKER_01: Then once you had found and solved the bugs, you had to replay the pipeline in the right order.
• 11:00
  SPEAKER_01: So it was also a complex task to make sure like all the DAGs are run in the right sequence so that the data gets generated like the way it should be.
• 11:08
  SPEAKER_01: And in addition to that, scaling was more difficult, as I said.
• 11:12
  SPEAKER_01: We would hit like resource issues, it was very compute intensive.
• 11:16
  SPEAKER_01: And like scaling in terms of development efforts was also more complicated.
• 11:21
  SPEAKER_01: Like when you wanted to add like a DAG in the middle of your pipeline, you had to make sure to specify all the right dependencies and whatnot.
• 11:27
  SPEAKER_01: So it was a bit of a nightmare, I would say, in the end.
• 11:31
  SPEAKER_03: So you have a situation where you have many tasks just sitting around waiting, which is what sensors do, obviously, um, which works great until you have a lot of dependencies based on them.
• 11:42
  SPEAKER_03: So you have pipelines that can't run.
• 11:44
  SPEAKER_03: Obviously, you don't want to have a sensor without a timeout for resource issues, but if they time out too early or something, then you've got these failures.
• 11:52
  SPEAKER_03: And then it's really hard to track what actually happened after the fact.
• 11:57
  SPEAKER_03: I can imagine that is not a great thing to wake up to in Europe's morning, especially regularly.
• 12:02
  SPEAKER_03: So tell me a little bit about what you did to fix it.
• 12:06
  SPEAKER_02: So when we went into this re-architecture, we really need to figure out the scaling and the re reliability issues.
• 12:12
  SPEAKER_02: Like, how do we address the sensor problem?
• 12:14
  SPEAKER_02: The obvious sort of move was to move to data sets, take out the sensors, swap in data set dependencies.
• 12:20
  SPEAKER_02: When we're doing started working with data sets, we noticed now we have this visibility issue in our end-to-end pipeline.
• 12:27
  SPEAKER_02: Because when a data set fails, you don't really see what you're missing below it, like what didn't trigger.
• 12:33
  SPEAKER_02: And that was a benefit we had of sensors because it blew up all the DAGs below it.
• 12:37
  SPEAKER_02: But the nice thing is when you've resolved the data set, it then naturally triggers everything below it.
• 12:42
  SPEAKER_02: So we don't have the full visibility when something, when there is a failure, but getting things stacked online quicker data sets do that.
• 12:50
  SPEAKER_02: But then how do we build this end-to-end visibility in a scalable way?
• 12:55
  SPEAKER_02: So we ended up deciding to build a control DAG, which basically would show us all of our DAGs in our deployment in a functional DAG.
• 13:02
  SPEAKER_02: And but how do we build this in an automated, scalable way?
• 13:05
  SPEAKER_02: We were like Airflow has our lineage.
• 13:07
  SPEAKER_02: When we're naturally defining our dependencies in Airflow, we get this rendered DAG dependency graph that shows us how all of our DAGs are interconnected.
• 13:16
  SPEAKER_02: And so we actually leverage that API to create our control DAG.
• 13:21
  SPEAKER_02: And so every node in our DAG dependency graph is actually a functional task that is sensing where we are in our end-to-end pipeline.
• 13:29
  SPEAKER_02: So for every run in our airflow instance, we see what's currently running, what's failed, what's passed, what got skipped because it's not in that schedule interval.
• 13:38
  SPEAKER_03: Interesting.
• 13:39
  SPEAKER_03: Okay, that makes sense.
• 13:40
  SPEAKER_03: So you're essentially solving for the fact that airflow right now isn't super well set up with the data sets feature to tell you when something doesn't happen.
• 13:49
  SPEAKER_03: So you can see the graph of visibility, but there's nothing to actually tell you proactively, hey, this didn't run.
• 13:58
  SPEAKER_03: So you have a separate DAG that does that.
• 14:01
  SPEAKER_02: Exactly.
• 14:02
  SPEAKER_02: Our end data product might have an SLA on it, but if something upstream of it fails, I don't necessarily get alerted that I'm gonna miss that SLA until we've created this sort of way of seeing our pipelines in a holistic view.
• 14:15
  SPEAKER_03: Interesting.
• 14:16
  SPEAKER_03: And does that control DAG just do alerting and monitoring?
• 14:21
  SPEAKER_03: So it say sends you an alert if something doesn't run, or is it providing more observability than so for every so it's on an hourly schedule?
• 14:29
  SPEAKER_02: That's our shortest refresh schedule for our data, basically.
• 14:32
  SPEAKER_02: And so for every hour, we see every single DAG within it and whether it's passed or failed, and every data set that's also getting produced, and it's a dependency of other DAGs.
• 14:42
  SPEAKER_02: So you can see, oh, this task failed.
• 14:45
  SPEAKER_02: So these downstream things won't get run.
• 14:48
  SPEAKER_02: And so you get the holistic view of everything, every hour that it's running and what failed and what succeeded.
• 14:55
  SPEAKER_01: Yeah, and the nice and the nice thing is that you get you get the root cause fail of the failure for free.
• 15:00
  SPEAKER_01: Like you just look at more than the graph, you look at the the list of tasks, which are really DAGs inside the list view and you can see the first failure and everything that that that trickle down from from it and then also solved.
• 15:15
  SPEAKER_02: Yeah.
• 15:16
  SPEAKER_02: And from that you can click into the error and go to the actual DAG that failed.
• 15:19
  SPEAKER_02: So that when you go to clear the data set and get it rerunning, you're already there.
• 15:23
  SPEAKER_02: So fixing it is one click away.
• 15:25
  SPEAKER_03: Okay.
• 15:26
  SPEAKER_03: And you fix just that one failure and then everything will run.
• 15:29
  SPEAKER_03: Yeah, that sounds much nicer than what you originally described.
• 15:34
  SPEAKER_03: And so talk a little bit how that helped with your scaling problem.
• 15:38
  SPEAKER_03: So I'd imagine the architecture you're describing isn't really limited by scale in the same way beyond the size of your control DAG, which I'd imagine is not particularly.
• 15:47
  SPEAKER_02: Yeah, I'll be curious when we hit some limit of using that DAG API to auto-generate a DAG.
• 15:52
  SPEAKER_02: But yeah, the scaling-wise, it actually because we have, I think it was crucial that we had a way of auto-creating this control DAG.
• 16:00
  SPEAKER_02: And so that when users are creating a new DAG, it's auto-monitored by the control DAG.
• 16:06
  SPEAKER_02: Um, I think if we had to maintain that separately, it would have gotten out of here that obviously not been scalable and sustainable.
• 16:12
  SPEAKER_02: I think the other thing that it's done for us is because we have this N10 visibility and because we are a small team.
• 16:19
  SPEAKER_02: Previously, our we used to create large DAGs with a lot of tasks in them because it was easier to main monitor 10, 10 DAGs.
• 16:27
  SPEAKER_02: We had more than 10 DAGs, but as an example, 10 DAGs.
• 16:31
  SPEAKER_02: But now because we know with the control DAG, we can see everything, nothing's getting missed.
• 16:36
  SPEAKER_02: The incentive is actually to break down our DAGs into much smaller units of work.
• 16:41
  SPEAKER_02: And so that the dependencies in each DAG are really only related to that area of focus and what's needed to produce that output, which really minimizes also the failures that we're seeing in our pipelines.
• 16:53
  SPEAKER_02: Previously, we have something upstream fail and it ends up taking a large portion of our pipelines down, just so that we could manage DAGs individually.
• 17:01
  SPEAKER_02: But now that we're not really managing DAGs individually, the idea is let's minimize failures that only make satisfying the true dependencies because not DAG.
• 17:11
  SPEAKER_03: Yeah, that makes sense.
• 17:12
  SPEAKER_03: That's one of the original kind of stated benefits of data sets is being able to have smaller pipelines with those visible dependencies.
• 17:20
  SPEAKER_03: I think you mentioned at the beginning, so you've got 30,000 tasks running every day, but only 100 or a couple of hundred DAGs.
• 17:27
  SPEAKER_03: That's a lot of tasks in every DAG, which are just they're unwieldy, as you said.
• 17:32
  SPEAKER_03: It's really hard to figure out just when something goes wrong or even look at them.
• 17:36
  SPEAKER_03: I think we've seen people even run into issues with just bumping up against kind of what the Airflow UI can manage in terms of like loading DAGs, which is not the scalability issue I think anybody wants to be dealing with.
• 17:49
  SPEAKER_03: And you have to get pretty big to hit that.
• 17:51
  SPEAKER_03: But obviously, having like much smaller numbers of tasks in each one, I think is a big benefit for a lot of teams.
• 17:59
  SPEAKER_03: Yeah, that's great.
• 18:00
  SPEAKER_03: And so now that you've had this in place for a little while now, do you feel that it's solved all of your original issues?
• 18:08
  SPEAKER_03: Marian, are you waking up to all green squares now?
• 18:12
  SPEAKER_01: Yeah, basically, that's what I'm wearing gray today as a celebration.
• 18:16
  SPEAKER_01: But yeah, now when there is a failure, it's a legit failure.
• 18:19
  SPEAKER_01: It's it's not a timeout, it's not a concurrency issue or resource issue.
• 18:25
  SPEAKER_01: It's like we messed something up.
• 18:28
  SPEAKER_01: Or more likely it's like the cell team.
• 18:32
  SPEAKER_01: No, it's like more like the sales team didn't enter the right data in Salesforce.
• 18:36
  SPEAKER_01: And so it broke our pipeline.
• 18:38
  SPEAKER_03: I was gonna blame the third party.
• 18:40
  SPEAKER_03: Yeah, you can blame the internal team.
• 18:42
  SPEAKER_03: Yeah, always blame internal team.
• 18:44
  SPEAKER_03: I'm joking, obviously.
• 18:45
  SPEAKER_03: Right.
• 18:46
  SPEAKER_03: Yeah, that's great.
• 18:47
  SPEAKER_03: And so now that you've lived with it for a little while and experienced those benefits, do you foresee any additional changes or kind of optimizations in the future?
• 18:56
  SPEAKER_03: What are you looking forward to now?
• 18:58
  SPEAKER_02: Yeah, so this has been in place for four months now, and we uh we're actually looking at the data this week, trying to put together our summit presentation, and we could measure our failure improved, failure rate improvement go from around 16% down to 3%.
• 19:12
  SPEAKER_02: There is absolute evidence to back up that our lives are easier now, and that's great.
• 19:17
  SPEAKER_02: So I we attribute that to a lot of moving to data set scheduling and then being able to break down our work into smaller units, the micro pipelines.
• 19:26
  SPEAKER_02: We're also seeing the total end-to-end pipeline DAG duration go down by around 35%.
• 19:33
  SPEAKER_02: So our tasks are also completing.
• 19:35
  SPEAKER_02: We're getting through the whole pipeline much faster now, too.
• 19:38
  SPEAKER_02: So, in terms of like improving this, we don't have a lot of active work to improve it.
• 19:43
  SPEAKER_02: We're focusing on other projects now because we don't have to worry about our pipelines.
• 19:47
  SPEAKER_02: But as new products and features are released in Airflow, I think we're always going to be looking for ways to incorporate them now into this control DAG.
• 19:55
  SPEAKER_02: And one of the things that's been on our list of maybe adding is data set listeners.
• 19:59
  SPEAKER_03: Got it.
• 19:59
  SPEAKER_03: Makes sense.
• 20:00
  SPEAKER_03: That's a very, very successful implementation.
• 20:03
  SPEAKER_03: Cool to see here or hear about those improvements.
• 20:06
  SPEAKER_03: That leads in pretty well to my last question, which I ask all of our guests.
• 20:11
  SPEAKER_03: And that is what would you most like to see from development of the Airflow project in the near future?
• 20:16
  SPEAKER_03: And I'll let you both answer.
• 20:18
  SPEAKER_02: Dad, I think the conversations within our company and also conversations with customers has really highlighted the that there is need of more lineage inside of the product and something more native.
• 20:30
  SPEAKER_02: So we'll actually see some announcements coming up about features being within Astro related to lineage and making a control light DAG available to customers so that they have the same visibility.
• 20:43
  SPEAKER_01: I second that.
• 20:44
  SPEAKER_01: When we monitor our pipeline, we like to make it clean.
• 20:48
  SPEAKER_01: So let's say we had some issues, some errors.
• 20:50
  SPEAKER_01: Once the failure is solved, we turn all the failed tasks to green to successful, right?
• 20:56
  SPEAKER_01: Just so that you wake up the next day and you make sure that everything is green.
• 21:00
  SPEAKER_01: And sometimes it's a bit time consuming to clean the failures.
• 21:04
  SPEAKER_01: Let's say we have a sub-hourly bag so that runs a lot and it failed a few times for a few days and we want to clean it.
• 21:12
  SPEAKER_01: You have like more than a hundred, more than a hundred failed paths.
• 21:16
  SPEAKER_01: You need to go through different pages of making it green.
• 21:20
  SPEAKER_01: It's silly, but I would like that to be solved.
• 21:23
  SPEAKER_03: Okay, so an easier way to clean to clean up the UI.
• 21:28
  SPEAKER_03: I always joke that our job at Astronomer is making the red squares turn green.
• 21:33
  SPEAKER_03: An easier way to make the red squares turn green.
• 21:36
  SPEAKER_03: That's actually a great request for the open source project.
• 21:39
  SPEAKER_03: I think you're probably not the first person who has identified something like that.
• 21:43
  SPEAKER_03: Yeah.
• 21:44
  SPEAKER_03: So more automation in the UI.
• 21:46
  SPEAKER_03: Cool.
• 21:47
  SPEAKER_03: I love it.
• 21:47
  SPEAKER_03: Well, thank you both so much for coming on.
• 21:50
  SPEAKER_03: This has been really cool to hear about what you've implemented, and we appreciate your time.
• 21:55
  SPEAKER_03: What's the best way for folks to get in touch with you?
• 21:57
  SPEAKER_03: You can find us on LinkedIn.
• 21:59
  SPEAKER_03: Okay.
• 22:00
  SPEAKER_03: Great.
• 22:01
  SPEAKER_03: Thanks again.
• 22:02
  SPEAKER_03: And I hope to see you both at Airflow Summit soon.
• 22:06
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering Airflow for data engineering and AI.
• 22:13
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 22:17
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.