Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty

Transcript

How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: And we realized, like, wait a minute, can't we just do this in Airflow?
• 0:03
  SPEAKER_01: Can't we just do this on Astro and have Airflow monitor other airflows?
• 0:07
  SPEAKER_01: And you absolutely can.
• 0:08
  SPEAKER_01: It's a great tool.
• 0:09
  SPEAKER_01: And also, of course, we get to actually become users of Airflow again, which many members of my team were helping other people use Airflow every day.
• 0:17
  SPEAKER_01: But we can get away from writing our own DAGs, doing our own work.
• 0:21
  SPEAKER_01: And this tool gives us the ability to get back in and do it again and experience it for ourselves.
• 0:26
  SPEAKER_01: And it was great.
• 0:27
  SPEAKER_01: And it's not an ETL pipeline at all.
• 0:35
  SPEAKER_02: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:45
  SPEAKER_00: Hey Airflow community, we've got exciting news.
• 0:48
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:51
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow and grab a free Airflow 3 fundamentals or DAG Authoring certification worth$150 each.
• 1:01
  SPEAKER_00: Plus, you can enter to win an exclusive workshop with Mark Lumberti.
• 1:05
  SPEAKER_00: Don't miss out.
• 1:07
  SPEAKER_00: The survey link is in the show notes and closes November 21st.
• 1:13
  SPEAKER_00: Hi everyone.
• 1:14
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:16
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Colin McNulty, who is a senior director of global support at Astronomer.
• 1:24
  SPEAKER_00: Colin, it's so good to chat with you.
• 1:25
  SPEAKER_00: How are you?
• 1:26
  SPEAKER_01: It's great to be on here, Ken.
• 1:28
  SPEAKER_00: Awesome.
• 1:28
  SPEAKER_00: Well, I am very excited to have you here.
• 1:30
  SPEAKER_00: I think our listeners are going to be really interested in the topic that we have today.
• 1:35
  SPEAKER_00: We're going to discuss some of the common issues that folks run into with airflow.
• 1:39
  SPEAKER_00: Always a very hot topic.
• 1:40
  SPEAKER_00: And this will be based on the experience of Astronomer CRE team.
• 1:44
  SPEAKER_00: So they have probably seen it all.
• 1:46
  SPEAKER_00: But I want to start before we dive into all of that with some background on you.
• 1:49
  SPEAKER_00: So maybe tell me how you first got started with Airflow and then how that led you to your current role.
• 1:55
  SPEAKER_01: My background is actually in mechanical engineering.
• 1:58
  SPEAKER_01: And I worked for GE for six years.
• 2:01
  SPEAKER_01: And as part of that, during their digital transformation, we were moving to use Airflow as part of a really big their monitoring and diagnostics program.
• 2:11
  SPEAKER_01: And it was this really big transition.
• 2:13
  SPEAKER_01: But what was so cool about Airflow is that even though we were using it in this really big way, because it was open source, me and some collaborators were like, actually, we have these other use cases for it around verification, like data verification, that we can run it in this really small way and try it out and become familiar with the tool and use it there.
• 2:33
  SPEAKER_01: And so that was my first introduction.
• 2:35
  SPEAKER_01: We were on running Airflow like 110, something like that.
• 2:39
  SPEAKER_01: And got really interested in data engineering and trying to follow the problems of you know, what do we need to work on?
• 2:45
  SPEAKER_01: What's the biggest problem facing the business or the organization?
• 2:49
  SPEAKER_01: And that led me further and further into data engineering and then into using Airflow.
• 2:53
  SPEAKER_01: And at my next employer, I did our Airflow 2 transition, as well as moving from Docker Compose to a Kubernetes-based setup of Airflow.
• 3:04
  SPEAKER_01: And that was a really interesting project.
• 3:07
  SPEAKER_01: And then I was following the Airflow 2 transition, which of course Astronomer was very involved in and really leading a lot of that work.
• 3:14
  SPEAKER_01: So I was following the company on GitHub.
• 3:16
  SPEAKER_01: And then I got a call to going to be interested in working with Astronomer.
• 3:21
  SPEAKER_01: And I was really excited.
• 3:22
  SPEAKER_01: I was really excited to be able to do that.
• 3:23
  SPEAKER_01: So yeah, that was my background that led me here and starting in the front line of our customer reliability engineering CRE team, which is our support team.
• 3:32
  SPEAKER_01: And over the last four years, working my way through to being the director.
• 3:36
  SPEAKER_00: There are two things that I love about this.
• 3:38
  SPEAKER_00: One is that you also have a traditional like engineering background.
• 3:42
  SPEAKER_00: This is my own background as well.
• 3:43
  SPEAKER_00: So I always love about hearing people who were engineers and then transitioned into like data and tech.
• 3:48
  SPEAKER_00: I think it's a natural transition, but I always get lots of questions about that for my personal self.
• 3:54
  SPEAKER_00: The other thing I love about that is you touched on working with Airflow in what I might consider a non-traditional use case.
• 4:00
  SPEAKER_00: Or I think maybe not a lot of people think about it in that way since it's so heavily used as an orchestrator for ETL pipelines, which makes sense and is the bread and butter, but it's also used for a lot of other things that you would have to orchestrate or schedule, including in traditional engineering.
• 4:16
  SPEAKER_01: I remember before we found Airflow asking, how can we schedule these tasks?
• 4:21
  SPEAKER_01: We run these tasks every day.
• 4:22
  SPEAKER_01: They're not very exciting reports, but reports that took up time and we had to do them.
• 4:27
  SPEAKER_01: They were just important analyses of are we doing this right?
• 4:32
  SPEAKER_01: Are we going to be sending the right safety reports to our customers?
• 4:36
  SPEAKER_01: Are we going to be telling they were power plant operators?
• 4:38
  SPEAKER_01: Are we going to be helping SIM manage their operations correctly?
• 4:42
  SPEAKER_01: And we need to do it every day.
• 4:44
  SPEAKER_01: And Airflow is an awesome tool for just anything that you need to do that way.
• 4:49
  SPEAKER_00: I personally, for astronomers, DevRel team have been doing a little bit of this myself recently of just trying to automate some of the stuff that we do internally for just general process stuff all the time and realizing, like, oh, this is a thing we do every month.
• 5:03
  SPEAKER_00: Airflow can do that for us.
• 5:05
  SPEAKER_01: Actually, we had a similar conversation within our CRE team where we wanted to get into something we do a lot now, which is proactive monitoring.
• 5:13
  SPEAKER_01: This was a couple of years ago, where we can monitor people's airflow instances for known problematic conditions and alert them ahead of time instead of waiting for them to notice or our customers to notice and then about us.
• 5:25
  SPEAKER_01: And we're looking at these other monitoring tools.
• 5:28
  SPEAKER_01: Like, oh, we need a monitoring tool.
• 5:30
  SPEAKER_01: And we realized wait a minute, can't we just do this in Airflow?
• 5:33
  SPEAKER_01: Can't we just do this on Astro and have Airflow monitor other airflows?
• 5:37
  SPEAKER_01: And you absolutely can.
• 5:38
  SPEAKER_01: It's a great tool.
• 5:39
  SPEAKER_01: And also, of course, we get to actually become users of Airflow again, which many members of my team were helping other people use Airflow every day.
• 5:47
  SPEAKER_01: But we can get away from writing our own DAGs, doing our own work.
• 5:51
  SPEAKER_01: And this tool gives us the ability to get back in and do it again and experience it for ourselves.
• 5:56
  SPEAKER_01: And it was great.
• 5:57
  SPEAKER_01: And it's not an ETL pipeline at all.
• 6:00
  SPEAKER_00: I think that leads really well into my next question, which I would love to dig in a little bit more to how astronomers CRE or customer reliability engineering, as you said, team works and is structured, and then how you use Airflow yourselves.
• 6:13
  SPEAKER_01: Absolutely.
• 6:14
  SPEAKER_01: So we are astronomer's support team.
• 6:15
  SPEAKER_01: If you go to support.astronomer.io, like that's us.
• 6:19
  SPEAKER_01: And so astronomer customers will send us questions about all kinds of things.
• 6:24
  SPEAKER_01: Our software product, our Astro, of course.
• 6:28
  SPEAKER_01: And what's unique about us is that we support, of course, our own infrastructure and platform and how do I do this or that in Astro?
• 6:37
  SPEAKER_01: But we also have a team, and this is my background where I came from, of helping you with Airflow with your DAGs.
• 6:43
  SPEAKER_01: So we focus on what we call like break fix support.
• 6:46
  SPEAKER_01: So something broke and we need to fix it, or what is the right answer to this?
• 6:50
  SPEAKER_01: Like, how do I do this particular thing in Airflow?
• 6:53
  SPEAKER_01: How do I have the right syntax or the right?
• 6:55
  SPEAKER_01: We have a lot of people with a lot of configuration.
• 6:58
  SPEAKER_01: Oh, can I do this?
• 6:58
  SPEAKER_01: Yes, you can do that with this configuration or with these DAG settings or with these best practices and answer their questions and help them get into back in business as usual.
• 7:08
  SPEAKER_01: We get all kinds of questions like I want to run DAGs, I want to run this task only after this, even if this task is skipped or even if it's upstream is like this.
• 7:16
  SPEAKER_01: Oh, okay.
• 7:16
  SPEAKER_01: Have you looked at trigger rules?
• 7:18
  SPEAKER_01: So you can use none failed instead of all success.
• 7:21
  SPEAKER_01: You can maybe you want to look at different timetables.
• 7:24
  SPEAKER_01: Most of the time, the default timetable gets you really far.
• 7:27
  SPEAKER_01: But then sometimes we have customers ask us, why can't I do this?
• 7:30
  SPEAKER_01: Oh, you can.
• 7:30
  SPEAKER_01: You just have to use a different timetable to achieve that same goal.
• 7:34
  SPEAKER_01: All kinds of questions like that.
• 7:35
  SPEAKER_01: But yeah, we answer questions and sometimes get on calls with customers and walk them through whatever we need to do to find the problem.
• 7:41
  SPEAKER_00: That's great.
• 7:42
  SPEAKER_00: And I think such a breadth of things you have to cover in everything that you just described because Airflow is so flexible.
• 7:48
  SPEAKER_00: Before I do definitely want to dig into common issues, but before we get into that, tell me a little bit more about how your team uses Airflow yourselves.
• 7:57
  SPEAKER_00: You led into this a little bit earlier with the predictive, trying to predict what might be going wrong in a customer's Airflow deployment before it does.
• 8:04
  SPEAKER_00: Is that your main use case for Airflow?
• 8:06
  SPEAKER_01: That's the main one, but we do have others.
• 8:08
  SPEAKER_01: Maybe the simplest is that when we have internal targets for how fast we want to respond to customer questions.
• 8:16
  SPEAKER_01: And if we're having a ticket close to that internal target, we want to inform us.
• 8:21
  SPEAKER_01: And our the software we use for running support does not have that inbuilt at the level of fine-grainedness that we need.
• 8:28
  SPEAKER_01: And so we build an airflow.
• 8:30
  SPEAKER_01: We monitor what tickets are there, determine if they're within a certain timeframe of our target, and inform the person who's assigned to it or the group that needs to pull someone in to become assigned to that ticket and get a response out.
• 8:43
  SPEAKER_01: So that's very straightforward, but extremely useful.
• 8:46
  SPEAKER_00: And another non-ETL use case.
• 8:48
  SPEAKER_00: Exactly.
• 8:49
  SPEAKER_00: Managing tickets.
• 8:50
  SPEAKER_00: Yeah.
• 8:51
  SPEAKER_01: We also are, like I said, we're doing the proactive monitoring.
• 8:55
  SPEAKER_01: So we determine really what we do is we watch what do I do as a person?
• 9:00
  SPEAKER_01: When I get an issue that kind of looks like this, I check my Prometheus metrics and I look for this pattern.
• 9:06
  SPEAKER_01: I look for this log in our logging tool, which is Splunk, or I'm looking at something in the Airflow metadata database, like some specific condition there.
• 9:14
  SPEAKER_01: And those are all tools that we have as human team members, and we give in those same permissions with appropriate restrictions to our Airflow instance.
• 9:24
  SPEAKER_01: And so we can say, I check for this log and then I check for these metrics and I check for this condition.
• 9:29
  SPEAKER_01: And if those things line up, then I'm going to actually create a ticket.
• 9:33
  SPEAKER_01: So we use our ticketing systems API to say, hey, this is a ticket as if a customer had raised it to us, but we're raising it to ourselves to go check it out.
• 9:42
  SPEAKER_01: And we populate it with all the relevant information on, hey, I saw this, I checked that.
• 9:48
  SPEAKER_01: And these are the conditions that lead me to believe there might be an issue.
• 9:51
  SPEAKER_01: And here's the next steps that you might want to do to verify that there's a real condition that the customer needs to become aware of.
• 9:58
  SPEAKER_00: How often are you able to look at the results of that and intervene before it actually ever becomes an issue for the customer?
• 10:05
  SPEAKER_01: Every day there's somebody that we're helping that way.
• 10:07
  SPEAKER_00: Wow, that's amazing.
• 10:09
  SPEAKER_00: I'm sure that saves you so much time too on the back end and obviously makes it life a lot easier for the customers who never even hit the issue, but usually easier to intervene proactively than react.
• 10:20
  SPEAKER_01: It's a much better experience to either we can just fix the issue for you or we can immediately tell you, I see that you were doing this.
• 10:27
  SPEAKER_01: And if you take these steps, this will no longer be happening to you.
• 10:32
  SPEAKER_01: Because we definitely have customers who inform us of things where it's like it's a really unintuitive solution.
• 10:37
  SPEAKER_01: If you're not in this every day, you might not think, oh, this is what I need to do to get out of this, or this is the thing that's causing the problem.
• 10:44
  SPEAKER_01: But we can detect that.
• 10:45
  SPEAKER_01: And it's also easier on our end to make sure that we're taking all the right steps.
• 10:49
  SPEAKER_01: Some of these processes are like eight checks, check this and then this, but not on this version, all kinds of stuff like that.
• 10:55
  SPEAKER_01: That's pretty tough to do the right way every time as a person, but we can make Airflow do that the right way every time.
• 11:01
  SPEAKER_01: And if we find a mistake in it, we can fix it and fix it forward forever.
• 11:06
  SPEAKER_00: That's so cool.
• 11:06
  SPEAKER_00: And I always love hearing stories internally.
• 11:09
  SPEAKER_00: In case anybody out there doubts whether astronomer really believes in open source airflow, yes, we are all relying on it on our own internal teams.
• 11:18
  SPEAKER_00: And yeah, I love Airflow using Airflow to fix Airflow.
• 11:21
  SPEAKER_00: That's very cool.
• 11:22
  SPEAKER_00: I think that's a good maybe segue into whether you're identifying the issues proactively or helping customers who have already hit something.
• 11:30
  SPEAKER_00: I talked a little bit already about how there can be like a huge breadth of things that you're hitting since Airflow is used for so many use cases and is so flexible.
• 11:38
  SPEAKER_00: I'm curious, just generally, with your experience touching the Airflow implementations of everybody who is using Airflow, what are the most common things that you see people do wrong?
• 11:52
  SPEAKER_00: Or maybe it's not even that they are doing something wrong, but that you see people encounter?
• 11:55
  SPEAKER_01: I definitely say one of the most important things to do with your Airflow instance is to stay upgraded with your patch versions.
• 12:04
  SPEAKER_01: With Astronomer and an Astro runtime, we do backport fixes to open source Apache Airflow.
• 12:10
  SPEAKER_01: Does not do backport fixes, but still it's important to get those patches when they are available.
• 12:16
  SPEAKER_01: The number of times I've seen users run into issues from upgrading too early is much less than those who end up, oh, you hit this bug and it really caused you a big problem.
• 12:30
  SPEAKER_01: This was fixed months ago.
• 12:32
  SPEAKER_01: And you didn't need to encounter this problem if you had an upgrade that was a little bit faster cadence.
• 12:38
  SPEAKER_01: I understand maybe not wanting to do it like literally day one of every release that comes out, but getting months behind is putting yourself at just an unnecessary amount of trouble.
• 12:49
  SPEAKER_01: With astronomers, uh astral runtime, especially with the backport fixes, we work really hard to make sure that we handle any sort of breaking changes or anything like that as called out in release notes.
• 12:59
  SPEAKER_01: So I would always check the release notes, make sure that you're not going to be impacted by a breaking change, but you really shouldn't see that unless you're going from Airflow 2 to Airflow 3, which of course is a bigger migration here.
• 13:09
  SPEAKER_01: And the other thing related to that is making sure to manage your provider versions explicitly.
• 13:15
  SPEAKER_01: We do see people get a little tied together with, oh, I can't upgrade or I don't want to upgrade my Apache Airflow because I have this Snowflake provider that I need this specific version.
• 13:26
  SPEAKER_01: That's fine.
• 13:27
  SPEAKER_01: Keep that version of the specific Snowflake provider and upgrade your underlying airflow package.
• 13:32
  SPEAKER_01: That's the one you really want to make sure stays up to date as much as possible and really shouldn't be having a big impact on your DAGs like code.
• 13:41
  SPEAKER_01: Again, unless you're making that Airflow two to three transition, which has plenty of tooling to help you with that.
• 13:58
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 14:10
  SPEAKER_02: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 14:18
  SPEAKER_00: I think about airflow upgrades is kind of like the proactive maintenance on your car, like getting your oil changed or stuff.
• 14:25
  SPEAKER_00: Like it's annoying and it's really easy to put off for forever, but like the longer you put it off, usually the worse it's going to be.
• 14:32
  SPEAKER_00: Like you said, definitely save you a lot of trouble.
• 14:34
  SPEAKER_00: And I do think that's a great point to make to folks that you don't necessarily have to upgrade the provider version since those do, it's just really hard to keep track of what's happening in provider versions, especially if you're using a lot of them and they go through their own major upgrades or whatever schedule they need to follow.
• 14:52
  SPEAKER_00: Yeah, and that can be really tricky to keep track of and can cause more breaking changes than at least like a patch version of Airflow, which shouldn't cause anything.
• 15:01
  SPEAKER_01: I think the decision is five years ago now, but to break providers separate from the core Airflow package was made for exactly this reason, so that you can keep your core airflow on the latest, patched up, and manage your providers as you need to for your specific use cases.
• 15:17
  SPEAKER_01: You actually refer other things I see people do.
• 15:19
  SPEAKER_01: Another one is not using deferrable operators when you have the chance to.
• 15:25
  SPEAKER_01: Man deferables are such a great tool.
• 15:29
  SPEAKER_01: They make your task so much more resilient.
• 15:32
  SPEAKER_01: And at this point, there are deferable options for almost all of the off-the-shelf community managed operators.
• 15:39
  SPEAKER_01: And uh you get such a huge benefit from doing so.
• 15:43
  SPEAKER_01: It's such a more efficient way to run it, it's a more robust way to run any of your sensor tasks.
• 15:49
  SPEAKER_01: Anytime you can use a deferable, we really should.
• 15:51
  SPEAKER_01: We see a lot of benefit from doing so.
• 15:53
  SPEAKER_01: It can be a little tough if you need to write your own sensor because there is a jump in learning curve from writing a synchronous normal sensor to writing a trigger that has to use Python async to run in the trigger.
• 16:10
  SPEAKER_01: And that can be a little tough.
• 16:11
  SPEAKER_01: But if you're using something that is off the shelf, that is community provided, go with a deferrable option wherever you can.
• 16:17
  SPEAKER_00: That's so interesting.
• 16:18
  SPEAKER_00: This isn't a best practice that I often hear talked about.
• 16:22
  SPEAKER_00: I guess for anybody listening who's not familiar, deferable operators are just a version of the existing operator that release their worker slot while they're waiting for something that's managed by a separate component called the triggerer.
• 16:35
  SPEAKER_00: And they were released in, I don't know, maybe Airflow 22, I think.
• 16:40
  SPEAKER_00: Okay.
• 16:40
  SPEAKER_00: Yeah.
• 16:40
  SPEAKER_00: So they've been around for quite a while and I think they are available for a lot of the existing operators that have been added since then in the couple years since 2.2 came out by the community.
• 16:51
  SPEAKER_00: What are the biggest issues people run into when they're not using them?
• 16:55
  SPEAKER_00: Is it like resource utilization type issues?
• 16:59
  SPEAKER_01: So it can be resource utilization issues, but it's more robustness or the customer reliability team.
• 17:05
  SPEAKER_01: So I'm focused on reliability.
• 17:06
  SPEAKER_01: And what we see is you're running a sensor and you want this sensor to run for 18 hours.
• 17:11
  SPEAKER_01: Well, that means you've got 18 hours for something bad to happen to this worker.
• 17:15
  SPEAKER_01: And maybe you're running it alongside a bunch of other tasks that are much more intensive.
• 17:20
  SPEAKER_01: Someone from a different DAG, it's much more intensive, and you hit an out-of-memory error.
• 17:24
  SPEAKER_01: And most of those tasks, they ran for just a little bit and it's fine.
• 17:28
  SPEAKER_01: But your sensor failed.
• 17:30
  SPEAKER_01: And often we see that the way you write a sensor is you're not expecting it to fail.
• 17:35
  SPEAKER_01: You're like, oh, there's like nothing that's happening here.
• 17:37
  SPEAKER_01: And deferable sensors just take that away from you.
• 17:39
  SPEAKER_01: You're not running on a worker, you're not co-located with anything that's likely to have a problem.
• 17:44
  SPEAKER_01: The trigger is much more, it's not doing as much intensive work.
• 17:47
  SPEAKER_01: And it is more efficient too, which is always great.
• 17:50
  SPEAKER_01: You can run, I think the default number of triggers to run the triggerer is a thousand.
• 17:55
  SPEAKER_01: So it's like way more, oh sure, everybody pile on here than the workers, which most people we see are running somewhere between 10 and maybe 60 tasks per worker.
• 18:05
  SPEAKER_00: Got it.
• 18:06
  SPEAKER_00: Yeah, that makes sense.
• 18:07
  SPEAKER_00: I hadn't really even thought about the reliability aspect of that, especially if you've used a pattern where you're having sensors wait to kick off the rest of a DAG.
• 18:18
  SPEAKER_00: As you said, like having that die in the middle of the hours-long process would be a huge bummer, and you can get pretty behind on those pipelines really quickly.
• 18:27
  SPEAKER_00: So yeah, that's a great tip.
• 18:29
  SPEAKER_01: Another thing we see people do, and this one takes a little bit more work to avoid, but in my opinion, the use of the external task sensor and trigger DAG run operator is something that if you can work around it, you should.
• 18:44
  SPEAKER_01: It's really easy to mess up.
• 18:46
  SPEAKER_01: Which DAG run is this external task sensors monitoring and that other DAG?
• 18:51
  SPEAKER_01: And oh, we changed that one from running Monday at seven to running Monday at eight, and now our external task sensor, it's a very difficult mental model to hold.
• 19:02
  SPEAKER_01: It reminds me a little bit of like you're running a Rube Goldberg machine where okay, this falls over and then that falls over, and then the mouse runs across the room.
• 19:08
  SPEAKER_01: But if you're using a data set or now called asset scheduling, the mental model is much simpler.
• 19:15
  SPEAKER_01: Okay, this task creates has an output asset, and this DAG runs based on that schedule.
• 19:22
  SPEAKER_01: And it usually means that you're going to have more smaller DAGs.
• 19:28
  SPEAKER_01: I think Bick from Coca did a Airflow Summit talk on this called micropipelines, I think, in 2023.
• 19:33
  SPEAKER_01: I think it's a great talk for refactoring DAGs to this pattern.
• 19:37
  SPEAKER_01: But we do see when people use a lot of trigger DAG run operators or a lot of external task sensors, it's common for something to go a little off at some point.
• 19:45
  SPEAKER_00: That makes total sense.
• 19:46
  SPEAKER_00: I think the external task sensor, I forget exactly what the parameter is called or how you set up like the polling interval that it's looking for.
• 19:53
  SPEAKER_01: Execution time underscore FA.
• 19:56
  SPEAKER_00: It's super confusing.
• 19:57
  SPEAKER_00: I always struggled to implement that.
• 19:59
  SPEAKER_00: And then, of course, I've totally abandoned it since data sets became available.
• 20:03
  SPEAKER_00: But I think they're also data sets and now assets, as you said, that are just a lot more flexible as well.
• 20:08
  SPEAKER_00: They give you a lot more power in terms of how you can schedule if you need to do one to many or many to one or combine with time schedules.
• 20:17
  SPEAKER_00: A lot easier to implement that and you have that visibility even in the airflow UI of how everything is connected.
• 20:24
  SPEAKER_01: I think whenever your code, your DAGs look on the page, the way they look in your head, things get so much easier.
• 20:32
  SPEAKER_01: When you're like, well, it's really this way, but we had to like transform it somehow into this other format.
• 20:39
  SPEAKER_01: That's always a recipe for some kind of challenge at some point.
• 20:43
  SPEAKER_01: And one of the roles of CRE is to see what customers are doing, what Airflow users are doing.
• 20:48
  SPEAKER_01: And if we find someone who has a really good reason for doing something that Airflow really doesn't want to do, we want to make sure that we communicate that back to the people within Astronomy who are focused on developing Airflow day to day and say, hey, look what this customer is doing.
• 21:04
  SPEAKER_01: This is why they're doing it.
• 21:06
  SPEAKER_01: What they're doing seems to make sense.
• 21:08
  SPEAKER_01: And Airflow doesn't like it yet.
• 21:10
  SPEAKER_01: So what can we do differently?
• 21:12
  SPEAKER_01: And we've had impact on, we talked a lot about DAG versioning, right?
• 21:15
  SPEAKER_01: When that came out, okay, what are some cases where we've seen problems?
• 21:19
  SPEAKER_01: What does DAG versioning mean to this person and this person?
• 21:22
  SPEAKER_01: And make sure that all that information is going back and into the conversation.
• 21:26
  SPEAKER_01: Of course, there's the dev list and many other places where this conversation happens, but that's something that for us we look for to are is someone trying to do something that makes sense, but not to airflow.
• 21:37
  SPEAKER_00: That's such a valuable feedback loop as well.
• 21:40
  SPEAKER_00: I talk all the time about the benefits of having a community driven project and the folks, as you said, at astronomer who are developing on open source airflow, as well as outside of astronomer, take that feedback really seriously.
• 21:51
  SPEAKER_00: I'm sure it's really helpful to see you just get such exposure on your team to how so many different people are doing it.
• 21:58
  SPEAKER_00: You can pull out some of those.
• 22:00
  SPEAKER_00: Patterns that you might not see if it were just individual folks speaking up in Airflow Slack or something like that.
• 22:05
  SPEAKER_01: That's right.
• 22:06
  SPEAKER_01: There really is such variety to what people are doing.
• 22:09
  SPEAKER_01: And often people who are doing something really unique don't even realize they are.
• 22:12
  SPEAKER_01: They like have a really good reason to do it.
• 22:14
  SPEAKER_01: They think, of course, there's tons of people doing this.
• 22:16
  SPEAKER_01: And I'll be like, I've never seen that before.
• 22:18
  SPEAKER_01: That's very interesting.
• 22:20
  SPEAKER_00: How unique is it that you see something that you've never seen before?
• 22:23
  SPEAKER_00: Like that somebody gets that badge on their airphones.
• 22:27
  SPEAKER_01: That's a good question.
• 22:28
  SPEAKER_01: Once every couple months we see something that's, oh wow.
• 22:31
  SPEAKER_01: There's lots of new mistakes to make, right?
• 22:32
  SPEAKER_01: But sometimes, oh, you're doing this, it makes sense you're doing this.
• 22:35
  SPEAKER_01: We saw just most recently, oh wow.
• 22:38
  SPEAKER_01: Like you'd have a lot of tasks start at the same time, like to the minute, or you're trying to launch a lot of tasks like in the same minute.
• 22:46
  SPEAKER_01: Oh, okay.
• 22:46
  SPEAKER_01: That's interesting.
• 22:47
  SPEAKER_01: That's different than what we'd seen before.
• 22:50
  SPEAKER_00: That's so interesting.
• 22:51
  SPEAKER_00: That's more frequently than I would have guessed.
• 22:54
  SPEAKER_00: And also, I think just points to how powerful Airflow is.
• 22:56
  SPEAKER_00: You can use it for so many different things that it's not hard to come up with a new one.
• 23:01
  SPEAKER_00: All right.
• 23:01
  SPEAKER_00: Well, I think we could probably keep talking about Airflow.
• 23:04
  SPEAKER_00: So good lead into my standard final question since we were just talking about feedback for the open source Airflow developers.
• 23:11
  SPEAKER_00: And my question is, what would you most like to see from the Airflow project in the near future?
• 23:15
  SPEAKER_01: So I am really excited about the new human in the loop AIP to be able to have a task that has a person like making a decision in the middle of a otherwise automated workflow.
• 23:25
  SPEAKER_01: I think that's really cool and unlocks a lot of great use cases.
• 23:28
  SPEAKER_01: Maybe more close to the CRE wheelhouse has been, or so the likes of C is the ability to cross-reference tasks and other things that have happened within this airflow instance and see, oh, what other tasks failed around this time?
• 23:43
  SPEAKER_01: Or what other tasks were running during this period when this task failed.
• 23:48
  SPEAKER_01: It's all possible.
• 23:50
  SPEAKER_01: Like the browse menu like gives you some sense of that, but it is a little challenging to answer some of those questions quickly.
• 23:57
  SPEAKER_01: And I think that leads to people not asking the question or not feeling like they can ask the question.
• 24:03
  SPEAKER_01: And certainly for me and my team when we're doing troubleshooting, like those are important questions to ask.
• 24:08
  SPEAKER_01: Sometimes we can ask them directly to the database or something like that.
• 24:11
  SPEAKER_01: But just having it be more, there's a ton of information there.
• 24:15
  SPEAKER_01: And we're usually used to seeing like just one DAG at a time.
• 24:18
  SPEAKER_01: But sometimes there are questions when things go wrong about things that are happening across DACs.
• 24:23
  SPEAKER_01: And I would love to see some more UI features to service that information in helpful ways.
• 24:28
  SPEAKER_00: That's great.
• 24:29
  SPEAKER_00: I love both of those answers.
• 24:30
  SPEAKER_00: Khan, thank you so much for coming on.
• 24:32
  SPEAKER_00: It's been really insightful hearing about your experience supporting all of our customers using Airflow.
• 24:38
  SPEAKER_00: Close by what's the best way for folks to get in touch with you?
• 24:42
  SPEAKER_01: If you are an astronomer customer and you send the support at astronomer.io, I'll see that.
• 24:47
  SPEAKER_01: But I'm also on LinkedIn and I have a blog at which I sometimes talk about data engineering related topics at carbon and silicon.
• 24:54
  SPEAKER_01: That's my Substack.
• 24:55
  SPEAKER_01: You can cut that out if you do.
• 24:57
  SPEAKER_01: So yeah, I'll sometimes blog on occasional topics.
• 25:00
  SPEAKER_01: Caxel read sent me some traffic once, and that was the big highlight of my micro blogging career.
• 25:06
  SPEAKER_00: Nice, nice.
• 25:07
  SPEAKER_00: I love that.
• 25:07
  SPEAKER_00: I will have to check that out myself.
• 25:09
  SPEAKER_00: Okay, that's great.
• 25:10
  SPEAKER_00: Thank you again for coming on.
• 25:11
  SPEAKER_00: It's been really great chatting.
• 25:15
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 25:22
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 25:26
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.