Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
Transcript
- 0:00SPEAKER_02: The world is full of invisible waypoints and airspaces and highways for planes in the sky.
- 0:07SPEAKER_02: And all of our work helps these pilots navigate and plan flights and stay safe in the sky.
- 0:14SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
- 0:25SPEAKER_01: Okay, hi everyone.
- 0:26SPEAKER_01: Welcome to the Data Flowcast.
- 0:28SPEAKER_01: I'm your host, Kenton Danis, and today I'm joined by Samantha Blaney Quibus, who is a software engineer at Jeppison for Flight.
- 0:35SPEAKER_01: Samantha, how are you?
- 0:37SPEAKER_01: Hi, I'm good.
- 0:38SPEAKER_01: Thank you so much for having me on.
- 0:39SPEAKER_01: Yeah, it's really great to have you.
- 0:41SPEAKER_01: I'm excited to get into this.
- 0:43SPEAKER_01: For our listeners for today's podcast, we're going to discuss how Jeppison for Flight manages aviation data pipelines with airflow.
- 0:50SPEAKER_01: And we'll also touch on where AI fits into that picture.
- 0:54SPEAKER_01: So really exciting topic.
- 0:56SPEAKER_01: But before we get into all of that, I want to start with some background on the company.
- 1:00SPEAKER_01: So can you tell me what Jefferson for Flight does, kind of where it sits in the aviation industry?
- 1:06SPEAKER_02: Yeah, absolutely.
- 1:07SPEAKER_02: Uh Jepson FloorFlight is a semi-recently and newly joined company where the leading providers in aeronautical aviation data and have been for their past 90 years or so.
- 1:18SPEAKER_02: Our Jeppison legacy brings some strong work in charting, navigation, data, and our FourFlight side brings some high-tech in our electronic flight bag products and offerings.
- 1:29SPEAKER_02: We support pilots all across the world from general aviation pilots to commercial aviation pilots to militaries across the world.
- 1:37SPEAKER_01: Yeah, that's great.
- 1:38SPEAKER_01: And can you maybe tell me like when a pilot is using your data, what does that help them do?
- 1:46SPEAKER_01: Or what does that look like for them?
- 1:48SPEAKER_02: Yeah, just as cars need to navigate and need instructions and directions, uh, planes do too.
- 1:53SPEAKER_02: The world is full of invisible waypoints and airspaces and highways for planes in the sky.
- 2:00SPEAKER_02: And all of our work helps these pilots navigate and plan flights and stay safe in the sky.
- 2:06SPEAKER_01: Okay, very cool.
- 2:07SPEAKER_01: This makes me think of those like images that you can look up of like all of the different flight paths that are happening at any given time.
- 2:15SPEAKER_01: And it's just like the entire globe is like this giant like spider web of all of them going.
- 2:20SPEAKER_01: It's just incredible that it all happens safely and mostly without hitches.
- 2:26SPEAKER_01: Uh you're helping make that happen.
- 2:28SPEAKER_01: That's very cool.
- 2:29SPEAKER_02: It's a huge interconnected system that is, you know, millions of people across the world responsible for safety and all working on this big publication uh cycle.
- 2:39SPEAKER_01: Yeah, awesome.
- 2:41SPEAKER_01: And so tell me a little bit more then about what your team does and how you support your business.
- 2:47SPEAKER_02: Yeah, I work on the global data pillar on the NAV data exchange platform team.
- 2:53SPEAKER_02: Uh, we are the stewards of what I like to call uh the most complete and largest aviation navigation database.
- 3:02SPEAKER_02: We have data for over 20,000 airports, runways, frequencies, airspaces, aircrafts, everything that you could think of.
- 3:10SPEAKER_02: Uh we consume and we publish.
- 3:13SPEAKER_02: We uh source more than 50 sources of data from the FAA, other countries, um, government agencies, and private sources from across the world.
- 3:24SPEAKER_02: The world operates on the same ARINC 28-day cycle.
- 3:28SPEAKER_02: We follow the same publication schedule that the FAA does, the NAV Canada and across the world.
- 3:33SPEAKER_02: We receive these sources at different times throughout this 28-day cycle, and we combine them all to create our most perfect picture of what the airport is called, the frequencies, um, and everything that is correct for that cycle.
- 3:46SPEAKER_01: Okay, interesting.
- 3:47SPEAKER_01: So very repetitive kind of data processing and everything that goes into that.
- 3:52SPEAKER_01: So I think I assume that will be very relevant to the next part of our conversation, which is can you tell me how airflow fits into all of this and what are you orchestrating with it in your day to day?
- 4:03SPEAKER_02: Airflow is the heartbeat of our data cycle.
- 4:05SPEAKER_02: It orchestrates almost all of our processes, all of our business logic, all of the um data transformations that we do.
- 4:12SPEAKER_02: We have uh many DAGs set up that sense for new data that sugar imports and run these processing export jobs.
- 4:22SPEAKER_02: Cool.
- 4:24SPEAKER_01: Yeah, that's great.
- 4:25SPEAKER_01: And uh so you mentioned that everything is happening on this repetitive 28-day cycle.
- 4:32SPEAKER_01: Um, and I would imagine that delivering the data on time on that cycle is very important.
- 4:38SPEAKER_01: Since imagine you don't want planes to have out-of-day information.
- 4:44SPEAKER_02: We have very strict time requirements of our cycle.
- 4:46SPEAKER_02: Um, we run into a lot of these hurry up and wait situations where we want to get the data out as fast as we can, um, but we might be still waiting for some sources.
- 4:56SPEAKER_02: So across this 28 days, we have very clearly defined weeks of busy times and slowdown weeks that we've programmatically entered into our airflow system through the use of custom timetables.
- 5:10SPEAKER_02: We created custom timetables to track this 28-day cycle, and the cycle starts on cutover.
- 5:16SPEAKER_02: Our cutover DAG updates are airflow variables or UI variables that set the cycle number that then trigger data sets to uh start all of our downstream processes for this new cycle.
- 5:29SPEAKER_01: Okay, interesting.
- 5:30SPEAKER_01: So, yeah, you're you're working within a lot of scheduling constraints to get this data on time.
- 5:36SPEAKER_01: Other than the custom timetables, is there anything that you've done within Airflow to kind of manage these constraints or or maybe not?
- 5:45SPEAKER_01: And just tell me a little bit more about the custom timetables.
- 5:48SPEAKER_02: Yeah, part of the constraints is um some of the sources arriving at the same time each cycle.
- 5:55SPEAKER_02: Sometimes we receive multiple deliveries of the same source, um, or we receive it a cycle late or half a cycle early.
- 6:02SPEAKER_02: So those we've um added into our Earthflow as extra sensing tasks that are aware if we've already received data for that cycle, if we expect to receive more, um, or if we don't expect to receive more and that source is marked completed for this cycle.
- 6:20SPEAKER_02: Okay.
- 6:20SPEAKER_02: Yeah, interesting.
- 6:22SPEAKER_01: I'm just I'm curious with the files arriving at different times.
- 6:25SPEAKER_01: So are you with using the custom timetable, are your DAGs scheduled at a certain time?
- 6:30SPEAKER_01: Like you don't use data sets or assets or something like that to do like more.
- 6:36SPEAKER_02: We have a mix of both.
- 6:38SPEAKER_01: A mix of both.
- 6:39SPEAKER_01: Okay.
- 6:40SPEAKER_02: Uh we have a mix of both.
- 6:42SPEAKER_02: The timetable is used for kind of the start of the cycle and processes that we know we want to happen seven days before we ship, 14 days before we ship, or one day after we ship.
- 6:54SPEAKER_02: Between that, we have a lot of data dependent sources and processes that have to wait on the upstream sources.
- 7:01SPEAKER_02: So we do have a mix of data sets that allow these downstream DAGs to wait um for all of the things to be ready that are both, you know, timed and beta sensitive.
- 7:13SPEAKER_01: Okay, got it.
- 7:14SPEAKER_01: Interesting.
- 7:15SPEAKER_01: Yeah, that's very cool.
- 7:17SPEAKER_01: And so I alluded to a little bit in the intro that I understand that you're also working with AI to make some of these processes more efficient.
- 7:26SPEAKER_01: Tell me a little bit more about that and what how what use case AI is supporting for you.
- 7:32SPEAKER_02: Our two biggest use cases for AI and Airflow as is one debugging.
- 7:37SPEAKER_02: Our pipelines are pretty well fleshed out.
- 7:39SPEAKER_02: Our team has been working on this 28-day cycle for 10 years or so, yeah, doing almost the same thing.
- 7:46SPEAKER_02: But things happen all the time.
- 7:47SPEAKER_02: We again uh source data from all across the world where sometimes words are spelled or spelled wrong in Excel sheets that causes pipeline failures and breakage.
- 7:58SPEAKER_02: Because our timing is so sensitive, some things happen over the weekend that we're not able to get to uh later on.
- 8:04SPEAKER_02: Um, the second requirement for AI is also just environment awareness for all of our developers and analysts.
- 8:10SPEAKER_02: The more DAGs that we have created and more dependencies, the you know, more complex it gets.
- 8:17SPEAKER_02: Yeah, our uses are uh debugging and awareness.
- 8:20SPEAKER_02: For debugging, we've been able to, or we're working on implementing what I call the data port data traffic agent.
- 8:29SPEAKER_02: It is a pretty simple two-task DAG that works as a Slack bot.
- 8:36SPEAKER_02: Um, our team's messaging is in Slack bot.
- 8:38SPEAKER_02: We work with GIS analysts, data operations analysts, and engineers working on this whole cycle together.
- 8:44SPEAKER_02: All of our messaging is in Slack, and our jobs post our their failures or successes in these channels as well.
- 8:51SPEAKER_02: We have our our our jobs post the success or failure messages with uh data port tags that then the bot is polling for looking for that tag.
- 9:00SPEAKER_02: The second task in our DAG is the process command.
- 9:03SPEAKER_02: Whatever command was processed after that data port tag is mapped to a list of predefined tasks that we've already said.
- 9:11SPEAKER_02: AI is really great and really smart at figuring a lot of things out, but so are we.
- 9:16SPEAKER_02: We've all we've already figured out a ton of you know common fixes and things to debug.
- 9:21SPEAKER_02: And so we built that into a list so we don't have to use uh too many tokens um by sending every request to AI and we're able to kind of process that ourselves.
- 9:30SPEAKER_02: I like to think of uh airflow operators almost as MCPs.
- 9:34SPEAKER_02: They're a way to connect to these different services that we've already defined, and we're able to utilize those operators as our connections.
- 9:42SPEAKER_02: So people are able to use that same data port tag in the Slack messages to either ask questions to submit a string to our AI token or process a command that we've already written, like submit job, debug this job, status of the cycle.
- 10:00SPEAKER_02: And that runs through the commands and outputs a message in that Slack channel.
- 10:05SPEAKER_02: It helps with the analyst being aware of where we are in the cycle, what's missing, if anything um has been broken.
- 10:12SPEAKER_01: That's so cool.
- 10:13SPEAKER_01: Uh this use case made me think of uh in a past job, I used to manage supply chain data for a nonprofit that did like supply chain traceability for um agricultural products that often cause deforestation.
- 10:26SPEAKER_01: So, anyways, I dealt with a lot of data in spreadsheets that was like the names of farms, you know, which are in very rural areas like all over the world and they're often in English.
- 10:36SPEAKER_01: And it was just such a nightmare from a perspective of cleaning the data because of course, like people spell things differently, or somebody writes it in English, and somebody writes it in Portuguese or whatever it is.
- 10:47SPEAKER_01: And so I it's one of those things that I really wish AI had existed at the time that I was managing all of that, you know, more than 10 years ago.
- 10:55SPEAKER_01: But uh I'm sure it would be a much easier job today.
- 10:59SPEAKER_02: Yeah, absolutely.
- 11:00SPEAKER_02: I feel like so much of our time as developers, I'm sure everybody can say, is really spent on debugging and reading through logs and, you know, data cleaning.
- 11:10SPEAKER_02: The industry has come so far with so many tools, but we're still limited by a human factor, uh original, you know, data ingestion and adding all these tools and steps along the way uh really to speed up our processes and let our brains think of new cool things and not just be bogged down by same, you know, failures.
- 11:33SPEAKER_01: Yeah, absolutely.
- 11:34SPEAKER_01: Everybody wants to be spending their time on that more interesting work.
- 11:38SPEAKER_01: Uh, and I like that you touched on adding, you know, the extra context in so that all of the work that your team has already done to kind of standardize this debugging process is also available to the AI.
- 11:50SPEAKER_01: I think that makes such a huge difference, obviously, in the output and the efficiency.
- 11:54SPEAKER_01: So it's cool that you've been able to automate all of that with Airflow.
- 11:57SPEAKER_02: Yeah.
- 11:58SPEAKER_02: Absolutely.
- 11:59SPEAKER_02: We've I've really taken a lot of inspiration from the skills that um, you know, have been published by all these companies, Astronomer, the Airflow skills, and understanding how all these agents work and how they're trying to work.
- 12:13SPEAKER_02: I think under the hood, it's not so complicated.
- 12:17SPEAKER_02: There's commands, and as developers, you're already doing these processes in your best practices of documentation, PRs, and testing.
- 12:26SPEAKER_02: And now it's a way for us to move beyond just the procedural aspects of development and into a lot more creativity and and yeah, awareness.
- 12:34SPEAKER_02: There is just so many things to think about all the time uh with data.
- 12:39SPEAKER_02: And yeah, it allows us more more brain space.
- 12:42SPEAKER_01: Yeah, totally.
- 12:44SPEAKER_01: And so with these two AI use cases, the debugging and the awareness, or maybe it's different for both of them, kind of where are you at in your implementation journey with this?
- 12:53SPEAKER_01: I know you mentioned one of them at least is integrated with Slack, but are these like fully running in production now, or is it still an ongoing process?
- 13:02SPEAKER_02: Fully running, we're definitely still in development and looking at different ways we can run things too.
- 13:08SPEAKER_02: Um, I've looked into ways that we can inject AI within the pipelines themselves as specific tasks.
- 13:16SPEAKER_02: And so we're doing a lot of strategy thinking of the best way to use it.
- 13:21SPEAKER_02: I feel very lucky to be in a company that has a bunch of resources and we haven't um been so confined in in pricing.
- 13:28SPEAKER_02: But, you know, thinking of the token usage and all these different models, still wanting to be very efficient with our workflow.
- 13:35SPEAKER_02: And and also, yeah, uh building on the knowledge and skills of the developers in our team already, AI is is you know a new coworker and a tool, and you know, not going to be used to replace people's brains and work here.
- 13:53SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
- 14:02SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
- 14:14SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
- 14:22SPEAKER_01: Yeah, definitely love the description of it as a coworker.
- 14:25SPEAKER_01: Cool.
- 14:25SPEAKER_01: And then I'm curious, just as you've gone through this process, what have you thought was the hardest part of operationalizing these AI agents, like in the context of doing it within an Airflow workflow?
- 14:38SPEAKER_02: Yeah, I gotta say that things were honestly easier than I thought.
- 14:42SPEAKER_02: Airflow has such great documentation and such a good community of people constantly putting out new ideas and new things, projects that they're working on, that I've really been able to kind of like consume a lot of this information and you know, try to use what's out there to build a lot of the AI agents that I've been using in my own development, you know, are very agreeable.
- 15:04SPEAKER_02: And I say, Can I do this?
- 15:05SPEAKER_02: And they say, Yeah, you can.
- 15:07SPEAKER_02: And here's how.
- 15:27SPEAKER_02: You know, we could do and should do, how much we wanted to give it power to run jobs, um, which environments it could pull logs from and what other tasks we wanted it to pay and either, you know, always our our QA staging or have it actually reach out to production.
- 15:46SPEAKER_02: There's a lot of, I feel like just so many different ways to then implement it to that figuring out the best case is is tricky.
- 15:54SPEAKER_02: So many good ideas, just uh hard to pick one.
- 15:57SPEAKER_01: Yeah, definitely.
- 15:59SPEAKER_01: I think um, yeah, it's interesting that you bring up kind of the just compliance and security aspect of it, is I think like it's definitely something that everybody is grappling with right now and how much autonomy to give everything, at least right away.
- 16:14SPEAKER_01: Uh, but I do love to hear that the challenges haven't really been so much on the Airflow side.
- 16:18SPEAKER_01: I think that's great news for the Airflow project.
- 16:21SPEAKER_02: Yeah, I think Airflow is with the REST API, it's really great, easy to connect to the database backend is super powerful, um, I think, and can be used in so many different ways of just you know, your your task history and writing DAGs has been really easy with AIs.
- 16:38SPEAKER_02: Um I'm the one that has to know what order things run in and the dependencies, and just being able to explain that um, you know, to an agent that then writes my DAG and then I can go and be like, okay, know this, no name this has been really fun and easy.
- 16:53SPEAKER_01: Yeah, that's great.
- 16:55SPEAKER_01: I think uh you might have mentioned this earlier, but yeah, Astronomer has open sourced our skills repo with Airflow, which I am just personally our DevRel team.
- 17:04SPEAKER_01: So my team works a lot on contributing to those to make the AI better at writing DAGs specifically.
- 17:11SPEAKER_01: And we've gotten a lot of great feedback on that and are really excited.
- 17:14SPEAKER_01: But yeah, it's definitely it's very cool and also a little crazy to me just how much better the AI is at doing very specific things like writing DAGs like now, especially compared to even like four to six months ago.
- 17:29SPEAKER_01: Uh yeah, it's it can save you a lot of time.
- 17:33SPEAKER_02: Yeah, one of the other great things I think about the skills and and the MCPs and agents is getting new developers on boarded with Airflow and also extending that to analysts as well.
- 17:44SPEAKER_02: Um, our analysts watch our DAGs and you know, sometimes trigger and are aware of things, but there is a bit of learning curved airflow.
- 17:51SPEAKER_02: And with the skills, it you know, you're able to learn along of okay, what are the best practices to debug or to trigger operators or you know, all these downstream um things.
- 18:02SPEAKER_02: And I think I'm really excited to have more people add their own DAGs and, you know, uh yeah, add to this our data port airflow.
- 18:11SPEAKER_01: Yeah, yeah, that's super exciting.
- 18:13SPEAKER_01: Well, I love to hear that you've made so much progress on this already.
- 18:17SPEAKER_01: And uh yeah, it would be interested to hear kind of how it evolves over time.
- 18:21SPEAKER_01: So thank you for going through that.
- 18:23SPEAKER_01: I always end with a standard final question.
- 18:26SPEAKER_01: This does not have to be AI specific, although maybe it could be.
- 18:30SPEAKER_01: Uh, but that's what would you most like to see from the development of the Airflow project in the near future?
- 18:35SPEAKER_02: Yeah, it's such a great question.
- 18:37SPEAKER_02: Um, there's so many good things coming out every day.
- 18:40SPEAKER_02: And I'm really excited about a lot of things in Airflow 3, the new UI, being in Rust.
- 18:47SPEAKER_02: One very specific thing that I would like to see is uh parameters tasks uh retries.
- 18:53SPEAKER_02: When we retry tasks, sometimes you know they fail because or tasks fail because a file name was wrong, or we need to change something slightly, but I don't want to reset our whole pipeline.
- 19:04SPEAKER_02: Be able to reset one task with a different parameter would be really cool and awesome.
- 19:08SPEAKER_01: Oh, interesting.
- 19:10SPEAKER_01: Yeah, I've I've actually never heard that request before, but that totally makes sense.
- 19:14SPEAKER_01: Even being able to, I can see, like provide, you know, a list of parameters or something to cycle through on failure or something like that, I can imagine, would be very helpful in certain cases.
- 19:26SPEAKER_02: Yeah.
- 19:27SPEAKER_02: I see another use for AI that we haven't integrated at all would be to kind of live edit our DAGs and re-upload our files to the DAG folder.
- 19:36SPEAKER_02: But the Airflow back end, you know, parsing the DAGs pretty quickly.
- 19:40SPEAKER_02: We can kind of make changes.
- 19:42SPEAKER_02: Of course, in a production environment, it's very risky, but that's kind of where we're trying to figure out how much can we use Airflow to, you know, maybe make these one-off fixes or or, you know, internal little changes um quickly so that we can move forward in our processes and not be stuck behind walls.
- 20:00SPEAKER_01: Yeah, absolutely.
- 20:01SPEAKER_01: I think that's certainly something that we think about a lot at astronomer is just how to make that process more efficient and where AI can come in.
- 20:09SPEAKER_01: Uh, I had an issue earlier today with a pipeline on um my team having a kind of transient failure, or I think it was just that the some file that got written ended up on the wrong worker because it took too long or whatever, you know, one of those standard just the things that like rerunning it will fix it, but a hashtag weird.
- 20:29SPEAKER_01: Yeah, exactly.
- 20:30SPEAKER_01: And I feel like it would have been great if we had an agent that was looking at that that probably would have realized, like, oh, this was a weird fluke, just rerun it before I went in and like did the same thing.
- 20:43SPEAKER_01: And so yeah, I think that sort of functionality will get better and better, especially when there's not a code change needed.
- 20:49SPEAKER_01: It's just a oh, we have to try a different parameter, or we have to just simply run the thing again, uh, or yeah, whatever it is.
- 20:57SPEAKER_02: Yeah, we have so many cases of that where it's uh yeah, hashtag weird, rerun it and it works.
- 21:03SPEAKER_02: And yeah, I'd love to see agents be able to debug that and have that same conclusion of let me just rerun the task, reset the task, and then and then the downstream clears.
- 21:13SPEAKER_02: A lot of fun stuff coming up.
- 21:14SPEAKER_01: Yeah, definitely lots to look forward to.
- 21:17SPEAKER_01: Well, Samantha, it's been really great chatting with you about this.
- 21:20SPEAKER_01: I think it was super interesting to learn more about your use case and you guys are kind of on the forefront of you know implementing AI into your Airflow instance.
- 21:29SPEAKER_01: So uh yeah, really appreciate you sharing that with us.
- 21:32SPEAKER_01: Just to close out, what is the best way for folks to get in touch with you?
- 21:36SPEAKER_02: Thank you.
- 21:36SPEAKER_02: Yeah, thanks again for having me on.
- 21:38SPEAKER_02: It's been a blast.
- 21:39SPEAKER_02: Um, you can find me on LinkedIn at Samantha Blaney.
- 21:41SPEAKER_02: I'm very excited to share what we're doing, and it's um great stuff.
- 21:45SPEAKER_02: Thank you.
- 21:46SPEAKER_01: Yeah, that's awesome.
- 21:47SPEAKER_01: Well, we'll put links uh to all of that stuff in the show notes.
- 21:50SPEAKER_01: So uh thank you for again for coming on.
- 21:52SPEAKER_01: It's been great talking to you.
- 21:54SPEAKER_01: Thanks again, Kenton.
- 21:55SPEAKER_01: Bye.
- 21:58SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
- 22:05SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
- 22:08SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.