Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Enhancing DAGs for Data Processing with William Orgertrice III at Cargill

Transcript

Enhancing DAGs for Data Processing with William Orgertrice III at Cargill · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: Adding more monitoring and uh and alerting is absolutely fantastic, especially when something breaks and the team is like, oh, can I get an RCA?
• 0:08
  SPEAKER_01: Can I get a root cause analysis on what's going on?
• 0:11
  SPEAKER_01: It helps you to catch those things a lot more quickly.
• 0:16
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:28
  SPEAKER_02: Okay, hi everyone.
• 0:29
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:31
  SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by William Orgertreis, who is a data engineer at Cargill.
• 0:37
  SPEAKER_02: William, how are you?
• 0:38
  SPEAKER_01: I'm doing well, Kenton.
• 0:40
  SPEAKER_01: How are you doing?
• 0:41
  SPEAKER_02: I'm good.
• 0:41
  SPEAKER_02: I'm good.
• 0:42
  SPEAKER_02: It's really great to have you on.
• 0:44
  SPEAKER_01: Yes, I'm glad to be here to be able to have this conversation.
• 0:47
  SPEAKER_01: Um, I see astronomer and airflow doing some great things.
• 0:52
  SPEAKER_01: So really it's a pleasure to be on here to speak with you today.
• 0:55
  SPEAKER_02: Yeah, well, we love to hear that.
• 0:57
  SPEAKER_02: And I'm excited to dig in more to how you are using Airflow.
• 1:02
  SPEAKER_02: Um, and yeah, for today's podcast, we're gonna discuss um how Cargill is using Airflow and then some strategies that you have for enhancing DAGs for data processing, which I think will be of great interest to a lot of our listeners.
• 1:16
  SPEAKER_02: Uh, but before we get into that, maybe we can start with some background.
• 1:19
  SPEAKER_02: Can you tell me what Cargill does?
• 1:21
  SPEAKER_01: Yeah, so uh Cargill is if you look it up and you look up the largest privately owned company in the United States, Cargill will show up, but I'm so happy to say that.
• 1:32
  SPEAKER_01: Right.
• 1:33
  SPEAKER_01: So um, what we actually do is a food company, simply put, is a food company.
• 1:38
  SPEAKER_01: It's been around for over 160 plus years, have 160 plus thousand employees, and we are in 70 different countries serving over 125 different markets.
• 1:50
  SPEAKER_01: Uh, so I'm happy to say that I'm working here at Cargill because there's a lot of impact that we have.
• 1:56
  SPEAKER_01: Uh, we impact people, billions of people daily.
• 2:00
  SPEAKER_01: So providing food and services to them um in several different countries.
• 2:05
  SPEAKER_01: So it's a really great company to work for.
• 2:08
  SPEAKER_02: Yeah, that's awesome.
• 2:09
  SPEAKER_02: I random fun fact from my own past life.
• 2:12
  SPEAKER_02: I used to work with Cargill um long before I even worked in like data engineering um at a previous job on some supply chain sustainability work.
• 2:21
  SPEAKER_02: And I I think it's just so cool like getting to work with a company like you said that is so large and also I think probably one that like not a lot of people have heard of in their day-to-day life, but that I mean, certainly every American has interacted with.
• 2:36
  SPEAKER_02: Like when you look at the food supply chain, as you said, it's they're just behind such a huge chunk of it.
• 2:42
  SPEAKER_02: It's you know, inescapable.
• 2:43
  SPEAKER_02: So one of those things that just happens kind of behind the scenes when you wonder, like, how does your food get there?
• 2:48
  SPEAKER_02: Uh right.
• 2:49
  SPEAKER_01: Yeah.
• 2:50
  SPEAKER_01: And you know, the the interesting thing about it is yes, uh, since we're so Cargill is headquartered in Waseta, Minnesota.
• 2:58
  SPEAKER_01: And uh, we opened up a brand new office last year in Atlanta, Georgia.
• 3:02
  SPEAKER_01: I want to say one of the best locations in Midtown Atlanta, right next to Georgia Tech.
• 3:07
  SPEAKER_01: And uh because we're new to that space in Atlanta, we're building out the brand.
• 3:13
  SPEAKER_01: Not too many people know about it yet, but I'm actually helping to uh be a brand ambassador of Cargill and help to um improve the brand so that way local businesses and individuals hear about how great the company is.
• 3:26
  SPEAKER_02: Oh, very cool.
• 3:27
  SPEAKER_02: That's great.
• 3:28
  SPEAKER_02: Love to hear about that expansion.
• 3:30
  SPEAKER_02: Uh I've only been to Atlanta once myself, and it was several years ago, but it seemed like a great place.
• 3:36
  SPEAKER_01: All right, yeah.
• 3:37
  SPEAKER_01: I'm born and raised there.
• 3:38
  SPEAKER_01: So, you know, I'm kind of biased.
• 3:40
  SPEAKER_01: I enjoy Atlanta.
• 3:41
  SPEAKER_02: So yeah, awesome.
• 3:43
  SPEAKER_02: That's a fair bias.
• 3:44
  SPEAKER_02: Cool.
• 3:45
  SPEAKER_02: Well, tell me a little bit more then about what your team does.
• 3:48
  SPEAKER_01: Yeah, so currently I um I support 60 plus different teams, and I work on our Cargill data platform team.
• 3:56
  SPEAKER_01: And uh so there essentially the way that uh my director and everybody put it is that CDP or the Cargo Data Platform essentially runs the business, right?
• 4:09
  SPEAKER_01: It is just one important part of uh business operations.
• 4:14
  SPEAKER_01: All the internal and external data comes through the Cargo Data platform, and uh we provide a lot of the data products and a lot of the data to the teens to enable them to be able to do their work.
• 4:25
  SPEAKER_01: So we kind of talked about it, like some of the things that Cargo does, like some of the customers that we work with would be like at Chick-fil-A, McDonald's, Delta, Starbucks, like several other companies, very many of them.
• 4:38
  SPEAKER_01: And the cool thing about it is that the data that we provide is so vast and plentiful that uh we're able to enable uh finance teams, uh inventory for the plants that we service, and several other organizations with a lot of the data and like some of the things that we do helps to drive a lot of the decisions that we make.
• 5:05
  SPEAKER_01: Right.
• 5:06
  SPEAKER_01: So we're here to nourish the world is our tagline, and uh we do that through um a lot of the work that we do locally and also through the data that we provide.
• 5:16
  SPEAKER_02: Yeah, awesome.
• 5:18
  SPEAKER_02: That's great.
• 5:18
  SPEAKER_02: Uh very big purview in terms of the data that you're managing and the impact that it has.
• 5:25
  SPEAKER_02: Then tell me a little bit about Airflow.
• 5:27
  SPEAKER_02: So I understand that you are maybe slightly newer in your Airflow journey at Cargill or where what does that look like right now?
• 5:37
  SPEAKER_01: Yeah, so me personally, I have a little bit over four years of experience with working with Airflow.
• 5:42
  SPEAKER_01: And then uh the cool thing is that so we have the Carga data platform, but something that we're going into now is a new data platform uh called Minerva.
• 5:52
  SPEAKER_01: And Minerva is a whole bunch of new technologies that we bring in to revolutionize uh what we have for uh the organization.
• 6:02
  SPEAKER_01: So new data products, more uh security, more technology.
• 6:06
  SPEAKER_01: So we're um using Airflow as one of our solutions, and then we have like things like AWS, Azure, several other platforms and and products that we use in order to improve our existing ETL pipelines and data pipelines.
• 6:24
  SPEAKER_02: Okay, great.
• 6:25
  SPEAKER_02: And so you mentioned that you have personally been using Airflow for a little bit longer.
• 6:30
  SPEAKER_02: I'm curious, are you trying to bring more Airflow use into your organization?
• 6:35
  SPEAKER_02: Or what made you choose Airflow to be part of this new Minerva data platform?
• 6:40
  SPEAKER_01: Yes, well, I'm just telling you this is that you know, I talked to a lot of the engineers who work at Cargill as well.
• 6:46
  SPEAKER_01: And I like I say, I'm kind of biased because Airflow is really cool, uh, because it just makes I just feel like it makes things easier, makes my my work easier, especially for like batch ingestion processes, like just setting up sensors to let us know oh, the SLAs aren't hitting.
• 7:02
  SPEAKER_01: That's something like that other platforms don't don't allow or don't make it easy for you to be able to do, or um like focusing on the dependencies, like something very simple of that.
• 7:14
  SPEAKER_01: Like, oh, okay, we're downloading libraries.
• 7:17
  SPEAKER_01: So these are some features that Airflow provides that uh Cargo said, okay, well, it actually is really good to be able to do this so that we can optimize our development project process for ETL pipelines.
• 7:32
  SPEAKER_01: Um, because before we were used before we're using like the legacy systems, like Impala and Kudu tables, and you know, it works, but it's really old technology.
• 7:44
  SPEAKER_01: And so we were trying to revolutionize and use newer technologies that have support and um just makes the developer's lifestyle easier as well.
• 7:54
  SPEAKER_01: So that's one of the reasons, right?
• 7:55
  SPEAKER_01: Makes makes my life as a developer easier, but then also I think that um with any organization, it comes into relationships as well, right?
• 8:04
  SPEAKER_01: Like you see what works with a dirt with a specific product and you see what they're offering, and then you may have a relationship with an organization, and that helps you to determine the decision to make when using a specific software.
• 8:19
  SPEAKER_02: Yeah, yeah, that totally makes sense.
• 8:22
  SPEAKER_02: And if you're able to speak about it, tell me about like how are you running Airflow internally or what does your implementation look like at a high level?
• 8:30
  SPEAKER_01: Yeah, so high level I can tell you is that uh so last year when we were first talking about it, right?
• 8:36
  SPEAKER_01: Uh, we were in development.
• 8:38
  SPEAKER_01: Now we do have some teams who are in production.
• 8:40
  SPEAKER_01: Okay.
• 8:41
  SPEAKER_01: Great.
• 8:41
  SPEAKER_01: So it is great to be able to say that we have um some really cool technologies that we're using to be able to make business decisions.
• 8:51
  SPEAKER_01: You have several different teams who like the people who work on the Cargo data platform are helping to uh migrate over into the new data platform, and that data is helping them to be able to use like Airflow.
• 9:05
  SPEAKER_01: So um, yeah, so some of the teams are actually using it in production.
• 9:10
  SPEAKER_02: Okay, very cool.
• 9:12
  SPEAKER_02: And are they the ones I understand um some of your teams are running Airflow on Astro, which is Astronomer's Managed Service?
• 9:18
  SPEAKER_02: Is that right?
• 9:19
  SPEAKER_01: Yes, that's exactly it.
• 9:20
  SPEAKER_01: So we are we are using Airflow through Astronomer uh as our managed solution.
• 9:26
  SPEAKER_02: Okay, great.
• 9:27
  SPEAKER_02: Well, very cool.
• 9:28
  SPEAKER_02: I will be so interested to hear, you know, at a later date, kind of how that implementation continues to go for you.
• 9:34
  SPEAKER_02: It sounds like it'll be really impactful.
• 9:37
  SPEAKER_02: Uh, I also want to talk on a little bit of a different topic about um something you've spoken about publicly.
• 9:43
  SPEAKER_02: I know you talked about this at last year's Airflow Summit.
• 9:46
  SPEAKER_02: Uh, and that was uh strategies for enhancing data pipelines and airflow or DAGs for data processing.
• 9:53
  SPEAKER_02: Again, I think this is a topic that a lot of Airflow users encounter.
• 9:57
  SPEAKER_02: So I'm curious to hear kind of your insights there.
• 10:00
  SPEAKER_02: Maybe you can start by just telling me what experience of yours in the past you said four years or so that you've been using Airflow, what made you interested in giving a talk like this about, you know, this best practices topic?
• 10:12
  SPEAKER_01: Yeah, so I'll just tell you this is that um I used Airflow at a previous organization and I use it in the capacity of building out ETL pipelines to be able to serve as an internal tool that utilizes AI to be able to search and match talent.
• 10:29
  SPEAKER_01: Really cool.
• 10:30
  SPEAKER_01: And I'm gonna tell you that if anything breaks with Airflow, like everything stops, right?
• 10:34
  SPEAKER_01: So it is uh one of those high-priority business tools.
• 10:38
  SPEAKER_01: And what got me interested in speaking at the Airflow Summit in 2025, I would just tell you it was it was a beautiful location.
• 10:45
  SPEAKER_01: It was the first time out there in uh Seattle, Washington.
• 10:48
  SPEAKER_01: It was um in a great hotel, great amount of people, um, extremely intelligent people.
• 10:54
  SPEAKER_01: I I actually really enjoyed coming up to the astronomer table to be able to speak with some of the engineers there or some of the uh people who actually work on Astronomer.
• 11:04
  SPEAKER_01: But I had I joined the Airflow community on Slack.
• 11:08
  SPEAKER_01: I think that was like one of my first uh introductions to it.
• 11:11
  SPEAKER_01: Actually, no, it what really got me into it was I was scrolling on LinkedIn and I saw that there was a book about the new astronomer like tools and technologies that you guys were coming out with.
• 11:24
  SPEAKER_01: I saw I saw that I signed up for a webinar that gave me the link to the Airflow community on Slack, and then um I heard about the Airflow Summit after signing up and going to some of the webinars, and I just showed up consistently and I was able to be able to be a speaker.
• 11:42
  SPEAKER_01: And since I already had experience, I was interested in the book.
• 11:45
  SPEAKER_01: Uh you guys actually gave up a free book as well, and I thought that was really cool about like Airflow 3, right?
• 11:52
  SPEAKER_01: So that was that was really awesome.
• 11:57
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:06
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:18
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:26
  SPEAKER_02: Yeah, that's so great to hear.
• 12:28
  SPEAKER_02: Obviously, like huge, shameless self-plug for the work that my team is doing at Astronomer.
• 12:32
  SPEAKER_02: Someone on my team wrote that book that you're referencing.
• 12:35
  SPEAKER_02: Um, so we're really glad to hear that it's of use.
• 12:37
  SPEAKER_02: But yeah, the Airflow community, of course, is so strong.
• 12:40
  SPEAKER_02: Uh, and there's so many things going on all the time.
• 12:42
  SPEAKER_02: There's just so many people out there using Airflow that it's great to be able to plug in wherever you're at.
• 12:48
  SPEAKER_02: Certainly, I am always happy to hear of people who had a great experience visiting Seattle.
• 12:53
  SPEAKER_02: Uh, this is where I'm, you know, from and where I'm based now.
• 12:57
  SPEAKER_02: I this year's Airflow Summit in 2026 will not be in Seattle, of course.
• 13:01
  SPEAKER_02: Uh, but it is gonna be in Austin.
• 13:03
  SPEAKER_02: So a place that I have never been, but have heard is really great.
• 13:06
  SPEAKER_01: Yeah, awesome.
• 13:08
  SPEAKER_01: And I will say this too is fantastic that it it travels, right?
• 13:12
  SPEAKER_01: So I'm glad that it's actually gonna be moving to Texas as well.
• 13:15
  SPEAKER_01: And uh the weather out there during the airflow summit.
• 13:19
  SPEAKER_01: I don't know if you guys knew ahead of time what the weather was gonna be like.
• 13:22
  SPEAKER_01: Absolutely fantastic.
• 13:24
  SPEAKER_01: Yeah, I was like, I hear that uh it rains a lot in Seattle, it was clear as skies, it wasn't that cold, it was beautiful on the waterfront.
• 13:32
  SPEAKER_01: Perfect, perfect timing.
• 13:34
  SPEAKER_02: Always a toss-up uh a lot of times of year here, but sometimes sometimes you roll the dice and you get lucky.
• 13:40
  SPEAKER_02: Yeah, well, that's so great that you had such a good experience there.
• 13:43
  SPEAKER_02: And then tell me a little bit more about like the best practices that you spoke about for enhancing DAGs.
• 13:51
  SPEAKER_01: Right.
• 13:52
  SPEAKER_01: Uh, so some of the things I I brought up during the speech was about like documentation, right?
• 13:58
  SPEAKER_01: So you can, I would say that you would you should always document each step of what you're doing so that way other developers can actually utilize some of the tools and can be able to troubleshoot things where things are breaking, right?
• 14:13
  SPEAKER_01: Every single task should have a specific purpose, right?
• 14:17
  SPEAKER_01: So yeah, the documentation tells me what it does, and then every single task has a specific purpose.
• 14:23
  SPEAKER_01: So when you're running your Python scripts, you say, okay, well, this function is doing this, and let's say that I'm just trying to pull the data from either like a data lake or I'm trying to pull data from like a flat file, like an Excel spreadsheet or something.
• 14:40
  SPEAKER_01: You want to outline, okay, this is what it's doing, and it only does this, and it moves on to the next task.
• 14:45
  SPEAKER_01: Uh, so the reason I say that is because uh through just trial and error, like when I was first learning airflow, we just say, okay, let's have one task.
• 14:55
  SPEAKER_01: I know it sounds terrible.
• 14:57
  SPEAKER_01: One task that just did everything.
• 14:59
  SPEAKER_01: And when I tell you, it was a complete nightmare.
• 15:03
  SPEAKER_01: But yeah, so so we had uh so we separated out into in several tasks, it makes it easier because you just have like an arrow to go to one task to the other, and then it sends and also send an email notification whenever it fails or whenever it succeeds, right?
• 15:19
  SPEAKER_01: So that's one of those things that I was always suggest.
• 15:21
  SPEAKER_01: Like if you're trying to optimize and make sure that you can troubleshoot, you know the documentation, you know the task only does one thing, and you send an email saying, Okay, this is what's gonna happen, then um, those are some good ways.
• 15:33
  SPEAKER_01: Those are just three good ways.
• 15:34
  SPEAKER_01: I can give you two more, but yeah, we have one more.
• 15:37
  SPEAKER_01: I would always say is um I I think I said it earlier on in this pre in this podcast, is that SLAs or surface level agreements?
• 15:46
  SPEAKER_01: You want to make sure that you have some sort of sensor to just check to see, hey, do you have a long-running job?
• 15:52
  SPEAKER_01: Right?
• 15:53
  SPEAKER_01: Like this DAG is taking too long to finish.
• 15:56
  SPEAKER_01: Like I've had it to where something didn't fail, right?
• 16:00
  SPEAKER_01: So it just was kept running and there is no notification or monitoring.
• 16:05
  SPEAKER_01: So you want to be able to add in that monitoring to say, okay, well, it is a heartbeat that says, okay, this is what's going on with your your cluster for your your airflow.
• 16:16
  SPEAKER_01: Something is not finishing on time.
• 16:19
  SPEAKER_01: Because I've had it to where uh DAGs will take 10 hours when it should only have taken one.
• 16:26
  SPEAKER_01: Right.
• 16:26
  SPEAKER_01: And if you're not checking, like if you're not checking this to check to see if all the deck of the jobs are finished, and you're not checking your email, then things can run on for an infinite amount of time.
• 16:37
  SPEAKER_01: You're just spending money.
• 16:38
  SPEAKER_01: Right.
• 16:39
  SPEAKER_01: So add in that monitoring, add in some sensors to make sure that your DAGs are are finishing, completing.
• 16:46
  SPEAKER_01: Another thing is um add in some contact information, right?
• 16:51
  SPEAKER_01: So one thing that I uh that I've noticed is sometimes you may have it to where in in some teams, you may have it to where you are using Airflow by yourself, like you're only your team is working on something, but then you may also have where you may you may be sharing airflow with other teams, and you guys are all adding in new dads and new jobs.
• 17:16
  SPEAKER_01: And if something breaks, I want to make sure, okay, well, if I want to fix something, let me reach out to the right person or let me send an email to the PDL or um the distribution list where all the teams are going to be able to work on this because you may own maybe you own the job, but you may not own the source data of where the data is coming from.
• 17:40
  SPEAKER_01: So um yeah, I would say add in some contact information that way you can reach out to the appropriate uh teams that may be owning that source.
• 17:48
  SPEAKER_02: Yeah, absolutely.
• 17:49
  SPEAKER_02: Do you do that with tags or something like that in Airflow on the DAG?
• 17:54
  SPEAKER_01: Yeah, you can.
• 17:55
  SPEAKER_01: You can add in um tags and um you can do that directly in in your DAG so that way you can add in some more documentation, right?
• 18:03
  SPEAKER_01: Some more, like, hey, you know, this is the PDL of who you who to reach out to.
• 18:07
  SPEAKER_01: You can also set it up to where the task will send it in email to the source teams as well um within a task.
• 18:14
  SPEAKER_02: Yeah, definitely the DAG documentation or DAG docs.
• 18:18
  SPEAKER_02: We have a tutorial on how to implement those.
• 18:19
  SPEAKER_02: That's a cool feature that came out in kind of like the mid-Airflow two minor releases, I think.
• 18:25
  SPEAKER_02: And yeah, you can put a lot of good stuff in there.
• 18:27
  SPEAKER_02: I've also seen our own data team uh uses the tags uh to add like an owner for each DAG so that again, as you said, if something breaks, you know like which person to go bother, um, as opposed to just sending messages to everybody.
• 18:42
  SPEAKER_02: So those are definitely good recommendations.
• 18:46
  SPEAKER_02: I think that SLAs is a good one as well.
• 18:49
  SPEAKER_02: We um I know the Airflow, at least when we're recording this, Airflow 3.2 was just released and the deadline alerts feature uh was expanded quite a lot.
• 19:01
  SPEAKER_02: I think it's I'm not sure if it's still experimental.
• 19:03
  SPEAKER_02: I think maybe.
• 19:04
  SPEAKER_02: Um, so it's pretty new, but that replaced Airflow's old like SLA feature.
• 19:09
  SPEAKER_02: And yeah, allows you to set up what you're talking about.
• 19:11
  SPEAKER_02: And then of course, like an Astro Astronomer's Managed Service, we have an alerting uh feature where you can set up SLAs because yeah, that proactive alerting, as we refer to it, is can be really important.
• 19:25
  SPEAKER_01: Right.
• 19:25
  SPEAKER_01: I mean, I would say there uh I've had it to where uh like let's say my director or my manager will reach out and say, Hey, you know, um this team reached out to me, they're they're escalating something, and you you look at it on your side and say, Okay, well, everything's running fine, right?
• 19:41
  SPEAKER_01: But you didn't get the alert that something is taking too long.
• 19:45
  SPEAKER_01: Um, and if something is taking too long, there may be something wrong within, you know, that uh within that pipeline.
• 19:53
  SPEAKER_01: And you can also have it set up to where, hey, I don't want it to run another ingestion at the same time, so it can only have like one run.
• 20:03
  SPEAKER_01: I think it's called one run, and then it doesn't start up a new ingestion until it finishes.
• 20:10
  SPEAKER_02: Yeah, definitely if you're controlling like the concurrency or parallelism, your airflow instance there.
• 20:16
  SPEAKER_02: Yeah, that can be an even bigger deal, of course, if something's just hanging.
• 20:22
  SPEAKER_02: Yeah.
• 20:23
  SPEAKER_02: Um, yeah, that's cool.
• 20:24
  SPEAKER_02: And did you say you were implementing those with sensors?
• 20:28
  SPEAKER_01: Yeah, that's that was one one of the use cases, right?
• 20:31
  SPEAKER_01: Like having like a I think it's called a heartbeat where you can just check to see to see exactly like okay, it what how long is something taking?
• 20:40
  SPEAKER_01: Right.
• 20:41
  SPEAKER_01: Um, you can also on the UI it'll tell you like how long um a job has been running as well.
• 20:46
  SPEAKER_01: That's that's something else you can check.
• 20:48
  SPEAKER_01: But if you're not on the UI or you know, not checking your email, you may not be able to see that.
• 20:56
  SPEAKER_02: Yeah, yeah.
• 20:56
  SPEAKER_02: Definitely easier if you can be alerted for sure.
• 21:00
  SPEAKER_01: Yep.
• 21:00
  SPEAKER_02: Yeah, well, those are all really good best practice recommendations.
• 21:04
  SPEAKER_02: I'm curious, what what do you think is the most important of those kind of for your team right now?
• 21:11
  SPEAKER_02: Or do you think you'll be adopting those strategies as your airflow usage grows?
• 21:15
  SPEAKER_01: I want to say all of them, all of them equally as important, right?
• 21:18
  SPEAKER_01: Uh like you need to have like, of course, um making things faster is a great thing, right?
• 21:23
  SPEAKER_01: So using like different libraries would be great.
• 21:26
  SPEAKER_01: So like you can use like pandas and generators within your Python and make things faster.
• 21:33
  SPEAKER_01: But I think that the monitoring is a really big piece that uh Airflow that Airflow and um Astronomer provides so that way it really stands out amongst like other ETL tools.
• 21:47
  SPEAKER_01: So yeah, adding more monitoring and uh and alerting is absolutely fantastic.
• 21:54
  SPEAKER_01: Especially when, like, especially when something breaks and the team is like, oh, can I get an RCA?
• 21:59
  SPEAKER_01: Can I get a root card?
• 22:00
  SPEAKER_01: Analysis on what's going on.
• 22:02
  SPEAKER_01: It helps you to catch those things a lot more quickly.
• 22:05
  SPEAKER_02: Yeah, absolutely.
• 22:07
  SPEAKER_02: That's great.
• 22:07
  SPEAKER_02: Well, I think um, I'm thanks for going through that again.
• 22:11
  SPEAKER_02: All really great best practice topics.
• 22:14
  SPEAKER_02: Uh, maybe a lead-in to my standard final question that I ask all of my guests.
• 22:19
  SPEAKER_02: Um, and it can maybe be related to these, or it doesn't have to be.
• 22:22
  SPEAKER_02: Uh, but that is what would you most like to see from development of the Airflow project in the near future?
• 22:27
  SPEAKER_01: Yes.
• 22:28
  SPEAKER_01: Uh so I hear, I hear people talking about this more like streaming ingestion, right?
• 22:35
  SPEAKER_01: With with Airflow and Astronomer.
• 22:37
  SPEAKER_01: Is that coming up?
• 22:38
  SPEAKER_02: I don't know that I would say streaming specifically.
• 22:42
  SPEAKER_02: I think there are, you know, sort of inroads to like more event-driven pipelines.
• 22:48
  SPEAKER_02: Um, so you can do event-driven scheduling in Airflow 3.
• 22:51
  SPEAKER_02: I think there's also, you know, potentially some work on just like lowering the latency.
• 22:55
  SPEAKER_02: I don't know that Airflow will ever be, you know, like a Kafka replacement or something like that in terms of streaming.
• 23:01
  SPEAKER_01: Right.
• 23:02
  unknown: Yeah.
• 23:02
  SPEAKER_01: Yeah, because we we still use Kafka.
• 23:04
  SPEAKER_01: Like, I can't, I think you can't really go without using Kafka if you're talking about streaming, right?
• 23:09
  SPEAKER_01: Right.
• 23:09
  SPEAKER_01: So, but yeah, definitely.
• 23:11
  SPEAKER_01: So we're using you're using Kafka, but um, yeah, something I would love to see from um from Astronomer or just from Airflow in general, is the ability to be able to test connections, right?
• 23:24
  SPEAKER_01: Like test your connections a little bit more uh more quickly.
• 23:28
  SPEAKER_01: Like I I remember I've worked on like a project where hey, I'm trying to connect to like Oracle or I'm trying to connect to Microsoft SQL Server, and you put in the connection string, and now I have to create the job, and then it doesn't work exactly, right?
• 23:43
  SPEAKER_01: So now I want to be able to say, hey, okay, well, let me go into Oracle and then when I when I define my connection, you can do like a quick test to see if it it if the connection works.
• 23:56
  SPEAKER_02: Okay.
• 23:56
  SPEAKER_02: I think it is there, uh, do you mean like outside of the connection form in the Airflow UI?
• 24:02
  SPEAKER_02: I think that functionality was I think it was always there.
• 24:04
  SPEAKER_02: It was just disabled by default, and I think now it's back to being enabled by default, I think.
• 24:10
  SPEAKER_02: Um I have to check that which uh version is it in the Airflow 3 version or yeah, there was there used to be an Airflow 2 like test connection button, and then like I said, it was uh there was a security kind of concern with it, and so it was disabled by default for a while.
• 24:27
  SPEAKER_02: But I think you could go in and turn it on.
• 24:30
  SPEAKER_02: Um, but I think maybe I'm making this up, but I I think now it's back to being enabled by default.
• 24:35
  SPEAKER_02: Anyway, something to look into because yeah, I think what you're describing is uh is something that you already have for most workflows, yeah.
• 24:42
  SPEAKER_02: I maybe um maybe it's not exactly what you're looking for, but yeah, I can totally see where you're coming from there.
• 24:48
  SPEAKER_01: Okay, awesome.
• 24:49
  SPEAKER_01: Well, I'll I'm it seems like you guys are already on, you know, already have it, right?
• 24:54
  SPEAKER_01: So I will have to check it out to see because it it also could depend on the organization as well.
• 24:59
  SPEAKER_01: Cause sometimes, like you how you said, it could be disabled too.
• 25:02
  SPEAKER_02: Yeah, yeah, that that could be it.
• 25:04
  SPEAKER_02: But um, yeah, regardless, great answer there.
• 25:07
  SPEAKER_02: Uh okay.
• 25:08
  SPEAKER_02: Well, William, it's been really nice speaking with you.
• 25:10
  SPEAKER_02: Really appreciate all of the insights that you've been able to share here.
• 25:14
  SPEAKER_02: Just to close us out, what is the best way for folks to get in touch with you?
• 25:18
  SPEAKER_01: Yeah, so you can reach out to me.
• 25:20
  SPEAKER_01: My name is William Orga Trice on on LinkedIn.
• 25:23
  SPEAKER_01: Uh, so that's really one of the best places that you can reach out to me.
• 25:27
  SPEAKER_01: I'm always active on LinkedIn.
• 25:29
  SPEAKER_01: Um, and yeah, I'm I'm looking forward to some more, some more great things from Airflow and Astronomer, and um love to be able to have some conversations with some people who are also interested in some of the great things that are coming up.
• 25:40
  SPEAKER_02: Yeah, absolutely.
• 25:42
  SPEAKER_02: Well, we'll throw links to your LinkedIn in the show notes.
• 25:45
  SPEAKER_02: And yeah, I hope to see you at uh some Airflow community thing coming up.
• 25:50
  SPEAKER_01: Awesome.
• 25:50
  SPEAKER_01: Thank you so much, Kenton.
• 25:51
  SPEAKER_01: I appreciate you for reaching out.
• 25:53
  SPEAKER_02: Yeah, of course.
• 25:54
  SPEAKER_02: Thank you.
• 25:57
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 26:04
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 26:07
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.