Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / From Task Failures to Operational Excellence at GumGum with Brendan Frick

Transcript

From Task Failures to Operational Excellence at GumGum with Brendan Frick · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We actually had a KPI request to start looking at task failures, and we just thought it felt wrong.
• 0:04
  SPEAKER_02: It was like everything's always successful.
• 0:06
  SPEAKER_02: We know things are failing, not being captured by this metric.
• 0:09
  SPEAKER_02: And so we just looked at it in a little bit more airflow way what really defines failure.
• 0:13
  SPEAKER_02: And it's actually retry because every task in our airflow instance is destined to succeed eventually as soon as the failure is fixed.
• 0:24
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:34
  SPEAKER_01: Hello everyone.
• 0:35
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:37
  SPEAKER_01: I'm your host, Mark Lometti, and today I'm joined by Brendan Freak, Senior Engineering Manager Data at Gum Gum.
• 0:46
  SPEAKER_01: And I would like to ask you, Brendan, how you're doing today?
• 0:50
  SPEAKER_02: I'm doing great.
• 0:51
  SPEAKER_02: It's a summer day, a little early in Santa Monica, but it's a great day.
• 0:54
  SPEAKER_02: A lot of Mark posts on LinkedIn and other materials throughout the year.
• 0:58
  SPEAKER_02: So it's great to talk to you here.
• 1:00
  SPEAKER_01: That's awesome.
• 1:00
  SPEAKER_01: I'm very happy to have you as well because I know that you have an interesting approach to task failures and DAG issues and so on.
• 1:08
  SPEAKER_01: So I'm sure that it's going to be very interesting for the people that are listening this episode.
• 1:12
  SPEAKER_01: And actually, just before getting into the questions, I would like to start with the primary takeaways that people will get out of this episode.
• 1:20
  SPEAKER_01: So, first, you will know how to approach task and DAG issues, which is very common, unfortunately.
• 1:27
  SPEAKER_01: What metrics to track, and also maybe some best practices or inspirations that has today.
• 1:35
  SPEAKER_01: So, without further ado, let's begin with the first one.
• 1:38
  SPEAKER_01: Just to give everyone a lay of the land, can you start by talking a little bit about your role at Gum Gum, like what you do?
• 1:45
  SPEAKER_02: Just give a bit of context, what Gum Gum does in the industry too.
• 1:48
  SPEAKER_02: We're an ad tech company.
• 1:49
  SPEAKER_02: So our goal is to connect agencies, advertisers, people with a product or brand that they want to share.
• 1:56
  SPEAKER_02: And our job is to connect them with potential audiences, looking at websites, watching videos on the open web.
• 2:02
  SPEAKER_02: And we try to do that.
• 2:03
  SPEAKER_02: Our kind of unique angle on it is we do it without user information.
• 2:07
  SPEAKER_02: So we offer contextual pathways.
• 2:09
  SPEAKER_02: We have all these really cool products that allow us to understand the mindset that the user's in as opposed to what page they looked at yesterday, what is in their cookies, all that more nefarious material that is in that privacy space.
• 2:23
  SPEAKER_02: And so when we're having, when we're making these connections, we have this ad exchange where we're talking to publisher partners like websites, CNN.com, Yahoo, things like that.
• 2:34
  SPEAKER_02: And we're talking to brands through a bunch of different channels.
• 2:37
  SPEAKER_02: And all these conversations are producing a lot of data and logs, and we're making a lot of decisions on this.
• 2:42
  SPEAKER_02: And so we want to operationalize that as a business.
• 2:44
  SPEAKER_02: And that's where my role comes in.
• 2:46
  SPEAKER_02: I oversee three teams: the data engineering team, which takes these conversations that are logged by all of these computing systems and brings them into a data warehouse space where we can start to operationalize them, model them, store them.
• 3:00
  SPEAKER_02: We have a BI team that takes this stored data in our data warehouse and operationalizes it for business use, uses it for financial reporting, troubleshooting integrations, all sorts of really cool explorations of material.
• 3:13
  SPEAKER_02: And then we have a data science team that does the same thing.
• 3:15
  SPEAKER_02: We're trying to operationalize that data, but do it automatically.
• 3:18
  SPEAKER_02: So using machine learning and AI to leverage that data and feed it back to our ad exchange and make smarter and smarter decisions.
• 3:25
  SPEAKER_02: So that's been my role.
• 3:26
  SPEAKER_02: And Airflow is a big part of all that because there's a lot of data movement, a lot of things moving around.
• 3:30
  SPEAKER_02: And so obviously, we need that system that can trace that and track that and manage that top down as well.
• 3:37
  SPEAKER_01: Actually, that's interesting because that's exactly my second question to you, which is how do you typically use Airflow in your organization today?
• 3:44
  SPEAKER_01: What are the use cases that you have?
• 3:46
  SPEAKER_01: You mentioned that there is a data science team.
• 3:49
  SPEAKER_01: So maybe are you running some machine learning models with using Airflow?
• 3:54
  SPEAKER_01: What kind of use cases do you have today?
• 3:56
  SPEAKER_02: Our foundation is a pretty typical data platform.
• 3:59
  SPEAKER_02: Out of the box, we have the process data every hour, every day, maybe weekly, maybe monthly.
• 4:05
  SPEAKER_02: And so that's where our Airflow Foundation started.
• 4:07
  SPEAKER_02: We had all these, we had a variety of scripts and different systems that were running agiles.
• 4:11
  SPEAKER_02: We had some other orchestration tools.
• 4:13
  SPEAKER_02: And then we unified all those into Airflow.
• 4:15
  SPEAKER_02: And a big bulk of our Airflow footprint is just bringing data in, moving it from S3 into data warehouses, launching Spark jobs that process data and curate it and moving that around.
• 4:26
  SPEAKER_02: So that's really our foundation.
• 4:28
  SPEAKER_02: And then once we built that foundation, we found we can do a lot of other cool things.
• 4:31
  SPEAKER_02: So, yeah, as you mentioned, our data science team uses it to not only train their models, but orchestrate their own data movement.
• 4:37
  SPEAKER_02: And so it's basically any automation that touches data, and even some automations that don't touch data find their way into Airflow, we would say.
• 4:44
  SPEAKER_02: So at this point, it's dual purpose for data orchestration, but also just general automation within the company as well.
• 4:51
  SPEAKER_01: Wow.
• 4:52
  SPEAKER_01: And out of curiosity, you mentioned Spark to process the data.
• 4:57
  SPEAKER_01: I imagine that you trigger Spark from Airflow to process the data in Spark.
• 5:02
  SPEAKER_01: You are not doing it in Airflow, right?
• 5:03
  SPEAKER_01: Is that correct?
• 5:04
  SPEAKER_02: Yeah.
• 5:04
  SPEAKER_02: So I think as a rule, we tend not to do too much data processing actually internally within Airflow.
• 5:10
  SPEAKER_02: We try to offset that to other systems.
• 5:12
  SPEAKER_02: So usually Spark, our primary data processing unit, our warehouses can do ELT, those spaces sometimes too.
• 5:19
  SPEAKER_02: We actually started experimenting with Kubernetes as well.
• 5:22
  SPEAKER_02: So wanting Kubernetes jobs to just do data processing.
• 5:25
  SPEAKER_02: That way, our Airflow footprint, we don't have to really worry about scaling our cluster for an increase in surge.
• 5:31
  SPEAKER_02: We want our airflow footprint to be really stable and consistent.
• 5:35
  SPEAKER_01: Yeah, the reason why I'm asking you this is because it's actually very common to have people processing data in Airflow, but also using uh other tools like Spark.
• 5:44
  SPEAKER_01: So that's interesting.
• 5:45
  SPEAKER_01: And as you mentioned, it looks like Airflow is pretty critical uh for you at GameGum.
• 5:49
  SPEAKER_01: How do you manage on a daily basis whenever a task failure happens or whenever you have a DAG issue, all of those wonderful things that we have in production, like how do you handle that?
• 6:02
  SPEAKER_01: How do you approach that?
• 6:03
  SPEAKER_02: It is absolutely critical.
• 6:05
  SPEAKER_02: And I think that's why originally, when I first joined Gum Gum, how we handled it is we raised a critical alert and we said, hey, this thing has failed, go and fix it.
• 6:14
  SPEAKER_02: And everyone drops what they're doing and goes and finds it.
• 6:17
  SPEAKER_02: Sometimes maybe we even miss that alert because alert fatigue is a real thing, and we're getting there's a lot of airflow failures.
• 6:24
  SPEAKER_02: We run over 200, 250,000 tasks a month.
• 6:28
  SPEAKER_02: And so what we wanted to do is handle it in a more organized and developer-friendly approach.
• 6:33
  SPEAKER_02: So what we've started doing is taking those critical failures and routing them to more actionable spaces.
• 6:40
  SPEAKER_02: So for us, that's been Jira has probably been the most important kind of pathway for handling production failures.
• 6:46
  SPEAKER_02: So we use Jira for all of our development work, right?
• 6:48
  SPEAKER_02: Every morning when a developer logs on, they can check this is what I'm working on today.
• 6:53
  SPEAKER_02: And so we thought let's let's funnel our airflow task failures to that same space.
• 6:56
  SPEAKER_02: And the underlying hypothesis is if we put these failures in an actionable space where developers are used to completing development work, it's going to encourage people not only just to rerun the task, make the bug fix, turn that square from red to green, but also to take a look and see if we can improve that task failure and prevent it from happening again.
• 7:16
  SPEAKER_02: And that's really been a huge shift that has helped us fix not only resolve the production issue, but fix the underlying issue, improve the system, make it more extensible and scalable going forward.
• 7:28
  SPEAKER_01: And out of curiosity, did you start using Jira right away, or did you start just getting notified by email first, maybe, or did you choose Jira right away?
• 7:38
  SPEAKER_02: No, so we probably started with more most people when we just had the default email callbacks.
• 7:43
  SPEAKER_02: And as people started missing those, we would add more.
• 7:46
  SPEAKER_02: We have a system called PagerDuty that's a vendor that rings your phone.
• 7:50
  SPEAKER_02: It's a pretty scary noise for developers.
• 7:52
  SPEAKER_02: And so we we started putting more and more critical alerts there.
• 7:55
  SPEAKER_02: And then we also tried Slack somewhere in between something that can pop up and show you something, but not necessarily wake you up if you're sleeping.
• 8:04
  SPEAKER_02: And they were all a mess.
• 8:06
  SPEAKER_02: It was it was just a lot of noise.
• 8:08
  SPEAKER_02: There were way too many things going on.
• 8:10
  SPEAKER_02: It was hard to manage the severity across three systems.
• 8:13
  SPEAKER_02: And as a manager, it was hard to actually say confidently this is the workload we're spending on it.
• 8:19
  SPEAKER_02: Because it you can't really compare number of emails to like the production work we're normally doing within Jira.
• 8:26
  SPEAKER_02: And so making that shift was important for a lot of reasons.
• 8:29
  SPEAKER_02: Yeah.
• 8:29
  SPEAKER_01: Yeah, that's true.
• 8:30
  SPEAKER_01: Keeping track on all the emails that you receive, especially when it's alerts, that's very challenging.
• 8:36
  SPEAKER_01: How is this implemented today?
• 8:38
  SPEAKER_01: Like how what features do you use in Airflow to make this possible?
• 8:42
  SPEAKER_02: Yeah, so we we use the we developed this about, I would say, two years ago.
• 8:46
  SPEAKER_02: So we implemented this originally just with the onfailure callback function at the DAG level.
• 8:51
  SPEAKER_02: And we just have built a pretty quick Jira API call in Python, call that from the on-failure callback, and then we have a severity system as well built into that uh Python function.
• 9:02
  SPEAKER_02: So when a developer is setting up a new DAG, uh if it's a really important DAG, uh they'll put in the DAG settings, they'll put that reusable class we have for the on-failure callbacks.
• 9:14
  SPEAKER_02: They'll say this is a severity one issue, and it'll trigger the appropriate response when a task in that DAG does fail.
• 9:22
  SPEAKER_01: Okay, and the the severity, how is it defined manually or is it automatic?
• 9:27
  SPEAKER_02: Like yeah, so it's defined manually when you develop the DAG, just like you set your DAG schedule.
• 9:32
  SPEAKER_02: So you're setting that the run schedule, it's just another property that goes right below that.
• 9:37
  SPEAKER_02: And it's severity one, okay, severity three, severity four, and then we build some custom logic in our reusable functions to map that to the appropriate action to be taken.
• 9:47
  SPEAKER_02: But when a new developer's setting up a DAG, they just say Jira callback function, severity one, basically.
• 9:54
  SPEAKER_01: Okay.
• 9:55
  SPEAKER_01: And just to make sure, when you said the severity, are we talking about the severity of a DAG or a specific task?
• 10:01
  SPEAKER_02: Generally at the DAG level, we we haven't really treated any tasks within a DAG, I would say, with different severities.
• 10:08
  SPEAKER_02: We try to keep our DAGs pretty focused functionally, I would say.
• 10:11
  SPEAKER_02: So is that possible?
• 10:12
  SPEAKER_02: I would think.
• 10:13
  SPEAKER_02: Yeah.
• 10:14
  SPEAKER_01: Okay.
• 10:15
  SPEAKER_01: Great.
• 10:15
  SPEAKER_01: And so actually, like how do you monitor your DAGs today?
• 10:21
  SPEAKER_01: What are the metrics maybe that you are monitoring today?
• 10:24
  SPEAKER_01: What do you have in place to make sure that things work as expected?
• 10:28
  SPEAKER_02: I think first developer views you log on to your airflow and green and red, right?
• 10:32
  SPEAKER_02: That's the basic then that can be pretty tedious, right?
• 10:35
  SPEAKER_02: And so what we try to monitor probably our favorite metric that's helped us really gauge where to spend our time in Airflow and in the platform space, spending the Airflow task retry rate.
• 10:47
  SPEAKER_02: We have a dashboard in our BI system in Looker that shows for every task, every operator type in every DAG, what that task failure rate looks like.
• 11:00
  SPEAKER_02: And essentially we just plot all of them most are around 1%.
• 11:03
  SPEAKER_02: And that task retry rate might be it failing and being manually corrected.
• 11:07
  SPEAKER_02: It might be the automated retries.
• 11:10
  SPEAKER_02: Um, I think those are the only two cases.
• 11:12
  SPEAKER_02: But essentially it means the DAG try, the task tried, something in the code failed and it had to retry at a later date.
• 11:17
  SPEAKER_02: And then when that task retry rate gets to around 5% or above, that's when we start to take a look at looking at the source cause.
• 11:26
  SPEAKER_02: There might not actually be anything failing.
• 11:28
  SPEAKER_02: It might not actually be waking up any developers.
• 11:30
  SPEAKER_02: It might be just doing that retry.
• 11:31
  SPEAKER_02: But it's an indicator to us that there's a potential for failure or there's a potential for raised wasted resources going on.
• 11:37
  SPEAKER_02: That gives us that trigger to go in and investigate and try to get that task failure rate or task retry rate back down below five.
• 11:46
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:55
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:07
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:16
  SPEAKER_01: Okay.
• 12:16
  SPEAKER_01: That's very interesting because I wonder why did you pick this metric?
• 12:21
  SPEAKER_01: Like why not another one?
• 12:22
  SPEAKER_01: Did do you remember why?
• 12:24
  SPEAKER_01: Like, how did you end up with that metric and not another one?
• 12:27
  SPEAKER_02: Functionally, we said we should know how much things are failing.
• 12:30
  SPEAKER_02: And we actually, when we started to try to implement that in Airflow and drive into that, failure is a state dependent thing.
• 12:36
  SPEAKER_02: So if something has failed it, but this time in five minutes, it should be fixed when the developer actually goes and resolves it.
• 12:43
  SPEAKER_02: And so we actually had a KPI request to start looking at task failures and it and we just thought it felt wrong.
• 12:48
  SPEAKER_02: It was like everything's always successful.
• 12:49
  SPEAKER_02: We know things are failing, not being captured by this metric.
• 12:53
  SPEAKER_02: And so we just looked at it in a little bit more airflow way what really defines failure.
• 12:57
  SPEAKER_02: And it's actually retry because every task in our Airflow instance is destined to succeed eventually as soon as the failure is fixed.
• 13:05
  SPEAKER_02: But yeah, I think the big part is we we we knew that a large percentage of our time is spent just rerunning tasks and investigating tasks.
• 13:12
  SPEAKER_02: A large reason why we have issues with stakeholders waiting for data is one task just didn't retry fast enough, or one task was retrying 10 times for every 30 minutes and we didn't get notified that it was taking that long.
• 13:26
  SPEAKER_02: So I think there were a lot of business issues, but uh the general essence is we wanted to understand when are things broken, basically.
• 13:32
  SPEAKER_02: Yeah, and since doing that, actually, we've been able to generalize this to other systems where the retry concept isn't necessarily as important.
• 13:39
  SPEAKER_02: So, like in Spark, for example, there's task retries within Spark, and we've actually found those to be a really good indicator of when a job failure is gonna happen because Spark retries a lot when there's like memory issues, there's balance issues, when there's connectivity issues.
• 13:53
  SPEAKER_02: And even though the job might succeed, it'll take a little bit longer.
• 13:56
  SPEAKER_02: There's gonna be a little bit more resources used, and that task retry is a symptom of a eventual cause or of an eventual problem that that might appear.
• 14:04
  SPEAKER_01: Yeah, that's a very good one.
• 14:05
  SPEAKER_01: And how do you actually fetch the data?
• 14:08
  SPEAKER_01: How do you fetch the task retries today with Airflow?
• 14:11
  SPEAKER_02: Yeah, I think we we just have a job that runs every hour and it hits the I think it's the task instances endpoint in the Airflow API.
• 14:22
  SPEAKER_02: It's a pretty easy metric to fetch, and then we store it in our warehouse and the rest is just modeling.
• 14:28
  SPEAKER_02: Yeah.
• 14:29
  SPEAKER_01: Okay.
• 14:30
  SPEAKER_01: Just just for your information, because I think that might be useful for you.
• 14:34
  SPEAKER_01: I saw that in 2.10 there is a new endpoint for the API that will allow you to get also the logs, the log details for every retry.
• 14:44
  SPEAKER_01: So that might be useful for you if you want to know what the task is.
• 14:48
  SPEAKER_02: Yeah, and even beyond that, too, the I think the event endpoint has some really interesting details.
• 14:53
  SPEAKER_02: We've been trying to figure out how to model aside that as well.
• 14:56
  SPEAKER_02: But yeah, there's a lot more detail to go into and find.
• 14:58
  SPEAKER_02: But just starting with task retry is great, a great indicator.
• 15:02
  SPEAKER_01: Definitely.
• 15:03
  SPEAKER_01: Do you have a dashboard in place?
• 15:05
  SPEAKER_01: Like, how do you like what does it look like?
• 15:08
  SPEAKER_02: Yeah, it's we have a dashboard, it's just a horizontal bar chart that shows task retry by DAG.
• 15:13
  SPEAKER_02: And so we have all our DAX.
• 15:15
  SPEAKER_02: We have the sizes weighted by the number of task runs that actually go.
• 15:18
  SPEAKER_02: So we can indicate maybe it's just five or six tasks that are running and one failed.
• 15:22
  SPEAKER_02: That's a really high retry rate.
• 15:23
  SPEAKER_02: And then we have a threshold line, and then anything to the right of the threshold line is something we look at.
• 15:28
  SPEAKER_02: And then we can also set alerts and look or anytime.
• 15:31
  SPEAKER_02: I think every month we just get an alert that says these are the DAGs that have task retry rates above that.
• 15:37
  SPEAKER_02: We also have an operator view, which is interesting.
• 15:39
  SPEAKER_02: So instead of looking at it by DAG, we look at it by operator.
• 15:42
  SPEAKER_02: And that was actually really interesting because we saw some things that we suspected, but weren't 100% sure.
• 15:47
  SPEAKER_02: We were using the Kubernetes operator a lot in the past, and we just found it it was really unreliable.
• 15:53
  SPEAKER_02: So we we actually moved away from it, at least the way we were implementing it was unreliable.
• 15:58
  SPEAKER_02: And so we were able just to refactor the way that we used that call and move to a separate system for processing that.
• 16:05
  SPEAKER_01: Oh, that's that that's a good one.
• 16:07
  SPEAKER_01: So looking at the operators, you are able to see like which one is more reliable than the others, and then just choose to replace it.
• 16:15
  SPEAKER_01: That's yeah, it's a good idea.
• 16:17
  SPEAKER_02: Or like we could see oh, our Databricks operators performing really poorly.
• 16:21
  SPEAKER_02: Maybe we we need to spend time fixing those jobs because it's causing these upstream issues as well.
• 16:26
  SPEAKER_01: That's good.
• 16:27
  SPEAKER_01: And any is there any other, I don't know, like maybe like patterns that you are looking for, or any other metrics that you're uh looking into today?
• 16:37
  SPEAKER_02: Yeah, the uh the other big one we use pretty routinely is just task duration.
• 16:41
  SPEAKER_01: Okay.
• 16:42
  SPEAKER_02: It's a it's not quite the metric we want.
• 16:45
  SPEAKER_02: What we're trying to get a sense of is SLAs and airflow.
• 16:49
  SPEAKER_02: And so task durations, they a good way to see when our SLAs might be at risk.
• 16:54
  SPEAKER_02: So we actually look at week over week and month over month task durations to see if new features that are being introduced, new data volumes, new bugs, really anything, um, might be causing our DAGs to take longer than they were previously.
• 17:08
  SPEAKER_02: And then if there's a significant amount of change, we get a small alert.
• 17:12
  SPEAKER_02: And that helps us track.
• 17:14
  SPEAKER_02: I think the biggest thing that helps us track is someone implements a new feature, like they maybe add a couple of columns to a Snowflake query or something like that.
• 17:23
  SPEAKER_02: And unexpectedly, the the query takes a lot longer to process now.
• 17:28
  SPEAKER_02: Maybe test or something like that.
• 17:30
  SPEAKER_02: And so this it's a good, it's a really good catch for feature changes that might endanger your SLAs or your task run time.
• 17:37
  SPEAKER_02: But as I mentioned, what we're really chasing there is the metric we've actually, I would say, struggled to implement effectively.
• 17:42
  SPEAKER_02: And that's that's the DAG SLA hit rate, essentially.
• 17:46
  SPEAKER_02: Okay.
• 17:47
  SPEAKER_02: So we look a lot at landing times and things like that in the airflow, but we I wouldn't say we've mastered the ability to say whether our DAG finished exactly when we expected it to, whether it was a little early or whether it was a little late.
• 18:00
  SPEAKER_02: That's been the golden goose for us in that task timing space.
• 18:02
  SPEAKER_02: And I don't think we've perfected that yet.
• 18:05
  SPEAKER_01: It's a pretty tough topic, honestly.
• 18:07
  SPEAKER_01: Like uh being able to do it, it's it takes time, it takes a lot of time.
• 18:11
  SPEAKER_01: And actually, you just I need to go back to the SLAs because you mentioned them.
• 18:15
  SPEAKER_01: Have you ever tried to use SLAs today in Airflow, like the built-in feature?
• 18:20
  SPEAKER_02: Yeah, we tried it around the same time we started task retry rates about two years ago.
• 18:26
  SPEAKER_02: And I think we we ended up having some success with it, but it was the observability we had in the native implementation wasn't that helpful.
• 18:36
  SPEAKER_02: Um so we ended up implementing a lot of custom SLAs to circumvent it.
• 18:41
  SPEAKER_02: It was also pretty difficult.
• 18:42
  SPEAKER_02: I think the biggest difficulty was navigating the difference between task and DAG SLAs.
• 18:47
  SPEAKER_02: I think most implementations in Airflow, it pushes the SLA object towards the task where we mostly cared about the DAG SLA.
• 18:56
  SPEAKER_02: Once all tasks are fed, what does the timing look like?
• 19:00
  SPEAKER_02: I think we ended up did finding a solution for that, but we ended up in that time period just implementing a separate process for being able to monitor those SLAs.
• 19:09
  SPEAKER_01: Okay.
• 19:10
  SPEAKER_01: Yeah, SLAs are the reason why I'm asking you this is because I know that SLAs are pretty how can I say that?
• 19:15
  SPEAKER_01: Fun to use in Airflow today.
• 19:18
  SPEAKER_01: So they like it's a challenge to implement them, and I know that they are getting improved, but it takes time.
• 19:24
  SPEAKER_01: But and actually, you mentioned earlier that obviously stakeholders are uh always pinging um data engineers whenever the data is late.
• 19:34
  SPEAKER_01: Do you have anything in place today to actually automate this process?
• 19:39
  SPEAKER_01: Like whenever you have a DAG that it is in failure, for example, and stakeholders are expecting their data to arrive at a specific time.
• 19:47
  SPEAKER_01: Do you have something in place to communicate that automatically, or is it a data engineer that tells, hey, sorry, we have an issue, data can be late?
• 19:55
  SPEAKER_02: Like today, it's still alerts, and then the data engineer has to translate that.
• 19:58
  SPEAKER_02: What we're working towards, though, is we're actually working with this, are they like a data catalog tool, data democratization tool called Atlin.
• 20:07
  SPEAKER_02: Okay, they we have full lineage using the open lineage features in Atlin from airflow to the end BI assets that people look at.
• 20:16
  SPEAKER_02: And so what we're trying to do right now is make those connections and when a SLA is missed for an airflow DAG, propagate that downwards onto the asset page for that specific dashboard or explore.
• 20:29
  SPEAKER_02: And so on every asset that's affected by a DAG, we can say, hey, data's running late for today and put a little warning sign.
• 20:36
  SPEAKER_02: So that's something we're we're trying to build out and experiment.
• 20:38
  SPEAKER_02: It's it's still really early.
• 20:39
  SPEAKER_02: We're not exactly sure what it looked like.
• 20:41
  SPEAKER_02: But yeah, the the manual process of having data engineers update is it's it's very error prone.
• 20:46
  SPEAKER_02: It's very unreliable.
• 20:47
  SPEAKER_02: It's it's a lot of work.
• 20:48
  SPEAKER_02: So we we we're we're definitely that's probably one of our top priorities in terms of automation today.
• 20:54
  SPEAKER_01: Yeah.
• 20:55
  SPEAKER_01: No, that that's this is something challenging.
• 20:57
  SPEAKER_01: Also, like I saw that some companies are implementing data contracts as well.
• 21:02
  SPEAKER_01: So I don't know if you have, do you have any thought on that?
• 21:05
  SPEAKER_02: Yep, that's a big part of that.
• 21:06
  SPEAKER_02: So we're trying to do data contracts within Atlan as well.
• 21:09
  SPEAKER_02: That way, your SLA, your data contract is all right there when you're looking at your dashboard or your explorer, your table in Snowflake, you'll have that data contract as well as whether or not you're actually hitting that data contract side by side.
• 21:24
  SPEAKER_02: Okay.
• 21:25
  SPEAKER_01: Okay, okay.
• 21:26
  SPEAKER_01: Great.
• 21:27
  SPEAKER_01: I guess maybe one last question.
• 21:29
  SPEAKER_01: Is there any I don't know, is there anything that in Airflow you would love to see in the future releases?
• 21:38
  SPEAKER_01: Is there any feature that you will say, oh, finally I have it in Airflow?
• 21:43
  SPEAKER_01: Is there anything like that?
• 21:45
  SPEAKER_02: Yeah, I think the we've talked about it a little bit here, but the whole idea of a truly encompassed, comprehensive SLA.
• 21:53
  SPEAKER_02: We're using a lot of external processes to build these SLAs, and we have our external data contracts.
• 21:59
  SPEAKER_02: It would be really cool if these were native features within within that Airflow environment.
• 22:03
  SPEAKER_02: When you build a DAG, you can say, this is my expectation for how it's going to perform, not just in terms of timing, quality, and other factors, reliability, things like that.
• 22:13
  SPEAKER_02: And able to track just natively within Airflow when you're violating those SLAs in that data contract.
• 22:19
  SPEAKER_02: That's probably the biggest thing.
• 22:20
  SPEAKER_02: I think for us, everything else I would say that we want is coming to fruition.
• 22:24
  SPEAKER_02: Like we spent a lot of time on that Jira on failure callback.
• 22:27
  SPEAKER_02: I know now there's like notifications that are released through and so like those features are really cool.
• 22:33
  SPEAKER_02: And like a lot of these other things I think that have been on our wish list for a while are actually getting built and released now.
• 22:39
  SPEAKER_02: Yeah, I would say the SLS feature is probably the number one thing.
• 22:42
  SPEAKER_01: Okay.
• 22:43
  SPEAKER_01: I hope it's gonna come soon then, because I'm pretty sure that you're not the only one asking for uh for better SLAs.
• 22:49
  SPEAKER_01: Um, and yes, there is definitely uh this feature called uh notifiers.
• 22:54
  SPEAKER_01: Um, I strongly everyone to uh to use it.
• 22:57
  SPEAKER_01: It's uh it's actually a pretty nice way to uh get notified without you reinventing the wheel, the logic.
• 23:03
  SPEAKER_01: You can get your notifier, for example.
• 23:05
  SPEAKER_01: I think there is one on Jira, I'm not sure, but I'm sure that there is uh one on Slack.
• 23:09
  SPEAKER_01: But yes, so take a look at notifiers, that might be helpful.
• 23:13
  SPEAKER_01: Great.
• 23:14
  SPEAKER_01: Thank you so much, Brendan.
• 23:15
  SPEAKER_01: That was so nice.
• 23:16
  SPEAKER_01: I guess what is the best way for folks to get in touch with you?
• 23:21
  SPEAKER_02: Yeah, I would just reach out on LinkedIn, just connect with me on LinkedIn, we're there.
• 23:26
  SPEAKER_02: Uh also check out the Gum Gum Tech blog.
• 23:28
  SPEAKER_02: There's a lot of what we talked about is on there.
• 23:30
  SPEAKER_02: There's like an article, eight KPIs that every data platform needs, things like that.
• 23:34
  SPEAKER_02: But yeah, reach out on LinkedIn, is I would say.
• 23:37
  SPEAKER_01: Awesome.
• 23:38
  SPEAKER_01: Awesome.
• 23:38
  SPEAKER_01: Thank you again.
• 23:39
  SPEAKER_01: I'm sure that people will enjoy that episode.
• 23:41
  SPEAKER_01: And I hope to see you very soon.
• 23:43
  SPEAKER_01: Great.
• 23:43
  SPEAKER_01: Thanks, man.
• 23:45
  SPEAKER_01: Thank you.
• 23:45
  SPEAKER_01: Bye.
• 23:47
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:54
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 23:58
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.