Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building an End-to-End Data Observability System at Netflix with Joseph Machado

Transcript

Building an End-to-End Data Observability System at Netflix with Joseph Machado · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: The standard pattern we have used in multiple places is you create your data set, you run some checks on it, and then you publish the data set to whatever table the end user is going to use.
• 0:12
  SPEAKER_03: So the write audit publish pattern.
• 0:15
  SPEAKER_03: At Netflix, we have a system that does that for you.
• 0:18
  SPEAKER_03: At previous companies, we used a similar pattern where you create a data set either as a Spark data frame or as a temporary table, whatever it might be.
• 0:26
  SPEAKER_03: You run some checks.
• 0:27
  SPEAKER_03: If it passes, you send it out.
• 0:29
  SPEAKER_03: If it fails, you erase an alert and fix it.
• 0:37
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:47
  SPEAKER_00: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:52
  SPEAKER_00: Join the Astronomer Roadshow for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 1:00
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:08
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:13
  SPEAKER_00: Register now.
• 1:14
  SPEAKER_00: The link is in the show notes.
• 1:18
  SPEAKER_02: All right, I'm super excited for this episode.
• 1:21
  SPEAKER_02: Hello everyone.
• 1:22
  SPEAKER_02: Welcome to the Data Flowcast.
• 1:23
  SPEAKER_02: I'm your host, Mark Lamati, and today I'm super excited because I'm joined by Joseph Machado, senior data engineer at Netflix.
• 1:31
  SPEAKER_02: How are you doing, Joseph?
• 1:33
  SPEAKER_02: It's been a while.
• 1:34
  SPEAKER_03: Hey Mark, happy to be here.
• 1:36
  SPEAKER_03: Been a long time listener.
• 1:37
  SPEAKER_03: And hello, audience.
• 1:38
  SPEAKER_03: My name is Joseph Machado.
• 1:40
  SPEAKER_03: I'm a senior data engineer at Netflix.
• 1:42
  SPEAKER_03: And I also blog at dustarddataengineering.com.
• 1:46
  SPEAKER_03: Excited to be.
• 1:46
  SPEAKER_02: Yeah, actually, even before starting the episode, I must say that if you want to learn about data engineering, I strongly recommend you to look at the Joseph website.
• 1:54
  SPEAKER_02: It's a gold mine.
• 1:55
  SPEAKER_02: And I actually would like to thank you for that website because I also learned a lot using what you published.
• 2:01
  SPEAKER_02: Thank you.
• 2:02
  SPEAKER_03: When I started Airflow, I took your course back in the day when Airflow was just becoming popular.
• 2:07
  SPEAKER_03: Yeah.
• 2:08
  SPEAKER_03: So thank you too.
• 2:09
  SPEAKER_02: That's awesome.
• 2:11
  SPEAKER_02: Great.
• 2:11
  SPEAKER_02: So in this episode, actually, you will learn a lot about how you can leverage data quality checks in your data pipelines to avoid the common pitfalls.
• 2:22
  SPEAKER_02: And it's very interesting because data quality checks are super important, but people don't really know exactly what they are, how to implement them.
• 2:29
  SPEAKER_02: It's almost like cybersecurity.
• 2:31
  SPEAKER_02: It's important, but it's really important when you have an issue.
• 2:36
  SPEAKER_02: You realize how important it is.
• 2:38
  SPEAKER_02: So I'm very excited about that episode because, again, for the listeners, you will learn more about data quality checks, what they are, what kind of data quality checks you should implement, and how to implement them using Airflow.
• 2:50
  SPEAKER_02: Alright, so before diving into the topic, can you just talk a little bit about you and your role at Netflix?
• 2:58
  SPEAKER_02: What do you do?
• 2:59
  SPEAKER_03: Yeah, for people who don't know, I've been in the data space for a little over a decade now.
• 3:03
  SPEAKER_03: Started back in the MapReduce world, Java came along the evolution.
• 3:08
  SPEAKER_03: So I've been around a while.
• 3:10
  SPEAKER_03: Right now, specifically at Netflix, I focus on certain verticals.
• 3:14
  SPEAKER_03: So Netflix has this concept of informed captains.
• 3:17
  SPEAKER_03: We primarily focus on privacy, engineering efficiency, making things easier for other engineers and my data engineers and my team.
• 3:26
  SPEAKER_03: That's what I specifically focus on.
• 3:29
  SPEAKER_03: Okay, includes data quality checks and defining data quality checks, understanding what types to run for, what type of pipelines, what type of metrics require, what type of check, how to measure the quality of your checks.
• 3:42
  SPEAKER_03: So checks on checks.
• 3:44
  SPEAKER_03: So those are some of the things I'm working on.
• 3:47
  SPEAKER_02: And do you use Airflow at Netflix today?
• 3:50
  SPEAKER_03: No, we don't.
• 3:51
  SPEAKER_03: Netflix did evaluate Airflow a while back, but I think they decided to go with the in-house system.
• 3:57
  SPEAKER_03: Similar but more YAML-based than Airflow.
• 4:01
  SPEAKER_03: But I have set up and used Airflow at a bunch of previous companies.
• 4:06
  SPEAKER_02: Yeah.
• 4:07
  SPEAKER_02: Yeah, I was asking you that question because I actually saw that I think like Netflix open sourced, or maybe they didn't open source, but I would open data source my straw.
• 4:16
  SPEAKER_02: Exactly.
• 4:17
  SPEAKER_02: Great.
• 4:18
  SPEAKER_02: So actually, can you tell me more about what kind of data pipelines do you build today?
• 4:22
  SPEAKER_02: What are your use cases?
• 4:24
  SPEAKER_03: Yeah, so I specifically work on the ads department at Netflix.
• 4:28
  SPEAKER_03: So Netflix recently launched ads a few years ago, and I'm on the data team supporting ads business.
• 4:34
  SPEAKER_03: I so just like ads as a whole, I have a specific business use case within ads that I support.
• 4:42
  SPEAKER_03: From a data pipeline perspective, most of our jobs are in Spark.
• 4:47
  SPEAKER_03: And as I said earlier, I work on engineering efficiency.
• 4:49
  SPEAKER_03: So one of my mandate is to ensure people are adopting best practices, using the right pipeline, building observability tools around the pipeline.
• 4:59
  SPEAKER_03: People have you can answer questions like what, how much, when, where, how, and why off your data and off your pipelines.
• 5:06
  SPEAKER_03: So that's the my vertical.
• 5:08
  SPEAKER_03: So enabling and also having an observability system where people can quickly answer these questions without having to dig into logs and data and all sorts of things.
• 5:17
  SPEAKER_03: So you can think of my work as combining outputs of multiple systems.
• 5:22
  SPEAKER_03: So I explode metadata, logs, some KP APIs we have internally, and putting together a system that just produces for simply said it's a dashboard that you can look at that gives you precise state of your data and workflow and your pipelines.
• 5:38
  SPEAKER_02: Okay, so I guess this is super interesting.
• 5:40
  SPEAKER_02: How like how critical would you say this dashboard is for your organization or for your team?
• 5:45
  SPEAKER_03: That's a good question.
• 5:45
  SPEAKER_03: I should probably sell myself better.
• 5:47
  SPEAKER_03: It's super critical because the team is growing super fast.
• 5:51
  SPEAKER_03: And if you see on Netflix jobboard, you're hiring a lot in the ads department.
• 5:56
  SPEAKER_03: And with so many people coming in, we are one data team and we power so many dashboards, so many UI features on the app that the ads department uses.
• 6:06
  SPEAKER_03: What's happening is the same old problem of, hey, where is this data coming from?
• 6:10
  SPEAKER_03: Hey, is this data late?
• 6:11
  SPEAKER_03: Hey, how much did this metric drop by compared to previous quarter or previous week, previous day?
• 6:17
  SPEAKER_03: And how did it fail or why did it fail?
• 6:19
  SPEAKER_03: These type of questions require a data engineer who knows the pipeline to dig in.
• 6:25
  SPEAKER_03: My mandate is to make that process easier.
• 6:30
  SPEAKER_03: So give metrics on why, where, how, when, and then figuring out, using that to figure out how to make our pipelines better and how to establish patterns, how to establish tools or utilities, whatever need be.
• 6:43
  SPEAKER_03: Yeah, so it's quite cost DLDR.
• 6:47
  SPEAKER_02: Yeah, I can feel that.
• 6:49
  SPEAKER_02: So I guess when you have a such critical dashboard that it looks like many people use to get answers to their questions, I imagine that the stakeholders want to make sure that whatever they get out of this dashboard is accurate, right?
• 7:02
  SPEAKER_02: How do you ensure that?
• 7:04
  SPEAKER_02: It's a very broad question, but how do you ensure that actually what you are doing to the stakeholders is accurate?
• 7:13
  SPEAKER_03: That that's a key question, especially with data engineering, because from what I've seen, we power a lot of revenue dashboards, a lot of not just at Netflix, but every company I've been, a lot of critical metrics for the business.
• 7:26
  SPEAKER_03: So if something looks wrong, all sorts of alerts go off everywhere, and there's like tons of meeting, incidents are open.
• 7:33
  SPEAKER_03: So you uh we really need to make sure that whatever you're presenting is accurate enough.
• 7:38
  SPEAKER_03: Sometimes you have to be exact precisely accurate, like with the compliance and finance.
• 7:43
  SPEAKER_03: Sometimes directionally accurate is fine.
• 7:46
  SPEAKER_03: For both those cases, I usually work backwards, figuring out who the end users are, how sensitive they are to issues in data.
• 7:56
  SPEAKER_03: Most of the time it's aggregated up, so it's okay to be like one point or two points off.
• 8:02
  SPEAKER_03: But if you see, oh, revenue dropped by 10%, that's gonna raise a lot of red flags.
• 8:07
  SPEAKER_03: So I usually start from who the end user is, and then the first thing I would think about is how do we ensure whatever data we are exposing to stakeholders is good enough?
• 8:19
  SPEAKER_03: Because again, as you said, testing data quality is a wide wide field.
• 8:25
  SPEAKER_03: You can test so many things, but you want to start with the critical metrics, presence of dimensions.
• 8:30
  SPEAKER_03: So that would be like, hey, is the metric variance within n person compared to yes yesterday?
• 8:36
  SPEAKER_03: Simple but very powerful.
• 8:37
  SPEAKER_03: And the other one would be are all the dimensions present?
• 8:40
  SPEAKER_03: If you have five states and suddenly one state is not showing up, something is up.
• 8:45
  SPEAKER_03: So I'll start with the basics, and a lot of times most teams don't have the basics, so you start there and then you build upon it as you see fit.
• 8:53
  SPEAKER_03: Or every use case, you start.
• 8:56
  SPEAKER_03: I usually start with the end user profile.
• 8:59
  SPEAKER_03: If it's an engineer, it's not too bad.
• 9:02
  SPEAKER_03: But if it's a business user, you need to be super careful with what you're presenting.
• 9:07
  SPEAKER_03: And then there's the downstream use case.
• 9:09
  SPEAKER_03: You cannot reverse data as easily as code because once it's out in the wild, that's it.
• 9:15
  SPEAKER_03: You can't backfill all the pipelines that depend on your data.
• 9:18
  SPEAKER_03: Yeah.
• 9:19
  SPEAKER_02: That's right.
• 9:20
  SPEAKER_02: And I guess like when you say when you say that you first you start by looking at who is the end user, does that mean that you also need to understand their business needs to better write your data quality checks?
• 9:31
  SPEAKER_03: And yeah, yeah, 100%.
• 9:34
  SPEAKER_03: I think the basic ones we always write, primary keychain, not nulls, those are standard constraint checks.
• 9:40
  SPEAKER_03: But on top of that, the metric variance is a big one, making sure whatever business uh is doing with our data set, working back from that.
• 9:49
  SPEAKER_03: For example, if our business is publishing this report to external clients, we need to really make sure that the numbers are precise because you cannot take that back.
• 9:59
  SPEAKER_03: If it's something like a quarterly report that you can wait, that someone will look at in an Excel for a week before presenting, you can be a little lax with the data quality checks.
• 10:08
  SPEAKER_03: Ideally, you should be very thorough.
• 10:11
  SPEAKER_03: You have to make trade-offs.
• 10:12
  SPEAKER_03: That would be the trade-off that I would make.
• 10:14
  SPEAKER_03: So if it's internal and there's some leeway, that's okay.
• 10:18
  SPEAKER_03: But if it's presenting to multiple stakeholders or external, you need to make sure that it's pretty accurate.
• 10:24
  SPEAKER_03: Yeah.
• 10:25
  SPEAKER_03: One of the common ways is just looking out for trends in your metrics.
• 10:31
  SPEAKER_02: Okay.
• 10:31
  SPEAKER_02: So I think you mentioned that there are like some basic data quality checks.
• 10:35
  SPEAKER_02: Can you tell us more about what are the different data quality checks that you can do actually?
• 10:40
  SPEAKER_02: Because you mentioned that there is the basic ones with okay, do you have some new values, maybe the correctness and so on?
• 10:47
  SPEAKER_02: But do you have other data quality checks?
• 10:49
  SPEAKER_02: I think you mentioned also the variance, but is there any anything else?
• 10:53
  SPEAKER_03: Yeah, yeah.
• 10:54
  SPEAKER_03: So, you know, the constraint as you said, not null unique, is in a set of predefined dimensions, relationship checks.
• 11:02
  SPEAKER_03: And then there are business checks, like, oh, there should be uh the min-max values of certain columns, things like that, and schema checks, obviously, and then variance check is basically like how much percentage was the change compared to yesterday or the previous run.
• 11:17
  SPEAKER_03: And I think another important thing these days is reconciliation check.
• 11:21
  SPEAKER_03: What that means is if you're transforming a fact table and you produce an output, you want to make sure the number of rows in the output matches the number of rows in the input.
• 11:32
  SPEAKER_03: So it's typically a simple transformation would be take a fact table, join all the dimensions, and then you do a count, some sort of aggregate at the end, you make sure that the counts are similar to the inputs.
• 11:45
  SPEAKER_03: So those are tricky because, but they're also super powerful at catching issues.
• 11:50
  SPEAKER_03: They're tricky because as your data is being processed, input changes.
• 11:55
  SPEAKER_03: So you need to make sure you're comparing to that input that was used at that state when this pipeline started, so things like that.
• 12:02
  SPEAKER_03: But reconciliation check is quite powerful.
• 12:04
  SPEAKER_03: It lets you confirm that your inputs are or your outputs correspond to your inputs.
• 12:12
  SPEAKER_03: In addition to that, you could also do input validation before starting processing.
• 12:18
  SPEAKER_03: So if your data process takes a long time, you can make sure that your join keys from the input are actually joining.
• 12:25
  SPEAKER_03: Sometimes join keys don't join, and then it either blows up or it reflers down your data set.
• 12:32
  SPEAKER_03: So you could do an input check as well.
• 12:34
  SPEAKER_03: But that's if your input is highly variable and subject to a lot of issues.
• 12:39
  SPEAKER_03: Trust but verify model there.
• 12:42
  SPEAKER_02: Okay, and like for those who are listening to the episode, I'm sure that they are super interested by data quality checks.
• 12:48
  SPEAKER_02: But as you are telling me the different checks that you can implement, I feel there is like a level of complexity depending on which one you want to implement.
• 12:56
  SPEAKER_02: So if you agree on that little exercise, can you actually uh tell me by level which checks can I implement?
• 13:03
  SPEAKER_02: Like, for example, if I start with level zero, what are the checks that I could implement?
• 13:08
  SPEAKER_02: And as I go higher, then in terms of complexity, if I want to start somewhere.
• 13:14
  unknown: Yeah.
• 13:15
  SPEAKER_03: If you're starting from the level like zero and so on, I would start with the constraint checks.
• 13:20
  SPEAKER_03: Are all the dimensions that you expect present?
• 13:23
  SPEAKER_03: Are they not null or whatever your constraints are?
• 13:26
  SPEAKER_03: And constraints usually fall into one or four primary key checks, not null checks, enum checks basically, and then relationship checks.
• 13:34
  SPEAKER_03: Does this table fully join with another table?
• 13:37
  SPEAKER_03: That's like the basic thing.
• 13:38
  SPEAKER_03: Okay.
• 13:39
  SPEAKER_03: That would be primarily for dimensions.
• 13:41
  SPEAKER_03: And then when you look at metrics, variance check is key or outlier direction, whatever you want to call it, basically looking at how it has changed over time and if there is a sudden spike or drop in a specific number.
• 13:54
  SPEAKER_03: This is a numerical number that really helps.
• 13:58
  SPEAKER_03: Still level zero for that one, or we are already on the that I would say like level zero slash one, because it's pretty critical from what I have seen.
• 14:07
  SPEAKER_03: Okay.
• 14:07
  SPEAKER_03: So many things can draw wrong, but if your numbers remain consistent, it gives you a level of guarantee that hey, this is somewhat okay.
• 14:16
  SPEAKER_03: And it catches a lot.
• 14:17
  SPEAKER_03: I have seen it catch so many different types of issues because, in a sense, you are condensing different issues into a few metrics.
• 14:25
  SPEAKER_03: That's basically what your pipeline is doing, right?
• 14:27
  SPEAKER_03: It joins and groups by.
• 14:28
  SPEAKER_03: And if you check that group by, you can see hey, something is off, and then you can dig in.
• 14:33
  SPEAKER_03: And it's not very specific, it catches a wide range of issues, but it's good in the fact that you don't expose bad data to your end user.
• 14:41
  SPEAKER_03: Okay.
• 14:41
  SPEAKER_03: Okay.
• 14:42
  SPEAKER_03: So I would say those are the primary ones you should probably start with.
• 14:46
  SPEAKER_03: And then there are like business rule checks, min-max values for specific rows, specific dimensions.
• 14:53
  SPEAKER_03: You have like default values like minus 9999 or it's 100% and level one would be row distribution check, row count distribution checks.
• 15:04
  SPEAKER_03: So basically, you will see if it's your row count remaining for all your data sets remaining within specific thresholds.
• 15:12
  SPEAKER_03: Again, similarly, it doesn't go outlier.
• 15:14
  SPEAKER_03: It's a little bit different from metric because metric you will be looking at specific numbers.
• 15:18
  SPEAKER_03: Row count is just the size of your data set over time.
• 15:22
  SPEAKER_03: Also add min max and null count checks to this.
• 15:25
  SPEAKER_03: Basically, for each data set, you will have historical information that you can compare your current run against.
• 15:32
  SPEAKER_03: And the last one would be reconciliation check, comparing output with input.
• 15:36
  SPEAKER_03: That is very difficult to do because, in a sense, you are recreating the pipeline because you have to do the joints and filters and it's hard to maintain.
• 15:44
  SPEAKER_03: So this is like the master level?
• 15:46
  SPEAKER_03: Yeah, I would say so.
• 15:47
  SPEAKER_03: There are way more, but if you are for a wide range of use cases, that's what I would say.
• 15:53
  SPEAKER_03: Yeah.
• 15:54
  SPEAKER_03: Awesome.
• 15:54
  SPEAKER_02: Okay, so now let's talk a little bit about how how your data quality checks run as a critical part of your of your pipeline.
• 16:03
  SPEAKER_02: Can you how do you actually implement them?
• 16:06
  SPEAKER_02: Uh, do you have any design pattern for that?
• 16:08
  SPEAKER_02: Can you tell us more about how they are implemented in general in your pipelines?
• 16:13
  SPEAKER_03: Yeah, the standard pattern we have used in multiple places is you create your data set, you run some checks on it, and then you publish the data set to whatever table the end user is going to use.
• 16:26
  SPEAKER_03: So the write audit publish pattern.
• 16:29
  SPEAKER_03: At Netflix, we have a system that does that for you.
• 16:32
  SPEAKER_03: At previous companies, we used a similar pattern where you create a data set either as a Spark data frame or as a temporary table, whatever it might be.
• 16:40
  SPEAKER_03: You run some checks.
• 16:41
  SPEAKER_03: If it passes, you send it out.
• 16:43
  SPEAKER_03: If it fails, you erase an alert and fix it.
• 16:46
  SPEAKER_03: Okay.
• 16:47
  SPEAKER_03: So that's the pattern.
• 16:48
  SPEAKER_03: That's like usually on the output side.
• 16:51
  SPEAKER_03: Yeah, I've also built a system that checks input.
• 16:54
  SPEAKER_03: Because we were working with multiple teams and we were having difficulty with the quality of input.
• 17:01
  SPEAKER_03: We didn't want to wait for our pipeline to run and then catch it at the output.
• 17:04
  SPEAKER_03: We wanted to alert quicker, and this will help us keep the latency down and meet our table SLAs.
• 17:11
  SPEAKER_03: Those sort of things, having an input check layer helped.
• 17:14
  SPEAKER_03: So before we start our pipelines, we will ensure that input is good enough, like primary keys, business constraint checks, schema changes, etc., depending on the pipeline.
• 17:26
  SPEAKER_03: And for pattern, we primarily use the declarative style, meaning you declare what type of checks you want, either in the JSON or like a YAML, and then the system will run it.
• 17:37
  SPEAKER_03: So you can think of this as oh, DBT checks.
• 17:40
  SPEAKER_03: They're similar.
• 17:41
  SPEAKER_03: You declare it somewhere in the YAML and it just runs it.
• 17:45
  SPEAKER_03: Similarly, great expectations, although it has multiple options.
• 17:48
  SPEAKER_03: We primarily put it on, we have used it in a way that you create your checks in a JSON, it just uses it depending on what pipeline it's running.
• 17:56
  SPEAKER_02: Okay.
• 17:57
  SPEAKER_02: And so that those checks run prior to prior to the pipeline, like they they run in a like before the data goes through the pipeline, or is it like a separated pipeline?
• 18:08
  SPEAKER_02: Like how exactly is it?
• 18:10
  SPEAKER_03: Oh yeah, that's a good point.
• 18:12
  SPEAKER_03: Typically, mostly we have two systems, right?
• 18:16
  SPEAKER_03: When it comes to data quality, one is part of the pipeline or in-band data quality checks.
• 18:21
  SPEAKER_03: These are critical for your pipeline to produce the expected output.
• 18:25
  SPEAKER_03: Things like null check, primary key check, and metric variants, those are critical.
• 18:30
  SPEAKER_03: We also sometimes have a separate pipeline, all it does is check data quality.
• 18:35
  SPEAKER_03: And these are issues that you are okay with catching light.
• 18:41
  SPEAKER_03: So things like, hey, the data has not arrived, it didn't arrive for one day out of the last seven days.
• 18:47
  SPEAKER_03: Why is that happening?
• 18:48
  SPEAKER_03: Things of that nature.
• 18:50
  SPEAKER_03: Where it's like data availability checks, maybe is a better way of phrasing it.
• 18:56
  SPEAKER_03: Sometimes we do have like separate pipeline to just to check the quality of the data, where the data quality checks may not themselves be super critical to the quality of data, but we do want to keep an eye out on it.
• 19:10
  SPEAKER_03: Yeah.
• 19:10
  SPEAKER_03: And one more thing I would like to add is whenever we run a data pipeline check, it's critical to log that somewhere.
• 19:17
  SPEAKER_03: You need to log past and failures so you can look back and see how it has changed over time.
• 19:23
  SPEAKER_03: Otherwise, you're just you're losing that information about how your pipeline is performing, how input quality is, how your output quality is, that gives a number that you can use to quantify how your system and how your team has improved over time.
• 19:37
  SPEAKER_02: Right.
• 19:38
  SPEAKER_02: Actually, that's interesting.
• 19:39
  SPEAKER_02: Yes.
• 19:39
  SPEAKER_02: Like you run your data quality checks, and for example, if you see that your checks are failing more than usually, you will be able to spot that.
• 19:48
  SPEAKER_02: And so that means maybe there is an issue somewhere else that you need to look at, right?
• 19:53
  SPEAKER_03: Yeah, basically you're looking for the outliers or seasonalities in the amount of data quality checks that are failing.
• 20:00
  SPEAKER_03: And then you either make your pipeline better, figure out a solution, and it depends on the problem.
• 20:07
  SPEAKER_03: But usually it's hey, this joint could be better.
• 20:09
  SPEAKER_03: Hey, you're coaling three fields that doesn't make any sense.
• 20:13
  SPEAKER_03: So yeah, it exposes like underlying assumptions with the way we process data.
• 20:22
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 20:32
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 20:43
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 20:52
  SPEAKER_02: We mentioned already a lot of different data quality checks, but I think we didn't talk about SLAs and also like freshness, maybe of your data.
• 21:01
  SPEAKER_02: Like if there I mean I imagine that it is two things that you also uh use quite a lot.
• 21:07
  SPEAKER_02: Can you say more about those?
• 21:10
  SPEAKER_03: Yeah, so just DLDR SLA usually we think of like when the data is available and is the data available, or are we only at 90% availability, things like that.
• 21:20
  SPEAKER_03: So for simplicity, let's just say SLA means data is available.
• 21:24
  SPEAKER_03: So that's a little bit different because SLA, you want to SLA is different from data quality in the sense that your quality of data is the correctness of data, and then there's also availability.
• 21:39
  SPEAKER_03: That's where the SLA comes in.
• 21:41
  SPEAKER_03: And that's how I think about it.
• 21:43
  SPEAKER_03: A lot of people put them in the same bucket, but I think of data quality specifically as how good your data is.
• 21:49
  SPEAKER_03: It can be late, but it is it good, and then there is also data availability, and yes, we need to use both of them.
• 21:55
  SPEAKER_03: And when it comes to SLA, it's about how many hours latency have you agreed to with your.
• 22:00
  SPEAKER_03: Stakeholders and are you meeting that?
• 22:01
  SPEAKER_03: And that's a little bit different to measure because you usually have a separate job that runs way more frequently.
• 22:08
  SPEAKER_03: Usually if it's a real-time system, you have it real time.
• 22:12
  SPEAKER_03: But usually this the SLA measuring system runs every five minutes or ten minutes.
• 22:18
  SPEAKER_03: Pipelines NR.
• 22:20
  SPEAKER_03: So every five minutes you will check, hey, is this table fresh?
• 22:23
  SPEAKER_03: Hey, is table late?
• 22:25
  SPEAKER_03: Things like that, and raise an alarm if you need to.
• 22:28
  SPEAKER_03: Most people are okay with waiting.
• 22:30
  SPEAKER_03: So you usually put SLA as like six hours or something, and you run pipeline every hour.
• 22:34
  SPEAKER_03: So you can make sure there's some time in case there's an issue.
• 22:38
  SPEAKER_03: But yeah, freshness usually is a separate job.
• 22:41
  SPEAKER_03: And it's also like data quality is what you promise your stakeholders, and data freshness is when you promise to deliver it to them in two different ways.
• 22:52
  SPEAKER_03: I don't like to mix them both.
• 22:53
  SPEAKER_03: So what, when, and then you can also think of like how use and whatnot, but let's not go into that.
• 22:59
  SPEAKER_03: But quality is what and when, and uh latency is when.
• 23:03
  SPEAKER_02: I love that.
• 23:03
  SPEAKER_02: I love that.
• 23:04
  SPEAKER_02: So just to be clear, SLAs and data freshness are the same, yeah, in the same bucket, right?
• 23:09
  SPEAKER_02: And then you have the data quality checks in another bucket.
• 23:12
  SPEAKER_02: Yeah.
• 23:13
  SPEAKER_02: Okay.
• 23:13
  SPEAKER_02: Because actually it's pretty common to see data freshness in that bucket of data quality checks.
• 23:19
  SPEAKER_03: Yeah, yeah.
• 23:20
  SPEAKER_03: I I have seen that in places.
• 23:23
  SPEAKER_02: I don't agree with that.
• 23:24
  SPEAKER_02: I think I actually think that many, I might be wrong, but I think like some data quality frameworks actually give you like data freshness checks, and it's more, I mean, it's like they are data quality checks for them.
• 23:36
  SPEAKER_03: I think of it a little bit differently.
• 23:37
  SPEAKER_03: That is like an implementation detail, right?
• 23:39
  SPEAKER_03: Great expectation has freshness checks, dbt has freshness checks.
• 23:42
  SPEAKER_03: Right.
• 23:42
  SPEAKER_03: That's an implementation detail, but how we conceptualize it with our end users may or may not be technical.
• 23:50
  SPEAKER_03: They need to be away from these implementation details.
• 23:55
  SPEAKER_03: So we can use great expectation, we can use dbt, they also do freshness, and we can do our own SQL.
• 24:00
  SPEAKER_03: Hey, it's daytime, right?
• 24:01
  SPEAKER_03: This table, right?
• 24:03
  SPEAKER_03: The way we present it for end users, it needs to be simple enough.
• 24:07
  SPEAKER_03: Like what, why, when those sort of questions, those are the that are like six W questions usually in most of business.
• 24:15
  SPEAKER_03: So trying to answer those, yeah.
• 24:18
  SPEAKER_02: Okay.
• 24:19
  SPEAKER_02: Can you tell us more about how do you uh specifically implement data equity checks, but in the context of Airflow?
• 24:25
  SPEAKER_02: Yeah.
• 24:26
  SPEAKER_02: So uh which by the way might not be that different from if we're implementing data equality checks for uh for a pipeline in general without using Airflow.
• 24:35
  SPEAKER_03: Yeah, it's basically the same everywhere.
• 24:38
  SPEAKER_03: Um it's just the infrastructure, underlying infrastructure is different.
• 24:41
  SPEAKER_03: Airflow or cron or whatever custom thing your company has, it it might vary, but the underlying architecture is pretty much the same.
• 24:50
  SPEAKER_03: However, I do say I would I would want to add when you're choosing your tools, make sure there's a good integration between the tools.
• 24:59
  SPEAKER_03: You don't want to have to build the integration yourself.
• 25:01
  SPEAKER_03: That's just a pain that you are not paid to do that.
• 25:03
  SPEAKER_03: Companies who do that are different.
• 25:06
  SPEAKER_03: So having said that, yes, I have used Airflow and grid expectations quite a bit.
• 25:10
  SPEAKER_03: That was my setup at my previous few like a startup and a company before that we were processing quite a bit of data.
• 25:18
  SPEAKER_03: I think one of the companies I was at one petabyte a week or something.
• 25:22
  SPEAKER_03: Oh wow, what a hair?
• 25:24
  SPEAKER_03: Yeah.
• 25:25
  SPEAKER_03: And what we did was we have a Spark process process the data, and then we have a get a great expectation, run checks on that data.
• 25:35
  SPEAKER_03: And then if that data frame passes, we push it out.
• 25:38
  SPEAKER_03: And it was all done through Airflow.
• 25:41
  SPEAKER_03: But the thing there though is if you keep it separate tasks, you need to write it to a table.
• 25:46
  SPEAKER_03: You cannot share Spark session because there are like separate tasks in Airflow.
• 25:50
  SPEAKER_03: So you want to make sure that they are in the same.
• 25:52
  SPEAKER_03: You usually put it in a Docker container and just deploy it.
• 25:55
  SPEAKER_03: That also helps us scale our ES Spark infrastructure instead of having to start one, stop one, and then start another, stop another for data quality check.
• 26:05
  SPEAKER_03: So we typically keep it in one as a script, and it also enables easier testing.
• 26:11
  SPEAKER_03: Side note, I am excited for the Airflow decorator to use the pipe to build pipelines, which I think is great because I like pure Python functions easy to test.
• 26:19
  SPEAKER_03: So I'm looking forward to that one.
• 26:21
  SPEAKER_03: You do.
• 26:22
  SPEAKER_03: Yeah, so typically we just have the processing, auditing, publishing, or write, audit, publish system in one script, and you just run it as a Spark, uh, Spark operator on Airflow.
• 26:36
  SPEAKER_03: Okay.
• 26:37
  SPEAKER_03: Yeah.
• 26:38
  SPEAKER_03: Or usually and deploy to EMR or Databricks or whatever we are using.
• 26:42
  SPEAKER_03: I think where Airflow helps a lot is the operational overview of your pipelines.
• 26:51
  SPEAKER_03: Is the pipeline running?
• 26:52
  SPEAKER_03: Is the pipeline running on time?
• 26:54
  SPEAKER_03: How long did it take?
• 26:55
  SPEAKER_03: I think people now have it easy.
• 26:57
  SPEAKER_03: When I started, I wrote like a Python script to look at all this stuff.
• 27:01
  SPEAKER_03: It was a pain.
• 27:02
  SPEAKER_03: It was painful.
• 27:03
  SPEAKER_03: And then when I use started using Airflow, I was like, oh my goodness, this is so easy to do.
• 27:07
  SPEAKER_03: I just have to deploy Docker and I get this uh UI with a Gantt charts and whatnot.
• 27:12
  SPEAKER_03: So I think people know it's it's easy to overlook that because a lot of companies do that, but it's amazing.
• 27:18
  SPEAKER_03: It gives a lot of insight into like time taken and ability to rerun.
• 27:23
  SPEAKER_03: So when something fails, if your quality check is not good, sometimes it alerts a lot, it fails a lot, and you have to do retries.
• 27:31
  SPEAKER_03: So setting that all up.
• 27:32
  SPEAKER_03: The pipeline patterns is where I see Airflow helping a lot.
• 27:37
  SPEAKER_03: It's like you once you said it once and you can reuse it multiple times.
• 27:41
  SPEAKER_03: So that's where I find a lot of value.
• 27:44
  SPEAKER_02: Branching as well as a good user when you have like maybe a test that fails and then you want to execute something else.
• 27:51
  SPEAKER_02: Yeah.
• 27:52
  SPEAKER_02: Is that something that you used as well?
• 27:54
  SPEAKER_03: Branching, I did not use a lot, but because the tests I typically run are like blocking.
• 28:02
  SPEAKER_03: So if it blocks, it does send an alert.
• 28:05
  SPEAKER_03: I just explicitly make it fail so people look at it.
• 28:08
  SPEAKER_03: Branching, I have used not in the context of data quality checks, but in the context of like logic, depending on time or end of day or end of quarter, you might want to do something a little bit different or send in different input parameters to the same function, things like that.
• 28:23
  SPEAKER_03: I do find branching helpful in it's it for me, it's more of a readability thing.
• 28:27
  SPEAKER_03: So if a new engineer comes in, they can look at the branching and say, hey, why do we have two different logic for the same?
• 28:33
  SPEAKER_03: And then they can investigate that.
• 28:34
  SPEAKER_03: Whereas if you had it all in one script, it's hard to understand without that visual element of a DAG helping you.
• 28:41
  SPEAKER_03: I agree.
• 28:42
  SPEAKER_02: Agree, agree.
• 28:43
  SPEAKER_02: Okay, I guess I need to ask you, and this is more I mean I'm curious about your your thoughts.
• 28:47
  SPEAKER_02: I tried to use great expectations a long time ago.
• 28:50
  SPEAKER_02: I'm not gonna lie, I tried great expectations like maybe two years ago, and I didn't like it.
• 28:57
  SPEAKER_02: I just found that it was way too hard just for writing data quantity checks, and then I discovered a soda, and I actually loved it, and I still love a soda, maybe it's different today, I don't know.
• 29:08
  SPEAKER_02: But what is your take on great expectations?
• 29:11
  SPEAKER_03: Um where do you think about it?
• 29:13
  SPEAKER_03: I actually built a framework on top of great expectations for LinkedIn where it makes things easier, basically abstracting away the difficulty.
• 29:24
  SPEAKER_03: Yeah, similar to you, I've had I use great expectations at a few companies.
• 29:29
  SPEAKER_03: It was really hard to set up, it was super hard to set up, and there are like nuanced bugs.
• 29:34
  SPEAKER_03: Yeah, I was chasing down this bug for three days where if I named the file, like an output file that great expectations produce as like a timestamp, it overrides it somehow.
• 29:46
  SPEAKER_03: I couldn't figure out what was happening.
• 29:47
  SPEAKER_03: I just added like a UUID up front and that fixed it.
• 29:51
  SPEAKER_03: But it's just like such a frustrating thing to set up.
• 29:54
  SPEAKER_03: However, once you set it up, once you have, oh, this is where our expectations are stored, this is where you want to store the results, this is where you want to use your Spark data frame, it works okay pretty much.
• 30:06
  SPEAKER_03: However, when you do upgrades, things break.
• 30:09
  SPEAKER_03: We I remember we upgraded minor version of great expectation and it the Spark data frame stopped working.
• 30:15
  SPEAKER_03: So it is quite useful.
• 30:17
  SPEAKER_03: And I have heard good things about Soda.
• 30:19
  SPEAKER_03: I haven't used it in production, it felt really easy to use.
• 30:22
  SPEAKER_03: Yeah, I think the data quality framework is ripe for the big if someone was going to build a data tool thing.
• 30:30
  SPEAKER_03: Yeah, primarily, I think these days most companies just do their own thing with like simple SQL returns yes or no.
• 30:38
  SPEAKER_03: It's not super complex, but it's also hard to evolve with a company.
• 30:43
  SPEAKER_03: So I think that's a good pain point for someone to solve.
• 30:47
  SPEAKER_02: Yeah, and like I just uh I just feel that, and maybe this is something that you that you agree on, but for some reason, data quality checks, we all agree that data quality checks are important, but I've I've spoken with so many different people saying data quality checks are important, but we don't have data quality checks in place today.
• 31:06
  SPEAKER_02: And so obviously, if you try your framework for data quality checks and you just realize how difficult it is just to set it up, you won't implement data quality checks because the reality is again like cybersecurity, you don't mind that much about data quality checks until you have data quality issues.
• 31:23
  SPEAKER_02: Same thing with cybersecurity.
• 31:25
  SPEAKER_02: So, to your point, and I would love to have your thoughts on that.
• 31:28
  SPEAKER_02: Like, why do you think everyone agrees on data quality checks are important, but they are not implemented.
• 31:37
  SPEAKER_02: Most of the time we don't implement them again.
• 31:39
  SPEAKER_03: Yeah, no, I've seen the same thing happen.
• 31:42
  SPEAKER_03: I think a hypothesis, it's a combination of two things.
• 31:45
  SPEAKER_03: In the US at least, there is lots of pressure on data engineers to deliver fast.
• 31:50
  SPEAKER_03: And data pipelining is in itself a time-consuming process.
• 31:54
  SPEAKER_03: It's not like we can just quickly build a feature and test it and deploy it.
• 31:58
  SPEAKER_03: We have to check the data, how it's changing, how it downstream impacts.
• 32:02
  SPEAKER_03: So it's it's a time-consuming process.
• 32:05
  SPEAKER_03: I wish it were not the case, but most of the time it is.
• 32:08
  SPEAKER_03: You can try to speed it up with like tools and whatnot, but at the end of the day, you have to make sure that your data is good.
• 32:14
  SPEAKER_03: And that's such a painful process sometimes, especially if you're in you don't know what you're doing with your inputs, or you don't know what upstream is doing with the inputs.
• 32:22
  SPEAKER_03: And with the mounting pressure to deliver so quickly, and the pressure causes people to just take shortcuts a lot of the time, um, although they know they have to do it, and also like in the US, you are rewarded based on outcomes, not always the quality of your work.
• 32:41
  SPEAKER_03: Uh I don't know, I don't want to get into trouble, but yeah, usually you want to just push, push.
• 32:46
  SPEAKER_03: Yeah, and people take shortcuts.
• 32:48
  SPEAKER_03: Obviously, um, I do too.
• 32:49
  SPEAKER_03: I'm not saying I'm and that causes a lot of issues.
• 32:53
  SPEAKER_03: And the reason people do realize that it's painful is when an issue happens, it just and if the issue is big enough, you get pings all over Slack, then you have to go to meetings, you have to figure out what the impact, you have to figure out all the downstream.
• 33:06
  SPEAKER_03: It's such a pain to, it's such a hassle to fix.
• 33:09
  SPEAKER_03: It's no there's no easy way to fix it, especially if your data set is used by multiple stakeholders.
• 33:15
  SPEAKER_03: So that's why people agree it's a much needed space.
• 33:18
  SPEAKER_03: Big companies that I worked at, they do have pretty good quality coverage and it's monitored pretty actively and it's taken as a big deal.
• 33:27
  SPEAKER_03: But some companies just shoot from the hip and be like, oh, I don't know.
• 33:31
  SPEAKER_03: So I think it's a cultural thing, and also the devx data pipelining in general is not the best, in my opinion.
• 33:38
  SPEAKER_03: For example, if you're developing, I don't know, a flask app, it's super easy.
• 33:42
  SPEAKER_03: You could just it's just and unit test.
• 33:45
  SPEAKER_03: Whereas in data, it's oh input, it's 20 GB, 30 GB, or 100 GB.
• 33:49
  SPEAKER_03: Output, how do you check?
• 33:50
  SPEAKER_03: You need to run it, you need to wait, and you need to do that loop.
• 33:54
  SPEAKER_03: So PLDR, that's business pressure and devx being super slow and feedback loop being super slow, and pressure to deliver is what I think the main reason for people not doing it is.
• 34:05
  SPEAKER_03: People do want to do it, but it's just I'm not sure how it's gonna change.
• 34:10
  SPEAKER_03: Yeah.
• 34:11
  SPEAKER_02: I actually I think I fully agree with with your take.
• 34:14
  SPEAKER_02: And I don't think it's actually only in the US as a French guy.
• 34:17
  SPEAKER_02: I I can tell you that even in France it was like that.
• 34:20
  SPEAKER_02: But yeah, to your point, a lot of pressure, and obviously you need to deliver and ability checks, tests, even tests.
• 34:26
  SPEAKER_02: Sometimes you don't even implement tests.
• 34:30
  SPEAKER_02: You need to deliver, so whatever it takes, right?
• 34:33
  SPEAKER_02: Deliver.
• 34:34
  SPEAKER_03: I do want to add one thing to that.
• 34:35
  SPEAKER_03: I I found this comment somewhere on Reddit.
• 34:40
  SPEAKER_03: When you look at a company, the UI engineers get a lot of recognition because it's very virtual.
• 34:45
  SPEAKER_03: Back end engineers do, but it's not as much as the front end.
• 34:49
  SPEAKER_03: And then data is like the back end of the back end.
• 34:52
  SPEAKER_03: It's like only when things go wrong you're brought in the picture.
• 34:56
  SPEAKER_03: Obviously, in the companies in certain companies is different, but as a general, that's how I see how it is, I think.
• 35:04
  SPEAKER_02: No, I fully agree with you.
• 35:05
  SPEAKER_02: And I think many data engineers will will recognize themselves in what you just said.
• 35:10
  SPEAKER_02: One last one last question to you.
• 35:11
  SPEAKER_02: What would be your recommendation if I want to uh start implementing data quality checks in my company?
• 35:17
  SPEAKER_02: Because I know that you are a big fan, and maybe I'm wrong, by the way.
• 35:20
  SPEAKER_02: Tell me if I'm wrong, but I know that you are a big fan of pure Python.
• 35:24
  SPEAKER_02: And so would you say, okay, you should go with a framework right away, or would you say start with just SQL request, maybe?
• 35:32
  SPEAKER_02: Like what is what are your recommendations?
• 35:34
  SPEAKER_03: I'm not like fully on Python or SQL, it's more about as I loosely mentioned.
• 35:40
  SPEAKER_03: I like the developer experience to be fast and smooth.
• 35:42
  SPEAKER_03: And I feel like with the IDE support, Python is easier to code.
• 35:46
  SPEAKER_03: So Spark, right?
• 35:47
  SPEAKER_03: SQL and PySpark use the same underlying engine.
• 35:51
  SPEAKER_03: Having said that, I wouldn't start thinking about underlying framework.
• 35:56
  SPEAKER_03: I would start with who's your user, how are they using it?
• 36:00
  SPEAKER_03: Is it okay if there's no data quality checks?
• 36:03
  SPEAKER_03: That's the best case scenario.
• 36:04
  SPEAKER_03: You don't need any more code.
• 36:05
  SPEAKER_03: But if it is not okay, think about the problems that they will face if they use the wrong data.
• 36:12
  SPEAKER_03: Are they presenting to leadership with wrong numbers?
• 36:14
  SPEAKER_03: That's going to be really bad.
• 36:16
  SPEAKER_03: And from there, you think about what types of data quality checks you need to use.
• 36:21
  SPEAKER_03: And if you have a framework, just use that.
• 36:23
  SPEAKER_03: Most of the time it's okay.
• 36:24
  SPEAKER_03: And as you as we briefly discussed, most tools are meh in this space or no.
• 36:29
  SPEAKER_03: So if you have something, just use that.
• 36:31
  SPEAKER_03: If you don't have something, I mean at this point, just use like a YAML-based like Soda or something of that nature.
• 36:38
  SPEAKER_03: Do not go for great expectations unless that's your primary project, like data quality is your primary project because it will take a lot of time.
• 36:46
  SPEAKER_03: Go for like an easy access tool at first, maybe.
• 36:50
  SPEAKER_03: The SE scale, you obviously need to evolve, but that would be a good starting point.
• 36:54
  SPEAKER_03: Whatever is easily available, it can just be SQL scripts to begin with, and you can just move that over to DBT if you're using DBT or SQL operator if you just want to keep it pure Airflow.
• 37:06
  SPEAKER_03: Okay.
• 37:07
  SPEAKER_03: That would be advice.
• 37:09
  SPEAKER_02: Yeah, I I think I think that's great recommendations.
• 37:12
  SPEAKER_02: Don't I don't remember the expression for that, but don't choose what's the you know, start small basically.
• 37:18
  SPEAKER_03: Yeah, there's a saying get it working, get it right, make it beautiful.
• 37:22
  SPEAKER_03: Or some or get it right, get it fast, and get it make it beautiful, or something like that.
• 37:27
  SPEAKER_02: Uh I say, yeah, but that's awesome.
• 37:30
  SPEAKER_02: That's awesome.
• 37:31
  SPEAKER_02: Thank you so much, uh Joseph.
• 37:32
  SPEAKER_02: Honestly, that was a super interesting episode.
• 37:34
  SPEAKER_02: And that's the first time I think in the podcast that we actually cover data quality checks.
• 37:39
  SPEAKER_02: We know that it is a critical topic.
• 37:41
  SPEAKER_02: So thank you for uh covering that.
• 37:43
  SPEAKER_02: I guess like for the people that would love to stay in touch with you, or maybe they if they have any questions, how can they reach out to you?
• 37:51
  SPEAKER_03: Um, yeah, primarily via LinkedIn DMs and also commenting on my website at startdataengineering.com.
• 37:59
  SPEAKER_03: I try to write articles that are actionable.
• 38:02
  SPEAKER_03: So, like code or how can you use X or how do you do Y compared to more abstract stuff?
• 38:10
  SPEAKER_03: Yeah.
• 38:11
  SPEAKER_03: So that's to reach me.
• 38:13
  SPEAKER_02: Again, Joseph didn't pay didn't pay me for saying that, but I recommend you to take a look at his website.
• 38:18
  SPEAKER_02: Thank you.
• 38:19
  unknown: Thank you.
• 38:20
  SPEAKER_02: I love it.
• 38:21
  SPEAKER_02: I love your website.
• 38:23
  SPEAKER_02: Awesome.
• 38:23
  SPEAKER_02: Thank you so much, Joseph.
• 38:24
  SPEAKER_02: That was nice to have you here, and I wish you the best.
• 38:27
  SPEAKER_03: Yeah, thanks for having me.
• 38:29
  SPEAKER_03: I'll see you again sometime.
• 38:31
  SPEAKER_03: See you.
• 38:32
  SPEAKER_03: Take care.
• 38:32
  SPEAKER_03: Bye-bye.
• 38:36
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 38:42
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 38:46
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.