Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam

Transcript

Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: One of the things that we do for our internal use case, sending data to maybe data science, we'll first do data quality checks on the data.
• 0:09
  SPEAKER_03: And then if everything looks good, then only we transform and send it to like our downstream.
• 0:18
  SPEAKER_02: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:30
  SPEAKER_00: Okay.
• 0:31
  SPEAKER_00: Hi everyone.
• 0:32
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:34
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Ashir Alam, who is a senior data engineer at Credit Karma.
• 0:41
  SPEAKER_00: Ashir, how are you doing?
• 0:42
  SPEAKER_03: Doing well.
• 0:43
  SPEAKER_03: How are you, Kenton?
• 0:44
  SPEAKER_00: I'm good.
• 0:45
  SPEAKER_00: I'm good.
• 0:45
  SPEAKER_00: Thanks.
• 0:46
  SPEAKER_00: It's so great to have you on.
• 0:48
  SPEAKER_00: We're happy to have you here.
• 0:50
  SPEAKER_00: Let's go ahead and dive in.
• 0:51
  SPEAKER_00: So for today's podcast, we're going to discuss how Credit Karma manages data quality and governance with Airflow.
• 0:57
  SPEAKER_00: I think this is an exciting topic.
• 1:00
  SPEAKER_00: But before we get into all of that, I want to start with a little bit of background.
• 1:04
  SPEAKER_00: I think a lot of folks might be familiar with Credit Karma, a pretty well-known company.
• 1:08
  SPEAKER_00: But for anybody who might not be aware or might not know the details, tell us a little bit about what Credit Karma does.
• 1:14
  SPEAKER_03: Yeah, so Credit Karma, if you think of it this way, we show great scores on the app from TransUnion, Equifax, but uh there's a lot more to that.
• 1:25
  SPEAKER_03: You have all kinds of financial products on our website, on our app.
• 1:30
  SPEAKER_03: Like you can go all the way from personal loan, home loans to insurance.
• 1:35
  SPEAKER_03: Uh, so many new things, so many exciting things.
• 1:37
  SPEAKER_03: And then uh now I think we also have connected accounts.
• 1:40
  SPEAKER_03: So you could track your net worth at in one place.
• 1:44
  SPEAKER_03: Yeah.
• 1:45
  SPEAKER_00: Yeah, very helpful to be able to see all of that disparate data in one interface.
• 1:50
  SPEAKER_00: I know lots of people who use Credit Karma uh or similar services.
• 1:55
  SPEAKER_00: So definitely a lot of value provided there.
• 1:57
  SPEAKER_00: And I would imagine a lot of data management that comes into providing all of that for folks.
• 2:02
  SPEAKER_00: So yeah, maybe that's a good lead-in to my next question, which is what does your team do?
• 2:07
  SPEAKER_03: So I think I mentioned transunion equifacts just a moment ago, but we are the gatekeepers of data getting into credit karma.
• 2:17
  SPEAKER_03: Uh, we are the primary team to ingest data from our different partners, and that includes transunion equifacts.
• 2:24
  SPEAKER_03: So name it uh credit scores, like credit data, uh holistic credit data as a whole for any member, for example, you, me, all of those things.
• 2:34
  SPEAKER_03: And then uh we will massage the data, do all kinds of transformations, make it usable for all, all of CK basically, and like all of CK, and then uh make it usable for everyone.
• 2:49
  SPEAKER_03: And also like maybe uh like do data quality, um, making sure data is clean enough uh that it can be used correctly.
• 2:59
  SPEAKER_03: So yeah.
• 3:01
  SPEAKER_00: Yeah, great.
• 3:01
  SPEAKER_00: I love your description of that you're the gatekeepers for data getting into the platform.
• 3:07
  SPEAKER_00: I think that'll be very relevant as we dig in a little more to the data quality piece a little bit later.
• 3:12
  SPEAKER_00: But tell me before that, where does airflow fit into your tech stack and to everything that you just described that your team does?
• 3:19
  SPEAKER_03: So when I joined in, we didn't have airflow at all uh within our team.
• 3:24
  SPEAKER_03: So what happened was uh we had a whole lot of Scala microservices.
• 3:29
  SPEAKER_03: Hey, you ingest data.
• 3:31
  SPEAKER_03: We didn't really care too much about data quality.
• 3:34
  SPEAKER_03: We were like, okay, some downstream teams will take care of things.
• 3:37
  SPEAKER_03: And uh that came uh that came into picture because a lot of data is PII and you want to handle all of that in a specific, you don't want to expose it to every single system, right?
• 3:50
  SPEAKER_03: Uh you want to make sure it's clean enough or PIIs are scraped, or if we have to store PII, it's in a secure database.
• 3:59
  SPEAKER_03: So the way now I think with uh everything that we have now, the way we have Airflow is uh we'll scrape off all PII.
• 4:09
  SPEAKER_03: And now we are using it to make the data available to all downstream systems who don't necessarily need any PII.
• 4:17
  SPEAKER_03: Um, that's one thing.
• 4:18
  SPEAKER_03: Uh the another use case is data quality, uh, which is I think the topic as well.
• 4:24
  SPEAKER_03: So we also end up doing data quality, dashboarding, reporting, all those things as well.
• 4:31
  SPEAKER_03: But I think mainly like data quality uh and uh making the data available to downstream systems without any uh PII data.
• 4:42
  SPEAKER_00: Okay, got it.
• 4:43
  SPEAKER_00: And that management of the data and sending it to other systems, the orchestration for that, is that primarily like ETL type pipelines, or what is that?
• 4:52
  SPEAKER_03: It is it is ETL type of pipeline.
• 4:54
  SPEAKER_03: So the data, so in the first layer, the data will land into our data warehouse.
• 5:00
  SPEAKER_03: In this case, it is um BigQuery, which is provided by GCP Google.
• 5:05
  SPEAKER_03: And then from there, uh from there on, we'll take the data, transform it into different types, maybe flatten the data if we have to.
• 5:13
  SPEAKER_03: And then all of those things are done via Airflow as an orchestrator.
• 5:18
  SPEAKER_03: And like you have custom functions, you have all of those things, but Airflow is the primary orchestrator.
• 5:24
  SPEAKER_00: Okay, interesting.
• 5:26
  SPEAKER_00: I love to hear that.
• 5:27
  SPEAKER_00: Well, let's go ahead and dive a little deeper into that data quality and governance use case.
• 5:33
  SPEAKER_00: It sounds like that's pretty central for you.
• 5:36
  SPEAKER_00: I feel like this is a bit of an obvious question given everything you've already said about what your company does and what your team does, but maybe tell me uh why is data quality and governance important for credit, Karma?
• 5:48
  SPEAKER_03: Maybe like in very lament terms, we want to make sure the score you see on app is tied to you and not a person-wide, right?
• 5:57
  SPEAKER_03: Uh, we don't want to make sure uh people like getting um cross, any kind of cross-contamination.
• 6:05
  SPEAKER_03: We want to make sure the data is clean as well as the all kinds of personalization that we are doing on your data is tied to you.
• 6:14
  SPEAKER_03: So that is that is one like piece of things.
• 6:17
  SPEAKER_03: Now we also want to make sure that hey, we have all these external providers of data.
• 6:24
  SPEAKER_03: And like in a normal case, you always want to assume that anyone can mess up.
• 6:29
  SPEAKER_03: So we want to make sure that you do the uh like we do our due diligence, we will always push people outside to do their due diligence, but we want to make sure we do our due diligence as well in any kinds of data that we are getting in.
• 6:43
  SPEAKER_03: And if people, I think people are extremely sensitive when it comes to their personal finances, which makes it even more important that we do our data quality and things.
• 6:55
  SPEAKER_00: Yeah, of course, that makes sense.
• 6:57
  SPEAKER_00: It's data that's very important for people's everyday lives and is used for really important decisions.
• 7:03
  SPEAKER_00: So uh, and I think that's interesting that you described it as because you're aggregating data from a couple of different sources, you're in kind of a unique situation where you can probably flag inconsistencies or things like that to let upstream, you know, companies or whatever know that hey, like something something has gone wrong here.
• 7:23
  SPEAKER_03: Happens all the time, like it happened last week, happened last month, it happens all the time.
• 7:29
  SPEAKER_00: I think it's great that you're doing that.
• 7:30
  SPEAKER_00: And also I'm like a little concerned to learn that it does happen all the time, right?
• 7:36
  SPEAKER_00: Not exactly what everybody wants to know, but at least at least we know that it's being managed.
• 7:41
  SPEAKER_01: So yeah, yeah, yeah.
• 7:42
  SPEAKER_00: Okay, cool.
• 7:44
  SPEAKER_00: And then you mentioned before that you are you did choose, you know, within the last couple of years that you've been um at Credit Karma to use Airflow for this use case.
• 7:53
  SPEAKER_00: What made you choose Airflow over other tools for managing your data quality in this way?
• 7:59
  SPEAKER_03: So since we started with NoTool, the the first things uh was getting a tool which was easy to use, right?
• 8:08
  SPEAKER_03: Uh Airflow with its Python syntax and everything, as well as like the basic syntax is so easy to learn uh by anyone.
• 8:16
  SPEAKER_03: I think that was the biggest driver for all of us.
• 8:20
  SPEAKER_03: One other thing that uh we thought of back in the day was we could use it for ETL, but we could also use it for data quality in a single pipeline.
• 8:28
  SPEAKER_03: For example, one of the things that we do for our internal use case, sending data to maybe data science, we'll first do data quality checks on the data.
• 8:40
  SPEAKER_03: And then if if everything looks good, then only we transform and send it to like our downstream or like vice versa, we'll transform, we'll do data quality checks and then only send it to downstream.
• 8:52
  SPEAKER_03: Uh, it is possible with other tools as well, but then with Airflow, you can do everything in one place, right?
• 8:59
  SPEAKER_03: We have also started using some other tools, but uh now we are able to complement all of them with Airflow, which makes it easier.
• 9:08
  SPEAKER_03: Like you can have one Airflow tag doing your main thing, but also like reaching out to um DPT or some other places and then uh doing things there.
• 9:18
  SPEAKER_03: So it's a very unified architecture that Airflow provides, as well as the flexibility of uh maybe picking and choosing the tools that you want for your use case, but being the main orchestrator uh for everything.
• 9:33
  SPEAKER_00: Yeah, that makes a lot of sense.
• 9:35
  SPEAKER_00: And then tell me a little bit more about how you've actually implemented the data quality framework within Airflow.
• 9:42
  SPEAKER_00: What does that look like?
• 9:43
  SPEAKER_03: Uh yeah.
• 9:44
  SPEAKER_03: So am I allowed to share screen?
• 9:47
  SPEAKER_00: No, just because uh it won't make sense for the audio.
• 9:50
  SPEAKER_00: Sorry.
• 9:50
  SPEAKER_03: Okay, okay.
• 9:51
  SPEAKER_03: Uh yeah.
• 9:52
  SPEAKER_03: So uh the way we do it is using something called DAC Factory.
• 9:57
  SPEAKER_03: Uh DAC Factory is open source library by Astronomer.
• 10:01
  SPEAKER_03: It is not by Airflow, it's an open source library that Astronomer provides.
• 10:07
  SPEAKER_03: What it does is uh you uh so Airflow as of now is Python-based.
• 10:12
  SPEAKER_03: Or like with Airflow3, we are moving to language agnostic tasks, but we'll go there in a bit.
• 10:19
  SPEAKER_03: Since it was primarily Python-based, we thought, why even have people do things via Python?
• 10:25
  SPEAKER_03: Uh instead, let's just have them define things in a YAML config file.
• 10:30
  SPEAKER_03: So what uh the way we do it is uh we'll ask a user to fill a form.
• 10:35
  SPEAKER_03: Uh so any kinds of abstraction or like them writing Python code is taken away.
• 10:41
  SPEAKER_03: Uh they know what they want, they'll fill in a form, and then that form internally uh is tied to a git GitHub executor, which ends up creating a PR on GitHub.
• 10:52
  SPEAKER_03: What that PR contains are the YAML file which will power Airflow DAG, which basically gets parsed into an Airflow DAG as a whole.
• 11:02
  SPEAKER_03: So, and then you have circle CI checks on top of that, and then uh it gets merged, and then you see the DAG on um like production and whatnot.
• 11:12
  SPEAKER_03: But uh specifically, like take sin, oh another thing, like I think because we knew that the data quality quality checks might be repetitive, like we might want one structured uh format.
• 11:25
  SPEAKER_03: We just thought, hey, uh let's plug everything through a YAML, let it have a very same structure, and then uh change where we need to.
• 11:34
  SPEAKER_03: Or based on the form, uh, we'll just uh update things uh for each data quality alert.
• 11:41
  SPEAKER_03: For example, like one alert could be for data completeness, one alert could be for data accuracy as an example, and then the GitHub executor on the back and takes care of everything.
• 11:52
  SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:02
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:13
  SPEAKER_02: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:22
  SPEAKER_00: Cool, that makes sense.
• 12:23
  SPEAKER_00: And so I think a lot of our listeners will probably be familiar with DAG Factory.
• 12:28
  SPEAKER_00: The package, as you said, is open source astronomer manages it now.
• 12:33
  SPEAKER_00: And I think used by a lot of teams who don't want to write their own Python for DAGs.
• 12:38
  SPEAKER_00: Uh, you might be the first person I've talked to who's using it for data quality specifically.
• 12:42
  SPEAKER_00: So I think that's an interesting case where, as you said, there's a repeatable pattern that you're trying to implement.
• 12:49
  SPEAKER_00: I'm curious what is happening underneath the hood.
• 12:53
  SPEAKER_00: So, like when somebody fills out the form and then that's translated into the DAG factory YAML, is that going to create like SQL check operators in the DAGs?
• 13:03
  SPEAKER_00: Or yes.
• 13:04
  SPEAKER_03: So you can define custom logic.
• 13:06
  SPEAKER_03: Uh, for example, let's say you said uh you want to do query check operator uh or like um some kind of big query operator or table is existent operator.
• 13:18
  SPEAKER_03: Uh, you can define in the form.
• 13:21
  SPEAKER_03: Uh, for example, you can do a tick for hey, I want to check if the table exists as uh first step.
• 13:27
  SPEAKER_03: So in the DAX factory YAML that gets passed at the end.
• 13:31
  SPEAKER_03: Step one might be a start the DAX.
• 13:33
  SPEAKER_03: Step two might be a check if the DAG, uh if the table exists or not.
• 13:38
  SPEAKER_03: Step three might be check null rate of a column X.
• 13:43
  SPEAKER_03: And everything uh given like we are using YAML, uh, it was easy to make it standardized and then be like, hey, yeah, everything is uh working.
• 13:53
  SPEAKER_03: And then like with after all the checks, we want to make sure alerting is set correctly.
• 13:58
  SPEAKER_03: So we had to write our custom uh Slack operator and PagerDuty operators, and then that way uh monitoring and alerting is set correctly.
• 14:08
  SPEAKER_00: Okay, that's great.
• 14:08
  SPEAKER_00: That would lead me into my next question, which is what happens when a check fails?
• 14:13
  SPEAKER_00: So are you implementing the checks within the ETL DAG and it's stopping the pipeline?
• 14:17
  SPEAKER_03: Or yeah, so in uh in two ways specifically.
• 14:21
  SPEAKER_03: So one of them being as you just said, uh in ETL pipelines, we have hey, uh we run transformations and then we run some extra checks uh as part of that.
• 14:32
  SPEAKER_03: If something failed, uh we'll just stop the pipeline, uh, meaning the data did not get into its destination from where uh downstream could have used.
• 14:41
  SPEAKER_03: And then simultaneously, Slack or Slack and email will be sent for like engineers and who all to take uh like take a look at the data and see what's going on.
• 14:54
  SPEAKER_03: So that is one case.
• 14:55
  SPEAKER_03: We do have cases where data is just sitting in the data warehouse, and then we'll also run checks on top of that uh on a set cadence, like once a day, four, every four hours, every six hours.
• 15:08
  SPEAKER_03: And then like something fails, uh, we'll just uh send HTTP Slack so that whenever like we have daily jobs which might run at 12, uh, our data quality checks will try to run by six or five to make sure that teams do have the latest and greatest data, as well as we have time to fix things on our side if something were to break at that time.
• 15:34
  SPEAKER_00: Interesting.
• 15:35
  SPEAKER_00: And so now that you've been working with DAG factory in this implementation for a while, are there any optimizations that you think you'd make to your framework or anything that you could think could be improved about DAG Factory as the tool?
• 15:48
  SPEAKER_00: I guess those are those are two separate questions.
• 15:50
  SPEAKER_00: We'll take them separately.
• 15:52
  SPEAKER_03: Yeah.
• 15:53
  SPEAKER_03: So in terms of framework, I think uh one thing I've like noted is uh oh we want to add new features.
• 16:04
  SPEAKER_03: For example, like we just added on failure callback uh for DAG as a whole to be sent to different sources, not just Slack and uh email.
• 16:16
  SPEAKER_03: Uh, we had to do some custom thing and then make sure everything is propagated everywhere, which was just a little bit of copy and paste to some XYZ place.
• 16:27
  SPEAKER_03: So maybe I want to think uh in terms of more in terms of how can I make it extensible so that I don't have to do a whole lot of copy and paste.
• 16:37
  SPEAKER_03: But in terms of that factory, uh I think initially when we did this, uh, one big challenge was uh like when I started when I was just playing around with it, one big challenge was DAGs were not getting passed correctly.
• 16:52
  SPEAKER_03: And then we I was like, hey, I made this change, I don't even see the DAG as a whole.
• 16:57
  SPEAKER_03: So and then uh I realized that cloud composer environment doesn't show a lot of details at all.
• 17:04
  SPEAKER_03: So you have to go to airflow, actual airflow thing, and then it might say a DAG is broken, but it won't say how and where, or will it say like a the syntax is broken?
• 17:15
  SPEAKER_03: So maybe if like we could do something around better error handling or like showing a uh error in a better way, I think that is one big thing.
• 17:25
  SPEAKER_03: And YAML, so I don't know how familiar people are with YAML, but it's very easy to miss one space or two spaces in a YAML file and then for everything to break.
• 17:37
  SPEAKER_03: So like maybe if you could have stricter or better YAML like validation, like YAML, uh, I think that will be one big thing as well.
• 17:49
  SPEAKER_00: Yeah, like YAML linting is always a chance.
• 17:51
  SPEAKER_00: YAMLECAR, very, very picky for sure.
• 17:55
  SPEAKER_03: Yeah, maybe one other thing is um I think we haven't reached there yet, but as we have like let's say thousands of DAX, uh, the way DAC factory does work is it will keep polling the folder for DAX every like every few minutes or whatever.
• 18:16
  SPEAKER_03: Uh but what happens is if you have thousands of DAX or like 5,000 DAG, uh, it's very easy to break that thing.
• 18:24
  SPEAKER_03: Um, there are obviously workarounds for that.
• 18:27
  SPEAKER_03: For example, instead of you having 5,000 DAG in one folder, you can just break it down by let's say 200, 300, 500 uh DACs per folder and then have one DAC factory instance manage one.
• 18:42
  SPEAKER_03: But if we can uh move everything from runtime runtime like we have right now, to if let's say compile time, uh I think we even we reduce that risk even further of the scheduler not picking up changes correctly uh for a DAC factory DAC.
• 19:00
  SPEAKER_00: Yeah, definitely.
• 19:01
  SPEAKER_00: I think with any sort of dynamic DAG generation scalability, it's always a big concern.
• 19:09
  SPEAKER_00: Yeah, a problem, something you have to tackle.
• 19:11
  SPEAKER_00: So yeah, I think uh a lot of teams deal with things like that.
• 19:16
  SPEAKER_00: And I uh it's fortunate that Airflow has so many knobs that you can tune to try and adjust and make it work that I think a lot of teams are successful with it, but you do have to know kind of I always think about like the big like airplane, you know, thing with all the like thousands of buttons to push.
• 19:34
  SPEAKER_03: Yeah, yeah.
• 19:35
  SPEAKER_03: And also like how how can you scale something to not just the current use case where you have 50 DAX?
• 19:41
  SPEAKER_03: How do you scale it to five years later?
• 19:43
  SPEAKER_03: How like the state will be?
• 19:45
  SPEAKER_00: Yeah, definitely.
• 19:46
  SPEAKER_00: Well, that's great that you're thinking forward in that way.
• 19:49
  SPEAKER_00: Uh, and appreciate you going through that use case.
• 19:51
  SPEAKER_00: I thought it was super interesting to hear about.
• 19:54
  SPEAKER_00: I think uh this looking forward conversation is a good tie-in to my last question, which I asked to all of the guests, and that is what would you most like to see from the Airflow project in the near future?
• 20:05
  SPEAKER_03: So with the last Airflow Summit, I was like, hey, I have to get my hands dirty with Airflow.
• 20:10
  SPEAKER_03: So I have started committing to the open source as well.
• 20:14
  SPEAKER_03: Uh, that's one thing uh for myself.
• 20:17
  SPEAKER_03: But one I think maybe if we do something like a make all the boring tasks by default, for example, you have retries, you have email sending, you have ownership.
• 20:29
  SPEAKER_03: If we can make everything by default, so that if I forget to set one flag, it doesn't break, hopefully, something, or there is a good handling for it.
• 20:39
  SPEAKER_03: I think that will be nice.
• 20:41
  SPEAKER_03: For example, like retries uh by default, or like alerting if we have a good hand good enough handling or by by default handling, I think that will be very helpful for everyone.
• 20:53
  SPEAKER_03: And I'm excited to see, I think, how the community is doing, going to do with Genai.
• 20:58
  SPEAKER_03: I think we have introduced human in the loop as an as a thing as well.
• 21:03
  SPEAKER_03: So I'm really looking forward to that too.
• 21:05
  SPEAKER_00: Yeah, definitely a priority for the project moving forward, more support for those types of workflows.
• 21:11
  SPEAKER_00: Uh, those are growing a lot.
• 21:12
  SPEAKER_00: So well, it's very cool to hear that you've started to contribute.
• 21:16
  SPEAKER_00: Uh, I always love hearing people.
• 21:18
  SPEAKER_01: Thank you.
• 21:18
  SPEAKER_00: Or yeah, joining the open source project.
• 21:21
  SPEAKER_00: Well, as year, it's been really nice speaking with you.
• 21:24
  SPEAKER_00: Uh, what is the best way for folks to get in touch with you?
• 21:27
  SPEAKER_03: Uh LinkedIn, uh, I think if you just look for me on LinkedIn, E S-H-I-R-H-A-L-A-M.
• 21:34
  SPEAKER_03: Uh, you'll find me there.
• 21:36
  SPEAKER_00: Great.
• 21:36
  SPEAKER_00: We'll throw that in the show notes.
• 21:38
  SPEAKER_00: Well, thank you everyone for coming on.
• 21:39
  SPEAKER_00: This has been a great discussion.
• 21:41
  SPEAKER_03: Thank you very much, Kenton.
• 21:42
  SPEAKER_03: Thank you.
• 21:46
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 21:52
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 21:56
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.