Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff
Transcript
- 0:00SPEAKER_01: Nurses will go on and create a profile with their background and experience, a set of preferences and like where they're looking for a job, what type of shifts they're open to, even things like down to the scrub color that they're looking for in their position, which I totally understand.
- 0:12SPEAKER_01: And then we also have recruiter users who will come onto the platform and try and place nurses in jobs that they have.
- 0:18SPEAKER_01: And so they can search similarly for nurses.
- 0:21SPEAKER_01: And we also use a lot of models to suggest jobs nurses might apply for or find like potential matches for jobs that the recruiters can then follow up with.
- 0:48SPEAKER_00: Hi everyone, and welcome to the Data Flowcast.
- 0:51SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Max Kilhuff, who is a lead data engineer at Vivian Health.
- 0:58SPEAKER_00: Max, how are you doing today?
- 0:59SPEAKER_01: I'm pretty good.
- 1:00SPEAKER_01: Yeah, just been in the process of doing some research to upgrade us to Airflow 3.
- 1:06SPEAKER_01: I've been going through that, and then we're hiring a new engineer soon to do a data engineer backfield job.
- 1:13SPEAKER_01: So I was hoping to have them like do a I forget what it's called.
- 1:17SPEAKER_01: You have a DBT competitor, right?
- 1:19SPEAKER_00: We have Cosmos, which is our use.
- 1:23SPEAKER_00: So for Airflow and DBT together.
- 1:25SPEAKER_01: Yes, exactly.
- 1:26SPEAKER_01: So yeah, I've been scoping out some tickets for that.
- 1:28SPEAKER_01: Yeah, it's exciting.
- 1:29SPEAKER_01: I've been looking at some of the features of Airflow 3, and it seems like exciting some stuff we might be able to use.
- 1:36SPEAKER_00: Great.
- 1:36SPEAKER_00: Yeah, sounds like there's a lot going on.
- 1:38SPEAKER_00: I definitely want to dig into the Airflow 3 piece a little more later.
- 1:43SPEAKER_00: But yeah, we'll go ahead and dive in for our listeners for today's podcast.
- 1:48SPEAKER_00: We're gonna discuss how Vivian Health uses Airflow for MLOps and regulatory compliance.
- 1:53SPEAKER_00: I think we'll touch on Airflow 3 as well.
- 1:56SPEAKER_00: So we'll get into all of that.
- 1:57SPEAKER_00: But yeah, first let's start with some background.
- 2:00SPEAKER_00: Can you tell everybody what Vivian Health does?
- 2:02SPEAKER_01: Vivian is the nation's largest healthcare talent marketplace slash placement tool for travel nurses.
- 2:07SPEAKER_01: Nurses will go on and create a profile with their like background and experience, a set of preferences and like where they're looking for a job, what type of shifts they're open to, even things like down to the scrub color that they're looking for in their position, which I totally understand.
- 2:21SPEAKER_01: And then we also have recruiter users who will come onto the platform and try and place nurses in jobs that they have.
- 2:28SPEAKER_01: And so they can search similarly for nurses.
- 2:31SPEAKER_01: And we also use a lot of models to suggest jobs nurses might apply for or find like potential matches for jobs that the recruiters can then follow up with.
- 2:39SPEAKER_01: Yeah.
- 2:40SPEAKER_00: Okay.
- 2:41SPEAKER_00: Very cool.
- 2:41SPEAKER_00: And what does your team do?
- 2:43SPEAKER_01: So we're the data team at Vivian.
- 2:46SPEAKER_01: We provide essentially anything that's built off of a data lake.
- 2:50SPEAKER_01: So we use DBT heavily to build models that then analysts use.
- 2:54SPEAKER_01: Some of it is like used in our front-end like web app.
- 2:57SPEAKER_01: We work pretty cross-functionally with both like core engineering and our customer success managers who interface with these recruiting agencies or user platform.
- 3:05SPEAKER_01: We also do like BI and Looker and create dashboards, publish reports.
- 3:10SPEAKER_01: And then we also handle our ML models, the training and deployment.
- 3:15SPEAKER_01: And those are, I touched on this earlier, those are things that like suggesting similar jobs after a nurse applies to one based on their preferences and a number of other parameters.
- 3:24SPEAKER_01: And then we'll also have potential matches that get generated for jobs with recruiters.
- 3:29SPEAKER_00: Very cool.
- 3:30SPEAKER_00: And where does Airflow fit into all of that?
- 3:33SPEAKER_01: So Airflow is our core orchestration tool.
- 3:36SPEAKER_01: We have a number of DAGs that are used for everything from retraining ML models to importing data from external API endpoints, updating our own API endpoints.
- 3:49SPEAKER_01: We have job speed that we generate every hour based on new analytics data coming in.
- 3:54SPEAKER_01: Yeah.
- 3:55SPEAKER_01: So really like a lot of the whole ETL pipeline essentially, we use it to like synchronize things.
- 4:01SPEAKER_01: We run in DBT cloud right now.
- 4:02SPEAKER_01: I was talking about a little earlier about wanting to maybe change over to Cosmos, but we will trigger like different DBT cloud jobs and then have like contingency based on down to like the individual models.
- 4:14SPEAKER_01: So if we do a run that selects like, say, like 100 models, we can like use Airflow to look at like we can parse the DBT manifest and see that a model succeeded or failed, and then run downstream tests as a result.
- 4:27SPEAKER_01: And the same thing for importing data from say uh Stitch or uploading it since all of this stuff can be triggered with Airflow.
- 4:35SPEAKER_01: And you can also do things like run like SQL statements that will process data from Snowflake and load it into say a Postgres table, which is typically pretty hard to do, but by building out like yep, leveraging like a custom operator we built, are we able to do that?
- 4:50SPEAKER_00: Okay.
- 4:50SPEAKER_00: Lots of stuff with Airflow.
- 4:52SPEAKER_00: It sounds like both on the ETL and the ML ops side.
- 4:55SPEAKER_00: Before we get into more of the use case specific stuff, how are you running Airflow for your team?
- 5:00SPEAKER_00: Do you use a managed service or do you run it yourself?
- 5:03SPEAKER_01: Yeah, we host it with Astronomer.
- 5:05SPEAKER_01: We use AWS Code Deploy to actually execute the build.
- 5:08SPEAKER_00: Okay.
- 5:09SPEAKER_00: Interesting.
- 5:09SPEAKER_00: Yeah, very cool.
- 5:10SPEAKER_00: Okay.
- 5:11SPEAKER_00: Well, I would love to dig in a little more to your ML ops use case since this is a growing use case with Airflow, especially over the past couple of years.
- 5:20SPEAKER_00: Tell me more broadly how you use Airflow to manage your ML models.
- 5:25SPEAKER_01: So at a very high level, we get data that we want.
- 5:27SPEAKER_01: The data that we use to train the ML models sits in Snowflake.
- 5:31SPEAKER_01: The data, I think I talked about this earlier.
- 5:32SPEAKER_01: The data comes from all kinds of sources and we process it.
- 5:35SPEAKER_01: And then we use AWS SageMaker to manage our models.
- 5:40SPEAKER_01: So we will push training data and trigger the model run, and then take that model and deploy it to the API endpoints that we also host in AWS.
- 5:52SPEAKER_01: And all of that is orchestrated in Airflow.
- 5:54SPEAKER_01: So everything from like getting the data, processing it into a DBT model, prepping it for export to SageMaker, triggering the training, like even testing to make sure that the models like adequately tuned.
- 6:07SPEAKER_01: We're currently actually working on refactoring our process for how we're going to handle feature drift.
- 6:12SPEAKER_01: Then once we have like the models deployed, we split to do A B testing.
- 6:16SPEAKER_01: And then once we find a model we want to deploy, we'll update Airflow and have that be the like current version.
- 6:25SPEAKER_01: And then once a week, typically, we will like then retrain that same model.
- 6:30SPEAKER_01: So it's kind of a higher level overview.
- 6:32SPEAKER_01: Yeah, and all that happens in a separate day called the weekly ML retraining pipeline.
- 6:37SPEAKER_00: Nice, very aptly named.
- 6:39SPEAKER_01: Yes.
- 6:39SPEAKER_00: Cool.
- 6:40SPEAKER_00: So it sounds like Airflow is pretty integral from the very beginning to the very end of that whole process.
- 6:47SPEAKER_00: I'm curious, just in the logic of choosing Airflow, not everybody thinks about Airflow as an ML ops tool since it's historically ETL has been its bread and butter.
- 6:57SPEAKER_00: But what do you think about Airflow makes it well suited for ML ops?
- 7:02SPEAKER_01: I just love the versatility of it.
- 7:04SPEAKER_01: It honestly reminds me of like earlier internet days where like you just had to like hack a lot of different services together.
- 7:10SPEAKER_01: But like the sky was really the limit.
- 7:12SPEAKER_01: And terms of like a lot of products now, I feel like kind of are restricted and can be like a little bit of a wall garden.
- 7:18SPEAKER_01: But the robust like operator library of Airflow and then the capability, if you have like a data engineer, you don't have to be like an all-star programmer to go in and extend an operator class.
- 7:29SPEAKER_01: You can do it, and it's it's even easier now with cursor.
- 7:33SPEAKER_01: So you can essentially do anything in whatever order you want, like the limits of using, for example, right?
- 7:39SPEAKER_01: Snowpark has its own model deployment and training like architecture, but you're like limited only to using Anaconda packages.
- 7:48SPEAKER_01: You can't like use anything else.
- 7:50SPEAKER_01: So that's because it's living in their virtual environment.
- 7:53SPEAKER_01: That was like a point of frustration.
- 7:55SPEAKER_01: And they also have, again, with that one, for example, like you can't really use, say, pandas efficiently to do data manipulation.
- 8:02SPEAKER_01: They have like their own proprietary version, which it does work, but it's just another like, why am I learning another Python package just to use do data manipulation as like one specific place when everywhere else?
- 8:12SPEAKER_01: And I'm using pandas.
- 8:13SPEAKER_01: So yeah, the fact that you can like just run any arbitrary Python code really, I think people like Python because of how versatile it is.
- 8:20SPEAKER_01: And you can just have any Python file and run it in Airflow.
- 8:23SPEAKER_01: That's very powerful, especially if you have a more technical team.
- 8:30SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed Airflow service designed for the modern data stack.
- 8:39SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
- 8:51SPEAKER_02: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
- 8:59SPEAKER_00: On my team at Astronomer, we always joke that like it's both the good thing and the bad thing about Airflow.
- 9:05SPEAKER_00: Is like if you can do it in Python, you can do it in Airflow, and that's both great and also can be dangerous sometimes.
- 9:12SPEAKER_01: Yeah, you have to have like pretty robust co-review and make sure that the people on your team are like, you know, yeah, it is like a tool for engineers.
- 9:20SPEAKER_01: I will say that it's a little tricky for like, for example, if you were gonna just go set up Stitch, okay, you like run a few commands in like snow sites, and then it fits itself together, and then you could say use DVT, but then going and like say spinning up like a Docker container and hosted service and getting airflow running.
- 9:40SPEAKER_01: We use like the managed service, but I know you need to have a different worker setup and redundancies, and so all that stuff takes a lot more knowledge than doing something that's more out of the box.
- 9:48SPEAKER_01: It is definitely like a tool for engineers, it's a really great tool for engineers.
- 9:52SPEAKER_00: I've been using it for years now, and yeah, best practices are important for sure, even with a managed service that helps out with some of those things.
- 10:01SPEAKER_01: The documentation for Airflow is pretty good, and astronomer has their own docs too that kind of expand on some of the more esoteric sections.
- 10:09SPEAKER_00: Yes, those are maintained by my team.
- 10:11SPEAKER_01: Oh, really?
- 10:11SPEAKER_01: Yeah, it's great.
- 10:12SPEAKER_01: I've not met a lot of companies that will like improve docs, like for I don't know, as just the public can use them.
- 10:19SPEAKER_01: I don't think you even need like a an astronomer account to read your docs.
- 10:23SPEAKER_01: Pretty good.
- 10:24SPEAKER_01: Yeah, yeah.
- 10:24SPEAKER_01: Anyone if you want to learn, they have like the Apache docs can be a little confusing sometimes.
- 10:29SPEAKER_01: Highly recommend astronomers docs.
- 10:31SPEAKER_00: Yeah, our stuff is there for the whole community.
- 10:33SPEAKER_01: So cool, cool.
- 10:34SPEAKER_00: Well, I'm curious also if you have on the topic of airflow for ML ops, is there anything that you think could be improved?
- 10:42SPEAKER_00: We spoke a little bit about the need for best practices, so that's certainly relevant there as well.
- 10:46SPEAKER_00: But anything else that you think in the project kind of feature-wise, that would make it better for that use case.
- 10:51SPEAKER_01: For ML ops specifically, I guess yeah, when we use AWS to retrain.
- 10:57SPEAKER_01: Do you know like of other users who like actually use like the say like us do the training within the container itself, or do they typically use like external services or another like remote machine that they can use in AWS?
- 11:10SPEAKER_00: Good question.
- 11:11SPEAKER_00: I think it does happen.
- 11:13SPEAKER_00: It's not as common.
- 11:15SPEAKER_00: I think most of the time folks are relying on some sort of like external service that's specifically for model posting and retraining, and they're orchestrating with Airflow.
- 11:25SPEAKER_00: Yeah, I have heard of people doing it directly within Airflow if it's their own models, but I don't think it's common.
- 11:32SPEAKER_01: That's the only thing I can think.
- 11:33SPEAKER_01: I don't know.
- 11:34SPEAKER_00: Okay, yeah, that's totally fine.
- 11:35SPEAKER_01: If it's useful with that specifically, I guess it is like you do have to look past these like files.
- 11:41SPEAKER_01: There's a lot of like file passing back and forth.
- 11:43SPEAKER_01: I think I talked about this earlier where we explained how it worked.
- 11:46SPEAKER_01: I don't know.
- 11:46SPEAKER_01: That's it.
- 11:47SPEAKER_01: That's not like nothing too crazy that I can think of that would like but that I guess from like a security or like data processing concern, like that could be something to cut down on.
- 11:56SPEAKER_01: And I guess it depends on where you're hosting like the source data from, but I guess mostly that would all live inside of AWS.
- 12:04SPEAKER_01: And correct me if I'm wrong, but the astronomer like services hosted in AWS too, right?
- 12:08SPEAKER_00: You can choose with our managed service where you want to run.
- 12:12SPEAKER_01: Ours is I remember, but yeah.
- 12:13SPEAKER_01: So I don't know.
- 12:14SPEAKER_01: I guess it depends.
- 12:15SPEAKER_01: Like we do import some data from BigQuery, and Google and Amazon do not make it easy to like say, like, okay, can I please just like export this BigQuery data to this S3 bucket where all of my other stuff lives?
- 12:26SPEAKER_01: And they're like, Yeah, we can't let you do that.
- 12:28SPEAKER_01: You're gonna have to build like parallel infrastructure for uh importing BigQuery data.
- 12:33SPEAKER_01: I can see if someone was like wanting to get combined data from Google and other sources, like running into problems with that, maybe, but I don't know.
- 12:42SPEAKER_01: This all seems like edge cases overall.
- 12:44SPEAKER_00: Yeah, totally valid to say it works great.
- 12:47SPEAKER_00: And it does work great.
- 12:48SPEAKER_00: I know that's how it is.
- 12:49SPEAKER_01: Pretty easy for the what we've for what we've built out.
- 12:52SPEAKER_00: Yeah, cool.
- 12:54SPEAKER_00: I would love to also touch briefly on your use case of using Airflow to anonymous user data, which I think you mentioned briefly at the beginning.
- 13:02SPEAKER_00: Tell me a little bit more about that and why it's important for your business.
- 13:05SPEAKER_01: Yeah, so where's like regulatory requirements for deleting expired anonymous data?
- 13:11SPEAKER_01: So, like when you have users, there's two different circumstances.
- 13:14SPEAKER_01: So, like people can request other data anonymized, or they're just like some person who comes to the website and they haven't made an account or logged in.
- 13:20SPEAKER_01: So we have they're like inherently anonymous to us.
- 13:23SPEAKER_01: So I believe what is it?
- 13:24SPEAKER_01: It's every I'm looking at the DAG right now, 1825 days.
- 13:30SPEAKER_01: So I believe it's like that's five years, right?
- 13:32SPEAKER_01: Approximately I think it's five years.
- 13:33SPEAKER_01: Yeah, five years.
- 13:34SPEAKER_01: So every five years, you have to get rid of this expired anonymous data.
- 13:39SPEAKER_01: So we built this DAD that essentially it just runs once a day, and like any data that like fits that criteria, it will drop from both the so we we clean it out of Snowflake and from we also have it run uh commands for our front end like Postgres database.
- 13:55SPEAKER_01: And yeah, that's really easy because then if like someone comes to audit us, it's like very simple to show them.
- 13:59SPEAKER_01: I don't know if you have to like really show them the Python.
- 14:01SPEAKER_01: I can give them like explain what the Python code does, but if I show them the DAG saying, look, it's executing this SQL statement on all these tables where we have everything.
- 14:09SPEAKER_01: So that's pretty cool to just be able to say, okay, this is restore anonymous data, then set up this DAG and then forget about it.
- 14:16SPEAKER_01: Like it just runs every day, and we don't have to have any ongoing maintenance or processes because that's what we were doing previously, just like once a month, is like, okay, I go in and execute this.
- 14:26SPEAKER_01: So that's been really good from a compliance and governance perspective.
- 14:31SPEAKER_00: Yeah, that makes sense.
- 14:32SPEAKER_00: We spoke about this a little bit at the very beginning of the episode, and it's not maybe directly related to this use case, but one of the things that we've seen with Airflow three because of the remote execution feature that we talked about is that a lot of companies in these highly regulated industries are moving to Airflow three so that they can make use of that, especially if they have regulations that say the data needs to stay in a specific place, and that's not necessarily where they're running Airflow.
- 15:01SPEAKER_00: Is that relevant for you or something you're looking at as you consider upgrading?
- 15:05SPEAKER_01: Definitely.
- 15:06SPEAKER_01: So, Vivian, it is like healthcare tech, but we don't work with any protected health information.
- 15:10SPEAKER_01: My old company had a ton of PHI, and just from a like development perspective, it was very frustrating to meet all those regulatory requirements.
- 15:19SPEAKER_01: And we were also using Airflow as well.
- 15:21SPEAKER_01: This was a number of years ago.
- 15:23SPEAKER_01: So there was no Airflow 3.
- 15:24SPEAKER_01: I remember I don't want to get too deep into the weeds, but there was like a ton of pain points with that for authentication and like local development, especially was really hard, like having to like essentially recreate like a ton of mock data or anonymize existing data every time you wanted to test something.
- 15:39SPEAKER_00: Okay.
- 15:40SPEAKER_00: Yeah, cool.
- 15:41SPEAKER_00: That totally makes sense.
- 15:42SPEAKER_00: I can see where that will be really helpful for a lot of folks going forward.
- 15:46SPEAKER_00: In the interest of time, Max has been really interesting talking about this, but I always like to end with a standard final question for folks.
- 15:53SPEAKER_00: And that is what would you most like to see from the Airflow project in the near future?
- 15:57SPEAKER_01: I might sound silly if this isn't something that already exists, but we really like the ability.
- 16:01SPEAKER_01: Like I built like an extension to some of our operators that allow us to like essentially like tag people in a Slack channel.
- 16:07SPEAKER_01: So you just set up like, okay, this is our like the Slack has some like web hooks you can set up, and then you can like map users to specific tasks.
- 16:14SPEAKER_01: There's a little bit of friction where you have to pull the like every user in Slack has like a unique hash ID and have like a you know, just a dictionary to look at that together.
- 16:22SPEAKER_01: I don't know.
- 16:23SPEAKER_01: So that is something that would be cool to like be able to have because to promote like engineering ownership over like certain things.
- 16:31SPEAKER_01: That way I don't have to like let's say you have someone who's you know, me leading the data team.
- 16:35SPEAKER_01: I see something breaks in Airflow and I have to like, oh okay, paying a specific person, like it can automatically do that.
- 16:41SPEAKER_01: I don't know.
- 16:42SPEAKER_01: That was like something we built out here internally, and it's been pretty helpful.
- 16:46SPEAKER_01: So I don't know, maybe see if people want to do that.
- 16:48SPEAKER_01: That would be cool.
- 16:49SPEAKER_01: Yeah, I haven't gone to play around with the UI too much in Airflow 3, but it does look a lot better.
- 16:54SPEAKER_01: Yeah, the Airflow 2 like UI was definitely showing its age and to the point, I think my favorite little I say favorite, but if you had like procedurally generated task names and they like got too big, they would take up the whole screen and you wouldn't really be able to see like the grid without like doing with messing around in it.
- 17:12SPEAKER_01: And that was just like this is clearly like just some CSS bug, but pretty frustrating.
- 17:17SPEAKER_01: So I don't know.
- 17:18SPEAKER_01: There's just like a lot of I guess like the main thing was like just like quality of life improvements that make airflow.
- 17:26SPEAKER_01: I think we talked about this earlier about things, it was like a very technical thing for engineers.
- 17:30SPEAKER_01: I think like removing some of that friction and there's a lot of low-hanging fruit, things I just discussed would go a long way towards addressing that.
- 17:38SPEAKER_01: Just kind of like bringing it up to the standards of like a product like Stitch or something.
- 17:44SPEAKER_01: I know it's never going to be there because they're obviously very different, but there are just like a few things, like for example, like the way the airflow variables are stored, having that a lot of people I know still like use the UI for that.
- 17:54SPEAKER_01: And there's not like it really an industry standard for like we use, for example, like AWS parameter store.
- 17:59SPEAKER_01: I know there's a number of other solutions, but yeah, maybe a little more like standardization there.
- 18:04SPEAKER_01: That's it, really.
- 18:05SPEAKER_00: Totally fair.
- 18:06SPEAKER_00: Sometimes it's those little things that make a big difference, and certainly a lot of improvements like that came in the Airflow 3 release.
- 18:13SPEAKER_01: Yeah, I'm excited to try it out.
- 18:14SPEAKER_01: It looked great.
- 18:15SPEAKER_01: Yeah, I downloaded it at Snow Summit this year.
- 18:18SPEAKER_00: So Okay, cool.
- 18:20SPEAKER_00: Yeah, very cool.
- 18:21SPEAKER_00: All right, Max, thank you so much for coming on.
- 18:24SPEAKER_00: It's been great chatting with you.
- 18:26SPEAKER_00: What is the best way for folks to get in touch with you?
- 18:28SPEAKER_01: You can find me on LinkedIn, Maxwell Kalehuff, C-A-L-E-H-U-F-F.
- 18:33SPEAKER_01: And if you have any questions about Vivian or data engineering, uh, my email is m kalehuff.
- 18:39SPEAKER_01: So yeah, m Kalehuff, my last name at Vivian.com.
- 18:43SPEAKER_01: But yeah, please shoot me an email, add me on LinkedIn.
- 18:45SPEAKER_01: Yeah, I love talking about this stuff.
- 18:47SPEAKER_01: I could be here for another hour.
- 18:50SPEAKER_00: We will throw those in the show notes.
- 18:52SPEAKER_00: I'm sure we have another hour of stuff to talk about, but I do have a hungry dog who is ready for something for his afternoon snack.
- 18:59SPEAKER_00: He's gonna get very bothersome here in a minute.
- 19:02SPEAKER_00: So I'll go take care of him.
- 19:04SPEAKER_00: But thank you again for coming on.
- 19:06SPEAKER_00: It's been really great speaking with you.
- 19:07SPEAKER_01: You too, Fenton.
- 19:08SPEAKER_01: Have a great day.
- 19:11SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
- 19:18SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
- 19:22SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.