Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building a Data-Driven HR Platform at 15Five with Guy Dassa
Transcript
- 0:00SPEAKER_02: Airflow is actually a central piece of our stack for our data platform.
- 0:04SPEAKER_02: So we use it to define the DAC, direct the cyclic graphs that execute the uh data ingestion and data transformation processes for us.
- 0:12SPEAKER_02: And then also the reverse ETL, the reverse ETL to our kind of destination databases where that feed into the product.
- 0:25SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
- 0:34SPEAKER_00: Hi everyone.
- 0:35SPEAKER_00: Welcome to the Data Flowcast.
- 0:37SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Guy Dassa, who's the CTO at 15.5.
- 0:44SPEAKER_00: Guy, how are you doing today?
- 0:45SPEAKER_02: I'm doing great, Kenton.
- 0:46SPEAKER_02: How are you doing?
- 0:47SPEAKER_00: That's great.
- 0:48SPEAKER_00: Yeah, I'm good.
- 0:49SPEAKER_00: I'm enjoying the last little bit of fall weather, or so I guess summer weather, the earliest bit of fall weather.
- 0:55SPEAKER_02: Yeah, it's finally getting a little chill in New York as well.
- 0:58SPEAKER_00: Okay.
- 0:58SPEAKER_00: Yeah.
- 0:59SPEAKER_00: I heard New York is having uh had some kind of hot days, even recently.
- 1:04SPEAKER_02: We haven't had rain in like a month plus.
- 1:07SPEAKER_02: Yeah, it's been pretty dry.
- 1:08SPEAKER_02: Yeah.
- 1:08SPEAKER_02: Okay.
- 1:09SPEAKER_02: Hopefully it changes later this week.
- 1:11SPEAKER_00: Yeah, that's a long time.
- 1:13SPEAKER_00: I think I'm always happy when the transition into fall happens, even though Seattle summers are usually pretty mild.
- 1:18SPEAKER_00: This one hasn't been bad, but um yeah.
- 1:21SPEAKER_02: The leaves are starting to fall at least.
- 1:23SPEAKER_00: We're seeing there.
- 1:24SPEAKER_00: I'm staring at this tree out my window that's just started to turn in the last couple of days.
- 1:28SPEAKER_00: It's really pretty.
- 1:29SPEAKER_00: Yeah, I love that.
- 1:31SPEAKER_00: Awesome.
- 1:31SPEAKER_00: Let's go ahead and dive in.
- 1:33SPEAKER_00: For our listeners for today's podcast, we're going to discuss how 15.5 is using Airflow to support their evolving data platform.
- 1:41SPEAKER_00: Uh so to start with an introduction.
- 1:43SPEAKER_00: 15.5, I understand, is in the HR tech space and provides a performance management platform.
- 1:49SPEAKER_00: Can you describe in more detail what 15.5 does and how your product works?
- 1:54SPEAKER_02: Yeah, certainly.
- 1:55SPEAKER_02: Yeah, we're we present ourselves as a command center for HR leaders to be able to measure and diagnose and take action on anything related to their people.
- 2:05SPEAKER_02: And this company exists for 13 years and over the years accumulated a whole ecosystem of products that help HR leaders and managers, including things like performance reviews, engagement surveys, tools that help help managers be more effective, like having a one-on-one, or setting goals, or giving recognition, high five, we call it, um, or doing a check-in to see how teams are team members are doing.
- 2:30SPEAKER_02: So all these tools combined help managers and HR leaders measure how people are doing and then take action to help improve those what we call outcomes of performance, engagement, and retention.
- 2:43SPEAKER_02: And I would say over the last year, we've started adding more data capabilities to make this product, providing true visualization of how these outcomes are showing up via dashboard and insights and much more data-driven.
- 2:55SPEAKER_02: And that's becoming much more exciting because we know there's high demand for major leaders to have a seat at the table and be able to show their executive team how these tools are actually helping them in improving the effectiveness and the productivity and retaining top talent in a company.
- 3:09SPEAKER_00: Yeah, definitely.
- 3:10SPEAKER_00: Those tools are super important for sure.
- 3:13SPEAKER_00: I remember I've been at Astronomer for about four years now.
- 3:17SPEAKER_00: And when I first started, we were, I don't know, maybe 30, 40 people and everything was managed in spreadsheets.
- 3:22SPEAKER_00: At the time, we were too small to have any kind of HR software implemented and have since kind of grown much past the point of needing that.
- 3:30SPEAKER_00: I've seen several things implemented here, but it helps a lot.
- 3:32SPEAKER_00: It's definitely really important, especially as you start to manage teams.
- 3:37SPEAKER_02: Yeah, and it's been evolving, I would say.
- 3:39SPEAKER_02: In the past, a lot of companies used to use what we call system of records or HRIS as just forums and workflows of people filling performance reviews or engagement surveys, and then someone will take a look at it and it will all be in Excel, as you're saying.
- 3:52SPEAKER_02: But it has been changing.
- 3:54SPEAKER_02: We're seeing a lot more trend, there's much more awareness around academically doing it right of how to measure engagement, what is engagement, and how to do calibrations properly, how to do that at scale, how to do to compare things and provide like aggregated insights of a people of an organization.
- 4:11SPEAKER_02: And that's uh that's something that you need tools to do that.
- 4:13SPEAKER_02: You can't just do everything in Excel, doesn't scale.
- 4:16SPEAKER_02: Um, yeah.
- 4:17SPEAKER_00: Yeah, totally.
- 4:19SPEAKER_00: So you mentioned a little bit already the evolution to having your product be more data driven, and I understand as well you're incorporating AI into that.
- 4:28SPEAKER_00: Can you talk about that transition and then we'll talk about where Airflow fits into that as well?
- 4:32SPEAKER_02: Yeah, definitely.
- 4:33SPEAKER_02: So as I as I mentioned, 15.5 exists for 13 years and there's so much data, right?
- 4:38SPEAKER_02: There's like hundreds of thousands of engagement surveys done on the platform and same with performance reviews.
- 4:44SPEAKER_02: And so we've taken the approach of let's activate that data, let's help, let's use this data in order to create uh meaningful predictive insights, right?
- 4:54SPEAKER_02: We measure a lot of things like engagement of people.
- 4:57SPEAKER_02: Uh huh.
- 4:57SPEAKER_02: Why not use that as a way to predict turnover?
- 5:00SPEAKER_02: Or we're measuring engagement via engagement surveys.
- 5:04SPEAKER_02: Why not provide guidance to HR leaders?
- 5:06SPEAKER_02: What are the problems they need to tackle?
- 5:09SPEAKER_02: After engagement surveys are done, typically people look at the questions, maybe they'll look at question by question, but it's hard to create action plans from them.
- 5:17SPEAKER_02: Okay, so this is I have 50 questions.
- 5:19SPEAKER_02: What do I do with all these questions?
- 5:20SPEAKER_02: How do I prioritize which of these questions need to be addressed?
- 5:23SPEAKER_02: Which of them are influencing more the score of engagement?
- 5:26SPEAKER_02: And so that kind of brought the need for us to get all this data into this centralized data platform so that we can start mining it and produce insights from it.
- 5:36SPEAKER_02: And a lot of these insights are generated by using AI models that either help with predicting something or help with prioritizing something.
- 5:44SPEAKER_02: And that's that's really how we're activating all these different attributes or pieces of data that we have across our different products into this kind of centralized data platform that we build.
- 5:55SPEAKER_00: Yeah, that makes sense.
- 5:56SPEAKER_00: And I suppose obvious question, but so all of your customers are using the platform and generating all of this data, and then that's all hosted by your company, and then the insights are provided by you.
- 6:07SPEAKER_02: That's correct.
- 6:07SPEAKER_02: Yeah, everything.
- 6:08SPEAKER_02: So because we we have the obviously the data confidentiality is extremely important to us.
- 6:13SPEAKER_02: So everything happens within our own environment.
- 6:16SPEAKER_02: Data is actually all our model training.
- 6:18SPEAKER_02: We sanitize data.
- 6:20SPEAKER_02: So there's no nothing that is personally identifiable information that is included.
- 6:24SPEAKER_02: And that's how we generalize the models.
- 6:26SPEAKER_02: And then, of course, we run in in our runtime environment, models are running as part of our regular customer data processing efforts that occur automatically.
- 6:34SPEAKER_00: Got it.
- 6:34SPEAKER_00: Makes sense.
- 6:35SPEAKER_00: And so as you already stated, that results in a lot of data that you've generated over quite some time of having done this.
- 6:42SPEAKER_00: So tell me a little bit about where Airflow fits into that.
- 6:45SPEAKER_00: I assume you use Airflow to manage all of that data and orchestrate some of those insights.
- 6:50SPEAKER_02: Yeah, Airflow is actually a central piece of our stack for our data platform.
- 6:55SPEAKER_02: So we use it to define the DAG, direct the account graphs that execute the uh data ingestion and data transformation processes for us.
- 7:03SPEAKER_02: And then also the reverse ETL, the reverse ETL to our kind of destination databases where that feed into the product.
- 7:11SPEAKER_02: And in addition, we also use Airflow to implement enrichment DAGs.
- 7:15SPEAKER_02: So these are the DAGs that provide additional, these are where the AI models come into the picture.
- 7:20SPEAKER_02: And so we primarily use it as an orchestrator for the different, I would say, uh transform data transformations via query.
- 7:27SPEAKER_02: So we use DBT as the tool for actually running the queries.
- 7:32SPEAKER_02: And Airflow is orchestrating the different DAGs, the DAG steps that transform the data from the lending zone all the way to a golden data set.
- 7:43SPEAKER_02: And so we have the ODS, and eventually we get to the golden data sets that we then reverse CTL into our product database.
- 7:51SPEAKER_02: We use Airflow, we use DBT, we also use Lambdas to trigger some of our AI models.
- 7:58SPEAKER_02: So we use DBT to trigger, so we use Airflow operators to trigger DBT calls that execute lambda calls to run some of our AI models and then reach the data.
- 8:10SPEAKER_00: So using Airflow for like kind of your bread and butter ETL, your reverse ETL, and then we often refer to it as like an LLM ops.
- 8:19SPEAKER_00: So managing the data required to use LLMs to get those analytics results out.
- 8:25SPEAKER_00: So yeah, that's a lot.
- 8:27SPEAKER_00: And I understand you have implemented Airflow relatively recently in this journey of evolving your data platform.
- 8:35SPEAKER_00: So you just mentioned Airflow being used for quite a lot of different things within your ecosystem.
- 8:42SPEAKER_00: Tell me more about how you chose Airflow and what you think it has benefited for those different use cases.
- 8:49SPEAKER_02: Yeah, first and foremost, it's been very extensible for us.
- 8:52SPEAKER_02: It's based on Python.
- 8:53SPEAKER_02: It's really easy to build uh wrappers around it and to wrappers to existing implementations.
- 8:58SPEAKER_02: We've actually been able to build Airflow DAGs, and people with no knowledge were able to spin up and you DAG in 15 minutes.
- 9:05SPEAKER_02: And that's been like incredible.
- 9:08SPEAKER_02: Yeah, that's been incredibly awesome for us because we are not we're not a company that traditionally has built data platforms, and so there's a lot of, I would say, knowledge gaps among our team.
- 9:17SPEAKER_02: And so it's been like really easy to train people to use it.
- 9:20SPEAKER_02: And we have a good number of DAGs at the moment and growing.
- 9:24SPEAKER_02: And so it's been really easy to do that with Airflow.
- 9:27SPEAKER_02: And I would say the the other thing is that Airflow just has a great community and there's constant improvements happening in the platform, and so it's been very easy to use and very easy to get, I guess, to get updates and to get help from the community whenever needed.
- 9:43SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
- 9:53SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
- 10:04SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
- 10:13SPEAKER_00: Yeah, that's so great to hear.
- 10:14SPEAKER_00: Obviously, we're a little biased, but think the Airflow community is great as well.
- 10:18SPEAKER_00: And my team does a lot of work to help folks learn how to use Airflow and use it most effectively.
- 10:24SPEAKER_00: So it warms my heart to hear that you guys have found it easy to implement.
- 10:28SPEAKER_00: Um Yeah, so you mentioned Airflow's extensibility and able to plug into all sorts of different tools.
- 10:37SPEAKER_00: You mentioned DBT.
- 10:38SPEAKER_00: I'm curious to dig in on that a little more.
- 10:40SPEAKER_00: I know a lot of our listeners use both DBT and Airflow.
- 10:44SPEAKER_00: We've seen huge popularity in like Cosmos, which is the open source package for integrating DBT workflows into Airflow that Astronomer has developed.
- 10:53SPEAKER_00: Tell me more about that integration and how you use those two things together.
- 10:57SPEAKER_02: We've been, I would say, primarily using the Airflow operators to trigger DBT calls from every node in our DAG.
- 11:06SPEAKER_02: Um, and in terms of we so we call DBT and DBT run queries, and then the queries result with new data, a transformation of the on top of the data, and then which feeds into the next step in the DAG.
- 11:20SPEAKER_02: I would say it's a pretty basic implementation, pretty basic process.
- 11:23SPEAKER_02: It's been very effective for us to use dbt because it's all in SQL.
- 11:27SPEAKER_02: If I actually start in the beginning, our DAG definitions, they are in GitHub, and then and it's easy for us to whenever we make a change, automatically updates the DAG definition, and then the DAG triggers DBT, and DBT is uh our main tool for the data transformation via querying.
- 11:44SPEAKER_02: And yeah, I would say it's pretty basic in that regard.
- 11:47SPEAKER_02: That the DBT Airflow combo works really well for us for what we need.
- 11:51SPEAKER_02: And we have, I would say actually the reverse ETL is where we have more complexity because we have to derive the metrics that matter for the product from the from our kind of golden data set that we produce in the ETL process.
- 12:06SPEAKER_02: And so there is like just multiple, multiple graphs that we have to execute.
- 12:11SPEAKER_02: And so each of them has different logic and have different dependencies, and there's uh synchronization issues that we have to account for.
- 12:17SPEAKER_02: But again, it's not like a tech stack uh like anything novel in how we use the tech stack, but that has been working for us very effectively.
- 12:25SPEAKER_02: Um, yeah, even from a hardware footprint.
- 12:27SPEAKER_02: It's been a couple of pods.
- 12:29SPEAKER_02: We're running on Kubernetes with scheduler and worker pods in the same node and one more for our UI for our web server.
- 12:37SPEAKER_02: Yeah.
- 12:37SPEAKER_02: Very easy to use, very effective, very easy to configure.
- 12:41SPEAKER_00: All right, sounds like you're the perfect airflow implementation of both like really easy to run it from the infrastructure side and easy to implement from the DAG authoring side.
- 12:51SPEAKER_00: That's really great that it's been so effective for you.
- 12:53SPEAKER_00: I'm curious about you mentioned a little more complexity in terms of the reverse ETL patterns that you've implemented and synchronizing all of that.
- 13:04SPEAKER_00: Is that something, that complexity, something that you've solved on the airflow side, or is that more just with the other tools you're working in, or how have you tackled that?
- 13:15SPEAKER_02: No, I think it's more about logic of readiness because if we're running uh multiple um uh DAGs that are have dependency on each other, uh that dependency we need to make sure that uh that that that one process and it uh finishes before another one uh starts.
- 13:31SPEAKER_02: I know we can configure um uh which of the nodes in the DAG are should be asynchronous versus synchronous, but sometimes there's just um different DAGs that need to be to coordinate because one is waiting for another one to finish and and the jobs that kick them off are not necessarily happening at the same time.
- 13:49SPEAKER_02: And there's different synchronization issues that we have to handle.
- 13:52SPEAKER_02: And it's more inherently from the way our product works and when data comes in and what triggers different jobs, and it less has to do with the infrastructure or the platform itself.
- 14:04SPEAKER_00: Got it.
- 14:04SPEAKER_00: So more about the scheduling.
- 14:06SPEAKER_02: That's right.
- 14:07SPEAKER_02: Yeah.
- 14:07SPEAKER_00: Okay, yeah, interesting.
- 14:09SPEAKER_00: Have you used data sets and airflow or other features like that to help with that scheduling complexity?
- 14:16SPEAKER_02: I'm not sure, to be honest.
- 14:18SPEAKER_00: Okay.
- 14:20SPEAKER_00: Um that's one we see very frequently that just allows you to create kind of the dependencies between different DAGs in your Airflow environment, but a little smarter than just saying this DAG needs to run before this stag.
- 14:32SPEAKER_00: So they came out, it's been a couple of years now, but there's been a lot of development on the feature recently.
- 14:37SPEAKER_00: So they're one we talk about internally a lot.
- 14:40SPEAKER_00: And it has definitely opened up how easy it is to implement something like you're talking about with Airflow.
- 14:45SPEAKER_02: So yeah, I will definitely look into that.
- 14:47SPEAKER_02: I would not be surprised if we are using it.
- 14:48SPEAKER_00: I'm just they're they're pretty easy to implement.
- 14:51SPEAKER_00: And again, if you're working with Airflow more recently, sometimes we hear about people who are on like a really old version of Airflow and haven't updated in a long time that haven't come across them.
- 15:00SPEAKER_00: But I think probably the original data sets feature came out, it was probably two years ago now.
- 15:06SPEAKER_00: So they've been around for quite a while.
- 15:08SPEAKER_00: Yeah, yeah.
- 15:10SPEAKER_00: I'll see.
- 15:10SPEAKER_00: Um yeah, that's really cool.
- 15:13SPEAKER_00: You mentioned so like now you've been living with it for a while.
- 15:16SPEAKER_00: Where do you see your airflow usage going next?
- 15:19SPEAKER_00: You mentioned that you've already got a fair number of DAGs and will probably have more in the future.
- 15:24SPEAKER_00: Do you have other plans for Airflow as your products evolve?
- 15:28SPEAKER_02: Yeah, I we definitely want to use Airflow for have more better integration with uh with either MLflow or an AI platform of choice.
- 15:38SPEAKER_02: Right now, as I mentioned, we're using Lambdas to trigger our models.
- 15:42SPEAKER_02: And uh it's it's good and bad because it's more ephemeral and it's more efficient in that regard, but we know that in the future, not so far future, we'll have way more models and we would want to have more of like AI as a service integrated.
- 15:55SPEAKER_02: And so how to integrate that more easily into our airflow environment is something that we are um we're definitely going to look into.
- 16:04SPEAKER_02: Yeah, so that that's one thing that I can definitely see us doing more.
- 16:08SPEAKER_02: And in general, I would say more enrichment.
- 16:09SPEAKER_02: And so, how do we plug in more data or more yeah, I guess more data into our airflow DAGs in a way that is more uh scalable, leveraging more operators and um yeah, should just expand the usage of what we're already doing, but do more enrichment today.
- 16:26SPEAKER_02: I feel like we're more like in a skeleton uh usage, whereas we could expand to more third-party capabilities.
- 16:33SPEAKER_00: Got it.
- 16:33SPEAKER_00: Yeah, that makes sense.
- 16:34SPEAKER_00: That's cool.
- 16:35SPEAKER_00: And I think integration with more AI ML tools is something that the project in general, I think, is focusing on right now.
- 16:44SPEAKER_00: There's been a lot of work in terms of the providers.
- 16:47SPEAKER_00: You mentioned ML flow.
- 16:48SPEAKER_00: There is an ML flow provider.
- 16:49SPEAKER_00: I think it might be a little early on in its journey, as some of them are, but that's a request that we've heard quite a lot is for more of that.
- 16:58SPEAKER_00: And I know some companies are implementing their own custom ones as well.
- 17:01SPEAKER_00: That leads me nicely into my final question, which I asked to everybody who comes on, and that is what would you most like to see from development of the Airflow project in the near future?
- 17:12SPEAKER_02: Yeah.
- 17:12SPEAKER_02: So one thing that's kind of came up to us is versioning, DAG versioning.
- 17:17SPEAKER_02: You probably hear that a lot.
- 17:18SPEAKER_02: Yeah, because today I think we I think there are overrides today.
- 17:21SPEAKER_02: So it would be good if we could run old and new side by side.
- 17:26SPEAKER_02: Yeah, so that that's one thing.
- 17:28SPEAKER_02: The other thing related to the dependencies that we called out, if we could make dependencies more visible or explicit in the UI, like which DAGs are depending on which tags, that would be also very useful.
- 17:40SPEAKER_02: And the final one I would say is having more granular access controls to the DAG, like who can access what, who can see what, especially from a logs perspective.
- 17:49SPEAKER_02: Those are the kind of top three things that come to mind on things that we could use would be great if we had.
- 17:54SPEAKER_00: Yeah, interesting.
- 17:55SPEAKER_00: So DAG versioning is definitely for a long time, I think that's been the number one request that we've seen from the community.
- 18:01SPEAKER_00: It's always number one on the Airflow survey.
- 18:04SPEAKER_00: It is coming in Airflow three, which should be out spring of next year's the tentative timeline.
- 18:10SPEAKER_00: That's something everybody, myself included, is very excited about.
- 18:14SPEAKER_00: And that should, yeah, have a good benefit for you folks.
- 18:17SPEAKER_00: Yeah, I don't know much about what the RBAC plans are in terms of updates there.
- 18:24SPEAKER_00: I do know for the UI, you can with the data sets view if you haven't seen that.
- 18:31SPEAKER_00: See if you do use the data sets feature, it does give you an overview into the dependencies between DAGs as you set them.
- 18:38SPEAKER_00: I think that's the best way to do that.
- 18:40SPEAKER_02: I know there is some way to see in the DAG definition if it what it's depending on, but I think there's a view where you see all the different DAGs and where it's less explicit, but uh, yeah, perhaps datasets solves that issue as well.
- 18:53SPEAKER_00: Yeah, some of them in the datasets view was just updated quite a bit in the most recent release, which is 210.
- 18:59SPEAKER_00: Do you know what airflow version you're running?
- 19:02SPEAKER_02: I'm not sure, but I would say probably one of the recent versions because we've been using it just for the last six months or so.
- 19:10SPEAKER_00: Okay.
- 19:11SPEAKER_00: That might be something to look into.
- 19:12SPEAKER_00: Otherwise, I think UI updates happen in every Airflow release.
- 19:16SPEAKER_00: And I know there's quite a lot planned for Airflow 3, which is going to be the next big release.
- 19:21SPEAKER_00: Definitely I encourage all teams to provide that feedback to the community, uh, because I know they they take it very seriously.
- 19:28SPEAKER_00: And we've got somebody on Astronomer's staff who contributes very heavily to the Airflow UI.
- 19:32SPEAKER_00: Um, and he always loves to hear that feedback.
- 19:34SPEAKER_00: That's great.
- 19:36SPEAKER_00: Um, cool.
- 19:36SPEAKER_00: Well, Guy, it was really great having you on.
- 19:38SPEAKER_00: Thanks so much for walking through your Airflow usage.
- 19:41SPEAKER_00: Uh, and asking what's the best way for folks to get in touch with you?
- 19:46SPEAKER_02: Yeah, first of all, thank you for having me.
- 19:48SPEAKER_02: Yeah, my best way to get in contact with me is through LinkedIn.
- 19:51SPEAKER_02: Just search for Guy Dassa.
- 19:53SPEAKER_02: That's also the suffix of my that's my LinkedIn name.
- 19:57SPEAKER_02: Yeah, hook me up there.
- 19:58SPEAKER_00: Awesome.
- 19:59SPEAKER_00: Great.
- 20:00SPEAKER_00: Sounds good.
- 20:00SPEAKER_00: Uh thanks again for coming on.
- 20:03SPEAKER_02: Thank you for having me.
- 20:06SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
- 20:13SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
- 20:16SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.