Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Managing travel platform data with Airflow at Headout

Transcript

Managing travel platform data with Airflow at Headout · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: You're listening to the Data Flowcast, the podcast about HD Airflow and the world of data and AI around it.
• 0:07
  SPEAKER_00: We are excited to invite you to orchestrate every day for a free online conference on September 16th.
• 0:14
  SPEAKER_00: Engineering data from Lyft RAM and WIC.
• 0:18
  SPEAKER_00: I don't care how they are shipping production AI workflows on AI core for the new Astronomer Certified AI Orchestration Fundamental exam.
• 0:28
  SPEAKER_00: Attendees get a discount code for a free certification.
• 0:32
  SPEAKER_00: A$150 value.
• 0:34
  SPEAKER_00: Go to astronomer.link slash data minus flowcast minus OE.
• 0:39
  SPEAKER_00: Link in the show notes.
• 0:41
  SPEAKER_00: Now let's get into the show.
• 0:43
  SPEAKER_01: Alright.
• 0:44
  SPEAKER_01: Hi everybody.
• 0:45
  SPEAKER_01: Welcome.
• 0:46
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:47
  SPEAKER_01: I'm your host, Mark Romati, and today I'm super excited because I'm joined by Marim Ralini, Data Platformat, head out.
• 0:55
  SPEAKER_01: And uh first thing first, how are you doing today?
• 0:58
  SPEAKER_02: I'm doing great, very excited to be here.
• 1:00
  SPEAKER_02: Um, but I've like love looking forward to having this conversation with you again.
• 1:05
  SPEAKER_01: Same, likewise, I'm super happy to have you here.
• 1:08
  SPEAKER_01: Um, so what's gonna what we're gonna discover in today's podcast is really interesting because we'll have a quick tour of how Head Out, which is a travel platform, um, how they run DBT, MLOps, alerting and for detection in Airflow.
• 1:23
  SPEAKER_01: So basically, we're gonna have this tour of the use cases that they have that they run in Airflow.
• 1:28
  SPEAKER_01: And uh maybe Rinalini will tell us more about some technical details and so on, but we'll see.
• 1:34
  SPEAKER_01: Uh okay.
• 1:36
  SPEAKER_01: So, first thing first, can you introduce yourself and tell us what you do at uh Head Out?
• 1:42
  SPEAKER_02: Okay.
• 1:42
  SPEAKER_01: And maybe maybe a little bit of your background as well, if you want.
• 1:46
  SPEAKER_02: Okay.
• 1:46
  SPEAKER_02: Um, so hi, I'm Renalini.
• 1:48
  SPEAKER_02: Um, I am a data platform engineer here at Head Out.
• 1:51
  SPEAKER_02: Head Out is basically a travel platform.
• 1:52
  SPEAKER_02: We try to bring real life experiences to people.
• 1:55
  SPEAKER_02: Um, so here I work on a big breadth of things.
• 1:59
  SPEAKER_02: That's the best part of my job, I feel like.
• 2:01
  SPEAKER_02: Um, I get because it's a lean team, I get to work on everything.
• 2:04
  SPEAKER_02: So I work from DVT to Airflow to MFOps to governance.
• 2:07
  SPEAKER_02: So all of it end-to-end.
• 2:09
  SPEAKER_02: Yeah, that's mostly it.
• 2:10
  SPEAKER_02: Um, I've been here for four years.
• 2:11
  SPEAKER_02: I enjoy it um a lot.
• 2:14
  SPEAKER_01: That's great.
• 2:15
  SPEAKER_01: Um, can you can you tell me more about like what what what exactly is a data platform engineer?
• 2:20
  SPEAKER_01: I guess like your data platform engineer.
• 2:22
  SPEAKER_01: Like, is it is it like your data engineer or just like you have also the DevOps part as well?
• 2:27
  SPEAKER_02: Like what Yeah, it's a combination of both, I would say.
• 2:30
  SPEAKER_02: So um I mostly handle the infra and enable people to do their best work.
• 2:34
  SPEAKER_02: So it includes a bunch of toolings to make sure um people have the best tools available to do their work, um, and in touch also with like um the infra bits, um everything is optimized, everything is running fine.
• 2:46
  SPEAKER_02: Um, and then also exploring newer tools, making sure everything is working fine.
• 2:51
  SPEAKER_02: Um, then also involves a bunch of DevOps.
• 2:53
  SPEAKER_02: We maintain a lot of services internally um for data.
• 2:56
  SPEAKER_02: So yeah, a little bit of everything.
• 2:59
  SPEAKER_01: Okay, okay.
• 2:59
  SPEAKER_01: So you are like a full-stack data engineer.
• 3:02
  SPEAKER_02: Yeah, I'm a jack of all trades.
• 3:04
  SPEAKER_01: Okay, okay, that's interesting.
• 3:06
  SPEAKER_01: Uh, just out of curiosity, um, I guess like, do you do you work with uh um data analysts, data scientists, and so on?
• 3:13
  SPEAKER_01: Like, do you work with different teams like that in your uh yeah?
• 3:16
  SPEAKER_02: Yeah, so um how it works in our team is basically we have a hub and spokes model.
• 3:21
  SPEAKER_02: So the data platform team sets in the center, and all the other data scientists, data um analysts, all these people sit at different spokes, different in within different pods.
• 3:29
  SPEAKER_02: Um so our work is to like kind of um make sure that they are doing their best work, um, help them in whatever way.
• 3:36
  SPEAKER_02: Um, because it's a very lean team, everyone does everything.
• 3:39
  SPEAKER_02: So our all of our analysts, all of our scientists also write their own data models.
• 3:42
  SPEAKER_02: So we need to make sure that we have good enough standards and good enough documentation so that they write code and like it's a well-enough code, um, even like optimization for things.
• 3:51
  SPEAKER_02: So yeah, it's like it sits in the middle.
• 3:52
  SPEAKER_02: We work a lot with um and we try to make their lives easier.
• 3:56
  SPEAKER_01: That's amazing.
• 3:57
  SPEAKER_01: Okay.
• 3:58
  SPEAKER_01: Um I guess the this was not uh uh um in the in the in the questions that we we discussed before, but I I I I'm just wondering like uh is Airflow was already there when you joined Head Out, or uh like did you decide to have Airflow in your company?
• 4:17
  SPEAKER_02: So it was already there when I was there, but like we were doing very smaller things back then.
• 4:23
  SPEAKER_02: Um it has now increased to do a lot more things.
• 4:26
  SPEAKER_01: Oh okay.
• 4:27
  SPEAKER_01: I was just wondering like if you uh you know if you had to go through uh the uh you know the install of Airflow and and also like maybe why did you choose Airflow and so on, but uh it's good to know that it was already there.
• 4:38
  SPEAKER_01: Um okay, I guess like uh another question to you.
• 4:42
  SPEAKER_01: So, what kind of data challenges uh come with running a travel platform?
• 4:46
  SPEAKER_01: Um I think that's actually the first time that I interview someone that it is working for uh a travel platform.
• 4:51
  SPEAKER_01: So I'm really interested to know more about that, uh, the data challenges that you have, and also where does Airflow sit in your uh in your stack?
• 5:01
  SPEAKER_02: Got it.
• 5:02
  SPEAKER_02: Okay, so um we already have like all the issues that come within a general data team of volume, data being wrong, or like the formatting being wrong, different kinds of file formats, different kinds of sources.
• 5:14
  SPEAKER_02: So all of that is there.
• 5:15
  SPEAKER_02: But specifically when you come to travel, there's like a few different things um that um are of a slightly bigger issue for us, especially things like inventory, where it changes very quickly, you need to be on top of things, you need to um run a lot of these things very quickly because a lot of your forecasting, a lot of your how much do you purchase, how much do you um not purchase, how much of it is going out.
• 5:36
  SPEAKER_02: Um a lot of revenue sits on top of that, right?
• 5:38
  SPEAKER_02: Um so inventory is a big thing.
• 5:40
  SPEAKER_02: Um, then um for real-time use case, especially like fraud, uh, you need to do a lot of real-time analytics, you need to do a lot of graph analytics, um, you need to process a lot of things very quickly in real time.
• 5:52
  SPEAKER_02: Uh, so that is one thing.
• 5:53
  SPEAKER_02: We also have a very big performance marketing heavy um domain travel, uh, where like a lot of our um like we have a lot of our customers coming in from campaigns um that we run online.
• 6:06
  SPEAKER_02: So that is a big data uh for us to manage um to maintain attribution um so that your money is actually going in the correct place where it should be going.
• 6:14
  SPEAKER_02: So those are some things um that are slightly different, like slightly different problem statements uh from like a general industry standpoint.
• 6:21
  SPEAKER_02: Um, Airflow sits at like, imagine like a nervous system of it all.
• 6:25
  SPEAKER_02: It's basically anywhere that we need.
• 6:27
  SPEAKER_02: So all the use cases that I told before, um, anything that we need to run, also like DVT, all the transformations happen through Airflow.
• 6:35
  SPEAKER_02: Like it's a tie-in for every kind of different system that we have from ingestion to kind of um serving some data to transformation, all of it just sits in Airflow.
• 6:45
  SPEAKER_01: Okay.
• 6:46
  SPEAKER_01: Um did you mention real time at some point?
• 6:49
  SPEAKER_01: Yes.
• 6:51
  SPEAKER_01: So how do you use Airflow with some real-time?
• 6:56
  SPEAKER_01: So real-time-ish orcha, I gotcha.
• 7:00
  SPEAKER_02: Um for some things you need to run it like at a two, three-minute interval, maybe a minute interval, five-minute interval, those.
• 7:07
  SPEAKER_02: Um for more real-time things we use Flink, um, but like for real-time-ish things.
• 7:12
  SPEAKER_01: Um, no, that that's interesting.
• 7:14
  SPEAKER_01: Um, yeah, I mean, I I was just wondering because like like you're not the first person that I hear about uh real-time use cases, and and also it was more like okay, two, two, two, three minutes.
• 7:25
  SPEAKER_01: Um you also mentioned dbt.
• 7:28
  SPEAKER_01: I mean, obviously dbt is a very, very popular um tool uh today.
• 7:33
  SPEAKER_01: But can you tell us more about like what is your dbt setup with Airflow?
• 7:39
  SPEAKER_01: Um, and uh you know how you uh interact with it.
• 7:42
  SPEAKER_02: Got it.
• 7:43
  SPEAKER_02: Um so our uh setup in dbt is a write audit publish setup.
• 7:48
  SPEAKER_02: Um we basically believe in that um stale data, slightly stale data is better than wrong data because a lot of our decisions are taken because of from data.
• 7:57
  SPEAKER_02: Like most of our teams are very data-driven.
• 7:59
  SPEAKER_02: All the decisions are go through um and are validated on top of data.
• 8:02
  SPEAKER_02: Now, if you give a wrong data that looks slightly correct, but might be wrong in a lot of ways, that does not um that would not lead to good results um if that gets in the pipeline.
• 8:12
  SPEAKER_02: So we went ahead with the right audit audit publish setup uh where we firstly basically pass through the manifest, um, see what tasks we want to build.
• 8:21
  SPEAKER_02: We will task groups per model, we'll run the model actually first in audit.
• 8:26
  SPEAKER_02: We'll run tests on top of it.
• 8:27
  SPEAKER_02: Um, if it passes, we replace the production model with that data.
• 8:31
  SPEAKER_02: If it doesn't, we um let people know so that um it gets fixed.
• 8:35
  SPEAKER_02: Um, slightly we have SLAs for all those things, um, so that we'd never have wrong data in production essentially.
• 8:43
  SPEAKER_01: That's actually very interesting.
• 8:44
  SPEAKER_01: Um because the write audit publish pattern is is not that I mean, to be honest, I I heard a quite a lot of people saying that they don't know about this pattern, uh, and I think it's a very useful one.
• 8:56
  SPEAKER_01: So just like in a very nutshell, um, can you just tell us a little bit more on how like what is exactly this pattern?
• 9:02
  SPEAKER_01: Like how does it work?
• 9:03
  SPEAKER_02: Um so basically right audit publish, or you might also have heard of like Broogle deployment, where you firstly deploy a thing, um, you test things, you test everything on top of it, and then if everything works fine on that audit instance or like that audit model.
• 9:18
  SPEAKER_02: So in case of dbt, we'll suppose build a bookings model.
• 9:21
  SPEAKER_02: Uh, we'll run XYZ text on top of it that we've defined in DBT.
• 9:25
  SPEAKER_02: If all tests tests run fine, we know the data is correct.
• 9:28
  SPEAKER_02: Um, and then we replace the production model that is user-facing and everyone in our company uses with that specific model.
• 9:34
  SPEAKER_02: If it's not, then older data continues to stay in production.
• 9:37
  SPEAKER_02: Um, so basically, you write it, you audit um by running tests, and then you replace it in production uh with the correct data if your tests run for correctly.
• 9:46
  SPEAKER_01: Okay, so so this pattern is really about avoiding having like some bad data in your production environment.
• 9:52
  SPEAKER_01: Okay, okay.
• 9:53
  SPEAKER_01: Uh it's uh actually, uh I mean, like you know, for the people that are listening to us and that don't know about this pattern, I highly recommend you to look into it because it's it's not a I mean it's not a hard pattern to uh to put in place, but it's a very efficient one.
• 10:06
  SPEAKER_01: Um great.
• 10:08
  SPEAKER_01: Uh but now like how how I mean I'm curious to know how do you interact with dbt from Airflow?
• 10:17
  SPEAKER_01: Sorry, from Airflow, because you know with Airflow, you have different ways for interacting with dbt.
• 10:22
  SPEAKER_01: Uh you can go with the bash operator and just run some dbt commands.
• 10:26
  SPEAKER_01: Uh you can also use like the dbt provider.
• 10:29
  SPEAKER_01: Um and there is also another way, but uh can you tell us more about your way?
• 10:34
  SPEAKER_01: Like how do you interact with with dbt?
• 10:36
  SPEAKER_02: Uh we use like a slightly different way, which many people might not agree with.
• 10:40
  SPEAKER_02: We use like a Python operator uh because uh we have like a certain level of customizations we have there, and we have like some alerting, some things that we send our send to our warehouse to see, okay, this task ran is much time.
• 10:54
  SPEAKER_02: Did it fail?
• 10:55
  SPEAKER_02: If the test failed, how many um rows it failed for?
• 10:57
  SPEAKER_02: We have a lot of logging and um auditing for all those data.
• 11:01
  SPEAKER_02: Um, so that is why we went ahead with a Python operator.
• 11:04
  SPEAKER_02: It like it allows you to do a lot more things.
• 11:06
  SPEAKER_02: Uh and that's in the main um five.
• 11:09
  SPEAKER_02: We went ahead with Python operator, yes.
• 11:11
  SPEAKER_01: Okay, and so this this Python operator, like is it um so it's like like so so you have this this custom operator, is that correct?
• 11:19
  SPEAKER_01: Where you interact with dbt?
• 11:21
  SPEAKER_01: Okay, but does it give you like the um because you know, like one issue that people have with uh with dbt and airflow is that they don't have the full picture of their of their uh dbt model, right?
• 11:32
  SPEAKER_01: Like when they look at their tag, they just see like one task, and then uh you know you have your dbt model behind this task, but you you don't see you know the the staging models and so on.
• 11:44
  SPEAKER_01: Uh is it something that you solved?
• 11:46
  SPEAKER_02: Yeah, so basically it's the same as like we build out the full lineage, we read the manifest, um, and then every um different model is a task, and within like it's the task group, and then audit is one task, tests are another task um that run after audit, and then we have a replace task.
• 12:05
  SPEAKER_01: Oh, okay.
• 12:05
  SPEAKER_01: And so how this is generated?
• 12:07
  SPEAKER_01: Like, do you have like a task?
• 12:09
  SPEAKER_02: We read the manifest file, and then we have like uh kind of like everything that dbt builds, uh, we kind of build it in how we have all those rules of partitions and all of those written custom.
• 12:20
  SPEAKER_01: Okay, and it is uh like what reads the manifest file?
• 12:23
  SPEAKER_01: Is it a script like a Python script?
• 12:25
  SPEAKER_02: Uh no, just the dbt manifest file um that dbt creates.
• 12:28
  SPEAKER_02: So dbt, whenever you run a dbt compile uh or a dbt um run, so it'll create a manifest file which will give you an information of all the different models, all the tasks, um the actual code that should be running, what are the columns, what are the column descriptions, um, if there's partition set.
• 12:43
  SPEAKER_02: So it'll give you a holistic information of what needs to be done to them, like what transformation needs to be made to that model.
• 12:49
  SPEAKER_02: Um, basically use all of that logic, apply it, and then we run it.
• 12:53
  SPEAKER_02: So we run it in two different projects.
• 12:55
  SPEAKER_02: So we run audit in one specific GCP project and then replace in another GCP project because of some governance issue where we did not want like the complete thing to be in one project.
• 13:04
  SPEAKER_02: Uh, we just wanted the final output data that we want to show to users in that specific project.
• 13:10
  SPEAKER_01: Okay, okay.
• 13:11
  SPEAKER_01: And just to be clear, like what reads this manifest file?
• 13:14
  SPEAKER_01: Is it like a script, a Python script?
• 13:16
  SPEAKER_01: Is it a task that generates the DAG with the manifest?
• 13:19
  SPEAKER_02: Um it's a just a JSON file.
• 13:21
  SPEAKER_02: Um, manifest file that's like the dbt um generated manifest file, it's a JSON file, and then we read through that and create like dynamic tasks um depending on what is there.
• 13:30
  SPEAKER_01: Okay.
• 13:31
  SPEAKER_01: Okay, okay, got it, got it, got it.
• 13:33
  SPEAKER_01: Uh it's interesting because that your your uh implementation looks very similar to uh Cosmos.
• 13:38
  SPEAKER_01: Uh I mean we discussed about that, but uh uh you you told me that you you had implemented that before even Cosmos existed.
• 13:45
  SPEAKER_01: Um but like for people that don't know about Cosmos, it's an open source project, you can use it for free.
• 13:50
  SPEAKER_01: Uh, I highly recommend you to look into it because you will save a lot of time.
• 13:55
  SPEAKER_01: Um it's just the best way to integrate um dbt with with Airflow.
• 13:59
  SPEAKER_01: You will have the full picture of your uh of your dbt models directly in Airflow, exactly as you uh explained.
• 14:06
  SPEAKER_00: This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
• 14:15
  SPEAKER_00: From better pipelines to training models and wiring up agents.
• 14:19
  SPEAKER_00: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 14:25
  SPEAKER_00: Check it out at astronomer.io.
• 14:29
  SPEAKER_01: Um great.
• 14:32
  SPEAKER_01: Now, uh, you know, I mean, obviously with AI, I mean AI is super popular right now, but um, before AI, actually, uh many people were talking about ML, like machine learning.
• 14:43
  SPEAKER_01: Um machine learning is still a thing, by the way, for people that wonder.
• 14:47
  SPEAKER_01: It's still something that works well.
• 14:50
  SPEAKER_01: Um I think you have some ML use cases uh that you run with Airflow.
• 14:56
  SPEAKER_01: Can you tell us a little bit more about that?
• 14:58
  SPEAKER_01: Because you know, for some people it's not obvious that you can use Airflow for uh ML.
• 15:02
  SPEAKER_02: So can you yeah, can you tell me more about so we like have like a few different ML use cases?
• 15:07
  SPEAKER_02: Some of them we run through Airflow uh specifically because it's like it's very easy to set up, it's very quick to because everyone is familiar.
• 15:14
  SPEAKER_02: Um they can very easily set up things.
• 15:16
  SPEAKER_02: Um, so there's different parts of ML ops um that we run on Airflow.
• 15:22
  SPEAKER_02: Um so feature store, we have feast as our feature store, a lot of things that flow into feature store, the things that we want to track, um etc.
• 15:29
  SPEAKER_02: Like they go through Airflow things that need to like on a scheduled interval, we send to a feature store that we want to track, that is done.
• 15:35
  SPEAKER_02: Uh, for some model training um and model inference, uh, we still use Airflow as our orchestrator, that's the main orchestrator.
• 15:43
  SPEAKER_02: Um, everything runs through there, like models, um, our models are very easy to train, very quick to run.
• 15:48
  SPEAKER_02: Um, then we also have like testing on top of it, right?
• 15:51
  SPEAKER_02: So you have data derived um and feature drift that can happen when you run all these airflow models.
• 15:56
  SPEAKER_02: So we have very extensive metric tracking there.
• 15:59
  SPEAKER_02: Um, and then we do calculations on top of it to see, okay, if there was any change, if we want to retune the model, etc.
• 16:05
  SPEAKER_02: Um, and then because most models are then retrained at X interval, Airflow makes it very easy to kind of do it on a schedule.
• 16:12
  SPEAKER_02: Um, same for inference for things that we have uh static um static recommendations for um or like static output for where like you just generate it one, like you generate a candidate set.
• 16:24
  SPEAKER_02: It's again very easy to do it 130 through Airflow.
• 16:26
  SPEAKER_02: So that is um there are some those are some ways where we um kind of use Airflow in our MLM system.
• 16:33
  SPEAKER_01: This is super interesting.
• 16:34
  SPEAKER_01: Um, I guess like when you say uh training your machine learning models, like um do you actually use to train your model or do you do you have like a different tool that you know to to train your model on on the data?
• 16:48
  SPEAKER_02: So for most of these use cases, like the small recommendation ranking, all these things, we do like all the training on Airflow itself.
• 16:55
  SPEAKER_01: Okay, okay, got it.
• 16:57
  SPEAKER_01: And uh do you have any like did you have any performance issues related to uh running Airflow with ML?
• 17:05
  SPEAKER_02: Uh so not specifically this ML likes for like the things that we run right now, it's fine.
• 17:11
  SPEAKER_02: Uh, but like for some bigger things, uh, we have like realized that memory and all those things, like it gets very expensive to do it um on our current, and it's easier to just like have another machine where you do all these things um and then push it uh rather than do it on Airflow because like continuously having that resource did not make sense.
• 17:29
  SPEAKER_01: Right.
• 17:30
  SPEAKER_01: Okay, okay.
• 17:31
  SPEAKER_01: Um okay, I guess like you know, when things go well, this is great, but as we know, right, in our world, many things go wrong first.
• 17:40
  SPEAKER_01: Uh how do you know that things go wrong in your uh you know in your Airflow instance?
• 17:46
  SPEAKER_01: Like, do you use the usual um you know alerting mechanism in Airflow, or do you did you create your own framework on top of Airflow?
• 17:53
  SPEAKER_01: Like, can you tell us more about how do you ensure that whenever something goes wrong in your Airflow instance, you know about it?
• 18:00
  SPEAKER_02: So there's a few different ways in we in which we track um how our Airflow instance is doing.
• 18:05
  SPEAKER_02: Um so first is we just have moved a lot of it to Grafana, where we have all of the metrics that we care about and the things that we care about.
• 18:13
  SPEAKER_02: Um, and also task level metrics where okay, it failed how many times?
• 18:17
  SPEAKER_02: What were like if it was like a general issue, who was the person who changed it, all those things.
• 18:22
  SPEAKER_02: So that is our central thing where we um track everything through Grafana.
• 18:25
  SPEAKER_02: There's a few alerts built on top of Grafana um that give us information.
• 18:29
  SPEAKER_02: Um the second way is at like a task level.
• 18:32
  SPEAKER_02: So uh we have like all the failure hooks, um success hooks.
• 18:36
  SPEAKER_02: So all for all failure hooks, we have uh we've made sure that okay, so we have owners in our DAGs, uh, we have tags in our DAGs for um teams.
• 18:44
  SPEAKER_02: We also check through GitHub who was the last person who changed that file.
• 18:48
  SPEAKER_02: So uh whenever uh a specific task fails, um, it'll send an alert on Slack that this is a person who last changed it, this is the commit that last changed it, this is the log link, and um, these are like the errors.
• 19:00
  SPEAKER_02: So it's very very easy to then like kind of converse on Slack and then okay, who is fixing it?
• 19:05
  SPEAKER_02: Can you fix it?
• 19:06
  SPEAKER_02: Also, very easy to kind of have all the links there.
• 19:09
  SPEAKER_02: So you have the commit link, so you can just directly see, okay, do you do I want to revert this PR?
• 19:13
  SPEAKER_02: Or um if it's something else, and you can directly go check the logs.
• 19:17
  SPEAKER_01: Wait, this is super interesting.
• 19:19
  SPEAKER_01: Like, like how how how does that work exactly?
• 19:22
  SPEAKER_01: Like, like like how do you how do you do that?
• 19:26
  SPEAKER_01: Like if you can enter a little bit more in the technical details, like how how did you implement this?
• 19:31
  SPEAKER_02: So um few like firstly, like if a task fails, all of the tasks will go to Slack.
• 19:37
  SPEAKER_02: Um that's like a failure hook.
• 19:39
  SPEAKER_01: Sorry.
• 19:39
  SPEAKER_01: Okay.
• 19:40
  SPEAKER_01: No, no, go ahead, go ahead, sorry.
• 19:42
  SPEAKER_02: Um, so there's a failure hook there.
• 19:44
  SPEAKER_02: Now on the failure hook, we have written a specific function.
• 19:47
  SPEAKER_02: Now I know uh what DAG is failing, so I know what kind of file is failing.
• 19:51
  SPEAKER_02: Um on GitHub because it's linked one-on-one, uh, because I can from Airflow get like what file um is the actual task failing in, what is the task and all those other information who's the author?
• 20:02
  SPEAKER_02: Um, so I can then go back checking GitHub.
• 20:05
  SPEAKER_02: Okay, what is the last commit on this particular file?
• 20:07
  SPEAKER_02: Um, and I know my teammates like Slack IDs and emails and all those things.
• 20:12
  SPEAKER_02: Okay.
• 20:12
  SPEAKER_02: So just stitch all of it together.
• 20:14
  SPEAKER_02: Um, you get Airflow logs, you get GitHub uh commit URLs.
• 20:17
  SPEAKER_02: So yeah, um, and then you just stitch it and send it to Slack.
• 20:20
  SPEAKER_01: And all of that is automated?
• 20:22
  SPEAKER_02: It's all automated.
• 20:24
  SPEAKER_02: Um, so because like user emails and all those things you can directly get, um, and then um Git commits, etc.
• 20:30
  SPEAKER_02: With GitHub API, it's very easy to um get all this information.
• 20:34
  SPEAKER_01: Awesome.
• 20:34
  SPEAKER_01: That's great.
• 20:35
  SPEAKER_01: That's the first one that I hear uh that I hear that this is this is cool.
• 20:39
  SPEAKER_01: Um so um I think that we discussed uh, you know, I mean uh previously, and I think is very common for people to have, maybe a little bit less today, but um I think you had some zombie tasks maybe in the past, I'm not sure if you still have uh but can you tell us more about how did you deal with this painful issue that I know many people had or at least had once uh with their with their own.
• 21:10
  SPEAKER_02: That that was a very long day.
• 21:12
  SPEAKER_02: Hang on, then again.
• 21:14
  SPEAKER_02: So basically, uh we had like a few uh zombie tasks there.
• 21:17
  SPEAKER_02: Um usually at Lowell Celery would kick them out.
• 21:20
  SPEAKER_02: Um, it is all okay, but that specific instance, sorry.
• 21:24
  SPEAKER_02: So sorry, very quickly, can you define what what a zombie task is for those who um so a zombie task is basically a task where like um it has ended.
• 21:33
  SPEAKER_02: Um on celery thinks it hasn't ended, but it has actually ended, it's not really running, but celery thinks it's running.
• 21:38
  SPEAKER_02: Um so there's a misma mismatch in status, um, and then it causes a lot of issues.
• 21:43
  SPEAKER_02: So for us specifically, what it caused was we weren't able to identify uh for a you know that it's running and it kept running in the background, right?
• 21:51
  SPEAKER_02: Um not because it did not happen to us before, so we were not tracking it that often.
• 21:55
  SPEAKER_02: Okay, this is something that's running, or like this is something that we need to take care of.
• 21:59
  SPEAKER_02: So the That time when it happened, what happened was it led to DV um size increasing because like it kept on adding logs, it kept on adding run states, etc., in the DV.
• 22:10
  SPEAKER_02: Now, DV very quickly, in like a few hours, um, changed the scale.
• 22:15
  SPEAKER_02: Um, and then it was causing locking issues on the DV.
• 22:18
  SPEAKER_02: Uh, and because of locking issues, no new tasks were getting um added to the queue.
• 22:23
  SPEAKER_02: Nothing was processing, it was all stuck.
• 22:26
  SPEAKER_02: And then it could also not cancel because even through API, even through um UI, because it doesn't know that the task is there, because according to it, the task is not there.
• 22:35
  SPEAKER_02: Um, so yeah, that was a big pain.
• 22:37
  SPEAKER_02: Now we have added like a bunch of alerting on top of it also.
• 22:41
  SPEAKER_02: Uh where now we track okay, what is the status, what is the task, how much longer has been it running, um, like memory and all those things.
• 22:50
  SPEAKER_02: So, like the Grafana and all those alerting and metrics came after this because we realized we need to build more exhaustive systems um to be able to monitor um air.
• 23:00
  SPEAKER_01: Okay, so so what allows you to spot those zombie tasks today are the alerts that you defined in Grafana.
• 23:07
  SPEAKER_02: Grafana plus also just um from the API.
• 23:10
  SPEAKER_02: So um in the Airflow API, you get like a lot of information, right?
• 23:13
  SPEAKER_02: Um, like how much tasks it's running, like all of those things.
• 23:15
  SPEAKER_02: So we use that also in um addition to Grafana.
• 23:18
  SPEAKER_01: That's cool.
• 23:19
  SPEAKER_01: Yeah, I mean I'm sure that people will uh will uh reflect a lot on what you what you just said.
• 23:24
  SPEAKER_01: Um all right, one last thing because I have to ask you, uh, you know, um AI is everywhere.
• 23:32
  SPEAKER_01: Yes.
• 23:32
  SPEAKER_01: So I have to ask you, uh, how do you use AI today in your in your um in your company?
• 23:38
  SPEAKER_01: Just if you can tell us more about that.
• 23:40
  SPEAKER_02: Yes.
• 23:41
  SPEAKER_02: So we use AI very extensively in our company now.
• 23:44
  SPEAKER_02: Like a lot of non-tech people also create commits.
• 23:47
  SPEAKER_02: Um, they have a lot of changes making the developer velocity, has changed a lot.
• 23:51
  SPEAKER_02: Specifically in the data team, um, we try to augment it, but also because data is a very sensitive thing, where like if AI hallucinates, there is a lot of issues, so you need to double check everything.
• 24:02
  SPEAKER_02: Um, we were building the self-serve um system, and then it was hallucinating columns that did not exist.
• 24:08
  SPEAKER_02: So you need to be also careful, careful about it.
• 24:11
  SPEAKER_02: Uh specifically, um coming to like coding um in data.
• 24:16
  SPEAKER_02: Um, in the start, we looked at things um where like people would create um PRs with um plot code or like any other agent.
• 24:24
  SPEAKER_02: Uh, but the thing was that was it was correct Python code, but it was not very airflow.
• 24:29
  SPEAKER_02: Um, like it would not follow it, like it would look like it's a fine Python code, but it's like it doesn't follow airflow conventions.
• 24:36
  SPEAKER_02: Like there would be things like where like um connections are created outside of tasks.
• 24:40
  SPEAKER_02: Um every time a DAG or bar, it'll create a connection.
• 24:44
  SPEAKER_02: Um so we realized we already have a lot of uh systems and um recommendations in place when you write DAGs.
• 24:53
  SPEAKER_02: There's always like this things like um good recommendation.
• 24:55
  SPEAKER_02: We also have internal things that this is the pattern that we follow.
• 24:58
  SPEAKER_02: Um so we took a two-step process here where firstly we encoded all of this as like a skill file.
• 25:05
  SPEAKER_02: Like if you want to develop, you have these um resources that this is the pattern that we follow, this these are the guidelines, patterns, anti-patterns, everything.
• 25:13
  SPEAKER_02: Like we try to make it as exhaustive as we can.
• 25:16
  SPEAKER_02: Um, so that whenever you're developing code, it goes in the right direction.
• 25:20
  SPEAKER_02: But actually, I missed this, but before that, you need to clean up your current current code because if it learns wrong patterns from your current code or current DAGs, like it's it's very difficult.
• 25:33
  SPEAKER_02: Um, it'll always create like not great code.
• 25:36
  SPEAKER_02: Uh that's right.
• 25:37
  SPEAKER_02: So so we cleaned it up, we created like a new system, and then we also have like a reviewer bot on top of it.
• 25:44
  SPEAKER_02: So there's a reviewer bot that reviews every PR in the company, tech, not like everything.
• 25:49
  SPEAKER_02: Uh, so we added specific uh instructions there that even if someone misses anything, even when they're building it through the skill, it's able to catch um these smaller things now.
• 25:58
  SPEAKER_02: Things like memory leaks are very difficult to spot.
• 26:02
  SPEAKER_02: So all those things you have added, like specific um edition.
• 26:05
  SPEAKER_02: It's still a work in progress.
• 26:06
  SPEAKER_02: There's still sometimes where we miss things and like uh things need to be modified, but again, like it's just an iterative process.
• 26:13
  SPEAKER_02: Like if something goes wrong, we try to fix it the next time.
• 26:16
  SPEAKER_01: But this is another AI that we've used the yes.
• 26:21
  SPEAKER_01: Oh, okay.
• 26:22
  SPEAKER_01: So it's like you have like you generate the code with AI and then you have another AI that checks whatever has been generated.
• 26:28
  SPEAKER_02: Yes, yes.
• 26:29
  SPEAKER_01: Okay, okay.
• 26:30
  SPEAKER_01: So it's like AI as a judge, right?
• 26:31
  SPEAKER_02: I mean kind of, yes.
• 26:33
  SPEAKER_02: So it does like first level of um checking for things, so that it's easier for like the reviewers to then go ahead and actually win the reviewing code.
• 26:40
  SPEAKER_02: Um, most of the obvious mistakes are fixed before then.
• 26:43
  SPEAKER_01: Right.
• 26:44
  SPEAKER_01: I think you you said something super uh super important is that um is that when you if you if you like if you decide to uh create a skill out of your code, then you better make sure that your code actually is great.
• 26:59
  SPEAKER_01: Uh and I think maybe this is one of the things that AI um bring you know brought back to life is the documentation as well, like ensuring that you document your uh your code properly.
• 27:12
  SPEAKER_01: Uh, you know, it it I mean today it's it's super important because obviously AI reads all of that.
• 27:19
  SPEAKER_01: Um yeah, I mean I think this is uh this is super interesting.
• 27:23
  SPEAKER_01: Uh this is super interesting the way that you are using uh AI and and again as you said, right?
• 27:28
  SPEAKER_01: AI can uh hallucinate uh quite a lot, so you better be careful with uh whatever you generate uh with it.
• 27:35
  SPEAKER_01: Well, uh I must say I'm very I'm very shocked because it's already uh over, and to be honest, that went super fast.
• 27:47
  SPEAKER_01: That went super fast.
• 27:48
  SPEAKER_01: Uh I'm sure that people will enjoy a lot uh this episode.
• 27:52
  SPEAKER_01: You obviously explained a lot of very interesting things and uh in details.
• 27:56
  SPEAKER_01: Uh so thank you so much for that.
• 27:58
  SPEAKER_01: Uh but before leaving, I have to ask uh what is the best way for uh folks to get in touch with you if they want to ask questions, maybe or you know if they want to learn more about uh head out.
• 28:09
  SPEAKER_02: Yeah, um, you can just uh connect with me on LinkedIn and maybe Blue Sky.
• 28:13
  SPEAKER_02: I don't use it as often, but you can shoot me at my probably replacing a few days.
• 28:17
  SPEAKER_01: Awesome, awesome.
• 28:19
  SPEAKER_01: Well, thank you so much again, Minalini.
• 28:21
  SPEAKER_01: Um, hope to see you again for another episode.
• 28:23
  SPEAKER_01: Who knows?
• 28:24
  SPEAKER_01: Uh but otherwise, I wish you the best.
• 28:26
  SPEAKER_01: Take care and uh see you.
• 28:28
  SPEAKER_02: You do.
• 28:29
  SPEAKER_02: Thank you so much for having me.
• 28:30
  SPEAKER_02: It was a very nice chat.
• 28:31
  SPEAKER_01: Thank you.
• 28:31
  SPEAKER_01: See you.
• 28:32
  SPEAKER_02: Bye.
• 28:34
  SPEAKER_00: Thank you for listening to the Data Flowcast.
• 28:37
  SPEAKER_00: Check the show notes for links to everything mentioned today.
• 28:41
  SPEAKER_00: If you're enjoying the podcast, subscribe so you never miss an episode.
• 28:46
  SPEAKER_00: And leave us a five-star review.