Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero

Transcript

Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: You need to be very careful because depending on the levels and depending on how complex your data is, it might take more time than expected.
• 0:08
  SPEAKER_02: So it's very helpful to have these kind of ML libraries or tools such as Azure Monitor or AWS Lookout because you will be able to identify from the very beginning or like from the moment you are implemented it.
• 0:23
  SPEAKER_02: You look at it and you say, Yeah, I knew this was going to take more time, and now I see it will take even more time in the future.
• 0:36
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:46
  SPEAKER_00: Hey Airflow community.
• 0:48
  SPEAKER_00: Airflow Theory is here and we are taking it on the road.
• 0:51
  SPEAKER_00: Join the Astronomer Road Show for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:59
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:08
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:13
  SPEAKER_00: Register now.
• 1:14
  SPEAKER_00: The link is in the show notes.
• 1:17
  SPEAKER_00: Okay, hi everyone.
• 1:18
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:20
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Adonis Castillo Cordero, who is an engineering manager at Procter and Gamble.
• 1:28
  SPEAKER_00: Adonis, how are you today?
• 1:30
  SPEAKER_02: I'm doing good.
• 1:31
  SPEAKER_02: What about you?
• 1:32
  SPEAKER_00: Yeah, I'm great.
• 1:33
  SPEAKER_00: I'm great.
• 1:34
  SPEAKER_00: Yeah, we're very busy over here at Astronomer.
• 1:37
  SPEAKER_00: A lot's going on in the Airflow project with Airflow 3 coming out in the past couple of weeks.
• 1:42
  SPEAKER_00: And so it's all good stuff, but definitely keeping us busy.
• 1:45
  SPEAKER_02: Yes, I can I can imagine.
• 1:47
  SPEAKER_00: Yeah, yeah, for sure.
• 1:49
  SPEAKER_00: Awesome.
• 1:49
  SPEAKER_00: Let's go ahead and dive in.
• 1:51
  SPEAKER_00: So for today's podcast, we're going to discuss some best practices for monitoring airflow and using it for AI workloads.
• 1:58
  SPEAKER_00: I think this will be very interesting for our listeners.
• 2:01
  SPEAKER_00: Best practices is always a really hot topic that everybody is looking for, especially with a tool that you can do so much with, like airflow.
• 2:09
  SPEAKER_00: But to start with some background for those that might not be familiar, maybe you can tell us what Proctor and Gamble does.
• 2:16
  SPEAKER_00: I think Procter and Gamble is pretty well known, but maybe there are some folks out there who aren't as familiar.
• 2:21
  SPEAKER_02: Yes.
• 2:22
  SPEAKER_02: So yeah, so it's a global leader when it comes to consumer goods.
• 2:27
  SPEAKER_02: Some brands such as Gillette, such as Pantine, for example, or Thai, they're in the states.
• 2:33
  SPEAKER_02: Those are some of the brands they they work on.
• 2:37
  SPEAKER_02: And of course, also when it comes to improving lives of consumers, so trying to give the sustainability as well.
• 2:46
  SPEAKER_02: So yeah, that's like overall.
• 2:48
  SPEAKER_00: That's great.
• 2:48
  SPEAKER_00: I think Procter Gamble is one of those big conglomerates that I think I suppose is true, certainly true for Americans that probably everybody uses something that is made by them, even if to your point, like a lot of things have other brand names, so they might not recognize the Proctor and Gamble, but it is the larger conglomerate behind a lot of extremely well-known and widely used brands for household goods.
• 3:11
  SPEAKER_02: That's right.
• 3:12
  SPEAKER_00: Yeah, that's awesome.
• 3:13
  SPEAKER_00: And what does your team do?
• 3:15
  SPEAKER_02: Yeah, so it is an engineering team.
• 3:17
  SPEAKER_02: I'm part of one of the engineering teams here at Proctor Gamble.
• 3:22
  SPEAKER_02: So since because of the size of the company, so it is very big, you might find like multiple departments being like, you know, like a small company themselves.
• 3:35
  SPEAKER_02: When it comes to architecture, in this case, this one, what it does is bringing all of the data for a specific area when it comes to master data specifically, because of the nature of the business.
• 3:51
  SPEAKER_02: So there's a lot of a lot of processes going on, like that you need to monitor almost live every time, because it comes to production, it comes to plants.
• 4:02
  SPEAKER_02: There's a lot of processes that are done there, and that you need to extract that data from like some legacy systems that are not like that.
• 4:11
  SPEAKER_02: You would say it is not that cloud-based, because of course it needs to monitor data and extract data from plants.
• 4:19
  SPEAKER_02: So, because of that, the team I work with, they have their own architecture, basically medallion architecture, which is very popular these times.
• 4:29
  SPEAKER_02: So it gets the data we could say it is like even before the bronze layer that it gets this data from different systems.
• 4:39
  SPEAKER_02: So it what it does is it consolidates this data uh into like a yeah, like a bronze layer, even carbon to to invent a word.
• 4:50
  SPEAKER_02: And then using Airflow in this case, what it does is uh it runs all of the orchestration, so all of the systems can come together to form like this base role layer that can be used moving forward with the rest of the of the processes and architectures that build the entire league, basically.
• 5:12
  SPEAKER_00: Okay, very interesting.
• 5:14
  SPEAKER_00: So to summarize that back, so you work with there are lots of internal technical teams at Proctor and Gamble doing really many different things with a lot of different tools.
• 5:25
  SPEAKER_00: But sometimes they need to modernize them or move off of legacy systems.
• 5:30
  SPEAKER_00: And so your team is responsible for helping them do that and providing the architecture setup that is going to help them be most successful.
• 5:37
  SPEAKER_00: Is that right?
• 5:39
  SPEAKER_00: Okay, great.
• 5:40
  SPEAKER_00: And so you touched on it a little bit, but I would imagine that a lot of the teams that you end up working with end up migrating from legacy systems to using Airflow as an orchestrator.
• 5:51
  SPEAKER_00: So tell me a little bit more about how Airflow is used for you mentioned that bronze layer, getting everything into the data lake.
• 6:00
  SPEAKER_00: Give me some more details on how that works.
• 6:03
  SPEAKER_02: Yeah, it varies a lot depending on the data you're using, depending on the systems you're extracting the data from as well.
• 6:10
  SPEAKER_02: So, for example, in some cases, it will be for streaming, for example.
• 6:16
  SPEAKER_02: So sometimes you might be using Kafka to monitor some events from, like I said, like a specific processes that they run.
• 6:25
  SPEAKER_02: So this data needs to be like as timely and accurate as possible by training a specific DACs using the Kafka connector, for example.
• 6:35
  SPEAKER_02: So you we will be getting this data and standardizing this data using other other technologies and most likely Spark because it's basically native there, right?
• 6:46
  SPEAKER_02: In Airflow.
• 6:47
  SPEAKER_02: So to transform this data and make it uh like uh on a very standardized uh state, uh, but not that much because of how this data is going to be used.
• 7:03
  SPEAKER_02: If you standardize it a lot, then you might not be able to use it in other processes, so you need to be very careful on that aspect.
• 7:11
  SPEAKER_02: And then once this process is finished, then you will be storing this on the different cloud storage that that we have.
• 7:19
  SPEAKER_02: And it depends if every cloud storage that you might know, we might use one of those at any point at any point because of, like I said, like how global it is.
• 7:30
  SPEAKER_02: So basically, that's like a little more detail on how it is done normally.
• 7:37
  SPEAKER_00: Yeah, that's great.
• 7:38
  SPEAKER_00: So it sounds like you're definitely leveraging Airflow's ability to connect with any system and that kind of plug and play aspect of using whatever different integrations are needed for the teams that you're working with, whether it be different cloud services or something like Kafka.
• 7:53
  SPEAKER_00: That's really interesting.
• 7:55
  SPEAKER_00: And I would imagine that given your experience helping a lot of different teams modernize their data orchestration and move on to Airflow, through that, are there any patterns that you've seen regarding best practices?
• 8:12
  SPEAKER_02: Yes.
• 8:12
  SPEAKER_02: So I would like to start with one that I consider very crucial at any stage of your architecture, and is to define the dependency like as soon as you can.
• 8:27
  SPEAKER_02: Because, for example, when it comes to things I have seen is like you build the architecture, you start building DAX and scripts, and then at some point, most of the engineering teams, like in in general, not only here, but in general, I have seen they find out there was something wrong at the very beginning because they did not see there was a dependency, that if this a specific DAC failed, then everything else will be sloped down or even to crash.
• 9:01
  SPEAKER_02: So, what I do recommend is to build this dependency mapping, even downstream or upstream, like a priority from the very beginning, because in that way, when you have this map, then it will be super easy for you to implement any other tool, such as, for example, open lineage.
• 9:21
  SPEAKER_02: So, because you build this dependency mapping, you can use tools such as this to handle the lineage properly.
• 9:29
  SPEAKER_02: And in that way, like you can move more into automating the monitoring, right?
• 9:34
  SPEAKER_02: Because it is not just now knowing that if an specific DAC fails, everything else fails, but it's also that you will know it beforehand or that you will be able to receive alerts when this happens.
• 9:47
  SPEAKER_02: And also, also for sure you will know like how to prevent it because you will have locks, you will see what are like those weak spots with points that it the architecture or the flow has.
• 10:01
  SPEAKER_02: So that's the first one I would like to mention.
• 10:07
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 10:16
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:28
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 10:37
  SPEAKER_00: I think several different aspects of that are really interesting to me.
• 10:40
  SPEAKER_00: The first is just uh what you touched on, like implementing the dependencies themselves, because Airflow has a lot of options for doing that, but it is quite a bit harder to do it after the fact if you don't already know what those dependencies are and you're trying to use data sets or like an Airflow 3 using assets.
• 10:59
  SPEAKER_00: But if your Airflow DAGs are in different Airflow environments, then it gets a little more complicated to implement and requires a lot more code change to do something like that.
• 11:11
  SPEAKER_00: And to your point, like if you're not planning that from the beginning, I think it adds quite a lot of work on the back end to retroactively do it.
• 11:18
  SPEAKER_00: And then the monitoring aspect that you touched on, like just being able to see all of that together is really important and planning for that from the beginning.
• 11:26
  SPEAKER_00: I love that you brought up open lineage and how that integration can be extremely helpful.
• 11:31
  SPEAKER_00: I know at Astronomer with Astro, our managed service, we've implemented a part of the platform called Astro Observe, which is built off of lineage data and kind of adds that extra visibility and monitoring into everything that's happening in all of your airflows so that you can see it kind of more holistically as opposed to just pipeline by pipeline.
• 11:52
  SPEAKER_00: So yeah, I think especially for larger teams, or I would imagine at Proctor and Gamble, where you've got so many interdependencies between all of the teams, that would be extremely important to prioritize up front.
• 12:06
  SPEAKER_02: Yes, I agree.
• 12:09
  SPEAKER_02: And actually, when it comes to yeah, to monitoring, that will be the second one, which is in this align also with ML or AI.
• 12:18
  SPEAKER_02: When it comes to the anomaly detection as well, so it's it goes with the handoff of the previous one.
• 12:25
  SPEAKER_02: Because, for example, with tools such as let's say Azure Monitor or AWS Lookout, with those tools, for example, or even you could like with FML libraries, you will be able to do this as well.
• 12:38
  SPEAKER_02: But this the point here is more focusing on predicting when something is going to, for example, a DAG or a pipeline is going to be sloped down, or that you can identify, let's say you're working, and I think this is very common.
• 12:57
  SPEAKER_02: Maybe you're working with JSON files and you need to flatten them, flatten them a little bit more so you can get a more structured data.
• 13:07
  SPEAKER_02: So in that in that kind of processes, you need to be very careful because depending on the levels and depending on how complex your data is, it might take more time than expected.
• 13:19
  SPEAKER_02: So it's very helpful to have these kind of ML libraries or tools such as Azure Monitor or AWS Lookout, because you will be able to identify from the very beginning or like from the moment you are implemented it, that this is you look at it and you say, Yeah, I knew this was going to take more time, and now I see it will take even more time in the future.
• 13:43
  SPEAKER_02: So with this, you are able not just to like not just to trust what by experience, but also to prove why you need to, or the team needs to pay attention to any specific pipeline or process.
• 13:59
  SPEAKER_00: Yeah, absolutely.
• 14:00
  SPEAKER_00: That what I often think of as like proactive alerting.
• 14:04
  SPEAKER_00: And so figuring out like when something is going to go wrong before it has actually gone wrong, or that like anomaly detection in terms of pipeline performance is something that we've seen so much attention paid towards recently.
• 14:17
  SPEAKER_00: And again, something that we have built and are building on in our observe platform.
• 14:23
  SPEAKER_00: And it sounds like you have worked with teams to implement something using other cloud services to integrate that with Airflow.
• 14:29
  SPEAKER_00: That's definitely something that's really important for teams, and I think becomes extra top of mind when you're working at like the scale that you're describing.
• 14:38
  SPEAKER_00: When if you've wait, waited until something has already gone wrong, then the downstream impacts are already going to be really huge.
• 14:45
  SPEAKER_00: So knowing ahead of time is very helpful.
• 14:49
  SPEAKER_02: Yes.
• 14:50
  SPEAKER_02: So the third one I had is on the quality monitoring.
• 14:55
  SPEAKER_02: This is very similar to what we have mentioned.
• 14:58
  SPEAKER_02: Teams tend to focus on like metrics, like focus on if the metrics are are fine, then everything else is fine, right?
• 15:08
  SPEAKER_02: So sometimes you might need to use other tools such as like Monicarlo or Great Expectations, for example, so you have better track of what's going on.
• 15:19
  SPEAKER_02: So you will be able to track, like, for example, if there's let's say you had a pipeline that applied a transformation based on a specific column, but then that column got removed, and your pipeline keeps running this process.
• 15:34
  SPEAKER_02: So it is going to add consumption at all in all aspects.
• 15:39
  SPEAKER_02: So when you include this kind of quality active quality monitoring, then you will be able to identify this.
• 15:47
  SPEAKER_02: So it is not just only about how that, yeah, the end data or the final product is fine, but also what was what is not being done anymore is not delaying or is not affecting the performance or of your pipelines.
• 16:05
  SPEAKER_02: So in that aspect, I consider this is something crucial also for you to make sure it's in us it is not that your data pipelines are fine, but also how they can be better by actively monitoring this.
• 16:21
  SPEAKER_02: And also, this also uh includes uh like defining the risk of uh like a fail or a failure when it comes to a pipeline, because sometimes like most of the teams maybe they just monitor if anything fails, and then you're like uh hey, it's 6 a.m.
• 16:43
  SPEAKER_02: The pipeline failed, what happened?
• 16:45
  SPEAKER_02: Everybody, let's start working on this, but then you realize it was like, hey, yeah, this failed, but it was actually not that that big of a of an issue.
• 16:55
  SPEAKER_02: It like uh the processes and the all of the pipelines could continue going on.
• 17:00
  SPEAKER_02: So if you are able to identify and set these priorities, then you you will be able in the future to have a better response from the team so you can like organize better and say, hey, yeah, this is we identified this failed, but this is not affecting everything.
• 17:19
  SPEAKER_02: This is just affecting this piece because we already tracked and we already tagged all of our DACs, and so we know how to prioritize them.
• 17:28
  SPEAKER_02: So I consider this to be something very important when it comes to prioritation in teams.
• 17:34
  SPEAKER_00: Yeah, absolutely.
• 17:35
  SPEAKER_00: I think so much good stuff in your answer there.
• 17:38
  SPEAKER_00: Data quality checks in general are so important, and you have a ton of options for implementing them with airflow.
• 17:44
  SPEAKER_00: And I think to your point about prioritizing them, this is something that our own data team, an astronomer, does in their pipelines, is they have a concept of data quality checks that a failure will stop the pipeline.
• 17:57
  SPEAKER_00: So if this fails, we are not moving forward.
• 17:59
  SPEAKER_00: It's going to impact stuff enough that we don't ever want it, you know, showing up in downstream tasks that might mistakenly update a dashboard or something like that.
• 18:08
  SPEAKER_00: They also have checks that fail that just notify them, but everything continues because to your point, it's not always something that needs immediate attention or needs to, again, stop everything downstream.
• 18:24
  SPEAKER_00: It can just be, hey, this is something that you might need to know about.
• 18:27
  SPEAKER_00: Like the data has changed in some way, but it's not maybe critical.
• 18:32
  SPEAKER_00: And I think, yeah, especially in our case, I think our team does that because they're a small team managing a lot of data.
• 18:38
  SPEAKER_00: And so they have to not create like too many fires where they don't need them.
• 18:43
  SPEAKER_00: And I think that's very relevant for probably every team that's trying to manage a lot of pipelines.
• 18:50
  SPEAKER_02: Yeah, yeah.
• 18:51
  SPEAKER_02: And the final one I had, this is something I will dare to say most of the teams do, and it's using a BI tool where monitoring, like, yeah, to have all of the big picture of what's going on.
• 19:05
  SPEAKER_02: My recommendation there is to be very careful and to keep it as simple as possible because sometimes, like and when it comes to people managing systems like engineering managers or so on, uh, they uh you know they need to be able to identify very specific points, like for example, not to see a full table with a lot of details, because maybe you just want to see, hey, how am I in capacity?
• 19:34
  SPEAKER_02: How am I on performance?
• 19:37
  SPEAKER_02: How is the availability of the data right now?
• 19:40
  SPEAKER_02: Uh how fresh it is.
• 19:42
  SPEAKER_02: So these kind of details might even you might be able to see them in a glance instead of having to go through multiple visuals.
• 19:52
  SPEAKER_02: So my recommendation there is to use all of the previous techniques mentioned, but also to keep this final layer as as simple as possible, and also to get like the most important metrics as possible.
• 20:11
  SPEAKER_00: Yeah, for sure.
• 20:12
  SPEAKER_00: Sometimes less is more in terms of monitoring, making sure that you're only looking at yeah, the most important stuff.
• 20:18
  SPEAKER_00: That is really great.
• 20:19
  SPEAKER_00: Those are four, I think, very insightful best practices, and it's neat to hear that, yeah, that information that you've gathered after working with so many different teams implementing Airflow.
• 20:30
  SPEAKER_00: Since we are almost at the end of our time, I always like to wrap things up with a standard final question for everybody.
• 20:36
  SPEAKER_00: And that is what would you most like to see from the Airflow project in the near future?
• 20:41
  SPEAKER_02: Actually, to be very honest, I haven't been able to take a look at the latest features in Apache Airflow 3.
• 20:51
  SPEAKER_02: So I will be testing it for the next weeks, but I think basically it will be that, just to see what is new, how all of these things I mentioned might be already there, or how they are made easier in the new version.
• 21:06
  SPEAKER_02: So I will say that for the moment.
• 21:08
  SPEAKER_00: Yeah, absolutely.
• 21:10
  SPEAKER_00: So much exciting stuff in the big release.
• 21:12
  SPEAKER_00: This is a particularly, I think, in my mind, funny time to be asking this question because we just hit such a huge milestone in the project.
• 21:19
  SPEAKER_00: Yeah, I'm excited to hear what you think about it after you've had a chance to kind of kick the tires a little bit.
• 21:26
  SPEAKER_00: So okay, great.
• 21:35
  SPEAKER_02: It will be through LinkedIn.
• 21:38
  SPEAKER_00: Great.
• 21:38
  SPEAKER_00: We will make sure to put that in the show notes.
• 21:41
  SPEAKER_00: Again, really appreciate you coming on and sharing your insights from Proctor and Gamble with us.
• 21:46
  SPEAKER_00: And yeah, we appreciate it.
• 21:49
  SPEAKER_02: Yeah, thank you very much.
• 21:51
  SPEAKER_00: Okay.
• 21:54
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 22:01
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 22:05
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.