Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira

Transcript

Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Okay.
• 0:00
  SPEAKER_03: We are talking our data lake today is about four petabytes.
• 0:06
  SPEAKER_03: And that's a lot of the data.
• 0:09
  SPEAKER_03: And data ability engineering assists change with maintenance, validation, corrections, and optimize both existing and new proxies.
• 0:20
  SPEAKER_03: This ensure we don't duplicate uh pipelines and uh help with the already existing pipelines here.
• 0:29
  SPEAKER_00: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:37
  SPEAKER_00: Let's get into it.
• 0:39
  SPEAKER_01: Alright, hello everyone.
• 0:41
  SPEAKER_01: Welcome, welcome to the Data Flowcast.
• 0:43
  SPEAKER_01: I'm your host, Marco Merti, and today I'm super excited because I'm joined by Matheus Ferrera, senior data engineer at Luisa Labs.
• 0:52
  SPEAKER_01: First thing first, how are you doing, Matheus?
• 0:56
  SPEAKER_03: I'm fine and you.
• 0:58
  SPEAKER_03: Uh I'm very glad to be here today.
• 1:01
  SPEAKER_03: Uh that's uh a nice opportunity to explain something about what you doing here.
• 1:08
  SPEAKER_01: Likewise, very happy to have you uh in this podcast.
• 1:11
  SPEAKER_01: So let's dive in for today's podcast.
• 1:13
  SPEAKER_01: Um, we are going to talk about how Luisa Labs orchestrates 2000s Airflow pipelines for one of the uh biggest, largest retailers in uh Brazil and why they are migrating off managed composer to run Airflow on their own on-premise on premise Kubernetes cloud.
• 1:32
  SPEAKER_01: So there is a ton of things to discuss today, but um I think the very first question that you know many people wonder is can you introduce yourself and tell us more about what you do at Lisa Labs?
• 1:43
  SPEAKER_03: Okay, nice.
• 1:44
  SPEAKER_03: Um, I'm Mateus.
• 1:45
  SPEAKER_03: Uh like I said, I'm 28 years old.
• 1:48
  SPEAKER_03: I'm from Brazil, I'm currently based in Sao Paulo.
• 1:51
  SPEAKER_03: I'm work as a data engineer here at Lisa Labs.
• 1:55
  SPEAKER_03: Uh I I do a lot of things here as a data engineer, but the main uh my main things are improving pipelines, cost reduction, developing eggs, maintaining data platform as a whole, like governance, security, performance, and more.
• 2:14
  SPEAKER_01: Okay, so a lot of uh of uh like the usual things that data engineers do, I think uh a lot a lot of people will uh will reflect in in what you said.
• 2:23
  SPEAKER_01: And um, can you tell us more about like Luiza Labs?
• 2:26
  SPEAKER_01: Like what do what this company typically does?
• 2:30
  SPEAKER_03: Okay, Luisa Labs is the arm of technology from Magazine Luiza or Magalu.
• 2:36
  SPEAKER_03: Uh this is the one of the largest uh retail companies here in Brazil.
• 2:41
  SPEAKER_03: Uh that's a very old company here.
• 2:44
  SPEAKER_03: Uh it's one of the largest companies here uh because we sell a lot of things, uh smartphones, um uh toils.
• 2:54
  SPEAKER_03: Um what do you imagine you can buy inside Magalu?
• 2:58
  SPEAKER_01: Uh it's like Amazon.
• 3:00
  SPEAKER_01: Is it like Amazon?
• 3:01
  SPEAKER_03: Yeah, yeah, it's like Amazon.
• 3:03
  SPEAKER_03: Uh and we focus here in digital and physical.
• 3:10
  SPEAKER_03: We are uh Magalu has more than 1,000 physical stores here in Brazil.
• 3:16
  SPEAKER_03: Okay, it's uh and Brazil is giant, it's a continental country with different regions here.
• 3:22
  SPEAKER_03: Uh that um is and the way of work is different inside different regions, and Magalu is is uh is spread in all all of the our country, and that's amazing to work with a lot of data from different places with different um destinations with different uh uh rules with different uh needs, and and that's amazing uh to work with that amount of data here.
• 3:57
  SPEAKER_01: Okay.
• 3:58
  SPEAKER_01: And so what uh what kind of uh data challenges you know come with uh you know running Magalu?
• 4:05
  SPEAKER_01: Like what typical challenges do you have?
• 4:08
  SPEAKER_03: The most typical challenge is to deal with different business areas, like logistically, financial.
• 4:15
  SPEAKER_03: Uh sometimes you need to apply some governance uh rules, uh sometimes you need to say, Oh, okay, that is using a lot of um data, or sometimes uh and now we are um magalu uh built uh an amazing thing, call it in Portuguese cerebro da lu, but in English we can say that's Lou Brain.
• 4:42
  SPEAKER_03: Lu is a chapter from Magalu, it's a woman.
• 4:46
  SPEAKER_03: Uh that's uh uh uh a nice, nice chapter.
• 4:49
  SPEAKER_03: You if you search uh in the web for Lou from Magalu, you will see uh the the how biggest is the chapter, and it's the face of Magalu.
• 5:02
  SPEAKER_03: And now we are building the Lou's brain, and that's it's a real challenge because when we nowadays we need to talk about AI how and how to use it, and Magalu is using AI to sell things like from using WhatsApp, and and and that is a new challenge.
• 5:22
  SPEAKER_03: No one did this.
• 5:24
  SPEAKER_03: This this is we we build that platform to say, okay, uh, you need a desk, we have a desk for you.
• 5:34
  SPEAKER_03: Uh and understand the needs from the consumer.
• 5:38
  SPEAKER_03: That's that's very interesting.
• 5:40
  SPEAKER_03: And the main challenge from for Magalu for uh looking uh by my eyes, looking as a data engineer, is to organize the house, uh to say, okay, this is from that business, they we use that, and that that is the main challenge.
• 5:59
  SPEAKER_01: Okay.
• 6:00
  SPEAKER_01: Um, I know that this question was not uh in the in the preparation code that we had, but uh just very quickly, do you use AI in your um data engine like in your pipelines today?
• 6:14
  SPEAKER_01: Or are you thinking of using AI at some point in your pipelines?
• 6:20
  SPEAKER_03: We are using both to develop uh to develop pipelines.
• 6:25
  SPEAKER_03: Yeah, okay.
• 6:25
  SPEAKER_03: Yeah, and we are building some steps of our platform like uh the data catalog, and that's very important for us because imagine we have 2000 pipelines where data come from, where data are going to, and now we are starting to use AI uh to integrate with some MECPs and say and see hey, your Spark is using a lot of um memory, but uh your executor is very very very larger than you need.
• 7:05
  SPEAKER_03: You can reduce your executor, for example.
• 7:08
  SPEAKER_03: Uh look more to execution, and and and we will talk uh more about this later, but data viability engineer is building uh a channel inside your chat to uh and and the AI is summarizing the error from our flow, for example.
• 7:30
  SPEAKER_03: Okay, you got uh out of memory error from Spark, and the AI say, Oh, I'm 99% uh uh sure that the problem is that and you are correct is that uh inside your the chat, and you can see the problem the chat and say, Okay, now I I know how to uh fix that problem, or say, okay, that's a stable, the platform, the pipeline is running, sometimes that got an error, or what do you need to do here?
• 8:01
  SPEAKER_03: Then that's uh our challenge with 2000 pipelines, that's a good solution.
• 8:06
  SPEAKER_01: Yeah, yeah, and that that's interesting that you are talking about this um way of using AI.
• 8:12
  SPEAKER_01: Um, because actually at Astronomer we have um uh what we call Astro Observe that allows you to uh observe your pipelines um and uh you know having like data lineage and so on.
• 8:25
  SPEAKER_01: And we actually have a root cause analysis feature where we use AI to actually uh know what happened in your uh in your pipelines and what might be the cause.
• 8:36
  SPEAKER_01: Uh well, I mean that that's really interesting.
• 8:38
  SPEAKER_01: I I think like many people are obviously using uh AI today for developing DAGs, uh, but I expect that more and more people will also have um uh workflows that use AI.
• 8:50
  SPEAKER_01: Uh so you you mentioned that you you are orchestrating around 2000 pipelines today.
• 8:57
  SPEAKER_02: Yeah.
• 8:58
  SPEAKER_01: Uh I have to ask you, like, how did Airflow become the heart of your uh data platform?
• 9:04
  SPEAKER_01: Like, why did you maybe you maybe you know why did you decide to use Airflow at your company?
• 9:11
  SPEAKER_03: Okay, nice.
• 9:12
  SPEAKER_03: Uh uh I I'm Matthias as an engineer.
• 9:16
  SPEAKER_03: I already worked with a lot of orchestrators, uh like uh Control M, uh Azure uh uh um AWS C AWS uh step functions, uh Azure data factor.
• 9:30
  SPEAKER_03: But what shine in my eyes was RFO, and when I came to uh Magalu, they are using Airflow, and I say okay, that's fine.
• 9:42
  SPEAKER_03: And um, but how that became the heart of our orchest orchestration.
• 9:49
  SPEAKER_03: Uh we started using GCP, uh okay.
• 9:52
  SPEAKER_03: Uh uh, and the focus uh on that uh was to use big query and storage.
• 10:00
  SPEAKER_03: Okay, big query and storage, that's fine, but how we will run our pipelines?
• 10:06
  SPEAKER_03: We can use other services uh inside GCP, but but with our demand of 2000 pipelines, and that will be a larger, larger cost.
• 10:19
  SPEAKER_03: And to organize the data proxies and centralize everything into a single scheduler, we began using Cloud Composer.
• 10:27
  SPEAKER_03: And Cloud Composer is their many managed Airflow.
• 10:39
  SPEAKER_03: And largely because the business units need data from various sources and applies a lot of transformations.
• 10:48
  SPEAKER_03: And this makes Airflow the center of data ingestion and from our operations.
• 10:55
  SPEAKER_03: Um they squad look to Airflow and say, okay, the task is green, my data is fine, but that's uh that's a way to see.
• 11:03
  SPEAKER_03: Uh, I know the we need more control than this, um, because we need to see that if the deck runs and the data is correct, uh the two uh part steps.
• 11:15
  SPEAKER_03: Um that is the way the full combo hurts of our orchestration, and today the data platform team manages and maintains the uh that health and ensure that's available to critical data processing processing to our business errors, okay.
• 11:36
  SPEAKER_01: And uh just uh out of curiosity, um when you started to use Airflow for the first time, did you find it hard to uh get started with, or did you find it okay, easy, like just out of curiosity, like you know, for beginners, for for people that maybe want to use Airflow at some point?
• 12:02
  SPEAKER_03: When I started to use Airflow, uh I got a uh a big gap between Control M and Airflow because uh when I was working with Control M, I worked for a bank and it's very, very, very bureaucratic.
• 12:19
  SPEAKER_03: That's a team that creates the pipeline dependencies.
• 12:24
  SPEAKER_03: That's this type of bureaucracy.
• 12:28
  SPEAKER_03: And when I started using RFU, I see the configurations needed and the concepts about operators.
• 12:37
  SPEAKER_03: I say, Oh, operators, it's a new thing, and uh and I see I have more operators than the the basic operator, I have custom operators, I can create my own operator, and that's incredible because uh when I started as data engineer, I worked with Hadoop and I created a Spark session with uh uh real uh difficulty because that's not easy to create a simple thing inside uh on-premise platform.
• 13:13
  SPEAKER_03: And when I started their flow, I was working using GCP.
• 13:18
  SPEAKER_03: I started with Composer, and I say, Okay, I need to implement my workflow here, and they are using just big query scheduled tables, and that's terrible for dependencies or to see okay, my table is updated, and that's terrible.
• 13:36
  SPEAKER_03: And I say to my manager, oh hey, can we can we use composer?
• 13:42
  SPEAKER_03: And uh that's my first time using composer.
• 13:45
  SPEAKER_03: So okay, you you can try.
• 13:47
  SPEAKER_03: I started using composer and their flow, I and exploring the airflow properties and developing with uh operators.
• 13:56
  SPEAKER_03: I'll say, okay, I will create just a simple thing, like okay, uh hello world script using Pyte operator, and that worked.
• 14:06
  SPEAKER_03: So I said, Oh, that's incredible.
• 14:10
  SPEAKER_03: Uh and after that, uh I started working with uh other operators and creating spark jobs and using a lot of things and integrating with some GCP things like uh cloud source and uh and big query.
• 14:28
  SPEAKER_01: Okay, so you you you would say that it was okay, right?
• 14:31
  SPEAKER_03: I mean it wasn't uh hard to no that's not hard because uh uh with my experience in when I started, I just worked with home-prem sphinx and Hadoop.
• 14:43
  SPEAKER_03: I I never had experience with Airflow, and they create uh entire uh pipeline dependencies with Airflow.
• 14:51
  SPEAKER_03: That's a very way to start.
• 14:55
  SPEAKER_00: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 15:03
  SPEAKER_00: From batch pipelines to training models and wiring up agents.
• 15:08
  SPEAKER_00: Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 15:14
  SPEAKER_00: Check it out at astronomer.io.
• 15:18
  SPEAKER_01: That's great, that's great.
• 15:19
  SPEAKER_01: Um, so actually, like um, you know, speaking about how you can like how hard it was to uh start with with Airflow, maybe for us as data engineers it's it's pretty easy, but what about non-engineers, right?
• 15:34
  SPEAKER_01: And I think um you told me that you built um a YAML wrapper um around Airflow so that you know actually like non-engineers can ship DAGs.
• 15:46
  SPEAKER_01: Um can you tell me more about that?
• 15:48
  SPEAKER_01: Like how does it work?
• 15:49
  SPEAKER_01: And you know, um, yeah, just like in in general, if you can tell me more about the solution, because I think many many actually companies uh use like YAML for generating DAGs in Airflow, but I would be curious to know about your implementation.
• 16:01
  SPEAKER_03: Oh, okay, nice.
• 16:02
  SPEAKER_03: Uh the and that's very important.
• 16:05
  SPEAKER_03: Like I said, the our main challenge is to uh attend all our business units here, and that's a real challenge.
• 16:14
  SPEAKER_03: Oh okay, let's go.
• 16:16
  SPEAKER_03: Uh our wrapper is a solution designed to simplify distract and load proxies.
• 16:21
  SPEAKER_03: Uh, system builders users most needed to collect data from database and and land it somewhere from analysis, in our case, data lake and big query.
• 16:32
  SPEAKER_03: These two bridge that gap, it allows them to join database information if various search for using dashboards like data size models and decision making.
• 16:44
  SPEAKER_03: The two uh how the two works and then the two works uh reading a YAML file contain like namespace tables and the other details, like executor size and driver size, and a lot of things from Spark, like options uh with call.
• 17:03
  SPEAKER_03: And and from there, uh it identifies the collection type like CDC, GDBC2, uh, it's an override pipeline, and GDBC incremental.
• 17:15
  SPEAKER_03: Uh the two manage the data read stage using checkpoints.
• 17:20
  SPEAKER_03: If needed, users can define column transformations and specify whether the data should live on only in the lake or lake and big query.
• 17:30
  SPEAKER_03: Uh and to keep it safe, we have a uh well-documented list of parameters defining what users can and cannot do.
• 17:40
  SPEAKER_03: Furthermore, every new implementation or modification goes through the data availability engineering and they approve and implement that uh modifications.
• 17:54
  SPEAKER_03: And that's that's a very good case for us because we with that wrapper we have more control over the pipelines.
• 18:04
  SPEAKER_01: Yeah, I mean, so so I guess like actually uh when you say more control over the pipelines, can you tell me more about that?
• 18:11
  SPEAKER_01: Like, what do you control exactly?
• 18:13
  SPEAKER_03: Like, do you control what operators it's more what yeah, it's uh different thing, it's more uh most valuable thing nowadays with AI, it's about metadata, okay?
• 18:28
  SPEAKER_03: Uh yeah, because uh with metadata from our pipelines like uh that came from that origin and that saves in that part of lake because we are using something like meta architecture here on our data lake, and uh we can say, hey, the data is coming from that MySQL database, and is it storing inside your trusted layer or or your gold layer or silver layer, and you can we can map where our data is being saved, saved that's cool.
• 19:09
  SPEAKER_01: Um I guess like uh um you know, like I I think like at some point you you talked about your um data reliability uh engineers um because obviously like self-servicing airflow in the way that you do uh requires uh data reliability engineering team.
• 19:32
  SPEAKER_01: Um so can you tell tell us more about this team?
• 19:35
  SPEAKER_01: Like how do they make sure that things do not uh turn on into a swamp?
• 19:41
  SPEAKER_03: Oh they are uh window to the company.
• 19:47
  SPEAKER_03: Okay, we can we can see that say that because every problem that business unit has, for example, oh my DAG is with a problem, I don't know what is that problem.
• 19:59
  SPEAKER_03: Because uh not all all chains from has a data engineer, they build the the proxies using our wrapper and say, Okay, that's working, and use the data inside BigQuery.
• 20:12
  SPEAKER_03: But uh to maintain that pipeline, we help them and say, Oh, we have we got a problem.
• 20:19
  SPEAKER_03: How you can uh how you can help and we start to analyze the pipelines, uh we look to the data, look mainly to airflow to see okay, airflow is correct, or airflow got an error, or the error is uh on our of pipe on our pipelines or everything that involves the pipeline.
• 20:44
  SPEAKER_03: But uh the focus to control our data is to uh focus on solution that ensure on extraction to any given source.
• 21:00
  SPEAKER_03: That's the most important thing, and because we are talking uh our data lake today is about four petabytes, and that's a lot of the data.
• 21:12
  SPEAKER_03: And data ability engineering assists change with maintenance, validation, corrections, and optimize both existing and new proxies.
• 21:23
  SPEAKER_03: This ensure we don't duplicate uh pipelines and uh help with the already existing pipelines, too.
• 21:33
  SPEAKER_01: That must be a lot of work uh for them.
• 21:36
  SPEAKER_01: Yeah, a lot, a lot of work.
• 21:37
  SPEAKER_03: It's about um ten uh search for words.
• 21:46
  SPEAKER_03: Ten people, yeah, yeah, ten people uh every day to uh help to help every everybody.
• 21:53
  SPEAKER_03: Yeah, yeah, but it's uh it's uh it's a real challenge.
• 22:00
  SPEAKER_01: Yeah, I imagine.
• 22:00
  SPEAKER_01: I imagine.
• 22:02
  SPEAKER_01: Um also like uh just to come back to what you said uh at the very beginning of the of the episode.
• 22:08
  SPEAKER_01: Um you mentioned about actually not very at the not at the very beginning because you also told us that uh when you um started using Airflow.
• 22:17
  SPEAKER_01: Uh can you tell us more about the custom operators?
• 22:20
  SPEAKER_01: I think like you are using some custom operators today, uh and custom operators are very popular um among the uh Airflow users.
• 22:29
  SPEAKER_01: So I'm curious, like how do you use custom operators?
• 22:31
  SPEAKER_01: And before how you use them, why did you have to create them in the first place?
• 22:38
  SPEAKER_01: Like, why did you create those custom operators?
• 22:41
  SPEAKER_03: That's a very good question.
• 22:43
  SPEAKER_03: Um we moved away from data proc operator mainly because it's very uh expensive.
• 22:52
  SPEAKER_03: And our strategy is to shift to Magaloo Cloud.
• 22:57
  SPEAKER_03: Um I will we will talk about this later, but Magaloo Cloud is uh public cloud from Magazine Luiza from Luiz from Magalu.
• 23:08
  SPEAKER_03: And we decide to focus on Kubernetes uh for process because our course operator does more than just enable Spark session, uh to give our greater control over Spark, job creation, session, parameter assignment, and the creation of specific Kubernetes resource needed to execution.
• 23:28
  SPEAKER_03: And we do all of this with the operator and uh creating the manifest, and that's a very we have a lot of control when we are creating a pipeline.
• 23:41
  SPEAKER_03: For example, when we are creating a pipeline using our airflow, we can say custom operator, you you use a small, medium, or large driver from Spark.
• 23:54
  SPEAKER_03: You can we can use a small executor, large, very ultra mega blaster, large executor.
• 24:04
  SPEAKER_03: And say, Oh, uh, you we can use that number of partitions, uh, and other values that we can pass through the operator, and that we replicate it inside the manifest that it starts the Spark session, and that helps a lot, a lot us to create uh like the YML wrapper, uh, to create a pattern to R 2000 pipelines.
• 24:39
  SPEAKER_01: Okay, yeah, I mean that that that makes a lot of sense.
• 24:42
  SPEAKER_01: Um actually I heard a couple of times people using custom operators to better control um how you know resources are used and so on, especially on Kubernetes.
• 24:54
  SPEAKER_01: Um I think like just like to um for the last question.
• 25:00
  SPEAKER_01: Um I think like you built a monitoring dashboard um on top of Airflow.
• 25:07
  SPEAKER_01: Um and so I'm curious because we know we always focus about I mean we always we always focus on data pipelines and so on, but we tend to forget that there is also like observability that is important.
• 25:20
  SPEAKER_01: Um and so can you tell us more about the dashboard that you built on top of Airflow?
• 25:25
  SPEAKER_01: Like what does it track, and you know, like what teams typically look at on this dashboard, and why did you build it?
• 25:34
  SPEAKER_03: Okay, and are you to take one step back just to explain a thing?
• 25:40
  SPEAKER_03: Uh sure uh when we are moving from GCP to MDC, MDC is the Magalu cloud, we are looking to more control over our pipelines.
• 25:52
  SPEAKER_03: Imagine to integrate a lot of pipelines to different cloud.
• 25:57
  SPEAKER_03: We need to orchestrate that pipelines to see what is production and what is uh testing, and we start using our flow tags.
• 26:07
  SPEAKER_03: Our flow tags are a magic thing because uh it's because when we start our monitoring process when I build, I start to query our our flow database and and what information we we can take from there.
• 26:27
  SPEAKER_03: Uh that's uh that's uh beautiful inside that database because uh today we m we are monitoring dag execution executions through the day.
• 26:41
  SPEAKER_03: We have a timeline, and and you can see okay, 2 a.m.
• 26:46
  SPEAKER_03: we have three pipelines running, 10 a.m.
• 26:50
  SPEAKER_03: we have 10 pipelines running, and how I can take that information.
• 26:54
  SPEAKER_03: I'm I'm selecting from the airflow database the scheduling times that from all our pipelines, and I'm creating a view inside your monitoring too.
• 27:06
  SPEAKER_03: And that's that uh worked very well.
• 27:08
  SPEAKER_03: And you can we can uh when we are building a new pipeline, for example, we can look to that information and say, okay, at 10 a.m.
• 27:18
  SPEAKER_03: that has a lot of pipelines, uh, I will place that 3 p.m.
• 27:24
  SPEAKER_03: That's better.
• 27:25
  SPEAKER_03: Uh that's create that type of decision, and that's not uh usual, but very important.
• 27:33
  SPEAKER_03: Um, and another thing that is very important is the error rates.
• 27:39
  SPEAKER_03: We can, like I said, we are using tags to see production things and test things, uh or you can create every tag you want, uh, but uh we are using that tags to separate in inside the dashboard all the running tags, all the error tags, uh that's very important.
• 28:04
  SPEAKER_03: And the one of the most important for me uh it's the classification of the DAGs.
• 28:12
  SPEAKER_03: How that works.
• 28:14
  SPEAKER_03: I'm collecting the last five executions from DAG and I classifying that way.
• 28:21
  SPEAKER_03: If uh the last five executions are green or correct, correct, I classify this DAG as success.
• 28:31
  SPEAKER_03: If that's one error or one stability, for example, uh it's partially correct.
• 28:39
  SPEAKER_03: And if that has more than one error and less than five, that is intermittent.
• 28:48
  SPEAKER_03: That's that DAG has a problem.
• 28:50
  SPEAKER_03: You don't know, but has a problem.
• 28:53
  SPEAKER_03: And if all the five last executions has a problem, okay, that's a total failure.
• 28:59
  SPEAKER_03: Your DAG is with a big problem.
• 29:02
  SPEAKER_03: And we apply that rule inside over all our DAGs.
• 29:09
  SPEAKER_03: Every execution from DAG creates that uh vision, and uh crucially, it also tracks if DAG is running with a normal duration based on median of research executions.
• 29:26
  SPEAKER_03: I believe everything should look closely to the DAG execution history and the health data provided by Airflow metadata.
• 29:37
  SPEAKER_03: Uh and it's the potential to improve your DAX, just seeing apply some small change, like uh for example, a Spark changing a partitioning, or if that is with wrong executor type, uh and that's uh uh very part.
• 30:02
  SPEAKER_01: Okay, so so I think like I mean this is super interesting.
• 30:05
  SPEAKER_01: Um especially like the I mean, you know, when you mentioned that you have this view where you know exactly for every hour how many DAGs are running.
• 30:16
  SPEAKER_01: I think this is a great one because like people tend to forget that if you have all of your pipelines running at the same time, it's actually very easy to end up with, especially when you are managing like thousands of pipelines, it's very easy to have like most of your pipelines running at the same time, and you are putting a lot of pressure on your uh you know Airflow scheduler and so on, and that can ultimately uh put some lag on on your uh pipelines, on your workflows.
• 30:43
  SPEAKER_01: Uh so that's actually a very nice way to okay, observe that and then make sure that your uh pipelines are spread in a way that they don't run all at the same time.
• 30:53
  SPEAKER_01: Thank you so much for uh for sharing uh all of those uh insights, Matheus.
• 30:58
  SPEAKER_01: I wonder, like you know, as a very last question to you, um if people want to, I mean if people want to contact you, what is the best way for for them to do it?
• 31:11
  SPEAKER_03: Okay, I don't have social media, but you can find me uh with my linkaging.
• 31:17
  SPEAKER_03: It's my name, uh Mateus Amy Ferreira.
• 31:21
  SPEAKER_01: Okay, perfect.
• 31:23
  SPEAKER_01: I will uh I will make sure that uh people will be able to find you in the description.
• 31:28
  SPEAKER_01: Okay.
• 31:28
  SPEAKER_01: Well, Marius, it was a wonderful pleasure to have you uh in this podcast.
• 31:32
  SPEAKER_01: Thank you so much again for uh sharing all of your uh great insights relating in relating to um uh airflow and uh data engineering, and uh yeah, hope to see you again for another one.
• 31:44
  SPEAKER_03: Oh okay, thank you.
• 31:45
  SPEAKER_03: Thank you so much.
• 31:46
  SPEAKER_03: Can I say just one sentence in Portuguese, uh just one sentence that I believe so much, uh if I can spread that word.
• 31:58
  SPEAKER_03: Uh sure.
• 31:59
  SPEAKER_03: Uh a gente nunca sabe quando está vivendo bons value steps.
• 32:04
  SPEAKER_03: Uh what that means in English, we don't know when we are living the good old times.
• 32:10
  SPEAKER_03: Uh that is a sentence like I like to say when I participate in podcast.
• 32:17
  SPEAKER_01: Amazing.
• 32:17
  SPEAKER_01: Thank you so much, Marius.
• 32:19
  SPEAKER_01: Have a wonderful day.
• 32:20
  SPEAKER_01: Take care to see you.
• 32:21
  SPEAKER_03: Thank you.
• 32:23
  SPEAKER_00: Thank you for listening to the Data Flowcast.
• 32:26
  SPEAKER_00: Check the show notes for links to everything mentioned today.
• 32:30
  SPEAKER_00: If you're enjoying the podcast, subscribe so you never miss an episode.
• 32:35
  SPEAKER_00: And leave us a five-star review.