Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi

Transcript

Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: The Zach Syntax to define data mesh is a decentralized data architecture, and every team trades data as a product.
• 0:06
  SPEAKER_03: Every team got the domain of their data, and their product is basically a table, and they provide the result to someone else.
• 0:13
  SPEAKER_03: That could be a business part or another team inside your company.
• 0:17
  SPEAKER_03: So we are got vertical team.
• 0:19
  SPEAKER_03: And when I speak about vertical team, it's someone that works really close with the business.
• 0:23
  SPEAKER_03: So it could be a marketing media team, sales team, and inside the team, there are both data engineers and data analytics.
• 0:35
  SPEAKER_00: You're listening to the Data Flow Cat, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:45
  SPEAKER_01: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:50
  SPEAKER_01: Join the Astronomer Road Show for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:58
  SPEAKER_01: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:07
  SPEAKER_01: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:12
  SPEAKER_01: Register now.
• 1:13
  SPEAKER_02: The link is in the show notes.com.
• 1:28
  SPEAKER_02: How are you doing, Alberto?
• 1:30
  SPEAKER_03: I'm fine.
• 1:30
  SPEAKER_03: I'm really glad to be here, Mark.
• 1:33
  SPEAKER_02: That's awesome.
• 1:34
  SPEAKER_02: I'm very glad that you are here too, because we're gonna go through very interesting topics today for that episode, especially regarding like why choosing airflow over many different orchestrators.
• 1:45
  SPEAKER_02: Also a bit of Dela Mesh.
• 1:47
  SPEAKER_02: Actually, this is I wouldn't say it's pretty new for me, but I don't know that much about Delamesh, so I'm very interested by what you will say about this approach.
• 1:55
  SPEAKER_02: And also we will highlight some interesting implementations at lastminute.com, such as how they use SQL sensors and other things that I'm sure you will be interested by.
• 2:07
  SPEAKER_02: Without further ado, can you tell us more about what you do at lastminute.com, Alberto?
• 2:12
  SPEAKER_02: What is your role and what is typically your day?
• 2:15
  SPEAKER_03: Yeah, I'm a data architect in lastminute.com and I'm a team composed by three people.
• 2:23
  SPEAKER_03: And we are in charge to choose the best practices and tools inside our department and also data modeling.
• 2:33
  SPEAKER_03: And we help all the other teams inside our department, basically.
• 2:38
  SPEAKER_03: And yes, when we find a tool, we will find a way to spread the knowledge inside all of the teams and find, as I said, the best practices and use them in the correct way.
• 2:51
  SPEAKER_02: Okay.
• 2:52
  SPEAKER_02: So what teams are you working with?
• 2:54
  SPEAKER_02: Like data engineers, data scientists, a mix?
• 2:58
  SPEAKER_03: Yes, I can say that we work from data platform engineers, so from the beginning, that bring the data inside our department.
• 3:06
  SPEAKER_03: Also network specialist, data engineer, data analyst is another other in here in Lasminute, data scientists is in another department, but we work also with them.
• 3:18
  SPEAKER_02: Okay, so very different profiles.
• 3:21
  SPEAKER_03: Yeah, all of the data world is is is in touch.
• 3:23
  SPEAKER_03: We are a tech company here in lasmino.com.
• 3:26
  SPEAKER_03: And of course, we sell company, we sell holidays, but our core business is to be a tech company.
• 3:32
  SPEAKER_02: Okay, okay.
• 3:34
  SPEAKER_02: I assume that it's a big responsibility for you to choose the right tool for the right team.
• 3:39
  SPEAKER_03: Yes, of course.
• 3:40
  SPEAKER_03: Uh we have got help from others, but yes, uh I agree with you.
• 3:46
  SPEAKER_02: Tell me more.
• 3:47
  SPEAKER_02: I think so today you are using Airflow at lastminute.com, but I don't think it was the case for I mean it was not the case at the beginning, right?
• 3:56
  SPEAKER_02: I think you introduced Airflow at some point.
• 3:58
  SPEAKER_02: Can you tell us more about what orchestrators or what schedulers you were using before Airflow and also why did you pick Airflow?
• 4:09
  SPEAKER_03: Yes, thank you for your question.
• 4:11
  SPEAKER_03: Yes, I joined the company in 2020 and the organization of the company was different, so I don't want to blame no one, but at the beginning, I counted like seven different orchestrators, starting from Randex, Crontup, Scheduler for the Cloud, Azkaban, custom scheduler.
• 4:33
  SPEAKER_03: So when we joined all of the teams, it was easy to understand that we needed one to rule them all.
• 4:40
  SPEAKER_03: And then we made a search on the market, we find Airflow.
• 4:44
  SPEAKER_03: It was the beginning of the peak.
• 4:46
  SPEAKER_03: I mean, because in 2022 was already the top of the market, Airflow.
• 4:50
  SPEAKER_03: And we found that everything of our requirements were fit.
• 4:54
  SPEAKER_03: So execute Java, execute Python, SQL integrated with our data rouse because we had more than one.
• 5:03
  SPEAKER_03: And Airflow fit really well and embrace it in step by step.
• 5:08
  SPEAKER_03: But at the end, uh Airflow is able to do whatever the all of the other schedulers did in the past.
• 5:15
  SPEAKER_02: And uh so you decided to go with Airflow in 2022, right?
• 5:20
  SPEAKER_03: We make a POC in 2020.
• 5:23
  SPEAKER_03: In the 2022, we was uh the migration handed, so all the I think that 2021 was the full migration of all of the artifacts.
• 5:33
  SPEAKER_02: And did you feel that for the learning curve of Airflow was steep?
• 5:38
  SPEAKER_02: Did you feel that it was hard to get started with Airflow?
• 5:41
  SPEAKER_02: And you can be honest, by the way.
• 5:42
  SPEAKER_02: Like you can truly say what you like, what was your feeling when you started with Airflow?
• 5:47
  SPEAKER_03: Thank you to you because when I started, I used all of your material and I learned thank you to you.
• 5:52
  SPEAKER_03: And also in lasminute.com, we had a learning path.
• 5:56
  SPEAKER_03: So our newbie followed the learning path, and some video is from you.
• 6:01
  SPEAKER_03: So maybe at the beginning it's different, it could be tough.
• 6:06
  SPEAKER_03: But if someone is able to introduce you and you've got the infrastructure already deployed, and you have got uh DAX to copy paste and task to copy paste, and someone that can help you, it's definitely easy.
• 6:20
  SPEAKER_03: At the beginning, maybe there is some tasks that are complicated, uh, not airflow task, infrastructure, CS CD, testing, understand the network.
• 6:30
  SPEAKER_03: That could be tough at the beginning, yeah.
• 6:32
  SPEAKER_02: Yeah, yeah.
• 6:33
  SPEAKER_02: So you would say like everything around Airflow, basically, like all the I will maybe like the DevOps part, like how do you scale Airflow, how do you implement your CICD pipeline, how do you like everything around is maybe complex, especially when you run want to run Airflow at scale.
• 6:47
  SPEAKER_03: Exactly.
• 6:48
  SPEAKER_03: And also because we got a lot of teams in our department, and basically we are using just one instance, or two instances, one of production and one on pre-product production.
• 6:58
  SPEAKER_03: So all of the teams schedule on the same and on the same environment.
• 7:02
  SPEAKER_03: So we need all of this uh DevOps work, like you said, working properly.
• 7:06
  SPEAKER_02: Yeah, yeah, that's interesting because like some people I saw on the internet that some people feel that Airflow is difficult to scale.
• 7:13
  SPEAKER_02: But the reality is Airflow scales, but you will get the same kind of issues that you will get with any other tools.
• 7:20
  SPEAKER_02: Whenever you want to scale your tool, you have so many things to think about.
• 7:24
  SPEAKER_02: Again, like in terms of your CICD, your resources, how you share your resources, how do you test your resources, and so on.
• 7:31
  SPEAKER_02: And that's why there is an astronomer if you want to manage that part for you, but still, it's always complicated to manage everything around the tool that you install in your.
• 7:40
  SPEAKER_03: Because the inner flow, the different types of job that you can uh deploy are very wide, yeah.
• 7:47
  SPEAKER_03: And maybe there are some SQL queries that uh give the resources to the data warehouse, or maybe other ones that uh use the resources.
• 7:56
  SPEAKER_03: So it depends.
• 7:57
  SPEAKER_03: Yeah, yeah.
• 7:59
  SPEAKER_02: Yeah, no, I agree.
• 8:00
  SPEAKER_02: I agree.
• 8:01
  SPEAKER_02: That's the I would say that's the most difficult part of having an orchestrator, right?
• 8:04
  SPEAKER_02: It's that you have so many things that you can orchestrate, so many different things that you can orchestrate.
• 8:09
  SPEAKER_02: So obviously it comes at some complexity.
• 8:12
  SPEAKER_02: Yeah.
• 8:14
  SPEAKER_02: So I think like the can you tell us more about the way during our conversation before that episode, I think you mentioned the fact that you have horizontal and vertical teams, and uh and also you have implemented the data mesh approach for those teams.
• 8:31
  SPEAKER_02: You know, like for the listeners that are not familiar to with this concept, and actually I'm not that familiar with that concept too.
• 8:38
  SPEAKER_02: Can you explain how you've organized your teams and how Airflow fits into this data mesh architecture?
• 8:45
  SPEAKER_02: Starting with what exactly is a data mesh for you?
• 8:48
  SPEAKER_03: Yeah, the Zach Synthes to define data mesh is a decentralized data architecture, and every team trades data as a product.
• 8:57
  SPEAKER_03: And every team got the domain of their data, and their product is basically a table, and they provide the result to someone else.
• 9:06
  SPEAKER_03: That could be a business part or another team inside your company.
• 9:10
  SPEAKER_03: And so we are got a vertical team, and when I speak about vertical team, is someone that works really close with the business.
• 9:18
  SPEAKER_03: So it could be marketing media team, sales team, and inside the team, there are both data engineers and data analytics, data analyst members.
• 9:27
  SPEAKER_03: And after that, we have got uh horizontal teams like my team, data architecture, that provide, like I said at the beginning, best practices to the detail, uh data modeling, whatever.
• 9:38
  SPEAKER_03: Another team like Data Platform that is in charge to bring all of the data inside our platform, and also data governance that do a lot of really tough work, like GDPR, naming convention, and lineage root grants, and blah, blah, blah.
• 9:54
  SPEAKER_03: And so the vertical teams think and work just for the business because all of the technicality has been held by us, like we can say, and they do not bother about platform, infrastructure, scaling, and they work close with the data analyst and also close to the business.
• 10:13
  SPEAKER_03: And in our department, we got we are got 12 teams, 13 teams, I could say, and we work more technical, so the horizontal team, and more deep on the business aspect, the vertical team.
• 10:26
  SPEAKER_02: Okay.
• 10:28
  SPEAKER_02: And so, yeah, go ahead.
• 10:29
  SPEAKER_03: Yeah, and answering your the your second question.
• 10:32
  SPEAKER_03: Because every single team has got his domain, he has got a lot of sources, a lot of requirements.
• 10:39
  SPEAKER_03: So maybe a finance team needs data from uh, I don't know, 30 ster 30 sources, and in their DAG, we need to put a SQL sensor.
• 10:50
  SPEAKER_03: SQL sensor is our main task used in source company because you need to understand when your data is ready and you can start with your transformation.
• 10:58
  SPEAKER_03: So, for example, the sales team, which is the at the end, all the pipeline, I got a lot of SQL sensors to understand all of the sources that we have that are more than 200, are waiting for.
• 11:10
  SPEAKER_03: And when they are ready, they could start.
• 11:13
  SPEAKER_03: And this is really nice because if you are able to see all of the pipelines coming from all of the different teams join together, and the SQL sensors are, I could say, that it's a mandatory task because they orchestrated really well all of the stuff.
• 11:28
  SPEAKER_02: Okay, that's super interesting.
• 11:30
  SPEAKER_02: Just to summarize, and you tell me if I'm wrong, so you have vertical teams that are close to the business, to the executives.
• 11:37
  SPEAKER_02: So, for example, the marketing.
• 11:38
  SPEAKER_02: So we can imagine truly those teams are silos.
• 11:42
  SPEAKER_02: Silos.
• 11:43
  SPEAKER_02: So you have the marketing team, you have maybe an I don't know, another team, whatever.
• 11:47
  SPEAKER_02: And then you have the horizontal teams that are actually walking across all of those vertical teams.
• 11:53
  SPEAKER_02: And so they provide what those vertical teams need to work.
• 11:58
  SPEAKER_02: Is that correct?
• 11:59
  SPEAKER_02: Is that the right way to say it?
• 12:00
  SPEAKER_03: Yeah, exactly.
• 12:00
  SPEAKER_03: If you try to on Google to write data mesh, that is exactly the image that you explain.
• 12:05
  SPEAKER_03: Vertical, like silos from finance, sales, uh, whatever you want, and horizontal, the team they provide the infrastructure and the best practices to work.
• 12:16
  SPEAKER_02: Okay.
• 12:16
  SPEAKER_02: And then you have like data products that you define as a table in a database or like a file, or is that correct?
• 12:25
  SPEAKER_03: Exactly.
• 12:25
  SPEAKER_03: A data product could be is something that you are supposed to be tested, and there are data that are uh are fine, are good.
• 12:32
  SPEAKER_03: And could be a table, could be a repo, could be a dashboard, or even if an Excel or a Google Sheet, or also a Slack channel.
• 12:39
  SPEAKER_03: What is meaningful is that the data is correct, and you provide the except data to the other team.
• 12:46
  SPEAKER_02: Okay.
• 12:46
  SPEAKER_02: And so, truly, like what those vertical teams care about are the data products.
• 12:52
  SPEAKER_02: Like they don't care that much about the architecture and so on, they just care about if the data is available or not, if the spreadsheet is available or not, and so on.
• 13:01
  SPEAKER_03: Every data engineer is different.
• 13:02
  SPEAKER_03: A lot of them are enthusiastic, so they want to know about how Airflow works, they want to know which cloud we are working on.
• 13:10
  SPEAKER_03: But it's because people are curious, they want to understand.
• 13:13
  SPEAKER_03: But to be honest, they could work without knowing nothing about a lot of stuff.
• 13:17
  SPEAKER_03: Okay.
• 13:18
  SPEAKER_02: Okay.
• 13:19
  SPEAKER_02: And so can you can you tell me more about the SQL sensor?
• 13:23
  SPEAKER_02: I know that you just touched on it, but can you give me uh just like a very specific example, maybe, of where it is useful for you?
• 13:31
  SPEAKER_03: Yeah, for example, like I said, the sales team, the sales team get data from all of the other teams because for the yesterday sales report, that was one of the my uh more important reports.
• 13:45
  SPEAKER_03: For example, the Google data came just once a day at the 7 a.m.
• 13:49
  SPEAKER_03: from from Sam from Web Analytics.
• 13:51
  SPEAKER_03: Uh, I'm saying about this stuff.
• 13:53
  SPEAKER_03: And uh so where the data is coming at 6 a.m., 7 a.m., you want to provide as fast as possible the results of the yesterday sales.
• 14:01
  SPEAKER_03: Of course, some data you have in real time.
• 14:03
  SPEAKER_03: But in the morning, in a really close time, you need to, you all of your ATL inside Airflow need to execute really quickly.
• 14:11
  SPEAKER_03: And when you got all of your dependencies, you put in your SQL sensor, you're able to understand which are your requirements and which are your sources.
• 14:20
  SPEAKER_03: And we create this custom operator called SQL sensor with Slack notification.
• 14:25
  SPEAKER_03: And maybe if if a data engineer is commuting by work or is in a meeting, he's able to check inside his Slack channel which is the source that is not available yet.
• 14:37
  SPEAKER_03: And is he able to drive to the people that need to solve this problem, or just because he understands it's not his fault.
• 14:46
  SPEAKER_03: And of course, you got a dashboard, but the notification in this case is far more helpful.
• 14:52
  SPEAKER_03: And and yes, you can think that there are few sources, but at the end of the day, uh many teams have got uh 20, 30, 40 sources inside that SQL sensor because they need to understand, they need to wait for all of the dependencies.
• 15:10
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 15:20
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 15:31
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 15:40
  SPEAKER_02: I have many questions for you.
• 15:42
  SPEAKER_02: So for the SQL sensor, do you wait for an entry in?
• 15:46
  SPEAKER_02: So do you have do you have one table, like one SQL sensor per table?
• 15:53
  SPEAKER_03: No, no, we have one SQL sensor for many tables.
• 15:56
  SPEAKER_02: Oh, okay.
• 15:57
  SPEAKER_02: So you have one sensor that checks multiple tables.
• 16:01
  SPEAKER_03: Exactly, yes.
• 16:02
  SPEAKER_03: We got right now we can call a semaphore table, and in this table, there are all the start and end or the of the previous ATL.
• 16:13
  SPEAKER_03: And you can write I started, I finished, I can finish with warning, and the result.
• 16:19
  SPEAKER_03: So the people that use the data check on that table if the table, if the data are available or not available.
• 16:25
  SPEAKER_03: And the SQL sense in some cases run every minute, in some cases every five minutes or 15 minutes.
• 16:31
  SPEAKER_03: It depends on use case, need to be really real time, or is the use less having the report at 10 a.m.
• 16:37
  SPEAKER_03: or 11 a.m.
• 16:38
  SPEAKER_02: Okay.
• 16:38
  SPEAKER_02: And that sensor is the first task of your DAG?
• 16:42
  SPEAKER_03: A lot of time, yes.
• 16:43
  SPEAKER_03: In 95% of cases, yes.
• 16:46
  SPEAKER_02: Okay, so you have a DAG that runs what daily, and then you have the SQL sensor that's gonna check every minute or every five minutes or every 15 minutes?
• 16:54
  SPEAKER_03: Yes, exactly.
• 16:55
  SPEAKER_03: Yes.
• 16:56
  SPEAKER_03: Okay.
• 16:57
  SPEAKER_03: For the day for the daily DAG, yes.
• 16:59
  SPEAKER_02: Okay, I have to ask, did you get any performance issue because of that?
• 17:05
  SPEAKER_02: Or not really?
• 17:07
  SPEAKER_03: We had performance issues, but we needed to improve the scheduler resources for CPU and RAM of the scheduler.
• 17:17
  SPEAKER_03: We are up and running with two schedulers and we faced with issue, but we understand that it was the scheduler, yes.
• 17:24
  SPEAKER_02: Okay, so that was because of the resources, like you didn't have enough resources to run all of those SQL sensors.
• 17:30
  SPEAKER_03: Yeah, I think that in in certain times, uh, for apps uh, the first team, so the data platform is not able to provide the data.
• 17:38
  SPEAKER_03: And maybe it's really late.
• 17:39
  SPEAKER_03: So at 10 a.m.
• 17:40
  SPEAKER_03: in the morning, and they provide yet data.
• 17:43
  SPEAKER_03: All of the DAX have been started, and all the SQL sensors are checking.
• 17:48
  SPEAKER_03: The data is available, the data is available, and uh there is a lot of task learning and the scheduler have pain.
• 17:55
  SPEAKER_03: So, in that case, we need a lot of resources.
• 17:58
  SPEAKER_02: Okay, got it.
• 17:59
  SPEAKER_02: That that that is super interesting.
• 18:01
  SPEAKER_02: And what what we are using, like what you were using before the SQL sensor, or you just started with the SQL sensor right away.
• 18:09
  SPEAKER_02: Did you use something else before?
• 18:12
  SPEAKER_03: It was hard because right now we are supposed to give our report early in the morning.
• 18:18
  SPEAKER_03: Before this, uh, the report was uh delivered in the afternoon or or maybe late.
• 18:24
  SPEAKER_03: So, of course, we will find a solution, but this is the quickest pollution that is possible.
• 18:29
  SPEAKER_03: So if you do not lose time, it's important to have SQL sensor or something that check in uh in a few seconds if the data is available.
• 18:39
  SPEAKER_02: You you always used the SQL sensor, like right at the beginning, you decided to do with the SQL sensor.
• 18:46
  SPEAKER_02: Oh, okay, okay.
• 18:46
  SPEAKER_02: Yes.
• 18:47
  SPEAKER_02: I thought that you were using something else before the SQL sensor.
• 18:50
  SPEAKER_02: Okay, you mentioned, I heard you right, you mentioned that you've created a custom operator.
• 18:56
  SPEAKER_02: So this SQL sensor is actually a custom operator, right?
• 19:00
  SPEAKER_02: So you have the SQL sensor with the Slack notification.
• 19:04
  SPEAKER_02: So it's interesting because like actually many people create their own custom operators, many companies do that.
• 19:11
  SPEAKER_02: Can you tell me more about one specific custom operator that I know you are using because we've talked about that, that allows you to execute uh something uh interesting?
• 19:20
  SPEAKER_03: Yeah, I don't know if if I'm proud about that for sure, work properly.
• 19:25
  SPEAKER_03: I should I don't know, because we are got legacy, legacy Java jar, and we need in some way to execute this code.
• 19:34
  SPEAKER_03: And as all of you know, Airflow is based upon Python.
• 19:38
  SPEAKER_03: And if you want to execute Java code, maybe you need to find another solution.
• 19:43
  SPEAKER_03: But we tried.
• 19:44
  SPEAKER_03: We tried in our team, we tried to create.
• 19:48
  SPEAKER_03: I'm a little bit technical right now, but with Python, you can create a subprocess, and basically you are going to create a terminal, a bash, and inside that terminal you can execute a terminal command.
• 19:59
  SPEAKER_03: And in that phase, we place the Java command to execute the jar.
• 20:06
  SPEAKER_03: It could be weird because it's something unorthodox to execute a Java jar inside a Python frame scheduler.
• 20:15
  SPEAKER_03: But at the end of the year, and after four years, we haven't got any problem.
• 20:19
  SPEAKER_03: It worked properly.
• 20:21
  SPEAKER_03: Maybe it's some millisecond slower, but for sure it's robust.
• 20:25
  SPEAKER_03: Yes, from the as a data architect, maybe I'm not proud.
• 20:29
  SPEAKER_03: As a result oriented, I'm really happy about that.
• 20:34
  SPEAKER_02: Look, I think it I don't know.
• 20:36
  SPEAKER_02: I think like some people will say it's okay to do that, some people will say it's a bit exotic to do that.
• 20:41
  SPEAKER_02: But I think like at the end of the day, it proves one point that Airflow is very flexible.
• 20:47
  SPEAKER_02: Ultimately, you can do a lot with Airflow.
• 20:49
  SPEAKER_02: And the reality, to your point, is when you are working in a company, obviously you have some tech depth that you need to deal with, and migration is definitely not something funny to do.
• 21:00
  SPEAKER_02: And you have deadlines, you have a lot of things to do, very fast, and blah blah blah.
• 21:04
  SPEAKER_02: So you just find the solutions that you can find as quickly as possible, and then when you have more time, then maybe you will improve the solutions.
• 21:12
  SPEAKER_02: But again, I think that proves one point that Airflow is flexible, and you can do a lot with it, even executing Java code.
• 21:19
  SPEAKER_02: Uh so that's really cool.
• 21:21
  SPEAKER_03: Yeah, yeah, I agree.
• 21:22
  SPEAKER_03: I agree.
• 21:22
  SPEAKER_03: Flexible is one of the main features of Airflow, for sure.
• 21:27
  SPEAKER_02: Did you create any other custom operators?
• 21:30
  SPEAKER_02: Any custom operators that you can think of are uh that could be interesting for the audience?
• 21:36
  SPEAKER_03: Something that I could mention that we created a repository that all of our custom repositories are inside.
• 21:43
  SPEAKER_03: It's managed by Has, by the data architecture team, managed by a merge request.
• 21:48
  SPEAKER_03: This is important because if you got more than one team as a data engineer, you need to take care of all this customer that has been created because they need to be resilient, they need to.
• 22:00
  SPEAKER_03: work properly, need to be tested, of course, because this could after that go in production.
• 22:05
  SPEAKER_03: So what I suggest is to create a dedicated repository centralized with of course merge request CD and all of the people can use that but centralized as a repository.
• 22:17
  SPEAKER_02: Okay.
• 22:18
  SPEAKER_02: Are you using DBT?
• 22:21
  SPEAKER_03: Yeah I'm a big fan of DBT.
• 22:23
  SPEAKER_03: Yes.
• 22:23
  SPEAKER_03: We are trying to migrate, yes.
• 22:26
  SPEAKER_02: Okay.
• 22:26
  SPEAKER_02: Are you using DBT with Airflow today?
• 22:29
  SPEAKER_03: Yes.
• 22:30
  SPEAKER_03: Right now, yes.
• 22:31
  SPEAKER_03: How?
• 22:32
  SPEAKER_03: Just the 5% of our transformation are in DBT.
• 22:36
  SPEAKER_03: So we are at the beginning of our path.
• 22:38
  SPEAKER_03: We are understanding which is the best way to to schedule the DBT image, the DB artifact.
• 22:45
  SPEAKER_03: So we are in some way in trial and error phase.
• 22:49
  SPEAKER_03: But definitely I'm a big fan of DBT.
• 22:51
  SPEAKER_03: And in my mind dbt and Airflow are close together and I think this is amazing using both of them.
• 22:59
  SPEAKER_02: And like how are you using DBT with Airflow today?
• 23:02
  SPEAKER_02: Are you using the bash operator or something else?
• 23:05
  SPEAKER_03: Right now we are using a Kubernetes pod operator.
• 23:08
  SPEAKER_03: So through the CI CD, GitLab we are using GitLab in our company, upload an image inside our artifact registry and after that we use a Kubernetes pod operator to execute there.
• 23:20
  SPEAKER_03: Maybe we use another another way maybe a serverless serverless option or something like that.
• 23:27
  SPEAKER_03: And we got also a demo from astronomer team that have inside the inside the platform dbt and our flow custom operator we have a follow-up on September.
• 23:38
  SPEAKER_03: Right now in our company we had so many custom operators we will see in the future if it will use them.
• 23:44
  SPEAKER_03: But for sure if we want to start for stretch uh it's something usual to take a look of astronomer.
• 23:49
  SPEAKER_02: Yeah so you got a chance to see uh cosmos right is that what you are exactly yes yeah yeah I think like for like for the people that are listening so obviously it depends on your architecture and on a lot of things but definitely if you are using dbt with airflow today I would recommend you to take a look at Cosmos.
• 24:08
  SPEAKER_02: It's an open source project you don't have to be an astronomer customer to use it and it will save you a lot of time to integrate dbt with with Airflow especially if you are using the bash operator or any other operators today because as and I think that's the case with the Kubernetes pod operator that you are using.
• 24:24
  SPEAKER_02: With dbt and airflow like you don't have you don't see what's going on in your dbt project on the Airflow UI right you only see the Kubernetes operator but you don't know what's going on in your dbt project when you run it.
• 24:37
  SPEAKER_02: You have to look at the yeah exactly yes it's correct so this is what like Cosmos brings to you that you have this lineage this observability over your DBT project right into Airflow.
• 24:50
  SPEAKER_02: But again depends on the way you are you you need to run dbt and so on.
• 24:54
  SPEAKER_02: I guess Alberto I have a question for you how you mentioned that you are working with different teams and I imagine that those teams have different skills right like some are data scientists some are data analysts data engineers and so on.
• 25:08
  SPEAKER_02: Do they all create data pipelines or do you have a like a framework for them so they can just create data pipelines but in the very uh restricted way I would say or like with some guidelines at least like how does it work yeah at the beginning we created a template okay but after one year no one used templates so far.
• 25:31
  SPEAKER_03: Okay.
• 25:32
  SPEAKER_03: Yeah and and yes right now we got uh a huge competencies inside our department so data engineering that analyz and data analysts are able to create to create DACs without problem at the beginning what we did uh in my teams is to create the first DAC together in peer programming with everyone and understand exactly what they pay in because someone has got already Python skills someone not and in peer-to-peer programming you are able to understand not so many lessons one or two and after that they are able to create an amazing DAG because at the end of the day a lot of data analysts create a lot of DAGs but it's just a copy paste of a lot of tasks and even if they are not they are coming for R for R and not from Python they are able to create a really good DAG there is one colleague for example which is an ABI expert A B testing expert sorry and he created an amazing DAG making Python script inside our conference page getting information from the data warehouse and at the end write the result of the A B testing is a back to to our confluence page this is an amazing result from a data analyst because he just wrote Python and he got all of the network opened and the infrastructure ready and he is able to he was able to create a really good result basically without being a data engineer.
• 27:02
  SPEAKER_02: That's awesome.
• 27:03
  SPEAKER_02: That's really awesome wow great I think like just to like for the people that are listening to us do you have any advices regarding airflow or maybe like the the experience that you had implementing that data mesh architecture and do you have any advices for them?
• 27:21
  SPEAKER_02: Maybe one or two?
• 27:23
  SPEAKER_03: Okay hard question hard question.
• 27:25
  SPEAKER_03: The first point that me and my colleagues in data architecture think is valuable is speak with the team.
• 27:32
  SPEAKER_03: So communication is the key.
• 27:34
  SPEAKER_03: And when you understand their requirements the the first point is it became easier because without speaking with all the people you're not able to understand what they do first of all with the remote working and all of the other all of the other cool ways to work but the communication is hard and understand the people and yes and after that of course there are videos there are a lot of tools to understand a little bit to peer program a little bit of peer programming together can help us to understand the level of the people can understand you if the people understand how to create DAGs or other stuff in our work.
• 28:12
  SPEAKER_03: And yes and after that of course in the data engineer world there is some personal tech lover and if you want to try something start with some tech lover start with him and after that the spread all the information go in the department and after that people came to you and said okay I want to get this and it will come as soon as possible after that because if something is cool and works properly a lot of people want to join and start to use them like I said as quick as possible.
• 28:42
  SPEAKER_02: That's true.
• 28:43
  SPEAKER_02: I think like truly I think like the maybe one of the key things that you said is communication.
• 28:49
  SPEAKER_02: I think like communication is to your point right like communication is key especially with the um like the stakeholders you need to make sure that you align on the same objectives between data engineers and the people that are waiting for your data pipelines to be ready and get their the things that they are that they want.
• 29:07
  SPEAKER_02: So it's very important to communicate on what is possible and what is not possible.
• 29:11
  SPEAKER_02: So I love your advices thank you so much Alberto for that I guess one last question unfortunately because we are already at the end of the episode what is the best way for uh folks to get in touch with you okay I'm on LinkedIn I'm on LinkedIn Alberto Crespi data in lasminu and it's the best way to chat with me here you go here we go thank you so much Alberto Mark it was a wonderful yeah thank you and I'm sure that the listeners are are enjoying that episode thank you so much see you for the next one bye bye bye bye thanks for joining us for the latest episode of the Data Flowcast mastering airflow for data engineering and AI.
• 29:57
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 30:01
  SPEAKER_00: If you're enjoying our podcast please subscribe so you never miss an episode and leave us a five star review