Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating data across 30 companies at itti

Transcript

Orchestrating data across 30 companies at itti · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: You are listening to the data flowcast and the world of data and AI around.
• 0:07
  SPEAKER_01: We are excited to announce the launch of the annual airflow survey.
• 0:11
  SPEAKER_01: The largest data entering survey ever.
• 0:14
  SPEAKER_01: With almost 6,000 respondents this year, we are offering a survey respondents.
• 0:20
  SPEAKER_01: A free code to take the new Astronomer Certified AI Orchestration Fundamentals.
• 0:26
  SPEAKER_01: $150 value.
• 0:28
  SPEAKER_01: Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
• 0:35
  SPEAKER_01: Now let's get into the show.
• 0:38
  SPEAKER_00: All right, hi everyone.
• 0:39
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:41
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Lucas Trubiano, who is a data engineer at ITTI, which is the technology company within Groupovasquez.
• 0:51
  SPEAKER_00: Lucas, how are you?
• 0:53
  SPEAKER_02: I'm doing great.
• 0:54
  SPEAKER_02: Thank you, Kenton.
• 0:57
  SPEAKER_02: Thanks for the space.
• 0:58
  SPEAKER_00: Yeah, of course.
• 0:59
  SPEAKER_00: It's great to have you on.
• 1:00
  SPEAKER_00: Well, we'll go ahead and dive in.
• 1:02
  SPEAKER_00: So for today, we're going to talk about how ITTI uses Airflow to orchestrate data across more than 30 companies.
• 1:10
  SPEAKER_00: And we'll dig into the custom YAML framework that you've built to make Airflow more accessible to non-technical teams.
• 1:17
  SPEAKER_00: Before we get into that, let's start with some background.
• 1:19
  SPEAKER_00: So tell me about what ITTI and more broadly what Groupovasquez does.
• 1:25
  SPEAKER_02: Perfect.
• 1:26
  SPEAKER_02: Well, I will start with Group of Vázquez, that it's the holding of, as you said, more than 30 companies in Paraguay, Latin American country.
• 1:38
  SPEAKER_02: And it's a really big holding, but split in two um very specific areas: financial and digital ecosystem, with services through payments, credits, and other uh complementary like uh e-commerce, uh food delivery, and mobility also.
• 2:00
  SPEAKER_00: Okay, perfect.
• 2:02
  SPEAKER_02: Um two of the main companies of this group are Weno Bank, that is the the leading bank in the country with uh more than 60% of active population.
• 2:15
  SPEAKER_02: So it's a lot of customers uh for us.
• 2:18
  SPEAKER_02: And the other is the the tech consultancy company that uh brings services to all the other companies, so it's uh a very big ecosystem.
• 2:29
  SPEAKER_00: Yeah, that sounds huge.
• 2:30
  SPEAKER_00: And what does your team do within that?
• 2:33
  SPEAKER_02: Well, uh right now I'm leading the data engineering center of excellence, and our main goal uh is to build a 360-degree uh view of our customers.
• 2:44
  SPEAKER_02: Uh so in order of that, uh we explore some new and challenging uh case business cases, like for example, streaming analytics or unstructured data processing, but not also we we create or explore some business uh solutions.
• 3:02
  SPEAKER_02: Also, we standardize and scale then uh those solutions to other teams across the organization.
• 3:10
  SPEAKER_00: Okay, very cool.
• 3:12
  SPEAKER_00: And where does Airflow fit into all of that?
• 3:16
  SPEAKER_02: Well, uh we use uh Airflow mainly as a central task orchestrator.
• 3:21
  SPEAKER_02: We know do not use it for uh data processing.
• 3:24
  SPEAKER_02: For this, uh we have Spark or DBT, for example.
• 3:28
  SPEAKER_02: Uh so we always uh try and our best practice is to use Airflow only to define uh dependencies, uh scheduling, also monitoring uh our data pipelines and to handle uh all types of failures and allerging.
• 3:48
  SPEAKER_00: Okay, so very traditional orchestrator kind of offloading the data processing paradigm.
• 3:54
  SPEAKER_02: Yeah, um uh right now we we have uh almost uh 300 uh processes uh in Airflow.
• 4:03
  SPEAKER_02: So uh yeah, um most of them grow in the in the last year, so it's uh a high speed uh creating of those processes.
• 4:14
  SPEAKER_00: Yeah, that sounds like a lot.
• 4:16
  SPEAKER_00: And I think it leads me into my next question, which is you've talked about how broad the business is and how all of these different business units need orchestration.
• 4:27
  SPEAKER_00: I imagine you have a lot of different people trying to use Airflow or requesting that your team do something in Airflow for them.
• 4:35
  SPEAKER_00: How do you manage all of that?
• 4:37
  SPEAKER_00: Like, how do you ensure that all of the users have the access they need to orchestration using Airflow?
• 4:44
  SPEAKER_02: Uh well, we can think of it in two uh ways.
• 4:47
  SPEAKER_02: Um, first one is uh the that at their enterprise scale, uh we prefer to use an Airflow as a service.
• 4:58
  SPEAKER_02: With that, uh users don't worry uh of manage uh infrastructure.
• 5:02
  SPEAKER_02: And the other thing is to uh enable self-service to our users, to our non-technical users, and we made that um by YAML code, uh by custom YAML code.
• 5:15
  SPEAKER_02: So users only need to focus on what they want to create, and then in a CI CD pipeline, we handle how to create how to convert uh that configuration into a fully functional um DAC airflow.
• 5:32
  SPEAKER_00: Okay, interesting.
• 5:34
  SPEAKER_00: And you mentioned at the top, I kind of buried the lead a little bit, that one of the ways that you've helped to manage this is developing a custom YAML framework.
• 5:44
  SPEAKER_00: So tell me a little bit more about that and what problem you were trying to solve.
• 5:49
  SPEAKER_02: Perfect.
• 5:50
  SPEAKER_02: Well well, we have uh two YAML frameworks right now implemented.
• 5:54
  SPEAKER_02: Uh the first one was uh more customizable uh based on uh the typical DAC factory uh framework.
• 6:01
  SPEAKER_02: This project we call Bowie, but uh the real problem was was the bottleneck uh in code review of those pipelines.
• 6:11
  SPEAKER_02: So because users uh not all the time use the correct operator or change some parameters uh in a wrong way, so we decided to uh create another framework where we uh abstract uh more the the complexity and only um uh live to the leave to user to configure what we what we think it's configurable and validate uh all other things to um to be as we want.
• 6:48
  SPEAKER_00: Yeah, that totally makes sense.
• 6:50
  SPEAKER_00: Makes it a lot easier for you to institute some guardrails and kind of manage everything with lots of different teams using it.
• 6:57
  SPEAKER_02: I would like to add uh another interesting insight that today more than 80% uh of our DACs are created with a new framework because it's simply um faster for the user to create those pipelines.
• 7:12
  SPEAKER_02: So it's a very big uh percentage of creation with that new framework.
• 7:17
  SPEAKER_00: Yeah, that's great.
• 7:18
  SPEAKER_00: Sounds like it's been very successful and you've had a really wide rollout.
• 7:23
  SPEAKER_00: Tell me a little bit more about what the framework looks like.
• 7:26
  SPEAKER_00: I know there are a lot of companies build sort of custom YAML frameworks, and then there are, of course, open source ones like DAG Factory.
• 7:33
  SPEAKER_00: I'm curious how yours works and what it looks like for somebody from your team to build a pipeline using it.
• 7:41
  SPEAKER_02: Uh well, behind a framework, we um we have the typical uh Python code.
• 7:47
  SPEAKER_02: So we as a center of excellence, we develop uh like the best uh DAG to solve some problem, and then we put it in a Ginja template.
• 7:58
  SPEAKER_02: Um so then a user only configure um what it's customizable in that pipeline through a JAML, and then we compile these parameters into the Ginja template and compile the Python code.
• 8:15
  SPEAKER_02: So it's uh very simple uh behind, uh, but it's uh also very powerful because uh behind we we wrote uh Python, but uh user don't know what's behind.
• 8:29
  SPEAKER_00: Yeah, so it sounds like very simple for the end user to interact with then.
• 8:35
  SPEAKER_02: Yeah, and uh one uh interesting thing is that um, for example, use I can uh give you an example um of a user trying to ingest, for example, a Google Sheet uh file into our lake.
• 8:52
  SPEAKER_02: Uh they for they, for example, uh only configure what uh what document uh by the URL and what sheet uh they want to convert into a table and the destination.
• 9:06
  SPEAKER_02: But behind uh we introduced a lot of um a lot of testing tasks uh and user don't know really uh if there is a pre-validation, then a download, and then a processing task, for example.
• 9:20
  SPEAKER_02: Uh but we introduced that uh because we think reliability uh is as much important uh as the speed.
• 9:29
  SPEAKER_02: Uh so uh we we ensured that in the in the framework.
• 9:36
  SPEAKER_00: Yeah, that makes sense.
• 9:37
  SPEAKER_00: I'd love to double-click a little bit on that reliability piece that you just mentioned, since it sounds like that's really important there in addition to the development speed that you're enabling with this framework.
• 9:50
  SPEAKER_00: So, how does that work from a data quality perspective?
• 9:53
  SPEAKER_00: How is that part of the framework that you've built?
• 9:57
  SPEAKER_02: Uh from the data quality perspective, uh it was very challenging for us.
• 10:02
  SPEAKER_02: And we uh develop our um self-made in-house uh tool because we want something uh faster that runs over a spark framework and that could test uh data per partition, not uh to fully scan the tables.
• 10:20
  SPEAKER_02: Um so we we decide to develop our own tool, and that tool is triggered by custom operators we have developed on Airflow.
• 10:31
  SPEAKER_00: Okay, interesting.
• 10:33
  SPEAKER_00: So I'd love to hear a little bit more about the custom operators part of the framework.
• 10:38
  SPEAKER_00: Can you give me some examples of what that looks like?
• 10:43
  SPEAKER_02: Yeah, we have almost uh three or four type kinds of operators.
• 10:48
  SPEAKER_02: Uh, most of us for um uh uh typical open source tools like DBT, for example, and another for uh our own data quality tool, and also some operators uh for Amazon Web Services.
• 11:03
  SPEAKER_00: Um yeah, that sounds like a great way of leveraging Airflow's flexibility.
• 11:10
  SPEAKER_02: Yeah, exactly.
• 11:11
  SPEAKER_02: Um also uh it's the uh the flexibility to uh not also to develop uh those uh custom operators, also to um explain or document uh these custom operators, maybe in some uh AI development framework.
• 11:32
  SPEAKER_02: Um because not only we we develop uh the operators, also our agents uh know how to use it, and it's simpler for the user to uh to explain what they want to create, and the our framework or development framework uh explain how to how to create it and how to use it.
• 11:53
  SPEAKER_00: Yeah, that's great.
• 11:54
  SPEAKER_00: Cool.
• 11:55
  SPEAKER_00: Well, thank you for going through that.
• 11:57
  SPEAKER_00: This is a really interesting example of a custom framework.
• 12:01
  SPEAKER_00: Sounds like it's really enabled your team to roll out airflow more broadly.
• 12:05
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 12:14
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 12:18
  SPEAKER_01: Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 12:24
  SPEAKER_01: Check it out at astronomer.io.
• 12:28
  SPEAKER_00: Okay, so I want to change topics a little bit and ask you about AI orchestration since this is a huge topic within the Airflow world right now.
• 12:39
  SPEAKER_00: I understand from speaking with you before that you are using a spec driven development approach so that product and business users can contribute to templates without needing to know Python.
• 12:49
  SPEAKER_00: So it sounds like kind of very similar in framework to the YAML framework that you've implemented just to allow more people to interact with your orchestration.
• 12:59
  SPEAKER_00: But let tell me a little bit more about how that works.
• 13:02
  SPEAKER_02: Uh yeah, we well, we created our own uh framework, but it's not new.
• 13:07
  SPEAKER_02: Uh, we based our framework on GitHub, uh SpecKit, uh already existing framework.
• 13:14
  SPEAKER_02: And as as GitHub uh standardized, we customize all the data uh lifecycle uh rules and create like new skills to help users to only explain what they want to create and understand what uh certified and available templates we have created.
• 13:34
  SPEAKER_02: And with that, in any AI tool, they can ask for creating or debugging maybe uh some error in a DAG, and all the tools uh with that skills will know how to solve it or how to at least uh guide the user to solve the problem.
• 13:52
  SPEAKER_00: Interesting.
• 13:53
  SPEAKER_00: That makes sense.
• 13:54
  SPEAKER_00: And tell me, or I guess give me an example of what that might look like in practice.
• 14:00
  SPEAKER_00: So if a business user has a bug fix or some sort of enhancement they need to implement, what happens then?
• 14:07
  SPEAKER_02: Perfect.
• 14:08
  SPEAKER_02: Well well, the the full um life cycle uh for us is when um when the process or a DAG fails, uh all DACs by the base template have a Slack alerting.
• 14:24
  SPEAKER_02: So uh every owner in Slack uh receives a notification with not only the um the logs of the error, uh also with uh some explanation uh if it's a common error.
• 14:38
  SPEAKER_02: So most of common issues are solved uh only by the user with the explanation in the in the Slack thread.
• 14:46
  SPEAKER_02: But uh if if there's uh a deeper uh issue, maybe in some operator, those tickets are scaled automatically to the central super team.
• 14:59
  SPEAKER_00: Yeah, that's really cool.
• 15:01
  SPEAKER_00: I love hearing how you've just made Airflow and all of your orchestration so much more accessible to all of these other teams through these frameworks.
• 15:10
  SPEAKER_00: That's great.
• 15:11
  SPEAKER_02: Yeah, um uh with that, um we have a lot of teams uh using the the tool.
• 15:17
  SPEAKER_02: So for for us, that users can uh can solve the the bugs uh by themselves, it's very important because uh yeah, because in the other hand, we we will uh be a bottleneck in the organization, so we don't want uh that to happens.
• 15:35
  SPEAKER_00: Yeah, I would imagine your team would spend a lot of time just fixing other people's bugs.
• 15:42
  SPEAKER_02: Yeah, exactly.
• 15:43
  SPEAKER_02: Yeah, yeah.
• 15:44
  SPEAKER_00: Yeah, that's great that you've been able to do that and kind of free up your team for more uh yeah, just more new work uh while also supporting so many business units.
• 15:56
  SPEAKER_00: That's really cool.
• 15:58
  SPEAKER_02: Yeah, we we think a lot of uh how we can continue um improving uh that that topic.
• 16:06
  SPEAKER_02: Um Airflow offers a lot of options.
• 16:09
  SPEAKER_02: Uh a few days ago we we were thinking about uh how to uh maybe um trigger some agent when we have um some alert or issue.
• 16:20
  SPEAKER_02: Uh also uh from the Airflow instance, uh maybe to create a code um yeah, to create a pull request with the um with the bug fix.
• 16:32
  SPEAKER_02: Uh so Airflow have have lots of uh features very interesting to explore and to implement.
• 16:39
  SPEAKER_00: Yeah, absolutely.
• 16:40
  SPEAKER_00: I think that kind of self-healing pipelines, as you're describing, is a very popular topic right now.
• 16:45
  SPEAKER_00: It's certainly one that astronomer is interested in as well.
• 16:49
  SPEAKER_00: I think speaking of kind of forward-looking airflow projects and features, that's a good lead-in to my standard final question for all my guests, which is what would you most like to see from the Airflow project in the near future?
• 17:03
  SPEAKER_02: Well, we are very anxious uh about moving to the new Airflow 3 version uh because in this new version, uh, well, you have solved a lot of uh things we we need to uh maybe create some legacy code that right now we are maintaining.
• 17:19
  SPEAKER_02: Uh some of these features are, for example, the backfilling features uh for people that need to reprocess uh every uh any table.
• 17:30
  SPEAKER_02: And another topic that it's uh growing a lot in in our organization, at least, uh it's even driven orchestration because we have a lot of um streaming pipelines right now, and I think it's uh this is a very interesting topic.
• 17:46
  SPEAKER_02: And another cool feature that we are uh expecting uh it's the human in the loop for launching agents uh in the in the same uh DAX processes.
• 17:58
  SPEAKER_02: Uh it's uh very useful for orchestrating uh generative AI projects.
• 18:05
  SPEAKER_00: Yeah, absolutely.
• 18:06
  SPEAKER_00: Human in the loop is a really cool feature.
• 18:08
  SPEAKER_00: There's a lot of AI enablement uh or AI features to enable AI orchestration in the Airflow kind of 3.1 to 3.3 where we're currently at.
• 18:21
  SPEAKER_00: So yeah, great to hear that you'll take advantage of some of those.
• 18:25
  SPEAKER_02: Yeah, and um not also to to leave a AI work alone, uh also to to introduce a human review in in that loop or in that process.
• 18:36
  SPEAKER_02: Uh yeah, it's a it's a very cool feature for for us.
• 18:40
  SPEAKER_00: Yeah, absolutely.
• 18:42
  SPEAKER_00: Cool.
• 18:42
  SPEAKER_00: Well, Lucas, thank you so much for going through all of that.
• 18:45
  SPEAKER_00: It's been really interesting to hear about how you're supporting so many different business units with Airflow.
• 18:51
  SPEAKER_00: Uh, just to close this out, what is the best way for folks to get in touch with you?
• 18:55
  SPEAKER_02: Well, uh by LinkedIn.
• 18:57
  SPEAKER_02: Uh the company you can search for the company or our personal LinkedIns uh are open.
• 19:04
  SPEAKER_02: It was a pleasure uh not only to talk but also to share how we solve some problems uh we have.
• 19:11
  SPEAKER_02: So thank you for the space.
• 19:13
  SPEAKER_00: Yeah, thank you again for joining.
• 19:16
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 19:19
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 19:22
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 19:27
  SPEAKER_01: And leave us a five-star review.