Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño

Transcript

Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: The importance of the culture being an ally of the technology and also the integrations.
• 0:06
  SPEAKER_02: And it's not only about fixing broken workflows, but again, how can you improve the reliability of your orchestrated pipelines so that you can have a more stable, scalable platform?
• 0:20
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:32
  SPEAKER_00: Okay, hi everyone.
• 0:33
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:35
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Carlos Porto Nino, who is a senior analytics engineer and data analyst at Addy.
• 0:43
  SPEAKER_00: Carlos, how are you?
• 0:45
  SPEAKER_02: I'm doing all right.
• 0:47
  SPEAKER_02: What about you?
• 0:48
  SPEAKER_00: Yeah, I'm good.
• 0:49
  SPEAKER_00: I'm good.
• 0:50
  SPEAKER_00: Uh this will not be true when this comes out, but we're just heading into the Thanksgiving holiday weekend here in the US at the time of recording.
• 0:57
  SPEAKER_00: So yeah, packing everything into a short week, which is always fun.
• 1:01
  SPEAKER_00: Cool.
• 1:02
  SPEAKER_00: Well, I know we've got a lot to get through today, so let's go ahead and dive in.
• 1:05
  SPEAKER_00: Uh for today's podcast for our listeners, we're going to discuss how ADI uses Airflow for batch orchestration and how they manage organizational complexity with their data platform.
• 1:17
  SPEAKER_00: Uh, I'm really excited to dig into this and maybe let's start with some background for folks who might not be familiar with ADI.
• 1:23
  SPEAKER_00: Uh, what does your company do?
• 1:25
  SPEAKER_02: So uh let me tell you and everyone who's listening.
• 1:28
  SPEAKER_02: So basically, ADI started operations around 2019.
• 1:33
  SPEAKER_02: Uh Adi is a Colombian company, and we are a FinTech, both a FinTech and a commerce platform.
• 1:40
  SPEAKER_02: We currently serve around 2.7 million customers, and also we are partnered with around or more than 30,000 merchants.
• 1:52
  SPEAKER_02: So there's a lot of people who rely on relies on us.
• 1:56
  SPEAKER_02: Yeah, so to tell you a little bit about what we do.
• 1:59
  SPEAKER_02: So we originally started as a buy-in-opulator solution.
• 2:02
  SPEAKER_02: Really, really short summary of what that means is basically providing short-time financing to our customers so that they can purchase, make their purchases, and pay them installments over a fixed period of time.
• 2:15
  SPEAKER_02: And then uh that's that's by now pay later.
• 2:18
  SPEAKER_02: After a couple of years, we decided to migrate and uh to increase the stakes, and we started to become a financial and e-commerce platform.
• 2:28
  SPEAKER_02: Uh, we're now a multi-product ecosystem.
• 2:31
  SPEAKER_02: We provide credit solutions for our customers, instant payments processing, working capital solutions for our allies slash merchants.
• 2:40
  SPEAKER_02: We have a fully fledged marketplace, and finally, we're going for uh or we're expecting to launch a Sabix Sabiex account really soon.
• 2:49
  SPEAKER_02: So uh yeah, that's that's what we are right now.
• 2:53
  SPEAKER_00: Very cool.
• 2:54
  SPEAKER_00: And I imagine, as with most fintech companies that are doing a lot of automation, as you described, and just working in a lot of different areas of finance, that data is really important to your business.
• 3:07
  SPEAKER_00: So maybe tell me about how the data team, or maybe it's multiple teams at ADI are structured and what sort of use cases those teams enable.
• 3:17
  SPEAKER_02: Of course.
• 3:18
  SPEAKER_02: So, as you may have guessed, a company that has been around the market for five, six years, uh, we have a lot of we have had a lot of changes, and not only in the way the strategy and the business is defined, but also in terms of how the data team is structured.
• 3:38
  SPEAKER_02: And this is like the latest iteration that corresponds to our appetite in terms of all the data that we are processing right now.
• 3:46
  SPEAKER_02: And as you mentioned, it's been crazy.
• 3:48
  SPEAKER_02: I mean, we've been changing a lot of paradigms within our company across the recent years due to the product and the ambition growing.
• 3:58
  SPEAKER_02: So definitely we had a lot of changes.
• 4:01
  SPEAKER_02: In summary, we have a data platform team, a centralized data platform team with around 20 people.
• 4:07
  SPEAKER_02: We're part of the engineering department, so we try and put in practice a lot of the good good the good put into practice a lot of the good practices of uh software development.
• 4:21
  SPEAKER_02: And basically we're composed of four teams, uh, one being the data engineers, another the data science team, the third one being the AI ML ops, and finally the one I'm part of, which is the analytics engineering engineering team.
• 4:38
  SPEAKER_02: Uh, I will tell you a little bit of each one of those so we can have like a good background of what each of those do.
• 4:46
  SPEAKER_02: So basically, the data engineering team manages infrastructure, they create ingestion, uh, batch and streaming jobs, API integrations, access control management, enforcing data governance.
• 4:58
  SPEAKER_02: The data science teams basically go about doing the descriptive, diagnostic, predictive analytics, machine learning, supervised and supervised models, prototyping, features, so a lot of stuff.
• 5:13
  SPEAKER_02: And the AIML ops team is basically in charge of deploying, monitoring models, the models that the data science team created, automating, controlling costs of those uh models and managing infrastructure.
• 5:27
  SPEAKER_02: And finally, um, my team, the analytics engineering team, what we do is create and maintain transformation pipelines.
• 5:34
  SPEAKER_02: We model the data, we perform or we do data warehousing, we're in charge of data observability, data quality.
• 5:42
  SPEAKER_02: And one key principle as well, or one key activity that we perform as well, is the stakeholder communication.
• 5:48
  SPEAKER_02: We not only talk with business analysts and managers, but also with the engineering backend team, which uh is in charge of actually managing the platform that provides a lot of the data that we consume.
• 6:03
  SPEAKER_02: So we have a really close relationship with them.
• 6:06
  SPEAKER_02: Um, in addition to that, I can tell you that we use Databricks as our unified managed data platform.
• 6:13
  SPEAKER_02: We follow the principle of uh the having a lake house and have the both the benefits of a data lake and a warehouse.
• 6:21
  SPEAKER_02: We have the medallion architecture in place, we do dimensional modeling.
• 6:27
  SPEAKER_02: And yeah, across the years we've been going through batch batch processing, both batch processing and streaming, a lot of data uh and due to the current in to the current architecture of our platform.
• 6:41
  SPEAKER_02: And um, in recent times, I want to highlight that we have a lot of integrations with external services, which provides several needs in terms of batch processing.
• 6:50
  SPEAKER_02: And definitely it's gaining a lot of significance uh integrating with all those.
• 6:56
  SPEAKER_02: And definitely Airflow has been one of the ways to do so in recent times.
• 7:02
  SPEAKER_00: Yeah, that's great.
• 7:03
  SPEAKER_00: Well, thank you for going through so much detail in terms of how you're set up, both from the team management side and from the tech stack side.
• 7:12
  SPEAKER_00: Uh it's really cool to hear that you have so much, such a mature setup.
• 7:15
  SPEAKER_00: We don't always hear that from companies of your size.
• 7:18
  SPEAKER_00: So that's really nice to hear about that investment.
• 7:22
  SPEAKER_00: You mentioned at the very end there a little bit about Airflow and batch processing.
• 7:26
  SPEAKER_00: Maybe you can tell me a little bit more about how Airflow fits into this and what sort of what it enables for your data teams.
• 7:33
  SPEAKER_02: Of course.
• 7:34
  SPEAKER_02: So I want to provide a little bit of context on how we evolved into the actually started using uh Airflow and how does it currently fit in our tech stack?
• 7:45
  SPEAKER_02: So the story goes about a few years ago.
• 7:49
  SPEAKER_02: We had a quite in main a quite basic platform.
• 7:53
  SPEAKER_02: We didn't have such a strong strategy as we have today.
• 7:56
  SPEAKER_02: So at some point of time, we definitely needed to invest and to mature our data stack.
• 8:03
  SPEAKER_02: We processed, we performed the batch processing via cron jobs in in an uh in a Postgres database, a replica per especially for analytics use cases.
• 8:16
  SPEAKER_02: And definitely that needed to change for us to scale, to serve our customers, to serve our increasing analytical needs.
• 8:23
  SPEAKER_02: And we found the solution, at least the first step of the solution, and the journey started with platforms like Databricks, frameworks like DBT, and tools like Airflow.
• 8:34
  SPEAKER_02: So all of those played an important role in this process.
• 8:38
  SPEAKER_02: And I want to say that they have enabled us to scale, to orchestrate, and to manage an increasing amount of data, uh, the cognitive load that derived from that and the complexity that it's always present in such an industry.
• 8:55
  SPEAKER_02: So uh I would say that our airflow use case goes uh or is best described as a single deployment, single development team.
• 9:03
  SPEAKER_02: We haven't reached that point of time in which we need to definitely scale it even more.
• 9:08
  SPEAKER_02: But yeah, so in terms of airflow, we started with uh MWAA.
• 9:14
  SPEAKER_02: It was we we use it a quite legacy airflow version.
• 9:18
  SPEAKER_02: A lot of people, or most of us, didn't know a bit about orchestrating via Airflow.
• 9:23
  SPEAKER_02: So that represented a lot of challenges.
• 9:26
  SPEAKER_02: We had a lot of hiccups.
• 9:28
  SPEAKER_02: Uh, but yeah, at the same time, it's what it was quite interesting to learn how to manage it properly, and especially uh something like the MWAA was quite difficult to tune.
• 9:40
  SPEAKER_02: So we decided, hey, why don't we go for a hosted solution that enables us to scale better and to monitor our uh instance, and yeah, basically that performs better and ends up as a better solution for us.
• 9:57
  SPEAKER_02: So we decided to go with astronomer with a hosted instance, and it's been quite a journey, a lot of learnings along the way.
• 10:04
  SPEAKER_02: I mean, I am not the expert, but definitely I have learned a lot as the versions were moving forward.
• 10:12
  SPEAKER_02: We realized that a lot of up the capabilities were being created, so we started to think how we can iterate what is currently in place so we can leverage on the good practices, so we can leverage on the new features that Airflow has been created across the years with the with the different versions.
• 10:31
  SPEAKER_02: We're currently running Airflow 2.10, and I'm while wide aware that we need to definitely at some point of time to think to migrate to the 3.1 and again leverage out of or leverage on the a lot of the features that have been going out, and uh definitely it could be great for our batch processes in um that we are working with in in recent months.
• 10:56
  SPEAKER_00: Yeah, for sure.
• 10:56
  SPEAKER_00: That's a great overview.
• 10:58
  SPEAKER_00: And now that you're on Astro, one of the things, one of our goals at Astronomer with our hosted service is that it allows people to spend more time with their pipelines and taking advantage of new features as opposed to just figuring out how to run airflow, especially when you're starting with it at a relatively smaller scale, but are in your case very likely going to need to scale up over time to support all of these different business units that are relying on it.
• 11:22
  SPEAKER_00: Has that been it sounds like that's been your experience that you're now able to look much more towards the new features and upgrading more promptly now that you're on this managed service?
• 11:32
  SPEAKER_02: Yeah, definitely, definitely.
• 11:34
  SPEAKER_02: The the challenge is when and how do you allocate enough time and resources to do so?
• 11:39
  SPEAKER_02: I mean, I know it's been like quite a lot of effort into making the process easier.
• 11:46
  SPEAKER_02: But yeah, the thing is that when you are part of a critical part of the business providing data, they want you to focus on that and not necessarily always have the time to allocate to these important but uh not that urgent task.
• 12:02
  SPEAKER_02: So yeah.
• 12:05
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:14
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:26
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:34
  SPEAKER_00: Yeah, definitely making time for upgrading in particular can always be a challenge, but it's usually worth it in the end once you've done it.
• 12:42
  SPEAKER_00: Speaking of that and new airflow features, and you've also mentioned a couple times that you're dealing with a very high degree of complexity when it comes to managing the data for your company, which I think is pretty typical in FinTech.
• 12:54
  SPEAKER_00: How do you think that Airflow helps you manage this complexity more broadly?
• 12:59
  SPEAKER_02: Yes, uh Airflow is one of the key components of the solution.
• 13:03
  SPEAKER_02: So to give you a little and to give the audience a little bit of context on numbers on the complexity itself.
• 13:10
  SPEAKER_02: So across years, we processed several hundred different messages or events with really complex payloads.
• 13:18
  SPEAKER_02: I mean, those are derived of our again from out of our architecture.
• 13:23
  SPEAKER_02: And this means that we can range from having several hundred thousand instances to even in a few instances, a couple of billion instances of publication of those events.
• 13:35
  SPEAKER_02: So it darts a lot of data, a lot of stuff that needs to be considered.
• 13:40
  SPEAKER_02: So uh, and that's not the end of it.
• 13:43
  SPEAKER_02: So we would you you would have also to consider that we could have payloads that go from less than a thousand primitive attributes to even in some few cases several hundred, or in in really few cases, more than a thousand primitive attributes.
• 13:57
  SPEAKER_02: So you can make or can can make an idea of how complex it could be to manage all of that, even with having those primitive attributes nested to four or even five levels.
• 14:07
  SPEAKER_02: So it's it's really, really complex.
• 14:10
  SPEAKER_02: So that requires concrete data contracts.
• 14:13
  SPEAKER_02: Those data contracts are constantly evolving, and uh and yeah, uh I would say that constant change means that we need to work and we need to deploy new stuff, but not only that, we also need to maintain, to evolve, to iterate existing data products.
• 14:32
  SPEAKER_02: And one of the ways that we are doing that is using custom solutions in Airflow uh and also using DVT.
• 14:41
  SPEAKER_02: So I will tell you a little bit about it.
• 14:43
  SPEAKER_02: So for anyone who's interested, we created a custom DAC factory solution.
• 14:47
  SPEAKER_02: I'm aware that Airflow uh has now a V1 for the DAC factory, but we created this like two or three years ago with help of a consulting firm, consulting tech firm.
• 15:00
  SPEAKER_02: And the idea is the same.
• 15:02
  SPEAKER_02: So you create these custom configuration files that are in YAML, and then you translate them into DACs with that that are easily orchestrated and created, and you don't necessarily need to know Python.
• 15:16
  SPEAKER_02: So yeah, that has helped us quite a lot, and especially the new joiners to deploy new DACs, to deploy new workflows and to orchestrate without having to know all the details and having to understand every single parameter that when you are setting up a Python uh DAC within Airflow.
• 15:35
  SPEAKER_02: So uh I would say that that mix of DAC factory solution in Airflow plus DBT has enabled us or has brought us with modularity, incrementality, and a great visibility so we can monitor and check assumptions really quickly.
• 15:53
  SPEAKER_02: I think that's one of the key components of this.
• 15:56
  SPEAKER_02: And this I'm pretty sure that we wouldn't, if we didn't have this in place, it would be quite difficult to assess did it run, when did it run, what's the length of the failed run, how many retries a single run had.
• 16:13
  SPEAKER_02: So these are concepts that are built in Airflow itself and that have, I would say that the way Airflow presents itself has helped us like rethink the way we orchestrate.
• 16:24
  SPEAKER_02: So those are the kind of things that I have learned across the years while using Airflow.
• 16:30
  SPEAKER_02: And and yeah, it's been great, especially the fact that you can group tasks to match the responsibility or slash the dependencies that you want to be created.
• 16:40
  SPEAKER_02: So you can really create something and orchestrate a set of tasks that match a certain criteria that the business needs, or that you that UNEDA is asking for to ensure consistency, reliability.
• 16:56
  SPEAKER_02: But yeah, it's that's uh a good way to to depict it.
• 17:00
  SPEAKER_00: Yeah, that's great.
• 17:01
  SPEAKER_00: That's a very clever use of that abstraction for DAG authoring with DAG Factory to manage what sounds like very highly unstructured data.
• 17:10
  SPEAKER_00: So really cool to hear about that.
• 17:12
  SPEAKER_00: I also want to dig into the monitoring aspect a little more because you mentioned a few times throughout there that keeping track of when things run and are they having to retry or things like that is really important.
• 17:27
  SPEAKER_00: And I think it's often pretty critical for you know data platform teams like your own that are managing a high degree of complexity and also serving really critical business needs.
• 17:37
  SPEAKER_00: So, how do you monitor your most critical data products, both from a tools perspective and from a process perspective?
• 17:45
  SPEAKER_02: I would say we have we have like a mix of culture, technology, and integrations of those technology systems slash tools.
• 17:58
  SPEAKER_02: So in terms of airflow, I would say one of the critical components is the on-failure callbacks.
• 18:05
  SPEAKER_02: In there, we have a custom integration with our platform, one of our alerting and incident managing platforms, which is Grafana.
• 18:14
  SPEAKER_02: And in on top of that, we have that integration to work with our communication platform, in this case a Slack.
• 18:21
  SPEAKER_02: So we are promptly alerted, and we can create the communication artifacts that enable us to tackle the problem.
• 18:31
  SPEAKER_02: So definitely across the years, this integration has enabled us to be alerted of minor and major problems with our data products.
• 18:40
  SPEAKER_02: Uh, I would say we have improved our communication, how effective it is, as everything is connected.
• 18:46
  SPEAKER_02: We don't need to jump from one platform to another to learn what which is the status of certain alert.
• 18:52
  SPEAKER_02: And I also mentioned the culture part of it, because here at Adi, we have like uh, I would say a mix of policies and group practices that we follow that basically state that whenever you are having an incident, either minor or major or really important, critical, you go through four stages.
• 19:16
  SPEAKER_02: And in those four stages, two of them are related to airflow itself.
• 19:21
  SPEAKER_02: So the first one is declaring the incident, the problem, the alert.
• 19:25
  SPEAKER_02: So you first have to identify it, then you have to respond to it, start to understand what's going on.
• 19:31
  SPEAKER_02: And I would I want to say that personally the logs from within Airflow are extremely useful.
• 19:37
  SPEAKER_02: I mean, I've always had this concept of logs at something that you would have to go into a really shady console somewhere in the internet, so you have you can have those logs, but definitely with Airflow, it's quite easy to have them.
• 19:51
  SPEAKER_02: And the third step would be the recovery.
• 19:53
  SPEAKER_02: How do you stop the bleeding for that data product?
• 19:56
  SPEAKER_02: How do you stop it failing?
• 19:58
  SPEAKER_02: And finally, the and one of the most important steps, it's learning.
• 20:02
  SPEAKER_02: So, how you can prevent in the future that alerts from being fired up, how you can be more stable, have a more stable product, to have a more stable uh orchestration platform, and so on.
• 20:18
  SPEAKER_02: So uh I just want to make quick, two quick like comments on those.
• 20:22
  SPEAKER_02: On the first one, again, airflow callbacks are critical triggers to understand where when something is failing.
• 20:29
  SPEAKER_02: That's like the declare first part of the of the workflow.
• 20:33
  SPEAKER_02: And on the learn on the end of the workflow, the fourth step, basically, uh we all we are all constantly learning how we can optimize the different parameters of our DACs, of our tasks.
• 20:46
  SPEAKER_02: Maybe we can we need to increase the frequency, maybe we need to increase or decrease the the timeouts.
• 20:54
  SPEAKER_02: All of those seem like simple, really easy to follow steps, but sometimes we forget that those could have like a really high gimpack in terms of reliability, in terms of monitoring capabilities, and so on.
• 21:09
  SPEAKER_02: So, yeah, I would say the importance of following that set of steps here at Adi, the importance of the culture being an ally of the technology and also the integrations.
• 21:22
  SPEAKER_02: And it's not only maybe a message for the audience, it's it's not only about fixing broken workflows, but again, how you can set new alerts, how can you improve the reliability of your uh orchestrated pipelines so that you can have a more stable, scalable platform.
• 21:39
  SPEAKER_00: That's great.
• 21:40
  SPEAKER_00: This is such a huge topic, monitoring and observability.
• 21:43
  SPEAKER_00: And I think we could talk for many more hours about that.
• 21:46
  SPEAKER_00: I do just want to call out that I absolutely love that you uh touch so heavily on the culture aspect of it because I think with this in particular and so many things, that's maybe even the more important part than the specific tech stack.
• 21:59
  SPEAKER_00: You do have to have good tools, but you also have to use them in a way that makes sense for your organization.
• 22:04
  SPEAKER_00: So much of that comes down to the culture.
• 22:07
  SPEAKER_00: Like I said, I would love to keep talking about this for a long time, but in the interest of time, I always like to end with a standard final question, and that is what would you most like to see from the Airflow project in the near future?
• 22:19
  SPEAKER_02: Yeah, I would say being having a technical background so strong in terms of BI and analytics, I would have the I would love to have the capability to access the data that Airflows produces in a maybe more scalable way.
• 22:33
  SPEAKER_02: I know that you can go and do some shady stuff and try to distract some data, but definitely something that have could enable me to connect with the other tech stack that I have to the other analytical capabilities.
• 22:46
  SPEAKER_02: So I can monitor more productively, I can leverage on that data to make some predictions in terms of the usage of the platform itself.
• 22:56
  SPEAKER_02: I would say that's something that maybe it's a little bit difficult for some of us when we don't have a lot of that deep infrastructure and DevOps knowledge that do you sometimes require to optimize your instance.
• 23:07
  SPEAKER_02: Um, but yeah, anything that helps in that regard in terms of monitoring our platform and debugging it, it's all it's amazing.
• 23:15
  SPEAKER_02: It would be amazing.
• 23:16
  SPEAKER_02: And a final little detail that I would love is definitely knowing who triggered who to perform a manual trigger on a DAC.
• 23:26
  SPEAKER_02: Currently, we don't know.
• 23:28
  SPEAKER_02: I'm not sure if that's in the plans of the development team, but definitely that could be useful so you can align better with other people who is solving something in terms of a DAC failing and so on.
• 23:39
  SPEAKER_02: So yeah, that could help.
• 23:41
  SPEAKER_00: Yeah, totally makes sense.
• 23:42
  SPEAKER_00: I think those are both great requests.
• 23:45
  SPEAKER_00: Well, Carlos, this has been really insightful.
• 23:47
  SPEAKER_00: I really appreciate you taking the time to come on just to close us out.
• 23:51
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 23:53
  SPEAKER_02: Uh I would say definitely you can reach out to our head of data, our CTO, or myself.
• 24:01
  SPEAKER_02: So the names are basically our head of data is Mauricio Hakome.
• 24:06
  SPEAKER_02: Our CTO is Carlos Toro, and uh who's speaking, Carlos Puerto Niño.
• 24:13
  SPEAKER_02: So feel free to reach out to us.
• 24:15
  SPEAKER_02: And yeah, thank you so much, Kentin, for this phase.
• 24:18
  SPEAKER_02: It was quite interesting to share our experience with Airflow.
• 24:22
  SPEAKER_00: Yeah, well, we really appreciate it.
• 24:24
  SPEAKER_00: Thanks again for coming on.
• 24:25
  SPEAKER_00: This has been great.
• 24:26
  SPEAKER_02: Awesome.
• 24:30
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:36
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:40
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five-star review.