Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye
Transcript
- 0:00SPEAKER_02: We tried to identify exposures on time, right?
- 0:02SPEAKER_02: Cases of failed transactions or downtimes in all the transfer services and payment gateways.
- 0:08SPEAKER_02: And as soon as we pick up these issues, we are able to alert the necessary individuals as well as the central bank and so many other things.
- 0:15SPEAKER_02: And Airflow helps because there are a lot of dynamic settings we have to run a lot of pipelines in parallel and also to experiment as well internally.
- 0:24SPEAKER_02: We've had times that have built Airflow DAGs just to analyze the order of people coming into meetings.
- 0:45SPEAKER_01: Hey Airflow community, exciting news.
- 0:48SPEAKER_01: This fall we're launching Beyond Analytics, a virtual data ops conference happening September 16th from 12 to 3 p.m.
- 0:56SPEAKER_01: Eastern.
- 0:56SPEAKER_01: You'll hear from data leaders on how modern data ops and orchestration are powering AI, machine learning, and production grade data products.
- 1:03SPEAKER_01: And here's the best part.
- 1:04SPEAKER_01: You can join Mark Lumberti for a fast track Airflow 3 certification prep session and get a discount code for a free certification, a$150 value.
- 1:14SPEAKER_01: The conference is virtual, it's free, and it's made just for you.
- 1:18SPEAKER_01: You can register now at astronomer.io slash beyond slash data flowcast.
- 1:26SPEAKER_01: Hi everyone.
- 1:27SPEAKER_01: Welcome to the Data Flowcast.
- 1:29SPEAKER_01: I'm your host, Kenton Dannis, and today I'm joined by Edio Lou Adaboy, who is a data engineer at MoneyPoint Group.
- 1:37SPEAKER_01: Lou, how are you doing?
- 1:38SPEAKER_02: I'm doing good.
- 1:40SPEAKER_02: Thank you.
- 1:40SPEAKER_02: How about you?
- 1:41SPEAKER_01: Great.
- 1:42SPEAKER_01: Yeah, yeah.
- 1:42SPEAKER_01: Doing well.
- 1:43SPEAKER_01: Enjoying the summer here.
- 1:44SPEAKER_01: For our listeners, today we are going to discuss how Money Point Group uses Airflow for payments and exposure monitoring.
- 1:52SPEAKER_01: So we'll go ahead and get right into it.
- 1:54SPEAKER_01: But let's start with some background for folks who might not be familiar.
- 1:57SPEAKER_01: Can you tell us what Money Point Group does as a company?
- 2:00SPEAKER_02: So Money Point is a financial services company.
- 2:04SPEAKER_02: We're the largest agency bank in Nigeria, right?
- 2:07SPEAKER_02: We drive a lot of financial inclusion and financial services across the country, largely through our POS point of sale terminals.
- 2:29SPEAKER_02: And beyond our services, agency banking service, we also cater straight to businesses, right?
- 2:35SPEAKER_02: So we have a B2B where we serve with virtual accounts and a lot of payment infrastructure and a payment gateway as well.
- 2:43SPEAKER_02: So yeah, in a nutshell, that's Money Point.
- 2:45SPEAKER_01: Okay, great.
- 2:46SPEAKER_01: And what does your team do?
- 2:48SPEAKER_02: Okay, so the data engineering team at Money Point is more or less the management information system for the company.
- 2:55SPEAKER_02: We cater to all the different departments in the company, such as offline sales, to marketing, to operations and finance and auditing, and also the legal team as well, because here regulations are high, especially in the financial space.
- 3:09SPEAKER_02: So compliance always has to be top priority as well.
- 3:13SPEAKER_02: So we feed all these different arms of the business through analytics, through dashboards, through deep analysis as well, looking into core parts of the business and looking at the trends and also helping our customers as well.
- 3:26SPEAKER_02: Always trying to tailor the best services for them as well as they grow as well.
- 3:31SPEAKER_02: Also, as they grow in businesses, as their businesses scale.
- 3:34SPEAKER_02: Also, the data also scales across the country and also into other places.
- 3:40SPEAKER_02: We've already have a footprint in the UK with our Great Britain office, and we hope to expand to more countries soon.
- 3:48SPEAKER_02: So the data team really is a powerhouse of information for all these arms of the businesses and also the new products that are still underway.
- 3:56SPEAKER_01: Yeah, amazing.
- 3:57SPEAKER_01: So you're managing both all of your internal data that powers pretty much every part of your business, as well as your products for a very large now international financial services company.
- 4:08SPEAKER_01: That's a big, that's a big umbrella that you're managing.
- 4:12SPEAKER_01: And so tell me how Airflow helps out with that.
- 4:16SPEAKER_02: Oh, yes.
- 4:17SPEAKER_02: Airflow helps us to monitor and orchestrate a lot of things, right?
- 4:21SPEAKER_02: Due to the different sides of the business.
- 4:23SPEAKER_02: We have different sub-teams, also in the data team, responsible for different arms.
- 4:29SPEAKER_02: I, in particular, I work with the payments team.
- 4:32SPEAKER_02: And our job with Airflow is to get all of our different major accounts for all our partner banks and institutions.
- 4:40SPEAKER_02: We need to get clarity on these transactions coming from these partner banks and going to them as well on a daily basis and also aggregating because we've been in the business for a while.
- 4:50SPEAKER_02: So we have almost four or five years worth of data that we try to get as much clarity on.
- 4:55SPEAKER_02: And also on a daily basis moving forward, we try to identify exposures on time, right?
- 5:01SPEAKER_02: Cases of failed transactions or downtimes in all the transfer services and payment gateways.
- 5:08SPEAKER_02: And as soon as we pick up these issues, we are able to alert the necessary individuals, as well as the central bank and so many other things.
- 5:17SPEAKER_02: And Airflow helps because there are a lot of dynamic settings we have to run a lot of pipelines in parallel and also to experiment as well internally.
- 5:26SPEAKER_02: We've had times that have built Airflow DAGs just to analyze the order of people coming into meetings.
- 5:32SPEAKER_02: So instead of just speaking on a first-come, first serve basis, I would randomize the names of all the attendees of the meeting and whoever gets first or whoever's in order speaks.
- 5:42SPEAKER_02: So yeah, that's how much of a tool Airflow has been for us.
- 5:46SPEAKER_02: So aside the serious stuff, we also use it for fun activities as well.
- 5:51SPEAKER_01: That's amazing.
- 5:52SPEAKER_01: I love that.
- 5:52SPEAKER_01: Did it work?
- 5:53SPEAKER_01: Did it help your meetings be more productive?
- 5:55SPEAKER_02: Yeah, it helped us keep the time.
- 5:56SPEAKER_02: Okay.
- 5:58SPEAKER_02: You have about 30 seconds to speak, right?
- 6:00SPEAKER_02: During stand-up, give your updates in 30 seconds, you get like a buzzer, and the next person has to take over.
- 6:07SPEAKER_02: So that was fun.
- 6:08SPEAKER_01: Yeah.
- 6:09SPEAKER_01: Okay.
- 6:09SPEAKER_01: I'm gonna tell my team about this.
- 6:11SPEAKER_01: They're gonna love this idea of using Airflow to gamify like actual work tasks.
- 6:16SPEAKER_01: That's a new one.
- 6:17SPEAKER_01: I haven't heard that before.
- 6:18SPEAKER_01: Yeah, that's really cool.
- 6:20SPEAKER_01: So I uh zooming back out, Airflow it sounds like is your primary orchestrator, you're using it for all these pipelines that you're running for all these different teams and for your product.
- 6:29SPEAKER_01: One of the use cases that I would love to dig into a little bit deeper is this exposure monitoring that you mentioned very briefly.
- 6:37SPEAKER_01: So can you tell me a little bit more about what that use case is at a high level?
- 6:43SPEAKER_02: So for us, what the I think for financial institutions as a whole, right, it's always a dicey game of getting your books in order because if you wait too long, by the end of a quarter or by the end of the year, you find out that you're in net negative positions or you have receivables or payables that are mounting up to sometimes millions, right?
- 7:05SPEAKER_02: Which could re put the company in a bad fashion.
- 7:08SPEAKER_02: So what the exposure monetary does is that it works on top of the existing pipelines that brings in data every day and it spots for unusual or risky transactions where we let's say a transaction gets reversed, but nobody sees it on a daily basis, but the system picks it up and traces if the provider ever gave us some feedback.
- 7:32SPEAKER_02: And if there's no feedback, then we know that there's likely a problem somewhere.
- 7:36SPEAKER_02: And if we don't act on time, then that's going to be a loss on our end that we might figure out months later.
- 7:43SPEAKER_02: So what the exposure system really does is it goes through our general ledger and also goes through what the other banks give us and it spots, it searches for those anomalies, right?
- 7:54SPEAKER_02: And once it finds that, it flags that and we're able to send that to the product team, they verify, sometimes they call, and once they get some feedback from the provider, we know what next to do.
- 8:05SPEAKER_02: So, like I said, we're talking about almost hundreds of accounts.
- 8:08SPEAKER_02: So a lot of them are easy to slip through the cracks, which is where Airflow really comes in.
- 8:13SPEAKER_02: So each account has its own dedicated Airflow pipeline, and then we have an aggregation pipeline as well that goes through all these accounts on a daily basis, actually, multiple times in a day, to uh categorize the data, to clear out the ones that have reconciled, and all the unusual or flagged transactions get sent into a separate table.
- 8:36SPEAKER_02: And from there, what we do is now we feed that to all our stakeholders so they get to visualize it, and then they can see what the problems are on time.
- 8:45SPEAKER_02: And we try to simplify it into layman's English because non-technical people don't need all the specific details, they don't need to learn about idempotency or all of that.
- 8:56SPEAKER_02: All they need to know is you know what's right, what's wrong, and what do we need to do.
- 9:01SPEAKER_02: And that's where we're able to connect from our data warehouse to their platforms.
- 9:06SPEAKER_02: And yeah, we give them the simple details they need to get on top of the job.
- 9:11SPEAKER_02: Yeah.
- 9:11SPEAKER_02: Airflow really helps a lot with the fundamental infrastructure of orchestrating all that information out, and then we just plug and play into other endpoints for them to work with.
- 9:21SPEAKER_01: Okay, interesting.
- 9:22SPEAKER_01: And so you're doing scale like proactive anomaly detection basically and orchestrating that with Airflow.
- 9:28SPEAKER_01: Obviously, there's a lot of technical detail that goes into the implementation of something like that, but you're actually managing this for a lot of very non-technical stakeholders who just need the end result.
- 9:41SPEAKER_01: How does Airflow play a role in that piece of it?
- 9:44SPEAKER_01: And are you creating dashboards for them to use?
- 9:47SPEAKER_01: Or you mentioned it's plugging into some tools that they might already use.
- 9:50SPEAKER_01: Is that something that you're just taking the anomaly detection and using Airflow to plug it into an API somewhere else?
- 9:57SPEAKER_01: Or how does that work?
- 9:58SPEAKER_02: Yes, we have some internal tools we use, imbued tools actually, that helps with internal settlement and reconciliation, but it feeds off the data that we process through Airflow.
- 10:10SPEAKER_02: So our data warehouse is on Click House, right?
- 10:14SPEAKER_02: It houses almost all of our data.
- 10:16SPEAKER_02: And what we use Airflow to do is it goes through the data we've stored on ClickHouse or on production, right?
- 10:22SPEAKER_02: We handle all the analysis there using SQL and DBT, and then from there, through Click House, we connect the data in Click House to those third-party tools or to our internal tools for the users to see.
- 10:37SPEAKER_02: Some tools like Grafana that visualizes the data ready-made.
- 10:42SPEAKER_02: And we've also explored tools like Luca on the GC P, just for end users to see.
- 10:48SPEAKER_02: So, yeah, so Airflow largely just helps us with that orchestration for the transformation itself.
- 10:53SPEAKER_02: And since the data is already in Click House, we just connect that to all the visualization tools for stakeholders here.
- 11:01SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
- 11:10SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
- 11:22SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
- 11:31SPEAKER_01: Did I catch right that you mentioned some of the tools you're sending data to are like custom internal things that you've built?
- 11:37SPEAKER_01: Isn't that right?
- 11:38SPEAKER_01: Was it difficult to plug airflow into those?
- 11:41SPEAKER_01: Or I imagine there's quite a lot of technical work that goes into maintaining something like that as well.
- 11:46SPEAKER_02: Yes.
- 11:46SPEAKER_02: I would say I'm running point we're very pro-innovation, right?
- 11:51SPEAKER_02: We're always encouraged to try and build our own tools.
- 11:56SPEAKER_02: If it doesn't work, then we outsource it.
- 11:58SPEAKER_02: But there's a high culture of innovation to try and build something custom because it comes in handy when you need to scale as well.
- 12:07SPEAKER_02: That uniqueness of your solution works.
- 12:10SPEAKER_02: So yeah, I'll say the team took a while to work on a lot of those internal tools, but it's helped us largely today.
- 12:17SPEAKER_01: Interesting.
- 12:18SPEAKER_01: Is that one of the reasons that you chose something like Airflow for the kind of open source flexible nature of it, that it's easy to customize?
- 12:26SPEAKER_02: Yes.
- 12:26SPEAKER_02: And also because I know Airflow also innovates a lot because a lot of new tools come out and we're able to play with that and experiment with that as well.
- 12:36SPEAKER_01: Yeah, definitely.
- 12:36SPEAKER_01: Or I guess I should have said easy in quotes to customize, possible to customize, maybe actually quite difficult to do, especially in production.
- 12:45SPEAKER_01: So yeah, that's really interesting.
- 12:47SPEAKER_01: Is there anything you found interesting about kind of your customization of airflow for this use case and plugging in with those tools?
- 12:57SPEAKER_02: No.
- 13:01SPEAKER_02: So far, I think what still gets me excited is a lot of dynamic DAGs, right?
- 13:07SPEAKER_02: Also dynamic tasks, just the ability to run a lot of things in parallel as well.
- 13:12SPEAKER_02: It's always quite interesting.
- 13:14SPEAKER_02: Right?
- 13:14SPEAKER_02: It comes in handy to save from a lot of stress, especially when I'm dealing with multiple data sources or multiple accounts for different stakeholders.
- 13:25SPEAKER_02: And once I'm able to find some similarity across board, I know that Airflow can handle the job of running all of that together.
- 13:32SPEAKER_02: So instead of doing multiple cron jobs manually, like a caveman, with Airflow everything in one place.
- 13:41SPEAKER_02: And even when we're making edits to the source code, I know it takes just about five minutes or so for those changes to reflect on Airflow's web server.
- 13:52SPEAKER_02: And that saves me a lot of time.
- 13:54SPEAKER_02: So I don't know.
- 13:55SPEAKER_02: It might be common to most people, but it gets me excited.
- 13:58SPEAKER_01: Yeah, absolutely.
- 14:00SPEAKER_01: And you mentioned dynamic DAGs.
- 14:01SPEAKER_01: I think if I heard right, you mentioned that part of your architecture for this particular use case is having one DAG per customer.
- 14:09SPEAKER_01: I'd imagine you're not doing that manually with the number of customers you have, that that's all dynamically done via code.
- 14:18SPEAKER_02: Yeah, it is.
- 14:19SPEAKER_02: And sometimes we even we've also automated allocation of accounts to different instances as well.
- 14:26SPEAKER_02: Oh interesting.
- 14:28SPEAKER_02: It makes it easy to onboard new accounts and everything's almost modular in some sense and it just keeps going.
- 14:35SPEAKER_01: Interesting.
- 14:36SPEAKER_01: So you actually divide like the airflow deployment or instance that your DAGs are running on automatically.
- 14:44SPEAKER_01: Oh, cool.
- 14:44SPEAKER_01: Yeah, that's an interesting implementation.
- 14:46SPEAKER_01: Is there what sort of things do you use to determine like which DAG should go to which deployment?
- 14:51SPEAKER_01: Or is it arbitrary and just splitting them up so that nothing gets overloaded with too many pipelines?
- 14:57SPEAKER_02: That was Miguel concern because we had cases before in the past where there was an overload of pipelines on certain instances.
- 15:05SPEAKER_02: So we just it's almost like load shedding just to be efficient.
- 15:10SPEAKER_02: Okay, yeah.
- 15:11SPEAKER_01: Yeah, yeah, interesting.
- 15:13SPEAKER_01: That sounds like a really cool implementation.
- 15:16SPEAKER_01: You also touched on just being excited about the innovation in the Airflow project as an open source project.
- 15:22SPEAKER_01: So I'm curious if there's anything there recently that you've been taking advantage of or want to touch on.
- 15:29SPEAKER_02: Currently, I'm not proud to say I haven't explored 3.0, FL 3.0 as much as I would want to.
- 15:38SPEAKER_02: So I know there's a I went for a presentation a few months back where I saw some cool things.
- 15:43SPEAKER_02: And I think that was right before the launch.
- 15:46SPEAKER_02: Right.
- 15:46SPEAKER_02: And one of the speakers had he had explored some of the new content there.
- 15:50SPEAKER_02: And I also noticed some changes to the UI, right?
- 15:54SPEAKER_02: Especially on the DAG overview page, where we could now see, we could see some of the patterns of the grid view of how the DAG runs right from the overview layer.
- 16:05SPEAKER_02: And there's a lot for me to explore, really.
- 16:08SPEAKER_02: I don't want to expose myself onto what I haven't checked out yet, but I'll say I'm really looking forward to it.
- 16:16SPEAKER_02: And hopefully I'll maybe I'll write an article or so on my best new features.
- 16:21SPEAKER_01: Yeah, absolutely.
- 16:23SPEAKER_01: I would love that for the Airflow community.
- 16:25SPEAKER_01: I think they'd be very interested to hear that.
- 16:26SPEAKER_01: And I think Airflow 3 is still relatively new.
- 16:29SPEAKER_01: I think most teams haven't actually gone through the upgrading process yet.
- 16:32SPEAKER_01: No, no embarrassment or anything needed there.
- 16:36SPEAKER_01: Uh, I guess that's a good tie-in, I think, to my standard final question that I always ask our guests.
- 16:42SPEAKER_01: So maybe related to Airflow 3 or maybe not, but that's what would you most like to see from the Airflow project in the near future?
- 16:49SPEAKER_02: I know I heard some talk about AI integration into a latest version already.
- 16:54SPEAKER_02: So I don't know how well that is, but if it's possible, maybe some partnerships with Gemini.
- 17:03SPEAKER_02: I don't know how minded Google might be about that.
- 17:08SPEAKER_02: But it would be fun to see that.
- 17:10SPEAKER_02: Because I think that would also really it would also get it would also get non-technical people curious, you know, about Airflow as well.
- 17:19SPEAKER_02: I'm still thinking of like use cases of Airflow for the average person.
- 17:24SPEAKER_02: That's not necessarily data inclined, because one personal project that I've been playing around with Airflow, I'm not done with it, but it's like to automate the birthdays or people in my contact lists and like automatically send out emails and messages to them on a personal basis.
- 17:42SPEAKER_02: And I just find that fun.
- 17:44SPEAKER_02: I don't know if the average person might use cases like that.
- 17:48SPEAKER_01: Yeah, yeah, interesting.
- 17:49SPEAKER_01: I uh you're making me realize that there are a lot of things I've never thought of Airflow to use for using it, and it really should.
- 17:55SPEAKER_01: I don't use it in my personal life at all, and maybe that's a missed opportunity.
- 17:58SPEAKER_01: I love that from the AI perspective.
- 18:00SPEAKER_01: Uh kind of would you like to see that from the DAG authoring?
- 18:04SPEAKER_01: So if there was an integration with Gemini or something, so it would help users write DAGs, or what do you envision there?
- 18:11SPEAKER_02: I think what DAG goes through, it would it could help, especially for maybe for beginners as well.
- 18:18SPEAKER_02: Help with them and getting stuck and maybe also debugging.
- 18:23SPEAKER_02: I think that could come in handy as well for debugging, especially in custom cases where I don't know how as many people that use Chat GPT.
- 18:31SPEAKER_02: It doesn't bring so much code as people think it does.
- 18:35SPEAKER_02: We still need that personal experience tailored.
- 18:38SPEAKER_02: So maybe a collab like that with AI that helps in the debugging process.
- 18:42SPEAKER_01: Yeah.
- 18:43SPEAKER_01: Absolutely.
- 18:44SPEAKER_01: That would be great.
- 18:45SPEAKER_01: I think debugging is such a difficult topic for a lot of folks.
- 18:48SPEAKER_01: We it will have passed by the time this is published, but we do have a webinar coming up on debugging and testing, and it's always it'll be August 6th.
- 18:56SPEAKER_01: For those listening to this later, I think the recording of the webinar will be available.
- 19:00SPEAKER_01: But yeah, we do that because it's an issue that a lot of people struggle with.
- 19:04SPEAKER_01: And I think it is a good potential place for AI to help out.
- 19:09SPEAKER_01: I think from a DAG authoring perspective, there's something that I know of planned in the open source Airflow project.
- 19:14SPEAKER_01: I know it's something Astronomer is thinking about a lot and working on with our own projects.
- 19:18SPEAKER_01: And I do know that in Airflow 3.1, they will have the human in the loop feature release.
- 19:24SPEAKER_01: I'm for more managing like AI workflows if you need to have points where a person needs to come in and check the work of the AI or something like that, or decide which path to take and things like that.
- 19:35SPEAKER_01: So I think there's a lot of excitement around that.
- 19:39SPEAKER_02: Sounds really cool.
- 19:40SPEAKER_02: Honestly, it sounds really cool.
- 19:42SPEAKER_02: It makes me want to go back into core AI work.
- 19:46SPEAKER_02: With data engineering, it's easy to get carried away into the crowd.
- 19:50SPEAKER_02: But AI projects also sound really fun.
- 19:52SPEAKER_02: Now, on Airflow, I have to try that out when it comes out.
- 19:55SPEAKER_01: I have yeah, definitely.
- 19:57SPEAKER_01: That should be, I think mid-September is the target for our 3.1, at least as of right now.
- 20:02SPEAKER_01: So pretty soon here.
- 20:03SPEAKER_01: It'll be here before we know it.
- 20:05SPEAKER_02: That's pretty close.
- 20:06SPEAKER_02: I was thinking you were going to say 2026 or something.
- 20:09SPEAKER_01: Oh gosh, no, no.
- 20:11SPEAKER_01: I think the Airflow project generally minor releases will go back now to being once a quarter-ish.
- 20:17SPEAKER_01: So every three to four months.
- 20:19SPEAKER_01: I know the schedule was a little bit different than it had been the past few years with the Airflow 3 release, obviously, it was a bigger step forward.
- 20:25SPEAKER_01: But yeah, there's still a lot more being added to the project even after that big release.
- 20:29SPEAKER_01: Yeah, cool.
- 20:30SPEAKER_01: So it was so great speaking with you today.
- 20:32SPEAKER_01: I know we're about at the end of our time.
- 20:34SPEAKER_01: So, what is the best way for folks to get in touch with you?
- 20:37SPEAKER_02: I would like to sound professional and say LinkedIn, but yeah, LinkedIn is a good place to reach out to me if you want, especially if you want to talk about data engineering work.
- 20:45SPEAKER_02: I'm active there.
- 20:47SPEAKER_02: Not so active on Twitter or social media, but if you find me there.
- 20:52SPEAKER_01: Okay, we'll throw the LinkedIn in the show notes for sure.
- 20:55SPEAKER_01: Thank you again for coming on.
- 20:56SPEAKER_01: It's been a pleasure talking to you about this.
- 20:58SPEAKER_02: Yeah, thank you so much for having me.
- 21:00SPEAKER_02: This was really fun.
- 21:01SPEAKER_02: I really hope to come for an astronomer event soon.
- 21:04SPEAKER_02: Wait to see what that would be like in person.
- 21:07SPEAKER_01: Yeah, that would be great.
- 21:08SPEAKER_01: We would love to have you somewhere.
- 21:10SPEAKER_02: Awesome.
- 21:10SPEAKER_01: Okay, thanks again.
- 21:14SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
- 21:21SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
- 21:24SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.