Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating cost-efficient data pipelines at Insignis

Transcript

Orchestrating cost-efficient data pipelines at Insignis · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: You are listening to the data of podcasts.
• 0:07
  SPEAKER_01: We are excited to announce the law of data.
• 0:17
  SPEAKER_01: We are offering Survey Respondents something special.
• 0:20
  SPEAKER_01: A free code to take the new Astronomer Certified AI orchestrate fundamental text.
• 0:26
  SPEAKER_01: $150 value.
• 0:28
  SPEAKER_01: Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
• 0:35
  SPEAKER_01: Now let's get into the show.
• 0:38
  SPEAKER_00: Okay.
• 0:38
  SPEAKER_00: Hi everyone.
• 0:39
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:41
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Anaruda Sengupta, who is a staff data engineer at Insignus.
• 0:48
  SPEAKER_00: Anaruda, how are you?
• 0:50
  SPEAKER_02: Yeah, I'm good.
• 0:51
  SPEAKER_02: Doing good.
• 0:51
  SPEAKER_02: Thanks.
• 0:52
  SPEAKER_02: How are you?
• 0:52
  SPEAKER_00: I'm good.
• 0:53
  SPEAKER_00: It's great to have you on.
• 0:54
  SPEAKER_00: I'm excited to chat today.
• 0:57
  SPEAKER_00: For our listeners, bit of a preview for today's podcast.
• 1:01
  SPEAKER_00: We're going to discuss how Insignus runs its data platform on Airflow and some advice from the team on controlling costs and implementing best practices, which I have to say, I think are two of maybe the most popular topics ever within Airflow and orchestration.
• 1:16
  SPEAKER_00: So I think, yeah, our listeners will get a lot out of this.
• 1:20
  SPEAKER_00: But as always, I want to start with a little bit of background.
• 1:23
  SPEAKER_00: So tell me first about the company you work for.
• 1:27
  SPEAKER_00: What does Insignus do?
• 1:29
  SPEAKER_02: Yeah, sure.
• 1:30
  SPEAKER_02: So Insignus is a FinTech based in the UK.
• 1:34
  SPEAKER_02: We've got offices in London and Cambridge.
• 1:37
  SPEAKER_02: We basically operate in the cash savings industry in the UK.
• 1:43
  SPEAKER_02: So we manage cash savings across businesses and people and basically make it as easy as possible for people to manage and diversify their cash holdings to get it to yield the highest possible returns.
• 1:59
  SPEAKER_02: And we do that by um, you know, we've built a client platform where clients basically instruct us to move their money into differing savings accounts that yield the highest interest rates.
• 2:14
  SPEAKER_02: So basically, we don't want um for charities, for schools, for example, we don't want their cash reserves sitting idle.
• 2:22
  SPEAKER_02: We also want it to do the hard work so that you know it it works hard alongside them in a it's it's not just a case of it sits there and does nothing.
• 2:33
  SPEAKER_00: Okay, yeah, that's great.
• 2:35
  SPEAKER_00: And I would imagine for a company like that, I mean FinSurf always has a lot of data to manage, I think, lots of sensitive data that you're dealing with.
• 2:45
  SPEAKER_00: Yeah, for sure.
• 2:46
  SPEAKER_02: Like, yeah, like um there's lots of transactional data, um, which you can imagine flying around.
• 2:52
  SPEAKER_02: Um, so yeah, it's a it's a case of um implementing uh best practices in a way so that you know we don't get bankrupted, but also keeping that uh kind of PII front and center, um, and just being aware uh of what is PII and what is not.
• 3:13
  SPEAKER_00: Yeah, absolutely.
• 3:14
  SPEAKER_00: And so tell me a little bit more about then how your data team is structured and kind of what you're responsible for.
• 3:21
  SPEAKER_02: Yeah, so we basically um our the the core um platform is built in C sharp.
• 3:30
  SPEAKER_02: And so we've got three kind of operational teams that look after that, which is you know look divided into three distinct uh areas.
• 3:38
  SPEAKER_02: The data team basically sits across the entire business um to kind of consume data from everywhere uh in the Insignus engine, as I like to call it, and try and deliver data-driven insights to the rest of the business and to the C-suite.
• 3:58
  SPEAKER_02: Um and uh we not only sit across the operational team, so we collect internal data, but we also collect a lot of external data as well.
• 4:09
  SPEAKER_02: So our um customer services team, for example, their calls such as they have with their with clients, get recorded in Ringover, so we export data from Ringover.
• 4:22
  SPEAKER_02: We've got our um CRM set up in Salesforce, so we um import data from Salesforce, and you know, so we try and build a kind of single plane of glass so that anyone in the company can just come to us with uh uh an insight they either want to know or or they want to look at data, where that single place where you can look at all the data of what's happening in the company.
• 4:50
  SPEAKER_02: I like to think of us as a basically a newspaper.
• 4:54
  SPEAKER_02: You you you kind of get an idea of what's happening in the business, but and we'll we'll get I'm sure we'll get into that later.
• 5:02
  SPEAKER_02: What is important for us is we don't have um core business kind of logic sitting within our data platform.
• 5:11
  SPEAKER_02: We are just simply there to be the newspaper for the rest of the business.
• 5:16
  SPEAKER_00: Okay, interesting.
• 5:17
  SPEAKER_00: I love the analogy of the newspaper, and so you're just providing all of the information, then what the other teams do with it is sort of up to them based on their use cases.
• 5:28
  SPEAKER_02: Yeah, it's almost like yeah, it like and sometimes I I think it's served us very well so far.
• 5:34
  SPEAKER_02: And sometimes a bit people have been like, uh, you know, why why is this broken?
• 5:39
  SPEAKER_02: It's like, don't know, like you have to ask the operational teams.
• 5:42
  SPEAKER_02: I'm just uh I'm almost like the messenger here.
• 5:44
  SPEAKER_02: I just report on what I see.
• 5:46
  SPEAKER_00: Yeah, yeah, that makes sense.
• 5:49
  SPEAKER_00: And I think you have told me previously, if I'm not mistaken, that you are the only data engineer at the company.
• 5:56
  SPEAKER_00: So your function, at least within that team, is of one person.
• 6:01
  SPEAKER_02: Yes, correct.
• 6:02
  SPEAKER_02: We've uh we've got um um another person.
• 6:06
  SPEAKER_02: He's he's more of a data analyst like his job title says data engineer, but we also need a data analyst within the team.
• 6:12
  SPEAKER_02: So he looks after the data analysis side, and yeah, I looked after mostly the data engineering side.
• 6:18
  SPEAKER_00: Okay.
• 6:19
  SPEAKER_00: And that means you probably got a lot of responsibility that's just on you.
• 6:25
  SPEAKER_00: I'm curious how does that shape you know the way that you work and yeah, like in in the beginning it was tough for sure.
• 6:32
  SPEAKER_02: Um I came from JLR from Jaguar Land Rover, and that had a very big data engineering team.
• 6:41
  SPEAKER_02: So in the beginning, really, I didn't if I had ideas that I wanted to discuss, I couldn't really discuss them with anybody else.
• 6:51
  SPEAKER_02: Um, so for me personally, AI has been really helpful with that, just as a like someone to have a back and forth, like a like to bounce ideas off.
• 7:02
  SPEAKER_02: I you know, I've I've definitely done it with my software engineering colleagues, um, and they have been helpful to an extent, but to like it's different when you're talking to like a data engineering colleague because they they might not have all the context.
• 7:17
  SPEAKER_02: Yeah, a software engineer might not have all the context a data engineering does, a data engineer does.
• 7:23
  SPEAKER_02: Um and but but what's also important for us is the way we shape our work is we still want the data engineering pipelines to be treated as first-class citizens.
• 7:36
  SPEAKER_02: So it has the same rigorous set of testing, see set of CICDs, extensive documentation.
• 7:43
  SPEAKER_02: You know, it's not I certainly don't allow it to be treated like an afterthought.
• 7:49
  SPEAKER_02: Um, you know, it it it deserves the same kind of um rigorousness as the other software engineering teams as well.
• 7:58
  SPEAKER_00: Yeah, absolutely.
• 8:00
  SPEAKER_00: Well, I'm sure that constraint will be relevant to some of our upcoming discussion about airflow as well.
• 8:06
  SPEAKER_00: I have also been myself not in my current role, but in pastorals in this position of being the only analyst at a company or something like that.
• 8:16
  SPEAKER_00: And yeah, it's a hard, it's a hard spot to be in sometimes, definitely.
• 8:19
  SPEAKER_02: Yeah, definitely.
• 8:21
  SPEAKER_00: Um yeah, so that's great that you've managed to make that work.
• 8:25
  SPEAKER_00: But again, I I do want to talk about how you know Airflow fits into this.
• 8:29
  SPEAKER_00: So tell me about how your data platform is structured and what role Airflow plays.
• 8:35
  SPEAKER_02: Yeah, so Airflow is basically at the very heart of our data, of our data infrastructure.
• 8:41
  SPEAKER_02: Um, it's our main tool for orchestrating data pipelines.
• 8:45
  SPEAKER_02: So we not only use it to orchestrate when we, for example, get uh third-party data from using APIs, but we also use it as an observation layer.
• 8:58
  SPEAKER_02: So we've got um a couple of real-time pipelines that we have, which is not built with Airflow, but Airflow kind of sits on top of those to build an observation layer to ensure that you know those real-time pipelines are definitely working uh as intended.
• 9:18
  SPEAKER_02: And that kind of observe like observability layers we found that it's very useful with what capability Airflow provides for us, uh, you know, as well as being you know our standard tool for kind of like you know, the standard ETL job.
• 9:36
  SPEAKER_02: Um, yeah.
• 9:38
  SPEAKER_02: And yeah, and we also use a lot of and again I'll get into that, but yeah, like for us, and I'm sure this is not a uh not a revolutionary thing, I'm gonna say, is Airflow for us is the the orchestrator for everything.
• 9:53
  SPEAKER_02: It it tells other things to do things and then returns for a result back, but doesn't actually do the things itself.
• 10:01
  SPEAKER_02: Um so it yeah, so it it like it it controls everything, it's like the brain, but it doesn't actually perform any of the heavy lifting as much as possible.
• 10:12
  SPEAKER_02: Other tools perform, you know, for example, SQL pipelines, they perform the head like in BigQuery, it performs the heavy lifting, and Airflow just orchestrates that.
• 10:22
  SPEAKER_00: Yeah, absolutely.
• 10:23
  SPEAKER_00: I think that is a really common pattern, and I do want to get into that a little more before we do, just high level.
• 10:29
  SPEAKER_00: Tell me a little bit about what kinds of data pipelines you're running with Airflow.
• 10:34
  SPEAKER_02: Yeah, so we've got um we've got a few operational ones, we've got a few where we use uh where we try and get data from external systems, and we try and do that as um like even if they're comp like even if we are using APIs, we try and use methods that are kind of best practices, not just to data engineering, but to software engineering, things like using uh watermarks when retrieving API results.
• 11:05
  SPEAKER_02: We try in terms of kind of cost saving, we try and stay two or three steps ahead.
• 11:13
  SPEAKER_02: You know, you get data from an API, you you get all the data there, it's fine, it works, you know.
• 11:20
  SPEAKER_02: But we've because we don't have that lock, that luxury of having a huge budget behind us, we kind of have to think two or three steps ahead, like right, it works, but now how can I make it as efficiently?
• 11:31
  SPEAKER_02: How can I get the data as efficiently as possible?
• 11:35
  SPEAKER_02: And on top of that as well, we've got some we've got some operational pipelines of our own, which the businesses rely on um to use every day.
• 11:46
  SPEAKER_02: So we also make sure that in our the data pipelines that we do have, the relevant observe observability layers, the relevant testing, making sure that you know things like if it fails or if it misses an SLA, it's all kind of there as in best practice to catch it so that incidents can be resolved as painlessly as possible.
• 12:09
  SPEAKER_00: Okay.
• 12:10
  SPEAKER_00: Yeah, got it.
• 12:12
  SPEAKER_00: And then you've alluded to this a little bit, but I know you've also spoken about building, you know, a low-cost, scalable platform for airflow.
• 12:22
  SPEAKER_00: And we've talked already about how you're working within the constraint of you know a very small team.
• 12:28
  SPEAKER_02: Yeah.
• 12:29
  SPEAKER_00: Tell me about how you have gone about implementing this, maybe anything else about kind of the importance of doing it.
• 12:35
  SPEAKER_00: And I guess, you know, what advice you would give to other small teams who are trying to use Airflow, but you know, need to be very conscious of cost and things like that, regardless of where or how they're running it.
• 12:47
  SPEAKER_02: Yeah, I think I think if you're in a similar position, one of the things I'd say first is um observability building in observability layers is should be a part of your definition of done and not just an afterthought.
• 13:03
  SPEAKER_02: And I don't and I don't necessarily mean observability in the sense that um I went to my Airflow pipeline, I went to the Airflow UI, it's all green.
• 13:14
  SPEAKER_02: That's not observability.
• 13:16
  SPEAKER_02: Observability is in my opinion, taking a step back and kind of seeing like is the output why expect?
• 13:25
  SPEAKER_02: Like am I like the pipeline runs fine, but are things being silently dropped?
• 13:32
  SPEAKER_02: And being very kind of curious about that, so building in observability layers where uh a great thing that's worked for us is uh dashboards built in in um in our stack that is kind of independent of the main data pipelines themselves.
• 13:49
  SPEAKER_02: So it kind of does his thing, the main data pipelines do their thing, so uh we kind of know that yes, the pipeline's passed, and yes, the data that is being fetched, that whatever it's trying to do is working as intended.
• 14:05
  SPEAKER_02: So that that will definitely help to keep costs uh um uh running as efficiently as possible.
• 14:12
  SPEAKER_02: And the other thing as well is when you're a small team, um automation is very important.
• 14:19
  SPEAKER_02: Uh so to automate and this might sound like an obvious thing, but you'd be surprised how many times I've I mean, when when I joined in Insignus, for example, uh, you know, this was happening.
• 14:30
  SPEAKER_02: Automate things that take the most time.
• 14:32
  SPEAKER_02: So uh it can be used intelligently as a substitution for headcount if you know what you're doing.
• 14:39
  SPEAKER_02: You don't uh the the um one of the aims of the big game is to uh uh try and scale the company uh but not necessarily scale it with headcount, right?
• 14:53
  SPEAKER_02: If that makes sense.
• 14:54
  SPEAKER_02: So be uh mindful and be intelligent of what you can automate, what takes you the most time, and go ahead and do it.
• 15:01
  SPEAKER_02: That that that will definitely help you with with kind of cost savings.
• 15:05
  SPEAKER_02: Um and then I I will also say the other thing is in our data platform we used a managed airflow version, we don't host things ourselves ourselves because whilst we do have a um platform team, it's not solely dedicated to the data team.
• 15:28
  SPEAKER_02: Um if you use the managed airflow version, uh like the things like the scheduler, the workers, the database is all is all kind of abstracted away from you.
• 15:38
  SPEAKER_02: So that leaves you free to write your code and write your documentation so that you know you don't have to spend your time putting on your platform hat and looking after those as well.
• 15:50
  SPEAKER_02: So that that will also help you to kind of manage your costs if you are a small team and you want to get started in Airflow.
• 15:59
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
• 16:08
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 16:12
  SPEAKER_01: Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 16:18
  SPEAKER_01: Check it out at astronomer.io.
• 16:23
  SPEAKER_00: Yeah, absolutely.
• 16:24
  SPEAKER_00: I there's so much good stuff there that I want to double-click on.
• 16:28
  SPEAKER_00: I do think the managed service, of course, we're uh very heavily biased at Astronomer, but we do think that this helps people quite a lot.
• 16:35
  SPEAKER_00: Um, again, as you said, so that you don't have to worry about the airflow infrastructure.
• 16:40
  SPEAKER_00: And I think that um relevant for automation as well, because I think it allows you to implement, you know, things like auto-scaling and stuff like that, where you can um just have a lot more intelligence built into the platform as opposed to trying to manually tweak those things.
• 16:55
  SPEAKER_00: Uh I'm curious actually, going back to the first thing you said on observability and how it's not just, you know, looking and seeing if you had any tasks fail, but also making sure that your output looks like what it should and all of that.
• 17:09
  SPEAKER_00: I, you know, oftentimes I think talk to people about that in the context of implementing things like data quality checks so that they're catching, you know, things where the task succeeded, but like something is wrong with the data.
• 17:21
  SPEAKER_00: I don't know if that's exactly what you're talking about.
• 17:24
  SPEAKER_02: Um I mean, I can give I can give a specific example without um without giving away company secrets.
• 17:31
  SPEAKER_02: So yeah, data quality checks are definitely important.
• 17:34
  SPEAKER_02: We we definitely we definitely use um a lot of them in our pipelines to make sure that the the data kind of before and and after you know it all looks and is all aligned.
• 17:48
  SPEAKER_02: But what I mean, you know, a great example of this is um our customer services team chat with clients every day and we record calls and transcripts, right?
• 18:03
  SPEAKER_02: Now we've got uh a data pipeline that gets calls and transcripts and every day it shows yeah, it's green, it's getting them.
• 18:14
  SPEAKER_02: But how do you know that they're getting all of them?
• 18:17
  SPEAKER_02: How do you know that they're not silently dropping anything?
• 18:20
  SPEAKER_02: How do you know that um uh you know things are not being muddled along the way?
• 18:25
  SPEAKER_02: So that's where building a separate observer, which I I I appreciate it can um it can be case specific, it can be project specific.
• 18:37
  SPEAKER_02: I mean, the example I'm giving is specific to this particular pipeline, it might not apply to others.
• 18:43
  SPEAKER_02: So, but we've built observable observability layers where we look at the output of what we've actually collected and then do like a kind of summary, like right, okay, um airflow pipeline is green.
• 18:58
  SPEAKER_02: Ah, but like look, we've got all the calls that Ringover said we've had, we've got all the transcripts, so there doesn't seem to be any errors, nothing is being dropped silently.
• 19:07
  SPEAKER_02: So that's why you know that observability layer in addition to your data quality checks for us is what has has been quite important.
• 19:17
  SPEAKER_00: Yeah, makes sense.
• 19:18
  SPEAKER_00: And tell me about how so, of course, having that data quality is really important for being able to make accurate business decisions, but you're also making the argument for cost optimization.
• 19:31
  SPEAKER_00: So, how does having that observability layer help with cost?
• 19:35
  SPEAKER_02: So it helps with costs because I think um you could argue like, well, you're building more things, so in the short term is costing you more, right?
• 19:48
  SPEAKER_02: But by having that observability layer, we're not we're actually we the thing that's gonna the thing that it will do is prevent incidents from happening.
• 20:00
  SPEAKER_02: So people downstream are not gonna scream like, well, I I can't see any of the ring over data, where is it?
• 20:05
  SPEAKER_02: That in itself will incur a much bigger cost than the kind of short-term costs you see of building those observability layers in the first place.
• 20:14
  SPEAKER_02: So the cost of having that incident downstream is gonna be much bigger than the cost, then the initial cost of building the observability layer from the start.
• 20:24
  SPEAKER_00: Got it.
• 20:25
  SPEAKER_00: Yeah, okay, I see.
• 20:27
  SPEAKER_00: And I guess related to that potentially, are there any patterns within your DAGs themselves in terms of implementing best practices that you think have made a big difference for you in terms of cost or scalability?
• 20:44
  SPEAKER_00: Maybe other than data quality checks?
• 20:46
  SPEAKER_02: Yeah, other than data quality and other than using, which I've already which we've spoken about, using Airflow as an orchestration engine, not an execution one.
• 20:56
  SPEAKER_02: Um I would say automating your CI in a deployment, don't be kind of tempted to leave those as an afterthought.
• 21:05
  SPEAKER_02: Um, and let your CI pipeline handle as much of the automated tests as possible.
• 21:10
  SPEAKER_02: Let machines pick up things so you don't have to manually kind of do it yourself.
• 21:16
  SPEAKER_02: Things like linting, unit testing, um documentation, and make sure that all your DAGs are kind of consistent, they all have a level playing field, and a great way to do that is enforcing it through your CI CD pipeline, um, and making sure that you know, like your DAGs have all the kind of best practices set out by Airflow and Astronomer, things like failure callbacks, um, being mindful of where you how you put the documentation, being mindful of where you're when your DAG starts and when it ends, and making sure it doesn't, you know, it doesn't go rogue and just do loads of which I have seen happen um before.
• 21:57
  SPEAKER_02: So um And the other thing as well for us is we purge our logs and our metadata because we find that the metadata table can build up very quickly.
• 22:12
  SPEAKER_02: So to kind of make sure that you know things are operating at that sweet spot, we purge them periodically.
• 22:18
  SPEAKER_02: Um so that you know we don't hold on to logs you know for you know that like a year old.
• 22:24
  SPEAKER_02: You know, we don't we don't need to see um you know that that it's just that you know logs that are recent that that may produce a failure, those are the ones that we want to see so that we can begin to debug if we need to.
• 22:38
  SPEAKER_00: Yeah, that's a recommendation we make very commonly to our customers as well.
• 22:42
  SPEAKER_00: It can have big performance impacts.
• 22:44
  SPEAKER_00: Yeah, definitely.
• 22:46
  SPEAKER_00: Okay, and then I understand that you have also built some internal tools kind of around your Airflow platform to help keep it running efficiently.
• 22:56
  SPEAKER_00: Tell me about those and how they help you.
• 22:59
  SPEAKER_02: Yeah, so the first thing is um uh we built a tool to show our SQL pipeline costs and we call that tariff.
• 23:10
  SPEAKER_02: Um so basically, uh Airflow is responsible for orchestrating our SQL pipelines as well.
• 23:17
  SPEAKER_02: And the the reason why we built this tool is one, we you know, we just want to see in a plain and clear way as possible how much our SQL pipelines are costing.
• 23:29
  SPEAKER_02: We don't want those to be creeping up and then you know um you know leaking into our airflow costs and our overall data infrastructure costs as well.
• 23:38
  SPEAKER_02: That's something that we've that you know it has a front end and a back end.
• 23:43
  SPEAKER_02: Um, and we find that it's definitely given us visibility and and some interesting insights, things like um, if you have kind of you know incremental mechanisms, what you might think is like, oh yeah, it's it's working really great, it's actually costing you a lot.
• 24:01
  SPEAKER_02: So it's actually helped us with the optimizations um a lot in our data infrastructure to kind of keep costs down.
• 24:08
  SPEAKER_02: The second thing is um we built an AI-powered tool to give recommendations on how to improve the Airflow platform in terms of like the workers, their memory, the scheduler, how big the database should be.
• 24:23
  SPEAKER_02: We call that Maestro.
• 24:24
  SPEAKER_02: Um previously this kind of took a long time, and it was it was for me to do.
• 24:30
  SPEAKER_02: Um, you know, it would you would kind of have to analyze your DAGs, analyze how they did over a period of time, analyze how your workers in your Airflow environment are doing.
• 24:43
  SPEAKER_02: You know, are they running out of memory frequently?
• 24:46
  SPEAKER_02: Are they just about to run out of memory or are they doing okay?
• 24:49
  SPEAKER_02: And then it was very much a finger in the air, like, oh I think the worker provision should be increased by X amount.
• 24:56
  SPEAKER_02: This kind of takes it all away, um, you know, and it takes it takes minutes, you know, to do and give recommendations.
• 25:04
  SPEAKER_02: Um, so no, so tools like that where it's really helped us to make sure that again that goes back to the point of automation.
• 25:12
  SPEAKER_02: You know, I could have easily not easily, but it would have taken me a while to kind of look through a SQL pipeline and go, right, you know, what are what is uh costing the most here and what is costing the least and why.
• 25:25
  SPEAKER_02: You know, we've kind of automated all of that.
• 25:27
  SPEAKER_02: Same for Maestra as well.
• 25:30
  SPEAKER_02: So it's tools like that that kind of helped us to uh you know build our data infrastructure, you know, well from the start, uh, and to make sure that it's it's not just about, like I said before, it's not just about short-term let's just get costs down, let's just spend a pound a day.
• 25:49
  SPEAKER_02: The the the point I want to stress here is you want to find that sweet spot.
• 25:55
  SPEAKER_02: Just you you want to give, for example, you want to give your airflow workers just enough juice so that they need they can they have what they need to do to do the work that they're asked to do, but not so much that you are it's costing you an an absolute arm and leg.
• 26:11
  SPEAKER_02: So you want to find that sweet spot, and so the tools we've built there is helping us to find that sweet spot to help us you know keep our data pipelines running every day.
• 26:21
  SPEAKER_00: Yeah, I think that's a great description of kind of striking the right balance between those short-term adaptations and the long-term sustainability of your platform.
• 26:31
  SPEAKER_00: Yeah, very interesting.
• 26:33
  SPEAKER_00: Well, on a slightly different topic, maybe, maybe it's related.
• 26:37
  SPEAKER_00: Uh, I know you've also mentioned that you just started exploring AI with Airflow, which is of course a really popular topic right now.
• 26:45
  SPEAKER_00: Um, I'm curious what sort of use cases are you looking at implementing there?
• 26:50
  SPEAKER_02: So one of some of the obvious UK use cases is probably some analysis of the movement of money.
• 26:58
  SPEAKER_02: So things like um our clients, how do they kind of invest with us?
• 27:04
  SPEAKER_02: Um, it's more kind of MLOPs, actually.
• 27:07
  SPEAKER_02: Um, you know, we want we want to kind of identify patterns and try and capitalize on them.
• 27:13
  SPEAKER_02: Um with move like you know, and maybe things like in our customer journey, things may not be as optimized as they can be.
• 27:22
  SPEAKER_02: So we what kind of you kind of you want to use AI and MLOPs to help us identify those.
• 27:27
  SPEAKER_02: One of the other things I found really inspiring from the Airflow Summit, which has just gone by, is you know, the the idea of self-healing pipelines, which is a very juicy topic to me.
• 27:39
  SPEAKER_02: So, you know, things like and again that comes back to the whole point, right?
• 27:43
  SPEAKER_02: Automating it away as much as I can.
• 27:46
  SPEAKER_02: Not automating it fully so that an AI just goes wild and on the rampage and changes things, but to help us as much as possible when there are things going wrong, you know, an AI can do the analysis much faster, looking through logs and seeing what's got what's gotten wrong, implementing uh suggesting a fix, an AI can do that much faster than a human can.
• 28:09
  SPEAKER_02: So definitely in our future, a kind of self-healing pipeline.
• 28:12
  SPEAKER_02: And and there was loads of cool talks, people had different ways of doing it.
• 28:17
  SPEAKER_02: There was, you know, one of the talks I think I went to was a DAG does it, a DAG that listens to other DAGs.
• 28:23
  SPEAKER_02: So that was cool as well.
• 28:25
  SPEAKER_02: We've got some ideas of how it can be done at Insignus for us, so we're definitely very interested to leverage that.
• 28:34
  SPEAKER_00: Yeah, absolutely.
• 28:35
  SPEAKER_00: This is something I'm doing myself as well.
• 28:37
  SPEAKER_00: Of course, Astronomer has our own perspective on how we can implement this within our platform.
• 28:41
  SPEAKER_00: As you said, there's many ways of doing it.
• 28:43
  SPEAKER_00: I love that the community are sharing those.
• 28:45
  SPEAKER_00: But yeah, it has certainly helped me a lot in the debugging of the pipelines that you know provide metrics and stuff for my team.
• 28:53
  SPEAKER_00: Um, it's often very quick at identifying any issues and fixing them compared to what it used to take me to debug them myself.
• 29:01
  SPEAKER_00: Yeah, that's great.
• 29:02
  SPEAKER_00: Well, I think uh in the interest of time, that's a good uh lead into my standard final question, which is what would you most like to see from the Airflow project in the near future?
• 29:11
  SPEAKER_02: I think one of the things actually I'd like to see is um in terms of when DAGs get passed, um, so that when they're when when they when they show up in the UI, I'd like to see that a little bit faster.
• 29:26
  SPEAKER_02: It's not a huge bottleneck for us, but it sometimes takes as long as longer than we like to see when we implement changes to then kind of see that in the UI.
• 29:36
  SPEAKER_02: Um and I know it a lot of it comes down to the DAG file processing, if you have things like heavy, heavy top-level imports.
• 29:44
  SPEAKER_02: Um, but I think the repassing at the moment works on an interval rather than an event-based change.
• 29:52
  SPEAKER_02: So I'd love for X Airflow to explore that more.
• 29:55
  SPEAKER_02: Um to you know, to make it you know a little bit more the DAC passing a little bit more event-based, so that you can kind of see changes close to instantaneous rather than kind of having waiting an interval.
• 30:09
  SPEAKER_00: Yeah, absolutely.
• 30:10
  SPEAKER_00: That's a great one.
• 30:11
  SPEAKER_00: Okay, well, Inurita, thank you so much for coming on.
• 30:14
  SPEAKER_00: This has been a really great discussion.
• 30:16
  SPEAKER_00: Like I said, I think it will be very helpful for a lot of our listeners.
• 30:19
  SPEAKER_00: Just to close this out, what is the best way for folks to get in touch with you?
• 30:23
  SPEAKER_02: Uh you can just yeah, find me on LinkedIn, probably.
• 30:26
  SPEAKER_02: I'm quite active there.
• 30:27
  SPEAKER_02: Um, but yeah, feel free to also reach out to me via email.
• 30:32
  SPEAKER_02: Um yeah, I'd say those are the two options.
• 30:36
  SPEAKER_00: Great.
• 30:37
  SPEAKER_00: We'll throw your LinkedIn in the show notes for sure.
• 30:39
  SPEAKER_00: Well, thank you again for coming on.
• 30:41
  SPEAKER_00: It's been great speaking with you.
• 30:42
  SPEAKER_02: No problem.
• 30:43
  SPEAKER_02: Yeah, thanks for having me.
• 30:44
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 30:47
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 30:51
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 30:56
  SPEAKER_01: And leave us a five star review.