Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer

Transcript

Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: So the way that it worked for us was that our analysts were working on DVT models in general.
• 0:04
  SPEAKER_02: As I said, there were some like bizarrely specific Python models that some people built, but 99% of the time we were working with these SQL models.
• 0:13
  SPEAKER_02: But what we did, we pre-invented a much worse version of that, like back in the past.
• 0:18
  SPEAKER_02: Obviously, you guys have done it better than us, but we pre-invented our own version of it.
• 0:21
  SPEAKER_02: The idea was that in CI, we would take the take all of the model definitions, generate all of the SQL for them, and then that's what Airflow would actually end up running.
• 0:29
  SPEAKER_02: And you'd also create a you'd also create from that a definition of the DAG that Airflow could consume.
• 0:34
  SPEAKER_02: So that's the kind of the mapping bit that's interesting.
• 0:41
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:50
  SPEAKER_00: Hello everyone.
• 0:52
  SPEAKER_00: Welcome to the Leda Flowcast.
• 0:54
  SPEAKER_00: I'm your host, Mark Lomberti, and I'm super excited because today I'm joined by Jonathan Rayner, staff engineer at Apollo GraphQL, previously the line platform engineer at Monzo Bank.
• 1:05
  SPEAKER_00: So how are you doing, Jonathan?
• 1:07
  SPEAKER_02: Hey, thank you very much for having me.
• 1:09
  SPEAKER_02: It's really great to be here.
• 1:10
  SPEAKER_02: Um yeah, doing well.
• 1:11
  SPEAKER_02: Lots of it's always this kind of period in the kind of window to Christmas when you suddenly realize everything you have to get done before you wind down.
• 1:17
  SPEAKER_02: And so we're in that kind of crazy period at the moment, but that's life sometimes, isn't it?
• 1:22
  SPEAKER_00: Yeah, I feel you, I feel you.
• 1:23
  SPEAKER_00: But I actually haven't prepared anything yet, so maybe I should.
• 1:26
  SPEAKER_00: Alright, so let's dive in.
• 1:28
  SPEAKER_00: For today's podcast, we will discuss about DAG consistency issues, the UI scaling challenges.
• 1:34
  SPEAKER_00: Honestly, this one for me is really interesting because when we talk about scaling issues, we always tend to think about resources like memory and CPU, but actually, how do you m use the UI when you have thousands of tasks?
• 1:48
  SPEAKER_00: That is something that you need to take into consideration.
• 1:52
  SPEAKER_00: Then how to monitor Airflow at scale and some practional lessons that you learned through your journey with Airflow at scale.
• 1:59
  SPEAKER_00: So maybe we could start with some background, like what did you do at Monzo Bank as a data platform engineer?
• 2:05
  SPEAKER_02: Yeah, of course.
• 2:06
  SPEAKER_02: So essentially, I was part of the data platform engineering team at Monzo.
• 2:10
  SPEAKER_02: And so the idea was we were responsible for the ingestion of data from all of our microservices and from the phone apps that we had deployed, the processing of that into raw events that we could use via Airflow, and then also the Airflow instance that ran the models created by our data analysts and that kind of thing.
• 2:30
  SPEAKER_02: Um, and then also we had a tangential responsibility for our Looker instance as well, which is running all of our kind of BI visualizations and that kind of thing.
• 2:37
  SPEAKER_02: So we're very much, it was very much the pipeline end-to-end, but we were very clear that we were trying to be responsible for giving our data analysts a platform to run models on rather than being responsible for the content of those models specifically.
• 2:48
  SPEAKER_02: So that was very much where we were situated.
• 2:50
  SPEAKER_02: So Airflow for us very much sat right in the middle of that.
• 2:54
  SPEAKER_02: So we had raw tables at one end, and then our analysts would create DBT models that would transform those raw events into various derived data products.
• 3:01
  SPEAKER_02: And then Airflow was really the workhorse that was scheduling all of the compute we needed.
• 3:06
  SPEAKER_02: So we very much used Airflow as a scheduling tool.
• 3:09
  SPEAKER_02: So we weren't running big compute intensive computation inside of it, but we were very much using it to schedule that, to schedule computation within BigQuery mostly.
• 3:18
  SPEAKER_02: We had a couple of like custom Python workloads that we ran that did actually run inside of Airflow itself in some sense.
• 3:25
  SPEAKER_02: But most of it was outsourced to BigQuery and was, and so we were using it as a scheduling mechanism essentially for those things.
• 3:32
  SPEAKER_00: Okay, that's interesting.
• 3:33
  SPEAKER_00: I actually did you try to run things within Airflow, or did you just wanted to use it as a scheduler?
• 3:42
  SPEAKER_02: So oh so it was quite an interesting kind of philosophical question almost that we tried to answer.
• 3:52
  SPEAKER_02: I remember there being this kind of interesting point where this came up in one of the panels, and someone was like, oh, maybe it should just be a compute platform, maybe it should be just a sketchy platform.
• 4:01
  SPEAKER_02: It's interesting.
• 4:02
  SPEAKER_02: I think the big I think a lot of that came from the history of where the data warehouse at Monzo came from originally.
• 4:08
  SPEAKER_02: So it started out as it started out in the kind of pre-DBT days as like hand stitched together SQL that existed.
• 4:15
  SPEAKER_02: And so in in most cases, because the data models were all expressed as SQL models, it made sense for them to just be things that were running BigQuery.
• 4:22
  SPEAKER_02: There was no real there wasn't a lot of heavy-duty Python processing or any of the kind of things you should you find in some other data pipelines.
• 4:29
  SPEAKER_02: So for us, that kind of made sense because of the kind of things we were processing.
• 4:32
  SPEAKER_02: But I can see in a world where if if your lingua franca is Python, if you're running machine learning models or whatever else, then I can see why you might want to schedule that within Airflow itself.
• 4:41
  SPEAKER_02: But I think it's interesting that Airflow allows you to take both views and isn't that opinionated on the way you want to go with it.
• 4:48
  SPEAKER_02: So I think so.
• 4:49
  SPEAKER_02: For us, it made sense for it to be a schedule for it to be just a scheduler, but that mostly is a sort of historical thing about where our models lived and the kind of models that our analysts wanted to build.
• 4:59
  SPEAKER_00: So that's a very good point.
• 5:00
  SPEAKER_00: Uh that's a very great point.
• 5:02
  SPEAKER_00: I I actually at the beginning I was extremely like I was very opinionated on the fact that you should not use Airflow as a way to uh run your task, but really use it as an orchestrator.
• 5:11
  SPEAKER_00: I I think I changed my opinion a little bit.
• 5:14
  SPEAKER_00: As long as you know what you are doing.
• 5:15
  SPEAKER_00: To your point, Airflow has this flexibility to offer you both ways.
• 5:19
  SPEAKER_00: Uh-huh.
• 5:23
  SPEAKER_02: I think it was great.
• 5:24
  SPEAKER_02: Um sorry.
• 5:25
  SPEAKER_02: No, yeah.
• 5:26
  SPEAKER_02: I think it's an interesting I think it is interesting because I think one of the problems that starts to arise is you then start having to use Airflow to consider things.
• 5:34
  SPEAKER_02: In our case, we are running a top community, so we're running our own kind of installation of it.
• 5:38
  SPEAKER_02: So you then have to start, you if you then start to use it as a compute, use it to run compute directly within Airflow, you then have to start considering things like, oh, do I want to, do I need my analyst to understand that you know this particular task needs resources for some reason, this particular one needs GPUs.
• 5:55
  SPEAKER_02: So you just end up with a much more complicated, or you can end up with a much more complicated kind of abstraction at the top level.
• 6:00
  SPEAKER_02: But it also depends upon, but it it depends upon so many dynamics, including things like your analyst level experience.
• 6:05
  SPEAKER_02: Like some of them under some and I some of them understand all this stuff, and other people are just, I wrote some Python, just run it for me, I don't care.
• 6:12
  SPEAKER_02: Like it's on a continuum somewhere, right?
• 6:14
  SPEAKER_00: Yeah, that's true.
• 6:15
  SPEAKER_00: That's very true.
• 6:16
  SPEAKER_00: Great.
• 6:16
  SPEAKER_00: So I guess like you you said a little bit about that, but what does your team use Airflow for when you were at Monzo Bank?
• 6:24
  SPEAKER_00: Like maybe if you have one use case in mind, because I imagine that you were using Airflow for many different use cases, but if you have one in mind.
• 6:31
  SPEAKER_00: Yes, and with some numbers as well, just to Oh, okay, yeah, yeah, no, Taylor.
• 6:36
  SPEAKER_02: So the big use case that we had for Airflow was our nightly DAG that we ran.
• 6:40
  SPEAKER_02: So this is a DAG that runs every 24 hours, it kicks off at one in the morning, and the idea, well the ideal, was that it would be finished processing the sort of eight or nine most significant models that we had by nine o'clock in the morning, so that it was ready for analysts to come in and do their work at nine.
• 6:57
  SPEAKER_02: That was the idea.
• 6:58
  SPEAKER_02: So these were models like the things that banks care a lot about, transactions, accounts, kind of stuff.
• 7:03
  SPEAKER_02: So these are and these are big models.
• 7:04
  SPEAKER_02: And we're talking about, and and so they're big models themselves, but also we're talking about a DAG with 8,000 or so tasks in it.
• 7:11
  SPEAKER_02: It's probably more now.
• 7:12
  SPEAKER_02: It was about 8,000 when I left, it's probably more now.
• 7:14
  SPEAKER_02: But so you're talking about something quite big, uh, essentially to work.
• 7:17
  SPEAKER_02: So a lot of it was so that was one use case was just doing the number crunching of working out what do all our accounts look like, what balances do people have, all that kind of stuff.
• 7:25
  SPEAKER_02: But then there were also other use cases in there.
• 7:27
  SPEAKER_02: So doing things like creating feature sets for our machine learning classifiers was another thing that we used it for.
• 7:33
  SPEAKER_02: Um and also even things like credit scoring, like you're you're building probabilistic scoring models and then using that.
• 7:40
  SPEAKER_02: Even though everything was mediated around these big SQL models, that what they were actually used and pushed into in many cases was was quite different.
• 7:47
  SPEAKER_02: The big and and to be honest, oftentimes the biggest consumer of that was our look at BI instance, essentially.
• 7:53
  SPEAKER_02: So we're trying to create models that you can therefore give insight to non-technical stakeholders.
• 7:58
  SPEAKER_02: So a lot of our biggest dashboards uh were being served directly to leadership and that kind of thing.
• 8:03
  SPEAKER_02: So there were some models, particularly around things like liquidity, that are really important that you get right, because obviously if you're a bank and you're not solvent, that's not particularly great.
• 8:12
  SPEAKER_02: There's a sort of there was there were interesting constraints on certain models around correctness and that kind of thing.
• 8:18
  SPEAKER_02: I guess the other big constraint or the other big use case for us was regulatory reporting.
• 8:21
  SPEAKER_02: So it's a big thing you've got, again, with being a bank, a big thing you've got to get right.
• 8:24
  SPEAKER_02: A lot of the regulators have uh very particular formats that they want data in and very particular kinds of data.
• 8:29
  SPEAKER_02: And they also have very specific restrictions about if you say you're being audited, if they come in and say you're being audited on a specific day, you have to have the data to them by a specific time on that day.
• 8:40
  SPEAKER_02: So it's a very specific format.
• 8:42
  SPEAKER_02: And it has to be because that's what you've got to be regulated in these industries, that makes sense.
• 8:46
  SPEAKER_02: But so that was another big deal for us with making sure that when our pipelines are running, we were getting the right data.
• 8:51
  SPEAKER_02: So if an audit were to happen at very short notice, it would not be a problem to compute the data required.
• 8:56
  SPEAKER_02: So those are a couple of the use cases that we had in mind for it.
• 8:59
  SPEAKER_00: This is awesome.
• 9:00
  SPEAKER_00: So I guess like it is safe to say that Airfo was pretty critical for us, for you.
• 9:04
  SPEAKER_02: Definitely, yeah, definitely.
• 9:06
  SPEAKER_02: It was something, especially in the world of regulatory reporting, where the penalties for getting that wrong were very high.
• 9:11
  SPEAKER_02: It was the kind of thing where if you were being paged about it at six in the morning or even earlier, you didn't you weren't like, I'm just gonna put my phone down, that's fine.
• 9:17
  SPEAKER_02: You know, you were on your laptop at whatever time in the morning to get fixed.
• 9:21
  SPEAKER_02: It was very important stuff.
• 9:23
  SPEAKER_00: Okay, yeah, I see.
• 9:24
  SPEAKER_00: So I guess when you are running Airflow at that scale, you mentioned that you had to run thousands of tasks.
• 9:30
  SPEAKER_00: And by the way, you said models.
• 9:32
  SPEAKER_00: Are you talking about DBT models or just like SQL?
• 9:35
  SPEAKER_02: Yeah, so yeah, so this is important, this is a bit of an important nuance to draw out because there's there was a bit of a difference in it so what so the way that it worked for us was that our analysts were working on DBT models in general.
• 9:45
  SPEAKER_02: As I said, there were some like bizarrely specific Python models that some people built, but 99% of the time we were working with the SQL models.
• 9:54
  SPEAKER_02: But what we did, we pre-invented a much worse version of that, like back in the past.
• 9:59
  SPEAKER_02: So we obviously you guys have done it better than us, but we pre-invented our own version of it.
• 10:03
  SPEAKER_02: The idea was that in CI, we would take the take all of the model definitions, generate all of the SQL for them, and then that's what Airflow would actually end up running.
• 10:11
  SPEAKER_02: And you'd also create a you'd also create from that a definition of the DAG that Airflow could consume.
• 10:16
  SPEAKER_02: So that's the kind of the mapping bit that's interesting.
• 10:18
  SPEAKER_02: So 97% of the time when I say task and model, they're interchangeable, but there is that weird little subtlety about the Python models, so it's not quite as precise as I might like it to be.
• 10:27
  SPEAKER_00: Okay, okay, yeah, okay, thank you for the distinction.
• 10:30
  SPEAKER_00: So I guess like just to cycle back on the number of tasks that you were running, did you did you have any issues like running so many tasks?
• 10:40
  SPEAKER_02: Yeah, so there's a whole there was a whole host of issues that that threw up, as you can imagine.
• 10:44
  SPEAKER_02: So from the point from the perspective of, as we've already alluded to in the introduction, the perspective of certain components of FO not scaling particularly well for that size of DAG in particular.
• 10:55
  SPEAKER_02: This is this is also to say that we weren't just running a one single DAG.
• 10:58
  SPEAKER_02: There were multiple DAGs, but the nightly one being the largest was the biggest problem, that the biggest problem that we had.
• 11:04
  SPEAKER_02: And so there was that kind of technical thing, but it also there was also a kind of more conceptual problem in the sense that so much of monitoring airflow is built around the idea that you can use the UI to observe what's going on in your DAG, and also that the reasoning element can be done by a human by inspection.
• 11:22
  SPEAKER_02: There's kind of an this is never written down in any of the after documentation, or it, but it isn't a kind of assumption that gets made as you go along.
• 11:30
  SPEAKER_02: So when we started having issues around DAGs being in inconsistent states, that was hard to track down initially because so much of that isn't a problem when your DAG is only, say, 20 things in size.
• 11:43
  SPEAKER_02: You can easily see if that's consistent or not.
• 11:46
  SPEAKER_02: But for us, it was much harder to track that kind of thing down.
• 11:49
  SPEAKER_02: And so we had to start thinking about how do we convert some of these kind of human insights into metrics rather than thinking about them from kind of insight levels.
• 11:56
  SPEAKER_02: Yeah, there were multiple different issues that that operating at that kind of a scale started to started to give us.
• 12:02
  SPEAKER_00: And I guess it is interesting because you just mentioned consistency.
• 12:05
  SPEAKER_00: Can you define what you what do we mean by that consistency?
• 12:08
  SPEAKER_02: Yeah, so if we think about so that consistency comes from the idea of task consistency.
• 12:13
  SPEAKER_02: So that's the the place to start.
• 12:15
  SPEAKER_02: Obviously, in the FO documentation, we talk about tasks, and obviously we have the nice state diagram that says tasks move from this state to this state, and that's how they do it, and that's what it means, and all that kind of stuff.
• 12:24
  SPEAKER_02: Obviously, for an individual task to be in a particular state, all of its upstreams should be in also in a specific state.
• 12:31
  SPEAKER_02: For example, if you're it it's not it should be impossible for a task to be in a successful state if a task upstream of it is in a waiting state, because obviously it doesn't make any sense.
• 12:41
  SPEAKER_02: How could how could it be successful if it had been if something is if it's still waiting for something?
• 12:45
  SPEAKER_02: Because the one that was interesting to us was if a task is in an upstream failed state, then it must have an upstream that has failed.
• 12:54
  SPEAKER_02: That's again, by definition, is obvious.
• 12:58
  SPEAKER_02: What what we meant by DAG consistency is to say that every task in the DAG is in a consistent state, by which we mean that the state the task is in is an accurate reflection of the upstreams that it has, and the downstreams, most of the upstreams.
• 13:11
  SPEAKER_02: And so what that so the way that manifested for us is that what we would start to find is we would occasionally end up with the throughput in the DAG going down to zero, so nothing was executing, because we had tasks that were we had tasks that were in an upstream failed state, but nothing upstream had actually failed.
• 13:30
  SPEAKER_02: And so this was and this was really hard to detect because from the DAG run's point of view, everything is fine.
• 13:37
  SPEAKER_02: Like it's still executing, nothing is raw in some sense.
• 13:41
  SPEAKER_02: But from our point of view, throughput's gone down to zero, no, nothing is executing, and the DAG's effect the DAG runs effectively stuck, like it can't do anything because it can't resolve this consistent the it can't resolve this inconsistency because a the DAG shouldn't be allowed to get into that state, and yet seemingly it does.
• 13:59
  SPEAKER_02: So that's those are the kind of problems that we started to encounter.
• 14:03
  SPEAKER_00: Wow.
• 14:03
  SPEAKER_00: Okay, that is very interesting.
• 14:05
  SPEAKER_00: So I guess I have some so many questions.
• 14:07
  SPEAKER_00: So the first thing is the first question that I have to you is how did you find out that you were having those issues?
• 14:15
  SPEAKER_00: Because to your point, like when you have like thousands of tasks, and I guess like maybe looking at the UI might not be easy, right?
• 14:22
  SPEAKER_02: I mean, yeah, that was definitely the case.
• 14:24
  SPEAKER_02: So a lot of it there's kind of a flippant answer to this, and then there's a more serious one.
• 14:28
  SPEAKER_02: The flippant answer is we kept being pulled into incidents, but this kept being a problem because especially because we had a kind of implicit time limit where it's like if this stuff's done after nine o'clock, that's a real issue.
• 14:38
  SPEAKER_02: As soon as we started getting to the point where where mod where you'd get to say like 10, 30, 11, and nothing had happened for a significant period of time, that was where then you'd start to go, you'd start to think inspect the DAC yourself and be like, oh, hold on a minute, this doesn't really make any sense.
• 14:53
  SPEAKER_02: But you're very much applying kind of human intuition on top of the data model that Airflow gives you, essentially.
• 15:01
  SPEAKER_02: So the way that we started to effectively see it was we get pulled into these incidents, and then in these incidents, you'd go and start to look at the set of upstream, the set of failed tasks, and then you'd start to go look at their upstreams and compare that to then what Airflow is telling you as an upstream failed state.
• 15:16
  SPEAKER_02: And then the sets would be disjoint and you'd be like, hold on, that doesn't make any sense.
• 15:20
  SPEAKER_02: How can it be an upstream how can it be an upstream failed state if it's not upstream of a failed thing?
• 15:25
  SPEAKER_02: That doesn't make any sense at all.
• 15:26
  SPEAKER_02: And so for a while we we dealt with it manually, basically.
• 15:29
  SPEAKER_02: We would find these in the inconsistent ones and clear the task with downstreams, and that would start to resolve the problem because you could it was relative because that was relatively easy to do, and it doesn't require a huge amount of kind of intelligence, but this happened quite a lot.
• 15:43
  SPEAKER_02: There was a point where it was happening, I think it was happening nearly every day that we ran the DAG, and so this was getting to be just ridiculous, just got to be ridiculous after a while.
• 15:50
  SPEAKER_02: We then started to look at it and think, hold on a minute, can we is there a way we could automate any of this?
• 15:57
  SPEAKER_02: Could we start to build something that would actually solve this for us?
• 16:00
  SPEAKER_02: Because and also I think to probably address it, to address at this point, what we were seeing was what we were seeing as far as we could tell is some problem with the way that Airflow does task state updates.
• 16:12
  SPEAKER_02: And so we think, although this isn't based on a huge amount of data because it was very hard to capture exactly what was going on, but we were pretty sure that what was happening was Airflow, when Airflow was trying to clear a lot of tasks at the same time, the order of events would be something like a task in the DAG failed quite early on and it's got a lot of downstreams.
• 16:32
  SPEAKER_02: So when Airflow is trying to is trying to set the state of all of those downstreams to be cleared, especially if one of those downstreams is also downstream of something else in that same set, you can end up in a sort of lost update problem where instead of it being upstream failed, upstream failed, cleared, it ends up being upstream failed, cleared, upstream failed.
• 16:54
  SPEAKER_02: And obviously that's incorrect.
• 16:56
  SPEAKER_02: So what we so what we obviously that's an issue with airflow, but because of the fact that we could, it was very hard to get the bug to reproduce in any sensible way.
• 17:06
  SPEAKER_02: We couldn't create a minimal example.
• 17:08
  SPEAKER_02: It was the proposition of going to the airflow community and being like, hey, we have this really ill-defined thing that we're not really sure if it's a problem or not.
• 17:16
  SPEAKER_02: It was a very hard sales.
• 17:17
  SPEAKER_02: We were like, okay, let's see what we can do in the shorter term to fix this.
• 17:20
  SPEAKER_02: So we built something that would that could do this consistency scanning for us.
• 17:24
  SPEAKER_02: Because all we were doing is graph algorithms.
• 17:26
  SPEAKER_02: And this is computer science 101.
• 17:28
  SPEAKER_02: This is back in the day, easy stuff.
• 17:30
  SPEAKER_02: We built we built a service that could do that and initially started it in just reporting mode to just tell us if there's a consistency issue.
• 17:37
  SPEAKER_02: And then we moved on to actually compute a solution and then and run the solution through properly.
• 17:43
  SPEAKER_02: So those were the those are the the steps were initially let's initially do it manually a couple of times, and then go on from there.
• 17:52
  SPEAKER_02: How can we build in a kind of automated process to to make this more consistent?
• 17:56
  SPEAKER_02: Obviously, the real solution would be let's go fix this properly in Airflow, but a lot of our but it was very much a case of we need a solution to this quickly, and also it was a case of we and and also at the time there were issues, not anymore, fortunately, but there were issues with us being stuck on older versions of Airflow for bots for the owner related reasons.
• 18:13
  SPEAKER_02: So it was like even if we do get a fix in the upstream, we can't even necessarily take it at this point because of all these other issues.
• 18:19
  SPEAKER_02: So it was very much a kind of like, how do we build something that can solve this problem for us?
• 18:25
  SPEAKER_02: We don't have to make big changes to Airflow at the current time, but that's the route we took.
• 18:30
  SPEAKER_00: Okay.
• 18:31
  SPEAKER_02: So I guess were you actually able to ultimately uh fix that issue, or you had to um So we built a s so we built a solution that worked, and in my last couple of months it saved uh I think it saved us about 11 or 12 incidents, I think, over the course of time.
• 18:46
  SPEAKER_02: So we definitely built something that worked, it could resolve the problem.
• 18:49
  SPEAKER_02: We we had quote so one of the so one of the reasons we thought one of the reasons we thought we might have been triggering this was because in the version of Airflow we were stuck on, which I think was 234 type.
• 19:02
  SPEAKER_02: Okay, the clear task API endpoint didn't have a with downstreams option.
• 19:06
  SPEAKER_02: It was added a bit later in.
• 19:08
  SPEAKER_02: We had to, we we built, we built the general pattern at Monzo was that you would build a service that ran in our cluster that fronted your API so that you didn't have to keep connected to the API, which so it finds in our service, we built a bit of extra intelligence that would that would pretend the clear with downstreams option existed, but actually that was all implemented in the service itself.
• 19:25
  SPEAKER_02: So we hoped that what when we upgraded and we got the clear with downstreams, that would start to solve this problem to some extent.
• 19:32
  SPEAKER_02: Because if you could make the if you could make the task clearing process atomic, so you basically say, okay, all these tasks clear at the same time, and the state is everything after the fact, rather than this kind of piecemeal process.
• 19:46
  SPEAKER_02: We can't we thought that would help.
• 19:47
  SPEAKER_02: However, we upgraded to latest Airflow 2 and we still didn't, we still saw this problem occurring.
• 19:52
  SPEAKER_02: There's clearly something else.
• 19:53
  SPEAKER_02: Whether I don't as I say, I've I've since left, but whether they've upgraded to Airflow 3 since then, I don't yet know.
• 19:59
  SPEAKER_02: It'll be interesting to see if that solves the issue as well.
• 20:02
  SPEAKER_02: But I think there's but I think it might be something.
• 20:05
  SPEAKER_02: The problem is it's also hard to know exactly where to where the problem might be, because it may well be really deep down in the database drivers level.
• 20:13
  SPEAKER_02: So it might even be not an Airflow issue at all, be something to do with the database libraries.
• 20:17
  SPEAKER_02: So it's something that it's something that it's very likely that something else will fix over the course of time, but obviously you can't, but it's hard because it's hard to identify exactly the root cause, very hard to fix in that specific way.
• 20:31
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 20:40
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 20:52
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astonomer.io.
• 21:01
  SPEAKER_00: Actually, I was a data engineer in Bank in France.
• 21:04
  SPEAKER_00: And I know what it is when you have this issue that you can that you can't reproduce for some reason, and it's just a nightmare.
• 21:11
  SPEAKER_00: Because how do you know it?
• 21:12
  SPEAKER_02: Yeah, how do you know exactly the root cause if you can't reproduce that error and it happens like exactly when it's so public as well, because that was our other kind of big thing, was like every all of these people, especially big execs, are looking at these data models and they're being like, why aren't they refreshing?
• 21:26
  SPEAKER_02: And it ultimately comes down when it comes down to or it feels comes down to your team, it's very difficult to be like, we know what the problem is, but we also don't it's a very frustrating thing to not be able to give them an answer of yes, we know we're doing our work on it.
• 21:38
  SPEAKER_02: Even on a sociological level, like it's so much more frustrating than you than just a technical problem.
• 21:44
  SPEAKER_00: Yeah, I imagine that many engineers uh listening to us right now will uh feel will feel you.
• 21:48
  SPEAKER_02: Yes, I'm sure that yeah, I'm sure there will there'll be a huge amount of sympathy, I'm sure.
• 21:52
  SPEAKER_00: So I guess like you mentioned that you build that you built that uh external system to monitor that consistency.
• 21:58
  SPEAKER_00: Uh maybe can you just maybe Explain us a little bit more what it is, like what you actually do.
• 22:03
  SPEAKER_02: Yeah, of course.
• 22:04
  SPEAKER_02: Yeah.
• 22:04
  SPEAKER_02: So as I said before, because we effectively built the DAG in CI before, that made it a lot easier because the structure of the DAG, so the non-dynamic part, was very easy.
• 22:14
  SPEAKER_02: We could get that.
• 22:15
  SPEAKER_02: We actually stored it in BigQuery as a kind of big list of big list of nodes and arcs.
• 22:19
  SPEAKER_02: So that's very easy.
• 22:21
  SPEAKER_02: Then so what we would then do is we would take that static structure, and then you would make a few requests to the FO API to ask it for various different things.
• 22:28
  SPEAKER_02: So you basically say, what tasks do you think have failed?
• 22:32
  SPEAKER_02: So that's one set.
• 22:33
  SPEAKER_02: And then what tasks do you think are in upstream failed?
• 22:36
  SPEAKER_02: And that's the second set.
• 22:37
  SPEAKER_02: And then essentially you would take each of those upstream failed tasks and work up, work back through the up work back through the DAG and basically say, if I start from here, can I eventually find something that's failed upstream before I hit the route?
• 22:52
  SPEAKER_02: And work through that for each of the for each of those upstreams.
• 22:54
  SPEAKER_02: And if you find one where you can't, then you flag the DAG as inconsistent.
• 22:58
  SPEAKER_02: In reality, you check all of them and then you give it a whole list rather than failing early, but that's the idea.
• 23:03
  SPEAKER_02: So you're basically taking the kind of static information from that we'd already recorded, and then using Airflow to layer the dynamic information on top of that, and that's how you ultimately make the decision.
• 23:14
  SPEAKER_02: Obviously, you could pull the whole DAG structure out of Airflow immediately.
• 23:17
  SPEAKER_02: You could do it all for the FO API, but for various reasons, uh, including the fact that some of our tasks were like some of our tasks were very big because they had these big JSON payloads attached to them for reasons that are reasons for reasons that are very complicated.
• 23:31
  SPEAKER_02: But basically it was so that was putting an awful lot of strain on on the API itself.
• 23:35
  SPEAKER_02: We decided not to go down that route and use this kind of pre-done version, effectively.
• 23:40
  SPEAKER_02: But yeah, but the important thing was going to Airflow for the dynamic information, because obviously information you can only get from Airflow.
• 23:45
  SPEAKER_02: And then once you've done that and you've worked out the you've worked out where the inconsistency lies, you essentially then you perform a slightly different process where you work downwards and work out what's the minimal number of tasks I need to clear in order to get all of this stuff cleared.
• 23:59
  SPEAKER_02: Because obviously, if you can work, if you can get to something quite high up a DAG's got a lot of downstreams, the likelihood is you're gonna have to issue fewer API calls overall in order to do that.
• 24:08
  SPEAKER_02: That's the kind of that was the kind of um system that we ended up with.
• 24:12
  SPEAKER_02: And the, as I say, the first version of that, it basically just reported into Slack and just said, hey, the DAGs in an inconsistent state, clear these tasks in this order and you'll be fine.
• 24:20
  SPEAKER_02: And then there was a kind of unattended mode, which we turned it into later on, which basically said, okay, here is, okay, that list I told you about, I'm going to now do that in this order.
• 24:32
  SPEAKER_02: And that would just work unattended.
• 24:33
  SPEAKER_02: Because the other thing that made this kind of more entertaining was that, well, entertaining is the wrong word, but kind of frustrating is probably the best way of describing it, is that this stuff would always happen at four in the morning or some kind of really on.
• 24:45
  SPEAKER_02: Yeah, but because because Murphy's law, right?
• 24:47
  SPEAKER_02: If it's gonna happen, it's gonna be the worst possible time.
• 24:49
  SPEAKER_02: So it's so it so it was so it was the kind of thing where like any solution had to be either any solution basically had to be automated because otherwise you'd end up with you'd end up with these huge problems.
• 25:00
  SPEAKER_02: So yeah.
• 25:00
  SPEAKER_02: So it started off as a kind of as just a reporting mechanism and then moved on to actually then try to fix the problem from there.
• 25:06
  SPEAKER_02: So that was the sort of rough kind of order of operations it took.
• 25:11
  SPEAKER_00: This is super smart.
• 25:12
  SPEAKER_00: Obviously, like I'm a huge, I'm a huge fan of automating everything we can because ultimately that that's the way that's the way to avoid becoming crazy, especially when we have heroes.
• 25:21
  SPEAKER_00: Yes, especially when you have like heroes at 4 a.m.
• 25:24
  SPEAKER_00: in the morning.
• 25:25
  SPEAKER_00: Great.
• 25:26
  SPEAKER_00: So I guess like moving on to again, like actually coming back to some scaling issues.
• 25:30
  SPEAKER_00: So we talked about the DAC consistency, but obviously the UI is also uh very important.
• 25:34
  SPEAKER_00: And again, we don't think about the UI that much.
• 25:36
  SPEAKER_00: We always think about hey, uh am I gonna have enough like CPUs and memory to scale Airflow?
• 25:43
  SPEAKER_00: Do I have the budget for that?
• 25:45
  SPEAKER_00: And so on.
• 25:45
  SPEAKER_00: But we don't think about truly like the UI.
• 25:48
  SPEAKER_00: Am I gonna be able to use the UI with the same way when I have two tasks than when I have thousands of tasks?
• 25:55
  SPEAKER_00: So to your point, actually, based on your experience, were you still able to use the UI to go to the graph view and then look at the tasks?
• 26:04
  SPEAKER_00: Or did you have to build something on top of it?
• 26:07
  SPEAKER_00: Or what was your experience with it?
• 26:09
  SPEAKER_02: Yeah.
• 26:10
  SPEAKER_02: So the U So the UI for the most part was pretty reasonable, up to about the thousand tasks or so.
• 26:17
  SPEAKER_02: That was, I think, where the kind of limit started to show.
• 26:19
  SPEAKER_02: So and so our big problems were things like the some of them were really minor things, like the fact that the default number of runs when you click on a DAG is 20, is a real pain for some when you have 8,000 tasks, because obviously that's maths are terrible, but that's 16 and there's 160,000 tasks it's trying to load up in this one kind of UI page.
• 26:39
  SPEAKER_02: So that's a real pain because we had to have this, we had this like special run book.
• 26:43
  SPEAKER_02: There's if you want to look at this, make sure you use this very specific URL, otherwise the web server might just crash because it's too big a thing to look at.
• 26:51
  SPEAKER_02: So so that's something so that's something that that we had to do.
• 26:54
  SPEAKER_02: The other thing were the the other, I mean, there were more problems when we were on a single instance Airflow cluster because if somebody took down the web server, they took down the executor as well, which obviously was, you know, that's bad, but that's an architectural problem, which is something else.
• 27:07
  SPEAKER_02: But you know, that that's that's a bit different.
• 27:09
  SPEAKER_02: But the other big problem we had with the UI was that it's very hard in the UI to get across the scale and or cost of what you're about to do.
• 27:18
  SPEAKER_02: So we had these incidents where people would like they mixed up upstream and downstream, so they cleared a model with upstreams, uh noise upstreams rather than all the downstreams.
• 27:26
  SPEAKER_02: And so this is so when you yeah, yeah, really that was a very expensive day for many reasons.
• 27:30
  SPEAKER_02: But but when when you have say 4,000 tasks upstream of yours because you happen to be downstream of whatever big model that you just want a bit of, for example, that's fine.
• 27:40
  SPEAKER_02: But it's very difficult in the UI to communicate that's what you're gonna do.
• 27:44
  SPEAKER_02: It says I'm gonna clear all these things, but I don't think that but it's very hard to to communicate sometimes either the cost in terms of time or the cost in terms of actual money that you're gonna spend doing certain things.
• 27:54
  SPEAKER_02: And obviously, when you're at a scale of three or four tasks, it's irrelevant.
• 27:57
  SPEAKER_02: If you make a mistake, it'll only be a few minutes for that sometimes.
• 28:00
  SPEAKER_02: But when it was when it in this particular one I'm thinking of, I think that took us like seven or eight hours to recover from.
• 28:06
  SPEAKER_02: And the person, obviously, the person who did it was very apologetic, which is it's not really it's not really their fault.
• 28:10
  SPEAKER_02: They made an honest mistake, but the UI should have had better guardrails to be like, hold on a minute, this seems this seems very odd what you're trying to do.
• 28:17
  SPEAKER_02: Are you really sure that's what you want to do?
• 28:19
  SPEAKER_02: Because that would have been helpful.
• 28:20
  SPEAKER_02: And so this so actually on that similar point, we actually had to we actually built in some we first thing they had to do was we had to start restricting access to the FL UI for one, but we also started to build in extra um extra approval levels, basically.
• 28:35
  SPEAKER_02: So somebody else had to approve what you were trying to do.
• 28:37
  SPEAKER_02: And we built that into a system that Monzo already had, which is called BizOps.
• 28:41
  SPEAKER_02: So they and that has a very kind of extensible interface for if you approve things, then you can make certain API calls so it makes sense.
• 28:47
  SPEAKER_02: But it was really frustrating for a lot of the data analysts coming in when we that when they were like, Oh, I've used Airflow at all these other companies, like, how do I get access to the UI?
• 28:55
  SPEAKER_02: And we were like, Oh, sorry, you can't because it's too dangerous.
• 28:59
  SPEAKER_02: And that's and even so even with even with reduced permissions models being what they are makes sense.
• 29:06
  SPEAKER_02: But it was but it was almost that they weren't granular enough for what we wanted to do because you almost wanted it to be within almost within DAGs rather than just like at the kind of whole DAG level and all that kind of stuff.
• 29:16
  SPEAKER_02: So it's so so yeah, there were some problems around just like the amount of data the UI is trying to show you, and also because of the fundamental issue that the human's ability to visually reason about more than about 50 things is probably is just is for the birds, basically.
• 29:30
  SPEAKER_02: You're not if if you have more than about a hundred things on a screen, it's very difficult to reason about any of it.
• 29:35
  SPEAKER_02: So you instantly start trying to cut it down.
• 29:38
  SPEAKER_02: And actually, to be fair, in the newer versions of Airflow, that was made a lot easier.
• 29:43
  SPEAKER_02: But being able to but then when you're at the when you were at the level of RT where we were saying, like, how is the whole thing running?
• 29:49
  SPEAKER_02: That's very difficult to get to.
• 29:50
  SPEAKER_02: So that was where, as I was saying before, we had to start moving into the world of what did these visual indications and human intuitions mean in metric terms.
• 29:59
  SPEAKER_02: And that's that was the big leap we had to make.
• 30:00
  SPEAKER_02: But yeah, so we had to yeah, so a lot of the problems we have with with the Airflu UI were basically just around the idea that it's not very useful at such large scale, and also that were and also it just was potentially quite dangerous for a lot of things that you could do with it.
• 30:15
  SPEAKER_02: So those are one of the big the bigger issues.
• 30:17
  SPEAKER_00: That that is super this is super interesting because that's actually the first time that I hear that.
• 30:22
  SPEAKER_00: And with with practical examples.
• 30:25
  SPEAKER_00: And do you have an example of what kind of permissions within a DAG you are thinking?
• 30:32
  SPEAKER_00: Like, because you said that sometimes it's not granular enough.
• 30:35
  SPEAKER_00: What kind of I think limiting the number of tasks that you want to clear, for example, for a specific DAG?
• 30:42
  SPEAKER_02: Yeah, or even some, or even like some level of some level of approval.
• 30:46
  SPEAKER_02: If you want to clear more than say 50 tasks or something, then it's hold on, you need X person to ping you for approval.
• 30:51
  SPEAKER_02: Because lots of products have those kind of mechanisms built in nowadays.
• 30:55
  SPEAKER_02: Things like Falanis, you can do that with Slack integrations, geo integrations, whatever you want.
• 30:58
  SPEAKER_02: It doesn't have to be particularly heavy-handed, but just something that says something that enables you to because it's all that thing about when you start restricting access to stuff, people often the criticism you get is, oh, do you not trust us?
• 31:10
  SPEAKER_02: And I think there is and I think and I think that's a very valid criticism to make.
• 31:13
  SPEAKER_02: I think it's perfectly valid, but it's always about it's always the kind of trust and verify thing, isn't it?
• 31:17
  SPEAKER_02: Is we do trust you, yes.
• 31:18
  SPEAKER_02: But if you're gonna do this, we really want to make sure you've got a good rationale for what you're doing, because that would have prevented so many issues previously.
• 31:25
  SPEAKER_02: And I think that's the kind of view that we came to.
• 31:27
  SPEAKER_02: But I think it's I think it's more the kind of all or nothing nature of the airflow permissions you have.
• 31:32
  SPEAKER_02: It's you either have all the permissions to do everything you need, or you don't.
• 31:37
  SPEAKER_02: And there's no way of saying we do want to give you these permissions, but we'd also just quite like to verify that they're actually okay.
• 31:43
  SPEAKER_02: Obviously, for some people it'd be fine to admin, that's totally reasonable.
• 31:46
  SPEAKER_02: But for other people, it's we just want to make sure that what you're doing is actually what you think you're doing, because sometimes that's not immediately apparent from from the commands you can make with like.
• 31:55
  SPEAKER_00: Yeah, no, that that makes a lot of sense.
• 31:57
  SPEAKER_00: Um, and so you mentioned that you you you had to monitor some metrics, right?
• 32:01
  SPEAKER_02: Um Yeah.
• 32:02
  SPEAKER_02: So we yeah, so we so we started to think essentially a lot of this came from the fact that we were struggling.
• 32:08
  SPEAKER_02: People would come to us and say things like, oh, like how healthy is Airflow?
• 32:12
  SPEAKER_02: And we would say it's running.
• 32:13
  SPEAKER_02: That was kind of that was about as about as good as it got.
• 32:16
  SPEAKER_02: And so we really started to think about, okay, what does it mean for Airflow to be in a good state?
• 32:20
  SPEAKER_02: Like, how would we know and how would we know that without having to go through this sort of folkloric, staring at the DAG, putting your finger in the air, and could be like, oh, it's about this?
• 32:28
  SPEAKER_02: How could we get away from that?
• 32:29
  SPEAKER_02: And so we started to think about defining metrics, and then therefore you could define monitors on the back of it.
• 32:33
  SPEAKER_02: And so for us, like throughput was the really big, was the really key one for us.
• 32:38
  SPEAKER_02: Like how many tasks are completing and and what's that been like over the last kind of while?
• 32:45
  SPEAKER_02: Because we often because the thing that we the thing that we found was quite hard was that often a lot of the underlying details of Airflow, or at least for us, were resigned in other systems.
• 32:56
  SPEAKER_02: Things like the the Kubernetes counter had its own set of metrics, BigQuery itself with a whole load of metrics you can come to.
• 33:02
  SPEAKER_02: So there was a lot of kind of incidental things, but the big question we were asking was how what rate are tasks completing and has that hit has that hit zero?
• 33:11
  SPEAKER_02: Is that a real problem?
• 33:12
  SPEAKER_02: Because if the DAG is still processing, that's the starting point.
• 33:15
  SPEAKER_02: And then for further refinements on that, you start to look into things like utilization, particularly of the underlying compute resources.
• 33:23
  SPEAKER_02: So for us, that was like reservations in BigQuery was like a big deal.
• 33:26
  SPEAKER_02: Less so on the Kubernetes end of things for us specifically, but I can imagine in other cases that might be true.
• 33:31
  SPEAKER_02: How well are you utilizing all of the things that how are they utilizing all the things you have?
• 33:34
  SPEAKER_02: And between those two metrics was a pretty good starting point for what we were looking for.
• 33:40
  SPEAKER_02: And but I think the big, but I think the big, or at least the interesting thing was it was very hard to find a lot of guidance on that kind of thing.
• 33:46
  SPEAKER_02: I would have been really interested to see like Airflow, the Airflow community coming up with this is what we think a golden path looks like for decent metrics.
• 33:54
  SPEAKER_02: Obviously, that's going to vary depending on a lot of specifics around Airflow.
• 33:58
  SPEAKER_02: But I think it would be really interesting to see what the community would think in terms of these are the metrics you really want to make sure that you're at least having thought about when you're thinking about monitoring, and then you can lay your own platform specific and all that other stuff on top of it.
• 34:10
  SPEAKER_02: Because I think that's because we had to, as I say, and to be fair, this was something that was starting up as I was leaving.
• 34:16
  SPEAKER_02: So it's probably, again, it's probably advanced a lot right now.
• 34:19
  SPEAKER_02: But it was quite difficult to start from not very much because coming from I'm kind of platform engineering background, so with things like humidities, there are an awful lot of very opinionated views about these are the key metrics that you should care about, and that's really useful.
• 34:32
  SPEAKER_02: So yeah, we really need to see what that would come, what that would be like in an Airflow context.
• 34:37
  SPEAKER_02: I'd be interested to see what could what the discussion about that would be like.
• 34:40
  SPEAKER_00: Now that's a super this is a very good point, because I I feel obviously when you begin with Airflow, you tend to think about how I'm gonna build a DAG, how I'm gonna write a task, and so on.
• 34:50
  SPEAKER_00: You don't think that much about monitoring.
• 34:52
  SPEAKER_03: Right?
• 34:53
  SPEAKER_00: Even if it is a crucial part of your work, um, you don't really think about that at first, but then you realize how important it is, and then this is when you realize that you don't have, to your point, that much guidelines on what metrics you should take a look at, even like what system you should set up to monitor Airflow.
• 35:10
  SPEAKER_00: Obviously, you have some rules that you can find on the Airflow documentation, but to your point it could be better, and I think this is a very valid point.
• 35:17
  SPEAKER_02: Because I think, yeah, could good because I think also it's it's a thing of I think it's something to do with kind of audience initially, because a lot of the time with a lot of the time with Airflow, you find these kind of organic installations have grown up over the course of time.
• 35:30
  SPEAKER_02: So it starts off as one person wants to schedule a few models and then become successful, and then it grows out, and then suddenly it becomes then owned by the platform, the by say a platform team.
• 35:39
  SPEAKER_02: And suddenly you then end up with these very different expectations and very different needs essentially from what it is.
• 35:45
  SPEAKER_02: When you're sort of first setting it up, you just want to get it set up, and you just want to run your data and you're not really that interested.
• 35:50
  SPEAKER_02: But when it becomes a, as you say, a critical piece of infrastructure, you actually care about is it running well and all that kind of stuff.
• 35:55
  SPEAKER_02: So I think it's very much a case of as the audience moves, the expectations move with it, but it'll be and and therefore will be interesting to see almost maybe like differentiated documentation about if you're running at scale, this is a different thing.
• 36:07
  SPEAKER_02: So I think there's definitely something to be said for the audience changing and therefore the expectations changing as well.
• 36:13
  SPEAKER_02: And I think that some of FO's history probably is some of the reason why that's the case, because it's because even talking to other people who've run it, you see it running in everything from teams of three people to teams of 3,000 people, and those are very different things.
• 36:27
  SPEAKER_02: But and supposedly it's to FO's credit that it can run at those multiple different scales, but it just brings with it different, it just brings with different expectations around what you're gonna need to do if you want to run it for multiple scales.
• 36:38
  SPEAKER_00: Yeah, absolutely, absolutely.
• 36:39
  SPEAKER_00: And and also like for the audience, you will find some Airflow best practices if you run Airflow in production in the documentation.
• 36:45
  SPEAKER_00: So they are still like a lot of information that you will find useful.
• 36:48
  SPEAKER_00: But yeah, to your point, it's again like very valid.
• 36:51
  SPEAKER_00: Your expectations are not the same when you are a team of three than when you are a team of thousands, at least like hundreds of people.
• 36:57
  SPEAKER_00: So you definitely have to know what you should look at.
• 37:01
  SPEAKER_00: Um I really enjoyed that conversation, uh, but I still have uh a last one question for you, Jonathan.
• 37:08
  SPEAKER_00: Honestly, I have many more questions, but we need to uh just look so looking back at your experience running Airflow with like thousands of tasks, I guess what advice will you give to teams who are starting to eat some scaling issues with their Airflow deployments?
• 37:23
  SPEAKER_00: What will you what like based on your experience, what are your recommendations?
• 37:28
  SPEAKER_02: Yeah, so I think to cap the end of our previous conversation, so metrics are really important.
• 37:34
  SPEAKER_02: Make sure that you make sure that you look, set set them up, go with what you think, track them for a while, and then when you start to have incidents, start to look at the metrics and start to work backwards and think, okay, what metric would I have needed in order to not have had this incident in the first place?
• 37:48
  SPEAKER_02: So that's definitely a kind of build up that operational excellence process and use the metrics to drive that.
• 37:53
  SPEAKER_02: It's a really kind of important thing, I think.
• 37:56
  SPEAKER_02: The other point is I think it's really being clear about it's really being clear about where and why you hit scaling issues fundamentally.
• 38:06
  SPEAKER_02: Like which specific component is it that's causing the issue?
• 38:10
  SPEAKER_02: Because it's very easy to just say, oh, we're having problems with scaling, whatever.
• 38:13
  SPEAKER_02: But be really specific about which part of the installation is the problem, and then start thinking how can we target stuff at this?
• 38:18
  SPEAKER_02: And maybe it's the case that we that there's another tool that you need, maybe it's more metrics, maybe it's practices you need to adopt around the kind of things that I was talking about with approvals and all that kind of stuff.
• 38:27
  SPEAKER_02: It's very much about it's very much about being specific about where the scaling issue is, because very often it will be in a single component or maybe one or two, because most things will generally tend to scale in a kind of reasonable way.
• 38:39
  SPEAKER_02: I think the other thing is I think also the other thing is don't necessarily be afraid to build tools outside of Airflow if you need to.
• 38:46
  SPEAKER_02: Like it's kind of obviously I'm I'm very much a support of open source.
• 38:49
  SPEAKER_02: Definitely go contribute to the upstream because it's a great experience, if nothing else.
• 38:52
  SPEAKER_02: But don't be afraid to use the best tool for the job fundamentally.
• 38:55
  SPEAKER_02: If the best tool for the job is going to get you there quickest is building something external that can solve your problem, then that's fine.
• 39:01
  SPEAKER_02: But if it's if it also is going to contribute to the upstream and that kind of thing, then also go do that because I think they're both important points.
• 39:07
  SPEAKER_02: But I think the final thing I would make point is that when you do hit scaling issues like this, is that you've got to be is that sometimes it's time to look at what you're doing and think, why is this getting as big as it is?
• 39:20
  SPEAKER_02: Because I think fundamentally, I think there are some times where we we had this a couple of times at Monzo where people would come along and say, Oh, well, what if we changed orchestrator?
• 39:26
  SPEAKER_02: What if we use Dagster or what if we use something else?
• 39:29
  SPEAKER_02: And I would constantly say, maybe, but let's focus on the fact that this DAG is really big.
• 39:34
  SPEAKER_02: How could we make it smaller?
• 39:35
  SPEAKER_02: Because then I think because otherwise you end up in the you end up in the kind of, I don't know, the kind of sugar rush of going to a new tool that's shiny and going to fix everything, rather than focusing on the actual fundamental problem, which is that some things are just too big.
• 39:48
  SPEAKER_02: And so foc and so if you have a choice between new tools and root causes and breaking things apart and all that kind of thing, take make sure the main thing is the main thing and split up the things that are big.
• 40:01
  SPEAKER_02: Because of because oftentimes with with scaling, you know what the problem is, you don't necessarily know what the solution is, but you do at least know what the problem is.
• 40:07
  SPEAKER_02: And so I think making sure that when you do hit scaling issues, make sure that you're focusing on the right thing and you're not trying to you're not trying to find some silver bullet somewhere, but you're actually focusing on how could I split this down and make it more composable and all that kind of stuff.
• 40:21
  SPEAKER_02: I don't think you'll go far wrong in that sense.
• 40:24
  SPEAKER_00: Wow.
• 40:25
  SPEAKER_00: I love that.
• 40:26
  SPEAKER_00: Actually, to your last point, I think it's just like human nature.
• 40:29
  SPEAKER_00: We tend to it's very easy for us to add new things, it's very hard to remove things that we've done.
• 40:35
  SPEAKER_00: It's like we we grow up, we don't we go back to small.
• 40:39
  SPEAKER_00: So I guess it's just like natural to do uh things like that, but to your point, um having a new trying to find a new tool, which is very natural again for engineers, we like new things, uh but adding a new tool, um adding new stuff basically, sometimes it's not the solution at all.
• 40:55
  SPEAKER_00: Sometimes it's gonna add more complexity and it's not gonna fix the root cause of your issue.
• 41:01
  SPEAKER_00: So I love what you said that you really should focus on like try to find the root cause and don't try to add new things and hope that it's gonna be better.
• 41:14
  SPEAKER_00: Yeah, and I think it's not gonna be the it's gonna not do it.
• 41:17
  SPEAKER_02: Yeah, and I think sometimes it's about I think sometimes it is about I think sometimes there can be a problem where it's about almost what can be palatable to your leadership in particular, because I think it can be a lot I think personally it can be a lot easier to go to them and say, we're gonna spend this money on this new tool and it's gonna be brilliant and whatever else, and you can show them fancy demos and this kind of thing.
• 41:38
  SPEAKER_02: I think the much harder sell and the much harder skill is to go in and say, hey, actually, if we spent the six months we were gonna spend evaluating this on actually breaking this thing down, then we would have this many fewer incidents.
• 41:50
  SPEAKER_02: I feel like that is, I don't feel like that is necessarily as compelling because ultimately a lot of technical leadership will were also engineers too.
• 41:57
  SPEAKER_02: But I feel like we really as a industry need to get better at selling the second version of that because that's what because that's ultimately how we get scalable, sustainable software, which is what we all want, but we sometimes but I think sometimes we're not necessarily the best sales put people for that kind of thing, and I think that's really important.
• 42:16
  SPEAKER_00: I think we're gonna we're gonna end on this on this on that because it's perfect.
• 42:20
  SPEAKER_00: It's a really nice way to summarize everything.
• 42:23
  SPEAKER_00: Thank you so much for joining.
• 42:25
  SPEAKER_00: Jonathan, it was a pleasure.
• 42:26
  SPEAKER_00: I guess one last question to you.
• 42:28
  SPEAKER_00: Uh, what is the best way to to stay in touch with you?
• 42:31
  SPEAKER_02: Cool, yeah.
• 42:31
  SPEAKER_02: So I'm on X Jonathan P.
• 42:33
  SPEAKER_02: Rayner, it's R-A-I-N-E-R, and you can find me there.
• 42:36
  SPEAKER_02: I'm also on Blue Sky.
• 42:36
  SPEAKER_02: I've just started there as well.
• 42:37
  SPEAKER_02: So you can find me Jonathan Rayner on Blue Sky.
• 42:39
  SPEAKER_02: And also if you want to email me, you can email me Jonathan at JonathanPythonRayner.com.
• 42:43
  SPEAKER_02: All three of those ways you can get in touch with me.
• 42:46
  SPEAKER_02: I tend to I I don't necessarily talk exclusively about tech on most of those platforms.
• 42:49
  SPEAKER_02: I also talk about my other interests, including baking and drumming and various other things.
• 42:52
  SPEAKER_02: It's a whole smoggus board of fun things you can come along, you can come on to enjoy.
• 42:56
  SPEAKER_02: But I'm always open to conversation and I love chatting with people.
• 42:59
  SPEAKER_02: So if you just so feel free to just open up either DM Thread or whatever on any of those platforms, and I'll be happy to have a chat.
• 43:05
  SPEAKER_00: Wow.
• 43:06
  SPEAKER_00: Thank you so much, Jonathan.
• 43:07
  SPEAKER_00: Again, looking forward to meeting you maybe for another episode.
• 43:10
  SPEAKER_00: But that was awesome.
• 43:11
  SPEAKER_00: I really learned a lot, and I'm sure that the audience learned a lot too.
• 43:14
  SPEAKER_00: So thank you so much.
• 43:15
  SPEAKER_00: Have a good one.
• 43:16
  SPEAKER_00: Thank you.
• 43:16
  SPEAKER_00: Thank you.
• 43:17
  SPEAKER_00: It's been great.
• 43:17
  SPEAKER_00: Take care.
• 43:20
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 43:27
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 43:31
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.