Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy

Transcript

Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: I think it is for me personally one of the biggest features that I'm looking for is actually backfield scale.
• 0:05
  SPEAKER_02: Since Backfield was a big bang for us, and a lot of us are using backfields, and we have to develop our own backfield solution for Airflow 1.
• 0:13
  SPEAKER_02: But for Airflow 2, we're really looking forward for those changes that are going to come up, especially making backfield as the REST API instead of you all know CLI wasn't always the best UX experience for the users and has some significant some challenges.
• 0:29
  SPEAKER_02: But I'm really happy to see that airflow communities, airflow street plan all moving to providing more backfield as it services actually.
• 0:54
  SPEAKER_00: Hello, everyone.
• 0:55
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:57
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by two folks from Stripe, Nick Blazarov, who is a senior data engineer, and Sharad Krishnamurti, who is an engineering manager.
• 1:08
  SPEAKER_00: How are you both doing today?
• 1:12
  SPEAKER_02: Thank you for enlarging us for the podcast.
• 1:14
  SPEAKER_02: And we're happy to be here and share our experience on an airflow scale and Stripe.
• 1:21
  SPEAKER_03: I'm doing wonderful.
• 1:22
  SPEAKER_03: Happy to be here.
• 1:22
  SPEAKER_03: Thank you for having us on.
• 1:24
  SPEAKER_00: Awesome.
• 1:24
  SPEAKER_00: Yeah, we're great.
• 1:25
  SPEAKER_00: Well, I'm I'm super excited to have you both join.
• 1:28
  SPEAKER_00: I think Stripe is such a cool company that a lot of folks are familiar with.
• 1:32
  SPEAKER_00: So very excited to hear your story here.
• 1:34
  SPEAKER_00: And I think we can go ahead and dive right into it.
• 1:37
  SPEAKER_00: So from our listeners for today, we're going to discuss how Stripe has customized her own airflow installation.
• 1:44
  SPEAKER_00: And to start off, for those, I I just said that everybody is familiar with Stripe.
• 1:49
  SPEAKER_00: I think in my world, like everybody knows what Stripe is.
• 1:51
  SPEAKER_00: But for maybe for those who might not necessarily be familiar, can you start by telling me what does Stripe do as a company and what is your business model?
• 2:00
  SPEAKER_03: Yeah, absolutely.
• 2:01
  SPEAKER_03: I can take uh a stab at doing that.
• 2:03
  SPEAKER_03: So Stripe's mission is to grow the GDP of the internet.
• 2:07
  SPEAKER_03: That's a pretty lofty goal.
• 2:08
  SPEAKER_03: What does that really mean?
• 2:10
  SPEAKER_03: Most people are familiar with who are familiar with Stripe think that that means they're growing, they're writing payment infrastructure for the internet.
• 2:17
  SPEAKER_03: And that is part of what Stripe does, but it's a lot more than that.
• 2:21
  SPEAKER_03: It's a lot more than just processing payments.
• 2:23
  SPEAKER_03: It's things like recurring invoices or hosted payments or usage-based billing or analytics.
• 2:30
  SPEAKER_03: And recurring invoices let people bill their recurring customers every month, say for a subscription for something that they've bought.
• 2:39
  SPEAKER_03: Hosted invoices let someone who's completely not internet savvy accept a payment online because Stripe does all the hosting of the payment flow.
• 2:49
  SPEAKER_03: Usage-based billing is something cool, especially in the age of AI, when every time you use a token, you're going to get built for it eventually.
• 2:56
  SPEAKER_03: These are all things that businesses need to function as an internet company, and Stripe provides all of this for them.
• 3:03
  SPEAKER_03: And the part that I really like is because it's like data, is even analytics for small and medium-sized enterprises to understand how to grow their business.
• 3:13
  SPEAKER_03: All of this together goes into that really lofty goal of growing the GDP of the internet.
• 3:18
  SPEAKER_00: Yeah, that's awesome.
• 3:19
  SPEAKER_00: I love that description.
• 3:20
  SPEAKER_00: And that's a really big deal, right?
• 3:22
  SPEAKER_00: I mean, I think if Stripe were to suddenly not work for probably many companies, that would be pretty bad.
• 3:28
  SPEAKER_00: It would be very disruptive to the businesses.
• 3:31
  SPEAKER_03: Yeah, absolutely.
• 3:32
  SPEAKER_03: In fact, uh, we have a stat there.
• 3:34
  SPEAKER_03: One of our publicly uh reported numbers is that our users process like a collective one trillion dollars on Stripe in 2023, and that's roughly equivalent to like 1% of the global GDP.
• 3:49
  SPEAKER_03: So at this point, we're at the size of small nations.
• 3:53
  SPEAKER_00: Just gonna say larger than the GDP of many countries.
• 3:57
  SPEAKER_03: So that's it's a pretty major responsibility, and I'm sure we'll get into it.
• 4:02
  SPEAKER_03: It also means that our systems have to be like super reliable and and a bunch of regulatory requirements.
• 4:06
  SPEAKER_03: That means our data has to be super reliable as well.
• 4:09
  SPEAKER_00: Yeah, totally.
• 4:10
  SPEAKER_00: And so speaking of your data, what does your team do specifically?
• 4:14
  SPEAKER_03: Yeah.
• 4:14
  SPEAKER_03: So Nick and I are on the data orchestration team.
• 4:18
  SPEAKER_03: Uh, we own the orchestration of the uh batch jobs that we write here at Stripe.
• 4:22
  SPEAKER_03: Uh I'm sure people who tune into this podcast will know exactly what we talk, we're talking about.
• 4:26
  SPEAKER_03: Uh, airflow is great for this sort of thing.
• 4:29
  SPEAKER_03: Uh, we also own the user experience of writing these jobs, operating them, making changes to them, rolling out those changes, cool terms like backfill and validations and you know, uh write, audit, release workflows, and so on.
• 4:43
  SPEAKER_03: So we own all of that, right?
• 4:45
  SPEAKER_03: The numbers are sort of illustrative of the responsibility and the scale of it.
• 4:49
  SPEAKER_03: At last count, we had something like 250 plus DAGs that together had something like 200,000 plus tasks that executed on them uh every day.
• 5:00
  SPEAKER_03: And about 500 teams internally at Stripe use in some way interact with the system uh to write these jobs and operate them.
• 5:08
  SPEAKER_03: And here's the crazy stat about 80% of all engineers interact with our system.
• 5:14
  SPEAKER_03: So just like a reflection of the world we live in today, where it used to be that you the data folks wrote data jobs, but the reality of today is that everyone is in some part a data engineer, and that's borne out by our statistics.
• 5:27
  SPEAKER_00: Yeah, that's great.
• 5:28
  SPEAKER_00: So so you said 80% of the engineers at Stripe interact with the systems that your team maintains.
• 5:34
  SPEAKER_00: Yes, that's right.
• 5:34
  SPEAKER_03: In some way, shape, or form, they're interacting with batch data orchestration.
• 5:39
  SPEAKER_03: Yeah, that's huge.
• 5:40
  SPEAKER_00: That's that's really cool.
• 5:42
  SPEAKER_00: Awesome.
• 5:43
  SPEAKER_00: Well, then so let's talk a little bit about how Airflow fits into that.
• 5:46
  SPEAKER_00: You mentioned I I think this will be kind of a bread and butter discussion for Airflow, is providing that sort of orchestration for a lot of companies.
• 5:54
  SPEAKER_00: And I think Stripe has been pretty vocal about their Airflow usage.
• 5:56
  SPEAKER_00: So tell me kind of how Airflow fits into your tech stack, and then we'll get into how you've customized it for your use cases.
• 6:05
  SPEAKER_03: Yeah.
• 6:06
  SPEAKER_03: So we use Airflow to orchestrate all data pipelines at Stripe.
• 6:10
  SPEAKER_03: This starts from getting data from our online systems, the ones that are actually the API systems.
• 6:17
  SPEAKER_03: We have an entire team that's dedicated to taking these petabytes of data that are arriving at like this really incredible scale and moving them seamlessly into what we call data land, into iceberg and all the familiar territory of batch data uh jobs.
• 6:32
  SPEAKER_03: We have so they are orchestrated on our Airflow stack.
• 6:37
  SPEAKER_03: We also have a number of teams that use this data then to generate important business reports that are used for regulatory requirements in a bunch of countries.
• 6:46
  SPEAKER_03: So you can think of it as this is required for Stripe to operate in those countries.
• 6:50
  SPEAKER_03: So you're used for that.
• 6:52
  SPEAKER_03: We have ML systems that again use both the source data as well as some of this derived data to uh write ML models and do scoring.
• 7:01
  SPEAKER_03: And we have an auxiliary system that is used to actually orchestrate the uh ML model workflows, but then the data itself comes through the system and the orchestration there happens with us.
• 7:11
  SPEAKER_03: And this is like pretty much everything, I would say.
• 7:14
  SPEAKER_03: Pretty much everything we do with data goes through the orchestration team and the orchestration stack, which is based on Talk of Airflow.
• 7:22
  SPEAKER_03: I won't go too much into the customizations.
• 7:24
  SPEAKER_03: I think there's there's a lot we could get into the technical details there, but at a high level, that's what we that's what we use it for.
• 7:30
  SPEAKER_00: Yeah, definitely.
• 7:31
  SPEAKER_00: That's great.
• 7:32
  SPEAKER_00: That's a huge scope.
• 7:33
  SPEAKER_00: And I think totally makes sense based on what you've described, you know, even your whole business model as being very data centric.
• 7:40
  SPEAKER_00: But I I do want to dive into those technical details though.
• 7:44
  SPEAKER_00: So we we've kind of led on a few times that you have made some customizations on top of Airflow.
• 7:50
  SPEAKER_00: So tell me uh maybe first what motivated you to do that.
• 7:54
  SPEAKER_00: So you're working with open source airflow, you have your own fork of the project.
• 7:58
  SPEAKER_00: Why did you decide to go down that route?
• 8:02
  SPEAKER_02: I think we had some historical reasons to expedition, but overall the main goal was to provide our users with their flexibility and burned faces to be more successful in the day to day-to-day job.
• 8:16
  SPEAKER_02: And for instance, exactly because of that, and we have a special task altering framework.
• 8:22
  SPEAKER_02: So basically, we don't use the standard Airflow One task definitions, and we made some custom implementation that forces users to explicitly provide input and output on the task definition in essentially in Python.
• 8:37
  SPEAKER_02: And in this way, it helps us to provide users with great flexibility, how to manage the dependencies between different pipelines, different decks.
• 8:47
  SPEAKER_02: At the same time, it helps us to develop, I think, something special that we have here, that I haven't seen anywhere else, which is called test framework for testing our batch uh jobs, which called internally user scope mode.
• 9:03
  SPEAKER_02: And what it allows our users uh to do is basically to test all the data pipelines and all the changes to pipelines locally without a fear that they're going to interfere with production, output or production works or workloads.
• 9:20
  SPEAKER_02: And that was uh the special task after development internally that's trying to help us to achieve us because uh when users use user scope mode, essentially they're running their pipelines on local machines, uh they are able to we are able to interfere and replace all the outputs and inputs with the safe locations essentially.
• 9:44
  SPEAKER_02: So instead of outputting to the production location, obviously they will put put on subscription zone on the SG and they can verify that the drop that the changes are safe to roll out the production.
• 9:57
  SPEAKER_02: In this way, we managed to stitch all the jobs together.
• 9:59
  SPEAKER_02: So essentially, users can just run any number of tasks and connect them together and test them in a safe, reliable way on the local machines.
• 10:09
  SPEAKER_02: Uh, obviously, this is something that sold us really well over the many years, but we don't want to stop here and currently have really ambitious projects to make it even better to make those tests in automatic way when you just push in their pipeline, the changes around all those tests in DCI and in the different environment to make sure that uh the changes are safe to deploy and then they will be automatically deployed as soon as that all tests are passed.
• 10:38
  SPEAKER_02: I think something that we make really interesting is for Airflow to immigration is to how our decks separate as a service.
• 10:49
  SPEAKER_02: So essentially every deck is a separate service, which deployed separately, and it display connection isolation.
• 10:57
  SPEAKER_00: Yeah, that's really interesting.
• 10:59
  SPEAKER_00: So you uh to uh summarize some of that, so you have basically created a fork that helps you democratize airflow usage within a very large organization where you've got many, many, many teams and lots of engineers working with a platform, and you obviously need to put some guardrails on that so that it's both easy for them and so that they're not, you know, potentially taking down very critical systems with their development.
• 11:25
  SPEAKER_00: So your user scope mode is, I think it sounds very analogous to like with us on, you know, astronomer platform, we often have folks have like dev deployments, which are just again kind of the isolated is very it's designed to mimic your production deployment, but it's not pointed at production databases or or anything like that.
• 11:44
  SPEAKER_00: But you've got something similar set up, but it's working on their local machine, so it's super easy for them to use without having to access those external systems.
• 11:53
  SPEAKER_02: Yes, that's really and that was the main goal to satisfy all the workflow we use and make them more successful in day-to-day work.
• 12:02
  SPEAKER_03: Yeah, I don't know if we mentioned this explicitly, but most of our development today, the the usage today is on a fork of Airflow One, and we are in the process of moving to an Airflow 2-based system, which we also hope is easier to upgrade in the future.
• 12:18
  SPEAKER_03: But the you mentioned earlier about why we went down this route.
• 12:21
  SPEAKER_03: We went down the fork route with Airflow One for a number of historical reasons, but with the reimagined Airflow 2-based system, it is very much a composition pattern that we are trying to use open source airflow as much as possible and add things through the many customization hooks that have been added since then.
• 12:42
  SPEAKER_00: Yeah, that makes sense.
• 12:43
  SPEAKER_00: So not quite such a distinct fork where you've branched off and then continued down your own path, but more just working from the existing open source project.
• 12:51
  SPEAKER_00: I think that makes sense.
• 12:52
  SPEAKER_00: So you uh being on, you know, you're currently migrating to Airflow 2.
• 12:57
  SPEAKER_00: That's quite even Airflow 2 is quite an old stage of the project.
• 13:01
  SPEAKER_00: Have you run into any challenges being on an older Airflow version, still working off of Airflow 1?
• 13:07
  SPEAKER_02: Yes.
• 13:08
  SPEAKER_02: So I think one of the those challenges, they're gonna get pretty similar to everyone.
• 13:13
  SPEAKER_02: So it's going to listen on the podcast.
• 13:15
  SPEAKER_02: They used to use one on Airflow One.
• 13:17
  SPEAKER_02: And the biggest sales obviously, like limitation on the scheduler and scalability of the running single-scale design that we are hoping that we are looking forward in IFO 2, and it helps us to scale much better and provide the performance for our system as well.
• 13:32
  SPEAKER_02: Uh general database and overall performance of the system.
• 13:36
  SPEAKER_02: Since in IRF one, what we noticed task execution schedule often suffer from performance degradation since we have a big number of the tasks and the number of the tasks increased, and as a result, a lot of insufficient querying going to database.
• 13:52
  SPEAKER_02: And for us, it was a really big concern since historically we had the department that have a lot of big monolithic decks that essentially have a lot of tasks in them.
• 14:03
  SPEAKER_02: It means all the queries that were running against make the database.
• 14:07
  SPEAKER_02: We need to scale them, and we are suffering a lot from different database performance issues, and we had to make some changes to our fork in order to make those queries more efficient.
• 14:18
  SPEAKER_02: And that we are hoping this is going to solve uh this problem is going to be solved with dexterization in RFL2 all the streets and the different optimization that we will make over the years to make RO2 more performant and more stable and improving task and bad performances.
• 14:39
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 14:48
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 15:00
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 15:08
  SPEAKER_00: I think this stability was one of the biggest benefits in my mind, at least, of the Airflow 2 release.
• 15:14
  SPEAKER_00: I also used to work with Airflow 1 in my in my previous job as well.
• 15:18
  SPEAKER_00: So I remember those days distinctly.
• 15:20
  SPEAKER_00: I'm curious how did you manage those challenges without you're just now going through the upgrading process, which I think will bring a lot of benefits.
• 15:29
  SPEAKER_00: But but you know, you mentioned like you're not working with a small airflow instance or anything.
• 15:34
  SPEAKER_00: And so dealing with um those performance issues, I'd imagine, would be quite challenging at your scale.
• 15:41
  SPEAKER_02: Yes.
• 15:42
  SPEAKER_02: So in order to tackle this performance scalar through IFO 1, we had to make some customization to the fork.
• 15:49
  SPEAKER_02: But with Airflow 2, I think like one of the biggest challenges for us right now on the integration side is to make it in seamless for i for our users as possible and to provide them a happy pass, basically move from iFO1 to IFLO 2.
• 16:04
  SPEAKER_02: And on this pass to solve different problems that we had in our digital limitations of our current system.
• 16:11
  SPEAKER_02: For instance, as I mentioned earlier, we have a lot of monolithic decks that we need to uh split on the smaller pipeline that would help our users to be more to have more stable pipelines.
• 16:23
  SPEAKER_02: But as well, what we do, we separate all the decks on the separate services in RFO2, which will help to achieve us great isolation and more safer deployments and safe uh change management for our users.
• 16:38
  SPEAKER_03: I'll just add on that we've been super lucky and blessed to have great quality of engineers who have sort of maintained a very complex system with limited resources.
• 16:48
  SPEAKER_03: But it is a challenge, and that's part of the reason why we are moving to a system based on Airflow to taking advantage of all the improvements in the community.
• 16:56
  SPEAKER_03: I would say that our it's sort of like a flywheel in the wrong direction because we early in the project's history, we took on, say, these large monolithic DAGs.
• 17:06
  SPEAKER_03: And as these grew and we uh decided not to pass the pain on to the rest of Stripe, we had to make some customizations to make the workflows work with all these large monolithic DAGs.
• 17:17
  SPEAKER_03: And then that took us down a path where we just kept adding more customizations.
• 17:20
  SPEAKER_03: And the way we maintain them is we've reached some level of stability for sure.
• 17:25
  SPEAKER_03: And it's not as hard as you might imagine because you have this practice of maintaining this very large and operating this very large system.
• 17:33
  SPEAKER_03: Uh, but the really cool thing is that uh Stripe is a growing company, so we're still growing, and we can clearly see on the horizon a point beyond which we cannot scale simply because we have a monolithic DAG that has like 10,000 or 100,000 tasks that kick off at UTC midnight, and there is no database that we can use that will that will absorb that level of pain, right?
• 17:55
  SPEAKER_03: So there are certain things we are we are dealing with it, but there's very clearly a point where we know we can't deal with it, which is why we've started that adoption of a new of a new platform.
• 18:05
  SPEAKER_00: Yeah, that makes sense.
• 18:06
  SPEAKER_00: I think a question I get all the time from Airflow users is like, at what level of task or number of tasks do I need to start being concerned about performance or about this feature not working?
• 18:17
  SPEAKER_00: Because you know, like high scale, I think means so many different things to different people, but certainly a hundred thousand tasks is is over the line, regardless of where you draw the line.
• 18:28
  SPEAKER_00: So yeah, I I can totally see where that would be prompting you to kind of make some longer-term adjustments for the stability of your platform.
• 18:36
  SPEAKER_00: I guess in terms of so you've talked uh a lot now about migrating to you know a newer fork off of the Airflow 2 project, but Airflow is also moving forward now to we're looking at the Airflow 3.0 release coming in another, gosh, it's gonna, it's gonna really sneak up on me, like end of March, early April kind of time frame, which is really right around the corner.
• 18:59
  SPEAKER_00: So, how are you thinking about, you know, kind of managing your own airflow usage with those continued upgrades in the project, you know, new features that are coming out, including new major releases.
• 19:11
  SPEAKER_00: Do you think you'll adjust so that it's easier for you to upgrade Airflow, or how are you thinking about that?
• 19:17
  SPEAKER_02: From our side, we obviously wanted to stick to your frequent upgrades and keep up with the community and uh we develop an upgradation strategy that we're going to just stick to during the whole creating process.
• 19:30
  SPEAKER_02: And currently already made one upgrade from 2.8 to 2.10.
• 19:36
  SPEAKER_02: Went pretty well.
• 19:37
  SPEAKER_02: And I think even moving further, it should be B pro for us.
• 19:40
  SPEAKER_02: But what we want to do for sure is to keep with all the features and to make sure that they take grantage of all the airflow development uh and keep them gradient and not to stick to the old version.
• 19:51
  SPEAKER_03: Yeah, the there's been a bunch of things that Nick and I were talking about in terms of what would we love to use in Airflow 3, right?
• 19:58
  SPEAKER_03: And I think Nick, you you had some really cool ideas there about like that versioning is something we really struggled with today.
• 20:05
  SPEAKER_03: I don't know, I don't know if you want to like touch upon that in terms of what we would like to get out of Airflow 3.
• 20:09
  SPEAKER_02: Yeah, definitely.
• 20:10
  SPEAKER_02: So that version it was a big ask probably for our users since a lot of the currently in Airflow, when you change a big deck, you basically assume that this deck was never changed and it uh takes it for granted.
• 20:24
  SPEAKER_02: But users quite often due to changes, and as a result, you have the deck running essentially with the you have your big deck while it's still running, which is not sometimes not great experience and can result in different issues, uh, and that they're looking we are looking forward to.
• 20:40
  SPEAKER_02: But I think it is for me personally one of the biggest features that I'm looking for is actually backfield scale.
• 20:45
  SPEAKER_02: Since backfield was a big pain for us, and a lot of us are using backfields, and we have to develop our own backfield solution for airflow one.
• 20:54
  SPEAKER_02: But for airflow two, we're really looking forward for those changes that are going to come up, especially making backfield as the rest API instead of you all know CLI wasn't always the best UX experience for the users and has some significant, some challenges.
• 21:10
  SPEAKER_02: But I'm really happy to see that Airflow community is airflow ship and all moving to providing more backfill as a source, essentially.
• 21:20
  SPEAKER_00: Yeah, definitely.
• 21:21
  SPEAKER_00: I think you know, DAG versioning is definitely getting a lot of publicity with the Airflow 3 release because it is a very widely requested feature, but I think actually backfills, you know, the are they not as sexy as some of the other new features or as exciting to talk about, but I do think they're one of the most impactful things.
• 21:39
  SPEAKER_00: They are also very, very highly requested or have been a highly requested feature for a long time.
• 21:45
  SPEAKER_00: And I think it's gonna be one of the biggest changes just in people's day to day use of Airflow.
• 21:50
  SPEAKER_00: As you said, having that be more of a first class feature as opposed to a thing that you can do, but like it's maybe not the best experience.
• 21:58
  SPEAKER_00: I think, I think a lot of teams.
• 22:00
  SPEAKER_00: Are gonna get a lot of use out of that.
• 22:05
  SPEAKER_03: Yeah, in in general, like I think the kind of challenges that we get when running Airflow on behalf of so many different teams, I would be really curious.
• 22:17
  SPEAKER_03: I see that the trend line is definitely towards an uh the community understanding there are different workflows we need to enable and people will customize their deployment of Airflow.
• 22:27
  SPEAKER_03: So I'm personally looking forward to what happens there.
• 22:30
  SPEAKER_03: I haven't gone through all the release notes or the architecture differences, but I'm really excited each time uh a new release comes out.
• 22:36
  SPEAKER_03: Like, how are we making it easier?
• 22:39
  SPEAKER_03: Initially, it was just like plugins, but then there's a whole different community now about how do you customize your Airflow?
• 22:45
  SPEAKER_03: The REST API really opened up a bunch of possibilities.
• 22:48
  SPEAKER_03: So yeah, I think every every company is very unique and its workflows are unique.
• 22:53
  SPEAKER_03: And I think uh Airflow being so central to the batch data workflow, it has to adapt to to suit every company that it's deployed at.
• 23:02
  SPEAKER_00: Yeah, yeah, absolutely.
• 23:04
  SPEAKER_00: Well, I think that actually leads in really well to my last question, which I always ask to every guest who joins us, and you've already sort of answered this, so maybe you can double down or give me a new answer, whichever you'd prefer.
• 23:15
  SPEAKER_00: Uh my last question is always what is one thing that you'd like to see from the Airflow project in the near future?
• 23:22
  SPEAKER_02: I think it is from my side something that I think during our current integration to IFLO2, and we were doing is to have a bit more flexibility in terms of extending UI part of the Rflow 2, Iflow 3 to make it possible to add specific features that are applicable to the companies, even like clicking a button on the task page or on the DAC page to make it a bit more flexible.
• 23:46
  SPEAKER_02: Currently, it is possible to create different pages, use the plugins, and play around with general like menu, but it would be nice if we can customize, have more ability to do to add different customization to the UI, to adjust to the need or to the specific needs of our users internally.
• 24:08
  SPEAKER_02: Sharad, how about you?
• 24:09
  SPEAKER_02: What are you looking for?
• 24:10
  SPEAKER_03: I I had I I had to struggle to keep this like in a small list.
• 24:14
  SPEAKER_03: So I'll I'll I'll I'll I'll take the top top items.
• 24:17
  SPEAKER_03: So the thing that I'm most excited about is I see that from the Airflow summit that the team attended last year, the we got a lot of value about how other companies were deploying their Airflow.
• 24:29
  SPEAKER_03: And you touched upon it, Yantin, as well.
• 24:31
  SPEAKER_03: Uh at scale, there isn't really convergence on this is how we run Airflow at scale, right?
• 24:36
  SPEAKER_03: Maybe it's because, again, there are different org structures, it's not just data engineering, or it's just data engineering, or like there's one monolithic team that does all the data, or like you have a you know, the data mesh approach where you have many teams doing the data.
• 24:49
  SPEAKER_03: We don't know.
• 24:50
  SPEAKER_03: But I think the theme, thing that's really curious, things that I would like to see from the community is more opinions on this is how you run Airflow at scale.
• 24:59
  SPEAKER_03: And that I think that will really in turn precipitate a bunch of improvements in reliability, like you said, and running this thing at scale.
• 25:06
  SPEAKER_03: So I'm really excited about that.
• 25:07
  SPEAKER_03: I'm also excited for for us to contribute because we do have a really large Airflow deployment.
• 25:12
  SPEAKER_03: Um, the other thing I'm really excited about and looking forward to in terms of improvements is I think the data becoming so critical.
• 25:19
  SPEAKER_03: And what, for example, I mentioned how for us regulatory reports are built on top of data and they have to be like absolutely of the highest quality.
• 25:28
  SPEAKER_03: Some I some thoughts about the change management workflow of data being baked into Airflow, I mean, I think might be really cool.
• 25:36
  SPEAKER_03: I know, for example, a Baxel itself is an opinion on data management, right?
• 25:41
  SPEAKER_03: Like you might have to go backsel your data when you add a new feature to a uh to a data set, right?
• 25:46
  SPEAKER_03: But similarly, like how do you actually release this thing to production and you have 200 teams with their downstreams depending on it?
• 25:53
  SPEAKER_03: Thinking about that and think I'm really excited to see how the community comes together and says this is how we solve this problem.
• 25:59
  SPEAKER_03: And again, excited to contribute uh back to that as we get on a more uh vanilla version uh of the system.
• 26:07
  SPEAKER_00: Yeah, that's great.
• 26:08
  SPEAKER_00: I love that.
• 26:08
  SPEAKER_00: Those are all very good requests.
• 26:10
  SPEAKER_00: Yeah, awesome.
• 26:11
  SPEAKER_00: Thank you for sharing.
• 26:12
  SPEAKER_00: Well, thank you so much for going through your use case.
• 26:15
  SPEAKER_00: I think this has been really interesting and it's been great to learn more about how you're using Airflow and where you're going with it in the future.
• 26:22
  SPEAKER_00: To close out, Nick and Shred, what is the best way for folks to get in touch with you?
• 26:28
  SPEAKER_03: Oh, yeah, you can you can email us.
• 26:30
  SPEAKER_03: We have our our Stripe handles, Sharad at Stripe.com or Nick Belozorov at Stripe.com.
• 26:36
  SPEAKER_03: Or even easier, like reach out to us on LinkedIn.
• 26:39
  SPEAKER_03: It's LinkedIn Nick Belozarov and LinkedIn Sharad K.
• 26:43
  SPEAKER_03: I'm sure that we'll be have we'll have some way to add this to like the show notes or something.
• 26:47
  SPEAKER_03: We can.
• 26:48
  SPEAKER_03: But yeah, that that would be a lot more convenient than me spelling it out.
• 26:51
  SPEAKER_03: So thank you for that.
• 26:52
  SPEAKER_03: But yeah, just reach out to us.
• 26:54
  SPEAKER_03: We're super approachable.
• 26:55
  SPEAKER_03: We're very eager to collaborate.
• 26:56
  SPEAKER_03: And like I said, I'm I'm personally really excited about the energy and the improvements in the space and seeing how we can collaborate and take advantage and contribute back to it.
• 27:07
  SPEAKER_00: Awesome.
• 27:08
  SPEAKER_00: That's great.
• 27:09
  SPEAKER_00: Well, again, thank you so much.
• 27:11
  SPEAKER_00: It's been really wonderful having you on.
• 27:13
  SPEAKER_02: Yeah, Angel Having us.
• 27:15
  SPEAKER_00: Thank you for having us, and uh yeah, take care.
• 27:17
  SPEAKER_00: Have a wonderful rest of your day.
• 27:18
  SPEAKER_00: Yeah, you guys too.
• 27:21
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 27:28
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 27:32
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.