Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development

Transcript

Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: But then we are in the era of of AI.
• 0:03
  SPEAKER_02: So all these closed source solutions that you know don't have like a code first approach are a little bit behind, or a lot behind, I would say, right?
• 0:15
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:28
  SPEAKER_00: Okay, hi everyone.
• 0:29
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:31
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Philip Kunchar, who is a platform director at Chipmunk.
• 0:38
  SPEAKER_00: Philip, how are you?
• 0:39
  SPEAKER_02: I'm good.
• 0:40
  SPEAKER_02: Thank you.
• 0:40
  SPEAKER_02: Hi, Kendon.
• 0:41
  SPEAKER_02: Thank you for having me.
• 0:42
  SPEAKER_00: Yeah, it's great to have you on.
• 0:44
  SPEAKER_00: I'm excited to get into it.
• 0:46
  SPEAKER_00: Yeah, hope your week is going well so far.
• 0:48
  SPEAKER_02: Yeah, it's it's been exciting, right?
• 0:50
  SPEAKER_02: With all the AI changes.
• 0:52
  SPEAKER_00: So goodness, I feel like that could apply to any given week.
• 0:58
  SPEAKER_00: There's certainly so much going on in the space right now.
• 1:00
  SPEAKER_00: It is both exciting and a little overwhelming.
• 1:04
  SPEAKER_02: True.
• 1:05
  SPEAKER_02: True.
• 1:05
  SPEAKER_02: Very true.
• 1:06
  SPEAKER_00: Yeah.
• 1:06
  SPEAKER_00: Well, we, I think we'll get into a little bit of that in the context of Airflow later in our discussion.
• 1:12
  SPEAKER_00: But for right now, let's dive in.
• 1:14
  SPEAKER_00: Uh for an overview for our listeners for today's podcast.
• 1:18
  SPEAKER_00: We're going to discuss how Shipmunk built their data platform on Airflow.
• 1:22
  SPEAKER_00: But before we get into even Airflow specifically, let's start with some background.
• 1:27
  SPEAKER_00: Can you tell me what Shipmunk does?
• 1:30
  SPEAKER_02: Sure.
• 1:30
  SPEAKER_02: Shipmun is a free bill company, which means for body logistics, what we focus on, like we want to take the burden from the e-commerce companies to store the goods and deliver to the customers and provide this as a service.
• 1:44
  SPEAKER_02: So we do everything like receiving the package and storing it into our warehouses, then picking it, boxing it, and then sending it out to the customers.
• 1:54
  SPEAKER_02: So that's all we take care about.
• 1:56
  SPEAKER_02: For every merchant that out there, we want to make this experience seamless so they don't have to care where the goods are and how they get to the customers and they can just rely on us.
• 2:07
  SPEAKER_00: So if I order something online that's coming from like halfway across the globe, like everything that happens in the middle is maybe being touched by shipmunk.
• 2:17
  SPEAKER_00: Is that right?
• 2:18
  SPEAKER_02: Exactly.
• 2:18
  SPEAKER_02: That's it.
• 2:20
  SPEAKER_02: We're most focused on the US market.
• 2:22
  SPEAKER_02: That's our biggest market.
• 2:24
  SPEAKER_02: We have some presence in Europe, but the US is definitely the biggest market that for us.
• 2:28
  SPEAKER_02: And we guarantee if if the customer wishes, so we guarantee the two-day delivery.
• 2:34
  SPEAKER_02: So and that that's pretty great for the whole whole US.
• 2:37
  SPEAKER_00: Wow, that's amazing.
• 2:39
  SPEAKER_00: I I live in Seattle, and so every time I'm down on the waterfront on Puget Sound is a big like shipping way for you know maritime shipping and just seeing all of the like cargo ships and containers and everything.
• 2:50
  SPEAKER_00: It's uh it's just sort of mind-blowing that it all works the way it does.
• 2:55
  SPEAKER_00: So yeah, very I'm excited to dig into like what powers this for your business.
• 2:59
  SPEAKER_00: Because again, talk about like huge impact that's happening mostly behind the scenes that people aren't even aware of.
• 3:06
  SPEAKER_00: Let's dig into what your team then uh assume the data platform team, what do you do specifically?
• 3:13
  SPEAKER_02: I lead, I lead the whole platform.
• 3:15
  SPEAKER_02: So we have the infrastructure team, security team, the developer experience, but the data platform is responsible for everything within this infrastructure for everything that is data related.
• 3:27
  SPEAKER_02: So uh we have the decentralized system or the decentralized architecture.
• 3:33
  SPEAKER_02: So all the developers and I mean really the developers, the full-stake developers, backend developers, they take care about their data, right?
• 3:40
  SPEAKER_02: So, for example, if if they own the data in the primary system, all the modeling and all the presenting to the customer or to the exec team is on the team.
• 3:52
  SPEAKER_02: So we as a data platform provide everything they need, every tooling to make this data available or or and model them into the right way.
• 4:01
  SPEAKER_02: And we have to do the whole experience as seamless as possible.
• 4:06
  SPEAKER_02: But what we say, we try to lower their cognitive load when they need to work with the data, right?
• 4:12
  SPEAKER_02: That's our motto.
• 4:13
  SPEAKER_02: Like make it possible as a natural and technique again.
• 4:17
  SPEAKER_02: It's not in their language, it's not in PHP, it's not in Java, they don't care, they just use the tool and they feel like it's it's native to them.
• 4:25
  SPEAKER_02: So that that that's our mission.
• 4:27
  SPEAKER_00: Yeah, that's very cool and quite a big goal, I'll say, for for a team.
• 4:34
  SPEAKER_00: Um, yeah, that's great.
• 4:36
  SPEAKER_00: And uh tell me a little bit how Airflow fits into that, and I guess maybe some history first.
• 4:43
  SPEAKER_00: I understand that you brought Airflow into Shipmunk.
• 4:47
  SPEAKER_00: Um, so tell me kind of when that happened and what the situation was before that, and why you made that decision.
• 4:54
  SPEAKER_02: We have Airflow from 2024, so for two years now.
• 4:59
  SPEAKER_02: I joined joined Shipmunk at the time, and and and I thought the uh change was necessary.
• 5:05
  SPEAKER_02: And and then that there is several reasons.
• 5:07
  SPEAKER_02: We were on the closed source solution before, and the most of the solutions, the data solutions, they are first, right?
• 5:15
  SPEAKER_02: So so you you don't have the code, and and that's something Airflow is just great for because because when you have the source code, you can you can understand what's happening on the background, first of all, but then we are in the era of of AI.
• 5:31
  SPEAKER_02: So all these closed source solutions that you know don't have like a code first approach are a little bit behind or a lot behind, I would say, right?
• 5:42
  SPEAKER_02: So so the the first motivation.
• 5:43
  SPEAKER_02: The second motivation, you just don't want to be limited by the features of of the closed source community that is developing that.
• 5:51
  SPEAKER_02: And obviously, Airflow is open source again.
• 5:54
  SPEAKER_02: Uh so that's great.
• 5:56
  SPEAKER_02: And then the the last reason for Airflow, you can call you know Daxter or or any other tool that you can think about.
• 6:04
  SPEAKER_02: But Airflow is the the community behind Airflow is so huge, and it's supported by all the major cloud providers, right?
• 6:11
  SPEAKER_02: So so this is a very huge point in the in the decision making.
• 6:16
  SPEAKER_02: So these are like the biggest reasons.
• 6:18
  SPEAKER_02: So we can open another use cases and don't be afraid that we will not be supported or we will lack some features.
• 6:25
  SPEAKER_02: So that's why I decided to go with the Airflow.
• 6:28
  SPEAKER_02: Obviously, me having the experience is uh a huge impact on the decision, definitely.
• 6:33
  SPEAKER_02: But but I think that was a great decision and made the whole thing not even not just better and more open, but also also cheaper, which is another which is another factor that that we definitely care about.
• 6:47
  SPEAKER_02: So yeah, that's that's a little bit of history.
• 6:50
  SPEAKER_02: And and today we have most of the data flows and the orchestration of everything, not just the data, but a lot of orchestration we have here is is on okay.
• 7:01
  SPEAKER_00: Yeah, very cool.
• 7:02
  SPEAKER_00: I think the customizability and flexibility of Airflow that you described are probably the two most common reasons that I hear of people choosing it.
• 7:11
  SPEAKER_00: And in my opinion, one of the reasons the community behind it has grown so much.
• 7:16
  SPEAKER_00: So it's cool to see to hear about that being so impactful for your team.
• 7:22
  SPEAKER_00: Tell me a little bit more about what those features of airflow, maybe over kind of the UI-based tool that you used previously, what have those allowed you to do that you couldn't be for at Chipmunk?
• 7:36
  SPEAKER_02: Yeah, so like the first thing I already mentioned, having the code in front of ourselves.
• 7:42
  SPEAKER_02: So so I I mean, I would say lineage is a great thing.
• 7:46
  SPEAKER_02: And and today, if you just can tell AI, hey, where does this specific data point come from, you know?
• 7:54
  SPEAKER_02: Okay, can you draw me the whole picture?
• 7:57
  SPEAKER_02: Just goes into the code and just seizes it immediately.
• 8:00
  SPEAKER_02: So that's one of the things that we really like before.
• 8:04
  SPEAKER_02: But also the use cases such as you know connecting to other systems and having the you can connect basically any API and you don't have to be limited by just what's written directly in the source code, was great.
• 8:16
  SPEAKER_02: So when we implemented Airflow or started using Airflow, we started to depend on like the whole DBT stack is now orchestrated through Airflow, so which enables again a lot of triggers uh outside of the ecosystem, right?
• 8:32
  SPEAKER_02: So you can have it one thing in dbt, then you just have something, for example, at the client side or in Google Sheets or somewhere somewhere else, and then you can just very easily play with that and and and it works.
• 8:44
  SPEAKER_02: So the use cases you're asking about what it enables us to do.
• 8:49
  SPEAKER_02: We are, for example, sharing the data with the merchants directly.
• 8:52
  SPEAKER_02: That's really great.
• 8:53
  SPEAKER_02: And I really appreciate that we can do that uh as seamlessly as we do it right now.
• 8:58
  SPEAKER_02: We are orchestrating everything within the data modeling, so the whole dbt path is orchestrated through through Airflow.
• 9:05
  SPEAKER_02: And then we do we play with AI, obviously.
• 9:08
  SPEAKER_02: So we orchestrate AI things and we orchestrate even some, I'd say maintenance things on the infrastructure or on the infrastructure side, or like what we call it maintenance.
• 9:18
  SPEAKER_02: So basically, what the infra infra team or the data platform team needs to do, and it's very easy to just put it into the DAG and go.
• 9:26
  SPEAKER_02: So these are the basic use cases we were not able to do before.
• 9:30
  SPEAKER_00: Okay, that's great.
• 9:31
  SPEAKER_00: I there's so much in there that I want to dig into.
• 9:35
  SPEAKER_00: So we'll I think we'll get to all of it.
• 9:37
  SPEAKER_00: But I want to start with the DBT and Airflow use case because I think this is a really common pairing for tools that a lot of teams use, and it just enables a lot of really great analytics and ETL and all of that.
• 9:50
  SPEAKER_00: So tell me first, what method are you using to orchestrate dbt and airflow together?
• 9:57
  SPEAKER_00: Or orchestrate DBT with Airflow, I should say.
• 9:59
  SPEAKER_02: Yeah, yeah.
• 10:00
  SPEAKER_02: So we use Cosmos from Astronomer, that's one, and then we also use just the native bash, you know, uh bash operator in and to to orchestrate.
• 10:11
  SPEAKER_02: They are two different things for two different purposes.
• 10:15
  SPEAKER_02: When we try to orchestrate for the rest of the team, so for the other developers, for the full stack developers, that that just tool for them and they need to have a perfect visibility and and see the lineage and uh what happened in the specific tasks and so on, we use Cosmos because the design is just out there.
• 10:34
  SPEAKER_02: They just come into the airflow and just see everything they need to see.
• 10:38
  SPEAKER_02: The disadvantage of this approach for us is it has a significant overhead.
• 10:43
  SPEAKER_02: If you have want to have everything you know perfectly prepared for somebody to be as visible as Cosmos allows to, you you need to have the overhead.
• 10:52
  SPEAKER_02: So you you have tons of tasks, you you have uh you are using resources that you wouldn't use with with the bash operator.
• 11:00
  SPEAKER_02: So with the internal things we use within the infrastructure, within the DBT, I mean, uh we use simple bash operator, and and it's much quicker and much resource intensive.
• 11:12
  SPEAKER_02: So so that this is the reason why we are using both of the methods.
• 11:15
  SPEAKER_00: Yeah, that makes sense.
• 11:16
  SPEAKER_00: Definitely some trade-offs there.
• 11:18
  SPEAKER_00: I think the visibility is a really great benefit of Cosmos, but certainly does add some complexity in some cases.
• 11:25
  SPEAKER_00: Maybe you've already mentioned it a little bit with kind of the trade-offs, but I'm curious if there are any particular challenges that you've encountered with using DBT and Airflow together.
• 11:34
  SPEAKER_02: Yeah, that's that's the the overhead I mentioned because specific example, we we were running one of the flows we need to run every 15 minutes or so.
• 11:45
  SPEAKER_02: Uh so pretty frequently.
• 11:47
  SPEAKER_02: And we need to be sure that you know it it runs that fast.
• 11:50
  SPEAKER_02: And and when we were on Cosmos, we were over the 15 minutes pretty pretty often.
• 11:56
  SPEAKER_02: When we just you know hit everything into the one bash command, we we we just we're just down to three minutes.
• 12:04
  SPEAKER_02: So the the you know the difference the of the runtime is is is pretty significant.
• 12:09
  SPEAKER_02: So so I I think that's the biggest challenge we we we had with the customer.
• 12:13
  SPEAKER_00: Yeah, yeah.
• 12:14
  SPEAKER_00: That makes sense.
• 12:16
  SPEAKER_00: Cool.
• 12:17
  SPEAKER_00: So changing track a little bit, you mentioned before that one of the things that Airflow has allowed you to do is provide data directly to your customers, and it sounds like Airflow is kind of key, like wrapped up within that product.
• 12:32
  SPEAKER_00: What does that actually look like?
• 12:34
  SPEAKER_02: The teams that are providing the data, we we call them domain teams, these are the developers, right?
• 12:40
  SPEAKER_02: So so when they provide the data, they schedule the full dbt experience.
• 12:46
  SPEAKER_02: So from the source to to the to the result, we call it marts, but whatever you call it, the the final layer that we want to share with with the with the customers.
• 12:56
  SPEAKER_02: And because you have the whole flow through through the triggers and and through what we call the data sets, right?
• 13:04
  SPEAKER_02: All the features that Airflow Airflow offers, we know exactly when hap when what happens, and then you can control it you do you have the SLA, is it on time, you know, all the features you need to have for such product when the customer actually pays for the delivery of the data on time and in certain format and so on.
• 13:24
  SPEAKER_02: And so because you have the whole chain in Airflow, we can just hook yet another DAG, and the DAG is configurable through YAML, JSON, whatever is there.
• 13:35
  SPEAKER_02: And then you just add, you know, this customer wants to have this specific data set with their data delivered to this specific destination with Snowflake, S3, you name it.
• 13:46
  SPEAKER_02: And then you just run the DAG after the whole flow is ready and you just deliver it directly directly to the merchant, to the customer.
• 13:54
  SPEAKER_02: Which is which is great because when you stay in the Airflow environment, we have it set up, I'd say pretty securely.
• 14:00
  SPEAKER_02: That there is like no it's pretty air gaped, right?
• 14:04
  SPEAKER_02: Like we we don't have any security issue with that.
• 14:07
  SPEAKER_02: Even sharing the secrets and stuff like that, it's already handled in Airflow, then then you just you just plug it in and it works, and everybody's happy, and the customer is a happy, happy customer.
• 14:18
  SPEAKER_02: So it's a pretty cool use case, I guess.
• 14:22
  SPEAKER_00: Yeah, that's great.
• 14:23
  SPEAKER_00: And so the so the customer themselves isn't themselves isn't interacting with your Airflow instance or an Airflow instance that you manage, but Airflow is being used to just deliver the data to whatever the source might be, and it might be different per customer.
• 14:38
  SPEAKER_00: Is that right?
• 14:39
  SPEAKER_00: Yes, it is.
• 14:40
  SPEAKER_00: It is.
• 14:41
  SPEAKER_00: Yeah, very cool.
• 14:42
  SPEAKER_00: I can definitely see where airflow would be pretty key in a use case like that over, you know, some sort of UI-based tool that it might be a lot harder to plug into a different destination system.
• 14:54
  SPEAKER_02: Yeah, yeah.
• 14:55
  SPEAKER_02: You you can we try to do it before in in a different tool, but you cannot track what's happening with the data at the specific time, or it's too expensive, you know, or you have so many hacks that you need to do to actually deliver it.
• 15:07
  SPEAKER_02: So Airflow is pretty straightforward in the sense.
• 15:10
  SPEAKER_00: Yeah, that's cool.
• 15:11
  SPEAKER_00: You mentioned that you have SLAs set up as part of that process so that things are delivered on time or you're aware if they're not.
• 15:18
  SPEAKER_00: How are you doing that in Airflow?
• 15:21
  SPEAKER_02: Each step in the chain has a specific alerting.
• 15:24
  SPEAKER_02: We use Obgeny or Obgeny's actually ending.
• 15:27
  SPEAKER_02: So we're moving to data doc alerting.
• 15:30
  SPEAKER_02: But but so we have all the steps hooked per team, per the owner.
• 15:35
  SPEAKER_02: Actually, we have we have it hooked to this alerting system.
• 15:38
  SPEAKER_02: So based on the importance or gravity of the specific DAG, we just alert the specific team, and and then you have if it's the most important, you have the P1 incident, right?
• 15:51
  SPEAKER_02: But thanks to the because it's very standard flow as any other thing in software development lifecycle, the developers are just used to it.
• 15:59
  SPEAKER_02: So they know you know if they receive the alert, they can react immediately, and then you know, the whole flow, you know, you need to alert the merchant if you're not fulfilling SLAs and something you are proactive and you can go.
• 16:11
  SPEAKER_02: So Airflow just allows to do customize it as much as you want and allows to do all of that.
• 16:19
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:28
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:40
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:49
  SPEAKER_00: Yeah, okay, that's great.
• 16:51
  SPEAKER_00: Part of why I was curious was they know Airflow 3.2 was just released and they have the kind of GA or um moving out of experimental phase for the deadline alerts feature, which replaced the old like legacy SLA feature in Airflow.
• 17:05
  SPEAKER_00: So there is some built-in SLA capabilities now, although I haven't played around with it too much myself yet.
• 17:11
  SPEAKER_00: But anyways, was just curious kind of how how you'd implemented that.
• 17:15
  SPEAKER_02: We are yet to get there.
• 17:16
  SPEAKER_02: Uh we are still on the 2.10.
• 17:20
  SPEAKER_00: Okay.
• 17:20
  SPEAKER_00: Okay.
• 17:21
  SPEAKER_00: Nice.
• 17:22
  SPEAKER_00: Well, that'll be something to play with maybe once you've upgraded.
• 17:25
  SPEAKER_00: Exactly.
• 17:26
  SPEAKER_00: Yeah, cool.
• 17:27
  SPEAKER_00: Uh well, one other thing that I really wanted to dig into that you mentioned a little bit earlier was you're experimenting with AI and Airflow.
• 17:36
  SPEAKER_00: And obviously a very hot topic.
• 17:38
  SPEAKER_00: I'm curious if that means you're using, you know, AI to write DAGs, or what my team refers to as like the AI for Airflow, or is it that you're orchestrating AI workflows with Airflow, or is it both?
• 17:52
  SPEAKER_00: We do everything you mentioned.
• 17:54
  SPEAKER_02: So yeah, exactly.
• 17:57
  SPEAKER_02: So so uh we are definitely authoring the DAGs with Airflow, and we are trying to have the skills, uh, what we call skills today, right?
• 18:07
  SPEAKER_02: And the and the plugins to to help with that, and to have very specified way, governed way how to do it with AI.
• 18:14
  SPEAKER_02: And that that proves to be pretty useful.
• 18:17
  SPEAKER_02: To write the DAG is even quicker than it was before.
• 18:20
  SPEAKER_02: So that that's one way.
• 18:21
  SPEAKER_02: Uh alterning the deck is is the one way.
• 18:24
  SPEAKER_02: The other way is orchestrating AI workflows with the deck itself, because again, like you can you can plug it in anything, you can plug Airflow and have have it control any runtime, right?
• 18:37
  SPEAKER_02: So we have lambdas, we have we have containers that we are controlling with Airflow.
• 18:42
  SPEAKER_02: And then within the runtime, we are doing the actual AI work with the tokens and everything.
• 18:49
  SPEAKER_02: Secrets are already there.
• 18:50
  SPEAKER_02: What I mentioned before, all the features of Airflow are already there, which is leveraging that to run the AI use cases.
• 18:59
  SPEAKER_02: And as as a the the these were the two things, and also the the the last thing is having the visibility into everything we do in Airflow with AI, because it's a code, that's definitely the third very important use case we have.
• 19:17
  SPEAKER_02: Because for one thing I want to mention, and that's I believe astronomers' work because uh I think Caxel work on that is today if you use his skill and you want to upgrade from 2.10, what we all do, to 3.1 or whatever, then you just type one command and ask, what do I need to do?
• 19:40
  SPEAKER_02: You know, yeah, and everybody who listens to the podcast and and had at least one experience, how painful it was to upgrade to anything, uh exactly knows what I'm talking about.
• 19:51
  SPEAKER_02: So so so like I'm I I even though it it might seem like it's a very obvious use case of the AI and Airflow, I still need to mention it because I think it's so powerful.
• 20:01
  SPEAKER_00: Yeah, absolutely.
• 20:02
  SPEAKER_00: That's a you're referring to Astronomers' AI agent tooling, which is our skills repo that is open source, so it's available for anybody to use.
• 20:10
  SPEAKER_00: There's a whole bunch of skills in there now.
• 20:12
  SPEAKER_00: My team has contributed to some of them, so we're all really excited about trying to make these things easier.
• 20:17
  SPEAKER_00: But yeah, the upgrading one you mentioned is huge.
• 20:20
  SPEAKER_00: I we've seen obviously doing a major upgrade, even with all the work that goes into the you know open source project to make that an easier process or as easy as possible.
• 20:30
  SPEAKER_00: Like it's still hard for people who have large, you know, amounts of DAGs and big Airflow instances.
• 20:35
  SPEAKER_00: So yeah, I'm excited to hear that you think that it's been useful for your own process because I think we're yeah, we're really excited internally about you know just how much this can enable people to do with Airflow when you don't have to spend hours and hours manually changing everything or debugging what's going wrong or, you know, all that.
• 20:56
  SPEAKER_00: Absolutely.
• 20:57
  SPEAKER_00: Yeah, yeah.
• 20:58
  SPEAKER_00: That's really cool.
• 20:59
  SPEAKER_00: In terms of the, you know, actually running AI workflows with Airflow, another thing that we're really excited about.
• 21:07
  SPEAKER_00: An astronomer and my team definitely spends a lot of time kind of playing around with this stuff and talking about it for the community because it's not always the first thing that comes to mind for Airflow users.
• 21:17
  SPEAKER_00: But as you've described, I think Airflow is very well suited to it.
• 21:22
  SPEAKER_00: Just curious, where where in that project are you?
• 21:26
  SPEAKER_00: Like what does it what does the path forward look like?
• 21:29
  SPEAKER_00: I assume that it's still relatively early stages as it is for most companies.
• 21:33
  SPEAKER_00: But what are you looking forward to?
• 21:35
  SPEAKER_02: With all the AI stuff, I think it's different every week.
• 21:40
  SPEAKER_02: So yes, to answer this question is really hard.
• 21:43
  SPEAKER_02: But I think orchestrating most of the things that we're trying to right now implement into the product, like not even well, just on the back end, but what the what the merchant sees or the customer sees is definitely the future.
• 21:59
  SPEAKER_02: So there's more and more of these uh implemented within the Airflow ecosystem.
• 22:04
  SPEAKER_02: I think that will become the standard.
• 22:07
  SPEAKER_02: So we will rely on that more and more.
• 22:09
  SPEAKER_02: That that's that's the number one thing.
• 22:12
  SPEAKER_02: The other thing is you know, the the AR platform platforms they develop so fast that that the only thing you you need to do is uh somehow keep up and have it governed.
• 22:24
  SPEAKER_02: And because I have also security under myself, so so that that's uh that's another another thing that we we're trying to figure out as we speak, like how to how to govern all the tools and and and all the possibilities we have.
• 22:38
  SPEAKER_02: So I think Airflow will play a huge role in this because I see it even today.
• 22:44
  SPEAKER_02: We need to control, you know, how what did you spend, what did you do, or what did you expose, what you didn't control, and and and all this stuff.
• 22:53
  SPEAKER_02: And and so I see Airflow as a huge enabler to do that and to schedule regular checks and to to be on top of things.
• 23:01
  SPEAKER_02: So that that that's what I think is is the future for us with with Airflow and the AI.
• 23:07
  SPEAKER_00: Yeah, I love that.
• 23:09
  SPEAKER_00: So much to think about there.
• 23:10
  SPEAKER_00: And I think that's a good tie-in to my standard final question that I always ask guests, which is what would you most like to see from the Airflow project in the near future?
• 23:19
  SPEAKER_02: That's that's not an easy question.
• 23:22
  SPEAKER_02: I I I I really love I have everything I I need right now in the Airflow.
• 23:26
  SPEAKER_02: I I as a joke, I would say I remember so many heated discussions about the multi-tenancy.
• 23:33
  SPEAKER_02: So so that would be just just a joke.
• 23:36
  SPEAKER_02: I I think we don't need it because we have so many possibilities to to do it uh otherwise, like the multi-tenancy today.
• 23:42
  SPEAKER_02: But that that's just one thing I I can think of from from the previous discussion, the next feature.
• 23:47
  SPEAKER_02: But other than that, other than that, I think the community is real life and and anything you can think of is is out there when you before even before you think of it.
• 23:57
  SPEAKER_02: So that's really great.
• 23:59
  SPEAKER_00: Yeah, I honestly always love it when the answer to this question is kind of nothing.
• 24:04
  SPEAKER_00: Like happy users, yeah.
• 24:06
  SPEAKER_00: We we love to hear about that.
• 24:07
  SPEAKER_00: That's great.
• 24:08
  SPEAKER_00: Awesome.
• 24:09
  SPEAKER_00: Well, Philip, this has been really interesting.
• 24:11
  SPEAKER_00: I really appreciate you coming on and talking about all of this.
• 24:14
  SPEAKER_00: Uh, it's been so cool to hear more about your use case and what you're doing with Airflow.
• 24:19
  SPEAKER_00: Uh, just to close us out, what is the best way for folks to get in touch with you?
• 24:22
  SPEAKER_00: LinkedIn, definitely.
• 24:24
  SPEAKER_02: Uh in the search my name and that that's that's the preferred way.
• 24:27
  SPEAKER_00: Yeah, great.
• 24:28
  SPEAKER_00: We'll throw that in the show notes.
• 24:30
  SPEAKER_00: Um, well, thank you again for coming on.
• 24:32
  SPEAKER_00: This has been a really great discussion.
• 24:34
  SPEAKER_00: Thank you for having me.
• 24:35
  SPEAKER_00: It's it's been a pleasure.
• 24:38
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:44
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:48
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.