Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Transforming the Airflow UI for Cloudera’s Users with Shubham Raj

Transcript

Transforming the Airflow UI for Cloudera’s Users with Shubham Raj · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: So basically, how this works in the back end is the whatever Python or the bash box you have dropped upon.
• 0:04
  SPEAKER_01: It uses the operators, like Python operator or the bash operator.
• 0:08
  SPEAKER_01: And whatever you wrote dipped upon it, eventually the overall system converts in the Python operator, adjust the Python operator and uses it.
• 0:14
  SPEAKER_01: And then you draw the lines between the two of the boxes and create the lineage of whatever dependencies you have.
• 0:19
  SPEAKER_01: And we have multiple uh segments like you can have the configuration based upon the DAG level also and on the UI level also, on the task level also.
• 0:33
  SPEAKER_02: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:43
  SPEAKER_00: Hey Airflow community, Airflow Theory is here and we are taking it on the road.
• 0:48
  SPEAKER_00: Join the astronomer Rojo for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:56
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:05
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:10
  SPEAKER_00: Register now.
• 1:11
  SPEAKER_00: The link is in the show notes.
• 1:13
  SPEAKER_03: Welcome everybody.
• 1:14
  SPEAKER_03: Hello everyone.
• 1:15
  SPEAKER_03: Welcome to the Data Flowcast.
• 1:17
  SPEAKER_03: I'm your host, Mark Amati, and today I'm super excited because I'm joined by Shuba Mraj, Software Engineer 2 at Cloudera.
• 1:25
  SPEAKER_03: How are you doing?
• 1:26
  SPEAKER_01: I'm doing great, Mark.
• 1:28
  SPEAKER_01: Thank you for inviting me here.
• 1:29
  SPEAKER_01: I whatever you are doing is very recommendable, and this podcast series is actually very helpful for people who are very much related to the Airflow.
• 1:36
  SPEAKER_01: Thanks for having me here.
• 1:38
  SPEAKER_03: Awesome.
• 1:38
  SPEAKER_03: Thank you.
• 1:39
  SPEAKER_03: I'm actually very happy to have you in this podcast episode, Shubam, because I know that we met, I think it was in the last Airflow summit.
• 1:46
  SPEAKER_03: Yes.
• 1:47
  SPEAKER_03: I'm really happy to finally see you for this for this podcast.
• 1:51
  SPEAKER_01: Yeah.
• 1:52
  SPEAKER_03: Okay, so today it's very interesting because you will learn how a software engineer, so in that case, Shubam, started to contribute to Airflow.
• 2:00
  SPEAKER_03: So, you know, if you want to improve something in Airflow, then you will learn how to contribute to it.
• 2:04
  SPEAKER_03: Helping build Airflow three.
• 2:15
  SPEAKER_03: I can't wait to dive into what you're gonna say today, Shubam.
• 2:19
  SPEAKER_03: But first, could you start by introducing yourself and telling us about your role at CloudR?
• 2:24
  SPEAKER_03: Like what does your day-to-day work look like?
• 2:27
  SPEAKER_01: Sure.
• 2:28
  SPEAKER_01: So my name is Shubham, and I am currently working at CloudRa as a software engineer too.
• 2:32
  SPEAKER_01: I work with the CloudRa data engineering team, specifically in the private cloud domain.
• 2:37
  SPEAKER_01: We make a data scalable data platform where users can run their Spark and Airflow workloads without thinking about the infrastructure setups and all.
• 2:46
  SPEAKER_01: So we basically make the data engineers' life easier by having it on.
• 2:50
  SPEAKER_01: And I started contributing to Airflow because of this team only.
• 2:54
  SPEAKER_01: So I know get to know about the Airflow because we use, we provide it as a service, Airflow.
• 3:00
  SPEAKER_01: So I got to know about it from there itself.
• 3:03
  SPEAKER_03: Okay.
• 3:04
  SPEAKER_03: Just like for quick, just to get some numbers like, are you how big is your team today?
• 3:08
  SPEAKER_01: Are you just a small team or my team particularly deals with the platform?
• 3:13
  SPEAKER_01: We are, I think, six, seven people.
• 3:14
  SPEAKER_01: And there is also an Airflow team also, which is beside from my team, which also which specifically works with the Airflow components.
• 3:21
  SPEAKER_01: My team specifically was providing the platform where the integration of the Spark and the Airflow things are done.
• 3:27
  SPEAKER_01: And all over we provide a CD platform where come and just run the your Spark or Airflow workloads.
• 3:35
  SPEAKER_03: Okay, awesome.
• 3:36
  SPEAKER_03: So I guess one question for you is how did you first discover the Airflow?
• 3:41
  SPEAKER_03: And yeah, actually, I just would like to start by there.
• 3:44
  SPEAKER_03: Like, how did you discover Airflow for the when did you discover Airflow for the first time?
• 3:49
  SPEAKER_01: Yeah, so I think when I joined CloudRa, it was just my out of my college, I joined and learning about the what CD is, and I came two terms, Spark and Airflow, and I was excited more about what it does and how it happens, what is the background, what is the back end things out there.
• 4:04
  SPEAKER_01: So I was very interested to contribute to the Airflow and Spark.
• 4:09
  SPEAKER_01: It was an open source contribution, and since we know that CloudRa is also in like very much deals with the open source things.
• 4:14
  SPEAKER_01: So we were also encouraged to do that.
• 4:16
  SPEAKER_01: But it was like, I think in 2023 summit, where we were already presenting a talk with one of my colleagues.
• 4:23
  SPEAKER_01: So there, I think my first commit went in that workshop in 2023 Airflow Summit.
• 4:28
  SPEAKER_01: So of course, like whoever is listening to this, I will highly recommend if you are in 2025 AirPlus Summit, we always have a workshop there.
• 4:35
  SPEAKER_01: So please do have it and you can get your first commit in.
• 4:37
  SPEAKER_01: So yeah, I mean, the first commit went there in 2023, and I got to learn more about Airflow.
• 4:42
  SPEAKER_01: I already have some knowledge, but having it on the summit, seeing multiple people and learning more about what Airflow is, how different companies are using it, or how people are usually utilizing it, give me more in-depth knowledge about Airflow.
• 4:54
  SPEAKER_01: And finally, in 2024, when there was a lot of buzz about 3.0, how these things are happening, and like, no, this time I can't procrastinate, and this is definitely I'm going to get rid of more and more about Airflow 3.0.
• 5:06
  SPEAKER_01: And then I started my contribution and uh yeah, and things I started, I think, from the UI Airflow 3.0 UI migration.
• 5:15
  SPEAKER_03: Okay.
• 5:15
  SPEAKER_03: You actually discovered Airflow at Cladera, is that correct?
• 5:19
  SPEAKER_01: Yes, I do.
• 5:20
  SPEAKER_03: Okay.
• 5:21
  SPEAKER_03: Was it difficult, you know, to start with Airflow?
• 5:23
  SPEAKER_03: Did you find it hard to learn about the tool?
• 5:27
  SPEAKER_01: So basically, no, the learning the documentations are very nice.
• 5:30
  SPEAKER_01: I will highly say that documentation all night.
• 5:33
  SPEAKER_01: So coming to the first, like the user perspective.
• 5:36
  SPEAKER_01: So when I was writing my first diag, I remember so I was referring a lot of documentations and all, and I basically wrote a sing simple Python operator job, and which I was felt like okay, it was a good thing, and I did not find a lot of problems.
• 5:51
  SPEAKER_01: And coming to the development side of the Airflow, I think that is also like we have a breeze thanks to the community members, they have made it very easy to onboard new contributors who wanted to contribute.
• 6:01
  SPEAKER_01: So basically, my like journey with Airflow was very seamless.
• 6:05
  SPEAKER_02: Okay.
• 6:12
  SPEAKER_01: Both of them were very seamless.
• 6:13
  SPEAKER_03: Okay, that's good to hear.
• 6:15
  SPEAKER_03: And so, like, why did you decide to contribute to Airflow?
• 6:19
  SPEAKER_03: You know, what because for example, I love Airflow, but I never contributed to Airflow.
• 6:23
  SPEAKER_03: Maybe I should, but uh, but why did you decide to to contribute to this project?
• 6:28
  SPEAKER_01: Yeah, as I told, I I knew Airflow is there, Spark are there, and all these open source software are there.
• 6:35
  SPEAKER_01: But when this did the first talk which we gave in 2023, so when I met the community, when I met people, when I saw everything around, so I found the community very helpful.
• 6:44
  SPEAKER_01: And I thought, okay, this is might be might I start because this was my first open source project.
• 6:49
  SPEAKER_01: I wanted to get into it, so I decided okay, this might be the good chance.
• 6:53
  SPEAKER_01: And basically, I saw the community and that motivated me more.
• 6:56
  SPEAKER_01: And of course, the Airflow, what it does, it actually is one of the powerful tools which we have.
• 7:01
  SPEAKER_01: And nowadays, whatever things are happening in the AI and things, I think Airflow agic is more important.
• 7:07
  SPEAKER_01: So, all this there were a lot of reasons, and of course, the major reason was the power of Airflow and the community members that motivated you more to choose a project of like Airflow and to contribute.
• 7:19
  SPEAKER_03: Okay.
• 7:20
  SPEAKER_03: And can you actually because you said that you migrated the Airflow 3.oUI, actually, the 2.0UI to 3.oUI using React.
• 7:29
  SPEAKER_03: Can you actually um tell us more about that?
• 7:31
  SPEAKER_03: Like specifically, what you what did you do on the Airflow 3.oUI and how was it?
• 7:36
  SPEAKER_01: So I think I started with the uh the first one was like uh triggering trigger DAG DAG run.
• 7:43
  SPEAKER_01: So that form basically designed and then uh slowly it was like we the whole design now the pop-up comes up and the UI is uh newly designed, of course, and then we migrated the good feature like the params to the inherited the paremps and all.
• 7:58
  SPEAKER_01: Thanks to Gens also, like he was a maintainer of that, and we both combined and we actually migrated it to the new AirPro 3.
• 8:06
  SPEAKER_01: And post that I think I worked upon the variables, pools, connections pages, all of them, and solving here and there small UI bugs.
• 8:16
  SPEAKER_03: Okay.
• 8:17
  SPEAKER_03: Can you tell us more about the pool page?
• 8:20
  SPEAKER_03: Because I think pools are useful, but for some reason, many people don't even know about pools.
• 8:25
  SPEAKER_03: I think at least like whenever I publish something on pools, that doesn't work that much.
• 8:30
  SPEAKER_03: So can you tell us more about pools?
• 8:32
  SPEAKER_03: Like why are they useful and what did you modify on the UI?
• 8:36
  SPEAKER_01: So I think this is basically like people always remain with the default pool section, but do not, but I will highly recommend that.
• 8:42
  SPEAKER_01: It's one of the biggest good features of having it and limiting the making different pools for different things and you utilizing it.
• 8:50
  SPEAKER_01: And you might have also seen the you there is a new UI.
• 8:53
  SPEAKER_01: I think Brent and me were discussing about I think the new page, and we wanted to show we wanted to move for different from the tabular format to the new line format where it more see like how much is remaining and then you hover over it and see the numbers of it.
• 9:09
  SPEAKER_03: Yeah, yeah.
• 9:10
  SPEAKER_03: So this is great.
• 9:11
  SPEAKER_03: Actually, um, I definitely use pools, and so for those who don't know, uh, a pool is very useful if you want to change the I would say the concurrency of uh a subset of tasks.
• 9:20
  SPEAKER_03: And so what you did, Shubam, is super interesting because now you know at which state your tasks are in that pool.
• 9:26
  SPEAKER_03: So this is super uh super useful.
• 9:28
  SPEAKER_03: I'm glad that that this page changed in this way.
• 9:31
  SPEAKER_03: Yeah, I think it's also like one interesting feature that you contributed was the ability to control XCOMs externally.
• 9:39
  SPEAKER_03: So, can you tell us more about that?
• 9:41
  SPEAKER_03: Like, why did you uh decide to create this uh API endpoint for creating XCOMs?
• 9:48
  SPEAKER_01: Okay, so I we got to have an issue there which was telling that the users might have a problem.
• 9:54
  SPEAKER_01: Suppose you have a very long-running workflows and something failed because of some wrong XCOM coming from the upstream, and the only solution you might have is rerunning the that same task again, which is problematic in in for the long-running workflows and often sometimes expensive also to do that.
• 10:11
  SPEAKER_01: And so we wanted to have something which is not that hard for the users and which make their life much more easier in basically also in debugging, also the coming incident recovery from a certain point of suppose you're on call and you wanted to see how the downstream is performing and what issues are there.
• 10:29
  SPEAKER_01: You just hit the API and you just update the values of the XCOM and check what is the situation or what else we basically made very comfortable for the users to change the XCOM and play around it without changing the DAG file, going back the changing the DAC file and then again doing all that things again.
• 10:46
  SPEAKER_03: Yeah, which is very cool.
• 10:47
  SPEAKER_03: Honestly, like when I saw that new API endpoint, I was pretty happy.
• 10:50
  SPEAKER_03: I think we had a few endpoints before for managing XCOMs, I think.
• 10:55
  SPEAKER_03: But like the you know, being able to create XCOMs through the REST API is definitely something cool, and I'm glad that we have this new feature in Airflow 3.0.
• 11:03
  SPEAKER_03: So if you did not know about that, now you can create XCOMs uh through the REST API.
• 11:08
  SPEAKER_03: Was it harm?
• 11:09
  SPEAKER_03: Actually, I have a question for you, Shuba Mlack, because you contributed on different aspects of Airflow.
• 11:16
  SPEAKER_03: Which one was the most challenging?
• 11:18
  SPEAKER_03: Was it more challenging for you to work on XCOMs or on the UI or like how tell us more about that?
• 11:25
  SPEAKER_01: I won't say it was very easy for any of the things, or it was not very hard for the things.
• 11:30
  SPEAKER_01: So basically, I started with uh UI things, but eventually I wanted to take up on the back end also and once also wanted to understand the back end of how Airflow usually works about.
• 11:41
  SPEAKER_01: So getting into the when I got the opportunity of working on this XCOM creation and update of the APIs, so I took it up and I thought, okay, this is the great chance of understanding how XCOM works because this is something which flows from task to task.
• 11:55
  SPEAKER_01: And you for working on XCOM, you might you need to understand how is the process happening and how is that that value is giving from one task to coming to the another task.
• 12:04
  SPEAKER_01: And after this, I think I also worked upon bulk APIs for the variables and the pools page where you wanted to do the bulk operations and all.
• 12:12
  SPEAKER_01: So eventually, like I wanted to touch both parts of the aspect, do not wanted to limit myself to one aspect of it.
• 12:17
  SPEAKER_01: And still, there is a lot of things to take care of.
• 12:22
  SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:31
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:43
  SPEAKER_02: To see how you can maximize airflow's potential within your data ecosystem, visit www.astonomer.io.
• 12:52
  SPEAKER_03: Is it more challenging to work on the UI than on, for example, XCOMs where you know when you are a software engineer?
• 12:59
  SPEAKER_03: Because for example, I know nothing about frontend.
• 13:01
  SPEAKER_03: So, like, how is it?
• 13:02
  SPEAKER_03: Did you feel that it was more challenging to work on React, or for you it was just okay?
• 13:07
  SPEAKER_01: So I don't find it challenging.
• 13:09
  SPEAKER_01: I think it's rather than the front-end you might have different much opinions when you are asking reviews.
• 13:15
  SPEAKER_01: It can be opinions are can be vary from person to person.
• 13:18
  SPEAKER_01: Backends, you do not you more focus on the optimization part rather than the looks about.
• 13:24
  SPEAKER_01: But UI is very like you should know that how user might feel about it, how use it should be very helpful.
• 13:30
  SPEAKER_01: You should not make UI that hard that people you should be like go-to thing for the people to have it.
• 13:36
  SPEAKER_01: So I think that was the major difference I saw.
• 13:38
  SPEAKER_01: And it is not related to Airflow, I think it's all over the industry.
• 13:40
  SPEAKER_01: This back end or front end things works like right.
• 13:44
  SPEAKER_03: I mean, I think if I were you, I would have preferred uh back end than front end, but yeah.
• 13:51
  SPEAKER_03: Okay, I know, and actually I really encourage people to look at the presentations you did in the previous Airflow summit.
• 13:57
  SPEAKER_03: But I know that uh Cloud Ara, you've built a drag and drop editor for Airflow that generates tag without coding.
• 14:04
  SPEAKER_03: How does this work, technically speaking?
• 14:06
  SPEAKER_03: Can you tell us more about this project?
• 14:08
  SPEAKER_03: Because I'm sure many people will be interested by it.
• 14:11
  SPEAKER_01: Sure.
• 14:12
  SPEAKER_01: As I told that, CD offers Airflow as a part of their offerings, and we offer our users to create the tag into either you upload it as a form of file or otherwise you have we call it editor.
• 14:23
  SPEAKER_01: So when you select editor, we provide a user with a canvas and you have multiple options there: Python, bash, or CD job run operator.
• 14:31
  SPEAKER_01: CD job run is basically you are you have some of the Spark jobs on our platform.
• 14:35
  SPEAKER_01: You can also have it as a task there in the Airflow canvas.
• 14:38
  SPEAKER_01: So you just drag and drop, and uh you so basically how this works in the back end is the whatever Python or the bash box you have dropped upon.
• 14:46
  SPEAKER_01: It uses the operators, like Python operator or the bash operator, and whatever you wrote script upon it.
• 14:51
  SPEAKER_01: So it uses it eventually, it converts it into the overall system converts in the Python operator, adjust the Python operator and uses it.
• 14:59
  SPEAKER_01: And then you create just draw the lines between the two of the boxes and create the lineage of whatever dependencies you have.
• 15:06
  SPEAKER_01: And we have multiple uh segments.
• 15:08
  SPEAKER_01: You can have the configuration based upon the tag level also and on the UI level also, on the task level also.
• 15:13
  SPEAKER_01: And basically you can save it and then it goes to the back end and like it converts the tag definition, whatever you have, and to goes to a translation engine, and then it converts the Python code of it.
• 15:24
  SPEAKER_01: The translation engine is nothing sort of, it's just you have the most template, it uses that and it fills all the values there, and it basically generates the Python code.
• 15:33
  SPEAKER_01: And then, of course, then it uses to get it to into the folder from where the scheduler takes it out, like the however the it works.
• 15:41
  SPEAKER_01: And we basically designed this like a feature for the people.
• 15:44
  SPEAKER_01: Suppose we wanted to have the smooth onboarding for the customers who might might be possible that they are not very much aware or something who wanted to try their hands on.
• 15:53
  SPEAKER_01: So at the very first stage, we uh everyone does not know every how to write a tag, or may maybe they have very good understanding of Python code or bash, but there is small things which need to know about the airflow to write a tag.
• 16:05
  SPEAKER_01: We wanted to make that also a very smooth onboarding for our users.
• 16:08
  SPEAKER_01: We made this tag brag and drop so that kick POCs or some someone who wanted to do their small tags or something very like quickly, they wanted to so we wanted to have some solution which helped them a lot.
• 16:20
  SPEAKER_03: Okay.
• 16:20
  SPEAKER_03: This is super interesting.
• 16:21
  SPEAKER_03: And but I need to ask one question to you is as you know, Airflow has many operators, right?
• 16:26
  SPEAKER_03: It has many types of tasks.
• 16:29
  SPEAKER_03: How do you deal with that?
• 16:31
  SPEAKER_03: Like, can I create any type of task with your solution?
• 16:34
  SPEAKER_03: Or do I have just a set of type of tasks that I can use?
• 16:38
  SPEAKER_03: And if so, then how did you choose, like, for example, the Python operator and the SQL execute query operator and so on?
• 16:45
  SPEAKER_01: Yeah, so basically, we wanted, since I told you this, we wanted to have the at least the basics to the medium level of operations all should work upon.
• 16:53
  SPEAKER_01: So we choose for the Python and the bash operator, you can put all your code, whatever you wanted there, and it will make a Python function.
• 17:00
  SPEAKER_01: And as we know, all how Python operator works, right?
• 17:03
  SPEAKER_01: Python callable function, and then it uses it.
• 17:05
  SPEAKER_01: So our main target was at least people who are actually coming to the Airflow, they have a smooth onboarding.
• 17:11
  SPEAKER_01: So at least these things are supported.
• 17:13
  SPEAKER_01: And of course, we also support the CD job run operator, which is basically the people who have already had their Spark jobs on our platform who have used it.
• 17:21
  SPEAKER_01: So they can also use it on the canvas and create a lineage out of it and make a pipeline out of it.
• 17:28
  SPEAKER_03: Okay.
• 17:28
  SPEAKER_03: You mentioned that the DAGs are generated.
• 17:30
  SPEAKER_03: Basically, you have a script or something, right, that fills the values in templates.
• 17:37
  SPEAKER_03: Do you have like one file, one DAG, or do you have one file and multiple DAGs?
• 17:43
  SPEAKER_01: Basically, the full one canvas with it has a one DAG.
• 17:46
  SPEAKER_01: Okay.
• 17:47
  SPEAKER_01: So that particular comes comes to the one tag itself.
• 17:50
  SPEAKER_01: So whatever you create on one single canvas, it will be constructed to the one DAG.
• 17:54
  SPEAKER_01: And as I told, right, it uses the Most S templating to do that.
• 17:58
  SPEAKER_01: So decide that it can have multiple tasks, as I mentioned, you drag and drop it.
• 18:03
  SPEAKER_01: So one file, one tag.
• 18:05
  SPEAKER_01: Um you can say yeah.
• 18:07
  SPEAKER_03: Okay.
• 18:08
  SPEAKER_03: Is there any any way to see your solution?
• 18:10
  SPEAKER_01: Like, I think during the presentation you did, you actually show uh we some uh I think we got into the live demo.
• 18:16
  SPEAKER_01: Uh I think yes.
• 18:18
  SPEAKER_01: So I think you can have a look.
• 18:19
  SPEAKER_01: We can have shown it on the live demo itself.
• 18:22
  SPEAKER_03: Yeah, I mean, for those who are listening, I strongly encourage you to look at the solution that Shubam Dim did on his presentation.
• 18:29
  SPEAKER_03: Okay, I guess like another question to you, because like definitely most of our listeners are on the user side of Airflow, but you are more on the infrastructure side.
• 18:40
  SPEAKER_03: So, what are like what are some common patterns or uh pain points that you see from customers, and how does that influence the platform features that you build at Cloud Air?
• 18:51
  SPEAKER_01: Sure.
• 18:52
  SPEAKER_01: So I think this is a very repetitive process for any of the product to revolutionize and all that, getting the feedbacks and then what are the customer pain points and all.
• 19:00
  SPEAKER_01: And one of the recent pain points we received was having the custom Python packages.
• 19:05
  SPEAKER_01: For example, we as a part of our offering, we provide the airflow in the cloud native fashion.
• 19:10
  SPEAKER_01: Employer uses the Kubernetes to do that.
• 19:13
  SPEAKER_01: Suppose you are using your DAG is using some of the library, which is not part of the Airflow environment.
• 19:18
  SPEAKER_01: So basically, you need to have it on all of the parts of the scheduler to the workers and everything.
• 19:24
  SPEAKER_01: So by default, if you try to do it, you cannot do that.
• 19:26
  SPEAKER_01: So we part we gave a solution on the VC level.
• 19:29
  SPEAKER_01: VC, in our terms, are nothing but the virtual cluster.
• 19:32
  SPEAKER_01: It is like you have an isolated environment for all particular instances of an Airflow.
• 19:37
  SPEAKER_01: So you have it there.
• 19:38
  SPEAKER_01: So what we take is we take the requirements from the requirement.txt file from the customer, and they can just upload it on the VC level.
• 19:46
  SPEAKER_01: And that requirement can contain all of the tags, whatever you are going to run, and all of the third-party libraries or something, you can have it.
• 19:53
  SPEAKER_01: And you just give them the paper repository link and all the requirements of texting.
• 19:57
  SPEAKER_01: It basically tries to build everything and creates a BVC and then mount it to all of the pods wherever it is required.
• 20:03
  SPEAKER_01: So eventually you will have that particular Python packages in all of the pods running, and eventually we solved the problems, something like this.
• 20:10
  SPEAKER_01: So this was also like coming feedbacks and all things, and we one by one source try to solve each of our customers' problem.
• 20:17
  SPEAKER_01: Wow.
• 20:19
  SPEAKER_03: Did you find that the solution difficult to implement?
• 20:21
  SPEAKER_01: I personally did not work on particular solution, but uh the team worked upon it, and I think it was not that difficult, but yeah, of course, it's a very must-to-have uh solution because I we cannot say in production a lot of DACs have the third-party libraries we they use it, and it's a must-have solution for any of the customers to have it.
• 20:39
  SPEAKER_01: So we also now offer as a part of like it's very old releases we given it, so now we have it and now product.
• 20:48
  SPEAKER_03: Awesome.
• 20:49
  SPEAKER_03: Thank you so much for sharing all of that, uh.
• 20:51
  SPEAKER_03: I guess like I have one last question to you.
• 20:53
  SPEAKER_03: Uh, what do you wish to see in the uh upcoming Airflow releases?
• 20:59
  SPEAKER_03: So if there is one feature that you would love to see in the next airflow release, what would it be?
• 21:05
  SPEAKER_01: There was a lot of things in my mind till if you asked me this question six months back.
• 21:10
  SPEAKER_01: But we have Airflow 3.0, and I think we have near about covered a lot of things and things.
• 21:17
  SPEAKER_01: But one of the things we I think there's a lot of AIP discussions going on the thread, which involves one of the things I recently saw on the EIP thread was MCP servers, integrations, and all, and human in the loop.
• 21:27
  SPEAKER_01: And there's a lot of things which are going coming in Airflow, and I'm very excited about it, how that's going to look upon.
• 21:33
  SPEAKER_01: And maybe hopefully we are going to have a lot of summit talks in this year.
• 21:36
  SPEAKER_01: So I'm looking forward to it.
• 21:38
  SPEAKER_03: Yeah, me too.
• 21:39
  SPEAKER_03: I think the future is exciting.
• 21:41
  SPEAKER_03: Awesome.
• 21:42
  SPEAKER_03: Thank you so much for joining, Shubam.
• 21:44
  SPEAKER_03: And I guess what is the best way for folks to get in touch with you?
• 21:48
  SPEAKER_01: People can connect me on the LinkedIn, which is Tumura's official via handle.
• 21:51
  SPEAKER_01: And of course, on the I'm available on the Airflow Slack.
• 21:54
  SPEAKER_01: You can just search Shubam Dodge and uh you can just talk to me there.
• 21:58
  SPEAKER_03: Awesome.
• 21:59
  SPEAKER_03: Thank you so much again, Shubam.
• 22:00
  SPEAKER_03: It was a pleasure to have you, and I hope to see you soon for uh who knows another episode.
• 22:04
  SPEAKER_03: Thank you so much.
• 22:06
  SPEAKER_01: Chef, thank you so much, Mark.
• 22:10
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 22:16
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 22:20
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.