Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik

Transcript

Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: What Airflow does well over here again is like you can define your MCP in the same way where Airflow defines its secrets, which is connections.
• 0:08
  SPEAKER_02: So you could define your entire set of MCP.json file inside an Airflow connection, and this will be utilized by the tool sets that we provide for you to utilize with Pidentic AI or just the command.ai provider.
• 0:23
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:36
  SPEAKER_00: Okay, hi everyone.
• 0:37
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:39
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by two guests, Kaxel Nike, who is a PMC member for Apache Airflow and a senior director of engineering at Astronomer, and Pavan Gopidesu, who is a lead data engineer at Experian and also a PMC member for Apache Airflow.
• 0:57
  SPEAKER_00: Kaxel, Pavan, how are you both doing?
• 1:00
  SPEAKER_02: Very good, very excited to talk about the command that AI provided and all the stuff that we are doing at the Airflow project.
• 1:08
  SPEAKER_03: Yeah.
• 1:08
  SPEAKER_03: We are very excited to talk about this one.
• 1:11
  SPEAKER_03: Yeah.
• 1:12
  SPEAKER_03: Looking forward to discuss more on this.
• 1:15
  SPEAKER_00: Yeah, absolutely.
• 1:16
  SPEAKER_00: Well, lots of exciting stuff going on in Airflow right now.
• 1:19
  SPEAKER_00: And of course, as you just said, so for today's podcast, we're going to discuss Airflow's new common AI provider.
• 1:26
  SPEAKER_00: Really, really exciting topic.
• 1:28
  SPEAKER_00: This is a huge milestone, I think, for the project.
• 1:30
  SPEAKER_00: But before we get into that, I want to start with some background on both of you.
• 1:34
  SPEAKER_00: Can you tell me what you do in your day jobs and then how you're involved with the Airflow project?
• 1:40
  SPEAKER_02: Sure.
• 1:40
  SPEAKER_02: I can I can go first.
• 1:42
  SPEAKER_02: Uh I've been involved with the Airflow project since nine, eight to nine years, I think now.
• 1:48
  SPEAKER_02: Been a release manager, have been contributing about various different things, Airflow 2.0, early Airflow 1.x days, and recently from last year, Airflow 3.
• 1:58
  SPEAKER_02: I work on open source projects most of my time.
• 2:01
  SPEAKER_02: And as part of my role at Astronomer, I also work on our product side and commercial strategy again around AI and making Airflow and Astronomer more uh agentic in nature.
• 2:13
  SPEAKER_03: Thanks, Katsel.
• 2:14
  SPEAKER_03: I think I think I started in around 2024.
• 2:18
  SPEAKER_03: It just started with some POC we were doing at my R, like where we wanted to experiment some of the Lambda functions.
• 2:25
  SPEAKER_03: We were doing some like Lambda operators.
• 2:27
  SPEAKER_03: There is some one of the future is missed.
• 2:30
  SPEAKER_03: Like that's when I started.
• 2:31
  SPEAKER_03: So it just is a small like a contribution to the project.
• 2:34
  SPEAKER_03: Then it feels me excited.
• 2:37
  SPEAKER_03: Like, okay, this is nice.
• 2:38
  SPEAKER_03: So I have to still like continue contributing.
• 2:40
  SPEAKER_03: That's how I started.
• 2:42
  SPEAKER_03: And yeah, so it's been almost more than two and a half years with airflow, and I'm still continuing contributions.
• 2:50
  SPEAKER_03: And yeah, I'm also had uh my org like experience I'm a late data in the where we work on building analytics platforms within the experience business net with uh different business units can use those platforms to onboard their assets.
• 3:06
  SPEAKER_00: Yeah, okay, great.
• 3:07
  SPEAKER_00: Uh it's crazy to think about everything that has happened with Airflow, even just in the past two years that you've been working on it and Caxel, certainly in the last eight or nine.
• 3:16
  SPEAKER_00: I yeah, I'm kind of closing in on probably close to seven years working with Airflow myself.
• 3:23
  SPEAKER_00: And I always think like it's sort of, you know, at any point if you ask me, I would say like now is the most exciting time in the project.
• 3:29
  SPEAKER_00: That's certainly true right now.
• 3:31
  SPEAKER_00: Um, so just keeps getting better.
• 3:34
  SPEAKER_00: Uh, related to that, definitely want to dig into this new common AI provider, which is what we're here mainly to talk about.
• 3:41
  SPEAKER_00: Obviously, as I said, lots going on in the Airflow project, but we'll keep this one relatively focused.
• 3:46
  SPEAKER_00: Uh, this new provider was introduced, I think, is AIP99.
• 3:50
  SPEAKER_00: Uh, and I want to start with just some background on kind of what it is and what problem it was solving, kind of what made you both want to introduce this and work on it.
• 4:00
  SPEAKER_02: Yeah, I can I can start for sure.
• 4:02
  SPEAKER_02: Um last year, I think last year or year before that, my God, uh, myself and Powen, we uh got talking because Powen had a real problem that he was trying to solve at his previous job.
• 4:13
  SPEAKER_02: And uh one of my goals was to make it very easy to utilize AI agents within Airflow.
• 4:19
  SPEAKER_02: Uh, there were two parts that I cared about how to make agents write better Airflow code and utilize Airflow.
• 4:27
  SPEAKER_02: And the second was how you could uh utilize Airflow itself to uh orchestrate AI agents, and then the problem that Pawan talked about resonated with things that I was doing on the private projects as well on the personal project side.
• 4:41
  SPEAKER_02: So we got talking, and then I let Pawan explain uh the concrete use case, and that that was the inception of it.
• 4:47
  SPEAKER_02: And then we said, okay, let's talk about it at Airflow Summit and see if it's uh resonates, if the problem resonates.
• 4:54
  SPEAKER_02: And the problem not only resonated, but we got a lot of people asking when is this going getting shipped?
• 4:59
  SPEAKER_02: I talked to a lot of teams afterwards and like okay, this is something great, let's do it.
• 5:04
  SPEAKER_03: Yeah, so um back in this, right?
• 5:07
  SPEAKER_03: Like in currently at as I said, like we are we are a core platform team in Xperian where we built data analytics platform, right?
• 5:16
  SPEAKER_03: So the platform whole purpose is to like onboard the data sets and uh uh then from there consume the assets, uh different business units in the org and also downstream like uh users.
• 5:29
  SPEAKER_03: So part of this one is right.
• 5:31
  SPEAKER_03: So it is you can think of it's a producer consumer model.
• 5:34
  SPEAKER_03: So we just provide a place to like different producers, they produce data sets and consumers like subscribe and use it.
• 5:41
  SPEAKER_03: So the key challenge is here, like how do we like make sure whether the data set is coming into the platform are like rightly aligned with the contracts?
• 5:51
  SPEAKER_03: So they have users have their own downstream systems to process after the consumer, right?
• 5:56
  SPEAKER_03: So if someone changes any schema, like the stats, a column, remove cycle, it's all like people definite, it gets breaks there, all their downstream systems, right?
• 6:05
  SPEAKER_03: How do you make sure as a platform like to make detect the changes early and break the system like uh offflow data flow into the platform?
• 6:13
  SPEAKER_03: So that's where the problem, like use case, like how do we like use?
• 6:17
  SPEAKER_03: Then I discussed with Fatcel, and uh so nowadays we know like agents are very good at like uh if you provide the right context, it like it basically detects I suggest right things to do in the next steps part of our data pipeline, right?
• 6:32
  SPEAKER_03: So that's how it started.
• 6:34
  SPEAKER_03: Overall, the problem is that's how it's yeah.
• 6:37
  SPEAKER_02: Yeah, and the pattern was I think very similar when I talked to a few teams, like they all when when they were using AI, uh it's like we built this nice prototype, we want to productionize it, but our secrets and stuff is all scattered in various different environment variables, very different places.
• 6:56
  SPEAKER_02: Like, is there a way I can productionize it?
• 6:58
  SPEAKER_02: And there were not really better tools for that.
• 7:01
  SPEAKER_02: And then, of course, it made sense like the tool that you have been using for the last 10 years to orchestrate all your data pipelines are the same tools that you could utilize for for this.
• 7:10
  SPEAKER_02: We just needed to provide them with better interfaces so that it feels completely native uh with how AI agents are designed.
• 7:19
  SPEAKER_02: And airflow already had the primitives around um production hardening of secrets, logging, monitoring, retries, and all the things that makes for a durable task execution.
• 7:31
  SPEAKER_02: So it just made sense that um we could make it into an um agent, we could make some agent primitives that makes it very seamless for folks to edit in their pipelines.
• 7:42
  SPEAKER_00: I think that's a great description.
• 7:44
  SPEAKER_00: And I like how you both touch on like Pavan, the aspect of it that this new AI functionality is so powerful for teams who are, you know, hopefully having to do less like manual intervention when things change and stuff like that.
• 7:56
  SPEAKER_00: But Caxil, to your point, I think like AI orchestration is kind of just like regular orchestration, like orchestrating everything else, you need sort of the same basic functionality.
• 8:06
  SPEAKER_00: And like Airflow has, you know, had that for a really long time and has obviously cemented itself as the standard.
• 8:12
  SPEAKER_00: So I love seeing those two things come together.
• 8:15
  SPEAKER_00: Tell me.
• 8:16
  SPEAKER_00: So I know at least at the time that we're recording this, the provider has just been released.
• 8:21
  SPEAKER_00: So it is out.
• 8:22
  SPEAKER_00: Anybody listening to this can use it, uh, which is very exciting.
• 8:26
  SPEAKER_00: Maybe tell me, at least in this first iteration, what what capabilities are in the provider right now?
• 8:31
  SPEAKER_02: Yeah.
• 8:31
  SPEAKER_02: Well, if you want to talk about the stuff that you are very excited about, and then I can tell about what I'm really excited about.
• 8:38
  SPEAKER_03: Yes.
• 8:39
  SPEAKER_03: So I'm really excited about the system, what we are really looking for, or like uh detecting the changes, right?
• 8:47
  SPEAKER_03: The schema come basically schema comparison, right?
• 8:50
  SPEAKER_03: You have a source system files and you have like physicist uh users which are using those data sets, right?
• 8:57
  SPEAKER_03: So when new data set comes, how do you like know like whether the new columns are or like uh that something is deleted or all these things, right?
• 9:06
  SPEAKER_03: So that's where we build like one of the operators is part of this first release where LLM schema compare operator, where what it does is right, so we have not like we have all the things which is used, which is already built in Airflow.
• 9:22
  SPEAKER_03: We just like connected all the dots.
• 9:24
  SPEAKER_03: How do you use like a proper way to detect the systems, right?
• 9:28
  SPEAKER_03: So uh you let's an example like let's say assume your data set coming to S3, that's a anywhere real pattern for most of the companies, like the data partners.
• 9:38
  SPEAKER_03: You coming to data into S3, and from there you want to process it right further.
• 9:42
  SPEAKER_03: And you have like example target as post-case.
• 9:46
  SPEAKER_03: And uh so this what it does is this schema comparator.
• 9:50
  SPEAKER_03: Under the hood, we use for data fusion, apart to data fusion, one of the frameworks we use.
• 9:56
  SPEAKER_03: So uh the data fusion like basically reads the data from the sources like S3 and it and it fetches the schema, and also uh on the other hood, like the airflow already have native DB hooks.
• 10:09
  SPEAKER_03: We use the DB hook to fetch the schema from the target database example, Postgres, and we provide both to the agent via schema help.
• 10:17
  SPEAKER_03: And the agent does the schema comparison, and the really nice thing is it clearly flags like what is the severity level.
• 10:24
  SPEAKER_03: Do we need really like a schema migration?
• 10:27
  SPEAKER_03: And it suggests also the all the queries what you need to run.
• 10:30
  SPEAKER_03: Uh, it also like uh differentiates where if you some things it suggests to in force example, let's say your target has like a where car, like example at like 200 cats and its source system always 100 cats coming.
• 10:43
  SPEAKER_03: It also suggests that kind of things, right?
• 10:45
  SPEAKER_03: So you may alter your query to change this because you don't need 200 characters, you always say source system is 100 cats.
• 10:53
  SPEAKER_03: So that's a really nice thing, right?
• 10:55
  SPEAKER_03: So think about like let's say you your hundred teams, like hundreds of teams on data sets are coming into your platform to manage this manually and write this code once, and you cannot reuse things, right?
• 11:08
  SPEAKER_03: You even though you write today a lot of like boilerplate code, you have to write the comparisons, but uh, with the help of agents, you you just provide the what information is right context, and the agent helps you to do all the things.
• 11:23
  SPEAKER_03: So that's one of the operator how like a schema compare operator.
• 11:28
  SPEAKER_03: Yeah.
• 11:29
  SPEAKER_03: I just want to add a point here.
• 11:30
  SPEAKER_03: Data fusion we use, like the reason behind is data fusion, like it's super fast, like this, like massively fast.
• 11:39
  SPEAKER_03: In like it is built on the Rust uh Rust and it has Python like bindings.
• 11:44
  SPEAKER_03: So with airflow, we are using Python bindings.
• 11:47
  SPEAKER_03: Most of the people already are aware of data fusion, I believe.
• 11:50
  SPEAKER_03: Like, let's we did a benchmarking, 50 million records, it just queried like within 10 seconds with Maclock different variants of aggregations.
• 11:59
  SPEAKER_03: You can think of speed.
• 12:01
  SPEAKER_03: Basically, you are not investing a lot of compute power, you are not investing a lot of money.
• 12:06
  SPEAKER_03: So, the goal, one of the goals also you don't want to like use compute power like to do while doing these tasks, right?
• 12:13
  SPEAKER_02: Yeah, I think my mind was blown when Powin told me that he did benchmark and send me message that this computer did in 10 seconds, and I was like, what?
• 12:22
  SPEAKER_02: That's amazing.
• 12:23
  SPEAKER_02: Um and like his use case to me, like I was first very, very, very excited about that, and was one of the motivations to say, okay, this is not some project that we are going to build for some user.
• 12:36
  SPEAKER_02: This is going to be utilized in production by Perwan and his team.
• 12:40
  SPEAKER_02: So we got real motivation, even before the Airflow Summit talk, and then Airflow Summit talk cemented the place because we talked with PayPal and various teams.
• 12:49
  SPEAKER_02: But back to the uh question around what I'm excited about.
• 12:53
  SPEAKER_02: Yes, that LLM schema operator uh or schema change operator is one of the things that I think probably all the data platform teams run into.
• 13:02
  SPEAKER_02: Uh, you've got data producers and consumers, and while there are contracts, they change sometimes without notice, and this sort of helps.
• 13:10
  SPEAKER_02: And I think this is complementary as well because um LLMs are good at writing SQL and uh when they're confined in within a box.
• 13:21
  SPEAKER_02: Like I don't, for example, I don't want an LLM to change the shape of my DAG yet.
• 13:27
  SPEAKER_02: I'm not comfortable with that, but I'm very, very comfortable with LLM taking care of one part of the box that I know what it will do, and that's the security guarantee that we want.
• 13:39
  SPEAKER_02: Take the power of Airflow uh as production uh ready tool and take the power of LLM and AI agents to do one part very, very well, and where we spend hours and hours, whether it's SQL queries, whether it's uh routing things from one place to other.
• 13:56
  SPEAKER_02: So I think this immense one of the use cases I was very excited about.
• 14:00
  SPEAKER_02: Now there are a few more things, and I'm trying to think which one I'm more excited about in this provider.
• 14:06
  SPEAKER_02: So, like one thing that um we added was hooked tool sets, which is we're thinking like Airflow has uh one of the main reasons that Airflow is very popular as an open source project is is integrations.
• 14:20
  SPEAKER_02: Like the we have got uh we recently released uh the registry which has like 99 providers and around 350 plus uh hooks that connects to an external system.
• 14:31
  SPEAKER_02: And we're thinking uh these agents are getting very good at tool calling.
• 14:35
  SPEAKER_02: Um, guess what tools are?
• 14:36
  SPEAKER_02: These are those are functions.
• 14:38
  SPEAKER_02: What are uh what are hooks?
• 14:40
  SPEAKER_02: Hooks are a way to connect to an external system with different functions.
• 14:43
  SPEAKER_02: So if you have an S3 hook, there will be list keys, read, uh blob, uh regen, read and object store, and all those methods.
• 14:50
  SPEAKER_02: So we said if we can give access to agents to that tools using all these hook and hook methods uh by providing guardrails.
• 14:59
  SPEAKER_02: So we do have something called allowed methods on hook tool sets, which will say the agent can utilize any methods on this as long as it is in this list.
• 15:09
  SPEAKER_02: So you can say, I only give I will only give this agent permission to list keys and read key method, but not delete key as an example.
• 15:17
  SPEAKER_02: So an agent uh we use Pydentic AI as one of the foundational libraries for the command.ai so that we can utilize multiple models.
• 15:27
  SPEAKER_02: But using that, the hook tool set will say, okay, the agent will have information to this model.
• 15:33
  SPEAKER_02: Now, what can it do for tool calls?
• 15:35
  SPEAKER_02: It can use S3 hook, it can use DB API hook to connect to databases, and now we can access these methods.
• 15:42
  SPEAKER_02: Awesome.
• 15:42
  SPEAKER_02: Done.
• 15:43
  SPEAKER_02: That way it will log properly, it will adhere to allowed methods.
• 15:47
  SPEAKER_02: So I'm not worried that it will delete something on my production system.
• 15:51
  SPEAKER_02: It will reuse all the connections, or you could create new connections for just agents, but then you get all the guarantees around retries and whatnot is within that.
• 15:59
  SPEAKER_02: So hook tool set is something that I'm really, really bullish on that uh actually cements a first class interface and makes airflow agent uh agentic because it combines the integrations and power of LM together.
• 16:13
  SPEAKER_02: That's one.
• 16:14
  SPEAKER_02: The second, the other one is the durable execution part where uh in one of the projects that we have in Astronomer, we recently blogged about it of how we utilize for impact analysis.
• 16:25
  SPEAKER_02: And it does around uh 35 to 40 tool use case, uh like tool uses every time.
• 16:32
  SPEAKER_02: Uh and I don't want that if it reaches 38th tool call and if it fails, it needs to start over all over again, and it will take 30 minutes and money wasted, time wasted, effort wasted.
• 16:45
  SPEAKER_02: So that uh if you just pass durable equal to true uh tool at LLM operators, or if you want to utilize it in various abstractions, what it will do is it will cache those tool calls, it will cache the LLM outputs and stuff, and it will resume from where you left off in the second try, which is very, very powerful.
• 17:05
  SPEAKER_02: It feels very simple, um, but it saves a ton of time, money, energy uh when when you are running it in production.
• 17:13
  SPEAKER_00: That's incredible.
• 17:14
  SPEAKER_02: Yes.
• 17:15
  SPEAKER_00: I think, and that's within within a task.
• 17:18
  SPEAKER_02: Exactly.
• 17:19
  SPEAKER_00: That's that's wild.
• 17:20
  SPEAKER_00: I mean, I think uh going you know, way back in like the older days of Airflow when we always recommended like keep tasks atomic for exactly that reason so that you didn't have to retry from a previous one.
• 17:30
  SPEAKER_00: But of course, if you're trying to coordinate a lot of different calls, that could mean you have a lot of tasks and a lot of like context to pass in between them.
• 17:38
  SPEAKER_00: So that's that's really, really cool.
• 17:40
  SPEAKER_02: Yeah, that that is one thing I'm really, really excited for sure.
• 17:44
  SPEAKER_02: Um, and I'm sorry, I I'll keep on saying that this is my favorite, this is my favorite, but the list is too long.
• 17:51
  SPEAKER_02: And the last one I promise is uh on that front.
• 17:55
  SPEAKER_02: I'm very happy that we have got abstractions that will cater to different people.
• 18:00
  SPEAKER_02: So um, like some that there is a group which loves operators.
• 18:05
  SPEAKER_02: Like I love operators, I don't want to care about hooks or I don't want to care about low-level interfaces.
• 18:10
  SPEAKER_02: We have got LLM operator, we have got LLM SQL operator, and those abstractions that will work for them awesome, awesomely, like easily, easy to use.
• 18:20
  SPEAKER_02: We have got good enough documentation.
• 18:22
  SPEAKER_02: Now we have got decorators as well for people who want to do pre- or post-processing and like the task flow API.
• 18:28
  SPEAKER_02: So task.agent as an example will help you with those.
• 18:32
  SPEAKER_02: All the operators that we have has a corresponding decorator.
• 18:36
  SPEAKER_02: So you can use it to utilize your task flow API.
• 18:39
  SPEAKER_02: The third part, which is less known of, which is you can utilize it within your current task flow API without the task.agent decorator.
• 18:48
  SPEAKER_02: So you can utilize your these agents and this entirety of abstractions within your Python operator or just add task um decorators by calling by creating Pydentic AI agent, but passing our tool sets uh like hook tool sets or SQL tool set within the tool sets argument that Pydentic provides.
• 19:10
  SPEAKER_02: And you'll still get all the niceties.
• 19:12
  SPEAKER_02: So that's the beauty of it that the all the hook tool sets are available with Pydentic AI within the constructs of Airflow.
• 19:19
  SPEAKER_02: And why this is interesting is like the three levels of abstractions goes from I don't need to worry about it versus ah, I can do more things with this, and I can have full control of this.
• 19:30
  SPEAKER_02: So it's like simplicity versus control, and we have got abstractions for all three levels, which is I think um I'm sure Porn will say the same that I'm proud of.
• 19:39
  SPEAKER_02: Yeah, yeah.
• 19:40
  SPEAKER_03: I mean, we have quite a nice abstractions.
• 19:43
  SPEAKER_03: I'll I just want to add another point, right?
• 19:45
  SPEAKER_03: Whatever we did, right?
• 19:46
  SPEAKER_03: All this, it has all guardrails, like SQL operator, agent operator.
• 19:50
  SPEAKER_03: If you use tools, it has all the god alone methods as Kachal mentioned, right?
• 19:54
  SPEAKER_03: If you use SQL operator, you can like say what kind of operations are allowed.
• 19:59
  SPEAKER_03: And under the hood, we use skillblot to detect if let's say if agent like agents can do many things, right?
• 20:06
  SPEAKER_03: If if it generates some delete query, right?
• 20:08
  SPEAKER_03: It basically detects all those guardrails on there.
• 20:11
  SPEAKER_03: If anyone wants to bypass it's an option, it's there, they can bypass, but by default, the guard else will protect uh any uh any anything that like breaks a system.
• 20:20
  SPEAKER_03: Like sorry by default, you can say like sol select by default, nothing insert, nothing delete all.
• 20:27
  SPEAKER_03: Yeah.
• 20:29
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 20:38
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 20:50
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 20:59
  SPEAKER_00: Yeah, that's great.
• 21:00
  SPEAKER_00: Wow, that is an impressive amount of functionality for this first release.
• 21:05
  SPEAKER_00: And I'm curious to know more about kind of the plan going forward for the provider.
• 21:10
  SPEAKER_00: So, Pavan, you mentioned that you were going to be using this in production.
• 21:14
  SPEAKER_00: Is that already happening?
• 21:15
  SPEAKER_00: And then maybe a follow-on question to that would be kind of where would you recommend folks get started considering where the provider is right now?
• 21:22
  SPEAKER_03: Yes, uh, we are in kind of discussions in our like on the phases of like how we move forward this one.
• 21:29
  SPEAKER_03: But on the other side, regularly I use for my personal task.
• 21:33
  SPEAKER_03: That's this provider.
• 21:34
  SPEAKER_03: So let's say if someone asks, like when data load completes, what's the query code?
• 21:38
  SPEAKER_03: I just have a hook tool set, like it just I just use to agent, just go and query from just simply Athena.
• 21:45
  SPEAKER_03: So it does for me a lot of manual efforts.
• 21:48
  SPEAKER_03: What I do, like I know need to log in paid bless, I don't have to do select query, right?
• 21:53
  SPEAKER_03: It's all agent for me with the airflow.
• 21:55
  SPEAKER_03: I use a lot of things, but uh, if so the discussions are going to like implement the One but yeah, we'll be there soon to get this into uh in live man.
• 22:06
  SPEAKER_02: I want to note a caveat that currently the provider only works with airflow three and above, and the eBay team tried to use it and they messaged me saying, Ah, we want to utilize it with airflow two, but uh we'll have to wait for airflow three or if you can add support for airflow two.
• 22:22
  SPEAKER_02: And I said, we'll see, we'll wait.
• 22:23
  SPEAKER_02: So, if folks who are listening to this podcast, if you give us signals that you are still stuck on airflow 2 for whatever reason, uh airflow 2.11, I hope at the very least, then let us know via some mechanism.
• 22:37
  SPEAKER_02: We could add 2.11 support if needed.
• 22:41
  SPEAKER_02: We don't want to, we want to keep it forward in Airflow 3.
• 22:44
  SPEAKER_02: But if there is high demand, we will supply to that need.
• 22:48
  SPEAKER_02: So currently it is still experimental, this provider.
• 22:51
  SPEAKER_02: Um, but that's the main reason of it being experimental.
• 22:54
  SPEAKER_02: We want a lot of feedback from you.
• 22:56
  SPEAKER_02: Send messages about what you like, what you don't like, what via what sort of things that you need for airflow 2.x support and stuff and good work on.
• 23:08
  SPEAKER_00: Yeah, that's great.
• 23:09
  SPEAKER_00: All right, I think that's a good tie-in uh to my standard final question.
• 23:13
  SPEAKER_00: We could of course keep talking about this all day, but we'll try and give keep everybody like within um the time I requested from you.
• 23:20
  SPEAKER_00: Uh so I I always ask my guests at the end, what are you what would you most like to see from the Airflow project in the near future?
• 23:26
  SPEAKER_00: For the purpose of this interview, I'm gonna tweak that a little bit and say, what are you most excited about with the common AI provider coming up?
• 23:35
  SPEAKER_03: Uh I think for me, right?
• 23:36
  SPEAKER_03: So there are quite a few things planned.
• 23:39
  SPEAKER_03: Uh I see people are started getting like into involved in this project, starting at APS coming, like Google ADK.
• 23:46
  SPEAKER_03: People are like, they want to integrate, and the next is LLM data quality operator is coming.
• 23:52
  SPEAKER_03: So uh it helps you to like run the quality rules on the flight, generates a rules and runs a new data test.
• 23:58
  SPEAKER_03: That's the most uh important thing.
• 24:00
  SPEAKER_03: And yeah, I'm really excited.
• 24:03
  SPEAKER_03: And most people are getting interested and started contributing.
• 24:07
  SPEAKER_02: For me, the answer is also two or three fold over here.
• 24:11
  SPEAKER_02: Just too excited about what what all is happening.
• 24:14
  SPEAKER_02: Like uh one thing we didn't touch upon uh is the MCP support.
• 24:18
  SPEAKER_02: So we do support MCP.
• 24:20
  SPEAKER_02: Uh MCP is another tool set as part of all this integration.
• 24:24
  SPEAKER_02: The good thing though is like you could like I always hated when I utilize plot code and stuff, and I need to define MCP in that mcp.json, or if I use cursor, I need to put the JSON and stuff.
• 24:36
  SPEAKER_02: And sometimes like JSON is still fine.
• 24:39
  SPEAKER_02: Sometimes I need to put credentials over there.
• 24:42
  SPEAKER_02: It was not long ago where they then started providing a way to inject your credentials, but that is still a lesser known fact.
• 24:50
  SPEAKER_02: But what Airflow does well over here again is like you can define your MCP in the same way where Airflow defines secrets, which is connections.
• 24:59
  SPEAKER_02: So you could define your entire set of mcp.json file inside an Airflow connection, and this will be utilized by the tool sets that we provide for you to utilize with Pydentic AI or just the command.ai provider, which means that any secrets will be redacted in logs and you get all the guarantees that connections provide.
• 25:19
  SPEAKER_02: So that is a great first step, I think, at least on the MCP support side.
• 25:23
  SPEAKER_02: And I'm excited on that sort of integrations where we could pair all the strong suits of airflow and combine it with what uh agent needs as well.
• 25:32
  SPEAKER_02: And like you said, Kenton at the start, uh it's like the commonal the commonality of data orchestration and AI orchestration is a lot more than people think it is.
• 25:42
  SPEAKER_02: Uh, you need all the guarantees that you needed for data orchestration or software in general, and all those principles apply.
• 25:49
  SPEAKER_02: And I want to strive towards getting that better and better.
• 25:54
  SPEAKER_02: That on like putting my software engineer head, of course.
• 25:57
  SPEAKER_02: But then on the AI side of things, again, uh Poen did a fantastic job with integrating human interloop story within within the realm of LLM operators and task.agent and stuff.
• 26:08
  SPEAKER_02: And I think we can do a lot more stuff on that.
• 26:10
  SPEAKER_02: Like there's already a foundation that Power added where you could go back and forth within the agent within the airflow deck.
• 26:17
  SPEAKER_02: So you can say it can generate SQL query, you could say, no, I want you to do this, it goes back to trigger and comes back, and you could say, no, no, no, but I wanted this, and you could iterate, and that's a great first step for airflow, and uh like that sort of a feedback cycle before the DAG is even completed.
• 26:33
  SPEAKER_02: So I think uh we can build more sort of those iterative feedback loops over here, and that is some things that I'm excited about, and I'm going to tie this into an AIP that's coming up, which is stateful set.
• 26:46
  SPEAKER_02: And that is where I feel the state management plus all the things that AI needs, common AI and statefulness will work well, very, very well together.
• 26:56
  SPEAKER_02: So I'm excited about that part.
• 26:58
  SPEAKER_03: I mean, when the AIP is posted, I really blow I I kind of like excited, like okay, this this is the time.
• 27:06
  SPEAKER_03: Like we have you have next steps for this to not not be other ties like what we have, what we had get what we have got the comments from part of the April proposal, like so, yeah.
• 27:18
  SPEAKER_00: Yeah, absolutely.
• 27:19
  SPEAKER_00: That will be huge for airflow.
• 27:21
  SPEAKER_00: Well, so much great stuff that we covered here, and I think there's a lot that hopefully folks will be excited to explore within the new provider.
• 27:29
  SPEAKER_00: We'll throw links to the documentation and the installation instructions and all that good stuff in the show notes.
• 27:35
  SPEAKER_00: Just to close us out, if anybody has feedback about the provider um or wants to help contribute, should they reach out to you in Airflow Slack in the Airflow repo?
• 27:45
  SPEAKER_00: What's your recommendation there?
• 27:47
  SPEAKER_02: Airflow Slack or GitHub issues, like if you need any quicker cycle, Slack is the best way to reach out to any of us.
• 27:54
  SPEAKER_02: There's a contributors channel as well.
• 27:56
  SPEAKER_02: So you are not uh at the mercy of a single developer.
• 28:00
  SPEAKER_02: But if you face or if you have any feature requests or feedback, GitHub issues are the best as well.
• 28:05
  SPEAKER_00: Okay, great.
• 28:06
  SPEAKER_00: Well, Pavan and Kaxel, thank you so much for coming on.
• 28:09
  SPEAKER_00: Really appreciate India.
• 28:10
  SPEAKER_00: This is really great work that you've been doing.
• 28:12
  SPEAKER_02: Thank you for having us.
• 28:13
  SPEAKER_02: It was really great.
• 28:17
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 28:23
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 28:27
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.