Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel

Transcript

Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: What are the hurdles we can take out from creating, managing, and scaling out pipelines that AI agents can help us to bits scale?
• 0:07
  SPEAKER_01: So we're super excited about those two bits client experience and innovational not.
• 0:13
  SPEAKER_02: BJ gave like a very uh awesome roadmap of everything that we're excited about for the partnership.
• 0:20
  SPEAKER_02: From my perspective, what I'm excited about is it's a very simple conversation for us to have with our customers.
• 0:26
  SPEAKER_02: The conversation usually goes something like this Hey, what'd your data stack look like?
• 0:36
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:46
  SPEAKER_04: Hey Airflow community.
• 0:48
  SPEAKER_04: Airflow Theory is here and we are taking it on the road.
• 0:51
  SPEAKER_04: Join the Astronomer Roadshow for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:59
  SPEAKER_04: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:08
  SPEAKER_04: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:12
  SPEAKER_04: Register now.
• 1:13
  SPEAKER_04: The link is in the show notes.
• 1:17
  SPEAKER_04: Hi everyone.
• 1:18
  SPEAKER_04: Welcome to the Data Flowcast.
• 1:20
  SPEAKER_04: I'm your host, Kenton Danis, and today I'm joined by two lovely folks from IBM, Ryan Yackle, who's a go-to-market strategy leader, and BJ Adisoji, who is a senior product manager.
• 1:32
  SPEAKER_04: Ryan, BJ, how are you both doing today?
• 1:35
  SPEAKER_02: Pretty good.
• 1:36
  SPEAKER_02: Doing awesome.
• 1:37
  SPEAKER_02: Thanks for having us, Kenton.
• 1:38
  SPEAKER_02: This is a lot of fun.
• 1:39
  SPEAKER_02: I'm glad you introduced us as lovely people.
• 1:41
  SPEAKER_02: I don't think I've ever been told I'm a lovely person.
• 1:44
  SPEAKER_02: I guess I don't know if that's that'd be that'd be kind of bad if I was ever viewed as lovely, but I've never been introduced as a lovely person.
• 1:51
  SPEAKER_04: I hope I'm not the first person who has said that to you.
• 1:56
  SPEAKER_04: No, I of course think all our guests are lovely, and I'm sure that you will uphold that throughout the duration of our episode.
• 2:03
  SPEAKER_04: Yeah.
• 2:03
  SPEAKER_04: All right, let's go ahead and dive in.
• 2:05
  SPEAKER_04: So for today's podcast, we're going to discuss how IBM's customers use Airflow and different methods for running Airflow in production.
• 2:12
  SPEAKER_04: Super excited to get into this because I think it'll be relevant to basically everybody using Airflow and a lot of our listeners.
• 2:20
  SPEAKER_04: But to start off, tell me a little bit about what the two of you do at IBM and what sort of customers you work with.
• 2:28
  SPEAKER_02: Yeah, so I'll go first.
• 2:29
  SPEAKER_02: So I'm a part of the data integration team over here at IBM.
• 2:33
  SPEAKER_02: So it runs the gamut of data replication, ETL, ELT, data observability, unstructured data.
• 2:40
  SPEAKER_02: I mean, anything really revolving around data movement and observability and data quality is kind of my wheelhouse.
• 2:47
  SPEAKER_02: I came over to IBM from a startup called IBM Data Band, which is a product that is a part of our technical suite as well now that just focuses on data observability.
• 2:56
  SPEAKER_02: That's kind of what I do.
• 2:57
  SPEAKER_02: I do a lot of go-to-market stuff and work with BJ a lot.
• 3:00
  SPEAKER_02: And I'm gonna pass it to him to let him know what he does.
• 3:04
  SPEAKER_01: Awesome.
• 3:04
  SPEAKER_01: Thank you, Ryan.
• 3:06
  SPEAKER_01: Um excited to be here.
• 3:07
  SPEAKER_01: I'm a senior product manager on the IBM Arts and X team, our data integration suite.
• 3:12
  SPEAKER_01: And I own the partnership and the relationship with Astronomer.
• 3:16
  SPEAKER_01: So from the vision to the strategy to actually launching our product back in March, that's something we're super excited about.
• 3:21
  SPEAKER_01: We think there's a huge opportunity for both our organizations to deliver lasting value to our clients.
• 3:28
  SPEAKER_01: And I think that's a great segue into why we think Airflow is important.
• 3:32
  SPEAKER_01: And I think one of the core in relation to what we see within our client base is Airflow is so ubiquitous within the client demography that we serve at IBM.
• 3:42
  SPEAKER_01: And it's just a core part of the ecosystem.
• 3:45
  SPEAKER_01: And so it was a natural segue for us to say, oh, let's explore the orchestration layer, right?
• 3:50
  SPEAKER_01: Let's find who the best players are there out there and let's try to partner with that.
• 3:53
  SPEAKER_01: So that's a core part of our strategy.
• 3:55
  SPEAKER_01: It's ecosystem friendliness.
• 3:57
  SPEAKER_01: So we're excited to be on this show to talk more about that and to double-click into what patterns we're seeing and how Airplay is being used at scale.
• 4:04
  SPEAKER_04: Yeah, that's great.
• 4:05
  SPEAKER_04: And for context, so the partnership that you're referring to is between Astronomer and IBM.
• 4:10
  SPEAKER_04: So we'll get more into how that helps folks run Airflow in production and at scale.
• 4:16
  SPEAKER_04: But I'm curious before we dive into that, if you can tell us a little bit more about the types of use cases that you see your customers working on and how Airflow fits in there.
• 4:27
  SPEAKER_02: Yeah, I'll start off and just say a lot, we saw a lot of customer use cases, even when I was at Databand before IBM acquired a Databan, just in terms of them running it either on on-premise environments or even with a managed solution like Astronomer.
• 4:42
  SPEAKER_02: We saw both four flavors of that in the market or in our customer base.
• 4:47
  SPEAKER_02: And I would say getting back to a little bit of the reporting you all did on the state of airflow report recently, I saw that basically Airflow had been primarily used for the longest time as like more of an internal for business analytics and dashboarding and things like that.
• 5:02
  SPEAKER_02: I think you guys said that it's 35% now.
• 5:05
  SPEAKER_02: People that have used it for more than five years are starting to do AI and ML use cases with Airflow now.
• 5:10
  SPEAKER_02: That's definitely what we see as well.
• 5:12
  SPEAKER_02: Like a couple of our customers that are out there today that use Airflow are we can name publicly are Autodesk and the Weather Company.
• 5:20
  SPEAKER_02: And they both use Airflow in both of those use cases.
• 5:23
  SPEAKER_02: So with Autodesk, they have a mixture of I think what a traditional setup of Airflow would be with like it's Airflow plus a bunch of other tools, just the weather company, but they're using things like Airflow plus Spark plus DBT plus Snowflake plus Python job.
• 5:38
  SPEAKER_02: They're using it across all of their analytics internally.
• 5:42
  SPEAKER_02: And with the weather company, they're using it as an orchestration solution for their SageMaker pipelines.
• 5:49
  SPEAKER_02: And they they're the the weather company is pretty interesting because you think, oh, they're using it for weather patterns and things like that.
• 5:55
  SPEAKER_02: Yes and no, but the people that we that use Airflow in our in our situation, our customer base, they're using it for predictive analytics around advertising.
• 6:04
  SPEAKER_02: Since you're pulling in the data, you're looking at the data, weather company has ads that they're on the weather company, weather channel, weather.com, all those areas.
• 6:12
  SPEAKER_02: What that allows them to do is use Airflow, orchestrate their ML stage maker pipelines to do predictive analytics.
• 6:19
  SPEAKER_02: And then that allows them to make business decisions based off of their those analytics.
• 6:23
  SPEAKER_02: Yeah, again, whether you're using it for ML or traditional, you'd say ETL or ELT processes.
• 6:28
  SPEAKER_02: We see those in both use cases for the customers.
• 6:30
  SPEAKER_02: And I'll pass it to BJ to talk more about some of the ones we're seeing currently right now.
• 6:35
  SPEAKER_01: Thank you for that, Ryan.
• 6:36
  SPEAKER_01: Before I double-click into the two clients I'm gonna mention, I was gonna just call out some of the patterns we typically see when it comes to like Airflow.
• 6:45
  SPEAKER_01: The first is it typically larger enterprises, right?
• 6:47
  SPEAKER_01: So they have dedicated data engineering teams to help them solve each of the use cases.
• 6:51
  SPEAKER_01: And they typically have like massive data volume needs, right?
• 6:55
  SPEAKER_01: So there's a lot of data to be processed, there's a lot of data to be analyzed, and this often leads to a segmentation.
• 7:00
  SPEAKER_01: So there's like the cloud-friendly teams, and then there's like the ones that are working on legacy systems and trying to build like a cohesive data strategy, right?
• 7:06
  SPEAKER_01: So Airflow is that glue that kind of tops on all the pieces to bring it together.
• 7:11
  SPEAKER_01: And the last but not the least pattern we see there is around the uh volume of regulation.
• 7:15
  SPEAKER_01: So I'm gonna mention two clients that are within that highly regulated space.
• 7:19
  SPEAKER_01: So from theme finance to insurance to healthcare, pretty sensitive data and pretty like strict compliance and audit reasons that makes them pay that to work Airflow for orchestration.
• 7:30
  SPEAKER_01: So for the anonymized clients, the first one is a national auto provider, like auto insurance provider.
• 7:37
  SPEAKER_01: And uh they run some of their most critical business uh workloads using Airflow.
• 7:42
  SPEAKER_01: Three of the projects that I can call out is the daily claims feature, right?
• 7:47
  SPEAKER_01: This is essentially helping them identify what claims are legitimate and which are high risk, and how can their teams and their team members respond to these, right?
• 7:56
  SPEAKER_01: That's super high value for an insurance provided.
• 7:58
  SPEAKER_01: They need to identify those quickly.
• 7:59
  SPEAKER_01: Next, they also do is in that same AI space, it's around the weekly churn rate analysis, right?
• 8:04
  SPEAKER_01: People know that people are constantly insurance shopping, right?
• 8:07
  SPEAKER_01: So how can they identify people that are going to turn suit and proactively offer them special discounts or special deals to keep them on board a little bit longer?
• 8:15
  SPEAKER_01: So those are like high value, high business value use cases for Airflow.
• 8:19
  SPEAKER_01: The last project we see them running with Airflow is for their insurance pricing dashboard.
• 8:26
  SPEAKER_01: Yet again, people are constantly shopping for pricing, right?
• 8:28
  SPEAKER_01: And there's a ton of data that goes into predicting how much your insurance is going to be from the city you live in to the car you drive to the mileage to your history, all these things they should be processed and run through their ML platform in order to get what price range they should quote you.
• 8:41
  SPEAKER_01: So the typical for this particular client, the data sources vary a bit, like from Snowflake to Hadoop, and even some object in S3.
• 8:48
  SPEAKER_01: And as I alluded to earlier on, the target here is like an ML platform called Domino that they use.
• 8:55
  SPEAKER_01: The second client case here is a major health insurance provider.
• 9:00
  SPEAKER_01: And for them, uh yet again, business critical workloads that are powered by Airflow.
• 9:06
  SPEAKER_01: One of that is the daily members lecture.
• 9:08
  SPEAKER_01: So imagine a scenario where you walk into a doctor's, you walk into a doctor's hospital or a clinic and you try to have a procedure done and they scan your card and it says, oh, you're not eligible and you know you're eligible.
• 9:18
  SPEAKER_01: That can create like a lot of friction that this client is trying to minimize.
• 9:22
  SPEAKER_01: So they do like daily snapshots of their members' eligibility as well as their billions and premiums every single day.
• 9:27
  SPEAKER_01: So these are this is like a really for one of the major providers in the US, this is like a lot, like tens of millions of people, or even hundreds, at scale being done every day.
• 9:38
  SPEAKER_01: The second project they leverage Airflow for is in the claims change data capture, right?
• 9:44
  SPEAKER_01: They're trying to identify uh what's a fraud, what's not a fraud, how can we get ahead of this, and how can we use this to proactively offer like better reimbursement to the valid claims.
• 9:53
  SPEAKER_01: Those are things that they power with Airflow.
• 9:56
  SPEAKER_01: The third project they also use here is the third project they power with Airflow within their organization is reporting.
• 10:02
  SPEAKER_01: So it's a big it's a big regulatory requirement from executive reporting to financial reporting and even like data privacy reporting.
• 10:08
  SPEAKER_01: And so these are pipelines that are powered by Airflow within this organization.
• 10:13
  SPEAKER_01: Their sources are also just as varied as the prior example I referenced.
• 10:17
  SPEAKER_01: And that's from like Prosgress to Mongo to DB2Z.
• 10:21
  SPEAKER_01: They have all sprawl of data, data systems that they need to pull data from, process data with, and feed into the centralized Cloudera data platform.
• 10:30
  SPEAKER_01: One thing to add here is in both of these cases, we see a pattern of like strict business requirements because these are not just hard hoc projects.
• 10:37
  SPEAKER_01: These are like business critical, right?
• 10:39
  SPEAKER_01: So they need the reliability, they need the security, they need the privacy, right?
• 10:43
  SPEAKER_01: Those are regular like conversations, like requirements that we surface doing these conversations as requirements, why they use Airflow and why they're looking to extend Airflow to provide all those things.
• 10:54
  SPEAKER_04: Yeah, absolutely.
• 10:55
  SPEAKER_04: These are great examples, and I think highlight the breadth of use of Airflow kind of across the world and basically every different industry.
• 11:02
  SPEAKER_04: So I love that you touched on a couple of different ones there.
• 11:06
  SPEAKER_04: Any one of those stories could be an episode in its own right.
• 11:10
  SPEAKER_04: Lots to talk about with how Airflow is used for each one of those use cases.
• 11:14
  SPEAKER_04: But I think one common thread that it would be interesting to talk about is kind of how organizations like that are running Airflow.
• 11:22
  SPEAKER_04: So at Astronomer, we've seen this kind of time and time again that companies like the ones you've described, especially in highly regulated industries, for them, the decision of how to run Airflow has a huge impact on the teams that are implementing it so often, data engineering or data science teams.
• 11:39
  SPEAKER_04: Can you talk to me a little bit more about that and some of the challenges that you've seen your customers run into when trying to manage Airflow at scale for those really critical use cases?
• 11:51
  SPEAKER_02: Yeah, I would say the one of the things to also know is when we talk about current IBM customers, they're a lot of them are obviously massive enterprises and, like you said, heavily regulated.
• 12:01
  SPEAKER_02: Lots of banks and insurance companies that we deal with as well.
• 12:05
  SPEAKER_02: And what you'll see is like there's tons of different tools that are inside of that organization.
• 12:11
  SPEAKER_02: And if they're pretty much doing like a hybrid environment where they have some in the cloud, some on-prem, they're pushing data back and forth between these kind of different environments.
• 12:20
  SPEAKER_02: They're gonna have some version of airflow in their environment.
• 12:25
  SPEAKER_02: Like we know that it's like you said it's I think something crazy, Kenton, like 35 million downloads.
• 12:30
  SPEAKER_02: Is it a day that Airflow has?
• 12:32
  SPEAKER_02: That's a month.
• 12:33
  SPEAKER_02: A month, sorry, yeah.
• 12:34
  SPEAKER_04: Still a pretty crazy number, but yeah, sorry, that would have been 300 million.
• 12:37
  SPEAKER_02: Okay, sorry.
• 12:38
  SPEAKER_02: So went a little too crazy there.
• 12:39
  SPEAKER_02: But yeah, come on, that's still an insane amount why Airflow is the most popular data pipeline tool, right?
• 12:44
  SPEAKER_02: And so we know that Airflow is in a lot of these different organizations alongside of IBM technology, other technology they're using alongside IBM, it's all over the place, right?
• 12:54
  SPEAKER_02: But I will say, like one of the big things that comes into play is because it's so separate and because it's got different lines of business using it, maybe, and other lines are not using it.
• 13:03
  SPEAKER_02: Things like upgrading and the configuration of Airflow and the centralization of administration of Airflow, that's a problem.
• 13:10
  SPEAKER_02: I know that you used to be, I think an Airflow administrator or a highly skilled Airflow developer.
• 13:16
  SPEAKER_02: I'm sure you still are.
• 13:17
  SPEAKER_02: But I think you mentioned you've dealt with a lot of those problems as well with uh admin configuration and things like that.
• 13:22
  SPEAKER_02: Also, the security aspect, right?
• 13:24
  SPEAKER_02: We are IBM, we have to have a very clear posture of security when we're deploying software into all these enterprise customers.
• 13:30
  SPEAKER_02: And they'll run into security issues when it comes to managing the open source version of Airflow.
• 13:35
  SPEAKER_02: Things like role-based access controls and single sign-on, all these things play into hey, should we like look to maybe do something else with Airflow?
• 13:44
  SPEAKER_02: How can we upgrade our airflow in a way that's not gonna break a bunch of stuff or step on anybody's toes?
• 13:49
  SPEAKER_02: Like that's a problem.
• 13:49
  SPEAKER_02: And then I would say the other one is like the deployment, resiliency, rollbacks, all the things that go into if you have to upgrade airflow or if you need to roll back something, all those things play into the administration part that is just annoying, right?
• 14:04
  SPEAKER_02: Now, if you're a small team, then that's fine.
• 14:07
  SPEAKER_02: If you're running 4,000 plus airflow DAGs, that's gonna be a problem, which is an actual use case we have with one of our customers that's using alongside our IBM data stage product.
• 14:18
  SPEAKER_02: But yeah, I mean, those are all kind of the main things that I would say that kind of pop up.
• 14:21
  SPEAKER_02: I know BJ, you have a lot more you could talk to, but those are some of the main things that pop up that we would love to help them out in that case.
• 14:29
  SPEAKER_01: Yeah, and I agree with you, Ryan.
• 14:30
  SPEAKER_01: Like those are the super critical things that we see.
• 14:33
  SPEAKER_01: And when it comes to our actual clients, I think to add to what you've said is broken down into three main points, right?
• 14:39
  SPEAKER_01: There's the deploying, like just deploying airflow skill can be a hassle, right?
• 14:43
  SPEAKER_01: When you to your point, oh well, not just a small team or like a small startup, but you can but if you have to think about the current data sprawl, right?
• 14:50
  SPEAKER_01: Every single team is looking at a different or ML, AI, dashboarding project, and they have like different data sources requirements.
• 14:56
  SPEAKER_01: Providing that level of abstraction and that level of control that is centralized across your organization can be very difficult.
• 15:02
  SPEAKER_01: So there's the deploying it, there's the securing it, and which you already answered absolutely.
• 15:07
  SPEAKER_01: And then there's the scaling it, right?
• 15:08
  SPEAKER_01: From cross-deployment visibilities, like different teams to your point around deployment space and workspaces, different teams being isolated from the next team, that way there's like minimal resource overlap, so like your pod contentions.
• 15:19
  SPEAKER_01: I think those are super important.
• 15:20
  SPEAKER_01: I was also going to add one thing around like technically, we see that there's a lot of requirements around this whole lineage optimization space.
• 15:28
  SPEAKER_01: Like we want to ensure that if customers are using Airflow for reporting and audits compliance reasons, they want to ensure that creating that particular audit trail that's immutable is as seamless as possible, leveraging open source airflow, as opposed to them having to do like a bunch of hoops to tie to like some external metadata catalog or some other tools out there.
• 15:48
  SPEAKER_01: Those are just something I wanted to add to what Ryan had fabulously answered.
• 15:54
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:03
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:15
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:23
  SPEAKER_04: All of what you are describing is kind of a why Astronomer created our original products to help folks run Airflow.
• 16:32
  SPEAKER_04: Those are real challenges that everybody hits.
• 16:34
  SPEAKER_04: And Ryan, you touched on this a little bit.
• 16:36
  SPEAKER_04: That's my background as well.
• 16:38
  SPEAKER_04: Like I have done this myself of running Airflow for at the time a relatively small team, but even that was pretty challenging.
• 16:45
  SPEAKER_04: And we had the classic issue that Astronomer now sees with a lot of our customers was we spent, me and one other developer, pretty much all of our time with Airflow, just managing Airflow and trying to get it to run on Kubernetes in a way that was scalable and so that we could manage RBAC and deal with all of these different things.
• 17:04
  SPEAKER_04: And we didn't have a lot of leftover time to focus on the actual pipelines.
• 17:09
  SPEAKER_04: And there was certainly no discussion at that time of implementing new use cases.
• 17:13
  SPEAKER_04: And we were still in the more kind of traditional ETL bucket at this time.
• 17:18
  SPEAKER_04: And so moving beyond that, I think you do have to have a much more stable and scalable platform.
• 17:25
  SPEAKER_04: And that is what Astronomer tries to do with our products.
• 17:28
  SPEAKER_04: And a lot of the things that you've touched on in terms of features like observability are areas that we're really invested in building out because we've seen that that's what's needed for teams to implement these more advanced use cases.
• 17:41
  SPEAKER_04: I think that's a good tie-in to my next question, which is that Astronomer and IBM have recently launched a collaboration that allows for integration between astronomer software, which is a way for software that we provide that helps folks run Airflow on a Kubernetes cluster so that they don't have to deal with all that configuration themselves.
• 18:02
  SPEAKER_04: And then integrating that with IBM's AI and data integration solutions.
• 18:07
  SPEAKER_04: So what sort of use cases or capabilities do you think this will unlock for your customers?
• 18:12
  SPEAKER_04: Or what are you excited about with this partnership?
• 18:16
  SPEAKER_01: At IBM, we're very excited to see what this partnership unlocks for both organizations and most importantly our clients, right?
• 18:23
  SPEAKER_01: At the intersection of yet again, our expertise and our experience in the data management space and astronomer's Apache Airflow dominance, right?
• 18:31
  SPEAKER_01: We'll love to see what unlocks we can achieve there.
• 18:33
  SPEAKER_01: When we think about unlocks, there's two main vectors we're thinking about.
• 18:36
  SPEAKER_01: The first is from a client experience perspective, what are we excited to see?
• 18:40
  SPEAKER_01: And the second is innovation-wise, what should we expect this whole space to birth?
• 18:44
  SPEAKER_01: So when we come to the client experience side, I think our goal is to identify how can we help improve the speed, the scale, and the reliability for critical workloads across the world, right?
• 18:55
  SPEAKER_01: How can we help our clients get faster time and value?
• 18:58
  SPEAKER_01: And to this effect, I think we have two main uh ideas that we are exploring right now.
• 19:02
  SPEAKER_01: The first is around the plug-and-play data integration uh strategy.
• 19:06
  SPEAKER_01: It's how can we bring our technologies to not just coexist within this client to what Ryan had alluded to, but to actually interact natively?
• 19:14
  SPEAKER_01: Like they're like native supports from IBM's long history of like data storage solutions from the DBTs to the object storage to like our newer like lakehouse solutions to what's next at data.
• 19:24
  SPEAKER_01: How can we ensure that enterprise ready pipelines to support these critical use cases that we already support them like in our own bubbles come together and are available within days and not weeks with minimal overhead?
• 19:37
  SPEAKER_01: The next play here or the next idea we have here is around centralizing and unifying the idea of data observability and governance, right?
• 19:45
  SPEAKER_01: Airflow is a critical piece of that at the equestration layer, but we're thinking for other alternate technologies, right?
• 19:51
  SPEAKER_01: He mentioned and Brian mentioned data stage and other different pipeline providers.
• 19:55
  SPEAKER_01: How can we ensure that we provide that end-to-end solution when it comes to our clients going to a single pane of glass for all pipeline types because these are hybrid environments?
• 20:03
  SPEAKER_01: So for that, we're excited to see yet again technical integrations, deeper dives between your technology, the Airflow technology, as well as our observability with data band or our mantechnology.
• 20:13
  SPEAKER_01: And we have a suite of governance solutions from like data governance to knowledge cataloging solution to actual What's in X governance, which is more around the AI use case and AI workloads.
• 20:24
  SPEAKER_01: How can we tie those together?
• 20:25
  SPEAKER_01: And even in standard, in standards like application monitoring performance monitoring, what's the stack of observability that we can provide that delivers a single view of value to clients?
• 20:34
  SPEAKER_01: We're excited about that opportunity and we're curious to see what this partnership helps the Airflow community drive.
• 20:39
  SPEAKER_01: When we double-click into the second idea I mentioned, which was like innovational, we're thinking around what AI-driven pipeline acceleration would look like in the future.
• 20:49
  SPEAKER_01: What are the new modalities?
• 20:50
  SPEAKER_01: What are the new frontiers we might expect to see?
• 20:52
  SPEAKER_01: And an idea that's been stuck in our head is at the intersection of IBM's Watson X Orchestrates, which is like our agency platform, our AI agentic AI platform that was recently announced, I think, with like 150 plus pre-built agents and astronomers' AI bets.
• 21:08
  SPEAKER_01: I know Astronomer in the Airflow space has been pushing it like from your Astrobots to the Astro AI SDK.
• 21:13
  SPEAKER_01: There's a lot of bet on AI being a useful tool within the Airflow community.
• 21:18
  SPEAKER_01: And we're curious to see how we can merge both together to help customers uh auto-generate or embed AI agents, whether within the pipeline or like alongside the pipeline, to automate things that are typically manual ETL efforts today, from uh data quality checks to in-depth anomaly detection to schema inferencing, enriching workflows without someone having to go do it manually.
• 21:40
  SPEAKER_01: But what are the hurdles we can take out from creating, managing, and scaling out pipelines that AI agents can help us do at scale?
• 21:48
  SPEAKER_01: So we're super excited about those two bits client experience and innovational knock.
• 21:55
  unknown: Yeah.
• 21:55
  SPEAKER_02: So BJ gave like a very, very awesome roadmap of everything that we're excited about for the partnership.
• 22:03
  SPEAKER_02: From my perspective, what I'm excited about is it's a very simple conversation for us to have with our customers.
• 22:09
  SPEAKER_02: The conversation usually goes something like this It's hey, what'd your data stack look like?
• 22:13
  SPEAKER_02: Hey, we have Airflow.
• 22:15
  SPEAKER_02: Oh, great.
• 22:16
  SPEAKER_02: Plan on moving that to a public cloud or hosted environment.
• 22:19
  SPEAKER_02: No.
• 22:20
  SPEAKER_02: Well, guess what?
• 22:21
  SPEAKER_02: That's great because we want to talk to you about our collaboration with Astronomer with IBM, which is a software version of that.
• 22:28
  SPEAKER_02: We can meet clients where they're at right away and give them the things that we talked about earlier: their operation simplicity, the security, the support to help manage their on-prem airflow environment, open source airflow environment in a way that is gonna be way more efficient going forward.
• 22:44
  SPEAKER_02: And yeah, all the things that BJ said, all those are amazing things we're gonna be working towards with the collaboration.
• 22:50
  SPEAKER_02: And again, it's like a very simple conversation for us to have.
• 22:53
  SPEAKER_02: And a lot of times, quite honestly, a lot of our customers don't even know that's an option.
• 22:57
  SPEAKER_02: They just think that, oh, I have to take on this heavy burden, kind of like what you were doing, Kenton, and spend a lot of your time maintaining and admitting all this stuff.
• 23:05
  SPEAKER_02: And it's no, there's a better way to do that.
• 23:07
  SPEAKER_02: And we like to talk to you about it.
• 23:08
  SPEAKER_02: So, again, for me, it's a very simple conversation to get them to all the things that BJ had mentioned.
• 23:15
  SPEAKER_04: Yeah, I love that.
• 23:16
  SPEAKER_04: That's a great way to, I think, summarize our conversation.
• 23:19
  SPEAKER_04: This partnership is such a great foundation for folks looking to implement new and more complex use cases, take advantage of all of the new AI features coming out without having to deal with the management burden or kind of operational overhead of running Airflow at scale themselves.
• 23:36
  SPEAKER_04: That's great.
• 23:37
  SPEAKER_04: I think that's a good place to close us out.
• 23:40
  SPEAKER_04: So I'll just end with Ryan and BJ, what is the best way for folks to get in touch with you?
• 23:45
  SPEAKER_02: I'm on LinkedIn, so you can get in touch with me on LinkedIn.
• 23:47
  SPEAKER_02: I have a weird last name, Yackle.
• 23:49
  SPEAKER_02: So just to look me up, I'll respond.
• 23:51
  SPEAKER_02: Or Ryan.yakle at ibm.com.
• 23:54
  SPEAKER_02: You can reach me there.
• 23:55
  SPEAKER_01: Same here.
• 23:56
  SPEAKER_01: I'm on LinkedIn, BJ at the Soji and on Twitter, like same name, BJ Addisoji.
• 24:02
  SPEAKER_01: My email's a little bit more complex, but we will throw it in the show notes.
• 24:06
  SPEAKER_04: We won't expect anybody to write it down from the audio here.
• 24:09
  SPEAKER_04: So no worries there.
• 24:11
  SPEAKER_04: All right.
• 24:11
  SPEAKER_04: Thank you so much, both of you, for coming on.
• 24:13
  SPEAKER_04: We, you know, at Astronomer are really excited about this partnership as well.
• 24:17
  SPEAKER_04: And it was great to talk more with you about it.
• 24:20
  SPEAKER_02: Awesome.
• 24:20
  SPEAKER_02: Thanks for having us, Ken.
• 24:24
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:30
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 24:34
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.