Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Running Airflow 3 in a regulated environment at OTPP

Transcript

Running Airflow 3 in a regulated environment at OTPP · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: Last week was our uh sunset of hosted uh deployments, and we're fully live on uh remote uh uh deployment model.
• 0:08
  SPEAKER_01: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:16
  SPEAKER_01: Let's get into it.
• 0:20
  SPEAKER_02: Okay, hi everyone.
• 0:21
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:23
  SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by Kauzi Narayan, who is a cloud data platform lead of data engineering at Ontario Teachers Pension Plan, which if it's okay, I think I will refer to as OTPP going forward in this interview.
• 0:39
  SPEAKER_02: Uh Kauzi, it's great to have you on.
• 0:41
  SPEAKER_02: How are you?
• 0:42
  SPEAKER_02: Hi, uh Kenton.
• 0:44
  SPEAKER_02: Uh, thanks for having me.
• 0:45
  SPEAKER_02: Uh, I'm good.
• 0:46
  SPEAKER_02: How do you?
• 0:47
  SPEAKER_02: Yeah, I'm doing really well.
• 0:48
  SPEAKER_02: I'm excited to have you on.
• 0:50
  SPEAKER_02: Uh well, let's go ahead and dive into it.
• 0:52
  SPEAKER_02: So, for today's podcast, we're going to discuss OTPP's cloud migration journey and your move to Airflow 3 and use of remote execution.
• 1:00
  SPEAKER_02: All very exciting topics.
• 1:02
  SPEAKER_02: Uh, but before we get into that, maybe let's start with some background.
• 1:05
  SPEAKER_02: Tell me just at a high level what OTPP does.
• 1:10
  SPEAKER_00: Yeah, Ontario Teachers Pension Plan is a global investor and is a fully funded defined benefit pension plan.
• 1:18
  SPEAKER_00: Uh, we invest in a wide variety of asset classes, uh, primarily to deliver retirement security for Ontario teachers.
• 1:25
  SPEAKER_00: Uh, we currently have like 346,000 active members in the fund, uh, both working as well as like retired teachers in the province of Ontario.
• 1:35
  SPEAKER_00: Okay, very cool.
• 1:37
  SPEAKER_00: And what do you and your team do?
• 1:40
  SPEAKER_00: Uh my team primarily is uh uh called like a data platform team.
• 1:44
  SPEAKER_00: Uh we're currently going through like a migration uh from all of our like an on-prem ecosystem to cloud.
• 1:51
  SPEAKER_00: So a lot of like uh the data platform uh initiatives uh are surrounding this cloud data technology.
• 1:58
  SPEAKER_00: So my team is primarily responsible for uh enabling standards, patents, framework, uh new capabilities within these data uh cloud platform technologies, as well as uh we also support the business technology teams with uh their project initiatives, also with like you know, onboarding training uh to help adopt these new tech new technologies and capabilities.
• 2:21
  SPEAKER_02: Okay, very cool.
• 2:23
  SPEAKER_02: And you mentioned as part of that that you're currently going through a cloud migration.
• 2:27
  SPEAKER_02: I'd love to hear a little bit more about that, kind of what's driving that change, what is that process looking like?
• 2:34
  SPEAKER_00: Yeah, so it's been a multi-year journey.
• 2:37
  SPEAKER_00: Uh it kickstarted uh a few years ago, and we're still in the midst of that journey, um, primarily to uh move away from like our on-prem data center and uh move to cloud um architecture.
• 2:51
  SPEAKER_00: The key capabilities is primarily like you know, enable uh scalability, resilience of the platform, and also like you know, as we all know, the business demands are rapidly growing.
• 3:00
  SPEAKER_00: We wanted to enable new capabilities as well as like you know meet the growing demands of the business.
• 3:06
  SPEAKER_00: So those are the key drivers that uh we wanted to enable uh cloud uh capabilities.
• 3:11
  SPEAKER_02: Yeah, that makes sense.
• 3:12
  SPEAKER_02: And what does the new tech stack look like?
• 3:15
  SPEAKER_00: Yeah, from the cloud data platform perspective, uh we have uh Snowflake as our uh enterprise data platform.
• 3:22
  SPEAKER_00: And uh we use uh DBT primarily for uh our data transformation, and um Airflow uh is uh sitting in the middle of that ecosystem primarily for data pipeline orchestration.
• 3:35
  SPEAKER_00: Okay, great.
• 3:36
  SPEAKER_02: Uh so it sounds like Airflow is pretty central there.
• 3:39
  SPEAKER_02: Maybe tell me a little bit about what sort of things you're orchestrating with Airflow.
• 3:44
  SPEAKER_00: Yeah, so it's uh it's positioned as a data pipeline orchestrator and as an investment uh organization, we bring in data from uh multiple vendor data sources, market data, um, also like our internal enterprise data, uh data from like an order management systems, data from book of record systems.
• 4:03
  SPEAKER_00: There are so many different data that is needed for our daily trading and investment decisions.
• 4:08
  SPEAKER_00: So this data engineering pipeline help bring these data, uh, either being it is scheduled or event-driven, kind of like use cases.
• 4:17
  SPEAKER_02: Yeah, absolutely.
• 4:19
  SPEAKER_02: That totally makes sense.
• 4:20
  SPEAKER_02: And I understand that OTPP is a customer of astronomers.
• 4:25
  SPEAKER_02: Maybe you can tell me a little bit about why you chose to run Airflow on Astro as part of this stack that you've put together.
• 4:33
  SPEAKER_00: Yeah, sure.
• 4:33
  SPEAKER_00: Uh like when we started exploring uh Airflow, uh specifically, like you know, the key features that we were looking for were available, um, and and specifically because of the nature of it being open source community and a very active community and uh able to release like you know features uh pretty rapidly.
• 4:52
  SPEAKER_00: Uh at the same time, uh we also looked at like the infrastructure side of things and managing that infrastructure on our own was an additional overhead and cost associated with it as well.
• 5:03
  SPEAKER_00: So uh we started looking into like a managed service providers, and uh, Astronomer stood out uh in that space, specifically because uh Astronomer is one of the strong contributors to the Airflow open source community.
• 5:16
  SPEAKER_00: Um, and as and when the Airflow releases, there is an Astronomer uh runtime releases right away and um other capabilities like uh monitoring, observability, an RBAC, all of that is built natively within Astronomer as a platform.
• 5:33
  SPEAKER_00: So these are the things that uh we don't want it to spend time investing, rather, we wanted to uh focus on business value and leverage some of these managed service providers.
• 5:42
  SPEAKER_02: Yeah, absolutely.
• 5:43
  SPEAKER_02: That's great.
• 5:43
  SPEAKER_02: I think that's you know, our sort of highest level value prep is we can just take care of some of those things so you don't have to.
• 5:50
  SPEAKER_02: So it's great to hear that that's working out for you.
• 5:53
  SPEAKER_02: Uh I also wanted to double-click very briefly on you mentioned uh that dbt is a big part of your tech stack.
• 6:01
  SPEAKER_02: And I'm curious how you are running DBT with Airflow and kind of what what does that look like?
• 6:07
  SPEAKER_00: Yeah, when we were originally experimenting um dbt, we were running uh a DBT uh core within Airflow natively using like you know bash operator and such.
• 6:19
  SPEAKER_00: Uh it was okay like for experimental uh work.
• 6:22
  SPEAKER_00: You know, we started with like something small models and it was all fine.
• 6:25
  SPEAKER_00: But then when we started bringing in like our real life scenarios and uh migrating our uh big uh uh data transformation projects, uh, we ran into issues of uh leveraging native airflow for DBT.
• 6:39
  SPEAKER_00: So that's when we found about like DBD Cosmos provided by Astronomer as well.
• 6:45
  SPEAKER_00: So we started looking into uh DBD Cosmos to specifically be able to give that granularity of all of our uh models, right?
• 6:56
  SPEAKER_00: Like it also enabled us to see much better lineage as well as be able to troubleshoot uh as well and uh be precise on where exactly the failure is and rerun from that point onwards rather than running the entire uh big monolith of data model.
• 7:12
  SPEAKER_00: So um that's how I think we started as like you know, plain vanilla, then slowly we evolved into leveraging Cosmos.
• 7:18
  SPEAKER_00: Um and uh another thing that we started looking at is also performance boost that we get out of Cosmos.
• 7:24
  SPEAKER_00: Now running it in like uh the new capability of uh running it in a watch mode, watcher mode, uh also brings in a significant uh performance boost for some of our uh uh transformation layer.
• 7:35
  SPEAKER_00: So uh that's that's how we started.
• 7:38
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 7:47
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 7:51
  SPEAKER_01: Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 7:57
  SPEAKER_01: Check it out at astronomer.io.
• 8:02
  SPEAKER_02: Yeah, absolutely.
• 8:03
  SPEAKER_02: Watcher mode was a really exciting uh feature in the release that happened, I think, a couple weeks ago as we're recording this, maybe a couple months ago by the time this has come out.
• 8:13
  SPEAKER_02: But yeah, definitely something that was widely requested.
• 8:15
  SPEAKER_02: I've seen a lot of excitement about it, including from my team who was, yeah, pretty impressed.
• 8:21
  SPEAKER_00: Yep.
• 8:22
  SPEAKER_02: Yeah, very cool.
• 8:23
  SPEAKER_02: Uh and then I also want to touch a little bit on your experience with Airflow 3, because I understand that you've just gone through migration.
• 8:31
  SPEAKER_02: You mentioned um Astronomer having runtime releases, right, with new uh Airflow releases, which is definitely something we want our customers to be able to um, you know, keep take advantage of all of the new things going on in the Airflow project, and Airflow 3 was a big one.
• 8:48
  SPEAKER_02: So, yeah, tell me about kind of what um what motivated you to do that upgrade uh when you did and kind of what that experience looked like.
• 8:58
  SPEAKER_00: Yeah, like I think we when we started Airflow, we weren't very too off given that our journey started recently.
• 9:04
  SPEAKER_00: We were at Airflow uh version 2.9.
• 9:07
  SPEAKER_00: And uh when uh Airflow 3 came about, like you know, the the main thing that we wanted to learn as part of this, like you know, how fast we can upgrade, uh, right, like from a version to another version.
• 9:18
  SPEAKER_00: Also, um, we started looking at like you know some of the new capabilities that Airflow 3 offered, which was uh remote execution capability.
• 9:25
  SPEAKER_00: Um, so that was something that we actually added from day one, but because we had to go live right away with Airflow 2.9, it wasn't available.
• 9:33
  SPEAKER_00: Uh so that is one of the primary drivers as well for us to make a decision to move to Airflow 3 and leverage the remote execution capabilities.
• 9:41
  SPEAKER_00: Uh, at the same time, exercise the upgrade procedures.
• 9:44
  SPEAKER_00: And it was pretty um easy for us to upgrade, mainly because we were pretty close to the version as well.
• 9:49
  SPEAKER_00: Uh, we weren't too far off.
• 9:51
  SPEAKER_00: At the same time, we leveraged um Astro CLI and some of the uh linters that can automatically catch some of the issues and uh any sort of like uh libraries that get deprecated and need to be upgraded.
• 10:04
  SPEAKER_00: All of that were pretty uh straightforward and uh very easy for us to manage.
• 10:08
  SPEAKER_00: So we made a quick call to move to three so that we can enable remote uh execution capabilities.
• 10:14
  SPEAKER_02: Yeah, great.
• 10:15
  SPEAKER_02: Let's talk a little bit more about the remote execution because this is uh definitely I think one of the most popular, uh, or I don't know most popular, but certainly one of the biggest impact features of Airflow three.
• 10:28
  SPEAKER_02: Uh I'm curious what drove uh your decision to use this feature, or why were you guys looking at it?
• 10:37
  SPEAKER_00: Yeah, I think like uh for us uh being uh an investment organization, uh like you know, there's a lot of like you know, security controls in place.
• 10:46
  SPEAKER_00: Uh and we wanted to make sure that the data doesn't get uh moved uh from our ecosystem, right?
• 10:53
  SPEAKER_00: Like there's always like uh info security concerns when the data gets moved in the cloud.
• 10:57
  SPEAKER_00: Um so that was one big uh factor.
• 10:59
  SPEAKER_00: And the other one is like the the scalability and flexibility that we get on our side.
• 11:04
  SPEAKER_00: Uh we can up and down the resources according to the different workloads, um, given that the execution plane is going to be running in our Kubernetes cluster.
• 11:13
  SPEAKER_00: So those are some of the key compelling reasons uh that uh we started exploring remote execution.
• 11:19
  SPEAKER_00: And um yeah, it was uh pretty successful uh migration and we did a quick uh POV uh working closely with the product engineers, and we learned a lot.
• 11:32
  SPEAKER_00: Uh also this capability is new from the product perspective as well.
• 11:35
  SPEAKER_00: So we provided feedback and we were able to get some updates uh in the next versions, and it was a very pretty collaborative uh work that we did uh with the product engineers, and uh it was a pretty successful migration.
• 11:49
  SPEAKER_02: Yeah, that's great to hear.
• 11:51
  SPEAKER_02: And I think the compliance piece is really interesting.
• 11:54
  SPEAKER_02: It's certainly something that we've heard a lot about with financial institutions of various kinds.
• 12:00
  SPEAKER_02: I think they're one of the top uh industries, I would say, that needs to make use of a feature like remote execution.
• 12:07
  SPEAKER_02: Before you had this available to you, did you have to do some sort of workaround to manage the data within your security policies?
• 12:16
  SPEAKER_00: Um Yeah, we had to come up with the, yeah.
• 12:19
  SPEAKER_00: So definitely I think like you know, compliance is always uh and infosecurity is always at the forefront of our uh any sort of integration with cloud platform.
• 12:27
  SPEAKER_00: Uh so when we had to do uh this uh without the the remote execution, there is a very complex network architecture that we have to come up with.
• 12:35
  SPEAKER_00: We have to enable like you know, VPN tunnel, and we have to handshake through that VPN.
• 12:40
  SPEAKER_00: And obviously, like you know, there's other network firewall rules and restrictions uh that we have to enforce, which made like the latency a concern as this data flows through.
• 12:51
  SPEAKER_00: Uh, there is a bit of that latency that we have uh noticed.
• 12:55
  SPEAKER_00: And uh yeah, so those the those are the work, I wouldn't call it like work crowd, like the complexity in the network architecture.
• 13:01
  SPEAKER_00: And uh by moving to remote execution, we were able to simplify that network architecture as well.
• 13:07
  SPEAKER_02: Yeah, absolutely.
• 13:08
  SPEAKER_02: I could see a huge benefit to doing that.
• 13:11
  SPEAKER_02: Uh you mentioned, well, maybe this seemed easy because it was a simplification, but I know uh remote execution setting that up can be not super trivial depending on the environment that you're working in.
• 13:24
  SPEAKER_02: You mentioned doing a POV before you actually made the move to doing this in production.
• 13:29
  SPEAKER_02: Kind of what did that process look like and how was it for you from an implementation standpoint?
• 13:36
  SPEAKER_00: Yeah, so I think like when we were in the hosted execution, we exactly know some of the pain points or concerns, specifically as I called out, like from the network complexity, uh data security, and all that stuff.
• 13:47
  SPEAKER_00: Um, and uh using those as kind of like our uh goal, uh, and we came up with like you know some um key success criteria uh metrics uh for us to be able to say, hey, we're we're good to go and move forward.
• 14:00
  SPEAKER_00: Uh so yeah, we were able to make those decisions pretty fast.
• 14:02
  SPEAKER_00: And as I mentioned, like, you know, we worked with like you know, astronomer product engineers, uh, we had some dedicated resources helping us uh to uh set this up on our side and making sure like you know the handshake is happening properly.
• 14:15
  SPEAKER_00: We're able to spin up like a sandbox uh environment uh for this uh use case as well to time box it uh for like I believe we did it like in a four to six weeks time box, and uh both from like uh infrastructure setup perspective as well as like you know deploying some of our workloads and validating those through those success criteria to make sure that we're able to meet all of those.
• 14:37
  SPEAKER_00: And uh then yeah, I think after that it's it's primarily socializing with our executive team and uh getting the the sign-offs from uh our our leadership to to move forward.
• 14:49
  SPEAKER_02: Okay, great.
• 14:50
  SPEAKER_02: And then so you have now at this point moved beyond that stage and you're using this in production.
• 14:55
  SPEAKER_02: Is that right?
• 14:56
  SPEAKER_02: Definitely, yes, yes.
• 14:57
  SPEAKER_00: So we are actually uh last week was our uh sunset of hosted uh deployments and we're fully live on uh remote uh deployment model.
• 15:07
  SPEAKER_00: Okay, cool.
• 15:08
  SPEAKER_00: That's exciting.
• 15:09
  SPEAKER_00: Yes, yes.
• 15:10
  SPEAKER_00: So we're closely monitoring, we're learning as well, uh, but uh so far, so good.
• 15:15
  SPEAKER_02: Yeah, that's great.
• 15:16
  SPEAKER_02: Love to hear that.
• 15:17
  SPEAKER_02: Like you said, I think the remote execution feature has been so impactful, especially for companies working with sensitive data as yeah, especially.
• 15:26
  SPEAKER_02: And so I it's really great to hear that success story from your team.
• 15:32
  SPEAKER_02: Very cool.
• 15:33
  SPEAKER_02: Uh well, I think that is a good um tie-in to my standard final question, which I ask of all of my guests, might be related to Airflow 3 and remote execution, or it doesn't have to be.
• 15:44
  SPEAKER_02: Uh, but that is what would you most like to see from the Airflow project in the near future?
• 15:49
  SPEAKER_00: I think like uh with Airflow 3, there are a lot of like you know new capabilities uh specifically around AI.
• 15:56
  SPEAKER_00: Uh everything is like you know, going at a rapid pace that we're trying to catch up.
• 16:01
  SPEAKER_00: Um I think like uh the focus for our organization as well is to uh bring in uh AI capabilities, a lot of the uh AI orchestration uh as well, um, and also uh like you know, stabilizing the remote execution uh platform.
• 16:17
  SPEAKER_00: So these are the key areas that uh we would be uh primarily focusing on and uh closely watching this uh community space as well to see how we can quickly enable these capabilities and evolve the platform.
• 16:29
  SPEAKER_02: Yeah, absolutely.
• 16:30
  SPEAKER_02: On the AI front, have you seen or tried the new common AI provider?
• 16:36
  SPEAKER_00: Yes, uh uh we we just recently upgraded to Airflow 3.2.
• 16:41
  SPEAKER_00: Um so we're in the midst of like you know evaluating some of those.
• 16:45
  SPEAKER_00: Along with that, uh AI, there's also other cool features like async operators, um, there's data partition, data assets.
• 16:52
  SPEAKER_00: Um, so a lot of cool features that primarily fits into our use case that we're looking at as well.
• 16:59
  SPEAKER_00: So definitely um exciting.
• 17:02
  SPEAKER_00: Um looking forward to do some POC and uh roll that out to production.
• 17:06
  SPEAKER_02: Yeah, that's great.
• 17:08
  SPEAKER_02: There's yeah, so much great stuff going on in the project right now.
• 17:11
  SPEAKER_02: And I know the common AI provider has been generating a lot of excitement.
• 17:15
  SPEAKER_02: Uh, I've just been working on today.
• 17:17
  SPEAKER_02: Actually, I will be giving some talks at upcoming meetups about the provider and some of its functionality because I think you are not the first person I have heard say that working with AI from Airflow is a top priority for their team.
• 17:30
  SPEAKER_00: So I think something that's definitely I'll I'll be the first one to sign up for those webinars.
• 17:34
  SPEAKER_00: I love the webinars uh that uh Airflow community runs.
• 17:38
  SPEAKER_00: It's amazing like uh to learn all the new capabilities as well as like you know to quickly implement as well with all the sample code repos that you provide.
• 17:48
  SPEAKER_00: Uh it's been amazing for uh to use all those uh resources.
• 17:51
  SPEAKER_02: Oh, great.
• 17:52
  SPEAKER_02: I love to hear that.
• 17:53
  SPEAKER_02: Um we will yeah, I'll pass along that feedback to to my team and other folks at Astronomer.
• 17:59
  SPEAKER_02: So okay, great.
• 18:00
  SPEAKER_02: Well, thank you so much for going through all of that.
• 18:03
  SPEAKER_02: So interesting to hear about yeah, the journey that your team has been on recently and the success that you've had with Airflow3.
• 18:10
  SPEAKER_02: Uh just to close us out, Kazi, what is the best way for folks to get in touch with you?
• 18:15
  SPEAKER_00: I'm available in LinkedIn.
• 18:17
  SPEAKER_00: So uh feel free to reach out and touch base with me.
• 18:20
  SPEAKER_00: I'm happy to share our experiences and able to share our uh journey as well.
• 18:25
  SPEAKER_00: Great.
• 18:25
  SPEAKER_02: Well, we will put a link to that in the show notes.
• 18:28
  SPEAKER_02: Uh well, thank you again for coming on.
• 18:30
  SPEAKER_02: It's been so great chatting with you.
• 18:32
  SPEAKER_02: Thank you.
• 18:33
  SPEAKER_02: Thanks for having us, uh Kinton.
• 18:35
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 18:38
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 18:41
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 18:46
  SPEAKER_01: And leave us a five star review.