Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building Airflow CTL with Buğra Öztürk at Mollie

Transcript

Building Airflow CTL with Buğra Öztürk at Mollie · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: Normally, uh what you usually do while developing tags, for example, you need to basically deplore your DAX with some automations, but behind the scene you need to test your DAG, you need to see that it's working.
• 0:13
  SPEAKER_03: And with the Airflow CTL, basically you can do these things even without going to UI.
• 0:20
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:32
  SPEAKER_00: Okay, hi everyone.
• 0:34
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:35
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Bura Osturk, who is a senior data engineer at Molly and is also a committer and PMC member on the Apache Airflow Project.
• 0:46
  SPEAKER_00: Bura, how are you?
• 0:48
  SPEAKER_03: I'm good, Kenton.
• 0:49
  SPEAKER_03: How about you?
• 0:50
  SPEAKER_00: I'm good.
• 0:51
  SPEAKER_00: I'm good.
• 0:51
  SPEAKER_00: It's great to have you on.
• 0:52
  SPEAKER_00: Uh I always love talking about things that are going on in the Airflow project, especially with committers and folks who are very involved.
• 0:59
  SPEAKER_00: So yeah, I'm excited for our conversation.
• 1:02
  SPEAKER_02: Yeah, likewise.
• 1:03
  SPEAKER_02: Thanks.
• 1:03
  SPEAKER_02: Thanks, Kentan.
• 1:04
  SPEAKER_00: Yeah, great.
• 1:05
  SPEAKER_02: So really happy to talk about the Airflowflow.
• 1:08
  SPEAKER_00: Yeah, awesome.
• 1:09
  SPEAKER_00: Well, let's go ahead and dive in.
• 1:11
  SPEAKER_00: So for our listeners for today's podcast, we're going to talk about Airflow CTL, uh, including what that is, how it works, and kind of where it's headed.
• 1:19
  SPEAKER_00: This is an exciting topic for the open source project.
• 1:22
  SPEAKER_00: But before we get into that, I want to start with a little bit of background on you.
• 1:26
  SPEAKER_00: So I know you're a data engineer in your day job, but as I mentioned, you're also a committer and PMC member for Airflow.
• 1:33
  SPEAKER_00: Uh, how did you first get involved with the project?
• 1:36
  SPEAKER_03: Yeah, thanks, Kentan, for the question.
• 1:38
  SPEAKER_03: Yeah, I I started my first issue, created creating the issue around like 2022, related to managing multi-team.
• 1:48
  SPEAKER_03: And then I remember my first contribution was on 2023, basically contributing to the Wild document, which I think one of the like the entry points for Airflow contribution.
• 2:00
  SPEAKER_03: And then I continue with operator contributions as well as continue worked with uh IAP44, which was called internal API.
• 2:10
  SPEAKER_03: Now it's totally replaced and deprecated, and some documentation changes.
• 2:16
  SPEAKER_03: I paused a bit there and then again started uh working with Apache Airflow, meaning contributing around uh 2024, around mid-July.
• 2:27
  SPEAKER_03: I think that was the time exactly the Airflow version 3 uh discussions already started and uh the features IAPs was going to finalize.
• 2:38
  SPEAKER_03: Uh and since then I have been contributing to Apache Airflow, and I have been committer since uh for over a year.
• 2:47
  SPEAKER_03: And since this year, early this year, I started my new role as well as a member of PMC, which yeah is really great.
• 2:55
  SPEAKER_03: And I also want to thank you like the all the PMC members and the contributors, committers that really helped beat my uh pet.
• 3:02
  SPEAKER_00: Yeah, that's great.
• 3:04
  SPEAKER_00: I love this story because and I hope it inspires people who might be a little intimidated to start contributing to the project, just because as you described, you started pretty small and then your role has grown to contributing quite significant things to the project.
• 3:20
  SPEAKER_00: But sometimes it starts with just those docs changes or things like that, and then you can still work your way up.
• 3:26
  SPEAKER_00: So uh hopefully some listeners take inspiration from that and decide to contribute themselves, and they might also end up being PMC members eventually.
• 3:34
  SPEAKER_02: Yeah, yeah, totally.
• 3:36
  SPEAKER_00: Yeah, I love that.
• 3:38
  SPEAKER_00: Okay, well, let's go ahead and talk about Airflow CTL.
• 3:41
  SPEAKER_00: I understand this is kind of your current biggest project that you're working on as a committer.
• 3:47
  SPEAKER_00: First, just tell me what is it, kind of what what problem is it solving?
• 3:51
  SPEAKER_03: Yeah.
• 3:52
  SPEAKER_03: Airflow CTL basically solving secure user interaction with Airflow environments.
• 3:58
  SPEAKER_03: What I mean by solving is it obstructs the database interactions using API and leverages the entire security layer that built on top, meaning that having the RPAC in place, audit logs, basically all the layers that we have in the API.
• 4:16
  SPEAKER_03: And with the Airflow CTL, you normally need the database connection and the credentials, which is highly sensitive, and like the entire key to the system itself.
• 4:27
  SPEAKER_03: So Airflow CTL abstract that with using the API and uh provide users a CLI interface, a control interface can be used.
• 4:36
  SPEAKER_03: And it's basically leveraging the REST API.
• 4:40
  SPEAKER_03: I also want to talk about a bit like we have, and I mean API, we have like three different different flavors.
• 4:46
  SPEAKER_03: One the execution API, which internal components uh communicates, one public API under the hood, it's core API, and one for the UI.
• 4:56
  SPEAKER_03: Which UI is not backward compatible, and Airflow CTL only uses the core API under the public API, meaning that any user that interacted with the Airflow securely is basically using that route, and Airflow CTL also using that.
• 5:13
  SPEAKER_00: Interesting.
• 5:14
  SPEAKER_00: And uh tell me a little bit more about how this is interacting with Airflow compared to like the classic Airflow CLI.
• 5:22
  SPEAKER_00: You mentioned kind of the security differences between the two.
• 5:25
  SPEAKER_03: Yeah, basically we can think of Airflow CLI is like the administration control panel, and the Airflow CTL is more for the users, it's also for the administrators as well.
• 5:37
  SPEAKER_03: But basically, they will live side by side and they will communicate with the uh Airflow environments in different paths.
• 5:46
  SPEAKER_03: We can think of like kubectl and kubeadmin, like in the kubeadmin, you are creating your environment, and with kubectl you are managing your environment.
• 5:55
  SPEAKER_03: This is the pretty similar abstraction and separation in Airflow CLI and CTL, where Airflow CLI will basically continue to run your like the uh components, managing your database, uh upgrades, and Airflow CTL more towards interacting with the environment securely.
• 6:16
  SPEAKER_00: Yeah, that makes sense.
• 6:17
  SPEAKER_00: That's a very clear way of thinking about it.
• 6:19
  SPEAKER_00: And tell me a little bit more about kind of the architectural overview, I guess I'll say, of like what does the Airflow CTL look like under the hood in terms of how it's interacting with these components?
• 6:31
  SPEAKER_03: Oh, Airflow CTL basically runs in users' computers in operating systems, and it's basically first authenticate through the API and acquires the JVT token.
• 6:43
  SPEAKER_03: It securely keeps in Keyring, uh, the operating system Keyrings or third-party key rings.
• 6:50
  SPEAKER_03: Uh, it also provides you an headless environment, which then you can leverage in uh a couple of different use cases.
• 6:58
  SPEAKER_03: Uh at the end, the Airflow CTL is basically connecting only to the API and uh lives within your computer.
• 7:07
  SPEAKER_03: And it's basically uh like managing your Airflow environment from anywhere.
• 7:13
  SPEAKER_00: Okay, got it.
• 7:15
  SPEAKER_00: And tell me what do you think that would look like in like a day-to-day practice?
• 7:20
  SPEAKER_00: Like, what's a concrete use case where somebody would reach for that?
• 7:24
  SPEAKER_03: Yeah, that that's a great question.
• 7:25
  SPEAKER_03: Uh basically, there are like multiple different examples.
• 7:29
  SPEAKER_03: I will try to keep giving like the important ones.
• 7:32
  SPEAKER_03: The first uh could be the like the faster local development.
• 7:37
  SPEAKER_03: Normally, uh, what you usually do while developing tags, for example, you need to basically deploy your DAGs with some automations, but behind the scene, you need to test your DAG, you need to see that it's working.
• 7:50
  SPEAKER_03: And with the Aflow CTL, basically you can uh do these things even without going to UI.
• 7:57
  SPEAKER_03: Let's say you are more of a command line person, and within the command line you can basically trigger your DAGs, see their status, and basically inspect their uh state, uh uh, meaning that it can help uh in faster iterations.
• 8:13
  SPEAKER_03: You can even create your like own scripts locally.
• 8:17
  SPEAKER_03: Uh one another example could be uh the CI CD automations.
• 8:22
  SPEAKER_03: Uh since Airflow CTL not using the database credentials, it's more secure to use in CI CD and those kind of places because then you will have short-lived token with the JVT using the API.
• 8:35
  SPEAKER_03: And in CI CD using even the headless mode, you can basically automate your deployments, meaning that you are deploying into a dev environment, and then you can basically trigger uh those DAX after deployments, do some smoke testing, and to see like you can even check if it succeeded or not, then you can even automate your entire end-to-end testing, integration testing with the airflow in your environment.
• 9:02
  SPEAKER_03: And uh I would like to also mention one important part of day-to-day work.
• 9:06
  SPEAKER_03: I think it changed a lot after AI agents uh landed, and AI agents also can use CLI uh very well.
• 9:14
  SPEAKER_03: So it's basically up to user how to leverage it, of course, but using the airflow skills while you are developing tech, you don't need to even basically develop your local secrets.
• 9:25
  SPEAKER_03: You can just ask an agent, an AI agent to basically, yeah, can you run my deck?
• 9:31
  SPEAKER_03: Uh it's deployed, check the status, and then yeah, that fix these parts, and then basically end-to-end, it can be a powerful tool for your uh AI agents as well.
• 9:42
  SPEAKER_00: Interesting.
• 9:43
  SPEAKER_00: That I would not have thought about that use case before.
• 9:46
  SPEAKER_00: That's very cool.
• 9:47
  SPEAKER_00: Uh, I know there are a lot of folks out there that struggle with getting a good local dev set up and do prefer to work in their terminal.
• 9:54
  SPEAKER_00: It makes me think of the old, old, maybe it's still in use, the like dag.test command that you can use in like an IDE, kind of like that, but like much more functionality.
• 10:03
  SPEAKER_00: So that's that's really cool.
• 10:05
  SPEAKER_00: I'm curious what thinking about like the long-term vision here.
• 10:10
  SPEAKER_00: So we've touched a lot on kind of the differences between Airflow CTL and then the existing CLI.
• 10:16
  SPEAKER_00: What is kind of the story there as you see it?
• 10:20
  SPEAKER_00: Will they be complementary over time or what does that look like?
• 10:26
  SPEAKER_03: Yeah, uh the Airflow CTS, like our first aim is to decouple uh the remote commands, meaning that we first need to understand what is remote.
• 10:35
  SPEAKER_03: Like remote is basically any action that we can do with the API itself, with the API endpoints, either with single API endpoints or multiple endpoints, and uh leveraging that API endpoint uh basically make those commands uh remote commands, meaning that we are aiming to decouple that part from the Airflow CLI and use the Airflow CTL entirely under the hood.
• 11:02
  SPEAKER_03: We have all to three center the IAP94, which exactly describe this long-term region, maybe more of mid plus in between mid and long term.
• 11:13
  SPEAKER_03: I mean, what we are aiming there is basically that there are two perspectives.
• 11:18
  SPEAKER_03: One is basically replacing these remote commands within the CLI and fully use to leverage the Airflow CTL behind the scene, meaning that in that part, the Airflow CLI will use Airflow CTL with the authentication part, and users won't be won't realize this change uh in their like their data work uh using the Airflow CLI.
• 11:42
  SPEAKER_03: In the second perspective, uh we are planning to bring up a migration tool, and that tool will aim to uh give users suggestions, we'll have them to basically migrate their current commands to the Airflow CTL.
• 11:57
  SPEAKER_03: We are trying to construct the CTL commands in a way that it should be easier to migrate towards the Airflow from CLI to Airflow CTL.
• 12:06
  SPEAKER_03: And in the in this vision, basically our aim is to completely decouple the remote part.
• 12:13
  SPEAKER_03: And after that, the Airflow CLI will stay with the its like the uh core commands.
• 12:21
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:30
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:42
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:50
  SPEAKER_00: And so in the shorter or medium term, users won't really have much of a transition, so to speak.
• 13:00
  SPEAKER_03: Yes, exactly.
• 13:01
  SPEAKER_03: They they can also start, of course, using Airflow CTL prom today, but the actual like fully uh decoupling part exactly will be happening in the uh midtime short term.
• 13:12
  SPEAKER_00: Okay, got it.
• 13:13
  SPEAKER_00: Uh and maybe I actually should have asked this before, but where are you at in this work today?
• 13:20
  SPEAKER_00: Like what is available now versus this part that you've described that is coming with AIP94 and more of the medium or long term, or even maybe what has already happened kind of in the recent past, if that makes sense.
• 13:33
  SPEAKER_03: Yeah, yeah, great questions.
• 13:35
  SPEAKER_03: Uh we uh basically brought up Airflow CTI to life, which wants CockSiel's ideation to manage the environments.
• 13:43
  SPEAKER_03: Uh with the IAP81, this basically comes back to life within the Airflow uh repository, fully functional and lurking.
• 13:52
  SPEAKER_03: We are currently at the 0.x versions, meaning that we are not fully backboard compatible yet, but we have lots of already implemented uh CLI commands that people can already start using, experimenting, and uh I already uh start hearing people start using in their CI automations, in their like deployment automations.
• 14:13
  SPEAKER_03: And we are aiming to move forwards 1.0, meaning that fully backward compatible and fully uh full parity with the airflow of CLI, meaning that any remote command lives in CLI already have in the CTL.
• 14:30
  SPEAKER_03: This part from 0.x to 1.0 will be part of the IAP94, as I mentioned.
• 14:37
  SPEAKER_03: But today they can start using, uh start exploring, and yeah, please.
• 14:42
  SPEAKER_03: I'm also suggesting strongly suggesting to do that.
• 14:46
  SPEAKER_00: Yeah, that's a great call to action.
• 14:47
  SPEAKER_00: And especially um if folks aren't always already aware, then they can provide you know any feedback through airflow issues in the GitHub repo or on Slack or for things like that.
• 14:59
  SPEAKER_00: I think it's great to hear that you've already seen feedback from folks using it for their CI workflows.
• 15:04
  SPEAKER_00: Um I think this would be super helpful, the just eliminating that kind of waiting period in development of deploying something and then waiting for the remote CI to run only for it to fail.
• 15:16
  SPEAKER_00: I think all of us have gone through that iteration so many times.
• 15:18
  SPEAKER_00: So being able to do that testing locally before you get there seems like it would save people a lot of time and just kind of general development energy.
• 15:28
  SPEAKER_03: Yeah, yeah, exactly.
• 15:29
  SPEAKER_03: And maybe one sentence to uh wrap this part as well.
• 15:34
  SPEAKER_03: Airflow CLI will uh continue as is uh with the core components, meaning that it will serve for administrators, it will help administrators to run components, manage the database tasks.
• 15:50
  SPEAKER_03: Those will uh still will uh live in the Airflow CLI and won't be migrated towards CTL.
• 15:58
  SPEAKER_00: Okay, got it.
• 15:59
  SPEAKER_00: Yeah, that's helpful for folks to know.
• 16:02
  SPEAKER_00: Cool.
• 16:02
  SPEAKER_00: Well, yeah, this sounds really exciting and it's great to let everybody know that it's already available uh and they can start using it now within their workflows.
• 16:12
  SPEAKER_00: Yeah, very cool.
• 16:13
  SPEAKER_00: I guess this brings me to kind of my standard final question, which is what would you most like to see from development of the Airflow project in the near future?
• 16:23
  SPEAKER_00: And I guess in this case, since we've just talked a bunch about Airflow CTL, maybe it's something else for you or something that you're not working on.
• 16:30
  SPEAKER_00: I guess I'll also ask as a sort of secondary question to that are there any other, you know, big things in the Airflow project that you're currently working on that you're excited about?
• 16:39
  SPEAKER_03: Yeah, yeah.
• 16:40
  SPEAKER_03: Uh there are actually like lots of great things already happened for 3.2, uh, which is about to release and already going to happen for the future versions.
• 16:50
  SPEAKER_03: Maybe my take would be uh the multi-team deployment to be generally available.
• 16:56
  SPEAKER_03: At the moment, it will be experimentally available uh in the 3.2, but seeing that uh generally available, yeah, uh would lead a lot for the airflow.
• 17:06
  SPEAKER_03: And of course, for Airflow CTL, what you mentioned, like uh as a commuter, I'm also like managing these projects under IAP94.
• 17:15
  SPEAKER_03: So I'm yeah, also would be happy to see those things actually happen.
• 17:19
  SPEAKER_00: Yeah, absolutely.
• 17:20
  SPEAKER_00: That's great.
• 17:21
  SPEAKER_00: And I know as we're recording this, Airflow 3.2 will be released imminently, uh, hopefully within the next couple of days, certainly.
• 17:30
  SPEAKER_00: So by the time this episode is live, hopefully Airflow 3.2 will have been out for a little while and folks can get their hands on it.
• 17:36
  SPEAKER_00: But yeah, there's a lot of great stuff in that release that we're looking forward to.
• 17:40
  SPEAKER_00: Okay, awesome.
• 17:41
  SPEAKER_00: Well, Bura, thank you so much for going through that again.
• 17:44
  SPEAKER_00: I think this is really great functionality that will benefit a lot of folks using the open source project and appreciate all of your hard work on it.
• 17:52
  SPEAKER_00: Just to close out, what is the best way for folks to get in touch with you?
• 17:55
  SPEAKER_03: Yeah, uh, and yeah, I would like to really uh say thank you, Canton, having me in Data Ploadcast and had a chance to talk about Airflow CTL.
• 18:03
  SPEAKER_03: And they can basically reach me out from Airflow Slack, Community Slack uh channels from LinkedIn.
• 18:09
  SPEAKER_03: And if it's related to contribution from, of course, GitHub issues and PRs would be great.
• 18:15
  SPEAKER_00: Yeah, I was about to say as a maybe a second follow-up to that closing question.
• 18:20
  SPEAKER_00: Do you have any advice for folks who may have listened to this and are really interested in helping with development of AIP94 or anything related?
• 18:28
  SPEAKER_00: Because I assume those contributions would be very much welcomed.
• 18:32
  SPEAKER_03: Yeah, yeah, exactly.
• 18:33
  SPEAKER_03: I would really like to show that any help would be really appreciated.
• 18:37
  SPEAKER_03: And after 3.2 released, we have really started working on this project.
• 18:43
  SPEAKER_03: We are waiting 3.2 to happen for managing to breaking changes easily.
• 18:48
  SPEAKER_03: Not breaking for Airflow users, but I mean internally.
• 18:52
  SPEAKER_03: And I would really like to see uh like contributors to check out the IAP from Confluence.
• 18:58
  SPEAKER_03: And we also created the GitHub projects that they can check for open tickets.
• 19:04
  SPEAKER_00: Okay, great.
• 19:05
  SPEAKER_00: Well, we will throw links to a lot of this stuff in the show notes uh in case anybody does not already have it, um, so that they can join the Slack or get on the Airflow repo if they need to.
• 19:16
  SPEAKER_00: But thank you again, Brura, for coming on.
• 19:18
  SPEAKER_00: It's been really great speaking with you.
• 19:19
  SPEAKER_03: You too.
• 19:20
  SPEAKER_03: Thank you very much.
• 19:23
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 19:30
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 19:34
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.