Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Improving security in Airflow 3

Transcript

Improving security in Airflow 3 · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:33
  SPEAKER_02: Or click the link in our show notes.
• 0:35
  SPEAKER_02: Now let's get into the show.
• 0:38
  SPEAKER_00: Okay, hi everyone.
• 0:39
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:41
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Anish Geary, who is a senior software engineer and active contributor to Apache Airflow focused on security.
• 0:51
  SPEAKER_00: Anish, how are you?
• 0:52
  SPEAKER_01: Yeah, I'm doing fine.
• 0:53
  SPEAKER_01: How are you?
• 0:54
  SPEAKER_01: Thanks for having me in today.
• 0:55
  SPEAKER_00: I'm good.
• 0:56
  SPEAKER_00: It's really great to have you on.
• 0:57
  SPEAKER_00: I'm excited to dive into this.
• 0:59
  SPEAKER_00: I always love having people who are contributing to the project on the podcast.
• 1:03
  SPEAKER_00: And for today, for our listeners, we're going to discuss your security contributions to Airflow, including things like connection testing, token revocation, and token scoping.
• 1:14
  SPEAKER_00: I think these are going to end up being things that probably every Airflow user encounters to some degree.
• 1:20
  SPEAKER_00: So it should be very helpful for all of our listeners.
• 1:23
  SPEAKER_00: But before we get into all of that, I want to start with some background.
• 1:26
  SPEAKER_00: So let's talk about you.
• 1:27
  SPEAKER_00: How long have you been using Airflow?
• 1:29
  SPEAKER_01: Yeah, before starting, I want to put a little disclaimer out that all the any views that I put out here is just mine and it does not relate to any organization.
• 1:39
  SPEAKER_01: And yeah, uh, well, yeah, it's funny thing that I'm not a data engineer and I've not uh run any pipelines in production yet.
• 1:47
  SPEAKER_01: I got into Airflow by reading the source code and docs itself.
• 1:52
  SPEAKER_01: And yeah, I see many people coming from outside in.
• 1:55
  SPEAKER_01: My journey was inside out, I believe.
• 1:59
  SPEAKER_00: Yeah, that's cool.
• 2:00
  SPEAKER_00: I think you're right that probably most people are Airflow users themselves and then start contributing.
• 2:06
  SPEAKER_00: Although we do certainly have people at Astronomer who are on the Airflow PMC or who are committers who spend most of their day job working on Airflow, who that is their background as well as they have worked in software and worked in open source software, but they are not using Airflow themselves.
• 2:23
  SPEAKER_00: So you're definitely not alone there.
• 2:26
  SPEAKER_00: And I understand that you, you know, started contributing to Airflow fairly recently, um, like at the end of last year.
• 2:33
  SPEAKER_00: I guess just kind of high-level what got you into open source contribution?
• 2:38
  SPEAKER_00: Like you mentioned, you are not an Airflow user yourself.
• 2:40
  SPEAKER_00: What made you interested in working with the project?
• 2:44
  SPEAKER_01: Yeah, well, well, my whole career is, you know, um dependent on users of uh the open source uh all the time.
• 2:53
  SPEAKER_01: And uh I think I know many people out there are spending their time, their precious time, building this for free.
• 3:00
  SPEAKER_01: And that always fascinated me.
• 3:02
  SPEAKER_01: And at one point of time I wanted to be in the giving side of the spectrum.
• 3:06
  SPEAKER_01: So I started small and got got hooked into it.
• 3:10
  SPEAKER_00: Yeah, that's amazing.
• 3:12
  SPEAKER_00: One of the best things I think about the Airflow project is just the number of people contributing.
• 3:16
  SPEAKER_00: I saw, I think they just passed 4,000 contributors uh to the project, which is so cool.
• 3:23
  SPEAKER_00: And so obviously, a very small fraction of those people are doing it as part of their day job.
• 3:28
  SPEAKER_00: And as you said, a lot of them are doing it just because they want to, they want to make the project better, whether they're using it themselves or not.
• 3:34
  SPEAKER_00: So that's really cool.
• 3:37
  SPEAKER_00: And then we're gonna talk a lot about your uh contributions focused on security.
• 3:43
  SPEAKER_00: And I'm curious what drew you to that sort of work specifically.
• 3:47
  SPEAKER_01: Well, yeah, well, while uh I think it likewise.
• 3:50
  SPEAKER_01: So while we think about software development, we think about uh what can this do.
• 3:55
  SPEAKER_01: But I think when it comes to security, uh the question slips and uh it becomes what can in uh any anyone make this do?
• 4:03
  SPEAKER_01: So this later part of the question always, you know, uh excited me about software development itself, and it drew always drew to me the security side of the engineering.
• 4:13
  SPEAKER_00: Yeah, very interesting.
• 4:15
  SPEAKER_00: Well, let's go ahead and dig into some of the work that you've done on the project recently.
• 4:20
  SPEAKER_00: And I'd love to talk uh first about connection testing.
• 4:25
  SPEAKER_00: So I know this is a feature that was disabled by default in Airflow 2.7 after a CVE was reported on it.
• 4:31
  SPEAKER_00: And I guess for some background, I think probably most Airflow users know this, but this is the feature within the Airflow connections UI where you can input all of the information and then just say test connection.
• 4:41
  SPEAKER_00: So it in theory makes the connection to the external system, but you don't have to run a DAG that uses that connection in a task to see, oh, something about the way that I formatted my credentials or whatever doesn't work.
• 4:55
  SPEAKER_00: So pretty useful feature.
• 4:56
  SPEAKER_00: I have always said myself and my own experience as a data engineer working with Airflow, I think connections are one of the hardest parts.
• 5:03
  SPEAKER_00: It can just be really tricky to get all of the information in them right based on all of these different external systems.
• 5:10
  SPEAKER_00: But again, this feature uh still exists but is disabled by default.
• 5:15
  SPEAKER_00: Um so tell me a little bit, yeah, about your work on this and how what change you contributed and kind of what that does to the way connection testing works.
• 5:25
  SPEAKER_01: Yeah, yeah, for sure.
• 5:26
  SPEAKER_01: So uh while Airflow executes this connection testing, it uh loads the whole driver.
• 5:32
  SPEAKER_01: And I think in some cases it loads it can load the entire class and also you know can do an outbound request with the user-supplied values that uh that comes from that form.
• 5:43
  SPEAKER_01: And that form basically acts as uh you know uh run code in the server uh form.
• 5:49
  SPEAKER_01: And uh and this is specifically why it was disabled in back in uh there in 2.7, I believe.
• 5:56
  SPEAKER_01: And you know, it this connection runs in that time it ran in the web server, which was the most crucial process of the Airflow itself that had all the DB connection and the uh secrets, and that is why it was twist off.
• 6:09
  SPEAKER_01: Now uh with my um contribution, uh what it made is it it just uh made it asynchronous currently.
• 6:16
  SPEAKER_01: What it does is in current mechanism, not the new design, what it does is it makes a connection request uh to the API server, it executes that right there, and UI uh waits for the API server to come back.
• 6:31
  SPEAKER_01: Uh, but with the new design, uh the API server does nothing, uh just records the connection testing uh that it gets and uh hands back UI uh token and the ID so that it can pull uh for the results later on.
• 6:44
  SPEAKER_01: And now the execution part of it happens in uh the worker.
• 6:49
  SPEAKER_01: Uh and here uh you know uh it executes uh with the same mechanism that uh the current tasks execution uh machinery uses.
• 6:58
  SPEAKER_01: So yeah, this is what the actual chains look like.
• 7:02
  SPEAKER_00: Yeah, that makes sense.
• 7:04
  SPEAKER_00: So changing kind of the architecture of how this feature works sounds like also based on the new Airflow 3 architecture now that you do have this separate API server and that workers no longer directly access the metadata database.
• 7:16
  SPEAKER_00: So am I getting that right?
• 7:17
  SPEAKER_00: That because this runs in a worker, now you're not accessing those secrets directly.
• 7:21
  SPEAKER_01: Yeah, yeah, with this new design, it actually became a job, you know.
• 7:25
  SPEAKER_01: And with job, none of the credential passes along with it.
• 7:29
  SPEAKER_01: Uh, and when it reaches workers, it uh does uh API requests to uh the API server spill itself to get the details uh of the connection testing to run it.
• 7:39
  SPEAKER_01: And API server, the pro the worker doesn't have the privilege that API server has.
• 7:44
  SPEAKER_00: Okay.
• 7:44
  SPEAKER_00: Yeah, that's great.
• 7:45
  SPEAKER_00: I would imagine that makes a big de difference for a lot of organizations on whether or not they can use this feature at all or whether their security teams are okay with somebody enabling it.
• 7:55
  SPEAKER_00: Yeah, very cool.
• 7:56
  SPEAKER_00: And I also am curious about the process for making this change.
• 8:01
  SPEAKER_00: So I think uh changes like this that have kind of an architectural component, especially with something that has previously had um, you know, a CVE tied to it or has ongoing potential security issues, something like connections, you know, those often go through a discussion on the dev list uh for airflow and you know an architecture review.
• 8:22
  SPEAKER_00: So is that something you did for this?
• 8:24
  SPEAKER_00: And what did that process look like?
• 8:26
  SPEAKER_01: Yeah, this story is actually interesting.
• 8:28
  SPEAKER_01: So this all started with the PR from a different commuter.
• 8:32
  SPEAKER_01: Uh and uh, you know, it was a bug fix in the current uh connection testing feature.
• 8:37
  SPEAKER_01: There, I saw a comment from Jarek, uh, which whose comment was, you know, uh this the ideal fix is to move this into the worker self and not patch it here.
• 8:49
  SPEAKER_01: So he wrote the all history there.
• 8:51
  SPEAKER_01: I read that comment and I was like, I had to build this uh stuff.
• 8:54
  SPEAKER_01: And yeah, I I uh built up my uh uh design and uh I wired up it and I um took it to the mailing list, and then all the PMC uh members and the commuters came came in, and you know, my design got torn apart and it was even rejected at first, and then it uh got revived along the way on uh you know different iteration.
• 9:18
  SPEAKER_01: I mean, it takes time, but you know, the design that we have uh at last is is, I believe, good, and the thing that is checked in, I'm I'm really proud of.
• 9:27
  SPEAKER_00: Yeah, that's great.
• 9:28
  SPEAKER_00: Definitely, I think a strength of open source projects is the kind of consensus-driven approach, but as you describe, it can also make it quite challenging.
• 9:38
  SPEAKER_00: I have heard stories from lots of people who have come with an idea uh that was then sort of torn apart by a lot of the Airflow PMC or committers and then eventually land on something that is much stronger.
• 9:51
  SPEAKER_00: And so that's that's really cool that you were able to um come to that consensus and um deliver something that, like I said, is really meaningful to a lot of users.
• 10:00
  SPEAKER_01: Yeah, for sure.
• 10:02
  SPEAKER_00: Yeah, that's great.
• 10:03
  SPEAKER_00: Well, I also want to talk about some of the other security-related contributions that you have worked on recently.
• 10:10
  SPEAKER_00: And one of them I understand was this token revocation in Airflow 3.
• 10:15
  SPEAKER_00: I think compared to what we just talked about, this is something that happens much more on the kind of behind the scenes on the back end, so that users probably aren't even aware of if everything is going right.
• 10:26
  SPEAKER_00: Uh, so explain to me a little bit kind of what is token revocation, how does that work, and what was the problem with how it was handled before?
• 10:35
  SPEAKER_01: Yeah, so this basically uh is uh drills down to the nature of the JWT itself.
• 10:42
  SPEAKER_01: You know, it's stateless and server can validate uh JWT without maintaining any state at all.
• 10:48
  SPEAKER_01: And it can uh always validate it if uh the signature is correct, the token is fine.
• 10:54
  SPEAKER_01: So in Airflow, when user did the logout, um uh we just uh in Airflow 3 specifically, um, you know, we just removed that uh token from the browser itself, uh, but the token was still valid because it was it not it was not expired.
• 11:10
  SPEAKER_01: Uh well it has certain default time uh where it would expire, but it was it's not it was not expired for that particular time.
• 11:17
  SPEAKER_01: And you know, anyone uh that could get hold of that token uh you know could act as you and do operation in behalf of you, which which which was not correct uh and ideal.
• 11:28
  SPEAKER_01: Uh so we uh built a revocation system to eliminate this.
• 11:34
  SPEAKER_01: And this uh this revocation system uh in this revocation system we built a tiny table which is just basically a dead list, and uh where we uh you know uh do the capture all of uh the tokens that are expired.
• 11:51
  SPEAKER_01: And uh whenever user clicks log out now, uh we add a record there with its JTI and the expiry token.
• 11:58
  SPEAKER_01: And whenever subsequent request comes with the token, we check that uh the token against that table, and if it exists here, we just reject it.
• 12:06
  SPEAKER_01: So now with this mechanism in place, the logout is actual logout.
• 12:10
  SPEAKER_01: And I want to be a bit clear, this uh revocation is just for the UI tokens, not on the execution side of the tokens that we have.
• 12:18
  SPEAKER_01: And uh yeah, this is how uh we uh we are able to solve uh the actual revocation problem in Airflow 3.
• 12:27
  SPEAKER_00: Okay, got it, makes sense.
• 12:29
  SPEAKER_00: Uh yeah, so I guess, like I said before, uh a change that is completely on the back end.
• 12:34
  SPEAKER_00: So hopefully most users are not aware of this at all when they're working with Airflow, but definitely a big security implication.
• 12:43
  SPEAKER_00: Yeah, very cool.
• 12:45
  SPEAKER_02: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 12:54
  SPEAKER_02: From beta pipelines to training models and wiring up agents.
• 13:04
  SPEAKER_02: Check it out at astronomer.io.
• 13:08
  SPEAKER_00: And then I understand you have also worked recently on uh uh a project you split Airflow's tokens into two types.
• 13:17
  SPEAKER_00: Was this related to token revocation or is this something totally different?
• 13:21
  SPEAKER_01: Well, yeah, this was totally, totally different.
• 13:24
  SPEAKER_01: Yeah.
• 13:25
  SPEAKER_01: Um it it came out uh it was a very strange issue, actually.
• 13:30
  SPEAKER_01: And uh uh one use it was uh in this case, uh the tax was in the queue for a very long time.
• 13:39
  SPEAKER_01: Uh and uh whenever worker picked the tax again up, uh the tax was uh getting 403 on the first call to the uh execution API.
• 13:49
  SPEAKER_01: And the strange thing was, you know, the Airflow 3 already had uh a refresh uh token mechanism built in, and it uh you know refreshes the token on every subsequent request that comes from uh the uh you know workers.
• 14:05
  SPEAKER_01: Uh but in this case the token was living there and none of the API calls was made to the server, so token refresh never happened.
• 14:14
  SPEAKER_01: And uh as it never happened, it eventually died.
• 14:17
  SPEAKER_01: And for this very case, the user was you know running a GPU pipeline and his uh his tasks was failing uh after 15 minutes, and he was he was uh kind of confused why that was happening because refresh also happens.
• 14:31
  SPEAKER_01: Uh but and also he tried multiple things to it, uh like increasing the uh task queue timeout, which is basically a timeout uh in uh uh until which uh tasks can stay in the queue itself, uh, but that didn't work because it was not tied to the DWT uh expiration at all.
• 14:52
  SPEAKER_01: So actually, with to solve these kind of you know uh invariation, we have to split that token into two.
• 14:59
  SPEAKER_00: Okay.
• 14:59
  SPEAKER_00: Got it.
• 15:00
  SPEAKER_00: That makes sense.
• 15:01
  SPEAKER_00: And then so tell me how does how does splitting those?
• 15:06
  SPEAKER_00: You gave the one example, um maybe more generally.
• 15:11
  SPEAKER_00: So how how does splitting those impact like the end user?
• 15:15
  SPEAKER_00: What would they see working differently?
• 15:17
  SPEAKER_01: So uh well I would say previous uh previous token uh you know uh was one token and having two entirely different jobs.
• 15:28
  SPEAKER_01: That is one was staying in the queue alive and one was authentication itself.
• 15:33
  SPEAKER_01: And uh for uh staying in the queue alive, it needs longer time.
• 15:39
  SPEAKER_01: Uh and uh for you know for authentication it needs a shorter time, actually.
• 15:44
  SPEAKER_01: Uh so basically, and uh with the longer time token as it uh as a workload travels through you know this executor transport, it is uh uh in a you know on a bit of an untrusted environment like uh broker logs and uh the KDS spec itself.
• 16:00
  SPEAKER_01: So it has to be tightly scoped so that the leak token is you know uh kind of unusable.
• 16:06
  SPEAKER_01: And in this case as well, uh the workload token is kind of unusable uh totally because it can only do one thing that is run uh the tax and that's it.
• 16:16
  SPEAKER_01: Uh then when it reaches worker, the execution uh uh you know API already have swapped that token uh while the first request happened.
• 16:24
  SPEAKER_01: So yeah, and the actual uh solving part happened here when we already increased the length of the workload token to be identical to the tax queue timeout timeout because it is now the same.
• 16:37
  SPEAKER_01: The same configuration is picked up for both workload token lifetime and the queue type uh queue tax timeout.
• 16:44
  SPEAKER_01: Uh, so that you know the two clock doesn't fight with each other, it is the same now.
• 16:48
  SPEAKER_01: And uh now, however time uh the uh you know tax is in that queue, it won't uh face the issue that user was facing.
• 16:56
  SPEAKER_01: And also, we already have uh uh uh scope so narrow that security is already baked into it.
• 17:04
  SPEAKER_00: Okay, got it.
• 17:05
  SPEAKER_00: Yeah, that's great.
• 17:06
  SPEAKER_00: What what an impactful change.
• 17:09
  SPEAKER_00: And all all three of these things that we've just talked about, you've done you know, this year basically, or since kind of the end of last year.
• 17:16
  SPEAKER_00: Yeah, that's awesome.
• 17:18
  SPEAKER_00: Um I'm curious, do you have any uh security-related projects that you're currently working on uh with Airflow, or are these uh three uh updates that we just mentioned, are they completely done?
• 17:35
  SPEAKER_00: Are they concluded projects, or is there are there still optimizations that you're hoping to make or future work planned?
• 17:42
  SPEAKER_01: Yeah, so basically two token mechanism is already checked in and it's already working in production.
• 17:48
  SPEAKER_01: And also uh the revocation system is already in production.
• 17:52
  SPEAKER_01: But with the test connection, um the groundwork and the base architecture is checked in, but we uh and only local executor is now uh you know migrating to the new uh format, new job that we have.
• 18:03
  SPEAKER_01: We still have to migrate all of the executors and uh then uh just swap up the um last piece of UI uh and yeah, then then we are done on that part as well.
• 18:14
  SPEAKER_00: Yeah, that's great.
• 18:15
  SPEAKER_00: And for the two that are completed, I should have asked this earlier, but for people to take advantage of those, what version of Airflow do they need to be on?
• 18:24
  SPEAKER_01: Uh they have to be on 3.2 uh and a higher.
• 18:28
  SPEAKER_00: Okay, great.
• 18:30
  SPEAKER_00: Makes sense.
• 18:31
  SPEAKER_00: Yeah, that's awesome.
• 18:32
  SPEAKER_00: Well, and then I always love to talk to people who have started contributing to the project relatively recently about any advice that you would give to someone who is thinking about contributing to Airflow.
• 18:44
  SPEAKER_00: Uh, we talk to folks in the community all the time who want to contribute but feel a little intimidated by both the process, I think can seem quite complex when you're new to it.
• 18:55
  SPEAKER_00: And also I think people have some just sort of factor of, you know, do I want to go through the review process?
• 19:02
  SPEAKER_00: Are people going to accept my changes?
• 19:04
  SPEAKER_00: You know, uh all of that stuff.
• 19:06
  SPEAKER_00: So yeah, what what would you say to anybody who's potentially on the fence about contributing?
• 19:12
  SPEAKER_01: Yeah, for sure.
• 19:13
  SPEAKER_01: I mean, yeah, uh, I would say just start small.
• 19:17
  SPEAKER_01: Uh, there are lots of good first uh issue labeled issues in uh, you know, GitHub.
• 19:23
  SPEAKER_01: And I would say don't be afraid to pull out a PR because you know, in the review process, there is where you get mentored by the best engineers that we have in the community of Airflow.
• 19:33
  SPEAKER_01: And yeah, I would say just don't be afraid, pull out a PR and go get uh started.
• 19:38
  SPEAKER_01: That's it, I think.
• 19:40
  SPEAKER_00: Yeah, that's great.
• 19:41
  SPEAKER_00: I would echo that.
• 19:42
  SPEAKER_00: Uh certainly the advice I always give people is start small.
• 19:45
  SPEAKER_00: Uh that's what I have done for my own contribution.
• 19:48
  SPEAKER_00: Same with my team.
• 19:49
  SPEAKER_00: And you know, everybody on the Airflow PMC or who are reviewing PRs, they actually prefer smaller changes.
• 19:55
  SPEAKER_00: It's a lot easier to review.
• 19:57
  SPEAKER_00: A lot of the stuff that you've been describing is pretty big in terms of functional changes to Airflow, but a contribution certainly does not need to be that large.
• 20:06
  SPEAKER_00: And yeah, I think I've also heard from you know the folks at Astronomer who are PMC members now and some of the most heavy contributors to the project over its history that they also started by just submitting a PR and they got tons of feedback on it and had to go back and rework.
• 20:23
  SPEAKER_00: And, you know, they think it makes them stronger engineers.
• 20:25
  SPEAKER_00: So I would say to people also don't be afraid of the feedback.
• 20:29
  SPEAKER_00: Like nobody expects you to submit something that's perfect the first time.
• 20:34
  SPEAKER_01: Yeah, I totally agree.
• 20:35
  SPEAKER_00: Yeah, that's great.
• 20:37
  SPEAKER_00: Uh well, Anish, I related to contributing, I guess this can be either related to something that you are working on or maybe not, and it's completely different.
• 20:46
  SPEAKER_00: But I always end uh by asking a standard final question, and that is what would you most like to see from development of the Airplug project in the near future?
• 20:55
  SPEAKER_01: Yeah, you know, uh we just checked in this uh uh principle of two token mechanism where we scope things uh of different parts of the airflow really tight so that if any uh authorization or credential linked in, it would be totally useless to attacker.
• 21:10
  SPEAKER_01: So I think I would like to see more of these kinds of scoping uh spread upon all of the creational part of the airflow.
• 21:16
  SPEAKER_01: And yeah, I'll be looking forward for it.
• 21:19
  SPEAKER_00: Okay, great.
• 21:20
  SPEAKER_00: That's a great answer.
• 21:21
  SPEAKER_00: Awesome.
• 21:22
  SPEAKER_00: Well, this has been so great.
• 21:24
  SPEAKER_00: Thank you so much for coming on and talking about your contributions.
• 21:27
  SPEAKER_00: Again, I'll say I think these are really impactful for everybody using Airflow.
• 21:31
  SPEAKER_00: So the community, of course, really appreciates all of the time that you spent on them.
• 21:36
  SPEAKER_00: Uh, just to close this out, what is the best way for folks to get in touch with you?
• 21:39
  SPEAKER_01: Yeah, for sure.
• 21:41
  SPEAKER_01: Anyone can connect me in the GitHub or Airflow Slack uh or LinkedIn if they want uh to connect there.
• 21:47
  SPEAKER_01: I'm always open to talk, uh, have a coffee any anytime, yeah, for sure.
• 21:52
  SPEAKER_00: Okay, great.
• 21:52
  SPEAKER_00: We'll put links to those things in the show notes.
• 21:54
  SPEAKER_00: Uh Anish, thank you again so much.
• 21:56
  SPEAKER_00: It's been great speaking with you.
• 21:58
  SPEAKER_01: Yeah, it was great um being here and thank you very much for uh having me here.
• 22:03
  SPEAKER_02: Thank you for listening to the Data Flowcast.
• 22:06
  SPEAKER_02: Check the show notes for links to everything mentioned today.
• 22:10
  SPEAKER_02: If you're enjoying the podcast, subscribe so you never miss an episode.
• 22:14
  SPEAKER_02: And leave us a five-star review.