Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Role-Based Access Control (RBAC)

Transcript

Role-Based Access Control (RBAC) · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: Hi there, and welcome to episode seven of the Airflow Podcast, produced by Astronomer.
• 0:05
  SPEAKER_03: In the first six episodes of the series, we've begun a deep investigation into Apache Airflow and have examined many sides of the project.
• 0:13
  SPEAKER_03: Before we get started, I'd like to quickly mention that Astronomer recently released what we're calling Space Camp, an airflow training and education course designed for teams looking to run airflow at scale.
• 0:23
  SPEAKER_03: Flexible enough for both small and large data engineering teams, Spacecamp seeks to help realize the full potential of Airflow through the instruction of operational best practices and the aid and setup of production grade airflow clusters.
• 0:35
  SPEAKER_03: We cater to the entire spectrum of users, from the complete novice looking to get started with Airflow to more experienced folks just looking to step up their game or help their team get up to speed.
• 0:44
  SPEAKER_03: Feel free to shoot me an email at pete at astronomer.io if you'd like to see a sample curriculum.
• 0:51
  SPEAKER_03: Okay, now back to this week's episode.
• 0:54
  SPEAKER_03: We're really excited to continue our investigation this week with an exploration of the recent role-based access control work done for the project.
• 1:01
  SPEAKER_03: This time, we met up with WiiPay engineer Joy Gao, who handled the bulk of the RBAC work that was released as part of the recent Airflow 1.10 update.
• 1:10
  SPEAKER_03: For those that don't know, role-based access control is an approach to restricting system access to authorized users.
• 1:16
  SPEAKER_03: It's long been a desired feature for Airflow users to limit access to specific parts of their instance, and Joy played a huge part in making that dream a reality for the community.
• 1:24
  SPEAKER_03: Here she is talking about her background and what inspired her to do all this work.
• 1:28
  SPEAKER_03: Hope you enjoy.
• 1:44
  SPEAKER_00: Yeah, I'm pretty excited.
• 1:45
  SPEAKER_00: Thanks for reaching out too.
• 1:47
  SPEAKER_03: Yeah, absolutely.
• 1:48
  SPEAKER_03: So um I guess we can just start with some intros from our side.
• 1:51
  SPEAKER_03: Um I'm Pete.
• 1:52
  SPEAKER_03: I help run the Airflow podcast with Virage.
• 1:55
  SPEAKER_03: Um I work at Astronomer.
• 1:56
  SPEAKER_03: Uh I work on the product team and do a little bit of sales engineering, um, also some uh business development work as well.
• 2:03
  SPEAKER_03: So uh wear a bunch of different hats, you know, it's the startup life, but um really enjoy it.
• 2:10
  SPEAKER_00: That's awesome.
• 2:10
  SPEAKER_02: Yeah, I'm Viraj.
• 2:12
  SPEAKER_02: Um I'm more on the uh data engineering side, just using Airflow to help our customers.
• 2:18
  SPEAKER_02: Um also now apparently a uh an amateur podcaster.
• 2:25
  SPEAKER_00: Nice.
• 2:27
  SPEAKER_02: And then Taylor, you want to introduce yourself?
• 2:30
  SPEAKER_01: Oh yeah, I'm Taylor.
• 2:31
  SPEAKER_01: I'm kind of one of our senior software engineers on the back end, so I'm um whatever we need to work on for the Astronomer Cloud and the Astronomer Enterprise Edition.
• 2:42
  SPEAKER_01: But mainly a lot of Airflow stuff.
• 2:45
  SPEAKER_00: Nice, nice to see you guys.
• 2:47
  SPEAKER_03: Awesome.
• 2:48
  SPEAKER_03: So do you mind just giving us a little bit of an introduction for yourself?
• 2:51
  SPEAKER_03: Tell us a little bit about your background, kind of how'd you end up at WePay?
• 2:56
  SPEAKER_00: Um, yeah, so this kind of comes back to like how I even got started in computer science to begin with.
• 3:02
  SPEAKER_00: Um so I went to school uh in Canada at University of Waterloo, and uh even though it's kind of a university that's known for its computer science program uh in the Bay Area, I went to school for accounting.
• 3:14
  SPEAKER_00: So I had an accounting uh major.
• 3:19
  SPEAKER_00: Um and it's mostly because kind of my parents um always sort of guided me that I should um look for a job with stabilities and high securities and a pretty stable career.
• 3:32
  SPEAKER_00: And I also didn't have a whole lot of um opinions and I went with that.
• 3:37
  SPEAKER_00: Then luckily for my first uh internship, I didn't get into one of the big four accounting firms that I was interested in.
• 3:46
  SPEAKER_00: Instead, I worked as a business analyst at a small uh startup, uh tech startup in Toronto.
• 3:53
  SPEAKER_00: And for uh that startup, I really got what it's like um working uh in a fast-paced environment, what it's like working with engineers, and what it's like um to basically um working in the tech industry in general, and I got really interested with the fellow software engineers at my company, um, and I kind of wanted to learn how to code and get better.
• 4:19
  SPEAKER_00: Um I started learning how to code and basically build uh build things instead of just you know auditing other people's work.
• 4:27
  SPEAKER_00: Kind of uncertain about the whole decision, so it wasn't until uh two and a half years later when I decided that I didn't want to be an accountant, so I switched major.
• 4:38
  SPEAKER_00: I switched major twice actually.
• 4:40
  SPEAKER_00: Um first I switched into kind of this uh joint program of both financial management and computer science, and then later on, complete full switch into computer science because I wanted to take some of the senior CS courses that only CS majors had uh had the opportunity to take.
• 5:01
  SPEAKER_00: And how the transition was pretty hard at the beginning, but it was extremely rewarding, and I felt like I was actually enjoying my university years again.
• 5:10
  SPEAKER_00: And then finally, um when it comes to my final internship, I thought I really wanted to work in Silicon Valley.
• 5:18
  SPEAKER_00: So uh I uh the Y Combinator website, and then I saw a list of the startups that I had there, and I basically applied to a and and ended up uh getting a job with Sweet Pay.
• 5:30
  SPEAKER_00: And my summer internshiping experience was really enjoyable, and then I decided to go back for full-time, and I've been basically working there ever since.
• 5:39
  SPEAKER_03: Very nice.
• 5:40
  SPEAKER_03: That's an awesome story.
• 5:41
  SPEAKER_03: Um, really, really cool to hear about the switch from accounting to computer science.
• 5:45
  SPEAKER_03: I know um if I could go back, I might have done something similar.
• 5:48
  SPEAKER_03: I studied chemistry um in college, but now that I'm working in tech, it would be really nice to have a CS degree.
• 5:54
  SPEAKER_03: Um, so what are you currently doing?
• 5:56
  SPEAKER_03: I'm sorry, go go ahead.
• 5:58
  SPEAKER_00: Oh, I would just say it's always nice to kind of have it like a dual uh a dual degree or at least like knowing multiple um disciplines, it really helps for most software engineers to to have a better understanding and a broader scope as well.
• 6:12
  SPEAKER_00: So I definitely think it's good to to to have that accounting knowledge or to have that chemistry knowledge as well.
• 6:18
  SPEAKER_02: Yeah.
• 6:18
  SPEAKER_02: Pete and I both went to uh liberal arts school, so we're very much on that train.
• 6:22
  SPEAKER_02: Yeah, yeah.
• 6:23
  SPEAKER_03: So wait, so how long how long have you been at WePay then?
• 6:27
  SPEAKER_00: Uh I've been at the company for about two years.
• 6:30
  SPEAKER_00: I started with working on the core payment team, and uh within the past year I switched into the data infrastructure team.
• 6:39
  SPEAKER_03: Very cool.
• 6:39
  SPEAKER_03: So so kind of how'd you end up getting your hands dirty with Airflow to start?
• 6:44
  SPEAKER_00: Um yeah, so so when I'm working um at uh WePay on the data infrastructure team, uh we use Airflow as one of our main toolings for our ETL pipelines and for our data warehousing uh stuff.
• 7:00
  SPEAKER_00: So because of this, um uh and because at the time Airflow was pretty new to our company and uh pretty new, it's a pretty new project to begin with.
• 7:09
  SPEAKER_00: So I had to basically read a lot of documentations and kind of get my hands dirty into the source code to uh to do like small bug fixes and whatnot to help with our production pipeline.
• 7:22
  SPEAKER_00: Um and I started, I think my first commit was actually uh making a change to the README to make it a little bit uh easier to read.
• 7:31
  SPEAKER_00: Um and then after that, I started working on smallback fixes in your flow.
• 7:35
  SPEAKER_00: And then later on, I started to work on some of the command line features and adding operators and whatnot.
• 7:42
  SPEAKER_00: Um and then yeah, finally I got to work on the robase access control, the RBAC stuff.
• 7:48
  SPEAKER_00: Um it's been kind of like gradual uh and steady involvement uh with Airflow.
• 7:56
  SPEAKER_00: And um part of the reason uh we had to do this for WePay is because we're a pretty small company, we're pretty small for data infrastructure, and we couldn't really afford Airflow and then building our custom feature on top, our forked version, um, because of how small we are and because of how fast the project is.
• 8:18
  SPEAKER_00: Um we had to do basically every time we need to make a change, we contribute it back to open source first, get approved, uh, get that merged, and then we, if we need the feature immediately, we would cherry pick that feature into um our uh we our version of Airflow.
• 8:34
  SPEAKER_00: And then later on, when the next release happens, we'll basically um use the latest release.
• 8:39
  SPEAKER_00: So that's kind of how we um kind of forced me to uh get more and more involved with Airflow open source community.
• 8:46
  SPEAKER_02: Oh that's really interesting.
• 8:47
  SPEAKER_02: Yeah.
• 8:48
  SPEAKER_02: What have you been your what's your experience been like working with the Airflow open source community?
• 8:52
  SPEAKER_02: Everyone we've talked to so far has uh spoke pretty highly of it.
• 8:56
  SPEAKER_00: Uh yeah, it's uh it's I think it's a really friendly environment.
• 9:02
  SPEAKER_00: I do hear a lot of like horror stories about some of the other open source communities, um, but with Airflow, um everyone has been very open and welcoming uh and kind.
• 9:13
  SPEAKER_00: So it's it's been a pretty and uh it's it's very active, and we get, I would say, maybe at least 10 pull requests or so every single day, maybe five to ten pull requests every single day.
• 9:26
  SPEAKER_00: Uh so it's constantly um changing, which is pretty exciting for us.
• 9:32
  SPEAKER_03: Oh yeah, definitely very exciting.
• 9:34
  SPEAKER_03: So so we're we're really interested in hearing a little bit more about the RBAC stuff that you've been working on.
• 9:38
  SPEAKER_03: I mean, it seems like such a cool, such a cool initiative and something that everybody um seems to have a pretty immediate use for.
• 9:44
  SPEAKER_03: So what what what kind of inspired the work on that and why now?
• 9:49
  SPEAKER_00: Uh yeah, so um this has to go back to kind of the nature of our company, uh where we pay.
• 9:57
  SPEAKER_00: We're a fintech company, and um we have a lot of requirements when it comes to legal supply, illegal compliances and securities and whatnot, there really isn't a whole lot of room for mistakes um because that could potentially put us out of business.
• 10:12
  SPEAKER_00: Um so I think our engineers at our company tend to get a little bit more paranoid about security in general, and I think that's more of a good thing because um for anyone, if you have um if you're an Airflow user and you get uh access granted to the Airflow UI, um you can do pretty much any administrative things with Airflow.
• 10:37
  SPEAKER_00: So this basically means you can um you know change the state of a DAG run or task instance, like Mac like marking something from fail to success uh without the job actually succeeding.
• 10:50
  SPEAKER_00: Um you can start any job, you can trigger a job, schedule, um just ad hoc.
• 11:00
  SPEAKER_00: Um and you can also modify um some of the shared um variables, sorry, some of the shared in airflow, like variables and connections.
• 11:10
  SPEAKER_00: Um and if you make a mistake there, that could potentially impact many jobs that depended on those um shared objects.
• 11:16
  SPEAKER_00: And then finally, there's a feature, uh ad hoc query in Airflow, that um if it's configured to read the metadata dB, you can essentially have the full access to the database.
• 11:29
  SPEAKER_00: So there's a lot of things that we're not very comfortable with uh with the Airflow UI.
• 11:34
  SPEAKER_00: So what we had to do at the start was we had to limit the Airflow UI to only a couple of folks at our company.
• 11:41
  SPEAKER_00: And for a lot of the DAG owners, what uh it kind of sucks for them because uh they basically had to rely on email alerts and on our downstream logging system to do any kind of uh production debugging.
• 11:55
  SPEAKER_00: Uh they can't test their DAGs locally, um, they can test it in our POC environment, but when it comes to production, it's pretty much a black box.
• 12:04
  SPEAKER_00: And this kind of really defeats um one of the main attractive um features of Airflow was its UI.
• 12:12
  SPEAKER_00: And over time, um we decided that we don't, if we want to continue to use Airflow, we want to expose the UI to um everyone, uh but uh at different degrees.
• 12:23
  SPEAKER_00: So if you're an Airflow uh maintenance and maybe all you need to do, uh maybe you need to um work mostly with connections and variables and just um the uh making uh setting how I say this, um making sure that it's um um up and running and whatnot, then you should have the admin access.
• 12:43
  SPEAKER_00: Uh if you're in DAG on nurse and you don't really care as much about the uptime of airflow, on the other hand, you care more so about your specific DAGs, then you can have these read, write, and execute access to the DAG.
• 12:57
  SPEAKER_00: And for everyone else who are more like spectators of Airflow, they should have read-only act be allowed to make any modification because granted, people make mistakes, right?
• 13:07
  SPEAKER_00: So we we want to add this level of um granularity um to access control for airflow.
• 13:17
  SPEAKER_02: Yeah, um, I think a lot of what you hit on is definitely something that would extend the usability of Airflow to just more types of corporations and more teams.
• 13:26
  SPEAKER_02: Uh the UI, like you said right now, is definitely a bit of a double-edged sword because uh it's so it's so user-friendly that it's almost uh almost giving the user too much power.
• 13:36
  SPEAKER_00: Yeah, exactly.
• 13:37
  SPEAKER_03: So is there any way, have you heard of any any ways that people have kind of gotten around the lack of um role-based authentication until you know up to this point with Airflow?
• 13:48
  SPEAKER_03: Or have people just has everybody been granted you know uh high-level permissions?
• 13:53
  SPEAKER_00: Yeah, so from my kind of involvement in the community and talking to other Airflow users, uh my um impression is that most people, most users of Airflow, they just assume the best uh uh best intent of the engineers they're doing and they should not make do anything that isn't you know that is dangerous or risky.
• 14:19
  SPEAKER_03: For sure.
• 14:20
  SPEAKER_03: Um definitely a dangerous game.
• 14:25
  SPEAKER_00: Yep, go ahead.
• 14:26
  SPEAKER_03: Oh, I was just gonna say definitely a dangerous game, especially if you're you're kind of just getting your feet wet with airflow and poking around in a production instance.
• 14:34
  SPEAKER_00: Yeah, exactly.
• 14:35
  SPEAKER_00: And and I do have um some of the few a fewer companies that uh said they had they did something similar to us where they limited the UI access.
• 14:44
  SPEAKER_00: Um but I think for the most part, it was just an you know, you seem innocent until proven guilty.
• 14:51
  SPEAKER_00: And um and I think for maybe ever since I started working on RBAC, we've been getting a lot more positive feedbacks um and kind of like advocating uh for this feature.
• 15:04
  SPEAKER_00: And actually, there's someone in the community that is working on an RBAC, which is on the DAG level.
• 15:11
  SPEAKER_00: Um, we call it DAG level access control or DLAC.
• 15:15
  SPEAKER_00: And what this basically enables them to do is to um DAG uh access specifically to users um as opposed to um sorry, specifically to s to rows, but on a DAG level basically.
• 15:29
  SPEAKER_03: Oh, that's really cool.
• 15:30
  SPEAKER_03: Yeah, that's exciting.
• 15:32
  SPEAKER_03: So how do you can you talk a little bit uh how you approached a project from a technical standpoint?
• 15:36
  SPEAKER_03: So how'd you actually go about implementing it?
• 15:40
  SPEAKER_00: Um yeah, so we started off with an um Airflow RBAC proposal.
• 15:47
  SPEAKER_00: This was actually something written by another colleague of mine that worked a security team at our company.
• 15:53
  SPEAKER_00: Um and then when it comes to actually the implementation, we got together as a community.
• 15:57
  SPEAKER_00: We had this a really long um mailing list thread that taught that discussed this.
• 16:03
  SPEAKER_00: And um, we had a couple options to decide between.
• 16:08
  SPEAKER_00: So the the first option was we thought about uh uh switching from uh Flask into Django.
• 16:15
  SPEAKER_00: Um because Django is a more mature framework.
• 16:18
  SPEAKER_00: I think it has like five years lead ahead of Flask, so there are more um security-related and role-based access control related features that are out of the box with this extensions.
• 16:29
  SPEAKER_00: Um scratch that one because it's a lot of work um just to migrate the framework just for RBAC.
• 16:37
  SPEAKER_00: Um, it didn't seem like it was worth it.
• 16:39
  SPEAKER_00: And then the second option we consider was actually implementing it from scratch, like to designing the data models, um, models and implementing them ourselves.
• 16:49
  SPEAKER_00: The downside of this is that we know there are already a lot of existing solutions out there, and us working on it um seems like a just a huge duplication of effort.
• 17:00
  SPEAKER_00: So um we also kind of went against it because um Airflow's um security model is actually pretty standard, and it didn't seem like we need to really customize it uh for a need.
• 17:12
  SPEAKER_00: So we kind of scratched that one as well.
• 17:14
  SPEAKER_00: And then we looked into some of the existing uh libraries and Flask extensions.
• 17:20
  SPEAKER_00: There were um a couple of them.
• 17:22
  SPEAKER_00: I think one was uh Flask RBAC, and then another was Flask Principal.
• 17:27
  SPEAKER_00: Um, both of them had one um issue with our implement uh with our implementation, is that it requires you to hard code the roles with the permissions.
• 17:41
  SPEAKER_00: So the roles have to be and cannot be configured based on the business need, and that's not really ideal.
• 17:48
  SPEAKER_00: So finally, um we evaluated um Flask app builder, uh short for RBAC, sorry, short for Fab.
• 17:56
  SPEAKER_00: And this is something that Maxime actually uh suggested to begin with because he is using this uh pro he's using this um extension for um Apache Superset, which is another project that he worked on.
• 18:11
  SPEAKER_00: Um and uh they've had some proven success with it, and we evaluate we evaluated um Flask App Builder.
• 18:19
  SPEAKER_00: And uh what it so basically is very similar to Flask Admin, which is the current extension that we're using for the Airflow web server.
• 18:28
  SPEAKER_00: Um both Flask Admin and Flask App Builder are inspired by Django, so they're kind of similar in nature.
• 18:35
  SPEAKER_00: Um they both use SQL Alchemy as their um ORM layer, and they both have um support for Bootstrap and Jinja templating um and all that.
• 18:45
  SPEAKER_00: So uh the migration work actually wasn't uh as much.
• 18:49
  SPEAKER_00: But what makes uh Flask app builder um kind of better than Flask admin for our use case is that it has RBAC out of the box, it has user session management out of the box, it has indication backends um out of the box.
• 19:07
  SPEAKER_00: So it's a much more opinion-native framework and um it fit really well with our needs, so we basically decided to um migrate to FastCap Builder.
• 19:18
  SPEAKER_00: Oh, um yeah, and to talk about like the actual migration process.
• 19:25
  SPEAKER_00: Um basically, um, because they both use SQL Alchemy as the underlying ORM layer, we didn't have to touch any of the model classes.
• 19:35
  SPEAKER_00: The only thing we needed to change were the views.
• 19:38
  SPEAKER_00: Um so we had to change all the views such that they inherit from Flask app builder's face view instead of Flask Admins based view, and also modify um some of the attributes for each of those views.
• 19:51
  SPEAKER_00: And then the next thing we had to do was basically tweak some of the HTML and uh and JavaScript component such that they will work with this uh with a Flask app builder framework.
• 20:02
  SPEAKER_00: And then finally, um, which is what actually took most of the time for me was adding some of the missing features um that are in Flask Admin but are not in Flask App Builder, um, so that we can reach feature parity between the old UI and the new UI.
• 20:19
  SPEAKER_00: And these are things like um um oh yeah, so uh so for example, in Flask Admin, you can um pre-programs before submission, and that's something we needed to add to Flask App Builder.
• 20:35
  SPEAKER_00: Um also uh Flask App Builder didn't support um comp didn't support composite primary keys, and it didn't support blob data types, and it didn't support custom data types um for SQL Alchemy.
• 20:48
  SPEAKER_00: So we had to add all of those support to Flask App Builder.
• 20:51
  SPEAKER_00: And then I ended up basically spending a lot of time making FAP work uh working with a new UI, sorry, working with Airflow, and that's how we kind of got the new Wow, yeah.
• 21:03
  SPEAKER_02: That seems like quite a rabbit hole to go down.
• 21:06
  SPEAKER_00: Yeah.
• 21:07
  SPEAKER_02: A lot of different parts work in there.
• 21:11
  SPEAKER_02: Um so just kind of veering back to Airflow itself.
• 21:14
  SPEAKER_02: Um sorry, lost my time of thought.
• 21:18
  SPEAKER_02: Uh just like veering back to Airflow itself.
• 21:21
  SPEAKER_02: Can you talk about some of the how you got started with the like some of the pain points you had when you were getting ramped up with the project?
• 21:28
  SPEAKER_02: Uh just learning to write DAGs and building in the framework.
• 21:48
  SPEAKER_03: Sorry.
• 21:50
  SPEAKER_03: Okay, sorry.
• 21:51
  SPEAKER_01: Yeah, I think I'm getting some lag or something.
• 21:53
  SPEAKER_03: All right, I think are we good now?
• 21:57
  SPEAKER_03: Okay, cool.
• 21:58
  SPEAKER_01: Sorry about that.
• 22:00
  SPEAKER_01: Can you repeat the last question?
• 22:02
  SPEAKER_02: Yeah.
• 22:02
  SPEAKER_02: Uh so kind of just veering back to the using Airflow itself, uh can you talk about some of the learning process, uh getting a feet wet with like actually writing DAGs and how that process was for you?
• 22:14
  SPEAKER_00: Um yeah, so I think the hardest part was actually setting up the environment for um because it's um when I first started, Airflow was a fairly young project.
• 22:27
  SPEAKER_00: So the the the setup part had some uh bugs and issues, and that took kind of some time to get used to.
• 22:36
  SPEAKER_00: Uh, there were some dependency conflicts, and from what I can remember.
• 22:40
  SPEAKER_00: Um, once you can have a UI that's running locally, and we basically recommend everyone who started using uh Airflow to first try to play around with it in a local environment to create their first um DAGs and whatnot.
• 22:56
  SPEAKER_00: And the actual deck building part is pretty, my opinion, as long as you're familiar with um Python, um uh it's it's it's it's a pretty good documentation um on uh on Airflow as well.
• 23:13
  SPEAKER_00: Uh the other thing that I guess there's a little bit unconventional to get used to was the way start date uh works in Airflow.
• 23:21
  SPEAKER_00: And I think many people are familiar with it or came across it, uh got confused by it when it first started, um, which is that um airflows is based on um intervals rather than rather than pointing time.
• 23:34
  SPEAKER_00: So the start uh so a job executes only when an interval ends from its start time and rather than when the rather than when the when the job started.
• 23:44
  SPEAKER_00: So this caused a lot of confusion.
• 23:46
  SPEAKER_00: But once you get hang of it, it becomes pretty straightforward as well.
• 23:51
  SPEAKER_02: Very nice, yeah.
• 23:52
  SPEAKER_02: Yeah.
• 23:52
  SPEAKER_02: Yeah, definitely the start date stuff and then catch up being true by default always kind of threw me off.
• 23:59
  SPEAKER_00: Yeah.
• 24:00
  SPEAKER_01: Taylor, was there something that you wanted to ask?
• 24:03
  SPEAKER_01: I I just thought it was interesting.
• 24:04
  SPEAKER_01: Uh the point that Joy mentioned about getting airflow set up and just running was the hardest part.
• 24:11
  SPEAKER_01: Um, you know, at Astronomer, we've heard a lot of customers say that, and we invested, not to like plug the product too much or anything, but we really invested in our CLI, being able to make it so like users can just download the CLI and then uh just type astro airflow start or and then start.
• 24:30
  SPEAKER_01: Um, and like all the Docker containers spin up and Docker Compose happens automatically and all this cool stuff for you.
• 24:36
  SPEAKER_01: Uh, I'd just be curious to know uh what you guys use internally.
• 24:40
  SPEAKER_01: Do you have some sort of like Docker Compose setup people run locally, or do you just do like virtual environments or something like that?
• 24:48
  SPEAKER_00: Yeah, so uh in terms of um for developers just do learning airflow, we don't have a Docker environment for it yet.
• 24:58
  SPEAKER_00: Um we do have like basically a set of guidelines on how to set up your virtual m and how to how to run locally.
• 25:05
  SPEAKER_00: Um that's definitely something that I think we're going to um add later on.
• 25:11
  SPEAKER_00: Um as more people adopt Airflow, we do want to make the props as frictionless as possible.
• 25:17
  SPEAKER_01: And kind of on the same token, I don't know if you saw the news uh this past week about I think it was a contractor for one of the music companies had deployed an Airflow instance.
• 25:27
  SPEAKER_01: And long story short, they had their production instance up with dev settings.
• 25:32
  SPEAKER_01: So like the web server was exposed by default, no authentication and all that.
• 25:37
  SPEAKER_01: And then, you know, shortly after there was some discussion on the Airflow mailing list around should we make it um production ready out of the box, but like a little bit harder for a dev to start with locally?
• 25:48
  SPEAKER_01: Should we do more like 12 factor where there's um dev settings and prod settings, and you just get both and you pick?
• 25:54
  SPEAKER_01: I'd just be curious to know if you have any more thoughts around that.
• 25:59
  SPEAKER_00: Um I so uh to be honest, I don't work as much in on DevOps and deployment process of Airflow, so I don't have a whole lot of experience or opinions about it, but um I definitely think it would be beneficial to not make the UI automatically, and it should be a way to um to basically prevent that to be to begin with, because it seems kind of yeah.
• 26:30
  SPEAKER_01: I I thought the I kind of want to double back to the point you mentioned on Django.
• 26:35
  SPEAKER_01: So uh personally I got started with Django in like 2014, and I had done some Flask stuff before that.
• 26:41
  SPEAKER_01: So it was a lot of um like what you said, you know, when you use Django and Flask side by side, um Django is like so batteries included, right?
• 26:50
  SPEAKER_01: Like the ORM is there out of the box, and the Django admin is like very full featured, and you can do all these things, you know, models views, everything is built in.
• 27:00
  SPEAKER_01: You don't need a bunch of packages like you do with Flask, but you know, the flip side, um well, if the Django admin doesn't work for you and you want to sub out a different admin, like you don't really have alternatives to a lot of the stuff it that comes out of the box.
• 27:14
  SPEAKER_01: Whereas with you know, with like Flask, there's so many different extensions and plugins, and like you said, you guys went through this process of trying a whole bunch of different alternatives to arrive at the RBAC.
• 27:25
  SPEAKER_01: Um I guess I'd just be curious to hear a little bit more about uh what the Django discussion looks like and and how far you went down that path.
• 27:34
  SPEAKER_01: I never really thought about um using Airflow in Django itself, but what I have seen at a higher level is like a few years in the Django community, you just see and obviously uh Django's been around for a few years, like it's more mature, the docs are like very polished, there's tutorials for everything, there's talks at conferences every year, and I think I'm seeing like the same thing happen in Airflow that happened in Django, it's just we're like five years after, because you know Airflow is still relatively young.
• 28:05
  SPEAKER_01: Um I'd just be curious if if you had seen or what your thoughts are on like the broader movement of the community, if you think we're doing something like that, if Airflow is the next big Python framework like Django is, or maybe I'm like reading into it a little too much.
• 28:21
  SPEAKER_01: I know it was kind of an open-ended question, but does that make sense?
• 28:26
  SPEAKER_00: So I think a potential movement to Django.
• 28:32
  SPEAKER_00: Um, I think unless there is a strong reason to do so, it doesn't uh we we are not going to modify right away because I mean the reason that we kind of went against uh migrating to Django was because the only thing we wanted out of it at the time was RBAC.
• 28:53
  SPEAKER_00: And to migrate an entire framework, there are we're going to have to deal with new issues and new bugs.
• 28:59
  SPEAKER_00: And uh and for a project like um Airflow, I think it's um it and given how fast it was growing at the time, um, it didn't make a whole lot of sense to switch framework entirely just for a feature.
• 29:14
  SPEAKER_00: Um especially that's going to take some time, and then uh all the existing features are going to be um added to Flask while we are Django, for example.
• 29:25
  SPEAKER_00: Then when we are trying to merge it back, there's going to be new conflict.
• 29:28
  SPEAKER_00: And it's unless there's really strong reason, I guess, then um I don't see airflow moving towards the Django direction.
• 29:37
  SPEAKER_00: And I think Flask has its benefits as well.
• 29:40
  SPEAKER_00: Um because it's a very light uh lightweight framework, it allows us to give us more flexibility um to building features that we care about.
• 29:50
  SPEAKER_00: From that perspective, um, we didn't we haven't came across with any like major performances issues or anything with Flask.
• 29:57
  SPEAKER_00: So yeah.
• 29:59
  SPEAKER_01: Yeah, I think that's important.
• 30:01
  SPEAKER_01: That's a good point you bring up, is like it's a lot easier to get started with Flask.
• 30:05
  SPEAKER_01: You don't have that big upfront learning curve like you have with Django.
• 30:08
  SPEAKER_01: So maybe there's a better approach for you know a big community-driven open source project like this.
• 30:18
  SPEAKER_03: Definitely.
• 30:19
  SPEAKER_03: So, Joe, I know I know we're kind of reaching the end of our time slot here, and I don't I don't want to um go over it all, but I have one more question for you that I'd love to kind of hear your perspective about.
• 30:27
  SPEAKER_03: Um we've interviewed a ton of people for this podcast, kind of dating all the way back to you know August or September or whenever we got it started.
• 30:34
  SPEAKER_03: And you're actually the first woman we've talked to, um, which you know could be uh more of a testament to the larger tech space in general than it is just to the Airflow project.
• 30:42
  SPEAKER_03: But I'm just wondering if you think there's anything that the open source community could do to better support female devs.
• 30:49
  SPEAKER_00: Um yeah, so I remember like a couple of years ago, I think uh I read this study um about open source and um gender discrimination.
• 31:00
  SPEAKER_00: They basically did a study that um if your gender is hidden while you're making an open source contribution, your report request, um female on average has a slightly higher merge rate than male, um maybe like by five percent.
• 31:17
  SPEAKER_00: It wasn't anything like too obvious.
• 31:19
  SPEAKER_00: Um, but um if your gender is visible for your um uh then women uh for those women's the average um merge rate is lower than male.
• 31:33
  SPEAKER_00: Um also not by a whole lot, but uh still this study kind of just got me aware about the gender issue on the whole uh with open source.
• 31:44
  SPEAKER_00: I first started um uh making contributions to Airflow.
• 31:49
  SPEAKER_00: I intentionally created a new account.
• 31:52
  SPEAKER_00: Um, I had an old GitHub account, JoyGao, uh, and then I create a new account, JGao54, which basically hide away my name, which hide away my gender.
• 32:01
  SPEAKER_00: And then I changed my profile picture as well to like a gender neutral.
• 32:06
  SPEAKER_00: Um and that kind of got me feeling a little bit more brave about you know contributing to open source.
• 32:13
  SPEAKER_00: And of course, like that's not something that um I advocate women should hide our gender just to get our foot in the door.
• 32:23
  SPEAKER_00: The fact that I I felt the need to do it is kind of um, I think it's kind of a sign that there um is somewhat of a gender issue um with open source and just in general in the tech industry.
• 32:36
  SPEAKER_00: Um I think as a as a committer and as a moderator for an open source project, um it's much more about you know contributing like code to the project.
• 32:48
  SPEAKER_00: Um a lot of it is about in a really safe environment for people to make contributions to grow the community.
• 32:56
  SPEAKER_00: This means that if someone is you know harassing another person or it's just being having bad behaviors in an issue in a discussion, then it's important for us as committers to call it out and tell them that it's not okay to do that.
• 33:12
  SPEAKER_00: Um similarly, um if some when people create pull requests, um we want to um give them the credits and give them um give them like um um like thank them basically for them feel more welcome.
• 33:26
  SPEAKER_00: And on the other hand, if they're creating a pull request that isn't quite up to part, isn't quite what we want it, um, rather than ignoring it or just you know dismissing it, um what we can do is tell them what needs to be changed in order for this pull request to be merged and just providing constructive feedback on the whole.
• 33:43
  SPEAKER_00: So I think that would help as well.
• 33:45
  SPEAKER_00: And finally, um, for open source projects, I think it's really important to have um very clear guidelines for newcomers and contributors to tell them how they can contribute to this project.
• 33:57
  SPEAKER_00: Um, just have it written uh written as a README in the repository itself.
• 34:03
  SPEAKER_00: So it's uh easy to find and easy for people to contribute.
• 34:07
  SPEAKER_00: And yeah, I think I think um when it comes to female in open source and phenome in tech specifically, it's a much deeper problem.
• 34:16
  SPEAKER_00: Um I think it's a top of the funnel problem because we just don't think studying computer science and um applying to CS jobs, and obviously it's not fair for um the company right now to require uh 50% of the engineers to be female when only maybe 30% of our um resumes uh are coming from um females.
• 34:42
  SPEAKER_00: But I think um I'm seeing a lot of changes personally uh now with um with kind of like this whole movement towards gender equality.
• 34:51
  SPEAKER_00: And I think um it's going to get better over time, and it's a slow process, um, and it's going to be somewhat of a struggle, but I think we're in the right path together.
• 35:04
  SPEAKER_03: Absolutely.
• 35:05
  SPEAKER_03: Yeah, I think it's really interesting that you brought up kind of the top of the funnel point.
• 35:08
  SPEAKER_03: Taylor and I were actually just talking about that last week.
• 35:11
  SPEAKER_03: Um, just how kind of university education um systematically, you know, whether, you know, like not necessarily um purposefully, but systematically can sometimes uh discourage women from pursuing, you know, more technical paths.
• 35:26
  SPEAKER_03: Uh I definitely felt that in my undergraduate institution, and um it's definitely definitely seen, but but glad to glad to know that you think it's getting better and um looking forward to that.
• 35:38
  SPEAKER_00: Yes, thank you.
• 35:42
  SPEAKER_01: I thought it was uh yeah, I I thought it was interesting that she mentioned that it's a top of the funnel problem.
• 35:47
  SPEAKER_01: I I think, like you said, that's something we all kind of agree on.
• 35:50
  SPEAKER_01: And obviously there's a lot of discussion around that in tech right now.
• 35:54
  SPEAKER_01: But um another thing I've heard recently is like, yes, it's a top of the funnel problem, but we need more female role models out there.
• 36:02
  SPEAKER_01: And it's kind of interesting just to hear uh Joy talk through her path of like, okay, when I was contributing, I started by like fixing things in the README, and then I did like small bugs, and then I fixed tests, and then I did operators, and then next thing you know, she's working on the hardest parts of like core airflow.
• 36:19
  SPEAKER_01: So I I think to some degree, just having that out there and other people can see that in the community, I think that helps as well.
• 36:26
  SPEAKER_01: So if the next um female dev that has not contributed to Airflow comes along and she says, Oh, look, one of the core contributors is female, and look, she's already done all these impressive things.
• 36:37
  SPEAKER_01: It's like, oh, okay, it won't be too hard for me to get there, too.
• 36:40
  SPEAKER_01: I don't know, it's just a thought, but I think that it helps.
• 36:43
  SPEAKER_03: Oh, definitely.
• 36:44
  SPEAKER_00: I think actually, yeah, and that's kind of part of the reason why I changed back my profile picture um with like an actual photo of myself, because I think that would actually encourage more um female engineers to also contribute, knowing that there's um not just male committers and male working on airflow um and on print source in general.
• 37:10
  SPEAKER_03: Thanks to Joy for coming on and thanks to her for all of her hard work on the project.
• 37:14
  SPEAKER_03: It's much appreciated by everybody involved.
• 37:17
  SPEAKER_03: Thanks for tuning in this week, and as always, feel free to reach out to me at pete at astronomer.io with any podcast feedback or if you'd like to be considered for an interview.