Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli

Transcript

Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: ST is Python's abstract syntax tree.
• 0:02
  SPEAKER_02: So each time your Python code is being run just before it being run, you can think of it as it is divided into a tree of syntax elements.
• 0:11
  SPEAKER_02: So you have functions, you have definitions, and you have variable assignments and classes.
• 0:16
  SPEAKER_02: So each Python script or code is being built into a tree of syntax elements.
• 0:23
  SPEAKER_02: And that way, when you have a tree of syntax elements which is predefined, you know how to treat each syntax element, then you can analyze your dogs structurally.
• 0:49
  SPEAKER_00: Hey Airflow community, Airflow Theory is here and we are taking it on the remote.
• 0:54
  SPEAKER_00: Join the Astronomer Road Show for an exclusive deep dive into the latest features, hands-on memos, and expert insights, all in a city near you.
• 1:02
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:11
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:16
  SPEAKER_00: Register now.
• 1:17
  SPEAKER_00: The link is in the show notes.
• 1:20
  SPEAKER_03: Alright.
• 1:21
  SPEAKER_03: Hello, hello everybody.
• 1:23
  SPEAKER_03: Welcome to the Data Flowcast.
• 1:24
  SPEAKER_03: My name is Mark Averti.
• 1:26
  SPEAKER_03: I'm your host.
• 1:26
  SPEAKER_03: And today I'm joined with Snear, Senior Data Engineer at Next Insurance.
• 1:32
  SPEAKER_03: How are you doing, Snear?
• 1:34
  SPEAKER_03: Hi, hi Mark.
• 1:35
  SPEAKER_03: Thank you for having me.
• 1:36
  SPEAKER_03: I'm doing great.
• 1:37
  SPEAKER_03: How are you?
• 1:38
  SPEAKER_03: I'm doing really great.
• 1:40
  SPEAKER_03: I'm very excited to have you in this episode, honestly.
• 1:43
  SPEAKER_03: Because for those who don't know about that, Snear wrote a very nice article about LinkedIn and so on, which is truly amazing, as you will discover in that episode.
• 1:55
  SPEAKER_03: For today's podcast with SNER, you will learn how to transform chaotic Airflow data pipelines into standardized maintable code through custom EST-based linking techniques that can dramatically improve engineering team efficiency and scalability.
• 2:11
  SPEAKER_03: As you can imagine, out of this episode, you will learn a lot of stuff that will be very useful, especially if you run Airflow in production today.
• 2:19
  SPEAKER_03: That being said, I would like to start by maybe actually a little bit of background about you, Snear.
• 2:25
  SPEAKER_03: What do you do today?
• 2:26
  SPEAKER_03: Where are you working and so on?
• 2:29
  SPEAKER_02: Yeah, my name is SNER.
• 2:30
  SPEAKER_02: I'm a data engineer at Next Insurance.
• 2:33
  SPEAKER_02: Before that, I also worked as a data engineer and a data engineering team lead and other companies.
• 2:38
  SPEAKER_02: And yes, uh very much fond of Airflow, to say the least.
• 2:44
  SPEAKER_02: I always use Airflow in prior works, like I think it's pretty much standard in our industry.
• 2:52
  SPEAKER_03: That's right.
• 2:52
  SPEAKER_03: That's awesome.
• 2:53
  SPEAKER_03: That's awesome.
• 2:54
  SPEAKER_03: So I guess as a data engineer, as there are many challenges that we have to face every day.
• 2:59
  SPEAKER_03: And definitely one of the challenges is how can I standardize my data pipelines with Airflow?
• 3:06
  SPEAKER_03: So can you tell us more about the specific challenges that you had and that led to the solution that you're gonna talk about later?
• 3:15
  SPEAKER_03: Like what kind of challenges you had that led to the standardization, to the need of finding a solution to standardize your data pipelines?
• 3:25
  SPEAKER_02: Yeah, absolutely.
• 3:27
  SPEAKER_02: So I think we can start with that in X Insurance, we have quite a big data engineering group.
• 3:34
  SPEAKER_02: We have lots of engineers, and we started noticing a kind of inconsistencies or growing inconsistency in how DAGs are written in the group.
• 3:45
  SPEAKER_02: Each engineer has different style, they have their convention.
• 3:49
  SPEAKER_02: Each of us like to do things a bit differently.
• 3:52
  SPEAKER_02: And sometimes, and we found out that it's really more than sometimes, it's pretty often, it made it hard to maintain chattered dogs.
• 4:02
  SPEAKER_02: So uh sometimes you need to take care of other dogs that you not necessarily written.
• 4:09
  SPEAKER_02: And it made maintainability really hard and also slowed down onboarding to other DAGs.
• 4:16
  SPEAKER_02: And yeah, it introduced practically a lot of a lot of slowness, also in code reviews, that we thought can be avoided as long as we have standards.
• 4:30
  SPEAKER_02: So that was our main idea.
• 4:33
  SPEAKER_02: And we realized that we need some kind of enforcement layer, right?
• 4:36
  SPEAKER_02: We want to enforce good practices, we want to enforce how things are being built or how they are generally structured in order for each of us to have good time and step into other people's dugs uh in a breeze.
• 4:52
  SPEAKER_02: And that's really where linting comes in, if for any of us who's not very familiar, it's a basically a tool that we can programmatically scan our code to and for best practices, style violations, code smells, anti-patterns.
• 5:09
  SPEAKER_02: And we thought, why not use it like we use it for Python, right?
• 5:13
  SPEAKER_02: And for other languages.
• 5:14
  SPEAKER_02: So we thought why not have it for Airflow?
• 5:17
  SPEAKER_02: Yeah, so that's why we that's all the challenges we had, and that led us to think about this linting tool.
• 5:24
  SPEAKER_02: This we called it Daglint internally, and yeah, like you said, it is using Python's own AST or abstract syntax tree, which is really fascinating.
• 5:36
  SPEAKER_03: Okay, that this is great, and just like for context, how many DAGs are we talking about?
• 5:41
  SPEAKER_03: How many DAGs do you manage, and maybe what is the size of your team?
• 5:45
  SPEAKER_02: Yeah, so we are almost, I say quite big because in in comparison to other companies I worked, but we have around 20 engineers divided into different teams dealing with different domains of the business.
• 6:00
  SPEAKER_02: Uh, but we are we have in production running, I think, close to 200 drugs.
• 6:08
  SPEAKER_02: Some of them, yeah, some of them are bigger, some of them are more simple.
• 6:13
  SPEAKER_03: But yeah, this is the yeah, so that that's actually more than enough to uh to have the need of standardizing the data pipelines.
• 6:20
  SPEAKER_03: You know, you mentioned about some challenges that you had.
• 6:23
  SPEAKER_03: Do you have any examples of actually bad practices or things that you saw in your data pipelines that led to your solution, the Daglint?
• 6:31
  SPEAKER_02: Yeah, so I like to divide it into three categories.
• 6:35
  SPEAKER_02: So, first, they're more style and conventions.
• 6:39
  SPEAKER_02: We wanted it to make it very easy.
• 6:41
  SPEAKER_02: So we want to, for example, enforce that task IDs for tasks in Airflow share the same name as the variable that holds them.
• 6:50
  SPEAKER_02: Or they are all lowercase, and DAGs are all lowercase, so they're not messing around in the UI and we're having our time finding them just because of capitalization and everything.
• 7:02
  SPEAKER_02: So that's the first category.
• 7:04
  SPEAKER_02: It's more like style and conventions.
• 7:06
  SPEAKER_02: The second category is more like around quality.
• 7:10
  SPEAKER_02: So we wanted to ensure all of our DAGs have a doc stream, like a doc MD in every markdown file with the real documentation, how to use the DAG, a playbook, what is it all about, what's the source, what's the output.
• 7:25
  SPEAKER_02: And these are more of the quality of the DAGs.
• 7:29
  SPEAKER_02: The third category, which I think is the most challenging, but also the most rewarding, usually is to identify anti-patterns, okay, airflow, and like anything that can maybe impact the performance of Airflow or the DAGs themselves.
• 7:47
  SPEAKER_02: For example, I think the clearer example is having slow code running at the top level of the DAG file.
• 7:55
  SPEAKER_02: Yeah.
• 7:56
  SPEAKER_02: So we db connection opening, not just for the Airflow Meta store, but also we had DAGs that communicate with external services like API just to build the graph.
• 8:07
  SPEAKER_02: So these are the examples of anti-patterns and things that might be harder to figure out and to flag that they are bad, but if we solve them, we will improve dramatically both our airflow instances and our DAGs.
• 8:24
  SPEAKER_03: Would you say that this third category is the one that got the most that has been the most impactful on your on the quality of your DAGs?
• 8:35
  SPEAKER_03: Or even the first category was already a huge win when you implemented it?
• 8:41
  SPEAKER_02: Yeah, I think the first category, which with the convention was the low-hanging code.
• 8:46
  SPEAKER_02: These were also the rules that were the most simple and easy to implement.
• 8:51
  SPEAKER_02: And we got into standard really quick.
• 8:55
  SPEAKER_02: The third category, which was very like much harder to implement, it drove the impact in terms of performance or resource optimization that we immediately saw, even in our AWS bill, for example.
• 9:10
  SPEAKER_02: So yeah, I think every category has its own impact, but on different aspects and areas.
• 9:16
  SPEAKER_03: Interesting.
• 9:17
  SPEAKER_03: So regardless of the category, you will say, if, for example, I'm just I'm very interested by what you say and I want to implement the same solution, you will say, okay, go to the first category.
• 9:27
  SPEAKER_03: This is already a big step.
• 9:28
  SPEAKER_03: Even if you are not able to implement the third category yet, the first one is already a huge improvement.
• 9:34
  SPEAKER_02: Yeah, I totally look up in conventions and also the quality category, like having a documented DAGs and everything.
• 9:43
  SPEAKER_02: This will itself get you to a way better place and way easy maintenance for your DAGs.
• 9:50
  SPEAKER_02: The third category is we'll save you kind of money, right?
• 9:57
  SPEAKER_03: That's right, that's right.
• 9:59
  SPEAKER_03: That's cool.
• 10:00
  SPEAKER_03: I guess maybe I forgot, but I'm not sure we actually defined what is AST.
• 10:06
  SPEAKER_03: Yeah, so can you tell us more about that?
• 10:08
  SPEAKER_03: I know about AAST because I actually use AST for automatically checking the assignments that learners do on the Astronomer Academy with Airflow.
• 10:18
  SPEAKER_03: But I would love to know if you can tell me more about AST for the audience.
• 10:22
  SPEAKER_02: Yeah, sure.
• 10:23
  SPEAKER_02: So uh AST is Python's abstract syntax tree.
• 10:27
  SPEAKER_02: So each time your Python code is being run just before it being run, it is you can think of it as it is divided into a tree of syntax elements.
• 10:37
  SPEAKER_02: So you have functions of definitions and you have variable assignments and classes and whatever.
• 10:43
  SPEAKER_02: So each one of them is each Python script or code is being built into a tree of syntax elements.
• 10:52
  SPEAKER_02: And that way, when you have a tree of syntax elements, which is predefined, the you know how to treat each syntax element, then you can analyze your DAG structurally and not just by regex or line by line or start to trying to find a piece of text in your DAG, which is very don't do that.
• 11:13
  SPEAKER_02: Yes.
• 11:14
  SPEAKER_02: Don't do it.
• 11:15
  SPEAKER_02: If you have your DAG, not just DAGs, but if you have your Python script structured into a tree of syntax elements, then you can really build custom rules tailored for your needs, and you can really test whatever you need and not rely on how people will write whatever they want to write.
• 11:30
  SPEAKER_02: Maybe it's lowercase, uppercase, and regex at all.
• 11:33
  SPEAKER_03: Yeah, no, this is super powerful.
• 11:35
  SPEAKER_03: For those who don't know this concept, AAE, definitely take a look at that concept because it's a really nice way, to your point, verify your Python code, and this way you can just apply any rules that you want, honestly.
• 11:48
  SPEAKER_03: So this is super powerful.
• 11:49
  SPEAKER_03: When I discovered that, I'm gonna be honest, like I discovered this concept pretty recently, and this is super, super uh useful.
• 11:57
  SPEAKER_03: I guess so.
• 11:58
  SPEAKER_03: You mentioned that you have created this lint, and that's the name of your solution, which is a linter that applies different rules to your data pipelines using AST to ensure the quality of the data pipelines and that there is no anti-pattern.
• 12:13
  SPEAKER_03: But I guess as like solutions are great, but they are really great when they are applied, when they are used, I would say.
• 12:20
  SPEAKER_03: How like beyond the technical implementation, how did you encourage uh adoption of your solution of Dag Vinting across your engineering teams?
• 12:28
  SPEAKER_03: Because as you mentioned, I think there are like many different schemes.
• 12:32
  SPEAKER_03: Uh, how do you ensure that they are gonna actually use the DAG Inter?
• 12:37
  SPEAKER_02: Yeah, this is a great point because I think adoption is always the harder part, right?
• 12:41
  SPEAKER_02: So you like build and especially in a tool like this that basically aims to slow down engine from doing whatever they want.
• 12:51
  SPEAKER_02: So adoption was a very key part here.
• 12:54
  SPEAKER_02: So it's interesting you're bringing this up and you immediately saw the potential issue here.
• 13:00
  SPEAKER_02: What we wanted to focus on when we created it is that the developer experience will be really good.
• 13:06
  SPEAKER_02: Because if it will be bad, nobody will use it.
• 13:09
  SPEAKER_02: Even if we will force their hands to use it.
• 13:13
  SPEAKER_02: If it's bad, then it's just a matter of time until we finally say we don't want to use it.
• 13:20
  SPEAKER_02: So what we decided to do to make the developer experience really good is to make it super fast, super easy, and a very immediate, quick and uh uh actionable feedback loop.
• 13:37
  SPEAKER_02: Okay.
• 13:37
  SPEAKER_02: You run this locally when you develop your DAGs, the same way that you push a commit or whatever in your terminal, you just run the command, and even the you can integrate it with your IDE so you don't need to run any command.
• 13:50
  SPEAKER_02: But as soon as you write your code, you get a clear uh indicator whether you are violating any rule or you are not.
• 13:58
  SPEAKER_02: So by making it in the command line and making it fast, you can just fix the issue and then immediately run it again and that it is solved.
• 14:08
  SPEAKER_02: And to make it a bit more fun, we used a grading mechanism.
• 14:14
  SPEAKER_02: We didn't want to say you're violating X rules.
• 14:16
  SPEAKER_02: We gave each DAG or each pipeline a score.
• 14:20
  SPEAKER_02: Okay.
• 14:21
  SPEAKER_02: You're like, I think you're naturally want to everything to be 100%, right?
• 14:26
  SPEAKER_02: Well, you write it at 100.
• 14:28
  SPEAKER_02: So I don't want to settle for an 80 or for a 92.
• 14:32
  SPEAKER_02: And so we did that as well.
• 14:34
  SPEAKER_02: And once we got a good feedback, then we decided okay, we can step that up and make it as part of our CI CD pipelines.
• 14:42
  SPEAKER_02: So we utilize GitHub workflows.
• 14:44
  SPEAKER_02: So every time a developer opens a PR, if the PR involves any DAG file, then this DAG file immediately being calculated and being graded with DAGLint.
• 14:56
  SPEAKER_02: So the both the reviewer and both the DAG author can see immediately the score of the DAGs they are about to change.
• 15:05
  SPEAKER_02: And once you know that, then it was really easy for us to say, okay, if your score is getting worse, or if you're getting less than 100, then you are not able to merge this PR.
• 15:20
  SPEAKER_02: Okay.
• 15:21
  SPEAKER_02: And by making it really clear and making it fast to fix violations and to identify them, it made it maybe easier to do such an enforcement.
• 15:35
  SPEAKER_02: And that led us to the third phase, which we wanted to make it visible for everyone and trying to put a bit of gamification into it.
• 15:46
  SPEAKER_02: So we created like a dashboard that lets you see each team, what's the score of each title of the tags, and then every Sunday we would look at this and say, hey, now this team is number one, now this thing is number one.
• 15:58
  SPEAKER_02: So it became like a it's a bit of a game.
• 16:01
  SPEAKER_02: Yeah, so it's it was a fun touch.
• 16:06
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:15
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:27
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:35
  SPEAKER_03: So how does it work?
• 16:36
  SPEAKER_03: Because, like, for example, if I write a data pipeline and for some reason I'm not reaching 100, am I gonna be able to push my DAG?
• 16:44
  SPEAKER_03: Is there a minimum?
• 16:45
  SPEAKER_03: Did you define a minimum, or do I have to reach 100 if I want to push my DAG?
• 16:51
  SPEAKER_02: At first, when we latched it, then we said then you must have a minimum of 60, right?
• 16:56
  SPEAKER_02: To make it production worthy.
• 16:59
  SPEAKER_02: But as it progressed and as we gained a trust in the tool, then we made it like a zero or nothing, like 100 or nothing.
• 17:07
  SPEAKER_02: If you're not 100, then you're not getting into production.
• 17:11
  SPEAKER_02: I would add that we eventually introduced a mechanism to skip certain rules or certain violation of the same rule because we started to see that this change might be in a it will make the scope of the task very big for that that are not yet standardized.
• 17:30
  SPEAKER_02: So we made it possible to skip several that's but then the reviewer also sees that there is a skipping of an error violation and they need to give their consent.
• 17:41
  SPEAKER_02: So we did introduce this auction.
• 17:44
  SPEAKER_03: That's amazing.
• 17:45
  SPEAKER_03: You just said that there are some rules that you can skip, but I guess there are some rules that are more important than others.
• 17:52
  SPEAKER_03: And if so, do I get more points if I'm applying those rules or like how is the grading?
• 17:59
  SPEAKER_03: But starting with maybe the most important rules if you have yeah.
• 18:03
  SPEAKER_02: So now each rule violation gets the same weight to the score, but we did think about maybe giving more emphasis on other rules than maybe the more simple rules or more easy to nice to have rules.
• 18:20
  SPEAKER_02: So we thought about it, we still didn't implement it, but we do have a set of rules that they are absolutely mandatory and you cannot even skip them.
• 18:29
  SPEAKER_02: So yeah, but it's a really good uh good idea to have different scales of points to each rule.
• 18:38
  SPEAKER_03: And I guess so.
• 18:39
  SPEAKER_03: You actually said that you gamified a little bit the linter, which I find that really amazing.
• 18:44
  SPEAKER_03: Obviously, once you have this board with the different teams and the scores, you want to be the best team.
• 18:51
  SPEAKER_03: I hope so.
• 18:52
  SPEAKER_03: Can you can you tell us maybe a little bit more about this dashboard?
• 18:57
  SPEAKER_03: And also, like, is there any additional metrics that you track on that dashboard?
• 19:00
  SPEAKER_03: Is it only the scores or is there anything else?
• 19:03
  SPEAKER_02: So the main metrics are the scores, and so the dashboard we built is both operational and for monitoring purposes.
• 19:11
  SPEAKER_02: So if we want to really dive in and we did things like that, for example, by like the dashboard can be operational and we had a really good use of it.
• 19:24
  SPEAKER_02: I'll give an example.
• 19:25
  SPEAKER_02: We wanted to uh migrate all of our DAGs to start using a new class that we built to do something.
• 19:33
  SPEAKER_02: So we wanted to make sure that each DAG is gradually increasing and finishing this migration.
• 19:39
  SPEAKER_02: So what we did is implement a specific rule to make sure that every change in a DAG file that was not migrated yet will be forced to migrate.
• 19:50
  SPEAKER_02: So whenever you touch it, and that way we really made the dashboard operational because you can filter on specific rule and see all the DAGs that violate it per team and what's their scores.
• 20:02
  SPEAKER_02: So you can really like create tickets based on that impact and everything, and or like in the monitoring side of this dashboard, then you have the main metrics, which are scores per team and DAGs per team, DAGs above certain score.
• 20:20
  SPEAKER_02: Uh, but yeah, the main metrics are the scores, and we have different colors based on the average score for each team.
• 20:28
  SPEAKER_03: Trying to make it that's good to know.
• 20:31
  SPEAKER_03: Just truly on a technical perspective, like how that dashboard is implemented.
• 20:37
  SPEAKER_02: I would say first that the dashboard is built in Tableau.
• 20:40
  SPEAKER_02: So we have a dashboard in Tableau visible for all our team and also management.
• 20:46
  SPEAKER_02: And the way we power it is we have a daily pipeline or even twice a day.
• 20:51
  SPEAKER_02: It scores all of our DAGs in our in the repo, in the master branch, it gives each of them a score, and then it stores the result into a table we can then use for the dashboard.
• 21:06
  SPEAKER_02: So every day we score the DAGs, and then you get these updates on a daily or semi-daily or whatever frequency we really like.
• 21:14
  SPEAKER_03: Okay, okay, I see, I see, I see.
• 21:16
  SPEAKER_03: Because for some reason in my head, I thought that the dashboard was by React or something like that, but there is definitely an easier solution than that, especially when you're a data engineer, you don't want to touch on front-end uh engineering.
• 21:27
  SPEAKER_03: I did that in the past and I will not do it again.
• 21:29
  SPEAKER_03: All right, so I guess one quick question.
• 21:32
  SPEAKER_03: Do you feel that now you have this solution in place?
• 21:34
  SPEAKER_03: Do you feel that for new engineers that onboard in your team, do you feel it's easier for them to actually know your guidelines and everything?
• 21:42
  SPEAKER_03: Did you actually find that it it improved the process of the onboarding process for new engineers?
• 21:48
  SPEAKER_02: Yes, 100%.
• 21:49
  SPEAKER_02: I think that a new engineer that even if they don't read any documentation we might have placed to how we write DAGs, but we want to make sure it self documents it because.
• 22:00
  SPEAKER_02: Because if they wrote some messy code, they will immediately face with the results of a bad score, and then they will be forced to improve the score.
• 22:08
  SPEAKER_02: I think we saw a really big improvement on onboarding time.
• 22:12
  SPEAKER_02: Also, code reviews are faster.
• 22:13
  SPEAKER_02: I think we don't need to deal with or discuss and maybe have disagreements about some things that we already agreed are standards.
• 22:23
  SPEAKER_02: So this is the standard and we all align to it.
• 22:26
  SPEAKER_02: And also for other engineers to interact with different teams' DAGs or even different team members of the same team, different DAGs, it's I think it's a bridge because it makes it the same language, right?
• 22:41
  SPEAKER_02: It doesn't feel like a foreign language.
• 22:42
  SPEAKER_02: This team builds it that way, our team doing this that way.
• 22:46
  SPEAKER_02: It's really helped collaboration, I think.
• 22:49
  SPEAKER_03: That's a very good point.
• 22:50
  SPEAKER_03: You have the guidelines, everything is standardized, you know what is right, you know what is wrong.
• 22:54
  SPEAKER_03: And there is nothing to discuss about it because to your point, you already agreed on that in the past, so no need to discuss about it again.
• 23:02
  SPEAKER_03: So I love that.
• 23:03
  SPEAKER_02: It reminds me of there is the formatter for Python called Black, and their or their slogan is this is not opinionated.
• 23:12
  SPEAKER_02: So this also not opinionated.
• 23:14
  SPEAKER_02: This is a standard question to follow it.
• 23:17
  SPEAKER_03: I agree.
• 23:18
  SPEAKER_03: I guess actually, is there any downsides of using a lighting?
• 23:23
  SPEAKER_03: Like, can you think about any downsides?
• 23:27
  SPEAKER_02: Yeah, I think there are downsides.
• 23:29
  SPEAKER_02: Both even if we made the developer experience amazing, we are still can get you slower, right?
• 23:36
  SPEAKER_02: Sometimes you just want to get your task done or there is a production issue you now need to focus on, and you don't need to focus on standardization on other things.
• 23:47
  SPEAKER_02: Like you just want to fix that.
• 23:49
  SPEAKER_02: And when you are blocked by something that you said it's less important now, we just need to fix that production error, then it can be a bit of a downside.
• 23:58
  SPEAKER_02: So we did, as I said, introduced ways to skip that.
• 24:02
  SPEAKER_02: And if you don't skip this, our team leads have the way to override the protection on the branch, and then they can merge it in in terms of if there is an emergency or they see how they see fit.
• 24:17
  SPEAKER_02: So we did empower it and try to reduce the downside of Linting.
• 24:21
  SPEAKER_02: And also we need to maintain these rules as because it's a matter, it's not a matter of if but when airflow will change, and maybe if something will change, and we'll need to adapt as long as it is not part of the of Airflow itself.
• 24:40
  SPEAKER_03: Yeah, that's right.
• 24:41
  SPEAKER_03: That's right.
• 24:41
  SPEAKER_03: Actually, did you deprecate any rules?
• 24:43
  SPEAKER_02: We did deprecate the two rules.
• 24:45
  SPEAKER_02: One opened was a it's not a real deprecation because it was a rule we created it just for a migration, but another one we deprecated because we found that we did like a group discussion and we voted and we said that was what we thought very is is something we need to enforce, we change online.
• 25:07
  SPEAKER_02: And it's the that's right, exactly.
• 25:11
  SPEAKER_03: And I guess uh to your point, I as a data engineer myself, I imagine that if I have a linter, obviously it adds frictions, and maybe at the beginning I will be like a bit upset about that.
• 25:24
  SPEAKER_03: I just want to push my DAG or I just want to write my DAG, and that's it, I just want to push it.
• 25:28
  SPEAKER_03: But even if it adds a bit of friction at the beginning, I think on the long term, it's actually like you will save a lot of time, right?
• 25:37
  SPEAKER_03: It's better, I think, to lose a bit of time when you are writing your DAGs than losing a lot of time debugging your DAGs because of rules that you haven't implemented at first.
• 25:49
  SPEAKER_03: Yeah, and totally agree.
• 25:51
  SPEAKER_03: And you know what it is, right?
• 25:52
  SPEAKER_03: Like when you don't have any standardization, it's very easy that your DAGs just become extremely messy.
• 25:59
  SPEAKER_03: So yeah.
• 26:01
  SPEAKER_03: I guess like maybe one last question to you Do you have any advice to people that to maybe leaders and so on that would love to implement a similar solution but actually are concerned about adding friction to the process or maybe having some discussions with the team and so on?
• 26:20
  SPEAKER_03: Like, what do you have any advice?
• 26:23
  SPEAKER_02: Yeah, I think if I need to think about it, I would say start small, right?
• 26:27
  SPEAKER_02: And and really put an emphasis on developer experience because it if it won't be easy to use and if it won't be meaningful, then people will just hate it, especially of the nature of a lintum, right?
• 26:40
  SPEAKER_02: Like it can make you slower and it can annoy you.
• 26:43
  SPEAKER_02: So make it fast, make it run locally, provide clear output.
• 26:48
  SPEAKER_02: And also, I think being like before we joked about it, but having it as part of a democratic process that everyone is voting on the rules really strengthened the involvement of all the team members and really makes everyone on the same page and understanding why the rules are important and why we should enforce them.
• 27:08
  SPEAKER_02: So I think this is really helpful.
• 27:11
  SPEAKER_02: So yeah, start small, build, like progress, and make sure it is easy to use and even fun.
• 27:20
  SPEAKER_03: Yeah, even fun.
• 27:22
  SPEAKER_03: So I love that.
• 27:22
  SPEAKER_03: I love the fact that you emphasized the the developer experience.
• 27:25
  SPEAKER_03: Adoption is key, right?
• 27:27
  SPEAKER_03: Adoption is key in everything, right?
• 27:28
  SPEAKER_03: You can build solutions, as many solutions as you want, but if people don't use the your solution, then it's just a waste of time for everybody.
• 27:36
  SPEAKER_03: People will wonder if your solution is open source.
• 27:39
  SPEAKER_03: So I have to ask.
• 27:40
  SPEAKER_02: I know the answer, unfortunately, but yeah, unfortunately, it is not open source.
• 27:46
  SPEAKER_02: We did have thoughts making it open source, but we this will mean also a lot of effort and time on our end to to really make it like uh ready to be consumed by others and make it like that you can easily extend on it, and it's very generic because right now it's pillared around our needs, but we definitely have it on our minds and on maybe on our mindset to maybe do it.
• 28:12
  SPEAKER_02: But I also heard that as part of working on Airflow 3.0, yeah, some developers are considering having such a thing built in.
• 28:22
  SPEAKER_02: So this is really exciting.
• 28:24
  SPEAKER_02: Right.
• 28:24
  SPEAKER_02: This will end up there.
• 28:27
  SPEAKER_03: Yep.
• 28:27
  SPEAKER_03: Yeah, actually, I think I think something like that is coming very soon for airflow 3.0.
• 28:32
  SPEAKER_03: But for those who wonder how to implement a similar solution than yours, I actually recommend anybody to go to your medium article.
• 28:41
  SPEAKER_03: I think the name, I don't have the name in mind, but you will find it if you type Airflow Linter Medium on Google.
• 28:48
  SPEAKER_03: This is the first post that you will see.
• 28:50
  SPEAKER_03: And I really recommend you to read that article because you will find how to implement, I will say, your solution because you even give some code snippets.
• 28:59
  SPEAKER_03: You see the dashboard as well, the CI CD process.
• 29:02
  SPEAKER_03: So even if it is not open source, I think it's more than enough to build your own solution, which is great.
• 29:08
  SPEAKER_03: So thank you so much, uh SNER, for that.
• 29:10
  SPEAKER_03: This is already the end, unfortunately.
• 29:13
  SPEAKER_03: Thank you so much for joining, SNEAR.
• 29:15
  SPEAKER_03: I guess what is the best way for folks to get in touch with you if they want to learn more about your solution or anything?
• 29:23
  SPEAKER_02: Yeah, I would say that everyone who's interested about it, like you said, read the article and shoot me down a message and LinkedIn.
• 29:30
  SPEAKER_02: And yeah, and I'm happy to share ideas and also learn from you guys, maybe suggestions for rules and stuff.
• 29:37
  SPEAKER_02: Like the only reason we did it open source it is because we don't feel like it is crowd ready, but we are I'm happy to share everything I know and everything we did in order to make it happen.
• 29:50
  SPEAKER_02: And then maybe if you have any questions and you need any help, then I'm happy to yeah, share.
• 29:56
  SPEAKER_03: I told you guys SNER is amazing.
• 29:59
  SPEAKER_03: Thank you so much.
• 30:00
  SPEAKER_03: Thank you so much.
• 30:01
  SPEAKER_03: It was a wonderful episode, and I hope to see you very soon.
• 30:04
  SPEAKER_03: Thank you, Mar.
• 30:05
  SPEAKER_03: Thank you for having me.
• 30:06
  SPEAKER_03: Thank you.
• 30:09
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 30:16
  SPEAKER_01: You'll find links in the show notes to any researches mentioned in today's show.
• 30:20
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.