Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich

Transcript

Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_04: My favorite part is that it has all the sources and the links to the source.
• 0:04
  SPEAKER_04: For example, if the new task is a Python operator that runs a certain script, so at the bottom of the DAG documentation, it will say the create table task is run by this script, linked to its GitHub file in this directory, and this task is run by this SQL, and here's the source of that SQL.
• 0:22
  SPEAKER_04: So because the same way it's able to create the tasks, it just takes all that information and throws it into a doc file.
• 0:44
  SPEAKER_00: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:49
  SPEAKER_00: Join the Astronomer Road Show for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:57
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:06
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:10
  SPEAKER_00: Register now.
• 1:11
  SPEAKER_01: The link is in the show notes.com.
• 1:27
  SPEAKER_01: How are you doing, Gil?
• 1:29
  SPEAKER_04: I'm doing well, thank you.
• 1:30
  SPEAKER_04: And I'm very excited too.
• 1:32
  SPEAKER_04: And congratulations on the release of Airflow 3.
• 1:35
  SPEAKER_04: All right.
• 1:37
  SPEAKER_04: And you're a legend here at Wix because most of us learned how to program Airflow by being told, here, watch these Mark Lamberti videos.
• 1:44
  SPEAKER_04: Thank you.
• 1:46
  SPEAKER_01: Thank you so much.
• 1:46
  SPEAKER_01: Thank you so much, Gil.
• 1:47
  SPEAKER_01: And indeed, as you said, Airflow 3.0 is out, so I recommend anybody to try it out.
• 1:52
  SPEAKER_01: You will enjoy that new version.
• 1:55
  SPEAKER_01: Alright, so that episode is going to be amazing because basically you will learn how to create more maintainable, efficient, and testable Airflow implementations through framework development, as you will discover later.
• 2:06
  SPEAKER_01: Gil implemented a really actually Gil created a framework that I'm super interested by.
• 2:12
  SPEAKER_01: And you will see some techniques and optimization strategies that significantly reduce execution time for your data preparation tasks.
• 2:22
  SPEAKER_01: So that being said, Gil, can you tell us more about you actually?
• 2:26
  SPEAKER_01: What do you do at Wix.com today?
• 2:29
  SPEAKER_01: And yeah, just a little bit of your background.
• 2:31
  SPEAKER_04: Okay, sure.
• 2:32
  SPEAKER_04: I'm originally from New York.
• 2:33
  SPEAKER_04: I'm living in Israel.
• 2:35
  SPEAKER_04: I've been writing code professionally for almost 35 years, since probably before most of the listeners here were born.
• 2:42
  SPEAKER_04: And I'm a data engineer for the wonderful data science team at Wix, where I basically create pipelines or manage, help the data scientists and the data scientist users run their models and get their data.
• 2:55
  SPEAKER_01: That's amazing.
• 2:56
  SPEAKER_01: Just like for information, like how big is your team today?
• 3:00
  SPEAKER_04: How big is our team?
• 3:01
  SPEAKER_04: The data science and engineering team itself is as I think four people now.
• 3:06
  SPEAKER_04: The data science team is I think we're about 50.
• 3:09
  SPEAKER_04: The data engineering team is 100 or something.
• 3:11
  SPEAKER_04: So it's a big company.
• 3:14
  SPEAKER_01: Okay.
• 3:14
  SPEAKER_01: So I guess like you have many airflow instances running, right?
• 3:18
  SPEAKER_04: Yes, yes.
• 3:19
  SPEAKER_04: I think we have I had the numbers.
• 3:22
  SPEAKER_04: It's many.
• 3:22
  SPEAKER_04: We have many pipelines running.
• 3:24
  SPEAKER_01: Yeah.
• 3:25
  SPEAKER_01: Okay, great.
• 3:26
  SPEAKER_01: So I guess I would like to start with this question for you.
• 3:29
  SPEAKER_01: When you joined uh Wix, you actually inherited from pretty complex Airflow codebase.
• 3:35
  SPEAKER_01: And what were your biggest lessons learned when you had to basically to start with all of that code base?
• 3:42
  SPEAKER_01: What were the lessons that you learned?
• 3:44
  SPEAKER_01: And what advice would you give to a team starting to experience like those kind of challenges when you inherit from a big codebase like you had?
• 3:52
  SPEAKER_04: Okay, yeah.
• 3:53
  SPEAKER_04: I think the biggest lesson I learned was it's very easy to create new airflow DAGs by copying and pasting existing airflow decks.
• 4:02
  SPEAKER_04: It's uh it's easy, it's safe.
• 4:04
  SPEAKER_04: In the short term, it's a brilliant idea.
• 4:06
  SPEAKER_04: You're not going to break anything existing.
• 4:08
  SPEAKER_04: You're copying code that worked for somebody else on something similar.
• 4:11
  SPEAKER_04: So it all feels great.
• 4:12
  SPEAKER_04: It's easy, it's a great way to get moving, but in the long term, it's a disaster.
• 4:17
  SPEAKER_04: And that that's the main thing I learned.
• 4:18
  SPEAKER_04: You just can't do it.
• 4:19
  SPEAKER_04: It is, it's very tempting.
• 4:21
  SPEAKER_04: And uh six months ago, my best advice for here would be stop copying and pasting code, or at least if you're tempted to copy and paste, stop and think about it and make sure it's really what you want to do.
• 4:31
  SPEAKER_04: And especially if there are already two or three or four versions of what you're copying and pasting, definitely stop and start again.
• 4:38
  SPEAKER_04: Now the problem is worse because now I go to PyCharm, I type two words, and GitHub Copilot already gives me the next, which is which is in some ways wonderful.
• 4:49
  SPEAKER_04: And in some is in short term wonderful and long term a disaster because it's just more and more duplicate code.
• 4:56
  SPEAKER_04: So I think you just really need to be on the lookout for duplicate and near duplicate code and realize that it's going to hurt you in the long run.
• 5:03
  SPEAKER_04: And uh, you and your future self will, and the people who inherit your code will thank you for taking control of it.
• 5:08
  SPEAKER_04: And you can start small.
• 5:10
  SPEAKER_04: We didn't start with building a framework.
• 5:11
  SPEAKER_04: We started with small refactoring.
• 5:13
  SPEAKER_04: We took the worst problems and we created variables and functions and whatever, and then we kept going.
• 5:18
  SPEAKER_04: And every small win led to inspired us to keep going until we had a framework that we were really happy with.
• 5:25
  SPEAKER_01: Yeah, and actually that's very true.
• 5:26
  SPEAKER_01: The interesting fact about your answer is that it is not related to airflow, it's just related to programming in general, right?
• 5:32
  SPEAKER_01: Like you just should avoid duplicating code as much as you can, because otherwise you're gonna end up with this massive code base with a lot of duplicates everywhere, and it's gonna be uh just a nightmare for you to you know to maintain that.
• 5:45
  SPEAKER_04: I think for Airflow, it was a little it was more challenging for me because it's an orchestrator, because it's doing a lot of clever and intelligent things that were not always intuitive to me.
• 5:54
  SPEAKER_04: It was the correct way to do them, but a lot of things were counterintuitive to me.
• 5:58
  SPEAKER_04: It was always hard for me to remember what gets evaluated in the operator, what has to be evaluated before, what when things happen, when you can resolve variables.
• 6:06
  SPEAKER_04: All these things were so confusing to me that I couldn't resist the temptation of copying an existing, exist working bag.
• 6:13
  SPEAKER_01: Yeah, yeah, I did the same as well.
• 6:15
  SPEAKER_01: So I guess one quick question for you because you mentioned a framework, and we're gonna circle back to that in a second.
• 6:22
  SPEAKER_01: But first, you also said that you started with small optimizations before creating that framework.
• 6:28
  SPEAKER_01: So can you maybe tell us a little bit more about that?
• 6:31
  SPEAKER_01: What did you start doing before creating that framework?
• 6:35
  SPEAKER_04: Okay, so let me give a quick example of the most egregious problem that was in our code.
• 6:39
  SPEAKER_04: Okay.
• 6:40
  SPEAKER_04: It was that so every day we all of our processes were creating these temporary tables.
• 6:46
  SPEAKER_04: Okay.
• 6:46
  SPEAKER_04: Okay, so every process creating temporary table.
• 6:49
  SPEAKER_04: And so originally we appended the DS, the timestamp to the or the date stamp, the execution date to the table.
• 6:56
  SPEAKER_04: And then we said, well, that's not enough because that's not unique, because you may be running a backfill and a daily run in the same day.
• 7:02
  SPEAKER_04: So you had to add the type.
• 7:04
  SPEAKER_04: And then we said, that's not enough because sometimes you run twice on the same day.
• 7:07
  SPEAKER_04: And what winds up happening is we actually had the same ginja template among like 30 different tables.
• 7:14
  SPEAKER_04: So we had 30 different tables that were defined by constant that was specific to that table, and then a ginja template that started as one variable and just kept growing and growing and growing.
• 7:24
  SPEAKER_04: And then we said, that's not enough.
• 7:26
  SPEAKER_04: We need to add the full timestamp with the uh dates and seconds.
• 7:28
  SPEAKER_04: And we said that didn't work because that has a T in the middle or the formatting was wrong, and it didn't create a proper table name.
• 7:34
  SPEAKER_04: And so everything wound up being more and more until we had this enormously ugly and difficult, ridiculous, which is in some ways was worse, because we weren't even consistent always with single quotes and double quotes.
• 7:45
  SPEAKER_04: So even searches and replaces became hard.
• 7:48
  SPEAKER_04: So it just became insane.
• 7:49
  SPEAKER_04: And hadn't, again, because ginsa templates and what gets evaluated when was so confusing to me.
• 7:54
  SPEAKER_04: So I didn't understand until somebody told me you can just make a variable for that.
• 7:58
  SPEAKER_04: So we just made a variable for what we're appending to the table names, one variable, updated in one place, and everything else uses that variable.
• 8:06
  SPEAKER_04: So that's something that's super obvious when you realize you can do it, and it just made the code so much more manageable.
• 8:13
  SPEAKER_01: Yeah, to your point, I mean, there are like some basic features, I would say, in Airflow that you should definitely leverage because that will help you a lot to make your DAGs cleaner, your tasks cleaner.
• 8:24
  SPEAKER_01: And variables is a good example of that.
• 8:26
  SPEAKER_01: Okay, that's definitely an interesting uh example.
• 8:29
  SPEAKER_01: I maybe that wasn't enough to avoid duplicating your code.
• 8:33
  SPEAKER_01: And I know that this is because we had a discussion previously, obviously, but and I know that this is when you decided to create a framework to actually help you combating uh duplicate code.
• 8:47
  SPEAKER_01: Can you tell us more about that?
• 8:49
  SPEAKER_01: Could you work us through the structure of this library and how it helps maintain clean and reasonable code in Airflow?
• 8:57
  SPEAKER_04: Okay, sure.
• 8:58
  SPEAKER_04: So again, so when we started, we had three orchestrators, three different files creating three very similar orchestrators and seven different files creating very similar, of course, similar framework or similar structured DAGs and a bunch of different copies of it append to history function, all these things.
• 9:14
  SPEAKER_04: So we created a framework that would organize all these things in the center of the framework is a class called DSDAG.
• 9:21
  SPEAKER_04: That DS is created by us, the data science team.
• 9:24
  SPEAKER_04: That's the central class that creates the DAGs and their tasks.
• 9:27
  SPEAKER_04: We have a DAG manager to class that the manages the DAG.
• 9:31
  SPEAKER_04: So there's one per project.
• 9:32
  SPEAKER_04: My favorite class is the doc creator class that we realize at some point that if we're creating, if we're automatically creating the DAGs, then we might as well create the documentation as we're doing it.
• 9:43
  SPEAKER_04: Yeah, which is great.
• 9:44
  SPEAKER_04: And that that's actually the main reason why, even on small, because the framework is great for complex projects.
• 9:50
  SPEAKER_04: Main reason I also use it for small projects is to get the automated documentation, which is fun.
• 9:57
  SPEAKER_04: And then a few other classes, but though those are the big ones.
• 10:00
  SPEAKER_04: And uh, I'll just give an example of what the DSDAG class does is many of our pipelines have have three different tasks for have for trigger model, wait for model run to complete, and uh and validate model results.
• 10:14
  SPEAKER_04: That's a common pattern.
• 10:16
  SPEAKER_04: So now we have one function inside DSDAG that's called create run model tasks, and that creates a Python operator to tr to asynchronously trigger a model.
• 10:25
  SPEAKER_04: It creates a Python sensor to see when that model finishes, and it creates another Python operator to run a validation code.
• 10:32
  SPEAKER_04: So it basically what became what was a lot of code duplicated in a lot of places became one very simple function call.
• 10:38
  SPEAKER_01: Okay, so that function call creates the three tasks that you just told about?
• 10:43
  SPEAKER_04: Yeah.
• 10:44
  unknown: Yeah.
• 10:45
  SPEAKER_01: Okay, I see.
• 10:46
  SPEAKER_01: And you actually mentioned the models.
• 10:48
  SPEAKER_01: Just to make sure that what we are talking about here, what do you mean by models?
• 10:51
  SPEAKER_01: Do you mean machine learning models?
• 10:53
  SPEAKER_01: Do you mean dbt?
• 10:54
  SPEAKER_01: What do you mean?
• 10:54
  SPEAKER_01: Machine learning models.
• 10:55
  SPEAKER_04: Oh, machine learning.
• 10:57
  SPEAKER_04: So most of the pipelines we run are running machine learning models in their center.
• 11:01
  SPEAKER_01: Okay, okay, okay.
• 11:02
  SPEAKER_01: And so you have a bunch of functions that you can call to generate tasks that you will automatically add to your DAGs.
• 11:11
  SPEAKER_01: Is that right?
• 11:12
  SPEAKER_01: Yes.
• 11:12
  SPEAKER_01: Okay.
• 11:13
  SPEAKER_01: And like how does it work exactly?
• 11:16
  SPEAKER_01: Like, can you tell us more a little bit about the implementation here?
• 11:19
  SPEAKER_01: Do you use a configuration file?
• 11:21
  SPEAKER_01: Do you just import the function in your DAG right away?
• 11:24
  SPEAKER_01: And then you call the function, and that's it.
• 11:26
  SPEAKER_01: Can you tell us more about the implementation here?
• 11:28
  SPEAKER_04: Sure.
• 11:29
  SPEAKER_04: So it's a so basically we separated the code into sort of three parts.
• 11:33
  SPEAKER_04: Or the first division is we took the shared whatever code could be shared, whatever functions could be shared, we put in the framework.
• 11:40
  SPEAKER_04: And then within each project, we separated between the code and the configuration.
• 11:44
  SPEAKER_04: So the config file, the can the Python dictionaries have all the things like table names and settings and things like that.
• 11:51
  SPEAKER_04: And then the code for each project actually it imports the library, it imports library and creates code on the library.
• 11:58
  SPEAKER_04: So for example, you create your DSDAGMANager object and you pass it the configuration, the conf the config object.
• 12:04
  SPEAKER_04: And then you do DAGMANager.create DAG and give it the name of the DAG.
• 12:08
  SPEAKER_04: And then you take these DAGs and you add you call the create tasks functions on each DAG.
• 12:14
  SPEAKER_04: So for example, then you so DAGMANager equals DSDAGMANager passing fig file.
• 12:20
  SPEAKER_04: Then DAG is dagmanager.create dag with the name of the DAG, and then DAG dot create tasks, create runmodel tasks.
• 12:28
  SPEAKER_04: And that's okay.
• 12:29
  SPEAKER_04: I see.
• 12:30
  SPEAKER_04: So basically there is three lines of code that really creates already a DAG with the run, wait, and validate tasks to it.
• 12:38
  SPEAKER_01: Okay, okay.
• 12:39
  SPEAKER_01: This is amazing.
• 12:40
  SPEAKER_01: So and that configuration file, is it like a YAML file or is it uh Python dictionary?
• 12:44
  SPEAKER_04: It's a Python.
• 12:45
  SPEAKER_01: Python dictionary.
• 12:46
  SPEAKER_01: Okay, okay.
• 12:47
  SPEAKER_01: And so I guess like you have a bunch of Python dictionaries corresponding to your uh configurations stored somewhere, and then you can Yeah, so for each project is one configuration object or one configuration file for each project.
• 12:59
  SPEAKER_04: Okay.
• 13:00
  SPEAKER_04: For example, is the main project as that that triggered this whole project, that inspired this whole project.
• 13:05
  SPEAKER_04: It was an orchestrator with uh with seven internal DAGs.
• 13:08
  SPEAKER_04: So I end up with with one configuration which has sections for each internal DAG, and then the DAG manager knows how to pull out each part for each internal DAG.
• 13:19
  SPEAKER_01: Okay.
• 13:20
  SPEAKER_01: This is super interesting because like usually in my previous episodes, most of the time I came across people that were generating DAGs using like a Python script, right?
• 13:30
  SPEAKER_01: Like they just write a Python script and that generates the corresponding code.
• 13:34
  SPEAKER_01: But this is not exactly what you do, right?
• 13:36
  SPEAKER_01: Like you actually have a Python function that just returns when I say just that returns tasks, and basically like those tasks are the some common patterns that you have in your data pipelines, and so you are calling functions that return tasks.
• 13:53
  SPEAKER_01: You are not calling a function that generates a DAG.
• 13:57
  SPEAKER_01: It's just uh it's actually different than what I used to see.
• 14:00
  SPEAKER_04: Yeah, I actually I started in that direction, and uh and my teammates talk me out of it, and they talk me into this direction, which I'm very happy about because this way it's understandable.
• 14:10
  SPEAKER_04: It's not one super function that creates all these DAGs out of anything from a computer.
• 14:14
  SPEAKER_04: It's no no, there's a program that has to go in and decide what task groups he wants to create, and uh and I'm much happier this way, and I find the code much easier too to understand and debug, etc., and maintain.
• 14:26
  SPEAKER_03: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 14:35
  SPEAKER_03: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 14:47
  SPEAKER_03: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 14:55
  SPEAKER_01: Did you create that framework for avoiding duplicating code?
• 15:01
  SPEAKER_01: Or did you actually create that framework to have a sense of standardization?
• 15:06
  SPEAKER_01: So even like data scientists, for example, can create DAGs as well, or maybe both.
• 15:11
  SPEAKER_01: Did you create that framework for both reasons?
• 15:14
  SPEAKER_04: Yeah.
• 15:14
  SPEAKER_04: My hope was that the entire data engineering team would start to just use this framework.
• 15:19
  SPEAKER_04: I thought it just made things much easier, especially within the data science group, because we're using so many of the same types of pipelines that this I hoped would make all the DAG writing much easier.
• 15:30
  SPEAKER_01: And so what about when you need to add a new task that is not in your framework?
• 15:35
  SPEAKER_01: Like, how does it work?
• 15:36
  SPEAKER_01: Do you add that task to your framework or just do you like how do you think that's it?
• 15:41
  SPEAKER_04: Yeah, so good question.
• 15:43
  SPEAKER_04: Right now, the complication is that we created a package out of the framework.
• 15:48
  SPEAKER_04: And what that means is that it's not that simple to just add a function to it and use it because it winds up being all sorts of deployment issues and it might be all sorts of things.
• 15:59
  SPEAKER_04: So what I do for right now is I either make the code locally and also change the framework so that it so I first have it locally, that is within the project, and which it can also be sometimes I can derive, I can derive from the DSDAC, drive from the package and add to a derived class of the package, and uh, and then when I get a chance, update the framework and then use that instead.
• 16:22
  SPEAKER_04: What we may also do is just bring it all back is get rid of the deploy package and just put the framework back as a separate directory inside of our code base.
• 16:32
  SPEAKER_04: And this way I can just go and update that more easily and use that.
• 16:36
  SPEAKER_01: Okay, that's cool.
• 16:37
  SPEAKER_01: And I guess just uh maybe one last question regarding the framework.
• 16:40
  SPEAKER_01: What would you say are the pros and cons of using that solution?
• 16:45
  SPEAKER_04: The main pros are that it's much easier for me.
• 16:49
  SPEAKER_04: Once you get used to it, it's easier to write the DAGs with, especially the complex DAGs I've been writing.
• 16:53
  SPEAKER_04: It's much easier to write and maintain them.
• 16:55
  SPEAKER_04: It's a much easier to have one place for all the code that's shared.
• 16:59
  SPEAKER_04: It's self-documenting, which is a great thing.
• 17:02
  SPEAKER_04: And it's not gen AI documentation, it's not approximate documentation, and it's not documentation that can be outdated, it's documentation that gets regenerated dynamically each time the DAG is generated.
• 17:12
  SPEAKER_04: So the documentation is always accurate and always up to date, which is fantastic.
• 17:16
  SPEAKER_04: How do you do that?
• 17:18
  SPEAKER_01: Because I almost missed that part.
• 17:20
  SPEAKER_01: So how can how do you do that?
• 17:23
  SPEAKER_04: Okay.
• 17:24
  SPEAKER_04: So when the DAG creator is creating the DAG and adding tasks, it also adds to the MD file that's sitting in the in the same directory.
• 17:34
  SPEAKER_04: Then you can add the set the documentation on the DAG.
• 17:36
  SPEAKER_04: Yeah.
• 17:36
  SPEAKER_04: So basically, as we're writing the DAG, we write the documentation.
• 17:39
  SPEAKER_04: And then we have special code that also understands.
• 17:41
  SPEAKER_04: So we have tops and bottom sections that will say these are all the assets that are modified by this DAG.
• 17:47
  SPEAKER_04: And my favorite part, these are all the tasks of the DAG.
• 17:50
  SPEAKER_04: Ideally, there are a few lines that are actually manually written.
• 17:52
  SPEAKER_04: So for example, have the essence of each DAG.
• 17:55
  SPEAKER_04: You write that once, and then that comes in, it gets pasted while you're creating the DAG.
• 17:59
  SPEAKER_04: But all the DAGs here in my favorite part is that it has all the sources and the links to the source.
• 18:04
  SPEAKER_04: For example, if the new task is a Python operator that runs a certain script, so at the bottom of the DAG documentation, it will say the create table task is run by this script, linked to its GitHub file in this directory, or it'll say this, or and this task is run by this SQL, and here's the source of that SQL.
• 18:23
  SPEAKER_04: Because the same way it's able to create the tasks, it just takes all that information and throws it into a doc file.
• 18:30
  SPEAKER_01: Yeah, this is cool.
• 18:32
  SPEAKER_04: It's really cool.
• 18:33
  SPEAKER_04: It's really cool.
• 18:34
  SPEAKER_04: I really like it.
• 18:34
  SPEAKER_01: Yeah.
• 18:35
  SPEAKER_04: And a second thing that the another pro of why I like using the framework is it has a mechanism to override the local configuration.
• 18:42
  SPEAKER_04: So for example, we run local airflow.
• 18:43
  SPEAKER_04: That is, we run an airflow in a dev Dockerized environment on our laptops.
• 18:48
  SPEAKER_04: And that's how we can test our code and play with it, and things like that.
• 18:52
  SPEAKER_04: And so we have a whole local mechanism there.
• 18:54
  SPEAKER_04: It will change the add prefixes to table names or change table names so that we're not overriding the real data.
• 19:00
  SPEAKER_04: And it will add settings like row limits, so only run on 10,000 lines instead of millions of lines, so we can more easily test and build locally.
• 19:08
  SPEAKER_04: So the framework has all those things built in, and those are the pros of using the framework.
• 19:13
  SPEAKER_04: And again, another pro is anytime you add something, like for example, you could trigger GitHub workflow, just something got needed for one project.
• 19:20
  SPEAKER_04: And a couple other people were you needed it too, and there was some duplicate code.
• 19:24
  SPEAKER_04: So I just added it to the framework.
• 19:25
  SPEAKER_04: And now it's a single function call, will trigger and wait for a GitHub workflow, and everything's built in there and it knows what pre variables to ask for if it can't just get it from the config.
• 19:34
  SPEAKER_04: So those are all the pros.
• 19:36
  SPEAKER_04: The con.
• 19:37
  SPEAKER_04: So two cons.
• 19:38
  SPEAKER_04: One is if the framework is in a separate package that's deployed separately, then modifying it is a bit of a headache because you have to wait for the deployment and you need this change now, and you can't wait for the deployment.
• 19:50
  SPEAKER_04: So that that's a headache.
• 19:51
  SPEAKER_04: And another problem is that the framework is in my native language.
• 19:55
  SPEAKER_04: It works for me.
• 19:56
  SPEAKER_04: To me, it's the most obvious thing in the world.
• 19:58
  SPEAKER_04: But I find that and I know the same thing.
• 20:01
  SPEAKER_04: When I try to learn somebody else's code, it was obvious to them.
• 20:04
  SPEAKER_04: It's not obvious to me.
• 20:05
  SPEAKER_04: In the same way, that also means that Google and ChatGPT and all these things are familiar with the normal way to do it.
• 20:10
  SPEAKER_04: They're not familiar with my framework's way of doing it.
• 20:13
  SPEAKER_04: So though those are their costs to going non-standard, to doing things differently than everybody else is doing.
• 20:19
  SPEAKER_04: Right.
• 20:19
  SPEAKER_04: Especially in a world where ChatGPT and Google give you so much information.
• 20:22
  SPEAKER_04: Right.
• 20:23
  SPEAKER_04: So those are the main cons.
• 20:24
  SPEAKER_04: And those are serious cons.
• 20:25
  SPEAKER_04: And if people wonder, but Gil, if you're on vacation or you leave the company or you're hit by a bus or whatever, you may think this is the easiest way to maintain your code.
• 20:34
  SPEAKER_04: But somebody who's not in your mind, it's not the easiest way for them to maintain your code.
• 20:39
  SPEAKER_04: And that that's a real question.
• 20:41
  SPEAKER_01: Yeah, that's very true.
• 20:43
  SPEAKER_01: I actually I hope that you will never be hit by a bus.
• 20:45
  SPEAKER_01: But to your point, actually, whenever you come with a new solution, I think like the hardest part, honestly, is to actually get the adoption of that new tool that you expect, right?
• 20:56
  SPEAKER_01: Because ultimately we all have our own habits, our own way of doing things.
• 21:01
  SPEAKER_01: And whenever you have another solution that is meant to standardize everything, I mean there are always some people that struggle with that.
• 21:08
  SPEAKER_01: But but again, I just love your solution.
• 21:11
  SPEAKER_01: I like just to summarize and let me know if I'm right here.
• 21:14
  SPEAKER_01: But basically, a data pipeline at Wix looks like a bunch of function calls, right?
• 21:22
  SPEAKER_01: And those function calls at the end generate a DAG with some tasks.
• 21:28
  SPEAKER_01: And again, those tasks are actually common patterns that you will find in many DAGs at Wix, and that's why they are behind functions, so you don't have to rewrite those tasks again and again.
• 21:39
  SPEAKER_01: I love that solution.
• 21:40
  SPEAKER_01: I think again it's very different than what we used to hear here about generating your DAG using a template, for example.
• 21:46
  SPEAKER_01: So this is a really cool solution.
• 21:50
  SPEAKER_01: I need to ask is it open source?
• 21:54
  SPEAKER_04: I shared the code.
• 21:56
  SPEAKER_04: I shared the code.
• 21:56
  SPEAKER_04: I'm not standing behind the code, I'm not the I provided the code so.
• 22:00
  SPEAKER_04: people can look at it and see it and and play with it.
• 22:03
  SPEAKER_04: What's the name of the framework again?
• 22:04
  SPEAKER_04: It's a DS tag framework.
• 22:06
  SPEAKER_04: Okay.
• 22:07
  SPEAKER_04: Uh DSTAG.
• 22:08
  SPEAKER_04: So yeah, you can I can send you a link to it or I don't have your show notes or whatever.
• 22:13
  SPEAKER_01: But yeah, and by the way, like for people that really wonder because I think you you actually made a talk about you did a talk about that in the previous Airflow Summit.
• 22:20
  SPEAKER_01: Feel free to check to check a guild talk out.
• 22:22
  SPEAKER_01: You will learn more about this framework specifically and it's worth the time.
• 22:28
  SPEAKER_01: Okay, so moving on to the next topic that I would like to touch on because as you said you are running machine learning models at Wix.com.
• 22:36
  SPEAKER_01: So what are the specific challenges of MN orchestration with Airflow that you had?
• 22:43
  SPEAKER_01: Do you have any like challenges that you had when you tried, when you actually started to orchestrate machine learning models with Airflow?
• 22:49
  SPEAKER_04: The main thing that the the machine learning team has been doing differently than some of the other teams here is in addition to all the scheduled airflow DAGs, we also have a lot of on-demand airflow DAGs.
• 23:00
  SPEAKER_04: We found that we have a lot of tasks that the that our data scientists are running and we provide them a UI and we basically those UIs and whether they know it or care about it or not, on the back end basically it's those UIs are triggering airflow DAGs.
• 23:14
  SPEAKER_04: So we basically have these airflow DAGs that are wrapped inside services and the DAGs up uh communicate back statuses and error messages or whatever to those services.
• 23:23
  SPEAKER_04: So that's I think the main thing that the that the machine learning task team is is doing differently.
• 23:28
  SPEAKER_04: For example, data scientist wants to train a model or run a stress test or create a data set or all sorts of things or even create a build a test set for a Gen AI project.
• 23:37
  SPEAKER_04: We have all these things functions that the data scientists are using and the and really what it is is an airflow DAG triggered in the background that's communicating back with the with the service.
• 23:49
  SPEAKER_04: That's the main difference that the the main challenge that the ML data engineering team took on that we think is been unique to us I think or different than most others are doing.
• 24:00
  SPEAKER_01: Just to to clarify here when you said the ML engineering team is different than the data science team?
• 24:07
  SPEAKER_04: Okay.
• 24:07
  SPEAKER_04: The data science team includes data scientists, data engineers and other engineers.
• 24:11
  SPEAKER_04: So here I'm referring to so the data scientists are using these tools created by both the engineering and the data engineering teams inside the data science.
• 24:20
  SPEAKER_01: So can you tell us more about maybe did you have actually to make some optimizations for your tasks because of those machine learning models that you are orchestrating on Airflow?
• 24:30
  SPEAKER_04: Are there any optimizations that you have on top of your mind that you had to do yeah so one optimization that we did was when one of our most complicated pipelines and pipeline that that drove drew me into the framework our customers wanted our results in the morning and we were delivering them in the late afternoon.
• 24:46
  SPEAKER_04: And the the reason that was happening again because we're running seven DAGs sequentially so each DAG was waiting for the upstream DAG to finish and that took a long time.
• 24:56
  SPEAKER_04: We had a sensor on each DAG and when that finished we moved on what we realized was that the each internal DAG did some post processing.
• 25:05
  SPEAKER_04: That is it had the output ready so the pipeline could really continue and then it continued on some of its own tasks storing the results in different tables and doing all sorts of things that the downstream DAGs didn't really need to wait for.
• 25:17
  SPEAKER_04: So what we're able to do is the the airflow DAG external task sensor if you don't send it a task ID it waits for the DAG but you can also give it a task ID and then it just waits for that task.
• 25:27
  SPEAKER_04: So now every every internal DAG it creates once it finishes with its output it it runs this empty operator of output ready.
• 25:37
  SPEAKER_04: And now our sensor tags tasks wait for that and that that cut down the pipeline by hours hours?
• 25:44
  SPEAKER_04: Wow by hours by hours yeah so it it was great it was great.
• 25:49
  SPEAKER_01: Yeah that that that's a huge improvement.
• 25:52
  SPEAKER_04: Yeah it was a huge improvement and it's the kind of thing it didn't need the framework and the refactorings or whatever except that it did because when you can't maintain your code and you have seven copies of everything then you don't even think in these directions because who wants to make a change like that and debug it and maintain it in seven different places.
• 26:12
  SPEAKER_04: First you need control of your code and then you can do smart things.
• 26:16
  SPEAKER_04: So as long as your code is out of control you can't optimize it and you can't extend it and you can't maintain it.
• 26:22
  SPEAKER_01: Yep that that is a very good advice okay just start by understanding your code and making sure that it is it is not a big mess so you can actually focus on optimizing your tasks.
• 26:33
  SPEAKER_01: How critical are the data pipelines that you run today with Airflow?
• 26:39
  SPEAKER_04: They're critical that it's okay.
• 26:41
  SPEAKER_04: Our data science teams customers are depending on them they're important.
• 26:46
  SPEAKER_01: So I guess do you have any uh testing strategies in place to ensure that your tasks work as expected that you don't have any bugs or something like that.
• 26:55
  SPEAKER_01: Do you have any strategies?
• 26:58
  unknown: Okay.
• 26:58
  SPEAKER_04: We have really two primary things that we're doing one is we have automated testing that we create the DAG and then we load the DAG back in from the DAG bag and automatically check that the DAG and the task and the values that we meant it to create actually got created.
• 27:16
  SPEAKER_04: So that's one level automated testing.
• 27:18
  SPEAKER_04: The second type of testing that we do is as I said we have we have local airflow.
• 27:21
  SPEAKER_04: We have airflow running in Dockerized environments on our own machines and then as I said we have a local good fig mechanism that's able that knows to change table names so we're not messing with real data and knows how to run on a more limited data set than on the full data set.
• 27:37
  SPEAKER_04: And so we're able to just run that.
• 27:38
  SPEAKER_04: So for code to go live it has to pass the automated tests that in theory show that the DAGs and the tasks and values are what they expected to be.
• 27:46
  SPEAKER_04: And also that in practice it it runs properly and successfully.
• 27:50
  SPEAKER_04: We do both those tests before anything goes live.
• 27:53
  SPEAKER_01: And do you have a staging environment or something like that?
• 27:55
  SPEAKER_04: The staging environment is our local airflow it is just is the Dockerized environment local computers.
• 28:01
  SPEAKER_01: I see I guess one last question regarding testing did you find any because I this is actually a very complex topic right listing your DAGs in Airflow is not easy.
• 28:11
  SPEAKER_01: So I guess did you find any challenges regarding testing your DAGs or tasks that you can think of or not really?
• 28:20
  SPEAKER_04: The biggest challenge that we had that we had I think that I the biggest challenge that I felt was that before before I put together the framework at least for where I was working there was no automated mechanism that said change table names so that when you run locally you don't write anything and there was no simple way of running specific things or only in certain safe environments.
• 28:41
  SPEAKER_04: And so that that was the biggest challenge.
• 28:42
  SPEAKER_04: And what we did at the time was you have to comment things out and change names on your local version and hope that you didn't make a mistake and commit any of those to master and all these things.
• 28:51
  SPEAKER_04: And so that that was one of the impetus of the framework is to to make that all safe working at the local environment for testing.
• 28:57
  SPEAKER_01: The more you talk about that framework the more I want to try it we I'm very excited by it Gil, one last question to you do you have any recommendations that you will make to data engineers that are listening to this episode today regarding airflow and how they can maintain a better code base and so on.
• 29:17
  SPEAKER_01: Do you have any recommendations?
• 29:19
  SPEAKER_04: The main recommendation is don't let things get out of control and solve things early and don't copy and paste and or at least if you're copying pasting just try not to and and especially now with with Copilot and AI and whatever don't you remember Joel Spolsky used to say that the problem with code is that it's easier to write than it is to read and that's how code bases get out of control.
• 29:41
  SPEAKER_04: Don't let your code base get out of control and again I think with with AI it's going to be an even bigger problem.
• 29:47
  SPEAKER_01: So just don't let it get out of control what do you think about AI in general related to engineering?
• 29:55
  SPEAKER_01: It's amazing yeah it's amazing it's terrifying it's powerful okay yeah I think it's all of that great thank you so much Gil I think it was extremely interesting to know about this framework that you built also you touched on a very interesting optimization so this is super interesting.
• 30:15
  SPEAKER_01: Do you have any you know like actually what is the best way for folks to get in touch with you if they have any questions the best way is probably LinkedIn or you can just email me at the gilr at Wix.com awesome awesome thank you again thank you so much for joining Gil and I'm looking forward to having another episode with you who knows that would be great.
• 30:37
  SPEAKER_01: Thank you thank you so much.
• 30:38
  SPEAKER_03: Have a good day thank you take care bye thanks for joining us for the latest episode of the Data Flowcast mastering airflow for data engineering and AI.
• 30:50
  SPEAKER_03: You'll find links in the show notes to any resources mentioned in today's show.
• 30:54
  SPEAKER_03: If you're enjoying our podcast please subscribe so you never miss an episode and leave us a five star review