Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias

Transcript

Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: We deal heavily with big data and we perform a lot of teletransformations and so we schedule and automate all these transformations with airflow.
• 0:09
  SPEAKER_01: Um basically we're using it for scheduling data and business analytics.
• 0:19
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:30
  SPEAKER_02: Okay, hi everyone.
• 0:31
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:32
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Christos BCS, who is an open source software engineer at G Research.
• 0:40
  SPEAKER_02: Christos, how are you?
• 0:42
  SPEAKER_01: I'm fine, thank you.
• 0:44
  SPEAKER_01: How about you?
• 0:45
  SPEAKER_02: Yeah, I'm doing well.
• 0:46
  SPEAKER_02: I'm doing well.
• 0:46
  SPEAKER_02: It's great to have you on.
• 0:48
  SPEAKER_02: Uh let's go ahead and dive in.
• 0:50
  SPEAKER_02: So for today's podcast, we're going to discuss how the G Research open source software team contributes to Airflow.
• 0:56
  SPEAKER_02: I'm very excited to get into this.
• 0:58
  SPEAKER_02: Uh always love hearing from uh contributors to the project.
• 1:02
  SPEAKER_02: Uh but before we get into that, let's start with some background.
• 1:05
  SPEAKER_02: Can you tell us about what G Research does?
• 1:08
  SPEAKER_01: Uh yes.
• 1:09
  SPEAKER_01: G Research is a leading quantitative research and technology firm.
• 1:14
  SPEAKER_01: We use a lot of machine learning, a lot of big data and emerging tech uh to predict movements in financial markets, try to answer the biggest questions in finance.
• 1:26
  SPEAKER_02: Okay, very interesting.
• 1:28
  SPEAKER_02: And at a high level, how does your company use Airflow?
• 1:32
  SPEAKER_01: We deal heavily with big data and we perform a lot of teletransformations.
• 1:37
  SPEAKER_01: And so we schedule and automate all these transformations with Airflow.
• 1:41
  SPEAKER_01: Um, basically, we're using it for scheduling data and business analytics.
• 1:46
  SPEAKER_02: Yeah, that makes sense.
• 1:48
  SPEAKER_02: And so what does your team specifically do?
• 1:51
  SPEAKER_01: As you mentioned, I'm part of the open source team.
• 1:54
  SPEAKER_01: Uh we have our own open source projects that we are actively uh developing and maintaining, and we also contribute to other open source projects like Airflow on behalf of G Research.
• 2:06
  SPEAKER_01: And we provide technical support to the other teams that we have, which are using the open source software that we are developing and maintaining.
• 2:16
  SPEAKER_02: Okay, got it.
• 2:17
  SPEAKER_02: So you're providing internal support to all of the teams that are using Airflow and also contributing to the project.
• 2:24
  SPEAKER_01: Yes, exactly.
• 2:25
  SPEAKER_02: Very cool.
• 2:26
  SPEAKER_02: And tell me a little bit more about why the company is investing in having you contribute to open source.
• 2:33
  SPEAKER_02: That's not something we always hear about from Airflow users.
• 2:36
  SPEAKER_02: And I would love to hear more about the why they think that that's a valuable use of time.
• 2:40
  SPEAKER_01: I mean, if you get down to it, it's just a business decision.
• 2:44
  SPEAKER_01: It's much cheaper to contribute to an open source software and maintain it than it is to uh buy it or license it.
• 2:52
  SPEAKER_01: And also it's nice to have your own in-house technical support uh that knows your exact use cases.
• 2:59
  SPEAKER_01: Because sometimes if you are depending on another company, a third party, and you need support from them, you cannot always share your exact uh business model and use cases and what you are doing.
• 3:12
  SPEAKER_02: Yeah, yeah, absolutely.
• 3:13
  SPEAKER_02: Uh I would love to use that for for other companies to encourage them to contribute to the open source project as well.
• 3:20
  SPEAKER_02: Of course, as you're saying, if you contribute, you get to influence the direction of the project in the way that works best for your company.
• 3:28
  SPEAKER_02: So yeah.
• 3:29
  SPEAKER_01: And it's it's always better to contribute your changes than to maintain a fork.
• 3:35
  SPEAKER_02: Yeah, of course.
• 3:36
  SPEAKER_02: Definitely maintaining a fork is a challenging route.
• 3:39
  SPEAKER_02: I we've seen some companies do that, but it they tend to run into issues, I think, at some point down the line.
• 3:46
  SPEAKER_02: It's definitely a lot of work.
• 3:48
  SPEAKER_02: Yeah, okay, very cool.
• 3:50
  SPEAKER_02: And so talking about a specific contribution, I you've spoken recently about open telemetry and airflow and using those two together.
• 4:01
  SPEAKER_02: And before we get into your contributions on that, I just want to hear more about this integration and why it's important to your team.
• 4:08
  SPEAKER_01: Okay.
• 4:09
  SPEAKER_01: I think that I should start with explaining what open telemetry is.
• 4:14
  SPEAKER_01: I don't think that a lot of people are familiar with open telemetry.
• 4:18
  SPEAKER_01: It's actually a collection of tools that we use to collect metrics, logs, and data.
• 4:24
  SPEAKER_01: And then we use all this data to monitor our system and get a better understanding in a deeper level.
• 4:31
  SPEAKER_01: So each data, each of three these three types, provides a different uh a different view.
• 4:37
  SPEAKER_01: Now, my presentation at the Airflow Summit focused on traces.
• 4:41
  SPEAKER_01: So, what's a trace?
• 4:43
  SPEAKER_01: Let's assume that we have an operation, like for example, we have a web app and we have a button, and that button makes uh makes a grand operation, makes a call to a back-end API, and the API makes a call to a database, and then you returns something to the API, and the API sends a response back to the front end.
• 5:04
  SPEAKER_01: That whole thing, all these steps are a trace.
• 5:08
  SPEAKER_01: And each individual step is a span.
• 5:11
  SPEAKER_01: And to put it in airflow context, a trace could be a dagger, and the entire daggram is a trace, and then each individual step, which is a task, it's a span.
• 5:22
  SPEAKER_01: Now, the thing is that so with trace and spans hold some data, like for example, the start time, the end time, and then they have the attributes which are key value pairs, and we can use them to expose internal values that we have during runtime.
• 5:39
  SPEAKER_01: Now, we know the start time and the end time of a task from the web server UI.
• 5:44
  SPEAKER_01: We don't need a span to tell us that.
• 5:47
  SPEAKER_01: But the thing is that we can also create our own spans under a task and monitor individual steps.
• 5:54
  SPEAKER_01: Like, for example, if a task uh consists of five or ten different steps and it's very slow, or for some reason it hangs, we could create our own spans for each individual step, see how long it takes, and that that's gonna give us a lot of options.
• 6:12
  SPEAKER_01: Like, for example, if if we find that uh step three of the task is where it takes most of the time, like uh 80% of the uh of the task execution, we can uh find a way to optimize it, uh increase performance, or maybe refactor it or find another way.
• 6:30
  SPEAKER_01: Anyway, you you get so many options when you have all the insights.
• 6:35
  SPEAKER_02: Okay, interesting.
• 6:36
  SPEAKER_02: So it's helping you profile those tasks and what's going on in them.
• 6:41
  SPEAKER_01: Yeah, yeah, it's it helps you gain better visibility.
• 6:45
  SPEAKER_01: Like, for example, one task, and this is very common, someone could be using a task to make a call to an external service.
• 6:53
  SPEAKER_01: And you can also monitor that, you can create another span for that, and you can know how long that call took and how long the operation, the external service took, and connect all of them.
• 7:05
  SPEAKER_01: And using visualization backends like Yakir, you have a triview where you know how long each node took to run.
• 7:15
  SPEAKER_02: Yeah, that sounds very helpful.
• 7:18
  SPEAKER_02: And then, so this is something that you have implemented at your company and are using to help other teams, the teams that use Airflow run more efficiently.
• 7:29
  SPEAKER_02: Is that right?
• 7:30
  SPEAKER_01: Yeah, but I've also contributed that all to the upstream ripple.
• 7:35
  SPEAKER_01: So that yeah, and and there are docs, and people can uh read the docs and enable open telemetry, they can set up an open telemetry collector and then enable that and uh start using it with Airflow and see how so there are there are some default spans that get exported once you enable open telemetry in your airflow configuration.
• 7:59
  SPEAKER_01: There are some default spans like there is the root span which represents the entire diagram, and then there are other spans from one span for each task.
• 8:09
  SPEAKER_01: But you can you can extend that and you can add your own code and create your own spans from from under the tasks.
• 8:18
  SPEAKER_01: I mean that that was part of my uh presentation at the Airflow Summit.
• 8:23
  SPEAKER_01: I had a few examples of how people can add their own spans under the tasks.
• 8:29
  SPEAKER_01: By the way, that all works in uh Airflow 3.0 and upwards.
• 8:33
  SPEAKER_01: If someone is running, yeah, if someone is running an older version like 2.10, it's not gonna work very well because the the initial implementation uh it was it wasn't working as expected.
• 8:46
  SPEAKER_02: Okay, I see.
• 8:47
  SPEAKER_02: Um that's great that it's already working on the newest major version.
• 8:52
  SPEAKER_02: I'm curious if you're during your talk at the Airflow Summit or maybe um remotely elsewhere, have you heard about other teams implementing the same thing now that you've contributed it back?
• 9:03
  SPEAKER_01: Yeah, yeah.
• 9:04
  SPEAKER_01: Uh after the presentation, a lot of people came to me.
• 9:07
  SPEAKER_01: Uh we I had a few discussions uh with people using it.
• 9:11
  SPEAKER_01: Um, or they were they were keen to use it.
• 9:14
  SPEAKER_01: I mean they they didn't know that it worked with Airflow 3.
• 9:17
  SPEAKER_01: Uh so I yeah, I I did a demo at the end of the presentation and showed it.
• 9:22
  SPEAKER_01: And yeah, not many people were familiar with it.
• 9:25
  SPEAKER_01: But a lot of people are familiar with OpenTelemetry, they just weren't so they didn't know it worked with Airflow.
• 9:34
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 9:43
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 9:55
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 10:03
  SPEAKER_02: And now that you've done this implementation for your team and you've contributed it to the open source project, are there any is there further work that you're looking to do on it?
• 10:14
  SPEAKER_02: Is it now it works and it's done, or are you looking to optimize it in any way?
• 10:20
  SPEAKER_01: Okay, I think that traces are in a uh in a very good state.
• 10:25
  SPEAKER_01: They're working, but I've been uh looking at other at other areas lately.
• 10:30
  SPEAKER_01: Like for example, metrics.
• 10:32
  SPEAKER_01: Now the the open telemetry metrics implementation worked very good.
• 10:36
  SPEAKER_01: Um but I found a few discrepancies between the docs and the actual metrics that get exported.
• 10:45
  SPEAKER_01: And so I've been doing some work around that.
• 10:48
  SPEAKER_01: I'm actually uh having an open PI right now, an open pull request, uh where I'm trying to add a YAML file that will keep track of all the metrics throughout the project.
• 10:59
  SPEAKER_01: And then the the idea is to enforce some checks uh in the CI so that when people add new metrics, they always have to exist in the YAML so that the YAML file uh will be the single source of truth.
• 11:16
  SPEAKER_01: And also the YAML will be used to build the docs, and that way they can make sure that there are no discrepancies.
• 11:23
  SPEAKER_01: And so when you read something in the docs, it's there.
• 11:26
  SPEAKER_01: If you enable OpenTelemetry, Prometheus or might be doing uh might be using Stats D or Datadog, uh whatever backend you're using.
• 11:36
  SPEAKER_01: Once you enable the configuration, you check Prometheus, Grafana, the metrics are are always there.
• 11:41
  SPEAKER_01: What you've read in the config in the docs is there.
• 11:45
  SPEAKER_02: Yeah, okay, got it.
• 11:47
  SPEAKER_02: Um love hearing about docs contributions, those are really important.
• 11:51
  SPEAKER_02: Outside of your work on open telemetry and airflow, are there any other airflow contributions that you and your team have worked on recently that you think are particularly exciting?
• 12:02
  SPEAKER_01: Another work that I have been doing, I have been looking at the scheduler performance.
• 12:08
  SPEAKER_01: I mean, we had a very powerful computer, we were running some DAGs and tasks, and we could see that it wasn't utilizing 100% of the CPUs, like for the scheduler and the workers.
• 12:20
  SPEAKER_01: And so we started doing some digging and we found out that the scheduler has this idea where it picks a lot of tasks to examine and queue, but in in fact, after after it picks all the tasks and retrieves them from the database, it applies uh a bunch of limits.
• 12:39
  SPEAKER_01: And uh that ends up in in case uh the schedule has already reached the limits and it cannot really queue any more tasks, it does it just does unnecessary computations.
• 12:53
  SPEAKER_01: And so the that was decreasing throughput.
• 12:57
  SPEAKER_01: The scheduler was doing all this work without actually scheduling anything.
• 13:01
  SPEAKER_01: And we found out that if we just include the limits in the initial query that retrieves the task from the database, that makes the scheduler always uh pick up the number of tasks that it can actually queue.
• 13:15
  SPEAKER_01: And also that's true for a DAG.
• 13:18
  SPEAKER_01: Like, for example, you could have a limit that can schedule only four tasks for a particular DAG, it would have reached that limit and then it would still go on and pick 30 or 40 tasks from that DAG, although it was at the maximum level, it couldn't schedule anymore.
• 13:34
  SPEAKER_01: So uh a whole a whole uh scheduler iteration would just go wasted.
• 13:40
  SPEAKER_01: Uh it wouldn't schedule anything because it concentrated in a DAG where it had reached max concurrency.
• 13:47
  SPEAKER_01: Um, and so by including the uh the limits at the start of it, that increased throughput a lot.
• 13:55
  SPEAKER_01: Like for example, if you have the the heavier the load, the greater the performance increase.
• 14:01
  SPEAKER_01: And that's something um I I've run a lot of tests.
• 14:05
  SPEAKER_01: There are some people from uh from astronomer that have also run a lot of tests, and they've seen that they have come to the same conclusion as me that that that's a huge improvement that makes a lot of difference.
• 14:20
  SPEAKER_01: And so we we are still running some more tests, but uh uh trying to cover all the edge cases and see that there's no regression.
• 14:28
  SPEAKER_01: But if all goes well, that that should be uh into the main repo soon.
• 14:33
  SPEAKER_02: Okay, that's great.
• 14:34
  SPEAKER_02: Would mean probably inclusion in the next minor release.
• 14:37
  SPEAKER_02: Yeah, that's that's a really exciting improvement that will impact everybody using Airflow, right?
• 14:43
  SPEAKER_01: Yeah.
• 14:44
  SPEAKER_02: That's a great example.
• 14:46
  SPEAKER_02: Uh any other projects that you wanted to cover that you're working on or contributions, I should say.
• 14:53
  SPEAKER_01: No, these are the major ones.
• 14:54
  SPEAKER_01: And I'm also doing some uh yeah, uh everything else is bug fixes.
• 14:59
  SPEAKER_02: Those are also very important, so maybe not as interesting to go through one by one.
• 15:04
  SPEAKER_02: So um, yeah.
• 15:06
  SPEAKER_02: Okay, very cool.
• 15:07
  SPEAKER_02: Well, thank you for giving the details on both of those uh projects.
• 15:12
  SPEAKER_02: Those are both really valuable contributions uh for airflow.
• 15:16
  SPEAKER_02: I always like to end with a standard final question for folks, and that is uh, what would you most like to see from the Airflow project in the near future?
• 15:24
  SPEAKER_02: Maybe that you're not already contributing on, but from others or something.
• 15:29
  SPEAKER_01: Um me and my team, we are very content with what airflow is today, but uh there are some very interesting changes in the task SDK.
• 15:38
  SPEAKER_01: I mean the whole separation with uh the task with the task execution path, that effort to make DAX language agnostic.
• 15:46
  SPEAKER_01: Like, I mean, in the future you could write DAX in whatever language you want.
• 15:51
  SPEAKER_01: It doesn't have to be Python.
• 15:52
  SPEAKER_01: At some point, it could even be Java.
• 15:55
  SPEAKER_01: And uh yeah, who knows?
• 15:57
  SPEAKER_01: Uh, with the whole separation of the task STK, it could be at some point people could be importing it into other projects and just using it uh as a dependency.
• 16:08
  SPEAKER_01: I mean, that's that's very interesting.
• 16:11
  SPEAKER_02: Yeah.
• 16:11
  SPEAKER_02: Are there uh other languages that you would write DAGs in if that was available today?
• 16:17
  SPEAKER_01: I mean, yeah, sure.
• 16:18
  SPEAKER_01: Java Java would be one.
• 16:20
  SPEAKER_01: And I mean, you know, not everyone is familiar with other languages.
• 16:24
  SPEAKER_01: Uh sometimes you end up in a project and you're writing a language that you hardly in before.
• 16:30
  SPEAKER_01: Like, for example, I wasn't that much familiar with Python until I started working in Airflow.
• 16:36
  SPEAKER_01: And yeah, if you can do the same thing in an language that he you are familiar with and you know best, yeah, why not?
• 16:44
  SPEAKER_02: Yeah, absolutely.
• 16:45
  SPEAKER_02: Definitely uh a feature that there's a lot of excitement around.
• 16:49
  SPEAKER_02: Okay, well, that's great.
• 16:50
  SPEAKER_02: Um, Christos, it's been really nice speaking with you.
• 16:53
  SPEAKER_02: Thanks so much for going through all of that.
• 16:55
  SPEAKER_02: Uh, to close us out, what is the best way for folks to get in touch with you?
• 16:58
  SPEAKER_01: I think the best way is my email, which is available on my GitHub.
• 17:04
  SPEAKER_01: And from my GitHub profile, you can also get my LinkedIn profile.
• 17:07
  SPEAKER_01: That's another way to reach me.
• 17:08
  SPEAKER_01: Uh yeah.
• 17:10
  SPEAKER_02: Okay, great.
• 17:11
  SPEAKER_02: We'll put all of that in the show notes uh so folks can get it, get it if they need it.
• 17:16
  SPEAKER_02: Um, okay, well, thank you again for coming on.
• 17:18
  SPEAKER_02: It's been a pleasure speaking with you.
• 17:21
  SPEAKER_01: Thank you, Rimas.
• 17:22
  SPEAKER_01: That's it for me too.
• 17:25
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 17:31
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 17:35
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.