Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Enhancing Business Metrics With Airflow at Artlist with Hannan Kravitz

Transcript

Enhancing Business Metrics With Airflow at Artlist with Hannan Kravitz · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We wanted to do that and also to be able to enrich it.
• 0:03
  SPEAKER_02: I mean, Airflow at the end of the day is an open source and it's quite easy to do some modifications according to your use case with Airflow and implement it on your end.
• 0:14
  SPEAKER_02: And by that, you can focus on enabling the business to do a lot of other things that some of the other tools are not designed to do.
• 0:25
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:34
  SPEAKER_00: Hello, everyone.
• 0:36
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:37
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Hanan Kravitz, who is a data engineering team leader at Artlist.
• 0:44
  SPEAKER_00: Hanan, how are you doing today?
• 0:46
  SPEAKER_02: I'm fine, thank you.
• 0:47
  SPEAKER_02: Excited to be here.
• 0:49
  SPEAKER_00: Thanks for accommodating our US-based time zones for this.
• 0:52
  SPEAKER_02: Sure.
• 0:53
  SPEAKER_00: Yeah.
• 0:53
  SPEAKER_00: All right, let's go ahead and dive in.
• 0:55
  SPEAKER_00: So for today's podcast for our listeners, we're going to discuss how Artlist uses Airflow to power their internal data orchestration.
• 1:02
  SPEAKER_00: And to start, Hunan, can you describe what Artlist does as a company for those who might not be familiar?
• 1:08
  SPEAKER_02: Sure.
• 1:08
  SPEAKER_02: So Artlist basically is a content creative company whose core mission is to empower people to create without limits.
• 1:18
  SPEAKER_02: Artlist believes that you have a story to tell, a message to share, or a vision to bring to life.
• 1:25
  SPEAKER_02: And basically all you need is an inspiration and the right tools to do that.
• 1:29
  SPEAKER_02: Artlist is a royalty-free asset platform which provides the best assets for video creation, such as highest quality music, sound effects, shootage, video templates, and plugins.
• 1:44
  SPEAKER_00: Okay.
• 1:45
  SPEAKER_00: Interesting.
• 1:46
  SPEAKER_00: Yeah, that's very cool.
• 1:47
  SPEAKER_00: I'm not as familiar with that space, but I can imagine that's very useful.
• 1:51
  SPEAKER_00: You work on the data engineering side.
• 1:53
  SPEAKER_00: How does your team support that business model?
• 1:56
  SPEAKER_02: Okay.
• 1:56
  SPEAKER_02: As I was saying, the main platform of Artlist is to enable video creators and content creators to enjoy our free our assets, our digital assets, and to basically use them with their creativity.
• 2:10
  SPEAKER_02: As you can imagine, everything that a user is doing on our platform, we want to measure and assess and understand what the journey that he's experiencing while using our platform.
• 2:21
  SPEAKER_02: A lot of data is being gathered.
• 2:24
  SPEAKER_02: And with all this data, we provide every business stakeholder in the company the right insight or the right focus on each domain that the company is going through on each domain that the product is used.
• 2:38
  SPEAKER_00: Okay, great.
• 2:39
  SPEAKER_00: So really looking at insight into how your product is used to help the company make the product better.
• 2:45
  SPEAKER_02: Exactly.
• 2:46
  SPEAKER_00: Okay, that makes sense.
• 2:47
  SPEAKER_00: We have a team internally that does something similar for our products.
• 2:51
  SPEAKER_00: I'd imagine that that generates a lot of data.
• 2:54
  SPEAKER_00: And I understand that you obviously use Airflow to help with this.
• 2:58
  SPEAKER_00: So maybe tell me a little bit about where Airflow fits in.
• 3:06
  SPEAKER_02: Sure.
• 3:06
  SPEAKER_02: My data engineering team is now six data engineers, including me.
• 3:11
  SPEAKER_02: And we collect data from multiple sources, such as internal sources of the company, external, such as the media platforms, such as LinkedIn, Meta, et cetera.
• 3:24
  SPEAKER_02: And also we collect data from other third parties that are working with our list.
• 3:28
  SPEAKER_02: We are using some third-party vendors to collect data from them as well.
• 3:33
  SPEAKER_02: So we need to combine all of this together and basically stream it into our platform, our data warehouse, in order for us to be able to start modeling the data according to the business domains.
• 3:47
  SPEAKER_02: Airflow comes in handy when we can connect to multiple sources such as payment providers, our internal RD team, third parties.
• 3:58
  SPEAKER_02: With Airflow, we can connect to all of these platforms and collect our data and store it into one place.
• 4:04
  SPEAKER_00: Awesome.
• 4:04
  SPEAKER_00: So I think a very classic use case for Airflow, right?
• 4:08
  SPEAKER_00: Connecting to lots of different systems, bringing data into a centralized location.
• 4:12
  SPEAKER_00: And then you're using that data to, you said power dashboards kind of for internal end users.
• 4:17
  SPEAKER_00: Is that right?
• 4:18
  SPEAKER_02: Exactly.
• 4:19
  SPEAKER_02: We collect all the data, we model it, we store it in the data marts for the analytics team to create dashboards out of it.
• 4:26
  SPEAKER_02: For every stakeholder, it can be on the push notifications or send on a schedule-based notifications.
• 4:33
  SPEAKER_02: So this is basically how it's done while we construct the pipeline and enable the business to understand the business KPIs and the metrics that are important for every domain in the company for the product.
• 4:46
  SPEAKER_00: The OG Airflow use case, exactly what it was designed for.
• 4:50
  SPEAKER_00: That's really cool.
• 4:51
  SPEAKER_00: I think also mentioned to me that you are then using that data to enhance those source systems again.
• 4:57
  SPEAKER_00: So like a reverse ETL pattern.
• 4:59
  SPEAKER_00: Is that right?
• 5:00
  SPEAKER_02: Yeah.
• 5:01
  SPEAKER_02: We also use Airflow.
• 5:03
  SPEAKER_02: Once we collect all the data into our data warehouse and we model it, we can also take this enriched data and also enrich the other sources that the other third parties that we use, such as our support team or our marketing team that uses some of their platform.
• 5:20
  SPEAKER_02: We enrich it with the data that we collected and we are enabling them the ability to understand more details on the user, more of its behavior, some kind of metrics that help the support team to understand whether there's a problem with some of the features in the product that the user can open a ticket for, or for the marketing to create segmentations for users and vice versa, and et cetera.
• 5:45
  SPEAKER_00: So you're empowering a lot of your business and all of the analytics with Airflow then.
• 5:50
  SPEAKER_02: Exactly.
• 5:51
  SPEAKER_00: That's great.
• 5:51
  SPEAKER_00: And I understand you've been using Airflow for quite some time within your company.
• 5:56
  SPEAKER_00: I think you mentioned me, over three years.
• 5:59
  SPEAKER_00: So I think we often see a lot of swirl around tools in the data ecosystem with people implementing something new and then using it for a while, something else pops up, and then they move to that.
• 6:09
  SPEAKER_00: Kind of what made you initially choose Airflow for your use cases and then stick with it for all this time?
• 6:16
  SPEAKER_02: To be honest, uh, when we first started here, we tried several tools that will do just that, move data or orchestrate a pipeline to move data from A to B.
• 6:26
  SPEAKER_02: And once we started growing and we got multiple sources to access to and multiple pipelines that we needed to work with, some of the tools weren't good enough to do that.
• 6:38
  SPEAKER_02: And with the simplicity of Airflow to connect with all these resources, there's a lot of open, there's a lot of resources in the community for Airflow, so you can utilize them in such in a lot of the use cases that will combine together into a single platform to do your orchestration pipeline.
• 6:57
  SPEAKER_02: And I think that was the main focus that we we thought about to centralize our pipeline in one place so we can use it and monitor it and do whatever the business is needing is requesting from us to do.
• 7:12
  SPEAKER_00: So before when you were looking at those other tools, was the issue that they were great for sort of part of your use case, but not the whole thing.
• 7:21
  SPEAKER_00: And so you were looking for one tool that could centralize all of it.
• 7:25
  SPEAKER_02: Exactly.
• 7:26
  SPEAKER_02: We wanted to do that and also to be able to enrich it.
• 7:30
  SPEAKER_02: I mean, Airflow at the end of the day is an open source and it's quite easy to do some modifications according to your use case with Airflow and implement it on your end.
• 7:40
  SPEAKER_02: And by that, you can focus on enabling the business to do a lot of other things that some of the other tools are not designed to do.
• 7:50
  SPEAKER_00: Yeah, that makes sense.
• 7:51
  SPEAKER_00: Obviously, having a centralized orchestrator, I think, is one of the biggest selling points of Airflow.
• 7:56
  SPEAKER_00: It's great to hear that's been a success for you.
• 7:58
  SPEAKER_00: You mentioned making modifications to it and the ability to do that because it's an open source tool and it's really flexible.
• 8:05
  SPEAKER_00: Are there any of those that you would want to touch on here?
• 8:07
  SPEAKER_02: Yeah, so I'll want to touch on two uh major use cases that we we do.
• 8:13
  SPEAKER_02: It's related to monitoring.
• 8:16
  SPEAKER_02: There are two kinds of monitoring that I can relate to.
• 8:19
  SPEAKER_02: One of them is an infrastructure monitoring to monitor your pipeline's SLA delays in the pipeline, and also for the business monitoring to basically monitor your business KPIs that are good enough to be shared with the rest of the company.
• 8:37
  SPEAKER_02: So I can take these two use cases and share that for the monitoring of the pipeline, we used Airflow itself to do that.
• 8:47
  SPEAKER_02: Airflow itself comes with an API that you can query all the time.
• 8:52
  SPEAKER_02: And by using that, we were able to monitor our pipelines and to alert if something is not working properly or exceeds SLA that we defined.
• 9:02
  SPEAKER_02: By that we can be proactive and before the business will come up with the issues that why is the data isn't updated, or what happened, this metric is not being updated for the last three hours.
• 9:15
  SPEAKER_02: By doing this monitoring on the pipeline itself with Airflow, we are able to be proactive.
• 9:20
  SPEAKER_00: The other thing that we did is sorry, before you go into the other use case, I want to dig in on that a little more.
• 9:26
  SPEAKER_00: So you've implemented like a DAG within your Airflow instance that monitors the other DAGs.
• 9:32
  SPEAKER_00: Is that right?
• 9:33
  SPEAKER_02: Exactly.
• 9:34
  SPEAKER_00: Okay.
• 9:34
  SPEAKER_02: We created a DAG that practically runs every 10 or 15 minutes or so.
• 9:41
  SPEAKER_02: I don't know exactly the exact schedule interval.
• 9:44
  SPEAKER_02: And it basically monitors what is currently running in Airflow in all the instances of Airflow that we have.
• 9:49
  SPEAKER_02: By that, we defined a configuration of alerting.
• 9:52
  SPEAKER_02: And suppose something is exceeding the threshold that we define, we get an alert.
• 9:57
  SPEAKER_02: And the alert is basically sent into Slack because everyone in the company uses Slack, and also us, the engineering team, we have the Slack open.
• 10:06
  SPEAKER_02: So every task that is exceeding from a threshold that we defined, we get an alert on that.
• 10:11
  SPEAKER_02: And this just gives us an attention.
• 10:14
  SPEAKER_02: This DAG is running too long.
• 10:16
  SPEAKER_02: And it can be from any other uh reasons.
• 10:18
  SPEAKER_02: It can be overload in the business, overload on Airflow, or overloaded in some of the processes that are that Airflow is running.
• 10:26
  SPEAKER_02: It can give you a great picture of what is going on with your pipeline all the time.
• 10:34
  SPEAKER_00: Got it.
• 10:34
  SPEAKER_00: Interesting.
• 10:35
  SPEAKER_00: So most of your monitoring is around kind of these SLAs of when you expect things to have finished and when they're if they don't, if they aren't finished.
• 10:44
  SPEAKER_00: Exactly.
• 10:45
  SPEAKER_00: Then you're alerting the team so that they're aware.
• 10:47
  SPEAKER_00: Yeah, that's an interesting customization.
• 10:49
  SPEAKER_00: And I would imagine comes out of a little bit like some limitations in Airflow's SLA built-in SLA mechanism, I should say.
• 10:57
  SPEAKER_00: I know there is a little bit of one, but it doesn't always work the way people think.
• 11:01
  SPEAKER_00: There's been some proposals to improve it, which would be great.
• 11:04
  SPEAKER_00: That's great about the flexibility that it allows you to implement your own solution.
• 11:09
  SPEAKER_02: It helped us a lot when to be proactive.
• 11:13
  SPEAKER_02: My team is developing a lot of pipelines, and the business requirements are not stopping to are always coming.
• 11:21
  SPEAKER_02: And we need to be able to basically the main goal of our team is to provide the data models in order for the business stakeholders to understand what is going on with the product.
• 11:31
  SPEAKER_02: So we need to build the most stable pipeline and robust pipeline that will answer this business requirement.
• 11:39
  SPEAKER_02: So once we're using a tool that basically runs our pipelines, we need to be able to monitor it.
• 11:44
  SPEAKER_02: Every DAG that is running, we need to be able to see that it is starting and it is completing successfully.
• 11:50
  SPEAKER_02: If they don't finish successfully, we'll be also getting alerts on that.
• 11:54
  SPEAKER_02: But we want to see that during the day and during the time that our DAGs are running, everything is monitored and we are on it.
• 12:02
  SPEAKER_02: My team is on it because we're the we're responsible for that.
• 12:06
  SPEAKER_02: And this is our responsibility for the stability of the pipeline.
• 12:12
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:21
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:33
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:40
  SPEAKER_00: Yeah, that makes sense for a really important business requirement.
• 12:44
  SPEAKER_00: But you mentioned that you're monitoring every single pipeline that you have running.
• 12:48
  SPEAKER_00: I'm curious in the way that you've set up your monitoring DAG, how much of a lift is it if you say add a new pipeline to then add that to be monitored?
• 12:59
  SPEAKER_00: Is it extensible in that way to adjust based on if you have to add a new business requirement or something like that?
• 13:05
  SPEAKER_02: Actually, it's a DAG that basically collects every single DAG run that is currently running.
• 13:12
  SPEAKER_02: So I mean, if I'm going to add new DAGs to Airflow, it is automatically being monitored.
• 13:17
  SPEAKER_02: But then we can adjust the configuration.
• 13:19
  SPEAKER_02: For some DAGs, we can say that the SLA of 20 minutes is enough.
• 13:23
  SPEAKER_02: And for others, it could be 40 minutes.
• 13:26
  SPEAKER_02: That's just something that we configure in order not to also make a lot of noises of these attentions.
• 13:32
  SPEAKER_02: But once a DAG is added to Airflow, a new DAG is deployed to Airflow, it is monitored with this DAG.
• 13:39
  SPEAKER_00: Got it.
• 13:39
  SPEAKER_00: That's really cool.
• 13:40
  SPEAKER_00: I'm sure that helps the team out a lot.
• 13:42
  SPEAKER_00: Okay, let's dive into you said the R2 monitoring use cases.
• 13:46
  SPEAKER_00: So let's talk about the other one.
• 13:47
  SPEAKER_02: Yeah.
• 13:48
  SPEAKER_02: So the other monitoring is actually the monitor that the entire company uses.
• 13:53
  SPEAKER_02: This is business metric KPI monitoring.
• 13:58
  SPEAKER_02: Our pipelines construct data models.
• 14:00
  SPEAKER_02: And these data models basically are being used with the dashboards that are displayed in the entire company.
• 14:08
  SPEAKER_02: And we took it with the business stakeholders to set up a threshold if something went wrong in terms of business metric that is decreasing or increasing.
• 14:21
  SPEAKER_02: Can be for both ways.
• 14:32
  SPEAKER_02: So we can immediately understand that it is related to this launch.
• 14:38
  SPEAKER_02: And this business metric API takes this data and sends it the alerts to the right team.
• 14:43
  SPEAKER_02: That's it, the product team.
• 14:45
  SPEAKER_02: It says that during the last two hours, there is a major decrease from this number to this number, and it is happening due to this launch.
• 14:55
  SPEAKER_02: Let's take uh go ahead and check it.
• 14:57
  SPEAKER_02: Maybe the user experience is not good right now.
• 15:01
  SPEAKER_02: Maybe the A-B testing that you did and it was okay.
• 15:05
  SPEAKER_02: And now once it's live, it's causing us a decrease.
• 15:08
  SPEAKER_02: So this business monitoring enables the business to understand what is going on based on data and not just by looking at it and try to guess if the feature is good or bad.
• 15:21
  SPEAKER_02: And a lot of other business metrics that we measure.
• 15:24
  SPEAKER_02: Let's say if for any reason, let's say we have a technical reason to charge money for from a user, he wants to buy a subscription.
• 15:31
  SPEAKER_02: But for any case, there is a problem with PayPal, let's say.
• 15:35
  SPEAKER_02: So the user won't be purchasing because there is an error or purchasing.
• 15:39
  SPEAKER_02: So we'll see there is a decrease in the purchase for the last three hours.
• 15:43
  SPEAKER_02: By that we can alert the bidding team to check if something went wrong.
• 15:48
  SPEAKER_02: And by that we're enabling action items to the business stakeholders.
• 15:52
  SPEAKER_02: And that we also implemented with Airflow, which is always running.
• 15:55
  SPEAKER_02: There is a DAG that always runs and collects the data from the business models.
• 16:00
  SPEAKER_00: Okay, got it.
• 16:01
  SPEAKER_00: Yeah, that's really interesting.
• 16:03
  SPEAKER_00: And so when you say always running because it's actually sending proactive alerts based on any changes in these metrics, how often does it actually run?
• 16:13
  SPEAKER_00: I'm curious when you say always.
• 16:14
  SPEAKER_02: Always.
• 16:15
  SPEAKER_02: It's uh kind of uh it's not every minute, but the business set up to a threshold of let's say every three hours in the during the day.
• 16:24
  SPEAKER_02: Every three hours this DAG is running and testing the business models that are generated during the other pipelines.
• 16:32
  SPEAKER_02: And suppose there is also a decrease or increase in some of the metrics that the business stakeholders defined, we trigger the alert and we send them directly to the right stack channel.
• 16:43
  SPEAKER_02: We have a lot of slack channels, we send it to the right team that will know that something went wrong.
• 16:49
  SPEAKER_02: Obviously, if there is a crisis, we send it to the entire company, but we try to focus it according to domains in the company.
• 16:55
  SPEAKER_02: And by that, it's a work in progress.
• 16:57
  SPEAKER_02: You collect all the requirements, you set up a configuration, then you put it into the DAG and go ahead and do that.
• 17:04
  SPEAKER_02: We can also change it.
• 17:05
  SPEAKER_02: The business stakeholders can change it.
• 17:07
  SPEAKER_02: We enable them to change these thresholds by creating an app, an application for them, which they can set up the thresholds and it immediately updates the configuration in the DAG.
• 17:18
  SPEAKER_02: We enable that so it will be flexible for them to do that.
• 17:21
  SPEAKER_00: Interesting.
• 17:22
  SPEAKER_00: That's cool.
• 17:22
  SPEAKER_00: Did you do that with a plug-in on the Airflow side?
• 17:25
  SPEAKER_02: No, actually, we did it with uh Streamlit application that we wrote with a Streamlit application.
• 17:31
  SPEAKER_02: It's a Python application that sends data into uh Snowflake and the configuration is read from Snowflake.
• 17:37
  SPEAKER_02: It's kind of a configuration read from Snowflake.
• 17:41
  SPEAKER_00: Okay, I see.
• 17:42
  SPEAKER_02: And then we we implemented into a DAG, into a YAML file that the DAG is actually consuming and according to that triggers the alerts if needed.
• 17:51
  SPEAKER_00: Got it.
• 17:52
  SPEAKER_00: Interesting.
• 17:52
  SPEAKER_00: So when the end user puts in that information, are they actually triggering the DAG by entering that?
• 17:59
  SPEAKER_00: Or is it DAG then uses that information the next time it runs on its time it runs, exactly.
• 18:05
  SPEAKER_00: Okay, interesting.
• 18:06
  SPEAKER_00: That's really cool.
• 18:07
  SPEAKER_00: I haven't heard about taking user input from Streamlet in that way.
• 18:10
  SPEAKER_00: That's interesting.
• 18:10
  SPEAKER_02: The screen was just for visual manner.
• 18:13
  SPEAKER_02: I mean, they can define uh thresholds and it is written into a table in Snowflake, and the configuration is read by that.
• 18:20
  SPEAKER_02: There is also a DAG that reads that and changed the YAML accordingly, and then the DAG is always running, always every three hours.
• 18:27
  SPEAKER_02: I mean, right.
• 18:28
  SPEAKER_00: On the the business scale that makes sense to you.
• 18:30
  SPEAKER_02: I just want to add another thing.
• 18:32
  SPEAKER_02: We also made it flexible that let's say if you don't want it to send some data during the night, for instance, we also enable them some sort of a configuration that says, okay, during the several hours, ignore, don't trigger the DAG.
• 18:47
  SPEAKER_02: We also made that uh possible.
• 18:49
  SPEAKER_02: It's a little bit risky because they can mess up with the other things, but uh, we did it supervised, so we can monitor that as well.
• 18:56
  SPEAKER_00: Okay, interesting.
• 18:58
  SPEAKER_00: Yeah, that's cool.
• 18:58
  SPEAKER_00: And then for the DAG that's actually checking the metrics and how much they've changed, did you implement that using like some data quality framework?
• 19:08
  SPEAKER_00: Or is it your own custom logic that you've no, it's basically an own custom logic.
• 19:13
  SPEAKER_02: Another thing that we implemented, we created a custom operator to do that, which is also being fed from a configuration to run a query.
• 19:22
  SPEAKER_02: According to the threshold that we're set in the streamed application, we generate a query that will run against the tables that holds the business metrics.
• 19:30
  SPEAKER_02: And we simply use it.
• 19:32
  SPEAKER_02: We actually did it with Airflow's capability of dynamic task mapping because it fires almost a thousand or more than a thousand queries against those tables to test it.
• 19:42
  SPEAKER_02: So we did it with dynamic task mapping, which was a great, great to implement it that way.
• 19:47
  SPEAKER_02: Before that, we did it in a Python way with some loops and uh multi-threaded, but once we had with the dynamic task mapping capabilities of Airflow, it made it so simple and so readable in the logs later on.
• 19:59
  SPEAKER_00: I'm sure speaking of monitoring, I'm sure that makes it a lot easier for you to monitor the results on your end using that as opposed to having everything lumped in a loop or generating those tasks some other way.
• 20:10
  SPEAKER_02: Exactly.
• 20:10
  SPEAKER_02: And to do another thing, we took it to the next level.
• 20:14
  SPEAKER_02: Each query that is being basically executed, its results set is being captured into a table.
• 20:21
  SPEAKER_02: And we have a stream with application that basically consumes this data and also provides us with the dashboarding internally for us, for the engineering team, to see that these tests were actually performed and uh these are the results, and we can also check that as well, rather than also sending the alerts.
• 20:38
  SPEAKER_02: The alerts are sent to the business domains, but we can also have visibility on what exactly happened to each query that uh was executed by the monitoring.
• 20:48
  SPEAKER_00: Okay, yeah, that makes sense.
• 20:50
  SPEAKER_00: That's really cool.
• 20:51
  SPEAKER_00: Thank you for going through that use case.
• 20:53
  SPEAKER_00: It's great to hear, yeah, just a classic case of Airflow being used for that internal orchestration and really tangibly helping your business.
• 21:01
  SPEAKER_00: I always like to end with a standard final question, and that is what would you most like to see from development of the Airflow project in the near future?
• 21:09
  SPEAKER_02: Actually, we're using now the latest version of Airflow, and we missed the old UI.
• 21:15
  SPEAKER_02: Actually, the old UI was pretty good as opposed to the current one.
• 21:19
  SPEAKER_00: When you say the old UI, do you mean from you're using 2.9 and you like something in the 2.8 UI, or are you talking about it?
• 21:26
  SPEAKER_02: No, two, let's say further back, it'd be for the 2.5, I think.
• 21:31
  SPEAKER_02: We've been with 2.5 and now we're at 2.9.
• 21:34
  SPEAKER_02: And obviously the capabilities are much uh wider, but from UI perspective, it's a little bit hideous to do some of the things.
• 21:42
  SPEAKER_02: So I would wish they returned to the previous uh UI.
• 21:46
  SPEAKER_02: I would like to have to see more capabilities of column lineage level.
• 21:50
  SPEAKER_02: There is DAG lineage level right now, but I want to be able to see every column level lineage that is being orchestrated in the task, for instance.
• 22:00
  SPEAKER_02: I want to see how it's related in all aspects of uh column level lineage.
• 22:05
  SPEAKER_02: This is something that I would like to have.
• 22:07
  SPEAKER_02: And I think more logging capabilities of usage and uh what exactly what I would like to be able to do is to assess which executor or how much memory consumption did every executor consumed during the execution.
• 22:25
  SPEAKER_02: More resources on the executors.
• 22:27
  SPEAKER_02: I know we have that in the dashboard now in 2.9, but I want to take it to the next uh level to dig in into more details of the usage of the executor themselves.
• 22:37
  SPEAKER_00: Yeah, got it.
• 22:38
  SPEAKER_00: That's a really good list and kind of a good breadth of different things within Airflow.
• 22:44
  SPEAKER_00: The lineage request is particularly interesting.
• 22:46
  SPEAKER_00: I think there's definitely some work behind the scenes in terms of the open lineage project, which has some integration with Airflow.
• 22:53
  SPEAKER_00: It's actually going to be talked about in the Airflow Town Hall this week, which will have already happened by the time that this airs, but in case you're interested in joining.
• 23:00
  SPEAKER_00: But yeah, those are really great.
• 23:02
  SPEAKER_00: And I think for any UI updates, certainly submit issues to the Airflow project.
• 23:06
  SPEAKER_00: I know we've got some folks internally who often contribute to the Airflow UI, and they're always interested in hearing that user feedback.
• 23:11
  SPEAKER_00: All right.
• 23:12
  SPEAKER_00: Well, thank you for going through that.
• 23:13
  SPEAKER_00: We really appreciate your time today.
• 23:14
  SPEAKER_00: And then what is the best way for folks to get in touch with you?
• 23:17
  SPEAKER_02: They can contact me through LinkedIn, look me up in LinkedIn, and it's the best way to approach me, I think.
• 23:23
  SPEAKER_00: Oh, great.
• 23:24
  SPEAKER_00: Sounds good.
• 23:25
  SPEAKER_00: Thanks again for joining us.
• 23:26
  SPEAKER_00: We really appreciate it.
• 23:28
  SPEAKER_02: I really appreciate your being you're hosting me.
• 23:30
  SPEAKER_00: Great.
• 23:30
  SPEAKER_00: Thanks, Hanan.
• 23:33
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:39
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 23:43
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.