Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant

Transcript

The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: I've read lots of things on internet saying that people like literally having debates like whether Airflow is ETL tool or not.
• 0:07
  SPEAKER_01: But I'll go with the ETL side because it totally depends upon your use cases, right?
• 0:11
  SPEAKER_01: If if you're not running like huge production level things, if you're basically going to end up with millions of pros in the end, then Airflow is like a best solution for you.
• 0:21
  SPEAKER_01: You can use it as an ETL.
• 0:22
  SPEAKER_01: Again, depending upon what kind of things you use in the background.
• 0:26
  SPEAKER_01: If you're going to use the batch of like basic operators, like Python operators only, then Airflow can easily manage that load.
• 0:38
  SPEAKER_02: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:48
  SPEAKER_00: Hi everyone.
• 0:49
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:50
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Mihir Samant, who is a senior data analyst at eTravel A Group.
• 0:57
  SPEAKER_00: Mihir, how are you doing?
• 0:59
  SPEAKER_00: I'm doing good.
• 1:00
  SPEAKER_00: How about you?
• 1:01
  SPEAKER_00: I'm good.
• 1:01
  SPEAKER_00: I'm good.
• 1:02
  SPEAKER_00: I just got back from AWS reInvent, so I'm a little tired.
• 1:06
  SPEAKER_00: Okay.
• 1:06
  SPEAKER_00: But it's been a busy week.
• 1:08
  SPEAKER_00: Yeah.
• 1:09
  SPEAKER_01: I'm here.
• 1:09
  SPEAKER_01: Like I'm in Toronto and the weather is quite negative two.
• 1:13
  SPEAKER_01: It's snowing outside.
• 1:15
  SPEAKER_01: Other than that, it's it's all good.
• 1:17
  SPEAKER_00: Is that negative two Celsius, right?
• 1:19
  SPEAKER_01: Yep, negative two degrees Celsius.
• 1:21
  SPEAKER_01: That's right.
• 1:22
  SPEAKER_01: That's right.
• 1:22
  SPEAKER_00: I have been in negative two Fahrenheit as well, which is a bit of a different temperature.
• 1:27
  SPEAKER_00: But yeah, I heard there's like a huge storm that hit the upper northeast coast and they got a bunch of snow.
• 1:33
  SPEAKER_00: Did you guys get a lot there?
• 1:34
  SPEAKER_01: No, not in Toronto.
• 1:36
  SPEAKER_01: Toronto, for some reason, is populated and warm.
• 1:39
  SPEAKER_01: So we got hit later, but like I heard like a couple of weeks back, there was like a storm-like situation, which hit the other side, but not the way I see.
• 1:49
  SPEAKER_00: Okay, got it.
• 1:50
  SPEAKER_00: I'm glad you're not having to deal with six feet of snow or something.
• 1:54
  SPEAKER_00: But it's yeah, definitely winter time.
• 1:57
  SPEAKER_00: All right.
• 1:57
  SPEAKER_00: Well, we'll go ahead and dive in.
• 1:59
  SPEAKER_00: So for our listeners for today's podcast, we're going to discuss how eTraveli group implemented Airflow for its finance automation team.
• 2:07
  SPEAKER_00: And I want to start with some background.
• 2:10
  SPEAKER_00: Can you tell our listeners what eTraveli does?
• 2:13
  SPEAKER_00: What is your business model?
• 2:14
  SPEAKER_01: Okay.
• 2:15
  SPEAKER_01: So eTraveli Group is a leading tech platform for selling tickets.
• 2:19
  SPEAKER_01: So what we do is we have different entities around the globe.
• 2:23
  SPEAKER_01: So if you say my trip.com or go to gate or flight network, it's all comes under a big umbrella of eTravelI Group.
• 2:30
  SPEAKER_01: So basically we sell tickets and not only tickets, we also sell some ancillary stuff as well, package options or seat selections or priority services, something like that.
• 2:40
  SPEAKER_01: So that's what eTraveler is doing.
• 2:43
  SPEAKER_01: And yes, the business model is basically selling the cheaper flight tickets on our websites.
• 2:48
  SPEAKER_01: And travelers can go on our website and they can select any allies they wanted to travel with and the cheapest options.
• 2:55
  SPEAKER_01: Plus, the business model is basically those ancillary services which we have.
• 3:00
  SPEAKER_01: We are in business from uh almost like two, two roughly two decades.
• 3:05
  SPEAKER_01: The earlier company got merged with different companies, and we finally have a name e-Traveler Group.
• 3:11
  SPEAKER_01: Our headquarters in Sweden, and we also have offices around, like I work in Toronto office.
• 3:16
  SPEAKER_01: So we also have offices in India, Sweden, and Greece.
• 3:19
  SPEAKER_00: Oh, okay.
• 3:20
  SPEAKER_00: Interesting.
• 3:21
  SPEAKER_00: A very international presence then.
• 3:23
  SPEAKER_00: That's great.
• 3:25
  SPEAKER_00: Cool.
• 3:25
  SPEAKER_00: Makes sense for a travel company.
• 3:27
  SPEAKER_00: Awesome.
• 3:28
  SPEAKER_00: And so what does your team do for e-Travel?
• 3:31
  SPEAKER_01: So I'm a part of a finance automation group, a finance automation team, and it's uh actually a small part of a big finance team.
• 3:38
  SPEAKER_01: So in finance team, we roughly have 30 or 40-ish peoples.
• 3:42
  SPEAKER_01: And finance automation is just four people managing all the data related need with the finance.
• 3:48
  SPEAKER_01: So the whole aim of having finance automation is to take advantage of the new technologies to the finance team.
• 3:56
  SPEAKER_01: So in finance team, mostly we have accountants and business controllers who deal with once the there's a like month close.
• 4:03
  SPEAKER_01: So they open the book and try to match all the credits and uh debit-related stuff.
• 4:08
  SPEAKER_01: So they always had that issue.
• 4:11
  SPEAKER_01: Like during month end, there's like always a bottleneck, right?
• 4:14
  SPEAKER_01: Because all whole office or whole e-travel eye trying to take data from a couple of databases, which we had a huge bottleneck.
• 4:22
  SPEAKER_01: And that's the first point when my manager discovered there is a scope where we can take advantage of technology and give this to the finance team.
• 4:30
  SPEAKER_01: So I was like, after my manager, I was the first employee for the finance automation team.
• 4:35
  SPEAKER_01: And that was the whole idea, right?
• 4:37
  SPEAKER_01: The things what we can do.
• 4:38
  SPEAKER_01: So I got like a clean slate when I joined the eTri.
• 4:42
  SPEAKER_01: So in an interview, I said this things like I have experience working with Airflow for a couple of years.
• 4:48
  SPEAKER_01: So I can literally can start from scratch with LittleHelp.
• 4:52
  SPEAKER_01: And we can do lots of good things, like not just scheduling stuff, but extracting things and everything we can do it with Airflow.
• 5:00
  SPEAKER_01: So that's what we do.
• 5:02
  SPEAKER_00: Yeah, that was perfect.
• 5:03
  SPEAKER_00: I think it's so interesting to start a team from scratch like that.
• 5:08
  SPEAKER_00: I have done that myself.
• 5:09
  SPEAKER_00: It's obviously a ton of work, but it is kind of exciting and it's interesting to be able to evaluate everything kind of in a new light where you have no basis and sort of everything is an active decision.
• 5:22
  SPEAKER_00: So obviously, Airflow was one of those decisions that you made to implement.
• 5:26
  SPEAKER_00: As you said, you came in and started with nothing.
• 5:28
  SPEAKER_00: You had some prior experience with Airflow, but walk me through in a little more detail why you decided to use Airflow for this team and kind of what has your transition to using it looked like.
• 5:40
  SPEAKER_01: Yep.
• 5:40
  SPEAKER_01: So earlier, my manager was performing lots of things on Python only.
• 5:44
  SPEAKER_01: And trust me, she's from a finance background.
• 5:47
  SPEAKER_01: So she'd done accounting and everything stuff.
• 5:49
  SPEAKER_01: She was learning at that time Python and on those scripts.
• 5:52
  SPEAKER_01: So I proposed this thing.
• 5:54
  SPEAKER_01: There's like a tool in market called as a Airflow.
• 5:57
  SPEAKER_01: And it runs on Python.
• 5:58
  SPEAKER_01: And if we create a small script into it, it can or we can automate stuff.
• 6:03
  SPEAKER_01: And basically we can assign some tasks, get some data from this particular database process over here, and then push it on, like maybe on email or on Slack.
• 6:12
  SPEAKER_01: That was a whole thing I proposed in an interview.
• 6:16
  SPEAKER_01: And I think that thing landed me on this job.
• 6:18
  SPEAKER_01: So last thing to add into this is basically we were like totally like my manager was starting a new team.
• 6:24
  SPEAKER_01: And when you start something new, you don't get too much budget.
• 6:27
  SPEAKER_01: The good thing about Airflow is like it's open source and there's no actual cost of running Airflow.
• 6:33
  SPEAKER_01: If you have servers, then it's a good thing, right?
• 6:36
  SPEAKER_01: So the on very first day, what I did is basically I went to I installed Docker locally.
• 6:42
  SPEAKER_01: Then I went to Airflow website, I took the image, and then I just followed the guide.
• 6:47
  SPEAKER_01: Like it's simple to install Airflow on your environment.
• 6:50
  SPEAKER_01: So I did it.
• 6:51
  SPEAKER_01: I I created a one DAG, which is basically going on particular API, taking the data from there, and then we process the data and we just create a file locally.
• 7:02
  SPEAKER_01: I create that DAG within like three, four hours, including the installation time.
• 7:08
  SPEAKER_01: That was a good move.
• 7:09
  SPEAKER_01: And my boss loved it.
• 7:11
  SPEAKER_01: And at this point, we have roughly 15, 16 DAGs up and running on our production environment.
• 7:18
  SPEAKER_01: And we do have two in different environments like we created.
• 7:21
  SPEAKER_01: And trust me, like Airflow is like one of the easiest things, which I believe it's easiest to manage and implement.
• 7:28
  SPEAKER_01: Once you know what things you have to do with Efno, it's super easy and super cool.
• 7:33
  SPEAKER_00: Yeah, that's great.
• 7:34
  SPEAKER_00: I love hearing about success stories with Airflow like that, where people find it really easy to use.
• 7:39
  SPEAKER_00: Getting it up and running in a DAG deployed in half a day is a that's a pretty good day's work.
• 7:44
  SPEAKER_00: That's awesome.
• 7:45
  SPEAKER_00: So you mentioned you have now you're using it in production, you've also got a dev environment, you have like 15 to 16 DAGs running.
• 7:54
  SPEAKER_00: And I assume those are managing all of the data for your finance team.
• 7:58
  SPEAKER_00: Since you've implemented those, are there any particular Airflow features that you have leaned on or that are core to how you've implemented Airflow for that finance data?
• 8:11
  SPEAKER_01: Yes.
• 8:12
  SPEAKER_01: So one of the best features I love is like sensors.
• 8:14
  SPEAKER_01: It's pretty cool.
• 8:16
  SPEAKER_01: Whether it's like a data-related database-related center where, you know, when you SQL sensors, when you when it keeps on checking any updates, or maybe it's data-aware data sets, that is also one of the cool features.
• 8:28
  SPEAKER_01: We don't use data aware data sets at this point, but that's something which is in my mind and we'll be implementing in next maybe quarter.
• 8:36
  SPEAKER_01: The best thing about Airflow is it's like a clay.
• 8:40
  SPEAKER_01: We can mole it as the way we want.
• 8:42
  SPEAKER_01: The best thing is if you wanted to change your executor, if you wanted to change your triggers, you can do that in the airflow level things.
• 8:50
  SPEAKER_01: That's also one of the great things we have.
• 8:55
  SPEAKER_00: Yeah, that's interesting.
• 8:56
  SPEAKER_00: So you're using lots of sensors, you're using data sets.
• 8:59
  SPEAKER_00: You mentioned Airflow is super flexible.
• 9:01
  SPEAKER_00: You can configure it in whichever way works for your team.
• 9:04
  SPEAKER_00: I mean, you mentioned you had some airflow experience prior to this role.
• 9:09
  SPEAKER_00: But one of the things that I hear about working with folks in the Airflow community a lot is that it's great that you have all of these knobs and things that you can turn to customize it for your solution.
• 9:20
  SPEAKER_00: Or I actually really liked your analogy of it's like the clay that you mold with.
• 9:25
  SPEAKER_00: But I think sometimes that can be also a little overwhelming for people or it's confusing.
• 9:31
  SPEAKER_00: They don't know exactly how they need to tune to implement their use case.
• 9:36
  SPEAKER_00: What has been your experience with that?
• 9:38
  SPEAKER_00: Do you think are you just relying on having used Airflow for a long time or anything that's helped you know the right things to focus on when getting it set up properly?
• 9:49
  SPEAKER_01: Yep.
• 9:49
  SPEAKER_01: So earlier when when we just started with changing a couple of things with Airflow, it was kind of difficult.
• 9:55
  SPEAKER_01: Then I start using documentation stuff.
• 9:58
  SPEAKER_01: I start going to Slack Overflow.
• 10:00
  SPEAKER_01: Sometimes I ask chat GPT, but I don't understand any of the Dockering widget stuff.
• 10:04
  SPEAKER_01: So I do that.
• 10:06
  SPEAKER_01: I think once you start experimenting stuff on your test environment or maybe on your local environment, you can slowly get to know if you're changing this particular statement or if this particular line, what exactly it reflects, right?
• 10:19
  SPEAKER_01: So that's how I learn.
• 10:20
  SPEAKER_01: One of the best things which I've did by myself is git sync.
• 10:24
  SPEAKER_01: So earlier I had issues with if I have production and uh test environment.
• 10:28
  SPEAKER_01: So what I do is basically I write a code, I test locally, and then I push it on my test environment with our sync command, right?
• 10:36
  SPEAKER_01: So sometimes if someone else is working on that particular file, that was causing issue.
• 10:41
  SPEAKER_01: So what we did is like we created a git sync.
• 10:44
  SPEAKER_01: So it runs on the container, it keeps on checking every five, 10 seconds, it keeps on checking the repository.
• 10:50
  SPEAKER_01: And once we commit on repository, it changes everything on the tag folder, which is on our Tarku Talker environment.
• 10:58
  SPEAKER_01: So that's one of the cool features.
• 11:00
  SPEAKER_01: Like I know like 3.0 coming with lots of good features, but this is like one of the features which we are lagging at this point on 2.10.
• 11:10
  SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:20
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 11:31
  SPEAKER_02: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 11:40
  SPEAKER_00: Deployment can be challenging for sure, and we have our own kind of opinionated solutions about that at Astronomer with our managed service.
• 11:48
  SPEAKER_00: I know I myself have used GitSync before I worked for Astronomer when I implemented Airflow at other companies.
• 11:54
  SPEAKER_00: GitSync was usually my method.
• 11:56
  SPEAKER_00: Granted, I'm gonna date my Airflow experience here a little bit, but this was back in like the 1.9 days.
• 12:01
  SPEAKER_00: So I think it was probably even more complex than it is now.
• 12:05
  SPEAKER_00: So yeah, that's definitely a challenge for sure, productionizing everything around Airflow.
• 12:11
  SPEAKER_00: And it sounds like so.
• 12:12
  SPEAKER_00: Did you say you're running it on-prem?
• 12:14
  SPEAKER_00: So you have everything set up within your servers?
• 12:18
  SPEAKER_01: So first, so like you know, the first couple of months when I started, we had one test environment.
• 12:24
  SPEAKER_01: We were just dumping all our codes into it.
• 12:27
  SPEAKER_01: And then after seven, eight months post that we had a production environment as well.
• 12:32
  SPEAKER_01: Right?
• 12:32
  SPEAKER_01: It's basically we were removing all the stuff one by one.
• 12:36
  SPEAKER_01: And it happened with me.
• 12:38
  SPEAKER_01: Like I mistakenly synced the whole folder from my local to the test.
• 12:43
  SPEAKER_01: And it changed a couple of codes, which I don't wanted.
• 12:45
  SPEAKER_01: And thankfully, it happened on test and not on the production.
• 12:49
  SPEAKER_01: Yep.
• 12:49
  SPEAKER_01: So that's that was the thing when I started researching about hit sync.
• 12:54
  SPEAKER_01: And I just simply again, I'm not bragging about myself, it's just like how community is built with the Airflow and how people are helping it.
• 13:02
  SPEAKER_01: Like I'm also on the Airflow Slack channels.
• 13:05
  SPEAKER_01: So people always type questions, community people jumps onto it, help them.
• 13:10
  SPEAKER_01: I went through lots of medium blogs, and I found one of the blogs is important.
• 13:15
  SPEAKER_01: And yeah, that helped me a lot.
• 13:17
  SPEAKER_01: In implementing GetSync was like a three-minute thing, actually, if you know what to exactly do.
• 13:24
  SPEAKER_00: Yeah, that's awesome.
• 13:25
  SPEAKER_00: I love uh also stories about the Airflow community being great.
• 13:30
  SPEAKER_00: This is obviously I manage our developer relations team.
• 13:33
  SPEAKER_00: This is my bread and butter.
• 13:35
  SPEAKER_00: So I'm really happy to hear when it's working for people.
• 13:38
  SPEAKER_00: I do think that the Airflow community being so strong is maybe one of its biggest proponents.
• 13:45
  SPEAKER_00: We hear a lot of folks say that that's one of the reasons that they choose to adopt it because there's so many other, as you said, people that you can lean on for help and inspiration.
• 13:54
  SPEAKER_00: And I think it's always great for me to see that that results in it both being used on really small teams, sort of, you know, teams like yours that have like a relatively small number of DAGs, but like it's powering a critical business function all the way up to giant companies with tens of thousands of DAGs and it's good for everything in between, right?
• 14:15
  SPEAKER_01: That's true.
• 14:16
  SPEAKER_01: And also, like, you know, I've read lots of things on the internet saying that people like literally having debates like whether Airflow is ETL tool or not.
• 14:24
  SPEAKER_01: But I'll go with the ETL side because it's totally depends upon your use cases, right?
• 14:29
  SPEAKER_01: If if you're not running like a huge production level things, if you're basically going to end up with millions of rows in the end, then Airflow is like a best solution for you.
• 14:38
  SPEAKER_01: You can use it as an ETL.
• 14:40
  SPEAKER_01: Again, depending upon what kind of things you use in the background.
• 14:43
  SPEAKER_01: If you're going to use the batch of like basic operators, like Python operators only, then Airflow can easily manage that load.
• 14:50
  SPEAKER_01: Again, depends upon your servers and all those things.
• 14:53
  SPEAKER_01: But yeah, Airflow is ETL2 for us, and it's doing a great job from last almost like two years now that nothing bothered us.
• 15:01
  SPEAKER_01: So all doing good.
• 15:03
  SPEAKER_00: Yeah, that's great.
• 15:04
  SPEAKER_00: Well, that kind of leads me to my next question, which you said you've been using it for like two years now, implemented when it was a clean slate.
• 15:12
  SPEAKER_00: Now that you've lived with it for a little while, where do you see your airflow usage going in the future?
• 15:17
  SPEAKER_00: Do you expect that you'll have more DAGs or use in other teams?
• 15:23
  SPEAKER_00: Or where are you guys at?
• 15:25
  SPEAKER_01: We also deal with other teams as well.
• 15:27
  SPEAKER_01: So depending upon their needs, they reach out to us.
• 15:30
  SPEAKER_01: So we create a solution for them.
• 15:31
  SPEAKER_01: So most of the time it will become part of our team.
• 15:34
  SPEAKER_01: Like that DAG become part of our team.
• 15:36
  SPEAKER_01: We won't be creating other environment for them.
• 15:39
  SPEAKER_01: We'll just run their part or on our server on server.
• 15:42
  SPEAKER_01: Talking about next plans, I think we like in coming months, we have we are implementing a solution.
• 15:49
  SPEAKER_01: So like the couple of teams, a couple of people from other teams are working on Databricks side.
• 15:55
  SPEAKER_01: So we're moving most of our databases over there.
• 15:59
  SPEAKER_01: So my solution is basically we will be connecting Airflow with Databricks.
• 16:04
  SPEAKER_01: So we will be taking data from Databricks, but we won't be processing it because we don't want to add any additional costs on top of it.
• 16:11
  SPEAKER_01: So the execution part will remain with the Airflow.
• 16:14
  SPEAKER_01: We'll just connect Airflow to Databricks.
• 16:16
  SPEAKER_01: Databricks will push the data back to Airflow, we'll process it, and we'll send it to users.
• 16:21
  SPEAKER_01: So like basically, we we send all the data to end users because it's like financial stuff.
• 16:26
  SPEAKER_01: They do their own calculations after that, depending about countries and laws and regulations.
• 16:32
  SPEAKER_01: So we get the data, we process, and we send it back to you.
• 16:36
  SPEAKER_01: We don't actually save anything on database most of the time.
• 16:41
  SPEAKER_00: Yeah, interesting.
• 16:42
  SPEAKER_00: That's uh and such funny timing.
• 16:43
  SPEAKER_00: We literally just about an hour ago, at least at the time we're recording, but did a webinar on using Databricks and Airflow together.
• 16:51
  SPEAKER_00: So this is a really popular topic.
• 16:53
  SPEAKER_00: I uh funny that the universe does things like this.
• 16:56
  SPEAKER_00: To pull things in one day.
• 16:58
  SPEAKER_00: But yeah, lots of teams combine those tools for similar, you know, reasons to what you mentioned, especially gives you a little more flexibility in terms of cost options.
• 17:09
  SPEAKER_00: And there's now, you know, the Databricks provider within Airflow has quite a lot of functionality that makes it pretty easy, especially if you're working with that kind of Databricks workflows and Databricks jobs ecosystem.
• 17:21
  SPEAKER_01: So you can do lots of things.
• 17:23
  SPEAKER_01: Like I was reading documentation both on Airflow and Databricks website.
• 17:27
  SPEAKER_01: So there are multiple ways to perform the same task.
• 17:29
  SPEAKER_01: So you can trigger your workflows, which you already saved in Databricks side, or maybe you can just use the SQL operators to pull the like you just send you SQL commands, square queries, and it will just push the records directly to you.
• 17:45
  SPEAKER_01: So that's kind of flexibility which I need.
• 17:47
  SPEAKER_01: I'm glad that Airflow and Databricks both created, I think Airflow side created such a great operators to work with.
• 17:55
  SPEAKER_01: So yeah, I'm super happy.
• 17:58
  SPEAKER_00: So we're about at the end of our time together.
• 18:00
  SPEAKER_00: So I want to always end with a standard or final question for everybody, and that is what would you most like to see from the Airflow project in the near future?
• 18:09
  SPEAKER_01: The one of the most popular demands, what every data analyst or data engineer wanted from like many years, it's basically tag versioning, which is coming up with 3.0.
• 18:19
  SPEAKER_01: So that's a good thing.
• 18:20
  SPEAKER_01: Yeah.
• 18:20
  SPEAKER_01: I personally want one thing, you know, like when you when you run multiple tags, or maybe when you run a tag and there's like multiple instances of that.
• 18:28
  SPEAKER_01: And when you go back in pass and you don't see the uh like the the code which you ran at that particular instance, the the code is basically the latest code which is running.
• 18:38
  SPEAKER_01: So that's the only that's the personally which I want because I want to see what caused that particular error, just jumping back into pass.
• 18:45
  SPEAKER_01: And maybe if we have uh pop-up errors or maybe notification on Airflow, which says this particular tag feel.
• 18:53
  SPEAKER_01: Not in my case, because I can easily manage all those stuffs with the the cluster uh activity stuff.
• 18:58
  SPEAKER_01: But if somebody having more than 50 hundred tags, it's basically hard to find what exactly feel unless it's like they have notification turned on.
• 19:06
  SPEAKER_01: So if the Airflow UI giving like a small pop-up notification, like this particular thing feel, will add up a good amount of value in it.
• 19:16
  SPEAKER_00: Yeah, those are two great requests.
• 19:19
  SPEAKER_00: I know with DAG versioning, what you're describing, being able to see past versions when they ran is coming.
• 19:25
  SPEAKER_00: I know that's definitely a big limitation, especially if you have any sort of dynamic DAG generation or things like or just your DAGs change frequently.
• 19:34
  SPEAKER_00: Yeah, it's really hard to see that historical context.
• 19:37
  SPEAKER_00: Um, and losing that can be quite challenging.
• 19:39
  SPEAKER_00: I haven't heard a request before for pop-ups in the UI.
• 19:42
  SPEAKER_00: I don't think that's a good one.
• 19:44
  SPEAKER_00: I know you can like filter for task status and stuff, but as you said, if you have a very high number of DAGs, sometimes even that can't help you.
• 19:50
  SPEAKER_00: So maybe you could do something like a plug-in.
• 19:53
  SPEAKER_00: Um yeah, I'm not sure.
• 19:55
  SPEAKER_01: So for in my case, what I have done is basically I created a function which basically calls or like send data on my Slack channel saying that this particular thing failed and I provide the U URL.
• 20:08
  SPEAKER_01: So whenever I click on that, it jumps back onto the Slack grid view of the of the airflow.
• 20:14
  SPEAKER_00: Yeah, interesting.
• 20:15
  SPEAKER_00: Yeah, I can see where that would be very helpful for sure.
• 20:18
  SPEAKER_00: Yeah, but yeah.
• 20:20
  SPEAKER_00: All right.
• 20:21
  SPEAKER_00: To close us out here, what is the best way for folks to get in touch with you?
• 20:25
  SPEAKER_01: You guys can reach me out on LinkedIn.
• 20:28
  SPEAKER_01: I am also write blogs on Medium.
• 20:30
  SPEAKER_01: Plus, I'm all also available on Apache's Airflow champions page.
• 20:34
  SPEAKER_01: I'm one of the champion for I think robot tree.
• 20:38
  SPEAKER_01: And yeah, you guys can ping me or just add me on LinkedIn.
• 20:42
  SPEAKER_00: That's great.
• 20:43
  SPEAKER_00: Well, we appreciate your contributions to the Airflow community that come along with being a champion.
• 20:48
  SPEAKER_00: So thank you for doing that.
• 20:50
  SPEAKER_00: And Mihir, it's been really great talking to you today.
• 20:53
  SPEAKER_00: Thanks so much for coming on.
• 20:54
  SPEAKER_01: Thank you so much, Kenda.
• 20:56
  SPEAKER_01: It was lovely talking to you.
• 20:59
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 21:06
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 21:10
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.