Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille

Transcript

Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We've used the Airflow Bulletin stats, hooked it up to a Grafana so that we also have better monitoring of whether or not there's any resource problems and we can really get behind that.
• 0:09
  SPEAKER_02: So that's gone away as well.
• 0:10
  SPEAKER_02: And uh I guess the final and more recent point that we had to deal with is the synchronizing data sets in multiple instances.
• 0:17
  SPEAKER_02: And for that, I made a dedicated service, but this was pre-Airflow 3.0, so I'm looking forward to comparing that with uh I know there's a new solution on Airflow 3.0 to synchronize data sets across multiple instances.
• 0:27
  SPEAKER_02: So I'm gonna have to check that out.
• 0:34
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:45
  SPEAKER_00: Hey Airflow community, exciting news.
• 0:48
  SPEAKER_00: This fall we're launching Beyond Analytics, a virtual data ops conference happening September 16th from 12 to 3 p.m.
• 0:55
  SPEAKER_00: Eastern.
• 0:55
  SPEAKER_00: You'll hear from data leaders on how modern data ops and orchestration are empowering AI, machine learning, and production grade data products.
• 1:02
  SPEAKER_00: And here's the best part.
• 1:04
  SPEAKER_00: You can join Mark Lumberke for a fast track Air Force 3 certification prep session and get a discount code for a free certification, a$150 value.
• 1:13
  SPEAKER_00: The conference is virtual, it's free, and it's made just for you.
• 1:17
  SPEAKER_00: You can register now at astronomer.io slash beyond slash data flowcast.
• 1:25
  SPEAKER_00: Hi everyone, and welcome to the Data Flowcast.
• 1:28
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Sebastian Crocovier, who is a data engineer and Numberly.
• 1:35
  SPEAKER_00: Sebastian, how are you?
• 1:37
  SPEAKER_02: Hi, Kenton.
• 1:37
  SPEAKER_02: I'm fine.
• 1:38
  SPEAKER_02: Thank you so much for having me.
• 1:39
  SPEAKER_02: I'm working remotely from Taiwan and it's a nice warm summer over here.
• 1:43
  SPEAKER_00: Amazing, amazing.
• 1:44
  SPEAKER_00: That's great.
• 1:45
  SPEAKER_00: Yeah, quite the time difference between Seattle and Taiwan, but so it's like we're both having good summers, making it work.
• 1:52
  SPEAKER_00: So definitely.
• 1:53
  SPEAKER_00: I love that.
• 1:54
  SPEAKER_00: All right, we'll go ahead and dive right into it.
• 1:57
  SPEAKER_00: So for today's podcast, we're going to discuss how Numberly has scaled their on-prem airflow environment.
• 2:03
  SPEAKER_00: I'm really excited to get into this topic with you.
• 2:06
  SPEAKER_00: But let's go ahead and start with some background.
• 2:08
  SPEAKER_00: So, first, can you tell us a little bit about what Numberly does?
• 2:11
  SPEAKER_02: Yeah, definitely.
• 2:12
  SPEAKER_02: So, Numberly, to give everyone an idea, is a company that has a revenue stream of around 70 to 80 million euros, and it has offices in Montreal, New York, London, Dubai, and the main office in Paris because it's a French company.
• 2:24
  SPEAKER_02: We're marketing technologists.
• 2:26
  SPEAKER_02: What that means is that on the marketing side, we do it all.
• 2:29
  SPEAKER_02: We do CRM, omnichannel, consulting, real-time bidding.
• 2:32
  SPEAKER_02: We have our own platform.
• 2:34
  SPEAKER_02: So I guess we also do SaaS.
• 2:35
  SPEAKER_02: I'm not going to get a lot into it because I'm more on the tech side, but we have some very established clients like HSBC, Air France, or Moleskin.
• 2:43
  SPEAKER_02: On the tech side, we have to have the means to make the marketing vision a reality.
• 2:48
  SPEAKER_02: So we also are big fans of owning our tech and the expertise that comes along with it.
• 2:53
  SPEAKER_02: So we're our own internet service provider.
• 2:55
  SPEAKER_02: Most of our infrastructure is hosted on bare metal servers that we have in multiple data centers where we rent racks.
• 3:01
  SPEAKER_02: And we have our own teams for most things like infrastructure, networking, front end, back end, and data engineering, data science, and data analysis.
• 3:09
  SPEAKER_02: We basically only use cloud for very specific use cases or clients that are locked in.
• 3:13
  SPEAKER_02: And we're big fans of self-hosting and open source.
• 3:16
  SPEAKER_02: We actually finance a few open source projects and we host our own open source projects on GitHub.
• 3:22
  SPEAKER_00: Yeah, that's so interesting.
• 3:24
  SPEAKER_00: Being your own internet service provider is a level of in-house that I have not really encountered before.
• 3:30
  SPEAKER_00: That's actually pretty cool that you're able to do that.
• 3:32
  SPEAKER_02: Yeah, it's really nice.
• 3:33
  SPEAKER_02: And that means that like we get on our infrastructure, we get uh internet speeds of superior to a gigabyte.
• 3:39
  SPEAKER_02: So for a lot of our tracking use cases, that's really nice.
• 3:43
  SPEAKER_00: Yeah, yeah, I bet.
• 3:44
  SPEAKER_00: And so you mentioned a little bit that you are big fans of open source software and managing that yourself.
• 3:51
  SPEAKER_00: So I assume that is where Airflow has come in.
• 3:54
  SPEAKER_00: So maybe tell me a little bit about what your team does for the business and then how you use Airflow.
• 3:59
  SPEAKER_02: So I'm a senior member of staff on a data engineering team of about like 20 people.
• 4:04
  SPEAKER_02: And we really do strictly data engineering, creating automated pipelines, services that deliver data and the monitoring for those pipelines and services because there's other teams that take care of the data science and data analysis parts.
• 4:15
  SPEAKER_02: Our stack is, I guess generally you could say Python, PySpark, some Kubernetes configuration, and then querying and other database operations.
• 4:24
  SPEAKER_02: And we use Airflow specifically for orchestration, obviously.
• 4:27
  SPEAKER_02: We have a little bit less than 2,000 jobs on our production instance.
• 4:32
  SPEAKER_02: And we also have development and sandbox instances for testing and migrations.
• 4:37
  SPEAKER_02: Our jobs vary a lot.
• 4:39
  SPEAKER_02: Like we can have small jobs that run every five to 15 minutes, and you can have bigger jobs that run only monthly.
• 4:45
  SPEAKER_02: And then you can have jobs that have only one task, jobs that have hundreds and hundreds of tasks.
• 4:49
  SPEAKER_02: So we really tested how robust Airflow is and how it works for each of these use cases.
• 4:54
  SPEAKER_02: Um, and the things we really love about Airflow, there's a couple.
• 4:58
  SPEAKER_02: It's first of all, it's in Python, and that's really the language that different technical expertise jobs have in common.
• 5:06
  SPEAKER_02: Like you won't find another language that you can talk about with the backend guys, infrastructure guys, and IT security guys.
• 5:13
  SPEAKER_02: It's open source, obviously.
• 5:14
  SPEAKER_02: I've said how much we love open source.
• 5:16
  SPEAKER_02: And even though it's open source, you don't always have a strong community, but in this case, Airflow does have one.
• 5:20
  SPEAKER_02: It's very well maintained.
• 5:22
  SPEAKER_02: It has extensive documentation, and you can find like strong communities on things like Stack Overflow or Reddit if you want to talk about it with people online.
• 5:30
  SPEAKER_02: And one of the things that might surprise you is that we really like about Airflow is we found that having such a nice UI actually helps you transfer responsibility away from the technical owners and back to more of the stakeholders of the data.
• 5:42
  SPEAKER_02: So we have maybe project managers or marketers whose clients depend on certain jobs.
• 5:48
  SPEAKER_02: And then when there's any issues or there's maybe worries on the client side, the marketers will just pull up the Airflow UI and be like, oh no, I saw your job.
• 5:55
  SPEAKER_02: It's working fine, it ran this morning.
• 5:57
  SPEAKER_02: You should have the files or you should have whatever the job does on your side.
• 6:01
  SPEAKER_02: Maybe you can check it.
• 6:02
  SPEAKER_02: And it really allows them to have a level of technical responsibility that I don't think we would be able to give them as easily otherwise.
• 6:10
  SPEAKER_00: Yeah, that's really cool.
• 6:12
  SPEAKER_00: I have not actually heard too many stories about the Airflow UI even being made available to teams like that, or especially for kind of customer-facing jobs.
• 6:21
  SPEAKER_00: So I actually love the feedback for the project that the UI is really simple to understand.
• 6:26
  SPEAKER_02: Um Yeah, and you have um what's it called?
• 6:28
  SPEAKER_02: You have like roles, so you can definitely give people who you don't want to read-only access so that they don't modify anything, but they still get all the information that they need.
• 6:36
  SPEAKER_02: So we it was really great.
• 6:37
  SPEAKER_00: Of course.
• 6:38
  SPEAKER_00: Okay, so that's really interesting.
• 6:39
  SPEAKER_00: So Airflow, so your team, uh, what you're doing as a data engineer seems very core to your business, and then you're using Airflow to support that orchestration.
• 6:48
  SPEAKER_00: Given that you've set the context of having everything on-prem and your company always manages everything in-house, I would love to dig into your Airflow setup because I also understand that you're running it both on-prem and at a fairly large scale.
• 7:03
  SPEAKER_00: So, how did you go about setting up Airflow this way?
• 7:05
  SPEAKER_02: We've had multiple Airflow setups over the year, but our latest setup is using the Airflow Helm charts.
• 7:12
  SPEAKER_02: You guys have Helm charts that are made available to people, and then we have to overload them a little bit because we have some configuration that doesn't exist that's on our side.
• 7:20
  SPEAKER_02: But I talked about it with one of my colleagues, his name is Cyril Labrosse, and he said that in the latest versions of the of the Helm charts, it's much more customizable, and he's had a much easier time to set it up for a lot of our use cases.
• 7:36
  SPEAKER_02: We especially we have a little bit of a painful tool that we use on the security side called Kiverno, and that tool requires some specific setup, and that recently has gotten much, much better.
• 7:46
  SPEAKER_02: So he's very happy about that.
• 7:47
  SPEAKER_02: As for the rest, we deploy Airflow on Kubernetes, and then we use the Kubernetes executor to spin up pods every time we we want to run a task.
• 7:55
  SPEAKER_02: So generally that's the setup.
• 7:57
  SPEAKER_02: It might be a little bit complicated for a single person, but I think it's very achievable for companies.
• 8:02
  SPEAKER_02: And uh, we've been running Airflow since at least 1.10.
• 8:05
  SPEAKER_02: So we've had a few, I wouldn't want to say performance issues or bottlenecks we've hit over the years.
• 8:11
  SPEAKER_02: If you want, we could discuss those.
• 8:13
  SPEAKER_00: Yeah, absolutely.
• 8:14
  SPEAKER_00: I'd love to hear more about that.
• 8:15
  SPEAKER_00: I also started with Airflow myself around the same time, maybe 1.9, coming into 1.10.
• 8:20
  SPEAKER_00: So it's, I think the project has improved a tremendous amount in terms of performance since then.
• 8:25
  SPEAKER_00: So yeah, tell me a little bit about that evolution.
• 8:28
  SPEAKER_02: So yeah, this the first thing is discussing with like open source communities or people who use Airflow since that time.
• 8:33
  SPEAKER_02: One of the main points of discussion is how do you how do you what do you get your jobs from?
• 8:39
  SPEAKER_02: How do you synchronize them?
• 8:40
  SPEAKER_02: How do you host them?
• 8:41
  SPEAKER_02: I know some people have mono repos where they just have all their Airflow jobs in one Git repo repository.
• 8:47
  SPEAKER_02: We're more we like to separate the concerns based off of the business use cases a little bit more.
• 8:53
  SPEAKER_02: So we have a single repository for a single DAG generally, or maybe like a group of dynamic DAGs or two or three DAGs that have work around a similar business use case.
• 9:03
  SPEAKER_02: So that means that if we have a little bit less than 2,000 DAGs to push on Airflow, we have a little bit less than 2,000 hub repositories that we want to sync.
• 9:13
  SPEAKER_02: And yeah, that's a lot of repos, especially let's say since this is all hosted on Kubernetes, your instance goes down for whatever reason, and then you want to spin it back up again, and then it has to synchronize all the repos.
• 9:24
  SPEAKER_02: It's down and your service is not working in the meanwhile.
• 9:27
  SPEAKER_02: So you don't want that to take 15, 20 minutes, you want it to take like a minute or something.
• 9:30
  SPEAKER_02: And in the past, we were using git sync for that, but we found that we had it was too slow, and also it was GDoSing our GitLab instance because we self-host lab.
• 9:40
  SPEAKER_02: So we've moved away from that, and now we use the NFS, which is a uh networking network file system, and we have our own custom service that basically only pushes the changes that we do.
• 9:51
  SPEAKER_02: So we only push to the NFS when there's a repository that actually makes some changes, not every single time it needs to reload everything.
• 9:58
  SPEAKER_02: And uh, because the file system is on the network, you don't need to synchronize, it's just there, and then you connect to it, and then you have all the DAGs.
• 10:05
  SPEAKER_02: So that that that's been much better for us.
• 10:07
  SPEAKER_02: We've had some issues in the past as well with like resource hogging or resource bottlenecks in terms of CPU, but I feel like that was along the versions, Airflow has gotten more efficient, and uh also we've gotten better at using things like resource pools, moving away from a single scheduler to multiple schedulers on multiple instances.
• 10:26
  SPEAKER_02: And I feel like the multiple scheduler on a single instance has gotten, I guess, seamless right now.
• 10:30
  SPEAKER_02: You just tell him, Oh, I want multiple schedulers, and poof, you have multiple schedulers.
• 10:34
  SPEAKER_02: And also, we've used uh Airflow Bulletin stats, hooked it up to a Grafana, so that we also have better monitoring of whether or not there's any resource problems, and we can really get behind that.
• 10:45
  SPEAKER_02: So that's gone away as well.
• 10:47
  SPEAKER_02: And uh, I guess the final and more recent point that we had to deal with is the synchronizing data sets in multiple instances.
• 10:54
  SPEAKER_02: And for that, I made a dedicated service, but this was pre-Airflow 3.0, so I'm looking forward to comparing that with, I know there's a new solution on your flow 3.0 to synchronize data sets across multiple instances.
• 11:05
  SPEAKER_02: So I'm gonna have to check that out.
• 11:09
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:18
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 11:30
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 11:38
  SPEAKER_00: I think using the API is not necessarily synchronized in the sense that Airflow is aware of it, but there are endpoints or more endpoints on the data sets, or now it's assets endpoint that would allow you to do something like that.
• 11:51
  SPEAKER_00: But that's really cool.
• 11:52
  SPEAKER_00: How does your service work at a high level?
• 11:55
  SPEAKER_02: It's fairly simple.
• 11:56
  SPEAKER_02: You basically, like you said, use the API to detect all the data sets, and then you have a single message queue.
• 12:03
  SPEAKER_02: In this case, it's Apache Kafka, and we just push all the data sets or data assets to the Kafka messaging queue, I guess you could call it.
• 12:12
  SPEAKER_02: And once you have them all there, with their the different configuration and all the metadata that tells you where they're from, how recent they are, then you just push them back to the other airflows.
• 12:22
  SPEAKER_02: What's making this easier is that it's not data set scheduling doesn't work the same way as external event scheduling.
• 12:29
  SPEAKER_02: You don't have a single job ROM for every single event.
• 12:32
  SPEAKER_02: So that's easier.
• 12:33
  SPEAKER_02: Every time you have a data set, you just check is this at least as recent as the latest one on my current instance?
• 12:40
  SPEAKER_02: If it is, I don't do anything because it's already run.
• 12:42
  SPEAKER_02: And so that actually allows you for a little bit more flexibility than just external event triggers where for every single event you have a DAG RAM.
• 12:49
  SPEAKER_00: Interesting.
• 12:49
  SPEAKER_00: So it sounds like you've implemented your own version of what in Airflow 3 they're called asset watchers, which is the implementation for event-driven scheduling.
• 12:56
  SPEAKER_00: So actually a Kafka one was just added.
• 12:59
  SPEAKER_02: Because I know there's something on AIP, I think it's 82 that does something like this.
• 13:04
  SPEAKER_02: And the only difference between this one and that one is that the one I saw that was implemented is Airflow has to constantly pull to check on the other instances to make sure that it's not missing anything.
• 13:15
  SPEAKER_02: Whereas here you're pushing.
• 13:17
  SPEAKER_02: So it's only when there's a new event that you verify.
• 13:20
  SPEAKER_02: So you use a little bit less compute, but then you have to uh make all the moving parts and then monitor all the moving parts, whereas the other one is just airflow.
• 13:27
  SPEAKER_00: I think here, yeah, have different in Airflow 3 different options for how you implement.
• 13:32
  SPEAKER_00: But yeah, that's really interesting.
• 13:33
  SPEAKER_00: And I'm curious to hear down the road what you think after you've evaluated it and how it works compared to your current solution.
• 13:41
  SPEAKER_00: You mentioned earlier, I guess I should back up and say that everything you've described, I would say is a very sophisticated Airflow implementation, especially for all on-prem that you've done yourself.
• 13:51
  SPEAKER_00: You mentioned that you have a fairly large team yourself, and then you've also got other folks at the company who are interacting with Airflow, even if just as viewers.
• 14:01
  SPEAKER_00: Are there things that you have done to democratize the word I usually use, make Airflow easier to work with other teams?
• 14:09
  SPEAKER_02: Yeah, that's a great question, actually.
• 14:11
  SPEAKER_02: The first thing I would say that we've done is in the data engineering team and in some of the other data teams, everyone has Airflow on their local machine.
• 14:19
  SPEAKER_02: It's before you even get to the development instances, you have a local installation of Airflow.
• 14:25
  SPEAKER_02: And this doesn't need to get very complex.
• 14:27
  SPEAKER_02: You don't need to set anything up.
• 14:28
  SPEAKER_02: You just pip install Apache Airflow.
• 14:30
  SPEAKER_02: Poof, you have already a web server, a database, and a scheduler on your machine.
• 14:35
  SPEAKER_02: And then I think it's like you need to initialize the database and then tell your standalone version to just spin everything up.
• 14:42
  SPEAKER_02: And you have a UI, you have everything you need, and that allows you to have the same kind of setup as you would have in production.
• 14:48
  SPEAKER_02: So you can already test everything locally before you even get to a development airflow instance or whatever.
• 14:54
  SPEAKER_02: So I think this is one of the strengths of Airflow is that it's installed so easily on a developer's laptop.
• 14:59
  SPEAKER_02: And that's one of the important things that allows us to make everyone Airflow literate on the team.
• 15:05
  SPEAKER_02: Another thing we've done is I'm not sure if this helps democratize Airflow.
• 15:09
  SPEAKER_02: It's more of a quirk of our organization.
• 15:11
  SPEAKER_02: I guess everyone, when they make a project their own, they set it up in the way that they want.
• 15:15
  SPEAKER_02: So we already had the internal library that allows us to connect to all the different databases, file systems, the messaging queues, et cetera, where we host data.
• 15:23
  SPEAKER_02: So instead of using all the range of operators that Airflow has, because Airflow has different operators for most of those kinds of services, we mostly rely on the Python operator and all of its little siblings to uh run Python code.
• 15:38
  SPEAKER_02: And on in that Python code, we'll use our internal library to connect to most of the other things we have.
• 15:44
  SPEAKER_02: Because our tech teams are already very literate and know that library very well, and maybe not are not as proficient with all the different Airflow operators, and it allows us to more easily connect to things.
• 15:56
  SPEAKER_02: Another interesting thing that we have is in terms of dependency management.
• 16:01
  SPEAKER_02: This is another thing that I think people talk about in terms of Airflow as well.
• 16:05
  SPEAKER_02: What dependencies does your code run with?
• 16:08
  SPEAKER_02: Uh, we found that the easiest way is to have uh a single big image that has all the dependencies for 99% of your use cases.
• 16:15
  SPEAKER_02: So we'll call that like our default image.
• 16:18
  SPEAKER_02: That's the one that Airflow runs in, and that's the one that'll be run by default if you just launch an Airflow job without any configuration.
• 16:24
  SPEAKER_02: And then if you're an outlier and you need another Python library, for example, you use a Python virtual env operator.
• 16:30
  SPEAKER_02: Or if you need like more hardcore dependencies, you'll run a Kubernetes spot operator with your own Docker image.
• 16:36
  SPEAKER_02: And that's basically how we do it.
• 16:38
  SPEAKER_02: I know some people will run every job has its Docker image and its Docker file inside the repo, but I find that that's a little bit too heavy for us.
• 16:47
  SPEAKER_02: We like to keep things simple and iterate quite often and make a lot of jobs.
• 16:51
  SPEAKER_02: So this is what we've settled on.
• 16:53
  SPEAKER_00: I think that makes a ton of sense.
• 16:54
  SPEAKER_00: I've heard of lots of teams narrowing the available operators to their teams, usually Python operator or Kubernetes Pod operator are the ones I hear of the most often that teams say you have to use this.
• 17:09
  SPEAKER_00: And I think exactly the reasons that you've said like it makes it easier for people to engage with, especially if you've already managed the dependencies for them.
• 17:18
  SPEAKER_00: So they don't have to even go into that foray of I always feel like that's a big rabbit hole of trying to get dependencies a lot of times.
• 17:26
  SPEAKER_00: And also I think it makes it easier on the teams maintaining airflow because you have a smaller possible window of the things you would have to review further that they run.
• 17:35
  SPEAKER_02: Yeah.
• 17:35
  SPEAKER_02: It's true.
• 17:36
  SPEAKER_02: And even for testing, I think it makes it much easier if you have your Python script or whatever, you can execute it locally and you can test that out.
• 17:42
  SPEAKER_02: And then all you do is put it inside a Python operator.
• 17:44
  SPEAKER_02: Whereas when you go into an operator, how do you test this?
• 17:47
  SPEAKER_02: You have to have separate code that tests it, or you have to use the airflow test commands on the CLI, which not everyone is familiar with.
• 17:55
  SPEAKER_02: I think that's a fairly good point.
• 17:57
  SPEAKER_02: But we still need to rely on some of the stuff Airflow uses.
• 18:00
  SPEAKER_02: And for that, we have another library that has only custom pure Airflow objects, so sensors, hooks, operators.
• 18:07
  SPEAKER_02: And that one's allowed us to like add in a few operators that maybe we wanted that didn't really exist out of the box, but we didn't want everyone to keep recoding them as Python code, because then you end up with a lot of boilerplate of everyone recoding the same operator.
• 18:21
  SPEAKER_02: And this library actually we've had to install as a plugin because uh one of the operators is uh external trigger operator, but that works across instances.
• 18:29
  SPEAKER_02: So it has a little bit of an effect on the UI.
• 18:32
  SPEAKER_02: And we found that installing a library as a plugin is incredibly simple, like deceptively simple.
• 18:38
  SPEAKER_02: All you had to do was add like a an entry point on the Py Project's Tommo file, and you were done basically.
• 18:45
  SPEAKER_02: As long as it's installed on the Airflow instance, it works.
• 18:48
  SPEAKER_00: That's great.
• 18:48
  SPEAKER_00: I'm eagerly awaiting Airflow 3.1 when plugins will be the interface has already been added back into the project with the new UI for Airflow 3, which is really exciting.
• 18:57
  SPEAKER_00: But I am kind of blown away always at how easy plugins can be, just how extensible it makes Airflow.
• 19:04
  SPEAKER_00: And so in your case, that UI component piece is that used by like an end user to trigger something in the external environment, or how does that work?
• 19:13
  SPEAKER_02: No, it's more like you have multiple instances and you're triggering from one job in one instance, you're triggering another job or another task in another instance.
• 19:21
  SPEAKER_02: And then, you know, if you're like a let's say a marketer or project manager, you're not going to go inside the code and then look at what's the URL of the instance and how do I know.
• 19:30
  SPEAKER_02: No, so you have just a neat little button, you click on it, it takes you to the job on the instance so that you can see that indeed it has launched and then it's running well.
• 19:37
  SPEAKER_02: And I don't know, it's a neat little trick.
• 19:39
  SPEAKER_02: And it was already built in there to the external trigger operator.
• 19:42
  SPEAKER_02: All we did was port it to an operator that works across instances instead of on the same instance.
• 19:47
  SPEAKER_00: Yeah, that's really cool.
• 19:48
  SPEAKER_00: Like the extra links you can put into operators, but more for like your whole Airflow instance.
• 19:53
  SPEAKER_00: That's great.
• 19:54
  SPEAKER_02: Yeah, it's simple, it's stupid, but it's very nice looking.
• 19:57
  SPEAKER_00: Yeah, I'm sure those simple things are the ones that everybody always appreciates the most, right?
• 20:02
  SPEAKER_02: Yeah, exactly.
• 20:02
  SPEAKER_00: Yeah, okay.
• 20:03
  SPEAKER_00: Well, that's really cool.
• 20:04
  SPEAKER_00: I think that's a good time to transition.
• 20:07
  SPEAKER_00: This has been a really insightful look into how you've used Airflow on-prem.
• 20:11
  SPEAKER_00: I love hearing stories like this where it's been so customized to fit your particular use case.
• 20:17
  SPEAKER_00: Speaking of, you've mentioned a little bit about kind of looking into Airflow 3 and some of the new features there, maybe related to that or maybe not, but I always end with a standard final question, and that is what would you most like to see from development of the Airflow project in the near future?
• 20:32
  SPEAKER_02: Oh, that's a big one.
• 20:34
  SPEAKER_02: Honestly, it's hard to have a direct answer to this, but I think because we're such huge fans of the UI, maybe the things will be that are like UI focused.
• 20:44
  SPEAKER_02: I think maybe one of them is so because we have such a big Airflow setup with different instances, different dependencies, and everything, when we migrate from let's say Airflow two to Airflow three, the migration setup is also quite long.
• 20:58
  SPEAKER_02: And then we have to pause DAGs on a specific instance and then transfer them to the other one, make sure they work, and then we cancel them.
• 21:05
  SPEAKER_02: And so that process of checking which DAGs have been migrated usually has us running custom scripts that will tell you, okay, these are the jobs on this instance that you've paused, these are the jobs on this instance that you've run successfully.
• 21:19
  SPEAKER_02: Uh and one of my colleagues said that if we could have a little button that gives you an that exports a snapshot of the current state of an instance, like the last state every job was triggered, which jobs are paused, which jobs aren't, that would help with that.
• 21:32
  SPEAKER_02: I'm not 100% sure if that would be helpful for everyone, but internally I think that would be interesting.
• 21:37
  SPEAKER_02: And another one that we've had is a little bit more technical this time.
• 21:41
  SPEAKER_02: You can use different backends for the connections, and we internally we use Vault, but we found that if you use the Vault backend for your connections, you they don't behave exactly the same way as connections that are hosted directly on the Airflow database.
• 21:54
  SPEAKER_02: Notably, I don't think they show up in the connections tab.
• 21:58
  SPEAKER_02: And I think that was a little bit frustrating for our infrastructure guys who were hoping that you could see them, but they'd be maybe grayed out with a read-only look and maybe a little sticker that says, Oh, this was important from Volts, this was important from Bitwarden, this was whatever.
• 22:13
  SPEAKER_02: Yeah, I think it's mostly this that kind of UI thing that makes it makes it a little bit more unified.
• 22:18
  SPEAKER_02: And but honestly, I feel like we're nitpicking here because we don't have too many bad things to say.
• 22:24
  SPEAKER_00: That's a good place to be.
• 22:25
  SPEAKER_00: That's where we want the requests to come in.
• 22:27
  SPEAKER_00: I actually really like connections one.
• 22:29
  SPEAKER_00: I think that would also be super helpful.
• 22:31
  SPEAKER_00: I have myself encountered this personally and also talked to many people who have unknowingly defined connections both in the UI and in an external back end, and then they're not even sure which one is being used.
• 22:44
  SPEAKER_00: And I have to go look at our docs every time to figure out what's more recent, why we don't have a single source of truth, you know.
• 22:51
  SPEAKER_00: Exactly.
• 22:51
  SPEAKER_00: I'm just not being able to, it's like it's defined.
• 22:54
  SPEAKER_00: You but you have no idea how I agree.
• 22:56
  SPEAKER_00: Having that visibility would be really cool.
• 22:58
  SPEAKER_00: I have no idea how feasible that is for the airflow project, but maybe.
• 23:03
  SPEAKER_00: Yeah, yeah.
• 23:04
  SPEAKER_02: Hopefully, someone will hear this.
• 23:05
  SPEAKER_00: Exactly.
• 23:06
  SPEAKER_00: I'll always forward these to our Airflow engineers, or we'll make sure to pull it out into the Airflow survey this year.
• 23:12
  SPEAKER_00: So nice.
• 23:13
  SPEAKER_00: So somebody sees it.
• 23:14
  SPEAKER_00: Okay.
• 23:14
  SPEAKER_00: I love both of those.
• 23:16
  SPEAKER_00: Sebastian, I won't keep you any longer.
• 23:18
  SPEAKER_00: Thank you so much for joining me.
• 23:19
  SPEAKER_00: It has been really great chatting with you.
• 23:21
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 23:24
  SPEAKER_02: Thank you for having me.
• 23:25
  SPEAKER_02: The best way for people to get in touch with me is definitely LinkedIn.
• 23:28
  SPEAKER_02: If they want to do it online, or if they want to do it in person, they can meet me at PyCon Taiwan in September or the Airflow Summit in October.
• 23:35
  SPEAKER_00: Oh, that's great.
• 23:36
  SPEAKER_00: Oh, I don't think I realized you were coming to Airflow Summit.
• 23:39
  SPEAKER_00: We can talk more about how excited I am for everyone to come to Seattle for Airflow Summit.
• 23:43
  SPEAKER_00: People are gonna get sick of hearing me say this.
• 23:45
  SPEAKER_00: That's great.
• 23:46
  SPEAKER_00: I will hopefully get to meet you in person there.
• 23:48
  SPEAKER_00: I'm not sadly going to PyCon, Taiwan, but hopefully some of our listeners are.
• 23:52
  SPEAKER_00: That sounds like fun.
• 23:53
  SPEAKER_00: Okay.
• 23:54
  SPEAKER_00: Well, thank you again for joining us.
• 23:55
  SPEAKER_00: It's been great.
• 23:56
  SPEAKER_02: Thank you for having me.
• 24:00
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:06
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:10
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.