Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler

Transcript

Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: And then I was really astonished how many people were jumping on a wagon and were also pushing this because it seems to be it was really some piece in Airflow missing.
• 0:09
  SPEAKER_02: And we tried to push it and develop it that we could use it in Airflow 2.
• 0:14
  SPEAKER_02: And it was a also leading factor to be officially supported in being GA in Airflow 3.
• 0:20
  SPEAKER_02: And then we can go a bit more to details later on this.
• 0:22
  SPEAKER_02: So actually the endeavor started earlier, but it contributed very much to the topic of Airflow 3 as a major release.
• 0:44
  SPEAKER_01: Hey Airflow community, exciting news.
• 0:47
  SPEAKER_01: This fall we're launching Beyond Analytics, a virtual data ops conference happening September 16th from 12 to 3 p.m.
• 0:54
  SPEAKER_01: Eastern.
• 0:55
  SPEAKER_01: You'll hear from data leaders on how modern data ops and orchestration are empowering AI, machine learning, and production grade data products.
• 1:02
  SPEAKER_01: And here's the best part.
• 1:03
  SPEAKER_01: You can join Mark Lumberti for a fast track Airflow 3 certification prep session and get a discount code for free certification, a$150 value.
• 1:12
  SPEAKER_01: The conference is virtual, it's free, and it's made just for you.
• 1:16
  SPEAKER_01: You can register now at astronomer.io slash beyond slash data flowcast.
• 1:24
  SPEAKER_01: Okay, hi everyone.
• 1:26
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:27
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm joined by Jens Scheffler, who is a technical architect of digital testing automation at Bosch.
• 1:35
  SPEAKER_01: Jens, how are you?
• 1:37
  SPEAKER_02: Good.
• 1:37
  SPEAKER_02: Yeah, so nice, cozy weather, and the summer is coming.
• 1:41
  SPEAKER_02: Maybe the same like in Seattle and looking forward to meeting in person then end of the year.
• 1:46
  SPEAKER_01: Yeah, yeah.
• 1:47
  SPEAKER_01: It is definitely summer here in Seattle.
• 1:49
  SPEAKER_01: It's pretty warm today.
• 1:51
  SPEAKER_01: And yeah, we're getting some good weather finally.
• 1:53
  SPEAKER_01: Seattle summers tend to start late and end late, which I hope means in your reference to the Airflow Summit that we'll still have good weather in early October.
• 2:01
  SPEAKER_01: So I'm very excited to see people in person there as well.
• 2:04
  SPEAKER_02: We'll promise good weather in October.
• 2:06
  SPEAKER_02: So let's see how it comes here.
• 2:08
  SPEAKER_01: I have promised so many people that good weather, honestly, and I'm gonna be really sad if it rains, which it might, but that's okay.
• 2:15
  SPEAKER_01: It'll still be a very beautiful time of year, regardless.
• 2:18
  SPEAKER_01: Yeah.
• 2:19
  SPEAKER_01: Well, speaking of Airflow 7, I think there will be a lot of discussion there about Airflow 3 and one of our topics today.
• 2:26
  SPEAKER_01: So let's go ahead and dive in.
• 2:28
  SPEAKER_01: For today's discussion, we're going to focus on the new remote execution functionality within Airflow 3, which is very exciting.
• 2:36
  SPEAKER_01: But first, for folks who might not know you, let's start with some background.
• 2:39
  SPEAKER_01: So, can you tell me a little bit more about what you do at Bosch and how you're involved in the open source AirFloor project?
• 2:46
  SPEAKER_02: At Bosch, we had a podcast earlier, which is maybe worth listening to.
• 2:51
  SPEAKER_02: We are doing automated driving, these other business we also do as Bosch, but my department is in the automated autonomous driving.
• 2:59
  SPEAKER_02: And I'm belonging to the part of the team which is doing the system-based testing.
• 3:04
  SPEAKER_02: So we test the software stack, not like a unit test or Py test, but the real integration of the ADUS functionality for the driving.
• 3:13
  SPEAKER_02: And since a long time we are using Airflow as an administrator to drive the test into our test infrastructure, at least we started with Airflow 1.10.
• 3:23
  SPEAKER_02: And as we used it, we first patched it a bit around, and then we got a bit more involved and we supplied a few kind of we upstream contributed a bit of patches.
• 3:33
  SPEAKER_02: And after some patches and bug fix, we also made some contributions, and that's also how it was dragged into the community of Airflow as being an active contributor and nowadays a PMC member to really bring features into Airflow, which actually is just helping us in making our life better.
• 3:51
  SPEAKER_02: So to contribute back to the community that we can use it on our own.
• 3:54
  SPEAKER_02: And yeah, the remote execution is actually one of those examples.
• 3:58
  SPEAKER_01: Yeah, that's great.
• 3:59
  SPEAKER_01: Thanks for that background.
• 4:00
  SPEAKER_01: We'll throw the link to your previous episode with the show in the show notes.
• 4:05
  SPEAKER_01: So if anybody's interested in learning more about your work at Bosch, that will be helpful.
• 4:10
  SPEAKER_01: Um, I love to hear that you're able to spend some time contributing back to the open source project with your company.
• 4:16
  SPEAKER_01: Of course, at astronomer, we do that a lot.
• 4:18
  SPEAKER_01: And I think it's such a mutually beneficial thing for everybody who uses the project.
• 4:22
  SPEAKER_01: So I love to hear that.
• 4:24
  SPEAKER_01: And as you mentioned, so you were very involved with the development of Airflow 3 over the past year or so.
• 4:31
  SPEAKER_01: Um, it was released, of course, in April, and we're particularly focused on this AIP69, which we've talked about remote execution already.
• 4:41
  SPEAKER_01: Also, the AIP was to add the new edge executor.
• 4:44
  SPEAKER_01: So tell me more really about this AIP.
• 4:47
  SPEAKER_01: What is the edge executor?
• 4:48
  SPEAKER_01: And what is remote execution, this thing that we keep talking about.
• 4:52
  SPEAKER_02: Yeah, so remote execution means like the term means it's not in the data center, but we also inside the Airflow community had a bit of a debate about the term because initially this Airflow improvement proposal, EAP69, as it was called remote execution.
• 5:08
  SPEAKER_02: And actually, I proposed that we have an option for remote execution.
• 5:13
  SPEAKER_02: And then the debate started that actually the term remote was already being used in the documentation because there's this inline inside the airflow, the platform execution when the scheduler is a component really executes also something like the local executor.
• 5:26
  SPEAKER_02: And the also very commonly used salary executor was already listed as being a remote executor because the salary cluster in theory could be outside of the Airflow ecosystem.
• 5:36
  SPEAKER_02: So the name was a bit clashing, and that was leading to the fact that actually finally we called it edge execution or edge worker, which also reflects the same term, but is a bit more precise to what remote actually means, because remote could be still a server in the same data center.
• 5:53
  SPEAKER_02: For our case, it is rather the way that we want to run tasks outside of the central data center.
• 5:59
  SPEAKER_02: As we are executing the main workload in the cloud, and we use all the fancy stuff in the cloud to scale up and scale down and orchestrate in the cloud.
• 6:08
  SPEAKER_02: But as we do autonomous driving, we have the problem that we also sometimes we need to get to our physical devices, like the units that are deployed into the vehicle and in the system integration, we need to deploy there.
• 6:20
  SPEAKER_02: And for us, the term remote means we also want to run jobs there.
• 6:25
  SPEAKER_02: I believe that was the driving factor.
• 6:27
  SPEAKER_02: But yeah, actually, this Airflow improvement proposal was issued before Airflow 3 started implementing, and we came to the point that we wanted to have that because we wanted to use the orchestrator for a seamless orchestration across the environment.
• 6:44
  SPEAKER_02: And actually, this improvement proposal was being dropped by me a bit naive to say.
• 6:52
  SPEAKER_02: There was not a technical concept behind the scene, it was just maybe developed saying there's a demand for this remote execution.
• 6:59
  SPEAKER_02: And then I was really astonished how many people were jumping on the wagon and were also pushing this because it seems to be it was really some piece in Airflow missing.
• 7:08
  SPEAKER_02: And we tried to push it and develop it that we could use it in Airflow 2, and it was also leading factor to be officially supported and being GA in Airflow 3.
• 7:20
  SPEAKER_02: And we can go a bit more to details later on this.
• 7:22
  SPEAKER_02: So actually, the endeavor started earlier, and yeah, but it contributed very much to the topic of Airflow 3 as a major release.
• 7:30
  SPEAKER_01: Makes sense.
• 7:30
  SPEAKER_01: So long, long time in the making.
• 7:32
  SPEAKER_01: That is very helpful background.
• 7:33
  SPEAKER_01: I actually didn't know that about the naming, but that totally makes sense that there's remote and then there's remote.
• 7:40
  SPEAKER_01: The other remote.
• 7:42
  SPEAKER_01: Yeah, very remote.
• 7:43
  SPEAKER_01: Great.
• 7:43
  SPEAKER_01: So with the new edge executor, and it sounds like in your motivation for proposing this, at least conceptually in the beginning, was from your work at Bosch, but as you said, like there was a lot of community buy-in from the beginning.
• 7:56
  SPEAKER_01: And I think this makes a lot of sense to me that it's something that I have heard a lot in my time working with folks who have used Airflow is this need to be able to run tasks outside of the infrastructure where your Airflow lives, not just on a you know, celery queue or in a Kubernetes pod.
• 8:16
  SPEAKER_01: But one of the things that my team has worked with folks to implement is like running with the Kubernetes pod operator in a different Kubernetes cluster or something like that as an alternative.
• 8:26
  SPEAKER_01: But it was quite difficult to do, a little bit hacky at times.
• 8:30
  SPEAKER_01: So I think a lot of companies, especially in sensitive industry or working with sensitive data or highly regulated industries, have that need.
• 8:38
  SPEAKER_01: So it sounds like there was a lot of buy-in on this from the beginning.
• 8:41
  SPEAKER_02: Yeah, and I can just underline this kind of it was the same for us.
• 8:45
  SPEAKER_02: For us, we have a corporate network and a corporate environment.
• 8:48
  SPEAKER_02: And from network point of security, this is very segregated and cut off the internet.
• 8:52
  SPEAKER_02: You need to go through chains of proxies to get to the internet.
• 8:56
  SPEAKER_02: So usually getting to the cloud means it can only be HTTP traffic and only outbound traffic, which then restricts a bit to get something linked together.
• 9:05
  SPEAKER_02: And the workaround also was discussed to say let's make a deployment on-prem and a deployment in the cloud, and then orchestrate between the deployments.
• 9:13
  SPEAKER_02: But yeah, actually, we didn't like that.
• 9:16
  SPEAKER_02: And so initially we tried to compensate.
• 9:19
  SPEAKER_02: Also, we used or attempted to use salary as an executor, and just maybe with an extension core to say you could spawn a VPN, but yeah, VPN was also not allowed from corporate regulations.
• 9:32
  SPEAKER_02: So we try to bring everything from the protocols to be HTTP.
• 9:36
  SPEAKER_02: And in Airflow 2, this means also the worker running somewhere far away from the central instance needs to have database connectivity.
• 9:44
  SPEAKER_02: That was the problem of Airflow 2 in the past.
• 9:47
  SPEAKER_02: And also salary depends on the queue to attach to, which is not really internet friendly.
• 9:52
  SPEAKER_02: And if the wire from the long distance breaks apart, that really affects the productivity and the production.
• 10:00
  SPEAKER_02: So we had many breakdowns in our remote data side, and it was negatively impacting the operations, and people were unhappy.
• 10:08
  SPEAKER_02: And that was one of the major drivers to say we need to have a remote working capability that runs standard internet protocols, HTTP.
• 10:16
  SPEAKER_02: You can pass security barriers and yeah, preventive proxies to get your work being scheduled.
• 10:23
  SPEAKER_02: And we needed this, and I was also a bit envy about how CI CD pipelines do it, because if you take a look to how GitHub or maybe Azure DevOps do it, there you can download a package and deploy that on an on-prem, whatever specific machine, and you can link your specific machine to the CI CD pipeline to run a build, to run a test.
• 10:44
  SPEAKER_02: And I thought that's also something that Airflow needs: that you can deploy a small package, less dependencies, and make that running on site so that individual jobs and tasks can be executed in in there from the remote.
• 10:57
  SPEAKER_01: Yeah, that makes sense.
• 10:58
  SPEAKER_01: A difficult thing to implement before this feature existed.
• 11:01
  SPEAKER_01: And I think that's a good segue into digging into some of the details for remote execution.
• 11:08
  SPEAKER_01: And I wanted to first talk about the implementation.
• 11:10
  SPEAKER_01: So you mentioned a little bit in the initial kind of AIP ideation that it was more just about the idea and less about the technical implementation.
• 11:19
  SPEAKER_01: I think this is the story of a lot of AIPs or a lot of even things that I have said of, hey, it would be great if Airflow did this.
• 11:26
  SPEAKER_01: Sounds really simple.
• 11:27
  SPEAKER_01: And this sounds like a particularly complex feature to implement.
• 11:31
  SPEAKER_01: Maybe can you describe at a high level just how it works?
• 11:34
  SPEAKER_01: And you led into this a little bit earlier, but why you needed to push this into Airflow 3, so into the major release as opposed to prior when the idea was first raised when we were still in Airflow 2.
• 11:45
  SPEAKER_02: And that's how I introduced it to say I was dropping that concept being a bit naive, and I thought it can be injected here or there into the tech stack.
• 11:54
  SPEAKER_02: And the initial impression I got from a couple of other contributors was, what are you trying to do?
• 11:59
  SPEAKER_02: That's impossible.
• 12:00
  SPEAKER_02: And I thought actually it's not too complex because the approach was that Airflow was building also in the 2.x chain with a modular concept, allowing multiple executors, as we had speak spoken about local execution, salary executor, Kubernetes executor.
• 12:16
  SPEAKER_02: So it was modular.
• 12:17
  SPEAKER_02: And I thought that's really the module where I would plug in the edge executor, or the remote executor to say.
• 12:23
  SPEAKER_02: And that's not really complex.
• 12:25
  SPEAKER_02: The executor is a modular component, it's running in the scheduler and it receives the tasks to be then scheduled and to be executed.
• 12:33
  SPEAKER_02: And so actually, that's not complex.
• 12:35
  SPEAKER_02: You get the task to be executed, it should run remote.
• 12:38
  SPEAKER_02: And actually, the technical solution is it's put to a queue, which is in this means a Postgres database table.
• 12:44
  SPEAKER_02: You just add an entry into the Postgres database table, and then the remote worker would use an API endpoint to pull from the table job and then to execute it.
• 12:54
  SPEAKER_02: In the normal salary executor, the task is being pushed via a queue.
• 12:58
  SPEAKER_02: In our edge work, as said it that goes via HTTP, the edge worker or the remote worker would pull from the queue and pull via an API.
• 13:07
  SPEAKER_02: And also, modular speaking, in Airflow, there's this plug-in concept so that you can extend web components and API endpoints.
• 13:15
  SPEAKER_02: So I believe on this level it was not too complex and a few hundred lines of code, and actually that was working.
• 13:22
  SPEAKER_02: The problem in Airflow 2 mainly was once you have the task on the edge and you want to execute it, and that was a bit naive, on my view, to say I thought we just need to instantiate the Python operator class in Python and run the execute method.
• 13:37
  SPEAKER_02: And I looked into the code where I can do it, and then I realized I need to pass the deck, and I need to have kind of the deck being passed to have the Python object, then to instantiate it and to make this.
• 13:48
  SPEAKER_02: The problem was the former task execution was very much like a hairball into the Python code.
• 13:54
  SPEAKER_02: It needed roughly 20 Python requests back and forth to the database to get all the context being loaded because you need to have the Python context, the deck run context, the task index context, you need to template the fields, you need to set up the logging and the task log handler.
• 14:10
  SPEAKER_02: And all of this was very tied into the database.
• 14:13
  SPEAKER_02: And luckily, in Airflow 2, there was an attempt for the multi-tenancy to encapsulate those database bindings with an API endpoint that was called the internal API.
• 14:24
  SPEAKER_02: And we tried to massage it in a way that the internal API was usable, and that's the way how it was able to be running in Airflow 2.
• 14:32
  SPEAKER_02: But it was also a learning curve for me and the community to see that this integration, cutting the hairball to an API, was not scalable.
• 14:42
  SPEAKER_02: And that was the foundation to the discussion: getting the task SDK as a clear API interface to unbundle the task execution from the core, which is a major concept, not only to make remote execution really productive, but also to cut the wires to be able to execute tasks in different languages, which is a major development.
• 15:01
  SPEAKER_02: And actually, this is really good that Airflow 3 brings this and this yeah really makes it usable in a way that you can happily run it in production.
• 15:12
  SPEAKER_02: Yeah, I would say we wired it in, and that's the way how we also made it working.
• 15:17
  SPEAKER_02: And actually, to be speaking, we are using it on Airflow 2 with this internal API, which then after we were cutting Airflow 3 was wiped off because it was not really good.
• 15:28
  SPEAKER_02: And we also need to go through the process of migrating to Airflow 3 to the task SDK, which makes it much better.
• 15:37
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 15:46
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 15:58
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:07
  SPEAKER_01: A lot of Airflow internals involved there.
• 16:10
  SPEAKER_01: And for those who are not aware, you just referenced that one of the big architectural changes in Airflow 3 was this new task SDK.
• 16:17
  SPEAKER_01: And so tasks no longer communicate directly with the metadata database.
• 16:22
  SPEAKER_01: There's a new API server, and that whole communication pathway has changed.
• 16:25
  SPEAKER_01: That's an interesting method, I guess, for working in remote execution.
• 16:30
  SPEAKER_01: So it seems like it worked out timing-wise with, as you said, a lot of unraveling the hairball of the internals.
• 16:36
  SPEAKER_01: That is one of the benefits of doing a new major version, is a lot of those breaking changes can be made.
• 16:42
  SPEAKER_01: So thanks for going into the details there.
• 16:45
  SPEAKER_01: I'm curious to hear your interpretation of working with the features.
• 16:50
  SPEAKER_01: So I know this is always a bit of a difficult thing to describe over audio, but how do users, if they have heard all of this and they think, oh, this is going to solve the problems that I've been having with Airflow, what do they need to do to use remote execution?
• 17:05
  SPEAKER_01: How do they set it up?
• 17:06
  SPEAKER_02: Yeah, that kind of like in Airflow, everything is modular.
• 17:10
  SPEAKER_02: This is a provider package.
• 17:11
  SPEAKER_02: It's not in the standard distribution, but you need to install the edge provider package.
• 17:16
  SPEAKER_02: And there's a fun story behind it.
• 17:18
  SPEAKER_02: It's the provider package is called Edge 3.
• 17:21
  SPEAKER_02: And the story behind why it's called Edge 3, we will reveal at Airflow Summit.
• 17:26
  SPEAKER_02: It's a bit of a fun story.
• 17:27
  SPEAKER_01: But I was on this dev call.
• 17:29
  SPEAKER_02: You need to install the edge three provider package and into your core and then configure it to be enabled.
• 17:36
  SPEAKER_02: So if the provider is installed, it's disabled per default for security purposes, not to open an API which is unwanted.
• 17:45
  SPEAKER_02: And you need to define a secret key to connect your remote workers to.
• 17:50
  SPEAKER_02: That would be maybe enhanced in the future to maybe have secret keys as a provisioning for multiple different endpoints, different secrets.
• 17:57
  SPEAKER_02: But at the moment, it is one secret, and you need to enable the API endpoint on your central side.
• 18:03
  SPEAKER_02: And on the edge side, you need to install the Airflow core and the provider package.
• 18:08
  SPEAKER_02: And then you run the worker as a standalone process.
• 18:12
  SPEAKER_02: So you can call it a command line.
• 18:15
  SPEAKER_02: And whoever deployed a salary worker is very similar.
• 18:18
  SPEAKER_02: Like in salary worker, you call Airflow salary workers and give it a command line.
• 18:23
  SPEAKER_02: On the edge, it's called you just type Airflow Edge worker, and then you define which queues it should listen to and how many jobs it should pull to execute.
• 18:33
  SPEAKER_02: So that's the basic deployment.
• 18:35
  SPEAKER_02: I'm looking forward that maybe we also supply a Docker Compose, that it's making it easier.
• 18:40
  SPEAKER_02: And also with Airflow 3, we maybe need to optimize it a bit to cut down the dependencies.
• 18:46
  SPEAKER_02: But the dependencies, if you just install in Airflow 3, the core and the provider package, and of course, you need to install the providers, you need to run your tasks.
• 18:55
  SPEAKER_02: If the decks are being synced, I believe it's a really lean setup that you need to deploy.
• 19:00
  SPEAKER_02: And I didn't test it actually on Airflow 3.
• 19:03
  SPEAKER_02: I need to repeat this, but on Airflow 2, we were also able to cut down the dependencies in a way that it is possible to run the task on a native Windows machine.
• 19:11
  SPEAKER_02: Because in the past it was always a bit limiting.
• 19:14
  SPEAKER_02: You always needed to run it in a Linux WSL container or you needed to run it on a Linux machine remote if you want to run a task on Windows.
• 19:22
  SPEAKER_02: But due to the cutdown dependencies, it's actually a bit light white, and you can deploy it then on your remote machine and run it.
• 19:29
  SPEAKER_02: And yeah, uh whoever used multiple executors, so you can use the edge executor as the only executor.
• 19:37
  SPEAKER_02: But I would propose to join it with the there's the it was formerly called hybrid executor.
• 19:42
  SPEAKER_02: You can run multiple executors and say everything that is running in my data center runs either on salary or Kubernetes executor, and only the tasks that need to run to the edge are running in the edge executor.
• 19:54
  SPEAKER_02: And then in your deck, you can mark every task that should run on the edge executor by labeling it which should run to the edge.
• 20:02
  SPEAKER_02: And besides defining which task should run on the edge executor also supports multiple cues.
• 20:07
  SPEAKER_02: So if you have a Windows machine, a Linux machine, whatever small, big machine in Michigan, and one in Calgary, you can then define cues so that you can drive the task where you need to have it.
• 20:19
  SPEAKER_02: And yeah, so actually for using it, it's not complex.
• 20:23
  SPEAKER_02: I believe the more complex thing is the operational part.
• 20:26
  SPEAKER_02: You have deployed the software, and if you have maybe multiple edge uh deployments, you need to maintain it, you need to monitor it.
• 20:33
  SPEAKER_02: And I believe there would be continuous improvement how to optimize the distributed operation as well.
• 20:39
  SPEAKER_02: So the edge executor brings also maintenance mode.
• 20:42
  SPEAKER_02: If a machine far away is broken and it's just producing errors from the central console, you can put it into maintenance mode.
• 20:49
  SPEAKER_02: You can even shut it down, and then somebody needs to run there and boot it up again.
• 20:54
  SPEAKER_02: So I would say the main endeavor is then an operational aspect, but getting it started is pretty easy and it could be continuously improved.
• 21:02
  SPEAKER_02: So looking forward also for community contributions, making it better and improving it, making it nicer.
• 21:07
  SPEAKER_01: That's great.
• 21:08
  SPEAKER_01: I think there's so much good stuff in there, especially I think the concept of having multiple remotes or multiple queues will be especially helpful for folks looking to use it for resource utilization optimization so that you're not setting all of your tasks to like a super big queue, or for those who are working with data that needs to stay in particular regions or things like that, being able to implement all of that within the same Airflow instance.
• 21:32
  SPEAKER_01: That's great.
• 21:33
  SPEAKER_01: I will throw in just for any folks who are listening who might be working with Airflow on Astronomer using Astro.
• 21:40
  SPEAKER_01: We have a similar implementation of this.
• 21:42
  SPEAKER_01: And so you can still use remote execution.
• 21:44
  SPEAKER_01: For us, it's called the Astro Executor.
• 21:46
  SPEAKER_01: It's kind of hard take and building on the edge executor, but same implementation where you assign that executor to your deployment and then you can set up multiple remotes if you need to, or I think on Astro, we've called it coordinating is the term.
• 22:00
  SPEAKER_01: For if you need to isolate one that's not working and stop sending new tasks to it, and that's all managed on the Airflow side gracefully, so that existing tasks will continue to run, and then it won't get any more if there are issues with that particular remote machine.
• 22:14
  SPEAKER_02: Yeah, and on Edge Executor, there will be a talk on the summit also that might be getting a bit more into detail.
• 22:19
  SPEAKER_02: Um, there was also a second talk, unfortunately, that was not accepted yet.
• 22:24
  SPEAKER_02: Looking forward, that maybe there's a spare slot.
• 22:26
  SPEAKER_02: Also, what we would like to share is the experience of having that now running in production for more than half a year, going through kind of the all the good and the bad things that you can do if you run the edge execution, the distributed operations, how do you monitor it, how do you see if it's liveness, how do you deploy the software, how do you maintain and upgrade the releases, how do you manage the cues?
• 22:50
  SPEAKER_02: I believe there's a lot of detail in setting this up on operational mode.
• 22:53
  SPEAKER_02: And yeah, I hope that if there's no talk on the summit, at least somebody is who is there would be able to uh talk to me directly, and then we can share experience also on having it in production since a long time and happily running in it.
• 23:08
  SPEAKER_02: And yeah, what I also need to share was it didn't have that in the plan yet.
• 23:13
  SPEAKER_02: But last year, when the PUC was done, last year at the summit, I pitched it and I was taking the attempt at that time.
• 23:19
  SPEAKER_02: The software, the edge executor, was not on main, but the pull request was in review.
• 23:24
  SPEAKER_02: And on the flight back from San Francisco to Germany, I was starting my laptop in the airplane on wireless in the airplane, and I started the edge worker on the airplane, and I started a DAG on my home server, and I was able to have also a task distributed across the wireless into the airplane and executed a Python operator in 10,000 meters of altitude.
• 23:47
  SPEAKER_02: Long distance needs to be challenged and looking forward that maybe someday the remote execution is running on the International Space Station or so.
• 23:56
  SPEAKER_02: So if somebody has really a long distance looking forward for the longest distance being bridged, we can be extra, extra remote.
• 24:03
  SPEAKER_01: I love that one.
• 24:04
  SPEAKER_01: I hope that talk is accepted at the summit.
• 24:06
  SPEAKER_01: I think that would be a great story.
• 24:08
  SPEAKER_01: I would love to hear more about that.
• 24:10
  SPEAKER_01: And what a great use case that, as you said, you've already been using it successfully in production for quite some time.
• 24:16
  SPEAKER_01: I'll throw out there as well as another resource.
• 24:18
  SPEAKER_01: I'll put a link to this in the show notes, but at least at the time we're recording later this week, we'll have a webinar on our side talking about a remote execution in Airflow 3 specifically and how you set it up.
• 24:29
  SPEAKER_01: We've got a great demo that might be easier to see some of the stuff that we've just talked about, as opposed to just listening here.
• 24:35
  SPEAKER_01: I think by the time this episode comes out, it will be a recording, but folks will still be able to access that for free.
• 24:41
  SPEAKER_01: So we'll throw that in as an extra resource.
• 24:44
  SPEAKER_01: But Jens, this is, I think, one of the most exciting features in my mind about within Airflow 3, even though there's a lot that could fit in that category.
• 24:52
  SPEAKER_01: So I appreciate you taking the time to, of course, both work on the development of the feature and then to come on and talk about it with us.
• 24:59
  SPEAKER_01: I think since we're about at the end of our time, I always like to ask folks a standard closing question, and that is what would you most like to see from the Airflow project in the near future?
• 25:08
  SPEAKER_01: It does not have to be the improvements to remote execution, although it could be.
• 25:12
  SPEAKER_02: I hope there are many to come.
• 25:13
  SPEAKER_02: And actually, if it would have been a recording two months ago, I would have said I would like to have a human in the loop also.
• 25:20
  SPEAKER_02: This is in implementation, looking forward for the next show, maybe highlighting this, and it was also dropping there and contributing to the AP.
• 25:27
  SPEAKER_02: What at the moment is a challenge for me is kind of, and I would wish that there's more support and looking forward, maybe also contributing to this.
• 25:35
  SPEAKER_02: And my colleague Marco is working on it.
• 25:37
  SPEAKER_02: Is at the moment we have the challenge we want to do testing really at a large scale.
• 25:42
  SPEAKER_02: And we so far in the past used the dynamic map task, which is really cool to map out the volume dynamically, but there's a soft limit of thousand per default.
• 25:52
  SPEAKER_02: And we thought, yeah, it's a soft limit, so just extended to 2000, 4000, and we realized it's not linearly scaling, so it's actually hitering some boundaries, and we have concrete demands that kind of people are approaching us to say we want to run a 10,000 tests or a hundred thousand tests, and we now have the challenge of really a large test suite to be executed.
• 26:15
  SPEAKER_02: And yeah, we want to still have a logical workflow, but we know it's not possible having a deck with a hundred thousand tasks today.
• 26:22
  SPEAKER_02: I would be looking forward if somebody has an idea, and if not, then probably it's something maybe we contribute in the future.
• 26:29
  SPEAKER_02: How to do really scale out workflow in complexity.
• 26:33
  SPEAKER_02: So yeah.
• 26:34
  SPEAKER_01: Yeah, that works.
• 26:35
  SPEAKER_01: Yeah, that would be very impactful.
• 26:37
  SPEAKER_01: I was scalability on dynamic task mapping.
• 26:39
  SPEAKER_01: I love that answer.
• 26:40
  SPEAKER_01: That's a good re a good additional answer to human in the loop, which as you said, we're very excited about and is coming soon.
• 26:46
  SPEAKER_01: I feel like this is a I don't intend it to be this way, but this seems like always a tricky question for folks to answer because I think they come up with something and then realize that it's already in development.
• 26:56
  SPEAKER_01: So the Airflow project is moving really fast.
• 26:59
  SPEAKER_01: Great.
• 26:59
  SPEAKER_01: That's all good stuff to look forward to.
• 27:01
  SPEAKER_01: Jans, it's been really great having you on.
• 27:03
  SPEAKER_01: To close out, what is the best way for folks to get in touch with you?
• 27:07
  SPEAKER_02: Yeah, so I I would say the usual way is LinkedIn, and probably you can drop the link.
• 27:11
  SPEAKER_02: Otherwise, I'm pretty active in Slack.
• 27:14
  SPEAKER_02: And yeah, last not least, I really hope to see most of the other folks and contributors in Seattle in the golden October, enjoying the kind of fall, the golden fall and the nice weather, and also meeting up and talking about yeah, directly in Seattle.
• 27:27
  SPEAKER_01: Absolutely.
• 27:28
  SPEAKER_01: I'm just gonna keep promoing it, but come meet Jens at Airflow Summit.
• 27:31
  SPEAKER_01: Come see me at Airflow Summit here in my hometown.
• 27:34
  SPEAKER_01: We'll keep, yeah, plugging away at Seattle's a great place to visit.
• 27:38
  SPEAKER_01: So we hope to see everybody there.
• 27:40
  SPEAKER_01: Okay, thanks so much, Jens.
• 27:41
  SPEAKER_01: This has been great.
• 27:44
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 27:51
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 27:55
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.