Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling

Transcript

Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: Normally we have around, I would say, between 700 and thousand CI runs per day, for example, where we have to run 20 minutes tests, 700 or 1000 times.
• 0:12
  SPEAKER_03: And for this big release test, we need to run around 10,000 of hours in two weeks.
• 0:20
  SPEAKER_01: And getting to this number, I believe one of the critical factors why we selected Airflow was also that if a test batch is maybe running for two weeks, we know that in between the two weeks something will break apart, or we need to deploy a patch into the infrastructure, we need to reboot the servers.
• 0:36
  SPEAKER_01: And with Airflow, we were sure that all the elements could be restarted and still driving the tests.
• 1:13
  SPEAKER_02: How are you doing, guys?
• 1:14
  SPEAKER_02: I'm very excited to have you in this episode.
• 1:17
  SPEAKER_03: Yeah, we are doing well.
• 1:19
  SPEAKER_03: Speaking for Jans as well, thank you for having us here.
• 1:22
  SPEAKER_03: So we're very excited to talk with you today.
• 1:26
  SPEAKER_02: Awesome.
• 1:27
  SPEAKER_02: Maybe I think so.
• 1:29
  SPEAKER_02: Before diving into this episode, I just would like to give a quick overview of what you should expect from this episode.
• 1:35
  SPEAKER_02: So for today's podcast, we will discuss how Bosch leverages Airflow to orchestrate complex testing scenarios for auto, for autonomous and driver assistance systems, scaling to millions of tests across both cloud and physical hardware to ensure vehicle safety before deployment on real road.
• 1:56
  SPEAKER_02: Honestly, I'm very excited about the topic because that's the first time I hear about using Airflow for that.
• 2:01
  SPEAKER_02: So I can't wait for you guys to tell me how you use Airflow with that use case.
• 2:06
  SPEAKER_02: But maybe Christian, starting with you, can you introduce yourself?
• 2:09
  SPEAKER_02: What do you do at Robot Bosch?
• 2:12
  SPEAKER_03: Yeah, of course.
• 2:13
  SPEAKER_03: Yeah, I'm Chris.
• 2:15
  SPEAKER_03: It's simple and short, it's also good.
• 2:18
  SPEAKER_03: And yeah, I'm working meanwhile since 12 years for Robert Bosch.
• 2:23
  SPEAKER_03: And since eight years, I have experience with Airflow, using Airflow in the context of automated driving and other systems.
• 2:33
  SPEAKER_03: I used it previously for data engineering and for a learning loop for machine learning models.
• 2:41
  SPEAKER_03: And so yeah, meanwhile, I am the product owner for the digital testing system we are presenting today.
• 2:48
  SPEAKER_03: And we're using Airflow here for automating all these tests that are needed around other systems.
• 2:56
  SPEAKER_02: Okay.
• 2:57
  SPEAKER_01: And yeah, I don't have such a long track record, but still, I believe pretty long.
• 3:03
  SPEAKER_01: So my career with Airflow started with 1.10.
• 3:06
  SPEAKER_01: And before joining Robert Bosch, I was working in the IT industry for building up large-scale complex cluster systems and solutions.
• 3:15
  SPEAKER_01: And from that, I bring in uh technical architect experience in how to bring up not a big but a large-scale system which is distributed.
• 3:25
  SPEAKER_01: And yeah, I joined Robert Bosch for especially a scaled-out solution for driving test load across big clusters.
• 3:32
  SPEAKER_01: And initially the decision went into Kubernetes as a workhorse for distributing the jobs.
• 3:39
  SPEAKER_01: And we saw that yeah, Kubernetes is very good for running the jobs, but for doing a workflow end-to-end, that brought us together and also meet to Airflow.
• 3:49
  SPEAKER_01: Yeah, and so that was the way starting with the challenges, and also brought me in from the point of being a user deploying that to being an active contributor in the community, also being very active in the community.
• 4:04
  SPEAKER_02: That's awesome.
• 4:05
  SPEAKER_02: That's awesome.
• 4:06
  SPEAKER_02: So I guess can you introduce us to um your team's work at Bosch and explain how you are using Apache Airflow for testing automated driving systems?
• 4:16
  SPEAKER_02: Because again, like this is a use case that I've never heard before.
• 4:22
  SPEAKER_03: Yeah, it's also not common use case, I would say, for Airflow here.
• 4:26
  SPEAKER_03: Yeah, let me maybe start with the introduction here of our team.
• 4:30
  SPEAKER_03: Yeah, as I mentioned already, our team is working in the ADAS domain.
• 4:34
  SPEAKER_03: ADAS means advanced driver systems.
• 4:38
  SPEAKER_03: And we are talking here about assistance systems in a car like cruise control or lane change assist and the combination of both, for example.
• 4:50
  SPEAKER_03: And in the levels above about autonomous driving, where no driver is needed anymore, basically, and the car can run on its own.
• 5:00
  SPEAKER_03: And um Robert Bosch is a tier one provider for the actual manufacturers of vehicles.
• 5:09
  SPEAKER_03: This means Robert Bosch is delivering software and hardware components.
• 5:16
  SPEAKER_03: And in the ADAS domain, these software components, which are basically the functional code for the car going autonomously.
• 5:25
  SPEAKER_03: What we are providing as a team is a framework for testing these components.
• 5:31
  SPEAKER_03: This means our framework is able to do a closed loop simulation or open loop simulation.
• 5:38
  SPEAKER_03: The difference is in the closed loop simulation, generated data is replayed with the latest software.
• 5:46
  SPEAKER_03: And in the open loop case, it is uh recorded data from the field.
• 5:52
  SPEAKER_03: And the important thing is this is really with the complete system of the car, software system of the car, and not only with some components.
• 6:02
  SPEAKER_03: So we are not talking here about unit tests or pie tests, but really big uh big tests.
• 6:08
  SPEAKER_03: And um, yeah, for this, we need a lot of data, we need a lot of um resources, power of infrastructure to do these tests, and we are using Airflow here to automate and scale all these tests.
• 6:25
  SPEAKER_03: This means so in the end, we're talking about hundreds and thousands of hours of data we need to replay.
• 6:33
  SPEAKER_03: And yeah, basically, we needed a software tool being able to orchestrate this amount of data and to orchestrate all these different parameters around it.
• 6:45
  SPEAKER_03: And yeah, in the end, the goal, the overall goal is to reduce the costs because when we can run it with airflow in in a virtual environment, we don't need to go to the car each time we have a change in the software, and so we reduce times, loop times of doing bug fixes, for example, and with that we reduce the overall costs.
• 7:09
  SPEAKER_03: Let me give some use cases also as examples here, what these tests are actually.
• 7:17
  SPEAKER_03: For example, we are talking here about data analysis tests, so we have new data from a car, running KPI evaluations on it and reporting them to a reporting engine in the project.
• 7:29
  SPEAKER_03: We integrate our airflow automation also in the CI pipeline, so for the software integration overall to do these tests.
• 7:39
  SPEAKER_03: We have the possibilities to provide airflow to all the developers so that they can also implement some special tests, which are not, let's say, these usual data analysis or integration tests.
• 7:54
  SPEAKER_03: And in the end, last but not least, and this is what we are really using Airflow a lot for, is these large regression test sets, which are then around 10,000 of hours of data we need to run in only a few days to say, okay, we can release the software or we can't, because there are too many failures in the software, for example.
• 8:18
  SPEAKER_02: This is super interesting.
• 8:19
  SPEAKER_02: I have so many questions already.
• 8:21
  SPEAKER_02: So just to make sure that everyone understands that correctly.
• 8:24
  SPEAKER_02: So when you are referring a test, you don't mean like a pie test or unit test, right?
• 8:30
  SPEAKER_02: Like you mean a test in uh, for example, I'm driving my car and there is a dog in front of me.
• 8:37
  SPEAKER_02: Then what should what the car should do, maybe?
• 8:40
  SPEAKER_02: Is that the kind of test you are you are doing with airflow just to make sure that the system is doing things correctly?
• 8:48
  SPEAKER_03: Exactly.
• 8:49
  SPEAKER_03: Yeah, so these data sets we're using have different scenes, like for example, a duck or elephant, maybe not an elephant in some in this western countries.
• 8:59
  SPEAKER_03: We have but having this in front of a car, and there is an expectation how the car should react.
• 9:06
  SPEAKER_03: And we are testing if the software reacts as expected.
• 9:10
  SPEAKER_03: Okay, yeah.
• 9:12
  SPEAKER_02: So I think it is safe to say that Airflow is pretty critical here, right?
• 9:16
  SPEAKER_03: Because that's true, yeah.
• 9:19
  SPEAKER_03: Because without Airflow, basically we would not be able to run so many data sets in this short time, and yeah, it's very critical for us to have a good result.
• 9:31
  SPEAKER_02: Okay, that this is this is amazing.
• 9:33
  SPEAKER_02: So, what makes existing autonomous and driver assistant systems particularly challenging?
• 9:39
  SPEAKER_02: And how has Airflow helped address these challenges?
• 9:44
  SPEAKER_03: Maybe for you.
• 9:45
  SPEAKER_03: So the biggest challenge is really the amount of data we have to run here and the amount of different configurations also of the software we have.
• 9:56
  SPEAKER_03: So, for this in our projects, we have a setup of that all the functional teams define their test scope.
• 10:04
  SPEAKER_03: So they are defining the parameters and the scenes they want to replay.
• 10:09
  SPEAKER_03: And this is done via Docs S Code approach.
• 10:12
  SPEAKER_03: So they are writing their test specifications in a documentation, and then we have a tool which is running also in Airflow in our instance, which is interpreting these specifications and is generating test configs.
• 10:28
  SPEAKER_03: And these test configs we're using in Airflow to massively parallelize then the following tests.
• 10:35
  SPEAKER_03: We are running hundreds or thousands of tests in parallel.
• 10:39
  SPEAKER_03: This is the job of Airflow in the end.
• 10:42
  SPEAKER_03: And output of each of these tests is actually a KPI or a fail-passed decision for a single test.
• 10:49
  SPEAKER_03: And then with Airflow, we are able to collect all these test results in the end, waiting until everything is finished, and aggregate this to one final decision that we can provide to the project management or to the overall management and Robert Posch to say, okay, we want to sell or we want to release the new software version.
• 11:11
  SPEAKER_03: And yeah, to do a big challenge is actually we need to be fast.
• 11:16
  SPEAKER_03: We need to be able to run a lot of tests in a short time, but we need to be able to make also enough or enough detail to say how it would run in the car.
• 11:28
  SPEAKER_03: Because I said we are not running in the car these tests to save costs.
• 11:32
  SPEAKER_03: Yes, we have two dimensions of tests, and yeah, challenges here, both must run in Airflow.
• 11:39
  SPEAKER_03: So we have to run on x86 tests on GPU machines, but also on the actual embedded hardware, which would run in the vehicle, but what we have in server centers.
• 11:50
  SPEAKER_03: So we need to orchestrate on different infrastructure in even different countries where this infrastructure is.
• 11:58
  SPEAKER_03: And here we are also using Airflow for.
• 12:01
  SPEAKER_03: And yeah, in the end, what we do with Airflow also in in complete on both these dimensions, and this is the orthogonal dimension we have again, is that Airflow has to collect all this data, has to execute this test, evaluating the results and reporting to other systems as well.
• 12:19
  SPEAKER_03: So this is we have to integrate Airflow in a complete project tool landscape.
• 12:25
  SPEAKER_03: And that the big advantage we see, especially with Airflow, here is that we have a single point of truth with it, because we have one GUI, one API for all the developers to run all these different test dimensions.
• 12:40
  SPEAKER_03: We can provide a simple way for them to run these tests, even if these tests are very uh challenging and yeah, a lot of dimension.
• 12:52
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 13:01
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 13:13
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:22
  SPEAKER_02: Chris, you said we have to run many tests in a very short period of time.
• 13:27
  SPEAKER_02: Can you give us some numbers here?
• 13:28
  SPEAKER_02: What are we talking about?
• 13:31
  SPEAKER_03: Yeah, so normally we have around, I would say, between 700 and 1000 CI runs per day, for example, where we have to run 20 minutes tests, 700 or 1000 times.
• 13:44
  SPEAKER_03: And for this big release test, we need to run around 10,000 of hours in two weeks.
• 13:52
  SPEAKER_01: And getting to this number, I believe one of the critical factors why we selected Airflow was also that if a test batch is maybe running for two weeks, we know that in between the two weeks something will break apart, or we need to deploy a patch into the infrastructure, we need to reboot the servers.
• 14:08
  SPEAKER_01: And with Airflow, we were sure that all the elements could be restarted and still driving the tests.
• 14:14
  SPEAKER_01: I remember coming a fresh colleague coming in from a bit of history, and he said, Let's give me a proof that this software can scale and run a million of jobs.
• 14:23
  SPEAKER_01: And I will take a week of time and try to run a million of jobs.
• 14:26
  SPEAKER_01: And he came back actually after two or three days and was done and said that he was convinced that okay, that thing can scale.
• 14:32
  SPEAKER_01: Besides, we made it away and we've proven that it can scale.
• 14:36
  SPEAKER_01: And I believe one important thing is also that as a kind of testing platform team, we are not able to drive all the tests on our own.
• 14:44
  SPEAKER_01: And in previous organizations, it was more in silos, one team was driving the software or creating the software, and another team was testing.
• 14:52
  SPEAKER_01: And meanwhile, it's more or less democratized, meaning the teams which are implementing the software they need to be capable and be self-sufficient to drive the tests on their own, to implement them and to run them.
• 15:05
  SPEAKER_01: And with Airflow, we yeah provide a self-service setup so that people can drop in their tests and they can directly use it without the central organization which they are blocking blocked by so they can directly work.
• 15:20
  SPEAKER_02: Okay.
• 15:20
  SPEAKER_02: I guess like when you say when you say you you made it self-sur self-serve, do you have any like how did you do that?
• 15:28
  SPEAKER_02: Maybe in the in the broad lines, like how people do use Airflow today?
• 15:36
  SPEAKER_01: Yeah, as Chris initially said, we have a docs as code approach where people can write tests, and that's I would say the gold standard.
• 15:43
  SPEAKER_01: People uh drive and write tests and bring them into the Git repo where they are automatically assessed and generated into tests.
• 15:50
  SPEAKER_01: And this should serve for the 80% of cases.
• 15:53
  SPEAKER_01: And for the 20% of cases where this is not sufficient, uh, people can use the same git repo and drop DAX.
• 15:59
  SPEAKER_01: And so multiple teams, functional teams, develop their own DAX based on some templates and operators that we have, and that's serving the other 20% of cases.
• 16:08
  SPEAKER_01: And this serves then for the left, left over 20%, 80% of the cases, and then for only the residual, maybe 20% of the 20%, then we develop specific tooling, or they also maybe extend the platform.
• 16:23
  SPEAKER_01: There also we have a contribution model similar, like open source, where functional teams, if they need a feature, then they can drop it in and do a PR and we can validate this.
• 16:34
  SPEAKER_02: Okay, this is super interesting because self-serving airflow in in companies is a such critical topic for actually many companies, like many companies are trying are trying to self-serving airflow because obviously you want to make sure that people do things as you expect they do things.
• 16:50
  SPEAKER_02: So that's why usually you rely on templates and so on.
• 16:53
  SPEAKER_02: So this is good to know.
• 16:55
  SPEAKER_02: I guess at some point you mentioned that you were testing different scenarios on both virtual and physical hardware.
• 17:02
  SPEAKER_02: So, could you explain this hybrid approach and why it's necessary for autonomous driving development?
• 17:09
  SPEAKER_02: Like, why do you have to actually I think you mentioned that at some point, Chris?
• 17:12
  SPEAKER_02: Like what you need to do on.
• 17:16
  SPEAKER_01: Yes, the the challenging situation is we are running on an embedded um CPU or ECU, you call it the uh the control unit in the vehicle later on.
• 17:25
  SPEAKER_01: And those control units they are on specific hardware, it's it's not off-the-shelf hardware.
• 17:31
  SPEAKER_01: They're running some real-time operating system.
• 17:34
  SPEAKER_01: Uh yeah, um, still it's POSIX, but it's sometime uh real-time.
• 17:38
  SPEAKER_01: And so the timing is very important, how the software behaves on the chip.
• 17:42
  SPEAKER_01: And the chip also has hardware accelerators, specific, for example, video decoders and accelerators for KA artificial intelligence, so that neural networks can be processed in high speed, similar like you do in GPUs with Qt.
• 17:56
  SPEAKER_01: But they are very specific to their APIs and to their instructions that you can simulate this on the x86, but to get an accurate timing, how long some process took to, for example, post-process an image, do segmatic segmentation.
• 18:12
  SPEAKER_01: Uh, this needs to be time measured.
• 18:14
  SPEAKER_01: And for everything that's timing related and integration to vendor specific for the ECU instructions, you need to have the embedded physical hardware as well as uh physical vehicle buses, like which is very common, the CAN bus or automotive Ethernet, those interfaces you can hardly emulate on x86.
• 18:33
  SPEAKER_01: Um, we try to do most of the tests we can do in software in x86, so we can leverage the cloud and we can scale up and down as we need the hardware.
• 18:44
  SPEAKER_01: And that's also a perfect fit for Kubernetes that you can scale your infrastructure in the cloud and you can distribute the workload.
• 18:51
  SPEAKER_01: And we can orchestrate with it.
• 18:53
  SPEAKER_01: And the other way around is when we know that certain tests need to um assure the timing is correct, or you need to have window-specific half accelerators, um, those need to be running on the physical hardware.
• 19:05
  SPEAKER_01: And we try to um uh yeah, depending on the test demand, schedule a test either on one or the other platform, but everything is abstracted.
• 19:15
  SPEAKER_01: So the user in his test definition can say what he wants to test and how he needs to test it, and then with our testing pipeline, the test is directed to the right AI hardware platform.
• 19:27
  SPEAKER_03: Got it.
• 19:29
  SPEAKER_03: And to add here, maybe another challenge we have is in the end, both platforms must be somehow comparable.
• 19:36
  SPEAKER_03: Uh, because if one platform would, let's say, lie, then it wouldn't be helpful.
• 19:43
  SPEAKER_03: So challenge for our team, especially, is here also to make them comparable and to make them credible for the users.
• 19:52
  SPEAKER_02: Okay.
• 19:54
  SPEAKER_02: So I guess like during our prep call before recording this episode, I think like at some point you mentioned that you were using dynamic task mapping in your tests, and this is a very interesting feature because obviously dynamic task mapping was expected by so many engineers as you can generate tasks truly dynamically without knowing the inputs in advanced.
• 20:15
  SPEAKER_02: So it's more complex than just a for loop and generating your tasks.
• 20:19
  SPEAKER_02: But I guess my question to you is how did uh dynamic test mapping help you to solve some challenges that you had with with your tests?
• 20:30
  SPEAKER_03: Yeah, as you said, the dynamic task mapping was really a game changer for us using Airflow here because we had a problem previously, or it's still a challenge.
• 20:42
  SPEAKER_03: We don't know when we started testing workflow how many tests we want to run.
• 20:47
  SPEAKER_03: As I mentioned, the teams, the functional teams are defining what should be tested, but we don't know exactly what and how many hours, how many parameters they want to test.
• 20:59
  SPEAKER_03: So previously we needed to implement some deck generators with a lot of complex code, a lot of complex functions actually, to say or to have the possibility to say I want 10 hundred thousand tests.
• 21:16
  SPEAKER_03: And then also afterwards, it was very hard to report uh the status of the test because each generated deck was a different deck on a different QI.
• 21:27
  SPEAKER_03: There was the report of the status of the task.
• 21:29
  SPEAKER_03: Yeah, it was really tough and hard to provide this to the users in an experienceable way.
• 21:36
  SPEAKER_03: And yeah, now with the dynamic test mapping, basically all these problems were gone because we can provide the numbers of tests without knowing before we start the deck to the users.
• 21:51
  SPEAKER_03: They can use always the same deck.
• 21:53
  SPEAKER_03: So we have a single API for all the tests they want to do, and we have a single reporting.
• 22:00
  SPEAKER_03: Page where they can look my test was successful or not.
• 22:04
  SPEAKER_02: Yeah, honestly, like for those who haven't tried dynamic task mapping yet, I strongly advise you to at least try this feature because it is to your point, Chris, it is a game because before dynamic task mapping, you had to do some really weird work workarounds.
• 22:21
  SPEAKER_02: And now with that feature, you can do you can do things in the in a much easier way, especially obviously when whenever you want to generate tasks on something that you don't know in advance.
• 22:30
  SPEAKER_01: And it's really strange looking back in the history how we decided to airflow and use it as an orchestrator, because in the beginning that feature was not there, but still we decided for airflow.
• 22:40
  SPEAKER_01: And I remember the times where we had a lot of branching gen uh operators and then code generators uh generating just a graph of tests which we were not knowing if it's to be executed or not.
• 22:51
  SPEAKER_01: Yeah, it's really strange.
• 22:53
  SPEAKER_02: Yeah, that that that was painful.
• 22:55
  SPEAKER_02: I actually started at the same airflow version than you gence.
• 22:58
  SPEAKER_02: It was 1.10, so that was a painful time for sure.
• 23:01
  SPEAKER_02: So I guess what I love with your use case, there are many things that I love, but particularly the fact that you I think it's safe to say that Airflow scales, right?
• 23:11
  SPEAKER_02: Some part sometimes you see, I would say, competitors saying that airflow doesn't scale.
• 23:16
  SPEAKER_02: We can agree that it's pretty false here.
• 23:19
  SPEAKER_02: But I guess scaling is definitely uh it's definitely it takes a lot of trials and errors.
• 23:25
  SPEAKER_02: And I guess, can you tell us more about the challenges that you had when you uh started to scale airflow at at the level that you are uh talking about here?
• 23:35
  SPEAKER_01: Yeah, so first when we started the airflow, we made kind of POCs and we tested if everything runs to expected numbers, but we were not sure to which number this it goes to.
• 23:46
  SPEAKER_01: And one big thing that we had as a challenge for maybe a year or one and a half years was that we also integrated Airflow with data events.
• 23:56
  SPEAKER_01: So, as Chris said earlier in the beginning, we wanted to have pipelines running whenever new data lands from the field.
• 24:03
  SPEAKER_01: And the volume was constantly by surprise changed, and it was not really between the departments correctly clear when which batch of data will arrive, and we were always faced with event storms unplanned, and that was always giving us a bit of a challenge increasing the performance and also the scale iteratively.
• 24:23
  SPEAKER_01: So we didn't do a test initially with millions of tasks because that's far way out of reality.
• 24:29
  SPEAKER_01: We always adjusted and tuned the system incrementally whenever there was a new challenge, and that was maybe an endeavor for one and a half years to see increasing volume and then tuning parameter here and there.
• 24:41
  SPEAKER_01: And it's not easy to make a cookbook because it very much depends on your use case, how the deck is being modeled, how the events are coming.
• 24:49
  SPEAKER_01: Um, but uh we used the good thing is in this data case, we know a deck was being triggered when the event came, and then if too many events came, they were just queued and too many deck runs.
• 25:00
  SPEAKER_01: And then we just had a peak of jobs and we worked them off.
• 25:04
  SPEAKER_01: Bit of a challenge was always that Friday evening it was hitting us hard, and then it took the whole weekend to work it off.
• 25:10
  SPEAKER_01: And then the next week we were uh trying to improve from yeah, increasing database, tuning parameters, uh scheduling, having more schedulers and uh tuning uh queries going into tracing.
• 25:23
  SPEAKER_01: Uh, I had a talk uh on the Airflow Summit in San Francisco, which is also recorded and which I can recommend.
• 25:29
  SPEAKER_01: That's maybe summarizing a bit of the storyline and takes a bit longer where we went to.
• 25:35
  SPEAKER_01: But yeah, it was also it's a good example for how we contributed to the community as well because we saw some queries were not optimal, and then we contributed some fixes and we tuned on all levels, and that's the beauty of open source.
• 25:48
  SPEAKER_01: You can directly go and trace into the source and you can optimize it where it is needed.
• 25:53
  SPEAKER_01: And yeah, hard to say.
• 25:55
  SPEAKER_01: So it was in our case an incremental approach where we started with kind of the colleague came in, wanted to have the proof that a million tasks were running.
• 26:04
  SPEAKER_01: We also, of course, did a couple couple of load test example decks to prove that it scales into the needs.
• 26:09
  SPEAKER_01: But when it comes to real life, you also have the challenge of the back-end Kubernetes need to scale, and then you need to archive off data and the background and your queue jobs, and then if the queue is larger, you have side effects.
• 26:22
  SPEAKER_01: Yeah, so it's a complex endeavor, but it's also a big learning story on our end.
• 26:28
  SPEAKER_02: Yeah, definitely it is hard.
• 26:29
  SPEAKER_02: And I can't recommend your your talk that you did at the Airflow Summit enough.
• 26:34
  SPEAKER_02: I actually watched it on the official Airflow YouTube channel.
• 26:38
  SPEAKER_02: And if you want to learn how to uh scale Airflow and what are the configuration settings that you can play with, especially for the scheduler, I strongly recommend anyone to look at your talk.
• 26:48
  SPEAKER_02: This is super, super interesting.
• 26:51
  SPEAKER_02: I learned a lot uh from it.
• 26:52
  SPEAKER_02: So thank you for that.
• 26:53
  SPEAKER_02: So I guess, like you mentioned that you've contributed to to Airflow.
• 26:57
  SPEAKER_02: Can you tell us more about some AIPs that you did?
• 27:00
  SPEAKER_02: Or because I think you did some wonderful contributions that's gonna be very useful.
• 27:05
  SPEAKER_01: Yes, so for our team, I would say every teammate from our team contributed upstream already.
• 27:11
  SPEAKER_01: Even Chris is a PO, he contributed also one fix upstream.
• 27:15
  SPEAKER_01: So we are really in the open source community.
• 27:17
  SPEAKER_01: And it started on our side with some patches and fixes, of course.
• 27:21
  SPEAKER_01: And the one thing that bothered me in the beginning was that the log wasn't auto-refreshing.
• 27:26
  SPEAKER_01: So I my first PR was a log tailing that you don't need to hit F5 when the task is still running.
• 27:32
  SPEAKER_01: Um, and then uh over the time we wanted to make Airflow better usable for the users, and uh, we had a big debate on how we are doing that.
• 27:41
  SPEAKER_01: Uh, because if you submit a job in this 1.10 or maybe 2.2 versions, if you submit a job, you can have parameters.
• 27:49
  SPEAKER_01: And for every manually submitted job, for us, it's important that the user can specify his parameters, for example, which git branch you want to use as a test for his system test or which parameters for the testing to use.
• 28:02
  SPEAKER_01: And in the past, it was only possible to provide a JSON dictionary, and then the user needed to know what to put into the JSON dict.
• 28:10
  SPEAKER_01: And so initially we had a web UI in front for the easierness of the user that the user has a configuration panel and he can specify what job options he wanted to have.
• 28:19
  SPEAKER_01: But it was always a bit of a challenge in maintaining another UI on top of Airflow just for users to submit a job.
• 28:26
  SPEAKER_01: So our first contribution, our first, I would say really larger contribution was the AAP50, so that the parameters that are defined on the DAC anyway are being used and treated as uh parameters to render a form for the user.
• 28:42
  SPEAKER_01: So today in the Airflow UI, if you press a button and you play a DAC, if you have uh parameters defined, then a nice form will be rendered where you can uh specify the parameters and you give the user some instructions.
• 28:54
  SPEAKER_01: That was one of our contributions, and I'm a bit proud of that.
• 28:59
  SPEAKER_02: I I think it is safe to say on behalf of the Airflow community that thank you so much for this contribution.
• 29:06
  SPEAKER_02: Because at the time where we had to use to put the JSON value, that was so painful and prone and error prone.
• 29:15
  SPEAKER_02: Thank you so much for your for your contributions here because that has super full.
• 29:21
  SPEAKER_01: Yeah, and one other AAP which we are working on with high focus at the moment is the so-called edge executor.
• 29:29
  SPEAKER_01: Um, and that's also we elaborated a bit about the hardware challenge that we have that we need to run on physical hardware.
• 29:36
  SPEAKER_01: And as you cannot rent this in standard cloud, usually everything of our test suits and also the airflow setup runs in the cloud.
• 29:43
  SPEAKER_01: Um, but the dedicated physical hardware is very specific to us.
• 29:47
  SPEAKER_01: So we we need to have them either on company premise in our enterprise infrastructure or also on co-located data centers.
• 29:54
  SPEAKER_01: So it's far away from the cloud.
• 29:56
  SPEAKER_01: And we saw severe stability problems because at the moment we are scheduling everything with salary, and we are really happy about the scheduling with the salary executor, distributing the workload in the cloud to whatever execution system.
• 30:10
  SPEAKER_01: But we really had a hard time to schedule workload on the physical hardware being stable because via VPN, corporate proxies, whatever enterprise infrastructure that lies between, salary needs to have a physical stable TCP connection, and then is from the operational quality really giving a lot of headache if you want to have a stable reliant infrastructure.
• 30:34
  SPEAKER_01: And we were a bit in the about comparing to other, for example, CI CD pipeline.
• 30:39
  SPEAKER_01: If you have a GitHub runner, you can uh set up your own GitHub local runner, self-hosted, and that's only communicating via HTTP.
• 30:48
  SPEAKER_01: And we thought also this is very much demanded on Airflow having that, and and so we started that AP69 for the edge um executor, which is a bit experimental.
• 31:00
  SPEAKER_01: It's in the source tray already, uh, but it's way much more stable for our remote use case, and we use it together with the hybrid execution today in our production already, even though it's experimental.
• 31:12
  SPEAKER_01: We are the main contributor there at the moment.
• 31:15
  SPEAKER_01: But our production farm runs happily and much more stable than with salary.
• 31:20
  SPEAKER_01: And for the hybrid scheduling today, everything that's scheduled in the cloud, we use a salary executor, and everything that's scheduled uh on the physical hardware, we use the edge executor quite happily.
• 31:32
  SPEAKER_01: And that is then officially released at Airflow 3.
• 31:35
  SPEAKER_01: But everybody who wants to try it out can take it and build it from source at the moment.
• 31:42
  SPEAKER_02: Awesome.
• 31:42
  SPEAKER_02: I don't know, I don't know what to say.
• 31:44
  SPEAKER_02: That was a super interesting episode.
• 31:46
  SPEAKER_02: Honestly, that was one of the most unusual uh use cases that I've ever heard with Airflow, and that was super interesting.
• 31:55
  SPEAKER_02: And again, it shows how Airflow scale, and it shows that you can do a lot of different things with Airflow.
• 32:02
  SPEAKER_02: And uh, thank you for sharing all of your insights and learnings.
• 32:06
  SPEAKER_02: And also thank you for contributing to Airflow because again, your contributions have been super helpful.
• 32:11
  SPEAKER_02: I guess one last question what is the best way for uh folks uh to get in touch with you?
• 32:16
  SPEAKER_02: Like, for example, if I maybe if I'm working in or if I'm very interested by this topic, like how can I get in touch with you?
• 32:26
  SPEAKER_03: Like it starts.
• 32:27
  SPEAKER_03: Uh so maybe the best way um to get in touch with us is uh via LinkedIn profile.
• 32:32
  SPEAKER_03: So we both have a LinkedIn profile.
• 32:35
  SPEAKER_03: I don't know how we could share maybe the link to it, but uh yeah, you will find it or find us with Jens Scheffler, Christian Schilling looking for it, and you will see both of us working at Robert Bosch.
• 32:48
  SPEAKER_03: Then just contact us and we are happy to answer your questions.
• 32:52
  SPEAKER_01: Yeah, and otherwise, also I personally am very active in the Slack community.
• 32:58
  SPEAKER_01: So if you hit a message in troubleshooting, then it might be very probable that I have an answer on this or also via Slack.
• 33:06
  SPEAKER_01: There's a very much uh response on it.
• 33:09
  SPEAKER_02: Awesome.
• 33:10
  SPEAKER_02: Thank you so much.
• 33:11
  SPEAKER_02: Thank you, Jens.
• 33:12
  SPEAKER_02: Thank you, Chris.
• 33:13
  SPEAKER_02: That was amazing.
• 33:14
  SPEAKER_02: I truly hope to see you again for another episode because uh there is so much to learn in your use case.
• 33:18
  SPEAKER_02: We'll see.
• 33:18
  SPEAKER_02: But have a good one.
• 33:20
  SPEAKER_02: Take care.
• 33:21
  SPEAKER_00: Thanks, Bye.
• 33:22
  SPEAKER_02: Thank you.
• 33:22
  SPEAKER_02: Bye-bye.
• 33:26
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 33:32
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 33:36
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.