Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie

Transcript

How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_04: Before Airflow, every simulation expert was building their own tool chain basically suited to their own needs, and they were able to run them on their environment.
• 0:08
  SPEAKER_04: But this was easily shareable to other simulation experts from the field, or maybe even other simulation experts from a slightly different field who were interested in the kind of simulation that they were doing because they needed them for their project as well.
• 0:20
  SPEAKER_04: And then they were asking them, okay, can I do the simulation?
• 0:23
  SPEAKER_04: And then it turned out, oh, it's not so easy to transfer the tailored script that they had to the system or to the environment that the other simulation expert had.
• 0:36
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering in AI.
• 0:46
  SPEAKER_00: Here, Airflow community, we've got exciting news.
• 0:49
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:52
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow and grab a free Airflow 3 Fundamentals or DAG Authoring certification worth$150 each.
• 1:02
  SPEAKER_00: Plus, you can enter to win an exclusive workshop with Mark Lumberti.
• 1:06
  SPEAKER_00: Don't miss out.
• 1:08
  SPEAKER_00: The survey link is in the show notes and closes November 21st.
• 1:14
  SPEAKER_03: Hello everyone, welcome to the Data Flowcast.
• 1:17
  SPEAKER_03: I'm your host, Mark Lamerty, and today I'm joined by Anya McKenzie from Covestro.
• 1:22
  SPEAKER_03: I'm super excited to have you here.
• 1:24
  SPEAKER_03: But first question, how are you doing?
• 1:26
  SPEAKER_04: Thank you very much.
• 1:27
  SPEAKER_04: Also, welcome to all of the listeners, and I'm doing fine, thank you.
• 1:31
  SPEAKER_04: How are about you?
• 1:32
  SPEAKER_03: Doing great.
• 1:32
  SPEAKER_03: I'm really looking forward to this episode because by the way, Anya is an expert for chemical informatics, which is pretty amazing.
• 1:41
  SPEAKER_03: And so today we are talking with Anya, who has taken Airflow in a completely different direction than most companies.
• 1:47
  SPEAKER_03: At Cavestro, leading polymer and materials science company, she's built a simulation workflow platform that turns Airflow into a toolbox for computational chemistry experts.
• 1:58
  SPEAKER_03: Honestly, I'm gonna stop there because this is a domain that I know nothing about.
• 2:03
  SPEAKER_03: And I'm very excited again to know more about how Airflow is used in your company because I'm sure many people wonder what you do at your company.
• 2:11
  SPEAKER_03: So let's begin with some context about Covestro.
• 2:15
  SPEAKER_03: What do you do?
• 2:15
  SPEAKER_03: Can you explain what computational simulations mean in the polymer and materials science field?
• 2:22
  SPEAKER_03: And then what kind of problems your simulation experts are trying to solve?
• 2:26
  SPEAKER_04: Yeah, sure.
• 2:27
  SPEAKER_04: Thank you very much.
• 2:27
  SPEAKER_04: Yeah, as you said, Covestro is a company in the field of high-quality polymers, though we produce high-quality polymers, and we have about 17,500 employees with operations across 46 production sites globally.
• 2:42
  SPEAKER_04: So we are quite a large company, and our products are used in everyday goods, like for example, electronics, so cases of laptops or smartphone cases, but also automotive, so seats or dashboards in the automotive industry are often including Covester products, but we also produce products for installations, for example, in fridges or housing, or also the matrices are often made from Covester products, so quite diverse set of products.
• 3:10
  SPEAKER_04: And Covestro is a very innovative company, so one of our research goals actually is to become fully circular, and for that we are doing a lot of research.
• 3:19
  SPEAKER_04: And I'm uh in the department which is called digital research of development.
• 3:24
  SPEAKER_04: And of course, we have a lot of lab uh scientists, but we are doing digital research and we're using digital research methods, and one of them are chemical simulations.
• 3:35
  SPEAKER_04: So, chemical informations you can think of are like experiments in the computer.
• 3:41
  SPEAKER_04: So, our lab colleagues are doing chemical experiments in the lab, and we are simulating the chemical experiments with digital methods.
• 3:50
  SPEAKER_04: Typical problems would be, for example, the tailored design of new products by establishing relations between the chemical structure and also the composition of the product to the physical chemical behavior.
• 4:02
  SPEAKER_04: So we can predict the physicochemical behavior of our products with our methods, but we can also investigate and simulate reaction pathways, for example, and this is used to optimize reaction conditions for our products or for the reactions that we are doing.
• 4:18
  SPEAKER_04: And we can also stimulate foaming, we can do foaming simulations, and we are using that to investigate the feasibility of a foaming experiment, so to say, and also to optimize manufacturing processes.
• 4:30
  SPEAKER_04: So these would be typical use cases, let's say.
• 4:33
  SPEAKER_03: So, well, I already have so many questions, but so let's take an example.
• 4:37
  SPEAKER_03: You said that, for example, in a computer, I can find a product from Covestro.
• 4:42
  SPEAKER_04: What would be that product exactly?
• 4:44
  SPEAKER_04: Covestro is not producing like products directly to the consumer but to businesses.
• 4:49
  SPEAKER_04: For example, one of the large products that we are producing are polyurethanes, and polyurethanes normally consist of isocyanides and polyoles, and they react to a foam.
• 5:00
  SPEAKER_04: This would be a very typical example.
• 5:02
  SPEAKER_04: So, for example, you buy polyol from Covestro and then isocyanide from Covestro, and they then react to a foam.
• 5:10
  SPEAKER_04: And this is then done at our customer's side.
• 5:12
  SPEAKER_04: So we are providing the ingredients, so the isocyanide and the polyol, and then our customer produces the foam, for example, a mattress, out of our two products.
• 5:21
  SPEAKER_04: And then you have very different mechanical properties for foams.
• 5:25
  SPEAKER_04: So, for example, matrices has to be very soft, but you can also imagine hard foams.
• 5:30
  SPEAKER_04: And this can be then steered by the composition of the polyol, mainly so there needs to be a lot of thoughts and engineering put into our polyols, for example, to produce very certain properties of foams.
• 5:45
  SPEAKER_04: And then we simulate these kinds of properties and also mechanical properties sometimes to design the polyoles and isocyanides to the needs of our customers, basically.
• 5:56
  SPEAKER_03: I hope that everyone is able to get that Covestro is actually everywhere.
• 6:00
  SPEAKER_03: This is super cool.
• 6:02
  SPEAKER_03: And so you mentioned that you have simulation experts that actually do simulations to ensure that everything will work out as expected.
• 6:10
  SPEAKER_03: Can you tell us more about how those simulations are done today?
• 6:15
  SPEAKER_03: Or I would say actually how they were done before Airflow.
• 6:19
  SPEAKER_04: So we have an on-premise cluster, and then our simulation experts typically write bash scripts or Python scripts to run their simulations.
• 6:28
  SPEAKER_04: So let's say a simulation has some components.
• 6:32
  SPEAKER_04: So first of all, you have a simulation engine.
• 6:34
  SPEAKER_04: Let's say, for example, you're doing a molecular dynamic simulation, then you have a very specific software tool which does this kind of simulation.
• 6:43
  SPEAKER_04: But of course, you also need some setup around this.
• 6:46
  SPEAKER_04: So you need to create an input file.
• 6:48
  SPEAKER_04: So you need to define the simulation and also provide information about the chemical system that you want to simulate.
• 6:54
  SPEAKER_04: This is part of the input file.
• 6:56
  SPEAKER_04: And also you need to provide us some information about the type of simulation that you're doing and the type of property you're interested in.
• 7:03
  SPEAKER_04: And this is all part of an input file.
• 7:05
  SPEAKER_04: And also then you sometimes need to provide additional files, for example, describing the chemical structure or certain properties that need to be part of the simulation.
• 7:15
  SPEAKER_04: So the input that needs to be done.
• 7:17
  SPEAKER_04: And this sometimes can be provided or can be generated by additional Python scripts and so on and so forth.
• 7:23
  SPEAKER_04: So there's some kind of tool chain involved.
• 7:25
  SPEAKER_04: And then after the simulation, you often analyze the results.
• 7:28
  SPEAKER_04: So you take a look at, for example, a trajectory.
• 7:32
  SPEAKER_04: So how did the molecule move during the simulation?
• 7:35
  SPEAKER_04: You get the trajectory, and then you analyze them, for example, compute certain properties like diffusion coefficients or whatever.
• 7:43
  SPEAKER_04: And this is then also another tool chain that comes behind that.
• 7:47
  SPEAKER_04: And before Airflow, every simulation expert was building their own tool chain basically suited to their own needs, and they were able to run them on their environment.
• 7:57
  SPEAKER_04: But this was not easily shareable to other simulation experts from the field, or maybe even other simulation experts from a slightly different field who were interested in the kind of simulation that they were doing because they needed them for their project as well.
• 8:09
  SPEAKER_04: And then they were asking them, okay, can I do the simulation?
• 8:12
  SPEAKER_04: And then it turned out, oh, it's not so easy to transfer the tailored script that they had to the system or to the environment that the other simulation expert had.
• 8:20
  SPEAKER_04: This was the situation we ended up with, and we wanted to make this shareable in a very easy way to make them accessible, these kind of simulation tool chains and also knowledge that is built into this to other people from the department.
• 8:35
  SPEAKER_03: It's super interesting because I don't know if you saw that on Reddit.
• 8:38
  SPEAKER_03: I don't even know if you use Reddit.
• 8:40
  SPEAKER_03: But there are many people wondering, hey, should I use Airflow?
• 8:43
  SPEAKER_03: Should I use an orchestrator?
• 8:44
  SPEAKER_03: And some people are saying you don't need an orchestrator, you just need cron and then some Python scripts.
• 8:50
  SPEAKER_03: Yes, it works until it doesn't.
• 8:52
  SPEAKER_03: And that's actually a very good example.
• 8:54
  SPEAKER_03: Like when you have many people working on different Python scripts, but on the same kind of bigger project, it's actually very hard to orchestrate all of those Python scripts together and also ensure that everyone is able to know what the other are doing.
• 9:10
  SPEAKER_03: And it's just like you end up in a giant big mess with a ton of Python scripts and bash scripts or whatever that you don't even know what they are doing anymore.
• 9:19
  SPEAKER_03: So it's just a nightmare.
• 9:21
  SPEAKER_03: So I'm glad that now you are using Airflow, right?
• 9:24
  SPEAKER_03: So tell me more about this.
• 9:25
  SPEAKER_03: What is your current setup with Airflow?
• 9:27
  SPEAKER_03: What do you do with it typically?
• 9:29
  SPEAKER_04: Yeah, so one feature that we use of Airflow is the web UI because this is a very special use case.
• 9:35
  SPEAKER_04: I mean, you already said that we are using Airflow in a quite different way.
• 9:38
  SPEAKER_04: So all of our workflows are manually triggered.
• 9:41
  SPEAKER_04: So we do not make use of sensors or something like this because we are triggering simulation workflows, and they are always triggered by a simulation exit because there needs to be a lot of expert knowledge put into this.
• 9:52
  SPEAKER_04: As I already said, each simulation workflow requires some parameters, and this is expert knowledge that needs to be put in from the respective experts.
• 10:01
  SPEAKER_04: So all of them are manually triggered.
• 10:03
  SPEAKER_04: And furthermore, we also have some compliance restrictions there.
• 10:06
  SPEAKER_04: So, like the username must be known.
• 10:08
  SPEAKER_04: So we always have a user ID connected to the generated data so that we know who is the data owner basically.
• 10:15
  SPEAKER_04: So that's why we like what we liked Airflow, that it directly provided a web UI.
• 10:20
  SPEAKER_04: And what we also really liked were the RP endpoints.
• 10:24
  SPEAKER_04: This is also really useful to have that all of the workflows that we have on Airflow already have an RP endpoint and can also be triggered by a third-party tool, let's say, which is also pretty cool.
• 10:34
  SPEAKER_04: Then that the workflows are scalable is also really useful.
• 10:38
  SPEAKER_04: And also that the possibility to customize Airflow so much is really good for us because we have very high requirements on the compute infrastructure.
• 10:46
  SPEAKER_04: And this is something where we also need to have a lot of control over.
• 10:50
  SPEAKER_04: So some simulation engines like to run on GPU nodes, some of them like to run on CPU nodes, so they have a very special need on the compute infrastructure, and this is something that we also have under control.
• 11:04
  SPEAKER_04: And then also that Airflow is written in Python, and that you write the DAX in Python is really good because most of our simulation experts know Python.
• 11:12
  SPEAKER_04: As I said, most of the scripts were written in Python before.
• 11:15
  SPEAKER_04: So this is the knowledge that is already present and people are used to it, which is really good to have.
• 11:22
  SPEAKER_04: And last but not least, the open source and very active that Airflow is open source and has a very active community is also really good, and is also something that we look out for.
• 11:33
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:42
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 11:54
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:02
  SPEAKER_03: I did not ask before to mention all of the benefits of using Airflow, but now you know why you should use Airflow.
• 12:08
  SPEAKER_03: So thank you so much, Anya, for showing that.
• 12:10
  SPEAKER_03: I have some questions about what you just said, but if we focus a little bit on scalability, and as you said, there is a lot of data, your workloads are pretty much compute intensive.
• 12:21
  SPEAKER_03: So one of the most interesting technical challenges that you solved is handling large simulation output files.
• 12:27
  SPEAKER_03: And so sometimes I think like gigabytes, are we talking about gigabytes of data to share between uh between your tasks?
• 12:35
  SPEAKER_03: So, can you explain a little bit how did you solve that in Airflow?
• 12:39
  SPEAKER_03: Like how are you able to share so much data between your tasks?
• 12:42
  SPEAKER_03: Because I'm sure it's one of the many use cases that people have and wonder what is your solution.
• 12:48
  SPEAKER_04: First of all, I have to say that we are running Airflow on Kubernetes, which is a required knowledge for our solution.
• 12:54
  SPEAKER_04: And as you said, so sometimes we have even data that is in file size of over 100 gigabytes, so it's really quite large data that sometimes gets produced during the simulations.
• 13:04
  SPEAKER_04: And since we are running Airflow on Kubernetes, all of our tasks run in pods.
• 13:09
  SPEAKER_04: And in principle, the data that is generated during a task run persists only for the pods lifetime and is not shared between the tasks or the pods, and also using XCOM is not really a viable option.
• 13:22
  SPEAKER_04: So yeah, we needed something else, as you already said.
• 13:25
  SPEAKER_04: So what we did is that we have at the beginning and the end of the workflow or the DAC required tasks, and they create a persistent volume claim.
• 13:35
  SPEAKER_04: So basically, we have a persistent volume claim and a corresponding volume that persists for the duration of a DAC run.
• 13:42
  SPEAKER_04: And this is then mounted to all of the pods running the tasks into so that they can access and share the data to the following tasks or can access the data from the previous tasks.
• 13:52
  SPEAKER_04: Yeah, this is what we came up with, and this works quite well.
• 13:56
  SPEAKER_03: Yeah, I think that's a very good solution.
• 13:58
  SPEAKER_03: I think I already heard that somewhere, but it's definitely a good thing to do.
• 14:02
  SPEAKER_03: Definitely don't use XCOMs for sharing that much data.
• 14:04
  SPEAKER_03: If you try, it won't work anyway, but you will end up in some weird issues.
• 14:08
  SPEAKER_03: So don't do that.
• 14:09
  SPEAKER_03: Great.
• 14:09
  SPEAKER_03: So can you tell me more about the Kubernetes pod operator?
• 14:12
  SPEAKER_03: Because it's a very common operator used across many companies.
• 14:16
  SPEAKER_03: Do you have a customized version of the Kubernetes pod operator or do you use it as it is?
• 14:22
  SPEAKER_03: Can you just tell me more about your use case with it?
• 14:25
  SPEAKER_04: Yeah, sure.
• 14:25
  SPEAKER_04: Yeah, we have a customized operator.
• 14:27
  SPEAKER_04: Yeah, it's very important to us actually because, first of all, what we would like to achieve with the custom operator is that we would like to decouple, let's say, the logic from the DAC from the infrastructure needs.
• 14:38
  SPEAKER_04: As I said, the people that develop the workflow and then also the DUCs are simulation experts and not software scientists or software engineers.
• 14:45
  SPEAKER_04: So we would like to make the life as easy as possible for them.
• 14:49
  SPEAKER_04: And that's why we try to hide as much of the, let's say, software logic behind custom operators or custom tasks or tools that we provide to them.
• 15:00
  SPEAKER_04: And so the custom operator, first of all, gets all of the, let's say, airflow, all of the Kubernetes config from airflow variables.
• 15:09
  SPEAKER_04: So the Kubernetes infrastructure exports all of the, let's say, affinities, configs, and so on and so forth as airflow variables.
• 15:16
  SPEAKER_04: And then we can, via the custom operator, basically load them and set them correspondingly to the pods.
• 15:23
  SPEAKER_04: And also the Kubernetes pod operator takes care of mounting the PVCs for the duck run.
• 15:28
  SPEAKER_04: Sometimes we also need init container, which are set in there.
• 15:32
  SPEAKER_04: It sets the certain environment variables for the DUCs.
• 15:35
  SPEAKER_04: For example, the mount path of the PVC or of the volume is also exported as an environment variable so that's easily accessible within the task.
• 15:45
  SPEAKER_04: Then and also, for example, what we often need is the dynamic setting of resources.
• 15:50
  SPEAKER_04: So sometimes the person triggering the duck also needs to have access to the resources that are used in the task because the person, for example, know the simulation expert, then know, okay, this is the quite large chemical system.
• 16:01
  SPEAKER_04: So I need a lot of compute resources.
• 16:03
  SPEAKER_04: And then the person should be able to say how much resources the task will need or background will need.
• 16:10
  SPEAKER_04: And then this can be set via ginger templating.
• 16:13
  SPEAKER_04: This is also something that we made available, let's say, during the custom operator.
• 16:18
  SPEAKER_03: No, that's cool.
• 16:19
  SPEAKER_03: It's actually trying to make Airflow server service is something that many people want because, to your point, the simulation experts are not, I mean, you don't want them to be experts in airflow, and you want them to spend time in the simulations and not in the Lego throwing part.
• 16:33
  SPEAKER_03: So this is cool.
• 16:34
  SPEAKER_03: I'm really glad that you shared the way you use the Kubernetes pod operator.
• 16:37
  SPEAKER_03: Are there any operators that you uh also customized on top of your mind that you can think of it?
• 16:42
  SPEAKER_04: Yeah.
• 16:43
  SPEAKER_04: So we also modified the trigger decron operator.
• 16:46
  SPEAKER_03: Okay.
• 16:47
  SPEAKER_04: And this is done for let's say multi-scale simulation workflows.
• 16:51
  SPEAKER_04: So what we typically or what we also have is that a workflow B depends on a workflow A.
• 16:58
  SPEAKER_04: So very often you have, let's say, tool chains in simulation tool or in the simulation workflow, where simulation workflow A produces parameters for the simulation workflow B.
• 17:10
  SPEAKER_04: And then we have a situation where each workflow or each DAC that we have in our platform can serve as a task, let's say, in another DAC.
• 17:19
  SPEAKER_04: So that they are basically building blocks that we can make use of in another DAC, and then we can build even larger, more complex workflows in a very easy way.
• 17:28
  SPEAKER_04: And then let's say the workflow B, which depends on workflow A, makes use of the trigger duckground operator to trigger the duck run A to get the parameters out of it for the following tasks within workflow B.
• 17:41
  SPEAKER_04: And so by modifying the trigger background operator, we try to make also this as smooth as possible for the people developing these kinds of workflows.
• 17:50
  SPEAKER_04: And what the modified version, for example, does is handling over or passing over the information about the PVC because then the workflow A also needs access to the PVC from workflow B and also pass the user ID of the trigger user to the workflow so that also the data generated there knows from which person it stems or it was created basically.
• 18:14
  SPEAKER_04: These are typical modifications that we made there.
• 18:18
  SPEAKER_03: That's cool.
• 18:18
  SPEAKER_03: That's really cool.
• 18:19
  SPEAKER_03: That's the first time that I hear about modifying the trigger diagonal operator to pass the PVC that you use between the tags.
• 18:25
  SPEAKER_03: So it's an interesting use case.
• 18:27
  SPEAKER_03: So you just mentioned about customizing operators, but do you customize anything else in Airflow in order to make the life of simulation experts easier?
• 18:36
  SPEAKER_04: Yeah, so we also modified the web interface via plugins.
• 18:40
  SPEAKER_04: Yeah.
• 18:40
  SPEAKER_04: So for example, one feature that we needed was the upload of files.
• 18:44
  SPEAKER_04: As I already said in the beginning, an input to a simulation workflow can also include a file input, for example, file describing molecular structures or molecular systems or larger systems, for example.
• 18:56
  SPEAKER_04: And these need to be uploaded at some point because you cannot have them as a normal parameter.
• 19:02
  SPEAKER_04: It's quite difficult.
• 19:03
  SPEAKER_04: And yeah, for that, we have created a new, let's say, endpoint, a new Flask view for uploading files where the front end we use drop zone for the file upload, and then on the back end, we accept the files from Python Flask.
• 19:15
  SPEAKER_04: And then if it's a chunk file, we are combining the chunks together again.
• 19:20
  SPEAKER_04: And then we have an own PVC and of course associated volume for the uploaded files where they are then stored.
• 19:26
  SPEAKER_04: And they are also stored like on a user basis, so every user can only see their own files.
• 19:32
  SPEAKER_04: Okay.
• 19:32
  SPEAKER_04: But they cannot have access to files from other users, of course.
• 19:35
  SPEAKER_04: And then this upload volume is also mounted to the respective pods where the task ran into when they needed to access uploaded files, for example.
• 19:43
  SPEAKER_04: This is something that we've modified in the web interface.
• 19:46
  SPEAKER_03: That proves something else about Airflow, is how customizable it is.
• 19:50
  SPEAKER_03: Because to your point, you can customize the like let's say the building blocks of your data pipelines, but you can also customize the user interface.
• 19:58
  SPEAKER_03: And for those who don't know that, in Airflow 3.1, now you can go one step further because you have this React-based plugin mechanism.
• 20:07
  SPEAKER_03: So now your plugins can be in React.
• 20:09
  SPEAKER_03: And so you can like pretty much do whatever you want now with DUI.
• 20:13
  SPEAKER_03: And so, again, if you want to make Airflow as easy as possible for your teams and ensure that the Airflow user interface fits with your needs, then again, like you just need to create your plugin or whatever, and you can customize whichever part of Airflow.
• 20:28
  SPEAKER_03: And again, you just shared that right now, which is really cool.
• 20:31
  SPEAKER_03: Before closing our amazing episode, I have one question for you that wasn't planned actually.
• 20:37
  SPEAKER_03: But can you tell me more about if you have any advice for new users of Airflow, right?
• 20:43
  SPEAKER_03: Like someone that wants to use Airflow, but it's the first time.
• 20:46
  SPEAKER_03: Do you have any advices to share about that?
• 20:49
  SPEAKER_04: When I started with Airflow, I took your Udemy courses, which were really great.
• 20:55
  SPEAKER_03: Thank you so much.
• 20:55
  SPEAKER_03: I just said that this question was not scheduled.
• 20:58
  SPEAKER_04: But yeah, it's the truth.
• 21:00
  SPEAKER_04: Also, this is something I share with all of the people who started with Airflow in our department.
• 21:04
  SPEAKER_04: I always say, take a look at your Udemy courses because they're really great.
• 21:09
  SPEAKER_04: I also have to say that I really liked the astronomer tutorials and the sources on the astronomer website.
• 21:16
  SPEAKER_04: From my side, I found the documentation on Airflow itself, so on the Airflow website.
• 21:20
  SPEAKER_04: Let's say easy to read or easy to see what's going on there.
• 21:23
  SPEAKER_04: I really liked the content provided by astronomer, which is also not scripted, let's say.
• 21:29
  SPEAKER_04: So yeah, these were two resources that I heavily used and also shared within the team.
• 21:34
  SPEAKER_04: Yeah.
• 21:34
  SPEAKER_03: Okay, that's cool.
• 21:35
  SPEAKER_03: That's cool.
• 21:36
  SPEAKER_03: Yeah.
• 21:36
  SPEAKER_03: I think like the point here is Airflow might be a bit difficult at first or maybe a bit overwhelming because you can do many things with it.
• 21:43
  SPEAKER_03: But as long as you have the right resources, it will be always pretty easy to get started with Airflow.
• 21:49
  SPEAKER_03: Obviously, if you use Kubernetes, it might be a different story, but you know that Kubernetes is pretty hard.
• 21:55
  SPEAKER_03: Great.
• 21:55
  SPEAKER_03: Again, thank you so much for joining.
• 21:57
  SPEAKER_03: One last question to you, Anya.
• 21:59
  SPEAKER_03: What is the best way for folks to get in touch with you?
• 22:02
  SPEAKER_04: So via LinkedIn or via my company mail.
• 22:05
  SPEAKER_04: So anja.mackenzie at cobest.com or via LinkedIn.
• 22:09
  SPEAKER_02: Maybe you can also put this in the show notes.
• 22:11
  SPEAKER_02: We just put Anya Mackenzie and then we find you on LinkedIn.
• 22:15
  SPEAKER_02: I hope so, yes.
• 22:16
  SPEAKER_02: Never mind.
• 22:21
  SPEAKER_02: That's good, yeah.
• 22:22
  SPEAKER_02: So I hope it works.
• 22:23
  SPEAKER_04: I think I can also give you the URL maybe to the LinkedIn that's listed under my LinkedIn profile, then it should work, yes.
• 22:29
  SPEAKER_02: That's perfect.
• 22:30
  unknown: Okay.
• 22:30
  SPEAKER_04: Yeah, cool.
• 22:31
  SPEAKER_04: Yeah, and also I want to say thank you very much for having me.
• 22:34
  SPEAKER_04: It was really great to be part of this podcast, and I really enjoyed it.
• 22:38
  SPEAKER_03: Awesome.
• 22:39
  SPEAKER_03: Thank you so much, Anya.
• 22:40
  SPEAKER_03: It was a pleasure to have you in this episode as well.
• 22:42
  SPEAKER_03: And who knows, maybe we will see you again for another one.
• 22:44
  SPEAKER_03: So thank you so much for joining me.
• 22:46
  SPEAKER_03: Take care.
• 22:47
  SPEAKER_02: Yeah, see you.
• 22:48
  SPEAKER_02: Bye bye.
• 22:51
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 22:58
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 23:02
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.