Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating strictly sequential ETL pipelines at Synechron

Transcript

Orchestrating strictly sequential ETL pipelines at Synechron · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:07
  SPEAKER_01: We are excited to invite you to orchestrate everything.
• 0:11
  SPEAKER_01: Our free online conference on September 60th.
• 0:15
  SPEAKER_01: Engineering leaders from Lyft, Ram, and Wix will share how they are shipping production AI workflows on Airflow.
• 0:22
  SPEAKER_01: Plus a crash course for the new Astronomer Certified AI Orchestration Fundamentals exam.
• 0:28
  SPEAKER_01: Attendees get a discount code for a free certification.
• 0:32
  SPEAKER_01: A$150 value.
• 0:34
  SPEAKER_01: Go to astronomer.link slash data minus flowcast minus OE.
• 0:39
  SPEAKER_01: Link in the show notes.
• 0:41
  SPEAKER_01: Now let's get into the show.
• 0:43
  SPEAKER_02: Hello everyone.
• 0:44
  SPEAKER_02: Welcome, welcome to the Data Flowcast.
• 0:47
  SPEAKER_02: I'm your host, Mark Lomerti, and today I'm super excited to be joined by Ivana Izajovich, big data engineer at Cinecron.
• 0:55
  SPEAKER_02: First thing first, how are you doing today?
• 0:58
  SPEAKER_00: Hi Mark, thank you so much for having me.
• 1:00
  SPEAKER_00: I'm really happy to be here and I'm doing great.
• 1:03
  SPEAKER_00: Thank you.
• 1:04
  SPEAKER_00: I'm especially excited about today's topic because uh airflow is something I worked uh with very close every day.
• 1:13
  SPEAKER_02: Yeah, I'm very excited about this.
• 1:14
  SPEAKER_02: Uh because actually, you know, about today's podcast, uh, we will deep dive into orchestrating, um, strictly say controlled ETL pipeline.
• 1:22
  SPEAKER_02: I think you have like a way of doing things that I think is truly interesting.
• 1:27
  SPEAKER_02: Um also how you're gonna how did you survive about the subdag deprecation?
• 1:33
  SPEAKER_02: Um, because subdags have been very popular for a while, but as you uh will tell us, you actually uh did the migration, the migration.
• 1:42
  SPEAKER_02: Uh and then also we will uh talk about adapting custom retrail logic all the way through uh Airflow 3.
• 1:48
  SPEAKER_02: So a lot of things to cover.
• 1:49
  SPEAKER_02: Uh so without further ado, let's dive into this amazing episode.
• 1:53
  SPEAKER_02: So, first thing first, um can you introduce yourself and tell us a bit more about what you do at your company?
• 2:01
  SPEAKER_00: Uh okay, as I said, uh as you said, uh my name is Ivana and I work as a senior big data engineer.
• 2:08
  SPEAKER_00: And in my daily work, I mostly focused on big data ETL processes, data processing, uh, and orchestration.
• 2:15
  SPEAKER_00: So basically, I work on making sure the data from different sources is important, transformed, validated, and then made available for reports or some kind of analytics.
• 2:25
  SPEAKER_00: And a big part of my role is orchestration, as I mentioned, uh connecting different steps in a pipeline, making sure that they run in the right order, handling retries, monitoring alerts, and overall pipeline stability.
• 2:41
  SPEAKER_00: And as you know, uh airflow plays a very important role in that.
• 2:46
  SPEAKER_00: Because uh, for us, it's not just a scheduler, it's the central place where we control and monitoring the whole data pipeline.
• 2:55
  SPEAKER_02: Well, I think what you described uh will resonate a lot uh with many data engineers.
• 2:59
  SPEAKER_02: I think like they will all uh they will all uh feel that they are doing exactly what you are doing.
• 3:05
  SPEAKER_02: Um so can you actually uh just out of curiosity, uh, because you just said that Airflow is uh is uh crucial.
• 3:12
  SPEAKER_02: Um but uh when you joined your company, Airflow was already there?
• 3:18
  SPEAKER_00: Uh yeah, uh I think so.
• 3:21
  SPEAKER_02: Oh okay, okay.
• 3:22
  SPEAKER_02: So actually, I'm just gonna keep that.
• 3:23
  SPEAKER_02: I I have a question for you.
• 3:25
  SPEAKER_02: But uh first thing first, uh can you tell us a little bit more about the data stack that you have today and uh where does Airflow fit in?
• 3:32
  SPEAKER_00: Okay.
• 3:33
  SPEAKER_00: On the current project, uh our stack has several main components.
• 3:37
  SPEAKER_00: Uh we use Snowflake as a data source.
• 3:40
  SPEAKER_00: Uh then we load and organize data in MongoDB uh using medallion architecture, brose, silver, and gold layers.
• 3:48
  SPEAKER_00: Uh we use Spark for transformations and processing, uh, Airflow for orchestration, uh for monitoring and error handling also, and uh Elasticsearch for storing uh reports that we are used for application later.
• 4:07
  SPEAKER_00: And uh Airflow is basically uh, as I said, control uh center of the system.
• 4:11
  SPEAKER_00: It decides when each ETL process starts, in which order task runs, um, uh what happens if uh something fails, and when notifications or alerts should be sent.
• 4:24
  SPEAKER_00: And without Airflow, the pipeline would be very hard to control.
• 4:28
  SPEAKER_00: And because we had many, many dependencies between processes.
• 4:32
  SPEAKER_00: And it's not just uh run a spark job.
• 4:35
  SPEAKER_00: Uh the whole flow has to be happened in the correct order.
• 4:40
  SPEAKER_02: Okay, well, actually that that that creates a bridge to the my next my next question, um, which is you just mentioned that that you are using Airfo for orchestrating your ETL um jobs, maybe more than just ETLs.
• 4:52
  SPEAKER_02: Do you have like any like maybe uh other, you know, um like maybe machine learning use cases, things like that also?
• 4:59
  SPEAKER_00: Or is it just like uh no this no, just ETL in this project, yeah.
• 5:04
  SPEAKER_02: Okay.
• 5:04
  SPEAKER_02: And you you just mentioned that dependencies, I mean, uh you definitely have to execute things in the right order, um, which is definitely what Airflow is made for.
• 5:13
  SPEAKER_02: But uh can you tell us a little bit more about you know the ETL uh processes that you have today?
• 5:20
  SPEAKER_02: Um and you know why do you actually require this sequence order that it is you know so important for you?
• 5:27
  SPEAKER_02: Because I think you also have a way to do uh you know uh things that are interesting.
• 5:33
  SPEAKER_00: Uh yeah, this was actually one of the points where things become really interesting and honestly a bit uh painful at the beginning, uh, because at first we tried to uh solve it using the standard airflow options.
• 5:47
  SPEAKER_00: Uh we looked at configurations, uh dog run limits, uh we uh tried to use pools, different uh dependency setups, and so on.
• 5:57
  SPEAKER_00: Basically everything that seems uh like it could help, but uh every solution uh solved just one part of the problem that we had.
• 6:05
  SPEAKER_00: And at the same point, in some point we were stuck and uh not completely desperate, but uh close because we really realized that the problem was bigger than just how we uh stopped two DUG from running at the same time.
• 6:19
  SPEAKER_00: And the real question was here uh how do we control the full execution flow across multiple DUGs, especially when some of them can be triggered at any uh moment from the application side?
• 6:31
  SPEAKER_00: And uh we had to take a step back, sit down, and literally draw the whole process uh end-to-end, uh, which DAGs uh uh which DAGs uh exist, uh, what they um touch, which collection uh they use, which layers they write to, and what could go wrong if two processes uh overlap.
• 6:52
  SPEAKER_00: And the main reason for strict sequential execution was that uh for uh was data consistency.
• 6:58
  SPEAKER_00: Uh all of these DUGs were used to the same MongoDB collections at the same layers.
• 7:05
  SPEAKER_00: And in the bronze layer, uh data could be overwritten by a new snapshot.
• 7:10
  SPEAKER_00: In silver, we had cleaned and prepared data.
• 7:13
  SPEAKER_00: In gold, we had aggregated data used by the applications, uh application reports.
• 7:19
  SPEAKER_00: And so if the two DUs run in the same time and change the same collection or layer in layers, we could uh easily end up with collision uh in uh inconsistency data or partially processed uh results.
• 7:33
  SPEAKER_00: And that's why uh the sequential order uh execution was not just nice to have, it was absolutely necessary for our case.
• 7:43
  SPEAKER_02: Wow.
• 7:43
  SPEAKER_02: So now I I need to learn more about like how did you solve that?
• 7:47
  SPEAKER_02: Um because I'm I'm sure that it's a very, very common issue uh you know in the in the like for many engineers.
• 7:53
  SPEAKER_02: Like how did you what did you try?
• 7:56
  SPEAKER_00: Oh uh before building a custom solution, uh we first tried to use uh waterflow already have.
• 8:02
  SPEAKER_00: Uh we started with the Dug level uh configuration, of course, like limiting the number of uh active DAG run.
• 8:10
  SPEAKER_00: Uh that helped a little because the same DUG could not run multiple times in parallel.
• 8:16
  SPEAKER_00: That was great, but it didn't uh solve the biggest problem here.
• 8:21
  SPEAKER_00: Different DAGs could still run at the same time.
• 8:23
  SPEAKER_00: And then we uh looked at pools also.
• 8:26
  SPEAKER_00: Pools are useful when you have to limit some many tasks, but in our case uh it wasn't a problem.
• 8:32
  SPEAKER_00: And uh uh our problem was uh on a higher level, we needed to control the execution order uh of entire DAGs, not just individually task or individually DAG.
• 8:43
  SPEAKER_02: And the biggest challenge, yeah, can you No, I was I was about to say so, yeah, it's it's really like you you actually want to have multiple DAGs that need to run one after the other.
• 8:53
  SPEAKER_02: It's not because, like you know, for people that that are uh wondering about pools, uh pools allow you to um you know control the the parallelism of your tasks.
• 9:02
  SPEAKER_02: So basically, if you want to execute one task after another, you can do that with pool and then you set one slot to that pool.
• 9:08
  SPEAKER_02: But you will have to attach uh all the tasks to that pool, but still it doesn't solve truly the the issue that uh now you have Ivana uh with the the DAGs that should not run, multiple DAGs that should not run at the same time.
• 9:22
  SPEAKER_02: Yeah, sorry.
• 9:24
  SPEAKER_00: Yeah, it was the problem here.
• 9:26
  SPEAKER_00: And the biggest challenge, as I said, uh was that the some DAGs are scheduled and uh the other DUG was even uh event-driven.
• 9:34
  SPEAKER_00: And uh that event-driven DUG uh could be triggered at any moment from the application.
• 9:39
  SPEAKER_00: And that's why we need to have uh control Q uh and uh uh and we need to have uh uh the right order when uh one DUG uh runs and so on.
• 9:53
  SPEAKER_00: So standard airflow configuration was not enough for us.
• 9:57
  SPEAKER_00: We needed something uh like control Q, as I said, that could decide which process was allowed to run next.
• 10:03
  SPEAKER_00: And that's why we built a custom orchestration layer for that.
• 10:08
  SPEAKER_02: Okay, so tell me tell me more about this uh this um this uh layer that you that you built.
• 10:16
  SPEAKER_00: Yeah, in the end we created a three-layer DAG architecture.
• 10:20
  SPEAKER_00: Uh the first layer is the scheduler or trigger DAGs.
• 10:24
  SPEAKER_00: Uh they are the entry point into the system.
• 10:27
  SPEAKER_00: Uh some are scheduled, for example, for daily imports, uh, and some one uh some are event-driven, triggered by uh user applications uh from application side.
• 10:38
  SPEAKER_00: And their job is not just to run the full uh ETL process directly.
• 10:43
  SPEAKER_00: Their job is to register a request and say this process needs to be executed.
• 10:48
  SPEAKER_00: And the next one, the second layer, is the proxy DAG.
• 10:52
  SPEAKER_00: And this uh here we uh get the magic.
• 10:56
  SPEAKER_00: This is the most important part of the solution because the proxy DUG controls the queue and decides which process can run next.
• 11:04
  SPEAKER_00: Uh, it checks if there is already an active uh DUG, uh, what is waiting uh in the queue, what has priority, or whether the next process is allowed to start.
• 11:15
  SPEAKER_00: And so instead of every DUG decide independently when to run, everything needs to go through the proxy layer.
• 11:23
  SPEAKER_00: And the proxy layer is the main uh brain here.
• 11:28
  SPEAKER_00: Uh the third layer is the main ETL DUGs.
• 11:31
  SPEAKER_00: Um, these DUGs contain the actual logic, data import transformation, validation, and so on.
• 11:38
  SPEAKER_00: And these design help us uh uh uh separate uh responsibilities, uh trigger DAG, as I said, initiate uh requests, the proxy controls uh execution order, and main DUGs go to the actual processing.
• 11:53
  SPEAKER_00: And basically we solve uh our problem with this with this architecture.
• 12:00
  SPEAKER_02: This is this is super interesting.
• 12:02
  SPEAKER_02: Um I have a few questions.
• 12:03
  SPEAKER_02: Um so the the um what exactly is this queue?
• 12:08
  SPEAKER_02: Like uh technically speaking, what is this queue?
• 12:10
  SPEAKER_02: Is it like uh Kafka queue?
• 12:12
  SPEAKER_02: Is it like like what is it?
• 12:14
  SPEAKER_00: It's uh DAG queues.
• 12:16
  SPEAKER_02: Okay, okay, okay, I see.
• 12:17
  SPEAKER_02: Um and uh uh so those those layers are they like how they are built?
• 12:24
  SPEAKER_02: Are they like all like like what is exactly a layer?
• 12:27
  SPEAKER_02: Is it like a DAG?
• 12:29
  SPEAKER_00: Yeah, yeah, everything yeah, it's DAG.
• 12:33
  SPEAKER_00: Yes.
• 12:34
  SPEAKER_00: We have uh scheduler, we have several scheduler DAGs, and we have just one proxy DAG, and we have several uh uh main DAGs.
• 12:44
  SPEAKER_02: Okay, I see, I see.
• 12:45
  SPEAKER_02: Because like I'm I'm just wondering because you know some some people may think that it's it might be like uh you know a plugin that you build or uh you know a Python scripts uh Python script that you run.
• 12:55
  SPEAKER_02: I think like you know, I mean we we also uh we also talked a little bit more about I mean previously uh before this uh this episode uh we discussed a little bit more about how you you know how your um your DAGs are uh um uh written and you know how they are generated and so on.
• 13:13
  SPEAKER_02: Uh I think you have some DAGs that have a lot of tasks.
• 13:16
  SPEAKER_02: Um and I was just wondering like how do you handle that?
• 13:21
  SPEAKER_02: Like how do you handle the creation of your DAGs, I would say?
• 13:24
  SPEAKER_02: Uh do you use like you know, uh a Python script to generate some of your DAGs?
• 13:29
  SPEAKER_02: Do you have like some templates?
• 13:31
  SPEAKER_02: Do you have you know dynamic loops, whatever?
• 13:33
  SPEAKER_02: Like can you tell us a little bit more of how you author those DAGs?
• 13:37
  SPEAKER_02: Because I imagine that you have all of those layers in place, I mean those three layers in place, but I imagine that your DAGs are really standardized as well, so that they can actually run.
• 13:46
  SPEAKER_00: Yeah, of course.
• 13:47
  SPEAKER_00: Uh we have uh DAGs with uh approximately 200 tasks, and of course, we definitely did not write uh 200 tasks manually.
• 13:57
  SPEAKER_00: Uh that will be impossible to maintain.
• 13:59
  SPEAKER_00: And the idea was to make the DAG structure usable and configuration driven.
• 14:05
  SPEAKER_00: And each main DAG has several logical uh sections here.
• 14:10
  SPEAKER_00: On the top level, it looks like uh it looks like uh much simpler, maybe around 30 or 40 main steps, but inside those steps we use task groups.
• 14:22
  SPEAKER_00: And it's in uh inside of task groups, we have more task groups.
• 14:27
  SPEAKER_00: And on that way, we create dynamically generated tasks here.
• 14:32
  SPEAKER_00: Uh uh when everything is expanded, we get around 200 tasks per DAG.
• 14:37
  SPEAKER_00: Yeah.
• 14:38
  SPEAKER_00: And also we use configuration file, define what uh changed between different DAG versions.
• 14:44
  SPEAKER_00: For example, which collections are used, which steps are active, uh, which parameters are needed, uh, how monitoring works, and so on.
• 14:53
  SPEAKER_00: Uh, also versioning uh was very important for us.
• 14:56
  SPEAKER_00: And after every release or hotfix, we increase the DAG version.
• 15:00
  SPEAKER_00: And DAG version, that number is related with the release number, and uh that helps us uh keep a clean and controlled pipeline without new logic uh affecting already running processes.
• 15:14
  SPEAKER_00: And we solve that on that way.
• 15:17
  SPEAKER_00: And with large DUGs like this, uh monitoring is crucial.
• 15:21
  SPEAKER_00: It's not enough to know uh what some uh uh that something uh failed.
• 15:26
  SPEAKER_00: You need to know uh where it failed and why it failed and uh whether you can safely restart only that part.
• 15:34
  SPEAKER_00: That was the logic behind all this.
• 15:38
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 15:46
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 15:51
  SPEAKER_01: Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 15:57
  SPEAKER_01: Check it out at astronomer.io.
• 16:01
  SPEAKER_02: Okay, um, and so this this uh those configuration files are they like YAML files?
• 16:06
  SPEAKER_00: Uh yeah, I use YAML file for the configuration file.
• 16:09
  SPEAKER_02: Great.
• 16:10
  SPEAKER_02: So you actually mentioned task groups, um, which I think is uh definitely an interesting feature.
• 16:14
  SPEAKER_02: It's a simple one, but it's uh actually a very useful one because to your point, when you have like hundreds of tasks, uh sometimes thousands of tasks, uh grouping your tasks together might be a good idea, um especially if you don't want like to actually almost crash the UI, uh or at least it's more it you won't be able to navigate through your DAG because there there will be way too many tasks.
• 16:37
  SPEAKER_02: Um but before task groups you were not using them, obviously.
• 16:42
  SPEAKER_02: You were some using something different, which I think uh was the sub DAGs.
• 16:47
  SPEAKER_02: Uh and subdags were very popular.
• 16:49
  SPEAKER_02: Can you tell us a little bit more of how how did you migrate from the sub DAGs to the task groups?
• 16:56
  SPEAKER_02: Um, because task groups are easier to use than sub DAGs, but you know, I think like many people actually had to migrate, and you are the first person that finally will tell me how they did that migration.
• 17:07
  SPEAKER_00: So yeah, yeah.
• 17:08
  SPEAKER_00: Uh subdags were useful for us because they allowed us to split one large ETL process, one large uh DAG into logical parts, as I said before.
• 17:18
  SPEAKER_00: For example, one part could be import data, another transformations, another validation, and so on.
• 17:24
  SPEAKER_00: If only one part failed, we could retry just part instead of rerunning the whole DUG.
• 17:31
  SPEAKER_00: That was the point here.
• 17:33
  SPEAKER_00: And when subdugs became deprecated, we had two main options.
• 17:37
  SPEAKER_00: The first one was to replace each subdugs with a separate DAG.
• 17:41
  SPEAKER_00: Uh, from retry perspective, uh, that made sense.
• 17:45
  SPEAKER_00: But uh each logical part could be triggered or retried independently.
• 17:50
  SPEAKER_00: And however, for our case, we would lose the visibility of the full pipeline in one place and introduce more complexity around uh dependencies, trigger triggering logic, uh status tracking, error handling, and so on.
• 18:04
  SPEAKER_00: Since we already had strict functional execution uh requirements, splitting everything into many separate DAGs would make the system harder to monitor and control.
• 18:14
  SPEAKER_00: And that was the main reason why we moved to task groups.
• 18:20
  SPEAKER_00: And task groups uh are better for organizing uh tasks, uh visualization inside one DAG, but they do not have behavior exactly like sub DAGs.
• 18:30
  SPEAKER_00: And that was the first challenge here.
• 18:33
  SPEAKER_00: So the main thing we lost was that availability to retry a whole group as one logical unit.
• 18:40
  SPEAKER_00: So we built custom retryal logic on the top of task groups.
• 18:45
  SPEAKER_00: In Airflow 2.9, uh, we solved this through the metadata uh data DB uh based on the task group identifier.
• 18:53
  SPEAKER_00: Uh we knew which task instance belongs to that group.
• 18:57
  SPEAKER_00: So we needed to retry the group, uh, our logic uh restart the state of the those um task instances so Airflow could run that part again.
• 19:09
  SPEAKER_00: Uh the important thing uh uh is that uh this uh this was not a random manually database update.
• 19:16
  SPEAKER_00: It was control logic inside our orchestration layer.
• 19:20
  SPEAKER_00: So that gave us the same practical benefit as before.
• 19:24
  SPEAKER_00: Now we had uh now we have uh task groups, but with a retry mechanism as subdag.
• 19:30
  SPEAKER_02: Which is amazing, right?
• 19:32
  SPEAKER_02: I mean I think like many many people actually had this issue when task groups um you know um uh came to uh I don't remember the the Airflow uh version, but when they actually replaced the sub DAGs, like people were like, okay, this is great, but also like now I cannot retry my task group.
• 19:49
  SPEAKER_02: I have to retry like each task one by one.
• 19:51
  SPEAKER_02: I I don't want to do that.
• 19:52
  SPEAKER_02: And so I think like many people actually did the exact same solution that you just described.
• 19:57
  SPEAKER_02: But in Airflow 3, I believe this is not possible anymore, right?
• 20:02
  SPEAKER_02: Yeah, of course.
• 20:03
  SPEAKER_02: How did you solve again this uh this you know with with Airflow 3?
• 20:08
  SPEAKER_00: Again, unexpected behavior.
• 20:10
  SPEAKER_00: Uh with the Airflow 3, we had to change this approach because relying directly on the metadata dB was no longer the right direction because of the new architecture, Airflow architecture.
• 20:23
  SPEAKER_00: So we moved to uh the retry logic to the Airflow REST API.
• 20:27
  SPEAKER_00: Now instead of directly changing the metadata dbs, db tables, uh we communicate with the airflow through the officially uh official uh APIs.
• 20:40
  SPEAKER_00: Uh now we need to control a dog run.
• 20:43
  SPEAKER_00: When we need to control DAG run uh task instance or anything else, uh or retry behavior, uh we do that through uh supported endpoints.
• 20:52
  SPEAKER_00: And this is better for several reasons.
• 20:55
  SPEAKER_00: The first one is more uh stable across Airflow versions, because uh if uh you rely on internal database table, something can change after an upgrade, as we had um experienced in previous uh versions, the Airflow versions.
• 21:11
  SPEAKER_00: And the second, uh, it's more secure.
• 21:14
  SPEAKER_00: Uh also I think that uh the architecture is clear uh clearer now, and Airflow remains responsible for its own uh internal uh state, and we interact with with uh with it through the official uh API layer.
• 21:29
  SPEAKER_00: Uh so finally it is easier to maintain and uh maintain and explain uh to the new team member uh also.
• 21:37
  SPEAKER_00: So in Airflow 2.9, our metadata db solution worked, but with Airflow 3, we modernize it basically.
• 21:46
  SPEAKER_02: Yeah, and I think this is um, you know, like for people that are wondering, so in Airflow 3 now you cannot access the Airflow metadatabase directly anymore.
• 21:55
  SPEAKER_02: Um for a good reason, like for security reason.
• 21:57
  SPEAKER_02: Like honestly, uh I mean it was good, it was good for us, but it was also a little bit hacky, right?
• 22:03
  SPEAKER_02: You you should not uh you should not expect that your DAGs could access the Airflow meta database, so that's why this has been changed.
• 22:10
  SPEAKER_02: Um but it's to your point, it's way more elegant uh to use the the the Airflow Rest API, uh especially as in Airflow 3, I think you know.
• 22:19
  SPEAKER_02: I mean basically they keep adding new endpoints so you can do more and more things with the Airflow SPI, it's much better than it used to be.
• 22:26
  SPEAKER_02: Um so I'm I'm glad that you overcame this issue again.
• 22:30
  SPEAKER_02: I hope that in Airflow 4 you won't have to come up with uh something else.
• 22:36
  SPEAKER_02: Um awesome, thank you so much, Ivana.
• 22:38
  SPEAKER_02: Uh time flies.
• 22:39
  SPEAKER_02: We are already uh I think it's already uh already uh three 30 minutes.
• 22:48
  SPEAKER_02: I have to ask you first, um what is the best way for folks to get in touch with you?
• 22:54
  SPEAKER_02: You know, if they have any questions regarding your solution and again like all of the common issues that you uh actually um solved.
• 23:02
  SPEAKER_00: Okay, thank you.
• 23:03
  SPEAKER_00: First, uh thank you for having me.
• 23:05
  SPEAKER_00: It was really great talking about this.
• 23:07
  SPEAKER_00: And the best way to get in touch with me is through LinkedIn, of course.
• 23:11
  SPEAKER_00: And I'm I'm always happy to connect with the people uh with the same topics and so on.
• 23:17
  SPEAKER_00: So I want to add something.
• 23:19
  SPEAKER_00: Uh I think these kind of challenges are very interesting uh because they show the orchestration is not just only for scheduling jobs, and uh, Airflow is not just for that.
• 23:29
  SPEAKER_00: Uh there are always unexpected situations where you have to learn something new and you know you need to adapt and find a practical solution.
• 23:38
  SPEAKER_00: So for me, uh that part of the work uh was really interesting.
• 23:43
  SPEAKER_00: And yeah, uh everyone can feel free to reach out on LinkedIn.
• 23:47
  SPEAKER_00: I'll be happy to connect.
• 23:50
  SPEAKER_02: That's awesome.
• 23:50
  SPEAKER_02: Well, first thing first, uh Ivana, I have to thank you for uh for being here.
• 23:55
  SPEAKER_02: Uh so thank you so much again for joining this episode.
• 23:58
  SPEAKER_02: It was a wonderful one.
• 23:59
  SPEAKER_02: I'm very happy uh about everything you said.
• 24:02
  SPEAKER_02: I think like people will enjoy all the insights that you just gave.
• 24:05
  SPEAKER_02: Um and well, I hope to see you again for another one.
• 24:08
  SPEAKER_02: Who knows?
• 24:09
  SPEAKER_02: But otherwise, I wish you a wonderful day.
• 24:11
  SPEAKER_02: Take care and soon.
• 24:14
  SPEAKER_01: Thank you for listening to the data flowcast.
• 24:17
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 24:21
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 24:26
  SPEAKER_01: And leave us a five-star review.