Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta

Transcript

Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: From my point of view, I I believe Airflow is really a rich framework.
• 0:04
  SPEAKER_02: And I believe Airflow is leader on the market with different orchestrators.
• 0:09
  SPEAKER_02: And uh when I I am using some tool, usually I don't think about what I want to see in this framework because I know what the framework can do, and I'm trying to use it somewhere in my architecture.
• 0:24
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:34
  SPEAKER_00: Airflow users, we have exciting news.
• 0:38
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:41
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow.
• 0:44
  SPEAKER_00: And as a thank you for filling it out, you'll get a free Airflow Fundamentals or DAG Authoring certification.
• 0:50
  SPEAKER_00: Those are worth$150 each.
• 0:53
  SPEAKER_00: Plus, you'll also be entered to win an exclusive workshop with Mark Lumberti.
• 0:57
  SPEAKER_00: So make sure you don't miss out.
• 0:59
  SPEAKER_00: The survey link is in the show notes and it closes December 20th.
• 1:04
  SPEAKER_00: Hi everyone.
• 1:05
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:07
  SPEAKER_00: I'm your host, Kenton Dennis, and today I'm joined by Vasile Vazuta, who is an RD team leader at Optimove.
• 1:15
  SPEAKER_00: Vasile, how are you?
• 1:17
  SPEAKER_00: Hi, thank you.
• 1:18
  SPEAKER_02: How are you?
• 1:19
  SPEAKER_00: Good, good.
• 1:20
  SPEAKER_00: I'm good.
• 1:21
  SPEAKER_00: Yeah, it's been a while since we chatted.
• 1:23
  SPEAKER_00: Are you gearing up for the holidays or a little forward?
• 1:29
  SPEAKER_02: I'm a hardworking person, so Okay.
• 1:32
  SPEAKER_00: Airflow all the time.
• 1:34
  SPEAKER_02: Yeah, almost.
• 1:35
  SPEAKER_00: Got it, got it.
• 1:36
  SPEAKER_00: Love to talk about that.
• 1:37
  SPEAKER_00: So I guess we'll go ahead and dive in.
• 1:39
  SPEAKER_00: So for our listeners for today's podcast, we're going to discuss how Optimove uses Airflow to power their data processing and machine learning models.
• 1:48
  SPEAKER_00: Let's start with a bit of background about Optimove.
• 1:51
  SPEAKER_00: So I understand that you are a customer-led marketing platform.
• 1:55
  SPEAKER_00: Tell me more about what that means and what Optimove does.
• 1:59
  SPEAKER_02: Yeah, sure.
• 2:00
  SPEAKER_02: Since we are a market market platform, we are sending marketing notifications to customers of our clients, which we are calling brands.
• 2:09
  SPEAKER_02: And why we are a customer-led platform?
• 2:12
  SPEAKER_02: Because we deeply believe that to reveal all the power of marketing, you should think about your customer first.
• 2:19
  SPEAKER_02: Namely, you you should create a specific and original journey for each of your customers, depending on their preferences, on their personality, etc.
• 2:29
  SPEAKER_02: etc.
• 2:30
  SPEAKER_02: as their behavior.
• 2:31
  SPEAKER_02: And yeah, this is what we are exactly doing.
• 2:34
  SPEAKER_02: Speaking technically, so this is like a business aim, but speaking technically, how it's looking like on high level, and each stage, at each stage of uh our processing pipeline, we are splitting users into different groups regarding their uh preferences, some personality or behavior.
• 2:54
  SPEAKER_02: And this is how we create original, original journey for each customer.
• 3:01
  SPEAKER_00: Okay, interesting.
• 3:02
  SPEAKER_00: So I imagine that grouping is very data-driven.
• 3:05
  SPEAKER_00: So we'll get into that.
• 3:06
  SPEAKER_00: And maybe related to my next question.
• 3:08
  SPEAKER_00: So, what is your role as the RD team leader?
• 3:11
  SPEAKER_00: What does your team do for the business?
• 3:14
  SPEAKER_02: Optimum is a big product, and we have a lot of responsibilities there.
• 3:18
  SPEAKER_02: And our responsibility is to find the best campaign for each customer.
• 3:23
  SPEAKER_02: What it means?
• 3:24
  SPEAKER_02: Campaign of it's like a rule what to send and to whom send and when.
• 3:30
  SPEAKER_02: And for our biggest tenants, we have thousands of campaigns for all customers.
• 3:35
  SPEAKER_02: It can be even five thousand different campaigns.
• 3:38
  SPEAKER_02: And we should split these campaigns to all customers.
• 3:42
  SPEAKER_02: And of course, that if since you have uh uh tens and even hundreds of millions of customers, that statistically there will be customers that will be eligible to hundreds of campaigns or even thousands of campaigns.
• 3:54
  SPEAKER_02: Of course, that bombardying them with such amount of emails or SMS or push notifications will be a nightmare for users.
• 4:02
  SPEAKER_02: And our aim, our goal is to find uh the best campaign for each customer.
• 4:10
  SPEAKER_00: Okay, that's great.
• 4:11
  SPEAKER_00: And so, where does Airflow fit into that?
• 4:14
  SPEAKER_00: I know you use Airflow for your data orchestration, I'd imagine.
• 4:18
  SPEAKER_00: Like you said before, that a lot of that grouping of customers is data driven.
• 4:22
  SPEAKER_00: So tell me a little bit more about how that works.
• 4:25
  SPEAKER_02: Yeah, exactly.
• 4:26
  SPEAKER_02: And here it is a point where Airflow is one of our main tools.
• 4:31
  SPEAKER_02: As I told, we have really big data because we have uh hundreds of millions of customers.
• 4:37
  SPEAKER_02: And of course, ordinary relational database cannot serve such a lot.
• 4:42
  SPEAKER_02: We're using Snowflake, we write storage procedures in Snowflake, but stored procedure is good, but you need to orchestrate them somehow.
• 4:50
  SPEAKER_02: And in this point, Airflow is a really uh nice orchestrator.
• 4:55
  SPEAKER_02: Honestly, when we were choosing something to orchestrate, some tool, we uh make quite intense research of different orchestrators that are on the market.
• 5:04
  SPEAKER_02: Uh in what we found, and we are deeply we deeply believe that Airflow is a leader on market right now.
• 5:11
  SPEAKER_02: So that's why we started to use Airflow.
• 5:14
  SPEAKER_02: Uh backing to uh to like our architecture, yeah.
• 5:16
  SPEAKER_02: So what we have Airflow, we have uh tens of DACs, to be honest, because we have a lot of possibilities, and each DAC is doing something, uh its own job.
• 5:26
  SPEAKER_02: And mainly DAC orchestrates stored procedures.
• 5:29
  SPEAKER_02: Uh we have uh DOCs with complicated topology, we have branching, we have a lot of things, and also created uh some infrastructure things that are really helpful for us.
• 5:41
  SPEAKER_02: Uh for example, we have something uh which is called joystick concept, namely we can uh move back and forth uh in time and replay all the data that we processed.
• 5:52
  SPEAKER_02: And this is like a plugin for Airflow and it is converted to Snowflake storage procedure.
• 5:58
  SPEAKER_02: And the developers who is creating the storage procedure, he even didn't think about it because it is fully powered by Airflow.
• 6:06
  SPEAKER_02: It's a nice thing.
• 6:07
  SPEAKER_02: Also, we have something like feature clext support and other uh cool things that is integrated deeply with Airflow.
• 6:17
  SPEAKER_00: Cool, that's really interesting.
• 6:19
  SPEAKER_00: So you have uh the plugin on Airflow exists so that not all of your users have to actually develop their own DAGs, they just write the stored procedures in Snowflake, and then the DAG creation is going on behind the scenes for them.
• 6:33
  SPEAKER_02: Some part, some part, because yeah, definitely need to define which storage procedure will be run after each.
• 6:40
  SPEAKER_02: And this is something that you cannot avoid to create, but there is some infrastructure that is added behind the scene, and you shouldn't think about it because it is out of the box, something like that.
• 6:52
  SPEAKER_00: Got it, got it, interesting.
• 6:54
  SPEAKER_00: And so, in terms of what the DAGs are actually doing, I think I understand from speaking with you before that you're orchestrating some machine learning models, and those might help with the grouping of your customers.
• 7:05
  SPEAKER_00: Is that the case?
• 7:07
  SPEAKER_02: Yes, so machine learning models is part of some part uh part of that.
• 7:12
  SPEAKER_02: We started development of machine learning models something like a year ago, at least our our team and our group, because we have the uh team of data scientists within our company, but they are working on separate uh features and they are not doing execution like we do.
• 7:30
  SPEAKER_02: And what we are doing exactly about machine learning model.
• 7:34
  SPEAKER_02: If you need to uh predict how each customer will behave in the future, for example, we we we know how customer was behaving uh one week ago during the past one week, uh and we know all the history how customers uh were behaving for the whole brand.
• 7:53
  SPEAKER_02: And we uh made something like Rainforest Learning, and uh based on data, we can predict what will be with each specific customer uh in uh in the next week, for example, uh when he will buy something, uh when he will churn potentially, or something like that.
• 8:10
  SPEAKER_02: Uh of course we cannot say that uh we know what a specific customer will do, but but statistically uh we have very nice precision.
• 8:20
  SPEAKER_02: And this exactly helped help us uh uh to understand uh when and to whom it's better to send campaign.
• 8:28
  SPEAKER_02: Because we have when I'm talking campaign at the beginning of our meeting, I thought that this is like a rule to whom, when, and what to send.
• 8:36
  SPEAKER_02: But in fact, we have really mature and seasoned business layer of that.
• 8:42
  SPEAKER_02: And under campaign, it's not only a rule, we have different A B testing there, we have out of priority and a lot of stuff there.
• 8:49
  SPEAKER_02: It's quite a complicated machine.
• 8:51
  SPEAKER_02: And yeah, this like our machine learning models, they help us to understand what will be this customer and what exactly campaigns should be sent to some customers at least.
• 9:04
  SPEAKER_02: We are running it ad hoc, namely, we have our process of campaign execution, let's say.
• 9:11
  SPEAKER_02: And when we run this process, as part of this process, we have this reinforced learning because this model it's relatively simple.
• 9:21
  SPEAKER_02: Runs of each model requires from five minutes to half an hour time.
• 9:27
  SPEAKER_02: That's why we can add this learning into our pipeline.
• 9:33
  SPEAKER_00: Yeah, interesting.
• 9:34
  SPEAKER_00: And so you use Airflow to orchestrate all that.
• 9:38
  SPEAKER_00: We've seen a lot more airflow use for kind of this MLOps use case in the last couple of years.
• 9:44
  SPEAKER_00: What made you choose Airflow for this use case?
• 9:48
  SPEAKER_00: I said it sounds like you used Airflow already previously, and then about a year ago, this was a new use case.
• 9:54
  SPEAKER_00: So I assume you already had that Airflow infrastructure set up.
• 9:58
  SPEAKER_02: Yes, exactly.
• 9:59
  SPEAKER_02: And we have a lot of infrastructure for Airflow, and we also have a lot of infrastructure from DevOps side.
• 10:06
  SPEAKER_02: We have robust pipelines with different unit testing, with integration testing, etc.
• 10:13
  SPEAKER_02: etc.
• 10:13
  SPEAKER_02: So yeah, we have a lot of stuff.
• 10:16
  SPEAKER_02: And our ops, for example, they also know how to work with Airflow.
• 10:21
  SPEAKER_02: They can read logs, they can make rollbacks by ourselves.
• 10:26
  SPEAKER_02: And that's why Airflow became a part of our infrastructure, not only uh technical infrastructure, but also from the point of view of Opsis or reliability team, et cetera, et cetera, which is also very important.
• 10:39
  SPEAKER_00: Yeah, yeah, that makes sense.
• 10:41
  SPEAKER_00: And so you your team works very closely with your data science team, who you said is doing more of the feature development and less of the actual writing of the pipelines.
• 10:52
  SPEAKER_00: Is that right?
• 10:54
  SPEAKER_02: Now, for the last couple of months, we have our own data scientists.
• 11:00
  SPEAKER_02: In the future, we we expect more machine learning and airflow.
• 11:04
  SPEAKER_00: Got it.
• 11:05
  SPEAKER_00: So a growing use case then.
• 11:08
  SPEAKER_02: Because it brings benefits, it gives through it.
• 11:12
  SPEAKER_02: Why not to use it?
• 11:14
  SPEAKER_00: So you talked a little bit already about some customizations that you've made to Airflow.
• 11:19
  SPEAKER_00: And I think I've learned that one other thing that you've done on top of the core Airflow is to add your own form of DAG versioning.
• 11:28
  SPEAKER_00: So tell me a little bit more about that.
• 11:30
  SPEAKER_00: What prompted you guys to make that customization and how does that work?
• 11:36
  SPEAKER_02: Yeah, sure.
• 11:37
  SPEAKER_02: In fact, when we see uh some tool like a framework like Airflow, you're thinking when we can use it and how we can use it, because we know how to use it.
• 11:46
  SPEAKER_02: And as I told previously, we are using it for machine learning models, but it's not all.
• 11:51
  SPEAKER_02: We are using it for data pipelines.
• 11:53
  SPEAKER_02: And we are crazy enough to run it very frequently, uh, what it means frequently.
• 11:59
  SPEAKER_02: Uh, we know that there are uh use cases, there are best practices for using some orchestrators like Airflow for long running tasks, etc.
• 12:08
  SPEAKER_02: etc.
• 12:09
  SPEAKER_02: That data pipeline.
• 12:10
  SPEAKER_02: But our goal is to run each dog for each tenant once per five minutes.
• 12:17
  SPEAKER_02: Uh namely, uh if we have almost 1000 tenant, 1000 brands, and we want to at peak load, we want to execute one 1000 DUGs per five minutes.
• 12:31
  SPEAKER_02: And you can imagine that having such a load and having some issue with deployment, with configuration or with whatever, and uh resolving this issue potentially can take some time.
• 12:45
  SPEAKER_02: And waiting for this time when our uh CICD pipelines run and all these automated tests will will be executed again, namely rollback can as it is can take some time.
• 13:00
  SPEAKER_02: That's why we want to have an ability to quick rollback of our DACs.
• 13:07
  SPEAKER_02: That's why we decided so there are different options.
• 13:10
  SPEAKER_02: Okay, yes, we can just uh take old version of uh old files of DAC and put it in DAX folder.
• 13:17
  SPEAKER_02: Yes, it's true.
• 13:18
  SPEAKER_02: That can be done.
• 13:19
  SPEAKER_02: And what why do we need to add versioning if we can do that if it is so simple?
• 13:25
  SPEAKER_02: But there are other restrictions.
• 13:27
  SPEAKER_02: For example, I mentioned that we have a reliability team, and we want uh our reliability team to be able to do that.
• 13:35
  SPEAKER_02: We also uh want it to be like automatic to minimize uh human errors, and because there is probability that you can take not the correct files and you will break again.
• 13:48
  SPEAKER_02: So something like that.
• 13:49
  SPEAKER_02: And instead of just replacing files, we we started to think about having different versions, but how you can have versions in Airflow.
• 13:59
  SPEAKER_02: So definitely you need to change uh name.
• 14:02
  SPEAKER_02: So we are adding, we have like uh doc name, and we add some suffix version one, version two, etc.
• 14:08
  SPEAKER_02: etc.
• 14:09
  SPEAKER_02: etc.
• 14:09
  SPEAKER_02: But the problem here is that our docs are triggered externally by other clients, by other docs or other services, and they are calling uh docs by name.
• 14:19
  SPEAKER_02: And how to resolve the issue?
• 14:22
  SPEAKER_02: Okay, we decided to create something like facing duck.
• 14:25
  SPEAKER_02: Something like it is a load balancer, but it's not load balancer, but rather version balancer.
• 14:30
  SPEAKER_02: We created a plugin for other ICD pipelines, and this plugin takes original DUC, it replaces a name, adding versions, and it builds a special facing DOG, which is DAC agnostic, which is business agnostic, which fits any DUC that we we have.
• 14:49
  SPEAKER_02: And uh it deploys different versions of DOGs, uh like my DUC version one, my DUC version two, etc.
• 14:55
  SPEAKER_02: etc.
• 14:56
  SPEAKER_02: But also this facing DOG.
• 14:57
  SPEAKER_02: And this facing DUC knows which uh DOC it should use.
• 15:01
  SPEAKER_02: It patches the same payload, so everything looks the same from external clients, and it is controlled by our version management system.
• 15:10
  SPEAKER_02: And the options they know how to use this version management system, and it's really uh easy for for them.
• 15:17
  SPEAKER_02: Also, there is one more problem that you solve uh with this approach, is this like versioning?
• 15:22
  SPEAKER_02: We can have gradual rollout.
• 15:24
  SPEAKER_02: Imagine that you are rolling out a new tag, new version with a new feature or whatever, and you want to open this feature only to 10% of your tenants or to specific tenants only.
• 15:36
  SPEAKER_02: And you also can do this by using a version management system because uh is this is exactly what this system is intended to do.
• 15:44
  SPEAKER_02: And uh this is like additional benefit that we gain from uh having this facing dog and uh yeah, this is our solution.
• 15:53
  SPEAKER_02: And to summarize how it's looking, so we created a CICD plugin, and uh this plugin hits any dog that we have, and it just in CICD pipeline it just makes some manipulation with files, with remains, and voila, everything is working, and DAOs even don't know what is going on there.
• 16:15
  SPEAKER_02: They are working locally with their local dog without versions.
• 16:18
  SPEAKER_00: Yeah, that's really cool.
• 16:20
  SPEAKER_00: It makes me think of our own data team who we interviewed on the show a couple months ago, has something similar.
• 16:27
  SPEAKER_00: It's not for not to control DAG versioning, but they refer to it as a control DAG that helps provide observability into all of their many DAGs.
• 16:37
  SPEAKER_00: So when there's a failure, they can more easily recover.
• 16:39
  SPEAKER_00: So having this like DAG that kind of sits in the middle that's controlled from outside of Airflow to orchestrate everything is a similar solution.
• 16:48
  SPEAKER_00: Yeah, that's really interesting.
• 16:49
  SPEAKER_00: I can totally see where versioning would be very important in the case where you've got such a high volume with such frequency, thousands of DAGs running every five minutes is I imagine would be very difficult to recover from if something went wrong.
• 17:03
  SPEAKER_00: I'm curious since you mentioned that you're required to change the names of the DAGs for the different version, which I think is a workaround for the fact that Airflow itself doesn't have the DAG version name currently.
• 17:16
  SPEAKER_00: Does that provide or does that cause any issues with maintaining the history of your DAG run?
• 17:22
  SPEAKER_00: So you have your DAG that controls the naming for the triggering.
• 17:27
  SPEAKER_00: But I assume once you change the name, the old DAG is gone, right?
• 17:31
  SPEAKER_00: Or do you maintain them all in the UI or how does that work?
• 17:34
  SPEAKER_02: Yes, nice question.
• 17:36
  SPEAKER_02: And we were thinking about it.
• 17:38
  SPEAKER_02: So we are keeping five versions of DAG, and we can do it in one because this uh this number definitely is a configurable number.
• 17:46
  SPEAKER_02: So and if we have five DAGs, historical five DAX, we have logs for them.
• 17:52
  SPEAKER_02: And after some time, when after some time we are removing DAX, probably history is not important for now.
• 17:59
  SPEAKER_02: And not no issues with log from that perspective.
• 18:02
  SPEAKER_02: But what we experience issue with what with logging, because names is a part of logs, and since we are changing name, name will be different, and you will see different service in Datadog and in any other logging system, but we also solve the issue.
• 18:22
  SPEAKER_02: So, yes, having different names it brings some complexities in other places, but this can default.
• 18:30
  SPEAKER_00: Yeah, gotcha.
• 18:31
  SPEAKER_00: Are you aware of the DAG versioning feature that is being added for Airflow 3.0?
• 18:38
  SPEAKER_02: Yeah, sure.
• 18:39
  SPEAKER_02: We're waiting for for it, and probably we will use it here.
• 18:44
  SPEAKER_02: Okay, so you can link out our trust.
• 18:46
  SPEAKER_02: You'll see you'll see because I'm I I tried to find, but I uh I wasn't able to find the exact documentation, how it will be look like.
• 18:55
  SPEAKER_02: So yeah.
• 18:56
  SPEAKER_00: Yeah, often the AIPs are a little bit vague to give them some wiggle room once they do start the development, but I know um astronomer is leading the development of that feature, and we're really excited about it.
• 19:09
  SPEAKER_00: It's been consistently like the top feature requested on the Airflow survey for the last couple of years.
• 19:16
  SPEAKER_00: So lots of teams really want DAG version name.
• 19:20
  SPEAKER_00: You're the first person I've talked to that has implemented such a sophisticated solution for it, where it seems like it's working pretty well for you now.
• 19:26
  SPEAKER_00: But that will lead me into my last question, which I always ask to everybody who joins us, and that is what is one thing that you would most like to see from the Airflow project in the near future?
• 19:37
  SPEAKER_00: It can be DAG versioning if you want or something else.
• 19:41
  SPEAKER_02: Definitely to DAG version because we needed this.
• 19:45
  SPEAKER_02: But uh also, you know, usually uh I have a lot of interviews.
• 19:49
  SPEAKER_02: I'm looking for uh some uh people into my team, and I like I like to ask questions.
• 19:55
  SPEAKER_02: Tell me about technology and like some uh limitations of this technology.
• 19:59
  SPEAKER_02: And since you are working with Airflow, I'm asking, okay, you you have experience in Airflow, so tell me what you want to have, or uh where did you struggle with something like that?
• 20:11
  SPEAKER_02: And honestly, no one answered something huge or something like that.
• 20:15
  SPEAKER_02: I don't know, maybe not enough experience, but yeah, it's not something like feedback from the people that I interviewed.
• 20:22
  SPEAKER_02: From my point of view, I I believe Airflow is really a rich framework, and I believe Airflow is a leader on the market with different orchestrators.
• 20:31
  SPEAKER_02: And uh when I'm using some tool, usually I don't think about what I want to see in this framework because I know what the framework can do, and I'm trying to use it somewhere in my architecture, something like that.
• 20:42
  SPEAKER_02: But what I was struggling with the most with Airflow, uh, I can say, except versioning, I can say probably two small things.
• 20:50
  SPEAKER_02: Uh, first of all, since we we are trying to run Airflow so frequently, like a thousand times per five minutes, we show some like some struggling, some throttling, something like that on this of the scales.
• 21:05
  SPEAKER_02: And we investigated the issue and we found that POSRES, which is on uh under the hood of Airflow, POSRES is a bottleneck for us.
• 21:14
  SPEAKER_02: So probably yes, probably if it is possible to improve performance on pro POSRES, maybe optimize optimizing the schema or adding some indexes that uh that will be helpful at these scales, so it will really help help us.
• 21:29
  SPEAKER_02: And another thing that I would like to have it's like imagine that you have dog and you have logs.
• 21:37
  SPEAKER_02: And uh logging in uh airflow is uh uh a bit more uh complicated than it is in pure Python, of course, for sure.
• 21:46
  SPEAKER_02: And what we want to have, each doc has some attributes, let's call it.
• 21:50
  SPEAKER_02: Some attributes are coming from parameters, some attributes maybe uh coming from Xcom or whatever.
• 21:56
  SPEAKER_02: Namely, if in our case, uh one of attributes is tenant ID, tenant name, and something like that, which is part of our uh parameters that we pass to Airflow.
• 22:07
  SPEAKER_02: And what we would like to have is to have an ability to configure Airflow to add these parameters as uh log properties, something like that.
• 22:18
  SPEAKER_02: Because we have a message, we can configure how a message can uh look like, but also we we would like to have it like a separate facet in data doc or something like that.
• 22:28
  SPEAKER_02: We found a custom solution, it's working production, uh it's working well.
• 22:32
  SPEAKER_02: It is working for task, but not for dock by itself.
• 22:35
  SPEAKER_02: So log from DACs cannot be uh cannot be enriched with such kind of parameters, but we found a solution with tasks.
• 22:43
  SPEAKER_02: So it is something that will be really helpful for us.
• 22:47
  SPEAKER_00: Okay, good.
• 22:48
  SPEAKER_00: That's a good wish list.
• 22:49
  SPEAKER_00: Very in the weeds.
• 22:50
  SPEAKER_00: I'm a little surprised to hear that the folks you are interviewing.
• 22:52
  SPEAKER_00: I love by the way that this is an interview question for you as well.
• 22:56
  SPEAKER_00: That that they're not giving you a lot of answers.
• 22:59
  SPEAKER_00: Maybe that's indicative of Airflow being fairly mature at this point.
• 23:03
  SPEAKER_00: The last time I used Airflow in production myself was in a much earlier version before I worked for Astronomer, and I would have had a long list for you of this was back in probably, I think I was on 1.9.
• 23:15
  SPEAKER_00: Definitely a long list of requests, but maybe it's gotten a lot better.
• 23:18
  SPEAKER_00: Interesting.
• 23:19
  SPEAKER_02: I believe the first version uh you'll be even more seasoned and mature.
• 23:23
  SPEAKER_02: So yeah.
• 23:24
  unknown: Yes.
• 23:26
  SPEAKER_00: Yeah, we're excited for 3.0 for sure.
• 23:29
  SPEAKER_00: Awesome.
• 23:29
  SPEAKER_00: I know we were about at time, so Vasila.
• 23:31
  SPEAKER_00: It was really great to have you on.
• 23:32
  SPEAKER_00: It was a pleasure talking to you.
• 23:34
  SPEAKER_00: Uh I'll end with what is the best way for folks to get in touch with you.
• 23:38
  SPEAKER_02: Uh my email.
• 23:39
  SPEAKER_02: Yeah, I can provide personal emails.
• 23:41
  SPEAKER_02: So feel free to reach me out uh whenever you need.
• 23:45
  SPEAKER_00: Cool.
• 23:45
  SPEAKER_00: Thanks again for coming on.
• 23:46
  SPEAKER_00: It was really great speaking with you and learning more about Optimove.
• 23:52
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:58
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:02
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.