Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing

Transcript

How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We were wasting the bulk of the resources we we were allocating.
• 0:03
  SPEAKER_02: And the reason is that in order to get pipelines to run, the convenient way you do it is you pick a number that corresponds to what your worst case is, and then everything gets that.
• 0:14
  SPEAKER_02: And if you're doing that and your worst case isn't a very common case, you end up just wasting enormous amounts of resources.
• 0:21
  SPEAKER_02: It's very easy to waste 80 or 90% of what you're paying for.
• 0:26
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:36
  SPEAKER_01: Hello, everyone.
• 0:37
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:39
  SPEAKER_01: I'm your host, Kenton Danis, and today I'm joined by two lovely folks from Vibrant Planet.
• 0:44
  SPEAKER_01: We have Cyrus Dukart, who is an engineering lead, and David Saserdote, who's a staff software engineer.
• 0:50
  SPEAKER_01: How are you both doing today?
• 0:51
  SPEAKER_02: Great, thanks.
• 0:53
  SPEAKER_01: Awesome.
• 0:54
  SPEAKER_01: How are things in the Bay Area?
• 0:56
  SPEAKER_02: Nice.
• 0:56
  SPEAKER_02: Summer and everything's starting to dry out.
• 0:58
  SPEAKER_01: So you know a good thing or a bad thing?
• 1:02
  SPEAKER_02: Both.
• 1:03
  SPEAKER_01: Both.
• 1:05
  SPEAKER_01: We feel bad.
• 1:06
  SPEAKER_01: I'm up here in Seattle in the Pacific Northwest, and we've had definitely a less rainy spring than is typical, which is somewhat nice for a bit of a break.
• 1:14
  SPEAKER_01: But yeah, we're also always a little worried about it for later in the summer.
• 1:17
  SPEAKER_01: So I think we'll talk more about that during this podcast, actually.
• 1:21
  SPEAKER_01: Well, that's great.
• 1:22
  SPEAKER_01: Let's go ahead and dive in.
• 1:24
  SPEAKER_01: So for folks listening for today's podcast, we're going to discuss how Vibrant Planet has implemented self-healing pipelines in Airflow.
• 1:31
  SPEAKER_01: That is something I know a lot of our listeners will be interested in.
• 1:34
  SPEAKER_01: I hear the phrase self-healing come up all the time.
• 1:36
  SPEAKER_01: So super excited to dive into that with you.
• 1:39
  SPEAKER_01: But before we get into all of the nitty-gritty of your Airflow implementation, which again, I think is going to be applicable to a huge range of Airflow users, I want to talk about your use case.
• 1:48
  SPEAKER_01: And this is something I'm particularly interested in, is very close to my heart.
• 1:51
  SPEAKER_01: But Vibrant Planet provides a common operating picture for achieving wildlife and climate resilience.
• 1:58
  SPEAKER_01: That's from your website.
• 1:59
  SPEAKER_01: So can you tell me more about what that means and what is the mission of your organization?
• 2:04
  SPEAKER_02: What that means is that we're looking for ways to reduce wildfire risk.
• 2:12
  SPEAKER_02: And wildfires, the really big ones, have become a lot more common recently.
• 2:17
  SPEAKER_02: And a lot of that is driven by higher temperature and lower humidity and sort of an accumulation of vegetation over time.
• 2:26
  SPEAKER_02: So what we're doing is we are using satellite imagery and geospatial data to create this sort of unified picture of what's growing and where, and what that means for community risk, and what they can actually go do about it to lower the risk.
• 2:47
  SPEAKER_02: Because we're engaged in this large-scale processing of geospatial data, and because a lot of the research was based on little sub-watershed units, we have sharded in ways that allow our results to be considered scientifically valid.
• 3:04
  SPEAKER_02: And that means that our shards are not all the same size and don't all require the same resources to process.
• 3:11
  SPEAKER_02: And that in turn means that we had a problem where when we ran our analysis pipeline, it would say run out of memory on various pieces.
• 3:23
  SPEAKER_02: And then we would have somebody, usually Cyrus, having to get up and adjust memory resource allocation for individual jobs and tasks.
• 3:33
  SPEAKER_02: And this gets old fast.
• 3:36
  SPEAKER_03: So our yeah, our challenge was we were scaling out this massive amount of geospatial processing and having variable resource requirements.
• 3:45
  SPEAKER_03: We entered this kind of tuning hell where you would have to go in and mod individual tasks or individual task instances with a certain amount of memory or CPU requirements.
• 3:58
  SPEAKER_03: And that wasn't scalable.
• 3:59
  SPEAKER_03: And that kept me out the night, literally.
• 4:01
  SPEAKER_01: Yeah, that makes a lot of sense.
• 4:03
  SPEAKER_01: And to sum up, so your use case, I think, is one that will a lot of folks will be able to relate to.
• 4:08
  SPEAKER_01: I mentioned this is an issue very close to my heart.
• 4:10
  SPEAKER_01: I'm a big trail runner.
• 4:11
  SPEAKER_01: Obviously, I live in the Western US, which is much, much more impacted by wildfires than many other areas of the globe and more than we used to be.
• 4:19
  SPEAKER_01: So I think that's really cool to see that use case.
• 4:22
  SPEAKER_01: And obviously, as you've described it, data, especially geospatial data, is very much core to what you do.
• 4:28
  SPEAKER_01: So you started touching on it a little bit, but let's talk more about the challenges that come along with that.
• 4:33
  SPEAKER_01: So you mentioned different scales, obviously, the amount of data that you're dealing with, kind of what exactly was it that was keeping you up at night, Cyrus?
• 4:41
  SPEAKER_01: And then maybe we can talk about how airflow fits into that picture.
• 4:44
  SPEAKER_03: Sure.
• 4:45
  SPEAKER_03: So airflow is our orchestration system here, and we are generally working with larger files.
• 4:51
  SPEAKER_03: So, like we work with uh what's called raster files or vector-based data.
• 4:56
  SPEAKER_03: Those are the outputs in the geospatial arena.
• 5:00
  SPEAKER_03: And like David said, we had this variable partitioning scheme where we're processing variable sizes of the earth, and each of those requires a different set of issues.
• 5:11
  SPEAKER_03: What would happen would be we would run these jobs and they would be fine the day before.
• 5:17
  SPEAKER_03: But then we would go and either our upstream files would change, or we would change like the amount of transform work that we were doing in the middle of the pipeline, and that would require more resources, and and it would just be this forever constantly manually adjusting things.
• 5:34
  SPEAKER_03: And instead of that, we realized like this is just not something that we can throw more people at.
• 5:40
  SPEAKER_03: We really have to come up with a solution that the tech can solve for us.
• 5:44
  SPEAKER_03: So David and I, and another engineer, took a step back and we started looking at it and like what would be the perfect solution for our case right here.
• 5:53
  SPEAKER_03: And it came down to a simple like self-healing in in our minds was self-healing in the concept of resource allocation.
• 6:00
  SPEAKER_03: There's a lot of different self-healing pipeline terminology that's been thrown out.
• 6:03
  SPEAKER_03: So let's just clarify it there.
• 6:05
  SPEAKER_03: We looked at it and said, What if the first task, if we can detect if a task fails due to memory, right?
• 6:12
  SPEAKER_03: How can we make the next task allocate more memory?
• 6:15
  SPEAKER_03: And that's just not, it wasn't something that was available out of the box.
• 6:18
  SPEAKER_03: So we kind of had to dig into the guts of things.
• 6:20
  SPEAKER_03: And it took probably a couple of weeks of just hacking, but we figured out so we use Kubernetes executor.
• 6:27
  SPEAKER_03: There's multiple executors, and with that, you can set pod resources for the machines that you spin up, but you can only set pod resources per task, not per individual task cry.
• 6:37
  SPEAKER_03: Um, so that was the thing that we needed to identify.
• 6:41
  SPEAKER_03: And we found it, it's just a little snippet of code, but we were able to in the retry callback, right?
• 6:48
  SPEAKER_03: It can't happen before a task is run, but on the retry callback of a failed task, we're able to mutate the next pod that comes out, and we're able to say allocate a certain amount of resources.
• 6:58
  SPEAKER_03: So in our minds, if we could do that, then we can try, and if we fail due to memory allocation, we could update, increase the resources on the next try.
• 7:08
  SPEAKER_03: And if that fails, we keep increasing to some reasonable extent until the task succeeds.
• 7:15
  SPEAKER_03: And that was the first concept we came up with.
• 7:17
  SPEAKER_03: And then I'll hand it to David because we can take it a little bit further.
• 7:21
  SPEAKER_02: So the next piece of that is that when you first run a task, you would actually like to get the right amount of memory the first time.
• 7:30
  SPEAKER_02: And so as we're running tasks, we actually have a monitor process that kicks off in the background when you start a task.
• 7:42
  SPEAKER_02: And that monitor process wakes up, I think, every 15 seconds and just looks at how much memory is being used and saves that information and saves peak utilization information in a database at the end.
• 7:57
  SPEAKER_02: And what that means is that when you start a task, you actually have the ability to run a database query that asks for similar tasks, what were their resource needs, and how did those resource needs vary in response to statistics about your input?
• 8:14
  SPEAKER_02: So as you start up a task, you can query the database and look and see if it's run before, here's exactly what my resource needs were.
• 8:22
  SPEAKER_02: And so I like, I'll make sure I have that plus a little buffer.
• 8:26
  SPEAKER_02: And if that plus a little buffer is not available in terms of like you can't load that from the database and say we've run this exact thing before.
• 9:01
  SPEAKER_02: Okay, I'll just run it.
• 9:02
  SPEAKER_02: And if it isn't, okay, I'll just fail fast since I know I'm going to run out of memory part way through running this analysis, anyways.
• 9:09
  SPEAKER_02: I can just fail fast and then set the memory on retry to be the amount I know I'm going to need.
• 9:17
  SPEAKER_02: And so the result of this is that by and large, you get a reasonable choice for the amount of memory for most of our tasks, most of the time.
• 9:26
  SPEAKER_02: And only once in a while depend on, gee, we ran it, it ran out of memory, and then we had to increase it.
• 9:33
  SPEAKER_01: That's great.
• 9:33
  SPEAKER_01: So you've not only implemented logic within your pipeline so that if a test does run out of memory, it will try again.
• 9:41
  SPEAKER_01: So you're eliminating this need for somebody to manually intervene like all the time, every time you have a new data set or something.
• 9:48
  SPEAKER_01: But then you've also recognized that memory needs differ a lot for different data sets, and you don't want to be wasting resources where you don't need to.
• 9:56
  SPEAKER_01: So you're trying to make that initial guess of how much memory you're going to need for any particular task smarter.
• 10:03
  SPEAKER_03: Yes.
• 10:04
  SPEAKER_03: That's great.
• 10:05
  SPEAKER_03: Excuse me.
• 10:06
  SPEAKER_03: Yeah, that's that's what made it cost efficient for us to even go this route because we are in some cases allocating more memory than we need, but it's remembered in the future.
• 10:16
  SPEAKER_03: So we know exactly what we use, and we'll only use that the next time that we rerun a task of similar size.
• 10:22
  SPEAKER_02: Without this system with the combination in there, we were wasting the bulk of the resources we we were allocating.
• 10:29
  SPEAKER_02: And the reason is that in order to get pipelines to run, the convenient way you do it is you pick a number that corresponds to what your worst case is, and then everything gets that.
• 10:40
  SPEAKER_02: And if you're doing that and your worst case isn't a very common case, you end up just wasting enormous amounts of resources.
• 10:47
  SPEAKER_02: It's very easy to waste 80 or 90 percent of what you're paying for.
• 10:51
  SPEAKER_01: Yeah, that's awesome.
• 10:52
  SPEAKER_01: I imagine with Vibrant Planet is a relatively small organization and you have pretty big goals in terms of increasing the analysis you're doing to different areas since they're all a little different.
• 11:03
  SPEAKER_01: I imagine that cost efficiency is really important for your business and the scalability of it.
• 11:08
  SPEAKER_01: Yeah.
• 11:08
  SPEAKER_01: That's awesome.
• 11:09
  SPEAKER_01: You spoke a little bit earlier, Harris, about executor choice and that you're using the Kubernetes executor.
• 11:16
  SPEAKER_01: Is there anything you want to talk more about there in terms of why you made that choice or why was that important for the architecture that you've set up?
• 11:25
  SPEAKER_03: Sure.
• 11:25
  SPEAKER_03: So in Airflow, you have the two choices right now.
• 11:28
  SPEAKER_03: Generally, you have salary and you have Kubernetes.
• 11:31
  SPEAKER_03: For us, Kubernetes made a lot more sense for what we were doing because we are doing a lot of large file processing and the capabilities to tune the resources was something we absolutely needed.
• 11:44
  SPEAKER_03: It just wouldn't have worked if we used a central Celery executor.
• 11:47
  SPEAKER_03: I do know that you can parallelize those and so on, but I don't think that would have fit our use case.
• 11:53
  SPEAKER_03: The Celery Executor is more beneficial if you are really partial towards startup times because there is a spin-up time for every node that we have going on.
• 12:05
  SPEAKER_03: But for our use case, we're generally not even running temporal daily pipelines.
• 12:11
  SPEAKER_03: We are running our pipelines with the mindset of a surrounding space on Earth, and we run those ad hoc as it is, which is very different than anything I've ever been working with before in terms of the Airflow and orchestration engine.
• 12:27
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:36
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:48
  SPEAKER_00: To see how you can maximize airflows potential within your data ecosystem, visit www.astronomer.io.
• 12:55
  SPEAKER_01: Yeah, and say more about that.
• 12:57
  SPEAKER_01: So how do you think that sort of scheduling difference has that impacted the solution that you've implemented at all, as opposed to a case where you might be running something over and over again?
• 13:07
  SPEAKER_03: So we might need to clarify that one.
• 13:09
  SPEAKER_03: We do run things over and over again, I would say, but generally it is not like a daily run.
• 13:17
  SPEAKER_03: And we are not really interested in five or 10 minutes, isn't a big deal to us to wait for things to spin up versus what some other clients or other businesses might require.
• 13:29
  SPEAKER_01: Got it.
• 13:30
  SPEAKER_01: So more that you're optimizing for resource utilization as opposed to a time-bound nature of something.
• 13:37
  SPEAKER_01: Yeah, if that makes sense.
• 13:39
  SPEAKER_01: And for that initial memory allocation, how did you implement that?
• 13:43
  SPEAKER_01: So that's another pipeline that's running in your Airflow instance that does those predictions.
• 13:48
  SPEAKER_02: And then how do those predictions actually get put into predictions are generated by a stub that runs as the first thing that we run when we run a task instance?
• 13:58
  SPEAKER_02: And so it calls out, it queries the database, it comes back with a number.
• 14:03
  SPEAKER_02: And if that number is nice and low and is smaller than the memory needed to run, just run our basic task, our basic stub, it'll keep running inside that stub.
• 14:14
  SPEAKER_02: And if it's more than that, it goes out and says, Well, okay, I'm going to set up the Kubernetes executor config for the retry and I'll fail fast.
• 14:22
  SPEAKER_03: Yeah.
• 14:23
  SPEAKER_03: For this full, our final product here, I think might be a a good thing to highlight.
• 14:28
  SPEAKER_03: We recognize that a lot of this logic, there was a bunch of code they had put in place for this.
• 14:33
  SPEAKER_03: This was a bunch of custom code.
• 14:35
  SPEAKER_03: And the only way to make it work for the rest of our developers, because not everybody needs to know all of the magic that's going on behind the scenes here, is we wrapped it all up in a custom decorator.
• 14:44
  SPEAKER_03: So instead of the at task decorator, we have an at task dot vp.
• 14:49
  SPEAKER_03: And that's literally all of our uh all our engineers need to know to include a bunch of the goodies that we needed to implement for this to work.
• 14:57
  SPEAKER_03: Um, and the list of those goodies are we had to implement stats logging for every task.
• 15:02
  SPEAKER_03: So, and what happens there is we load the actual memory utilized into a table and 15-second increments, and that's what David can use later to like get that initial memory recommendation.
• 15:14
  SPEAKER_03: So we implemented stats logging, we implemented initial memory recommendation logic, and then we implemented the memory ramp, is what we call it, which enables us to self-heal on retry.
• 15:28
  SPEAKER_03: And we have some additional logic in there that helps us determine what actually is an out-of-memory error versus an out-of-memory error because we don't want a memory ramp on things that are, for instance, like a value error or something that we wouldn't want to use extra resources where it would just fail our memory gap.
• 15:46
  unknown: Yeah.
• 15:47
  SPEAKER_02: The list of things that look like an out-of-memory error is a little bit funny because we're using Python and a lot of language features in Python allocate memory along the way.
• 15:59
  SPEAKER_02: So if you go into like a tri block or something, you're allocating memory for just because you went into the block.
• 16:05
  SPEAKER_02: And so for tasks that are on the edge, there's actually a fairly long list of things you will see that are actually memory out-of-memory failures.
• 16:15
  SPEAKER_01: Yeah, that's interesting.
• 16:16
  SPEAKER_01: I am personally familiar with the many different causes of, or not causes, but I suppose log outputs that could indicate an out-of-memory.
• 16:24
  SPEAKER_01: We actually did a previous episode of this podcast where we interviewed an internal team and astronomer that are currently working on an AI solution that would help with translating log outputs, so into a more human readable output so that people have an easier time figuring out what's going on in them, which you're obviously doing it on the fly within your task, which is a little different than somebody reading it.
• 16:46
  SPEAKER_01: But yeah, those can definitely be tricky.
• 16:48
  SPEAKER_01: Fascinating.
• 16:49
  SPEAKER_01: Love hearing about the customization that you've done.
• 16:52
  SPEAKER_01: I think this is something that we hear a lot about, especially with teams using Airflow at scale, is really leveraging the flexibility for implementing as you have like a custom decorator or something so that you have all of your task logic wrapped up and repeatable.
• 17:07
  SPEAKER_01: So it's I I've found actually relatively a little more rare that teams are having engineers go and just write DAGs from scratch all of the time.
• 17:15
  SPEAKER_01: Usually there's some sort of internal logic that's built in to help with whatever the particular use case is.
• 17:21
  SPEAKER_03: Are you talking DAG factories and patterns like that more often or decorators?
• 17:26
  SPEAKER_01: Not necessarily DAG factories.
• 17:29
  SPEAKER_01: Some teams definitely do use that, but more building some sort of structure around just a blank Python file where you would be starting out with some sort of template.
• 17:39
  SPEAKER_01: So in your case, it might be something like a custom operator or a custom decorator.
• 17:43
  SPEAKER_01: It could also be like a DAG generator.
• 17:46
  SPEAKER_01: But again, just leveraging the flexibility that Airflow brings and for you to implement things like that.
• 17:51
  SPEAKER_01: One other thing, I've seen you talk about this in public meetups where you've spoken about this use case.
• 17:57
  SPEAKER_01: But one of the Airflow features that you've used in the solution is dynamic task mapping, which for folks who haven't seen it, a little harder to describe over audio, but allows you to create copies of tasks dynamically on at runtime based on some input that you give.
• 18:14
  SPEAKER_01: Can you talk a little bit about how that was uh or how that influenced your solution?
• 18:19
  SPEAKER_03: Certainly.
• 18:20
  SPEAKER_03: So that was a big piece that we had to solve.
• 18:23
  SPEAKER_03: We jumped right into dynamic task mapping because we recognized that it would work for our partitioning scheme relatively early.
• 18:30
  SPEAKER_03: We don't plan on doing a partitioning scheme that would max out what dynamic task mapping can do, which I think is 1024 individual task instances for for each piece.
• 18:41
  SPEAKER_03: So it's really nice.
• 18:43
  SPEAKER_03: You just have to feed it a list essentially, and it will spin up an additional, like a task instance that's a copy given some parameter for whatever lists you enter.
• 18:53
  SPEAKER_03: And for us, we break it down by what we call watersheds.
• 18:56
  SPEAKER_03: Those are just areas on the map that are were useful for us because we have a nice clean partitioning scheme across all of the United States with these.
• 19:05
  SPEAKER_03: So we'll feed in basically a list of those watersheds, and then it will fan out into processing an individual task instance per watershed.
• 19:14
  SPEAKER_03: And each of those task instances will spin up its own node with its own resources to it.
• 19:20
  SPEAKER_03: And then by default, dynamic task mapping will work just like tasks.
• 19:24
  SPEAKER_03: You you assign the resources that you need to use at the task level.
• 19:29
  SPEAKER_03: But if it fans out and say you're running 24 individual task instances, they would all have the same resources by default.
• 19:36
  SPEAKER_03: And that didn't work for us because each of these watersheds had a different acreage, essentially, with different acorages, just more data.
• 19:45
  SPEAKER_03: So that's where this idea of initial memory recommendation came into play, and that was so important for us because this was a cost saver in general, where we would have to allocate memory to be towards the largest size of a watershed that we had by default before we implemented this uh initial memory recommendation system that David has put in place.
• 20:10
  SPEAKER_01: Which I'd imagine is expensive.
• 20:12
  SPEAKER_01: Yes, and default is the maximum.
• 20:15
  SPEAKER_03: David can talk about how many extra resources we were using.
• 20:19
  SPEAKER_02: It was on the order of we were using about 10 times what we actually needed, just because there's this huge gap between the biggest and the smallest.
• 20:28
  SPEAKER_03: But we we really like dynamic task mapping because honestly, it kept us uh didn't require us to go out and get another tool.
• 20:34
  SPEAKER_03: So we didn't have to go and leverage Ray or Dask or some system like that just yet.
• 20:39
  SPEAKER_03: We can do it within Airflow.
• 20:42
  SPEAKER_03: And then with this solve, we were able to say, like, hey, what we're what we actually do, and this is a little crazy right now, is we can't mutate a pod until we hit the retry callback.
• 20:55
  SPEAKER_03: So in some cases, we have to fail the first try, and we'll do that instantly, and we'll immediately go out and look on our database and determine the right resource to apply.
• 21:05
  SPEAKER_03: And then on the first try, we give it on retry callback that mutates the next try.
• 21:10
  SPEAKER_03: And that kind of sets us up for success from the initial.
• 21:14
  SPEAKER_02: That model of you fail fast works really well when you have long-running tasks.
• 21:20
  SPEAKER_02: We were designing this around, you know, the biggest ones here tend to run for hours and the smallest ones often for several minutes.
• 21:28
  SPEAKER_02: If it's a situation where you have very short-running tasks where they're taking seconds to run, this is not usually the right answer.
• 21:39
  SPEAKER_02: Sure.
• 21:40
  SPEAKER_02: But if it's large tasks with highly variable memory requirements, this works really well.
• 21:46
  SPEAKER_01: Yeah, that's awesome to hear.
• 21:48
  SPEAKER_01: Well, I know we're getting close to the end of our allotted time, and we appreciate you both taking the time today.
• 21:54
  SPEAKER_01: Uh, I always like to end with a standard final question, and that's what would you most like to see from development of the Airflow project in the near future?
• 22:02
  SPEAKER_03: This is great.
• 22:03
  SPEAKER_03: Well, I think we just highlighted it is we got to come up with some solution so we can do memory requests for us.
• 22:11
  SPEAKER_03: This is important before in the first try.
• 22:13
  SPEAKER_03: I think there's a lot of extra like dancing that we're doing right now that we don't need to be doing.
• 22:18
  SPEAKER_03: And then having this more integrated and like kind of core to the airflow system would be very interesting.
• 22:26
  SPEAKER_03: So having the capability to mutate pods out of the box would be great.
• 22:30
  SPEAKER_03: We've certainly shared our code, and people are more than welcome to use that.
• 22:34
  SPEAKER_03: But I'd be very interested in working with anybody from the community that wanted to take a look at what we're doing and find a way to make it make sense for anybody else.
• 22:43
  SPEAKER_01: Yeah, that's great.
• 22:44
  SPEAKER_01: I think this use case is certainly far from uncommon what you're trying to do.
• 22:49
  SPEAKER_01: So I'm sure that would benefit a lot of different folks in the communities.
• 22:53
  SPEAKER_01: Obviously, very grateful for any contributions from outside of the core project, especially ones that have been used in production already and been successful.
• 23:01
  SPEAKER_01: So that's great.
• 23:03
  SPEAKER_01: All right.
• 23:03
  SPEAKER_01: Well, thank you both so much for joining today.
• 23:05
  SPEAKER_01: This has been really fun.
• 23:06
  SPEAKER_01: What's the best way for folks to get in touch with you?
• 23:09
  SPEAKER_03: I would say they can certainly reach out by email.
• 23:12
  SPEAKER_03: So you can either find that on our website or feel free to reach out to me at Cyrus at vibrantplanet.net.
• 23:21
  SPEAKER_01: Okay, that's great.
• 23:22
  SPEAKER_01: We can throw that in the show notes as well.
• 23:24
  SPEAKER_01: Well, thank you both again.
• 23:25
  SPEAKER_01: It was really great talking to you.
• 23:26
  SPEAKER_01: And yeah, we hope to help you on again sometime.
• 23:29
  SPEAKER_03: Pleasure, thank you.
• 23:33
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:39
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 23:43
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.