Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith

Transcript

Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: There's these really cool, like clever little tricks and things that you could do with Airflow that make your processing easier.
• 0:05
  SPEAKER_03: And they're kind of fun, honestly.
• 0:06
  SPEAKER_03: Like it was a little intimidating at first getting into it just because it's something so different than writing like normal Python code, but it's kind of enjoyable.
• 0:12
  SPEAKER_04: And working with the dependencies, like setting up dependencies in Airflow tags have been, you know, when you set them up properly.
• 0:20
  SPEAKER_04: And then if you happen to have to run just this one task and to have it flow through and only run everything else that's dependent on it, but just those things.
• 0:51
  SPEAKER_02: In just seven minutes, you can help shape the future of Airflow and grab a free Airflow 3 fundamentals or DAG Authoring certification worth$150 each.
• 1:02
  SPEAKER_02: Plus, you can enter to win an exclusive workshop with Mark Lumberti.
• 1:06
  SPEAKER_02: Don't miss out.
• 1:07
  SPEAKER_02: The survey link is in the show notes and closes November 21st.
• 1:14
  SPEAKER_02: Hi everyone, welcome to the Data Flowcast.
• 1:17
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Alex Ianicelli, who is an open source software engineer at Overture Maps Foundation, and Daniel Smith, who is a senior solutions architect at Wearabots.
• 1:29
  SPEAKER_02: How are you both doing?
• 1:31
  SPEAKER_04: Great.
• 1:31
  SPEAKER_04: How are you?
• 1:32
  SPEAKER_04: I'm doing great.
• 1:32
  SPEAKER_04: Thank you.
• 1:33
  SPEAKER_02: Join the end of summer here.
• 1:35
  SPEAKER_02: So yeah, okay.
• 1:36
  SPEAKER_02: For our listeners for today's podcast, we're going to discuss using Airflow to manage geospatial data.
• 1:43
  SPEAKER_02: I think this is a super interesting topic.
• 1:45
  SPEAKER_02: I had a very tiny bit of experience working with geospatial data in a previous life, many, many years ago, and have not touched it since.
• 1:53
  SPEAKER_02: So this was long before Airflow, I think, was even a thing.
• 1:56
  SPEAKER_02: So I'm excited to dive into it with you both.
• 2:00
  SPEAKER_02: But before we do that, I want to start with some background.
• 2:02
  SPEAKER_02: So maybe you can both tell me about what you do and what your companies do.
• 2:07
  SPEAKER_03: Yeah, definitely.
• 2:07
  SPEAKER_03: My name is Daniel Smith.
• 2:08
  SPEAKER_03: I am a senior solutions architect at WareBots.
• 2:11
  SPEAKER_03: I've been in the geospatial industry for 20 or 25 years now.
• 2:14
  SPEAKER_03: But WareBots provides a SaaS platform for running Apache Sedona.
• 2:19
  SPEAKER_03: And so in the same way that PostGIS brings spatial functionality to the widely used Postgres database, Apache Sedona brings spatial functionality to the widely used Apache Spark system.
• 2:34
  SPEAKER_03: And so Warebots is actually run and created by the original authors and the maintainers of the Apache Sedona project.
• 2:42
  SPEAKER_03: And we have additional optimizations that run on top of it that kind of take it from one to 11, if you will, as well as some additional value add products on top of that.
• 2:52
  SPEAKER_02: That's great.
• 2:52
  SPEAKER_02: We had Astronomer, we love open source companies or open source focus companies, of course.
• 2:57
  SPEAKER_02: Cool to see another Apache Software Foundation project.
• 2:59
  SPEAKER_02: Yeah, getting some getting a bump up to 11, as you said.
• 3:03
  SPEAKER_02: There we go.
• 3:03
  SPEAKER_02: I love that.
• 3:04
  SPEAKER_02: And Alex, what about you?
• 3:05
  SPEAKER_04: I am uh Alex Yonicelli.
• 3:07
  SPEAKER_04: I'm principal software engineer at Overture Maps Foundation.
• 3:10
  SPEAKER_04: I've been in the GAS industry for only about five years.
• 3:15
  SPEAKER_04: Before that, lots of different cloud computing and storage and networking protocols and things like that.
• 3:21
  SPEAKER_04: At Overture Maps Foundation, we're using open data.
• 3:25
  SPEAKER_04: We get open data from multiple different sources and we need to conflate them all together, merge them all together for our monthly release that is completely free to use for anyone on the internet.
• 3:37
  SPEAKER_02: Very cool.
• 3:37
  SPEAKER_02: And how do Whereabouts and Overture Maps Foundation work together?
• 3:42
  SPEAKER_04: At Overture Maps Foundation, we use the Spark processing for our spatial joins and spatial merging of all this data.
• 3:53
  SPEAKER_04: And we were using just AWS Glue and Azure Databricks for that processing.
• 4:00
  SPEAKER_04: And Wearabots came along and said they'd like to be part of that.
• 4:04
  SPEAKER_04: And we work together with them.
• 4:06
  SPEAKER_04: We have an agnostic Spark class so we can kind of point using Azure's parameter, one parameter in Airflow, using an Airflow parameter, we can choose what platform we run all of our Spark processing on.
• 4:20
  SPEAKER_04: And Wearabots is one of our default ones that we use for a lot of our pipelines.
• 4:24
  SPEAKER_03: Yeah.
• 4:24
  SPEAKER_03: I think that relationship goes back to I think the middle of 2024.
• 4:29
  SPEAKER_03: And originally, I think that we just kind of wanted to see where WareBots can fit into this project.
• 4:35
  SPEAKER_03: And initially, we started to do some benchmarking and some testing between the default settings and the settings that you guys were using in Databricks and in EMR and running those against Warebots.
• 4:46
  SPEAKER_03: I think whether they were running, whether Alex and Overture runs their pipelines wherever, like they're using Sedona at the core, right?
• 4:53
  SPEAKER_03: These are Sedona pipelines.
• 4:55
  SPEAKER_03: And so just doing a pure lift and shift, right?
• 4:57
  SPEAKER_03: We just took the main business logic from doing some of this conflation, right?
• 5:01
  SPEAKER_03: Taking OSM buildings and then other machine learning and other generated building data sets specifically for this pipeline and just replicated that within the platform and were able to see some nice performance gains, which drove down the cost and increased performance overall.
• 5:17
  SPEAKER_03: And so I think from there, that's when we started to really team up like with Alex's team and the other folks at Overture and the other folks at WareBots to migrate these workflows and rewrite them so that they can be run within WareBots using our airflow operator.
• 5:32
  SPEAKER_03: And so when Alex talks about the agnostic class, from our perspective, what we've helped with there is being able to point the pipeline and run it, like Alex said, with only one parameter change using the WareBots run operator.
• 5:48
  SPEAKER_03: And so it's this open source operator that we maintain specifically for running the geospatial jobs within the WareBots environment.
• 5:54
  SPEAKER_03: We actually have two different operators that we run.
• 5:56
  SPEAKER_03: We have a basic SQL operator, which returns just as you would expect, it returns the results from a specific SQL query.
• 6:03
  SPEAKER_03: And we have a larger runs operator, which is what we use with Overture.
• 6:07
  SPEAKER_03: And this allows a much larger process or script to be automated.
• 6:11
  SPEAKER_03: I think that those are the ways that we really work together.
• 6:13
  SPEAKER_03: And so we've got a couple of guys that work with Alex directly to help implement some of these more technical things.
• 6:19
  SPEAKER_02: Okay, very cool.
• 6:21
  SPEAKER_02: And maybe just zooming out a little bit, talk to me a little bit about where Airflow generally fits in with the Wearabots platform.
• 6:29
  SPEAKER_02: And you mentioned that's a managed or adding additional features on top of Apache Sedona.
• 6:35
  SPEAKER_02: And so how do you think those two tools most effectively work together?
• 6:39
  SPEAKER_03: I think that the amount of geospatial data being created 20 years ago was smaller than it is now, obviously.
• 6:49
  SPEAKER_03: And it was less frequent, right?
• 6:51
  SPEAKER_03: You had different processes and different techniques and tasks that would generate this data that was either pushed out daily or weekly or whatever.
• 6:57
  SPEAKER_03: With the proliferation of like mobile devices and cheap IoT devices and the sensor networks that we have out, we have terabytes and petabytes of data coming in.
• 7:07
  SPEAKER_03: So airflow and whereabots kind of mesh is in that processing, right?
• 7:12
  SPEAKER_03: We have our interactive workloads.
• 7:14
  SPEAKER_03: It's a Jupyter notebook.
• 7:15
  SPEAKER_03: So you can come in and define your process and work out what you want to do, you know, how that business logic needs to transform or load or analyze that data.
• 7:24
  SPEAKER_03: And typically, what our customers do, we'll take that work and then automate it with airflow, right?
• 7:29
  SPEAKER_03: Like that data comes in every day, sometimes every hour, sometimes every minute, frankly.
• 7:33
  SPEAKER_03: And that's really where Webots and Airflow kind of start to mesh is in that recurring constant processing of this stream of data that is constantly being generated by our cell phones, by our connected devices, by assets and tractors in the field, by delivery vans, like all of this stuff.
• 7:51
  SPEAKER_03: DoorDash, Uber, like everybody has cell phones.
• 7:54
  SPEAKER_03: Everybody is collecting GPS data, whether they like it or not, like that's being fed somewhere and that data needs to be processed.
• 8:00
  SPEAKER_03: And that's where Webots and Airflow operator start to shine.
• 8:03
  SPEAKER_04: Yeah.
• 8:03
  SPEAKER_04: And at Overture Maps Foundation, so we have a few different levels of data processing.
• 8:09
  SPEAKER_04: The very first one is just getting the open data from the vendor and bringing it into our environment and converting it to our schema.
• 8:17
  SPEAKER_04: And that's one set of pipelines.
• 8:19
  SPEAKER_04: And then we have a separate set of pipelines that for each theme, we've got a number of different themes like buildings is a theme, places, transportation, those types of themes.
• 8:30
  SPEAKER_04: Each theme has a pipeline that takes the data, the multiple different open data sets and converges them into the data set that we will present for our theme.
• 8:42
  SPEAKER_04: And then finally, and that's a set of different pipelines.
• 8:45
  SPEAKER_04: And then we have a one big pipeline at the end that takes each one of those themes and processes that for a data set that can be released.
• 8:53
  SPEAKER_04: And that is every month.
• 8:54
  SPEAKER_04: But then a lot of those themes are being run multiple times a month just to make sure anytime we make a change, we're doing testing, things like that.
• 9:02
  SPEAKER_04: And Airflow makes it really easy to see the progress of all the themes, pipelines, and see where we have issues and be able to drill into the logs and be able to debug things too.
• 9:14
  SPEAKER_02: Yeah, got it.
• 9:15
  SPEAKER_02: And so to vastly oversimplify what you've just said, whereabots built on top of Sedona is the processing engine that's specifically built for geospatial data, and you're using Airflow as the orchestrator, which totally, totally makes sense.
• 9:28
  SPEAKER_02: I love that.
• 9:30
  SPEAKER_02: If any, are there special considerations do you think are required of an orchestrator like Airflow in this case that are being used for geospatial pipelines?
• 9:40
  SPEAKER_04: I look at each of the tasks that the Airflow pipeline or DAG is executing as a job that needs to be done and it takes a certain amount of time.
• 9:49
  SPEAKER_04: I personally love how Airflow allows me to visually see what's running, click into it, see the logs, and be able to watch the progress of whatever that thing is.
• 10:01
  SPEAKER_04: So when we're working with Webots, has all their logs coming from Spark feeding into that log, and we're able to see whether things are going off the rails or not, because sometimes they do, and we don't want to spend extra resources when things are going bad.
• 10:16
  SPEAKER_04: So I guess that's not very specific to G GIS, but like we have these massive data loads that we have to process the whole entire world of features.
• 10:28
  SPEAKER_04: And so they take hours at times, and being able to just see that this is in progress and be able to drill into it easily using a GUI has been great.
• 10:36
  SPEAKER_04: Yeah.
• 10:36
  SPEAKER_03: And I think that dovetailing into the scale of the spatial data, the scale that Airflow can scale to is really the benefit here, right?
• 10:44
  SPEAKER_03: Like we have so much data being processed, and Airflow is not a bottleneck for orchestration, right?
• 10:51
  SPEAKER_03: It will kick these tasks off.
• 10:52
  SPEAKER_03: It doesn't necessarily have to, it's not memory bound from when you use our operator, right?
• 10:57
  SPEAKER_03: You're deploying and running this code within our processing engine.
• 11:00
  SPEAKER_03: And so I think for me, the specialness of Airflow in this is the fact that it just scales and it scales in processing time and it scales with the size of our data and the number of processes that we have.
• 11:11
  SPEAKER_03: And it bear is mentioning that for folks that aren't familiar with Overger Maps Foundation, like they have a gigantic amount of data.
• 11:20
  SPEAKER_03: We're talking about global building footprints.
• 11:22
  SPEAKER_03: I think last time I checked, this is in 2.6 to 3 billion polygon range.
• 11:27
  SPEAKER_04: Those are the ones that we release.
• 11:29
  SPEAKER_04: So we're processing way more than that and then filtering out.
• 11:32
  SPEAKER_04: Yeah.
• 11:33
  SPEAKER_03: Right.
• 11:33
  SPEAKER_03: And that that process includes, I think, five or six different building footprints data sources.
• 11:39
  SPEAKER_03: And that's just one of their themes, right?
• 11:40
  SPEAKER_03: They support unique POIs and places data sets.
• 11:43
  SPEAKER_03: They have administrative boundaries and natural features, you know, water bodies, parks, all kinds of stuff that comes in.
• 11:50
  SPEAKER_03: I think that it's an amazing group that has decided to come together to do this, right?
• 11:54
  SPEAKER_03: Like it's all built on top of, I think, OSM, like the OpenStreetMap data, it serves as the base layer for a lot of these.
• 12:00
  SPEAKER_03: And that data in and of itself is huge.
• 12:03
  SPEAKER_03: And so being able to process all of that, like Alex said, with seven, 10 different sources coming in and not having the orchestration tool be the bottleneck.
• 12:14
  SPEAKER_03: And with Webots, the processing is no longer the bottleneck.
• 12:17
  SPEAKER_03: It allows for faster iteration.
• 12:19
  SPEAKER_03: It allows for, like Alex said, observability into the process and just keeps the boat moving.
• 12:24
  SPEAKER_03: After spending a lot of time in this area, we'd spend time manually dividing up the data like Alex processes by country or by county or by state, right?
• 12:34
  SPEAKER_03: And just having to manage that code, right, uh is onerous and it is time consuming.
• 12:41
  SPEAKER_03: And so parsing and writing those tools in code that orchestrates subsetting down is uh you know removed from the process.
• 12:50
  SPEAKER_03: Like we don't have to manage that part anymore.
• 12:53
  SPEAKER_03: And we don't have to worry about the processing time or the amount of memory that has to go in.
• 12:57
  SPEAKER_03: So like these two barriers and these two things that had to be in the front of an engineer like Alex's mind are kind of removed.
• 13:03
  SPEAKER_03: And it allows him to focus on the business logic, focus on bringing in new data sets, focus on all these other things that are more creative outlets for his skills than orchestrating multi-threading in Python or orchestrating multi-processing in Python, whatever the case may be.
• 13:19
  SPEAKER_03: I speak for myself.
• 13:20
  SPEAKER_03: I can't speak for you directly, Alex, but it frees me up to do the things that I enjoy more, which I think is really the goal with orchestration tools and yeah.
• 13:28
  SPEAKER_04: Yeah, absolutely.
• 13:29
  SPEAKER_04: And we talk about orchestrating the pipelines through Airflow, and then all of a sudden we go, what else can we orchestrate using Airflow that we do often?
• 13:38
  SPEAKER_04: And we end up creating all these other pipelines that aren't exactly part of the normal release pipeline, but I can push button, do this, do that, all these different tasks that we have that we do often.
• 13:49
  SPEAKER_04: It's really been great.
• 13:51
  SPEAKER_03: And the other things I think I really like about Airflow, just some of the fun things that I've used are like the circuit breaker operator, right?
• 13:57
  SPEAKER_03: There's these really cool, like clever little tricks and things that you could do with Airflow that make your processing easier.
• 14:02
  SPEAKER_03: And they're kind of fun, honestly.
• 14:04
  SPEAKER_03: Like it was a little intimidating at first getting into it just because it's something so different than writing like normal Python code, or but it's kind of enjoyable.
• 14:10
  SPEAKER_04: And working with the dependencies, like setting up dependencies in airflow tags have been, you know, when you set them up properly.
• 14:19
  SPEAKER_04: And then if you happen to have to run just this one task and to have it flow through and only run everything else that's dependent on it, but just those things, not the other things.
• 14:30
  SPEAKER_04: That's been very cool to see because some of these tasks that we have, like I said, take hours to do.
• 14:35
  SPEAKER_04: And like very often we have, like Daniel said, we have multiple, but just thinking about the buildings, we have like seven different sources, and OSM is updated the most frequent, and it happens to be kind of at the top of our priority.
• 14:49
  SPEAKER_04: And so if that has to be updated, we can go in and just say, run that one task, update the whole thing, and everything downstream of that will run.
• 14:59
  SPEAKER_04: But the other sources, they don't have to be reconverted because they haven't changed.
• 15:04
  SPEAKER_04: And so a lot of that work is simplified with the dependencies.
• 15:07
  SPEAKER_04: It's been great.
• 15:10
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 15:19
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 15:31
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 15:40
  SPEAKER_02: You've done a very good job of describing kind of the feature richness of something like Airflow, which to your point can be, I think there's a lot of unknown stuff that people don't even realize is in there.
• 15:51
  SPEAKER_02: But it is does cover almost all of the use cases that you would need for orchestration.
• 15:55
  SPEAKER_02: I think to your point, if you implement it correctly, can scale quite effectively, especially if you're doing so, you know, with a separate processing engine.
• 16:05
  SPEAKER_02: I think one of the questions that my team gets a lot is can you process data within Airflow?
• 16:09
  SPEAKER_02: And so I was like, yes, sort of, to some extent.
• 16:14
  SPEAKER_02: Certainly not when you're talking about like the scale of data that you're discussing here with geospatial data.
• 16:21
  SPEAKER_02: You would not want to do that on your Airflow worker.
• 16:23
  SPEAKER_02: So yeah, I think using it in combination with a tool like Wearabots is really cool.
• 16:28
  SPEAKER_02: And I do want to kind of circle back to we talked a little bit about the Wearabots operators, but just speaking about like the feature richness of Airflow, one of the things that I think is really cool about the open source project is that so many people have committed providers or made available, even if they're not within the project themselves.
• 16:45
  SPEAKER_02: In fact, most of them are not ways of interacting with external tools that make it a lot easier.
• 16:50
  SPEAKER_02: And it sounds like so you have done this for Wearabots with your operators.
• 16:54
  SPEAKER_02: Maybe just tell me, you know, one more time a little bit about kind of what those look like and are they open source and how does a user interact with those?
• 17:02
  SPEAKER_03: Yeah, definitely.
• 17:03
  SPEAKER_03: So there's two different operators that we support.
• 17:05
  SPEAKER_03: The more basic operator is the SQL operator.
• 17:08
  SPEAKER_03: And so the parameters that it takes are super straightforward, right?
• 17:11
  SPEAKER_03: You take an API key and you've got a connection string, right?
• 17:15
  SPEAKER_03: And that's how you make a connection to our cloud offering.
• 17:18
  SPEAKER_03: We've actually do this with the free version of WareBots.
• 17:20
  SPEAKER_03: We have a community version that you can sign up for.
• 17:22
  SPEAKER_03: It gets you access to smaller run times.
• 17:24
  SPEAKER_03: It's available on the marketplace.
• 17:25
  SPEAKER_03: You can go to our website and check it out too.
• 17:28
  SPEAKER_03: The more comprehensive operator is the runs operator.
• 17:31
  SPEAKER_03: And this really encapsulates our entire runs API in an easy-to-use airflow operator, right?
• 17:37
  SPEAKER_03: So the REST API has all of the Git put, like all of those fine tune-in points that allow you to orchestrate the job manually.
• 17:45
  SPEAKER_03: But we've packaged that up and it's directly available on GitHub.
• 17:48
  SPEAKER_03: And for that, it's a really basic install, right?
• 17:50
  SPEAKER_03: You just have it as part of your requirements file, whether you're using a managed airflow instance or you're rolling your own.
• 17:55
  SPEAKER_03: And then when you rebuild, like that operator is available to you.
• 17:58
  SPEAKER_03: And so I think that some of the good parts about it is that you can use all of your command line arguments.
• 18:04
  SPEAKER_03: And so if your process takes inputs for, say, an AOI, like an area of interest, a polygon, that can be coded and sent through if it just has different parameters.
• 18:13
  SPEAKER_03: Like we actually maintain a mirror of the Overture Maps Foundation data within our spatial catalog within WareBots.
• 18:19
  SPEAKER_03: And so I actually run our mirror task as just this, right?
• 18:24
  SPEAKER_03: And it just uses the WareBots operator.
• 18:25
  SPEAKER_03: And what it really does is it calls a Python script or like a jar file that's either like in managed storage on our end or in your guys' S3 buckets or a user's S3 buckets if it's attached.
• 18:37
  SPEAKER_03: And then it calls that script within the environment.
• 18:40
  SPEAKER_03: And you can specify your size of your environment.
• 18:43
  SPEAKER_03: Like I was saying, that there's different size environments that you can use depending on your workload, which is another great part about using Airflow and using the DAGs and the tasks.
• 18:51
  SPEAKER_03: If you have a task that imports data or ingest data, and the first task is to migrate that data, you can get a rough estimate of that data size.
• 18:59
  SPEAKER_03: And then using like the XCOM, communicate back to the Warebox operator and dynamically size the runtime selection that you have there, which is a really nice feature of being able to dovetail these processes together.
• 19:11
  SPEAKER_03: Those are nice.
• 19:12
  SPEAKER_03: We also support like all of your individual dependencies.
• 19:14
  SPEAKER_03: And so you can specify either as part of the runtime or part of the dependencies as the command or the parameter within the Webots operator to say, we need these different packages.
• 19:24
  SPEAKER_03: Please install those first before you start the actual environment.
• 19:27
  SPEAKER_03: So they're all available if you have additional dependencies.
• 19:29
  SPEAKER_03: And so it's pretty well feature complete from what I've seen.
• 19:32
  SPEAKER_03: But it really didn't start that way, to be honest with you.
• 19:34
  SPEAKER_03: I think when we first started working with Alex and the operator, like we didn't have some of these pieces in here.
• 19:39
  SPEAKER_03: This was a really great learning experience iterating with Alex and the Overture Maps team to extend and expand our operator for a real world use case.
• 19:49
  SPEAKER_03: It was a really good opportunity for us to finish that build.
• 19:53
  SPEAKER_03: And like I said, if it's completely open source, you can go into GitHub and look at both operators there.
• 19:57
  SPEAKER_03: Yeah, and they're available on Pi too.
• 19:58
  SPEAKER_03: So when you go to install, like that's really just how you do it.
• 20:01
  SPEAKER_03: So it's available on PyPy.
• 20:02
  SPEAKER_03: That's how you get to it with your requirements file for installing on the actual Airflow instance.
• 20:07
  SPEAKER_02: Yeah, great.
• 20:08
  SPEAKER_02: Well definitely include links to the GitHub repo and stuff in the show notes.
• 20:12
  SPEAKER_02: That sounds like a really cool partnership.
• 20:14
  SPEAKER_02: And like I said, I always think one of the best things about Airflow is having operators available so you don't have to figure out how to write all that code yourself.
• 20:22
  SPEAKER_02: It sounds especially in this case, that would save you quite a lot of time.
• 20:26
  SPEAKER_02: Yeah, very cool.
• 20:27
  SPEAKER_02: Speaking of Airflow features and providers and such, I always like to end with a standard final question for folks.
• 20:33
  SPEAKER_02: And that is what would you most like to see from the Airflow project in the near future?
• 20:37
  SPEAKER_04: Oh this is one of those things that I don't know whether it exists or not.
• 20:42
  SPEAKER_04: I may be missing this, but I'd love to see an easy way for DAGs to trigger DAGs.
• 20:50
  SPEAKER_04: So when a full pipeline is finished based on some output, trigger another DAG to execute.
• 20:58
  SPEAKER_02: Is that in like a different way than data sets or now they're called assets in Airflow 3 in a different way than those would work?
• 21:05
  SPEAKER_04: Or I don't know.
• 21:07
  SPEAKER_04: We are on Airflow 2, aren't we?
• 21:10
  SPEAKER_04: Yeah.
• 21:11
  SPEAKER_04: So we're not in Airflow three.
• 21:13
  SPEAKER_02: So as long as we're on, I think it was 2.4 that data sets were introduced.
• 21:18
  SPEAKER_02: So you should be able to use a feature design for exactly that where you would indicate a task and it could be the last task in a DAG that updates data set or asset.
• 21:28
  SPEAKER_02: I'm using quotes because Airflow doesn't actually have any awareness of what that underlying thing might be of the name.
• 21:35
  SPEAKER_02: But then you can have that as the schedule for a DAG.
• 21:38
  SPEAKER_02: So it runs whenever that is updated.
• 21:40
  SPEAKER_04: Cool.
• 21:41
  SPEAKER_04: All right.
• 21:41
  SPEAKER_04: One other little thing.
• 21:43
  SPEAKER_04: In the GUI, being able to group things by like directories or groups, that'd be cool.
• 21:50
  SPEAKER_04: But just visually.
• 21:50
  SPEAKER_04: It's just nice.
• 21:51
  SPEAKER_02: Yeah.
• 21:52
  SPEAKER_02: That's definitely a frequent request.
• 21:54
  SPEAKER_02: I have seen in the very early stages 3.1, which should come out.
• 21:58
  SPEAKER_02: Well, by the time this is released, I think it will already be out.
• 22:01
  SPEAKER_02: But like mid-September-ish time frame, there is at least now the ability to favorite DAGs.
• 22:06
  SPEAKER_02: So you can pin them like in the UI, which at least for the ones that you need to look at every day is nice.
• 22:12
  SPEAKER_02: But yeah, grouping being able to group them or organize them in a different way, I think, is something a lot of people want.
• 22:19
  SPEAKER_02: Yeah.
• 22:19
  SPEAKER_03: Very nice.
• 22:20
  SPEAKER_03: I don't know.
• 22:21
  SPEAKER_03: I don't think I have a lot of requests right now.
• 22:22
  SPEAKER_03: I'll go back to our engineers and see what they say.
• 22:24
  SPEAKER_03: Frankly, I'm a geospatial guy at heart.
• 22:27
  SPEAKER_03: So, you know, any amount of spatial awareness that could be built in, it could even just be a map somewhere.
• 22:31
  SPEAKER_03: Like just embed a map anywhere in there and I'll be a happy camper.
• 22:33
  SPEAKER_03: Just put a new map there.
• 22:34
  SPEAKER_02: Maps are great.
• 22:35
  SPEAKER_02: I love that suggestion.
• 22:37
  SPEAKER_02: Okay, cool.
• 22:38
  SPEAKER_02: Thank you for going through all of that.
• 22:39
  SPEAKER_02: This has been really interesting.
• 22:40
  SPEAKER_02: What is the best way for folks to get in touch with you?
• 22:43
  SPEAKER_03: It's a super difficult email.
• 22:44
  SPEAKER_03: It's just Daniel at wearbots.com.
• 22:46
  SPEAKER_03: You can just hit me up there at any time or support at Webots.com.
• 22:49
  SPEAKER_03: Both of those will come to me.
• 22:51
  SPEAKER_03: Or you can jump onto our website, like I said, and have a support request if you want to do a demo or just see what Webots is all about.
• 22:56
  SPEAKER_03: Those are going to be the best ways to reach out and kind of do that.
• 22:58
  SPEAKER_03: And like I said, jump onto the marketplace, sign up for a free tier account, jump onto our website, sign up for a free tier account.
• 23:04
  SPEAKER_03: There's a bunch of data there for you to work with, inbuilt examples and notebooks.
• 23:08
  SPEAKER_03: It should be pretty easy to get started.
• 23:09
  SPEAKER_03: And if you have any issues, then I'll go talk to my boss about it.
• 23:12
  SPEAKER_03: So you can go from there.
• 23:13
  SPEAKER_04: Yeah, and Overture Maps is Overtramaps.org.
• 23:16
  SPEAKER_04: And you can get the data that's released every month for free.
• 23:21
  SPEAKER_04: So check it out.
• 23:22
  SPEAKER_04: Go look at the GitHub repos that we have that are public.
• 23:25
  SPEAKER_04: And if you have issues, post them there.
• 23:27
  SPEAKER_04: We do listen.
• 23:28
  SPEAKER_04: And you can reach me at alex at overture maps.org.
• 23:32
  SPEAKER_02: Super easy.
• 23:33
  SPEAKER_02: We'll throw all of those links in for folks who want to access them.
• 23:37
  SPEAKER_02: Thank you both again for joining me today.
• 23:39
  SPEAKER_02: This has been really great.
• 23:40
  SPEAKER_04: Thanks, Kesim.
• 23:41
  SPEAKER_04: Always it, please.
• 23:42
  SPEAKER_04: Thank you.
• 23:44
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:51
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 23:55
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.