Latest / Satellite image deep learning / Challenges and opportunities for Ai mapping
Transcript
- 0:05SPEAKER_01: Hi, Toby.
- 0:06SPEAKER_01: Welcome to the podcast.
- 0:08SPEAKER_00: Yeah.
- 0:08SPEAKER_00: Hi, Robert.
- 0:09SPEAKER_00: Thank you for having me.
- 0:11SPEAKER_01: It's my pleasure.
- 0:12SPEAKER_01: All right.
- 0:12SPEAKER_01: Let's begin with where you work and what you do.
- 0:16SPEAKER_00: So, yeah, my name is Tobias Augsburger.
- 0:19SPEAKER_00: I'm working for Open Energy Transition.
- 0:22SPEAKER_00: I'm leading the grid mapping activity.
- 0:26um we just started a small initiative called um map your grid
- 0:32uh that is funded
- 0:33actually by break for energy um that's the company of bill
- 0:37gates that tries to do a
- 0:39SPEAKER_00: lot in the energy space um
- 0:44So Bill Gates actually envisioned to create an open source
- 0:48operating system in the
- 0:49SPEAKER_00: energy sector.
- 0:51And we are supporting this with different open source
- 0:54developments within this company.
- 0:57um yeah i have a background in actually aerospace engineering
- 1:02so actually i'm an
- 1:03aerospace engineer i worked in aerodynamics for quite some time
- 1:07instrumentation
- 1:08remote sensing so remote sensing instrumentation then i
- 1:12switched to um evv codes
- 1:16and automation of evv codes did a lot of with open source and
- 1:19robotics so i really
- 1:20SPEAKER_00: got into open source actually by
- 1:23SPEAKER_00: using it in robotics,
- 1:25automating big trucks to move containers on a yard was
- 1:32something we did for DHL for
- 1:34SPEAKER_00: quite some time with open source.
- 1:37And then I lost my job and did the open sustainable technology
- 1:43project as a COVID project.
- 1:47SPEAKER_00: Yeah, right.
- 1:48SPEAKER_01: So you had a bit of time.
- 1:49SPEAKER_01: You thought, I'll get into open source.
- 1:52SPEAKER_01: Interesting.
- 1:53And so there's kind of an arc here towards understanding energy
- 1:56infrastructure and
- 1:59SPEAKER_01: even sort of making sense of that on a bigger scale.
- 2:03SPEAKER_00: Exactly.
- 2:03SPEAKER_00: Yes.
- 2:05We really try to create more or less a digital twin of the
- 2:09global electrical grid.
- 2:11SPEAKER_00: Yeah.
- 2:12SPEAKER_00: All right.
- 2:12SPEAKER_01: Okay.
- 2:12SPEAKER_01: Yeah.
- 2:14SPEAKER_01: Fascinating.
- 2:15SPEAKER_01: So just what are we talking about?
- 2:17We're talking about pylons,
- 2:19we're talking about substations,
- 2:21that kind of infrastructure,
- 2:22right?
- 2:24exactly um so we map the lines because you can also see the
- 2:29lines and satellite
- 2:30imagery so our main source uh for mapping or raw data for
- 2:36mapping is satellite
- 2:37imagery and we can see the pylons or power towers we can see
- 2:42substations and we can
- 2:44SPEAKER_00: see even the transmission lines in many cases
- 2:47right right so literally those wires across the landscape that
- 2:50are probably thin as
- 2:52what your wrist or something um exactly yeah we really uh we
- 3:00normally start with
- 3:01the substations and then from that you see all the wires going
- 3:05through all the
- 3:06SPEAKER_00: regions
- 3:09SPEAKER_00: connecting actually the energy systems across all the countries.
- 3:16SPEAKER_01: Right,
- 3:18SPEAKER_01: so high resolution imagery,
- 3:20SPEAKER_01: you look for the substations,
- 3:21SPEAKER_01: you know that these things are connected by the wires.
- 3:24SPEAKER_01: Sounds pretty straightforward.
- 3:26I think there's plenty of machine learning models out there
- 3:28that can segment objects.
- 3:30SPEAKER_01: What are the challenges there?
- 3:33SPEAKER_00: Yeah,
- 3:33SPEAKER_00: we,
- 3:34SPEAKER_00: we use,
- 3:36SPEAKER_00: um,
- 3:36SPEAKER_00: mainly at the moment,
- 3:38SPEAKER_00: um,
- 3:39SPEAKER_00: human beings actually that are,
- 3:41SPEAKER_00: uh,
- 3:42SPEAKER_00: uh,
- 3:42SPEAKER_00: mapping the grid and open street maps.
- 3:44SPEAKER_00: So 70% or roughly about 70% of the grid has,
- 3:48SPEAKER_00: uh,
- 3:48SPEAKER_00: mostly actually been mapped over the last 17 years by,
- 3:53SPEAKER_00: um,
- 3:53SPEAKER_00: human beings.
- 3:54SPEAKER_00: Um,
- 3:56SPEAKER_00: But we are now,
- 3:58SPEAKER_00: let's say on the last 30%,
- 3:59we are now also investigating machine learning approaches to
- 4:07detect substations.
- 4:08That's the most easiest object you can,
- 4:12and biggest object also you can find in the grid.
- 4:19SPEAKER_00: But we also investigating to detect the next power tower
- 4:25SPEAKER_00: But this comes with a lot of challenges.
- 4:27SPEAKER_01: Right,
- 4:30SPEAKER_01: so substations,
- 4:31SPEAKER_01: quite large,
- 4:32SPEAKER_01: regular shapes,
- 4:33SPEAKER_01: you think those are pretty straightforward,
- 4:35SPEAKER_01: but you have other infrastructure,
- 4:36SPEAKER_01: which is more challenging.
- 4:38And that's why the human oversight is required to kind of
- 4:41connect the dots and
- 4:43SPEAKER_01: identify all these things as being part of the same network.
- 4:46SPEAKER_01: Is that right?
- 4:47SPEAKER_00: Yeah, that's right.
- 4:48But it's also the case that you find a lot of different grid
- 4:54infrastructure that
- 4:55SPEAKER_00: has actually been synthetically created by AI.
- 4:59And at the end,
- 5:02the challenge is to harmonize all this data into one map that
- 5:05is validated.
- 5:07SPEAKER_00: And this is where the human being comes into the process.
- 5:13SPEAKER_00: Interesting.
- 5:15Yeah,
- 5:16you have for one country sometimes five or six different grid
- 5:19maps or grid data sets.
- 5:23SPEAKER_00: And what we normally do, we don't copy paste anything in OpenStreetMap.
- 5:28SPEAKER_00: That's very important to understand.
- 5:30SPEAKER_00: We use them as a hint layer.
- 5:33so we have our open street map editors and then we show the
- 5:39open street map data
- 5:40and then we put on another layer actually this ai generated or
- 5:45however this data
- 5:46has been generated and then we have to integrate this data set
- 5:52by validating every
- 5:53data point right so power tower got it so there's basically
- 5:59false positives from
- 6:00SPEAKER_01: the machine learning models is that what you're saying
- 6:04um yes it's about false positives um but it's really also about
- 6:10the connections so
- 6:12um for example it is relevant um at which side of the
- 6:18substation a transmission
- 6:19SPEAKER_00: line is connected to estimate the voltage of the line it's sometimes very
- 6:27SPEAKER_00: interesting to um also see what is
- 6:32um on the end of a substation and this is also hard to uh to do
- 6:36with AI because you
- 6:38find sometimes very different infrastructure at the end you
- 6:42find bigger loads you
- 6:43find sometimes just a distribution substation of the city you
- 6:48find sometimes a
- 6:49SPEAKER_00: generate a big car plant and
- 6:54this final mapping process of the grid allows us to bring in
- 7:01all this context into
- 7:03SPEAKER_00: the data set.
- 7:04SPEAKER_00: So it's really a journey that you have in grid mapping.
- 7:11SPEAKER_00: And with machine learning, we would not have this.
- 7:14SPEAKER_00: So we could not bring all this knowledge into the grid.
- 7:19SPEAKER_00: We would not have the opportunity also to
- 7:26bring metadata into the grid that demands a certain context
- 7:33understanding of the grid.
- 7:35So there's a lot of different layers and context understanding
- 7:39that you need to map the grid.
- 7:44SPEAKER_01: I guess my next question is,
- 7:46presumably you could train models to do things like identify
- 7:48which side of wire was
- 7:50connected and start to build up some of this context itself or
- 7:53input those features
- 7:55to another model that could make predictions about how these
- 7:59things are connected
- 8:00SPEAKER_01: or which class they are.
- 8:04SPEAKER_00: Let's say this way.
- 8:05SPEAKER_00: It's technically possible, yes, but there are limitations to that.
- 8:13one is just the data availability for machine learning of this
- 8:16high resolution data.
- 8:19SPEAKER_00: So in OpenStreetMap,
- 8:22SPEAKER_00: we have S3 world imagery,
- 8:25SPEAKER_00: we have Bing aerial imagery,
- 8:28but this data is just being provided for the OpenStreetMap
- 8:32community for free to do
- 8:34SPEAKER_00: manual work mapping.
- 8:36SPEAKER_01: Right.
- 8:39SPEAKER_00: Every machine learning approach,
- 8:41we would actually need to buy the satellite imagery and not
- 8:47just one imagery
- 8:49SPEAKER_00: product.
- 8:50SPEAKER_00: And that's the challenge.
- 8:51SPEAKER_00: So for us as a human to validate this data,
- 8:54SPEAKER_00: you have to switch between sometimes four different satellite imagery.
- 8:59SPEAKER_01: Right.
- 9:01Because the transmission lines in,
- 9:04for example,
- 9:05in Africa,
- 9:06they go through areas where
- 9:09age of the satellite imagery and the quality of the satellite
- 9:12imagery is changing a
- 9:13lot so you go 100 kilometers then you have to change to another
- 9:18satellite imagery
- 9:19product then you have to change again you have change again and
- 9:22sometimes for us we
- 9:23really have to switch four times between the same imagery to
- 9:28ensure that this is
- 9:29SPEAKER_00: actually a power tower right
- 9:33So you would basically have four times the problem for a model
- 9:36because typically a
- 9:36SPEAKER_01: model is trained on one kind of imagery,
- 9:39SPEAKER_01: one kind of target.
- 9:40SPEAKER_01: But here we have a lot more diversity to handle.
- 9:43SPEAKER_01: Still sounds like maybe a solvable problem for machine learning,
- 9:47SPEAKER_01: but I'm getting a good impression of the sort of key challenges.
- 9:52SPEAKER_00: It definitely is.
- 9:53There is a very interesting open source prototype that has been
- 9:58developed by a
- 10:00SPEAKER_00: group in New York from Lindsay Labs.
- 10:05SPEAKER_00: And they do substation detection using Sentinel-2.
- 10:11And since Sentinel-2 is an open data set,
- 10:15it's allowed as a raw data source for OpenStreetMap.
- 10:22And we are investigating this because one of the main
- 10:28challenges is to have the substation.
- 10:30If you have the substations of a country,
- 10:33I would say this is 50% of the work.
- 10:37SPEAKER_00: Then you just have to connect the dots.
- 10:42SPEAKER_00: And this is possible using...
- 10:49all the different spectres of the uh of centenary two right
- 10:54right so there was a
- 10:55solution already developed i think i've seen that a little
- 10:57while back um how
- 10:59SPEAKER_00: effective was it i think it the false positives were very little um the
- 11:07segmentation was uh the intersection of union was not perfect
- 11:11but still very good
- 11:13SPEAKER_00: at least from the paper and from the um
- 11:18uh from the code i've seen uh it works actually very well um
- 11:23and it would be for us
- 11:25good enough because even if you would say even if you would
- 11:28have i don't know uh 10
- 11:31false positives since we are this has been validated by a human
- 11:36being at the end
- 11:37SPEAKER_00: it's not a problem
- 11:40But it's really key for us because every power plant comes with
- 11:44a substation.
- 11:46SPEAKER_00: Every bigger industrial loads comes with a substation.
- 11:52You have actually the whole power infrastructure if you have a
- 11:55good substation detection.
- 11:56SPEAKER_01: All right.
- 11:57So that sounds like a pretty promising model,
- 12:00just for that particular task.
- 12:03SPEAKER_01: Does it work globally?
- 12:04Are there some reasons why it can't be deployed for this
- 12:09initial vetting of where
- 12:12SPEAKER_01: the substations are?
- 12:13SPEAKER_00: So the training data set...
- 12:19that has been created for this is using OpenStreetMap data as a
- 12:25ground truth.
- 12:25SPEAKER_00: So OpenStreetMap is used a lot as ground truth validated data set.
- 12:30And I used actually the polygons and projected then on Sentinel
- 12:35two data and use
- 12:37SPEAKER_00: this,
- 12:37SPEAKER_00: I think 50,000 substations all around the world.
- 12:41SPEAKER_00: Yeah, the results were pretty good.
- 12:50and i think the only limitation that you actually have is areas
- 12:57where there's just
- 12:58a lot of clouds it's always right so optical data set some
- 13:02places amazon or
- 13:04wherever lots of clouds potentially you need more than one
- 13:09modality to detect the
- 13:10substations globally yes i we investigated also sentinel one
- 13:16data if it might be
- 13:18SPEAKER_00: possible
- 13:23SPEAKER_00: I think it could be done.
- 13:28And it's,
- 13:29it,
- 13:29it,
- 13:30but it also depends on the size of the substation itself.
- 13:34SPEAKER_00: Like this really high voltage substations.
- 13:37SPEAKER_00: They are so big.
- 13:39Um,
- 13:39you can actually work with very low resolution satellite data
- 13:43to try and detect them.
- 13:45SPEAKER_00: Um, but those are actually easy to find for this.
- 13:50SPEAKER_00: You don't have detection.
- 13:52SPEAKER_00: Um,
- 13:55The harder ones are the small distribution substations that are
- 13:59at the end of the grid.
- 14:03SPEAKER_00: But those are especially interested because the interesting for us,
- 14:10because the distribution grid is something that you cannot map
- 14:18very well with even
- 14:21SPEAKER_00: with high resolution satellite data.
- 14:25SPEAKER_00: because it's mostly in urban areas,
- 14:28SPEAKER_00: you have the poles sometimes,
- 14:30SPEAKER_00: you can't see them,
- 14:31SPEAKER_00: but normally you see poles based on the shadow or power towers,
- 14:35you see always based on the shadow of those infrastructure
- 14:38because they are so
- 14:39SPEAKER_00: tall.
- 14:40SPEAKER_00: And in this urban areas, it's sometimes pretty hard to detect this.
- 14:46SPEAKER_00: But if you have the distribution substations,
- 14:49and it can use again machine learning and try and this is
- 14:53there's a lot of a lot of
- 14:55groups trying to estimate the distribution grid using machine
- 15:00learning approaches
- 15:01SPEAKER_00: and this can be um supported a lot if you know the nodes the distribution
- 15:09SPEAKER_00: substation that are the nodes of the distribution grid
- 15:13SPEAKER_01: Right.
- 15:14So this sort of comes back to your earlier point that the
- 15:16experts use multiple
- 15:17SPEAKER_01: sources of imagery.
- 15:18SPEAKER_01: They've done a lot of personal training.
- 15:20SPEAKER_01: They have an understanding of how grid networks work.
- 15:23And they are able to identify these smaller substations in more
- 15:27challenging areas.
- 15:28SPEAKER_01: But currently, machine learning methods are not there.
- 15:34SPEAKER_00: We have sort of the substation detection that has been developed so far.
- 15:43SPEAKER_00: using mostly bigger substations, yes.
- 15:46SPEAKER_01: Yeah.
- 15:46SPEAKER_01: Okay, brilliant.
- 15:48So I think that really helps sort of clarify where the state of
- 15:50the art is and what
- 15:51SPEAKER_01: the sort of frontier looks like and what the challenges are.
- 15:55SPEAKER_01: Obviously, you've been working to curate a lot of truth data.
- 15:59SPEAKER_01: As you mentioned,
- 16:00SPEAKER_01: OpenStreetMap is often used for generating,
- 16:04SPEAKER_01: you know,
- 16:04SPEAKER_01: annotated data sets for training models.
- 16:06SPEAKER_01: Yes.
- 16:08I'm not sure how many people were aware of that and also the
- 16:10richness of the data
- 16:11SPEAKER_01: that you've curated.
- 16:13So I think one thing we want to do is just highlight that that
- 16:15resource exists and
- 16:17SPEAKER_01: is being developed for this purpose,
- 16:19SPEAKER_01: right?
- 16:21SPEAKER_00: Yes,
- 16:23SPEAKER_00: it's not...
- 16:24SPEAKER_00: It's used for machine learning a lot,
- 16:28SPEAKER_00: but it's also used for energy system modeling,
- 16:30SPEAKER_00: like actually half a ground truth data set.
- 16:34SPEAKER_00: It's used a lot...
- 16:38SPEAKER_00: for regional studies, for larger studies.
- 16:40SPEAKER_00: It's used by the World Bank to actually,
- 16:45SPEAKER_00: to improve the electrification of,
- 16:51SPEAKER_00: for example,
- 16:52SPEAKER_00: areas in Africa and South America.
- 16:56SPEAKER_00: So this data is really one
- 16:59actually the only data set that is standardized for the
- 17:03electrical grid worldwide.
- 17:05Many low to middle income countries,
- 17:06there's actually no good data about the electrical grid.
- 17:12And this is a real problem for investors and for the expansion
- 17:18of this electrical grid.
- 17:22And yeah,
- 17:23OpenStreetMap is used as one of the best sources because it's
- 17:28validated.
- 17:29SPEAKER_00: And it's accepted by the community,
- 17:33because it's used for years by larger organizations like the
- 17:37World Bank to do
- 17:38SPEAKER_00: investments into the grid.
- 17:39SPEAKER_01: Right.
- 17:40SPEAKER_01: So significant uses for this data set,
- 17:45SPEAKER_01: you know,
- 17:45SPEAKER_01: not just for machine learning and expanding the data set,
- 17:48SPEAKER_01: but also for understanding infrastructure in a country,
- 17:51SPEAKER_01: enabling investment,
- 17:54SPEAKER_01: etc.
- 17:55SPEAKER_00: Exactly, yeah.
- 17:56SPEAKER_01: Brilliant.
- 17:58SPEAKER_01: And interesting constraints,
- 17:59SPEAKER_01: as you mentioned,
- 18:00SPEAKER_01: around sort of licensing of data for particular uses.
- 18:03And then also when you try and deploy these generated locations
- 18:09into OpenStreetMap,
- 18:10SPEAKER_01: there's certain criteria that must meet,
- 18:13like you must be able to independently validate those results
- 18:17using open data.
- 18:17So it's a really interesting context there that kind of guides
- 18:21the solutions people
- 18:22SPEAKER_01: might be looking to develop.
- 18:25SPEAKER_01: Brilliant.
- 18:25SPEAKER_01: That's a really interesting conversation.
- 18:27SPEAKER_01: I think it's a very interesting problem as well.
- 18:28SPEAKER_01: As you mentioned,
- 18:29SPEAKER_01: it's kind of not just detect a thing,
- 18:31SPEAKER_01: but understand how it fits into a bigger network and challenges like the
- 18:36SPEAKER_01: multimodality potentially helping.
- 18:38And I think lots of potential there for machine learning and
- 18:43really highlights kind
- 18:44SPEAKER_01: of the limitations of the current technology.
- 18:47SPEAKER_01: I guess.
- 18:48SPEAKER_01: In terms of just to sort of bring things to a head,
- 18:51where are the main platforms where this information is
- 18:54communicated and where you
- 18:56SPEAKER_01: share your own kind of resources on this topic?
- 19:01so um i think one of the best um websites to actually have a
- 19:07look on this data is
- 19:08um open infrastructure map or you find it at openinframap.org
- 19:15and this is a
- 19:17visualization of the global electrical grid data this is an
- 19:21open street map
- 19:22SPEAKER_00: including power plants substations and um
- 19:28SPEAKER_00: but also all the underground infrastructure.
- 19:33SPEAKER_00: So you will find,
- 19:36SPEAKER_00: especially for Europe and the US,
- 19:38SPEAKER_00: this data is from data quality,
- 19:44SPEAKER_00: similar to official data from system operators.
- 19:51If you visit,
- 19:51if you want to improve this data,
- 19:53you would visit map your grid.org.
- 19:57SPEAKER_00: That's our website.
- 19:59SPEAKER_00: Um,
- 20:00SPEAKER_00: here you can find our starter kit or videos,
- 20:03but also some tooling that is just simplifying and automating
- 20:07the process around
- 20:09SPEAKER_00: transmission grid mapping.
- 20:11SPEAKER_00: So we, we try to standardize a lot around this topic.
- 20:16SPEAKER_00: Um,
- 20:18But also the OpenStreetMap Wikipedia page has a lot of very
- 20:23interesting content.
- 20:28You will find it actually if you search for OpenStreetMap Wiki
- 20:32and Power Networks.
- 20:36Debbie will find like a landing page that will give you an
- 20:40introduction into also
- 20:42SPEAKER_00: the topic,
- 20:43but also the local pages from the local groups of people that
- 20:52are focusing on
- 20:54SPEAKER_00: curating,
- 20:55SPEAKER_00: improving the grid data for their country.
- 20:58SPEAKER_00: And they are collecting data there on their country pages.
- 21:02SPEAKER_00: So I think this is the main
- 21:05SPEAKER_00: um, websites, um, when it comes to open street map.
- 21:10It's great to hear about these,
- 21:13these global communities working to map their areas.
- 21:16And I just think it's an amazing resource that the community
- 21:20has put together.
- 21:22Yes,
- 21:23it's really,
- 21:24uh,
- 21:25I must personally say I'm really,
- 21:27um,
- 21:28uh,
- 21:30impressed by the data quality.
- 21:31SPEAKER_00: Um,
- 21:34It's a little bit hard to prove the data quality within
- 21:37OpenSuiteMap and we are
- 21:38SPEAKER_00: investigating this at the moment.
- 21:40SPEAKER_00: Also using machine learning,
- 21:41SPEAKER_00: do random sample approach to take data,
- 21:44SPEAKER_00: try if you can detect it like it's been mapped.
- 21:47SPEAKER_00: Mm-hmm.
- 21:49SPEAKER_00: But it's really,
- 21:50SPEAKER_00: yeah,
- 21:52what I think this is something very special that a lot of
- 21:56people don't really
- 21:57SPEAKER_00: understand.
- 21:58SPEAKER_00: How do you curate a data set over 17 years?
- 22:03SPEAKER_00: What is the strategy to improve this data on and on and on,
- 22:08SPEAKER_00: especially when it comes to machine learning,
- 22:10SPEAKER_00: where there's a lot of people that actually are generating data that
- 22:18is being potentially a source,
- 22:22but someone at the end still needs to harmonize all of this.
- 22:29SPEAKER_00: And OpenStreetMap is really a great place to harmonize GIS data.
- 22:35SPEAKER_01: Humans still play a very critical role there.
- 22:38SPEAKER_01: So that's important for us all.
- 22:42SPEAKER_01: Brilliant.
- 22:43SPEAKER_01: Okay.
- 22:43SPEAKER_01: And you're personally active on LinkedIn, I think.
- 22:46SPEAKER_01: So that's probably another place people can follow.
- 22:48SPEAKER_00: Yes.
- 22:51SPEAKER_00: Yeah, and have a look at our YouTube videos.
- 22:54I think they are the best way to have a quick understanding
- 22:57about what we're actually doing.
- 22:58SPEAKER_01: Perfect.
- 23:00SPEAKER_01: I'll place those links in the show notes.
- 23:04But once again,
- 23:04thanks for joining me today and telling me about this
- 23:06fascinating topic.
- 23:07SPEAKER_01: And I wish your project all the best in the future.
- 23:11SPEAKER_00: Yeah, thank you so much, Robin.