Latest / Satellite image deep learning / Benchmarking AI for Critical Infrastructure Mapping

Transcript

Benchmarking AI for Critical Infrastructure Mapping · Satellite image deep learning

• 0:00
  [music]
• 0:05
  SPEAKER_00: Hi Justin, welcome to the podcast.
• 0:08
  SPEAKER_01: Hi Robin, it's great to be here.
• 0:10
  SPEAKER_00: Let's begin with a brief intro to yourself.
• 0:13
  SPEAKER_01: Sure, yeah.
• 0:14
  SPEAKER_01: So my name is Justin Guthrie.
• 0:17
  SPEAKER_01: I am a recent bachelor's graduate from Northeastern University in Boston.
• 0:23
  SPEAKER_01: Originally, I come from an informatics background, so my degree was data science and business.
• 0:28
  SPEAKER_01: But ever since then, I've been Slowly transitioning over into the Earth observation,
• 0:34
  SPEAKER_01: geospatial, and civil engineering side of things.
• 0:39
  SPEAKER_01: As of now, I am currently working full-time
• 0:43
  SPEAKER_01: at Northeastern University's Defense Industrial Base Institute,
• 0:47
  SPEAKER_01: where I am a research data analyst there.
• 0:50
  SPEAKER_01: And in addition, I'm also part of the Sustainability and Data Sciences Lab at Northeastern.
• 0:57
  SPEAKER_01: The Geoinformation Lab at George Mason University,
• 1:01
  SPEAKER_01: which is where my most recent paper has come from.
• 1:04
  SPEAKER_01: Fantastic.
• 1:06
  SPEAKER_00: So we're connected over that paper, InfraBench CLS.
• 1:10
  SPEAKER_00: Do you want to tell us a bit about it?
• 1:13
  SPEAKER_01: Yeah, absolutely.
• 1:14
  SPEAKER_01: So InfraBench CLS kind of came out
• 1:19
  SPEAKER_01: as a result of me taking a look on LinkedIn,
• 1:24
  SPEAKER_01: for example, and seeing All of these foundation models that are popping up one
• 1:29
  SPEAKER_01: or two every month, it seems like.
• 1:31
  SPEAKER_01: And I realized after having some exposure to that,
• 1:38
  SPEAKER_01: it seemed like there were a lot of foundation models that were out there,
• 1:41
  SPEAKER_01: but there wasn't a whole lot of work done into looking
• 1:45
  SPEAKER_01: at how well these foundation models can perform,
• 1:48
  SPEAKER_01: specifically where they're pitted against each other, essentially.
• 1:53
  SPEAKER_01: So that kind of formed the idea of InfraBench CLS,
• 1:58
  SPEAKER_01: which is a benchmarking paper that compares all these foundation models specifically
• 2:05
  SPEAKER_01: on critical infrastructure.
• 2:07
  SPEAKER_01: And critical infrastructure, in my view, at least for this paper,
• 2:12
  SPEAKER_01: are the power systems, water systems, transportation,
• 2:16
  SPEAKER_01: and telecommunications systems that underpin a lot of society.
• 2:24
  SPEAKER_01: In combination with not really seeing a lot of benchmarks out there,
• 2:28
  SPEAKER_01: specifically it seemed like if there were any benchmarks,
• 2:32
  SPEAKER_01: they were focused more on agriculture, urban science, like human movement,
• 2:39
  SPEAKER_01: or just kind of broad earth observation tasks.
• 2:42
  SPEAKER_01: So I felt like there was a real gap looking at critical infrastructure and felt
• 2:47
  SPEAKER_01: that was really important considering its role in And, you know,
• 2:51
  SPEAKER_01: natural disaster resilience and things like that.
• 2:55
  SPEAKER_00: Indeed. In many parts of the world, this infrastructure is not mapped at all.
• 3:00
  SPEAKER_01: Yeah, absolutely.
• 3:01
  SPEAKER_01: I mean, there's, yeah.
• 3:03
  SPEAKER_01: And this hopefully can help, you know,
• 3:06
  SPEAKER_01: put a first step out to that goal of being able to better map infrastructure
• 3:12
  SPEAKER_01: across the world as well.
• 3:13
  SPEAKER_00: Right, right. So to create that data set, I think you used OSM?
• 3:18
  SPEAKER_01: Yeah, so I used OSM OpenStreetMap for weak labels to inform the data set.
• 3:25
  SPEAKER_01: And the data set itself is Sentinel-2
• 3:29
  SPEAKER_01: and Sentinel-1 imagery centered
• 3:32
  SPEAKER_01: on these assets where we derive their locations from OpenStreetMap.
• 3:38
  SPEAKER_01: I can talk a little more about the data set if you'd like.
• 3:41
  SPEAKER_00: Yes.
• 3:42
  SPEAKER_00: Well, I mean, where does it cover, you know, how you chose the places that you chose?
• 3:47
  SPEAKER_01: Yeah, so from the beginning, we wanted this to be global.
• 3:52
  SPEAKER_01: It didn't really make much sense to focus on just one country.
• 3:57
  SPEAKER_01: And so it covers the globe.
• 4:01
  SPEAKER_01: And OSM is kind of the basis for where these assets are.
• 4:07
  SPEAKER_01: All of these assets, as I mentioned earlier, cover power, water, transportation,
• 4:11
  SPEAKER_01: and telecommunication systems.
• 4:13
  SPEAKER_01: For the power system, we look at Things like power generators, power substations,
• 4:19
  SPEAKER_01: both transmission and distribution.
• 4:22
  SPEAKER_01: For the water side,
• 4:24
  SPEAKER_01: we look at water treatment plants and transportation,
• 4:30
  SPEAKER_01: things like train stations, airports, and telecommunications.
• 4:34
  SPEAKER_01: We actually just have data centers.
• 4:36
  SPEAKER_01: The original intention was to look at cell towers.
• 4:40
  SPEAKER_01: All of the imagery that we're using is 10-meter resolution,
• 4:45
  SPEAKER_01: publicly available Sentinel imagery,
• 4:46
  SPEAKER_01: so they would be a little bit difficult to detect.
• 4:49
  SPEAKER_01: So we stuck with something a little broader.
• 4:53
  SPEAKER_00: The larger structures.
• 4:55
  SPEAKER_00: I'd imagine even within those,
• 4:56
  SPEAKER_00: probably a railway station in North America that's quite different from one
• 5:00
  SPEAKER_00: in Africa, for instance.
• 5:01
  SPEAKER_00: How did you get the geographic coverage for those classes?
• 5:06
  SPEAKER_01: Yeah, so really that kind of came down
• 5:09
  SPEAKER_01: to OpenStreetMap's understanding of what those would be.
• 5:14
  SPEAKER_01: And thankfully for transportation,
• 5:16
  SPEAKER_01: that's one of the sectors where OpenStreetMap tends to be a little more consistent
• 5:22
  SPEAKER_01: and accurate in terms of labeling those types of assets.
• 5:28
  SPEAKER_01: So we kind of ended up using that.
• 5:30
  SPEAKER_01: And because we're looking at, you know,
• 5:34
  SPEAKER_01: The dataset itself comprises just around
• 5:39
  SPEAKER_01: 20,000 imagery tiles across the globe.
• 5:43
  SPEAKER_01: There's a lot for the foundation models to train on,
• 5:47
  SPEAKER_01: which would help it to understand,
• 5:50
  SPEAKER_01: even while there may be some slight regional differences,
• 5:53
  SPEAKER_01: to get a better idea of what the train station class would look like,
• 6:00
  SPEAKER_01: even across country lines.
• 6:03
  SPEAKER_00: Got it.
• 6:04
  SPEAKER_00: And you mentioned that each image essentially has a label.
• 6:07
  SPEAKER_00: Why did you go with this kind of level of classification rather than something like,
• 6:11
  SPEAKER_00: you know, a box around a structure or a segmentation mask?
• 6:17
  SPEAKER_01: Yeah, so for classification,
• 6:20
  SPEAKER_01: what I believe is this is kind of the most achievable first step.
• 6:26
  SPEAKER_01: I think that segmentation, for example,
• 6:29
  SPEAKER_01: naturally follows what we do and I I really see this paper as sort
• 6:35
  SPEAKER_01: of a waypoint rather than the final destination for something like this.
• 6:39
  SPEAKER_01: But we went with classification as it was an easy first step to do.
• 6:45
  SPEAKER_01: I didn't do classification and segmentation because that was just
• 6:51
  SPEAKER_01: a little bit too out of scope for me.
• 6:54
  SPEAKER_01: But yeah, we've done classification because we still think that that can be of use
• 7:01
  SPEAKER_01: without having to expand into looking at pixel-level boundaries.
• 7:08
  SPEAKER_01: And we could really utilize OpenStreetMap in that process for classification.
• 7:13
  SPEAKER_00: Yeah, makes a lot of sense.
• 7:15
  SPEAKER_00: So you've created this global benchmark data set.
• 7:18
  SPEAKER_00: You want to evaluate different foundational models.
• 7:19
  SPEAKER_00: As you mentioned, there's a new model every week or so.
• 7:22
  SPEAKER_00: How did you choose the models that you selected for the benchmarking?
• 7:26
  SPEAKER_01: Yeah.
• 7:29
  SPEAKER_01: I did kind of a combination of what I've personally been exposed to,
• 7:36
  SPEAKER_01: both through my LinkedIn adventures looking
• 7:40
  SPEAKER_01: and seeing from different professors
• 7:43
  SPEAKER_01: and lab groups what foundation models that they've been doing.
• 7:47
  SPEAKER_01: I also chatted with Isaac Corley a lot,
• 7:51
  SPEAKER_01: who is one of the Torch Geo maintainers and is also a co-author on this paper.
• 7:58
  SPEAKER_01: And he gave me some suggestions as well on different foundation models to look at.
• 8:03
  SPEAKER_01: What I really tried to do was kind of take a combination of, you know,
• 8:07
  SPEAKER_01: from my exposure to these foundation models, take a combination of, you know,
• 8:14
  SPEAKER_01: models that were more well-known with models that maybe are not as well-known,
• 8:18
  SPEAKER_01: models from, you know, a more academic Labs and sources versus ones
• 8:25
  SPEAKER_01: that were more developed by large corporations or governments or things like NASA
• 8:31
  SPEAKER_01: or Google, for example.
• 8:34
  SPEAKER_01: And I tried to do the best I could to kind of span
• 8:38
  SPEAKER_01: a variety of different architectures as well.
• 8:43
  SPEAKER_01: If you were to look at just one type of foundation model architecture,
• 8:49
  SPEAKER_01: it wouldn't really do you that much good if there's all these sorts
• 8:51
  SPEAKER_01: of potential architectures that you could choose from.
• 8:54
  SPEAKER_00: Right.
• 8:55
  SPEAKER_00: So you've selected your set of foundation models.
• 8:59
  SPEAKER_00: These models themselves might have different requirements.
• 9:02
  SPEAKER_00: They might take different modalities.
• 9:04
  SPEAKER_00: How did you handle that complexity?
• 9:07
  SPEAKER_01: Yeah, well, complexity is certainly the right word there.
• 9:12
  SPEAKER_01: And what we attempted to do was make it
• 9:17
  SPEAKER_01: as consistent as we could across all
• 9:20
  SPEAKER_01: the models while also still respecting each model's individual specifications
• 9:26
  SPEAKER_01: that were outlined in their documentations.
• 9:30
  SPEAKER_01: For example, in terms of the consistency,
• 9:34
  SPEAKER_01: we ran each model through linear probing as well as fine-tuning.
• 9:39
  SPEAKER_01: And something like this let us compare Google Alpha Earth Foundation's pre-computed
• 9:45
  SPEAKER_01: embeddings with other models' performances.
• 9:51
  SPEAKER_01: And for, you know, so we, the dataset itself, you know,
• 9:55
  SPEAKER_01: comprises Sentinel-1 SAR and Sentinel-2 data.
• 9:59
  SPEAKER_01: So for the models that just took in Sentinel-1 data, for example,
• 10:04
  SPEAKER_01: Satellis pre-trains S1 backbone, You know,
• 10:07
  SPEAKER_01: we gave it the SAR data for their S2 backbone.
• 10:10
  SPEAKER_01: We gave it the Sentinel-2 data.
• 10:13
  SPEAKER_01: So these things are outlined in the paper as, you know,
• 10:19
  SPEAKER_01: notes of sometimes those things aren't directly comparable
• 10:23
  SPEAKER_01: because of the different types of modalities,
• 10:25
  SPEAKER_01: but it still provides an interesting reference point regardless based on the imagery
• 10:32
  SPEAKER_01: that you have available and what assets that you want to classify.
• 10:37
  SPEAKER_00: Right, right.
• 10:38
  SPEAKER_00: And did you work within an existing codebase,
• 10:40
  SPEAKER_00: or did this require a lot of engineering work to bring all these models together?
• 10:46
  SPEAKER_01: So, and hopefully this will be the answer to your question,
• 10:50
  SPEAKER_01: but essentially what I did was, yeah,
• 10:54
  SPEAKER_01: so this training Phase required a lot of GPU,
• 10:58
  SPEAKER_01: which my home computer or my laptop doesn't have.
• 11:03
  SPEAKER_01: So what I did for that is all the code itself is code
• 11:08
  SPEAKER_01: that I created in my VS Code IDE.
• 11:12
  SPEAKER_01: It's all published on GitHub.
• 11:15
  SPEAKER_01: And what I did for the GPU requirements is I actually used Google Colab.
• 11:23
  SPEAKER_01: And use their hosted runtimes to be able to accomplish that.
• 11:28
  SPEAKER_01: And as we're currently in the peer review process,
• 11:33
  SPEAKER_01: so what I'm doing now is looking
• 11:35
  SPEAKER_01: into George Mason's
• 11:41
  SPEAKER_01: high-powered compute research cluster
• 11:43
  SPEAKER_01: to be able to utilize GPU
• 11:48
  SPEAKER_01: that way and do it in a little more cost-effective way as well.
• 11:52
  SPEAKER_00: Yeah, I'm sure that would be much more practical once you've got it all set up.
• 11:56
  SPEAKER_00: So you've got the benchmark data set.
• 11:58
  SPEAKER_00: You've selected your models.
• 12:00
  SPEAKER_00: You've created the pipeline.
• 12:02
  SPEAKER_00: What do the results look like?
• 12:05
  SPEAKER_01: Yeah, so this is really interesting.
• 12:09
  SPEAKER_01: And we alluded to it in the paper.
• 12:11
  SPEAKER_01: And I'll kind of start with overall which model performed well.
• 12:16
  SPEAKER_01: And then I'll kind of get into the class comparisons a little bit more.
• 12:21
  SPEAKER_01: What was really interesting was Dyno V3, which is a meta-model,
• 12:27
  SPEAKER_01: meta-foundation model, performed best overall.
• 12:32
  SPEAKER_01: And importantly, because I've had many people come back to me,
• 12:37
  SPEAKER_01: I confirmed up and down that the pre-training of the foundation model
• 12:44
  SPEAKER_01: that we used was their natural imagery dataset.
• 12:47
  SPEAKER_01: It was their LVD1689M dataset.
• 12:51
  SPEAKER_01: Rather than their Earth observation data set.
• 12:55
  SPEAKER_01: And there's, of course, many possible ways that this could have performed better,
• 13:02
  SPEAKER_01: some of which are less of a fluke than others.
• 13:08
  SPEAKER_01: But it's a point for further exploration regardless.
• 13:11
  SPEAKER_01: And so to kind of break this down,
• 13:16
  SPEAKER_01: Dynav3 performed best on A good majority of classes.
• 13:22
  SPEAKER_01: And if I remember correctly,
• 13:23
  SPEAKER_01: it performed best in terms of their F1 score across all classes
• 13:30
  SPEAKER_01: when you add them all together,
• 13:33
  SPEAKER_01: including when you're utilizing your weighted F1 as well,
• 13:38
  SPEAKER_01: based on the distributions of these classes or these assets around the globe.
• 13:45
  SPEAKER_01: And for people who may not be familiar, the Dyno V3 model You know,
• 13:51
  SPEAKER_01: pre-trained on the LVD 1689M data set.
• 13:56
  SPEAKER_01: That data set is all natural imagery or, you know, camera level,
• 14:01
  SPEAKER_01: you take it with your phone, that kind of imagery rather than satellite imagery.
• 14:07
  SPEAKER_01: And this is a genuinely massive data set.
• 14:11
  SPEAKER_01: 1689M is representative
• 14:15
  SPEAKER_01: of 1,689 million Images
• 14:20
  SPEAKER_01: that it's pre-trained on.
• 14:22
  SPEAKER_01: And what I find is interesting is
• 14:26
  SPEAKER_01: it did beat out these Earth observation-specific models.
• 14:30
  SPEAKER_01: And it could potentially be, because the data set is genuinely just so big,
• 14:37
  SPEAKER_01: could be a real testament to this idea of more data is better
• 14:42
  SPEAKER_01: in the machine learning world.
• 14:45
  SPEAKER_01: It's also possible that That somehow along the training pathway,
• 14:53
  SPEAKER_01: the images that Dyno V3 was fed allowed the model to start
• 14:59
  SPEAKER_01: to pick up geometric patterns of different infrastructure sectors and classes,
• 15:05
  SPEAKER_01: rather, to then be applied into Earth observation imagery,
• 15:09
  SPEAKER_01: which is honestly what I hope is what happened because then that would be
• 15:12
  SPEAKER_01: a really interesting thing to explore further.
• 15:15
  SPEAKER_01: However, it's also possible that the Parameters
• 15:18
  SPEAKER_01: that we gave Dyno v3 were just unusually well-suited
• 15:22
  SPEAKER_01: to be able to allow the model
• 15:26
  SPEAKER_01: to classify these infrastructure assets.
• 15:30
  SPEAKER_01: And any other models that may have had, for example,
• 15:35
  SPEAKER_01: some zeros filled in or things like that in their parameters might have gave
• 15:41
  SPEAKER_01: that a disadvantage.
• 15:42
  SPEAKER_01: So there's definitely things to explore in regards to that result.
• 15:47
  SPEAKER_01: But off of this first exploration, that is what happened.
• 15:54
  SPEAKER_01: In addition, you know, kind of going into some of the more class-specific findings,
• 16:01
  SPEAKER_01: we found that airports, train stations,
• 16:06
  SPEAKER_01: and data centers performed the best overall,
• 16:09
  SPEAKER_01: meaning across All of the foundation models,
• 16:15
  SPEAKER_01: they generally performed well,
• 16:17
  SPEAKER_01: and overall across three seeds for all of these models,
• 16:22
  SPEAKER_01: they were consistently classified achieving
• 16:28
  SPEAKER_01: the highest macro F1, essentially.
• 16:31
  SPEAKER_01: And I think that is primarily due to, at least for airports and train stations,
• 16:38
  SPEAKER_01: just the completeness of OpenStreetMap data on those specific assets.
• 16:44
  SPEAKER_01: Airports in particular are notoriously very easy to classify.
• 16:49
  SPEAKER_01: They've got these very large footprints.
• 16:52
  SPEAKER_01: There's literally planes that you can see.
• 16:55
  SPEAKER_01: It's one of the more Exhibit A kind of assets to classify.
• 17:02
  SPEAKER_01: So I think that's kind of one of the reasons is
• 17:08
  SPEAKER_01: the large geometric footprints
• 17:12
  SPEAKER_01: of The airports, as well as the data centers,
• 17:17
  SPEAKER_01: as well as for the airports and train stations,
• 17:20
  SPEAKER_01: how complete they're documented in OpenStreetMap.
• 17:26
  SPEAKER_01: So those are kind of the better classes in terms of how they were classified.
• 17:32
  SPEAKER_01: Some of the classes that didn't perform as well were primarily concentrated
• 17:36
  SPEAKER_01: in the power sector and there's a few possible explanations that I personally have
• 17:40
  SPEAKER_01: for this.
• 17:43
  SPEAKER_01: One that I believe is most likely the culprit is because what we did
• 17:48
  SPEAKER_01: for the power stations, and this is because some of the work
• 17:52
  SPEAKER_01: that I've done recently has looked into the power system more granularly.
• 17:56
  SPEAKER_01: So what I did was I subtyped
• 18:00
  SPEAKER_01: the power substation asset into transmission and distribution.
• 18:05
  SPEAKER_01: Transmission, very broadly speaking, is You know,
• 18:09
  SPEAKER_01: those very large power poles,
• 18:12
  SPEAKER_01: power lines that go across large swaths of the country.
• 18:16
  SPEAKER_01: The distribution substations and the distribution system is what actually takes
• 18:22
  SPEAKER_01: that transmission power and lets you have it within your house.
• 18:27
  SPEAKER_01: It's generally a little more smaller scale.
• 18:30
  SPEAKER_01: So I essentially subtyped the substations that way.
• 18:36
  SPEAKER_01: The problem is To be able to distinguish between those two is not generally done
• 18:42
  SPEAKER_01: at the 10 meter resolution level.
• 18:45
  SPEAKER_01: This is something where higher resolution imagery is needed and is a next step for us.
• 18:52
  SPEAKER_01: And because of that, I think that's one of the main reasons why it didn't perform well.
• 18:58
  SPEAKER_01: The second reason is because from what I've observed within OSM,
• 19:05
  SPEAKER_01: OpenStreetMap is
• 19:08
  SPEAKER_01: Especially for substations,
• 19:14
  SPEAKER_01: there's often a lot of mistyping
• 19:17
  SPEAKER_01: or mislabeling substations based on whether they are transmission or distribution.
• 19:24
  SPEAKER_01: I did a manual,
• 19:28
  SPEAKER_01: hand-curated dataset of Maine and New Hampshire substations,
• 19:34
  SPEAKER_01: which if you're not from the US are two of the US states here
• 19:38
  SPEAKER_01: in the very northeast part of the country.
• 19:41
  SPEAKER_01: And I'm currently based in Maine.
• 19:43
  SPEAKER_01: And what I did was I essentially looked at the OpenStreetMap labels
• 19:49
  SPEAKER_01: for these substations, noted whether they were classified as transmission
• 19:52
  SPEAKER_01: or distribution and where they're located.
• 19:55
  SPEAKER_01: And then I also compared this with our
• 20:00
  SPEAKER_01: Uh, national kind of transmission, uh, data system, uh, sorry,
• 20:05
  SPEAKER_01: transmission system data, um, as well as specific, uh,
• 20:10
  SPEAKER_01: utility maps for the Maine and New Hampshire region, uh,
• 20:14
  SPEAKER_01: for the distribution layer and found that, you know, somewhere around half of the,
• 20:20
  SPEAKER_01: uh, the substations from open street map were not properly, uh,
• 20:25
  SPEAKER_01: labeled according to these authoritative sources.
• 20:28
  SPEAKER_01: Now, of course, things can change.
• 20:30
  SPEAKER_01: And it's possible that construction work is being done or something like that,
• 20:35
  SPEAKER_01: but it was still something interesting that I found regardless.
• 20:38
  SPEAKER_01: And because of those ambiguous labels,
• 20:40
  SPEAKER_01: that could also be a reason why these power substations weren't classified as well,
• 20:45
  SPEAKER_01: too.
• 20:46
  SPEAKER_01: And of course, there were other assets in the power sector, for example, power generators.
• 20:52
  SPEAKER_01: Substations and their subtypes comprised a good majority of the imagery that we had
• 20:58
  SPEAKER_01: on that sector. So that's kind of why I detailed that more clearly.
• 21:02
  SPEAKER_00: Yeah. I suppose in the long run, there might be a feedback loop where models are not
• 21:06
  SPEAKER_00: so strong on certain classes.
• 21:07
  SPEAKER_00: So there's just less truth available for them
• 21:10
  SPEAKER_00: and they perpetually remain challenging versus something like airports,
• 21:14
  SPEAKER_00: which I think it sounds like it's pretty much saturated in the performance.
• 21:19
  SPEAKER_01: Yeah, no, exactly.
• 21:20
  SPEAKER_01: There definitely is this sort of feedback loop because the models are only as good
• 21:25
  SPEAKER_01: as what they're trained on. And if there's not a lot of training data available
• 21:30
  SPEAKER_01: for certain infrastructure classes or within certain regions,
• 21:34
  SPEAKER_01: that definitely perpetuates the ability of the foundation model
• 21:37
  SPEAKER_01: to properly classify, detect, or segment these different assets.
• 21:43
  SPEAKER_00: Absolutely. It always comes back to the dataset, doesn't it?
• 21:46
  SPEAKER_01: I know, right? It really does.
• 21:49
  SPEAKER_00: So some really interesting findings there, and obviously a lot of work's gone into this.
• 21:55
  SPEAKER_00: Given what you've learned, if you're doing this again or advising somebody who wants
• 21:58
  SPEAKER_00: to take it further, what would be your suggestions?
• 22:03
  SPEAKER_01: Yeah, and there's so many possible things that I could point to.
• 22:10
  SPEAKER_01: I mean, benchmark papers are difficult, so there's a lot of things to kind of keep in mind.
• 22:19
  SPEAKER_01: What I, so I guess in terms of kind of next steps, you know,
• 22:26
  SPEAKER_01: the direction I see this most going is moving on to segmentation, for example.
• 22:32
  SPEAKER_01: I think that, again, classification is kind of that first step.
• 22:37
  SPEAKER_01: If we're able to Identify and bound multiple assets,
• 22:42
  SPEAKER_01: critical infrastructure assets within a single image.
• 22:46
  SPEAKER_01: This is something that I think will really help
• 22:48
  SPEAKER_01: the critical infrastructure interdependency modeling world,
• 22:54
  SPEAKER_01: which is also one that I'm a part of.
• 22:57
  SPEAKER_01: And I think that this would be really advantageous for them.
• 23:00
  SPEAKER_01: And this is definitely a direction that I would like to take this work in.
• 23:04
  SPEAKER_01: I'd also like to, as I mentioned earlier, test on higher resolution imagery.
• 23:08
  SPEAKER_01: That I think is really essential.
• 23:11
  SPEAKER_01: And we just have to do some more digging into finding ways
• 23:16
  SPEAKER_01: to query high-resolution imagery down to the sub-meter level efficiently.
• 23:24
  SPEAKER_01: And that's something that's a little bit of a challenge, but we're looking into that as well.
• 23:29
  SPEAKER_01: If I were to advise someone
• 23:33
  SPEAKER_01: on a similar paper like this, I mean,
• 23:37
  SPEAKER_01: which one would be really cool?
• 23:39
  SPEAKER_01: But secondly, I think really taking the time to understand the models beforehand
• 23:47
  SPEAKER_01: and making sure that the experiment is set up thoughtfully.
• 23:51
  SPEAKER_01: Considering this is actually my first lead author paper,
• 23:55
  SPEAKER_01: this is something where I came into it a little bit brash
• 24:01
  SPEAKER_01: because I was really excited and kind of got starry-eyed in terms of Being able
• 24:06
  SPEAKER_01: to look at all these foundation models and being able to try to train them.
• 24:10
  SPEAKER_01: And I think that while this paper is definitely a valuable resource in terms
• 24:15
  SPEAKER_01: of what we have found, I think there's still a lot of room for improvement in terms
• 24:21
  SPEAKER_01: of the experimental design
• 24:24
  SPEAKER_01: and being able to give each model a fair
• 24:29
  SPEAKER_01: and consistent shake at classifying these assets.
• 24:33
  SPEAKER_01: And I think just Really going down to basics,
• 24:37
  SPEAKER_01: understanding the documentation of these foundation models,
• 24:40
  SPEAKER_01: what they were pre-trained on, how they've performed in other scenarios,
• 24:47
  SPEAKER_01: things like that can really help to not only help guide
• 24:52
  SPEAKER_01: the foundation model choice itself,
• 24:54
  SPEAKER_01: but to be able to prepare an experiment that is truly fair and consistent
• 25:00
  SPEAKER_01: and I think that that's something that I would definitely suggest to someone
• 25:05
  SPEAKER_01: who wants to do something similar.
• 25:08
  SPEAKER_01: I'd say, and I mentioned this
• 25:11
  SPEAKER_01: because you mentioned in some questions that you gave
• 25:16
  SPEAKER_01: to me is, you know, if someone wants to map their own infrastructure, you know,
• 25:22
  SPEAKER_01: what should they take from this?
• 25:24
  SPEAKER_01: And I want to highlight this because me personally, like I,
• 25:29
  SPEAKER_01: I'm someone who wants to bridge academia and practice.
• 25:33
  SPEAKER_01: I think that academia shouldn't be siloed to just the academic world.
• 25:38
  SPEAKER_01: I think that if a practitioner, for example,
• 25:42
  SPEAKER_01: an emergency management official wants to map their infrastructure assets,
• 25:47
  SPEAKER_01: I think that they should look into our paper firstly
• 25:53
  SPEAKER_01: and kind of Really be open
• 25:57
  SPEAKER_01: to seeing what's out there in the academic world.
• 26:01
  SPEAKER_01: And for the academics out there,
• 26:04
  SPEAKER_01: making sure that the research
• 26:08
  SPEAKER_01: that we're doing isn't necessarily bogged down
• 26:12
  SPEAKER_01: by technical details, or at the least there's an alternate version for practitioners
• 26:16
  SPEAKER_01: who may not understand the technical details,
• 26:18
  SPEAKER_01: but still would like to take a look at these models, maybe for the first time.
• 26:24
  SPEAKER_01: What I would suggest to someone who does want
• 26:26
  SPEAKER_01: to map their infrastructure is genuinely check out our paper.
• 26:31
  SPEAKER_01: And I'm not just saying that because I was an author on it, but, you know,
• 26:35
  SPEAKER_01: even though there were some limitations that I mentioned during this chat,
• 26:40
  SPEAKER_01: I think that by and large, this is one of,
• 26:43
  SPEAKER_01: if not the first instance of comparing foundation models
• 26:47
  SPEAKER_01: for critical infrastructure specifically,
• 26:50
  SPEAKER_01: especially in such a systematic way and across multiple sectors.
• 26:55
  SPEAKER_01: As we're all aware, these infrastructure assets and sectors will interact with each other.
• 27:01
  SPEAKER_01: So that's why I think it's important to take a look at some of the results
• 27:05
  SPEAKER_01: that we found and be able to look at, for example,
• 27:09
  SPEAKER_01: if you'd like to model your power generators or your water treatment plants, well,
• 27:14
  SPEAKER_01: which foundation model is going to do the best at classifying that and helping you
• 27:18
  SPEAKER_01: to build out that data set for yourself?
• 27:23
  SPEAKER_01: Secondly, another thing that we looked at that I didn't really touch on much
• 27:29
  SPEAKER_01: in this chat so far is, in addition to training these models
• 27:35
  SPEAKER_01: on the full training data set that we have,
• 27:38
  SPEAKER_01: we also looked at how they would perform on a training set with only 30%
• 27:45
  SPEAKER_01: of the labels, so a subset of that.
• 27:48
  SPEAKER_01: And this is something that We thought it would be really interesting because a lot
• 27:53
  SPEAKER_01: of people are coming from positions
• 27:55
  SPEAKER_01: where they might not have very comprehensive label data,
• 28:00
  SPEAKER_01: especially in regions like the Global South,
• 28:02
  SPEAKER_01: where it may be a little harder to come by.
• 28:05
  SPEAKER_01: So we included that as sort
• 28:09
  SPEAKER_01: of a secondary result or secondary experiment
• 28:13
  SPEAKER_01: in this whole paper to be able to look at if foundation models are only given 30%
• 28:19
  SPEAKER_01: of the labels, can they still perform well?
• 28:22
  SPEAKER_01: And overall, they did.
• 28:24
  SPEAKER_01: So it was an interesting finding.
• 28:26
  SPEAKER_01: And I think that if you're coming to this from a practitioner's standpoint or even
• 28:31
  SPEAKER_01: as an academic and you're struggling to find data out there, which at least for me,
• 28:35
  SPEAKER_01: I know is definitely the case outside of OpenStreetMap especially,
• 28:40
  SPEAKER_01: being able to only use a subset of labels and still use these foundation models
• 28:46
  SPEAKER_01: for good results I think it's definitely something
• 28:49
  SPEAKER_01: that could be helpful in terms
• 28:53
  SPEAKER_01: of future research or future practice.
• 28:56
  SPEAKER_00: Fantastic. So lots of practical implications from this work. People should dig in.
• 29:02
  SPEAKER_00: The paper itself, I think the preprint is out.
• 29:04
  SPEAKER_00: The paper is in review at the moment.
• 29:08
  SPEAKER_00: The data set itself, I think that's published?
• 29:11
  SPEAKER_01: Yes. The data set itself is on Zenodo as we speak.
• 29:16
  SPEAKER_01: And yes, the preprint is out.
• 29:18
  SPEAKER_01: We're currently in the peer review process.
• 29:22
  SPEAKER_01: So hopefully that'll be out soon as well.
• 29:25
  SPEAKER_00: Fantastic.
• 29:26
  SPEAKER_00: If people want to follow the latest updates, I presume LinkedIn is the best place?
• 29:31
  SPEAKER_01: Yeah. Yeah, shoot.
• 29:33
  SPEAKER_01: Feel free to follow me on LinkedIn.
• 29:37
  SPEAKER_01: I, you know, I engage with a lot of LinkedIn comments.
• 29:41
  SPEAKER_01: So even if, you know, You miss some of the posts that I put out.
• 29:47
  SPEAKER_01: I'm always liking and commenting on other people's work as well and can provide
• 29:51
  SPEAKER_01: a bridge to check out other people's work in the Earth Observation,
• 29:56
  SPEAKER_01: Civil Engineering, GIS space.
• 29:58
  SPEAKER_01: So if that's of interest to anyone, feel free to follow me on LinkedIn or GitHub.
• 30:07
  SPEAKER_01: I'll put some stuff out on GitHub.
• 30:09
  SPEAKER_01: There's all sorts of ways.
• 30:10
  SPEAKER_01: And honestly, if you have any questions,
• 30:13
  SPEAKER_01: I'm not sure what sort of contact information will be shared after this,
• 30:17
  SPEAKER_01: but feel free to shoot me an email as well.
• 30:22
  SPEAKER_01: It's jay.guthrie at northeastern.edu,
• 30:25
  SPEAKER_01: and we can maybe see if we can get my email sent out as well.
• 30:28
  SPEAKER_01: But yeah, I mean, I'm happy to talk about my paper, any of the work,
• 30:35
  SPEAKER_01: or look into opportunities for collaboration. So yeah.
• 30:39
  SPEAKER_00: Fantastic. Brilliant.
• 30:40
  SPEAKER_00: Well, I'll put all those links in the show notes.
• 30:44
  SPEAKER_00: Once again, thanks for joining me, Justin.
• 30:45
  SPEAKER_00: Until next time, take care.
• 30:48
  SPEAKER_01: Thanks. You too, Robin.