Latest / Satellite image deep learning / Building Damage Assessment

Transcript

Building Damage Assessment · Satellite image deep learning

• 0:04
  Hello,
• 0:05
  I'm Robin Cole and welcome to the Satellite Image Deep Learning Podcast.
• 0:09
  SPEAKER_00: In this episode,
• 0:10
  I caught up with Caleb Robinson to learn about the Building
• 0:12
  Damage Assessment
• 0:13
  SPEAKER_00: Toolkit from the Microsoft AI for Good Lab.
• 0:16
  This toolkit enables first responders to carry out an
• 0:18
  end-to-end workflow for
• 0:20
  assessing damage to buildings after natural disasters using
• 0:23
  post-disaster satellite imagery.
• 0:25
  SPEAKER_00: It includes tools for annotating imagery,
• 0:27
  SPEAKER_00: fine-tuning deep learning models,
• 0:29
  SPEAKER_00: and visualizing model predictions on a map.
• 0:32
  Caleb shared an example where an organization was able to train
• 0:34
  a useful model with
• 0:35
  SPEAKER_00: just 100 annotations and complete the entire workflow in half a day.
• 0:40
  SPEAKER_00: I believe this represents a significant new capability,
• 0:43
  SPEAKER_00: enabling more rapid response in times of crisis.
• 0:46
  SPEAKER_00: If you're watching this on YouTube,
• 0:47
  SPEAKER_00: please consider liking the video and subscribing to the channel,
• 0:50
  SPEAKER_00: as this really helps to reach a wider audience.
• 0:52
  SPEAKER_00: Thank you.
• 0:53
  SPEAKER_00: Hi Caleb, welcome back to the podcast.
• 0:56
  SPEAKER_01: Awesome.
• 0:56
  SPEAKER_01: Thanks for having me, Robin.
• 0:58
  SPEAKER_01: Good to be here again.
• 0:59
  SPEAKER_00: Fantastic.
• 1:00
  So today we're going to be talking about damage assessment in
• 1:04
  the context of disasters.
• 1:05
  SPEAKER_00: Why don't you tell me what you've been up to?
• 1:09
  SPEAKER_01: Yeah.
• 1:09
  So I'm a principal research scientist at Microsoft on the AI
• 1:13
  for Good team.
• 1:14
  SPEAKER_01: One of our big focus areas is doing disaster response work.
• 1:20
  SPEAKER_01: And over the past couple of years,
• 1:23
  SPEAKER_01: throughout responding to many disasters around the world,
• 1:26
  SPEAKER_01: we've developed this building damage assessment toolkit.
• 1:30
  And it's really a workflow that allows people to reproduce the
• 1:35
  damage assessments
• 1:36
  SPEAKER_01: that we do quickly using post-disaster optical satellite imagery.
• 1:43
  So this is RGB imagery typically captured a day or so after the
• 1:47
  actual kind of disaster event.
• 1:50
  SPEAKER_01: Yeah, and it can be RGB, it can be multispectral.
• 1:54
  The pipeline itself is really agnostic to the type of imagery
• 1:57
  or the provider of the imagery.
• 2:00
  The whole point of it is to be able to train a model to the
• 2:04
  specific disaster that
• 2:07
  SPEAKER_01: has happened,
• 2:09
  SPEAKER_01: run the model over all the imagery covering that particular area,
• 2:13
  SPEAKER_01: and then get the results to a partner as quickly as possible.
• 2:16
  SPEAKER_00: So models train for specific events, producing results very quickly.
• 2:21
  SPEAKER_00: Fantastic.
• 2:22
  SPEAKER_00: I know you've got a short presentation.
• 2:24
  SPEAKER_00: Why don't we fire it up and start talking through this workflow?
• 2:27
  SPEAKER_01: Cool.
• 2:28
  SPEAKER_01: So yeah, went through all this.
• 2:31
  SPEAKER_01: Like I said, we've open sourced this workflow.
• 2:35
  I think stepping through the code line by line is not
• 2:39
  particularly interesting,
• 2:40
  but the point is it's out there on GitHub at github.com slash
• 2:45
  Microsoft slash
• 2:46
  SPEAKER_01: building damage assessment.
• 2:47
  SPEAKER_01: And really this repo, again, gives people
• 2:52
  step-by-step tutorials to reproduce the work that we've done in
• 2:57
  responding to
• 2:58
  SPEAKER_01: different disasters around the world.
• 3:00
  The demo is how we responded to the wildfires that hit Hawaii,
• 3:05
  specifically Maui,
• 3:07
  last August.
• 3:10
  SPEAKER_00: How many years work has gone into getting to this workflow?
• 3:14
  SPEAKER_01: Yeah, good question.
• 3:15
  SPEAKER_01: We started with the earthquakes in Turkey, if you remember those.
• 3:20
  I think that was sort of springtime,
• 3:24
  at least in the northern hemisphere,
• 3:26
  for 2022.
• 3:27
  SPEAKER_01: And we've sort of been iterating on this process ever since.
• 3:32
  So it's gone through quite a bit of refinement over a couple of
• 3:35
  years then.
• 3:37
  SPEAKER_01: This is the kind of basic problem set up.
• 3:40
  You assume that you're given sort of post-event high resolution
• 3:44
  satellite or aerial imagery.
• 3:46
  SPEAKER_01: It's agnostic.
• 3:48
  SPEAKER_01: Here you can see visual signs of damage of the disaster and the imagery.
• 3:53
  SPEAKER_01: And our goal is always to do these building levels assessments of damage.
• 3:58
  SPEAKER_01: We want to summarize for each building footprint,
• 4:01
  SPEAKER_01: whether or not we think it's damaged and for some types of disasters,
• 4:05
  SPEAKER_01: how damaged we think it is.
• 4:09
  And I really describe kind of the purpose of making this a
• 4:14
  workflow using this chart.
• 4:18
  SPEAKER_01: So here you have on the x-axis,
• 4:20
  SPEAKER_01: number of hours after disaster,
• 4:22
  SPEAKER_01: y-axis,
• 4:23
  SPEAKER_01: number of buildings that have been assessed as damaged or not.
• 4:27
  The goal is of course to get to 100% of buildings assessed and
• 4:30
  you wanna do that as
• 4:31
  SPEAKER_01: quickly as possible.
• 4:32
  SPEAKER_01: And when we started off with this work,
• 4:34
  we were trying to train these models that would work under any conditions.
• 4:40
  And that really led us to this red line here,
• 4:43
  which is the model doesn't work workflow.
• 4:47
  SPEAKER_01: You have this model,
• 4:47
  SPEAKER_01: you get imagery from a new disaster,
• 4:49
  SPEAKER_01: you try to run it and the results just aren't where they need to be.
• 4:53
  And here you really don't have the time in disaster response
• 4:57
  scenarios to turn it
• 4:57
  SPEAKER_01: into a research project and maybe try domain adaptation methods.
• 5:02
  You're just not going to get to that 100% damage assessment
• 5:07
  result with the data.
• 5:08
  quality that you need to ship out to the first responders and
• 5:13
  really in that
• 5:14
  scenario your time would be better spent with this blue line
• 5:17
  doing a manual
• 5:18
  labeling of the buildings you go one by one maybe do 10 seconds
• 5:23
  per building and
• 5:23
  you can eventually get to that 100 line and really where we
• 5:26
  want to be is this
• 5:27
  SPEAKER_01: green line which is a human in the loop workflow where human labelers
• 5:32
  SPEAKER_01: Domain experts,
• 5:33
  people can tune the models for a specific disaster and have
• 5:38
  this like low variance
• 5:39
  SPEAKER_01: way of getting results reliably and of good quality.
• 5:45
  SPEAKER_00: Yeah.
• 5:47
  And for an example of why a lot of the ways that you train will
• 5:54
  end up in this red zone,
• 5:55
  SPEAKER_01: I've just got a couple of examples of post-disaster imagery.
• 5:58
  SPEAKER_01: This is not post-disaster.
• 6:00
  SPEAKER_01: This is aerial imagery of the Microsoft campus.
• 6:03
  SPEAKER_01: And it's really high quality, high resolution, really no artifacts.
• 6:08
  But when you get into imagery captured after disaster,
• 6:11
  it's often off nadir.
• 6:13
  SPEAKER_01: There might be artifacts, different lighting conditions.
• 6:17
  And your model has to deal with all of these scenarios if it's
• 6:21
  going to be this
• 6:22
  SPEAKER_01: general purpose thing.
• 6:26
  SPEAKER_01: We often don't have labels for all these scenarios.
• 6:30
  SPEAKER_01: So such a model is very hard to train.
• 6:32
  SPEAKER_00: Right.
• 6:33
  So here I'm just flicking through some of the images that we
• 6:36
  actually had to deal
• 6:36
  SPEAKER_01: with in the Turkey earthquake response.
• 6:40
  So these are from different satellites,
• 6:43
  different times of day,
• 6:44
  different positions.
• 6:45
  SPEAKER_01: Exactly.
• 6:46
  SPEAKER_01: And even Mosaic, all of them really shows how...
• 6:52
  You have to deal with these different conditions,
• 6:54
  different times of day,
• 6:55
  clouds,
• 6:56
  artifacts.
• 6:58
  SPEAKER_01: Your model has to work over all of these variety of conditions,
• 7:02
  SPEAKER_01: no matter what,
• 7:03
  SPEAKER_01: because the important thing here is getting the results out.
• 7:07
  That really led us to this workflow that's implemented in that
• 7:11
  GitHub repo from the
• 7:12
  SPEAKER_01: first slide,
• 7:13
  SPEAKER_01: and it couldn't be more simple.
• 7:15
  Here,
• 7:16
  the steps are you set up this labeling tool with post-disaster imagery.
• 7:21
  SPEAKER_01: That's a different open-source project that we have.
• 7:24
  This satellite imagery labeling tool is a very lightweight way
• 7:26
  to create polygon labels over
• 7:30
  SPEAKER_01: pretty much any imagery that can be served in XYZ tile format.
• 7:35
  SPEAKER_01: And then just repeat the steps of annotating examples of buildings,
• 7:40
  SPEAKER_01: damaged buildings,
• 7:41
  SPEAKER_01: background class,
• 7:42
  SPEAKER_01: and occasionally clouds,
• 7:43
  SPEAKER_01: depending on the quality of your scene,
• 7:45
  fine tuning a model from scratch and evaluating that
• 7:48
  performance on a held out
• 7:50
  SPEAKER_01: validation set.
• 7:52
  SPEAKER_01: And if you just repeat this for a couple of iterations,
• 7:55
  SPEAKER_01: we found that you're often getting really,
• 7:57
  SPEAKER_01: really good performance with a small number of labels.
• 8:01
  SPEAKER_01: So here's an example of Rolling Fork, Mississippi.
• 8:04
  SPEAKER_01: There was a tornado that hit this town,
• 8:07
  and we partnered with American Red Cross in order to do a
• 8:12
  disaster response here.
• 8:14
  SPEAKER_01: So for training,
• 8:15
  we did 153 polygon annotations spreading in between the
• 8:19
  background undamaged
• 8:20
  SPEAKER_01: building and damage building classes,
• 8:24
  SPEAKER_01: tuned a unit with basic ImageNet pre-training setup.
• 8:28
  SPEAKER_01: All of this was implemented with Torch.io.
• 8:31
  And this is the results that you get after maybe like 30
• 8:34
  minutes of fine tuning the model.
• 8:37
  SPEAKER_00: So let me just check.
• 8:38
  So that is literally representing the number of annotations
• 8:41
  that this model was trained on.
• 8:42
  SPEAKER_00: That's seen there approximately.
• 8:44
  SPEAKER_01: Yeah, 100 annotations.
• 8:45
  SPEAKER_00: Yeah.
• 8:47
  SPEAKER_01: And this is what you're seeing now is the output of those 100 annotations
• 8:52
  SPEAKER_01: summarized over the building footprints,
• 8:54
  existing kind of post or pre-disaster building footprints for
• 8:57
  this location.
• 8:59
  SPEAKER_00: And is this model,
• 9:00
  is it just trained from scratch or it's been trained in other
• 9:02
  disaster situations?
• 9:04
  SPEAKER_00: Yeah.
• 9:05
  SPEAKER_01: No, just trained from an ImageNet initialization.
• 9:08
  SPEAKER_00: OK, from scratch, right.
• 9:10
  SPEAKER_01: Yep.
• 9:12
  SPEAKER_01: And the surprising thing here is that you only need 150 labels,
• 9:18
  as the point of this model isn't to be a general purpose
• 9:22
  disaster response model.
• 9:24
  It's to get good annotations for this specific scene over
• 9:29
  Rolling Fork,
• 9:29
  Mississippi.
• 9:31
  And once that's your goal,
• 9:33
  the number of labels that you need really,
• 9:34
  really decreases.
• 9:36
  SPEAKER_00: On the technical side,
• 9:37
  SPEAKER_00: do you have to do lots of augmentation to get good results,
• 9:39
  SPEAKER_00: or is it pretty much just it's a very kind of consistent scene,
• 9:43
  SPEAKER_00: so it's not that difficult a task?
• 9:45
  SPEAKER_01: Yes, consistent scene, so it's not that difficult of a task.
• 9:48
  SPEAKER_01: I think there's augmentation built into the workflow,
• 9:52
  SPEAKER_01: but I would be surprised if that was a necessary component.
• 9:56
  Mm-hmm.
• 9:58
  And of course,
• 9:59
  there's a lot of room for research work and improvement here.
• 10:03
  SPEAKER_01: Like,
• 10:04
  SPEAKER_01: is there just a lightweight,
• 10:06
  self-supervised way that you can initialize a model specific
• 10:09
  for this scene,
• 10:10
  SPEAKER_01: for example,
• 10:10
  SPEAKER_01: or specific for the sensor?
• 10:12
  SPEAKER_01: So you do have a better starting point than ImageNet.
• 10:16
  But again,
• 10:19
  150 labels is not that much of a effort from an annotation standpoint.
• 10:24
  SPEAKER_01: If it can get you to a good point, then
• 10:28
  SPEAKER_01: That's something that we can work with.
• 10:30
  SPEAKER_00: Yeah.
• 10:31
  SPEAKER_00: So if you had a very large image,
• 10:32
  you might only annotate 10% of it and then apply this model to
• 10:35
  the remaining 90%
• 10:36
  SPEAKER_00: and see good results.
• 10:38
  SPEAKER_01: Yeah, and that's exactly the workflow.
• 10:40
  So we usually start off with a very local part of the image
• 10:43
  where we know that
• 10:45
  SPEAKER_01: there are examples of that disaster class.
• 10:50
  SPEAKER_01: And then after the first iteration,
• 10:51
  SPEAKER_01: if the results are not working well somewhere else,
• 10:54
  SPEAKER_01: then the image will jump and label a little bit there.
• 10:56
  SPEAKER_01: Mm-hmm.
• 10:57
  So that's that iterative feedback that you get from just kind
• 11:01
  of looking at the
• 11:02
  predictions or labeling randomly for a validation set is what
• 11:07
  drives the stopping
• 11:07
  SPEAKER_01: criteria there.
• 11:08
  SPEAKER_00: I might be jumping the gun here,
• 11:10
  but do you visualize the predictions of the model on the same
• 11:13
  tool while you're
• 11:13
  annotating so you can kind of have a single place where you do
• 11:16
  the entire workflow?
• 11:19
  SPEAKER_01: Not the same tool,
• 11:21
  SPEAKER_01: but the inputs and outputs of the workflow are geotiff,
• 11:25
  SPEAKER_01: so it's quite easy to drop into QGIS,
• 11:30
  SPEAKER_01: your favorite GIS software.
• 11:34
  SPEAKER_01: That's our standard way of approaching it.
• 11:39
  SPEAKER_01: So for this disaster,
• 11:42
  SPEAKER_01: specifically,
• 11:43
  the American Red Cross was on the ground here and they
• 11:45
  collected 909 ground truth
• 11:48
  SPEAKER_01: labels from actually going door to door and inspecting damage.
• 11:52
  And if you compare our model with that,
• 11:54
  we see we get 96 percent average precision here.
• 12:00
  SPEAKER_01: We did the same thing for the wildfires in Maui.
• 12:04
  SPEAKER_01: Here, only 108 annotations were needed.
• 12:08
  We happened to stop after that from a selection of 250 random
• 12:15
  building footprints
• 12:15
  SPEAKER_01: that we annotated based on just interpreting the imagery,
• 12:19
  SPEAKER_01: 99% recall and 96% precision on the damaged or not damaged task.
• 12:26
  SPEAKER_01: And this whole thing intent,
• 12:27
  SPEAKER_01: was just four hours for the MAUI response.
• 12:32
  SPEAKER_01: And this can actually have impact in the world.
• 12:34
  SPEAKER_01: So at the AI for Good Lab, our goal really is to make impact.
• 12:38
  And here we were able to see that the American Red Cross was
• 12:41
  able to use this to
• 12:42
  SPEAKER_01: speed up their development
• 12:46
  SPEAKER_01: kind of damage assessment process,
• 12:47
  SPEAKER_01: which is a requirement for them to release funds to people who need it.
• 12:52
  SPEAKER_01: Same thing with Maui United Way.
• 12:54
  They used these outputs to actually generate receipts for a
• 13:00
  disbursement that they
• 13:00
  SPEAKER_01: were doing on ground.
• 13:02
  And I just wanted to end off here with a visualize that we made
• 13:07
  just last week for
• 13:08
  SPEAKER_01: the floods that hit Valencia,
• 13:10
  SPEAKER_01: Spain.
• 13:11
  SPEAKER_01: So real-time output from our model here.
• 13:15
  This is the visualizer,
• 13:17
  and we usually create these to share with partners that we have.
• 13:22
  SPEAKER_01: But here,
• 13:22
  SPEAKER_01: instead of building damage assessment,
• 13:25
  SPEAKER_01: we're doing flood extent mapping,
• 13:26
  SPEAKER_01: but it's the exact same workflow.
• 13:28
  SPEAKER_01: Instead of labeling actually buildings as damaged,
• 13:31
  SPEAKER_01: you attempt to label where you see flood extent.
• 13:36
  SPEAKER_01: Here, this is just a typical pre-post map where
• 13:42
  SPEAKER_01: you can see what things look like in a base map,
• 13:47
  SPEAKER_01: what things look like in the post-disaster imagery,
• 13:49
  SPEAKER_01: and then the model predictions.
• 13:53
  SPEAKER_01: So red is flood extent, gray is clouds.
• 13:57
  There are some places here that were really,
• 14:02
  really hit hard,
• 14:04
  particularly along the rivers.
• 14:06
  SPEAKER_01: So if you zoom in here, I think this was
• 14:10
  um a big parking lot it's along the dry riverbed and then here
• 14:15
  you can see the
• 14:16
  flood waters kind of wiped out half yeah really great results
• 14:25
  and again there's
• 14:26
  like two 200 labels um a couple hours worth of work to produce
• 14:31
  this and then uh you
• 14:32
  SPEAKER_01: can run
• 14:33
  SPEAKER_01: this analysis on a much, much larger scene.
• 14:36
  SPEAKER_01: If you had planet scope imagery kind of extending past this,
• 14:39
  SPEAKER_01: I would expect the same model to work in that scenario.
• 14:43
  SPEAKER_01: So it really saves a lot of time when you're doing these responses.
• 14:48
  SPEAKER_00: Yeah,
• 14:48
  SPEAKER_00: I mean,
• 14:49
  getting the sort of output in half a day is just stunning and
• 14:54
  really flies against
• 14:55
  SPEAKER_00: typical expectations about how time consuming these projects can be.
• 15:01
  SPEAKER_01: Yeah, yeah.
• 15:03
  SPEAKER_01: So yeah,
• 15:04
  SPEAKER_01: like I said,
• 15:05
  the repository actually has step-by-step instructions where you
• 15:09
  can reproduce the
• 15:10
  SPEAKER_01: work that we did in Maui.
• 15:12
  SPEAKER_01: We also look at how to do flood extent mapping from SAR imagery.
• 15:17
  SPEAKER_01: There's a lot of great work that's been done in the community here.
• 15:22
  SPEAKER_01: But yeah,
• 15:23
  we hope people will contribute to this repo and will definitely
• 15:27
  contribute to it as
• 15:28
  SPEAKER_01: we work through more of these disasters.
• 15:30
  SPEAKER_00: Fantastic.
• 15:31
  SPEAKER_00: Well, thank you so much for the walkthrough.
• 15:33
  SPEAKER_00: One final question.
• 15:34
  SPEAKER_00: You mentioned you've been working on this for two years.
• 15:36
  I'm wondering what you feel like you've learned in the process
• 15:38
  of that process.
• 15:41
  SPEAKER_01: Yeah, a lot.
• 15:43
  SPEAKER_01: So always working with domain experts,
• 15:45
  SPEAKER_01: whether it's in disaster response or other topics,
• 15:49
  SPEAKER_01: you learn a lot about how the responses are actually executed,
• 15:56
  SPEAKER_01: what types of damage you should be looking through in the image.
• 16:00
  SPEAKER_01: Sometimes that's down trees.
• 16:01
  SPEAKER_01: Sometimes that's flood extent.
• 16:04
  debris,
• 16:06
  what buildings are actually important for responses after hurricanes.
• 16:13
  Not all building footprints are relevant for calibrating the
• 16:17
  level of response.
• 16:19
  SPEAKER_01: You're really looking for residential areas.
• 16:21
  SPEAKER_01: Sheds, for example, are important to include.
• 16:27
  SPEAKER_01: Yeah, the list goes on and on.
• 16:30
  SPEAKER_01: From the technical side,
• 16:31
  SPEAKER_01: it's about how to create,
• 16:34
  SPEAKER_01: again,
• 16:34
  these low-variance workflows that allow you to get good results
• 16:38
  consistently and
• 16:40
  not have cases where your model is just outputting things that
• 16:44
  you can't ship.
• 16:45
  SPEAKER_01: Yeah.
• 16:46
  And of course,
• 16:47
  this workflow can be adapted for non-disaster tasks as well.
• 16:52
  SPEAKER_00: Yeah.
• 16:53
  SPEAKER_00: Yeah, fantastic.
• 16:55
  SPEAKER_00: Okay, well, it's been brilliant having you on again.
• 16:57
  SPEAKER_00: I think we'll do a third one in the future,
• 16:59
  SPEAKER_00: but for now,
• 17:01
  SPEAKER_00: thank you for joining me and all the best.
• 17:04
  SPEAKER_01: Thanks so much, Robin.
• 17:05
  SPEAKER_01: You too.