Latest / Satellite image deep learning / Methane Plume Detection with AutoML

Transcript

Methane Plume Detection with AutoML · Satellite image deep learning

• 0:04
  SPEAKER_01: Hi, Julia.
• 0:04
  SPEAKER_01: Welcome to the podcast.
• 0:05
  SPEAKER_00: Hi, Robin.
• 0:07
  SPEAKER_00: So nice to be here.
• 0:08
  SPEAKER_01: Let's begin with where you work and what you do.
• 0:11
  SPEAKER_00: So I'm a PhD student at Leiden University.
• 0:13
  I study computer science and I work together with atmospheric
• 0:18
  scientists at Space
• 0:20
  SPEAKER_00: Research Organisation Netherlands.
• 0:23
  And we work on machine learning pipelines to find atmospheric
• 0:27
  gases like methane.
• 0:29
  SPEAKER_01: Right.
• 0:29
  SPEAKER_01: So methane detection, it's kind of a big topic at the moment.
• 0:33
  SPEAKER_01: What's significant about methane?
• 0:36
  SPEAKER_00: So methane is a very strong greenhouse gas,
• 0:39
  but it only stays in the atmosphere for a relatively short time
• 0:43
  compared to CO2.
• 0:44
  So any methane we can avoid emitting right now could make big
• 0:52
  impacts on our global
• 0:54
  SPEAKER_00: warming on a short time scale.
• 0:58
  SPEAKER_01: Great.
• 0:58
  SPEAKER_01: So how is your research connected to that?
• 1:02
  So I design machine learning pipelines for plume detection in
• 1:08
  the data of a very
• 1:09
  SPEAKER_00: specific instrument,
• 1:10
  SPEAKER_00: TROPOMI,
• 1:11
  SPEAKER_00: on a board of Sentinel-5P.
• 1:14
  And this is a spectrometer that measures concentrations of
• 1:17
  different gases.
• 1:19
  SPEAKER_00: It looks very different.
• 1:20
  SPEAKER_00: If you look at the images,
• 1:21
  SPEAKER_00: it looks very different from optical instruments like Sentinel-2,
• 1:26
  SPEAKER_00: which is much better known.
• 1:27
  SPEAKER_00: So it has its own quirks and requires separate pipelines.
• 1:33
  SPEAKER_01: Right.
• 1:33
  Could you briefly describe what a typical image might look like
• 1:36
  and how it differs?
• 1:37
  SPEAKER_00: Yeah, so one pixel for methane is about 5 by 7 kilometers.
• 1:43
  SPEAKER_00: So if we're looking for methane point sources,
• 1:47
  SPEAKER_00: that sometimes can even be a whole city at that scale.
• 1:54
  SPEAKER_00: There's just one channel,
• 1:56
  SPEAKER_00: and the channel has methane concentration,
• 1:58
  but the satellite data product also has other fields that I use
• 2:02
  with auxiliary
• 2:03
  SPEAKER_00: information like cloud masks,
• 2:06
  SPEAKER_00: surface pressure,
• 2:08
  SPEAKER_00: albedo,
• 2:09
  SPEAKER_00: all kind of information.
• 2:12
  that is used in the retrieval of the methane concentrations
• 2:15
  from the light spectra
• 2:17
  SPEAKER_00: that are measured.
• 2:17
  SPEAKER_00: And sometimes when a piece of land has very high reflectivity,
• 2:25
  SPEAKER_00: then this can lead to errors in the retrieval.
• 2:28
  So then it might sometimes look like there's a methane bloom
• 2:31
  when there's not.
• 2:32
  SPEAKER_00: But in fact, it looks...
• 2:35
  grainy is not the right word because it's pixely image very low
• 2:41
  resolution but high
• 2:43
  spectral resolution right so you might even have just like a
• 2:46
  single pixel that's
• 2:48
  telling you there's a plume sometimes yes those are tricky
• 2:52
  incredible it sounds
• 2:53
  SPEAKER_01: very challenging how have you gone about detecting them
• 2:57
  So the atmospheric scientists I work with already developed a
• 3:01
  pipeline that kind of
• 3:02
  SPEAKER_00: uses divide-and-conquer approach,
• 3:05
  SPEAKER_00: first with a CNN to find all the images with a plume-like shape,
• 3:09
  and then they use the support vector machine with
• 3:12
  physics-derived features to
• 3:16
  SPEAKER_00: really discern the real plumes from the artifacts,
• 3:19
  SPEAKER_00: the things that look like plumes but are not.
• 3:22
  So I wanted to extend this approach also to other gases
• 3:26
  measured by tropomy.
• 3:27
  So I wanted to use automated feature extraction,
• 3:30
  which brings us to neural networks.
• 3:31
  SPEAKER_00: So end-to-end CNNs.
• 3:34
  SPEAKER_00: But still there are differences between the gases.
• 3:38
  SPEAKER_00: Plumes looked
• 3:41
  background levels of the gas too if you compare for example
• 3:45
  methane and carbon
• 3:46
  monoxide so i developed a neuroarchitecture search approach
• 3:51
  that automatically
• 3:52
  designs neural networks to find plumes in these different
• 3:55
  tropomy gases interesting
• 3:58
  so just so i understand the first network is doing a
• 4:00
  classification just saying
• 4:02
  SPEAKER_01: this image has or does not have a plume in it
• 4:07
  Yeah,
• 4:07
  so the original approach does as the two steps,
• 4:10
  and we have bundled this into one.
• 4:13
  So we have now one neural network that has access to multiple
• 4:18
  fields of the Tripomi
• 4:19
  SPEAKER_00: data product.
• 4:20
  SPEAKER_00: Well, in the original pipeline, this was
• 4:23
  SPEAKER_00: The CNN had only access to methane,
• 4:26
  SPEAKER_00: and then there was a classical machine learning model on top,
• 4:29
  which would use many other fields from the data product to
• 4:34
  calculate all sorts of
• 4:35
  SPEAKER_00: related features.
• 4:37
  SPEAKER_00: This was very specific to methane,
• 4:39
  SPEAKER_00: and by designing a neural network that takes a raster data cube as input,
• 4:45
  SPEAKER_00: we could generalize the solution to have our gases too.
• 4:49
  SPEAKER_01: Okay,
• 4:49
  SPEAKER_01: so going from the two-stage solution to a single neural network,
• 4:52
  SPEAKER_01: but lots of optimization to find the right parameters for that network.
• 4:57
  SPEAKER_01: Yes.
• 4:58
  SPEAKER_01: Okay.
• 4:59
  SPEAKER_01: Do you mind giving us an idea about neural architecture search then?
• 5:04
  SPEAKER_00: Yeah,
• 5:05
  SPEAKER_00: so in this approach,
• 5:07
  SPEAKER_00: we use optimization algorithm,
• 5:11
  SPEAKER_00: Bayesian optimization to fine tune both the architecture and the training
• 5:16
  SPEAKER_00: hyperparameters of these CNNs that we use for data fusion.
• 5:21
  SPEAKER_00: And in this neuroarchitecture search, we tweak a few things.
• 5:28
  We have designed a backbone,
• 5:30
  CNN,
• 5:31
  that has effectively multiple input branches.
• 5:34
  SPEAKER_00: So if you have multiple input modalities,
• 5:36
  SPEAKER_00: so each modality is a different layer from the data product,
• 5:39
  SPEAKER_00: it gets its own input branch.
• 5:41
  SPEAKER_00: And the neuroarchitecture search optimizes how deep these branches are,
• 5:47
  SPEAKER_00: and how the branches are then fused in the neural network,
• 5:51
  SPEAKER_00: and also what the backbone architecture is,
• 5:53
  because you can have a vanilla CNN or ResNet or even a vision
• 5:58
  transformer.
• 5:59
  SPEAKER_00: And the optimization also takes care of
• 6:03
  the hyperparameters because when you change the depth of these
• 6:09
  multi-branch
• 6:09
  SPEAKER_00: networks,
• 6:11
  SPEAKER_00: then the optimal hyperparameters can also change drastically.
• 6:14
  SPEAKER_00: So it's very important to tune those in tandem.
• 6:18
  SPEAKER_01: Okay.
• 6:18
  SPEAKER_01: So you basically define the inputs,
• 6:20
  give it some range to search over and it somehow intelligently
• 6:25
  finds the right
• 6:26
  SPEAKER_01: parameters or identifies a few different possible solutions.
• 6:30
  SPEAKER_00: Yes.
• 6:31
  SPEAKER_01: Right, okay.
• 6:32
  SPEAKER_01: And what are the results like using this approach?
• 6:36
  SPEAKER_00: So on the test set, you got really good results of 95% accuracy.
• 6:43
  SPEAKER_00: This was for methane and similar results for carbon monoxide.
• 6:47
  SPEAKER_00: So we're very happy with that.
• 6:49
  SPEAKER_00: But of course,
• 6:51
  SPEAKER_00: for us,
• 6:52
  SPEAKER_00: or for me personally,
• 6:53
  SPEAKER_00: the goal was to make a pipeline that would also be usable in real life.
• 6:57
  SPEAKER_00: So we also conducted...
• 7:01
  SPEAKER_00: deployment scenario analysis,
• 7:03
  SPEAKER_00: reuse case application,
• 7:05
  SPEAKER_00: where we took just a week of satellite data,
• 7:10
  SPEAKER_00: which was only filtered for quality.
• 7:12
  SPEAKER_00: And there have to be sufficient valid pixels also in each image.
• 7:17
  And we fed that into our model and we labeled all the images
• 7:21
  that were detected as plume.
• 7:23
  SPEAKER_00: So not everything,
• 7:24
  SPEAKER_00: because there were more or less 17,000 images,
• 7:27
  SPEAKER_00: and it's not doable to label all of those in a reasonable amount of time.
• 7:32
  SPEAKER_00: So we labeled the detections,
• 7:33
  and we compared our results with analysis from the pre-existing
• 7:38
  pipeline to see,
• 7:39
  SPEAKER_00: like,
• 7:39
  SPEAKER_00: is this something,
• 7:40
  SPEAKER_00: this automatically created network,
• 7:43
  SPEAKER_00: could we use this operationally?
• 7:47
  SPEAKER_00: So there the results were surprising, because methane plums are very...
• 7:52
  SPEAKER_00: Very rare.
• 7:54
  But our data set was mostly balanced because you cannot train a
• 7:56
  neuron network on
• 7:57
  SPEAKER_00: extremely imbalanced data.
• 8:00
  But in a real world scenario,
• 8:02
  we did find a lot of plumes,
• 8:05
  but also about as many artifacts.
• 8:08
  So the model flagged things that looked kind of like plumes,
• 8:11
  but were not.
• 8:14
  SPEAKER_00: More than we'd like.
• 8:16
  SPEAKER_00: because that means that in the operational case,
• 8:18
  SPEAKER_00: the precision was about 50%,
• 8:21
  SPEAKER_00: 60%,
• 8:21
  SPEAKER_00: even though on the label test set,
• 8:25
  SPEAKER_00: we got like 95%.
• 8:26
  SPEAKER_00: So there's a generalization gap there,
• 8:28
  which of course also comes partly from the difference in class
• 8:32
  balances between the
• 8:32
  SPEAKER_00: datasets.
• 8:34
  And this is an issue we're still working on right now on how to
• 8:40
  get this gap
• 8:41
  SPEAKER_00: smaller,
• 8:43
  SPEAKER_00: because on the one hand,
• 8:45
  SPEAKER_00: If you increase the precision, then you might lose too many plumes.
• 8:49
  SPEAKER_00: But if the precision is not high enough,
• 8:51
  SPEAKER_00: then you have to label a lot of images that might not be plumes.
• 8:55
  SPEAKER_00: So this is something I'd like to improve in the next project.
• 8:59
  SPEAKER_01: And do you think that needs to be addressed in training or in the data?
• 9:02
  SPEAKER_01: What are you thinking?
• 9:04
  SPEAKER_00: Yeah, so we've looked at the data in detail.
• 9:08
  SPEAKER_00: I think it has to do with the data in some way,
• 9:13
  SPEAKER_00: because of course we selected,
• 9:15
  SPEAKER_00: or not me,
• 9:16
  SPEAKER_00: but the PhD student who made the original dataset,
• 9:20
  SPEAKER_00: selected images based on their knowledge of where blooms occur.
• 9:25
  SPEAKER_00: So there is always some kind of selection bias.
• 9:29
  SPEAKER_00: Even though they selected a very large rift,
• 9:33
  SPEAKER_00: different locations and types of plumes,
• 9:36
  SPEAKER_00: I suspect there may be something with sampling going on.
• 9:43
  So maybe there are some specific types of plumes that we have a
• 9:47
  lot of in our data
• 9:48
  SPEAKER_00: sets,
• 9:48
  SPEAKER_00: but that occur a bit less in real life.
• 9:51
  SPEAKER_00: I think it's something slightly like that.
• 9:52
  SPEAKER_01: Okay.
• 9:55
  So it sounds like you could almost do some kind of
• 9:57
  investigation to try and
• 9:58
  SPEAKER_01: understand the different plumes,
• 9:59
  get an idea of how the different classes of plumes are maybe
• 10:02
  represented and start
• 10:05
  SPEAKER_01: to address any imbalances.
• 10:07
  SPEAKER_00: Yes, definitely.
• 10:08
  And now Svon has a lot of knowledge on this because they have
• 10:11
  been running
• 10:12
  SPEAKER_00: operational detections for a few years now,
• 10:14
  where every week there's two different methane analysis experts
• 10:18
  who look at the
• 10:19
  SPEAKER_00: predictions of their model.
• 10:23
  So we are working on this together to see how we can improve
• 10:27
  the data set for my model.
• 10:29
  Because I think the key distinction here is that the original
• 10:31
  data set was designed
• 10:33
  SPEAKER_00: for this two-step approach.
• 10:35
  SPEAKER_00: And my model,
• 10:37
  SPEAKER_00: because it's end-to-end,
• 10:39
  SPEAKER_00: might require something different,
• 10:41
  but we're not yet sure what we would need to change on the
• 10:44
  original dataset.
• 10:45
  SPEAKER_01: Fascinating.
• 10:47
  SPEAKER_01: I just want to cover the architecture search a little bit more.
• 10:50
  SPEAKER_01: So you had two gases.
• 10:52
  SPEAKER_01: Do the models end up being the same?
• 10:53
  SPEAKER_01: Are they different somehow?
• 10:56
  SPEAKER_00: They were a little bit different.
• 10:59
  SPEAKER_00: The same backbones tended to be selected,
• 11:02
  SPEAKER_00: because I found that we included a lot of different backbones.
• 11:05
  So as I said,
• 11:06
  very simple ones,
• 11:06
  just the vanilla CNN,
• 11:08
  two more complex models.
• 11:11
  SPEAKER_00: The more complex backbones,
• 11:12
  SPEAKER_00: like vision transformers,
• 11:13
  SPEAKER_00: were never selected in the best configurations.
• 11:16
  SPEAKER_00: And the best models for both gases tended to be the simple CNNs.
• 11:22
  SPEAKER_00: which was interesting finding because it's not,
• 11:26
  SPEAKER_00: it kind of contradicts the trend in general computer vision.
• 11:30
  I think it just has to do with that our data sets are still
• 11:33
  relatively small and
• 11:34
  SPEAKER_00: remote sensing data can also be a bit more complex than natural images,
• 11:39
  SPEAKER_00: smaller features,
• 11:40
  SPEAKER_00: for instance.
• 11:42
  SPEAKER_00: So that was part one.
• 11:44
  SPEAKER_00: And then,
• 11:46
  There was a small difference between architectures that came
• 11:50
  out for the two gases.
• 11:53
  It was mostly the depth of the input branches was different,
• 11:58
  the ideal input depth.
• 12:01
  SPEAKER_01: Right.
• 12:01
  SPEAKER_01: Okay.
• 12:01
  Possibly related to the complexity of identifying one gas
• 12:05
  versus other classes.
• 12:08
  SPEAKER_00: Yes.
• 12:09
  SPEAKER_00: Yes.
• 12:09
  Because carbon monoxide is a slightly easier problem because
• 12:12
  the background levels
• 12:14
  SPEAKER_00: of carbon monoxide in the atmosphere are lower.
• 12:17
  SPEAKER_00: So the plumes are a bit clearer.
• 12:19
  SPEAKER_01: Right.
• 12:20
  SPEAKER_01: Okay.
• 12:20
  SPEAKER_01: And this approach, is it open source?
• 12:24
  SPEAKER_01: Is it something that people can apply to their own problems?
• 12:29
  SPEAKER_00: Yes, definitely.
• 12:30
  SPEAKER_00: We have the code on GitHub.
• 12:33
  SPEAKER_00: And so we evaluated this specifically with Tripomi or Sentinel 5P data,
• 12:37
  SPEAKER_00: but in principle,
• 12:38
  SPEAKER_00: it could work for any data fusion approach,
• 12:42
  SPEAKER_00: also with different numbers of modalities.
• 12:45
  So the standard data fusion scenario in remote sensing is with
• 12:49
  two modalities,
• 12:50
  SPEAKER_00: for example,
• 12:51
  SPEAKER_00: optical and LIDAR or radar.
• 12:54
  SPEAKER_00: But for us, there's no such restriction.
• 12:56
  The model will automatically adapt to the number of modalities
• 13:00
  that you need.
• 13:02
  SPEAKER_01: Amazing.
• 13:03
  SPEAKER_01: Okay.
• 13:03
  SPEAKER_01: Well, thanks for giving us an overview.
• 13:05
  SPEAKER_01: You've got a paper out at the moment.
• 13:08
  SPEAKER_00: Yes.
• 13:09
  SPEAKER_01: And where is that?
• 13:11
  SPEAKER_00: It's an IEEE JSTARS.
• 13:15
  SPEAKER_00: Oh God, let me look up.
• 13:17
  I always forget what the full name of this journal is because
• 13:20
  it's very long.
• 13:24
  It's the Journal of Selected Topics and Applied Earth
• 13:27
  Observations and Remote Sensing.
• 13:29
  SPEAKER_01: Right, okay.
• 13:30
  SPEAKER_01: And the title of the paper is?
• 13:32
  SPEAKER_00: Auto Merge Net.
• 13:34
  SPEAKER_01: Auto Merge Net.
• 13:36
  SPEAKER_01: Okay.
• 13:36
  SPEAKER_01: Yeah, that's easy to remember, isn't it?
• 13:38
  SPEAKER_01: Auto Merge Net.
• 13:38
  SPEAKER_01: Brilliant.
• 13:39
  SPEAKER_01: Okay.
• 13:40
  SPEAKER_01: So paper's coming out.
• 13:41
  SPEAKER_01: Code is out there.
• 13:42
  SPEAKER_01: The solutions are being deployed and kind of refined in time.
• 13:47
  How does this tie in with everything else you want to achieve
• 13:49
  in your PhD?
• 13:52
  SPEAKER_00: Yeah, so it's now the last year of my PhD.
• 13:54
  I'm very happy that this paper is out because we've worked on
• 13:57
  this for a very long time.
• 13:58
  Now I'm thinking about how I can find a satisfying ending to
• 14:05
  this story before I wrap up.
• 14:07
  So we are working now on a new project,
• 14:11
  specifically looking at the data we discussed earlier.
• 14:14
  SPEAKER_00: And I also have some architectural ideas
• 14:17
  to further leverage the knowledge Esron mainly has about the
• 14:21
  problem in our approach.
• 14:26
  And we already looked a little bit at the data and we found
• 14:29
  only one small issue
• 14:31
  related to representation bias that we addressed in a workshop
• 14:35
  paper at the machine
• 14:37
  SPEAKER_00: learning workshop.
• 14:38
  SPEAKER_00: from ECML PKDD,
• 14:40
  where we found that our dataset accidentally had a correlation
• 14:46
  between the number
• 14:46
  SPEAKER_00: of missing pixels in an image and the label.
• 14:49
  So we had a lot of images without plumes with many missing
• 14:53
  pixels and very few
• 14:56
  SPEAKER_00: images of plumes with a lot of missing pixels.
• 14:59
  SPEAKER_00: So the models were learning how to count missing pixels a little bit.
• 15:07
  SPEAKER_00: But it was fairly easy to solve and we published also those techniques.
• 15:10
  SPEAKER_00: So that was step one of diving into the data.
• 15:14
  SPEAKER_00: And before the end of my PhD, I hope to
• 15:18
  SPEAKER_00: at least have a better understanding of what causes this knowledge,
• 15:22
  this generalization gap between the test scores that we found
• 15:25
  and operational keys.
• 15:26
  SPEAKER_00: Yeah.
• 15:28
  SPEAKER_00: So fingers crossed for that.
• 15:30
  SPEAKER_01: Brilliant.
• 15:30
  SPEAKER_01: Okay.
• 15:31
  SPEAKER_01: Well, exciting stuff, getting close to the end.
• 15:33
  SPEAKER_01: Lots of interesting work coming in the pipeline as well.
• 15:36
  SPEAKER_01: If people want to follow along your updates, which is the best platform?
• 15:40
  SPEAKER_00: Yeah, so probably LinkedIn or follow my Google Scholar.
• 15:43
  SPEAKER_00: I also post updates about my research sometimes on my website.
• 15:49
  SPEAKER_00: Specifically,
• 15:49
  SPEAKER_00: I've written a few blog posts on machine learning for Earth observation,
• 15:53
  SPEAKER_00: specifically for computer scientists,
• 15:55
  SPEAKER_00: because when I did my first project with remote sensing data,
• 16:01
  I had to learn very quickly that it's not just like any other
• 16:05
  data and there's
• 16:05
  SPEAKER_00: quite a lot of extra knowledge even about file types,
• 16:09
  SPEAKER_00: et cetera,
• 16:10
  SPEAKER_00: that you need.
• 16:11
  SPEAKER_00: So I wrote some blog posts about that that people can read.
• 16:15
  SPEAKER_01: Brilliant.
• 16:16
  SPEAKER_01: Okay.
• 16:16
  SPEAKER_01: I'll put all those links in the show notes.
• 16:19
  SPEAKER_01: Once again, thank you for joining me today.
• 16:21
  SPEAKER_01: Fascinating topic and all the best with the final year.
• 16:24
  SPEAKER_00: Thanks.
• 16:25
  SPEAKER_01: See ya.
• 16:27
  SPEAKER_00: See ya.