Latest / Satellite image deep learning / Meta-learning with Meteor

Transcript

Meta-learning with Meteor · Satellite image deep learning

• 0:05
  SPEAKER_00: Hello, I'm Robin Cole and welcome to the Satellite Image Deep Learning Podcast.
• 0:09
  SPEAKER_00: In this episode, I caught up with Mark Rusram to learn about meta-learning with Meteor.
• 0:14
  SPEAKER_00: Our conversation starts with a discussion about meta-learning and the training of
• 0:17
  SPEAKER_00: Meteor and how this approach differs from the typical approaches taken to train
• 0:21
  SPEAKER_00: foundational models.
• 0:23
  SPEAKER_00: We cover the advantages and challenges of this technique and discuss the
• 0:26
  SPEAKER_00: fine-tuning of Meteor with minimal examples,
• 0:29
  SPEAKER_00: as few as five,
• 0:30
  SPEAKER_00: for tasks like deforestation monitoring and change detection,
• 0:34
  SPEAKER_00: and consider what the future could hold for this approach.
• 0:37
  SPEAKER_00: Meteor showcases the significant potential of few-shot learning for processing
• 0:41
  SPEAKER_00: remote sensing imagery and proves that it's possible to tackle tasks even when very
• 0:45
  SPEAKER_00: few training examples are available.
• 0:47
  SPEAKER_00: I hope you enjoy this episode.
• 0:49
  SPEAKER_00: Hi, Mark.
• 0:49
  SPEAKER_00: Welcome to the podcast.
• 0:51
  SPEAKER_00: Hiya, thank you so much for having me.
• 0:52
  SPEAKER_00: It's my pleasure.
• 0:53
  SPEAKER_00: Let's begin with where you work and what you do.
• 0:56
  SPEAKER_01: Yeah, thanks.
• 0:57
  SPEAKER_01: I'm Marc Roswell.
• 0:58
  SPEAKER_01: I'm assistant professor for machine learning and remote sensing at Wageningen University.
• 1:03
  SPEAKER_01: And I have some background in geodesy, so from the earth science.
• 1:07
  SPEAKER_01: And then I moved from computer vision into machine learning.
• 1:10
  SPEAKER_01: And I'm now really in the intersection between both fields.
• 1:13
  SPEAKER_01: And especially in Wageningen, we do a lot of applications.
• 1:16
  SPEAKER_01: So we use modern machine learning methods effectively to bring them to do something good with them.
• 1:24
  SPEAKER_01: That's the main topic here.
• 1:26
  SPEAKER_00: Absolutely.
• 1:27
  SPEAKER_00: You recently worked on a piece of work called Meteor.
• 1:29
  SPEAKER_00: I'm looking forward to hearing what that's about.
• 1:32
  SPEAKER_01: Yeah, that fits very well, the recent work on big machine learning models.
• 1:37
  SPEAKER_01: So Meteor is about meta learning, deep learning models for various tasks.
• 1:43
  SPEAKER_01: So I think everybody is familiar with foundation models.
• 1:46
  SPEAKER_01: So one model is supposed to solve a lot of different tasks.
• 1:50
  SPEAKER_01: meta learning is a way to formalize that in in a mathematical way so we are really
• 1:56
  SPEAKER_01: optimizing a model on the objective to be good at various different problems at the
• 2:01
  SPEAKER_01: same time already in the training stage where like the classic gpt type of
• 2:07
  SPEAKER_01: pre-training models they are trained on a
• 2:09
  SPEAKER_01: different pre-training objective and then we hope they are good on a set of
• 2:13
  SPEAKER_01: downstream tasks and they are very effective in that.
• 2:15
  SPEAKER_01: But the idea is when we actually train them explicitly to multitask,
• 2:20
  SPEAKER_01: then they also should be better on those individual tasks.
• 2:23
  SPEAKER_01: And model-agnostic meta-learning,
• 2:25
  SPEAKER_01: that's the algorithm that we explored in that paper,
• 2:29
  SPEAKER_01: is a way to formalize that with its own advantages and disadvantages,
• 2:32
  SPEAKER_01: of course.
• 2:33
  SPEAKER_00: Fascinating.
• 2:34
  SPEAKER_00: So foundational models are trained with some kind of pretext task,
• 2:37
  SPEAKER_00: like fill in the missing pixels or something like that.
• 2:40
  SPEAKER_00: In your case, meta learning means you're training for specific tasks, I think classification, is it?
• 2:45
  SPEAKER_00: Yes,
• 2:46
  SPEAKER_01: primarily land cover classification,
• 2:48
  SPEAKER_01: but it's more like a contextual framework,
• 2:50
  SPEAKER_01: how we see data around the globe.
• 2:52
  SPEAKER_01: Because when we look at land cover problems at different locations on the earth,
• 2:57
  SPEAKER_01: we can see each of that as a different task,
• 2:59
  SPEAKER_01: as a different problem.
• 3:00
  SPEAKER_01: And we see then the world as a data set of data sets.
• 3:04
  SPEAKER_01: And there's some really fundamental advantages on this.
• 3:07
  SPEAKER_01: So in classic land cover classification, we have situations where we have like...
• 3:13
  SPEAKER_01: ice and desert, two classes next to each other that the model needs to classify.
• 3:17
  SPEAKER_01: But that doesn't not really make much sense because these classes don't really appear next to each other.
• 3:22
  SPEAKER_01: So it's a much more realistic
• 3:25
  SPEAKER_01: framework to think about a classification a length of a classification problem in
• 3:30
  SPEAKER_01: the arctic there's a problem including ice and tundra and a problem in in like the
• 3:34
  SPEAKER_01: sahara including like shrublands and deserts and classic supervised learning does
• 3:40
  SPEAKER_01: not really capture this so meta learning is a way to formulate it as a data set of
• 3:46
  SPEAKER_00: data set that's nice so
• 3:49
  SPEAKER_00: It's not like with a GPT kind of model where you just train once on the massive data set.
• 3:54
  SPEAKER_00: In this case, you're training once on smaller data sets or many times on many smaller data sets?
• 4:00
  SPEAKER_01: We split this one big data set of land cover problems around the Earth into many
• 4:05
  SPEAKER_01: thousands of very small data sets.
• 4:07
  SPEAKER_01: So we are going in this few-shot domain where we have global land cover data,
• 4:12
  SPEAKER_01: and then we only look at data in the Sahara.
• 4:15
  SPEAKER_01: And we take only 30,
• 4:16
  SPEAKER_01: 40 images from the Sahara and train the model to distinguish different length of
• 4:21
  SPEAKER_01: the classes there.
• 4:22
  SPEAKER_01: and then optimize the initialization of this model,
• 4:25
  SPEAKER_01: so where it kind of starts,
• 4:27
  SPEAKER_01: simultaneously to be good in the Sahara case,
• 4:30
  SPEAKER_01: but also be good in the Tundra case from the same starting point.
• 4:34
  SPEAKER_01: So that's the core idea here.
• 4:35
  SPEAKER_01: Right.
• 4:36
  SPEAKER_01: And is that supervised or unsupervised at the moment?
• 4:38
  SPEAKER_01: It's supervised.
• 4:39
  SPEAKER_01: We still need labels,
• 4:41
  SPEAKER_01: but it's the learning algorithm that incorporates this hierarchical two levels of
• 4:47
  SPEAKER_00: learning here.
• 4:48
  SPEAKER_00: Okay.
• 4:48
  SPEAKER_00: So, and I gather there's some sort of novelty to the architecture as well.
• 4:54
  SPEAKER_01: Yeah.
• 4:54
  SPEAKER_01: So the model is a bit smaller through to this algorithm, but it basically learns to compare.
• 5:01
  SPEAKER_01: So you learn to compare few training images with the test images.
• 5:07
  SPEAKER_01: And that also has a potential to learn very fine grained patterns from those
• 5:12
  SPEAKER_01: training images in different locations,
• 5:14
  SPEAKER_01: rather than needing to learn
• 5:16
  SPEAKER_01: all possible patterns of land cover across the entire globe in one large model.
• 5:21
  SPEAKER_01: So that's an advantage here that the models are smaller.
• 5:24
  SPEAKER_01: And the disadvantage, of course, is that the dependence on the training data is much larger.
• 5:29
  SPEAKER_01: So you need really high quality training data, optimally with hand-picked samples.
• 5:32
  SPEAKER_01: So I think like human interloop mining would be a good application there.
• 5:36
  SPEAKER_00: Okay.
• 5:36
  SPEAKER_00: So let's say I'm just starting totally from scratch and I want to do deforestation.
• 5:40
  SPEAKER_00: So how much data do I collect?
• 5:42
  SPEAKER_00: How much training is involved?
• 5:44
  SPEAKER_00: How do I evaluate when I'm ready to fine tune?
• 5:47
  SPEAKER_01: Well, the good part here is the model is already pre-trained on those global land cover problems.
• 5:51
  SPEAKER_01: So for a new problem in deforestation, you basically just need to annotate a few examples.
• 5:56
  SPEAKER_01: So we have experiments on five shot learning, for instance, so five examples per class.
• 6:01
  SPEAKER_01: In that case, it would be deforested and non-deforested.
• 6:04
  SPEAKER_01: And the model would learn to distinguish these two classes with these few examples
• 6:09
  SPEAKER_01: and then be able to predict.
• 6:11
  SPEAKER_01: So that's when it's explicitly trained on this future learning objective.
• 6:16
  SPEAKER_01: That's the point.
• 6:17
  SPEAKER_00: And it was pre-trained on a Sentinel dataset, was it?
• 6:19
  SPEAKER_01: Yeah.
• 6:20
  SPEAKER_01: Sentinel-1 and 2MS dataset, basically.
• 6:24
  SPEAKER_01: The main advantage is that it's globally distributed with land cover data.
• 6:31
  SPEAKER_00: And does this mean I can pick and choose if I use Sentinel-1 or Sentinel-2,
• 6:36
  SPEAKER_00: or I need to use both of them?
• 6:38
  SPEAKER_01: Yeah,
• 6:39
  SPEAKER_01: we have some good experiments here as well,
• 6:40
  SPEAKER_01: because there's this second fine-tuning phase where the modeler is able to adapt to
• 6:45
  SPEAKER_01: the individual downstream task.
• 6:47
  SPEAKER_01: And we can actually change the number of channels and only select the Sentinel-2 channels,
• 6:52
  SPEAKER_01: only the Sentinel-1 channels.
• 6:54
  SPEAKER_01: And usually that wouldn't work particularly well because there's a lot of
• 6:57
  SPEAKER_01: dependencies from the first layers to the next layers in the network.
• 7:01
  SPEAKER_01: But in this case, because the entire smaller network is again trained with gradient descent and updated,
• 7:07
  SPEAKER_01: it can adapt fairly well on even the first few layers.
• 7:11
  SPEAKER_01: So that's a really nice thing here that we are kind of flexible.
• 7:13
  SPEAKER_01: We can also go,
• 7:14
  SPEAKER_01: we train it on those multispectral images,
• 7:16
  SPEAKER_01: but we can go to RGB three-channel images by switching out the first layers and it
• 7:22
  SPEAKER_00: works relatively well.
• 7:23
  SPEAKER_00: Fascinating.
• 7:23
  SPEAKER_00: And how was it to train, to do the pre-training in general?
• 7:30
  SPEAKER_01: How is it to train?
• 7:31
  SPEAKER_00: There's a lot of specialist hardware experience in the training process.
• 7:38
  SPEAKER_01: Like in any big deep learning model.
• 7:40
  SPEAKER_01: So it took about like two days, like 48 hours on a GPU, but on a single GPU.
• 7:47
  SPEAKER_01: So we didn't scale up things too much.
• 7:50
  SPEAKER_01: We took this data set, which is about 200 gigabytes in size and land cover data.
• 7:55
  SPEAKER_01: We trained it for two days.
• 7:57
  SPEAKER_01: The core thing is we need to build this dataset of dataset structure.
• 8:01
  SPEAKER_01: But that's basically how we got our pre-trained models that are then available on
• 8:06
  SPEAKER_01: the GitHub repository,
• 8:07
  SPEAKER_01: which is somewhat reasonable.
• 8:10
  SPEAKER_01: But I think if you would scale it up a bit more with bigger compute,
• 8:15
  SPEAKER_01: you would probably get a bit more benefit out of it.
• 8:17
  SPEAKER_01: But that was not necessarily the core idea of this paper.
• 8:20
  SPEAKER_01: The core idea of the paper was to
• 8:22
  SPEAKER_01: show that this training algorithm works from land cover data.
• 8:26
  SPEAKER_01: And then even though it's only trained on land cover problems globally,
• 8:30
  SPEAKER_01: it can still be fine tuned on very different other problems simultaneously.
• 8:36
  SPEAKER_01: So that was kind of the main takeaway.
• 8:38
  SPEAKER_00: Fantastic.
• 8:40
  SPEAKER_00: And what's some of the tasks that you fine tuned on?
• 8:43
  SPEAKER_01: So, um, that's the, that's the cool idea that this can be actually quite different.
• 8:48
  SPEAKER_01: So training tasks,
• 8:49
  SPEAKER_01: the land cover,
• 8:50
  SPEAKER_01: and then at the downstream tasks,
• 8:51
  SPEAKER_01: we have change detection on a planet scope images.
• 8:55
  SPEAKER_01: So fewer channels, higher resolution.
• 8:57
  SPEAKER_01: And we provide five images per class of deforested images and five images per class
• 9:03
  SPEAKER_01: for non-deforested images.
• 9:05
  SPEAKER_01: And the model learns fairly well to classify then these two prototypes, class prototypes.
• 9:12
  SPEAKER_01: Then we have urban density.
• 9:16
  SPEAKER_01: classification.
• 9:17
  SPEAKER_01: So dense urban areas with sparse urban areas and so on,
• 9:21
  SPEAKER_01: which is particularly difficult if you think about it,
• 9:24
  SPEAKER_01: that it's only trained on land cover,
• 9:26
  SPEAKER_01: which just has an urban class.
• 9:27
  SPEAKER_01: And now it's really asked to distinguish different densities of urban areas.
• 9:32
  SPEAKER_01: And we can even like play with the
• 9:34
  SPEAKER_01: definition of classes and really go into a more like temporary change detection idea.
• 9:40
  SPEAKER_01: Because at the end,
• 9:41
  SPEAKER_01: what we do in this fine-tuning task,
• 9:43
  SPEAKER_01: we define a prototype of classes with those few training examples.
• 9:47
  SPEAKER_01: And what we did is we looked at the Beirut example where this one warehouse
• 9:53
  SPEAKER_01: exploded a few years ago,
• 9:55
  SPEAKER_01: which left quite a big crater.
• 9:57
  SPEAKER_01: And we took the first images in a time series and learned a class pre-event.
• 10:02
  SPEAKER_01: and then took the last images in a time series in the post-event class.
• 10:07
  SPEAKER_01: So it's, again, a classification task in that sense.
• 10:09
  SPEAKER_01: And then when we look at the whole time series and the probability for each of those,
• 10:14
  SPEAKER_01: for both of these classes,
• 10:16
  SPEAKER_01: we see that there is a sharp drop in the moment where the explosion happens.
• 10:20
  SPEAKER_01: So that was more like a conceptual idea that we can go away from this classic class
• 10:25
  SPEAKER_01: definitions of class A and class B and
• 10:28
  SPEAKER_01: by defining few examples per class to really have something more dynamic of what a
• 10:34
  SPEAKER_01: class is supposed to look like,
• 10:35
  SPEAKER_01: like pre and post event.
• 10:36
  SPEAKER_00: Right.
• 10:37
  SPEAKER_00: So in that sort of context,
• 10:38
  SPEAKER_00: somebody that wanted to know when it happened,
• 10:40
  SPEAKER_00: they would just have to find a before and after image and quickly train a model.
• 10:44
  SPEAKER_00: I think you mentioned five examples, possibly even fewer examples for that.
• 10:48
  SPEAKER_00: And then you would have an algorithm that could tell you when this kind of significant change occurred.
• 10:54
  SPEAKER_01: It touches a bit this question of what is a task in the first place.
• 10:58
  SPEAKER_01: In meta-learning, we talk a lot about tasks and we express tasks as a small data set effectively.
• 11:04
  SPEAKER_01: And what we do is we transfer our understanding of a problem to the machine
• 11:10
  SPEAKER_01: learning model and the way we transfer this
• 11:14
  SPEAKER_01: Understanding is through trained images.
• 11:16
  SPEAKER_01: Classically in supervised learning,
• 11:18
  SPEAKER_01: we need a lot of trained images to really train a deep learning model with many examples.
• 11:24
  SPEAKER_01: But because a meta-learning model is trained explicitly on future tasks,
• 11:29
  SPEAKER_01: we can just have a small set of images.
• 11:32
  SPEAKER_01: and then be fairly efficient on how we communicate these tasks that we have and
• 11:39
  SPEAKER_01: that we would like to solve.
• 11:40
  SPEAKER_01: That's a bit different to like the classic foundation contrastive model methods
• 11:45
  SPEAKER_01: where we define this pretext tasks that we can easily sample at a large scale.
• 11:51
  SPEAKER_01: But then we don't really know how well those pretext tasks align with our downstream target tasks.
• 11:57
  SPEAKER_01: In that sense, it's fairly complementary, but it serves a different purpose.
• 12:02
  SPEAKER_01: It's explicitly trained on this few shot setting with fewer training images.
• 12:06
  SPEAKER_00: I guess the natural question,
• 12:07
  SPEAKER_00: is there a future where you can combine and get the benefits of the larger data set
• 12:11
  SPEAKER_00: with meta learning approach?
• 12:13
  SPEAKER_01: Yeah, it's a good question.
• 12:15
  SPEAKER_01: There's two challenges.
• 12:16
  SPEAKER_01: One is the computational efficiency of this meta-learning algorithm.
• 12:20
  SPEAKER_01: So scaling it up would probably help a bit because we still didn't perform too big
• 12:27
  SPEAKER_01: of a training around this.
• 12:29
  SPEAKER_01: But there's still limits on how big you can do those models.
• 12:32
  SPEAKER_01: So that would probably require...
• 12:35
  SPEAKER_01: changing this algorithm a bit to be more computationally efficient and be easier to scale up.
• 12:42
  SPEAKER_01: But I think a key takeaway is more to...
• 12:46
  SPEAKER_01: be aware of this idea of tasks in the Earth observation domain,
• 12:50
  SPEAKER_01: because we all think about data sets because that's what we have like in a grabbable,
• 12:55
  SPEAKER_01: touchable way.
• 12:56
  SPEAKER_01: But what is behind the data set,
• 12:58
  SPEAKER_01: those are just samples from a data distribution and this data distribution is our task.
• 13:04
  SPEAKER_01: And if we start to think about the Earth as an environment of different tasks where
• 13:11
  SPEAKER_01: we now need to build models
• 13:13
  SPEAKER_01: to solve a variety of tasks simultaneously,
• 13:15
  SPEAKER_01: then we can learn from this meta-learning framework to actually build new models
• 13:20
  SPEAKER_01: specifically for those unique properties that we have in geographic data.
• 13:23
  SPEAKER_00: Absolutely.
• 13:24
  SPEAKER_00: Really interesting stuff.
• 13:26
  SPEAKER_00: So the code itself and the model is on GitHub.
• 13:31
  SPEAKER_00: What do you want people to do with it?
• 13:33
  SPEAKER_01: I mean, there's basically two ways to use it.
• 13:37
  SPEAKER_01: So one thing is we have the model of eight, so everybody can download it and
• 13:42
  SPEAKER_01: train it on their own downstream tasks.
• 13:44
  SPEAKER_01: So that's more like the user perspective that would be nice to know what people do with it.
• 13:50
  SPEAKER_01: The other side is to use the training algorithm and improve upon it.
• 13:54
  SPEAKER_01: So include more
• 13:56
  SPEAKER_01: training tasks, for example.
• 13:58
  SPEAKER_01: So we have like all the training scripts in there as well, obviously.
• 14:02
  SPEAKER_01: So one can basically take more training data from different problems,
• 14:07
  SPEAKER_01: not only land cover classification problems,
• 14:10
  SPEAKER_01: and just include,
• 14:12
  SPEAKER_01: increase the variety of training tasks and then use our evaluation scripts to
• 14:17
  SPEAKER_01: measure how good it improves actually on those
• 14:20
  SPEAKER_01: downstream task that we defined in the paper.
• 14:22
  SPEAKER_01: Because that's how we limited ourselves a bit.
• 14:25
  SPEAKER_01: Because at some point you need to make a cut when you write a paper.
• 14:28
  SPEAKER_01: But how far do you go?
• 14:30
  SPEAKER_01: And we said,
• 14:30
  SPEAKER_01: okay,
• 14:30
  SPEAKER_01: let's stick with the Sen1-2MS,
• 14:32
  SPEAKER_01: which is like a fairly established data set only for like LandCover and Sentinel-2.
• 14:39
  SPEAKER_01: And show what we can do just from this set of
• 14:43
  SPEAKER_01: training tasks, how many downstream tasks can we address?
• 14:47
  SPEAKER_01: So I feel if somebody starts to train the same algorithm on like much more diverse
• 14:52
  SPEAKER_01: training tasks,
• 14:53
  SPEAKER_01: there should be quite a bit of a boost in performance.
• 14:56
  SPEAKER_00: Absolutely.
• 14:56
  SPEAKER_00: Well, thank you very much for giving us the background on the project.
• 14:59
  SPEAKER_00: It's been really fascinating to learn about meta-learning.
• 15:01
  SPEAKER_00: If people want to follow your updates, which is the best platform?
• 15:05
  SPEAKER_01: I mean, I'm on LinkedIn and Twitter.
• 15:08
  SPEAKER_01: Mainly, but I think everybody can just find my email readily, easily, and just write me an email.
• 15:14
  SPEAKER_01: I answer usually the email, so as much as I can.
• 15:17
  SPEAKER_00: All right, fantastic.
• 15:18
  SPEAKER_00: I'll put that in the show notes.
• 15:19
  SPEAKER_00: Well, been brilliant, Mark.
• 15:20
  SPEAKER_00: Once again, thank you, and I'll talk to you in the future.
• 15:23
  SPEAKER_01: Yeah, thank you so much for having me.
• 15:24
  SPEAKER_01: It's a pleasure.
• 15:24
  SPEAKER_01: Likewise.
• 15:25
  SPEAKER_01: Bye.