Latest / Satellite image deep learning / A Single GPU Is All You Need for Self-Supervised Pretraining

Transcript

A Single GPU Is All You Need for Self-Supervised Pretraining · Satellite image deep learning

• 0:05
  SPEAKER_01: Hi Lakshay, welcome to the podcast.
• 0:08
  SPEAKER_00: Hey Robin, nice to be here, thanks for having me.
• 0:11
  SPEAKER_01: Fantastic, let's begin with a brief intro to yourself.
• 0:16
  SPEAKER_00: Sure.
• 0:16
  SPEAKER_00: So my name is Lakshir Sharma.
• 0:18
  SPEAKER_00: I'm a computer vision slash ML scientist and engineer.
• 0:24
  SPEAKER_00: Presently,
• 0:24
  SPEAKER_00: I'm with Instacart based out of New York City,
• 0:27
  SPEAKER_00: where I work on problems related to estimated time of arrival prediction.
• 0:32
  SPEAKER_00: So using AI, ML for that sort of stuff.
• 0:36
  SPEAKER_00: Before that,
• 0:36
  SPEAKER_00: I spent about five years with Microsoft's geospatial division,
• 0:41
  SPEAKER_00: where I was using computer vision for
• 0:44
  SPEAKER_00: things like finding roads and buildings and aerial imagery,
• 0:48
  SPEAKER_00: finding traffic signs,
• 0:50
  SPEAKER_00: traffic lights,
• 0:52
  SPEAKER_00: crosswalks,
• 0:53
  SPEAKER_00: that sort of stuff and street-side imagery.
• 0:55
  Also doing sort of more traffic and mobility-oriented tasks
• 1:00
  like figuring out when
• 1:01
  SPEAKER_00: traffic builds up in cities,
• 1:05
  SPEAKER_00: things like that,
• 1:06
  SPEAKER_00: yeah.
• 1:07
  SPEAKER_00: Fascinating.
• 1:09
  SPEAKER_01: So we connected on Torchgeo.
• 1:11
  SPEAKER_01: I'm very interested to learn a bit more about your research.
• 1:14
  SPEAKER_01: Do you mind giving us some intro to that?
• 1:18
  SPEAKER_00: Absolutely.
• 1:19
  So my areas of interest are broadly,
• 1:23
  I think,
• 1:23
  things that I am very passionate about.
• 1:24
  SPEAKER_00: Those include self-supervision and computer vision,
• 1:28
  and more specifically self-supervision as it's applied to
• 1:32
  geospatial pre-training.
• 1:34
  I've also worked a little bit at the intersection of vision and
• 1:39
  language,
• 1:39
  so pre-training,
• 1:42
  SPEAKER_00: Very,
• 1:42
  very classical contrastive style pre-training for vision
• 1:46
  language alignment,
• 1:47
  SPEAKER_00: that sort of stuff.
• 1:49
  And self-supervised learning and pre-training has become very
• 1:52
  significant recently
• 1:53
  SPEAKER_01: because of the advent of foundational models.
• 1:55
  SPEAKER_01: But for those that are not familiar with self-supervised learning,
• 1:58
  SPEAKER_01: can you give a small overview?
• 2:01
  SPEAKER_00: Yes, absolutely.
• 2:03
  So generally in computer vision and machine learning,
• 2:06
  there are tasks that we care about.
• 2:10
  SPEAKER_00: So a very common one is semantic segmentation, right?
• 2:13
  Where there's a certain predefined set of classes we're trying
• 2:16
  to classify every pixel into.
• 2:19
  However,
• 2:19
  very often these tasks,
• 2:22
  because they fall in this paradigm of supervised learning,
• 2:25
  They require labels which can be a very expensive and
• 2:29
  time-consuming undertaking.
• 2:31
  So self-supervised learning is this idea where we train these
• 2:36
  models on some kind
• 2:38
  SPEAKER_00: of auxiliary task first
• 2:41
  And the idea there is that the labels or the supervision for
• 2:44
  the auxiliary task is
• 2:46
  it's free because it's some inherent property of the data
• 2:50
  distribution itself or of
• 2:52
  SPEAKER_00: the images that we're dealing with.
• 2:54
  SPEAKER_00: And so the idea there is that if you train
• 2:58
  if you train on an auxiliary task which is sufficiently
• 3:00
  challenging and you do it
• 3:02
  with enough data for enough number of epochs for enough time
• 3:07
  you obtain some useful
• 3:10
  representation of the image that this model can produce and
• 3:13
  then this property
• 3:15
  these parameters of the model can then transfer better to the
• 3:18
  downstream task as
• 3:20
  SPEAKER_00: opposed to directly trying to train on the downstream task.
• 3:23
  SPEAKER_01: Mm-hmm.
• 3:24
  And how is this similar or different to the original ImageNet
• 3:28
  pre-training that
• 3:28
  SPEAKER_00: people might be familiar with?
• 3:32
  So it's similar in that both of these ideas are classified as
• 3:35
  what we would call
• 3:36
  SPEAKER_00: transfer learning,
• 3:38
  where we train a model on something first and then we train it
• 3:43
  on the tasks that we
• 3:44
  SPEAKER_00: actually care about.
• 3:45
  SPEAKER_00: The difference here would be that ImageNet pre-training
• 3:48
  SPEAKER_00: does rely on human annotated labels.
• 3:51
  SPEAKER_00: So the ImageNet dataset has about 1,000 classes exactly, actually.
• 3:58
  And so this idea of transfer learning sort of began around then
• 4:02
  when folks realized
• 4:03
  SPEAKER_00: that,
• 4:03
  SPEAKER_00: okay,
• 4:04
  we can pre-train a model on ImageNet and after that do the
• 4:08
  tasks that we actually
• 4:09
  SPEAKER_00: care about and then those properties,
• 4:11
  those desirable properties in the model transfer better to the
• 4:14
  downstream task.
• 4:15
  SPEAKER_00: The difference with self-supervision is that
• 4:19
  SPEAKER_00: It does borrow this idea of transfer learning,
• 4:23
  SPEAKER_00: but the labels in the tasks that we tackle first,
• 4:28
  SPEAKER_00: they don't require human annotation.
• 4:30
  SPEAKER_00: So in some ways,
• 4:33
  SPEAKER_00: if you have enough images,
• 4:35
  SPEAKER_00: you essentially have infinite labels with self-supervision.
• 4:39
  SPEAKER_01: Right.
• 4:41
  SPEAKER_01: So more recently, as you say, we've moved to our own
• 4:45
  SPEAKER_01: self-supervised approach,
• 4:46
  SPEAKER_01: unlabeled data that allows you to use basically every image that you have
• 4:50
  SPEAKER_01: available.
• 4:51
  What kind of approaches have people taken and how has that
• 4:53
  evolved over the last
• 4:54
  SPEAKER_01: couple of years?
• 4:56
  So within vision specifically,
• 4:59
  broadly I would say there are three umbrellas of approaches.
• 5:05
  SPEAKER_00: The very first one that was probably very popular
• 5:09
  SPEAKER_00: maybe about 10, 8 years before now.
• 5:12
  SPEAKER_00: I'd call it predictive-style self-supervised pre-training.
• 5:18
  SPEAKER_00: Okay, how about this?
• 5:19
  I'll list all of them out first,
• 5:20
  and then I'll do a deep dive into each one.
• 5:23
  SPEAKER_00: So there's predictive self-supervision,
• 5:26
  SPEAKER_00: there's generative self-supervision,
• 5:28
  SPEAKER_00: and then finally there's contrastive self-supervision.
• 5:32
  SPEAKER_00: Predictive-style self-supervision is where
• 5:37
  where we try and predict some sort of label that is directly
• 5:41
  derived from the image.
• 5:43
  SPEAKER_00: It does not require human annotation.
• 5:45
  SPEAKER_00: It's some property of the image itself.
• 5:48
  SPEAKER_00: So that's predictive.
• 5:50
  SPEAKER_00: The second one is generative.
• 5:52
  SPEAKER_00: So that's as the name suggests,
• 5:56
  SPEAKER_00: You're basically trying to reconstruct the image.
• 5:59
  SPEAKER_00: So the most natural way of doing this is autoencoder style.
• 6:05
  SPEAKER_00: Pre-training where there's some original image that is fed through a few
• 6:10
  SPEAKER_00: neural network layers through a bottleneck,
• 6:13
  SPEAKER_00: typically,
• 6:13
  because you don't want to make the task trivially simple such
• 6:16
  that an identity
• 6:17
  SPEAKER_00: matrix can just recreate it.
• 6:19
  SPEAKER_00: And then from that bottleneck,
• 6:21
  SPEAKER_00: the image is then reconstructed through expanding neural network layers.
• 6:27
  SPEAKER_00: So that would be an example of generative.
• 6:29
  SPEAKER_00: Of course, there are other more popular ones now, like GANs, where
• 6:34
  SPEAKER_00: quite a popular one for a while.
• 6:36
  SPEAKER_00: And now there's more like diffusion-style generator pre-training.
• 6:41
  SPEAKER_00: And finally,
• 6:41
  SPEAKER_00: there's contrastive pre-training,
• 6:43
  SPEAKER_00: which is by and large,
• 6:45
  SPEAKER_00: I'd say,
• 6:46
  SPEAKER_00: the sort of dominant paradigm at this time.
• 6:49
  SPEAKER_00: So this is where the network needs to sort of
• 6:56
  cluster or contrast semantically similar inputs together and
• 7:01
  then also push apart
• 7:03
  SPEAKER_00: dissimilar inputs.
• 7:06
  SPEAKER_00: So an example of this would be maybe the very first
• 7:11
  The initial set of contrastive pre-training approaches would do
• 7:15
  this sort of thing
• 7:15
  SPEAKER_00: where you have the original image,
• 7:17
  SPEAKER_00: you apply some sort of transformation,
• 7:19
  SPEAKER_00: some sort of distortion,
• 7:22
  SPEAKER_00: some alteration to that original image,
• 7:24
  SPEAKER_00: pass both of these through maybe a Siamese-style neural network,
• 7:28
  SPEAKER_00: and then try to push these two views of the same image together.
• 7:32
  SPEAKER_00: And anything else that is not from the same image is pushed apart.
• 7:38
  And this idea has then broadly been expanded to other things
• 7:42
  where you can take the
• 7:43
  SPEAKER_00: same,
• 7:45
  an aerial image of the same area but from two different points
• 7:48
  in time and then you
• 7:49
  can sort of push them closer together using some sort of
• 7:51
  contrastive loss and then
• 7:53
  SPEAKER_00: push apart images from other areas.
• 7:55
  SPEAKER_00: Right, fantastic.
• 7:56
  SPEAKER_01: So you just touched on some of the sort of remote sensing
• 8:00
  twists on the approach that'd be a nice point to dive into
• 8:04
  maybe some of the other
• 8:05
  approaches that you've seen or specifically what your research
• 8:09
  has tackled yep
• 8:11
  absolutely so as I said earlier the contrast of style
• 8:16
  pre-training that's perhaps
• 8:18
  SPEAKER_00: the dominant
• 8:20
  SPEAKER_00: paradigm at this time.
• 8:21
  SPEAKER_00: And a lot of the state-of-the-art models that we see use this idea.
• 8:27
  SPEAKER_00: So the Dino series of models from Meta,
• 8:33
  which that same idea that's been borrowed by some geospatial
• 8:38
  machine learning
• 8:39
  SPEAKER_00: teams,
• 8:40
  SPEAKER_00: that's a combination of using a contrastive style
• 8:45
  SPEAKER_00: learning loss,
• 8:45
  but also they combine this idea of self-distillation where
• 8:50
  there's a teacher model
• 8:51
  and then there's often a smaller student model and you're
• 8:54
  trying to sort of
• 8:56
  SPEAKER_00: contrast views from the teacher model and the student model together.
• 8:59
  SPEAKER_00: The thing though with contrastive style pre-training is that
• 9:04
  SPEAKER_00: While it's a very elegant approach,
• 9:06
  SPEAKER_00: a very elegant mathematical formulation,
• 9:09
  SPEAKER_00: it does require a vast amount of data.
• 9:12
  SPEAKER_00: So you probably need hundreds of thousands,
• 9:15
  SPEAKER_00: maybe even millions of images to make it work pretty well.
• 9:21
  SPEAKER_00: So that's with contrastive style pre-training.
• 9:25
  SPEAKER_00: With more traditional predictive style pre-training,
• 9:29
  some examples of that that were very popular a few years ago
• 9:32
  actually were
• 9:34
  where you could drop the color channels from an image and then
• 9:42
  you have a grayscale
• 9:43
  SPEAKER_00: image and then you're trying to predict the color.
• 9:45
  SPEAKER_00: So you're trying to predict some inherent property of the image.
• 9:47
  SPEAKER_00: So if you see there's grass and trees,
• 9:50
  SPEAKER_00: but they're grayscale,
• 9:52
  you could naturally infer that they're green and then you hope
• 9:54
  that the model can
• 9:55
  SPEAKER_00: predict this from the image.
• 9:57
  SPEAKER_00: Some other popular approaches were
• 10:00
  SPEAKER_00: randomly rotating an image.
• 10:01
  So maybe there's a 90 degree rotation to the image and the
• 10:05
  model needs to classify
• 10:07
  SPEAKER_00: what the angle of rotation is.
• 10:09
  SPEAKER_00: This could be a classification task.
• 10:11
  You can just have four predefined rotation angles or you can
• 10:15
  make it a regression
• 10:16
  SPEAKER_00: task where it's trying to predict a specific number.
• 10:19
  The reason a lot of these approaches have actually been applied
• 10:24
  to geospatial data,
• 10:26
  but
• 10:27
  SPEAKER_00: Very often these approaches,
• 10:29
  they sometimes just don't make sense for geospatial data
• 10:33
  because of the fact that
• 10:34
  SPEAKER_00: geospatial images have rotational invariants.
• 10:37
  So you could rotate a picture of a dog and you can tell that,
• 10:40
  okay,
• 10:40
  this dog is upside down.
• 10:42
  But if you have an aerial image,
• 10:43
  you could rotate it and it still makes sense.
• 10:46
  So sometimes these approaches just don't transfer well to the
• 10:52
  geospatial domain.
• 10:54
  SPEAKER_00: Fantastic.
• 10:55
  You touched on,
• 10:56
  I guess,
• 10:57
  some of the challenges in terms of the data volumes,
• 10:59
  etc.
• 11:01
  SPEAKER_01: And I guess that would create a barrier to use, right?
• 11:05
  SPEAKER_01: So efficiency is a big part of the equation in recent years?
• 11:10
  SPEAKER_00: Yeah, yeah, absolutely.
• 11:12
  So what we're seeing with the most successful models is the
• 11:16
  fact that there's
• 11:18
  a vast amount of training data and therefore a vast amount of
• 11:21
  compute that is
• 11:22
  required in this pre-training exercise to then get sort of a
• 11:26
  general purpose model
• 11:27
  SPEAKER_00: that can transfer well to a multitude of geospatial remote sensing tasks.
• 11:34
  There has been less of a focus on sort of efficient
• 11:38
  pre-training and more targeted
• 11:40
  pre-training and that's where my co-author Alex and I focused
• 11:46
  our energies on was
• 11:47
  SPEAKER_00: trying to figure out
• 11:49
  Can we pre-train efficiently and do it in a way that works well
• 11:53
  for the very
• 11:54
  SPEAKER_00: specific downstream task that we care about?
• 11:57
  SPEAKER_00: Right.
• 11:59
  SPEAKER_00: So the downstream task being classification?
• 12:03
  SPEAKER_00: In our case, specifically, it was semantic segmentation, actually.
• 12:07
  SPEAKER_00: So we,
• 12:08
  SPEAKER_00: because pixel-level classification,
• 12:10
  SPEAKER_00: you know,
• 12:11
  SPEAKER_00: to this day remains such a central,
• 12:14
  maybe one of the most common things that you typically do in
• 12:18
  remote sensing.
• 12:19
  So some examples would be land cover classification where
• 12:22
  you're trying to
• 12:24
  figure out if some land area belongs to some predefined set of
• 12:28
  classes like water
• 12:29
  SPEAKER_00: bodies or forest cover or things like that,
• 12:32
  SPEAKER_00: buildings,
• 12:33
  SPEAKER_00: roads.
• 12:34
  SPEAKER_00: Chain detection,
• 12:35
  SPEAKER_00: of course,
• 12:36
  SPEAKER_00: is also a very popular task where given two views of the same area,
• 12:40
  SPEAKER_00: you're trying to figure out if some land
• 12:42
  SPEAKER_00: pixel in that image had undergone some change or not.
• 12:46
  SPEAKER_00: And so therefore we picked semantic segmentation,
• 12:48
  SPEAKER_00: A,
• 12:49
  SPEAKER_00: because it's a very common task in remote sensing,
• 12:51
  SPEAKER_00: and B,
• 12:52
  SPEAKER_00: because it's also very expensive to do pixel-wise enabling.
• 12:58
  SPEAKER_01: Yeah, probably the most common task, at least medium resolution, imagery.
• 13:04
  As you say,
• 13:05
  annotation at the pixel level is incredibly time-consuming and
• 13:09
  therefore expensive.
• 13:11
  SPEAKER_01: You have set up your objectives in terms of the downstream task.
• 13:16
  SPEAKER_01: You said you want to be more efficient.
• 13:18
  SPEAKER_01: How did you go about tackling that?
• 13:20
  So we tried to design a pre-training task that was sort of more
• 13:25
  closer in the
• 13:28
  SPEAKER_00: mechanics of how it operates with our target task,
• 13:31
  SPEAKER_00: which was semantic segmentation in this case.
• 13:34
  So I can speak to the specifics of the idea,
• 13:38
  which was actually fairly simple.
• 13:41
  SPEAKER_00: So we reasoned that given an image,
• 13:44
  SPEAKER_00: if there's a sub-image within that image,
• 13:47
  or I try not to use the word patch because it's such an
• 13:50
  overloaded term now with
• 13:51
  SPEAKER_00: VIT and such.
• 13:52
  SPEAKER_00: So I'm going to say sub-image, but
• 13:54
  SPEAKER_00: It's a patch, really, is what it is.
• 13:57
  So given some image and a sub-image within it that you can sort
• 14:03
  of copy over,
• 14:05
  if a network is provided both of these and then asked to
• 14:09
  localize within the full
• 14:11
  SPEAKER_00: image where this patch was pulled out of,
• 14:14
  important distinction here is that we're not cropping this
• 14:17
  patch or the sub-image
• 14:18
  SPEAKER_00: out.
• 14:18
  We're leaving it untouched in the original image,
• 14:21
  but then we're just making a copy of it.
• 14:24
  so to speak and then if both of these are passed into a network
• 14:27
  and then this can
• 14:28
  be treated as a semantic segmentation task right because the
• 14:34
  network is then tasked
• 14:36
  with doing a binary pixel-wise classification where it can say
• 14:39
  that this specific
• 14:40
  pixel within the original image belongs to the extracted
• 14:45
  sub-image or it's a
• 14:46
  SPEAKER_00: background pixel that does not belong
• 14:49
  SPEAKER_00: to the original sub-image.
• 14:50
  SPEAKER_00: So yeah, so the idea is pretty simple.
• 14:52
  It's just localizing within a whole image where the sub-image
• 14:56
  came from as a
• 14:57
  SPEAKER_00: pixel-wise classification task.
• 15:00
  SPEAKER_01: Yeah, that sounds very intuitive.
• 15:02
  And in terms of,
• 15:03
  I guess,
• 15:03
  how you set that up,
• 15:04
  what kind of distances are kind of reasonable?
• 15:06
  Is there a gradient that you have to apply in terms of starting
• 15:09
  with small
• 15:10
  SPEAKER_01: distances moving to larger distances,
• 15:11
  SPEAKER_00: etc.?
• 15:11
  SPEAKER_00: ?
• 15:13
  SPEAKER_00: I think so.
• 15:14
  There were definitely a lot of hyperparameters that we had to
• 15:17
  play around with to
• 15:18
  sort of make this work because this could be a trivial task if
• 15:22
  it's just a
• 15:23
  SPEAKER_00: pixel-wise matching exercise effectively,
• 15:27
  SPEAKER_00: right?
• 15:28
  SPEAKER_00: some of the things we had to do is we had to apply image augmentations
• 15:32
  independently to the sub-image and the original image so you
• 15:36
  can think of this as
• 15:37
  once this sub-image has been copied over has been extracted you
• 15:41
  could rotate it you
• 15:42
  SPEAKER_00: can add some Gaussian noise you can jitter the color you can do
• 15:46
  all manner of things to it so that it doesn't remain a
• 15:49
  trivially simple task for the network.
• 15:53
  SPEAKER_00: We also found that the size of the sub-image matters a lot.
• 15:57
  SPEAKER_00: If it's too small, it becomes a really difficult task.
• 16:00
  task and the network doesn't really truly learn useful
• 16:04
  semantics from the pre-training.
• 16:06
  SPEAKER_00: If it's too big of an image, then it becomes too easy.
• 16:11
  SPEAKER_00: Because this is just a binary pixel-wise classification,
• 16:14
  SPEAKER_00: if the sub-image is say 80% of the area of the original image,
• 16:18
  you can randomly pick a pixel and classify it to be a positive
• 16:22
  label and you have
• 16:24
  SPEAKER_00: an 80% chance then of being right.
• 16:25
  SPEAKER_00: So there's that like balance.
• 16:29
  SPEAKER_00: I think you mentioned something about the gradient.
• 16:33
  SPEAKER_00: I think we didn't focus,
• 16:36
  we didn't end up having to focus a lot on things like vanishing
• 16:41
  gradients,
• 16:41
  SPEAKER_00: exploding gradients,
• 16:42
  SPEAKER_00: but we did find that
• 16:44
  the loss function can affect the pre-trading performance and
• 16:48
  therefore the
• 16:49
  SPEAKER_00: downstream performance.
• 16:50
  SPEAKER_00: So because this is a... Because the pixel is kind of a smaller...
• 16:59
  SPEAKER_00: Sorry, I don't mean the pixel.
• 17:00
  SPEAKER_00: I mean the sub-image is a minority of pixels in the original image.
• 17:04
  SPEAKER_00: They're this class imbalance.
• 17:06
  SPEAKER_00: And so there we rely quite heavily on very traditional...
• 17:11
  very well studied loss functions like the focal loss is a good
• 17:15
  one for this
• 17:17
  situation where you have an imbalanced set of pixels where
• 17:21
  pixels that are harder
• 17:22
  to classify or are in a minority are penalized harder in the
• 17:26
  loss function and
• 17:26
  therefore the gradients from those tends to be stronger than
• 17:30
  easier to classify
• 17:31
  SPEAKER_00: pixels.
• 17:33
  SPEAKER_01: Makes sense.
• 17:34
  SPEAKER_01: And in terms of the evaluation,
• 17:36
  what kind of evaluation did you perform and how did it compare
• 17:38
  to I guess the
• 17:39
  SPEAKER_01: alternatives?
• 17:41
  SPEAKER_00: So since this was a pixel-wise classification task,
• 17:44
  SPEAKER_00: we went with a mean intersection over union,
• 17:48
  SPEAKER_00: which is generally
• 17:51
  SPEAKER_00: I think a good intuitive way to evaluate pixel-wide classification.
• 17:56
  The alternative would be you could do something very simple
• 18:00
  like class-wide
• 18:02
  SPEAKER_00: accuracy,
• 18:03
  but that doesn't take into account the fact that you have this
• 18:06
  sort of uneven
• 18:08
  SPEAKER_00: distribution of classes,
• 18:09
  SPEAKER_00: right?
• 18:09
  So the positive labels are always going to be in a minority
• 18:12
  because we make this
• 18:13
  SPEAKER_00: task sufficiently hard.
• 18:14
  So MIOU is the evaluation metric that we went with for this
• 18:21
  pixel-wise classification task.
• 18:23
  SPEAKER_00: And I mean, this was not our brainwave.
• 18:26
  SPEAKER_00: We see that this is very common in semantic segmentation.
• 18:31
  SPEAKER_00: Fantastic.
• 18:34
  SPEAKER_01: So you've pre-trained and you want to demonstrate,
• 18:38
  SPEAKER_01: I think,
• 18:39
  SPEAKER_01: superior performance and downstream tasks.
• 18:42
  SPEAKER_01: How did it perform then?
• 18:44
  SPEAKER_00: So we had a few different training setups that we attempted.
• 18:49
  SPEAKER_00: So what are the things we can vary here?
• 18:52
  SPEAKER_00: We can attempt different network architectures.
• 18:55
  SPEAKER_00: We can try a few different data sets.
• 18:58
  SPEAKER_00: We can vary the downstream data sets so that it's a different data set,
• 19:03
  SPEAKER_00: a different distribution from the upstream data set.
• 19:05
  SPEAKER_00: And so these were some of the things we tried.
• 19:08
  SPEAKER_00: I can walk over them.
• 19:11
  SPEAKER_00: one by one.
• 19:12
  SPEAKER_00: So we attempted two architectures.
• 19:15
  SPEAKER_00: One was a VIT style,
• 19:17
  SPEAKER_00: a dyno V2 model specifically,
• 19:20
  SPEAKER_00: which is a vision transformer,
• 19:23
  which is sort of the more popular architecture now because it
• 19:29
  tends to model
• 19:30
  SPEAKER_00: long-read dependencies better than traditional continents.
• 19:33
  Well,
• 19:34
  I say traditional,
• 19:34
  but they're only about 10 or 15 years old at this point.
• 19:38
  SPEAKER_00: And then we also tried CNNs,
• 19:40
  SPEAKER_00: a Resonate 50-style backbone,
• 19:42
  SPEAKER_00: because those are still very effective.
• 19:45
  SPEAKER_00: As we know,
• 19:47
  SPEAKER_00: they're less parameter-heavy very often than transformers,
• 19:52
  SPEAKER_00: more efficient because,
• 19:54
  SPEAKER_00: as we know,
• 19:54
  SPEAKER_00: transformers,
• 19:56
  SPEAKER_00: the
• 19:58
  SPEAKER_00: The runtime complexity,
• 19:59
  it scales as a quadratic function of the number of patches that
• 20:03
  we divide the image
• 20:04
  SPEAKER_00: into.
• 20:04
  SPEAKER_00: So we saw value in trying both of these architectures.
• 20:09
  SPEAKER_00: We pre-trained one of our first,
• 20:12
  SPEAKER_00: our initial set of experiments,
• 20:13
  SPEAKER_00: we pre-trained on the,
• 20:16
  SPEAKER_00: let me see,
• 20:17
  SPEAKER_00: I'm forgetting the dataset name very quickly,
• 20:20
  SPEAKER_00: let me look it up.
• 20:21
  We tried a few different datasets and okay,
• 20:23
  we started with Land Cover AI actually,
• 20:25
  so we
• 20:26
  SPEAKER_00: pre-trained on the land cover AI dataset,
• 20:29
  SPEAKER_00: which is aerial imagery from,
• 20:32
  SPEAKER_00: I think,
• 20:33
  SPEAKER_00: somewhere in Poland,
• 20:34
  SPEAKER_00: if I remember correctly.
• 20:35
  SPEAKER_00: And there are, I think, five, maybe seven distinct land cover classes.
• 20:41
  So in the pre-training,
• 20:42
  of course,
• 20:42
  we don't need the labels from the dataset.
• 20:45
  SPEAKER_00: We don't need these land cover classes.
• 20:46
  SPEAKER_00: We just do this localization sub-image overlap pre-training task.
• 20:53
  And so we pre-trained for about 150 epochs with some of those
• 20:58
  augmentations and
• 21:00
  SPEAKER_00: some of those things that I was describing earlier.
• 21:03
  SPEAKER_00: What we found was that the augmentations do matter.
• 21:06
  SPEAKER_00: So doing something like jittering the colors of the image,
• 21:13
  SPEAKER_00: that makes the task quite hard and therefore it's more difficult.
• 21:18
  SPEAKER_00: for the model to learn from this.
• 21:19
  And the training,
• 21:20
  if we see the validation loss,
• 21:22
  it's quite unstable in the pre-training.
• 21:25
  SPEAKER_00: We found that just doing simple flipping of the image,
• 21:29
  flipping and horizontal vertical flips and mirror images of
• 21:34
  either the patch or the
• 21:37
  And remember,
• 21:38
  these are augmented very independently to make the task
• 21:40
  challenging.
• 21:42
  SPEAKER_00: So doing that did help the pre-training quite a bit.
• 21:45
  And so once we did this pre-training and we picked our
• 21:50
  best-performing pre-training
• 21:51
  SPEAKER_00: model,
• 21:52
  SPEAKER_00: then we fine-tuned on the same data set on LandCover AI downstream.
• 21:57
  SPEAKER_00: And we did find that our model actually
• 22:01
  not only had better performance than a model that used
• 22:08
  pre-trained weights from,
• 22:09
  SPEAKER_00: say,
• 22:09
  SPEAKER_00: ImageNet or from the LVDE dataset,
• 22:12
  SPEAKER_00: which is what Dyno is pre-trained on.
• 22:15
  So we had better performance by about one to three percentage
• 22:19
  points,
• 22:20
  which
• 22:22
  SPEAKER_00: maybe isn't a huge difference,
• 22:23
  SPEAKER_00: but I think the biggest difference we saw is in the convergence speed.
• 22:27
  So our model tends to converge significantly faster than the
• 22:31
  Dyno V2 pre-trained
• 22:32
  SPEAKER_00: model,
• 22:34
  SPEAKER_00: which was pre-trained on LVD or say ImageNet ResNet 50.
• 22:39
  So this was one set of experiments where we had the same data
• 22:43
  set downstream.
• 22:44
  And then the question,
• 22:45
  the natural question we asked was that,
• 22:47
  hey,
• 22:47
  we don't want to
• 22:48
  SPEAKER_00: because this whole paper was about being efficient.
• 22:50
  SPEAKER_00: We don't want to pre-train every time on a new data set,
• 22:55
  SPEAKER_00: given that we want to do the downstream task on some other data set.
• 23:00
  SPEAKER_00: So therefore, the next thing we attempted was we
• 23:03
  SPEAKER_00: use the same weights as the LandCover AI dataset pre-trained model.
• 23:08
  SPEAKER_00: And then we varied the downstream dataset.
• 23:11
  SPEAKER_00: So we tried two different datasets.
• 23:13
  We tried the Love DA and DeepClobe datasets,
• 23:17
  both of which also are semantic segmentation.
• 23:20
  SPEAKER_00: These are aerial images and there are semantic segmentation
• 23:24
  labels for land cover classification available for both of
• 23:27
  these datasets.
• 23:28
  And we were able to reproduce the same result where we had a
• 23:31
  bit of a bump in
• 23:32
  SPEAKER_00: performance with pre-training versus without.
• 23:35
  But the big win here was that the convergence was significantly
• 23:39
  faster across all
• 23:41
  SPEAKER_00: these experiments.
• 23:42
  SPEAKER_00: Even when we changed the downstream distribution,
• 23:45
  SPEAKER_00: we pre-trained on land cover AI,
• 23:46
  SPEAKER_00: but then the fine-tuning happened on Love DA or on Deep Lobe.
• 23:51
  so I guess the takeaway there is you pre-train once you perform
• 23:55
  that pre-training
• 23:56
  for all different downstream tasks and then the downstream
• 24:00
  fine-tuning is just much
• 24:02
  SPEAKER_01: faster therefore more practical
• 24:05
  SPEAKER_00: Right, right.
• 24:06
  SPEAKER_00: The one caveat I'd add here is that when you say downstream task,
• 24:11
  SPEAKER_00: we tried this only on semantic segmentation,
• 24:13
  SPEAKER_00: but we did change the dataset.
• 24:15
  SPEAKER_00: It is a natural question, though.
• 24:17
  SPEAKER_00: What if this was not a semantic segmentation task at all?
• 24:20
  SPEAKER_00: What if this was, say, classification?
• 24:23
  SPEAKER_00: What if it was object detection?
• 24:24
  SPEAKER_00: That is not something we were able to try in the setup experiments,
• 24:27
  SPEAKER_00: but we did prove that doing the same task,
• 24:30
  SPEAKER_00: i.e.
• 24:31
  SPEAKER_00: semantic segmentation, but on a different dataset does transfer well.
• 24:36
  and is this something that you see other teams taking this sort
• 24:41
  of approach you
• 24:42
  know varying the pre-training task and if so what other kind of
• 24:46
  approaches do you
• 24:47
  see being experimented with I think within geospatial
• 24:57
  pre-training specifically
• 24:59
  What we're seeing is that there's what I think I personally
• 25:03
  have seen as a trend
• 25:04
  towards more large-scale pre-training with approaches that
• 25:09
  generalize well to a
• 25:11
  SPEAKER_00: number of different tasks.
• 25:13
  SPEAKER_00: And so, which is, I mean, it's a very worthy endeavor.
• 25:16
  SPEAKER_00: I'm not taking that away.
• 25:18
  SPEAKER_00: from those pre-training approaches.
• 25:19
  But of course,
• 25:19
  that comes at the cost of you need a lot of data to pre-train
• 25:23
  with.
• 25:24
  And so we actually did do some experiments where we try to
• 25:28
  measure the efficiency
• 25:29
  SPEAKER_00: of our pre-training with some of the other well-known
• 25:34
  SPEAKER_00: geospatial pre-training tasks, pre-training models.
• 25:39
  SPEAKER_00: So one of the experiments that we tried was we had our pre-trained model,
• 25:47
  we had a ResNet-50 pre-trained model that we pre-trained on our
• 25:52
  task and then
• 25:54
  We took some of these other weights that are openly available
• 25:59
  in TorGio,
• 25:59
  actually.
• 26:00
  SPEAKER_00: So this is a good plug for TorGio.
• 26:02
  SPEAKER_00: If you want to do some experimentation and you need weights from Seasonal
• 26:06
  SPEAKER_00: Contrastive or Satvis Pre-Train or a lot of the models you can think of,
• 26:10
  SPEAKER_00: TorGio makes it super easy to load those weights.
• 26:15
  So the experiment that we tried specifically here was we
• 26:19
  pre-train using our
• 26:20
  SPEAKER_00: pre-training task and then we fine-tune downstream
• 26:25
  SPEAKER_00: on,
• 26:26
  SPEAKER_00: I'm forgetting the specific dataset we use,
• 26:27
  SPEAKER_00: but if I'm remembering correctly,
• 26:29
  SPEAKER_00: it was DeepGlobe.
• 26:31
  SPEAKER_00: And then we have some performance numbers for our pre-trained weights.
• 26:36
  SPEAKER_00: Following that,
• 26:36
  SPEAKER_00: we then use the same architecture,
• 26:39
  SPEAKER_00: ResNet50,
• 26:40
  but we preload weights from a few different well-known
• 26:44
  pre-trained models.
• 26:45
  SPEAKER_00: So these included Satellis pre-trained,
• 26:49
  SPEAKER_00: GASL,
• 26:49
  SPEAKER_00: which stands for Geography Aware Cell Supervised Learning,
• 26:55
  SPEAKER_00: SECO,
• 26:55
  SPEAKER_00: which stands for Seasonal Contrastive.
• 26:58
  We also tried ImageNet Retrained Rates and Random
• 27:02
  Initialization at a baseline.
• 27:05
  So the goal here was to not just benchmark performance and
• 27:12
  convergence speed on
• 27:13
  SPEAKER_00: this downstream deep-globe task or deep-globe segmentation task,
• 27:17
  but also to then compare what is the size of the pre-training
• 27:21
  dataset that we're
• 27:21
  SPEAKER_00: using.
• 27:23
  SPEAKER_00: So what we found was that we did
• 27:26
  SPEAKER_00: better as well as all of the other models.
• 27:29
  SPEAKER_00: There was,
• 27:29
  SPEAKER_00: I think,
• 27:30
  SPEAKER_00: SSL for EOS 12,
• 27:32
  SPEAKER_00: which was slightly ahead of ours in terms of the MIOU performance.
• 27:37
  But we were able to beat all of the other models,
• 27:39
  all of the other pre-training.
• 27:41
  And we actually converged faster than all of these other
• 27:45
  pre-trained weights that we tried.
• 27:48
  And the big takeaway here is that the reason this result is so
• 27:51
  significant is because we...
• 27:54
  use orders of magnitude,
• 27:57
  fewer images,
• 27:58
  fewer spatial pixels in our pre-training.
• 28:00
  So we only had about 7,000 pre-training images from LandCover
• 28:06
  AI and I think about
• 28:08
  SPEAKER_00: 300 validation images,
• 28:10
  whereas all of the other models they use anywhere between a
• 28:14
  million to three
• 28:16
  SPEAKER_00: million images,
• 28:17
  SPEAKER_00: which is the case of SSL for EOS 12.
• 28:20
  SPEAKER_00: So yeah, we had models that
• 28:23
  SPEAKER_00: used maybe over 100x the number of images,
• 28:29
  SPEAKER_00: over 100x the number of spatial pixels as we did,
• 28:32
  SPEAKER_00: and we are as good,
• 28:33
  SPEAKER_00: if not better,
• 28:34
  SPEAKER_00: than those models.
• 28:37
  One other important finding that I actually forgot to mention
• 28:40
  in our initial set of
• 28:41
  SPEAKER_00: experiments with
• 28:43
  pre-training on land cover AI and then also fine-tuning on land
• 28:46
  cover AI is I
• 28:47
  SPEAKER_00: mentioned that gap in performance and convergence.
• 28:51
  If the amount of label data is reduced,
• 28:54
  that gap increases quite drastically.
• 28:57
  SPEAKER_00: So as an example, I'm going back to my paper.
• 29:01
  SPEAKER_00: So when 100% of the downstream label data are used,
• 29:04
  there's maybe about a percentage points worth of difference in
• 29:08
  performance.
• 29:10
  But if you use 50% of the label data,
• 29:12
  that gap increases to somewhere between 3% to 5%.
• 29:15
  SPEAKER_00: If you use a quarter,
• 29:17
  SPEAKER_00: 25% of the label data,
• 29:19
  SPEAKER_00: that gap in performance and convergence actually keeps increasing.
• 29:23
  SPEAKER_00: So that was another important finding we had.
• 29:27
  And there's lots of kind of contemporary criticism about
• 29:30
  foundational models and
• 29:31
  SPEAKER_01: pre-training just being very kind of resource-intensive.
• 29:36
  This sounds like a very promising alternative direction for
• 29:39
  those kind of scenarios
• 29:41
  SPEAKER_01: where you're constrained on compute or available data.
• 29:45
  SPEAKER_01: Maybe you'd be working on some kind of niche platform,
• 29:48
  SPEAKER_01: say,
• 29:49
  SPEAKER_01: or you have a limited data set in total.
• 29:53
  SPEAKER_01: This sounds like a really promising direction.
• 29:56
  SPEAKER_00: Yeah, absolutely.
• 29:58
  SPEAKER_00: I mean,
• 29:58
  SPEAKER_00: of course,
• 29:58
  SPEAKER_00: the most common and the most,
• 30:01
  SPEAKER_00: I think,
• 30:02
  SPEAKER_00: practical way of doing these things is try to find a model that has been
• 30:06
  pre-trained on a vast amount of data that is known to
• 30:09
  generalize well and then
• 30:10
  SPEAKER_00: fine-tune it on whatever it is that you care about.
• 30:14
  But what if you just have a very niche task and you just want
• 30:18
  to do some very
• 30:19
  SPEAKER_00: specific pre-training targeted towards your specific task,
• 30:22
  SPEAKER_00: towards your specific data set?
• 30:24
  SPEAKER_00: That's, I think, really where
• 30:26
  SPEAKER_00: our approach brings value.
• 30:28
  One other thing that we didn't actually try because we were
• 30:31
  trying to make all this
• 30:32
  work on one single T4 GPU is you could actually pre-train on a
• 30:38
  lot of data.
• 30:38
  SPEAKER_00: We only pre-train on about 7,000 to 10,000 images,
• 30:41
  SPEAKER_00: but what if you could pre-train on a million images,
• 30:43
  SPEAKER_00: right?
• 30:44
  What if you could do the sub-image overlap task on a million
• 30:47
  images and then
• 30:47
  SPEAKER_00: compare it and then see how the performance downstream varies?
• 30:51
  SPEAKER_00: That is something that I'd be very curious to see.
• 30:56
  SPEAKER_01: I'll throw that out there for the next paper.
• 30:59
  SPEAKER_01: Well, thanks for covering that topic.
• 31:02
  SPEAKER_01: Really interesting and,
• 31:04
  SPEAKER_01: like I say,
• 31:04
  slightly taking it in a different direction from what a lot of
• 31:08
  what's happening in
• 31:08
  SPEAKER_01: the field is taking.
• 31:10
  SPEAKER_01: What's the name of the paper and where is it published?
• 31:13
  SPEAKER_00: So this paper,
• 31:14
  it's titled Sub-Image Overlap Prediction Task-Aligned
• 31:18
  Self-Supervised Pre-Training
• 31:19
  SPEAKER_00: for Semantic Segmentation in Remote Sensing Imagery.
• 31:23
  Very wordy,
• 31:23
  but if you just search for sub-image overlap prediction,
• 31:26
  it should show up.
• 31:28
  This was published at the Computer Vision for Earth Observation
• 31:31
  Workshop in March
• 31:34
  SPEAKER_00: of 2026,
• 31:35
  SPEAKER_00: March of this year.
• 31:37
  SPEAKER_00: And this was a part of the WACV conference,
• 31:40
  which I think stands for Winter Conference on the Application
• 31:43
  of Computer Vision.
• 31:45
  SPEAKER_00: Happened in Tucson, Arizona.
• 31:46
  SPEAKER_00: Happened in Tucson, Arizona this year.
• 31:48
  SPEAKER_00: It's in Orlando next year.
• 31:51
  SPEAKER_00: One of my favorite conferences.
• 31:52
  SPEAKER_00: Highly recommend if you're considering attending a conference.
• 31:56
  SPEAKER_01: Mm-hmm.
• 31:57
  SPEAKER_00: and hit the water parks afterwards, I guess.
• 32:00
  SPEAKER_00: Exactly, yeah.
• 32:00
  SPEAKER_00: I think that's definitely appealing.
• 32:02
  SPEAKER_00: Fantastic.
• 32:03
  All right,
• 32:04
  so if people want to follow along your updates,
• 32:06
  is LinkedIn the best platform?
• 32:08
  SPEAKER_00: Yes, yeah.
• 32:09
  SPEAKER_00: So I can be found on LinkedIn, Lakshay Sharma, my full name.
• 32:14
  SPEAKER_00: You may find other people with the same name,
• 32:15
  SPEAKER_00: but if you suffix that with Instacart or Microsoft,
• 32:19
  SPEAKER_00: I think that should be my profile that shows up.
• 32:23
  SPEAKER_01: Fantastic.
• 32:23
  SPEAKER_01: I'll drop by the details in the show notes.
• 32:26
  SPEAKER_01: And once again, thank you for joining me and all the dust for the future.
• 32:29
  SPEAKER_00: Awesome.
• 32:30
  SPEAKER_00: Thanks so much, Robin.
• 32:31
  SPEAKER_00: Thank you.
• 32:32
  SPEAKER_01: Yeah.