Latest / Satellite image deep learning / Tessera: A Temporal Foundation Model for Earth Observation

Transcript

Tessera: A Temporal Foundation Model for Earth Observation · Satellite image deep learning

• 0:05
  SPEAKER_00: Hi guys, welcome to the podcast.
• 0:08
  SPEAKER_00: Hi.
• 0:10
  SPEAKER_00: Let's begin with a brief introduction to yourselves.
• 0:13
  SPEAKER_01: So I'm Sadiq Jaffa.
• 0:15
  I'm an assistant research professor here in the Energy and
• 0:18
  Environment Group in the
• 0:20
  Department of Computer Science and Technology at the University
• 0:22
  of Cambridge.
• 0:23
  SPEAKER_01: I specialize in AI for climate and nature projects.
• 0:27
  SPEAKER_02: Thank you.
• 0:27
  SPEAKER_02: Frank, yourself?
• 0:29
  SPEAKER_02: Yeah, so hi, I'm Jampeng Feng, and also I go by Frank.
• 0:33
  I'm a second year PhD student from University of Cambridge from
• 0:37
  Department of Computer Science.
• 0:38
  And I've been working with SEDIC and also a bunch of other of
• 0:41
  my colleagues on,
• 0:42
  SPEAKER_02: you know,
• 0:43
  SPEAKER_02: Tessera,
• 0:44
  and also my research interests a lot in the intersection of AI
• 0:48
  and remote sensing.
• 0:49
  SPEAKER_00: Fantastic.
• 0:50
  SPEAKER_00: Thank you both.
• 0:51
  SPEAKER_00: Let's begin with the name Tessera.
• 0:53
  SPEAKER_00: What does it mean?
• 0:53
  SPEAKER_00: What does it stand for?
• 0:55
  SPEAKER_01: Oh, no.
• 0:56
  SPEAKER_01: Can we remember what the Bacronym is?
• 0:59
  SPEAKER_01: The Bacronym?
• 1:02
  SPEAKER_01: Temporal Embeddings.
• 1:03
  SPEAKER_01: It's on the paper title.
• 1:05
  I think it's Temporal Embeddings for Earth Surfaces,
• 1:10
  for Earth Representations and Analysis.
• 1:14
  SPEAKER_02: So in short, it's Tessera.
• 1:16
  SPEAKER_00: I like the name, to be honest.
• 1:18
  SPEAKER_00: Let's begin with the temporal bit.
• 1:19
  SPEAKER_00: What's special about this model?
• 1:24
  SPEAKER_02: Do you want to start first?
• 1:26
  SPEAKER_01: Yeah.
• 1:26
  So,
• 1:26
  um,
• 1:27
  so,
• 1:28
  uh,
• 1:28
  the temporal in the sense that,
• 1:30
  um,
• 1:30
  it's not a point in time snapshot.
• 1:33
  SPEAKER_01: Um,
• 1:33
  the embeddings are annualized embeddings that give you the
• 1:38
  temporal temporal
• 1:39
  SPEAKER_01: spectral characteristics of that particular point of earth over one year.
• 1:44
  SPEAKER_01: So, and we'd use a calendar year.
• 1:47
  SPEAKER_00: Fantastic.
• 1:48
  SPEAKER_00: So if you wanted to compare change from one year to the next,
• 1:50
  SPEAKER_00: this is the logical way to segment.
• 1:53
  SPEAKER_01: Yep.
• 1:54
  SPEAKER_00: Fantastic.
• 1:54
  SPEAKER_00: And what does it mean in practical terms,
• 1:56
  SPEAKER_00: in terms of the data that's available,
• 1:58
  SPEAKER_00: maybe how you train the model,
• 1:59
  SPEAKER_00: et cetera?
• 2:00
  SPEAKER_01: So,
• 2:01
  SPEAKER_01: I mean,
• 2:01
  SPEAKER_01: the reason why we capture that temporal,
• 2:05
  SPEAKER_01: those temporal characteristics is for things like plants,
• 2:08
  SPEAKER_01: the phenology,
• 2:09
  how the spectral characteristics of those plants change over
• 2:12
  the year are very
• 2:13
  SPEAKER_01: distinctive.
• 2:14
  SPEAKER_01: And so by kind of capturing it over a year,
• 2:17
  SPEAKER_01: we're able to discriminate things,
• 2:18
  which if you just took a point in time snapshot from one
• 2:22
  capture from Sentinel-2,
• 2:23
  you wouldn't be able to discriminate between two different
• 2:28
  plants that are very
• 2:28
  SPEAKER_01: close by.
• 2:29
  SPEAKER_01: In terms of training,
• 2:31
  it does mean that we need to pull down a lot of data because
• 2:36
  every inference run we
• 2:37
  do to produce embeddings for a year requires us to download the
• 2:40
  entire year's worth
• 2:41
  SPEAKER_01: of data for that particular point.
• 2:44
  SPEAKER_00: Makes sense.
• 2:45
  SPEAKER_00: And you chose which modalities?
• 2:48
  SPEAKER_02: So we use... I think probably I can answer this.
• 2:52
  So I think one thing I want to add to the previous question is
• 2:58
  that actually we
• 2:59
  SPEAKER_02: need to download basically all the Sensor 1,
• 3:01
  SPEAKER_02: Sensor 2 data.
• 3:02
  SPEAKER_02: So that also answers your third question.
• 3:05
  SPEAKER_02: So we use Sensor 1, Sensor 2.
• 3:07
  And so since the Tessera is a temporal model,
• 3:11
  and also we put down the whole year data.
• 3:13
  SPEAKER_02: And right now we're doing embeddings from 2017 to 2025.
• 3:17
  So that means we have to put down all the available like
• 3:20
  Sentinel-1,
• 3:20
  Sentinel-2 data.
• 3:22
  SPEAKER_02: I think that would be over 25 petabyte of data.
• 3:25
  SPEAKER_02: So it's like a massive huge amount of data and we're doing.
• 3:30
  SPEAKER_00: And do you just train on all of it or do you have to curate it somehow?
• 3:34
  So in training,
• 3:34
  we select actually over 3,000 MGRS tiles across the globe.
• 3:42
  SPEAKER_02: And also, when choosing these tiles, we also consider their distribution.
• 3:47
  SPEAKER_02: That means, let's say we sample some data from forests.
• 3:50
  SPEAKER_02: We also have some tiles from deserts, rainforests.
• 3:54
  SPEAKER_02: So we don't want to make it as rich as possible that we can do.
• 3:58
  SPEAKER_00: And does that...
• 4:02
  SPEAKER_00: involve a lot of work just choosing those locations,
• 4:04
  SPEAKER_00: or is it kind of a straightforward query and you're happy?
• 4:08
  SPEAKER_02: I would say it's kind of straightforward.
• 4:10
  SPEAKER_02: So basically what we do is we select randomly,
• 4:13
  but also one thing that we have to keep in mind is that we need
• 4:17
  to avoid choosing
• 4:18
  SPEAKER_02: some tiles that has overlaps with our validation data.
• 4:22
  SPEAKER_02: So for instance, we have a crop data set in our validation pipeline,
• 4:27
  SPEAKER_02: So when choosing the training data,
• 4:29
  SPEAKER_02: we have to exclude the tiles that falls into that region of interest.
• 4:35
  SPEAKER_00: Very interesting.
• 4:35
  SPEAKER_00: So the validation is influencing the data that's used to train a model.
• 4:41
  SPEAKER_02: Yes.
• 4:42
  SPEAKER_00: Interesting.
• 4:43
  And it's worth pointing out,
• 4:44
  so we got Tessera is unusual when it comes to model training.
• 4:48
  Usually the hard part is training the model and then inference
• 4:52
  is relatively simple
• 4:53
  SPEAKER_01: and cheap,
• 4:54
  SPEAKER_01: relatively.
• 4:56
  SPEAKER_01: In Tessera's case, it's the other way around.
• 4:58
  So training the model still requires a fair bit of work and
• 5:02
  data movement,
• 5:02
  SPEAKER_01: but actually inference is where it gets really expensive.
• 5:05
  Because we then have to have all of the data we need in order
• 5:08
  to produce those
• 5:09
  SPEAKER_01: embeddings for all of those years across both those modalities.
• 5:13
  SPEAKER_00: Yeah.
• 5:14
  What is the data requirement for a single embedding given it's
• 5:17
  a whole year and
• 5:17
  SPEAKER_00: it's two modalities?
• 5:21
  SPEAKER_02: That's probably a franc.
• 5:24
  SPEAKER_02: Yeah, you mean like the number of observations in the year?
• 5:27
  Yeah,
• 5:27
  I'm just trying to get a sense for you say it's,
• 5:29
  you know,
• 5:29
  costly data wise.
• 5:31
  SPEAKER_00: I'm just trying to get a sense for that.
• 5:33
  So usually,
• 5:34
  I think on average,
• 5:37
  we have 76 Sentinel-2 observations across the globe.
• 5:42
  SPEAKER_02: And I think for Sentinel-1,
• 5:43
  SPEAKER_02: I think we have more data in Europe,
• 5:46
  SPEAKER_02: but also maybe less data in Africa.
• 5:49
  But so,
• 5:49
  yeah,
• 5:50
  I would say probably for Sentinel-2,
• 5:51
  we have 76 observations.
• 5:53
  SPEAKER_02: And for Sentinel-1, we have probably 16 observations.
• 5:57
  SPEAKER_02: So that is actually quite a lot of data when producing the embeddings.
• 6:01
  SPEAKER_00: So approximately 120, 130 images get compressed down to a small vector.
• 6:08
  SPEAKER_01: Yes, exactly.
• 6:09
  SPEAKER_00: And it requires,
• 6:10
  SPEAKER_01: I think we worked out it was about 180,000 GPU hours for the eight,
• 6:16
  SPEAKER_01: seven years of embeddings across those seven years.
• 6:22
  SPEAKER_00: Right.
• 6:23
  SPEAKER_00: So in practice, you're not expecting everyone to compute the embeddings.
• 6:25
  SPEAKER_00: You've just done it and distributed them.
• 6:28
  SPEAKER_00: Correct.
• 6:28
  SPEAKER_00: Very cool.
• 6:30
  And what decisions did you have to make when you chose your
• 6:32
  training strategies?
• 6:37
  Well,
• 6:38
  actually,
• 6:38
  I think in terms of training,
• 6:41
  of course,
• 6:41
  the Tessera is a pixel-based model.
• 6:43
  SPEAKER_02: So it's quite different from any other foundation models,
• 6:45
  SPEAKER_02: which I know most of the foundation models,
• 6:47
  SPEAKER_02: they're patch-based.
• 6:48
  So I think in terms of training strategy,
• 6:50
  we adopt several different ways to train the model.
• 6:56
  So first of all,
• 6:56
  what we do is,
• 6:57
  since if we only select all the pixels within the patch,
• 7:01
  SPEAKER_02: then it means that basically most of the pixels, they look the same.
• 7:04
  SPEAKER_02: They look very similar within the patch.
• 7:06
  So what we do during the training is that we actually gather as
• 7:09
  many pixels as
• 7:10
  SPEAKER_02: possible within the globe,
• 7:14
  SPEAKER_02: and then we just randomly sample them.
• 7:17
  SPEAKER_02: Basically, we just scramble them.
• 7:18
  SPEAKER_02: This makes sure that the loss converges smoothly,
• 7:23
  SPEAKER_02: and also this prevents from the feature collapse.
• 7:26
  SPEAKER_02: And also, there are some other stuff that we do.
• 7:28
  SPEAKER_02: So first, we have a lot of GPUs, of course.
• 7:32
  SPEAKER_02: So we set up a very large batch size.
• 7:35
  SPEAKER_02: I think it's over 32,000 in terms of batch size.
• 7:39
  SPEAKER_02: And also, there are some other tricks that we do in terms of optimizer.
• 7:43
  SPEAKER_02: And also, we're doing this.
• 7:47
  SPEAKER_02: learning rate warm up and then cosine annealing stuff.
• 7:50
  SPEAKER_02: So yeah, so there's a lot of tricks that we do for the training.
• 7:54
  SPEAKER_02: And yes, everyone can find the training details in our paper as well.
• 7:57
  I'm really curious because you mentioned it's pixel based and
• 8:00
  obviously there's
• 8:00
  SPEAKER_00: other models where they're image based and you do this kind of masked
• 8:04
  SPEAKER_00: reconstruction.
• 8:04
  SPEAKER_00: How does it differ for pixel based then?
• 8:07
  Yeah,
• 8:08
  so we adopt a different way when it comes to self-supervised
• 8:12
  training.
• 8:12
  SPEAKER_02: So like you said,
• 8:13
  SPEAKER_02: like you mentioned,
• 8:14
  a lot of patch-based methods that you actually use mask
• 8:18
  autoencoder to do the SSL
• 8:21
  SPEAKER_02: training.
• 8:22
  SPEAKER_02: But for pixel-based, you can do that.
• 8:24
  SPEAKER_02: I mean,
• 8:24
  SPEAKER_02: you can just simply just mask along the temporal dimension,
• 8:28
  SPEAKER_02: not on the spatial dimension.
• 8:30
  SPEAKER_02: You can actually mask some of the observations.
• 8:33
  SPEAKER_02: And then you predict the missing...
• 8:36
  observation you can do that as well but what we do is quite
• 8:38
  different so we adopted
• 8:39
  a method it's called bottle twins so essentially what bulletins
• 8:43
  does is that we
• 8:44
  sample two augmentations or actually two subsets of the whole
• 8:49
  year's observations
• 8:50
  and then we just want to make sure that although these two
• 8:53
  subsets are different
• 8:55
  but we want to reduce the like the reduction within these two
• 8:59
  subsets by only
• 9:00
  SPEAKER_02: maintaining the most essential information
• 9:03
  SPEAKER_02: So this is what we do for the Cesario,
• 9:06
  SPEAKER_02: like self-supervised training,
• 9:07
  SPEAKER_02: which is,
• 9:08
  SPEAKER_02: I think,
• 9:08
  SPEAKER_02: this is actually the first foundation model that uses bottle twins,
• 9:14
  SPEAKER_02: uses this reduction,
• 9:16
  SPEAKER_02: sorry,
• 9:17
  SPEAKER_02: redundancy reduction method.
• 9:20
  And the intuition there is that it should reconstruct similar
• 9:26
  embeddings if it's
• 9:27
  SPEAKER_00: got the second half versus the first half of the year then.
• 9:31
  SPEAKER_01: Yeah, something like that.
• 9:33
  SPEAKER_01: But they're not different parts of the year.
• 9:37
  They're two sets of observations that span the year,
• 9:40
  but they might be different.
• 9:42
  So given two sets of observations at the same point,
• 9:47
  you want the same representation.
• 9:49
  SPEAKER_00: Right.
• 9:49
  SPEAKER_00: The observations being different modalities or...
• 9:52
  SPEAKER_01: The observations being a subset of the total observations.
• 9:57
  SPEAKER_01: So we basically take two different samples.
• 9:59
  SPEAKER_00: Right.
• 10:00
  So you randomly select some fracture and for set A and the
• 10:03
  remainder of a set B.
• 10:04
  Okay.
• 10:05
  SPEAKER_00: Yes.
• 10:05
  SPEAKER_00: Okay.
• 10:06
  SPEAKER_00: Very interesting.
• 10:07
  And Frank is underselling the amount of architectural work and
• 10:11
  high parameter
• 10:12
  SPEAKER_01: optimization stuff.
• 10:13
  SPEAKER_01: Actually trying to get Barlow Twins to continue to scale is quite tricky.
• 10:18
  And it requires some of the tricks that we've kind of got in
• 10:21
  the paper to be able
• 10:23
  SPEAKER_01: to not have the representation collapse after a while.
• 10:26
  SPEAKER_00: Right.
• 10:28
  And those are tricks that you learn along the way or they're
• 10:31
  kind of standard
• 10:31
  SPEAKER_00: tricks that people are adopting now?
• 10:35
  SPEAKER_02: Actually, we adopted some common practices.
• 10:42
  SPEAKER_02: And also, actually, when training the model along the way.
• 10:45
  Actually,
• 10:46
  at first,
• 10:46
  I almost gave up the model training,
• 10:48
  because at first,
• 10:48
  it's really a pain.
• 10:50
  SPEAKER_02: The loss won't converge, and also the embedding just sucks.
• 10:54
  SPEAKER_02: It really sucks.
• 10:55
  SPEAKER_02: But then at some point, we just realize some issues.
• 10:59
  SPEAKER_02: Then we just dive into that.
• 11:00
  SPEAKER_02: We just analyze what can be the reason behind that.
• 11:04
  SPEAKER_02: Then we just fix that, fix that.
• 11:05
  And after iterating it for several times,
• 11:08
  then finally the loss converges.
• 11:11
  SPEAKER_02: And also the embedding gets better.
• 11:12
  SPEAKER_02: At least from the map, it looks pretty.
• 11:16
  SPEAKER_02: So yeah, I think we learn a lot along the way.
• 11:20
  SPEAKER_00: So first sort of iteration phase is get the loss down.
• 11:24
  I want to circle back to the evaluation approach because I was
• 11:28
  kind of intrigued
• 11:29
  SPEAKER_00: that it also influenced the selection of the training data.
• 11:31
  SPEAKER_00: Can you talk a bit more about that?
• 11:35
  SPEAKER_02: It's quite simple because when training this AI model,
• 11:38
  SPEAKER_02: especially when you use the AI model for evaluation,
• 11:42
  SPEAKER_02: I think one concern is that is there a data leakage?
• 11:47
  So when training the data,
• 11:48
  so when training the model,
• 11:49
  we just want to exclude
• 11:51
  We just want to make sure that all the data used for evaluation
• 11:55
  are not in the training set.
• 11:57
  I think training with Earth data is quite different with
• 12:03
  training with language
• 12:04
  SPEAKER_02: model and also other models.
• 12:06
  SPEAKER_02: Because in remote sensing, basically we have unlimited data, right?
• 12:10
  SPEAKER_02: We have a lot of satellite images.
• 12:13
  SPEAKER_02: So I think this is possible.
• 12:15
  So that's why we just exclude the region of interest that falls
• 12:20
  into the validation data set.
• 12:22
  SPEAKER_02: So, yeah.
• 12:23
  SPEAKER_00: So quite a simple data set design in that sense.
• 12:26
  SPEAKER_00: I'm just curious,
• 12:27
  SPEAKER_00: across the different research teams,
• 12:29
  do they agree on the evaluation tasks and data sets or is that
• 12:34
  still something
• 12:34
  SPEAKER_00: that's being figured out?
• 12:37
  SPEAKER_01: So we had a couple of
• 12:41
  evaluation tasks or validation tasks that we also had reported
• 12:45
  during training.
• 12:47
  And the reason for that is you don't just want your loss to
• 12:49
  converge and then find
• 12:50
  out later on in training that actually your loss being low had
• 12:54
  absolutely nothing
• 12:54
  SPEAKER_01: to do with the actual performance on downstream tasks.
• 12:58
  So there were a couple of different tasks that we had in there
• 13:02
  that we had
• 13:03
  SPEAKER_01: of the training loop and we could actually see that their performance was
• 13:05
  increasing as we scaled parameters and scaled data um it's
• 13:09
  worth pointing out that
• 13:09
  this is actually like a much larger team uh we have many
• 13:12
  colleagues all of whom are
• 13:14
  on the paper um and uh so there was a lot of as we were
• 13:18
  evaluating the model there
• 13:19
  was a lot of trying to kind of figure out from colleagues where
• 13:22
  where else we
• 13:23
  SPEAKER_01: should look how else we should try and kind of fix it fantastic
• 13:30
  SPEAKER_00: You've got a couple of tasks that you evaluated on.
• 13:32
  SPEAKER_00: I know you have some visualizations.
• 13:34
  SPEAKER_00: Maybe now is a good time to dive into those.
• 13:37
  SPEAKER_01: Yeah, sure.
• 13:37
  SPEAKER_01: So I kind of want to just show two.
• 13:39
  SPEAKER_01: It says sharing is not turned on.
• 13:41
  SPEAKER_00: Oh, God.
• 13:42
  SPEAKER_00: They changed Zoom.
• 13:44
  SPEAKER_00: And here we go.
• 13:45
  SPEAKER_00: It should be working now.
• 13:47
  SPEAKER_00: Allow you to present.
• 13:52
  SPEAKER_00: We've lost your audio, though.
• 13:54
  SPEAKER_00: You've muted yourself.
• 13:55
  SPEAKER_00: Yeah.
• 14:01
  SPEAKER_02: Sorry, Sadik, we can't hear you.
• 14:03
  SPEAKER_01: No, no, it's fine.
• 14:04
  SPEAKER_01: It's fine.
• 14:04
  SPEAKER_01: I can't use Zoom.
• 14:05
  SPEAKER_01: Okay.
• 14:06
  SPEAKER_01: So I want to just show two downstream tasks.
• 14:09
  SPEAKER_01: So like I said,
• 14:10
  there's a big team around Tessera and we work with a lot of
• 14:14
  ecologists and plant
• 14:15
  SPEAKER_01: scientists and now conservationists.
• 14:18
  And many people are using Tessera to do downstream tasks where
• 14:24
  it's dramatically
• 14:24
  speeding up their ability to use remote sensing data to kind of
• 14:28
  answer vital
• 14:29
  SPEAKER_01: questions in ecology and conservation.
• 14:32
  SPEAKER_01: But this is just one example here of our colleague Yovina.
• 14:37
  She took two sets,
• 14:41
  two areas where there have been wildfires in California.
• 14:45
  SPEAKER_01: And she took samples from those areas and then did a UMAP,
• 14:53
  SPEAKER_01: so unsupervised dimension anti-reduction.
• 14:56
  SPEAKER_01: So from 128 dimensions down to two dimensions.
• 15:00
  SPEAKER_01: This is purely unsupervised.
• 15:01
  SPEAKER_01: There's no use of labels.
• 15:03
  But then what she did is she then colored it based on whether
• 15:06
  there was a fire and
• 15:08
  SPEAKER_01: then when the fire happened and how severe it was.
• 15:11
  And you can see that kind of there is clear separation here for
• 15:17
  where there was a
• 15:17
  SPEAKER_01: fire and what time of year it happened and how severe it was.
• 15:21
  So kind of that Tesla representation has already learned those
• 15:26
  features and it's there.
• 15:28
  SPEAKER_01: And so obviously,
• 15:30
  SPEAKER_01: if you have that kind of separation,
• 15:32
  SPEAKER_01: you know,
• 15:32
  SPEAKER_01: a model kind of built on this is going to perform quite well.
• 15:36
  SPEAKER_00: It's interesting that the two fires are on different clusters as well.
• 15:39
  SPEAKER_01: Exactly.
• 15:40
  SPEAKER_01: So that temporal information,
• 15:41
  SPEAKER_01: despite the fact that you're only got one representation for the year,
• 15:44
  there is temporal information in there as to when things happen
• 15:47
  during the year.
• 15:48
  And we're still trying to understand just how much and how
• 15:51
  precise it is.
• 15:53
  And just to give you another example,
• 15:55
  this is an example from our colleague,
• 15:58
  Madeleine Esaes.
• 16:00
  She did something very similar in Senegal where she took a
• 16:04
  bunch of points in
• 16:04
  Senegal and then she did a two dimensional unsupervised
• 16:11
  dimensional anti-reduction.
• 16:13
  And then use the land cover ground truce from the agricultural
• 16:18
  monitoring JECAM project.
• 16:21
  SPEAKER_01: And again,
• 16:21
  you can see this separation showing you that the land cover
• 16:25
  kind of very nicely
• 16:26
  SPEAKER_01: separates out in these different areas.
• 16:28
  And that's another good example of where you would then expect
• 16:31
  the kind of a
• 16:33
  SPEAKER_01: downstream map produced from this to be a very high quality.
• 16:37
  SPEAKER_00: Absolutely.
• 16:39
  It almost looks like these embeddings have solved the annual
• 16:43
  tasks at this point.
• 16:46
  SPEAKER_00: Is that controversial?
• 16:49
  SPEAKER_01: I don't know.
• 16:51
  SPEAKER_01: I mean, there's still more to go.
• 16:52
  SPEAKER_01: We know with, I'm just going to stop showing this for a second.
• 16:57
  SPEAKER_01: We know,
• 16:58
  and the paper has a bunch of these scaling work that we've done
• 17:02
  to show that as we
• 17:03
  SPEAKER_01: add more data and add more parameters,
• 17:05
  the embeddings get better and our performance on downstream
• 17:08
  tasks improves.
• 17:10
  SPEAKER_01: So maybe, but this is also as bad as they're going to get.
• 17:15
  SPEAKER_01: They will only get better from here.
• 17:17
  SPEAKER_00: Right.
• 17:18
  SPEAKER_00: And so what do you see the immediate impact being of this new capability?
• 17:23
  So we have colleagues in ecology where they've had projects
• 17:28
  that would normally
• 17:28
  SPEAKER_01: have been kind of 18 months with a postdoc.
• 17:31
  And we've had some master's students over the summer produce
• 17:34
  maps in a week or two.
• 17:36
  SPEAKER_01: Because instead of having to spend time with,
• 17:38
  SPEAKER_01: you know,
• 17:39
  SPEAKER_01: dealing with clouds and dealing with,
• 17:40
  SPEAKER_01: you know,
• 17:41
  SPEAKER_01: the gaps in observations or missing observations,
• 17:43
  SPEAKER_01: that kind of thing,
• 17:44
  SPEAKER_01: it's very easy to take Tessera for a year and even put a very,
• 17:47
  SPEAKER_01: very simple model on top of it.
• 17:50
  So increasing the velocity,
• 17:51
  reducing the barriers to the use of the remote sensing data.
• 17:56
  SPEAKER_01: Yes,
• 17:57
  while also making sure that people are not completely reliant
• 18:00
  on us in that the
• 18:01
  SPEAKER_01: model is open source,
• 18:02
  SPEAKER_01: the weights are open source,
• 18:04
  SPEAKER_01: the embeddings under a Creative Commons batch,
• 18:05
  SPEAKER_01: we should license.
• 18:06
  SPEAKER_01: So, you know, you don't have to rely on us to use it.
• 18:10
  SPEAKER_00: Right.
• 18:12
  And so I guess in that context,
• 18:14
  you know,
• 18:14
  you'd be able to get to a certain point faster.
• 18:16
  Do you think most people or will it be task by task would be
• 18:20
  using just the
• 18:21
  SPEAKER_00: embeddings or do you think the embeddings are a stepping stone?
• 18:26
  The advice I've been giving to people is that Tessera should be
• 18:28
  your baseline.
• 18:29
  If you've got a remote sensing task that you always do,
• 18:32
  try it with Tessera.
• 18:34
  SPEAKER_01: You may find that actually the performance is exactly what you want.
• 18:38
  SPEAKER_01: And certainly in many of the tasks,
• 18:41
  SPEAKER_01: my ecologist colleagues and plant scientists,
• 18:43
  SPEAKER_01: it is more than sufficient for what they want.
• 18:46
  It has another property which surprised us,
• 18:49
  which is incredibly label efficient.
• 18:51
  SPEAKER_01: So because you are fitting a model model
• 18:54
  only to the 128 dimension embeddings and you're not trying to
• 18:58
  learn those physical
• 19:00
  features of the raw observations you're not spending those
• 19:03
  precious labels fitting
• 19:04
  SPEAKER_01: parameters that are actually more about the data distribution of the raw
• 19:08
  observations and so actually you end up with needing a tiny
• 19:11
  fraction of the labels
• 19:13
  SPEAKER_01: So where you've got a problem that is very label scarce,
• 19:16
  SPEAKER_01: and maybe you've tried remote sensing raw data in the past,
• 19:20
  SPEAKER_01: it's very much worth trying with Tessera again,
• 19:23
  SPEAKER_01: and you may be surprised in the performance.
• 19:27
  SPEAKER_00: Fantastic.
• 19:27
  SPEAKER_00: That's a really good summary there.
• 19:29
  I just want to close up by saying what's next on the research
• 19:34
  dimension then.
• 19:35
  SPEAKER_00: It sounds like scaling is one possible avenue of the ones.
• 19:41
  SPEAKER_01: Yeah,
• 19:41
  SPEAKER_01: so once you've got this kind of compact representation,
• 19:44
  SPEAKER_01: there are things you can do with it,
• 19:46
  SPEAKER_01: like change detection.
• 19:47
  You can look at the embeddings year to year and try and
• 19:50
  understand has something
• 19:51
  SPEAKER_01: changed and what.
• 19:54
  And the idea is that because you've abstracted away a lot of
• 19:57
  those things like
• 19:58
  SPEAKER_01: clouds and temporal gaps and that kind of thing.
• 20:01
  SPEAKER_01: I mean,
• 20:01
  SPEAKER_01: the seasonality,
• 20:03
  SPEAKER_01: you're able to get a much more precise answer to yes,
• 20:05
  SPEAKER_01: something has changed in a way that doesn't get fooled by those.
• 20:09
  SPEAKER_01: So we've got colleagues looking at using test for change detection.
• 20:13
  SPEAKER_01: And there's also things like forecasting,
• 20:15
  SPEAKER_01: because,
• 20:16
  SPEAKER_01: again,
• 20:16
  SPEAKER_01: you've now got this nice kind of latent space that you could forecast in,
• 20:20
  SPEAKER_01: as opposed to the forecasting,
• 20:23
  SPEAKER_01: the raw observations themselves,
• 20:24
  which mean you have to model things like clouds and temporal
• 20:27
  gaps and speckle and
• 20:28
  SPEAKER_01: those kind of things.
• 20:30
  SPEAKER_01: So those are kind of new areas beyond just scaling the existing model.
• 20:36
  SPEAKER_00: Right.
• 20:38
  SPEAKER_00: And also modalities,
• 20:40
  SPEAKER_00: the different modalities,
• 20:41
  SPEAKER_00: are they going to be a part of future models,
• 20:44
  SPEAKER_00: you think?
• 20:46
  SPEAKER_02: Yes, this is actually something we've been trying to do.
• 20:49
  SPEAKER_02: So I think like change detection.
• 20:51
  SPEAKER_02: Sudeikis is very into change detection,
• 20:53
  while one of my major tasks right now is to develop like
• 20:57
  Tessera version 2.
• 20:59
  SPEAKER_02: So maybe in version 2.
• 21:00
  SPEAKER_02: It's not maybe, actually it is.
• 21:02
  SPEAKER_02: So in version 2.
• 21:03
  So we're going to have then set into the Tessera embedding,
• 21:07
  into the Tessera model.
• 21:08
  SPEAKER_02: So there will be sensor one, sensor two, and Landsat.
• 21:11
  And also what's more fascinating about Landsat is that we have
• 21:14
  data for Landsat.
• 21:16
  SPEAKER_02: We have data since 1970s.
• 21:18
  SPEAKER_02: So let's say, think about that.
• 21:20
  SPEAKER_02: Once we have Landsat into our model,
• 21:22
  SPEAKER_02: then it is possible that we can generate embeddings from 1970s onwards.
• 21:28
  So that will give you actually a very long time series of
• 21:31
  embedding that also open
• 21:33
  SPEAKER_02: the doors for all the other tasks.
• 21:36
  especially I think it's for ecological applications and also
• 21:42
  for forest monitoring.
• 21:44
  SPEAKER_02: So yes, this is something I've been doing and also I've been testing.
• 21:49
  SPEAKER_00: Well, thank you so much for sharing that.
• 21:50
  SPEAKER_01: There's also alternative SAR data sources we're looking at,
• 21:56
  SPEAKER_01: which will then increase the temporal resolution.
• 21:58
  SPEAKER_00: Right.
• 21:59
  SPEAKER_00: And so that's not Sentinel-1, that's something else.
• 22:02
  Yeah,
• 22:03
  so we're looking at Pulsar 2 as a possible data source as well
• 22:06
  going in.
• 22:07
  SPEAKER_00: Okay.
• 22:08
  SPEAKER_00: Very exciting.
• 22:09
  SPEAKER_00: Lots of developments happening.
• 22:12
  SPEAKER_00: You've got the paper that's published already.
• 22:15
  SPEAKER_01: That is on archive and it's under peer review at the moment.
• 22:19
  SPEAKER_00: Fantastic.
• 22:20
  SPEAKER_00: So it should be out pretty soon.
• 22:22
  SPEAKER_00: I'll put the links in the show notes.
• 22:23
  Lastly,
• 22:24
  if people want to follow along with the project,
• 22:27
  where should they be watching?
• 22:29
  So you can look at our GitHub,
• 22:30
  the GeoTessera project under the uCameo GitHub project.
• 22:37
  That has the library,
• 22:38
  GeoTessera,
• 22:39
  which should make working with Tessera projects.
• 22:42
  much simpler if you're in the python world and there's bindings
• 22:45
  coming from other
• 22:46
  languages as well um there's other projects going at the moment
• 22:49
  um with some of our
• 22:50
  uh some we've been helped by a lot of organizations amd intel
• 22:54
  the university's own
• 22:55
  computing resource um and uh hopefully there should be now uh
• 22:59
  the ability to get
• 22:59
  SPEAKER_01: the embeddings on s3 soon via aws's open data program okay
• 23:06
  So a few different places,
• 23:07
  GitHub,
• 23:07
  probably the main one for developers,
• 23:09
  et cetera.
• 23:10
  SPEAKER_00: Thank you so much.
• 23:11
  SPEAKER_00: I'll put all that in the show notes.
• 23:13
  SPEAKER_00: Once again, thanks for coming and sharing all this information.
• 23:15
  SPEAKER_00: Really exciting model, really exciting developments in this space.
• 23:19
  SPEAKER_00: And I can't wait to see what comes for version two.
• 23:22
  SPEAKER_02: Thank you, Robin.
• 23:23
  SPEAKER_02: Thanks, Robbie.
• 23:24
  SPEAKER_00: See ya.