Latest / Satellite image deep learning / BetaEarth: Open Embeddings of Sentinel-2 and Sentinel-1 with a Little Help of AlphaEarth

Transcript

BetaEarth: Open Embeddings of Sentinel-2 and Sentinel-1 with a Little Help of AlphaEarth · Satellite image deep learning

• 0:06
  SPEAKER_00: Hi Miko, welcome back to the podcast.
• 0:08
  SPEAKER_01: Hey, it's good to be back.
• 0:10
  SPEAKER_00: Excellent.
• 0:11
  SPEAKER_00: It's been a little while.
• 0:11
  SPEAKER_00: What's the news on your side?
• 0:15
  SPEAKER_01: Plenty of news.
• 0:15
  SPEAKER_01: I'm not sure where to start.
• 0:17
  SPEAKER_01: I know we'll cover a better earth today,
• 0:19
  but there have been plenty of things we've been involved with
• 0:23
  in Adasterisk Labs.
• 0:25
  SPEAKER_01: We are a worker-led cooperative research lab, and we mostly
• 0:29
  look for interesting applications of open data that tells us
• 0:34
  something about the Earth.
• 0:36
  SPEAKER_01: So plenty of things going on.
• 0:37
  SPEAKER_01: I don't want to go into too much detail given the time constraint.
• 0:40
  But betaEarth is one of our organic contributions where I've
• 0:46
  just spun a site
• 0:49
  SPEAKER_01: project for a few days.
• 0:50
  SPEAKER_01: And it turned out to be interesting enough to share with the community.
• 0:54
  SPEAKER_00: Amazing.
• 0:54
  SPEAKER_00: All right.
• 0:55
  SPEAKER_00: And betaEarth, that sounds familiar.
• 0:57
  SPEAKER_00: AlphaEarth, is it related?
• 0:59
  SPEAKER_01: Yes,
• 1:01
  I think at some point there was this opportunity to write a
• 1:08
  short paper for the IRS
• 1:10
  SPEAKER_01: PRS session on embeddings,
• 1:12
  SPEAKER_01: and there was one idea that I had on my mind for a while.
• 1:16
  SPEAKER_01: which was basically related to the Alpha Earth release from last year.
• 1:20
  And I think it was one of the first major embedding product
• 1:27
  releases that we've seen.
• 1:28
  SPEAKER_01: Right now, this is getting more popular, but
• 1:31
  SPEAKER_01: I think it officially,
• 1:33
  SPEAKER_01: well,
• 1:33
  maybe semi-officially launched during Living Planet Symposium
• 1:37
  in June at the ESA
• 1:39
  SPEAKER_01: event.
• 1:40
  And it was quite interesting because while being the first
• 1:45
  major embedding product
• 1:47
  SPEAKER_01: release,
• 1:47
  it was also the first one where we've seen a closed model and
• 1:51
  an open embedding
• 1:52
  SPEAKER_01: product.
• 1:53
  SPEAKER_01: And that was quite interesting to me because
• 1:56
  It seems like an obvious thing to do as well in the sense that
• 2:01
  you can protect your
• 2:02
  SPEAKER_01: IP of your model.
• 2:03
  SPEAKER_01: You have full control of who runs the model and where and how.
• 2:08
  SPEAKER_01: And at the same time,
• 2:08
  SPEAKER_01: you can provide a global and a huge product to the community for free,
• 2:15
  SPEAKER_01: essentially.
• 2:17
  What was quite interesting to me is apart from being a really
• 2:20
  capable model,
• 2:22
  SPEAKER_01: it was interesting to me to think about how
• 2:25
  SPEAKER_01: safe this is.
• 2:27
  SPEAKER_01: And obviously,
• 2:28
  whenever we use an AI model and real workflows with Earth
• 2:32
  Observation data,
• 2:34
  we often want to have this access and get an idea of how the
• 2:37
  model behaves rather
• 2:38
  SPEAKER_01: than just the end product.
• 2:40
  So it immediately was interesting to me to think about how much
• 2:44
  you can actually
• 2:44
  SPEAKER_01: retrieve,
• 2:46
  how much you can actually emulate the original model's behavior
• 2:49
  from its embedding
• 2:50
  SPEAKER_01: outputs alone.
• 2:52
  SPEAKER_00: Right.
• 2:53
  SPEAKER_00: Well, that's the motivation.
• 2:55
  SPEAKER_00: What's the results look like?
• 2:58
  OK,
• 2:58
  so I guess I can share the screen and just give you a quick run
• 3:02
  through the repo.
• 3:05
  SPEAKER_01: And at the same time,
• 3:06
  I don't know if we want to explain what AlphaRF does for people
• 3:11
  who might not be
• 3:11
  SPEAKER_01: familiar.
• 3:12
  SPEAKER_01: Yeah, let's do that.
• 3:15
  SPEAKER_01: Let me just share the screen.
• 3:19
  Alpha Earth takes in many inputs,
• 3:21
  probably more than any other model I've seen.
• 3:24
  SPEAKER_01: Yes, it's a huge product.
• 3:27
  SPEAKER_01: It's probably a fairly complex model, unlike Beta Earth.
• 3:34
  SPEAKER_01: It mostly relies,
• 3:35
  SPEAKER_01: interestingly,
• 3:36
  SPEAKER_01: on public free data,
• 3:40
  SPEAKER_01: essentially,
• 3:41
  SPEAKER_01: from Sentinel-2,
• 3:42
  SPEAKER_01: Sentinel-1,
• 3:43
  SPEAKER_01: Landsat.
• 3:44
  SPEAKER_01: and a bunch of other supporting sensors.
• 3:47
  And it's quite interesting because it uses some of these
• 3:52
  sensors as targets or some
• 3:54
  SPEAKER_01: of the products as targets,
• 3:55
  SPEAKER_01: some of them as inputs,
• 3:57
  SPEAKER_01: so mostly Sentinel-2,
• 3:58
  SPEAKER_01: Sentinel-1,
• 3:58
  SPEAKER_01: and Landsat.
• 3:59
  SPEAKER_01: And some of them it uses as both inputs and targets.
• 4:02
  SPEAKER_01: So it's sort of like a hybrid workflow.
• 4:07
  And what's also interesting is that it's a fairly complex model
• 4:09
  that has these
• 4:11
  really interesting modules that take care of the temporal
• 4:17
  understanding of these
• 4:18
  SPEAKER_01: sequences.
• 4:18
  So in principle,
• 4:19
  you can create an embedded product that describes any range of
• 4:23
  time from
• 4:26
  SPEAKER_01: I would assume any subset of the modalities.
• 4:29
  So you can basically say that you want descriptor for this sort
• 4:31
  of time range,
• 4:33
  SPEAKER_01: and these are the sources that you have available.
• 4:35
  And in the case of the embedding product that everyone has
• 4:38
  access to,
• 4:38
  it's an annual product.
• 4:40
  So they try to describe a full year of observations with the
• 4:43
  data that exists there.
• 4:45
  If I scroll down to the very bottom,
• 4:47
  there should be,
• 4:48
  or maybe not the bottom,
• 4:49
  somewhere here,
• 4:51
  SPEAKER_01: There should be another, I don't know if you can see.
• 4:54
  SPEAKER_01: You can go to the original product and access it for free.
• 4:57
  There is also a mirror on source co-op where you can access
• 5:02
  this data without
• 5:03
  SPEAKER_01: Google Earth Engine as well.
• 5:05
  SPEAKER_01: But in any case,
• 5:06
  SPEAKER_01: we only have access to the annual product,
• 5:08
  SPEAKER_01: so that obviously limits what we can emulate.
• 5:10
  We can only emulate the annual product,
• 5:12
  really,
• 5:13
  because that's what we have access to.
• 5:14
  SPEAKER_01: But in principle,
• 5:16
  SPEAKER_01: Alpha Earth could produce higher granularity,
• 5:19
  SPEAKER_01: and a lot of users were interested in that.
• 5:22
  SPEAKER_01: And at the same time,
• 5:22
  SPEAKER_01: there's also a discussion about how much sense it makes to go into final
• 5:27
  granularity because very often the annual product is supposed
• 5:32
  to encode all of this
• 5:33
  SPEAKER_01: information from a full year.
• 5:34
  SPEAKER_01: So in principle,
• 5:36
  SPEAKER_01: even if you're interested in a specific month of the year,
• 5:38
  SPEAKER_01: in a specific dynamic,
• 5:40
  SPEAKER_01: it should
• 5:42
  it would ideally be also covered by the annual product and you
• 5:44
  have multiple annual
• 5:45
  products so in principle we have access to everything um so i
• 5:49
  know like the teams
• 5:49
  are like alpha earth and tessera do work on that quite a lot on
• 5:53
  like trying to find
• 5:54
  these right parameters to produce embedded products that make
• 5:57
  it practical and also
• 6:00
  SPEAKER_01: efficient enough to store and distribute.
• 6:02
  SPEAKER_01: Because it's a huge product.
• 6:03
  SPEAKER_01: And there was an interesting discussion as well in the very early days of
• 6:07
  SPEAKER_01: embeddings where they were mostly interpreted as tools of compression.
• 6:12
  SPEAKER_01: And it turns out that if you have,
• 6:14
  SPEAKER_01: for many different,
• 6:15
  SPEAKER_01: for many types of embedding products,
• 6:17
  SPEAKER_01: you actually inflate your data in a way.
• 6:20
  SPEAKER_01: Or at least you create a product that's very, very bulky.
• 6:23
  SPEAKER_01: And that's the case for Altverif as well.
• 6:29
  SPEAKER_00: So I think Alpha Earth is 64 channels,
• 6:33
  SPEAKER_00: which would be more than Sentinel-2 as a global mosaic,
• 6:37
  SPEAKER_00: but possibly better than if you had all these different data sources.
• 6:41
  SPEAKER_01: Exactly.
• 6:42
  SPEAKER_01: It's supposed to compress multiple sources.
• 6:44
  So I haven't seen much work into that,
• 6:47
  but it would be interesting to also
• 6:49
  SPEAKER_01: explore this compression angle.
• 6:52
  SPEAKER_01: Because you are compressing multiple temporal samples.
• 6:55
  SPEAKER_01: So it's 64 channels,
• 6:59
  but it's derived from probably quite a few Sentinel-2
• 7:02
  observations Sentinel-1.
• 7:04
  SPEAKER_01: I mean, it's up to the designer how much goes into that.
• 7:08
  SPEAKER_00: Yeah.
• 7:08
  SPEAKER_00: And it's not just compression,
• 7:09
  because it should be somehow more meaningful than the
• 7:13
  combination of the inputs.
• 7:15
  SPEAKER_01: Yeah, exactly.
• 7:15
  SPEAKER_01: So I mean, that's another aspect that
• 7:19
  SPEAKER_01: depending on the model architecture,
• 7:22
  unless your model workflow is operating exclusively in pixel
• 7:29
  domain where you just
• 7:29
  process temporal sequences of pixels and you never actually use
• 7:35
  any of the
• 7:36
  SPEAKER_01: neighborhood context,
• 7:38
  SPEAKER_01: unless you do that,
• 7:38
  you actually have a lot of additional information flow that
• 7:41
  goes into every single
• 7:43
  SPEAKER_01: pixel,
• 7:44
  SPEAKER_01: embedding pixel,
• 7:45
  SPEAKER_01: from its neighbors.
• 7:46
  SPEAKER_01: So it could be an additional context that tells you that
• 7:49
  SPEAKER_01: hey,
• 7:49
  SPEAKER_01: this is an agricultural field,
• 7:51
  SPEAKER_01: but it also happens to be quite close to,
• 7:54
  SPEAKER_01: I don't know,
• 7:54
  SPEAKER_01: this type of infrastructure or whatever.
• 7:56
  SPEAKER_01: So it's already determined by the architecture that you use.
• 8:01
  And in case of the beta earth,
• 8:04
  we do use a segformer and we also use dyno backbone.
• 8:09
  SPEAKER_01: So actually this sort of neighborhood context can be leveraged as well.
• 8:15
  SPEAKER_00: OK, that's a good place to start.
• 8:16
  So the architecture you mentioned,
• 8:17
  it's basically a segmentation approach.
• 8:21
  SPEAKER_01: And again, this was mostly an experimental workflow.
• 8:24
  SPEAKER_01: So if I just find the right place here,
• 8:26
  there's like eight models that I've released with various
• 8:33
  training strategies,
• 8:34
  SPEAKER_01: with various architectures.
• 8:36
  SPEAKER_01: The Dino is an interesting experiment because in principle,
• 8:40
  some of the experiments I've done for Beta Earth were mostly
• 8:42
  exploring how much,
• 8:44
  SPEAKER_01: how cheap you can make this,
• 8:46
  SPEAKER_01: how close you can replicate this without changing models.
• 8:50
  SPEAKER_01: much of some pre-training weights.
• 8:52
  SPEAKER_01: And dyno is an interesting backbone.
• 8:55
  SPEAKER_01: So we've used that, but it's only with three-channel input.
• 8:59
  SPEAKER_01: So what we can do is actually create these primitives,
• 9:02
  SPEAKER_01: three-channel primitives,
• 9:03
  SPEAKER_01: not only for the RGB,
• 9:04
  SPEAKER_01: but also for infrared bands and so on and so on.
• 9:08
  And surprisingly,
• 9:09
  you can get fairly comparable cosine similarity with these
• 9:14
  primitives.
• 9:16
  SPEAKER_01: Same with LandUseLandCover.
• 9:18
  SPEAKER_01: And then there's an even simpler version,
• 9:21
  SPEAKER_01: and that's the segformer model,
• 9:23
  SPEAKER_01: where I only use Sentinel-2 RGB bands,
• 9:26
  SPEAKER_01: and we still get about 0.83 cosine similarity,
• 9:30
  SPEAKER_01: and fairly OK.
• 9:31
  land cover not amazing and none of the models are as good as
• 9:34
  alpha earth is so
• 9:35
  that's like that's one disclaimer it's a it's a fairly
• 9:37
  lightweight model that just
• 9:38
  allows you to play with these embeddings more but they are not
• 9:41
  the same quality as
• 9:43
  alpha earth but they can be very cheap to run and again like an
• 9:46
  rgb is an amazing
• 9:47
  example we just use optical rgb to approximate a lot of the
• 9:50
  information in the
• 9:51
  signal which i think that a lot of people on linkedin have
• 9:54
  already like asked about
• 9:56
  SPEAKER_01: that and now i think like
• 9:59
  There's probably an interesting finding here where a lot of the
• 10:02
  information in
• 10:03
  SPEAKER_01: Alpha Earth,
• 10:04
  SPEAKER_01: despite this being a temporal descriptor,
• 10:06
  SPEAKER_01: is actually static.
• 10:07
  So it does contain a lot of time-agnostic information about a
• 10:14
  given region that
• 10:15
  SPEAKER_01: just tells you the overall context,
• 10:17
  SPEAKER_01: the type of land cover,
• 10:18
  SPEAKER_01: and so on.
• 10:19
  So that's why I would assume that we can get fairly high
• 10:23
  performance from just one
• 10:25
  SPEAKER_01: non-temporal sample in just RGB channels.
• 10:28
  Yeah,
• 10:29
  so do you think LULC is maybe just one example you might want
• 10:32
  to compare against?
• 10:34
  SPEAKER_01: Yeah, it's not a particularly challenging benchmark.
• 10:37
  SPEAKER_01: And again, this goes back to this being a side project.
• 10:40
  So I'd be keen to know what kind of performance people manage
• 10:44
  to get with these encoders.
• 10:48
  SPEAKER_01: I would imagine they won't be as high quality as Alpha Earth.
• 10:54
  SPEAKER_01: But I think they also can be
• 10:58
  SPEAKER_01: quite agnostic.
• 11:02
  They could be still quite useful embeddings or encodings of the
• 11:08
  observational data.
• 11:09
  And the nice thing about it is that you have full control of
• 11:12
  which samples go into
• 11:13
  SPEAKER_01: your embedding.
• 11:14
  SPEAKER_01: So when we run the demo, we can actually specify
• 11:18
  SPEAKER_01: the time range, the sensors, and how basically they're weighted down.
• 11:25
  SPEAKER_01: The core model is monotemporal,
• 11:27
  SPEAKER_01: so you just specify which day of year your samples are from,
• 11:32
  SPEAKER_01: you pass them through it,
• 11:33
  SPEAKER_01: and you have some approximation of the annual product,
• 11:36
  SPEAKER_01: and then you can do it for multiple
• 11:38
  time stamps and average in any way you like,
• 11:41
  which also gives you some interesting...
• 11:42
  SPEAKER_01: So,
• 11:43
  SPEAKER_01: like,
• 11:43
  most of the experiments I've done weren't really with very
• 11:46
  specific downstream
• 11:47
  SPEAKER_01: tasks,
• 11:47
  but instead they were focused on cosine similarity to test
• 11:52
  different strategies for
• 11:53
  merging these embeddings from different times of the year and
• 11:56
  from different
• 11:57
  SPEAKER_01: sensors.
• 11:57
  SPEAKER_01: Like, for instance,
• 11:59
  Sentinel-2 will always give you the highest performance and the
• 12:01
  best approximation
• 12:02
  SPEAKER_01: compared to,
• 12:03
  SPEAKER_01: for instance,
• 12:04
  SPEAKER_01: Sentinel-1.
• 12:05
  SPEAKER_01: So sometimes the question is like,
• 12:06
  does it even make sense to add Sentinel-1 if you only care
• 12:09
  about cosine similarity?
• 12:11
  SPEAKER_01: And that will also be task specific.
• 12:12
  So there's like plenty of experimental work you can do there to
• 12:16
  kind of see how
• 12:17
  SPEAKER_01: different modalities and timestamps,
• 12:19
  SPEAKER_01: depending on location,
• 12:20
  SPEAKER_01: affect your final embedding and ultimately the performance on your task.
• 12:25
  SPEAKER_00: Right,
• 12:25
  so the objective is to put these models out there and then see
• 12:28
  how people can
• 12:29
  SPEAKER_00: evaluate them for different kind of tasks.
• 12:32
  SPEAKER_01: Yeah, so I have other things going on, obviously.
• 12:36
  SPEAKER_01: So this is a side project, so I will probably test them myself.
• 12:40
  SPEAKER_01: Currently, I'm mostly interested in
• 12:43
  and trying to link existing embeddings or existing latent
• 12:49
  spaces of the models that
• 12:51
  SPEAKER_01: we have with representations that are slightly easier to control,
• 12:56
  SPEAKER_01: such as text,
• 12:57
  SPEAKER_01: such as multilingual text.
• 12:58
  SPEAKER_01: And hopefully,
• 13:00
  SPEAKER_01: in the next few months,
• 13:00
  SPEAKER_01: we'll see another release that will actually be,
• 13:04
  SPEAKER_01: I think,
• 13:04
  SPEAKER_01: more interesting than BetaEarth.
• 13:05
  But BetaEarth is probably something I will also test with these
• 13:09
  specific bridges
• 13:11
  SPEAKER_00: Fantastic, very exciting.
• 13:13
  SPEAKER_00: So that's the models.
• 13:16
  SPEAKER_01: I can maybe show you the demo then just to,
• 13:19
  SPEAKER_01: so like if you go to the GitHub repo,
• 13:21
  you will find links to all of the models and there will be
• 13:26
  different ways of
• 13:27
  SPEAKER_01: running the models.
• 13:29
  SPEAKER_01: So you can use the pip install, run it locally.
• 13:33
  You can even instantiate your own streamlet app the same way I
• 13:37
  did for the demo,
• 13:39
  SPEAKER_01: but you will see in a minute.
• 13:41
  SPEAKER_01: You can use the existing call-up notebooks.
• 13:43
  You can load it from,
• 13:44
  like you can,
• 13:45
  the first one is for a monotemporal example with Major Tom.
• 13:48
  And the second one allows you to do anything you like over any
• 13:50
  spatial temporal context.
• 13:52
  So you can,
• 13:52
  there's many ways to get started,
• 13:54
  but I will just click on the demo.
• 13:58
  SPEAKER_01: So this is the demo on Hugging Face.
• 14:00
  SPEAKER_01: And just for the sake of time,
• 14:02
  I will use the local version of the same demo just so it runs a
• 14:08
  little bit faster.
• 14:09
  SPEAKER_00: All right,
• 14:09
  SPEAKER_00: so it's a pretty,
• 14:11
  SPEAKER_00: I guess,
• 14:11
  standard sort of map-based interface where you can draw the
• 14:14
  region you care about
• 14:15
  and select some other parameters and then hit a button and in
• 14:18
  the background the
• 14:19
  SPEAKER_00: model is going to generate some embeddings.
• 14:21
  SPEAKER_01: Exactly.
• 14:22
  SPEAKER_01: And I've probably spent more time making this demo,
• 14:24
  SPEAKER_01: like,
• 14:24
  SPEAKER_01: same amount of time as I have,
• 14:26
  SPEAKER_01: like,
• 14:26
  SPEAKER_01: writing the actual model.
• 14:27
  SPEAKER_01: So, like, both models and the app were, like, super, super...
• 14:31
  SPEAKER_01: fast to do.
• 14:34
  SPEAKER_01: And I guess,
• 14:34
  SPEAKER_01: like,
• 14:35
  SPEAKER_01: it's less,
• 14:35
  SPEAKER_01: the app itself is less trivial than the model because I mean,
• 14:40
  Gradio is probably the most common way of producing these
• 14:43
  hugging face demos,
• 14:44
  SPEAKER_01: but it doesn't work that good with maps.
• 14:47
  SPEAKER_01: You can use,
• 14:47
  SPEAKER_01: like,
• 14:48
  SPEAKER_01: a folium extension for Gradio,
• 14:50
  SPEAKER_01: but it doesn't work very well with accepting the input.
• 14:53
  SPEAKER_01: So, in this case, we're using a streamlet version.
• 14:57
  SPEAKER_01: And yeah, it works.
• 14:58
  SPEAKER_01: So, I can, I don't know.
• 15:00
  SPEAKER_01: I can maybe select some footprint
• 15:05
  SPEAKER_01: And we have pretty much full control here.
• 15:09
  SPEAKER_01: So we can use an annual product, which obviously takes more time.
• 15:14
  SPEAKER_01: Or we can use a custom date.
• 15:15
  SPEAKER_01: So let's use one month of July in 2023.
• 15:21
  SPEAKER_01: We can save per time stamp embedding.
• 15:23
  SPEAKER_01: So in case you want to limit the memory usage of your app,
• 15:27
  SPEAKER_01: you can switch that off if you just want the final embedding.
• 15:30
  But I'm going to switch it on just so we can see how these
• 15:32
  different components
• 15:33
  SPEAKER_01: actually contribute to the final descriptor.
• 15:36
  SPEAKER_01: And we can also save that prior timestamp input.
• 15:39
  It also saves a manifest JSON that tells you which products
• 15:42
  went into this
• 15:43
  SPEAKER_01: embedding,
• 15:43
  which is another nice thing to have when you actually work with
• 15:47
  these products.
• 15:48
  And that's one thing that's maybe missing that we'll hopefully
• 15:51
  see more in the
• 15:52
  future when you actually know how a specific embedding product
• 15:55
  was created.
• 15:56
  SPEAKER_01: So let's generate it.
• 15:58
  SPEAKER_01: It first loads a model,
• 16:00
  SPEAKER_01: it downloads the DEM,
• 16:01
  SPEAKER_01: and then from Planetary Computer,
• 16:03
  SPEAKER_01: it will download Sentinel-2 and Sentinel-1 data,
• 16:05
  SPEAKER_01: I think.
• 16:09
  SPEAKER_00: It's only one month, so it's pretty good.
• 16:13
  Yes,
• 16:14
  and pretty much you can connect it to any source of Sentinel-2
• 16:17
  or Sentinel-1 data.
• 16:20
  SPEAKER_01: These are the two modalities that I just found most universe,
• 16:24
  SPEAKER_01: like most versatile,
• 16:27
  SPEAKER_01: and yeah,
• 16:28
  SPEAKER_01: it's...
• 16:29
  SPEAKER_01: Here it is.
• 16:30
  SPEAKER_01: So right now we see the 2023 average from all products,
• 16:35
  but we can also observe individual products from a specific
• 16:38
  date and also the
• 16:40
  SPEAKER_01: products that went into them.
• 16:41
  SPEAKER_01: In this case, we have a lot of cloud coverage from Sentinel Tools.
• 16:44
  SPEAKER_01: This makes it an interesting example because it's Sentinel.
• 16:48
  SPEAKER_01: You can see that the Sentinel one product is much cleaner.
• 16:51
  It produces a less affected embedding where you can see how
• 16:55
  cloud affected the embedding here.
• 16:59
  SPEAKER_01: And the final output,
• 17:00
  SPEAKER_01: once you average it out,
• 17:01
  you actually get something that's quite similar to the original
• 17:03
  Sentinel one
• 17:04
  SPEAKER_01: product just because that's the more reliable source of data.
• 17:07
  SPEAKER_01: You can play with a macro cloud cover if you like.
• 17:09
  SPEAKER_01: Obviously in Scotland,
• 17:12
  SPEAKER_01: bring variable results, but you can always rely on Sentinel-1.
• 17:16
  SPEAKER_01: But then in general,
• 17:17
  SPEAKER_01: like for many regions,
• 17:18
  SPEAKER_01: Sentinel-1 will give you a worse approximation of the annual embedding.
• 17:24
  SPEAKER_01: So there's that.
• 17:25
  SPEAKER_01: So you generally want,
• 17:27
  SPEAKER_01: I think,
• 17:27
  to have a healthy amount of Sentinel-2 data if you want to
• 17:30
  approximate Alpha Earth.
• 17:32
  SPEAKER_00: Interesting.
• 17:33
  So it's basically got more information,
• 17:35
  Sentinel-2,
• 17:36
  as far as Alpha Earth is concerned.
• 17:37
  SPEAKER_01: Mm-hmm.
• 17:38
  SPEAKER_01: And obviously,
• 17:39
  a large portion of the pipeline is also figuring out how to
• 17:43
  deal with maybe
• 17:46
  SPEAKER_01: non-standard footprints.
• 17:49
  SPEAKER_01: Most of the models will have a limited receptive field.
• 17:53
  By the way,
• 17:53
  you can download this as zip so you have access to all the
• 17:56
  information.
• 17:56
  SPEAKER_01: But I'm going to rerun for this on this weird footprint.
• 18:01
  But a lot of this comes down to how expensive it is to do some
• 18:05
  overlap and to merge
• 18:06
  SPEAKER_01: embeddings from different sources and different crops.
• 18:09
  SPEAKER_00: Question on the detail there,
• 18:11
  because I've seen a few examples of embeddings over large areas
• 18:14
  where you can see
• 18:15
  SPEAKER_00: this kind of tiling artifact.
• 18:17
  SPEAKER_00: So how do you handle that with embeddings versus just RGB?
• 18:21
  SPEAKER_01: It's a good question.
• 18:22
  SPEAKER_01: I think this is maybe a slightly under,
• 18:26
  SPEAKER_01: at least in the space of open source,
• 18:28
  SPEAKER_01: it's a slightly under-explored aspect.
• 18:32
  SPEAKER_01: Because effectively what you do is you apply some type of like top result
• 18:36
  linear merging and you you merge well i'm not an expert in
• 18:39
  creating mosaics to be
• 18:40
  fair so i don't know what's the state of the art when it comes
• 18:43
  to mixing rgb data
• 18:45
  but in terms of embeddings you often just like average and do
• 18:48
  some like smoothing
• 18:49
  between different crops and i think that's about it it would be
• 18:52
  really again it's
• 18:53
  an interesting it's an interesting direction for the future is
• 18:55
  to like find like
• 18:57
  better ways and smarter ways to mix data from from different
• 19:00
  crops yeah because it
• 19:03
  the difference is that one patch has a limited context you move
• 19:06
  along that has
• 19:07
  different context that affects the embeddings yeah and
• 19:11
  sometimes it's it's it's
• 19:12
  really tricky like the simple simple uh simple mixing like
• 19:17
  linear mixtures are
• 19:19
  often suboptimal um but again it goes back to to what you
• 19:24
  really want to do with
• 19:24
  SPEAKER_01: the embeddings because
• 19:26
  In order for an embedding product to be useful,
• 19:28
  it doesn't necessarily have to look great.
• 19:31
  We like looking at PCAs because it looks fun,
• 19:33
  but PCAs are not really useful on their own.
• 19:38
  It only gets useful once you connect to another model and
• 19:40
  perform your downstream task.
• 19:42
  So the chances are that the models can still be fine with these
• 19:47
  artifacts.
• 19:48
  SPEAKER_01: Especially if you train on artifacts,
• 19:51
  SPEAKER_01: you might be able to get some immunity against this effect.
• 19:55
  SPEAKER_01: Definitely.
• 19:56
  SPEAKER_00: Looks like it's generated something over the water there as well.
• 19:59
  SPEAKER_00: What's that?
• 20:00
  SPEAKER_00: Sorry?
• 20:01
  SPEAKER_00: Looking at that latest embedding, what am I saying?
• 20:03
  SPEAKER_00: Some details over the water itself?
• 20:06
  SPEAKER_01: Yeah.
• 20:06
  SPEAKER_01: I mean, I would expect there to be some detail.
• 20:09
  I don't know how useful these details are,
• 20:11
  but it will usually produce something like that.
• 20:15
  SPEAKER_01: Obviously, you get a lot of speckle in Sentinel-1.
• 20:18
  SPEAKER_01: Again, an interesting...
• 20:21
  direction would be to look into how,
• 20:23
  like,
• 20:23
  Sentinel-1 behaves and how it affects embeddings.
• 20:25
  Because,
• 20:25
  again,
• 20:25
  like,
• 20:25
  what I found is,
• 20:26
  like,
• 20:27
  GEM probably tells you the least.
• 20:29
  SPEAKER_01: Obviously, it just tells you the terrain.
• 20:31
  So it's just,
• 20:32
  like,
• 20:33
  it gets you to,
• 20:34
  let me see,
• 20:36
  it gets you to,
• 20:36
  like,
• 20:36
  some interesting,
• 20:39
  SPEAKER_01: 0.541 cosine similarity.
• 20:42
  SPEAKER_01: So not trivial either.
• 20:43
  You can see that a large amount of information comes from the
• 20:46
  shape of the terrain itself.
• 20:48
  SPEAKER_01: But then Sentinel-1 is 0.71, for instance.
• 20:51
  SPEAKER_01: And then you can get as high as 80s when you add Sentinel-2.
• 20:59
  SPEAKER_01: And then finally, when you combine all of them, it's on average is 872.
• 21:02
  SPEAKER_01: But what's interesting as well is this dynamic changes.
• 21:07
  modalities are useful to combine depends on where you are and
• 21:12
  also what exact
• 21:14
  SPEAKER_01: effect you want to model.
• 21:16
  SPEAKER_01: So again,
• 21:17
  SPEAKER_01: I'm not sure how performant this model is,
• 21:20
  SPEAKER_01: but someone can always reuse the same code and train a better model.
• 21:25
  But what it does allow us to experiment with these sort of flow
• 21:28
  dynamics from
• 21:29
  SPEAKER_01: different modalities and sampling and so on,
• 21:31
  SPEAKER_01: because a lot of this could really
• 21:35
  The optimal configuration could really depend on your task and
• 21:37
  spatiotemporal
• 21:39
  SPEAKER_01: context of interest.
• 21:41
  SPEAKER_00: Absolutely.
• 21:41
  I think it's really opening the Pandora's box of opportunities
• 21:46
  for experimentation.
• 21:47
  I can see that quite a few masters and PhD students are going
• 21:51
  to be kept busy.
• 21:54
  SPEAKER_01: There are quite a few groups, really amazing work.
• 21:58
  SPEAKER_01: There's definitely,
• 21:58
  SPEAKER_01: like,
• 21:58
  SPEAKER_01: some papers that actually measure,
• 22:01
  try to approximate the information content depending on where
• 22:04
  you are in the
• 22:05
  SPEAKER_01: embedding itself.
• 22:07
  SPEAKER_01: So I think,
• 22:07
  SPEAKER_01: like,
• 22:07
  SPEAKER_01: there's,
• 22:08
  SPEAKER_01: just like with any,
• 22:09
  SPEAKER_01: like,
• 22:09
  SPEAKER_01: deep,
• 22:10
  SPEAKER_01: like,
• 22:11
  SPEAKER_01: multilayer model,
• 22:12
  SPEAKER_01: there's a lot to be discovered once you start really probing things.
• 22:15
  SPEAKER_01: And, like, this is, I think, what we need in the community.
• 22:18
  SPEAKER_01: Like, thankfully, we have
• 22:19
  SPEAKER_01: other models like Tessera, like OlmoEarth that we have more access to.
• 22:23
  SPEAKER_01: So we can do a lot of that work.
• 22:25
  BetaEarth is more like an experiment that tells you how easy it
• 22:29
  is to maybe
• 22:29
  SPEAKER_01: reconstruct these models from embeddings alone.
• 22:33
  And I also think that potentially embeddings could be an
• 22:36
  interesting additional
• 22:37
  SPEAKER_01: regularization in your training regime.
• 22:39
  SPEAKER_01: So when you don't have that much data,
• 22:42
  SPEAKER_01: you can also use this as a model,
• 22:44
  SPEAKER_01: as a signal that you want to approximate.
• 22:45
  SPEAKER_01: Or you can reuse BetaEarth
• 22:49
  pre-trained weights as well if you want something that's quite
• 22:52
  agnostic and also
• 22:52
  cheap to do or maybe you have a different architecture that you
• 22:56
  want to use not
• 22:58
  that that is not covered uh within within this github repo so
• 23:01
  you just take that
• 23:02
  architecture take alpha rf embeddings and you do some
• 23:04
  pre-training on that and then
• 23:05
  maybe you you have some specific task in mind that you later
• 23:08
  want to find you know
• 23:09
  yeah and just a really i guess niche point but what's what's
• 23:13
  the licensing
• 23:15
  SPEAKER_00: situation around using embeddings in that way
• 23:18
  SPEAKER_01: So I think it really depends on who releases the embeddings and how.
• 23:22
  SPEAKER_01: In the case of Google embeddings, it's a Creative Commons license.
• 23:29
  SPEAKER_01: Yeah.
• 23:30
  And technically,
• 23:31
  you have to use this specific attribution text when you
• 23:38
  re-release embeddings.
• 23:39
  SPEAKER_01: In our case, we are not re-releasing embeddings.
• 23:42
  SPEAKER_01: We are releasing a model that approximates these embeddings
• 23:46
  SPEAKER_01: But still,
• 23:47
  SPEAKER_01: I've created the links to the original data set,
• 23:50
  SPEAKER_01: and I've added the attribution at the very end.
• 23:52
  SPEAKER_01: But it's another interesting point.
• 23:54
  SPEAKER_01: And I mean,
• 23:55
  SPEAKER_01: obviously,
• 23:55
  like with AI and with models that regenerate some specific data
• 23:59
  artifacts,
• 23:59
  SPEAKER_01: there is an interesting discussion
• 24:01
  SPEAKER_01: about how they link to their training data.
• 24:06
  SPEAKER_01: And obviously there's a lot of unresolved issues,
• 24:09
  especially in the computer vision community or like creative
• 24:12
  arts community in the
• 24:13
  SPEAKER_01: mainstream AI that,
• 24:14
  SPEAKER_01: I mean,
• 24:15
  SPEAKER_01: there's still a lot
• 24:16
  SPEAKER_01: a lot of gray area and there's a lot of,
• 24:19
  SPEAKER_01: obviously,
• 24:20
  even violation where you release a model even though the
• 24:23
  training data didn't
• 24:24
  SPEAKER_01: really necessarily allow for this type of output.
• 24:29
  SPEAKER_01: So maybe this is, I mean,
• 24:31
  SPEAKER_01: I'm really not sure where this leads in the error observation community.
• 24:36
  SPEAKER_01: I think it will be slightly different and maybe slightly less sensitive.
• 24:39
  I think it gets very sensitive when you have artists produce
• 24:43
  artworks and then be
• 24:45
  SPEAKER_01: completely detached from the product.
• 24:48
  SPEAKER_01: And in this case,
• 24:49
  we have an open source solution built on top of another open
• 24:52
  source solution
• 24:53
  SPEAKER_01: effectively in case of the training data.
• 24:57
  SPEAKER_00: very interesting topic in itself.
• 24:58
  SPEAKER_00: Well, thanks so much for the overview and the demo.
• 25:01
  SPEAKER_00: I just wanted to finish by asking you why you chose the carrot.
• 25:05
  I was just looking for something I could represent with an
• 25:10
  emoji and something that
• 25:12
  SPEAKER_01: is easy to remember.
• 25:13
  SPEAKER_01: And Beta Earth is because of Alpha Earth and then
• 25:19
  And the only thing I could really think about is beta-carotene
• 25:22
  for the emoji association.
• 25:27
  SPEAKER_01: So I chose a carrot.
• 25:28
  There was a lot of internal discussions,
• 25:30
  because at some point the carrot had a face.
• 25:32
  SPEAKER_01: And it was a very controversial topic, an asterisk.
• 25:36
  And right now,
• 25:37
  we have this faceless carrot that I hope will be good enough.
• 25:41
  SPEAKER_00: I don't know.
• 25:41
  SPEAKER_00: I can see a mouth and a couple of eyes and that.
• 25:45
  SPEAKER_00: Maybe.
• 25:47
  SPEAKER_00: Wonderful.
• 25:47
  SPEAKER_00: Well, thanks again.
• 25:48
  Looking forward to seeing the impact in the community and
• 25:52
  seeing how people pick this up.
• 25:54
  SPEAKER_00: And as you said, you have a few ideas how it could be taken forward.
• 25:58
  SPEAKER_00: So let's see what comes along.
• 26:00
  SPEAKER_00: I'll put the link to the repository in the show notes.
• 26:03
  But once again,
• 26:03
  thanks for joining me and look forward to hearing more in the
• 26:06
  future.
• 26:07
  SPEAKER_01: Perfect.
• 26:08
  SPEAKER_01: Thank you for having me.
• 26:09
  SPEAKER_01: Cheers.
• 26:10
  SPEAKER_01: Cheers.