Latest / Satellite image deep learning / Major TOM: Expandable EO Datasets

Transcript

Major TOM: Expandable EO Datasets · Satellite image deep learning

• 0:05
  SPEAKER_01: Hello and welcome to the satellite image deep learning podcast.
• 0:08
  SPEAKER_01: I'm Robin Cole and it's my pleasure to present another technically focused episode in the series.
• 0:13
  SPEAKER_01: In this episode, I caught up with Alastair and Miko to learn about Major TOM, which is a significant new public dataset of Sentinel-2 imagery.
• 0:21
  SPEAKER_01: Noteworthy for its immense size of 45 terabytes, Major TOM also introduces a set of standards for dataset filtering and integration with other datasets.
• 0:31
  SPEAKER_01: Their aim in releasing this dataset is to foster a community-centered ecosystem of datasets open to bias evaluation and adaptable to new domains and sensors.
• 0:40
  SPEAKER_01: The potential of Major Tom to spur innovation in our field is truly exciting, and I hope you enjoy this episode.
• 0:45
  SPEAKER_01: Hi, guys.
• 0:47
  SPEAKER_01: Welcome to the podcast.
• 0:48
  SPEAKER_01: How are you?
• 0:48
  SPEAKER_02: Hey, good to see you.
• 0:51
  SPEAKER_02: Very well.
• 0:53
  SPEAKER_01: All right.
• 0:53
  SPEAKER_01: So really excited to hear about what you've been up to.
• 0:55
  SPEAKER_01: Why don't we begin with just a quick intro to where you are and what you're doing?
• 0:59
  SPEAKER_01: Over to you, Alison.
• 1:02
  SPEAKER_02: Yeah, so Miko and I are both research fellows at the Phi Lab in Esrin in Italy and we're sort of working on a bunch of different projects related to representation learning across big data, kind of data-centric ML, some generative stuff on Miko's side as well and we're kind of
• 1:27
  SPEAKER_02: Yeah, we were just really motivated basically to help all of our projects and the projects of our colleagues at the Phelab and also hopefully the projects of others outside the Phelab with some sort of high-quality, you know, just baseline datasets that people can plug into for kind of big data applications with EO.
• 1:46
  SPEAKER_01: Right, and so what's this new dataset that you're announcing?
• 1:51
  SPEAKER_02: Major TOM stands for Terrestrial Observation Metaset.
• 1:56
  SPEAKER_02: It's both a dataset that we're going to talk about today, that we've released today, but it's also a wider project that we hope to build with the community.
• 2:06
  SPEAKER_02: Currently, Miko and I have just been working on it, but we're hoping to engage people both through this podcast and through the dataset itself to build a
• 2:15
  SPEAKER_02: whole range of datasets that can be used kind of like building blocks together and that you can piece them together and filter them and combine them to kind of generate multimodal datasets or generate datasets over specific places and times with a sort of rich time series or any other kind of application where you can imagine like combining data across different datasets.
• 2:37
  SPEAKER_02: The dataset that we're sort of releasing today is so-called Major TOM Core Sentinel-2 and we think it's the largest Sentinel-2 dataset
• 2:50
  SPEAKER_02: that's yet been released for ML applications.
• 2:54
  SPEAKER_02: And it's about 50, well, it's about 45, 46 terabytes of data across the two levels covering about half the Earth and pretty much all the land.
• 3:05
  SPEAKER_02: There's a few missed areas where we couldn't get a good scene, et cetera, et cetera, but for the most part, all the land is covered.
• 3:13
  SPEAKER_02: and we have about two and a half trillion pixels in the dataset if you think about it in 10 meters obviously there's all the bands and our kind of design principle was to do as little to the data as possible so the data is in its original pixel values it's not re-sampled it's not re-projected which has pros and cons which maybe we can talk about later but yeah that's the that's the data and it's hosted on Hugging Face which I think is maybe not a first for EO data but maybe
• 3:41
  SPEAKER_02: A bit of a first for a kind of really big EO dataset to be hosted in this way.
• 3:46
  SPEAKER_02: And again, there's pros and cons with that.
• 3:49
  SPEAKER_02: So there's some really, really nice functionality that we can do that we can show you later in terms of streaming data, in terms of sampling it from the cloud, like from the Hugging Phase repo.
• 3:58
  SPEAKER_02: And it also offers like very fast download if you just want to download the whole thing.
• 4:04
  SPEAKER_02: So yeah, that's the kind of broad picture, the broad overview.
• 4:07
  SPEAKER_02: Mika, do you want to add anything?
• 4:11
  SPEAKER_00: No, I think that all makes sense.
• 4:15
  SPEAKER_00: Yeah, we're basically trying to build some more organized effort in building the next generation of EO datasets.
• 4:25
  SPEAKER_00: And we will definitely be following these rules to make sure most of the datasets that we
• 4:33
  SPEAKER_00: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 4:51
  SPEAKER_00: And I think with Core, we really have achieved that you can basically make any subsets across the entire Earth and capture the area of interest or maybe just sample uniformly across the Earth.
• 5:01
  SPEAKER_00: So it's kind of up to the user what they find interesting, whether it's urban areas or something completely, completely different.
• 5:06
  SPEAKER_00: I feel like it's not very flexible to kind of work under the assumptions that the dataset designers always have to make about what areas are being sampled over the others.
• 5:19
  SPEAKER_01: Right so I think that's as you mentioned kind of a I guess limitation of existing data sets they tend to focus on North America or some specific continent and obviously we want models that work over the entire world so employers need a data set of the entire world and I think the really interesting aspect for me is that you mentioned this would be a core data set and it will be able to join on with other data sets what are the considerations to make that possible then I suppose you need geospatial imagery for a start
• 5:47
  SPEAKER_02: Yeah, so we have a kind of few building blocks that I think will also kind of reify a bit more as we see more datasets added to the space because we kind of come up with some specifications and I'm sure the specifications will break in a few datasets when someone comes up with something quite different.
• 6:06
  SPEAKER_02: But for now, the central component of Major TOM is the grid system that we have, which basically is like an index for the samples.
• 6:18
  SPEAKER_02: You know,
• 6:20
  SPEAKER_02: It's not hard to get data over the same place from different instruments if you want to.
• 6:25
  SPEAKER_02: But what's annoying is dealing with different extents and different overlaps and trying to find consistent windows of interest across all of these different scenes and different products and different datasets.
• 6:45
  SPEAKER_02: So what we've tried to do is basically just say, here's a pretty good
• 6:50
  SPEAKER_02: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 7:20
  SPEAKER_02: with a kind of easy to understand indexing system with up and down and left and right from the zero point.
• 7:27
  SPEAKER_02: So with that, what we're just hoping that people do is kind of just follow that system and put their data
• 7:35
  SPEAKER_02: with sort of understanding that if you're putting data there of like a raster image from a satellite, then you try and sort of make sure that it covers most of that area without excessive no data values.
• 7:48
  SPEAKER_02: We have some no data values because obviously sometimes you sample in a place where there is no Sentinel-2 scene that has like full data across the 10 kilometers by 10 kilometers of grid.
• 7:58
  SPEAKER_02: So it's inevitable, but we kind of
• 8:00
  SPEAKER_02: you know best estimate um and alongside that currently we have a kind of metadata specification um that is now probably a bit too specific for like the general purpose thing in the future that we that we envision but we're kind of going to build up the specification as we go and that metadata is you know um the grid cell it's in the time it was taken the product id um
• 8:27
  SPEAKER_02: in the case of Sentinel-2, cloud cover, no data, things like that in a big metadata table that you can download without having to download all the data and it also contains basically the link to the file in which you'll find that data because on Hugging Face we're using so-called Part A file format which is maybe a bit unfamiliar to people in EO and it's maybe a bit of a learning curve but it's basically a column based
• 8:52
  SPEAKER_02: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 9:11
  SPEAKER_02: So maybe this is a bit specific to this dataset and maybe we'll come up with a kind of more general way of of interfacing with with different Major TOM datasets but the kind of conceptual framework is that we're going to be putting it in this grid system the Major TOM grid system and that kind of just ensures that across different datasets if I say grid cell 512 up 32 left
• 9:35
  SPEAKER_02: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 9:59
  SPEAKER_01: You briefly mentioned the Parquet format.
• 10:02
  SPEAKER_01: Obviously, there's geospatial formats for indexing data.
• 10:05
  SPEAKER_01: Stack is an indexing format that's getting a lot of attention.
• 10:09
  SPEAKER_01: What are the considerations?
• 10:13
  SPEAKER_02: So I think that's one other thing I forgot to mention is we are actually looking at sort of stackifying what we've done so far and kind of encouraging others to do that as well.
• 10:25
  SPEAKER_02: But I don't know if you've worked with Stack, but it's maybe a bit easier from the user side than it is from the server side to kind of generate this metadata for all of the different files that you're using.
• 10:35
  SPEAKER_02: So we're kind of working through that at the moment, but it's, I mean,
• 10:39
  SPEAKER_02: It will be done.
• 10:40
  SPEAKER_02: It's not like a pressing concern because I think the really good stuff about stack is going to come when you have this whole ecosystem of different datasets that you can filter and play around with within the stack search.
• 10:53
  SPEAKER_01: And of course, you've got platforms like Planetary Computer, whatever, Stack Index, so you can imagine, say, I've got my XYZ dataset, let's join it with the imagery here, and then we'll do some pre-training on the entire dataset and maybe some fine-tuning over a specific geography, something like that.
• 11:08
  SPEAKER_01: Really interesting stuff.
• 11:09
  SPEAKER_01: Cool.
• 11:10
  SPEAKER_01: I'd love to hear about the practicalities of working with this dataset.
• 11:13
  SPEAKER_01: Obviously, 46 terabytes is kind of a massive dataset.
• 11:17
  SPEAKER_01: Nobody's going to down... Well, unless you've got a huge machine, download it locally.
• 11:20
  SPEAKER_01: How do you anticipate people actually working with this then?
• 11:25
  SPEAKER_02: Yeah, it was hard enough building the thing.
• 11:28
  SPEAKER_02: We annoyed several very, very brilliant and patient engineers that helped us with the kind of compute side of this, and we were constantly running out of space as we went and things, so kudos to the team there.
• 11:45
  SPEAKER_02: But yeah, I mean, the key thing is that
• 11:50
  SPEAKER_02: Well, you know, there's kind of two modes of operation, right?
• 11:53
  SPEAKER_02: You can search and filter the dataset very easily using the kind of metadata table that's only 200 megabytes or something like that big.
• 12:02
  SPEAKER_02: And you could say, right, I'm going to take all the images over France, you get a polygon for France or a bounding box and you quickly search and filter and you say, okay, I don't want cloud cover above
• 12:12
  SPEAKER_02: 10%.
• 12:13
  SPEAKER_02: We never have excessive cloud cover, but we have a bit of cloud sometimes.
• 12:17
  SPEAKER_02: And then you might end up with a few hundred gig dataset and you can use that how you want.
• 12:26
  SPEAKER_02: The other option, of course, is just downloading the whole thing.
• 12:28
  SPEAKER_02: People are going to need disk space if they want to do that, I'm afraid.
• 12:36
  SPEAKER_02: They could try to do some kind of streaming approach to training because Hugging Face does allow that.
• 12:43
  SPEAKER_02: we're kind of doing our own testing at the moment and it seems pretty good for doing it for streaming in order but we're not sure quite yet how we get kind of good performance on like a shuffled streaming so these are kind of questions open questions I suppose but ultimately if you want to use the whole data set you're probably going to have to fork out some terabytes of storage to do so yeah really interesting yeah sorry
• 13:14
  SPEAKER_00: We have some notebooks already set up, so for anyone who's interested in this aspect, there's a call-up notebook where you can actually take this basically for a spin and filter your own version of the dataset, download it and basically immediately incorporate it into your PyTorch framework.
• 13:35
  SPEAKER_00: Yeah, we're still kind of investigating how we can go between, how we can fill the space between like downloading your own small copy
• 13:44
  SPEAKER_00: and downloading the entire copy and obviously streaming can sometimes satisfy this context but we might still need to do a little bit of work in terms of the framework to make sure that it's kind of easy enough to work with and fast enough for actual training of deep neural networks.
• 14:03
  SPEAKER_01: Yeah, you mentioned that the data is on a high-gain base.
• 14:05
  SPEAKER_01: I've not got any personal experience with streaming off that, but obviously the streaming bandwidth is going to influence how effectively you can utilize whatever GPU hardware you've got.
• 14:15
  SPEAKER_01: And potentially there's different considerations, whether you stream it from platform A or move it to platform B and stream from B. Do you have any kind of data on costs and bandwidth available?
• 14:27
  SPEAKER_02: Well, we'll see in the demo, but we found that the download speed from Hugging Face is pretty good.
• 14:36
  SPEAKER_02: It's probably not at the kind of speed you need to create batches and fully maximize a GPU during training.
• 14:43
  SPEAKER_02: You're probably going to have some downtime between batches if you're fully just streaming it.
• 14:49
  SPEAKER_02: That being said, to just navigate and search through the data and filter it, it's pretty responsive.
• 14:57
  SPEAKER_01: Fantastic.
• 14:57
  SPEAKER_01: Sounds like just an engineering challenge then to get the maximum utilization.
• 15:02
  SPEAKER_01: Really exciting stuff.
• 15:03
  SPEAKER_01: Thank you so much for the overview.
• 15:05
  SPEAKER_01: I want to briefly move on to downstream models and the state of the art at the moment.
• 15:11
  SPEAKER_01: There's lots of interest in foundational models and obviously datasets like this are used in the process, the training of those kind of models.
• 15:17
  SPEAKER_01: I'd love to hear both of your thoughts on where the models are at at the moment and where you see things going.
• 15:25
  SPEAKER_02: Yeah, I think there's some really exciting stuff going on.
• 15:28
  SPEAKER_02: I think a great example is a previous guest of yours, Konstantin, who was talking about SACLIT.
• 15:37
  SPEAKER_02: That kind of global approach, really big dataset that he was using, I think he said it was 100,000, 200,000.
• 15:43
  SPEAKER_02: Obviously, we're beginning to see this scaling up of approaches to big data.
• 15:55
  SPEAKER_02: I think what we're really kind of concerned about or we want to help encourage is that these models that are you know I don't really like the term foundation model but let's say foundation model that's going to be used in all of these different applications in all of these different areas by all different people with greater or lesser understanding of how the model works or what the model is doing
• 16:20
  SPEAKER_02: I think what we think is really important is basically that we have kind of data providence and we have an idea of like okay this model we know what went into it we know it's relatively unbiased hopefully because it's globally distributed it's not just over Europe and America we can trace you know the training data that they used to the kind of model output and we can recreate it if we if we have the resources and if we have the inclination we can reproduce that that
• 16:48
  SPEAKER_02: That's that.
• 16:50
  SPEAKER_02: And so Major TOM is kind of trying to
• 16:55
  SPEAKER_02: be as generally useful as possible to all these different applications and all these different kind of ways of thinking about representational learning or self-supervised learning at a large scale so that we can inject some kind of traceability into all of this work that's happening now?
• 17:10
  SPEAKER_02: Because I think what we want to avoid is maybe something that we see more in like, you know, large language modeling where, okay, there are, you know, there are amazing things going on in the open source world there and there are amazing datasets out there as well, but
• 17:25
  SPEAKER_02: The people at the front of the field are basically competing with their proprietary large datasets and their massive computational resources.
• 17:38
  SPEAKER_02: We can't really fix the computational resources part yet, but we can at least make sure that in EO, the best datasets that are out there are openly available, traceable,
• 17:49
  SPEAKER_02: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 18:17
  SPEAKER_02: They're not just, not just are they open, but they're also sort of traceable in the sense that they have data going in that we understand.
• 18:26
  SPEAKER_02: And so we can understand then maybe better how they're performing and why they're performing in certain ways.
• 18:31
  SPEAKER_01: Yeah.
• 18:31
  SPEAKER_01: So we don't want just sort of black box solutions and want to be able to audit the data, understand the bias potentially, and obviously this is where you're leading that datasets can change over time.
• 18:41
  SPEAKER_01: You might address imbalances in the data, for example.
• 18:43
  SPEAKER_01: Sorry, Miko.
• 18:45
  SPEAKER_00: I think,
• 18:48
  SPEAKER_00: An important context here as well is not only whether we can access the data, but how easy we can access that data.
• 18:55
  SPEAKER_00: Copernicus is an amazing program and we've got tons and tons and tons of data.
• 19:01
  SPEAKER_00: Everybody keeps recreating datasets for their own purpose.
• 19:04
  SPEAKER_00: Sometimes they're open, sometimes they're not.
• 19:08
  SPEAKER_00: In essence, if we want to create something quite general, it's not just a matter of providing supposed access to everyone who can go through the entire process of acquiring the data from an open program in one way or another, and then
• 19:22
  SPEAKER_00: doing what they want to do to basically utilize that data we also want to actually make it really easy to use if we want these models to be traceable and Major TOM tries to do that for Copernicus I mean Major TOM Core is what it tries to do that for Copernicus but we could do it with all sorts of data sources I think like there's this like we're basically trying to fill that big gap between huge data archives of EO imagery
• 19:52
  SPEAKER_00: and datasets that we use for deep learning because there's still a big divide and I feel like these two communities are slightly isolated in some sense and these it seems like there's still quite a lot of quite a lot of improvements we can make for like creating a bridge between these two contexts
• 20:16
  SPEAKER_02: I totally agree.
• 20:19
  SPEAKER_02: If we go back to the analogy of language, there's a great irony that for large language models and also image generation models,
• 20:27
  SPEAKER_02: The secret source, the proprietary information, if you like, is the way that these companies search and filter through the vast quantities of actually public information that is out there on the internet and in other sources.
• 20:41
  SPEAKER_02: And this is all information that is accessible to everyone all the time if they have the inclination to go and find it.
• 20:47
  SPEAKER_02: The difficulty is kind of searching, filtering,
• 20:51
  SPEAKER_02: Pairing down that vast trove of information, deduplicating whatever, into something that can be maximally useful for training their model.
• 21:00
  SPEAKER_02: And actually with EO, we see a really similar thing.
• 21:03
  SPEAKER_02: We have this huge trove of public information from these publicly funded programs, Copernicus, USGS and NASA in America and others.
• 21:14
  SPEAKER_02: We saw great things with the Indian Space
• 21:17
  SPEAKER_02: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 21:41
  SPEAKER_01: Yeah, I think it's really great that you're taking the lead and also just raising that issue because, you know, things move so fast with language models, it seems almost like a retrospective to try and unpick that problem.
• 21:53
  SPEAKER_01: Very finally, I just want to understand the evolution and the ownership of this dataset.
• 21:58
  SPEAKER_01: Is it something that's going to continue to grow under your direct stewardship or does the team own it?
• 22:04
  SPEAKER_01: Does the organization own it?
• 22:05
  SPEAKER_01: Who will be managing it in five, 10 years time?
• 22:09
  SPEAKER_02: Well, currently, I mean, it is just me and Miko who have been working on it, and we've kind of kept it fairly close to our chests until now because we kind of wanted a bit of fanfare with this release.
• 22:22
  SPEAKER_02: But I think we're both really, really keen to understand more about what the community wants.
• 22:30
  SPEAKER_02: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 22:48
  SPEAKER_02: from a broader perspective and anyone can kind of request to join and we can we can definitely accommodate that but from a wider perspective you know there's nothing stopping someone making a Major TOM compliant dataset outside of those organizations right it's still it's still the same format and specification but we will be looking I think if we generate kind of a big
• 23:10
  SPEAKER_02: community of people who are interested in some kind of structure by which we can maybe make decisions you know about the format in a sort of democratic way how we actually want to structure these data sets how we want them to interoperate and that really can only come from a large number of people because we are ultimately just going to think about applications that are relevant to us because we don't know about applications
• 23:37
  SPEAKER_02: other applications right and so if people come with all sorts of ideas about what this could be used for and how this could be uh engineered then i think it will be a much richer system for it um so we're we're yeah so there's cool yeah yeah i just wanted to reiterate that uh we really want people to join Major TOM because there's quite a few ways of contributing to the project because like even right now
• 24:06
  SPEAKER_00: Just by running the filtering notebook on your own machine, you can create a subset of Major TOM that works for you and maybe for some other people.
• 24:15
  SPEAKER_00: So you can actually create a new dataset definition with very minimal effort and publish this as a subset of Major TOM that works for this and that.
• 24:25
  SPEAKER_00: then obviously even more interesting will be when if people decide to contribute with their with new data samples and also with new labels I mean so far we've been quite focused on self-supervised and generative applications so this is just like a large bulk of input images so to speak but labels are quite interesting for too many people working with the old data so we definitely want to find ways how we can also kind of join
• 24:52
  SPEAKER_00: uh forces and create richer data sets and maybe mixes of data sets that also involve um labels um and i think this is really interesting because if we have like labels that don't change with time as much we can actually elevate other data sets to help us with various tasks and like spontaneously we can sometimes maybe get like temporal series and whatnot
• 25:13
  SPEAKER_00: Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets Satellite image deep learning featuring Robin Cole titled Major TOM Expandable EO Datasets
• 25:33
  SPEAKER_00: The following contributions will definitely be based on what people actually tell us they're looking for, and maybe they will want to join the forces and actually contribute actively to that.
• 25:45
  SPEAKER_02: The most pressing issue is that our hard drives are full, so we definitely need help from other people because it's getting harder and harder to collect more of this data, so we need to help.
• 25:54
  SPEAKER_01: Yeah, there's only so many drives available on Amazon.
• 26:00
  SPEAKER_01: Awesome.
• 26:01
  SPEAKER_01: Well, once again, thank you so much for the overview of the dataset.
• 26:04
  SPEAKER_01: Really exciting.
• 26:05
  SPEAKER_01: Really looking forward to seeing how the community grows around this.
• 26:08
  SPEAKER_01: Sounds like Hugging Face is the prime location to connect with the dataset and yourselves.
• 26:14
  SPEAKER_01: And I'm really looking forward to future updates.
• 26:16
  SPEAKER_00: Thank you.
• 26:17
  SPEAKER_00: Thanks.