Latest / Satellite image deep learning / Mapping South America and Beyond with Fields of The World V2

Transcript

Mapping South America and Beyond with Fields of The World V2 · Satellite image deep learning

0:06
SPEAKER_02: Hi, Hannah.
0:06
SPEAKER_02: Hi, Tristan.
0:07
SPEAKER_02: Welcome to the podcast.
0:08
SPEAKER_01: Thank you.
0:09
SPEAKER_02: Thanks.
0:11
SPEAKER_02: Let's begin with where you work and what you do, Hannah.
0:15
SPEAKER_01: Sure.
0:16
SPEAKER_01: So I'm Hannah Kerner.
0:17
I'm an assistant professor at Arizona State University in the
0:21
School of Computing
0:22
SPEAKER_01: and Augmented Intelligence.
0:25
I'm also the AI lead for the NASA Harvest and NASA Acres
0:28
programs,
0:29
which are NASA's agriculture,
0:33
SPEAKER_01: and food security programs focused globally and on the US.
0:38
SPEAKER_01: And I'm also a research advisor for Taylor Geospatial.
0:42
So my lab at ASU focuses on developing machine learning and AI
0:47
algorithms,
0:48
SPEAKER_01: data sets,
0:50
SPEAKER_01: knowledge,
0:51
SPEAKER_01: and scientific advances that can support
0:56
making AI work for real people and delivering on a lot of the
1:00
challenges that we
1:01
SPEAKER_01: have in the real world,
1:03
SPEAKER_01: particularly related to environmental challenges.
1:07
SPEAKER_01: So we do a lot of work on machine learning for satellite data,
1:12
SPEAKER_01: including developing systems often that support agricultural monitoring,
1:17
SPEAKER_01: which we're able to deploy for real users through
1:21
SPEAKER_01: NASA Acres and NASA Harvest and now through Taylor Geospatial as well.
1:28
And work on trying to actually get these research advances into
1:34
the hands of end users.
1:35
SPEAKER_02: Fantastic.
1:37
SPEAKER_02: How about yourself, Christian?
1:39
SPEAKER_00: Hi, Robin.
1:40
SPEAKER_00: Thanks so much for having me on the podcast.
1:41
SPEAKER_00: I can't wait to chat.
1:43
SPEAKER_00: My name is Tristan Grupp.
1:44
SPEAKER_00: I'm an agricultural data scientist at World Resources Institute.
1:47
I've been really fascinated in my work by characterizing
1:50
complex systems using geospatial data.
1:53
SPEAKER_00: Land use systems, water systems, food systems.
1:57
SPEAKER_00: WRI is an awesome place to characterize these complex systems.
2:01
SPEAKER_00: We're an international independent research organization.
2:03
I'm based out of Washington,
2:05
D.C.,
2:05
but we have offices all over the world.
2:07
Some of our offices are in Kenya,
2:10
Brazil,
2:11
Colombia,
2:12
Mexico,
2:12
China,
2:13
Indonesia,
2:14
Europe.
2:15
SPEAKER_00: And we leverage this global network,
2:17
our expertise and data to influence policy and create more
2:21
positive outcomes for
2:22
SPEAKER_00: people,
2:23
SPEAKER_00: nature and climate.
2:24
SPEAKER_00: I'm a data lab.
2:25
SPEAKER_00: We deliver a lot of data products.
2:28
as part of WRI,
2:29
such as Global Forest Watch,
2:30
which tracks deforestation globally.
2:32
And before we dive into the applied AI work and the fields of
2:35
the World Research
2:36
SPEAKER_00: Consortium,
2:36
I wanted to talk about why we're doing this kind of work and
2:40
who's funded us.
2:41
So the Walmart Foundation funded WRI and Hannah Kerner's lab to
2:45
map the fields of South America.
2:47
SPEAKER_00: And why we're doing this is to address agricultural-driven deforestation.
2:52
SPEAKER_00: Agriculture causes deforestation globally,
2:54
and a few commodities are associated with this deforestation
2:57
and nature loss.
2:58
SPEAKER_00: Pasture, so cattle, soybeans, coffee, cocoa, palm oil.
3:06
SPEAKER_00: The European Union is trying to influence this.
3:09
SPEAKER_00: It's probably the biggest regulation, environmental regulation to date.
3:15
SPEAKER_00: It's called the European Union Regulation on Deforestation, EUDR.
3:18
And what EUDR is doing is saying that if you are going to be
3:22
selling to the
3:22
SPEAKER_00: European Union market,
3:24
SPEAKER_00: you have to prove that this food commodity,
3:26
SPEAKER_00: this commodity,
3:27
did not cause deforestation down to the plot of land that it
3:30
was grown on.
3:31
SPEAKER_00: And this is a huge gap.
3:33
SPEAKER_00: There is a big gap in data, in South America particularly.
3:38
SPEAKER_00: Fields are not mapped globally.
3:40
SPEAKER_00: South America has some of the most intact native ecosystems on Earth.
3:45
SPEAKER_00: And so we're filling that gap by mapping the field boundaries,
3:49
SPEAKER_00: mapping the plots of land in South America.
3:52
SPEAKER_00: South America has some critical native ecosystems,
3:55
SPEAKER_00: the Amazon rainforest probably being the most famous,
3:57
SPEAKER_00: but also the Cerrado,
3:58
SPEAKER_00: the Chiquitania,
3:59
SPEAKER_00: the Chaco,
4:00
SPEAKER_00: the Pampas,
4:01
SPEAKER_00: Alto Paraná.
4:03
SPEAKER_00: And these ecoregions are heavily impacted by deforestation.
4:06
SPEAKER_00: So by mapping field boundaries,
4:08
SPEAKER_00: we can help companies that produce these products,
4:11
SPEAKER_00: that sell and trade in these food products,
4:14
comply with the regulation and show that their plots of land
4:16
have not caused deforestation.
4:18
SPEAKER_00: And we're hoping that these data science products,
4:19
SPEAKER_00: these data products can help reduce deforestation and support the EUDR.
4:25
Can't wait to talk about the AI and the applied data science
4:31
work here and the
4:32
SPEAKER_00: Fields of the World Research Consortium.
4:34
SPEAKER_00: Thanks so much.
4:34
SPEAKER_02: Excellent.
4:37
SPEAKER_02: Thanks, Kristen.
4:38
SPEAKER_02: So we're connected over Fields of the World.
4:40
SPEAKER_02: Hannah, maybe you give us a bit of background to Fields of the World.
4:44
SPEAKER_01: Yeah,
4:44
SPEAKER_01: so Fields of the World is really it started as a benchmark data set,
4:49
but has evolved into a whole ecosystem of tools for supporting
4:54
field boundary
4:55
SPEAKER_01: segmentation globally.
4:57
So that's the idea that we want to take a satellite image and
5:01
be able to draw a
5:03
vector polygon representing every field along its boundary and
5:10
be able to create
5:10
SPEAKER_01: solutions that work well anywhere in the world.
5:13
So Fields of the World has a benchmark data set that we can use
5:19
for training models
5:20
and benchmarking progress of new models that are developed
5:24
using that data set for
5:26
SPEAKER_01: field boundary segmentation.
5:27
SPEAKER_01: That includes, I think, 26 countries globally.
5:32
SPEAKER_01: So we have not just like a single performance over the whole world,
5:37
SPEAKER_01: but you can get group wise,
5:39
SPEAKER_01: country wise performance and models are scored based on
5:43
how well you do on average across these countries to kind of
5:47
address the bias that
5:49
SPEAKER_01: many models have on performing well in data rich regions.
5:54
SPEAKER_01: And we also have lots of tools like a CLI for running these models.
6:00
SPEAKER_01: So you can really easily use our CLI to download,
6:05
SPEAKER_01: search for images,
6:06
SPEAKER_01: download satellite images that go into the model,
6:09
SPEAKER_01: download different model checkpoints that are hosted,
6:12
SPEAKER_01: publicly and run model inference for new regions.
6:19
We also have a web app that you can use if you don't want to
6:22
run a CLI or write any code.
6:26
SPEAKER_01: The project is entirely open source,
6:28
so you can go in and see all of our code on GitHub and even
6:33
make pull requests,
6:35
SPEAKER_01: assign some issues to yourself if you are interested.
6:38
It's really an open source community effort to develop the
6:43
infrastructure that's
6:44
SPEAKER_01: needed for achieving global field boundary segmentation.
6:47
SPEAKER_02: That is fantastic.
6:49
And I'm curious what state-of-the-art performance looks like on
6:52
Field of the World.
6:54
SPEAKER_01: Yeah, that's a good question.
6:55
State of PR performance looks different in different places,
6:59
as you might imagine.
7:00
I don't think anyone listening to this podcast is surprised
7:04
that models perform
7:06
SPEAKER_01: differently in different parts of the world.
7:09
SPEAKER_01: So most of the data that's in fields of the world,
7:11
most of the data that's publicly available for ground truth
7:15
field boundaries that
7:16
SPEAKER_01: we can use to train the model are in larger fields.
7:20
agricultural systems that have more regular field boundaries or
7:24
larger fields,
7:26
SPEAKER_01: which are easier to identify the boundary of in Sentinel-2 data,
7:31
SPEAKER_01: which is 10 meter per pixel.
7:33
So in certain places that are in distribution to the model and
7:38
have fields,
7:39
SPEAKER_01: even if they're out of distribution to the model,
7:40
SPEAKER_01: that have fields similar to a lot of what the model has seen,
7:46
SPEAKER_01: State of the art performance looks really good.
7:49
SPEAKER_01: You know,
7:50
SPEAKER_01: it looks generally like if you looked at the boundaries,
7:53
SPEAKER_01: maybe you see some errors where you're like,
7:55
maybe I would have drawn a line in between that or you might
7:58
have done something
7:59
SPEAKER_01: slightly differently.
8:00
SPEAKER_01: But it's it's pretty much solved, I would say, in a lot of those places.
8:06
SPEAKER_01: In places where the fields are much smaller,
8:08
SPEAKER_01: there's more heterogeneous mixed cropping systems,
8:13
especially when those boundary sizes start to be below the
8:18
scale of Sentinel-2.
8:18
SPEAKER_01: That's where we see more challenges.
8:23
SPEAKER_01: If you're familiar with this problem,
8:25
I think of field boundary segmentation and these kinds of
8:28
resolution issues,
8:30
still state of the art performance in regions that are totally
8:32
out of distribution
8:33
SPEAKER_01: and very small challenging areas.
8:36
SPEAKER_01: we get, I think, pretty surprisingly good starting results.
8:41
And then it's pretty straightforward to improve the model by
8:45
adding new training data.
8:47
SPEAKER_01: It's very, very responsive to new training labels in a new region.
8:51
SPEAKER_01: And that's something Tristan will tell you more about.
8:54
SPEAKER_02: Fantastic.
8:56
SPEAKER_02: Tristan, over to you.
8:57
SPEAKER_02: Can you argue with a question?
9:02
SPEAKER_02: Extending to new locations.
9:05
SPEAKER_01: Yeah.
9:05
SPEAKER_01: So Tristan,
9:06
I just was thinking that in the effort to improve the model in
9:11
South America,
9:12
the results look pretty good in some of these regular large
9:16
scale cropping systems.
9:17
But to improve it on the mixed systems we see in South America,
9:21
training data was super helpful.
9:23
SPEAKER_00: Yeah, exactly.
9:24
And I think that's how we approach creating new training data
9:26
for South America was
9:27
trying to capture the diversity of agricultural practices that
9:30
you see there.
9:31
SPEAKER_00: I think in the word agriculture,
9:33
SPEAKER_00: it's agriculture,
9:34
SPEAKER_00: like culture,
9:36
SPEAKER_00: a specific culture produces agriculture.
9:39
SPEAKER_00: It varies around the world.
9:41
And so capturing diversity and training data was really
9:43
important for this project.
9:47
You can see huge diversity in agricultural practices across
9:51
South America.
9:51
SPEAKER_00: You see shifting cultivation in the Amazon.
9:54
You see really vast pasture and soybean fields in northern
9:59
Chaco,
10:00
in Mato Grosso.
10:02
SPEAKER_00: And then...
10:03
SPEAKER_00: In mountains, there are very small, snaky fields.
10:05
SPEAKER_00: And so agriculture varies a lot with the culture that practices them,
10:10
SPEAKER_00: the ecology that it exists in.
10:12
SPEAKER_00: And so we approach training data collection with diversity,
10:15
SPEAKER_00: capturing a bunch of different conditions.
10:18
The training data set in Fields of the World was this small
10:22
segment of fields.
10:23
SPEAKER_00: I think it's
10:26
SPEAKER_00: 0.11% of the overall FTW training data.
10:29
So we really wanted to expand it,
10:30
make Fields of the Worlds work in South America.
10:33
And so we added a bunch of new labels through targeting this
10:36
diversity and
10:36
SPEAKER_00: challenging conditions.
10:38
And we saw that it really helped the model generalize across a
10:41
bunch of different contexts.
10:43
So yeah,
10:43
the diversity of training data was really important for this
10:45
work.
10:46
SPEAKER_02: Right.
10:46
And did you consult an expert on South American fields or did
10:51
you have to become
10:51
SPEAKER_02: one to identify which areas needed annotation?
10:58
So I've been working on South American agriculture for about
11:01
eight years.
11:02
SPEAKER_00: I started my work in the Peruvian Amazon.
11:04
They have this really specific style of shifting cultivation
11:08
that I've been characterizing.
11:10
SPEAKER_00: And then just generally working on agriculture in South America,
11:14
SPEAKER_00: you've come across all these different practices,
11:16
SPEAKER_00: read about them.
11:17
And so I came in with a certain familiarity with the diversity
11:21
of practices,
11:21
SPEAKER_00: but I had to learn a lot as well.
11:23
SPEAKER_00: And so to really get a sense for all the diversity of practices,
11:28
we ran the fields of the world model in South America and saw
11:31
where the model
11:32
SPEAKER_00: wasn't performing well.
11:33
SPEAKER_00: And in those areas, we hand annotated field boundaries.
11:36
SPEAKER_00: So we looked to see, okay, is there a little model performance here?
11:39
SPEAKER_00: Let's sample here.
11:40
SPEAKER_00: Let's create an initialization baseline.
11:42
SPEAKER_00: Let's create a nice base to work from.
11:46
SPEAKER_00: And we focused on areas of fields of the world
11:50
the fields of the world model under performance and sampling in
11:53
17 ecoregions in South America.
11:55
SPEAKER_00: And so that diversity was our approach.
11:59
SPEAKER_00: And I had to look at a lot of model output results,
12:02
SPEAKER_00: make sure that we were capturing a bunch of different systems.
12:04
So a lot of that was qualitative and getting used to and
12:09
getting familiar with the
12:10
SPEAKER_00: different types of agriculture.
12:12
So I learned so much about the different systems and that was a
12:14
really fascinating
12:15
SPEAKER_00: part of this work.
12:17
i'm sure some are very kind of niche and localized so how do
12:21
you decide when you
12:22
have covered and captured enough data for you know each of
12:26
these different kind of
12:26
subclasses vibes no no i mean we annotation effort is costly uh
12:38
even when we have
12:39
SPEAKER_01: these annotators on our own team it's
12:44
SPEAKER_01: you really are careful to not overestimate.
12:48
SPEAKER_01: So,
12:48
SPEAKER_01: you know,
12:49
we try to do what seems like a minimum amount and see how
12:53
things go and then add
12:54
SPEAKER_01: more as needed.
12:55
SPEAKER_01: So Tristan can tell you more about this,
12:58
but we essentially started with what we thought seemed like a
13:01
low reasonable
13:02
SPEAKER_01: number,
13:03
SPEAKER_01: about 50 chips in each eco region for this sampling of
13:09
SPEAKER_01: low performing areas in each eco region.
13:12
And then we also use the updated model to look at where things
13:16
were performing
13:17
poorly and also collect whatever data we could from various
13:21
partners to add into
13:24
SPEAKER_01: the model.
13:24
And I think in the end that ended up being about 900 chips
13:28
added to fields of the
13:30
SPEAKER_01: world,
13:30
SPEAKER_01: which is really a small amount compared to the overall data set,
13:34
SPEAKER_01: which is about 70,000 chips.
13:36
SPEAKER_01: But even that small amount with no re-weighting,
13:39
SPEAKER_01: just adding them into the model,
13:42
SPEAKER_01: you know,
13:43
appending it with the existing training data set has a huge
13:47
impact on performance
13:48
SPEAKER_01: in the region.
13:50
SPEAKER_02: And you mentioned the model.
13:51
So there is a single global model and objective and stuff to
13:54
bring that one up
13:55
SPEAKER_02: rather than great kind of location-specific models.
14:01
Yeah,
14:01
Tristan,
14:01
maybe you want to share a bit more about the experience of...
14:05
SPEAKER_01: trying to fine tune these models in different places.
14:08
I think what many people assume in agricultural monitoring or
14:15
machine learning with
14:15
satellite data is that specializing local models will typically
14:20
do better.
14:21
But I think we find in a lot of cases and especially in this
14:25
project that it's
14:26
really important to retain the global size of the training data
14:31
and diversity of
14:32
SPEAKER_01: the samples that were there.
14:33
SPEAKER_01: Again, because
14:35
The cropping systems in South America are so,
14:39
so diverse across the entire continent.
14:41
SPEAKER_01: And so we were,
14:43
SPEAKER_01: you know,
14:43
SPEAKER_01: adding some specialized training data,
14:46
but it's still training on that alone was not as good as
14:50
training with the global
14:52
SPEAKER_01: data as well.
14:53
And you really want to train in a way that retains that
14:57
previous knowledge.
15:00
SPEAKER_00: Just to jump back to this point about training data,
15:02
we originally thought to do 100 samples in each ecoregion and I
15:05
was labeling and my
15:06
SPEAKER_00: hands started cramping.
15:07
SPEAKER_00: I was like, we got to stop it at 50.
15:09
SPEAKER_00: This is way too much.
15:12
But that first 50 samples that we did,
15:14
I think it was actually 83 that we dropped down to 50.
15:18
SPEAKER_00: We did fine-tuning in Bolivia.
15:20
And there's this really particular pattern in the Bolivian
15:23
Chiquitania with these
15:24
SPEAKER_00: huge,
15:25
SPEAKER_00: large strip soy fields and these radial,
15:28
SPEAKER_00: massive fields.
15:29
SPEAKER_00: And that was our first fine-tuning process.
15:32
training data set and when we did fine tuning we saw model
15:35
performance drop
15:36
significantly so with full fine tuning um it went from the base
15:41
model in brazil so
15:42
against the brazil test set being 96 pixel iou and it dropped
15:46
down to 42 even after
15:48
we added this chiquitania training data and so um we saw model
15:52
performance fall uh
15:53
SPEAKER_00: and so that i think that speaks to um hannah's point that uh
15:57
SPEAKER_00: You can't just add local data.
15:59
SPEAKER_00: It's not always going to make a better local model.
16:01
SPEAKER_00: There's a lot of great knowledge from other contexts.
16:06
And so the combination is why we decided to go with these fully
16:09
trained models with
16:10
SPEAKER_00: the full FTW baseline plus the South America data,
16:13
because we didn't want to have this issue of catastrophic
16:15
forgetting or losing
16:17
SPEAKER_00: really interesting or really valuable data.
16:23
SPEAKER_00: data from other contexts, training data from other contexts.
16:26
SPEAKER_00: Yeah.
16:27
SPEAKER_01: And part of,
16:28
SPEAKER_01: sorry,
16:28
I just add to that part of what we want to do in adding new
16:32
training data is
16:32
specializing to the types of patterns that humans are paying
16:36
attention to,
16:37
SPEAKER_01: like the different types of agriculture.
16:39
But there's a lot that models are paying attention to that are
16:42
not necessarily relevant.
16:44
And so having more training data often helps the model become
16:48
more robust to these things like
16:51
SPEAKER_01: cloud cover or certain colorations,
16:54
SPEAKER_01: you know,
16:54
SPEAKER_01: these subtle things that humans are ignoring because,
16:57
SPEAKER_01: you know,
16:58
SPEAKER_01: we're well-evolved,
16:59
SPEAKER_01: but models can be quite sensitive too.
17:01
I'd love to understand a bit more about the process you went
17:06
through to choose the
17:07
SPEAKER_02: particular model because you've obviously got CNNs,
17:10
SPEAKER_02: Transformers,
17:10
SPEAKER_02: they might have different areas they perform better or worse in.
17:14
SPEAKER_02: Is that something that you can talk about?
17:17
SPEAKER_01: Yeah,
17:18
so in the first fields of the world kind of sprint that we did
17:23
with the first paper
17:24
SPEAKER_01: that we published on this benchmark data set,
17:27
we basically had the hypothesis that the issue in solving this
17:32
problem of global
17:34
field boundary segmentation was not that we didn't have a good
17:37
enough model
17:38
SPEAKER_01: architecture.
17:39
SPEAKER_01: It was that we didn't have enough data.
17:41
And so we focused that first effort on creating this large
17:45
scale field boundary
17:47
SPEAKER_01: segmentation data set that we could use to train any model architecture.
17:52
And we did very little experimentation on which architectures
17:56
we experimented with
17:57
SPEAKER_01: in that first paper,
17:59
SPEAKER_01: which was published in AAAI last year.
18:04
And so there we basically said,
18:07
we know units work well for segmentation.
18:10
SPEAKER_01: We're going to use a unit.
18:11
Maybe we'll sweep a few backbones like efficient nets or deep
18:17
lab just to see how
18:19
SPEAKER_01: much sensitivity there is.
18:21
SPEAKER_01: We found that there was not much sensitivity.
18:23
SPEAKER_01: Most of the sensitivity was coming from the data.
18:26
So we kind of kept that as just a pretty standard lightweight
18:30
model and focused on
18:31
SPEAKER_01: the training data development.
18:33
SPEAKER_01: Then after we published that data set,
18:36
SPEAKER_01: we said,
18:36
SPEAKER_01: well,
18:36
how much can we juice out of this benchmark data set with model
18:40
improvements?
18:41
SPEAKER_01: So then we started an effort that we called
18:44
SPEAKER_01: the Bake Off,
18:46
SPEAKER_01: which as somebody living in the UK,
18:50
SPEAKER_01: I think you're probably very familiar with PRU.
18:52
SPEAKER_01: So we ended up calling our eventual model PRU,
18:57
SPEAKER_01: which is some tortured acronym for Practical Recipe for Field Boundary
19:02
SPEAKER_01: Segmentation.
19:02
SPEAKER_01: I'll let you work that one out, how it comes out to PRU.
19:07
SPEAKER_01: But basically there,
19:09
SPEAKER_01: we said,
19:10
SPEAKER_01: okay,
19:10
let's now keep the data set fixed and try out a whole bunch of
19:16
different modeling
19:16
SPEAKER_01: choices.
19:18
SPEAKER_01: So we tried a whole bunch of different choices within units.
19:23
SPEAKER_01: So changing...
19:24
SPEAKER_01: different weightings on the loss function,
19:27
SPEAKER_01: changing different loss functions,
19:28
SPEAKER_01: using different backbones in terms of the backbone architecture,
19:32
SPEAKER_01: but also the size,
19:34
SPEAKER_01: like efficient at B3 up to B7,
19:37
SPEAKER_01: as well as other architectures.
19:39
SPEAKER_01: We tried other types of models,
19:43
SPEAKER_01: like treating it as instant segmentation,
19:46
SPEAKER_01: using models like mass two former instead.
19:50
SPEAKER_01: We also looked at geospatial foundation models,
19:53
SPEAKER_01: fine tuning them on the fields of the world data set.
19:58
SPEAKER_01: tested like I think 11 different models on the data set.
20:02
SPEAKER_01: So it was a really, really big modeling effort.
20:04
SPEAKER_01: I don't know exactly how many checkpoints we created here.
20:07
SPEAKER_01: I think it's like over 100 for sure.
20:12
But really pushing on seeing what limit we could hit with the
20:15
modeling now that we
20:18
SPEAKER_01: have this training data available.
20:19
SPEAKER_01: And we were able to improve the performance quite a bit by
20:25
uh,
20:25
pushing on that and getting,
20:26
uh,
20:27
I think over 10% improvement on the benchmark performance.
20:33
SPEAKER_01: Um,
20:34
SPEAKER_01: and again,
20:35
SPEAKER_01: uh,
20:36
SPEAKER_01: that,
20:38
what came out of that essentially was a recipe that worked well
20:41
for doing,
20:41
SPEAKER_01: uh,
20:42
SPEAKER_01: failed boundary segmentation that is not just applicable,
20:46
SPEAKER_01: really segmentation in general,
20:47
SPEAKER_01: we're finding when,
20:48
SPEAKER_01: uh,
20:49
SPEAKER_01: we've tested this on other projects too,
20:51
SPEAKER_01: but,
20:52
SPEAKER_01: um,
20:53
that's something that can be applied for field boundary
20:55
segmentation in general,
20:57
SPEAKER_01: not just for the fields of the world data set.
21:02
SPEAKER_01: And another thing we looked at there was testing these models,
21:07
SPEAKER_01: not just in terms of benchmark performance,
21:11
SPEAKER_01: as in,
21:11
SPEAKER_01: you know,
21:11
SPEAKER_01: IOU,
21:12
SPEAKER_01: precision recall,
21:13
SPEAKER_01: blah,
21:13
SPEAKER_01: blah,
21:14
SPEAKER_01: blah on the test set,
21:16
SPEAKER_01: but trying to create additional metrics that would
21:20
better approximate how the model is performing on new samples
21:24
out of distribution
21:25
because what we were finding after fields of the world v1 with
21:30
the you know
21:31
standard unit we had from that paper was that there were a lot
21:36
of systematic issues
21:37
that weren't really captured in the benchmark metrics this no
21:42
surprise to anyone
21:44
SPEAKER_01: i'm sure but
21:46
SPEAKER_01: So we created a few metrics like one that we called consistency,
21:50
SPEAKER_01: which is looking at how consistent the predictions are.
21:53
If you slide the window over a little bit to try to get at
21:56
these edge artifacts
21:58
SPEAKER_01: that often occur with models.
22:00
SPEAKER_01: We also ranked models on throughput.
22:03
So how many kilometers squared per second could be processed as
22:07
a proxy for cost,
22:08
SPEAKER_01: which is hugely important for making maps.
22:11
And some other metrics looking at robustness to like brightness
22:16
or orientation kinds of things.
22:21
Very interesting because often get fixated on the IOU metric or
22:25
something.
22:25
SPEAKER_02: But as you say, end users don't really care.
22:27
SPEAKER_02: They have some other experience that they want to.
22:30
measure somehow very interesting okay so we've moved on version
22:35
one version two
22:36
both uh model and the data set by the sound of things uh what
22:41
was the kind of
22:41
performance jump for those south american regions with the
22:45
version two then
22:48
SPEAKER_01: Yeah,
22:48
SPEAKER_01: Tristan,
22:49
SPEAKER_01: do you want to speak to,
22:51
SPEAKER_01: I think in particular,
22:52
when we integrated some of the models that were developed along
22:57
the way in this
22:58
SPEAKER_01: bake-off,
22:59
some different choices like changing the boundary class
23:01
weighting and things like
23:04
SPEAKER_01: this,
23:05
SPEAKER_01: we saw pretty major improvements in a lot of places.
23:10
SPEAKER_00: Yeah, we really did.
23:12
So I think one of the starkest jumps was that the Fields of the
23:16
World-based model
23:17
SPEAKER_00: on SOI in South America was looking at 47% pixel IOU.
23:22
SPEAKER_00: And then after adding new training data, it jumped up to 77%.
23:25
SPEAKER_00: And that was just the first iteration.
23:29
Adding new training data on top of that,
23:31
so we trained four different model variants.
23:33
SPEAKER_00: And the final one included these challenging active learning samples.
23:36
SPEAKER_00: And that's where we saw the biggest jump up in object recall.
23:40
SPEAKER_00: Pixel value didn't change much, but recall shot up by more than 20%.
23:45
So we were able to see huge gains by adding just a small amount
23:47
of training data.
23:49
SPEAKER_00: And again,
23:50
SPEAKER_00: this number of samples,
23:52
SPEAKER_00: I think it ended up being,
23:54
SPEAKER_00: after active learning,
23:54
SPEAKER_00: about 1,200 new samples,
23:56
SPEAKER_00: makes up such a small number of the overall FCW training data set.
24:00
SPEAKER_00: But we see huge gains, especially in object recall.
24:05
yeah so these uh increasing training data is hugely important
24:08
for making these
24:09
these models um performant but on the modeling side some of
24:13
these elements of the
24:15
pru recipe um that we applied to the south america mapping we i
24:20
think most notably
24:21
saw improvements in over prediction of the boundary class so
24:25
fields of the world
24:26
SPEAKER_01: models predict
24:27
SPEAKER_01: boundary, field interior, and background.
24:31
SPEAKER_01: And we emphasize that boundary class to really make it draw the boundary,
24:35
SPEAKER_01: not just merge fields together.
24:38
SPEAKER_01: But the first version model kind of took that too seriously.
24:42
SPEAKER_01: And we saw just prediction of boundary class everywhere.
24:47
SPEAKER_01: And so when we actually segment fields from that,
24:51
SPEAKER_01: when we vectorize fields from those raster predictions,
24:54
SPEAKER_01: we just keep the field interior class.
24:56
SPEAKER_01: So that meant that
24:58
all this stuff that was being predicted as boundary was just
25:00
missed by the model.
25:03
And so it also led to a lot of really eroded field boundaries
25:07
because the boundary
25:09
class was being predicted so much and there wasn't much
25:12
interior class being
25:13
SPEAKER_01: predicted,
25:13
SPEAKER_01: which is again,
25:15
SPEAKER_01: the part that gets vectorized as the field object.
25:20
SPEAKER_01: So the pre-recipe really improved that boundary over prediction issue.
25:24
SPEAKER_00: And we actually shared some of these model results.
25:26
SPEAKER_00: So that first version of the model,
25:28
SPEAKER_00: which overweighted the boundary as Hannah was talking about,
25:31
we produced model results for that version and then the change
25:36
in the loss function
25:36
SPEAKER_00: weighting version.
25:38
And we shared these boundaries produced from these models with
25:41
actual users.
25:42
And one of them was someone from the federal finance
25:45
institution that invests in
25:47
SPEAKER_00: sustainable agriculture.
25:48
SPEAKER_00: and they noticed the difference.
25:49
So they looked at the version one model with the higher
25:53
boundary weight and they
25:54
SPEAKER_00: said it looked buffered in.
25:56
SPEAKER_00: But then on a more recent version,
26:00
SPEAKER_00: I can directly quote for them and say,
26:01
SPEAKER_00: this data strongly aligns with our expectations.
26:04
SPEAKER_00: Precision,
26:05
SPEAKER_00: spatial coherence,
26:05
and compatibility with national land use data sets make it
26:09
highly useful for
26:09
SPEAKER_00: monitoring,
26:10
SPEAKER_00: planning,
26:10
SPEAKER_00: and agricultural intelligence.
26:13
SPEAKER_00: So
26:14
They also said that the latest version of the model would be
26:17
great for area-based
26:17
SPEAKER_00: estimations,
26:18
SPEAKER_00: again,
26:19
SPEAKER_00: solved by that boundary cost re-rating.
26:22
SPEAKER_00: So we had users directly give us feedback about these modeling changes,
26:26
SPEAKER_00: and they saw the improvements and were excited by them.
26:30
SPEAKER_02: It sounds...
26:32
significant that only a small amount of actual annotation had a
26:35
big impact on the model itself.
26:38
SPEAKER_02: And we can imagine that in other parts of the world,
26:40
SPEAKER_02: people might be thinking,
26:41
SPEAKER_02: can I do the same thing?
26:43
SPEAKER_02: What do you recommend for people that might be thinking that?
26:47
SPEAKER_01: I would say probably.
26:49
SPEAKER_01: So we have a lot of documentation on the GitHub page for
26:56
SPEAKER_01: how to train these models.
26:59
But with this project that when WRI is releasing different
27:05
additional tools that we
27:06
SPEAKER_01: developed to adapt the model for the South America use case,
27:11
there's going to be more documentation about how to create new
27:14
training data and
27:16
add that data to models for people to improve if it's not
27:20
working well in their
27:21
SPEAKER_01: region.
27:22
SPEAKER_01: So those are resources that people can build on
27:25
SPEAKER_01: That might help to add more context from the Fields of the World repo.
27:31
Fields of the World community is also working on tools to help
27:36
with annotation in
27:39
SPEAKER_01: new regions,
27:40
help people add or even edit the boundaries to give feedback
27:44
and incorporate new
27:45
SPEAKER_01: updates into the model in the future.
27:46
SPEAKER_01: So...
27:48
SPEAKER_01: Basically, you can do this on your own.
27:50
And if it's not working well for you right now,
27:53
that's the first place I would start.
27:54
SPEAKER_01: Don't be trying to get a geospatial foundation model to fine tune.
27:58
SPEAKER_01: It's not going to work as well, I promise, as one of these units.
28:04
And then stay tuned for our next version of the data set that's
28:11
going to be coming
28:13
SPEAKER_01: in the next year.
28:15
year i'll say i'm not going to make promises on exactly when in
28:17
the next year but
28:19
i'm really excited for it it's going to add a lot of new data
28:22
and have new labeling
28:24
efforts to fill in the gaps in poor performing places too
28:28
that's really great to
28:29
hear and we're looking forward to hearing what comes in version
28:32
three so just uh to
28:34
bring things to a close so version two the paper the dates are
28:38
imminently released
28:40
SPEAKER_02: is that right
28:41
SPEAKER_01: Yeah, so the first paper is already released.
28:43
You can find it search fields of the world archive or maybe
28:48
it's linked with this episode.
28:50
SPEAKER_01: Like I said, that was published in AAA last year.
28:53
SPEAKER_01: So that published the data set and really the baseline models.
28:57
The second paper will be on archive any day now and will be
29:01
published in the CBPR
29:02
SPEAKER_01: proceedings this summer.
29:04
And that has all the results for this big model bake-off and
29:09
describes the PRU recipe.
29:11
But the PRU model checkpoint is already available through the
29:15
FTW ecosystem.
29:16
And also very exciting,
29:18
in the next several weeks,
29:19
we're going to be,
29:20
I'll say several months.
29:22
SPEAKER_01: Again, I don't want to overpromise, so I'm super excited.
29:25
we're going to be releasing a global map of predicted field
29:30
boundaries to make it
29:31
even easier for people to get the initial boundaries from
29:34
fields of the world.
29:35
SPEAKER_02: Fantastic.
29:36
So people just go and go to their neighborhood and see how the
29:39
model performs there
29:40
SPEAKER_02: and access those boundaries.
29:42
SPEAKER_02: Wonderful.
29:42
SPEAKER_02: Yes.
29:44
SPEAKER_02: Well, really interesting project, very impactful.
29:46
I'm delighted to hear that it's going to keep going and address
29:50
other areas and so forth.
29:52
SPEAKER_02: Obviously,
29:53
sounds like the website for Fields of the World is the main
29:55
place for people to go
29:58
SPEAKER_02: to get started.
30:00
SPEAKER_01: Yeah, the website is good if you want to get started.
30:03
We're trying to update that website to make the entry points
30:06
for different types of
30:08
SPEAKER_01: interested people more clear.
30:10
SPEAKER_01: And if you're a developer,
30:11
SPEAKER_01: want to work on open source projects,
30:13
SPEAKER_01: then definitely come to our GitHub.
30:15
SPEAKER_02: Fantastic.
30:17
SPEAKER_02: I'll put the links in the show notes.
30:19
SPEAKER_02: And once again, thank you for joining me today.
30:21
SPEAKER_02: And all the best of version three.
30:23
SPEAKER_01: Thank you.
30:24
SPEAKER_02: Thank you.