Latest / Satellite image deep learning / Mapping The World at Taylor Geospatial

Transcript

Mapping The World at Taylor Geospatial · Satellite image deep learning

0:05
SPEAKER_01: Hi, Jen.
0:06
SPEAKER_01: Hi, Isaac.
0:07
SPEAKER_01: Welcome to the podcast.
0:08
SPEAKER_02: Hi, Robin.
0:09
SPEAKER_02: Thanks for having us.
0:10
SPEAKER_01: Thanks for having us.
0:11
SPEAKER_01: All right.
0:12
SPEAKER_01: Let's begin with you, Jen.
0:13
SPEAKER_01: Where do you work and what do you do?
0:16
SPEAKER_02: I'm Jen Marcus.
0:17
I am the vice president for strategic innovation programs at
0:21
Taylor Geospatial.
0:23
SPEAKER_01: Wonderful.
0:23
SPEAKER_01: Thank you.
0:24
SPEAKER_01: And Isaac?
0:26
SPEAKER_00: I'm Isaac.
0:26
SPEAKER_00: I am director of research at Taylor Geospatial,
0:29
and I worked primarily on the fields of the world project to do
0:33
the large scale
0:34
SPEAKER_00: global inference.
0:35
SPEAKER_01: Wonderful.
0:36
SPEAKER_01: Jen, maybe you can give us the intro to Taylor Geospatial.
0:40
SPEAKER_02: Sure, happy to.
0:42
SPEAKER_02: Yeah, we're excited about the organization that we're in right now.
0:46
SPEAKER_02: We established ourselves as a new organization in January of this year.
0:52
SPEAKER_02: There were a couple of predecessor organizations,
0:56
SPEAKER_02: Taylor Geospatial Institute and Taylor Geospatial Engine.
1:01
SPEAKER_02: And we were obviously funded by the same philanthropist.
1:05
We took a look at what we were doing across all of that and
1:09
actually created a
1:10
SPEAKER_02: brand new organization.
1:13
We,
1:14
as we were going along,
1:15
we thought it was a merger and then we realized it's new.
1:21
We're a new organization with a really acting as a focused
1:25
research organization.
1:27
So we have a problem we're focused on solving and that is
1:32
unlocking the bottleneck
1:34
SPEAKER_02: that there is on
1:36
all of the information held in the vast amounts of satellite
1:38
imagery that's been collected.
1:41
And as I think everyone in this room and probably your whole
1:44
audience knows,
1:45
we need the machines to help us get all of that information out
1:49
of satellite
1:50
SPEAKER_02: imagery.
1:50
SPEAKER_02: So we are focused on the application of AI,
1:54
SPEAKER_02: computer vision,
1:55
SPEAKER_02: and machine learning to satellite imagery to create global data sets,
2:01
SPEAKER_02: which is an important fact,
2:03
or important feature of what we're doing and doing all that for
2:07
the digital public good.
2:09
SPEAKER_02: So everything we do is made open.
2:11
SPEAKER_01: Wonderful.
2:13
SPEAKER_01: I recently did an episode on Fields of the World.
2:15
SPEAKER_01: That's your product as well.
2:18
How about a small intro to that for the people that might not
2:21
be familiar with it?
2:23
SPEAKER_02: Well, Fields of the World was our first...
2:26
SPEAKER_02: effort at playing out our mission.
2:29
SPEAKER_02: And so we started that project a couple of years ago,
2:34
SPEAKER_02: actually,
2:35
SPEAKER_02: with the intention to create a global data set of field boundaries.
2:40
SPEAKER_02: So field boundaries wasn't our end-all and be-all.
2:44
SPEAKER_02: It was our first step.
2:46
And we built the teams and the tech amazingly in what we feel
2:51
was a pretty short
2:52
SPEAKER_02: amount of time to create a global data set of field boundaries.
2:56
right and what are the unique challenges when you go from a
3:00
local to more global
3:02
kind of task Isaac do you want to take that one yeah there's
3:08
there's way too many I
3:10
think yeah I think what you see is generalization issues so
3:14
it's very much like
3:15
machine learning 101 you know you're you're training on some in
3:20
distribution areas
3:22
SPEAKER_00: and then you're doing
3:22
SPEAKER_00: and doing some out-of-distribution,
3:25
SPEAKER_00: large-scale predictions where you don't actually have label data.
3:28
SPEAKER_00: And so in our case,
3:30
we had to do a huge effort just to make the benchmark dataset
3:34
by working with
3:35
SPEAKER_00: organizations,
3:36
SPEAKER_00: getting global,
3:37
SPEAKER_00: other localized field boundary data sets.
3:40
SPEAKER_00: Because if you look at the literature,
3:42
most of the existing field boundary delineation models from
3:46
satellite imagery,
3:47
SPEAKER_00: they tend to be very Eurocentric.
3:51
SPEAKER_00: France has really good field boundaries.
3:54
SPEAKER_00: Some other countries have very good field boundaries data sets,
3:58
but they don't work when you try to predict on the lower
4:02
latitudes in South Africa
4:03
SPEAKER_00: or Brazil,
4:06
SPEAKER_00: for example.
4:06
SPEAKER_00: And so
4:07
We first had to do that effort of tackling how do we get a
4:12
dataset that is
4:13
sufficiently large enough that it can generalize to different
4:16
areas before we even
4:19
SPEAKER_00: consider predicting.
4:21
We went the whole route of spending a year on just getting
4:25
benchmark data and
4:27
SPEAKER_00: making sure the dataset is clean and high quality.
4:30
and doing a bunch of evaluations and experiments just to test
4:33
generalization and
4:34
SPEAKER_00: what countries are hard to predict on,
4:37
SPEAKER_00: what should you use in your holdout set.
4:40
SPEAKER_00: And so trying to do the science right first before we consider doing this
4:45
SPEAKER_00: large-scale inference,
4:47
not to even consider there's all these parameters during
4:50
inference time that make
4:51
SPEAKER_00: it really hard to work across different locations.
4:54
SPEAKER_00: So yeah, I think it's not a surprise to anybody that the model
4:59
SPEAKER_00: doesn't work amazingly in certain areas.
5:01
SPEAKER_00: But I think the driver is that this is like version one.
5:04
SPEAKER_00: And we can clearly see there's improvements that need to be made.
5:10
SPEAKER_00: And so we're working on annotating,
5:12
getting collaborators to help annotate field boundaries in
5:15
areas that it does
5:16
SPEAKER_00: perform poorly or the labels are not available.
5:19
So there's a lot of more work that's even come out of just this
5:23
initial global run.
5:26
SPEAKER_01: Wonderful.
5:27
SPEAKER_01: And in terms of the objectives of Taylor Geospatial,
5:30
SPEAKER_01: how do you kind of map those out in terms of the global context?
5:34
Do you say we want each continent to kind of be balanced or
5:38
like how do you organize that?
5:41
SPEAKER_02: Do you mean in terms of like one data set, the field boundaries data set?
5:46
In terms of just the objectives of the project and it being a
5:49
global project?
5:54
SPEAKER_02: Well,
5:55
what we're really it's an interesting question because and I
6:00
think I think the
6:01
question actually gets at the way our field typically operates
6:08
our field being not
6:09
a field that you plant things in but our industry in that and
6:16
it's in this is
6:17
actually the other side of it is what we're working on so the
6:21
question that you're
6:22
SPEAKER_02: asking I think is
6:24
SPEAKER_02: Are we trying to focus in certain regions?
6:26
SPEAKER_02: Is that what you're asking?
6:27
Yeah,
6:28
just what the objectives are and what that means to say we're
6:31
doing global.
6:32
SPEAKER_02: Yeah, yeah.
6:33
SPEAKER_02: So I've been in this field my entire career.
6:37
And what I have seen is that all of the tech gets developed and
6:43
innovated around a
6:45
SPEAKER_02: use case,
6:45
SPEAKER_02: a set of use cases,
6:47
SPEAKER_02: a particular place in the world.
6:50
SPEAKER_02: and some of that is transferable to other places in the world,
6:54
SPEAKER_02: but we haven't gotten there with satellite imagery and AI yet.
7:00
SPEAKER_02: And so we really started from,
7:04
we know there are fields everywhere in the world and we know
7:08
there are people who
7:09
SPEAKER_02: want to do something with those fields everywhere in the world,
7:12
SPEAKER_02: whether it's in Kenya
7:15
and a coffee farmer needs to prove they didn't deforest their
7:20
land so that they can
7:22
continue to sell their crop or whether it's looking at
7:27
large-scale farming that's
7:29
SPEAKER_02: happening in another country for drought purposes.
7:34
SPEAKER_02: So what we really did was not ask the question,
7:37
SPEAKER_02: what use case are we solving for or what part of the world?
7:39
SPEAKER_02: We said, what can we contribute
7:43
SPEAKER_02: to this body of work,
7:46
SPEAKER_02: to our field,
7:46
SPEAKER_02: to the innovation and technology that's available,
7:50
that would lower the barrier of entry for other practitioners
7:54
who are going to
7:55
SPEAKER_02: answer their specific use case for their specific region.
8:00
SPEAKER_02: And what I saw at Planet was,
8:05
SPEAKER_02: I was at Planet for a number of years,
8:07
and what I saw was that we could only solve a problem if a
8:10
customer wanted us to
8:11
SPEAKER_02: solve the problem.
8:13
SPEAKER_02: And when you're in companies,
8:15
SPEAKER_02: not just Planet,
8:16
SPEAKER_02: but in companies that are publicly traded or have to make a profit,
8:22
there's not the incentive to do something globally that you can
8:26
keep using over and
8:27
SPEAKER_02: over again.
8:29
And what I realized over a number of years is that there isn't
8:34
an organization that
8:35
SPEAKER_02: is in a position to do that.
8:37
SPEAKER_02: If there was one,
8:39
SPEAKER_02: I think that,
8:40
SPEAKER_02: and this is the hypothesis of this whole organization,
8:42
is that we can actually help the whole industry move forward
8:46
and the promise of all
8:48
the investment that's happened in the proliferation of
8:51
satellites that are out
8:52
SPEAKER_02: there.
8:53
SPEAKER_02: You know,
8:54
SPEAKER_02: that promise wasn't,
8:56
SPEAKER_02: hey,
8:56
SPEAKER_02: give us,
8:56
SPEAKER_02: you know,
8:57
SPEAKER_02: to venture capital,
8:58
SPEAKER_02: give us a bunch of money so we can put data collectors in space.
9:01
SPEAKER_02: It was, give us this funding so that we can answer all these problems.
9:06
SPEAKER_02: But getting data collectors in space is very expensive.
9:10
SPEAKER_02: And so the next problem set,
9:12
SPEAKER_02: which is what we're here to help push along,
9:17
is let's answer some of those things globally that aren't
9:21
unique to a particular
9:23
use case or unique to a particular region so that the people
9:26
with special questions
9:27
SPEAKER_02: and problems can work on those.
9:30
SPEAKER_02: And that's why we're doing it in the open.
9:33
SPEAKER_02: You know,
9:34
we believe there's some foundational data that is relevant to
9:39
answering tons of
9:40
SPEAKER_02: problems,
9:40
SPEAKER_02: whether it's climate change,
9:42
SPEAKER_02: food security,
9:45
SPEAKER_02: you know,
9:45
SPEAKER_02: deforestation,
9:47
SPEAKER_02: those kind of things.
9:47
SPEAKER_02: So we really truly are not focused on a particular use case.
9:53
SPEAKER_02: However,
9:55
SPEAKER_02: now that we've created a global data set,
9:59
we are going to focus on those things with particular
10:01
stakeholders and particular
10:02
SPEAKER_02: users because
10:04
making the dataset better for their needs will feed back in to
10:10
improving the
10:10
SPEAKER_02: dataset everywhere.
10:11
SPEAKER_02: So our hypothesis was,
10:14
SPEAKER_02: let's do it global,
10:15
SPEAKER_02: tune it,
10:17
SPEAKER_02: again,
10:17
SPEAKER_02: not in the machine learning sense,
10:19
SPEAKER_02: but figure out what actual users need from what we've put out there,
10:24
SPEAKER_02: improve that,
10:24
SPEAKER_02: and have that cycle continue.
10:27
right and obviously field boundaries are essential for any kind
10:30
of crop related
10:32
task could be you know detection of what crop is growing or
10:35
health etc so that's
10:38
why that makes total sense I guess Isaac maybe coming back to
10:41
you you mentioned
10:43
that the first step was finding the data that was available
10:46
what did that look like
10:47
SPEAKER_01: was that global already or was it quite patchy
10:51
SPEAKER_00: It's definitely quite patchy, very messy.
10:55
SPEAKER_00: As you can imagine,
10:56
different countries and organizations have different levels of
11:01
quality for their
11:01
SPEAKER_00: label datasets.
11:03
SPEAKER_00: And some have crop types, some don't.
11:07
And so there is an inherent noise in the benchmark dataset that
11:11
is just unfortunately
11:13
SPEAKER_00: there.
11:14
I think you find that very common in a lot of remote sensing
11:17
benchmark datasets.
11:19
SPEAKER_00: And so the goal really is if I have a sufficient amount of labeled data,
11:24
a machine learning model will just learn through the noise and
11:26
predict whatever
11:28
SPEAKER_00: data-in-law.
11:43
SPEAKER_00: from a lot of conversations that we've had since we've done the release.
11:47
I've heard it get compared to the equivalent of OpenStreetMap
11:51
for field boundaries
11:52
SPEAKER_00: where we have this global data set.
11:54
SPEAKER_00: We're working actually on annotation and editing tools.
11:58
Hannah Kerner runs an organization that teaches people how to
12:02
annotate field
12:03
SPEAKER_00: boundaries correctly since it actually is quite particular.
12:06
SPEAKER_00: And we get compared to OpenStreetMap where you can actually edit.
12:10
SPEAKER_00: It's like community
12:13
you know annotated and over time and so over time we find that
12:18
a lot of people
12:20
notice that oh in this area like I noticed you missed some
12:23
field boundaries I'd
12:24
like to like annotate them to add to the data set for everybody
12:27
to use and I think
12:28
that that's really inspiring for us because somebody had to
12:34
like do it first before
12:36
to kind of like bring everyone to to come and start annotating
12:39
for free as a
12:40
SPEAKER_00: community effort
12:42
And we will be retraining a new model based on a lot of these
12:47
annotations that
12:48
SPEAKER_00: hopefully will be even better.
12:49
SPEAKER_00: We'll do more runs.
12:51
SPEAKER_00: We'll do some backfilling of prior years as well.
12:54
SPEAKER_00: And that unlocks a lot of applications, right?
12:57
You see a lot of times global deforestation datasets or
13:00
above-ground biomass
13:01
SPEAKER_00: datasets that are annual that you can do change over time.
13:05
There isn't in existence a field boundary dataset that we can
13:10
do this with.
13:11
SPEAKER_00: the global scale lets us get some kind of number.
13:14
SPEAKER_00: Even if it's an estimate, it's not perfect.
13:17
It lets us have some kind of idea of what we're dealing with at
13:21
global scale
13:22
SPEAKER_00: instead of just,
13:23
SPEAKER_00: as you described,
13:24
SPEAKER_00: it's very patchy.
13:25
SPEAKER_00: Certain countries,
13:26
SPEAKER_00: predominantly first world countries,
13:28
SPEAKER_00: third world countries don't have it as good label data.
13:31
So I think we just don't want it to be as biased towards
13:35
certain areas and be
13:37
SPEAKER_00: global and openly licensed for everyone to use.
13:40
SPEAKER_01: Yeah.
13:41
SPEAKER_01: And just so I could understand how you kind of selected the Verse 1 data,
13:45
did you have a big map and look at the coverage of your labels
13:48
over a map?
13:49
SPEAKER_01: What kind of methodology was there to say,
13:51
SPEAKER_01: yes,
13:52
SPEAKER_01: okay,
13:52
we're happy that we now covered the areas that we wanted to
13:56
cover to the right
13:57
SPEAKER_01: extent?
13:58
yeah I think it was it wasn't like a single time it was
14:01
definitely very iterative
14:02
so like as you're collecting data you're running experiments
14:06
and running predicting
14:07
on holdout countries to see like what the performance is and
14:12
you're visually
14:13
SPEAKER_00: inspecting
14:14
SPEAKER_00: quantitatively and qualitatively.
14:16
Because there's a lot of issues with if you're just comparing
14:19
pixel IOU versus how
14:22
SPEAKER_00: after polygonization,
14:23
SPEAKER_00: do the polygons look good or not?
14:26
SPEAKER_00: Are they sufficient or not?
14:27
So yeah,
14:27
there's a lot of very particular details that you have to
14:31
figure out.
14:32
SPEAKER_00: But yeah, it's definitely more of an iterative process.
14:35
SPEAKER_02: And I would like to add something there and to say it's not good.
14:41
SPEAKER_02: right?
14:42
The first time I saw the map of the benchmark training data
14:48
set,
14:49
I was like,
14:49
this isn't global.
14:51
SPEAKER_02: What is this?
14:53
SPEAKER_02: It's not enough.
14:54
And so that's one point we wanted to make here today is this
14:59
was our first pass at
15:01
SPEAKER_02: both,
15:02
SPEAKER_02: even though like Isaac said,
15:03
SPEAKER_02: it was iterative of creating the benchmark data set,
15:08
SPEAKER_02: experimenting with a whole bunch of models,
15:10
SPEAKER_02: it,
15:11
SPEAKER_02: publishing it,
15:13
but we're immediately going back and going to expand that
15:16
benchmark dataset
15:18
SPEAKER_02: massively.
15:19
SPEAKER_02: It's not enough.
15:22
But the success there is that we have all those component
15:26
pieces in place that we
15:27
SPEAKER_02: know how to add to the dataset.
15:30
SPEAKER_02: We know where we want to add to it.
15:33
SPEAKER_02: We've tested tons.
15:34
SPEAKER_02: And how many models did we evaluate?
15:36
SPEAKER_02: I mean, at least a couple dozen.
15:41
um so you know we're really looking at now where where can we
15:45
improve along all
15:46
those moving parts um the output plus having the feedback loop
15:52
um from ground truth
15:54
so that that was the first year that got you to fill the world
15:57
version one and then
15:58
uh the recent release was version two which I think primarily
16:02
addressed South
16:03
SPEAKER_01: America
16:06
SPEAKER_00: Yeah,
16:07
SPEAKER_00: so we haven't done a global run with that,
16:10
SPEAKER_00: but yes,
16:11
the work that I believe WRI and Hannah were working on was for
16:17
South America in
16:18
SPEAKER_00: particular,
16:18
SPEAKER_00: and that will be added to the new version 2 of the benchmark dataset,
16:22
SPEAKER_00: which has not
16:23
has not been released yet so that is that is getting at like
16:27
wrote back in because
16:29
we we do have a lot of stakeholders in South America that are
16:33
very interested the
16:34
fields are wildly different than in other locations and so that
16:39
in particular is
16:40
SPEAKER_00: very important for like global food security mm-hmm
16:44
SPEAKER_01: So,
16:44
SPEAKER_01: Jen,
16:45
SPEAKER_01: I think how you see this working is that you put the data out there,
16:48
SPEAKER_01: people adopt it,
16:50
SPEAKER_01: identify which parts of the world maybe it's not forming so well,
16:54
and they then engage to bring up the data set and kind of
16:59
balance things out
16:59
SPEAKER_01: better.
17:01
Yes,
17:02
at the same time that we are doing things on the technical side
17:05
that we already know.
17:08
SPEAKER_02: When we put that data set out, one of the questions we kept asking is,
17:12
How do we put this out in a way that also lets people know we
17:16
know where the weaknesses are?
17:18
And I would say less,
17:23
well,
17:23
it's both parts of the world,
17:25
but it's also use case.
17:30
SPEAKER_02: Some use cases, you don't really need a perfect boundary.
17:34
In some use cases,
17:35
you just want to look across a whole bunch of fields and do
17:39
analytics on you know a bigger scale other use cases you know
17:44
like a precision
17:45
farming use case is the best one to say like we're not there
17:48
yet at all if you're
17:49
trying to look at like how much pesticide you put in a
17:52
particular field so we're
17:55
we're engaging folks who have particular use cases that either
18:00
where it's a good
18:01
SPEAKER_02: fit for or not so that we can learn from that
18:05
SPEAKER_01: And possibly a technical sort of point I'd like to explore there.
18:08
Do different locations have sort of different objectives or
18:12
different measures that
18:13
SPEAKER_01: are meaningful to them?
18:14
So I can imagine if you have lots of small fields that you
18:17
might have different
18:18
SPEAKER_01: objectives to having just large fields,
18:20
SPEAKER_01: for instance.
18:22
SPEAKER_00: Yeah,
18:22
SPEAKER_00: we definitely record metrics that also break it down,
18:26
SPEAKER_00: similar to object detection where you have average precision for large,
18:30
SPEAKER_00: medium,
18:30
SPEAKER_00: small.
18:31
We do similar metrics and analysis based on hectares,
18:36
like area of the field.
18:38
um so that that is an important piece because um you know we're
18:42
predicting at 10
18:42
meter per pixel so uh in a lot of cases like a field boundary
18:46
on very very tiny
18:47
small holder fields is not even visible really so or if you
18:51
look at like the
18:51
central 2 image it's a pixel um even not so I think like yeah
18:57
there there are
18:58
SPEAKER_00: limitations with central 2 um
19:01
and we are looking for partners for higher resolution imagery
19:07
to rerun on the same
19:09
SPEAKER_00: the same labels and
19:11
SPEAKER_00: do some more targeted runs,
19:12
SPEAKER_00: I think,
19:13
of certain countries that have a high amount of smallholder
19:16
fields to get those
19:17
SPEAKER_00: right.
19:18
SPEAKER_00: Because I think,
19:19
SPEAKER_00: yeah,
19:19
what you normally see with the smaller fields is you see that
19:23
they'll either get
19:23
merged together into one large field or they get broken up into
19:28
significantly
19:29
SPEAKER_00: smaller fields.
19:31
SPEAKER_00: But yeah,
19:31
SPEAKER_00: as Jen mentioned,
19:33
we've seen a lot of stakeholders and people that have reached
19:37
out to us that
19:38
SPEAKER_00: actually are not
19:40
they don't care too much about like the precision per boundary
19:43
like they kind of
19:44
care more about like area and estimates at that scale so that
19:49
they can make these
19:49
like larger sort of higher level claims of like their
19:53
percentage of you know land
19:56
in a certain area is like primarily field so it's not always
20:00
that everyone is using
20:02
SPEAKER_00: the field boundaries themselves like
20:04
um we have some folks who have been using like the raw softmax
20:09
you know probability
20:10
predictions of the semantic segmentation model that we store in
20:13
Coggins r format so
20:15
there because we release all of that it actually allows people
20:18
to you know come up
20:19
with their own polygonization algorithm and that's kind of what
20:21
we wanted is we
20:22
wanted to release all the intermediate artifacts um because
20:25
we're not prescribing
20:26
SPEAKER_00: you know this polygon is a polygonization method is the best um or this
20:30
SPEAKER_00: thresholding method is the best like we want
20:32
to allow to just have it fully transparent open so anyone can
20:35
kind of pick their
20:36
SPEAKER_00: own parameters.
20:39
I'm really curious to understand you mentioned the different
20:41
kind of layers to the product.
20:43
SPEAKER_01: There's obviously a kind of web interface where you can see the polygons.
20:47
SPEAKER_01: You mentioned that you can actually get the outputs of the models.
20:50
Can you just briefly describe all of those layers so we can
20:52
just get an idea of the
20:53
SPEAKER_01: richness of what was released?
20:55
yeah so we we initially have these so the model takes in
20:59
planting and harvest
21:00
season imagery from central to that is selected based on like
21:06
the location based on
21:07
the world cereal crop calendar and that's because different
21:12
areas are going to have
21:13
different planting harvest seasons and so if you if you just
21:16
pick one date
21:17
basically it's not going to be the correct like a crop will be
21:21
harvested there are
21:23
SPEAKER_00: some details of like you know
21:25
Having a single crop calendar doesn't really work across all
21:29
different types of annual crops.
21:30
SPEAKER_00: Um, but it is like a good starting point.
21:32
SPEAKER_00: You have to pick something to start with.
21:35
SPEAKER_00: Um, so we do release the Sentinel-2, uh, global mosaics in
21:41
SPEAKER_00: So we have per tile,
21:42
SPEAKER_00: per UTM zone,
21:44
but we also have a global czar that has all of the planting and
21:47
harvest season
21:48
SPEAKER_00: bands in there.
21:50
SPEAKER_00: We do that for two years, so the 2024 and 2025.
21:53
SPEAKER_00: That's all in Source Cooperative.
21:54
SPEAKER_00: So all of our data is in Source Cooperative S3 buckets.
21:57
It's not request or pay,
21:58
so you can query it as much as you want for free anonymously.
22:02
SPEAKER_00: And so it's all partitioned with overviews and multiscale
22:08
um and then we have the raw predictions of the model per pixel
22:12
um at 10 meter
22:13
resolution in the same format so Cog and Czar um and so and
22:17
then from there we
22:18
SPEAKER_00: polygonize those predictions for the field
22:23
SPEAKER_00: category in particular into Geoparquet, into the geometries.
22:28
SPEAKER_00: And then we have the raw predictions there.
22:30
And then in the paper that's led by Caleb for the global field
22:34
boundary data set,
22:35
SPEAKER_00: we do a bunch more interesting,
22:37
SPEAKER_00: like a confidence layer and more filtering.
22:39
SPEAKER_00: Because initially, like there's about 3 billion
22:42
SPEAKER_00: polygons in our unfiltered raw dataset.
22:45
SPEAKER_00: But after you do filtering, it comes down to about 860 million per year.
22:51
And so that seems more reasonable compared to other global
22:56
estimates that have been
22:57
SPEAKER_00: made.
22:58
SPEAKER_00: And so all of this is out there.
23:00
SPEAKER_00: Geoparquet is chunked very nicely.
23:02
And the interesting thing is our Explorer dashboard is just a
23:05
static front-end.
23:06
SPEAKER_00: There's nothing particularly special about it.
23:11
SPEAKER_00: We built it.
23:11
SPEAKER_00: It has
23:13
SPEAKER_00: zoom levels.
23:14
SPEAKER_00: But all of that is built into what's on source co-op.
23:17
SPEAKER_00: So there's no backend that's hosting it.
23:19
SPEAKER_00: We're just streaming it from cloud-native sources.
23:21
So what you're seeing when you're zooming in is the PM tiles
23:24
version of the polygons.
23:26
SPEAKER_00: And then as you
23:27
You get to a particular zoom level,
23:28
it starts overlaying the polygons in your point of view.
23:31
SPEAKER_00: But this is all just streaming from cloud storage for free.
23:35
SPEAKER_00: So if you don't like the application or you have your own specific,
23:40
SPEAKER_00: you know,
23:41
SPEAKER_00: custom application you want to build,
23:42
there's nothing stopping you from like just taking the URL and
23:46
just building on top
23:47
SPEAKER_00: of that for your own case.
23:49
SPEAKER_01: Wonderful.
23:49
SPEAKER_01: So all of those assets are there.
23:51
SPEAKER_01: People can just access them,
23:52
SPEAKER_01: no cost,
23:53
SPEAKER_01: build their own front ends,
23:55
SPEAKER_01: process the data in a different way.
23:57
SPEAKER_01: Fantastic.
23:59
SPEAKER_02: I'd like to add something there if you don't mind,
24:01
SPEAKER_02: Robin,
24:03
SPEAKER_02: because everything Isaac just described,
24:07
SPEAKER_02: and he's talking about COGS and Czar and Source Cooperative,
24:13
SPEAKER_02: this is academic work,
24:15
SPEAKER_02: actually.
24:16
like the progress that we made was done with research teams in
24:21
universities and
24:23
SPEAKER_02: some researchers who are in companies.
24:26
And another thing that we decided to do with Taylor Geospatial
24:31
as a model because
24:34
of how our industry is sort of lagging behind the progress that
24:39
just generally in
24:40
SPEAKER_02: AI and computer vision for satellite imagery
24:43
We decided that we didn't want to wait for academic work to get
24:48
published and then
24:48
SPEAKER_02: someone to go replicate it and try it out and scale it up.
24:53
What we're doing as an organization is doing that as the
24:56
research is happening.
24:57
So we fund research teams,
25:00
but we also fund really talented geospatial technical engineer
25:07
types.
25:09
SPEAKER_02: to match up with the research teams.
25:12
SPEAKER_02: And as the researchers are doing work,
25:14
SPEAKER_02: we're kind of watching and saying,
25:16
SPEAKER_02: hey,
25:17
SPEAKER_02: can we put that in this format?
25:18
Because what I saw was you go and look at the artifacts of
25:22
research and you're
25:23
SPEAKER_02: like,
25:24
SPEAKER_02: those words make sense to me,
25:26
SPEAKER_02: but what I'm looking at isn't recognizable from industry standards.
25:31
So the fact that Isaac is saying COG and Czar and Source
25:35
Cooperative was a very
25:37
deliberate decision by our organization to make sure that's
25:40
happening right
25:41
SPEAKER_02: alongside the research.
25:42
SPEAKER_02: So papers get published on the same day if we're allowed to.
25:47
SPEAKER_02: We put this stuff out there for use at scale.
25:51
And that's one thing that I'm super proud of our organization
25:55
for doing because we
25:57
SPEAKER_02: can then really dig into the impact of the work
26:01
SPEAKER_02: and improve the research questions,
26:04
SPEAKER_02: really,
26:06
kind of hand-in-hand rather than those things being staggered
26:09
or kind of diverging,
26:11
SPEAKER_02: which I think often happens.
26:12
SPEAKER_01: Yeah,
26:13
I think that definitely addresses a real kind of common pain
26:15
point that academic
26:17
SPEAKER_01: breakthroughs don't often materialize into,
26:20
SPEAKER_01: you know,
26:20
real-world applications because of the friction of moving it
26:23
across the production
26:26
SPEAKER_01: setting.
26:27
SPEAKER_02: And I think that people criticize academics for that.
26:32
SPEAKER_02: and they shouldn't be criticized.
26:35
SPEAKER_02: That's the incentive structure that they are employed within requires.
26:41
SPEAKER_02: And the funding mechanisms usually say,
26:44
SPEAKER_02: oh,
26:45
SPEAKER_02: it should be commercializable,
26:46
SPEAKER_02: but that's like a check the box kind of thing,
26:49
SPEAKER_02: not a like someone really did it at scale.
26:53
SPEAKER_02: And so what we, from the beginning, our position was we can't expect them
27:00
to be able to understand all these formats and understand what
27:03
eventual users might want to do.
27:05
SPEAKER_02: They have to do what they do, and we have to augment that.
27:10
SPEAKER_02: And that part has been super fun, I think, for all of us.
27:15
SPEAKER_02: At first,
27:16
SPEAKER_02: Hannah Kerner was the first one that we kind of talked,
27:22
SPEAKER_02: and I was like,
27:23
SPEAKER_02: do you think we could do this?
27:24
SPEAKER_02: Would you be willing to try?
27:26
SPEAKER_02: In the beginning,
27:27
SPEAKER_02: I called her almost every week because I was like,
27:29
SPEAKER_02: are we stepping on your toes?
27:30
SPEAKER_02: Are we getting in your way?
27:32
SPEAKER_02: And it took a little bit of time.
27:33
SPEAKER_02: I mean, she always said we weren't.
27:35
SPEAKER_02: And it took her some time to then realize like,
27:37
SPEAKER_02: oh,
27:38
SPEAKER_02: this is something I can throw over to the tech fellows.
27:41
SPEAKER_02: And at the end of the first year,
27:43
SPEAKER_02: we had a meeting and she stood up and she was like,
27:45
SPEAKER_02: this is incredible.
27:46
SPEAKER_02: It's improved our ability to do the research.
27:50
And like,
27:50
I get chills talking about it because I think there's really
27:54
something there when
27:56
And it's when you let people kind of do what their lane is
28:00
designed to do.
28:02
SPEAKER_02: And that's been our philosophy also in all of our partnerships,
28:06
SPEAKER_02: you know,
28:06
SPEAKER_02: with Source Cooperative.
28:07
SPEAKER_02: Like, we're not going to recreate a place to store all this data.
28:09
We're going to partner with somebody who's doing that and has a
28:13
similar mission to us.
28:15
SPEAKER_02: And that's been, it's been really fun to see how that plays out.
28:19
SPEAKER_01: Yeah,
28:20
SPEAKER_01: and I think it's really delivered,
28:21
SPEAKER_01: you know,
28:21
with this first version of Fields of the World and everything
28:25
that comes with that,
28:26
SPEAKER_01: all the documentation,
28:27
SPEAKER_01: software,
28:28
SPEAKER_01: publications,
28:29
SPEAKER_01: it's really making a huge impact.
28:33
SPEAKER_01: I'd love to finish out just by understanding maybe what's next for Taylor
28:36
Geospatial and now that you've learned what you have and
28:39
established this way of
28:40
SPEAKER_01: working.
28:42
SPEAKER_02: Well, I touched on it a little bit, more fields.
28:46
SPEAKER_02: More fields of the world.
28:48
SPEAKER_02: We are going to expand that benchmark.
28:52
We are also going to expand the information we collect with it,
28:56
like crop type.
28:58
So we'll make that predictions and the output data richer and
29:03
along those lines
29:04
SPEAKER_02: because we have this work out there.
29:08
We're working with stakeholders to understand their needs and
29:12
loop that back in.
29:14
SPEAKER_02: But we are also kicking off three other threads of work.
29:20
SPEAKER_02: So one we're calling features of the world where,
29:24
SPEAKER_02: and this was our plan from the beginning,
29:26
SPEAKER_02: was,
29:27
SPEAKER_02: you know,
29:28
identify all these component parts in the ecosystem to create a
29:33
global data set and
29:35
SPEAKER_02: then come back in and say,
29:37
SPEAKER_02: okay,
29:37
SPEAKER_02: we're going to do a different feature type.
29:40
and how much can we use of what we've already created and how
29:44
much can we expand that?
29:46
How much work is it to do the second feature after we've sort
29:50
of learned what we've
29:50
SPEAKER_02: learned on the first global feature?
29:53
So we're right now in the process of deciding what those
29:57
features will be and we're
29:59
really thinking about what would be most impactful and what can
30:03
we accomplish in
30:06
SPEAKER_02: some timeframe that's reasonable.
30:09
so that's exciting and we have a team that's lined up to do
30:12
that and and that's
30:13
going to be a lot of fun and then the other the second project
30:19
that we're kicking
30:20
off that's new is around benchmarking really and Isaac can talk
30:25
about that in a
30:25
second so I'll turn that over to Isaac in a second but to me
30:29
that one's sort of not
30:31
SPEAKER_02: very sexy
30:33
but could have a very big impact on how the industry
30:37
understands the state of the art.
30:39
SPEAKER_02: And especially in uptake,
30:41
if we can provide information on what benchmark and model is
30:47
best for what use
30:48
SPEAKER_02: case,
30:49
SPEAKER_02: I think we'll see more uptake in industry.
30:53
SPEAKER_02: And then the third project is our most innovative.
30:56
And it's really kind of turning all of this on its head and
30:59
asking,
30:59
what if
31:00
What if just growing benchmark data sets forever and tweaking
31:05
these models in their
31:06
SPEAKER_02: current form isn't going to get us much more improvement?
31:09
SPEAKER_02: And maybe there's an entirely different way.
31:12
So we're also looking,
31:13
that's like kind of the biggest risk,
31:16
biggest reward area.
31:18
SPEAKER_02: But Isaac, do you want to talk about the benchmarking a little bit?
31:22
SPEAKER_00: Yeah, I'll spend a little bit of time.
31:24
So we just released a paper on Archive called No One Knows the
31:28
State of the Art in
31:29
SPEAKER_00: Geospatial Foundation Models,
31:31
SPEAKER_00: which basically details issues in current evals in GeoAI and
31:37
foundation models and issues with benchmarks and some
31:42
recommendations for what we
31:43
SPEAKER_00: should be doing in the future to get out of this rut that we're in.
31:49
This paper is actually a precursor to an evaluation toolkit
31:53
that we'll be releasing
31:55
SPEAKER_00: here soon,
31:57
SPEAKER_00: which tries to standardize.
31:58
I think one of the recommendations in the paper at the end is
32:02
that we need to have
32:04
SPEAKER_00: an independent third-party
32:06
SPEAKER_00: community-led evaluation toolkit that researchers can fork and use.
32:13
SPEAKER_00: And we're all using the same grading rubric, right?
32:18
SPEAKER_00: So that's sort of like what that's about.
32:20
SPEAKER_00: And I think
32:22
SPEAKER_00: We're planning to maintain it long-term,
32:24
which has been probably one of the major flaws in some other
32:28
benchmarking toolkits
32:30
SPEAKER_00: in geospatial AI because they're normally led by
32:36
searchers and students and then they move on to the next one so
32:39
it's not maintained
32:40
SPEAKER_00: long term and then everyone kind of just makes their own eventually.
32:43
SPEAKER_00: And I think that that needs to get fixed.
32:46
SPEAKER_00: The paper coming out though has been really huge.
32:48
SPEAKER_00: We've had a lot of people reach out and talk about specific
32:51
and specifically creating more curated evaluation benchmark
32:57
datasets that have
32:58
SPEAKER_00: better translation to real-world applications.
33:01
I think a lot of the benchmark datasets we've used in the past
33:04
don't have this
33:05
SPEAKER_00: actual translation
33:07
are more of like a sanity check and have kind of been saturated
33:10
as far as like
33:11
SPEAKER_00: accuracy and performance.
33:12
SPEAKER_00: So we need to sort of pivot.
33:14
SPEAKER_00: Those are all great and great to still benchmark on,
33:16
but we do need to pivot into very specific tasks like for
33:21
agriculture and forestry
33:22
SPEAKER_00: and all urban planning and other applications that people care about.
33:25
SPEAKER_00: Because I think in industry,
33:27
People don't care about,
33:29
like on average,
33:29
does something do well on Eurostat or Big EarthNet?
33:31
They probably care more about their particular case that
33:35
they're working on or task.
33:36
SPEAKER_00: So that's really what that's trying to solve.
33:38
SPEAKER_01: Wonderful.
33:39
Well,
33:39
these are wonderful and I think very meaningful initiatives
33:43
that you're working on there.
33:45
Thank you again for joining me today to discuss them and
33:48
looking forward to seeing
33:49
SPEAKER_01: the publications and the future work coming.
33:52
SPEAKER_00: Awesome.
33:52
SPEAKER_01: Thanks for having us, Rob.
33:54
SPEAKER_01: Bye, Deb.
33:55
SPEAKER_02: Bye.