Latest / Satellite image deep learning / Field boundary detection with Segment Anything
Transcript
- 0:05SPEAKER_01: Hello, I'm Robin Cole and welcome to the Satellite Image Deep Learning Podcast.
- 0:09SPEAKER_01: In this episode,
- 0:10SPEAKER_01: I caught up with Samuel Bancroft to learn about segmenting field boundaries using
- 0:14SPEAKER_01: Segment Anything,
- 0:15SPEAKER_01: also known as SAM.
- 0:17SPEAKER_01: SAM is a foundational model for vision released by Meta, which is capable of zero-shot segmentation.
- 0:24SPEAKER_01: However, there are many open questions about how to make use of SAM with remote sensing imagery.
- 0:29SPEAKER_01: In this conversation,
- 0:30SPEAKER_01: Samuel describes how he used SAM to perform segmentation of field boundaries with
- 0:35SPEAKER_01: Sentinel-2 imagery over the UK.
- 0:37SPEAKER_01: His best results were obtained not by fine-tuning SAM,
- 0:40SPEAKER_01: but by carefully pre-processing a time series of images into HSV color space and
- 0:46SPEAKER_01: using SAM without any modifications.
- 0:48SPEAKER_01: This is a surprising result,
- 0:50SPEAKER_01: and using this kind of approach significantly reduces the amount of work necessary
- 0:54SPEAKER_01: to develop useful remote sensing applications utilizing SAM.
- 0:58SPEAKER_01: I hope you enjoy this episode.
- 1:01SPEAKER_01: Hi, Sam.
- 1:01SPEAKER_01: Welcome to the podcast.
- 1:03SPEAKER_00: Hiya.
- 1:03SPEAKER_00: Thanks for inviting me.
- 1:05SPEAKER_01: It's my pleasure.
- 1:05SPEAKER_01: Let's begin with where you work and what you do.
- 1:08SPEAKER_00: Sure.
- 1:10SPEAKER_00: I'm a PhD student at the University of Leeds.
- 1:12SPEAKER_00: I've been doing my PhD since 2020, and it's all been about crop type classification.
- 1:18SPEAKER_00: So like the rest of machine learning, you're always on the hunt for more and more labels.
- 1:23SPEAKER_00: Crop type classification is no different.
- 1:26SPEAKER_00: So I'm trying to develop new machine learning methods that rely less on the labels
- 1:32SPEAKER_00: that are very expensive to acquire and just making things that are more
- 1:35SPEAKER_00: generalizable so I can...
- 1:38SPEAKER_00: do big mapping campaigns and work out where the crops are growing across the whole country.
- 1:42SPEAKER_00: Right, so that's the goal.
- 1:44SPEAKER_00: Yeah, that's the goal.
- 1:45SPEAKER_01: Limited data or annotated data, I know I end.
- 1:48SPEAKER_01: A model that would generalize across the whole country, in this case, the United Kingdom.
- 1:52SPEAKER_01: Okay, well, where do we begin?
- 1:53SPEAKER_01: What solution have you come up with?
- 1:56SPEAKER_00: Sure.
- 1:56SPEAKER_00: So it's been a long process, but I'm talking to you today about UK fields.
- 2:03SPEAKER_00: So that's a data set containing millions of field boundaries across England,
- 2:08SPEAKER_00: Scotland,
- 2:09SPEAKER_00: Wales and Northern Ireland.
- 2:11SPEAKER_00: That has been in collaboration with Jake Wilkins, who I believe has been on this podcast previously.
- 2:17SPEAKER_00: So we started a conversation about using Meta or Facebook's new machine learning
- 2:23SPEAKER_00: model that came out April last year called Segment Anything.
- 2:26SPEAKER_00: And a lot of people have tried to use that to apply to Earth observation.
- 2:30SPEAKER_00: And we've applied that to the UK to create field boundaries.
- 2:34SPEAKER_01: Right.
- 2:34SPEAKER_01: And what are some of the limitations that SAM has?
- 2:40SPEAKER_00: Sure.
- 2:40SPEAKER_00: So...
- 2:43SPEAKER_00: SAM is built to,
- 2:46SPEAKER_00: or Segment Anything,
- 2:47SPEAKER_00: otherwise known as SAM,
- 2:49SPEAKER_00: is built to segment a wide range of many different things from people in
- 2:53SPEAKER_00: photographs to cats and dogs and road signs.
- 2:56SPEAKER_00: It's not necessarily adapted to aerial imagery,
- 2:59SPEAKER_00: although perhaps aerial imagery was part of the original training set.
- 3:04SPEAKER_00: It's also built around
- 3:06SPEAKER_00: rgb red green blue images and we all know that earth observation images don't
- 3:10SPEAKER_00: necessarily need to just be rgb so when we take a big foundation we'll like segment
- 3:17SPEAKER_00: anything and we apply it to earth observation we don't necessarily get the results
- 3:20SPEAKER_00: we want so we want to do feature engineering or fine tuning and stuff like that to
- 3:24SPEAKER_01: bring it more into our domain interesting so obviously you want to apply sam to the
- 3:28SPEAKER_01: the boundary detection what's some of the stages that you went through in your
- 3:31SPEAKER_01: experimentation and where did you arrive at
- 3:35SPEAKER_00: So the story actually began at a workshop last year that I attended and some
- 3:42SPEAKER_00: academics from a university in Germany set out with this initial task of how can we
- 3:49SPEAKER_00: apply segment anything
- 3:51SPEAKER_00: to field detection.
- 3:53SPEAKER_00: And I think my original attempt was to look at things like NDVI and edge detection
- 3:59SPEAKER_00: and feed that in,
- 4:00SPEAKER_00: stuff like that.
- 4:01SPEAKER_00: We also tried to fine tune the model, which I can talk a little about that later as well.
- 4:06SPEAKER_00: And there was some success,
- 4:09SPEAKER_00: particularly lots of success in like easy agricultural landscapes like the US
- 4:14SPEAKER_00: Midwest where the fields are really,
- 4:16SPEAKER_00: really obvious.
- 4:17SPEAKER_00: But when I applied it to the UK, just for my own personal bias,
- 4:21SPEAKER_00: It didn't do well across the board,
- 4:23SPEAKER_00: not well enough to outcompete the traditional field boundary detection methods that
- 4:28SPEAKER_00: are out there.
- 4:28SPEAKER_01: So North America, the fields are very large and irregular grids in the UK.
- 4:33SPEAKER_01: They're maybe not so regular.
- 4:35SPEAKER_01: Is that the case?
- 4:36SPEAKER_00: Definitely.
- 4:36SPEAKER_00: Yes.
- 4:38SPEAKER_00: Very flat, very big fields in the US.
- 4:42SPEAKER_00: across huge areas in the uk it can change very very quickly particularly between
- 4:47SPEAKER_01: that urban and rural areas right okay very interesting and what data did you use
- 4:52SPEAKER_01: for this or did you restrict yourself to any one data source sure so uh the uk
- 4:57SPEAKER_00: fields data set as is released um is built upon sentinel 2 imagery uh mainly
- 5:03SPEAKER_00: because it's a framework
- 5:05SPEAKER_00: hopefully anyone will want to use for Sentinel-2 and it gets good high quality
- 5:09SPEAKER_00: field boundaries at the 10 meter scale.
- 5:12SPEAKER_00: That doesn't necessarily mean that you just need to use Sentinel-2.
- 5:15SPEAKER_00: We're releasing the code so you can just swap out
- 5:19SPEAKER_00: for any other data source.
- 5:23SPEAKER_00: So for example, we've been playing around with planet imagery as well.
- 5:26SPEAKER_00: It works with some quite interesting results at the five meter scale as well.
- 5:30SPEAKER_01: Very interesting.
- 5:31SPEAKER_01: Do you want to talk us through the process of using maybe Sentinel to begin with?
- 5:35SPEAKER_01: And maybe we can start at the beginning, how you use multispectral data with a three channel model.
- 5:42SPEAKER_01: Sure.
- 5:43SPEAKER_00: Yes.
- 5:43SPEAKER_00: Should I bring up my slides?
- 5:45SPEAKER_00: Yeah, go for it.
- 5:46SPEAKER_00: Definitely.
- 5:47SPEAKER_00: Okay.
- 5:55SPEAKER_00: All right, so to begin with the.
- 6:00SPEAKER_00: solution to how to make UK fields work consistently across the whole country.
- 6:05SPEAKER_00: very much lies on making harmonic composites now this isn't a completely fresh idea
- 6:12SPEAKER_00: harmonic composites exist and existed a long time.
- 6:16SPEAKER_00: for a variety of different reasons.
- 6:18SPEAKER_00: I've used Harmonix to do crop type classification in an unsupervised way,
- 6:24SPEAKER_00: so using no labels whatsoever,
- 6:25SPEAKER_00: trying to identify distinct different crop types automatically.
- 6:32SPEAKER_00: And you can do this in Google Earth Engine relatively easily.
- 6:36SPEAKER_00: And other people, no doubt, have been using this with their own field delineation algorithms.
- 6:41SPEAKER_01: Just to clarify for the readers,
- 6:42SPEAKER_01: the listeners,
- 6:43SPEAKER_01: sorry,
- 6:43SPEAKER_01: the harmonics are fitted to a time series,
- 6:46SPEAKER_01: essentially.
- 6:48SPEAKER_00: Yes.
- 6:49SPEAKER_00: So what you're looking at here is a little bit of a kaleidoscopic image,
- 6:54SPEAKER_00: but it's representing a time series of images.
- 6:57SPEAKER_00: And how that's done is it's taking a vegetation index,
- 7:01SPEAKER_00: in this case NDVI,
- 7:02SPEAKER_00: across a 12-month growing period.
- 7:05SPEAKER_00: And it's fitting a harmonic model to that time series.
- 7:09SPEAKER_00: And from that harmonic model,
- 7:11SPEAKER_00: from each pixel,
- 7:12SPEAKER_00: you can get the phase,
- 7:14SPEAKER_00: the amplitude,
- 7:15SPEAKER_00: and the mean NDVI of that particular pixel.
- 7:20SPEAKER_00: And we map that to the HSV hue saturation value color space.
- 7:24SPEAKER_00: So the phase, because it's cyclical,
- 7:29SPEAKER_00: can be the hue, saturation can be the amplitude, and value can be the mean NDVI.
- 7:35SPEAKER_00: And once you pop that and encode that as this kind of static image here,
- 7:40SPEAKER_00: you get loads of different colors,
- 7:41SPEAKER_00: right?
- 7:42SPEAKER_00: And the hope is that something like this is much more easily interpretable by
- 7:48SPEAKER_00: segment anything than median RGB composites just in a normal way.
- 7:55SPEAKER_01: At a high level, we can see the boundaries are quite clear there.
- 7:59SPEAKER_01: I suppose in any one time, just in RGB, the boundaries wouldn't be quite so clear.
- 8:07SPEAKER_00: Yes, definitely.
- 8:09SPEAKER_00: There's a few good examples about just using an RGB composite and comparing it to
- 8:14SPEAKER_00: this and seeing the accuracy jump up a little bit.
- 8:16SPEAKER_00: So what we did, actually, is we tested Segment Anything just with normal color imagery.
- 8:23SPEAKER_00: compared to the harmonic approach.
- 8:24SPEAKER_00: And it bumps up what's known as the dice score, which is a nice accuracy metric.
- 8:30SPEAKER_00: It just bumps it up by 0.2 on average.
- 8:34SPEAKER_00: So a significant improvement, basically.
- 8:36SPEAKER_00: So it really does fix a lot of the problems with a few of the odd fields that don't
- 8:40SPEAKER_00: get segmented properly.
- 8:42SPEAKER_01: Okay.
- 8:42SPEAKER_01: So the basic process is take your time series over 12 months,
- 8:46SPEAKER_01: generate one of these HSV images,
- 8:48SPEAKER_01: and then pass it through,
- 8:50SPEAKER_01: segment anything without any kind of fine tuning or modification for this situation.
- 8:57SPEAKER_01: Yeah.
- 8:58SPEAKER_00: Fantastic.
- 8:59SPEAKER_00: It's a good example of don't reinvent the wheel,
- 9:03SPEAKER_00: as perhaps I tried to do when I initially set out to do this and just rely a little
- 9:08SPEAKER_00: bit on feature engineering.
- 9:09SPEAKER_01: Right.
- 9:10SPEAKER_01: Excellent.
- 9:11SPEAKER_01: And how do the results, metrically speaking, look across the UK?
- 9:17SPEAKER_00: Sure.
- 9:17SPEAKER_00: So if I bring up the right slides.
- 9:23SPEAKER_00: These are results scoring against the next best available ground truth is validation data.
- 9:31SPEAKER_00: So we've taken our UK fields data set.
- 9:35SPEAKER_00: We've split it up into the grid cells and worked out the average
- 9:40SPEAKER_00: dice score,
- 9:41SPEAKER_00: intersection over union,
- 9:42SPEAKER_00: under-segmentation and over-segmentation rates compared to the Ordnance Survey.
- 9:46SPEAKER_00: So that's the national mapping agency,
- 9:48SPEAKER_00: government agency for the UK,
- 9:51SPEAKER_00: compared to the fields in the MasterMap dataset that they've got available.
- 9:57SPEAKER_00: So for those thinking about harmonic composites already, you'll notice that the fields that
- 10:04SPEAKER_00: strongly agricultural that are growing crops that have very distinct growing curves
- 10:08SPEAKER_00: will do very very nicely in a harmonic composite and you can see that reflected
- 10:11SPEAKER_00: here you can see the east coast uh big growing regions of the uk get very nice
- 10:16SPEAKER_00: scores you know 0.7 0.8 dice score which is really really good um when you get to
- 10:24SPEAKER_00: uh
- 10:25SPEAKER_00: non-agricultural areas like Scottish Highlands or where there's not many crops growing,
- 10:30SPEAKER_00: for example Cumbria,
- 10:32SPEAKER_00: it doesn't do so well.
- 10:34SPEAKER_00: But the hope is UK Fields is releasing so many agricultural fields,
- 10:41SPEAKER_00: you're not really very interested in fields
- 10:44SPEAKER_00: in non-agricultural areas, right?
- 10:46SPEAKER_00: So when we release the data set, there won't be fields over the Scottish Highlands.
- 10:51SPEAKER_00: And I don't think anyone's necessarily going to be too worried about the lack of them.
- 10:56SPEAKER_01: Yeah, makes a lot of sense.
- 10:59SPEAKER_01: Fantastic.
- 10:59SPEAKER_01: Do you want to dive into a few more of the details around maybe the pre-processing?
- 11:04SPEAKER_01: I think you had a couple of extra slides there.
- 11:07SPEAKER_00: Definitely, yes.
- 11:07SPEAKER_00: Yeah.
- 11:09SPEAKER_00: I should say...
- 11:11SPEAKER_00: Releasing this kind of data set is probably borne out a lot of frustration in my
- 11:16SPEAKER_00: PhD when I tried to get my hands on field data very easily.
- 11:19SPEAKER_00: There's a few offerings in the UK that you can make use of,
- 11:24SPEAKER_00: but sometimes their paywalls or the methods aren't very easy to work out compared
- 11:29SPEAKER_00: to other countries in mainland Europe,
- 11:33SPEAKER_00: which is much,
- 11:33SPEAKER_00: much easier to get hold of more timely data.
- 11:37SPEAKER_00: And I told myself,
- 11:38SPEAKER_00: if I ever made the field boundaries myself for my own research,
- 11:43SPEAKER_00: I'd make it available so people wouldn't have to go through the same old slog to
- 11:47SPEAKER_00: get the same data.
- 11:48SPEAKER_00: So the ultimate aim of this is,
- 11:50SPEAKER_00: yes,
- 11:51SPEAKER_00: so other researchers can look at fields over the UK,
- 11:54SPEAKER_00: but by releasing the code as well,
- 11:56SPEAKER_00: We're hoping that other people create more field boundaries over loads of other
- 12:00SPEAKER_00: regions all across the world.
- 12:02SPEAKER_00: So it's just a single line import in Google Earth Engine and suddenly you've got
- 12:06SPEAKER_00: your fields to work with because this kind of data is just a stepping stone for
- 12:11SPEAKER_00: next further bits of analysis that you need to do.
- 12:15SPEAKER_00: So that's an important thing that hopefully releasing this will make a big difference.
- 12:21SPEAKER_00: There's also a few quality,
- 12:23SPEAKER_00: useful features in the code release that I hope people can make some good use of.
- 12:31SPEAKER_00: We could have released the code and just told you how to segment different tiles,
- 12:35SPEAKER_00: because if you're mapping the whole UK,
- 12:38SPEAKER_00: you've got to chip it up into lots of different adjacent areas and do your results.
- 12:45SPEAKER_00: But when you stitch it all together,
- 12:47SPEAKER_00: if it detects one field in one chip and then the next chip over the next adjacent chip,
- 12:52SPEAKER_00: it doesn't necessarily get the same field.
- 12:55SPEAKER_00: Merging it together becomes a complete headache.
- 12:57SPEAKER_00: So what we've got here is what we first do in Google Earth Engine is we export
- 13:03SPEAKER_00: overlapping chips,
- 13:05SPEAKER_00: probably about 10%.
- 13:07SPEAKER_00: either side, we do that in an overlapping fashion such that we can merge them together later.
- 13:13SPEAKER_00: And this is just a quick visualization of how we do merge them together.
- 13:16SPEAKER_00: So we do like a radial weighting.
- 13:19SPEAKER_00: So we take a kernel,
- 13:21SPEAKER_00: a radial weighting kernel on the left-hand side that you can see there,
- 13:24SPEAKER_00: and we apply that to the inference result we get from segment anything on the
- 13:28SPEAKER_00: harmonic composite,
- 13:29SPEAKER_00: which you can see in the middle tile.
- 13:31SPEAKER_00: Merge them together, and you can say that the results in the very center of that tile
- 13:36SPEAKER_00: we're most confident about, and those on the periphery, not so much.
- 13:41SPEAKER_00: And then when we stitch together all the overlapping ones,
- 13:44SPEAKER_00: so if we go left to right here,
- 13:45SPEAKER_00: we take the first tile,
- 13:46SPEAKER_00: weight it,
- 13:47SPEAKER_00: apply it to the next weighted tile,
- 13:49SPEAKER_00: and then the next one,
- 13:50SPEAKER_00: and then the next one.
- 13:51SPEAKER_00: And it just builds up the weighted average, right?
- 13:53SPEAKER_00: Such that if you...
- 13:55SPEAKER_00: take all of the weightings, you add them all together, then you binarize it at the end.
- 14:00SPEAKER_00: It just means that there's no ugly discontinuities between the different tiles and
- 14:06SPEAKER_00: information isn't lost.
- 14:08SPEAKER_00: And sometimes this code, when you're doing it at scale, can be a little bit tricky.
- 14:13SPEAKER_00: So we're releasing it so you can do this in Google Cloud and the Google Cloud platform.
- 14:18SPEAKER_00: So you don't even need to bring any data onto your own laptop to do this kind of stuff.
- 14:23SPEAKER_00: And also, if you do want to use stuff on your own laptop, we have that code available as well.
- 14:27SPEAKER_01: Right, okay.
- 14:28SPEAKER_01: I'm not too familiar with how that would work on Google.
- 14:31SPEAKER_01: You just give it the function and when API access.
- 14:35SPEAKER_00: Yeah, and there's also some very useful GDAL commands, but...
- 14:41SPEAKER_00: it it took me a long time to to get it all in the right place and working the way
- 14:46SPEAKER_01: incantation yeah yes awesome that's that's an amazing result and some really nice
- 14:53SPEAKER_01: technology built around it to make it very practical for use um i'm also interested
- 14:58SPEAKER_01: a bit in the paths that you went down that didn't work out you mentioned uh fine
- 15:02SPEAKER_01: tuning of sam this is something that so people have talked about doing but nobody's
- 15:07SPEAKER_01: really sure how to do do you mind talking a bit about that
- 15:09SPEAKER_00: Yes, no, definitely.
- 15:11SPEAKER_00: It's very important to talk about all the failures because if I'd gone through all
- 15:17SPEAKER_00: of this and not released it,
- 15:19SPEAKER_00: it's been hidden away.
- 15:22SPEAKER_00: One of the first big things I tried was fine tuning the Segment Anything to work better on RGB images.
- 15:31SPEAKER_00: And I was inspired by, I think it was called MedSam.
- 15:34SPEAKER_00: So it was Segment Anything fine tuned against medical images.
- 15:39SPEAKER_00: So I took some code off GitHub and tried to play around with it and swapped out the
- 15:44SPEAKER_00: medical image datasets for a popular field boundary benchmark dataset called AI for Boundaries,
- 15:52SPEAKER_00: which contains loads of Sentinel-2 imagery and corresponding field boundaries
- 15:57SPEAKER_00: across a huge part of Europe,
- 16:00SPEAKER_00: I believe.
- 16:00SPEAKER_00: And I fine-tuned that and
- 16:06SPEAKER_00: At first,
- 16:07SPEAKER_00: I thought I was getting better results,
- 16:08SPEAKER_00: but this is also,
- 16:10SPEAKER_00: it was very much just a pet project in the background.
- 16:12SPEAKER_00: I was training for many hours at a time, using a lot of GPU time, just playing around with this.
- 16:18SPEAKER_00: But the end result was no matter what type of model I use,
- 16:22SPEAKER_00: there's three different versions of Segment Anything with different numbers of parameters.
- 16:26SPEAKER_00: No matter what I tried, it didn't work.
- 16:30SPEAKER_00: it didn't properly bump up the accuracy in any meaningful way.
- 16:33SPEAKER_00: In many cases, it made it worse.
- 16:35SPEAKER_00: Now, I'm not saying that it's impossible to fine-tune it.
- 16:39SPEAKER_00: I'm just saying that my attempt at doing it in very much a background fashion just
- 16:45SPEAKER_00: didn't work out in the end.
- 16:46SPEAKER_00: It was a lot of work for very little gain.
- 16:49SPEAKER_00: And this harmonic composite approach was a lot less work for a lot more gain and a
- 16:56SPEAKER_00: lot more consistent results across the board.
- 16:59SPEAKER_01: Yeah,
- 16:59SPEAKER_01: I think for me,
- 16:59SPEAKER_01: that's a standout sort of takeaway that,
- 17:02SPEAKER_01: you know,
- 17:03SPEAKER_01: you can fine tune models or maybe you can take existing models and experiment with
- 17:06SPEAKER_01: the feature engineering,
- 17:08SPEAKER_01: as you put it,
- 17:09SPEAKER_01: and potentially that's a more viable route.
- 17:13SPEAKER_00: Yeah, definitely.
- 17:14SPEAKER_00: Sometimes there comes a point where if you've got something that works, you've just got to release it.
- 17:20SPEAKER_00: You've got to make sure that other people can build on that as well.
- 17:23SPEAKER_01: Really exciting stuff.
- 17:24SPEAKER_01: I think on your last slide, you've got the GitHub link.
- 17:26SPEAKER_01: Maybe we just put that one up and then we can just say,
- 17:30SPEAKER_01: okay,
- 17:31SPEAKER_01: so everything's being published at the moment.
- 17:33SPEAKER_01: You're finishing your PhD now.
- 17:36SPEAKER_01: What's next for you and what are your ambitions after having done this?
- 17:42SPEAKER_00: So I came into my PhD from industry and very much at the end of my PhD,
- 17:47SPEAKER_00: I'll be looking to return to industry.
- 17:49SPEAKER_00: I'm all about agriculture, earth observation, machine learning.
- 17:53SPEAKER_00: So I'll be looking for a job in those kind of fields very soon.
- 17:58SPEAKER_00: My PhD finishes at the end of October.
- 18:02SPEAKER_00: That's October 24.
- 18:02SPEAKER_00: So only a few months away.
- 18:05SPEAKER_01: Exciting.
- 18:06SPEAKER_01: And if people want to follow along your updates, which is the best platform for that?
- 18:11SPEAKER_00: uh best uh way to be best informed when this is all released is to look at my
- 18:15SPEAKER_01: twitter so that's at spirit well okay fantastic once again thank you uh fascinating
- 18:21SPEAKER_01: conversation and all the best for the write-up and hopefully we'll hear from you
- 18:24SPEAKER_00: again in the future lovely thank you very much for your time rogan bye cheers