Latest / Satellite image deep learning / Machine learning with SAR at ASTERRA

Transcript

Machine learning with SAR at ASTERRA · Satellite image deep learning

• 0:04
  SPEAKER_00: Hello and welcome to the satellite image deep learning podcast.
• 0:07
  SPEAKER_00: I'm Robin Cole and it's my pleasure to present another technically focused episode in the series.
• 0:13
  SPEAKER_00: In this episode, I caught up with Enon Sharini to learn about the fascinating world of machine learning with SAR imagery.
• 0:19
  SPEAKER_00: The unique attributes of SAR imagery, such as its intensity, phase and polarisation, provide rich information for deep learning models to learn features from.
• 0:28
  SPEAKER_00: We discussed the innovative applications Asterra is developing and covered the nuances of performing machine learning with SAR imagery.
• 0:35
  SPEAKER_00: Coming from a background with limited SAR experience, this conversation was incredibly enlightening.
• 0:39
  SPEAKER_00: I hope you enjoy this episode.
• 0:44
  SPEAKER_00: Hi, Arnon.
• 0:44
  SPEAKER_00: Welcome to the show.
• 0:45
  SPEAKER_00: Why don't you begin by telling us where you work and what you do?
• 0:50
  SPEAKER_01: Hi Robin, thanks for having me.
• 0:51
  SPEAKER_01: So, my name is Inon Sharroni, I work at Asterra, which is a company, we have several offices, our main office is in Israel, in Tel Aviv, and we have offices in the US, San Diego, in London, UK, and also in Japan, China, different locations.
• 1:11
  SPEAKER_01: What we do is we do SAR analytics, so we study SAR imagery,
• 1:18
  SPEAKER_01: for different applications and yeah.
• 1:22
  SPEAKER_00: Cool.
• 1:23
  SPEAKER_00: Why don't we begin by saying what's special about SAR then?
• 1:27
  SPEAKER_01: Okay, so SAR for people who don't know is a remote sensing technology which is
• 1:34
  SPEAKER_01: in the overall radio frequency spectrum is somewhere in the radio frequency, so it's somewhere between infrared and the longer-range radio.
• 1:48
  SPEAKER_01: It's about, I'd say, tens of centimeters in wavelength, which translates to about a gigahertz to ten thereabouts in terms of frequency.
• 2:00
  SPEAKER_01: What's special about this wavelength is, well, several things.
• 2:03
  SPEAKER_01: First of all, it is basically the atmosphere is transparent at this frequency, meaning that the water vapor in the atmosphere does not interfere.
• 2:14
  SPEAKER_01: And that basically means that what we use is capable of operating through clouds in any weather conditions, pretty much.
• 2:27
  SPEAKER_01: There's passive sensors at this frequency and active, and SAR is active in that it transmits as well as receives its own signal, so we're not dependent on external illumination as opposed to usual optical imagery.
• 2:46
  SPEAKER_01: yeah so those are the main I would say two strengths a part of that one of the things that works to our advantage is that while the water vapor in the atmosphere is kind of transparent to us the water
• 3:01
  SPEAKER_01: in the ground or water moisture in the ground isn't so at the bands of frequencies that we're talking about and they're named in letters so you have K X C S P L which is what we use we are very sensitive to the electromagnetic permeability of water sorry permittivity so that means that
• 3:29
  SPEAKER_01: We are able to discern not only quantities of water in the ground, but we're also able to discern the salinity content of that water.
• 3:37
  SPEAKER_01: So we'd be able to differentiate between rainwater, potable water, wastewater, anything like that.
• 3:46
  SPEAKER_01: And there's the usual stuff that you can do with electromagnetic
• 3:52
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled Machine
• 4:09
  SPEAKER_01: And we can also play around with the phase.
• 4:11
  SPEAKER_01: So if we judge the signal coming back as a complex value, it has backscatter amplitude or intensity, and also we can check how much the phase has changed.
• 4:24
  SPEAKER_01: And that gives us two things.
• 4:26
  SPEAKER_01: First of all, it's yet another degree of freedom, another source of information about the structure of the target that we interrogated as well as to perform interferometry, which gives us sentiment resolution in the vertical axis, let's say.
• 4:42
  SPEAKER_00: So really strong properties for interrogating the ground and understanding both what's happening inside the ground, moisture, but also small changes to the level of the ground.
• 4:52
  SPEAKER_00: That's a really interesting capability.
• 4:54
  SPEAKER_01: Yeah.
• 4:54
  SPEAKER_00: Fantastic.
• 4:56
  SPEAKER_00: So you've mentioned some of the potential uses.
• 5:00
  SPEAKER_00: What are the particular applications that you've been working on at Asterra?
• 5:05
  SPEAKER_01: So as I started mentioning, we are mostly interested in things that are, let's say, water content in the ground, whether that is for detecting water where water shouldn't be.
• 5:19
  SPEAKER_01: So as a general statement, you could say that we are targeting a lot of critical infrastructures.
• 5:25
  SPEAKER_01: So that could be, let's say, water pipes or water utilities that have their own infrastructure of water pipes.
• 5:32
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled Machine learning featuring Asterra
• 5:51
  SPEAKER_01: So we're helping that industry in recovering the water that's lost and they don't get revenue for, which is the non-revenue water.
• 6:04
  SPEAKER_01: So that's one thing.
• 6:05
  SPEAKER_01: And another thing is, let's say,
• 6:08
  SPEAKER_01: transportation infrastructure mining dams all these places where you don't want to see excess soil moisture because that basically will possibly lead to I would say about 80% of all earthworks projects that go wrong probably have that water element or factor that that contributed so that's another
• 6:37
  SPEAKER_01: product line that we're dealing with we have a few more there's you know we had a spin-off recently where we were able to detect mineral content beneath the surface and that was for example lithium which is a you know very important element this day and age right really interesting so many of these will be I guess typical measurements where does machine learning fit in
• 7:05
  SPEAKER_01: okay so as I mentioned in the beginning the SAR image or data in general is quite complex so we do have all these degrees of freedom the polarization and the phase and so on which means it's not like an RGB image or it's more like you would see in I guess what what you're used to it with with multiband or
• 7:32
  SPEAKER_01: multi-spectral or possibly even more than that.
• 7:35
  SPEAKER_01: So we have high dimensionality input that we need to make sense of.
• 7:42
  SPEAKER_01: Another challenge we have is that because we're dealing with a wavelength that's very different from visible light
• 7:50
  SPEAKER_01: We don't have the benefit of being able to manually label or annotate our data so we're working in an area where machine learning is a very important tool both in making sense of the data that we have and also in completing the full cycle of
• 8:10
  SPEAKER_01: building the application so yeah we use it for anything between you know making sure that the data that we get is similar to in terms of data distribution to previous images to rule out any anomalies and onwards to pre-processing it and and trying to extract features which are as I mentioned
• 8:36
  SPEAKER_01: probably going to be deep features for us to be for them to be useful and then we have you know the regular supervised or unsupervised or semi-supervised self-supervised we have the entire array that mostly depends on the kind of labels that we have for supervision in some cases those are more forthcoming but in our case usually they're not so we have to be creative about it
• 9:05
  SPEAKER_00: Does that mean that you have to have models that identify anomalies and then somebody reviews if you think it could have been a water event or something else?
• 9:16
  SPEAKER_01: That's a good question.
• 9:18
  SPEAKER_01: For the case of, let's say, the basic product that we have, which is the water leak detection,
• 9:26
  SPEAKER_01: in that case we are building on existing experience where known leaks were identified in fieldwork which is expensive and extensive and you need to have good design partners for that so we did have a kernel of ground truth samples that we could start and build from so in that case you know we started with the basic vanilla supervised convolutional neural networks in those cases
• 9:55
  SPEAKER_01: and those worked out good, well, so we went on to use that system more and more and now it's basically the system that we use in production.
• 10:07
  SPEAKER_01: For other cases, we did not have as much luck with getting ground truth data, whether it's from
• 10:14
  SPEAKER_01: So, you know, I could give the extreme example of looking for something like lithium, which is not well, in recent years, it has boomed, but there has not been a whole lot of existing work on where lithium would be found in substantial quantities.
• 10:34
  SPEAKER_01: So in those cases, we did have to resort to things like cell supervise and things like that.
• 10:41
  SPEAKER_01: and then you know at the other end of that spectrum you also have the anomaly detection and unsupervised where by that I'm talking about things that are really outliers like defective imagery detection for which we do have known failure modes of SAR or generally of any you know satellite data product but
• 11:04
  SPEAKER_01: We also want to catch the unknown unknowns.
• 11:07
  SPEAKER_01: In that case, you're just comparing any new example to the distribution of existing samples and trying to flag anything that somebody will need to look at.
• 11:21
  SPEAKER_00: Satellite image deep learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring
• 11:40
  SPEAKER_01: very large amounts of data coming in but not necessarily as much annotation or labeled examples thereof so I mean for people starting off in the field they will go to let's say some Kaggle competition or some other challenge that was set forth and these are really good primers to get you know kind of a foothold on what it would mean to work with this or other kind of imagery but
• 12:10
  SPEAKER_01: In the real world, you're not always presented with pristine and well-ordered datasets, and they're not always on the scale that you would want to train, let's say, a vision classifier, a vision transformer from scratch.
• 12:25
  SPEAKER_01: So those are the real-world challenges we have.
• 12:28
  SPEAKER_00: Right, okay.
• 12:30
  SPEAKER_00: And I guess, what's the state of the art in SAR at the moment?
• 12:33
  SPEAKER_00: I'm very familiar with optical sensors, but less so with SAR.
• 12:36
  SPEAKER_00: I'd love to hear about that.
• 12:38
  SPEAKER_01: Yeah, so I think for the reasons I mentioned as well as the preference of the different satellite operators towards the visible and IR sensors that there are not as much SAR data available.
• 12:53
  SPEAKER_01: There's of course Sentinel-1.
• 12:55
  SPEAKER_01: from ESA, which is C-band and dual-pole.
• 12:59
  SPEAKER_01: We use quadruple polarized, so that's even more information in that image.
• 13:06
  SPEAKER_01: So the state of the art is based on, let's say, Sentinel-1 imagery.
• 13:10
  SPEAKER_01: It's usually based on, you know, kinds of cross-modality things that people play with, like trying to remove clouds in Sentinel-2 images using a Sentinel-1 image which corresponds somehow.
• 13:25
  SPEAKER_01: You know, the challenge is there being first and foremost getting the two images co-registered so you're looking at the same thing.
• 13:32
  SPEAKER_01: But I think what most people are playing around with, as I mentioned, are things like self-supervised learning, using different engineered features from, in our case, the polarimetric aspects of SAR.
• 13:47
  SPEAKER_01: So those would be things like the decoherence metrics or things of that nature.
• 13:52
  SPEAKER_01: so you're using some kind of a physical first principles understanding for at least the initial features you want to play around with but you know more cases than not those are piped into the network alongside the raw inputs and we just leave leave it to the deep learning model to figure out how it wants to play around with those things and get the most value
• 14:18
  SPEAKER_00: Yeah, I feel like there's probably ample room for domain expertise in the pre-processing and the feature engineering side.
• 14:26
  SPEAKER_00: Is the data also amenable to classical techniques like random forest?
• 14:33
  SPEAKER_01: Yeah, so we, for example, we did play around with that I think for the most part nowadays for what you would expect from state-of-the-art with any kind of computer vision.
• 14:44
  SPEAKER_01: The classical or less deep methods are either going out of style or, as we found out ourselves, by trying both.
• 14:55
  SPEAKER_01: you end up looking for deeper features because the SAR input as I said is both it's complex both in the sense that it has different kinds of noise distributions as well as well we're looking for physical phenomena which are just not as simple as is it a cat or dog you know I mean if I look at an image and try to judge of you know whether it's a
• 15:21
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled
• 15:48
  SPEAKER_01: and therefore, yeah, our go-to tool is deep learning and we try to use, as I mentioned, the classical feature and engineering techniques as something to get us that, you know, first step closer to the end goal, but yeah, it will probably not get us all the way there.
• 16:07
  SPEAKER_00: Cool.
• 16:08
  SPEAKER_00: In terms of the actual architectures you're using, are they architectures I would be familiar with, like the U-Net or are there particular architectures?
• 16:16
  SPEAKER_01: Yeah, so we, you know, we're not shy about using anything, so we will reach for the classics, the convnets and the unets if, you know, we want to do semantic segmentation and so on.
• 16:30
  SPEAKER_01: So those are usually what we would try to get our baseline on.
• 16:36
  SPEAKER_01: and then we will try different well stuff we see in papers usually or you know different thoughts that people have in the team of you know how to combine features how to try to perform a multi-stepped training procedure as I mentioned with SSL for instance usually we have to we have to go as far as that in order to get you know the level of detection performance we want to achieve
• 17:06
  SPEAKER_01: But yeah, even the reference or baseline architectures like just a very basic convolutional neural network will probably already outperform classical algorithms in the field.
• 17:19
  SPEAKER_00: Fantastic.
• 17:23
  SPEAKER_00: Well, thanks so much for the technical dive.
• 17:25
  SPEAKER_00: I'm interested a bit in the product side as well.
• 17:27
  SPEAKER_00: How does the product look?
• 17:28
  SPEAKER_00: Is it just a map?
• 17:29
  SPEAKER_00: Is it images?
• 17:30
  SPEAKER_00: Is it alerts?
• 17:32
  SPEAKER_01: Right, so Asterra's product is basically a software as a service, a SaaS product.
• 17:38
  SPEAKER_01: So we have an online application which is known as EO Discover, EO standing for Earth Observation.
• 17:50
  SPEAKER_01: on that platform we serve all our solutions be they the you know the leak detection or recover product or solution as we call it earthworks and so on and yeah the idea is to have everything
• 18:07
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled Machine learning with SAR at Asterra
• 18:29
  SPEAKER_01: Yeah, we have the level of basic algorithms and trying to process the data and get it to a level that we think is insightful, but we have to remember that our customers are not necessarily data scientists and this is true for everything you do.
• 18:45
  SPEAKER_01: So we're looking for that specific business value that we can present to the customer and show them what they need to know.
• 18:57
  SPEAKER_01: My aim is to try to get the model as end-to-end as possible, so if we know the business metric that we're trying to hit, we'll try to train the model and optimize for that directly.
• 19:09
  SPEAKER_01: because deep learning models will do whatever it takes to optimize for that.
• 19:13
  SPEAKER_01: And we don't always know how to build a model that will go halfway and then manually post-process the other half of the way.
• 19:25
  SPEAKER_01: So we're trying to do it all in one shot and yeah, it's been working out for us so far.
• 19:32
  SPEAKER_00: Yeah, a bit about those business metrics.
• 19:34
  SPEAKER_00: So there must be some practicalities like you're alerting on a leak, but a leak can start small and grow.
• 19:39
  SPEAKER_00: How do you decide when a leak is significant?
• 19:42
  SPEAKER_00: Something as basic as that.
• 19:45
  SPEAKER_01: So as I mentioned in the beginning, the problem we have, I mean, not we have, but our customers have with their infrastructure is that it's very old and it's not maintained as much as they would want, which basically means that the problem space is ripe for disruption, right?
• 20:06
  SPEAKER_01: So they may have a leak that's been going for weeks or months and they won't even know about it.
• 20:14
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring Robin Cole titled Machine learning featuring
• 20:39
  SPEAKER_01: yeah so as I mentioned a bit regarding the previous question so when we train a binary classifier we can say is this a leak or not and try to optimize for accuracy but for the business aspect that would be sub-optimal because
• 20:58
  SPEAKER_01: We are not treating, let's call them, missed leaks like false negatives with the same weight as we would a false positive.
• 21:12
  SPEAKER_01: It kind of depends also on the context of the customer.
• 21:17
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled Machine learning featuring Asterra
• 21:39
  SPEAKER_01: a composite of KPIs.
• 21:43
  SPEAKER_01: Yeah, but we can, you know, we can always kind of weight our samples or, you know, make some kind of, customize somehow our loss function to try to get it as
• 21:59
  SPEAKER_01: much as possible aligned with the business metrics that we're interested in optimizing.
• 22:06
  SPEAKER_00: That's really a fascinating insight.
• 22:07
  SPEAKER_00: Thanks for sharing that.
• 22:09
  SPEAKER_00: So does Asterra publish their techniques?
• 22:13
  SPEAKER_00: Do you publish in conferences?
• 22:16
  SPEAKER_01: yeah so one thing we always do to try to protect our intellectual property is file patents so the company has about I think 15 patents I'm not sure I'm even up to date on that we usually would send people to present in conferences that are broadly about space tech
• 22:40
  SPEAKER_01: Earth observation specifically I mean that's that's what we do we also you know participate in different machine learning conferences so GTC you may see us with you know Nvidia we're in a partnership with them with Amazon Web Services and and you know there are other things that I'm not sure I'm supposed to talk about
• 23:08
  SPEAKER_01: probably some announcements about partnerships with Earth observation companies that people will know so yeah we try to get get our names where you know in the right company known in the right company and not necessarily plastered all over the place sick so people that want to follow along the the news and the updates I'm assuming LinkedIn you also have a web page
• 23:36
  SPEAKER_01: yeah we have a blog on our web page that has different things from so if you're interested more in the industry aspect industry news aspects of things then you will be able to see that there as well as you know more technical stuff technical blog posts and yeah depending on the kind of the kind of content you're looking for you will find it there
• 24:01
  SPEAKER_00: fantastic and you're also on LinkedIn yourself if people want to follow your personal updates yep I am all right I'll put all the details in the show notes well thanks again for this fascinating introduction to SAR and these interesting applications and I look forward to hearing again in the future from you thank you very much Robin thank you