Latest / Satellite image deep learning / AI at Nearmap with Michael Bewley

Transcript

AI at Nearmap with Michael Bewley · Satellite image deep learning

• 0:05
  SPEAKER_00: Hi Michael, how are you doing?
• 0:08
  SPEAKER_00: I'm great, yourself Robin?
• 0:09
  SPEAKER_00: Yeah, fantastic, thanks.
• 0:11
  SPEAKER_00: I'm really excited to be on the show today.
• 0:12
  SPEAKER_00: I've been following lots of your posts on LinkedIn.
• 0:15
  SPEAKER_00: It looks like you're out to some really interesting stuff.
• 0:17
  SPEAKER_00: Maybe to get the ball rolling, you can tell us where you are and what you do.
• 0:22
  SPEAKER_01: Yeah, so I'm based in Sydney in the early morning.
• 0:25
  SPEAKER_01: You might hear some birds in the background at a company called Nearmap where I lead the AI and computer vision group.
• 0:33
  SPEAKER_01: What we really do is we design and build our own camera systems.
• 0:37
  SPEAKER_01: Well, my group doesn't, the other one does.
• 0:39
  SPEAKER_01: Design and build camera systems, fly them to planes, do aerial imagery capture over like 85% of the US population and 95% of the Australian population and New Zealand and Canada too.
• 0:50
  SPEAKER_01: We turn the images into these big photo maps and 3D structures.
• 0:56
  SPEAKER_01: We use machine learning to turn that into information.
• 1:00
  SPEAKER_01: And then we have apps and APIs and things which expose them to customers.
• 1:03
  SPEAKER_01: So yeah, my thing mainly has been in the past, the machine learning bit.
• 1:09
  SPEAKER_00: That's fantastic.
• 1:10
  SPEAKER_00: So this is very high resolution imagery then?
• 1:13
  SPEAKER_01: Oh, absolutely.
• 1:14
  SPEAKER_01: So we, our kind of standard base has been about seven and a half centimetres a pixel.
• 1:20
  SPEAKER_01: A few years ago introduced a new camera system and we've been flying that at about five and a half centimetres a pixel and just in the process of rolling out a shiny new hyper camera three system, which, which is a little bit similar show solution, but, but you get some really nice clarity improvements and things.
• 1:39
  SPEAKER_00: I suppose that with that high-resolution comes very high-quality analytics and insights that you can generate.
• 1:47
  SPEAKER_01: You know, one of the most fascinating conversations I've had with people with more of a satellite background is the interplay with what you can do with the texture on high-resolution imagery compared to multiband on a satellite.
• 2:01
  SPEAKER_01: So, for example, if you're trying to look at the material of a roof, I think in satellite you're kind of looking for reflectance properties and different bands and stuff like that.
• 2:09
  SPEAKER_01: Satellite image deep learning featuring Robin Cole
• 2:33
  SPEAKER_00: understanding the built environment in a digital way there's a very interesting conversation to be had there about how like the sort of different trade-offs of different resolutions because you know convolutional neural networks typically pick up on textures so there must be some sort of threshold at which you suddenly gather this extra information and that will you know affect the uh the ability of models trained on that data
• 2:55
  SPEAKER_01: Oh, definitely.
• 2:56
  SPEAKER_01: You've got, I think probably a nice example of that trade-off would be, we're up to our fifth generation production system now, but between our first and our second, in our first, we were all about the maximum resolution of our imagery, basically training and predicting at zoom 21 on web Mercado, which is normally seven and a half centimeters a pixel.
• 3:18
  SPEAKER_01: and the problem is that at that zoom level looking at texture you get these big commercial buildings and you're limited by the amount of memory in your GPU like you've kind of got a close to infinite image at that zoom level because you've got a thousand square kilometers in one image but at that zoom level the whole context window that you can fit in memory
• 3:39
  SPEAKER_01: is just this sort of bumpy, gravelly surface.
• 3:43
  SPEAKER_01: And it's not until you zoom out a few times that you realise, oh, that's a big commercial warehouse of some kind.
• 3:48
  SPEAKER_00: Right, that's a really interesting observation there.
• 3:53
  SPEAKER_00: And this is something I'd love to dive into, how you would fuse information maybe from different resolutions.
• 3:59
  SPEAKER_01: Yeah, well, I guess maybe the important thing to start with is the fact that we run one single global model.
• 4:06
  SPEAKER_01: uh so we like if you didn't have that then you just you just train a model for your particular task right detecting a light pole or um or some light discoloration on a part of a roof is a different task from a massive commercial warehouse uh in in many respects but um i gave the team the goal uh which was you know there was some uh there were some challenges early on with it but we want one model uh because we wanted the efficiency of running
• 4:35
  SPEAKER_01: So we're now up to 78 different layers in a single semantic segmentation model.
• 4:43
  SPEAKER_01: And each one of them is an independent, it's kind of a multi-label output.
• 4:47
  SPEAKER_01: So you can have a pixel which is both tree and roof, and then you can work out tree overhang.
• 4:52
  SPEAKER_01: So you're jamming all that into one model that you run everywhere.
• 4:57
  SPEAKER_01: And the restriction there is, well, that model has to solve a variety of tasks.
• 5:01
  SPEAKER_01: It has to pick up
• 5:03
  SPEAKER_01: PowerPole by just a few pixels of the kind of crossbars on top of the PowerPole.
• 5:08
  SPEAKER_01: And it has to be able to pick up clean outlines of, you know, maybe you've got a built-up commercial roof with kind of gravelly stuff on it, and maybe it's in an area that's got kind of gravelly ground.
• 5:19
  SPEAKER_01: So you just lose the fine texture.
• 5:22
  SPEAKER_00: That is fascinating.
• 5:23
  SPEAKER_00: I never would have thought to choose one model for such a wide range of, you know, scales, for example.
• 5:31
  SPEAKER_01: Well, it really comes down to what you're going to do with it.
• 5:37
  SPEAKER_01: There's two real benefits.
• 5:39
  SPEAKER_01: One is efficiency.
• 5:42
  SPEAKER_01: How many times you need to take the RGB pixels and convert them into low-level features like edges and corners and things, and then up and up and up, and there's so much shared information.
• 5:55
  SPEAKER_01: It becomes really expensive to run that, whereas we can pack additional layers into our semantic segmentation model.
• 6:01
  SPEAKER_01: We've got some sneaky tricks, which I won't talk about, but we can basically pack as many layers as we want in.
• 6:09
  SPEAKER_01: And you're then basically paying the cost of running one ConvNet in production to produce 78 output layers for different customers.
• 6:19
  SPEAKER_01: So if you're kind of going all in on AI like we have,
• 6:23
  SPEAKER_01: um it makes a massive difference to execution cost and we're running on on petabytes of imagery so you kind of each cent per square kilometer matters fantastic is it possible to have a look at your your platform and start looking at some of the some of the outputs of this yeah i can show you some pictures for those who are um watching the video why don't i show you um and this will get us into a slightly different domain as well um this is
• 6:54
  SPEAKER_01: An area in... Let me click this one to get it right.
• 6:59
  SPEAKER_01: These are post-catastrophe survey that we've done.
• 7:03
  SPEAKER_01: So we've got this impact response program where we fly as soon as we can.
• 7:06
  SPEAKER_01: So you can see this one was captured a day after the event.
• 7:11
  SPEAKER_01: Now you won't see it clearly here yet, but this outline is where a tornado went through.
• 7:14
  SPEAKER_01: But I've got a junk and wreckage layer here.
• 7:18
  SPEAKER_01: And if I fade out the background, I don't know when you'll start seeing it.
• 7:21
  SPEAKER_01: Maybe let me know when you start seeing it on your...
• 7:24
  SPEAKER_00: Yeah, there's some red lines and sort of a band going across there.
• 7:30
  SPEAKER_01: So you'll see this band is actually the swath of damage from the hurricane zoomed out, sorry from the tornado zoomed out.
• 7:37
  SPEAKER_01: The yellow is junk and wreckage.
• 7:38
  SPEAKER_01: I'll kind of zoom in here a little bit.
• 7:41
  SPEAKER_01: um the yellow is junk and wreckage the red is vegetation debris and i've got pink in there for structural damage for roofs and and this kind of thing so you can kind of move down and see um and see that damage i'll give a little background so you can see what's going on
• 7:56
  SPEAKER_01: Sorry, I'm fighting bandwidth with Zoom here.
• 8:01
  SPEAKER_01: And I'm on a 4K monitor, so it's loading a little slowly.
• 8:05
  SPEAKER_00: Just so I can understand what I'm looking at, this is the model is detecting pixels associated with particular damage, but it's not doing a change detection.
• 8:13
  SPEAKER_00: It's literally processing a single image here.
• 8:17
  SPEAKER_01: Yeah, I mean, for kind of maximum speed and efficiency, so here you can see some of our imagery at around about its full resolution.
• 8:31
  SPEAKER_01: So, yeah, it's just looking at single-date imagery here because
• 8:37
  SPEAKER_01: I guess when there's an event that goes through, the vast majority of the damage is from that date.
• 8:43
  SPEAKER_01: You don't tend to have structurally damaged roofs left around for too long.
• 8:46
  SPEAKER_01: You can always go back and look at the previous date, but it just, it lets you move a bit faster and a bit more easily to do single date.
• 8:53
  SPEAKER_01: As you can see here.
• 8:55
  SPEAKER_00: The pink color, is that associated with damage to roofs or is that any kind of damage?
• 8:59
  SPEAKER_00: This is just structural damage to roofs.
• 9:01
  SPEAKER_01: So if I fade that out,
• 9:04
  SPEAKER_01: you can see that that's basically the semantic segmentation output from our model in a fairly raw form and we then do a whole bunch of geospatial post-processing to clean it up and vectorize it and link attributes and all sorts of stuff but this is basically the model and then you can see there's if I jump in there's a long long list here but if I go for junk and wreckage I can um
• 9:29
  SPEAKER_01: I can bring that yellow stuff in and out.
• 9:31
  SPEAKER_01: And that's just all the stuff which is, you know, we've actually got an ontologist who does all the definitions.
• 9:40
  SPEAKER_01: So the junk and wreckage layer is any junk or wreckage manufactured by humans and now discarded from its original use.
• 9:46
  SPEAKER_01: That's the short definition.
• 9:47
  SPEAKER_01: And then the long one is, you know, all about what's included and what's excluded.
• 9:50
  SPEAKER_01: Because we're training this model on a dataset that's extensively labeled.
• 9:56
  SPEAKER_01: I think it's more than a million labeled images so each each layer has kind of a definition set like this and a whole bunch of labeling instructions on what is and isn't considered part of that because as your listeners know
• 10:09
  SPEAKER_01: basically a deep learning model can learn to reliably detect what is in an image as long as a human can reliably detect it so it actually becomes less about the algorithm when you've got lots of labeling going on high quality imagery it's actually much more about the definitions and and being consistent with them than it is about fancy algorithm we do do lots of tweaks to the algorithms to improve over time like there's
• 10:33
  SPEAKER_01: Gosh, with our current generation, we originally started way back with kind of a deep lab V3 type model, but it doesn't look much like that anymore because of all these customizations.
• 10:43
  SPEAKER_01: Like how do you tile when you're running?
• 10:47
  SPEAKER_01: How do you tile a model kind of next to itself and get perfect edges?
• 10:51
  SPEAKER_01: How do you run at different scales or how do you run efficiently and pack all those extra layers in without blowing your memory out?
• 10:58
  SPEAKER_01: So yeah, the team's done an incredible job of
• 11:03
  SPEAKER_01: building our own kind of architecture that is pretty unique.
• 11:07
  SPEAKER_00: That's really exciting and interesting to hear about.
• 11:10
  SPEAKER_00: I'm curious also, is it able to absorb imagery at different resolutions or do you choose a particular resolution and stick with that always?
• 11:23
  SPEAKER_01: I can share we do so we all we're always running at some 21 kind of that um that native resolution a list of our early camera systems um because that gives the best balance between a cost trade-off and and being able to detect even even a few missing shingles kind of little little patch after a storm or something like that um however we do do some multi-scale stuff because of things like those large roofs and but we it is genuinely one model um and we have one um
• 11:53
  SPEAKER_01: you know we have one execution pipeline that's this whole computational graph of algorithms that run
• 11:59
  SPEAKER_01: Satellite image deep learning featuring Robin Cole
• 12:25
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled AI at Nearmap with Michael Bewley
• 12:41
  SPEAKER_01: you know so the generation is kind of the major version of the data and then you know the latest one is Tranquil Pool which is just uh we've got some nuances to what the names mean and then there's there's things that happen on the fly to the data um so you can go back and look at our changelog all the way back to you know go back to gen 3 you can see the various versions of gen 3 that got released and gen 4.
• 13:04
  SPEAKER_01: and this is this is really important for a lot of our customers who like yes it's great to train a machine learning model as a one-off or for a project but if they're basing things like risk and underwriting decisions on it or you're trying to assess you know what what has happened in your in your town in terms of vegetation history you kind of need to have proper lineage of the data and where it comes from so so even little patch tweaks that we make and end up in our versioning system
• 13:33
  SPEAKER_01: That's fascinating.
• 13:35
  SPEAKER_00: And is it a global model or do you find that you get sort of better precision if you target, say, Australia versus North America?
• 13:44
  SPEAKER_01: 100% global model at the moment.
• 13:46
  SPEAKER_01: And we've done, so we run in four countries, we run the same model.
• 13:52
  SPEAKER_01: In fact, we had a really interesting test point when we started off just running and training it on data from the US and Australia.
• 13:59
  SPEAKER_01: And then we expanded it to Canada and New Zealand.
• 14:02
  SPEAKER_01: And as you validate it,
• 14:04
  SPEAKER_01: we kind of found that the model actually already performed really well in Canada and New Zealand even though it wasn't trained on their data because there's enough kind of variation between Australia and the US that it captures that sort of stuff but then we wind in more training data so each generation where we're constantly you know got kind of 100 or more labellers just labelling data so every time we fly surveys we run the full survey
• 14:29
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled AI at Nearmap with Michael Bewley Satellite image deep learning featuring Robin Cole titled AI at Nearmap with Michael Bewley Satellite image deep learning featuring Robin Cole titled AI at Nearmap with Michael Bewley
• 14:49
  SPEAKER_01: most deep learning models you kind of jammed between you jammed near zero percent and 100 confidence output um but if you're in the middle somewhere the model's probably confused so you might want to go and label that area so it's this constant cycle of get the best and latest model um then go and um uh and run it and label it and and one through so typically we do a
• 15:14
  SPEAKER_01: with each of our generations we roll out a new deep learning model which sometimes has improved architecture and things as well but mostly it's got new classes in it I think Gen 4 to Gen 5 went from 49 layers to 78 layers so you can imagine the team kind of figuring out how to make sure each of the layers performs well and you've got a it's this dance of all these layers and the performance of them and you're watching the model converge and you're trying to make sure
• 15:40
  SPEAKER_01: Satellite image deep learning featuring Robin Cole titled AI at Nearmap with Michael Bewley
• 16:09
  SPEAKER_01: Well, I mean, the metrics, the same metric means different things for different layers.
• 16:14
  SPEAKER_01: So if you look at your IOU, your intersection over union, that's kind of a pretty common measure of quality for a prismatic segmentation output.
• 16:27
  SPEAKER_01: but the way it works is very heavily biased against small objects compared to large objects.
• 16:33
  SPEAKER_01: So you might see a much lower IOU for a power pole or a little swimming pool compared to a large building.
• 16:42
  SPEAKER_01: But when you visually look at the data, you say, oh, that's actually a really good result because even when it's a small object, even a slight wobble around the edge really hurts that score.
• 16:51
  SPEAKER_01: So we do produce scores.
• 16:55
  SPEAKER_01: We're actually really pleased by something we started doing in Gen 5, which was an automated difference report.
• 17:02
  SPEAKER_01: So we'll run on thousands of square kilometres.
• 17:04
  SPEAKER_01: We'll do unsupervised comparisons head-to-head with the models and also supervised comparisons.
• 17:10
  SPEAKER_01: And then to our customers, we're giving this report that kind of says exactly how does the model behave differently compared to the previous one so that they know what they need to change in their systems.
• 17:20
  SPEAKER_01: And usually it's kind of nipping off corner cases, improving a bit here, improving a bit there.
• 17:26
  SPEAKER_01: Satellite image deep learning featuring
• 17:56
  SPEAKER_00: You hinted at lots of architectural improvements as well.
• 17:59
  SPEAKER_00: I'm curious how you select directions to go in.
• 18:04
  SPEAKER_00: Do you follow published research?
• 18:06
  SPEAKER_00: Do you have internal experiments that you run?
• 18:12
  SPEAKER_01: A bit of both.
• 18:12
  SPEAKER_01: I mean, at our size, when you've got kind of that 20 or 30-person group and, I don't know, maybe 10 or so are deep learning people, you can't afford to be out there doing the groundbreaking research that publishes a new academic paper and changes how everyone does things.
• 18:29
  SPEAKER_01: We've got stuff that we could actually publish papers on, but that's the trade-off between IP and getting things out there.
• 18:37
  SPEAKER_01: But what we tend to do is look at the latest research because there's people doing some really great stuff out there with imagery.
• 18:42
  SPEAKER_01: I don't know how many billions of dollars is being poured into improving, for example, semantic segmentation results in imagery.
• 18:50
  SPEAKER_01: So we'll take the best of that and then we'll think, well, how's our problem different?
• 18:53
  SPEAKER_01: How's our problem special?
• 18:55
  SPEAKER_01: And you've got this infinite image, you know about the scale, so you're not dealing with arbitrarily scaled objects.
• 19:01
  SPEAKER_01: um the rotations kind of mean something different when you're looking top down so so the team is really very good at looking at the latest research and then adapting it to say well what's special about Nearmap it's not even just that we've got that 2D um
• 19:15
  SPEAKER_01: you know we're doing models now that include a depth channel as well from our 3D models so you know you're kind of trying to predict you know is it a building or not if you look perfectly top down and it looks like a car park and you know if it's a car park on top of the building or not so you kind of start to need that elevation channel to to get into the next layer and you know you don't find that many stock models out there that
• 19:40
  SPEAKER_01: I think geospatial data and elevation channel.
• 19:43
  SPEAKER_01: So there's a lot of customization and adaption that has to happen and adaptation.
• 19:49
  SPEAKER_01: But yeah, we certainly do look at the latest research.
• 19:53
  SPEAKER_00: And are there any other sort of modalities that you commonly would fuse with the data maybe for some classes?
• 20:00
  SPEAKER_00: You mentioned elevation maps for buildings.
• 20:04
  SPEAKER_00: Is it just elevation maps or?
• 20:07
  SPEAKER_00: Anything else?
• 20:08
  SPEAKER_01: Yeah, so early on we focused really heavily on Nearmap-generated content.
• 20:13
  SPEAKER_01: And that's because we have this unique data set that's tens of petabytes in size and we know exactly everything about it.
• 20:20
  SPEAKER_01: So you can really optimize for it.
• 20:23
  SPEAKER_01: um now um we're i mean we're constantly doing more things like the new camera system has a near infrared channel which we're really excited to get to find out what new information is genuinely in that channel um you know historically people used to find vegetation but we can find that from texture but they're also using for vegetation health and water content and that kind of thing so i'm expecting more information there um
• 20:50
  SPEAKER_01: and we are starting to weave in some some other third-party data but really the front and center this is about not just doing deep learning on aerial imagery it's about deep learning on near map imagery and you know when you kind of sit with the guys who did the color balancing and deal with saturation and and know how the surveys were processed
• 21:13
  SPEAKER_01: You can do things in a way that you simply can't if you're trying to look at it as just like a generic set of aerial imagery.
• 21:21
  SPEAKER_00: Fantastic.
• 21:23
  SPEAKER_00: One last question.
• 21:23
  SPEAKER_00: Can I ask you about the tech stack?
• 21:26
  SPEAKER_00: Are you Python people or is there a mixture of languages in there?
• 21:30
  SPEAKER_01: We had a bit of Golang for a bit because our API teams use a lot of Golang.
• 21:40
  SPEAKER_01: So we've had some of that in the backend, but when you're doing imagery, geospatial,
• 21:47
  SPEAKER_01: Deep learning, all of it is an extensive Python ecosystem.
• 21:51
  SPEAKER_01: So we're probably 90% Python, maybe 10% C++.
• 21:57
  SPEAKER_01: In the Vision team, we're much more C++ because that's been going on as a much longer effort and you've got to really optimize every single scrap you get out of it.
• 22:09
  SPEAKER_01: um yeah and we've just migrated actually from from TensorFlow to PyTorch um that's the big controversial move we were talking about doing that for a couple of years and we're now there um that's a good one was there a particular reason for the jump
• 22:23
  SPEAKER_01: There are a number.
• 22:25
  SPEAKER_01: I'm pretty resistant to changes generally for the sake of it.
• 22:29
  SPEAKER_01: So I was like, we'll stick with TensorFlow.
• 22:31
  SPEAKER_01: That should be great for everything.
• 22:34
  SPEAKER_01: And you saw that groundswell of movement towards PyTorch in the research field.
• 22:38
  SPEAKER_01: So I had my R&D team saying, look, all these papers that are out there, they come with PyTorch code, and it's just so easy to work with and manipulate.
• 22:45
  SPEAKER_01: um but then I had my my kind of engineering teams also saying we should switch to PyTorch because we would at that point we were looking at um you know we were doing multi-GPU training on a couple of GPUs but as you want to expand to a full node of like eight GPUs and then do multi-node um they just kept hitting problems with the way we were trying to do it in TensorFlow that just worked much more easily in PyTorch so when I had all my teams telling me PyTorch PyTorch we
• 23:12
  SPEAKER_01: who eventually ripped the Band-Aid and made the switch and it's been good so far.
• 23:17
  SPEAKER_00: That's really good to hear, fantastic.
• 23:20
  SPEAKER_00: Thank you so much for sharing so many details and presenting this really fascinating look into the work you're doing there.
• 23:25
  SPEAKER_00: If people want to follow along the developments, where's the best place for them to do that?
• 23:31
  SPEAKER_01: Well, so you can get both the company and myself on LinkedIn and Twitter, probably more active on LinkedIn than Twitter.
• 23:40
  SPEAKER_01: at Nearmap.com, N-E-A-R-M-A-P.com is the website.
• 23:46
  SPEAKER_01: Docs.nearmap.com is if you want to actually see the technical stuff that's written up about our product.
• 23:51
  SPEAKER_01: So for your listeners, that's probably the place.
• 23:53
  SPEAKER_01: Just search for Nearmap AI.
• 23:55
  SPEAKER_00: Fantastic.
• 23:56
  SPEAKER_00: I'll put that in the show notes as well for everybody.
• 23:58
  SPEAKER_00: Well, once again, thank you and hope to catch up with you again in the future.
• 24:03
  SPEAKER_01: Great.
• 24:03
  SPEAKER_01: Great to chat, Robin.
• 24:04
  SPEAKER_01: Looking forward to it.