Latest / What's Your Problem? / Using Pokémon Go to Map the World

Transcript

Using Pokémon Go to Map the World · What's Your Problem?

0:15
Speaker 1: Pushkin. I'm Jacob Goldstein and this is What's Your Problem.
0:21
Speaker 1: Two quick things before we start the show today. One,
0:25
Speaker 1: we're going to take a little break of a few
0:27
Speaker 1: weeks after this week's episode, but we'll be back soon.
0:30
Speaker 1: And two, I want to say thanks for listening. I
0:33
Speaker 1: really appreciate it. I want to keep making a show
0:36
Speaker 1: that you like, so please feel free to email us
0:39
Speaker 1: at problem at pushkin dot fm. You can also find
0:42
Speaker 1: me on LinkedIn or on x formerly known as Twitter
0:46
Speaker 1: at Jacob Goldstein. Okay, that's the housekeeping. Now here's today's show.
1:01
Speaker 1: My guest today is Brian McClendon. He's the chief technology
1:06
Speaker 1: officer at a company called Niantic Spatial, and his problem
1:10
Speaker 1: is this, how do you make maps better? Specifically, how
1:15
Speaker 1: do you take the two dimensional maps that we have
1:17
Speaker 1: today and turn them into three dimensional maps that change
1:21
Speaker 1: over time to keep up with changes in the real world.
1:25
Speaker 1: Maps like this would be very useful for robots. For
1:28
Speaker 1: one thing, Brian's company is in fact doing a pilot
1:31
Speaker 1: with a robot delivery company, those little robots that drive
1:35
Speaker 1: down the sidewalk to bring you a pizza or whatever.
1:38
Speaker 1: But the maps could also be useful for people. For example,
1:41
Speaker 1: the company is working on a project with the US
1:43
Speaker 1: Coast Guard to build better flight simulators for helicopter pilots,
1:47
Speaker 1: and the kind of maps that the company is building
1:50
Speaker 1: could also be useful for augmented reality, and in fact,
1:54
Speaker 1: Niantic Spatial spun out of the company that made Pokemon Go,
1:58
Speaker 1: the augmented reality game. Brian's been working on maps for
2:02
Speaker 1: a long time. In the early two thousands, he worked
2:06
Speaker 1: at a mapping startup that got acquired by Google in
2:09
Speaker 1: two thousand and four, and Google turned the project he
2:11
Speaker 1: was working on into Google Earth, and after that, Brian
2:15
Speaker 1: kept working on mapping at Google until twenty fifteen. In
2:19
Speaker 1: our conversation, Brian and I discussed the problems he's working
2:22
Speaker 1: on now trying to build better maps, including why trees
2:26
Speaker 1: are his nemesis. Also, we talked about what it will
2:29
Speaker 1: take to teach Ai to understand the physical world, but
2:33
Speaker 1: to start. Brian told me the biggest lesson that he
2:36
Speaker 1: learned about mapping during his time at Google.
2:39
Speaker 2: The most interesting thing is that map data in two
2:41
Speaker 2: thousand and four, you know, was already digital. We had
2:44
Speaker 2: suppliers that you could buy map data from, and you know,
2:47
Speaker 2: map quest was out there showing you a little little
2:50
Speaker 2: map on the screen on your website, on their website.
2:53
Speaker 2: But what we discovered when we started build Google Maps,
2:57
Speaker 2: we used that same data and we showed it to
2:59
Speaker 2: our users and launched in two thousand and five. We
3:03
Speaker 2: came up with this idea of street view, where we'd
3:05
Speaker 2: go take pictures of a lot of places around around
3:08
Speaker 2: the city and publish them.
3:10
Speaker 1: Can I say, by the way, because I remember when
3:12
Speaker 1: that happened and when I heard of that, and I
3:16
Speaker 1: when I first heard of it, I was like, surely
3:18
Speaker 1: they're not going to drive cars with cameras on every
3:22
Speaker 1: street and take pictures, Like, surely that's too much. But no,
3:26
Speaker 1: that's what it was like so wildly ambitious, just on
3:30
Speaker 1: a logistical level.
3:31
Speaker 3: Absolutely, that was It was a crazy project.
3:34
Speaker 2: You know, Larry Page gets credit for coming up with
3:36
Speaker 2: the idea and then following through and supporting us as
3:39
Speaker 2: we built it.
3:40
Speaker 1: What did you say when someone was it Larry Page
3:44
Speaker 1: who said to you, hey, let's do this, Who told
3:46
Speaker 1: you about it?
3:47
Speaker 2: You know, Larry, Larry had actually taken a video camera
3:50
Speaker 2: and driven around the Stanford campus, you know, you know,
3:53
Speaker 2: sort of systematically and said, you know, why can't we
3:56
Speaker 2: do this for the whole world?
3:57
Speaker 1: What did you say when he said, why can't we
4:00
Speaker 1: do this for the whole world? What did you say?
4:02
Speaker 3: I said, he's Larry Page and it's his money. Man.
4:04
Speaker 3: If he wants to do it, we'll give it a shot.
4:06
Speaker 2: And and honestly that's you know, the support that Google
4:10
Speaker 2: had for organizing the world's information was incredibly strong.
4:14
Speaker 3: I mean, Larry was serious about this.
4:16
Speaker 2: And we initially built a very expensive data collection van
4:20
Speaker 2: that drove around a few cities. Then we started to
4:23
Speaker 2: drive other cars that were cheaper but still okay, and
4:28
Speaker 2: then that's when we launched the service. And then we
4:31
Speaker 2: eventually said we need to do this for the entire
4:34
Speaker 2: United States. And the reason we did this was that
4:37
Speaker 2: when we launched the first cities, the first thing that
4:41
Speaker 2: people said was they looked at the pictures on the screen,
4:44
Speaker 2: and then they looked at their map data, and they
4:47
Speaker 2: started complaining about the map data. How can you get
4:49
Speaker 2: this wrong? I have a picture right here, this shows
4:52
Speaker 2: a street sign and a street number, and you have
4:54
Speaker 2: it completely in a different location. So a picture is
4:57
Speaker 2: worth a thousand words and probably megabytes of actual useful data.
5:01
Speaker 2: And there hadn't been any systematic sort of extraction of
5:06
Speaker 2: the value from those pictures, because that's not how maps
5:08
Speaker 2: had been made in the past. And so the big
5:11
Speaker 2: insight we ever had was that map data to that
5:14
Speaker 2: moment was not good.
5:16
Speaker 1: And by taking pictures for the first time, people could
5:20
Speaker 1: in a scaled way look at a photo right next
5:25
Speaker 1: to right on top of the map data and see like, oh,
5:28
Speaker 1: the map is wrong, and this photo proves that. Like
5:30
Speaker 1: it was the moment when people understood how wrong maps
5:33
Speaker 1: in general were exactly.
5:35
Speaker 2: And people always kind of knew this because they always
5:38
Speaker 2: got frustrated every now and then when they'd use the
5:40
Speaker 2: maps and they would fail. But now we had this
5:43
Speaker 2: much larger complaint body, and we knew that the data
5:48
Speaker 2: that we'd been buying from third parties wasn't good, and
5:50
Speaker 2: so we decided to make our own maps. And the
5:53
Speaker 2: biggest basis for that was the street view pictures themselves.
5:56
Speaker 2: And we drove all of the US, Canada, Mexico in
5:59
Speaker 2: two thousand and eight, and by two thousand and thirteen
6:03
Speaker 2: we had driven fifty countries.
6:05
Speaker 1: So the point of street view was not just for
6:08
Speaker 1: the like, hey, you can look at pictures, it's to
6:11
Speaker 1: get better data as an input to the sort of
6:14
Speaker 1: more traditional maps. Absolutely, I never knew that. Is that
6:19
Speaker 1: why Apple Maps sucked when it first came out, because
6:23
Speaker 1: they didn't have them.
6:24
Speaker 3: I'm so glad you said. It keeps me from saying that.
6:26
Speaker 1: I remember when it happened. I remember they fired the guy.
6:29
Speaker 1: Remember they fired the guy. But is that why? Because
6:33
Speaker 1: they just were buying the normal map data and everybody
6:37
Speaker 1: was comparing it to Google and you guys had literally
6:39
Speaker 1: driven cars on every street in America to figure it out.
6:42
Speaker 2: That's exactly right, and they eventually got their Apple Apple
6:45
Speaker 2: has driven you know, many millions of miles themselves, But
6:49
Speaker 2: if you look at it worldwide, I would say that
6:51
Speaker 2: Google is still probably better.
6:55
Speaker 1: I don't have a dog in that fight. It is
6:57
Speaker 1: interesting how both Google and Apple had these We're just
7:02
Speaker 1: throwing off crazy amounts of money, right each for their
7:05
Speaker 1: own special reasons, but we're reinvesting it in an interesting
7:09
Speaker 1: way that like x Ante, I would not have guessed. Right,
7:13
Speaker 1: Like this sort of search giant and the fancy phone
7:16
Speaker 1: maker becoming kind of going to the vanguard of mapping
7:21
Speaker 1: is not obvious.
7:22
Speaker 2: So I think when Steve Jobs added Google Maps to
7:26
Speaker 2: the iPhone very very quickly. He realized it was strategic
7:31
Speaker 2: and you know, he knew that because there was a
7:33
Speaker 2: GPS on the phone and because he now had a
7:36
Speaker 2: you know, pan and zoom capable touch screen, that he
7:41
Speaker 2: had a you know, an amazing new method for helping
7:44
Speaker 2: people understand the world.
7:45
Speaker 3: And so getting that map data on there was very important.
7:49
Speaker 2: And Android and iOS both had it very quickly. But
7:53
Speaker 2: the moment you do that, you have the same problem
7:55
Speaker 2: with street view showed was that people walk around, they
7:57
Speaker 2: look at their map and they look up at the
7:59
Speaker 2: street around them, and it's wrong. And so they knew that,
8:02
Speaker 2: you know, the biggest complaint they would be getting was
8:05
Speaker 2: this factual failure from their users about their maps.
8:09
Speaker 1: Yeah, well a map that's wrong is worse than no map, arguably.
8:14
Speaker 3: Yes.
8:16
Speaker 1: I mean, were there surprises to you about, you know,
8:19
Speaker 1: sort of what what developed, what didn't developed, how people
8:22
Speaker 1: used Google Maps, things you thought might happen that didn't happen.
8:25
Speaker 2: I think one of the things that you know, we
8:29
Speaker 2: probably speculated about. But if you realized that Google Earth
8:34
Speaker 2: and Maps launched in February and June of two thousand
8:37
Speaker 2: and five.
8:38
Speaker 3: In August, Katrina happened, and.
8:43
Speaker 1: Katrina was Hurricane Katrina in New Orleans.
8:45
Speaker 2: Yeah, yeah, it was an incredible hurricane, did huge amounts
8:49
Speaker 2: of damage and the city of New Orleans was flooded,
8:52
Speaker 2: you know, completely, and this turned into a challenge for
8:57
Speaker 2: you know, everybody involved, you know, both for just escaping
9:00
Speaker 2: the city and going to the right places, but then
9:02
Speaker 2: it became about rescuing people on rooftops. And the problem
9:06
Speaker 2: now was the streets were gone and they couldn't figure
9:09
Speaker 2: out where things were. And so it turned out that
9:12
Speaker 2: the helicopter pilots at the time used Google Earth to
9:17
Speaker 2: find out where the address was on the picture and
9:21
Speaker 2: then use the picture to navigate their helicopter to find
9:24
Speaker 2: the people waiting on the roof. And there were literally
9:28
Speaker 2: hundreds of rescues done using Google Earth's bipilots, you know,
9:32
Speaker 2: a few months after we'd released it, and that was
9:34
Speaker 2: that was a big surprise for us.
9:37
Speaker 1: It is extremely interesting to me that the company right
9:41
Speaker 1: now spun out of the company that made Pokemon Go,
9:45
Speaker 1: and that there was this kind of kernel of an
9:46
Speaker 1: idea which was like, oh, hey, people you have playing
9:51
Speaker 1: Pokemon Go have taken a billion photos in the world,
9:54
Speaker 1: like truly a billion photos. This is an interesting data set.
9:58
Speaker 1: Maybe we can do something with it. I mean, if
10:00
Speaker 1: I understand right, that was before you got there, But like,
10:03
Speaker 1: how did that playoff? Did they know when they were
10:05
Speaker 1: launching Pokemon Go that like, oh, we're going to get
10:07
Speaker 1: this data? Did somebody realize it?
10:09
Speaker 2: So interestingly, I think the uh, you know, John John
10:13
Speaker 2: Hankey definitely had a very long term vision. He was
10:16
Speaker 2: thinking about not just the game itself, but about games
10:19
Speaker 2: in the real world. Now, the photos that were taken
10:22
Speaker 2: initially by Pokemon Go players were you know, taking a
10:25
Speaker 2: picture of a Pokemon into location. You know that those
10:28
Speaker 2: photos stayed private. What happened was that John, seeing the future, said,
10:34
Speaker 2: can we create a game mode where players consciously go
10:38
Speaker 2: out and record videos for us of poke stops and
10:41
Speaker 2: help us map these locations by recording videos. So it
10:44
Speaker 2: was a conscious gameplay activity where you would get in
10:47
Speaker 2: game rewards if you would upload a thirty second video
10:52
Speaker 2: to the to the game.
10:54
Speaker 3: And that was where that data came from.
10:55
Speaker 2: So it was consciously contributed by the by the players
10:58
Speaker 2: rather than sort of background collected.
11:00
Speaker 1: And and what was what was the point of that
11:05
Speaker 1: From the point of view of the company, so that
11:06
Speaker 1: we wanted to create more ar gaming experiences.
11:10
Speaker 2: And the Pokemon Go already used augmented reality to create
11:14
Speaker 2: those photos that you were talking about. Put a Pokemon
11:17
Speaker 2: sitting on a chair and take a picture of it,
11:19
Speaker 2: or so it was already using augmented reality. What we
11:22
Speaker 2: wanted to do is to be able to augment the
11:24
Speaker 2: poke stops in the outdoors, and so to do that accurately,
11:28
Speaker 2: you actually need to have a very accurate map of
11:30
Speaker 2: those pokey stops, not just where they are with a
11:32
Speaker 2: blue dot, but actually the three D model of the
11:35
Speaker 2: benches around it and the statue itself, and so creating
11:39
Speaker 2: that three D model and then being able to point
11:43
Speaker 2: your phone at it and say, I know exactly where
11:45
Speaker 2: you're standing relative to that statue. Now the Pokemon that
11:49
Speaker 2: was placed by the last person is exactly the same
11:52
Speaker 2: place that it is for you.
11:54
Speaker 1: So you need a really sophisticated model of the physical
11:57
Speaker 1: world for that to work, otherwise it'll be like floating
12:00
Speaker 1: in space a foot from where it's supposed to be.
12:02
Speaker 2: That it is exactly the problem we spent several years solving,
12:05
Speaker 2: and we only launched the product that used it. I
12:08
Speaker 2: think in twenty twenty three, and that was part of
12:13
Speaker 2: Pokemon Go and it's called Pokemon Go playgrounds, and it's
12:17
Speaker 2: the ability for players to leave Pokemon at locations in
12:21
Speaker 2: interesting formations and interesting combinations and then have other players
12:25
Speaker 2: discover them or add to them.
12:28
Speaker 1: And at what point did somebody think, oh, we can
12:32
Speaker 1: do something wholly other than gaming here, we can build
12:35
Speaker 1: a new kind of map that is, you know, more
12:37
Speaker 1: legible to a robot.
12:39
Speaker 2: I think that in some people's minds, you know, that
12:43
Speaker 2: was the thought from day one. I think John would
12:45
Speaker 2: argue in his mind he was he was really thinking
12:48
Speaker 2: in that direction. But I think, you know, from a
12:52
Speaker 2: product perspective, in a business perspective, being able to allow
12:57
Speaker 2: developers of applications to go scan their own places, not
13:01
Speaker 2: Poke stops, but any location around the world and then
13:04
Speaker 2: create these experiences. That was our first step. I think
13:07
Speaker 2: the robotics ass inspect came into play, you know, more recently,
13:11
Speaker 2: probably in the last eighteen to twenty four months, as
13:13
Speaker 2: we realize that, you know, robots struggle with finding themselves
13:18
Speaker 2: in the real world, and so they need a form
13:21
Speaker 2: of maps, don't we all.
13:26
Speaker 1: And we'll be back in just a minute. So let's
13:37
Speaker 1: talk about where you are now, like what's in the
13:40
Speaker 1: world now that you're making.
13:43
Speaker 2: I think the uh, you know, the experience of working
13:46
Speaker 2: with the data from the phones of the players taught
13:50
Speaker 2: us a lot about how to reconstruct and create these
13:53
Speaker 2: precise maps, not with super fancy survey tools but with
13:57
Speaker 2: just consumer devices.
13:59
Speaker 3: And what we got good at was using.
14:01
Speaker 2: Low quality data, which I mean, it's still very good,
14:05
Speaker 2: but it's you know, it's not professional great anything, and
14:08
Speaker 2: being able to turn that into professional grade data. And
14:11
Speaker 2: so I think that was the skill that we're bringing
14:15
Speaker 2: into the enterprise applications that you hear now.
14:18
Speaker 3: And we can do it not just at the scale
14:20
Speaker 3: of a.
14:21
Speaker 2: Statue or a single location, but we can do it
14:24
Speaker 2: at city scale. We can collect an entire city and
14:28
Speaker 2: help you find yourself.
14:29
Speaker 1: Have you collected an entire city?
14:31
Speaker 3: Yes?
14:33
Speaker 1: What city?
14:34
Speaker 2: We have all of San Francisco and pieces of other cities,
14:38
Speaker 2: and they're usually collected from drone data. Basically commissioning and
14:42
Speaker 2: saying we wanted a drone based camera, you know, follow
14:46
Speaker 2: a path, take a picture every several meters and take
14:50
Speaker 2: it from multiple angles, and then use photogrammetry to reconstruct
14:55
Speaker 2: that city at a level of detail, you know, beyond
14:58
Speaker 2: what Google can do it.
15:00
Speaker 1: And when you say drone, you mean like a flying drone,
15:02
Speaker 1: like a little quad copter what exactly.
15:04
Speaker 2: A very lightweight, you know, very limited under two hundred
15:08
Speaker 2: and fifty grams smallest ones.
15:09
Speaker 1: And so what are you doing now with your three
15:12
Speaker 1: D robot legible map of San Francisco.
15:15
Speaker 2: We are combining all of that interesting data collected on
15:19
Speaker 2: the ground with phones and with three sixty cameras with
15:22
Speaker 2: this drone data and also with satellite data and proving
15:26
Speaker 2: out that we can help robots and humans find themselves,
15:31
Speaker 2: localize themselves, you know, anywhere in the city without having
15:35
Speaker 2: to pre map it, you know, specifically at that location.
15:38
Speaker 1: So you mean like you a robot, It's just like
15:41
Speaker 1: you blindfold the robot for lack of a better word,
15:44
Speaker 1: and then take the blindfold off, and it knows right
15:47
Speaker 1: away where it is and which way it's looking.
15:49
Speaker 3: That's absolutely the goal.
15:50
Speaker 1: Yes, what do you have to do to get there?
15:55
Speaker 2: I think the challenge is that you know, the world
15:58
Speaker 2: is a complicated and varied place, and you know, trees.
16:01
Speaker 2: Trees are my nemesis in my career, and you know
16:05
Speaker 2: they are they are absence.
16:06
Speaker 1: I'm proachry. For the record, I'm approached.
16:09
Speaker 2: I understand they are visually amazing, but they are in
16:13
Speaker 2: fact fractally based. Right they have very high detail, and
16:17
Speaker 2: worse than that, they're leaves. They flutter in the wind.
16:20
Speaker 2: It's deeply inconvenient, and so when you take multiple pictures
16:24
Speaker 2: of a tree, no two pictures have the tree in
16:27
Speaker 2: the same spot.
16:28
Speaker 1: When they change rapidly with tom.
16:30
Speaker 2: Absolutely so another problem. The old way of doing this
16:34
Speaker 2: was that, you know, satellites and airplanes would take pictures
16:37
Speaker 2: of cities in the winter because the leaves were off,
16:40
Speaker 2: and that gave them more information. But if you're trying
16:43
Speaker 2: to build a model the city and localize, you actually
16:45
Speaker 2: need to see them in all their states, not just
16:48
Speaker 2: the leaf off state.
16:49
Speaker 1: So is the short answer to how you solve that problem? AI?
16:53
Speaker 2: How do you solve the tree problem? You try to
16:56
Speaker 2: look beyond the trees. You know, trees other than the
16:59
Speaker 2: tree trunk. There's not a lot of stability there, but
17:02
Speaker 2: there are many things behind the tree or next to
17:05
Speaker 2: the tree that are useful and being able to effectively
17:09
Speaker 2: in an AI. To ignore the tree is usually your
17:12
Speaker 2: best bet because there's enough remaining features in view that
17:17
Speaker 2: you can find enough to connect yourself to that location.
17:23
Speaker 1: I know you just made a deal with a delivery
17:25
Speaker 1: robot company. Like, is that the first use case? Like,
17:29
Speaker 1: what do you think the first use cases are going
17:31
Speaker 1: to be?
17:32
Speaker 2: I mean, definitely is the first use case they struggle with,
17:35
Speaker 2: effectively the blindfold problem that sometimes they don't don't have
17:39
Speaker 2: enough context to know where they are, and they need
17:41
Speaker 2: to rediscover their location, and they need to need do
17:45
Speaker 2: so accurately enough that either the autonomous controller or some
17:49
Speaker 2: human who drops in to look at where it is
17:51
Speaker 2: can know where they are in the city because city
17:55
Speaker 2: blocks tend to look the same.
17:56
Speaker 3: Unless you have that information.
17:58
Speaker 1: How does the robot get lost in the first place? Like,
18:01
Speaker 1: how does the robot find itself in that situation?
18:03
Speaker 3: There's two ways.
18:05
Speaker 2: One is I mean they do it they hit a reboot,
18:07
Speaker 2: and they literally lose connection if they are cover if
18:12
Speaker 2: for any reason enough of the world is blocked from
18:16
Speaker 2: them for long enough, they lose lose control. GPS by
18:20
Speaker 2: itself is rarely good enough because, especially in cities, the
18:24
Speaker 2: GPS has you know, reflections off of buildings, and you
18:27
Speaker 2: can be a city block away and the GPS will
18:30
Speaker 2: happily tell you bigger there so.
18:33
Speaker 1: Hu So it's insufficiently precise, particularly in cities.
18:37
Speaker 2: Yes, and big cities with big, tall buildings are absolutely
18:41
Speaker 2: the hardest challenge other than tunnels and parking critches.
18:45
Speaker 1: But that's good for you. Like that that weakness is
18:49
Speaker 1: your opportunity exactly. What tell me about the broader industry
18:57
Speaker 1: is the right word, but the broader context, you know,
18:59
Speaker 1: who else is working on things like you're working on.
19:03
Speaker 2: I mean, I think we've had, you know, companies that
19:06
Speaker 2: have worked on building digital twins of the world. They're
19:09
Speaker 2: in the micro or the macro for you know, since
19:12
Speaker 2: before Google, and Google certainly is the biggest wholesale attempt
19:17
Speaker 2: at this, but there's a need for more resolution and
19:22
Speaker 2: also to have it from suppliers other than Google, because
19:26
Speaker 2: one of the areas that I think is hard is
19:28
Speaker 2: that Google is all about recreating the public world, but
19:31
Speaker 2: they don't really have any way for a company to
19:34
Speaker 2: do their own location and add it to the real world.
19:39
Speaker 1: Oh interesting, the hospital did I read hospital as an example.
19:43
Speaker 1: We were talking about that when we're doing the prep
19:45
Speaker 1: of like how hard it is to find your way
19:47
Speaker 1: around a hospital?
19:49
Speaker 3: It is, and I just experienced that yesterday at Stanford.
19:52
Speaker 3: I think the.
19:54
Speaker 2: Challenges building a map and with the hospital, there's actually
19:58
Speaker 2: a lot of change that happens, you know, in the
20:00
Speaker 2: corridors of a hospital at any given time. The big
20:03
Speaker 2: quarriors are stable, but most of the actual patient areas
20:07
Speaker 2: are you know, constantly changeing their visual appearance, and so
20:11
Speaker 2: getting robots that are able to navigate that space accurately
20:15
Speaker 2: and reliably is certainly one of our potential use cases.
20:19
Speaker 1: So there's an app called scan a Verse that's you right,
20:23
Speaker 1: the app scan a Verse. So I downloaded that this
20:27
Speaker 1: week and astonishingly, they're estonishingly to me. Within one hundred
20:33
Speaker 1: yards of my house, there are two things, two objects
20:37
Speaker 1: that have been scanned in three D and uploaded to
20:40
Speaker 1: this app, which is interesting cool. I had no idea
20:44
Speaker 1: what's going on there? What is it? And why are
20:46
Speaker 1: people on my block taking pictures of things.
20:49
Speaker 2: Scan Versus is a great app, and we actually acquired
20:52
Speaker 2: the one person.
20:53
Speaker 3: Who wrote it, a brilliant programmer.
20:56
Speaker 2: And turned it into this product that allows capture of
21:00
Speaker 2: three D objects. And what you're seeing there is that
21:03
Speaker 2: a lot of users love to be able to go
21:05
Speaker 2: out and scan their area their location or a statue
21:08
Speaker 2: or whatever, and recreated in three D visually almost completely
21:13
Speaker 2: photo realistically, and then share that with the rest of
21:16
Speaker 2: the world. And so you're experiencing sort of the map
21:19
Speaker 2: that scanners puts up at the front of the front
21:21
Speaker 2: of them.
21:23
Speaker 1: So it's in the same way that people like to
21:26
Speaker 1: do whatever geokashig and geo guessing and whatever. It's some
21:30
Speaker 1: version of kind of sharing the real world in a
21:33
Speaker 1: digital way.
21:34
Speaker 3: Exactly is it big?
21:36
Speaker 1: I infer from the fact that they're too on my
21:39
Speaker 1: block or one on my block and one around the
21:40
Speaker 1: corner that lots of people are using it? Is that true?
21:43
Speaker 1: Do I live in a weirdly dense area?
21:46
Speaker 2: There are a lot of people using We have hundreds
21:48
Speaker 2: of thousands of users and they are passionate about this ability,
21:54
Speaker 2: and we have used we've recently, very recently added the
21:58
Speaker 2: enterprise capabilities to that app. People want to create a
22:02
Speaker 2: digital twin of a tiny little place, they can do
22:04
Speaker 2: it on their phone and upload it like you saw.
22:07
Speaker 2: If they want to recreate their factor or their business,
22:11
Speaker 2: you know, hundreds of square meters, they can do that
22:13
Speaker 2: with our enterprise version and then allow for modeling and
22:18
Speaker 2: allow for robotic training within their site using that tool.
22:22
Speaker 1: So you can use your phone and walk around your factory,
22:27
Speaker 1: say you're a little, you know, small factory or your
22:29
Speaker 1: construction site or whatever, and the three D rendering it
22:34
Speaker 1: generates is like robot legible. Then you can train your
22:37
Speaker 1: robot on the thing you made with your iPhone.
22:40
Speaker 3: That's correct.
22:41
Speaker 1: So there's this phrase you hear an AI world model, right,
22:47
Speaker 1: And it's sort of kind of a complement to a
22:50
Speaker 1: language model.
22:51
Speaker 3: Right.
22:51
Speaker 1: Instead of AI that's trained on language, it's AI in
22:54
Speaker 1: some way that's trained on the physical world. Like, tell
22:57
Speaker 1: me about that and how your work fits in with that.
23:00
Speaker 2: So the world models have been mostly focused to date
23:03
Speaker 2: on solving a sort of very specific problem, which is
23:06
Speaker 2: how to translate video, which they're mostly programmed with, into
23:12
Speaker 2: understanding of the world that's good enough that it can
23:15
Speaker 2: predict the future.
23:16
Speaker 1: Yeah, and when you say predict the future, you don't
23:18
Speaker 1: mean any like weird oracle thing, right, you mean like
23:20
Speaker 1: predict the future as well as a human can that. Like,
23:23
Speaker 1: if a fork starts to fall off the table, it's
23:26
Speaker 1: going to fall all the way to.
23:27
Speaker 3: The grounds, that's exactly right.
23:28
Speaker 2: So if something is initiated, how would it play out
23:33
Speaker 2: with the objects. And if you've seen enough forks fall
23:35
Speaker 2: and videos, if you've seen enough objects fall, then you
23:38
Speaker 2: can start to predict it.
23:39
Speaker 1: And we were so used to now AI being so
23:42
Speaker 1: good at language, it's easy to forget that it has
23:45
Speaker 1: no understanding of the physical world. I mean except in language.
23:49
Speaker 3: That's right, so weird. Yeah, it can, it can think
23:52
Speaker 3: in its brain.
23:53
Speaker 2: I know what the rules of Newton's law are, but
23:55
Speaker 2: you know, turning that into pixels on the screen is
23:57
Speaker 2: actually pretty hard. But if you train enough video models
24:01
Speaker 2: on it, then then it can figure it out. And
24:04
Speaker 2: so the I think the connecting the language model to
24:09
Speaker 2: the physical world is in some ways. There's two ways
24:13
Speaker 2: to do it, and one is to have the model
24:16
Speaker 2: understand the physics video. The other is to turn the
24:19
Speaker 2: world itself into words or into meaning. And this is
24:24
Speaker 2: you know, typically called semantics. And there's all sorts of
24:28
Speaker 2: you know, technologies that try to label information about a picture.
24:33
Speaker 2: And if you train enough of models on the labeling
24:37
Speaker 2: of the pictures, you can then label any but any
24:39
Speaker 2: picture of the real world.
24:41
Speaker 1: And this is kind of the first big breakthrough of
24:44
Speaker 1: neural networks, right, I mean, it sounds like.
24:46
Speaker 3: That cats, cats and dogs, cats and dogs.
24:48
Speaker 1: Yeah, and image net right.
24:50
Speaker 2: So if you give the information about the embeddings, it's
24:54
Speaker 2: it's called and you do that not just in cat
24:57
Speaker 2: and dog dimension, but you do it in hundreds of
24:59
Speaker 2: dimensions of objects. You can give these language models a
25:03
Speaker 2: the equivalent of a language understanding of the real world,
25:07
Speaker 2: and then they can do amazing things with it.
25:09
Speaker 1: Huh. And so how does all that relate to the
25:12
Speaker 1: work you're doing.
25:13
Speaker 2: We believe that to do that successfully you need to
25:16
Speaker 2: start with three D data because the ability to label
25:20
Speaker 2: something in three dimensions is much more accurate and with
25:24
Speaker 2: and argue with much more information than if all you
25:26
Speaker 2: have is a two D picture of it. Because in
25:29
Speaker 2: two D pictures there's always, uh, you know, obscuration of data,
25:34
Speaker 2: and things are behind things and you can't see the
25:37
Speaker 2: whole picture. But in three D you eventually see everything
25:40
Speaker 2: that is seeable.
25:41
Speaker 1: I mean is that the fun not to be reductive,
25:44
Speaker 1: but kind of to be reductive, is that the fundamental
25:47
Speaker 1: thing you were doing that is novel is making a
25:50
Speaker 1: three dimensional map and world where maps have always been
25:53
Speaker 1: two dimensions.
25:53
Speaker 2: Before I would say yes, I'm there, there are there
25:57
Speaker 2: are three D visual maps. There are you know, rare
26:01
Speaker 2: one off three maps, but trying to do a fully
26:05
Speaker 2: language embedded three D semantic map of the world so
26:08
Speaker 2: that a language, models and robots and others can understand
26:12
Speaker 2: it is what we're working on.
26:15
Speaker 1: So for zoom out like, well, two things, I guess
26:20
Speaker 1: there's two things we'll do, the sad one and the
26:21
Speaker 1: happy one. Like what might go wrong for you? Like
26:24
Speaker 1: what has to go Yeah, what might go wrong for
26:26
Speaker 1: you over the next years. It seems like you're at
26:28
Speaker 1: this moment when you have this interesting technology, you're sort
26:30
Speaker 1: of proving it out. It's not widely used yet. What
26:33
Speaker 1: are the sort of pitfalls?
26:34
Speaker 2: I mean, it was hard hard mapping the world in
26:37
Speaker 2: two D for humans. Mapping the world in three D
26:39
Speaker 2: four robots requires a level of detail and precision that
26:43
Speaker 2: nobody's ever done before. And the only way to get
26:46
Speaker 2: it done is to apply AI to the problem, to
26:49
Speaker 2: figure out how to make it more accurate, more quickly
26:52
Speaker 2: and more cheaply than has been done in the past, because,
26:57
Speaker 2: you know, carefully reconstructing the world, every square meter of it.
27:02
Speaker 2: The old way is you know, billions, if not trillion dollars,
27:06
Speaker 2: and this can't be that nobody's going to pay that.
27:10
Speaker 1: Yeah, billions maybe, but trillions definitely not. And so if
27:17
Speaker 1: you if you get it right, like give me the
27:22
Speaker 1: like the big exciting outcome, you know what I mean,
27:25
Speaker 1: like if it goes, well, what's the world look like?
27:27
Speaker 1: How is the world better?
27:29
Speaker 2: I think that if you know, as we're seeing, the
27:32
Speaker 2: ais have been able to dig into human knowledge and
27:35
Speaker 2: you know ingest all of the text of all of
27:37
Speaker 2: our scientific papers and all of our blog posts and
27:41
Speaker 2: all of the Instagram and they've synthesized this in many
27:44
Speaker 2: cases into some very interesting information that answers at least
27:48
Speaker 2: you know, personally relevant questions today. And if you talk
27:51
Speaker 2: to some mathematicians, you know they're they're they're doing pretty
27:54
Speaker 2: well in math as well. But the questions about the
27:57
Speaker 2: real world are are are simply not there yet. And
28:01
Speaker 2: but the the advantage of an AI is it doesn't
28:04
Speaker 2: stop and it doesn't stumble on too much data, and
28:08
Speaker 2: if you can give it enough information about the world,
28:11
Speaker 2: it can start to answer like really interesting questions about
28:15
Speaker 2: the city, about how the city works, about which intersections
28:18
Speaker 2: are you know too busy, and about you know, you know,
28:23
Speaker 2: flow of people and you know I think things that
28:26
Speaker 2: things that could make the city more efficient because you
28:29
Speaker 2: have a more complete vision of it, you know, especially
28:34
Speaker 2: in you know, three dimensions. You know, the two D
28:36
Speaker 2: maps for a street are fine, but the moment you
28:38
Speaker 2: go down to the subways in New York, some of
28:40
Speaker 2: those are you know, three four levels deep, and some
28:43
Speaker 2: of them are extremely popular in others are ghost towns.
28:45
Speaker 2: And and you know, I think there's opportunities to improve
28:51
Speaker 2: the infrastructure that we already have built without having to
28:55
Speaker 2: like rebuild it from scrap.
29:00
Speaker 1: We'll be back in a minute with the lightning round.
29:14
Speaker 1: We're gonna finish with the lightning round. Tell me about
29:17
Speaker 1: the Meadowbrook apartments in Lawrence, Kansas.
29:20
Speaker 2: I grew up in Meadowbrook. It was a fine apartment complex.
29:24
Speaker 2: I moved there when I was four and moved out
29:28
Speaker 2: when I was eighteen. And so the center of Google
29:34
Speaker 2: Earth on the windows PC in particular, will zoom you
29:38
Speaker 2: into the bedroom of the apartment of my apartment building.
29:42
Speaker 1: Still still yeah, that's a robust easter egg.
29:49
Speaker 2: And there's another one in that category, which is that
29:51
Speaker 2: I a friend of mine who I went to the
29:54
Speaker 2: University of Kansas with, I hired him in two thousand
29:57
Speaker 2: and five to port Google Earth to the Macintosh, and
30:00
Speaker 2: he secretly changed the center of Google Earth on the
30:03
Speaker 2: mac to be Shanook, Kansas, where he's from. But he
30:07
Speaker 2: did it to the mains main intersection there in Chanut
30:11
Speaker 2: and the city is she who loved it so much.
30:13
Speaker 2: They actually built a mural on the street at that location.
30:16
Speaker 2: So if you ever run a mac give it a
30:17
Speaker 2: try and you'll see what it says.
30:22
Speaker 1: What's one tip for becoming a pac Man champion?
30:27
Speaker 3: The old old machines have patterns.
30:29
Speaker 2: And when I was, you know, the very first pac
30:32
Speaker 2: Man machine, if you knew the five key pattern and above,
30:36
Speaker 2: you know I was able to break a million. They're
30:39
Speaker 2: really professional people. I think the high score of the
30:41
Speaker 2: Infinite Perfect Score is three point three million, but I
30:45
Speaker 2: was able to get a million. But these days, you know,
30:48
Speaker 2: I still play Miss pac Man at home. I have
30:51
Speaker 2: a home machine, and you know, it's really about reaction,
30:55
Speaker 2: and you know it's if you play it on fast mode,
30:58
Speaker 2: it's all about speed.
31:01
Speaker 1: Do I recall that Miss pac Man was more deterministic
31:04
Speaker 1: than pac Man? Is that right?
31:05
Speaker 3: It's it's the other way around.
31:06
Speaker 2: When they launched Miss pac Man, they surprised people because
31:08
Speaker 2: they had extra logic and they actually randomized it, so
31:12
Speaker 2: literally no pattern really works even from the.
31:15
Speaker 1: Oh okay, so it's a it's a kind of more
31:17
Speaker 1: elegant game maybe at some level, or you could play
31:19
Speaker 1: it for longer. Anyways, how are your skills?
31:23
Speaker 2: My my I score is turned seventy nine thousand, My
31:26
Speaker 2: friend's high score is turned in seventy five thousand and
31:28
Speaker 2: and and that's where we're at.
31:32
Speaker 1: Are you currently training to win any other video game championships? No?
31:36
Speaker 3: I I my my only my only games.
31:40
Speaker 2: I've played a lot of video games in the early
31:42
Speaker 2: eighties and some Missile Command, miss pac Man, pac Man,
31:45
Speaker 2: and Galaga.
31:47
Speaker 3: I think we're one of my main games.
31:50
Speaker 1: I liked the Big Ball on Missile Command, the Arcade
31:53
Speaker 1: Missile Command, remember had that giant ball.
31:56
Speaker 3: Almost a bowling ball, and it had had momentum.
32:00
Speaker 1: But the the only games I.
32:02
Speaker 2: Really play now are are the New York Times word games,
32:06
Speaker 2: and I still play pokem let Go.
32:12
Speaker 1: Do you have any like favorite maps from history or
32:15
Speaker 1: whatever their maps you love, or like genius cartographers.
32:19
Speaker 3: There's there's one map.
32:22
Speaker 2: There's more of a sort of a data science presentation.
32:25
Speaker 2: There was a tough tie was a guy who wrote
32:28
Speaker 2: some really great books about data visualization, and he highlighted
32:31
Speaker 2: this map which has always stuck with me.
32:33
Speaker 3: And it was a single.
32:35
Speaker 2: Sheet map that showed Napoleon's you know, march to Russia,
32:39
Speaker 2: including the size of the army as it progressed, and
32:43
Speaker 2: then it decimated itself in Russia and then shrunk and
32:46
Speaker 2: shrunk and shrunk as it returned because they were, you know,
32:49
Speaker 2: going through winter with no food and just the visual
32:52
Speaker 2: The amount of information in that map, beyond the map
32:56
Speaker 2: itself is incredible.
32:59
Speaker 1: Anything you think shouldn't be mapped.
33:04
Speaker 3: The inside of people's homes, you know, you have to
33:08
Speaker 3: have a private space.
33:10
Speaker 1: What was the last time you got lost?
33:15
Speaker 3: It's a good question.
33:17
Speaker 2: Never for very long, you know, I don't remember that.
33:22
Speaker 2: Blue dot's pretty powerful. Tell me about catalog in your life.
33:28
Speaker 2: So I started doing it in nineteen ninety three. I
33:30
Speaker 2: started recording books, I read, movies I watched, and you know,
33:33
Speaker 2: various other events in my life. And I actually backpropagated
33:36
Speaker 2: and re remembered movies I'd seen before nineteen ninety three.
33:40
Speaker 2: So I have a list of every book, every movie,
33:43
Speaker 2: as well as many other life events all in a spreadsheet,
33:47
Speaker 2: and it's you know, as I grow older, it is
33:51
Speaker 2: you know, it becomes my memory because it's not super detailed,
33:56
Speaker 2: but you know, in many cases just the event itself
33:58
Speaker 2: existing reminds me of what I was doing that year,
34:01
Speaker 2: that month, that day.
34:03
Speaker 1: What like do you look back on it, like in
34:06
Speaker 1: the way people might look at old photos or something like,
34:08
Speaker 1: how do you how do you kind of use it
34:10
Speaker 1: or engage with it?
34:11
Speaker 3: I do.
34:11
Speaker 2: I looked back at like what movies was I watching
34:14
Speaker 2: in the last semester before I graduated high school? You know,
34:17
Speaker 2: how did I spend my time that year? And was
34:20
Speaker 2: what was available? And what's going on?
34:22
Speaker 1: And you know you have qualitative notes? Do you have
34:26
Speaker 1: ranking like are there interesting columns in those?
34:29
Speaker 2: I've gone back through and I have my top forty
34:31
Speaker 2: for let's see, what do I have my top forty
34:34
Speaker 2: for books? And I have my top two hundred for movies,
34:37
Speaker 2: and I have them somewhat ranked within.
34:39
Speaker 1: That top movie, top book.
34:42
Speaker 2: Let's see. I'll still say Contact for reasons. On the movie,
34:47
Speaker 2: I just find it very compelling, And on the book,
34:53
Speaker 2: I love Scholzie and Hugh Howie, So I almost want
34:59
Speaker 2: to say Silo but I'll still say three body problem.
35:02
Speaker 1: So so both both science fiction.
35:05
Speaker 2: I am unapassionately almost entirely science fiction.
35:10
Speaker 1: It's really I guess it shouldn't be surprising, but it's
35:13
Speaker 1: particularly it's persistently notable to me how influential science fiction
35:19
Speaker 1: is in the world. Like you know, the world we
35:23
Speaker 1: live in, people built it in some significant degree because
35:27
Speaker 1: they were inspired by science fiction in really specific ways.
35:30
Speaker 2: Absolutely, Carl Sagan wrote Contact, and you know it was
35:33
Speaker 2: hugely influential that way. But Neil Stephenson wrote snow Crash
35:40
Speaker 2: and described a version of Google Earth before we built it,
35:43
Speaker 2: and both John and I had read that book and
35:47
Speaker 2: feel at least partially inspired by that. There were other
35:50
Speaker 2: things that inspired us too, but that very specific description
35:53
Speaker 2: of a visualizable, holographic three D model of the planet
35:57
Speaker 2: that you could zoom into is very powerful.
36:00
Speaker 1: Thank you so much for your time, was very interesting
36:04
Speaker 1: to talk with you.
36:05
Speaker 3: Thank you.
36:13
Speaker 1: Brian McLendon is the chief technology officer at Niantic Spatial.
36:18
Speaker 1: Our show Today was produced by Gabriel Hunter Chang. It
36:22
Speaker 1: was edited by Lydia Jean Kott and engineered by Hansdale.
36:25
Speaker 1: She I'm Jacob Goldstein and We'll be back in a
36:27
Speaker 1: couple weeks with more episodes of What's Your Problem. Thanks
36:31
Speaker 1: for listening.