Latest / Automated with Brian Heater / Waymo’s Vincent Vanhoucke on Embodied AI and Robotics

Transcript

Waymo’s Vincent Vanhoucke on Embodied AI and Robotics · Automated with Brian Heater

• 0:00
  SPEAKER_01: It turned out that it had been the first time they were in a Waymo, and they hadn't told me that it was their first time riding.
• 0:08
  SPEAKER_01: So the first 10 minutes you can see them darting their eyes around and trying to get situated and experiencing the joys of autonomous driving for the first time with a little bit of you know nervousness initially.
• 0:21
  SPEAKER_01: And then they had the same experience as you know everybody who's ridden in a Waymo, which is that after 10 minutes you basically forget that you're in an autonomous car and it's a very comfortable ride and you don't really think about it anymore.
• 0:44
  SPEAKER_00: Happy New Year and welcome to another episode of Automated.
• 0:47
  SPEAKER_00: I am Brian Heater, the managing editor of A3, and I am very excited to bring you this week's conversation with Vincent Van Nook.
• 0:57
  SPEAKER_00: Vincent has a fascinating history in the space, including a long stint at Google Robotics.
• 1:04
  SPEAKER_00: These days, he is a distinguished engineer at Waymo, as well as the host of his own interview podcast, AI in Motion, which is not unlike automated, where it recorded in a moving vehicle.
• 1:18
  SPEAKER_00: Check that out on the podcasting platform of your choice.
• 1:22
  SPEAKER_00: And while you're there, why not like and subscribe to this program?
• 1:26
  SPEAKER_00: Easy peasy.
• 1:27
  SPEAKER_00: Thank you, Vincent, for taking the time to chat.
• 1:30
  SPEAKER_00: And I will see you on the other side.
• 1:34
  SPEAKER_00: Slightly different circumstances for you.
• 1:37
  SPEAKER_00: Is this um is it difficult getting prepared for an interview when you're stationary?
• 1:43
  SPEAKER_01: Yeah, it's been fun to do this series of video uh uh podcasts in the backseat of a WEMO in San Francisco, being mobile and uh also having a great conversation at the same time.
• 1:59
  SPEAKER_01: The uh experience of riding around the city while being able to have a quiet conversation and interesting one is one of the value propositions of having an autonomous driving car.
• 2:12
  SPEAKER_01: Uh being able to actually do work in the backseat.
• 2:16
  SPEAKER_01: It's becoming um my uh my favorite commute uh is to take a Waymo to work and uh get a head start on the on the work in the morning.
• 2:28
  SPEAKER_00: Yeah, I asked partially because I know a lot of very uh frustrated, sleep-deprived new parents they'll often uh take the child and kind of like drive around for a few hours to lull them to sleep.
• 2:40
  SPEAKER_00: So I assume that moving from the studio into an autonomous vehicle probably has some sort of impact on maybe the quality or or the content of the conversation.
• 2:53
  SPEAKER_01: It's been surprisingly uh relaxing to have the city around us, but also being able to have a conversation at the same time.
• 3:04
  SPEAKER_01: It's an environment that you know is surprisingly comfortable.
• 3:08
  SPEAKER_01: Uh you don't think of you know driving and having a conversation at the same time as something that uh you know you should be doing routinely.
• 3:17
  SPEAKER_01: Um but once you have uh an autonomous driver in the front seat, uh it's it's a very different experience.
• 3:25
  SPEAKER_00: I think that's right.
• 3:26
  SPEAKER_00: And and something I figured out a long time ago that I need to get back to is one of the keys to having a a really good and really candid conversation is to make the other person kind of forget that they're having a conversation.
• 3:39
  SPEAKER_00: And when you're just driving around in a car and you've just got your like lav mic on, then they probably, you know, they just feel like they're talking, they're chatting AI with a colleague.
• 3:49
  SPEAKER_01: I I had one guest that um I I won't say who, um, but I it turned out that it had been the first time they were in a Waymo, and they hadn't told me that it was their first time riding.
• 4:02
  SPEAKER_01: So the first 10 minutes you can see the them uh you know darting their eyes around and trying to get situated and experiencing the the joys of autonomous driving for the first time with a little bit of you know nervousness initially.
• 4:17
  SPEAKER_01: And then they had the same experience as you know, everybody who's ridden in a Waymo, uh, which is that after 10 minutes you basically forget that you're in an autonomous car and it's a very comfortable ride and you you you don't really think about it anymore.
• 4:31
  SPEAKER_01: But the first 10 minutes where uh I could tell that there was uh you know some novelty there, and I had no idea.
• 4:38
  SPEAKER_00: Aaron Powell One of the things I'm I'm really interested in is this new, I don't know if it's a new stage for Google or like a new approach, because um in March, AstroTeller launched the uh the Moonshot podcast, and then now you're hosting this AI in motion, which is uh you know what we're discussing happening in the back of a Waymo.
• 4:59
  SPEAKER_00: Is this like a new thing for Google?
• 5:01
  SPEAKER_00: Is this a bit to maybe like uh like increase transparency or like speak more directly with the consumers?
• 5:08
  SPEAKER_01: We've always tried to speak very directly to uh our consumers and to the the people that uh are our writers and the public and the the general public.
• 5:22
  SPEAKER_01: This series is a little bit more tailored towards the AI community, people who are interested in embodied AI in general.
• 5:32
  SPEAKER_01: Um, there is a lot of talk about AI in general uh right now, but not so much uh discussion around this next wave of AI, which connects AI to the physical world.
• 5:46
  SPEAKER_01: So we thought it would be a good opportunity to start having that conversation because Waymo is to some degree the very first uh company to bring embodied AI to the real world in a way that uh sort of connects to uh the general public, not just to sort of, it's very different from industrial robotics or uh logistics application of robotics.
• 6:13
  SPEAKER_01: It's something that lives in the human world and is tailored to really sort of impact uh individuals in the communities where they live.
• 6:26
  SPEAKER_00: It's interesting because uh obviously Astro, do you just call them Astro?
• 6:32
  SPEAKER_00: Just Astro?
• 6:33
  SPEAKER_00: Astro's uh Astro's podcast is obviously tailored to to a general audience, you know, watching yours, like certainly you you should have some familiarity with with AI.
• 6:45
  SPEAKER_00: Uh and I I know that you know, in terms of what I do, something that's made me successful is probably too strong a word for my career.
• 6:55
  SPEAKER_00: But you know, something that that's that's been good for my career, I'll say, is my ability to sort of speak to a more general audience.
• 7:02
  SPEAKER_00: Um you know, in in in terms of you sitting in the back of a car and talking to some very smart people about some very high-concept ideas, are you attempting to distill it for a more general audience?
• 7:18
  SPEAKER_01: It's always a tension that I always in the back of my mind want to talk shop and you know use all the secret acronyms that uh they will understand and I will understand, and all the shortcuts, the cognitive shortcuts that we are used to uh use when we talk with people in the field.
• 7:37
  SPEAKER_01: It takes a very deliberate effort to uh unpack all that and really try to communicate at a level that they understand what I'm talking about, I understand what I'm talking about, but also the public that we are trying to reach out to understands uh and has a shared basis that they they can understand where the conversation is going.
• 7:58
  SPEAKER_01: So it's new to me.
• 8:00
  SPEAKER_01: Uh this is kind of uh a new I I always uh said in in my career, I've always tried to try and maximize my imposter syndrome.
• 8:09
  SPEAKER_01: Uh basically uh live at the edge of what I'm comfortable with.
• 8:15
  SPEAKER_01: And as long as I'm surrounded with people who know the area much better than me and can help me and can uh improve my uh ability to operate.
• 8:27
  SPEAKER_01: So this is definitely something at the edge of my uh comfort zone, if you will, to uh be interviewing other people and trying to communicate things.
• 8:38
  SPEAKER_01: Uh I'm very used to speaking in uh you know conferences and in settings that are very much tailored to uh a technical audience and reaching out to a more general public uh is very exciting.
• 8:53
  SPEAKER_01: It's and it's uh it's it's a bit of a new experience for me.
• 8:57
  SPEAKER_00: One of the, I guess, um something that you and I definitely have in common is that I would say, and let me know if you don't agree with this, but um we've both taken what I would classify as a uh circuitous route into robotics.
• 9:16
  SPEAKER_01: Yes, very much so.
• 9:18
  SPEAKER_01: I started from AI.
• 9:22
  SPEAKER_01: I in fact I did my PhD in uh uh speech recognition originally, and moved on to work on more general uh machine learning, including computer vision, and took that route to going into robotics uh much later, trying to bring kind of a more fresh perspective on robotics through the lens of AI.
• 9:48
  SPEAKER_00: Did you is this a place that you I mean, not necessarily autonomous vehicles, but is robotics a place that you anticipated you would end up eventually?
• 9:59
  SPEAKER_01: Not really.
• 10:00
  SPEAKER_01: Uh it was very serendipitous in the sense that uh we had been working on AI and deep learning for a number of years.
• 10:10
  SPEAKER_01: And uh a lot of the work there initially was very revolutionary.
• 10:15
  SPEAKER_01: There was a lot of very uh you know zero-to-one innovation, and at some point sort of evolved, particularly in computer vision, towards something a bit more incremental, something where uh you know you were basically iterating on different architectures and improving the efficiency of the models, but essentially solving the same problems.
• 10:37
  SPEAKER_01: And I kind of took at some point the perspective of you know, assume computer vision is going to be solved uh in on some time horizon.
• 10:47
  SPEAKER_01: That's you know, now that we are on the right track, we have the right basic framework to think about computer vision, assume that other smart people will be able to take this all the way to human level or you know, superhuman performance.
• 11:05
  SPEAKER_01: What are the consequences of that?
• 11:08
  SPEAKER_01: And when we went down that thought process, two things kind of came out.
• 11:13
  SPEAKER_01: Uh, we thought that uh healthcare was going to be uh an area where you know radiology and uh imaging was going to be revolutionized, and a number of my colleagues went into that route from the perspective of computer vision.
• 11:29
  SPEAKER_01: And uh robotics was the other one where we thought suddenly if we can really apply, if we have a perfect perception system, uh robotics will look completely different.
• 11:40
  SPEAKER_01: So at the time I started uh sort of on the basis of that assumption, you know, how do we make that happen?
• 11:47
  SPEAKER_01: Like how do we uh leverage that new uh way of uh thinking about perception and apply it to robotics?
• 11:57
  SPEAKER_01: At the time, um we had the opportunity to work with Google X.
• 12:04
  SPEAKER_01: A number of my colleagues uh were there and um they had a bunch of arms lying around that had been the result of an acquisition that robot arms to be called.
• 12:17
  SPEAKER_01: Robot arms, yes, robot arms.
• 12:19
  SPEAKER_01: And uh they were basically there gathering dust.
• 12:23
  SPEAKER_01: And uh we decided what if we use those arms and build a data collection system for essentially picking objects, right?
• 12:35
  SPEAKER_01: Having training um AI model to learn how to pick objects and basically apply all the recipes that worked well in computer vision, not to recognize pixels on an image, but to recognize what is the correct action to be taking uh given a specific roboactic task, which in that case was about picking.
• 12:59
  SPEAKER_01: And the that was a bit of a skunk work project.
• 13:02
  SPEAKER_01: Uh, and the the result of that was uh very, very surprising in the sense that we were able to quickly learn uh with a reasonable amount of data picking strategies that generalized very well, that supported basically picking any object that we presented in front of the robot.
• 13:23
  SPEAKER_01: And at the time, this idea of being able to generalize in robotics was not really a dominant concept.
• 13:32
  SPEAKER_01: A lot of the strategies for manipulation, for picking, uh were very bespoke to, you know, you assume that you had knew what object you were going to be manipulating, and then optimized a robotic policy to that object.
• 13:48
  SPEAKER_01: So suddenly we had that opportunity to think about it in a generalized framework.
• 13:55
  SPEAKER_01: And we ended up deciding to really pivot our computer vision efforts to robotics on that premise and trying to see where we could take it and how far we could go with this mix of AI and robotics.
• 14:09
  SPEAKER_00: Yeah, it's it's it's a new way of thinking about systems, obviously, you know, for decades.
• 14:15
  SPEAKER_00: And even still, uh robots are largely built for like single purpose activities, right?
• 14:22
  SPEAKER_00: And and for doing them over and over again.
• 14:25
  SPEAKER_00: And I wonder now, now that we're talking about um general purpose through these AI systems, if that's maybe like shifted perception and certainly the public's perception to give them kind of an unrealistic expectation of what systems are uh actually able to do now and how far we are from like true general purpose robotics.
• 14:50
  SPEAKER_01: Yeah, there is this tension between sort of generalists versus specialists and uh you know how much progress we can make on very specialist tasks using very generalist systems.
• 15:05
  SPEAKER_01: Uh, in fact, recently interviewed on the podcast of Sergei Levine, who was one of my co-conspirators in this journey of bringing computer vision and robotics together, and asked him, like, do you think it's uh a tension really that uh, you know, there is the specialists on one end of the spectrum and the generalists at the other end of the spectrum?
• 15:28
  SPEAKER_01: His perspective was that no, that there is very much a uh, you know, a that uh we will end up in a place where the generalist systems are also the best specialists.
• 15:41
  SPEAKER_01: That transition happened in AI, in the sense that it used to be that you'd build machine learning models for every specific task independently, right?
• 15:52
  SPEAKER_01: For machine translation, for computer vision, for uh natural language processing.
• 15:57
  SPEAKER_01: And suddenly we're in a place now with the large language models and large multimodal models where the generalists basically beat the specialist at every task.
• 16:08
  SPEAKER_01: Or you can turn a generalist into a better specialist with very little fine-tuning or even just by prompting the model.
• 16:16
  SPEAKER_01: So that transition has happened in a lot of areas of AI.
• 16:21
  SPEAKER_01: And the question today is: are we going to see this transition happen in embodied AI in the same way?
• 16:28
  SPEAKER_01: And what it will it take for that transition to happen?
• 16:32
  SPEAKER_01: A lot of people are making the bet on it's really just a matter of scale.
• 16:38
  SPEAKER_01: So we need more data and we need uh you know, more scalable ways of collecting data.
• 16:43
  SPEAKER_01: We'll see if that's uh if that's true.
• 16:46
  SPEAKER_01: Um, there are also a lot of bets on simulation, like can we do a lot of simulation and generate a lot of data in that way?
• 16:54
  SPEAKER_01: Um, I think the jury is still relatively out on what will win out and if you know we're going to be in a place where specialist systems are going to be much better than generalists for a long time.
• 17:08
  SPEAKER_01: Um but that question is a is a very important scientific question to be answering for the future of robotics.
• 17:16
  SPEAKER_00: Yeah, it's really interesting because I feel like there's almost this tension, even like, I won't say at Google.
• 17:22
  SPEAKER_00: Obviously, intrinsic is kind of its own thing now, maybe you know, under the larger alphabet umbrella.
• 17:28
  SPEAKER_00: But um to a certain extent, they are really kind of still working on these specialized systems.
• 17:34
  SPEAKER_00: Google's obviously placed its bet into general systems to such a point that you know they're like, oh, Gemini, Gemini works for LOMs, let's just port that over.
• 17:44
  SPEAKER_00: We'll keep the name to um and and and we'll we'll bring that into the physical AI world.
• 17:49
  SPEAKER_00: And it sounds like you're leaning in that latter direction that even if I've got a very simple at some point in the future, even if I have an extremely simple robot arm, that I'm still gonna be in some way using one of these really sort of large, robust models to have it continue to do its job.
• 18:09
  SPEAKER_01: Back in the robotics world at Google, we collaborated a lot with intrinsic.
• 18:14
  SPEAKER_01: In fact, we were just you know a few desks away from each other, and there was a lot of uh collaboration going on on uh trying to see how much of bootstrapping from a generalized model can improve uh bespoke tasks.
• 18:29
  SPEAKER_01: They were working on very, very hard tasks.
• 18:33
  SPEAKER_01: Uh, and a lot of the focus was on very precise uh dextrus manipulation at a time, uh that's a few years back, where dexterity was not at the forte of those journalist models.
• 18:48
  SPEAKER_01: They were very clumsy in the sense that they would do well at very high-level grasping tasks, for example.
• 18:57
  SPEAKER_01: But as soon as you were trying to do very fine manipulation, the the there was just no good way of doing that.
• 19:05
  SPEAKER_01: That is evolving rapidly, right?
• 19:07
  SPEAKER_01: So if you look at the latest kind of policies that are being learned, uh whether it's with uh Gemini for Robotics or companies like uh Physical Intelligence or Dyna or others that are bringing more dexterity to this very uh to this large model space?
• 19:31
  SPEAKER_01: Uh there's a lot of innovation happening in that space right now that I think is opening up the path to you know having your cake and eating it too, or eating your cake and having it too.
• 19:43
  SPEAKER_00: So a lot of this ultimately I think comes down to timeline, right?
• 19:46
  SPEAKER_00: So the question is are we making a product that we want to ship right now, or are we making something that's gonna work at some time in the future?
• 19:52
  SPEAKER_00: And one of the things that you know, one of the benefits I suppose that you get from working under Google, this is a benefit that I would say Waymo has had in the long run, is the ability to really sort of like develop these products over time.
• 20:08
  SPEAKER_00: And that that's what Gemini, Gemini is gonna require, right, that this development for a long period of time to get to that state where a real sort of generalized system is able to adapt to any kind of embodiment.
• 20:21
  SPEAKER_01: The question is, what are you trying to maximize?
• 20:24
  SPEAKER_01: Are you trying to maximize utility today, or are you trying to maximize your rate of learning?
• 20:30
  SPEAKER_01: Right?
• 20:31
  SPEAKER_01: Right now, the most learning we are getting is in this space of very large uh learning, large AI models, because it's it's very um it's very greenfield.
• 20:44
  SPEAKER_01: There is a lot to explore in that space, and the capability that we're seeing uh emerging from that line of research are really novel.
• 20:55
  SPEAKER_01: It's a lot about generalization, it's a lot about uh being able to do things cross-embodiment, it's about um new workflows that enable you to uh learn policies directly from data, not from uh sort of essentially engineering and elbow grease.
• 21:15
  SPEAKER_01: Um so, in terms of like the where the research should be, it's definitely uh very there's a lot a lot of things happening in that space, in contrast to sort of more bespoke approaches that um have, you know, basically where we're not evolving nearly as fast from a capability standpoint.
• 21:38
  SPEAKER_01: If you're trying to get to a product today, it's a different story, right?
• 21:43
  SPEAKER_01: So I think there is a lot more uh uncertainty on how quickly those systems will be ready for prime time.
• 21:53
  SPEAKER_01: Um, in a sense, there's this interesting question uh parallel to the AV world, right, to the autonomous driving world.
• 22:00
  SPEAKER_01: World where essentially when Waymo started back in 2009, there was the promise of self-driving as a product on the horizon, but the technology wasn't there.
• 22:14
  SPEAKER_01: And in fact, it took several generations of technology before Waymo got into a place where a commercial product was viable.
• 22:24
  SPEAKER_01: And being able to sustain an investment through this cycle of technology evolution is what enabled Waymo to be where it is today.
• 22:35
  SPEAKER_01: So the question is: are those AI-driven robotics uh approaches going to have to go through the same cycles of technology development?
• 22:47
  SPEAKER_01: I think the shortcut that they have is that we're in the right sort of technology generation for things to work.
• 22:57
  SPEAKER_01: Like Wemo had to go through the COVNET revolution and then had to go through the Transformer Revolution and uh sort of learn from all those advances.
• 23:09
  SPEAKER_01: I don't think there is the need for that many revolutions for robotics and AI to work well together.
• 23:18
  SPEAKER_01: But we'll see.
• 23:19
  SPEAKER_01: Uh, I might be too optimistic about this, right?
• 23:21
  SPEAKER_01: It's also possible, you know, it took Waymo uh basically uh more than 10 years, 15 years to get to the point where we are today.
• 23:31
  SPEAKER_01: Um, it's very possible that the embodied AI for manipulation in general will also take many years to uh emerge as a uh as a commercial product that's viable.
• 23:44
  SPEAKER_00: Uh so obviously at that point in the early days, you were um an outsider specifically to Waymo, to self-driving cars.
• 23:51
  SPEAKER_00: Um I mean, is it fair to say that, you know, in hindsight, that that it has taken a fair bit longer than you anticipated to get to a place where we'll have full uh uh autonomy on the roads?
• 24:09
  SPEAKER_01: So I started uh collaborating with Waymo actually uh a while back, uh in back in 2013.
• 24:19
  SPEAKER_01: Uh I was on the Google side and working on computer vision then, and we were as part of the Google Brain team, we had just gone through the big deep learning revolution with uh Jeffrey Hinton and uh Iliasitzgiver and Alex Khrushewski to bring the early versions of Alexnet into Waymo to do very simple perception tasks at the time.
• 24:44
  SPEAKER_01: So I got to see a little bit the early days of sort of what it took to basically reinvent the stack or part of the stack uh at the time and how really being nimble about the technology developments and following what the technology uh what evolutions of technology uh were available enabled the company to sort of pivot very quickly towards whatever the state of the art uh was quickly becoming.
• 25:21
  SPEAKER_01: Um it took a long time, but a driver that punctuated that timeline was really that the general technology was evolving at the same time.
• 25:32
  SPEAKER_01: Again, there was like the pre-AlexNet days, uh the pre-deep learning days, and then there was the convolutional net days, and then the transformer days, and now we're going more into the foundation model days.
• 25:45
  SPEAKER_01: Uh, all of these generations of technology had to happen for the self-driving system to be at the level uh that it is today.
• 25:57
  SPEAKER_01: The AI and robotics may not have to go through the same number of upheavals to be able to become productive.
• 26:05
  SPEAKER_01: We'll see.
• 26:06
  SPEAKER_01: I think it's an interesting question of how many generations of technology do we need to uh achieve the same outcomes in general sort of navigation manipulation.
• 26:18
  SPEAKER_00: I obviously come at a lot of this from the consumer side of things, having covered consumer electronics for many years.
• 26:24
  SPEAKER_00: And in that world, they've got this concept of sure locking, right?
• 26:29
  SPEAKER_00: Which is specific to Apple and an Apple app where Apple developed a feature and baked a feature into the operating system that effectively killed a third party that was doing a similar thing.
• 26:43
  SPEAKER_00: And I think there's an analogy here, and and this really speaks to what you're talking about when you talk about being nimble.
• 26:50
  SPEAKER_00: Um and this is especially the case with AI, because as AI develops, we're seeing all of these really interesting and uh hopeful startups kind of be wiped away because of like just a complete uh a new feature to AI.
• 27:11
  SPEAKER_00: Um how as a startup and and as a larger startup, how how can you stay nimble and how can you continue to pivot when things are changing so quickly?
• 27:25
  SPEAKER_01: What I've seen happening more is that there is a ton of experimentation in the startup world right now.
• 27:32
  SPEAKER_01: Yes, there is a lot of churn as a result.
• 27:34
  SPEAKER_01: A lot of people are trying to figure out hey, this is a new technology, this is uh a new way of thinking about even the business model has to change because the technology sort of has different implications from uh from you know where the value add is coming and uh how the consumer interface is with the technology.
• 27:55
  SPEAKER_01: So there is a lot of you know uh sort of creative destruction that's happening in the general AI space.
• 28:02
  SPEAKER_01: I think that's that's a very healthy uh state for things to be right now, given how uh new the AI uh space is, or renewed, if you will, uh through the the modern breed of large language model and large multimodal models is.
• 28:21
  SPEAKER_01: So I think that's going to continue to some degree, but uh it's also um it's it's it's an age of innovation.
• 28:31
  SPEAKER_00: So you referred to some of your uh early work or I guess early experimentation uh in and around robotics at Google as being skunk works, right?
• 28:41
  SPEAKER_00: Which you know obviously like kind of puts it in line uh with the work that X was doing.
• 28:46
  SPEAKER_00: Um at that point, was Google considering robotics a moonshot in the same way that it had considered self-driving cars a moonshot early on?
• 28:59
  SPEAKER_01: There was a big effort at Alphabet at large uh in uh before my time, uh that uh before I got involved in robotics to really give an earnest shot at uh building something significant in the robotics space.
• 29:20
  SPEAKER_01: Uh that was uh all of that was part of Google X.
• 29:25
  SPEAKER_01: There were a bunch of acquisitions, uh uh, Boston Dynamics, uh Shaft, and a bunch of others got acquired around that time.
• 29:34
  SPEAKER_01: That effort basically was a little bit ahead of its time, right?
• 29:38
  SPEAKER_01: I think the goal there that uh Larry and Sergei had was to bring this embodied AI about at a time where uh there was not a lot of uh connections between machine learning, AI, large data systems, uh the kind of things that had served Google well in other areas and robotics.
• 30:03
  SPEAKER_01: And they tried to sort of bring that about at the time, and it was probably too early.
• 30:07
  SPEAKER_01: I I don't know, I wasn't involved in it, but you know, from uh how things went, uh the this was probably premature at the time.
• 30:17
  SPEAKER_01: That said, sort of really good things came out of that, you know, sort of melting pot at the time.
• 30:23
  SPEAKER_01: Like you mentioned intrinsic earlier, uh, that's one of the companies that emerged from uh that um initial uh thrust and that has have been very successful uh uh since.
• 30:38
  SPEAKER_01: At the time, the piece that was missing was that I think a lot of robotics was really focused, uh those robotics efforts were really focused on hardware.
• 30:50
  SPEAKER_01: And uh the hardware innovation was ahead of the software in a sense, was ahead of the intelligence.
• 31:00
  SPEAKER_01: Uh, we're in a much better place today where sort of the software and hardware, now it's a lot more of a question of how do we make the hardware and the software work together well.
• 31:13
  SPEAKER_01: But at the time, uh a lot of the the intelligence and the software and the ability to control the robots uh for the kind of tasks that were envisioned, the kind of tasks that were not easy to um sort of control where the environment wasn't controlled, where uh you had to work in an environment where uh humans were involved, and uh the those there was very, very unstructured, that ability wasn't there.
• 31:47
  SPEAKER_01: Um, it's still not there today, but at least the path uh is a lot clearer towards that.
• 31:53
  SPEAKER_00: It's interesting and and fairly um enlightening to hear you describe it in that way, because again, I was covering it from the outside at the time, and something that struck us all was how, I guess I would say disparate, how diverse the kinds of companies that a Google was acquiring were at the time.
• 32:13
  SPEAKER_00: Like in addition to some of the ones you mentioned, I think they acquired Boton Dumbly, which was like a special effects company.
• 32:18
  SPEAKER_00: They did special effects for gravity.
• 32:20
  SPEAKER_00: And the big question was like, how are they taking all of these companies that work in very different spaces and mashing them together to create a final product?
• 32:30
  SPEAKER_00: And and and I think it makes more sense now in hindsight if you're describing it in what's kind of perhaps more abstract than any of us on the outside were thinking of at the time, was to build a language or you know, to build a physical AI system for robotics.
• 32:51
  SPEAKER_01: Yeah, I think that was the that that was the the governing idea behind this.
• 32:58
  SPEAKER_01: Uh and you're right that uh it ended up being a number of very disparate companies.
• 33:04
  SPEAKER_01: Uh that and it it turned out that uh you know putting a bunch of hardware companies and putting them into a room and saying, now please do AI was a lot harder.
• 33:13
  SPEAKER_01: Uh it was worth a shot, yeah.
• 33:16
  SPEAKER_01: It was worth a shot.
• 33:17
  SPEAKER_01: Well, it's it's it's been an interesting sort of uh you know, sort of uh I'm I'm very glad that uh uh some of those companies sort of uh were able to uh go through that upheaval and actually uh survive.
• 33:29
  SPEAKER_01: Like Boston Dynamics is a is is a good example of that, where they are they're very successful today.
• 33:35
  SPEAKER_01: Um the the experiment was to some degree a very expensive experiment, uh but uh one that ended up uh sort of forging uh a very good uh idea of what it took to really bring about uh sort of the kind of embodied intelligence that was envisioned at the time.
• 33:57
  SPEAKER_01: Um we learned from this mostly this lesson of we need to bring a software engineering perspective to that equation.
• 34:07
  SPEAKER_01: We need to bring a machine learning and AI perspective to that equation.
• 34:12
  SPEAKER_01: Um at the time, the the ML community and the robotics community, there was not a lot of common uh ground between the two.
• 34:23
  SPEAKER_01: In fact, I started a conference, an academic conference called the Conference on Robot Learning, specifically to bring the robotics and machine learning community into the same room.
• 34:35
  SPEAKER_01: It's been going on for nine years, and it's been very successful at really bringing the communities uh to talk and uh forging sort of a common language that the communities could uh bring back to their um to the broader sort of AI community and uh and build benchmarks and build uh ways of uh evaluating performance, ways of measuring progress.
• 35:02
  SPEAKER_01: Um and I'm I'm I'm very happy that this is now a thing, that uh this AI plus robotics community is um is uh is is a thriving community.
• 35:14
  SPEAKER_00: Yeah, anybody who watches this show with any regularity knows that one of my special interests is Willow Garage.
• 35:21
  SPEAKER_00: And that's because also ahead of its time, also I think it's fair to say it didn't achieve its end goal, but still managed to leave a really large mark on robotics, right?
• 35:32
  SPEAKER_00: And still launched a thousand ships.
• 35:34
  SPEAKER_00: And I think that there is a very real parallel that you can draw to um to this Google Robotics project, you know, like uh obviously, like some unfortunate consequences, you know, not every single company made it, but I think probably one of the upsides of it that you do have the situation when you stick a bunch of roboticists in the same room and and ask to work on a project, is you've got a lot of a lot of really smart people cross-pollinating and working with each other.
• 36:00
  SPEAKER_00: And that's why some of this work came out of that.
• 36:04
  SPEAKER_01: This is also one thing that um I'm very proud of uh from um my my time at uh Google Robotics is that uh if you look at a lot of the companies that are innovating today, uh a lot of the people that are uh leading the AI efforts in those companies are people who at one time were uh part of uh Google Robotics.
• 36:29
  SPEAKER_01: So it's uh physical intelligence, it's uh figure, uh it's OneX, uh it's a number of those companies that uh have uh even at Tesla, uh the number of folks that have gone through that uh sort of new way of thinking about uh robotics that was emerging at the time, have gone on to uh work on uh kind of the next frontier of robotics companies.
• 36:59
  SPEAKER_00: I'm looking back at, I remember covering everyday robots, and I don't know how familiar people are with them, but you know, they they're kind of in that in-between space, right?
• 37:10
  SPEAKER_00: They they they came after that initial push and uh ahead of where we are now, and they kind of effectively like that team sort of like migrated into um deep mind robotics.
• 37:22
  SPEAKER_00: Is that pretty fair to say?
• 37:24
  SPEAKER_00: But um I'm looking back on the work that they were doing at the time, and like it's pretty clearly they they were having the same conversations about physical AI that we're having right now.
• 37:36
  SPEAKER_01: Yeah, yeah.
• 37:37
  SPEAKER_01: They were very much ahead of the time and they they were a very you know key partner for us at Brain Robotics to really enable uh that junction.
• 37:49
  SPEAKER_01: They had very talented people both on the AI side and on the hardware side who understood that you know something had to change and evolve if we wanted to build robots that were driven by uh AI uh approaches.
• 38:06
  SPEAKER_01: Uh it went against a lot of the sort of predominant uh orthodoxy.
• 38:14
  SPEAKER_01: Uh a lot of the uh robotic systems in the past were very much designed around so the, I want to say the ROS framework.
• 38:23
  SPEAKER_01: Uh, this idea that everything is extremely modular and they only talk to each other through uh message passing.
• 38:30
  SPEAKER_01: Uh, that kind of went against the idea of being able to optimize things end-to-end.
• 38:37
  SPEAKER_01: And so we had to really rethink the stack so that we could do sort of end-to-end optimization and connect the perception all the way to the controls without having uh a very complicated software stack uh to stitch things in between.
• 38:54
  SPEAKER_01: Um, and understanding that there was a big shift that went all the way down to the operating system essentially, that didn't need to happen.
• 39:04
  SPEAKER_01: Uh, and having that flexibility to think about the problem in a different way was clearly a huge enabler for the kind of research that we were doing.
• 39:16
  SPEAKER_01: They also had a very strong sort of uh product vision of let's build a very uh inexpensive robot that has that can is mobile, that has a manipulator, and that has enough uh compute uh on the robots that we can do very intelligent things.
• 39:37
  SPEAKER_01: And at the time, there was not really any off-the-shelf robots that had this combination of mobile manipulation and compute, uh, and that really sort of turbocharged in uh in effect the research.
• 39:50
  SPEAKER_00: Is there anything now that like fills that void?
• 39:53
  SPEAKER_00: I know that like NVIDIA's got some of these kind of like platform robots that they're using right now, but I'm trying to think of something that's like a direct analogy.
• 40:02
  SPEAKER_01: There is a lot of startups that are building mobile manipulator robots today, uh, whether it's humanoids or uh non-humanoids, uh, there's a lot of wheeled robots with the colour.
• 40:15
  SPEAKER_01: Kinesi.
• 40:17
  SPEAKER_01: Uh yeah, yeah.
• 40:19
  SPEAKER_01: And uh a lot of those uh now it's you can actually buy robots off the shelf.
• 40:25
  SPEAKER_01: The price has gone down dramatically in the last five years.
• 40:30
  SPEAKER_01: Um, and uh so there is a lot more diversity of platforms.
• 40:34
  SPEAKER_01: We've also kind of established more and more that you could build a foundation that is um essentially embodiment agnostic and then fine-tune it to uh all of those different platforms.
• 40:49
  SPEAKER_01: So there is a lot more belief in the ability to bring to build a very generalist uh robotic brain, essentially, and be able to retarget it to a specific platform uh relatively easily with uh some amount of data.
• 41:06
  SPEAKER_01: Um but uh the the world has really the the ecosystem has exploded uh essentially.
• 41:13
  SPEAKER_00: 2011, you know, in real terms feels like you know yesterday, but in robotics terms feels like a million years ago.
• 41:22
  SPEAKER_00: What you were working, I think, at Google Brain at the time.
• 41:27
  SPEAKER_00: Um what was deep learning in 2011?
• 41:32
  SPEAKER_01: In 2011, I was working mostly on speech recognition at the time.
• 41:38
  SPEAKER_01: And uh we had a very different set of tools than deep learning uh powering all the speech recognition algorithms.
• 41:46
  SPEAKER_01: Um and then I started collaborating with a gentleman called Jeffrey Hinton, uh, who uh is now uh called sort of the godfather of AI or one of the godfathers of AI.
• 42:01
  SPEAKER_01: And he at the time he, one of his students, came to work with us and promptly uh over the course of a summer completely blew our out-of-the-water or performance on all the benchmarks we had on speech recognition.
• 42:18
  SPEAKER_01: Um, and we ended up uh deploying what I think uh was the very first large-scale deep learning system anywhere uh with Google Voice Search.
• 42:29
  SPEAKER_01: And that was kind of the embryo of um Google Brain existed at the time as a project at Google X.
• 42:41
  SPEAKER_01: And uh quickly people sort of realized as a result of this results that there was a lot of very direct applications of deep learning that could really benefit Google.
• 42:51
  SPEAKER_01: And so we brought uh Google Brain back into Google, and that's when I joined uh the effort then.
• 42:59
  SPEAKER_01: Um, there was really a belief or a uh an idea then that this new way of doing very data-driven uh learning in a way that was very generic, that could cut across all the different products at Google, could really revolutionize the way we think about how do we do translation, how do we do search, how do we do ads, how do we do uh image recognition and things like this, and bring all of that onto a common substrate that if Google could invest in that substrate, suddenly it could really make an impact on all those products with very good efficiency scales.
• 43:40
  SPEAKER_01: Um, so the the next few years were basically spent uh looking at how do we essentially disrupt Google from within by bringing deep learning to all the different areas of natural language processing, computer vision, uh, machine translation, and everything.
• 43:59
  SPEAKER_01: And uh A lot of that work sort of basically laid the foundation for this large-scale AI that we see today.
• 44:08
  SPEAKER_01: At the time, it was very much a it was a big leap of faith, right?
• 44:14
  SPEAKER_01: It was uh, you know, there was this idea, um, and uh a few people uh were, you know, like Jeffrey Hinton or Andrew Ng or uh Jeff Dean were really the visionaries that really saw this as a possibility and uh decided to sort of bring the investments uh to be commensurate with that potential opportunity.
• 44:37
  SPEAKER_00: Looking at your CV from with like the 10,000-foot view like just doesn't make a lot of sense until you realize that that's kind of the inflection point right there.
• 44:48
  SPEAKER_00: Like that's how you go from Raya to Waymo, right?
• 44:52
  SPEAKER_00: That's how you go from you know facial recognition to uh you know a shopping site and then eventually end up at self-driving cars.
• 45:02
  SPEAKER_01: Yeah, before I joined Google, I was in this small startup that was basically trying to do computer vision before any of it really worked.
• 45:12
  SPEAKER_01: Uh some of it worked, you could do face detection uh quite reliably at the time, but anything beyond that was very, very challenging.
• 45:21
  SPEAKER_01: But we were trying to bring computer vision to the general public, very much like Google Photos is uh, you know, has brought a lot of computer vision capabilities to your photo collection today.
• 45:35
  SPEAKER_01: Um we were trying to do that uh pre-deep learning, and so uh we had some very targeted successes, um, but it was uh it was again a little bit ahead of its time, but still in the vein of, hey, how can what can we do with the modern or what was the modern AI technology at the time, and how do we bring that to uh users in a way that uh they can connect with it?
• 46:02
  SPEAKER_00: At this point, you're working on something that I guess it's fair to say is much closer than to, you know, I mean it it is technically, it's out there, you know, obviously again, like in San Francisco and now an increasing number of cities, like you can catch a Waymo.
• 46:18
  SPEAKER_00: You had that big highway news, which I was like, I think maybe even like undercovered as far as how substantial of news that was.
• 46:27
  SPEAKER_00: How does your job shift when that kind of change is taking place at a company?
• 46:34
  SPEAKER_01: Well, a lot of the work today is about enabling scale, right?
• 46:39
  SPEAKER_01: So you're saying you were talking about sort of the freeway uh launching on a uh our service on freeways has some similarities to driving on surface streets, but also has material differences.
• 46:54
  SPEAKER_01: Things, you know, cars go faster.
• 46:56
  SPEAKER_01: Um, when something goes wrong, you cannot just stop.
• 47:00
  SPEAKER_01: You have to uh sort of react to it in a way that doesn't disrupt uh the traffic.
• 47:07
  SPEAKER_01: There's a lot of uh different uh and the geometry of freeways changes all the time, so you have to understand uh all the the dynamic uh how dynamic those things are.
• 47:20
  SPEAKER_01: A lot of those challenges sort of uh build on top of the existing challenges.
• 47:28
  SPEAKER_01: Um to some degree the scaling uh problem always brings about very different things to work on.
• 47:38
  SPEAKER_01: I often say if you if you drive a million miles a week, you see the things that happen once in a million every week.
• 47:47
  SPEAKER_01: If you drive 10 million miles a week, you see the things that happen once in 10 million uh every week.
• 47:54
  SPEAKER_00: And it turns out cases are no longer that far on the edge.
• 47:57
  SPEAKER_01: No, and it turns out something that happens once in 10 million miles is very different from something that happens once in one million miles.
• 48:07
  SPEAKER_01: They all have their own very different uh peculiarities.
• 48:12
  SPEAKER_01: And so scaling and generalizing sort of go hand to hand.
• 48:16
  SPEAKER_01: And uh what AI really brings to the table is this ability to generalize well and to bring about this level of scalability.
• 48:25
  SPEAKER_00: Aaron Powell One of the sort of big and ever-present lessons that I like, I feel like I've learned indirectly coming over to A3 is that people seem to consistently underestimate how difficult it is to scale, both in terms of data we're talking about now, but also, and this is kind of gonna be a huge thing with um humanoid robots uh deployment and manufacturing.
• 48:51
  SPEAKER_00: People just tend to just underestimate what it takes to scale.
• 48:55
  SPEAKER_01: The humanoid space is interesting in that respect, right?
• 48:58
  SPEAKER_01: To if you squint at it, it does feel a little bit like humanoid robotics is where Waymo was maybe in 2013, right?
• 49:08
  SPEAKER_01: In a sense that there is a lot of demos, there is a lot of uh expectations, a lot of investments going towards uh that space.
• 49:18
  SPEAKER_01: And the question is, are we going to see a replay of the autonomous driving um uh story play out in the humanoid robotics space?
• 49:31
  SPEAKER_01: Uh if if that's the case, then they have a long road ahead.
• 49:35
  SPEAKER_01: And the question is uh is uh is the um is the are the fundamentals of uh uh that uh industry similar enough that you can imagine replaying the same story, or are the formals very different?
• 49:53
  SPEAKER_01: The product story is slightly different, right?
• 49:56
  SPEAKER_01: It's uh there is a question of you know, is humanoids the right endpoint?
• 50:00
  SPEAKER_01: Is it something that the world needs to be successful in robotics?
• 50:05
  SPEAKER_01: Or are we going to be building more bespoke robots, more targeted robots for a long time, and that's what will eventually win out?
• 50:14
  SPEAKER_01: Um, are we in the right generation of technology to enable humanoids to work?
• 50:21
  SPEAKER_01: Do they have the same kind of characteristics in terms of the autonomous driving space?
• 50:27
  SPEAKER_01: What is the dominant factor is really safety, right?
• 50:31
  SPEAKER_01: You have to be able to drive safely, you have cars uh on the roads around people, uh, and uh sort of the safety equation sort of dominates the design space.
• 50:45
  SPEAKER_01: Um, I think there is uh in many circles uh in humanoid robotics a belief that maybe safety is less of an issue uh in the development of humanoid robots.
• 50:58
  SPEAKER_01: Um I vehemently disagree with that.
• 51:01
  SPEAKER_01: I think uh the the safety uh question in humanoid robotics has to be central.
• 51:08
  SPEAKER_01: It's the different question than what we see in autonomous driving, but it's still a uh very central problem.
• 51:16
  SPEAKER_01: And in some ways, even a harder problem uh that needs to be solved as a uh, you know, and it that will greatly influence the design space of humanoid robots.
• 51:29
  SPEAKER_00: Well, Vincent, it's been an absolute pleasure.
• 51:32
  SPEAKER_00: Thank you so much for taking the time.
• 51:34
  SPEAKER_00: Thanks for having me.
• 51:36
  SPEAKER_00: Thank you so much to Vincent and Waymo for that great conversation.
• 51:39
  SPEAKER_00: Thanks to you as always for helping us pump up our watch hours by sticking around until the bitter end of the show.
• 51:49
  SPEAKER_00: Please like and subscribe while you're at it.
• 51:52
  SPEAKER_00: And don't forget to check out the automated weekly newsletter.
• 51:56
  SPEAKER_00: And with that, we will see you next week for another episode of Automated.