Latest / Automated with Brian Heater / Carolina Parada on Gemini Robotics, Robot Data and Physical AGI

Transcript

Carolina Parada on Gemini Robotics, Robot Data and Physical AGI · Automated with Brian Heater

• 0:00
  SPEAKER_01: 10 years ago, we were looking at robotics and saying we're just gonna solve it totally differently.
• 0:05
  SPEAKER_01: We're also gonna solve it by trying to do a single model that can go from pixels to motor control, and everybody thought we were crazy.
• 0:12
  SPEAKER_01: Our goal is to build the AI layer that can be used by any robot to solve things that basically require human-level understanding and human-level precision to accomplish.
• 0:22
  SPEAKER_01: I don't really believe in a single robot is going to win it all.
• 0:25
  SPEAKER_01: I think there's gonna be a broad ecosystem of many different robot types, so that's why we think building the AI layer that can power all of them is actually the most important way to bring impact into this space and to accelerate it.
• 0:37
  SPEAKER_00: I'm looking right now at humanoids and there's sort of a fork in the road right now.
• 0:42
  SPEAKER_00: There's these companies that are talking about, hey, let's deploy our systems right now.
• 0:46
  SPEAKER_00: Then there are other companies that are like, we're gonna wait until these systems are AGI to really commercialize the product.
• 0:53
  SPEAKER_01: You don't need to wait until you have full AGI in the physical world to make it valuable.
• 0:57
  SPEAKER_01: Our Gemini Robotics ER model is getting used in robotic applications.
• 1:01
  SPEAKER_01: Our partner at Boston Dynamics is now launched this with the spot robot.
• 1:05
  SPEAKER_01: I hope that we can bring a lot of value before full AGI in the physical world is there, but the intention is to continue pushing it all the way until it's there because that's when the opportunity explodes.
• 1:15
  SPEAKER_00: Do you see this work that you're doing with Gemini Robotics as leading to some of these innovations that will sort of like continue to form existing systems beyond like big AGI models?
• 1:39
  SPEAKER_00: Hello and welcome to another episode of Automated.
• 1:42
  SPEAKER_00: My name is Brian Heater.
• 1:44
  SPEAKER_00: I am the managing editor at the Association for Advancing Automation.
• 1:47
  SPEAKER_00: Today we have someone on the show you have been asking for since she stepped into her current role back in May of this year.
• 1:55
  SPEAKER_00: Carolina Prada is the VP of robotics for Google DeepMind.
• 1:59
  SPEAKER_00: She has a ton of insight to share around the company's work in physical AI through its Gemini robotics models.
• 2:06
  SPEAKER_00: Carolina and Google are bringing some really novel approaches to the table.
• 2:11
  SPEAKER_00: I think you're going to get a lot out of that as well as her uh personal journey from search to robotics.
• 2:17
  SPEAKER_00: Thank you so much to her and to DeepMind for setting up that conversation.
• 2:22
  SPEAKER_00: Thanks to you as always for tuning in.
• 2:25
  SPEAKER_00: Don't forget to like, subscribe, review, and uh just generally juice our algorithms and check out our newsletter and news site over at automated.fm.
• 2:36
  SPEAKER_00: Automated's next happy hour and live podcast recording will be in the Bay Area on Tuesday, September 22nd.
• 2:44
  SPEAKER_00: We will be at circuit launch in Mountain View.
• 2:46
  SPEAKER_00: Thanks to our friends at Silicon Valley Robotics for helping to set that up.
• 2:50
  SPEAKER_00: More information and a link to sign up in the show notes.
• 2:54
  SPEAKER_00: Now please enjoy this episode of Automated.
• 2:57
  SPEAKER_00: Vision and AI are changing what's possible in industrial automation, from quality inspection and predictive maintenance to robotics, logistics, medical devices, aerospace, and manufacturing.
• 3:09
  SPEAKER_00: At the Advanced Vision and AI conference, you'll hear from industry leaders, see real-world AI applications in action, explore the latest in machine vision, computer vision, sensors, edge computing, and automation technology.
• 3:21
  SPEAKER_00: You'll also be able to connect with the people who are shaping what comes next.
• 3:25
  SPEAKER_00: Whether you're building, buying, or implementing these systems, this is a chance to stay ahead of the curve and see how vision and AI are moving from promise to practical impact.
• 3:36
  SPEAKER_00: Learn more and make sure to buy your ticket early at the Advanced Vision and AI conference September 23rd and 24th in Santa Clara.
• 3:46
  SPEAKER_00: How does one get into how does one start studying electrical engineering?
• 3:52
  SPEAKER_00: Why was that specifically of interest to you?
• 3:55
  SPEAKER_01: Hi, Brian.
• 3:56
  SPEAKER_01: Thanks for having me.
• 4:08
  SPEAKER_01: And then once I was there and I started seeing all the people playing around with circuits, and I was like, oh, that seems way cooler.
• 4:15
  SPEAKER_01: So I ended up switching to electronic engineering.
• 4:18
  SPEAKER_01: But I knew generally that I wanted to be in engineering.
• 4:22
  SPEAKER_01: That uh I've always wanted to, I always thought thought about robotics from cartoons as like the coolest thing to build when I grew up.
• 4:31
  SPEAKER_01: And so that actually motivated me to go into the into that path.
• 4:35
  SPEAKER_01: And then once you start studying there, there's a million paths to go into, right?
• 4:38
  SPEAKER_00: So that is a very familiar part of the story too.
• 4:41
  SPEAKER_00: Um, you know, of just I think it's one, just oh yeah, like that, oh, that's actually like feasible.
• 4:48
  SPEAKER_00: And now it like is it is feasible.
• 4:51
  SPEAKER_00: Uh, you know, as I've seen, there's just been so many people that I've spoken to over the years who have studied something like, you know, maybe a little bit broader, you know, maybe just some kind of engineering.
• 4:59
  SPEAKER_00: And then once it got to the point where they're like, oh, there's actually like this is this is a something that I could do for a living.
• 5:06
  SPEAKER_00: Um, there is some infrastructure here, then they actually, you know, do start studying that.
• 5:11
  SPEAKER_00: So when when you were in your undergrad when you were at Washington State, um, was it specifically in robotics?
• 5:21
  SPEAKER_01: So I was, I did my undergrad in electrical and computer engineering in Washington State.
• 5:25
  SPEAKER_01: And then I knew I wanted to go into grad school to get more specialized.
• 5:30
  SPEAKER_01: And in grad school, I actually went for control systems because, again, my intention was to um build a career in robotics.
• 5:38
  SPEAKER_01: And then uh while I was doing control systems, I started doing digital signal processing as well, you know, a lot of those um other areas.
• 5:46
  SPEAKER_01: Um and at that time, speech recognition was going through this really hot transition into like be more focused on learning based.
• 5:57
  SPEAKER_01: It used to be very much like a grammar-based approach, was what most people would use.
• 6:01
  SPEAKER_01: People don't think, don't remember this now, but back then just speech didn't work at all, right?
• 6:07
  SPEAKER_01: And so I thought for my PhD, I would actually take a diversion and go into machine learning for speech recognition.
• 6:15
  SPEAKER_01: It was just an area that was getting transformed.
• 6:17
  SPEAKER_01: It was also an electrical computer engineering discipline.
• 6:21
  SPEAKER_01: And so that's how I ended up doing speech recognition in my PhD.
• 6:24
  SPEAKER_01: Um, and I work with one of the grades, Fred Jelinek was my advisor, and he transformed the field.
• 6:30
  SPEAKER_01: So that was a really, really great uh way to get introduced into machine learning and deep learning.
• 6:36
  SPEAKER_01: But it's not the I went after it because it was it felt like a place that was going through a transformation and that it was combining a lot of things that I was learning on signal processing, control systems, and machine learning altogether.
• 6:50
  SPEAKER_00: Yeah, that's another through line that I've I've noticed as well.
• 6:53
  SPEAKER_00: Like there are a lot of people now who are um really prominent, specifically in the in the robotics field, who like again didn't study robotics, but it seems like there are so many other, especially technical uh or mathematical fields that just end up being transferable in one way or another to robotics.
• 7:14
  SPEAKER_01: Yeah, I mean it's all quite similar.
• 7:16
  SPEAKER_01: Like if you're learning, if you're doing machine learning, regardless of the application, it's all about learning patterns and the algorithms that you use in order to extract information out of data.
• 7:27
  SPEAKER_01: And so speech recognition went through it, and language models actually were the first ones that went through very big transformations, and then you see how vision then followed, and then now robotics is following.
• 7:38
  SPEAKER_01: But it's all the same underlying techniques.
• 7:41
  SPEAKER_00: You said you use the word diversion, but like we're talking about, I mean, that's a significant diversion here, right?
• 7:46
  SPEAKER_00: I mean, we're talking about like 2000, 2006, uh, you know, up until I I guess I would say maybe what your time at uh NVIDIA, like like 2017.
• 7:57
  SPEAKER_00: So like, you know, maybe maybe about like a decade or so or so that you were in speech?
• 8:02
  SPEAKER_01: Yeah, I did about seven years uh in speech.
• 8:05
  SPEAKER_01: Uh was my PhD, well, after my PhD, I did my PhD on at Jones Hopkins on speech recognition.
• 8:11
  SPEAKER_01: And then I came as an intern to Google.
• 8:14
  SPEAKER_01: And then once I came as an intern, and and I also interned in other places, but I fell in love with Google pretty early on.
• 8:20
  SPEAKER_01: And uh I joined at a pretty fun time because back then, again, speech was not a product, it was just a research area.
• 8:27
  SPEAKER_01: And then as we went through this transformation into making it deep learning-based, it all of a sudden became something that could understand anyone speaking open vocabulary.
• 8:37
  SPEAKER_01: Like back to back in those days, you would do closed vocabulary speech recognition.
• 8:42
  SPEAKER_01: So you would have a list of states and then you mentioned the state, and then a list of stores, and then you mentioned the stores.
• 8:48
  SPEAKER_01: And then all of a sudden, we were able to just listen in a conversation and understand a five-second query uh with voice, and that was pretty exciting.
• 8:58
  SPEAKER_01: So then once we launched speech recognition in many Google surfaces, we launched it on voice search, uh, so like the microphone now, which then went in Chrome, it went on Android, it went on Google Home.
• 9:10
  SPEAKER_01: Then I was ready for a new challenge, and then I went on to NVIDIA to do self-driving cars.
• 9:15
  SPEAKER_01: And it was essentially applying the same kind of techniques, but for vision now.
• 9:19
  SPEAKER_01: So I was leading some of the vision teams there that were doing like lane detection and sign detection and just all of the different components that you have in your perception system for a self-driving car.
• 9:30
  SPEAKER_00: Yeah, it seems to me that you've got a knack for being in the right place at the right time.
• 9:34
  SPEAKER_00: Um, you know, because like one of the things that that that you like one of the additional places that um, because it was just an internship, but um you were you were at nuance, and and nuance um uh was an SRI.
• 9:47
  SPEAKER_00: So, you know, Stanford Research spinoff built the, I think they're owned by Microsoft now, built like the foundational technology for Siri.
• 9:54
  SPEAKER_00: You know, you're at NVIDIA around the time for self-driving.
• 9:57
  SPEAKER_00: Like how like how do you put yourself in these positions?
• 10:01
  SPEAKER_00: Is it uh how much of it is is is luck?
• 10:05
  SPEAKER_01: Yeah, I mean, I'm sure that there's a lot of luck into it, but I think what I what it is is that I'm looking for um what area feels like ripe to be disrupted.
• 10:15
  SPEAKER_01: And um definitely the speech, the speech area was getting quite more significantly more mature, and you could feel that uh it was those same techniques could apply to other fields.
• 10:27
  SPEAKER_01: So that's when I started looking to go back into robotics.
• 10:29
  SPEAKER_01: And so when I was looking through robotics, I was like self-driving feels like it's now ready.
• 10:33
  SPEAKER_01: And then since then, as you can see, self-driving has come a really long way.
• 10:37
  SPEAKER_01: None of these fields are like fully solved, to be clear, but I was certainly like ready for which one feels that is ready to be disrupted.
• 10:44
  SPEAKER_01: And then I came to robotics about eight years ago, so I would say it was still a little early, but just in the last three, four years, it's been significantly disrupted, as you've seen again.
• 10:55
  SPEAKER_01: So, yeah, robotics is hard.
• 10:57
  SPEAKER_01: But and when I say robotics, I really mean just general purpose robotics, like a robot that can do anything you and I can.
• 11:04
  SPEAKER_01: And that feels like a pinnacle of showing full artificial general intelligence in the physical world.
• 11:11
  SPEAKER_01: And so that I still think that we got some work to do to solve it really.
• 11:16
  SPEAKER_00: Yeah, this is gonna be an abstract question.
• 11:20
  SPEAKER_00: So um, you know, it might be a bit of a tough one.
• 11:22
  SPEAKER_00: But uh, you know, you said none of those are fully solved.
• 11:25
  SPEAKER_00: Obviously, like robotics, you know, general purpose robotics is far from fully solved.
• 11:30
  SPEAKER_00: Self-driving is, you know, pretty far from fully solved.
• 11:33
  SPEAKER_00: Yeah, I mean, I would say speech is definitely not solved.
• 11:35
  SPEAKER_00: Like, what is it abstractly speaking, like what does it mean to be solved?
• 11:43
  SPEAKER_01: Yeah, I mean, I would say that I'm comparing it to human performance.
• 11:47
  SPEAKER_01: So I would say speech recognition today is has improved dramatically.
• 11:52
  SPEAKER_01: And today we rely on auto captioning for all kinds of videos.
• 11:57
  SPEAKER_01: But still, if you put a microphone at a distance in a bar full of people or in a cocktail party, you're not gonna be able to solve it the same way a human can.
• 12:05
  SPEAKER_01: So there's definitely aspects of it.
• 12:06
  SPEAKER_00: If you put me in a bar, I'm probably gonna have the same problems.
• 12:09
  SPEAKER_00: But I guess like just generally speaking, yeah.
• 12:12
  SPEAKER_01: Yeah, I think humans are still superior in terms of performance when you're putting them in a cocktail party, or even uh anything that is a little bit farther than close microphone range is hard still.
• 12:23
  SPEAKER_01: Um, or when you have multiple speakers and you have to discern between them.
• 12:27
  SPEAKER_01: So there's still aspects where humans are still superior, and then same with self-driving.
• 12:31
  SPEAKER_01: I think self-driving has gone so far.
• 12:34
  SPEAKER_01: I'm just very impressed.
• 12:35
  SPEAKER_01: I ride Waymo pretty often, and it's always really exciting to see how it handles almost like a human, so many situations.
• 12:44
  SPEAKER_01: But still, there's still edge cases that humans are much better at reasoning through that um the self-driving still has a ways to go.
• 12:51
  SPEAKER_01: And then finally, um, for general purpose robots, it's incredibly hard because now, at least in self-driving, you have rules of the road.
• 12:58
  SPEAKER_01: You have lanes, you have traffic lights.
• 13:00
  SPEAKER_01: When you're and you're mostly trying to stay away from interaction.
• 13:05
  SPEAKER_01: But when you're talking about general purpose robots, it's all about interaction.
• 13:08
  SPEAKER_01: You have to interact with the very unstructured world.
• 13:11
  SPEAKER_01: There's no rules of the road telling you how to behave, you just sort of have to read the room.
• 13:15
  SPEAKER_01: Again, if you're working on human-centric spaces, so it really puts together all of these really difficult problems of understanding the physical world, interacting with it in a meaningful way, and then um interacting with humans in that physical world, and then being able to reason about what it means to solve a problem.
• 13:34
  SPEAKER_01: So it is uh really bringing a whole lot of really complex problems into one.
• 13:39
  SPEAKER_00: So yeah, I mean it sounds like in all three cases that human standard is sort of the the gold for each of them.
• 13:47
  SPEAKER_00: Is that fair?
• 13:48
  SPEAKER_01: Yeah, that's fair.
• 13:49
  SPEAKER_01: But that's definitely what I'm trying to achieve is how do we get these machines to be human-level performance?
• 13:55
  SPEAKER_00: Yeah, I would say that like self-driving, I mean, this is you know, again, like a whole nother conversation.
• 14:00
  SPEAKER_00: Um, but like the self-driving is maybe has to be a little bit better than humans, right?
• 14:06
  SPEAKER_00: I mean, it's and and and maybe and and in certain ways, like uh maybe maybe maybe robotics do as well, because you know, obviously, again, the first time there's an accident or the first time there, you know, something's like it goes wrong with a self-driving car, even if a human couldn't have corrected for it, like, you know, then there's a lot of pushback.
• 14:28
  SPEAKER_01: Yeah, you're right.
• 14:29
  SPEAKER_01: I think on the safety, the safety bar is quite high for anything that you put uh out there higher than a human.
• 14:35
  SPEAKER_01: But yeah, I meant to say that definitely human performance is already a really, really high bar for all of these technologies.
• 14:43
  SPEAKER_01: And then, yeah, of course, we can wherever we can make it even safer than a human, that's obviously something we want to we want to go after.
• 14:50
  SPEAKER_00: Yeah, yeah.
• 14:51
  SPEAKER_00: And and we're we're we're gonna loop around to to the work that you're doing um with Gemini right now.
• 14:55
  SPEAKER_00: Uh, you know, I know like Gasimov um was was a big piece of that too, but it really and I'm wondering to what extent working in the self-driving world really kind of hammered home how important it is to bake some of these safety standards into robots from the outset.
• 15:10
  SPEAKER_01: Mm-hmm.
• 15:12
  SPEAKER_01: Yeah, certainly.
• 15:13
  SPEAKER_01: I think the the self-driving world has definitely, I mean, my entire career has shaped how I see solving each one of these problems, but self-driving is certainly the closest.
• 15:22
  SPEAKER_01: And it's one that I think it still has a ways to go and it's it still feels it represents a small part portion of all that needs to be solved for general purpose robots.
• 15:33
  SPEAKER_01: So there's still a bit more challenges with with general purpose robotics.
• 15:36
  SPEAKER_00: Yeah, so so so let's talk about that because I'm I'm curious, like specifically, we were talking earlier about the the transferable skills, and and again, um one of the things that I that I find uh again when I talk to a lot of people in this field is it really comes down to not necessarily the things that you've studied, but people's ways of thinking and people's ways of of problem solving.
• 15:59
  SPEAKER_00: Like that's the thing often that is transferable and helpful.
• 16:04
  SPEAKER_00: Um all of these years, you know, the decade plus that we mentioned that you were in the world of speech, you know, pattern recognition.
• 16:13
  SPEAKER_00: Um I'm going down your Google Scholar page, and a lot of the cited papers that you have are in speech.
• 16:19
  SPEAKER_00: Uh what are some of the learnings from that world that were transferable to the robotics work that you're doing right now?
• 16:26
  SPEAKER_01: I mean, um I like it, like you said, some of it is the approach with which you see a problem and and how you go about solving it.
• 16:35
  SPEAKER_01: Certainly, one of the things that I've learned with each one of them, and if you notice in each one of them, I was there trying to change the way the current the current community was solving the problem.
• 16:46
  SPEAKER_01: So even when I started working on speech recognition, one of the first things that I introduced was how to use deep learning for keyword spotting, which is at the time was done in a totally different way.
• 16:56
  SPEAKER_01: So I think in most of these cases, I try to go at it from how can I learn this from data in a way that feels robust and resilient to noise?
• 17:08
  SPEAKER_01: And also, how do I try to solve this without necessarily evaluating against the current techniques, but not just trying to do an incremental improvement on them?
• 17:16
  SPEAKER_01: So it's perfectly okay to sort of start from scratch and remove the existing approach before um before before evaluating against it.
• 17:27
  SPEAKER_01: So um in yeah, in speech recognition, by the time that I was in the in that team, we were rewriting everything.
• 17:34
  SPEAKER_01: How do you rewrite acoustic models, language models?
• 17:37
  SPEAKER_01: At the time, we even had multiple systems.
• 17:40
  SPEAKER_01: We have acoustic language and pronunciation models, which then we rewrote again to be a single transformer model.
• 17:47
  SPEAKER_01: So just like that, this regard for the status quo and for just rewriting how you solve a problem is something that I saw consistently across my career.
• 17:57
  SPEAKER_01: And then in self-driving, it's happening, it happened again, right?
• 18:00
  SPEAKER_01: At the very beginning, we started with many different systems, one for like lane detection and one for sign detection, and one for each one of the different things that you thought ahead of time you have to put together.
• 18:12
  SPEAKER_01: And then a lot of that became just a single perception system that was thinking through all the cameras and all the sensors together in the car.
• 18:20
  SPEAKER_01: And so, like, I think that's one theme that I brought into robotics again.
• 18:25
  SPEAKER_01: And honestly, the job the team that I joined was already thinking that way.
• 18:28
  SPEAKER_01: So I tend to be also connected to teams that are have that mindset of how do we change robotics, how do we transform it.
• 18:36
  SPEAKER_01: And then I think pretty early on, 10 years ago, we were looking at robotics and saying we're just gonna solve it totally differently.
• 18:43
  SPEAKER_01: We're also gonna solve it by trying to do a single model that can go from pixels to motor control.
• 18:49
  SPEAKER_01: And everybody thought we were crazy.
• 18:51
  SPEAKER_01: And over time, it has been adopted more and more.
• 18:55
  SPEAKER_01: And today, pretty much everyone that does gender purpose robotics has adopted a lot of the techniques that we have introduced.
• 19:00
  SPEAKER_00: So when you say we, you mean the deep mind team?
• 19:03
  SPEAKER_01: The deep mind team, yes, of course.
• 19:05
  SPEAKER_00: Yeah.
• 19:06
  SPEAKER_01: So we pretty early on, maybe seven years ago, we introduced reinforcement learning for whole body control, for example, in simulation.
• 19:13
  SPEAKER_01: And now pretty much every robot that you see dancing, walking is using reinforcement learning.
• 19:18
  SPEAKER_01: And that was not back then, most people were using just MPC to control robots and modeling everything about the robot perfectly.
• 19:27
  SPEAKER_01: And then the same with uh manipulation.
• 19:29
  SPEAKER_01: Manipulation had uh and still today, there's quite a lot of systems that are deployed today.
• 19:34
  SPEAKER_01: Most systems that are deployed today are using traditional control approaches and traditional perception.
• 19:40
  SPEAKER_01: But the entire industry is trying to really rewrite that into how do you learn from pixels into motor control to handle uncertainty, to be able to understand what's happening in the world, to be able to do eye hand coordination.
• 19:56
  SPEAKER_01: This is something that you couldn't do before.
• 19:59
  SPEAKER_01: And so, yeah, we're we're basically again trying to rewrite um how to do manipulation for general purpose robots.
• 20:06
  SPEAKER_01: And I don't think this will be the last time.
• 20:08
  SPEAKER_01: We will continue to do this.
• 20:10
  SPEAKER_00: Yeah, it it sounds like to a certain extent you're you're kind of advocating for um every so often maybe switching your focus or bringing people in from the outside because like that's an opportunity to bring in a fresh set of eyes.
• 20:25
  SPEAKER_01: Definitely challenging your assumptions is what I'm advocating for.
• 20:28
  SPEAKER_01: And I ask folks that join the team all the time that I say we're very proud of having a very diverse mix of folks from different backgrounds.
• 20:37
  SPEAKER_01: And I ask them all to look at what we have and to be completely candid and challenge our assumptions on how they would do it differently.
• 20:44
  SPEAKER_01: And so it's it's usually really great to see to get fresh perspectives.
• 20:48
  SPEAKER_00: Yeah, I mean it's uh it's nice to hear that like they were thinking similarly because like it it must be a little tough to see like you coming through the door on day one and just be like, all right, well, it's time to it's time to erase the whiteboard and like start over from day one.
• 21:05
  SPEAKER_00: Is it is it like some is it is it is it there there has to be a degree in which I mean what do they call it, sunk cost fallacy.
• 21:12
  SPEAKER_00: Like there has to be a degree in which it's like it's frustrating, right?
• 21:16
  SPEAKER_00: It's frustrating to work on a problem for a really long time and then be asked to sort of like rethink it and then start over again.
• 21:23
  SPEAKER_01: Yeah, I mean, I I think we're quite used to it by now.
• 21:26
  SPEAKER_01: Uh we certainly have some things that we have in place.
• 21:29
  SPEAKER_01: Like, for example, we have beliefs of we want to be able to solve robotics with a single model that can do a broad range of tasks.
• 21:36
  SPEAKER_01: And so that is a fundamental assumption that we have on how we solve the problem.
• 21:40
  SPEAKER_01: But the solution, like what is the architecture that we use to solve it, what is the way that we learn, that we feed data into that model?
• 21:48
  SPEAKER_01: There's a lot of open opportunities to challenge everything that we do about that.
• 21:52
  SPEAKER_01: And then even the chat, even the assumption of a single model gets challenged every so often, because why not, right?
• 21:58
  SPEAKER_01: If you're challenging all your other assumptions.
• 22:00
  SPEAKER_01: That also gets challenged.
• 22:01
  SPEAKER_01: But um for the most part, I think that has been a pretty consistent theme within the team is why don't we build a system that can reason about its environment, reason about itself, and then reason about the task and put it all together to understand whether it is done or not.
• 22:18
  SPEAKER_01: That continues to be a theme that has been consistent in the team of the main problem we want to solve.
• 22:25
  SPEAKER_00: Yeah.
• 22:26
  SPEAKER_00: Do you feel like across the industry that there's not a consensus, but that there's starting to be a consensus around single model?
• 22:34
  SPEAKER_00: Because there's definitely still some debate there.
• 22:36
  SPEAKER_01: Yeah.
• 22:36
  SPEAKER_01: I mean, I think there is significantly more attention to how do you solve general purpose robotics with a single foundation model?
• 22:43
  SPEAKER_01: That is now something that the community has adopted as the way forward to really open the door for many more applications for robotics.
• 22:52
  SPEAKER_01: I think there's still a lot of debate about whether you have multiple systems, like a system zero that does very high frequency control, a system one, and then a system two.
• 23:02
  SPEAKER_01: And even within our own team, we actually have all of those systems, right?
• 23:07
  SPEAKER_01: Even so, you still think that one of the key ideas that you want to bring forward is that there needs to be a system that's really understanding what is happening when you're interacting with that object and whether that is whether you're moving and progressing in the right direction and how to respond to changes.
• 23:24
  SPEAKER_01: And that is what I mean by the single model.
• 23:26
  SPEAKER_01: But we really have multiple models in our family of solutions, right?
• 23:32
  SPEAKER_01: So I'm happy to go into that if you like.
• 23:35
  SPEAKER_00: Let's cut right into that right now because uh when you know Gemini comes out and there are like effectively three different flavors of Gemini 2, there's a main version.
• 23:44
  SPEAKER_00: But you know, one of the things I think maybe you're talking about right now is the edge version of it that's meant to run on the machines themselves without the cloud connection.
• 23:52
  SPEAKER_00: And I assume that that requires like maybe like a significantly smaller model or a significantly smaller data set, at least.
• 24:00
  SPEAKER_01: So definitely the the way to think about Gemini Robotics is a family of models, and that we always have running in the in the when you're doing, say, a 30-minute sequence of the robot doing a task, say the robot is organizing your office.
• 24:16
  SPEAKER_01: That robot is essentially having a high-level reasoning layer.
• 24:19
  SPEAKER_01: It's just the orchestrator.
• 24:20
  SPEAKER_01: This is the Gemini Robotics ER model.
• 24:23
  SPEAKER_01: It actually understands the environment, it understands what it means to organize the office.
• 24:27
  SPEAKER_01: Maybe you show it a picture of how your office was organized before, and now it has to fix it to that.
• 24:33
  SPEAKER_01: So it tracks the progress towards that.
• 24:35
  SPEAKER_01: So it will put the books in the right place and then how to where to put the chairs and what what the progress is towards that goal, and it's actually able to talk to you.
• 24:42
  SPEAKER_01: So that's the reasoning model, that's Gemini Robotics ER.
• 24:45
  SPEAKER_01: And then that model operates with an action model, which is Gemini Robotics the VLA.
• 24:51
  SPEAKER_01: And this action model translates the visual input into the motor commands that the robot executes.
• 24:56
  SPEAKER_01: So if the robot is putting away the books, the message to that model would be put away the blue book into this bookshelf, second shelf on the right.
• 25:05
  SPEAKER_01: So it's pretty specific about what it wants it to do.
• 25:07
  SPEAKER_01: And then the action model translates that into the joints, moving to grab the object and to move it and put it over there.
• 25:14
  SPEAKER_01: It actually executes the actions.
• 25:16
  SPEAKER_01: That action model is still one model that takes its input natural language and visual input and produces all the joint control for the entire robot.
• 25:25
  SPEAKER_01: And actually, underneath that, there might be a system zero.
• 25:29
  SPEAKER_01: If you have a humanoid, you need to balance.
• 25:30
  SPEAKER_01: And there might be a system zero that is a whole body controller that in our case is usually an RL whole body controller, so reinforcement learned, that is maintaining stability of the system.
• 25:40
  SPEAKER_01: So actually, when you're running, organizing your office, you actually have three systems that are in play.
• 25:46
  SPEAKER_01: And then the action model that I explained, uh, the VLA also can run fully on device.
• 25:50
  SPEAKER_01: So sometimes you might have a situation where there's like no connectivity or very poor connectivity, and you just care about the reliability of that system, even if it's less smart, because obviously a model that runs on your device is going to be smaller and it's gonna be less smart than the model that can run on cloud.
• 26:08
  SPEAKER_01: At least that's the case today.
• 26:09
  SPEAKER_01: Maybe it will change.
• 26:10
  SPEAKER_01: Maybe in two years we won't need a cloud at all.
• 26:13
  SPEAKER_01: Um and so that's why we have three models that we offer.
• 26:17
  SPEAKER_01: Does that make sense?
• 26:18
  SPEAKER_01: So we start with high-level, we have action models, and then we have something that maintains stability of the system.
• 26:25
  SPEAKER_00: Yeah, and and I'm guessing uh this is another place maybe where your speech background is coming into play when we're talking about so you mentioned ER, which is embodied reasoning.
• 26:37
  SPEAKER_00: Um, and the way the way I saw it described uh by I think it was you on the blog is embodied reasoning is what allows it to bridge the gap between digital intelligence and physical action.
• 26:50
  SPEAKER_00: So yeah, I mean, for it to actually like execute on an action effectively.
• 26:56
  SPEAKER_01: Yeah, the ER model is keeping this long understanding of what it means to do a task and tracking progress towards that task.
• 27:04
  SPEAKER_01: It doesn't ever move the robot.
• 27:06
  SPEAKER_01: It's the input is literally the goal and the images, and then the output is instructions to the action model to execute this and then that and then that.
• 27:16
  SPEAKER_01: It's basically um doing orchestration, the same way that any agent that operates on um on your code base is doing orchestration, making different calls is exactly the same.
• 27:27
  SPEAKER_01: It's actually we call it our agent.
• 27:29
  SPEAKER_00: Is is natural language, is that a piece of that?
• 27:33
  SPEAKER_01: Yeah.
• 27:33
  SPEAKER_01: I mean, the modalities that it can use to communicate with the action model can be changed.
• 27:39
  SPEAKER_01: Uh, and we certainly explore with that.
• 27:41
  SPEAKER_01: So sometimes when you're explaining maybe to your kid how to uh where to put the toys, you might not even like say it all with natural language, you might show your kid, right?
• 27:51
  SPEAKER_01: Or you might point to it.
• 27:52
  SPEAKER_01: So those are other modalities that the agent could decide to use with the action model.
• 27:58
  SPEAKER_01: Um, but these are much earlier in our in our research, so we don't talk about them that much.
• 28:04
  SPEAKER_01: The majority of how this model was trained was using natural language.
• 28:09
  SPEAKER_01: And the reason was simple.
• 28:10
  SPEAKER_01: Like there is a significant amount of data out there on how to translate complex tasks into a sequence of steps using natural language, so it comes out very naturally as a way to orchestrate a long, a long task.
• 28:26
  SPEAKER_01: But we do believe that we're gonna need probably multiple modalities to take this all the way to, again, human-level performance.
• 28:33
  SPEAKER_00: Yeah, one of the things uh so I I saw your talk at Machina in Paris.
• 28:38
  SPEAKER_00: And one of the things that you broke down in natural language, one of the things that you broke down really well is um the distinction between traditional hard coding and and what AI is bringing to the table.
• 28:50
  SPEAKER_00: And I think that that's a really big piece of this, right?
• 28:52
  SPEAKER_00: Is actually having to go in and having to code every single action versus these models which were able to carry out like not only fluid action, but like multi-step actions.
• 29:04
  SPEAKER_01: Yeah, I think today there is a significant, like I mentioned in that talk, today there's significant automation out there.
• 29:11
  SPEAKER_01: And you can automate pretty much anything you want if you can manipulate the environment and put it the way you need, and then you can hard code the entire sequence, right?
• 29:20
  SPEAKER_01: But that's obviously not very scalable, especially in a world that is constantly changing.
• 29:24
  SPEAKER_01: And so you can think of the reasoning model as the model that takes a whole lot of uh world understanding and brings into this application, and you can now orchestrate any sequence of skills.
• 29:38
  SPEAKER_01: And then the action model is basically generating any skill that you want based on natural language, and the two come together to do something pretty general.
• 29:47
  SPEAKER_00: Yeah, uh one of the one of the things I wanted to back up a little bit because one of the things you said a little bit earlier that I thought was really interesting um during your time on speech was that uh speech was not a product, right?
• 30:00
  SPEAKER_00: So that it was like a fully that it was fully research in in all these places.
• 30:04
  SPEAKER_00: So again, like we're talking prior to Google Assistant, Google Home Siri, all these things.
• 30:13
  SPEAKER_00: Um and certainly that was the state for self-driving, and like that's the state of of robotics right now.
• 30:19
  SPEAKER_00: And and it and and it seems like I mean, is that the place in which you are the most comfortable operating?
• 30:26
  SPEAKER_01: Yeah, I mean, definitely I'll tell you that my intention is for sure to get to a world in which these robots are useful and they're doing all kinds of things around us.
• 30:37
  SPEAKER_01: And that's not the case today.
• 30:38
  SPEAKER_01: Today we're very much in research land, but my goal is how to could how do we continue to iterate on the solution such that it can actually get to useful.
• 30:48
  SPEAKER_01: And for it to get to useful, it really needs to be able to handle all that diversity that exists that you and I handle really well.
• 30:56
  SPEAKER_01: And then all of that variability in objects is never seen, the environment is never seen, and just reactivity to changes in that environment.
• 31:05
  SPEAKER_01: That's really what's going to transform it from something that can be built and be left alone for 10 years to do a single task versus something that is you and I experience every single day.
• 31:16
  SPEAKER_01: So that's that's certainly where I want to get to.
• 31:19
  SPEAKER_01: It's a really exciting place to be.
• 31:21
  SPEAKER_01: I think uh robotics obviously significantly harder when it's combining hardware that is also evolving with AI that is also evolving, and humans all in one is is what makes a super exciting area to work on.
• 31:35
  SPEAKER_00: Yeah, I think it's really interesting again, if you draw the parallels between where we're at in robotics right now and where autonomous driving was 10 years ago.
• 31:46
  SPEAKER_00: Um there were a lot of uh early startups in there, and very few of them have survived.
• 31:53
  SPEAKER_00: Waymo is one of them.
• 31:55
  SPEAKER_00: Um, and a lot of that is, you know, because it's it it's a it's a Google company, it's been around, you know, it's it's it's had sort of the resources to continue and it and it's and it's built and it's built those really sort of like deep and and complex models.
• 32:07
  SPEAKER_00: I'm looking right now at humanoids, and there are there's sort of a fork in the road right now, right?
• 32:16
  SPEAKER_00: There's um there's these companies that are talking about, hey, let's deploy our systems right now, let's get these robots out, uh, have them do uh one or two tasks pretty well, you know, move totes.
• 32:32
  SPEAKER_00: And then there are these then then there are other companies that are like, okay, AGI.
• 32:36
  SPEAKER_00: We're gonna wait until these systems are are AGI to really commercialize the product.
• 32:41
  SPEAKER_00: Um I know that like Google doesn't necessarily have a dog in that race, but like how do you do you see this playing out in a certain way?
• 32:50
  SPEAKER_00: Is it going to sort of parallel self-driving?
• 32:54
  SPEAKER_01: So, well, first, I think we certainly care about solving the whole problem.
• 32:58
  SPEAKER_01: So we on the our goal is to build the AI layer, right, that can be used by any robot to solve things like basically require human-level understanding and human-level precision to accomplish.
• 33:10
  SPEAKER_01: So that's our goal.
• 33:11
  SPEAKER_01: I think robotics is one of those situations where you're we've envisioned having a really broad set of different robot types that are operating in different environments, are doing different tasks.
• 33:23
  SPEAKER_01: I don't really believe in a single robot is going to win it all.
• 33:27
  SPEAKER_01: I think there's going to be a broad ecosystem of many different robot types.
• 33:30
  SPEAKER_01: So that's why we think building the AI layer that can power all of them is actually the most important way to bring impact into this space and to accelerate it.
• 33:41
  SPEAKER_01: Um, having said that, I think you don't need to wait until you have full AGI in the physical world to make it valuable.
• 33:47
  SPEAKER_01: I think, and we're already seeing this, we're already putting out, for example, our agent systems like our Gemini Robotics ER model, it's getting used in robotic applications to combine existing skills with reasoning so that you can do more complex behaviors.
• 34:02
  SPEAKER_01: So, one example is our partner at Boston Dynamics is now launched this with the spot robot.
• 34:07
  SPEAKER_01: And so they use it for as part of their inspection routine.
• 34:10
  SPEAKER_01: And in addition to build using their regular autonomy, they now combine it with an agent to understand what's happening, to read instruments, to decide what to do next.
• 34:21
  SPEAKER_01: And these are things that they couldn't do before because you had to pre-program it.
• 34:25
  SPEAKER_01: So I think like that, there's going to be a whole lot of different applications that are going to start showing up in many different spaces, probably in logistics and in manufacturing, it will come sooner because these are more structured spaces where the safety problem is a little bit more constrained.
• 34:40
  SPEAKER_01: But similar to self-driving, how they started.
• 34:43
  SPEAKER_01: I don't know if you remember, they started in I think um Pittsburgh.
• 34:48
  SPEAKER_01: Pittsburgh.
• 34:49
  SPEAKER_01: And then they started in, and then eventually they got to San Francisco.
• 34:53
  SPEAKER_01: And even in San Francisco, they haven't gotten to the full Bay Area.
• 34:56
  SPEAKER_01: So there will just be stages, right, of difficulty that they will we will be posting for ourselves.
• 35:01
  SPEAKER_01: And I hope that we can bring a lot of value before full AGI in the physical world is there.
• 35:06
  SPEAKER_01: Uh, but the intention is to continue pushing it all the way until it's there because that's when the opportunity explodes in terms of applications.
• 35:14
  SPEAKER_00: Yeah, and that that that's a really good, that's a good point, and that's a good example.
• 35:18
  SPEAKER_00: And I think like that works specifically with spot really well because we're talking about systems that are like already deployed, right?
• 35:24
  SPEAKER_00: Boston Dynamics did a really good job of getting those spots out there without AGI, and now you can sort of add this layer.
• 35:32
  SPEAKER_00: Are you do you see this work that you're doing with Gemini Robotics broadly speaking, as you know, maybe leading to some of these innovations that will sort of like continue to inform existing systems beyond, again, these sort of like big AGI models?
• 35:52
  SPEAKER_01: Yeah, I think we see it as there is going to be value to be brought immediately with existing systems that are already deployed, where you're combining these agents as a way to bring reasoning and to bring orchestration and to bring uh more flexibility on understanding the environment and the task.
• 36:10
  SPEAKER_01: And then the action models will start also bringing value in different in different spaces.
• 36:15
  SPEAKER_01: And that is our intention is to actually take existing systems, augment them, but at the same time, we have other bets where we're like, we're gonna do something that there's no robotic deployments out there today because they couldn't operate in such a complex environment.
• 36:30
  SPEAKER_01: And so anything that has to do with human-centric spaces, you really don't have solutions today with general purpose robots that can operate there.
• 36:38
  SPEAKER_01: And we care about those too.
• 36:40
  SPEAKER_01: But we hope that we learned a lot from deployments in these early systems as well.
• 36:45
  SPEAKER_00: How I mean, I don't I I don't know if this is something, so you know, I don't know this is something that you've disclosed.
• 36:51
  SPEAKER_00: How are are you able to say how large the team is, the DeepMind Robotics team?
• 36:55
  SPEAKER_01: I don't think they let us say that, but it's certainly like it's certainly a small team compared to like the full GDM team.
• 37:03
  SPEAKER_01: But one thing to consider is like when we're solving general purpose robotics, we are also participating and deeply integrated with all of the other Gemini teams.
• 37:13
  SPEAKER_01: So we work with Gemini and we work with Omni at different stages to shape those models.
• 37:18
  SPEAKER_01: So even though even if our model is a small, our team is a smaller portion of GDM, it certainly leverages and contributes and deeply embeds with all the other teams to be able to shape those models as well.
• 37:31
  SPEAKER_00: Yeah, and that was something that I really um I really started kind of realizing uh a couple of years ago too is that like it it's clear now in hindsight, as I talk to people who have uh continued you know to make their way up in the world of you know deep mind robotics or or in intrinsic or you know any sort of like anything on the AI side, it's that it seems like every bet that Google has made in robotics over the years, whether it's some like you know, some of those acquisitions or everyday robots, like has built the foundation.
• 38:09
  SPEAKER_00: And Waymo obviously has built the foundation.
• 38:11
  SPEAKER_00: And like, and I'm seeing like all these people, like, you know, I've got like keep on behind me over here, like all of these people who like came in through these acquisitions in in a lot of cases who are like still there and are still like working with the team.
• 38:26
  SPEAKER_01: Yeah, we have some actually quite a bit of people that have been here the entire history.
• 38:31
  SPEAKER_01: There is some people that are here that were here before I was uh even part of the team.
• 38:35
  SPEAKER_01: So it's exciting to see how deep it goes, the history of the team, and then how much we've learned through each one of those transitions.
• 38:44
  SPEAKER_01: Um I think in many of those transitions, it felt like it was maybe too early to do product.
• 38:50
  SPEAKER_01: And I think this time we're like really taking it seriously to solve the foundations.
• 38:56
  SPEAKER_01: Uh, so like the AI first layer.
• 38:59
  SPEAKER_01: Uh and we're partnering with different companies in order to evaluate that in different applications, bring those learning back, and continue to um push the technology forward.
• 39:09
  SPEAKER_00: Yeah.
• 39:11
  SPEAKER_00: Part of the reason why I asked about the size of the team is because you know, I'm I'm curious when we're talking about solving like the biggest problem or solving AGI, or at least like these really like robust models, and then we're talking about solving some of these sort of you know, smaller things that can kind of almost be like patched onto existing systems, whether these are complementary to one another and whether the same people are working on both sides.
• 39:38
  SPEAKER_01: So we only have basically the three models that we put out as three efforts, and we don't create separate solutions for each application that I'm talking about.
• 39:49
  SPEAKER_01: Like even this bot application is simply an application that captures a model that does a lot more than that.
• 39:57
  SPEAKER_01: So we one thing that we avoid is trying to create like a team per application or per solution, and instead we bring applications as benchmarks that help us guide the overall system.
• 40:10
  SPEAKER_01: And so that enables us to stay nimble, build single models that can do many different tasks, but then track performance across multiple applications as they evolve.
• 40:20
  SPEAKER_01: And we leverage, of course, uh the deep collaboration that we have with Gemini.
• 40:25
  SPEAKER_01: And sometimes we just get a huge push-up just because Gemini fundamentally improved.
• 40:31
  SPEAKER_01: And so that immediately improves all of robotics.
• 40:34
  SPEAKER_01: And other times we are bringing important applications of robotics into Gemini that then boost Gemini's vision capabilities.
• 40:41
  SPEAKER_01: And so it is uh a cycle that helps us stay focused on a few single models that have a broad range of capabilities.
• 40:49
  SPEAKER_00: I I know, you know, like I alluded to one one of the reasons why Botswynamics with Spot makes a lot of sense again, because like there's so many deployed out there.
• 40:59
  SPEAKER_00: Um I'm I'm curious, and and I don't think I've heard you or anybody really talk about this.
• 41:05
  SPEAKER_00: And if this is something that you do talk about, but why Apptronic was the company, you know, why that was a fit, why that made sense for this like really strong ongoing partnership that the the two companies have.
• 41:16
  SPEAKER_01: Yeah, I mean, we have multiple, as you know, we have multiple partners.
• 41:19
  SPEAKER_01: We have electronic, uh, boss and dynamics, agile robots, and then these are some partners that we work with the deepest, but then we also have a broad range of other trusted testers.
• 41:29
  SPEAKER_00: So, but like I look at the Gemini blog, like Gemini 2 blog, and like you know, Apollo's all over that.
• 41:33
  SPEAKER_00: So, like it's it is a very like it's a strong partnership.
• 41:37
  SPEAKER_01: Yeah, I mean, I think it was uh certainly for Gemini Robotics 2, and for each one of these um releases that we do, we try to capture just what has we are always multiple having multiple research threads going at all times.
• 41:52
  SPEAKER_01: And we think about each one of these releases as checkpoints along the way.
• 41:56
  SPEAKER_01: So we don't want to wait, as you said, to get all the way to AGI to show our progress.
• 42:00
  SPEAKER_01: And so simply the electronic partnership was one that we started earliest and therefore had all of this work around whole body general actions going.
• 42:09
  SPEAKER_01: And it was a great time to show what we have accomplished with them.
• 42:12
  SPEAKER_01: And so that's why it's all over there.
• 42:14
  SPEAKER_01: But it's certainly one that has helped us uh earliest sort of explore that question of can we get these models to really understand not only things on a table, because even though that's actually still quite complex, you know, understanding everything that can be done in a table, but we wanted to really understand whether the models could really get to understanding an entire room and understand, especially 3D reorientation and understand how their position with respect to objects.
• 42:43
  SPEAKER_01: And that is something that the abstract partnership helped us evaluate pretty early on.
• 42:48
  SPEAKER_01: Um so that's why it's all over.
• 42:50
  SPEAKER_01: I think that um we are what we look on partners is partners that can help us, that can complement what we're doing.
• 42:57
  SPEAKER_01: So we're very much focused on the models and they can very much be focused on the evolution of the hardware and that can work closely with us because we believe, I'm I'm sure, well, I imagine everybody believes that the robots that we have today are going to evolve dramatically over the next two years.
• 43:13
  SPEAKER_01: So I think what is most important is that you have a strong team that you can work with to evolve both the hardware and the AI together so that you can get to the best possible point in a couple of years' time.
• 43:24
  SPEAKER_01: And so, yeah, along the way, we're very excited with and very thankful to our to all our partners for what we've been able to do.
• 43:31
  SPEAKER_01: And what could you should expect that we'll continue to show the new partners as well in the in the follow-up releases.
• 43:38
  SPEAKER_00: Yeah, it it's interesting because I feel like the the two ways um, you know, I I mentioned this sort of like big bite of the apple that uh the boiling the ocean, as they say in in in your neck of the woods, of like, you know, do do we get these things out there or do we solve AGI?
• 43:53
  SPEAKER_00: And then the other piece of that is like big part of the reason why we're seeing these like huge valuations.
• 44:00
  SPEAKER_00: And huge investments is because at least a handful of these companies are really trying to go as full stack as possible, you know, including uh the the AI, the software, uh, everything else.
• 44:13
  SPEAKER_00: Um, but then there are a lot of other companies that that are doing AI, but are also partnering with you at the same time.
• 44:20
  SPEAKER_00: Um and and you alluded to this a bit.
• 44:24
  SPEAKER_00: I'm curious when when you talk about that partnership, you know, how much of a two-way street is it as far as the hardware is concerned?
• 44:32
  SPEAKER_00: Like how direct is that conversation?
• 44:34
  SPEAKER_00: And like ultimately, abstractly speaking, like how much you influence sort of some of the hardware decisions that they make.
• 44:42
  SPEAKER_01: Yeah, I mean, in general, the reason that we went together towards a partnership was how do we help accelerate each other and the field?
• 44:50
  SPEAKER_01: That's that's the mindset that we come into it.
• 44:52
  SPEAKER_01: And so we certainly know that we're bringing different parts of the problem into it.
• 44:57
  SPEAKER_01: And I think we're very open to get feedback in both directions, right?
• 45:01
  SPEAKER_01: So we and this matters because in many times you can make your problem significantly harder if you have the wrong sensing modalities or the wrong cameras, the wrong actuators.
• 45:12
  SPEAKER_01: All of this actually contributes to how hard it is to solve the problem.
• 45:15
  SPEAKER_01: So we do send feedback back and forth, and they also send us feedback on the performance of the models, the applications that they care about, because they each each one of our partners has made partnerships with uh with potential customers, right?
• 45:29
  SPEAKER_01: And so they know deeply how they want to deploy these models and they bring that feedback back to us.
• 45:34
  SPEAKER_01: So it actually goes both ways, and I think it's really important to do that at this stage.
• 45:38
  SPEAKER_01: It is still pretty early for humanoid, for humanoid general purpose robots.
• 45:43
  SPEAKER_01: So you want to be able to evolve it together.
• 45:46
  SPEAKER_00: I think one of the big points uh in favor of large models, one of the best arguments that I've seen in favor of large models, is um I've spoken to a number of physical AI companies uh that have said, oh, we're noticing now, like much to our surprise, that uh it's actually useful to train the system on this completely unrelated task and that will inform the job that we're trying to do here.
• 46:13
  SPEAKER_00: Is are you finding that to some extent that like some unrelated actions are actually helping unrelated actions?
• 46:25
  SPEAKER_01: So, I mean, we definitely believe that solving the problem in a general way while it's harder, it actually leads to discerning intelligence about what it means to solve a particular part of the task.
• 46:37
  SPEAKER_01: So, to give you an example, if you're actually doing, say, logistic tasks, and all you do in logistic tasks is objects that are red, let's just say.
• 46:46
  SPEAKER_01: And all your data is red.
• 46:48
  SPEAKER_01: And all of a sudden you put this robot in a totally different scene.
• 46:51
  SPEAKER_01: The objects are exactly the same, but they're all blue.
• 46:54
  SPEAKER_01: I think unless you have two things, unless you have diversity on your data that involves variety of environments, which means different colors and different objects, and you have a backbone that understands, understands the difference between different colors, you are going to memorize the color as an associated trait of an object.
• 47:15
  SPEAKER_01: And so it is going to be a model that is only going to predict to pick up the one red object that is in that place.
• 47:22
  SPEAKER_01: Um, so that is an exaggeration, but it's a lot of what happens, right?
• 47:26
  SPEAKER_01: If you is it's the same as probably, I imagine humans will be a lot better than this, but if humans grow in only a particular type of world and then all of a sudden you put them in a moon with the very different objects, they will be confused and it will take them some time to adjust.
• 47:40
  SPEAKER_01: And so robots are the extreme version of that.
• 47:43
  SPEAKER_01: If they don't see variety, they can't discern what matters and what doesn't matter when it comes to doing a task.
• 47:50
  SPEAKER_01: And so I think that relates to what you're mentioning it, that what you're mentioning, which is does learning a broad range of tasks make me better at learning one task?
• 47:59
  SPEAKER_01: I we believe so.
• 48:01
  SPEAKER_01: We believe so because it enables you to be able to reason about what's different about this task and how I can solve it in under different conditions.
• 48:09
  SPEAKER_00: And like it teaches you about physics and other sort of broadly important things like that.
• 48:13
  SPEAKER_01: Precisely.
• 48:14
  SPEAKER_00: Yeah.
• 48:15
  SPEAKER_00: So we'll we're we're at time and and we can close on this, but so I'm obviously gonna ask you like a big broad question as we're closing.
• 48:21
  SPEAKER_00: But this this kind of relates to the um the aptronic uh conversation to a certain extent, because around the same time that uh Gemini 2 was announced, they announced uh Robot Park, which I found out was something that apparently Jeff Cardenas he had the name Robot Park since childhood and finally was able to uh to use it for this huge like 90,000-foot robot training facility, like robots in the real world collecting data.
• 48:46
  SPEAKER_00: It seems like a lot of physical AI companies right now have moved in that direction, like brute forcing it, getting as many out in the world.
• 48:53
  SPEAKER_00: There's still a lot of companies uh who are convinced it's gonna largely be video for pre-training.
• 49:00
  SPEAKER_00: Um, I'm seeing more and more sort of telepresence, egocentric telepresence start start to pop up.
• 49:07
  SPEAKER_00: Um Do you have, you know, Go Google being or DeepMind having been ahead of a lot of these trends, I mean, do you have a sense that one form of data training when it comes to like really getting this flywheel started is going to be more valuable than any other?
• 49:27
  SPEAKER_01: Yeah, I mean, this is something that we've spoken out about for a while now, is like we certainly believe that for robotics to be scalable and to be operating in every human-centric space the same way you and I can, they need to have the ability to learn from very rich, large-scale data.
• 49:44
  SPEAKER_01: And certainly robot data is the most accurate one if you want to map a robot to be able to behave in a certain way, but it's not scalable.
• 49:54
  SPEAKER_01: It only scales with the number of robots and number of hours in the day.
• 49:56
  SPEAKER_01: And so we've always had uh multiple threads going, including learning from simulation, learning from video human data, robots learning from their own mistakes.
• 50:06
  SPEAKER_01: And this is something that um the entire industry now, of course, has adopted.
• 50:10
  SPEAKER_01: And I think that honestly, there's still like research breakthroughs to be done to really properly absorb information, this the same way humans can.
• 50:20
  SPEAKER_01: Um, but we're all making significant strides in that direction.
• 50:24
  SPEAKER_01: So I hope that certainly the goal at the end of the day is one where robots learn the majority of what they know from observing videos, and then they are able to practice a little bit to adjust it to their own body and be able to improve uh as they continue to do their work.
• 50:42
  SPEAKER_01: So that's the that's where we all want to go.
• 50:45
  SPEAKER_01: I think that there's still some breakthroughs to get there.
• 50:48
  SPEAKER_00: Well, Carolina, it's been an absolute pleasure.
• 50:51
  SPEAKER_00: Thank you so much.
• 50:52
  SPEAKER_01: Thank you so much.
• 50:54
  SPEAKER_00: Thanks so much to Carolina and Google DeepMind for that great conversation.
• 50:58
  SPEAKER_00: Thanks to you for tuning in.
• 51:00
  SPEAKER_00: Please like, subscribe, share, review, and check out the rest of our work over at automated.fm.
• 51:07
  SPEAKER_00: Automated's next happy hour and live podcast recording will be in the Bay Area.
• 51:12
  SPEAKER_00: On Tuesday, September 22nd, we will be at Circuit Launch in Mountain View.
• 51:17
  SPEAKER_00: Thank you to our friends over at Silicon Valley Robotics for helping to set that up.
• 51:21
  SPEAKER_00: More information and a link to sign up in the show notes.
• 51:25
  SPEAKER_00: Stick around, and we will see you back here very soon for another episode of Automated.