Latest / Automated with Brian Heater / Daniela Rus on Humanoid Robots, Physical AI, and the Future of Robotics

Transcript

Daniela Rus on Humanoid Robots, Physical AI, and the Future of Robotics · Automated with Brian Heater

• 0:00
  SPEAKER_00: It's not so difficult to get a car to steer on an empty road.
• 0:04
  SPEAKER_00: The big challenges have to do with dealing with the messiness of the world.
• 0:08
  SPEAKER_00: If your car is driving at 60 miles an hour, does the car have the luxury to wait for the cloud to tell it what to do?
• 0:15
  SPEAKER_00: I would say no.
• 0:17
  SPEAKER_00: We are at such an extraordinary moment in computing, in robotics, in AI.
• 0:22
  SPEAKER_00: I just want to live every second of it and make uh the most contribution I can.
• 0:28
  SPEAKER_01: You're pretty down on humanoids, or at least bipedal humanoids.
• 0:31
  SPEAKER_01: Is that have you softened at all?
• 0:33
  SPEAKER_00: There are many advantages that there are for the humanoid shape, but we're not ready for prime time with humanoids just yet.
• 0:42
  SPEAKER_01: I've spoken with a lot of uh physical AI companies over the last several months, and I feel like everyone is again moving towards these really large models for robotics.
• 0:52
  SPEAKER_01: Is that the direction ultimately that you see things going in when we're talking about embodied AI or physical AI?
• 0:58
  SPEAKER_00: We are at a point in time where it's important to explore a lot of different directions.
• 1:03
  SPEAKER_00: And so we we see university research, we see startups exploring in many different directions.
• 1:10
  SPEAKER_00: My own view is that hello and welcome to another episode of Automated.
• 1:27
  SPEAKER_01: My name is Bright Heater.
• 1:29
  SPEAKER_01: I am the managing editor at the Association for Advancing Automation.
• 1:33
  SPEAKER_01: We are back with our second ever live show, and one of my favorite people in the whole of robotics, Daniela Rus, is the director of MITC sale, among many other things.
• 1:46
  SPEAKER_01: Uh, chatting with her always offers some incredible insight into just how many world-changing aspects are currently being impacted by robotics.
• 1:55
  SPEAKER_01: Uh, self-driving cars, we've got crowning sperm whales.
• 1:59
  SPEAKER_01: This conversation has it all.
• 2:01
  SPEAKER_01: Thank you so much to Daniela and her team, and of course, Mass Robotics for hosting us.
• 2:06
  SPEAKER_01: If you're enjoying the show, don't forget to like and subscribe, and please check out the newsletter over at automated.fm.
• 2:12
  SPEAKER_01: And with that, please enjoy this conversation with Daniela Roos.
• 2:17
  SPEAKER_01: You know, we talk a lot about what's coming up next in Automation on the show, but if you really want to see the future motion, you've got to be there in person.
• 2:24
  SPEAKER_01: Automate 2026 is where the world's leading innovators, builders, and dreamers come together to show you what's possible.
• 2:31
  SPEAKER_01: Robots AI, machine vision, motion control, you name it, all automation under one roof.
• 2:37
  SPEAKER_01: And as part of Automate this year, the Humanoid Robot Forum brings together leaders, engineers, and researchers for a two-day deep dive into the real-world development, deployment, and commercialization of humanoid robotics.
• 2:50
  SPEAKER_01: Register for free at automateshow.com to join us in Chicago June 22nd through the 25th.
• 2:58
  SPEAKER_01: We will see you there.
• 2:59
  SPEAKER_01: The last time that I saw you, we were, I think we were at the automate show, and uh it was very brief.
• 3:06
  SPEAKER_01: And I think I was chasing you through the halls.
• 3:07
  SPEAKER_01: You had your suitcase with you because you had to jump on a plane.
• 3:11
  SPEAKER_01: Um thank you for doing this.
• 3:14
  SPEAKER_01: I feel like I'm a busy person.
• 3:15
  SPEAKER_01: I don't know how you possibly maintain your schedule at all.
• 3:20
  SPEAKER_00: Brian, just like you and everyone else does.
• 3:23
  SPEAKER_00: So yeah.
• 3:24
  SPEAKER_00: Um sometimes I have to run through hallways with a suitcase.
• 3:27
  SPEAKER_01: Yeah.
• 3:28
  SPEAKER_01: Yeah.
• 3:28
  SPEAKER_01: I mean, obviously you've you've got a lot going on.
• 3:30
  SPEAKER_01: You've got your teaching schedule, you're on a number of boards, it seems like you're you're flying all over the place.
• 3:35
  SPEAKER_01: I mean, how do you determine what you prioritize at any one given time?
• 3:40
  SPEAKER_00: You know, honestly, I just want to do it all.
• 3:42
  SPEAKER_00: Yeah.
• 3:43
  SPEAKER_00: I feel like we are at such an extraordinary moment uh in our field and uh in computing, in robotics, in AI.
• 3:52
  SPEAKER_00: And I just want to live every second of it and make uh the most contribution I can.
• 3:58
  SPEAKER_00: It's uh I think we're very lucky.
• 4:00
  SPEAKER_00: We're really the generation that is bringing uh intelligence and uh robots and automation uh to the world in a way that supports people and supports the planet, and uh and it's an extraordinary time.
• 4:14
  SPEAKER_01: Yeah, because you you and I have been speaking for a number of years now, like I probably close to 10 at this point.
• 4:20
  SPEAKER_01: And when we first started speaking, there was a lot of you know, five to ten years, five to ten years, but it really feels like the last few years, a lot of these things you've been talking about have really started coming to fruition.
• 4:31
  SPEAKER_00: Well, um uh yes.
• 4:32
  SPEAKER_00: So 10 years ago, Brian, you were the only one who wanted to talk with me.
• 4:37
  SPEAKER_00: It was really great.
• 4:38
  SPEAKER_00: Um, but uh look at uh technology development is like this.
• 4:43
  SPEAKER_00: Um, you make uh steps, you have you have people who think about the future, who think about what might happen 10 years from now, 25 years from now.
• 4:52
  SPEAKER_00: And um at MIT CSEL, we have always been focused on what we can do today that still has relevance and impact 25 years from today.
• 5:02
  SPEAKER_00: And we really have been thinking about living in the future.
• 5:06
  SPEAKER_00: Uh the exciting thing is that the future we were dreaming about 10 years ago is now.
• 5:10
  SPEAKER_00: And now we have to kind of figure out how do we think about the future 10 years from now, but also be in the moment, be in the now, because um there is so much excitement and opportunity.
• 5:23
  SPEAKER_00: And you see, universities have the mission of advancing the knowledge, of showing that ideas uh can um can take flight, um, that capabilities are possible.
• 5:36
  SPEAKER_00: This is what we do at the university, and there is this extraordinary um attraction to thinking about what we used to think about back then um and and and to see that is uh to see it happening right now.
• 5:53
  SPEAKER_00: And I would say that it's not trivial to take a research grade solution and turn it into a solution that is impacting the world today into a product.
• 6:04
  SPEAKER_00: And so that part of the innovation cycle is also exciting and and important.
• 6:11
  SPEAKER_00: And um I I'm just trying to figure out how I can do some of that and I can continue to do the things that I've loved doing all my life.
• 6:20
  SPEAKER_01: I want to take two pieces of technology that that I know um CSEAL has been involved in and have obviously have taken two very different paths.
• 6:27
  SPEAKER_01: There's um autonomous driving and then there are LLMs.
• 6:31
  SPEAKER_01: In terms of where you expected things would be and where things are, it seems like self-driving has maybe taken a little bit longer.
• 6:39
  SPEAKER_01: LLMs seem to have kind of surprised everybody.
• 6:42
  SPEAKER_01: You know, even people who worked on them seem surprised at like how quickly they got as good as they got.
• 6:48
  SPEAKER_00: Yeah, so it's exciting.
• 6:50
  SPEAKER_00: Um, self-driving um cars have been, and and in some sense, navigating robots uh have been um part of the conversation in the robotics community for decades.
• 7:02
  SPEAKER_00: In our research group, we had our first uh self-driving research vehicle was navigating the streets of Singapore in 2011, 15 years ago.
• 7:13
  SPEAKER_00: It's taken some uh research demonstrations and then companies like Google Waymo and others uh to take those ideas and make them robust.
• 7:24
  SPEAKER_00: The the challenge is the long tail, right?
• 7:27
  SPEAKER_00: I mean it's not so difficult to get a car to steer on an empty road.
• 7:31
  SPEAKER_00: Uh what it's challenging.
• 7:34
  SPEAKER_00: But well, in today's in today's landscape, that is not um that that is not where the big challenge is.
• 7:40
  SPEAKER_00: Uh the big challenges um have to do with dealing with the messiness of the world, dealing uh with people who break the traffic rules, edge cases, uh dealing with weather.
• 7:53
  SPEAKER_00: So there are a lot of edge cases um and what we call the long tail.
• 7:58
  SPEAKER_00: And and so we continue to work on uh on these ideas.
• 8:04
  SPEAKER_00: Sensors don't work well in snow and in rain.
• 8:07
  SPEAKER_00: That's why we don't have uh rides by Waymo in Boston.
• 8:11
  SPEAKER_00: Boston is too chaotic, actually, even the way people drive in Boston makes it really difficult for a robot car that's expecting people to follow the rules to operate effectively.
• 8:22
  SPEAKER_00: But um we continue to make progress and we already see a lot of um uh interesting capabilities from the self-driving um industry.
• 8:32
  SPEAKER_00: We see, of course, the cars that are offering autonomous rides to people, but then there are a lot of other powerful applications that the world is talking about less.
• 8:42
  SPEAKER_00: For example, we have autonomous driving in industrial settings that is making a huge difference.
• 8:49
  SPEAKER_00: Our company, Venti Technologies, has been uh deploying fleets of autonomous 40-foot-long trucks that are supporting the operations in the port of Singapore.
• 9:00
  SPEAKER_00: And they've been operating 24-7 for months, for more than a year now, and they will be deployed in other settings.
• 9:08
  SPEAKER_00: There are other companies that are built that are bringing these self-driving solutions uh in industrial settings, in um on campuses, in um in um areas that are off public roads.
• 9:23
  SPEAKER_00: And the technologies are really good for those kinds of environments.
• 9:26
  SPEAKER_00: The technology is really good for um either uh closed environments which are very carefully mapped, or for environments that are simple, where things don't move very fast, where we don't have so much uncertainty from the perception system of the car.
• 9:46
  SPEAKER_00: So great progress in self-driving vehicles, but still we have a long way to go.
• 9:51
  SPEAKER_01: Also, just you know, it's I uh I like to point out that probably nothing that we're talking about in robotics right now would be really possible in the way it is without all of that progress that had been done with self-driving cars.
• 10:01
  SPEAKER_00: But exactly, exactly.
• 10:03
  SPEAKER_00: Um we, I mean, you know, this this progress in robotics is enabled by advancements in hardware, by advancements in computing, and by advancements in algorithms.
• 10:14
  SPEAKER_00: And for autonomous driving, the key contributions um had to do with making maps, uh, with perception, with detecting things reliably so that you don't bump into things, the car doesn't bump into things, uh, also with planning and control.
• 10:30
  SPEAKER_00: But the hardware plays a very important role.
• 10:34
  SPEAKER_00: Um, so having accurate um um maps uh really depended on having uh laser scanners, the LIDAR sensors, and then figuring out how to couple the feedback from a LIDAR sensor to the feedback from a camera.
• 10:51
  SPEAKER_00: Back in the day when I was a graduate student, um, I used to build navigation algorithms using sonar.
• 10:58
  SPEAKER_00: And sonar is extraordinarily uh noisy and uh nothing worked back then.
• 11:04
  SPEAKER_00: After the LIDAR was introduced, all the algorithms that didn't work before with sonar, all of a sudden they all started working because measuring distance with light is much more precise than measuring distance with sonar.
• 11:17
  SPEAKER_00: And so the hardware that is available to us is very important uh in uh solving the problem.
• 11:24
  SPEAKER_00: So uh the sense uh sensors and actuators are important, but also the speed of computation uh is important, and we will continue to see great progress moving in the future.
• 11:35
  SPEAKER_00: For autonomous driving, uh, what's difficult now is uh driving in inclement weather.
• 11:41
  SPEAKER_00: Um so what do you do when within trains really hard?
• 11:45
  SPEAKER_00: What do you do when it snows and you don't see the road?
• 11:48
  SPEAKER_00: And at MIT, we've been looking into this use case in this part of the uh self-driving problem, and we um we have a solution that is not uh deployed in uh commercial products yet.
• 12:03
  SPEAKER_00: But it turns out that if you uh put a ground penetrating uh radar under the car and you look at the texture of the soil, uh we see unique features that could serve uh as landmarks, just like the features uh we collect with LIDAR for locating inside a map do.
• 12:26
  SPEAKER_00: So, like it's possible to deal with weather by looking down rather than looking up.
• 12:31
  SPEAKER_00: Isn't this interesting?
• 12:32
  SPEAKER_00: And this is a capability that is enabled by a hardware advancement coupled with a software advancement.
• 12:39
  SPEAKER_01: I I I want to back up a second because I'm curious about the the sonar thing.
• 12:42
  SPEAKER_01: It's just always been really interesting to me.
• 12:45
  SPEAKER_01: Um, you know, in cases where you're working on a technology like that and and really kind of in a certain sense, you know, banging your head against the wall, and then something a couple of years comes along and it makes it a lot easier.
• 12:57
  SPEAKER_01: Can can that be frustrating that you were, you know, that it was this really difficult research and then no?
• 13:03
  SPEAKER_00: No, no, it's exciting, right?
• 13:04
  SPEAKER_00: Like you want you want the work to uh to have an impact.
• 13:09
  SPEAKER_00: And so uh the algorithm, the the basis of the invention remains the algorithm.
• 13:14
  SPEAKER_00: The algorithm couldn't work when um the the sensors were imprecise, but all of a sudden we got a precise sensor, and boom, everything started um working.
• 13:24
  SPEAKER_00: That was an extraordinary moment.
• 13:26
  SPEAKER_01: Yeah.
• 13:27
  SPEAKER_01: You you had a a moment, uh it was it was John Hopcroft, right?
• 13:32
  SPEAKER_01: That was the visiting Iowa professor, and it sounds like not long after you had moved to the States, he kind of changed everything for you as far as um your career path.
• 13:42
  SPEAKER_00: Oh, I still remember that day.
• 13:44
  SPEAKER_00: So I um I went to a lecture that was given by one of the founding fathers of the field of computer science, John Hopcroft.
• 13:53
  SPEAKER_00: And uh interestingly, in this lecture he said, computer science is dead.
• 13:58
  SPEAKER_00: And well, what he meant was a lot of the problems that were posed by um uh by the um early founders of the field had solutions.
• 14:09
  SPEAKER_00: So he actually didn't mean it.
• 14:10
  SPEAKER_00: Maybe he didn't say computer science is dead, maybe he said computer science is solved.
• 14:14
  SPEAKER_00: And now it's time for the grand applications.
• 14:17
  SPEAKER_00: And the grand applications were about computation that interacts with the world.
• 14:22
  SPEAKER_00: And in in some sense, this is what robotics is.
• 14:25
  SPEAKER_00: And so I was very excited by this vision of building machines that interact with the physical world, and I went to work with him for um for this objective.
• 14:35
  SPEAKER_01: What was it about him specifically that made you really?
• 14:37
  SPEAKER_01: I mean, you f you followed him, you went and worked with him.
• 14:40
  SPEAKER_01: I mean, he really did change that path for you.
• 14:42
  SPEAKER_00: He is a visionary.
• 14:43
  SPEAKER_01: Yeah.
• 14:44
  SPEAKER_00: He really sees into the future with great clarity.
• 14:48
  SPEAKER_00: And back in the day when I was a graduate student, um, he had the vision of robots.
• 14:54
  SPEAKER_00: He also had the vision of uh simulation environments where you can test how machines work and that could be shared by labs across the world.
• 15:05
  SPEAKER_00: There were um the uh networking was not good enough to do it back then, but look where we are now.
• 15:12
  SPEAKER_00: All the predictions he made back then have come to pass.
• 15:16
  SPEAKER_01: Yeah.
• 15:17
  SPEAKER_01: So that that sounds like a familiar story then.
• 15:20
  SPEAKER_01: A lot of times um there's just sort of a bottleneck.
• 15:23
  SPEAKER_01: You know, there's a lot of these kind of ideas behind the scenes, but the technology just isn't there to execute them.
• 15:28
  SPEAKER_00: Well, the technology and the ideas, um, that's why it's important to follow the path and build on everyone's prior work.
• 15:37
  SPEAKER_01: What do you see your role being at a place like CSEL?
• 15:41
  SPEAKER_00: Well, I uh as uh the director of CSale, I see my role as thinking about the future and building programs and bringing ideas for my colleagues to help develop.
• 15:56
  SPEAKER_00: I'm very interested in topics that are bigger than an individual researcher's um area of expertise.
• 16:05
  SPEAKER_00: So I've been looking at well, what are some of the um challenges and problems that require expertise from multiple aspects of computing, maybe even reaching out to other disciplines?
• 16:19
  SPEAKER_00: And how can we then create an opportunity for people to come together and address those challenges?
• 16:27
  SPEAKER_00: So, from an intellectual point of view, I'm very focused on new ideas that will build the future 10 years from now, 25 years from now.
• 16:37
  SPEAKER_01: Yeah, I uh a very long time ago now, I think I sat in on one of those those meetings, uh, it was sort of almost like a pitch meeting, it felt like.
• 16:45
  SPEAKER_01: Um how how do those projects really sort of start to spin out?
• 16:51
  SPEAKER_00: It has to be a kind of a combination of top-down and bottom-up conversations.
• 16:56
  SPEAKER_00: So top-down and bottom-up, because uh if the leader of an organization like MIT says, let's work on X, um, well, that path is not successful unless people are interested in working on X.
• 17:11
  SPEAKER_00: And so at the same time, if the group decides that Y is interesting, but there's no way of identifying the resources uh to enable Y to proceed, again, we're stuck.
• 17:25
  SPEAKER_00: So we have we have to combine the um the ability to identify resources for a project with a people's interests to work on those problems.
• 17:38
  SPEAKER_01: So let's take a recent example.
• 17:40
  SPEAKER_01: This is something, again, that you and I spoke about a number of years ago.
• 17:43
  SPEAKER_01: You were very excited.
• 17:44
  SPEAKER_01: You said we can't talk about this yet.
• 17:46
  SPEAKER_01: And then I looked in the New York Times not too long ago, and it seems like you're finally really ready to talk about it.
• 17:52
  SPEAKER_01: A lot of the research that you had been connected with around um sperm whale breeding.
• 17:58
  SPEAKER_01: That news has finally surfaced.
• 18:01
  SPEAKER_01: Um, MIT CSAL was involved in that.
• 18:04
  SPEAKER_01: What was your role and how did you get involved with that?
• 18:08
  SPEAKER_00: I've been uh interested in uh whales and uh how technology can help us understand uh their lives for more than a decade.
• 18:19
  SPEAKER_00: And that all came to be because I was at the MacArthur Fellows gathering meeting and I heard Roger Payne um give a lecture about his beloved whales, and I was just mesmerized.
• 18:34
  SPEAKER_00: And so I still remember going to Roger to say, Hey, I love your uh your whales song.
• 18:39
  SPEAKER_00: What can I do for you?
• 18:40
  SPEAKER_00: Can I make you some robots?
• 18:41
  SPEAKER_00: Is there something that uh you need that would help you um study the whales better?
• 18:47
  SPEAKER_00: And so we so back then, this was 2007, 2008, 2009, it was really a long time ago.
• 18:55
  SPEAKER_00: Uh back then, um my students were building drones.
• 18:59
  SPEAKER_00: And so we thought, oh, maybe we can take some drones um to observe whales close up.
• 19:07
  SPEAKER_00: And we did indeed uh have a project where we took drones and we studied uh whales off the coast of Argentina uh in 2009, and the kind of uh behaviors and footage that um we could uh get with the drones was was extraordinary.
• 19:26
  SPEAKER_00: Um Roger spent years on a cliff with paper and pencil and binoculars, uh essentially looking at who goes by.
• 19:36
  SPEAKER_00: And he knew who went by because whales have closities on their heads, southern right whales do, and these closities are unique to every individual.
• 19:46
  SPEAKER_00: And so for about 40 years, Rogers stood on a cliff and um and kept track of who he could see went by.
• 19:55
  SPEAKER_00: So now imagine the change that um that he experienced when he could essentially just throw a drone uh out at sea to really look close by.
• 20:06
  SPEAKER_00: Well, who are they?
• 20:07
  SPEAKER_00: What are they doing?
• 20:09
  SPEAKER_00: Uh the kind of um of uh data and resolution that he could get with a drone was uh something that he didn't even um imagine, and Roger was quite a visionary.
• 20:21
  SPEAKER_00: So it was transformational for his work.
• 20:24
  SPEAKER_00: And then a few years later, we were at a workshop at Harvard, and we started talking about well, you know, what if we could uh exchange communication with whales?
• 20:37
  SPEAKER_00: What if we could someday understand what whales are saying to each other?
• 20:42
  SPEAKER_00: What is uh what is part of the um the communication mechanism, the vocalization and the song of whales.
• 20:50
  SPEAKER_00: So we wrote a um proposal for a Ted Audacious prize to reverse engineer the language of sperm whales.
• 20:58
  SPEAKER_00: And we were awarded the prize.
• 21:01
  SPEAKER_00: A lot of people were involved uh in this program.
• 21:04
  SPEAKER_00: Uh Roger certainly was, I was part of it.
• 21:08
  SPEAKER_00: David Gruber, uh Rob Wood, and others were all part of writing this proposal.
• 21:13
  SPEAKER_00: We got the funding, and we've been um studying the recordings of uh the sperm whales in Dominica.
• 21:21
  SPEAKER_00: The sperm whales uh click, so they just go something like that.
• 21:26
  SPEAKER_00: And it's interesting because their language is discrete, and as a result, it's much easier to analyze using computer science techniques.
• 21:34
  SPEAKER_00: And um we've had several uh breakthroughs in um in our understanding of um of the language of uh sperm whales, but we're still quite far from really knowing what they say and um what is the right way to think about uh this um these insights and innovation.
• 21:57
  SPEAKER_00: Um we've uh analyzed.
• 22:01
  SPEAKER_00: The alphabet, and we've shown using AI, we were able to show that in fact there are many more units of speech, which are called codas, than previously imagined.
• 22:16
  SPEAKER_00: We showed that whales have the possibility of expressing about 540 different types of sounds.
• 22:25
  SPEAKER_00: And prior to our work, it was believed that they could only do 21, 20 or 21.
• 22:31
  SPEAKER_00: And so this actually matters a lot because when you can make a lot of sounds, that means you have the capacity to transmit and encode a lot of complex information.
• 22:43
  SPEAKER_00: And this is what makes the difference between animals that can vocalize simple things and animals that can transmit complex ideas.
• 22:56
  SPEAKER_01: I mean, I feel like I've had a lot of conversations at cocktail parties where it's just like, hey, let's you know figure out if we can get whales to talk, but you're actually able to like execute on that, which is that's a nice part of your job.
• 23:07
  SPEAKER_00: Well, it's so exciting.
• 23:09
  SPEAKER_00: I told you it's uh it's an extraordinary time.
• 23:12
  SPEAKER_00: So um look, we first have to understand uh the vocalization and the communication structure, and then we have to understand how that connects to their behaviors and their lives, and then we have to understand what do we do with that so that we do not endanger the whales, but rather we protect them.
• 23:34
  SPEAKER_01: Yeah, I I think that that's an underrated probably part of your job, um, certainly part of my job, and maybe just part of you knowtics in general is um uh a curiosity and an interest in in other fields.
• 23:49
  SPEAKER_01: I mean, that's sort of what what unites us.
• 23:51
  SPEAKER_01: It's a big part of what you do is um, you know, being interested in whales, being interested in in sea turtles in landmarks, and that's what all of a sudden opens up all of these other different fields, and that's what connects all these dots for you.
• 24:04
  SPEAKER_00: There are definitely great opportunities across disciplines.
• 24:09
  SPEAKER_00: And computing and robotics and AI have matured to the point where we can connect these tools to other disciplines to enable discovery in other disciplines.
• 24:21
  SPEAKER_01: Yeah, uh we took I mentioned uh alluded to uh LLMs a little bit before, and and I think you were one of the first people I spoke to who was really talking about the black box problem.
• 24:31
  SPEAKER_01: Um, and when I talked about this sort of uh surprise that researchers had around um a lot of these large language models, it was because of the black box, right?
• 24:41
  SPEAKER_01: You know, all this information was going in, they didn't really know why the results were coming out.
• 24:46
  SPEAKER_01: Um, and one of the things that you have been advocating for, and I think really still are through Liquid AI, are a lot of these um these smaller, these smaller models.
• 24:54
  SPEAKER_01: Why is that something that you have been so focused on?
• 24:58
  SPEAKER_00: Uh well, Brian, the extraordinary pace of progress in AI is really inspiring.
• 25:04
  SPEAKER_00: And you have to understand that many of the ideas that are enabling these extraordinary capabilities are ideas that were invented decades ago.
• 25:15
  SPEAKER_00: And back then we didn't have enough data and we didn't have strong enough computation.
• 25:22
  SPEAKER_00: Uh, we didn't have enough memory.
• 25:23
  SPEAKER_00: And so back that uh back then they didn't work so well.
• 25:27
  SPEAKER_00: But uh then we had an inflection point where the hardware platforms and the and the data sources uh got to the point where all of a sudden those ideas started working, and that was extraordinary.
• 25:41
  SPEAKER_00: In fact, I still remember uh the talk that Sebastian Bubeck did at MIT in March of 2023, where he could reveal um some of the capabilities of GPT-4.
• 25:54
  SPEAKER_00: The world did not know that GPT-4 existed, and um he showed extraordinary programming capabilities, he showed extraordinary language generation capabilities, reasoning.
• 26:08
  SPEAKER_00: It was stunning.
• 26:09
  SPEAKER_00: Um, it was absolutely stunning.
• 26:11
  SPEAKER_00: And since then, of course, the field um uh uh continued to make progress and the capabilities increased.
• 26:20
  SPEAKER_00: And right now, what we see with agentic AI is is unbelievable.
• 26:26
  SPEAKER_00: The ability of uh the gener uh the tools to generate code and to give us organized access to information is absolutely extraordinary.
• 26:38
  SPEAKER_00: What's interesting is that now we have these huge models that require industrial scale installations and um that have a lot of benefit, but you you sort of need deep pockets in order to take full advantage of the capabilities in the cloud.
• 27:01
  SPEAKER_00: And so I I see this moment as being analogous to where computing was back in the day when we had mainframe computers, and only companies with very deep pockets were able to take advantage of computing.
• 27:17
  SPEAKER_00: And then forward-looking researchers, some at Lincoln Lab, Bill Gates, Steve Jobs, their friends, they invented the personal computer.
• 27:26
  SPEAKER_00: And the personal computer completely democratized access to computing and created an economy that we didn't imagine could exist.
• 27:36
  SPEAKER_00: A lot of our everyday comforts and the way of life today is due to computing.
• 27:43
  SPEAKER_00: Just imagine a day in your life without computing and everything it enables.
• 27:48
  SPEAKER_00: No email, no social media.
• 27:50
  SPEAKER_00: Maybe no email is good.
• 27:51
  SPEAKER_00: I mean no social media, um, no diagnosis imaging, uh, no electronic commerce, right?
• 27:58
  SPEAKER_00: Like it's it's impossible to think about it uh today.
• 28:02
  SPEAKER_00: So I am very excited about the possibility of bringing AI from the cloud uh down to earth.
• 28:09
  SPEAKER_00: Um there are many devices uh on our planet.
• 28:13
  SPEAKER_00: In fact, a large fraction of processors are in phones, in um iPads, in uh computers, in um sensors.
• 28:24
  SPEAKER_00: They're in everyday devices.
• 28:26
  SPEAKER_00: So the question is: can we empower those devices with the ability to reason with um with a kind of AI that offers the maximum level of intelligence according to the hardware specifications of the device without the need of a cloud?
• 28:44
  SPEAKER_00: I personally believe in a hybrid future where there continues to be a need of large-scale operations in the cloud because ultimately the current technology really depends on how much information you can put inside the model.
• 29:03
  SPEAKER_00: So the larger the model, the more information you can package in that model.
• 29:08
  SPEAKER_00: But not all problems require access to all the world's information.
• 29:13
  SPEAKER_00: So, what if we actually had uh customized models that run on your phone, run on your computer, run on your sensors and on your robots, and that run and deliver the capabilities that you need without the need to talk to the cloud, without the energy cost, without the water cost that is required in order to bring large AI to the world?
• 29:37
  SPEAKER_00: This is what I have been thinking about for many years now.
• 29:42
  SPEAKER_00: And, you know, honestly, this was originally motivated by robotic problems, because we wanted the models to run on device.
• 29:52
  SPEAKER_00: If your car is driving at 60 miles an hour, does the car have the luxury to wait for the cloud to tell it what to do?
• 30:00
  SPEAKER_00: I would say no, but if you have a um a well-tuned model uh that runs on device, that understands physics, that understands uh common sense in the context of driving, then you can have the AI reasoning capability on device and you can have much uh safer driving.
• 30:21
  SPEAKER_00: When we started working with Soyuta in 2015, we had this vision of the car that will never be responsible for a collision and become your friend.
• 30:31
  SPEAKER_00: So now as we think about the state of the art in driving, we've made a lot of progress on the autonomous car that um that would not be responsible for a collision.
• 30:44
  SPEAKER_00: The friend part was a little bit slower to come uh to come to fruition, but with large language models, um, the car can become your friend by putting the language inside the car.
• 30:55
  SPEAKER_01: I I've spoken with a lot of uh physical AI companies uh over the last several months, and and I feel like they're going down a similar path where um, you know, as we're talking about sort of general, generalized AI, um, you know, general purpose robots, that everyone is again moving towards these really large models for robotics.
• 31:15
  SPEAKER_01: Is that the direction ultimately that you see things going in when we're talking about embodied AI or physical AI?
• 31:21
  SPEAKER_00: We are at a point in time where it's important to explore a lot of different directions.
• 31:27
  SPEAKER_00: And so we we see um university research, we see startups exploring in many different directions.
• 31:36
  SPEAKER_00: My own view is that for AI to really deliver um good capabilities for robots, safe capabilities for robots, we need those AI systems to run on device, we need those AI systems to understand physics, and we need those uh AI systems to essentially understand the task.
• 32:01
  SPEAKER_00: And robotic tasks are primarily about the interaction between the machine and the physical world.
• 32:09
  SPEAKER_00: And interaction requires knowledge of forces and torques.
• 32:13
  SPEAKER_00: So the kind of um physical AI that we are developing in my research group at MIT aims to capture that.
• 32:21
  SPEAKER_00: And the forces and torques are really difficult to capture if your robot uh learns how to do things by watching videos or even by teleoperation.
• 32:32
  SPEAKER_00: So um I think it's interesting to consider what kind of tasks can be taught by purely visual feedback, what kind of tasks can be taught by teleoperation, and for what kind of tasks do you need uh more detailed information like forces and torques?
• 32:53
  SPEAKER_00: And when we think about robots and whether AI is an important part of the robot solution, I think it's important to consider whether the task of the robot uh is something that can be described from first principles, because if that is the case, then we don't need complex models to guess um correctly what to do next.
• 33:16
  SPEAKER_00: Um we can we can deliver a solution from for for from first principles.
• 33:22
  SPEAKER_00: However, if the task is so complex that it cannot be described from first principles, then we need to figure out how to collect the data to enable the machine to learn.
• 33:36
  SPEAKER_00: And at MIT we're looking at complex tasks uh that are done by humans that are difficult to uh map onto uh onto equations.
• 33:49
  SPEAKER_00: And um and so what we do is we instrument people doing, let's say, kitchen tasks or construction tasks.
• 33:56
  SPEAKER_00: We instrument people and we collect uh information about the muscles, um forces and torques.
• 34:03
  SPEAKER_00: Uh we also collect pose information, uh, we collect uh eye gaze information, we collect environmental information, and so from all of this data, we are then able to synthesize policies that teach machines how to do the task in a manner similar to what the human did.
• 34:23
  SPEAKER_01: Yeah, uh the conversation that that I've been having a lot lately with a lot of these people is is the flywheel problem, right?
• 34:30
  SPEAKER_01: Is is how do we actually get enough of these robots out into the the world?
• 34:34
  SPEAKER_01: Um do you get the sense that it's ultimately going to require uh physical, physical real-world data?
• 34:43
  SPEAKER_01: Will a certain um how much of this will actually be able to be executed with like synthetic data and simulation, for example?
• 34:51
  SPEAKER_01: Or do the robots actually have to be out there performing tasks in order to really get a good idea of physics and long tail problems?
• 35:00
  SPEAKER_00: I think it depends on the task.
• 35:02
  SPEAKER_01: Yeah.
• 35:03
  SPEAKER_00: And so I believe that there are some tasks that can be uh can be trained from purely visual data.
• 35:12
  SPEAKER_00: We don't know exactly what those tasks are, but the research community is working on those kinds of policies.
• 35:18
  SPEAKER_00: But then there are also tasks that I believe require understanding of the world, understanding of physics.
• 35:24
  SPEAKER_00: And so for these tasks, what should we do?
• 35:28
  SPEAKER_00: Well, one option is to try to collect more physical data.
• 35:33
  SPEAKER_00: Another option is to think about different kinds of AI models that have a notion of physics that are baked into them, that are alternatives to the transformer models that are being broadly distributed.
• 35:47
  SPEAKER_00: And um, then there are other ideas where maybe you can take what your transformer-based model uh produced as a potential solution, and then you can simulate it to see if the solution um matches the constraints and the physics of the tasks.
• 36:05
  SPEAKER_00: Um, and so I I we don't have a clear path forward.
• 36:11
  SPEAKER_00: Uh, this is a very interesting point in time uh where we are exploring all these different avenues, and eventually we will have something really amazing.
• 36:21
  SPEAKER_01: Yeah, well, one of the things that's been really interesting to me the more I've been thinking about this is when we talk about embodied AI or physical AI, is you know, when we tend to think of our bodies, we tend to think of sort of a central brain, you know, moving all over different parts, but physical AI, it could be, you know, like a per sensor, a per actuator basis.
• 36:40
  SPEAKER_00: What is important to remember about robots is that they have a body in the brain.
• 36:45
  SPEAKER_00: Yeah.
• 36:46
  SPEAKER_00: And so we need to think about what we want the body to do and um and create the morphology that is capable of the task, and then we need to have a brain that can control the body to do what it's meant to do.
• 37:00
  SPEAKER_00: And these two pieces are interrelated.
• 37:04
  SPEAKER_00: Um, we can um we can put the brain anywhere.
• 37:08
  SPEAKER_00: We can put the brain uh as a as a central coordination mechanism.
• 37:13
  SPEAKER_00: We can also create decentralized systems uh where the brain is spread across the robot body.
• 37:21
  SPEAKER_00: And interestingly, there are organisms in nature that operate according to very interesting distributed intelligence, like the octopus, for example.
• 37:32
  SPEAKER_00: So in my research lab, we have a new project where we are looking at how do we create intelligence in a decentralized way, in a way that is inspired by the way the octopus distributes uh its neurons across the entire body.
• 37:51
  SPEAKER_00: How do we get machines to learn skills?
• 37:54
  SPEAKER_00: How do we get machines to compose those skills?
• 37:59
  SPEAKER_00: And this is actually uh an important detail because most solutions that we have today for robots learn skill in context.
• 38:09
  SPEAKER_00: So that is not a very general solution for skills.
• 38:12
  SPEAKER_00: So are there new ideas that enable the learning of skill that can be then applied to any kind of context through some form of adaptation?
• 38:25
  SPEAKER_00: And then also, how how would a machine that has this kind of sophisticated ability to adapt and learn new things?
• 38:34
  SPEAKER_00: How does that machine learn to do new things?
• 38:38
  SPEAKER_00: So, like imagine you have a robot with one arm, and then all of a sudden you add a second arm to the robot.
• 38:45
  SPEAKER_00: What can the robot do?
• 38:47
  SPEAKER_00: Right?
• 38:48
  SPEAKER_00: So there are certain things you can do with one arm, but when you have two arms, then you can have uh extraordinary richness of things you could do that that one arm uh is insufficient for.
• 39:01
  SPEAKER_00: So there are some really profound questions related to uh learning when we have decentralized intelligence in the machine.
• 39:10
  SPEAKER_01: Yeah, and this brings to mind other um biologically inspired work that your team and other teams have done as far as like swarm-based robots.
• 39:18
  SPEAKER_01: So we're when we're talking about decentralized intelligence, not just on the robot itself, but just like decentralized across different robots.
• 39:27
  SPEAKER_00: Well, exactly.
• 39:28
  SPEAKER_00: So I've been very interested in decentralized intelligence for decades, uh, really thinking about collaboration, how do robots work together to do more together than what a robot can do alone.
• 39:41
  SPEAKER_00: And uh it's interesting to think um collect uh about collectives of robots.
• 39:46
  SPEAKER_00: Uh, back in the day, I um I started wondering whether we could imagine self-organizing robot systems where we had uh robotic unit modules that could connect with other modules and um realize sort of cellular uh organisms, and the cellular organisms uh could change.
• 40:07
  SPEAKER_00: They could grow a second arm if the task required a second arm, or they could uh grow a very tall neck if the robot had to go look up uh whether a tool is on the on the top of a shelf.
• 40:20
  SPEAKER_00: Right.
• 40:21
  SPEAKER_00: So so there is there has been a lot of um exciting work uh in this in this area of decentralized cellular robotics.
• 40:31
  SPEAKER_00: And um interestingly, the biggest challenge here is the hardware platform.
• 40:35
  SPEAKER_00: So we have a lot of algorithms that allow us to reason about what a group of cells can do together.
• 40:43
  SPEAKER_00: And um when it comes to realizing those capabilities at um at the hardware level, it becomes tricky because um uh because the the components that we have right now are really uh largish, so you cannot make a small uh unit module.
• 41:03
  SPEAKER_00: And then if you have a large unit module, even um even its behaviors are challenging because making and breaking connections and doing so robustly is difficult.
• 41:17
  SPEAKER_00: So there's still a lot of work that we have to do to understand decentralized intelligence when the cells are connected to each other.
• 41:25
  SPEAKER_00: We have made a lot of progress on decentralized intelligence where the units are connected by Wi-Fi by communication.
• 41:34
  SPEAKER_00: And so the swarm um work that uh you hear about most often is really about this kind of um this kind of work, a group of ground robots doing together or coordinating with a group of flying robots to get better access to information and better situation awareness.
• 41:54
  SPEAKER_01: I I interview a lot of uh a lot of artists and and musicians and creative people, and a lot of the a lot of times we talk about constraints and the power of constraints when it comes to actually sort of like focusing your creativity.
• 42:08
  SPEAKER_01: And I think that there's a parallel to be drawn here.
• 42:10
  SPEAKER_01: Something that that I hadn't really realized, and and I hope this is accurate, but something I hadn't really realized about some of your work is part of the reason why you had started doing 3D printing was kind of out of necessity because that's what you had access to.
• 42:24
  SPEAKER_01: I you know, I'm wondering when it comes to robotic hardware, whether in your history there has been power in those constraints, in not having necessarily always had access to sort of the latest and greatest and most expensive hardware.
• 42:40
  SPEAKER_00: I've always said engineering is a very creative discipline because the the real constraints we have to do our work uh force us to think outside of the box.
• 42:52
  SPEAKER_00: So I started 3D printing um in uh my first year uh as assistant professor.
• 43:00
  SPEAKER_01: That would have been around when?
• 43:01
  SPEAKER_00: Hey, let's not go there.
• 43:04
  SPEAKER_01: 2024, 2025.
• 43:06
  SPEAKER_00: Something like that.
• 43:07
  SPEAKER_00: Um it it seems it doesn't seem that long ago, uh honestly.
• 43:10
  SPEAKER_00: But I um I took this job at Dartmouth and I was in a computer science department where I had students who were brilliant programmers, but they did not know how to make stuff.
• 43:21
  SPEAKER_00: And so all of a sudden the 3D printer, the fuse deposition uh machine came uh online.
• 43:27
  SPEAKER_00: I spent my entire startup package to buy this machine, and then we started making uh robots through programming.
• 43:35
  SPEAKER_00: We started making components that were colorful, the opportunities uh opened up to be creative about how we design the robot so that the robot is not just a bunch of metal parts that are stuck together, but um, there could be a sense of aesthetics about the body of the robot.
• 43:52
  SPEAKER_00: So um that was very uh very interesting and exciting.
• 44:00
  SPEAKER_01: were when we were first really starting to talk about generative AI and I'd asked you you know what applications you were excited about as far as generative AI and robotics, you were really excited about design and and you know the ability of generative AI to actually design robots, which I've actually s seen some research recently come out from your lab on that front.
• 44:18
  SPEAKER_00: Well we are in the process of writing a paper about an an extraordinarily exciting system that could be the beginning of an AI engineer where the idea is that you could specify the function of something that you want to build in language.
• 44:37
  SPEAKER_00: And then you could have a um a generative AI system maybe a language based system study the world knowledge to understand what are the functions that you um you need for a device that meets your task specification.
• 44:54
  SPEAKER_00: So for instance let's say you want to operate a syringe you want to create a custom robot that automates a lab and needs to operate the syringe well that robot has to hold the syringe and it has to operate the plunger.
• 45:09
  SPEAKER_00: And so these are the degrees of freedom that ultimately have to be mapped to mechanisms.
• 45:13
  SPEAKER_00: And so then our system is able to reason through um uh through these degrees of freedom and create a mechanism that realizes that task under the specifications that we give.
• 45:26
  SPEAKER_00: So this is actually quite exceptional because now you don't have to program a 3D printer to make a mechanism that you have to design bottom up with great detail.
• 45:38
  SPEAKER_00: What you do is you tell your AI engineer hey I need a robot to operate a syringe and the AI engineer will crank through the world's knowledge will identify the the primitives will create mechanisms for those primitives will then feed this information to a simulator to see if indeed the design meets the specifications and it will optimize it will repeat the loop and optimize.
• 46:08
  SPEAKER_00: So in a matter of hours you could in principle generate what used to take people um months and sometimes even years to get right.
• 46:18
  SPEAKER_01: So so to really uh boil that down to the the basic like you know a newspaper headline it's like chat GPT for robots like build build me a build me a pizza robot.
• 46:31
  SPEAKER_00: It's not just chat GPT right I mean chat GPT is uh or let's just call it the chat bot let's not show pref uh preference for one of the models so let's just say the chatbot um is essential as an interface as a way of mapping a concept or a high level idea which is what humans are used to uh deal with uh into constraints and mathematics that machines can deal with and then from those constraints and mathematics you could have another AI system that can generate the solutions for those constraints well we're uh at about time right now but I but I have to ask you um you did an interview last year you were pretty down on humanoids or at least bipedal humanoids is that uh have you softened at all I think humanoids are extraordinarily exciting we have a robot lab at uh MIT C cell we call it the living lab you may have called them overhyped maybe that's where where we have created a uh an apartment and we have humanoid robots um and a lot of students who are uh thinking about how to make those humanoid robots more capable I think it's very easy to get a humanoid to um delight us uh with an extraordinary video but then uh getting the humanoid to work robustly and um and systematically on a wide range of tasks is complex.
• 48:03
  SPEAKER_00: It's complex because humanoids are complex mechanisms that are difficult to control and we still have to figure out the uh the technical aspects that enable the humanoids to interact with the world in a dynamical way.
• 48:19
  SPEAKER_00: So not just statically where you move very slowly or you only move your upper body but dynamically.
• 48:27
  SPEAKER_00: And we still have to think about how the humanoids will understand the constraints of the world and common sense when they interact with us in the world.
• 48:38
  SPEAKER_00: I was at a meeting a few months ago and there was a demonstration of a humanoid it did pretty well slowly at watering a plant and um it interacted with me with by voice.
• 48:52
  SPEAKER_00: And after the humanoid watered the plant I said now can you water my friend here?
• 48:57
  SPEAKER_00: And the humanoid was ready to dump water on the expensive Italian shoes of my friend.
• 49:03
  SPEAKER_00: So this is the kind of common sense that humans would never um mistake make a mistake but for robots this kind of common sense reasoning has to be built as part of the robot function.
• 49:19
  SPEAKER_00: So it's one thing to put a humanoid robot in a constraint manufacturing setting it's a different constraint it's a different story to put that robot in a human-centered environment so I think it's important to continue to push the boundaries to develop humanoids.
• 49:40
  SPEAKER_00: There are many advantages that there are uh for the humanoid shape but we're not ready for prime time with humanoids just yet I think it's gonna take a while longer to get humanoids to be controllable um in the way we imagine and to be interactive in the world in the way we imagine.
• 50:01
  SPEAKER_01: Well Daniela always a pleasure thank you so much thank you thanks to Daniela and the whole team at Mass Robotics for hosting us thanks as always for tuning in please like and subscribe please rate the show check out the newsletter uh that hits inboxes on Thursdays and LinkedIn on Friday and with that we will see you next week with another episode of Automated