Latest / Automated with Brian Heater / Jan Liphardt on Building the AI Brain for Humanoid Robots
Transcript
- 0:00SPEAKER_00: I basically bought a humanoid and quadruped dogs from my house.
- 0:03SPEAKER_00: I was imagining the experience to be out of the movies, but then when all the boxes arrived and you open them, it's standing there.
- 0:11SPEAKER_00: And it's not doing anything at all.
- 0:13SPEAKER_00: That then immediately led to how do we start turning this inert piece of plastic and metal into a friend, companion, helper, co-worker, and how do we personalize its behavior for specific use cases and scenarios?
- 0:28SPEAKER_01: A lot of the physical AI companies right now that I talk to seems like they are trying to solve for like all of the physical AI stuff.
- 0:35SPEAKER_01: And it seems to me that you're focused on, I don't want to say a few, but select and important problems.
- 0:41SPEAKER_00: We mix and match and integrate and assemble other people's models in addition to the models that we build.
- 0:47SPEAKER_00: That's what allows us to move very, very quickly and specifically not boil the ocean.
- 0:53SPEAKER_01: The humanoid industry is shooting itself in the foot right now with all of these flashy demos and just continuing to create these otherworldly expectations as far as what humanoid robots can and can't do at the moment.
- 1:13SPEAKER_00: And that's exactly true.
- 1:14SPEAKER_00: The amount of time it takes to try to get all of that to work well outside the lab is astronomically large.
- 1:40SPEAKER_01: I am the managing editor at the Association for Advancing Automation.
- 1:45SPEAKER_01: Today we are speaking with Open Mind founder Jan Liphard.
- 1:49SPEAKER_01: We speak to the world's most cutting-edge physical AI companies on uh pretty much a weekly basis on the show.
- 1:56SPEAKER_01: And I feel confident in saying that Jan and Open Mind are doing stuff that's uh very unique in the space.
- 2:03SPEAKER_01: We talk about that along with his academic work and a fun time as had by all.
- 2:09SPEAKER_01: Uh, thank you so much to Jan and to Open Mind.
- 2:11SPEAKER_01: Uh, thanks to you as always for tuning in.
- 2:14SPEAKER_01: Don't forget to like, subscribe, review, and check out our uh newsletter and new site over at automated.fm.
- 2:22SPEAKER_01: Automated's next happy hour and live podcast recording are going to be in the Bay Area on Tuesday, September 22nd.
- 2:30SPEAKER_01: We will be at circuit launch in Mountain View.
- 2:33SPEAKER_01: Thank you to our friends at Silicon Valley Robotics for helping to set that up.
- 2:37SPEAKER_01: More information and a link to sign up can be found in the show notes.
- 2:42SPEAKER_01: Now please enjoy this episode of Automated.
- 2:46SPEAKER_01: Vision and AI are changing what's possible in industrial automation, from quality inspection and predictive maintenance to robotics, logistics, medical devices, aerospace, and manufacturing.
- 2:58SPEAKER_01: At the Advanced Vision and AI conference, you'll hear from industry leaders, see real-world AI applications in action, explore the latest in machine vision, computer vision, sensors, edge computing, and automation technology.
- 3:10SPEAKER_01: You'll also be able to connect with the people who are shaping what comes next.
- 3:14SPEAKER_01: Whether you're building, buying, or implementing these systems, this is a chance to stay ahead of the curve and see how vision and AI are moving from promise to practical impact.
- 3:25SPEAKER_01: Learn more and make sure to buy your ticket early at the Advanced Vision and AI conference September 23rd and 24th in Santa Clara.
- 3:36SPEAKER_01: I had a friend who went to Reed College and we we used to debate who did their undergrad at the more hippie school.
- 3:50SPEAKER_01: I went to UC Santa Cruz.
- 3:51SPEAKER_01: He was convinced that Reed was a little bit more sort of like like hippie liberal arts.
- 3:56SPEAKER_01: I don't know where you where you come out on this debate.
- 4:01SPEAKER_00: I had a really uh great experience to read.
- 4:04SPEAKER_00: I thought it was just what I was looking for.
- 4:06SPEAKER_00: It's one of the very few places that has its own nuclear reactor and has very strong math, physics, and computer science, and of course, interest in typography and things like that, which is then what led Steve Jobs to make sure that the Apple computers were very good at typography, among other things.
- 4:33SPEAKER_00: But um, I had a really great uh academic experience at Reed and graduate school, and I won't mention names here.
- 4:41SPEAKER_00: Graduate school was a little bit uh disappointing after that.
- 4:45SPEAKER_00: I do have to say Reed had like two camps.
- 4:48SPEAKER_00: There was the uh more outgoing, stylish, worldly camp, and then there were the uh introverted science-y types who got their nuclear reactor uh operator certificate.
- 5:04SPEAKER_00: And uh I was very much in the latter crowd.
- 5:09SPEAKER_01: Yeah, yeah.
- 5:10SPEAKER_01: I mean, I it it's helped me, I think, significantly in my career that I I did my undergrad there.
- 5:17SPEAKER_01: Uh I was a uh BA in creative writing, but also uh the school also just happened to map the human genome.
- 5:26SPEAKER_01: So that's that's helped with my I think that's helped with my credibility over the long run.
- 5:32SPEAKER_01: Was there a little bit of bleed over though?
- 5:33SPEAKER_01: Did you get a little bit of that kind of like weird uh Oregon art school life?
- 5:38SPEAKER_00: Oh, well, Oregon is very beautiful.
- 5:40SPEAKER_00: Everything's green, there's lots of rain, there's mountains, the ocean is not too far away, and things like that.
- 5:46SPEAKER_00: So absolutely, I thought it was a um just overall, I had a very good experience uh in undergrad there.
- 5:52SPEAKER_01: Yeah, so so the nuclear reactor, it sounds like that was really the big draw for you.
- 5:57SPEAKER_00: Well, uh that that wasn't the only thing, but definitely something I remember.
- 6:02SPEAKER_00: But all of us typically only go to one college, so it's very hard to uh compare, of course.
- 6:07SPEAKER_00: Um certainly compared to graduate school.
- 6:10SPEAKER_00: I think it was uh really focused on learning and teaching the undergrads things.
- 6:14SPEAKER_00: So really appreciated that.
- 6:16SPEAKER_01: Yeah, I'm curious like specifically what it was about the nuclear reactor that well, no, I mean I ask because obviously, you know, and and and I I think that the the pendulum has to a certain extent swung back on this, but uh nuclear energy in particular has gotten a pretty you know bad rap over the last you know 50 or so decades.
- 6:39SPEAKER_00: 50 decades?
- 6:41SPEAKER_01: Um I would say I would say since maybe you know the this the the Cold War kind of put a damper on it.
- 6:47SPEAKER_00: Well, I think for any science nerd in high school, one of their dreams must be looking at Cerenkov radiation and the blue glow around a nuclear reactor core just 20 feet away from them in a pool of water.
- 7:04SPEAKER_00: I can't imagine how that wouldn't be a dream uh for most high schoolers.
- 7:10SPEAKER_01: Yeah, I so one of the things that I'm trying to sort of, I guess reconcile is the wrong word, but I'm I guess I'm sort of kind of trying to triangulate uh in terms of not only your studies, but also what you've been teaching.
- 7:25SPEAKER_01: And now, you know, we're talking about being on staff with Berkeley Physics, we're talking about, I think like bioengineering.
- 7:33SPEAKER_01: Now, obviously, like what you're doing with with open minds, AI and robotics.
- 7:39SPEAKER_01: Is there is there a specific through line between all of these different studies?
- 7:48SPEAKER_00: Well, no.
- 7:48SPEAKER_01: I mean, I think like, you know, I I I think this is a common thing.
- 7:52SPEAKER_01: I mean, obviously, like most of the people that I talk to in this world, like, you know, they'll they'll have like a computer science or or an engineering background, but it's not super uncommon that somebody does come from uh uh a different study.
- 8:06SPEAKER_00: Yeah, well, there's certainly in the last few years, there's been a few bioengineers who've had a good run.
- 8:12SPEAKER_00: There's one thing that has always helped me stay sane, which is a quote from Feynman's lectures.
- 8:19SPEAKER_00: And in the introduction, he says, nature doesn't care what you call it.
- 8:24SPEAKER_00: I've been in so many situations, especially in academe, where there are long debates about whether something is truly computer science, or what the definition of computer science is, or whether data science is actually a science, or and I sit in these meetings thinking, oh, you know, this is all really bizarre.
- 8:45SPEAKER_00: These are all just labels.
- 8:47SPEAKER_00: And the fundamental question is: is the problem a good problem?
- 8:52SPEAKER_00: And is the opportunity a good opportunity?
- 8:55SPEAKER_00: And it really doesn't matter what you call it.
- 8:58SPEAKER_00: And that's why, sure, I was like tenured physics professor at Berkeley, who then all of a sudden is a Bangerian professor at Stanford, and then all of a sudden thinking about software and compute architectures for autonomous machines and unstructured environments.
- 9:16SPEAKER_00: But in my mind, uh all of these things are very at the at the core, they are all kind of similar.
- 9:23SPEAKER_00: And so certainly one thread in my career is needing to get hardware to play well with software and to do that very reliably and elegantly.
- 9:36SPEAKER_00: And so whether I was building instrumentation for soft condensed matter physics or trying to think of good ways of doing experiments in the wet lab in a sort of bi-engineering context, a common thread there is really the sort of hardware-software integration.
- 9:50SPEAKER_00: That's obviously really important if you have humanoids walking around San Francisco.
- 9:55SPEAKER_00: That's not a pure software problem.
- 9:57SPEAKER_00: That requires cloud, latency, networking, um local GPU, city planning, sociology.
- 10:07SPEAKER_00: Well, that is well a little bit, and then it involves uh you know avoiding mean dogs that are scared of the humanoid because they've never seen one.
- 10:16SPEAKER_00: So it's just requires um coordinating different kinds of things.
- 10:20SPEAKER_01: Yeah.
- 10:21SPEAKER_01: So I mean, I one of the things I'm curious then about, you know, if you're like during the hiring process, like let's say you're building out a hardware team and you're looking for people to build your humanoid.
- 10:35SPEAKER_01: Is the first batch of candidates you're looking for people who like specifically studied that, you know, people who were uh studying computer science, who were studying hardware?
- 10:45SPEAKER_01: Or do you think that there's value in looking towards maybe a little bit outside of the box candidates?
- 10:52SPEAKER_00: I think for most people that hire humans, it's generally pretty clear pretty quickly if someone is potentially uh interesting.
- 11:06SPEAKER_00: And there's a few factors, and generally they're they're not on a CV.
- 11:13SPEAKER_00: And generally it's all based on how you meet the person.
- 11:18SPEAKER_00: A great example is your cell phone rings, and you're like, oh, I don't know whose number this is.
- 11:24SPEAKER_00: And then the person on the other side says, you know, hey, I'm Sean.
- 11:29SPEAKER_00: Um, I'm 17, I'm in a forest in Germany, and my four-wheeled robot that I built in high school is being evaluated by the US Army.
- 11:41SPEAKER_00: And oh, wait, I'll call you right back.
- 11:44SPEAKER_00: I need to do some PID tuning um for the second set of wheels.
- 11:50SPEAKER_00: I'll call you right back.
- 11:51SPEAKER_00: And then he hangs up, and that's like a guaranteed we're gonna hire you.
- 11:57SPEAKER_00: And then, like a few weeks later, you know, he's now here.
- 12:01SPEAKER_00: And that's an example of like, I don't know what his undergraduate degree is, and he probably doesn't have one.
- 12:10SPEAKER_00: I don't know what languages he speaks or where he was born.
- 12:14SPEAKER_00: None of that is relevant.
- 12:15SPEAKER_00: But the fact that he's calling from a forest in Germany as a 17-year-old, and he's built something that is being evaluated in a field trial is just like an immediate no-brainer.
- 12:28SPEAKER_00: It's like, okay, this is someone who can put things together, build things, and then has the agency to get on an airplane and like make sure his robot uh is doing well while in a wet forest.
- 12:42SPEAKER_00: And so that just like every possible thing you could want is coming together there.
- 12:46SPEAKER_00: So we've had a few examples now of where like people knock on the door and we have like this two-minute conversation, and it's just like completely crystal clear that we're gonna hire them and they're gonna come work for us.
- 12:58SPEAKER_00: And so I don't know, maybe this is like a startup thing, or maybe this is an SF thing or a Stanford thing or something, but we've had all of our critical hires are just completely bizarre stories exactly like that.
- 13:15SPEAKER_01: Well, it's the last let's extrapolate the last part of that because that's particularly interesting to me to say that that's a Stanford thing, because it strikes me that, at least on the face of it, that that might be a somewhat controversial opinion, specifically in the world of Stanford, in the world of like higher ed, in the world of Ivies, that the person doesn't have to have a degree.
- 13:39SPEAKER_00: I I I've met many people who have extremely fancy degrees, and it's not immediately clear how well they'll do as an engineer or academic or program manager.
- 13:52SPEAKER_00: And I've met many people who don't have degrees that I admire incredibly and try every single day to learn from.
- 14:00SPEAKER_00: So in my orbit are people um with high school degrees that are astoundingly smart and astoundingly capable.
- 14:09SPEAKER_00: So I've learned to just have a conversation with someone.
- 14:13SPEAKER_00: I generally never ask, like, hey, where do you go to school?
- 14:15SPEAKER_00: Um, what kind of degrees you have?
- 14:17SPEAKER_00: Um, but I'm I'm much more concerned with what have you done and what motivated you to do that, and how did you solve problems?
- 14:27SPEAKER_00: Did you assemble a team?
- 14:29SPEAKER_00: And so I'm looking for that uh like adventurous, can-do liquid, flowing around problem, relentless energy.
- 14:40SPEAKER_00: And really that has nothing to do with the degree.
- 14:42SPEAKER_00: What degrees do is they sue they serve as like a useful funnel sometimes or pre-filter.
- 14:49SPEAKER_00: Places like MIT, Harvard, Stanford, Beijing, they serve that role of drawing people from all over the world to come meet there.
- 15:01SPEAKER_00: And so that makes it easier for me to meet those people, just you know, when I grab a coffee or I run around campus.
- 15:08SPEAKER_00: But I'm generally much more concerned with having a really simple conversation with that person, understanding um what they've done.
- 15:15SPEAKER_01: I I'm curious, you know, especially um, you know, you you were saying you had you had a great time at Reed, and maybe not all of the um all of the rest of your higher education was was like as good of a time.
- 15:28SPEAKER_01: It sounds like maybe you went into higher ed from uh you know a student standpoint with different expectations um than you maybe ultimately got out of it.
- 15:42SPEAKER_01: Um do do you feel like that's the case?
- 15:45SPEAKER_01: Do you feel like your expectations shifted?
- 15:47SPEAKER_01: Uh do you feel like maybe I don't know if jaded is the right word, but you know, do you do you feel like no not at all jaded.
- 15:57SPEAKER_00: I think let's just talk science for a moment or research or education.
- 16:02SPEAKER_00: When people think of a PhD, they generally many people think that a PhD is this, you know, ancient construct.
- 16:13SPEAKER_00: But of course, PhDs have really only been around since about World War One.
- 16:19SPEAKER_00: So the notion of a PhD is um a very uh is is really young.
- 16:27SPEAKER_00: And somehow uh humanity managed to do whatever it was doing uh for hundreds of generations, and this notion notion of a PhD simply didn't exist.
- 16:40SPEAKER_00: And so then you start to think, oh, you know, why do PhDs in Cambridge and Oxford take three years, but PhDs at MIT, Harvard, and Stanford take six?
- 16:52SPEAKER_00: So are Americans not as smart?
- 16:55SPEAKER_00: No, I'm no, I think that's clearly ridiculous.
- 16:58SPEAKER_00: So even as as you look at something like the this this notion of a PhD, there are incredible regional differences.
- 17:06SPEAKER_00: So I got my PhD from Cambridge, and which is like three years.
- 17:10SPEAKER_00: I've never been discriminated against.
- 17:12SPEAKER_00: Um people have never said, oh Jan, your Cambridge PhD, it only took you three years.
- 17:17SPEAKER_00: Uh it must be like a lightweight miniature version of a real PhD.
- 17:22SPEAKER_00: I never heard that.
- 17:23SPEAKER_00: So as you start to look into how academe and research is organized, there's a lot of things worth asking questions about, like three years versus six years for a PhD.
- 17:36SPEAKER_00: Another question is uh, what is the role of a PhD student and what is the role of a PhD advisor?
- 17:44SPEAKER_00: And one perspective is the role of the PhD advisor is to um optimize learning in the PhD student.
- 17:55SPEAKER_00: And then once they've learned everything they can possibly learn, then immediately the supervisor should say, Hey, you're done learning, you're ready, and I'm gonna kick you out now.
- 18:04SPEAKER_00: So that's one perspective that optimizes uh student learning.
- 18:09SPEAKER_00: And then there's another perspective that says, Oh, you've been here for six and a half years.
- 18:14SPEAKER_00: Um, maybe one more paper would really seal the deal.
- 18:17SPEAKER_00: And then you end up with these incredibly uh long-winded PhDs.
- 18:22SPEAKER_00: And so it's important, I think, for everyone involved to understand what the motivations are of all the different players in something like uh PhD level research.
- 18:32SPEAKER_00: And that means that in general, students, for example, should be a lot more diligent in how they evaluate their options and their timelines and stuff like that.
- 18:40SPEAKER_01: Now that you mentioned it, to to get your your perspective on this, like, you know, getting your PhD from Cambridge and now being at at Stanford, um what is the difference?
- 18:53SPEAKER_01: You know, what is the difference between a three and a six-year degree?
- 18:57SPEAKER_00: Um, I don't know.
- 18:59unknown: Okay.
- 19:00SPEAKER_00: So I mean, uh, aside from the glaringly obvious um that you save three years of your life, I mean, a practical operational thing was that um Cambridge had a up or out perspective.
- 19:14SPEAKER_00: So they don't admit students for PhDs, they admit students for uh like MPhils.
- 19:21SPEAKER_00: And then after your one year MPhil, you do an exam and then you either then go on to do a PhD or they kick you out.
- 19:33SPEAKER_00: And there, uh the fraction of people they kick out at year one is very high.
- 19:39SPEAKER_00: Whereas in the US, you apply to a PhD program, and unless something goes horrifically wrong, you'll typically end up with a PhD at year five and a half, six, uh, six and a half, or whatever the number is.
- 19:54SPEAKER_00: And so they just have a little bit of a different architecture.
- 19:58SPEAKER_01: I'm always curious, you know, going into it, whether you were anticipating staying in academia for as long as you did or or have.
- 20:09SPEAKER_00: That's a fascinating question.
- 20:11SPEAKER_00: And I have found academe amazing in some ways, and the real world amazing in other ways.
- 20:20SPEAKER_00: So my recommendation for anyone is that if there's a way you can combine academe with real world, that is by far the best combination of things.
- 20:34SPEAKER_00: And I think it also benefits the students because when students take a lecture, they kind of want that lecture to be uh relevant and timely and interesting and help them.
- 20:49SPEAKER_00: They don't want to hear a lecture about Maxwell's equations where I did the PowerPoint in 15 years ago.
- 20:58SPEAKER_00: And so I like to think that when faculty um spend time in the real world and then come back, um, that has a benefit for the students and also for research and research questions and so forth.
- 21:10SPEAKER_00: So I've been incredibly appreciative of Stanford's flexibility and also the extent to which they support their faculty doing stuff outside the university, like building computer or software companies, and then coming back and having that feed into their teaching.
- 21:29SPEAKER_00: I think that's uh uh that's a really great way to do things.
- 21:33SPEAKER_01: When you say the real world, is that a little bit tongue-in-cheek, you know, in terms of drawing that delineation?
- 21:39SPEAKER_00: Oh, you're saying that uh the San Francisco tech bubble is not the real world?
- 21:43SPEAKER_01: No, I'm saying I'm saying of saying that that uh academia like is is explicitly not the real world.
- 21:51SPEAKER_00: Well, um some academics pride themselves in isolation.
- 21:58SPEAKER_00: They they actually approach.
- 22:00SPEAKER_00: The concept of an ivory tower.
- 22:04SPEAKER_00: You can say, you know, it's it's quieter or more platonic in its organization, or it gives people the ability to spend years ruminating uh over something that's beautiful or fascinating.
- 22:21SPEAKER_00: And that may well be true, and different people are wired in different ways, but the thing I really appreciate is when I build something, and then a normal person raises their hand and says, Hey Jan, this is not so dumb.
- 22:39SPEAKER_00: Thank you.
- 22:40SPEAKER_00: Because as an academic, I've had so many examples of my esteemed colleagues responding in truly bizarre ways to things that I've done as an academic.
- 22:53SPEAKER_00: A great example would be I wrote a paper on violently perturbed small systems, and I was very proud of this, and it wasn't science, and uh it was all very exciting to me.
- 23:05SPEAKER_00: And then on Christmas Eve, I got an email, and the email said, Oh, Jan, um, you know, I read your recent paper.
- 23:15SPEAKER_00: Um, I could not help but notice that my review wasn't cited in the first paragraph.
- 23:23SPEAKER_00: And then the email went on to explain in like multi-page detail why I really should have cited this review earlier.
- 23:32SPEAKER_00: And I remember thinking in that instant, like, look, it's Christmas Eve.
- 23:38SPEAKER_00: And are you seriously telling me that you're using Christmas Eve to send me a multi-page email about where your review appeared?
- 23:48SPEAKER_00: And for anyone outside of academia, they're probably listening to this and thinking, oh man, this is like this is really bizarre.
- 23:55SPEAKER_00: And I agree, it's truly bizarre.
- 23:57SPEAKER_00: And um, I had a few examples like that where I really struggle to understand how people were prioritizing their time.
- 24:06SPEAKER_00: And so I really like this outside world as well, where you know, people maybe have other priorities.
- 24:13SPEAKER_00: I really like that juxtaposition and that tension.
- 24:16SPEAKER_01: Yeah, it it's it strikes me that Start X is maybe a good place to begin that transition.
- 24:24SPEAKER_01: It's a safe place, right?
- 24:25SPEAKER_01: In in that it is, it is like it's sort of dipping your toe in that in the water to a certain extent.
- 24:30SPEAKER_00: Oh, yeah.
- 24:31SPEAKER_00: Um, well, so Stardex is uh um Stanford-aligned um accelerator or incubator, although they kind of don't like either of those words, as far as I know.
- 24:41SPEAKER_00: But the original intent of Stardex was to facilitate Stanford uh people, students, and faculty to start and grow their companies.
- 24:52SPEAKER_00: And um sure for me, yes, that was one of the first experiences where I started to get dragged more and more um into the uh startup or company side.
- 25:02SPEAKER_00: Another experience for me that was very important is Stanford Biodesign.
- 25:06SPEAKER_00: And they have a program for faculty where they give you a highly condensed version of that class, and that focuses on medtech devices and regulatory and business and needs finding and things like that.
- 25:22SPEAKER_00: And for me, especially that needs finding aspect of Stanford Bidesign was incredibly valuable because as an academic, you end up thinking that um the new equation you just discovered or the paper you wrote has got to be world-changing.
- 25:43SPEAKER_00: And what Stanford BiDesign teaches is a completely different perspective, which is go around in the world, look around, ask lots of questions, find lots of opportunities.
- 25:56SPEAKER_00: Uh, and once you have like 100 or 200 of them, then prioritize them and then only think about the set of technologies that is relevant to addressing that need.
- 26:06SPEAKER_00: So for me, it was a really great way of getting out of this mindset that the thing I'm in love with right now also somehow being um relevant to big problems in the world outside of me.
- 26:19SPEAKER_00: So uh yeah, Stardex, uh Stardex is awesome, uh, but Stanford by design was also in retrospect really important.
- 26:27SPEAKER_01: Yeah, this is a problem that that that I hear all the time from um, you know, specifically when I when I talk to to VCs and and they meet with a lot of you know very smart um folks trying to launch startups, especially from the academic world, is the problem of sort of like trying to backwards engineer a problem, starting with the solution.
- 26:47SPEAKER_01: That's that's like horrible.
- 26:48SPEAKER_01: That's the death.
- 26:49SPEAKER_01: And and and and and and I heard you describe what I I think is sort of the genesis of open mind.
- 26:56SPEAKER_01: And it really, it's like that it's that in that classic infomercial thing of like there's gotta be a better way uh moment.
- 27:04SPEAKER_01: And and it sounds like you were like literally you were opening uh up a humanoid robot and realized that it just couldn't really do anything.
- 27:12SPEAKER_00: Yeah, exactly.
- 27:13SPEAKER_00: Um I basically bought a humanoid and three quadruped dog, quadruped dogs for my house.
- 27:17SPEAKER_00: Uh I have two sons.
- 27:18SPEAKER_00: And I thought it'd be really fun to um just uh uh parachute a humanoid and some quadrupeds into that mix and just see what happens.
- 27:28SPEAKER_00: And I was imagining the experience to be out of the movies, like R2D2, Terminator, The Matrix.
- 27:36SPEAKER_00: And so I was imagining the box would arrive, the box would open, the humanoid would jump out and say, Hey, hey, Yan, I'm your new humanoid.
- 27:45SPEAKER_00: Uh, let me start helping you and learning about you and making your life better.
- 27:49SPEAKER_00: But then when all the boxes arrived and you open them, uh, you know, first you start charging lots of batteries, then you plug in the batteries, and then with a lot of luck, the robot may be able to stand up and then it's standing there and it's not doing anything at all.
- 28:05SPEAKER_00: And that then immediately led to oh, what's the right software architecture here?
- 28:10SPEAKER_00: And uh what kind of chips are in the system?
- 28:13SPEAKER_00: And how do we start um turning this inert piece of plastic and metal into a friend, companion, helper, co-worker?
- 28:22SPEAKER_00: How do we make it speak 50 languages?
- 28:24SPEAKER_00: How do we make it navigate my neighborhood?
- 28:27SPEAKER_00: Um, how do we make it remember uh everything I've told it?
- 28:32SPEAKER_00: And how do we personalize its behavior um for specific uh use cases and scenarios?
- 28:38SPEAKER_00: Yeah, so that that's basically just uh having a bunch of hardware being drop shipped uh to my living room and then realizing it that this was like two years ago, that it basically just didn't do anything at all.
- 28:50SPEAKER_01: Yeah, it it it if that was your expectation, somebody who like has been in academia and research for so long, like I can only imagine what the lay person's expectation is, especially now.
- 29:02SPEAKER_01: And I feel like I don't feel like I know that the robotics industry, specifically the humanoid industry, is doing itself such a disservice and shooting itself in the foot right now with all of these flashy demos and just completely continuing to create these otherworldly expectations as far as what humanoid robots can and can't do at the moment.
- 29:25SPEAKER_00: It is a bunch of tech nerds um trying to build the future, and so there's going to be a lot of friction in all of that.
- 29:33SPEAKER_00: It's an incredibly hard problem, and it's incredibly hard to do safely and reliably.
- 29:39SPEAKER_00: Yeah.
- 29:40SPEAKER_00: So the main thing I'm seeing right now is that the basic technology is coming together quite quickly.
- 29:47SPEAKER_00: But there is, you know, there's like a famous saying in engineering, which is like the last 3% take 80% of the time or 97% of the time.
- 29:58SPEAKER_00: And that's exactly true.
- 30:01SPEAKER_00: So the difference between a humanoid that does a lot of useful things in a lab and then opens the door, walks out onto the street, avoids the Waymo, and then the dog comes running and it's yapping, and then there's moisture on the ground, and the leaves are moving, and then it has to speak Cantonese.
- 30:19SPEAKER_00: The amount of time it takes to try to get all of that to work well outside the lab is astronomically large.
- 30:29SPEAKER_01: So take me through that next step as far as, okay, so I've identified this issue, knowing that again, these are going to be very difficult problems to solve.
- 30:42SPEAKER_01: And again, that this isn't necessarily my background or my my uh expertise, although I do have a science background and I am good at problem solving.
- 30:52SPEAKER_01: What what are those what are those steps?
- 30:54SPEAKER_01: And why is this the thing that really drew you into, as you called it, the real world?
- 31:01SPEAKER_00: Oh, well, I can't speak for everyone, but I would like to think that for every five-year-old on Earth, one of their dreams has got to be to be a brain architect for humanoids.
- 31:13SPEAKER_00: I mean, how much more sci-fi could that possibly be?
- 31:16SPEAKER_00: And it's no longer sci-fi, it's like a real job title now.
- 31:20SPEAKER_00: Yeah.
- 31:20SPEAKER_00: And that means what all of us are experiencing, and sorry for being like all hyperbole and all Elon and stuff like that, but this has got to be one of the most fascinating moments in uh human history ever.
- 31:35SPEAKER_00: There's absolutely no question.
- 31:38SPEAKER_00: And the notion of a machine being able to get up, go to the park, chat with people, make friends, little old ladies get teary-eyed, and then it goes to the cafe, gets you some croissants, comes back.
- 31:53SPEAKER_00: For for me, that is really just like a movie or the future, like coming alive right in front of me.
- 32:01SPEAKER_00: And I can't imagine uh like anyone who would have that opportunity to start doing that wouldn't immediately drop everything in their life and and just go for that.
- 32:13SPEAKER_01: One of the things that that interested me, and one of the reasons why I wanted to talk to you specifically, again, like as I'm sure you're aware, we we talked, we talked to many, many physical AI companies.
- 32:25SPEAKER_01: And I know you were at Automate.
- 32:26SPEAKER_01: I don't know if you were you at GTC as well.
- 32:29SPEAKER_01: Yeah.
- 32:29SPEAKER_01: Yeah.
- 32:30SPEAKER_01: So so like shows like that, right now at this point in time, I'm sure obviously like there's a lot of signal, but there's there's a lot of noise, and it can be difficult to really set yourself apart from all the other physical AI companies, certainly at a show like that, as I'm walking around and trying to like distinguish one from the other.
- 32:48SPEAKER_01: Um and and one of the things that jump that jumped out at me is and correct me if I'm mischaracterizing this, but um, a lot of the physical AI companies right now that I talk to seems like they're they are, as they say in Silicon Valley trying to boil the ocean and trying to solve for like all of the physical, all of the AGI stuff, you know, all of the big physical AI stuff.
- 33:12SPEAKER_01: And it seems to me that you're focused on uh um I don't want to say a few, but select and important problems.
- 33:22SPEAKER_00: Well, uh certainly we like to think so.
- 33:25SPEAKER_00: Um there's a lot of things we do not do.
- 33:28SPEAKER_00: Because otherwise we would go absolutely crazy.
- 33:30SPEAKER_00: One thing we don't do is we don't do anything that is fast.
- 33:35SPEAKER_00: So, for example, if you need like drone terminal guidance to hit tank and you're operating at 500 hertz, we don't go near that.
- 33:44SPEAKER_00: That's like really fast, 500 hertz.
- 33:48SPEAKER_00: And we also don't do anything with hands and arms and fingers, because hands and arms and fingers is an enormous industry on the data side and the hands side and the industrial use cases side, the painting and the grinding and the neurosurgery and the iPhone assembly, and it's computationally very difficult.
- 34:09SPEAKER_00: And of course, I love hands, and I'm fortunate to have colleagues such as Cynthia, who started physical intelligence, who cares about hands.
- 34:21SPEAKER_00: And there are thousands of incredibly smart, incredibly well-funded people who care about arms and hands.
- 34:29SPEAKER_00: And from my perspective, that's a that's an awesome thing because that means we don't have to do it.
- 34:36SPEAKER_00: We can just sit here and focus on other stuff, and then we know without us doing anything at all, that the hands will keep getting better and the arms will keep getting better.
- 34:45SPEAKER_00: And of course, when they get to the point where they're good at unlocking the front door, uh, we'll just start using them more and more.
- 34:51SPEAKER_00: The other thing we don't do is we don't try to reinvent the wheel, for example, when it comes to um understanding voice, ASR.
- 35:02SPEAKER_00: So the ability of our robots to understand 50 languages just reflects the fact that we repurpose um tech other people build and sell, and video's ReVA, 11 Labs, some of the stuff that Google is building.
- 35:16SPEAKER_00: And for us, that's like a plug-in capability.
- 35:19SPEAKER_00: Just the problem of good ASR, like voice recognition, that actually involves fusion of video and audio.
- 35:28SPEAKER_00: Because when humans are speaking, of course, we use the fact that our lips are moving to make better decisions about what is actually signal noise, among other things.
- 35:36SPEAKER_00: So even that is an enormously difficult problem.
- 35:38SPEAKER_00: And we just reuse what other people are doing.
- 35:41SPEAKER_00: And it's our it's the fact that we mix and match and integrate and assemble other people's models in addition to the models that we build.
- 35:51SPEAKER_00: That's what allows us to move very, very quickly and specifically not boil the ocean.
- 35:56SPEAKER_00: We're specifically concerned with robots in unstructured environments that are interacting with people.
- 36:02SPEAKER_00: And it's okay if our robots are kind of slow and kind of clumsy.
- 36:08SPEAKER_00: It's okay if they sometimes need help.
- 36:11SPEAKER_00: In fact, that strengthens the bond of the human with humanoid in front of them if the humanoid comes across as imperfect, if it's kind of wobbly and sometimes slow and sometimes misunderstands.
- 36:25SPEAKER_00: It actually gives a much more compelling user experience than this um terminator-like, optimist-like, um glint of perfection type object in front of you.
- 36:37SPEAKER_00: And so that's how we simplify our engineering through modularity and also relaxing the temporal constraints.
- 36:48SPEAKER_00: We're fine if our machines take a second or 1.5 seconds to come up with a good response, as opposed to trying to do that at like video frame rates or something.
- 37:01SPEAKER_01: Now, when you say response, um, you know, the a lot of the demos that I've seen are are specifically it uh I I know I know the phrase social intelligence comes up a lot of a lot of the the videos I've seen are specifically around these sort of like like conversations.
- 37:16SPEAKER_01: Is that what you mean in terms of response?
- 37:18SPEAKER_01: Is like ask effectively asking the robot a question and having it respond to you, or do you mean executing an action?
- 37:25SPEAKER_00: Generally, those things are connected.
- 37:29SPEAKER_00: If you watch someone on Zoom or Google Meet and you turn off the audio, there's a lot of stuff going on.
- 37:38SPEAKER_00: Maybe the human looks bored, or the eyes are closing, or they're sitting attentive in their seat and looking and tracking and following, or using their hands.
- 37:47SPEAKER_00: So humans use just lots and lots of different ways of communicating interest, intent, enthusiasm, listening.
- 37:56SPEAKER_00: And we certainly expect our humanoids to do the same.
- 37:59SPEAKER_00: If the humanoid is just, you know, duct taped to a pole and just voice in, voice out, that gets incredibly boring incredibly quickly.
- 38:08SPEAKER_00: And it also confuses people.
- 38:11SPEAKER_00: People are used to the person in front of them to tracking them and moving and wiggling and wobbling and lifting their head and moving their eyes and looking at them, but not staring at them.
- 38:21SPEAKER_00: And so when we talk about response, of course we want the voice response to be awesome.
- 38:30SPEAKER_00: But in addition, we expect the robot to move its shoulders slightly and use its hands to accentuate certain points.
- 38:39SPEAKER_00: And if possible, move its head.
- 38:42SPEAKER_00: We expect the directional microphone to be tracking the moving mouth of the person of interest.
- 38:50SPEAKER_00: And so a good engaging response by a humanoid, even if it's largely around conversation, typically involves moving the hands, moving the arms, uh, rotating the torso, adjusting the directional microphone.
- 39:04SPEAKER_00: Um, and you're trying to do that all pretty smoothly to get this good experience.
- 39:10SPEAKER_01: You you mentioned physical intelligence.
- 39:13SPEAKER_01: Um was part of, I suppose, um determining the focus of the company, really surveying the landscape, looking at the problems that again, all these like really smart people were already uh tackling, and then just sort of determining what the effectively the gaps in the market were.
- 39:31SPEAKER_00: No, imagine a five-year-old and you give the five-year-old a bunch of awesome stuff, and you ask him or for you know, go do something fun.
- 39:43SPEAKER_00: And just to emphasize um the incredible value of ignorance.
- 39:49SPEAKER_00: No, seriously.
- 39:50SPEAKER_00: No, I know you are.
- 39:51SPEAKER_01: I I just I think that's that's that's great.
- 39:53SPEAKER_01: That's a great t-shirt if you know ever you ever do uh branding.
- 39:57SPEAKER_00: Yeah, celebration of ignorance.
- 39:59SPEAKER_00: Um so if you ask someone who's been like hardcore robotics for like 25 years and you ask them, enumerate all the hard problems, I guarantee you like one of the first things will be like the hand problem.
- 40:12SPEAKER_01: And I agree that you're you're talking about having sort of like a fresh set of eyes, effectively.
- 40:17SPEAKER_00: Yeah, a totally fresh set of eyes, which is and we can do the same experiment with you.
- 40:22SPEAKER_00: Like if I drop ship you a humanoid into your house and then it stares at you in complete silence.
- 40:30SPEAKER_00: I think naturally the first thing you'll want is you'll want to have a good conversation and ask the humanoid, hey man, what can you actually do?
- 40:36SPEAKER_00: But that involves a conversation.
- 40:38SPEAKER_00: And then you don't want the humanoid to run into your wall or kill your pet.
- 40:43SPEAKER_00: So that involves safety and avoidance and path planning.
- 40:47SPEAKER_00: And then you don't want to have to charge the humanoid every two hours.
- 40:51SPEAKER_00: You want it to have its own self-charging bed or chair, and that again involves navigation.
- 40:57SPEAKER_00: And then if the humanoid does really um dumb, annoying things like have a verbal tick.
- 41:03SPEAKER_00: Of course, you want to tell the humanoid, please do not preface preface every utterance with, oh, Brian is awesome.
- 41:12SPEAKER_00: Well, I don't know.
- 41:13SPEAKER_00: You know, I'm on the side.
- 41:14SPEAKER_00: You of course, maybe maybe you want that, but um, that means, okay, now you need a memory and personalization system for the humanoid, and then you want the humanoid to be safe.
- 41:23SPEAKER_00: And all of this are like immediate needs you perceive as a normal person that has nothing to do with hands.
- 41:32SPEAKER_00: But if you had asked the roboticist, oh, what's the most important thing in robotics?
- 41:36SPEAKER_00: Uh, maybe they would have said like slam or visual inertial navigation for denied environments or whatever the answer might be.
- 41:46SPEAKER_00: But you as a human being who has like kids running around, um, you're going to have a totally different list of priorities.
- 41:52SPEAKER_00: And so that that's really maybe that's it's it's because we started in this completely ignorant and naive manner, just focusing on the immediate experiences my kids and I had.
- 42:05SPEAKER_00: So the fact, for example, that we emphasize also education by humanoids is based on my son Ben.
- 42:12SPEAKER_00: Um, the quadruped dog came up to Ben and Ben said, Hey, can you do square roots?
- 42:16SPEAKER_00: And the dog said, Of course I can do square roots.
- 42:18SPEAKER_00: And so that then led to um what does an educational experience look like of a middle schooler with a uh with a robot?
- 42:27SPEAKER_00: And so all of this is driven just by juxtaposing the technology into a family.
- 42:33SPEAKER_01: Yeah, it what one of the interesting bits that um as as I was looking through the website, and I think this was part of the mission statement, uh, one of the things it says is uh if we are going to live with thinking machines, humans should be able to understand how they think and make decisions, and all of us should be able to trust those decisions.
- 42:50SPEAKER_01: And I don't think this is, I mean, I know this is not a case of roboticists intentionally trying to obfuscate uh why robots do the things they do.
- 43:00SPEAKER_01: It's just maybe maybe it's not something necessarily that's either either not been prioritized or it's just traditionally been very difficult to translate that into sort of uh plain language.
- 43:16SPEAKER_00: You're absolutely right.
- 43:18SPEAKER_00: Um, if you think about a standard robotics task, which is you know, pick something up, don't smoosh the grapefruit, um uh translate it and put it on the table, then there isn't such a big need for an outside observer to inspect the reasoning and make sure that the reasoning is safe, for example.
- 43:43SPEAKER_00: But as you get into robots that are performing much longer, more complex tasks in our living rooms and in schools and in hospitals and so forth, you quickly get into a territory of Needing to guardrail all of that.
- 44:02SPEAKER_00: And if I give you a 100 gigabyte matrix of floats, and I ask you, hey Brian, um go have a look at this matrix and tell me how it will behave in these 100 edge cases.
- 44:19SPEAKER_00: You're gonna look at that and say, Oh, now I'm gonna build this like really complex simulator and andor do lots of research or something.
- 44:29SPEAKER_00: And um our position was uh was super pragmatic.
- 44:34SPEAKER_00: Our position was, look, um, why don't we interconnect all the sensors and all the models with natural language?
- 44:40SPEAKER_00: So the camera says, hey, I just saw Jan, open brackets, he's wearing a blue shirt, close brackets, uh, he looks bored, period.
- 44:48SPEAKER_00: And if that kind of information now flows to an orchestrator, then you as a human being are able to inspect which sensor said what, how was that fused, and how did that input information result in various models making decisions?
- 45:07SPEAKER_00: And so it gives you um, it's almost like you're able to eavesdrop in a conversation with many different participants, and it's that ability to eavesdrop that helps debug and also helps with uh alignment and governance.
- 45:25SPEAKER_00: Typically, if you're dealing with tasks that need to be done very, very quickly, like you know, 500 hertz drone terminal guidance, uh, there's no way you use natural language.
- 45:37SPEAKER_00: You just need to much more directly connect a sensor with the you know servo flight control system.
- 45:45SPEAKER_00: But for other tasks, um, you know, a humanoid uh helping a patient or teaching something, things can be slower.
- 45:53SPEAKER_00: And then uh the benefits of using data uh like an interconnect um system that is based on easier to understand communications um uh becomes more and more valuable.
- 46:08SPEAKER_00: So we're basically trading off speed with intelligibility and ability to guardrail at the level of the data interconnect system.
- 46:20SPEAKER_01: Another distinction that I see is again, whether we're talking about physical intelligence or road AI or field AI or any number of companies, the two big problems again is manipulation, which we talked about, and the other one is the data problem, right?
- 46:34SPEAKER_01: Like collecting enough data to build these models.
- 46:37SPEAKER_01: Um the way that I've seen you characterize the work that you're doing is it's focused around um two primary things.
- 46:44SPEAKER_01: So OM1 and Fabric, and OM1 being effectively an operating system, you characterize fabric as being a decentralized coordination layer.
- 46:53SPEAKER_01: Now, are these distinctly different concepts from these kinds of like models or world models that the other companies are building?
- 47:03SPEAKER_00: I mean, uh so in in terms of uh let's leave the sort of decentralized perspective out of things because that's really just a that's like a technical nuance uh really at the sort of database level more than anything else.
- 47:17SPEAKER_00: But the main thing we care about for an individual robot is that individual robot needs to be safe.
- 47:24SPEAKER_00: We need to understand what it's doing and why it's doing what it's doing, and we need the user experience to be really good because we want the kid or the coworker or the patient to be happy with the experience.
- 47:36SPEAKER_00: So that's all focused on individual machines.
- 47:39SPEAKER_00: When it comes to the data problem, one has to be incredibly nuanced and careful about which problem you're trying to solve.
- 47:48SPEAKER_00: For example, if you're trying to build a robot to full t-shirts, that's a very specific thing that requires very specific data.
- 47:57SPEAKER_00: If you're trying to build a robot to be a great teacher, then that has very different ramifications for software architecture, latency, and training data.
- 48:07SPEAKER_00: For example, if you're building a humanoid to be a great math teacher, uh, we're very fortunate that um a lot of the data relating to that particular skill is already available publicly.
- 48:18SPEAKER_00: You know, there's um hundreds of thousands of hours of math teaching examples available.
- 48:24SPEAKER_00: And um, and we are very fortunate that for some of those skills like um helping patients or being a great teacher, a lot of that information is already available in various forms, but there's a totally different perspective.
- 48:40SPEAKER_00: And so let's now think completely differently.
- 48:46SPEAKER_00: Why do humans do things in the way they do them?
- 48:50SPEAKER_00: And a lot of that simply reflects that we have like two arms and uh most of us have ten fingers.
- 48:58SPEAKER_00: So when we think about like an assembly task, one way to teach uh a robot to do that assembly task is to look at a lot of videos of people performing that assembly task.
- 49:15SPEAKER_00: But a different perspective is well, what's actually governing that assembly task, it's not how people usually do it, it's the physics, right?
- 49:26SPEAKER_00: So another perspective on an assembly task is if you tell the robot what you want done and you don't show it any videos at all, but tell it to, given its understanding of the governing physics and what you're trying to optimize, come up with the optimal path.
- 49:44SPEAKER_00: And I'm very much in the um second camp, uh perhaps reflecting a sort of background as a physics professor.
- 49:53SPEAKER_00: I'm much more comfortable with situations where um I give the system the fundamental um constraints and barriers, and then I tell it to figure out the optimal path.
- 50:05SPEAKER_00: So our perspective there is much more um uh like a CNC programming type task, where I'm not going to show a CNC machine millions of hours of how humans would do it.
- 50:19SPEAKER_00: I'm much more interested in the CNC machine figuring out the optimal way to do it, given its specific capabilities.
- 50:27SPEAKER_00: And so that's why we do not spend an incredible amount of time and money um either collecting or purchasing um videos of how humans are doing things.
- 50:41SPEAKER_00: But that said, there's incredibly good reasons to do that for other um use cases that just happen to not be what we're focusing on.
- 50:52SPEAKER_01: So we'll we'll close on this because we're at time, but uh longtime listeners of the show know that uh one of my ongoing uh obsessions is a strong word, but uh areas of interest is willow garage.
- 51:05SPEAKER_01: Uh so I have to ask you, um, Steve Cousins being on as an advisor, you know, somebody who has been robots, humanoid robots, Silicon Valley robots longer than just about anybody else.
- 51:18SPEAKER_01: I mean, what are give me one or two things that you've learned directly from his involvement?
- 51:24SPEAKER_00: Steve is awesome.
- 51:25SPEAKER_00: And um uh I apologize, uh Steve, if you're listening to this um for all my um uh downloads on uh everything that Ross 2 um is not doing perfectly.
- 51:39SPEAKER_00: Um and I just like to emphasize publicly that I love Ross 2 and Ross 2 is awesome.
- 51:45SPEAKER_00: Um and the best part about Steve is his scars because um he has tried so much and run into brick walls in terms of robotics, and that means we don't have to run into exactly those same brick walls, but we can just go ask Steve, and Steve is like, you know what?
- 52:05SPEAKER_00: I've done that four times, and here's all the different ways this will break.
- 52:11SPEAKER_00: And that just is um is of course uh amazingly useful for us.
- 52:16SPEAKER_01: Well, uh it has been an absolute pleasure, Jan.
- 52:19SPEAKER_01: Thank you so much.
- 52:21SPEAKER_00: Oh, Brian, this is awesome.
- 52:23SPEAKER_00: Um, I've had lots of fun, and uh your questions were fascinating.
- 52:31SPEAKER_01: Thank you so much to Jan and Open Mind.
- 52:33SPEAKER_01: Thanks to you for tuning in.
- 52:34SPEAKER_01: Please like, subscribe, share, review, and check out all of the rest of our work over at automated.fm.
- 52:41SPEAKER_01: Automated's next happy hour and live podcast recording are going to be in the Bay Area on Tuesday, September 22nd, which is very soon.
- 52:49SPEAKER_01: We will be at circuit launch in Mountain View.
- 52:52SPEAKER_01: Thank you to our friends at Silicon Valley Robotics for helping to set that up.
- 52:57SPEAKER_01: More information and a link to sign up in the show notes.
- 53:00SPEAKER_01: Stick around, and we will be back here very soon with another episode of Automated.