Latest / Automated with Brian Heater / Andrew Barry on Why Dexterity Is the Next Breakthrough in Physical AI

Transcript

Andrew Barry on Why Dexterity Is the Next Breakthrough in Physical AI · Automated with Brian Heater

• 0:00
  SPEAKER_01: The spot we're starting at is dexterous robots.
• 0:03
  SPEAKER_01: If you can solve dexterity, there's just a huge number of things you can do.
• 0:08
  SPEAKER_01: I have worked on many, many robots in my career that have been very difficult to commercialize.
• 0:14
  SPEAKER_01: And so when we started the company, we said we're gonna go out and make sure that our goals and our benchmarks are real tasks people are paying money for today.
• 0:24
  SPEAKER_01: We have built a handheld device that allows us to do data capture very inexpensively compared to many other techniques.
• 0:30
  SPEAKER_01: And we have scaled that.
• 0:31
  SPEAKER_00: How cognizant are people of the fact that they're training an automation system that may essentially do that task in the future?
• 0:39
  SPEAKER_01: Yeah, you know, I mean we're very, very upfront with people about it.
• 0:44
  SPEAKER_01: We had taught the robot to pick this baggie up with the left hand and then like shake it.
• 0:49
  SPEAKER_01: It picked the baggie up with the right hand and shook it.
• 0:52
  SPEAKER_01: And we're like, we never taught it to do that.
• 0:55
  SPEAKER_01: And we all just didn't believe it.
• 1:19
  SPEAKER_00: The conversation was recorded uh shortly after Generalist debuted their Gen 1 model.
• 1:25
  SPEAKER_00: We were extremely impressed with the demos and invited co-founder, CTO, and Boston resident Andy Berry to sit down for a chat.
• 1:35
  SPEAKER_00: Uh got a lot of really great insight into physical AI, and I think you will as well.
• 1:40
  SPEAKER_00: If you're enjoying the show, don't forget to like and subscribe.
• 1:43
  SPEAKER_00: Check out the newsletter over at automated.fm.
• 1:46
  SPEAKER_00: And with that, here's Andy Berry of Generalist.
• 1:50
  SPEAKER_00: We know we talk a lot about what's coming up next in automation on the show, but if you really want to do feature motion, you've got to be there in person.
• 1:57
  SPEAKER_00: Automate 2026 is where the world's leading innovators, builders, and remote come together to show you what's possible of robots AI, machine vision, motion control, you name it, all automation under one roof.
• 2:10
  SPEAKER_00: And as part of Automate this year, the Humanoid Robot Forum brings together leaders, engineers, and researchers for a two-day deep dive into the real world development, deployment, and commercialization of humanoid robotics.
• 2:23
  SPEAKER_00: Register for free at automateshow.com to join us in Chicago June 22nd through the 25th.
• 2:30
  SPEAKER_00: We will see you there.
• 2:32
  SPEAKER_00: So this is probably wildly inappropriate.
• 2:34
  SPEAKER_00: So I'm going to start by paraphrasing your co-founder, Pete, uh, in a blog post that went up um I think within the last day or two.
• 2:43
  SPEAKER_00: Uh, but he was pointing out the fact he said we've never referred to our models as either VLAs or world models.
• 2:50
  SPEAKER_00: And I'm wondering why that's important.
• 2:52
  SPEAKER_01: Yeah, that's a great question.
• 2:53
  SPEAKER_01: So, you know, fundamentally we think of our models as something different, right?
• 2:58
  SPEAKER_01: And that's because they are trained from scratch.
• 3:02
  SPEAKER_01: You know, not quite, right?
• 3:04
  SPEAKER_01: 99% of the parameters in the model is from scratch.
• 3:06
  SPEAKER_00: Why why why is there always a 1%?
• 3:09
  SPEAKER_01: Yeah, that's a great question.
• 3:10
  SPEAKER_01: I mean, you know, we do the thing that makes the model work as well as we could possibly make it work.
• 3:14
  SPEAKER_01: Yeah.
• 3:15
  SPEAKER_01: Right.
• 3:15
  SPEAKER_01: And so uh in that case, I I'm not we ha so a while ago, before we had a huge amount of data, we tried getting rid of the 1% and it caused a performance regression.
• 3:30
  SPEAKER_01: Uh and we haven't redone that experiment.
• 3:33
  SPEAKER_01: So I suspect that we could go to 100% at this point.
• 3:37
  SPEAKER_01: Um but you know, we like to be honest.
• 3:40
  SPEAKER_00: Yeah, I mean you you would you would be going 1%.
• 3:42
  SPEAKER_00: That would like be like a purely like branding thing, I suspect, right?
• 3:45
  SPEAKER_00: In order to say that.
• 3:46
  SPEAKER_00: And and it is, and and you know, and in robotics, like you do say 99% because like everything is kind of like 99.99% at this point, right?
• 3:54
  SPEAKER_00: Uh so actually it's funny because th this was something that actually came up uh I I'm assuming this is this way or after our conversation with Russ, but I was talking to him a little bit about uh Pete's comments as far as um starting from scratch and whether or not that was that was necessary.
• 4:10
  SPEAKER_00: I wanted to get his take.
• 4:12
  SPEAKER_00: Um and I'm curious why that is so important because that sounds like a pretty daunting undertaking.
• 4:19
  SPEAKER_01: Yeah, it's not, you know, we didn't set out and say we're gonna do a model from scratch because being from scratch is really important, right?
• 4:27
  SPEAKER_01: The the reason we're doing it is that with such a large data set, you can do it from scratch, right?
• 4:34
  SPEAKER_01: You don't need the crutch of a VLA or some other set of parameters that are are chosen for other reasons, right?
• 4:42
  SPEAKER_01: And so if you have a large enough data set, you can just basically in some ways fill up all of the parameters in the model with exactly the thing you want to do.
• 4:54
  SPEAKER_01: And there's a lot of evidence from the rest of the field, and and we see it too, that that gives you just better performance.
• 5:00
  SPEAKER_00: So so the idea then then is that you can just sort of like what just scoop it up all at once, basically?
• 5:06
  SPEAKER_01: It's it's more like you know, if if you have a certain amount of capacity in your model, right, and you're gonna fill that capacity with as much goodness as you can possibly fill it with.
• 5:16
  SPEAKER_01: Sure, right.
• 5:17
  SPEAKER_01: And so you can say, well, sure, let me fill it with Wikipedia first.
• 5:20
  SPEAKER_01: Um, but you can do that, but what we find is for dexterous robots, Wikipedia is not all that helpful compared to more and more dexterous data.
• 5:32
  SPEAKER_01: What does the friction look like?
• 5:33
  SPEAKER_01: What do the objects look like from the system we use?
• 5:37
  SPEAKER_01: So that's why we do it.
• 5:38
  SPEAKER_00: So I so okay, then I mean fair enough.
• 5:41
  SPEAKER_00: And the what are, insofar as you're like willing and able to talk about it, and I know like I'm sure video is really big for pre-training, it seems like everybody has kind of gone in that direction.
• 5:53
  SPEAKER_00: Um, what are you really using to build these models?
• 5:56
  SPEAKER_01: Yeah, so so we've we've said now, you know, we have built a handheld device that allows us to do data capture very inexpensively compared to many other techniques, right?
• 6:07
  SPEAKER_01: And we have scaled that a huge amount, right?
• 6:09
  SPEAKER_01: So we have half a million hours of that.
• 6:12
  SPEAKER_01: Uh we've built thousands and thousands of these devices and deployed them all over the world.
• 6:18
  SPEAKER_00: So it it's effectively teleop in a way.
• 6:20
  SPEAKER_00: I mean, it's real-world data collection.
• 6:22
  SPEAKER_00: It's real-world data collection is not teleop, right?
• 6:25
  SPEAKER_01: These are these are gloves people wear.
• 6:27
  SPEAKER_01: Um and that gives you a huge number of advantages because if you need to do teleop, well, you have to have a robot.
• 6:33
  SPEAKER_01: Yeah.
• 6:33
  SPEAKER_01: That robot has to work, and you have to have power, and you have to have all these things.
• 6:37
  SPEAKER_01: But it um if you have a person do it, then you just get diversity, right?
• 6:43
  SPEAKER_01: You you're going out into the world as opposed to bringing the world to you.
• 6:47
  SPEAKER_01: And so you you're able to get a data set that's much richer.
• 6:50
  SPEAKER_00: So obviously, like scale and size is a big question.
• 6:54
  SPEAKER_00: I mean, how were you able to compound that much data that quickly?
• 6:59
  SPEAKER_01: Well, two years ago when we started the company, we said this is what we're gonna do.
• 7:04
  SPEAKER_01: Yeah.
• 7:04
  SPEAKER_01: And and, you know, at that time, uh my co-founders Pete and Andy were at DeepMind, right?
• 7:09
  SPEAKER_01: And we had known that this was a way to collect data since like 2018, right?
• 7:15
  SPEAKER_01: Pete made a video in Russ's lab doing this, and then Andy had written a number of papers uh about this kind of technique, but in 2018 we didn't know what to do with the data.
• 7:26
  SPEAKER_01: Right?
• 7:26
  SPEAKER_01: It was like, okay, great, you get a ton of data, what are we gonna do?
• 7:28
  SPEAKER_01: Right.
• 7:29
  SPEAKER_01: Well, you fast forward to when we started the company in 2024, and it's like, okay, well, obviously LLM's work, we can basically use those techniques.
• 7:37
  SPEAKER_01: And Andy and Pete were doing exactly that at DeepMind, but then the scale was was clearly what was required.
• 7:45
  SPEAKER_01: So, you know, at that time we said, okay, we're just gonna go after scale.
• 7:51
  SPEAKER_01: And so we spent two years building an incredible pipeline to get a huge amount of scale, and that's one of the things we've done that has allowed us to build Gen 1.
• 8:00
  SPEAKER_00: Yeah, so what why you know logistically, like pragmatically, what is what is what is the pipeline?
• 8:06
  SPEAKER_00: You know, how are you actually like getting that?
• 8:08
  SPEAKER_00: Like who's doing it?
• 8:09
  SPEAKER_00: Is it is it just a team of three people who are getting bullets of hours of data?
• 8:12
  SPEAKER_01: No, no, no, definitely not.
• 8:13
  SPEAKER_01: Yeah, we we tried everything.
• 8:15
  SPEAKER_01: Yeah, right.
• 8:16
  SPEAKER_01: The first thing you you have to do is you have to make a device that you can mass manufacture, and it has to be inexpensive and it has to be incredibly easy to use, right?
• 8:24
  SPEAKER_01: And then the second thing you do is is you have to find people to use it.
• 8:27
  SPEAKER_01: And and we tried everything.
• 8:29
  SPEAKER_01: We tried mailing it to people and seeing what would happen.
• 8:32
  SPEAKER_01: We tried bringing people into our offices.
• 8:34
  SPEAKER_01: Uh we tried working with a bunch of partners uh who had access to large labor forces.
• 8:41
  SPEAKER_01: We did all of those things.
• 8:43
  SPEAKER_00: When when you were uh just to like to pause for a second, when you were mailing it to people, what were you telling them?
• 8:49
  SPEAKER_00: We tried everything there too.
• 8:50
  SPEAKER_01: Okay.
• 8:51
  SPEAKER_01: And uh we told them things like uh do your laundry.
• 8:56
  SPEAKER_01: We told them things like do whatever you can think of.
• 8:58
  SPEAKER_00: Well, those are the tasks, but but you were pretty upfront, you're like, hey, this is the thing that we're doing.
• 9:02
  SPEAKER_00: We're eventually training robots, and we want you to be a part of that.
• 9:05
  SPEAKER_00: Okay.
• 9:06
  SPEAKER_01: Yeah, yeah.
• 9:06
  SPEAKER_01: Yeah.
• 9:06
  SPEAKER_01: There was no cloak and dagger.
• 9:08
  SPEAKER_01: Yeah.
• 9:08
  SPEAKER_01: It was, yeah, we were very clear about what it was.
• 9:11
  SPEAKER_00: Yeah, and and and people um I i i you know, maybe it's something that people are like hypothetically excited about, but actually when it comes to doing the work that nobody actually really wants to do it.
• 9:23
  SPEAKER_00: People were mostly confused.
• 9:25
  SPEAKER_01: They were like, what you want?
• 9:26
  SPEAKER_01: Yeah.
• 9:27
  SPEAKER_01: And we were like, just put these gloves on and go do your stuff and then uh mail us the data.
• 9:32
  SPEAKER_01: Yeah.
• 9:33
  SPEAKER_01: And, you know, yeah, a bunch of people were like, I don't know what this is, I'm too confused.
• 9:37
  SPEAKER_01: Like, no.
• 9:38
  SPEAKER_01: And then a bunch of people were like, okay, this seems cool.
• 9:40
  SPEAKER_01: And then they dug a little bit and they were like, oh wow, you're training robots with this.
• 9:44
  SPEAKER_01: That seems cool.
• 9:45
  SPEAKER_01: You know, and then pretty quickly we had robots that actually did some things, and so we could show them those videos, and they were like, Oh, okay, I get it.
• 9:51
  SPEAKER_00: That specific group of people, how how did you like find that?
• 9:55
  SPEAKER_00: Because it sounds like like they didn't, you didn't know them directly, they didn't know like specifically what you were doing right at the top.
• 10:00
  SPEAKER_00: Like, yeah, how did you find this large group of people to do laundry?
• 10:04
  SPEAKER_01: Uh do a lot more than laundry.
• 10:07
  SPEAKER_01: Yeah, sure.
• 10:08
  SPEAKER_01: Yeah, yeah.
• 10:08
  SPEAKER_01: I mean, and this is you know, this isn't even just in the United States, right?
• 10:12
  SPEAKER_01: So we have partners that we work with now that that has has really helped us.
• 10:17
  SPEAKER_01: And um that has allowed us to basically train our partners and then they train their labor forces as well.
• 10:23
  SPEAKER_01: Okay.
• 10:24
  SPEAKER_00: So so it's so it's basically like it's people who are are are kind of like being compensated for that job anyway, and then they're wearing the okay.
• 10:31
  SPEAKER_01: So yeah, so we have we have people who are being compensated and then are also wearing the device, and then they're additionally compensated for that.
• 10:38
  SPEAKER_01: And then we also have folks who are are compensated just to wear the device and do whatever they want to do.
• 10:43
  SPEAKER_01: Got it.
• 10:43
  SPEAKER_01: Yeah.
• 10:44
  SPEAKER_00: Yeah, I mean, yeah, you I'm I I I'm sure that you want the professionals, the people who do it for a living, to be the ones who are Well, there was nobody who really did this for a living.
• 10:52
  SPEAKER_01: Well, who do laundry for a living?
• 10:54
  SPEAKER_00: Like, for example, right?
• 10:55
  SPEAKER_01: So we do that.
• 10:56
  SPEAKER_00: We actually do a lot of laundry isn't the only thing, but some of these tasks.
• 11:00
  SPEAKER_01: A lot of the data is not just commercial data.
• 11:04
  SPEAKER_01: It's like in people's houses and stuff, yeah, doing whatever they want to do.
• 11:08
  SPEAKER_01: So it's not necessarily professionals doing a specific job.
• 11:12
  SPEAKER_00: So I think I think we might have jumped ahead a little bit.
• 11:14
  SPEAKER_00: Um, you know, you kind of alluded to what sounds like a solution to a certain extent there.
• 11:19
  SPEAKER_00: Um yeah, how did you how did you go from just sending these things out and nobody wanting them to actually being able to collect that data?
• 11:28
  SPEAKER_01: Uh I would say we tried everything and threw the spaghetti at the wall and saw it stuck.
• 11:33
  SPEAKER_01: Right.
• 11:33
  SPEAKER_01: And so we pretty quickly learned um kind of how to explain it to people that that they could understand it and be excited about it.
• 11:40
  SPEAKER_01: And then we pretty quickly learned how to run the logistics for it.
• 11:44
  SPEAKER_01: Um it's a complicated thing.
• 11:46
  SPEAKER_01: We're moving you know, petabytes of data around, and that's not something you can do on your home internet connections.
• 11:53
  SPEAKER_01: So um there are a huge number of moving pieces there that that's really what we spent the first year of the company figuring out how to do.
• 12:02
  SPEAKER_00: I I'm wondering, i are are people like especially in in those cases when it is something that is professional, like how cognizant of people are people of the fact that they're essentially like training an automation system that may essentially do that task in the future?
• 12:16
  SPEAKER_01: Yeah, you know, I mean we we're very very upfront with people about it.
• 12:21
  SPEAKER_01: And uh some people are really excited about it, you know, especially in jobs that they're not that excited to do, right?
• 12:27
  SPEAKER_01: Yeah, yeah.
• 12:28
  SPEAKER_01: Uh and then there are some people who are less excited about it.
• 12:31
  SPEAKER_01: I thought that would be a much bigger problem than it was.
• 12:34
  SPEAKER_01: You know, for the most part, folks were were pretty happy to to try it out and and collect data.
• 12:41
  SPEAKER_00: So it's really it's it's manipulation is that that's that's the problem right now that you're trying to solve.
• 12:47
  SPEAKER_00: Yeah.
• 12:47
  SPEAKER_01: So, you know, we're a generalist intelligence company for the physical world.
• 12:51
  SPEAKER_01: It's right there in the name.
• 12:52
  SPEAKER_01: It's in the name, right?
• 12:54
  SPEAKER_01: And so that's a really big problem, right?
• 12:56
  SPEAKER_01: So we want to break it down.
• 12:58
  SPEAKER_01: And the the spot we s we're starting at is dexterous robots.
• 13:02
  SPEAKER_01: Right.
• 13:03
  SPEAKER_01: So it's because if you can solve dexterity, there's just a huge number of things you can do.
• 13:09
  SPEAKER_01: Yeah.
• 13:09
  SPEAKER_01: Right?
• 13:10
  SPEAKER_01: We have all these robots in the world.
• 13:12
  SPEAKER_01: Myself, the team has worked on a huge number of robots.
• 13:15
  SPEAKER_01: In a lot of cases, it's been like, well, we have an incredible mobility platform, but, and the but is always we can't touch the world, or we're really limited on our ability to touch the world.
• 13:27
  SPEAKER_01: And so the robot is just much less valuable than it could be.
• 13:30
  SPEAKER_01: And so we think if we can crack dexterity and we're, you know, we're starting to see signs that are very exciting in my mind, uh, then that opens a huge number of things we can do.
• 13:41
  SPEAKER_00: So so you're starting with the most difficult problem first.
• 13:44
  SPEAKER_00: We're starting with the most valuable problem first.
• 13:47
  SPEAKER_00: But but which is like effectively, I mean, at least what I hear, you know, um, but it is, I mean, it is one of, if not the most difficult, certainly in humanoids.
• 13:55
  SPEAKER_01: Uh it's it is very hard.
• 13:57
  SPEAKER_01: Yeah.
• 13:58
  SPEAKER_01: Yeah.
• 13:58
  SPEAKER_01: But it's we're not doing it because it's hard, we're doing it because it's valuable.
• 14:02
  SPEAKER_00: One of the things I like at I often uh talk about uh with people, and and you know, I guess this this is sort of um it may be a philosophical thing as far as like launching a Surf.
• 14:12
  SPEAKER_00: Like those are there are those startups that they've got their go to market, right?
• 14:15
  SPEAKER_00: And and and then um and then certainly like in the physical AI space, and certainly when we're talking about uh like general intelligence and and like really large models, there's just it's just sort of like accepted that it's just gonna kind of like take a while to get there.
• 14:28
  SPEAKER_00: And it's the problem that you're trying to solve, like the first problem that you're going after is a big problem that's gonna take a little while to solve.
• 14:35
  SPEAKER_01: Yeah, yeah.
• 14:36
  SPEAKER_01: No, it's definitely gonna take a little while, but I think what we are seeing is strong parallels to the language space.
• 14:43
  SPEAKER_01: And so, you know, GPT-2 was very cool and not commercially useful in any way, right?
• 14:50
  SPEAKER_01: You couldn't really use it to do anything.
• 14:53
  SPEAKER_01: GPT-3 was like, wow, this is definitely more capable.
• 14:56
  SPEAKER_01: And we started to see glimpses of commercial viability, right?
• 14:59
  SPEAKER_01: There were some cases in summarization, in like certain types of ad copywriting where GPT-3 really was good enough.
• 15:07
  SPEAKER_01: And then GPT-4 comes out, and you're like, okay, actually this opens up a whole bunch of tasks.
• 15:12
  SPEAKER_01: But GPT-4 cannot be a lawyer, right?
• 15:14
  SPEAKER_01: It it still is quite limited.
• 15:16
  SPEAKER_01: GPT-5 comes out, and so we're starting to see the same exact progression here where it's like Gen Zero, incredible ability to do things.
• 15:24
  SPEAKER_01: Gen 1, you say, okay, there's actually starting to be glimpses of places this could be really commercially viable.
• 15:31
  SPEAKER_01: 99% is not 99.99%, right?
• 15:35
  SPEAKER_01: But we're seeing that start to happen.
• 15:37
  SPEAKER_01: And then the second piece is, you know, Gen Zero and Gen 1 are five months apart.
• 15:41
  SPEAKER_01: Yeah.
• 15:41
  SPEAKER_01: And so the rate of progress, you know, is like 66% to 99% in five months.
• 15:49
  SPEAKER_01: You know, we'll see what the line looks like, but yeah.
• 15:53
  SPEAKER_00: Yeah, so so when when you say you're like seeing signs or you're getting those glimpses, you know, I'm curious what that means.
• 15:59
  SPEAKER_00: And and uh I think it'd be helpful to break that down for a general audience as far as the difference between seeing, you know, a demo of um I would like well, I know I keep going back to laundry, but like that well, like closed folding has been the big thing that everybody's been talking about, right?
• 16:17
  SPEAKER_01: We did closed folding partly because everybody asked us.
• 16:19
  SPEAKER_01: They were like, Can you do close folding?
• 16:21
  SPEAKER_01: Like, fine.
• 16:21
  SPEAKER_00: Yeah, but it but it was a thing.
• 16:23
  SPEAKER_00: But but but like what what is the difference between like dozen or so close folding demos that we've seen over the last few months and actually like seeing a sign for commercial viability?
• 16:32
  SPEAKER_00: Yeah, I mean, so let me put it this way.
• 16:37
  SPEAKER_01: I have worked on many, many robots in my career that have been very difficult to commercialize, right?
• 16:42
  SPEAKER_01: And so when we started the company, we said we are not going to just build technology and then somehow they will come, right?
• 16:51
  SPEAKER_01: We said from the very beginning, we're gonna go out and make sure that our goals and our benchmarks are real tasks people are paying money for today, right?
• 17:00
  SPEAKER_01: Because if we can, that means if we can hit our benchmarks, we will have reached commercial liability, right?
• 17:06
  SPEAKER_01: And so we went and did that.
• 17:07
  SPEAKER_01: And so we've been working with partners where we've had we've taken exactly their tasks, sometimes even their specific parts, we brought them in-house, and then we've run our benchmarks on those.
• 17:18
  SPEAKER_01: And we've trained models on them, and we've you know used them to guide our research, right?
• 17:23
  SPEAKER_01: And so when I say glimpses of commercial liability, what I say is some partners are getting very excited about where their tasks are and um you know how close we could be to deploying those.
• 17:38
  SPEAKER_00: And and the and this model built, you know, on dexterity.
• 17:42
  SPEAKER_00: Do you see this being the foundation for sort of like your future general purpose robot model will be built on top of this?
• 17:47
  SPEAKER_00: Yes.
• 17:48
  SPEAKER_01: Yeah.
• 17:48
  SPEAKER_01: Yeah.
• 17:48
  SPEAKER_01: So so this is like a big change in my mind, right?
• 17:51
  SPEAKER_01: So like I worked on Spot from like 2016, you know, through like 2021.
• 17:57
  SPEAKER_01: We did like door opening, right?
• 17:58
  SPEAKER_01: You know, we made some great YouTube videos, had a lot of fun doing it.
• 18:01
  SPEAKER_00: Which which like now it's like uh door opening, but like at the time, opening a door was robot really hard, right?
• 18:07
  SPEAKER_01: And and the way we built, and like I'm very proud of that behavior.
• 18:10
  SPEAKER_01: That behavior works really well, right?
• 18:12
  SPEAKER_01: And to be clear, I wasn't the only one who worked on it.
• 18:14
  SPEAKER_01: You know, a bunch of people in Boston who worked on that.
• 18:17
  SPEAKER_01: But each section of that was a hard-coded controller, you know, and so specific moves, right?
• 18:24
  SPEAKER_01: It's a hard-coded whole controller.
• 18:25
  SPEAKER_01: So in the Rust Hedrake version of the world, right, you have a funnel in state space, and each each controller kind of filled in one of those funnels, right?
• 18:35
  SPEAKER_01: But what we see now in these kinds of models is that you get this kind of improvisational intelligence, is what we're calling it, where the model is just the funnel of the model is just wildly bigger because uh the model sees a case it has never seen before, it's never been in the training, and it will do something very reasonable.
• 18:58
  SPEAKER_01: Yeah.
• 18:58
  SPEAKER_01: Right.
• 18:59
  SPEAKER_01: And so we see that uh improving reliability, we see it improving speed, we see it, and we also just think it's worth pursuing by itself, right?
• 19:08
  SPEAKER_01: And so that type of improvisational intelligence is kind of one of the key things we see.
• 19:14
  SPEAKER_01: And let me tell you about reliability in that case, right?
• 19:17
  SPEAKER_01: So, you know, if you watch a human do a task, like their accuracy is not incredible.
• 19:22
  SPEAKER_01: Like they do not have like, you know, one misspick out of a million, right?
• 19:26
  SPEAKER_01: But their ability to complete the task is really, really high because of course they misspick, they just pick again, right?
• 19:31
  SPEAKER_01: And we see exactly the same kinds of things coming out of these models.
• 19:35
  SPEAKER_01: Of course, not all the time, and but we see glimpses of this where the model will just do something that is exactly what it should be doing, and we know it's not in the training data.
• 19:45
  SPEAKER_00: I was speaking with uh uh the CEO of Rhoda AI, and you know, obviously like they're doing some really cool stuff in space with video.
• 19:54
  SPEAKER_00: Um, and one of the things that he said to me was that, and and this was one of those sort of surprise things in hindsight, was that um uh for a lot of tasks, they found that uh uh uh they were actually more efficient when trained on like stuff like outside of that specific task, like videos that weren't that specific task as far as like get you know learning physics and getting to know the world.
• 20:19
  SPEAKER_01: Yes.
• 20:19
  SPEAKER_01: But we see the exact same thing, right?
• 20:21
  SPEAKER_01: You know, we kind of showed this in the Gen Zero blog post, which is we have a specific set of tasks, and every time we add more data into the model, and this is data completely unrelated to the task, not even in the same country as the task is being operated, the task gets better.
• 20:36
  SPEAKER_01: Everybody's waiting to see how far that will go, but there are no signs of that stopping.
• 20:41
  SPEAKER_00: Yeah, how far that will go as far as like diversity of tasks, but also like settings as well?
• 20:47
  SPEAKER_01: Settings we stopped worrying about.
• 20:49
  SPEAKER_01: Like, you know, so we did a demo at GTC, we had no data at GTC ever.
• 20:53
  SPEAKER_01: We never took any data there.
• 20:55
  SPEAKER_01: Uh we have two offices, so half of our company's in San Mateo, California, half of us are in uh Davis Square in Somerville, and the tasks immediately transfer across offices uh with no additional data.
• 21:07
  SPEAKER_00: So uh uh yeah, I mean I I I I guess I mean like as far as like unstructured environments and lighting and things like that.
• 21:13
  SPEAKER_01: We had somebody come in and we were running a demo for them and they said what happens if you turn the lights off, and we just turn the lights off, and the you can't even tell.
• 21:21
  SPEAKER_01: Okay.
• 21:22
  SPEAKER_00: So okay.
• 21:23
  SPEAKER_00: Because I mean that's that that's like the big that's the big thing.
• 21:25
  SPEAKER_00: But I mean like you know in real world, and maybe it's because these are uh I mean are these are these are these mobile tasks that require actual like movement beyond that or so what we're focused on right now, you know, we certainly are adding mobility, yeah, but we started with the simplest possible problem, right?
• 21:43
  SPEAKER_01: Which is like two arms at a desk.
• 21:45
  SPEAKER_01: Yeah.
• 21:45
  SPEAKER_01: Uh and so the tasks, a lot of the tasks we've shown have have not been mobile based tasks.
• 21:50
  SPEAKER_01: Not because our system can't do it, but because you know we're focused on robotic dexterity and mobility works, you know.
• 22:00
  SPEAKER_00: Mobility works great.
• 22:01
  SPEAKER_00: No, yeah, but but this mobility plus dexterity, is it like that much more complex?
• 22:08
  SPEAKER_00: Mobile manipulation is a whole field for sure.
• 22:10
  SPEAKER_01: Yeah.
• 22:11
  SPEAKER_01: Uh what I will tell you is we have put our robots on wheels and run them around with the models running, and the models do not seem to care very much.
• 22:18
  SPEAKER_01: Yeah.
• 22:19
  SPEAKER_01: We actually we do have one video, I think, posted of uh of a little semi-humanoid running around doing stuff.
• 22:24
  SPEAKER_01: Yeah.
• 22:25
  SPEAKER_00: Yeah, I know I know one of the things that you were there was a video that you seemed like particularly excited about was uh the the cable threading.
• 22:31
  SPEAKER_00: And I can't remember we were talking to somebody, you know, I've heard this from several people, but we were talking to somebody recently, you know, who pointed out that automating on the automotive line that it's always like the cable threading that's always the most difficult thing.
• 22:42
  SPEAKER_00: Why has that been such a difficult challenge to solve?
• 22:46
  SPEAKER_01: That one's really hard to model, right?
• 22:48
  SPEAKER_01: You have like these flexible objects that just they're just tough to model.
• 22:52
  SPEAKER_01: And so when you're trying to write code for them, it's just really hard to do.
• 22:57
  SPEAKER_01: And then when you take this technique, so it's like the material like flexible materials, right?
• 23:03
  SPEAKER_01: We never did any of that on on spot when I was at Boss Dynamics, right?
• 23:07
  SPEAKER_01: It was we just didn't know how to, right?
• 23:09
  SPEAKER_01: And there's a whole bunch of techniques you can try, but boy, just pointing a camera at it and collecting a little bit of data, putting it on top of a foundation model, just works a heck of a lot better than anything else I've ever tried.
• 23:21
  SPEAKER_00: And I mean it it it sounds like you know, again, it's dexterity, but it but it sounds like you're talking about cross-embodiment as well.
• 23:29
  SPEAKER_00: 100%.
• 23:30
  SPEAKER_01: Yeah.
• 23:30
  SPEAKER_01: We have cross-embodiment is, you know, we've shown um elite robots, flexive UR, um a couple different semi-humanoids, uh, that stuff works.
• 23:45
  SPEAKER_00: You know, we you're talking a little bit about the kind of the origin of the project, and you know, it sounded like a lot of it goes way back uh to Pete's research.
• 23:54
  SPEAKER_00: I mean, at what point was it clear for you and the founders that, oh, like we might actually have a a startup in this?
• 24:02
  SPEAKER_01: Yeah, so the the story is that Pete had just had a kid and I had just had a kid.
• 24:09
  SPEAKER_00: Just people tell me this, they're like, oh, I just got married, I just had a kid, which to me like that seems like the worst possible time.
• 24:15
  SPEAKER_01: It's pretty much the worst possible time.
• 24:16
  SPEAKER_01: Yeah, and so I was in a molecular biology lab.
• 24:19
  SPEAKER_01: I I went from Boston Dynamics to a product, yeah.
• 24:23
  SPEAKER_01: And oh my gosh, I love molecular biology.
• 24:25
  SPEAKER_01: I can talk your ear off about molecular biology.
• 24:27
  SPEAKER_01: Let's get to the origin, sorry.
• 24:29
  SPEAKER_01: Uh but anyway, we just had a phone call and he was like, We we gotta scale this thing.
• 24:35
  SPEAKER_00: This thing that I've been working on for a while now, now is the time.
• 24:38
  SPEAKER_01: Yeah, but but they they didn't have the ability there to to just scale the data like we wanted to.
• 24:44
  SPEAKER_00: But so but so was he was he he was he had been working on it in the background or he had been working on?
• 24:50
  SPEAKER_00: He had been working on on like VLAs, right?
• 24:52
  SPEAKER_01: And he was working on Palmy and those kinds of things.
• 24:54
  SPEAKER_01: Yeah, he had just gotten back from leave.
• 24:56
  SPEAKER_01: Um and so, you know, we're just like, maybe we should do our own thing.
• 25:02
  SPEAKER_01: I had a kid a month prior, and it and I was like, well, this will take like six months to raise money for, and like I'll take all my leave, and you know, it'll be really nice.
• 25:12
  SPEAKER_01: And um we raised money very, very quickly uh and then I had to quit my job.
• 25:18
  SPEAKER_00: Yeah, I'm I'm curious.
• 25:19
  SPEAKER_00: You know, I I actually uh I think I think I might have asked Russ the same question because you know, again, in this time when there are all these like physical AI companies coming out, um, when you're pitching yourself to uh to a VC, um obviously a lot of it is like, you know, here are my co-founders and and here's a here's our background, but you know, beyond that, how are you kind of differentiating yourself as a physical AI company?
• 25:43
  SPEAKER_01: You're talking about now or you're talking about two years ago?
• 25:45
  SPEAKER_01: Two years ago, pitch deck, this is how we're different.
• 25:48
  SPEAKER_01: Yeah, two years ago, uh we had no strategy and we s we just said all of the things we could think of that were risks.
• 25:56
  SPEAKER_01: And so we said this is a hugely deep technical.
• 25:59
  SPEAKER_01: Led with all the negatives.
• 26:00
  SPEAKER_01: By accident.
• 26:01
  SPEAKER_01: Yeah.
• 26:02
  SPEAKER_01: Okay.
• 26:02
  SPEAKER_01: I mean, we'd never done a company before.
• 26:05
  SPEAKER_01: You know, who's to say we knew that?
• 26:06
  SPEAKER_01: No, no, I mean probably it worked out.
• 26:08
  SPEAKER_01: So they, you know, I but I I think it I think a lot of it is around the honesty of and and also, you know This is why this is really, really hard.
• 26:17
  SPEAKER_01: Yeah, this is why we thought about each one of these problems, and these are why these are the right problems to work on, right?
• 26:23
  SPEAKER_01: And so why didn't we go work on a mobile base?
• 26:27
  SPEAKER_01: Because we said dexterity is where all the value is, right?
• 26:30
  SPEAKER_01: And why aren't we starting with a humanoid?
• 26:33
  SPEAKER_01: Because again, dexterity is where the value is, and uh if you have dexterity, you can put it on a humanoid, right?
• 26:40
  SPEAKER_01: And so why do we buy other people's arms as opposed to building our arms?
• 26:44
  SPEAKER_01: Because we know from teleop alone that the arms are good enough.
• 26:48
  SPEAKER_01: So that's not where we should focus the company's risk, right?
• 26:51
  SPEAKER_01: All of the risk I don't want to do take any risk at all, except in the thing that really matters, right?
• 26:58
  SPEAKER_01: In dexterity.
• 26:59
  SPEAKER_01: And so that's that's what we said at the beginning, and and I think the investors recognize that.
• 27:04
  SPEAKER_00: So, you know, I guess I guess between you it was an open question of like, are we a robotics company?
• 27:09
  SPEAKER_00: Are we an AI company?
• 27:11
  SPEAKER_01: Yeah, we're we're a general intelligence company.
• 27:14
  SPEAKER_01: Yeah.
• 27:14
  SPEAKER_01: And and I think you know, we're starting with robots.
• 27:16
  SPEAKER_01: I think we will see.
• 27:17
  SPEAKER_00: Well, I guess sorry, let me back up, but you're not, but you're not like, you know, because you you said we're not starting with like building a human.
• 27:22
  SPEAKER_00: So you're not a hardware company in that way.
• 27:24
  SPEAKER_00: I think being a hardware company is really hard.
• 27:26
  SPEAKER_00: Yeah, no, it's yeah, unintended.
• 27:28
  SPEAKER_00: Yeah, absolutely.
• 27:28
  SPEAKER_01: That's yeah, no, yeah.
• 27:30
  SPEAKER_01: And it, you know, we we do build some of our own hardware and we build it in service of the dexterity, right?
• 27:35
  SPEAKER_01: So we build our own grippers, and that that really comes down to we couldn't find the grippers that kind of matched the type of data collection we wanted to do in the right way.
• 27:47
  SPEAKER_01: And so we basically felt forced to build our own grippers.
• 27:50
  SPEAKER_01: Uh and I we have I love our grippers.
• 27:52
  SPEAKER_01: We have incredible grippers.
• 27:53
  SPEAKER_01: And we are also going beyond our grippers, right?
• 27:55
  SPEAKER_01: You know, that's really all about making sure the data set and the inference time matches, right?
• 28:01
  SPEAKER_01: But when you start, what we're starting to see is when you have enough diversity, well, all of a sudden now it's pretty easy to add in new types of hands, different types of grippers, all these kinds of things are starting to happen.
• 28:12
  SPEAKER_01: And and you're building the grippers specifically for data collection?
• 28:16
  SPEAKER_01: So we build both, right?
• 28:18
  SPEAKER_01: We build the data collection device and the robot gripper, uh, so we can make them similar.
• 28:23
  SPEAKER_00: But but I I I I I guess I guess what I'm getting at is say is the the robot grippers themselves, like the those potentially have real-world applications or will.
• 28:31
  SPEAKER_00: Okay, for sure.
• 28:32
  SPEAKER_00: Yeah.
• 28:32
  SPEAKER_01: Yeah.
• 28:33
  SPEAKER_01: Yeah.
• 28:33
  SPEAKER_01: And and we we we see people in some cases, very serious people, wanting to purchase the grippers, and and we've talked to various people about that.
• 28:42
  SPEAKER_00: Right.
• 28:42
  SPEAKER_00: So two years ago you said, right?
• 28:44
  SPEAKER_00: Yeah, about about two years ago.
• 28:45
  SPEAKER_00: A little longer now.
• 28:46
  SPEAKER_00: Uh so you know it sounds like things out of the gate really, really quickly.
• 28:51
  SPEAKER_00: And would would you say that uh I I don't know, I don't know if GTC was quite like coming out of stealth for you, but it was definitely a big like It was our first public demo.
• 29:00
  SPEAKER_00: So yeah, okay.
• 29:00
  SPEAKER_00: So so yeah, so so so two years, I mean, what is that two what does that period look like in the interim?
• 29:07
  SPEAKER_01: Yeah, so the first real year of the company was all about understanding how to do data collection, right?
• 29:14
  SPEAKER_01: So we knew it was all about data collection at scale uh in a really reliable and inexpensive way, so so you can scale.
• 29:23
  SPEAKER_01: Right.
• 29:23
  SPEAKER_01: And so we spent a year basically only focused on data collection.
• 29:28
  SPEAKER_01: Plus, of course, you do the robot side, but at that time, the only reason to do the robot side was to make sure we were doing the data collection correctly, right?
• 29:34
  SPEAKER_01: Because you really didn't want to end up with half a million hours of useless data, right?
• 29:38
  SPEAKER_01: That would be a disaster, right?
• 29:39
  SPEAKER_01: So we built the robots just to prove to ourselves that the data collection we were doing was high fidelity enough that it could transfer.
• 29:47
  SPEAKER_01: And once we saw that transfer, we scaled that data collection all the way to half a million hours and beyond and out.
• 29:54
  SPEAKER_01: So that's what we spent, yeah, the first year of the company.
• 29:56
  SPEAKER_00: And and does does or or will video have any role in pre-training for you?
• 30:01
  SPEAKER_01: Yeah, of course.
• 30:02
  SPEAKER_01: Yeah, yeah, yeah.
• 30:02
  SPEAKER_01: We have this kind of very unique data set around the this UMI style uh data.
• 30:10
  SPEAKER_01: And then of course you can add to that, right?
• 30:13
  SPEAKER_01: If you if you have you know all of YouTube, great.
• 30:17
  SPEAKER_01: You know, you can you can do exactly what like Rota does, but then you can also fine-tune the model on an enormous data set that's very, very similar to what the robot's gonna see.
• 30:27
  SPEAKER_01: You're gonna get really good performance.
• 30:28
  SPEAKER_01: And the same thing comes in with language, right?
• 30:30
  SPEAKER_01: You want to add language to it, great.
• 30:32
  SPEAKER_01: Now you have a very steerable system, and we've done a little bit of that.
• 30:35
  SPEAKER_01: Um, but it's really about the base of the system is kind of the fundamental friction of the world, touching objects, understanding objects.
• 30:45
  SPEAKER_01: Yeah.
• 30:45
  SPEAKER_01: That's kind of the fundamental base of the system, and then you can add all these things on top.
• 30:50
  SPEAKER_01: As opposed to like Wikipedia being the base of the system.
• 30:53
  SPEAKER_01: Well, you can read Wikipedia all you want, you're never gonna be a good skier, right?
• 30:57
  SPEAKER_01: And so that's kind of how we think about it.
• 30:59
  SPEAKER_00: It's it's physics of materials.
• 31:01
  SPEAKER_00: Exactly.
• 31:01
  SPEAKER_01: Yeah, and you know, objects, how they move.
• 31:04
  SPEAKER_00: Yeah, and and you know, and and and the thing that that that I keep talking to people about lately is is this idea of the flywheel.
• 31:12
  SPEAKER_00: Um do you feel like we're at a point where or we're getting close to a point where there's gonna be enough of those signs where it will be legitimately useful, where it can scale to a point where or or do you feel like like I guess have you hit enough scale yourselves that you feel like you have started kind of a data flywheel?
• 31:34
  SPEAKER_01: We have started it and and we are learning a huge amount about it.
• 31:38
  SPEAKER_01: I don't think the data flywheel will look like what people thought it would look like two years ago.
• 31:44
  SPEAKER_01: If you look at where robots are deployed today, they do not have 10 gigabit internet connections to S3 out of those warehouses, right?
• 31:52
  SPEAKER_01: You can barely get internet out of those warehouses.
• 31:54
  SPEAKER_01: And so, uh oh, like that's a problem, right?
• 31:58
  SPEAKER_01: And then the other piece is well, are the customers actually interested in helping you with your data flywheel?
• 32:04
  SPEAKER_01: You know, in general, no.
• 32:06
  SPEAKER_01: Yeah, right?
• 32:06
  SPEAKER_01: Like, what's the value of the thing?
• 32:08
  SPEAKER_01: Are they gonna do the hard work of training your robot for you?
• 32:10
  SPEAKER_01: Yeah.
• 32:10
  SPEAKER_01: Right.
• 32:11
  SPEAKER_01: And so we are seeing uh, you know, we are thinking very hard about how we design the product and so that we are giving the value back to the customer in a way that makes them really excited to help us with our data flywheel.
• 32:26
  SPEAKER_01: Right.
• 32:26
  SPEAKER_01: And so I think we'll have a lot more to say about this.
• 32:29
  SPEAKER_00: So interesting.
• 32:30
  SPEAKER_00: So, so I you know, it'll I I don't know gamifying isn't the right word, but like adding value into the process.
• 32:37
  SPEAKER_00: Yeah, and it's all about making their system better, right?
• 32:39
  SPEAKER_01: That's what the customer cares about.
• 32:41
  SPEAKER_01: Like, I I'm a customer, I want my system to work.
• 32:45
  SPEAKER_01: I want every single time I give you data, I want to see that result in my improvement, right?
• 32:51
  SPEAKER_01: And so if you can connect that feedback loop, well, now you've totally flipped the script because now they're very motivated to give you the data, they're really excited about it, they're getting value out of it, you're gonna get a lot more data, you're gonna be a lot happier, and you're gonna get better quality data.
• 33:06
  SPEAKER_00: One of the one of the things I really like talking to to folks about who are dealing especially with you know these really gigantic models are the the things that surprise them along the way.
• 33:15
  SPEAKER_00: Yeah.
• 33:16
  SPEAKER_00: Um, you know, have have you in the last two years, is there anything I mean it sounds it sounds like some of these signs maybe even are are um and and things that it's picked up that it maybe maybe maybe that it's picked up with more accuracy or more quickly than you anticipated it.
• 33:33
  SPEAKER_01: There's many surprises.
• 33:35
  SPEAKER_01: I'll give you the first one that it was just like a jaw on the form floor moment for the whole company, which is we were doing this test this very early on, you know, things were working, and we had taught the robot to pick this baggie up with the left hand and then like shake it.
• 33:52
  SPEAKER_01: And we were running the task, and on one of the rollouts, it picked the baggie up with the right hand and shook it, and it did the task, and we all just stopped and we were like, we never taught it to do that.
• 34:05
  SPEAKER_01: That and it was one of the first models that had a pre-trained base, right?
• 34:09
  SPEAKER_01: And so, but at that time the data set was small enough that Andy Zhang got sick the next week, and he spent his entire flu watching every single minute of the data collection for this task because he was sure that somebody had done it with the right hand, and he looked at every single minute of the data, nobody did it.
• 34:29
  SPEAKER_01: And so that was kind of the first like oh wow moment where like there is something here that is just very different than what we have done before.
• 34:39
  SPEAKER_01: And then uh yeah, since then the the robot has uh in fact we released in in Gen 1.
• 34:45
  SPEAKER_01: Um, actually, I was holding the camera at this moment, so we taught it right to pick up this washer and put it on a shelf, and then the other hand kind of comes in and picks it up and puts it in a slot.
• 34:56
  SPEAKER_01: And we are trying to get some nice shots of like smacking the robot with a hockey stick, right?
• 35:00
  SPEAKER_01: Which is, of course, I love to do that.
• 35:02
  SPEAKER_01: The classic, you know.
• 35:02
  SPEAKER_01: But we're smacking it away, and the the washer gets all messed up and it's holding it in one hand, and the other hand just comes in in mid-air and re-orients the washer in the first hand, and then it just puts it in the slot.
• 35:12
  SPEAKER_01: And like it's not in the data set.
• 35:14
  SPEAKER_01: Yeah.
• 35:14
  SPEAKER_01: In this case, I can't prove it to you because it, you know, yeah, we've scaled our data past uh past Andy's ability to get sick and and watch literally every minute of it.
• 35:24
  SPEAKER_01: Okay.
• 35:24
  SPEAKER_00: So you're not you're not like intentionally trying to infect Andy in order to No, we we try to avoid that, yes.
• 35:30
  SPEAKER_00: Yeah, it it it another thing that's really fascinating to me too is like at a certain point it it seems like it does.
• 35:35
  SPEAKER_00: I mean, you know, you're you're kind of getting at this, that it does kind of get away from you a little bit, right?
• 35:39
  SPEAKER_00: That you can't like sit there and crawl and read all the the data.
• 35:43
  SPEAKER_00: So what you do, because you don't really have the learnings and you don't quite understand exactly how things are working under the bus.
• 35:50
  SPEAKER_01: Part of the reason it's so exciting uh to work on this, which is that we are training the robot to do things, but that doesn't mean it always does the task successfully.
• 36:00
  SPEAKER_01: Yeah.
• 36:00
  SPEAKER_01: Right.
• 36:00
  SPEAKER_01: So what we've seen is we've seen cases where we trained the robot to always miss first and then pick up the object.
• 36:08
  SPEAKER_01: And there there are even I can point them out on little moments of the video that we're gonna do.
• 36:13
  SPEAKER_00: Wait, to be clear, you're training it to miss first?
• 36:15
  SPEAKER_01: No, no, no, by accident, right?
• 36:16
  SPEAKER_01: It's just if you're not careful with the data set and and you the user miss and then you regress, which humans do all the time, yeah, the robot will learn that.
• 36:27
  SPEAKER_01: Yeah, yeah, yeah.
• 36:28
  SPEAKER_01: And there's no sense of like, you know, it's just it there's nothing that tells it don't do that.
• 36:35
  SPEAKER_01: In a way, it's becoming more human.
• 36:37
  SPEAKER_01: Yeah.
• 36:38
  SPEAKER_01: In in some way, yeah.
• 36:39
  SPEAKER_01: Uh more human-like.
• 36:41
  SPEAKER_01: And and so this is something we're learning a lot about and and learning how to steer, right?
• 36:45
  SPEAKER_01: Because we want to be able to say, actually, don't do that, but do do this, and we don't want to delete the whole data set.
• 36:51
  SPEAKER_01: So you end up uh doing kind of all sorts of things there.
• 36:55
  SPEAKER_01: Um the other pieces you can take these models and and just fix little parts of them by and it's it's not you know, I'm a coder, right?
• 37:06
  SPEAKER_01: It's not coding, it's coding uh programming via data collection.
• 37:11
  SPEAKER_01: So you so in this one case, you know, we're uh in fact it's in one of the the videos we released, we're trying to stuff this hose into the the foam pad.
• 37:19
  SPEAKER_01: And we just didn't when we did the data collection, we just didn't quite get it right.
• 37:23
  SPEAKER_01: And so we really needed the robot to like do one extra action.
• 37:27
  SPEAKER_01: And I spent about two hours just doing that action over and over again and then trained a model overnight, and the next day the robot just did it.
• 37:35
  SPEAKER_01: Right.
• 37:36
  SPEAKER_01: So we're learning kind of how to adjust these little parameters and and fix little moments in it.
• 37:42
  SPEAKER_00: So I did want to talk a little bit about you and bad and back up um because it because it is an interesting progression in in terms of how you got uh uh to to generalists.
• 37:54
  SPEAKER_00: So, you know, going Boston Dynamics to generalists, that makes sense to me, right?
• 37:57
  SPEAKER_00: Yeah, yeah um you were we were working on you were working on spot, you were working on on the ARM, um, but this this period in between, so the broad institute, yeah.
• 38:08
  SPEAKER_00: So it's uh biomedical and genomic research.
• 38:12
  SPEAKER_00: Yes.
• 38:12
  SPEAKER_00: How how and why did you end up there?
• 38:15
  SPEAKER_01: Yeah, I love molecular biology is the answer.
• 38:18
  SPEAKER_01: And so I was at uh Boston Dynamics, and uh my fiance at the time, now wife, uh she is a biologist, right?
• 38:26
  SPEAKER_01: Uh and so you did it for love.
• 38:29
  SPEAKER_01: No.
• 38:30
  SPEAKER_01: I mean, you could say that.
• 38:32
  SPEAKER_01: I could say that, but uh she didn't she wasn't like you should do this.
• 38:36
  SPEAKER_01: Uh, we so the the way it happened is she told me, you know, you can buy a DNA sequencer for a thousand bucks now.
• 38:44
  SPEAKER_01: And this was like 2017, and I was like, no way.
• 38:48
  SPEAKER_01: And then I look it up, sure enough, there's a little thing called a minion, incredible technology.
• 38:53
  SPEAKER_01: Uh it's a nanopore technology, I can tell you all about it, but a really cool company.
• 38:57
  SPEAKER_01: And so I I decided to buy her one for Christmas, right?
• 39:01
  SPEAKER_01: And and so, you know, I just put my order, add to cart, and then they called me and they were like, Who are you?
• 39:08
  SPEAKER_01: And I was like, uh oh.
• 39:09
  SPEAKER_01: Uh and so I I fessed up to my wife what her Christmas present was, and she gave me a note card for like what to say to the sales rep.
• 39:16
  SPEAKER_01: They were like vetting you?
• 39:17
  SPEAKER_01: Yeah.
• 39:17
  SPEAKER_01: Yeah.
• 39:18
  SPEAKER_01: And it turned out later they didn't they had people who thought it was like 23andMe and didn't understand that this is like a piece of lab lab equipment.
• 39:25
  SPEAKER_00: It's an expensive 23andMe.
• 39:27
  SPEAKER_01: Yeah.
• 39:27
  SPEAKER_01: Yeah.
• 39:28
  SPEAKER_01: But you know, I don't know, whatever.
• 39:29
  SPEAKER_01: So so I read my note card and the sales rep like totally didn't buy it.
• 39:34
  SPEAKER_01: And then I was eventually like, okay, look, my wife's a biologist, like, I'm buying it for Christmas.
• 39:39
  SPEAKER_01: And the sales rep was like, oh, that's totally cool.
• 39:41
  SPEAKER_01: Like, sure, we'll ship it to you.
• 39:43
  SPEAKER_01: And then I was like, Do you do gift wrapping?
• 39:44
  SPEAKER_01: And they were like, no.
• 39:46
  SPEAKER_01: So we got this thing and we had an amazing time.
• 39:49
  SPEAKER_01: You know, we did like a little PCR, and and we, you know, we just lived in a little apartment in Cambridge, and we sequenced the bacteria off our tongues, right?
• 39:58
  SPEAKER_01: And I just thought it was the coolest thing.
• 39:59
  SPEAKER_01: It's so romantic.
• 40:01
  SPEAKER_01: Uh the bacteria off our tongues, less so, you know, it smells like Lady in the Tramp.
• 40:06
  SPEAKER_01: Uh we have very normal um mouth flora.
• 40:10
  SPEAKER_01: No, muzzle tub.
• 40:11
  SPEAKER_01: Yeah, thank you.
• 40:13
  SPEAKER_01: And anyway, so I started doing this, and and I just had a great time doing it.
• 40:17
  SPEAKER_01: And then I just get more and more interested in it, and we got to the point where you know, Boston Dynamics was changing as a company, and I said, okay, I'm gonna go do something new.
• 40:26
  SPEAKER_01: They were commercializing.
• 40:27
  SPEAKER_01: Yeah, exactly.
• 40:28
  SPEAKER_01: And they were growing a lot.
• 40:29
  SPEAKER_01: And and at that time, it I was like, okay, you know, I've had an amazing run here, I'm gonna do something new, and I want to do more ML to, you know, especially in that time, they weren't doing a huge amount of ML.
• 40:39
  SPEAKER_01: That has, of course, changed.
• 40:41
  SPEAKER_00: It is interesting to have left a robotics company to do more ML, like at the insight.
• 40:44
  SPEAKER_00: Yeah.
• 40:45
  SPEAKER_00: Yeah, but but I mean at that time that you know we were doing a little bit, but I really want to do like a lot.
• 40:50
  SPEAKER_00: They they were doing, yeah.
• 40:51
  SPEAKER_00: I mean, so they they were really I I remember having conversations like specifically with I guess it would have been like Rob at the time, Mark and Rob at the time, about this kind of like app store idea that they had of like having people program um different systems versus having it be this purely like ML play.
• 41:08
  SPEAKER_01: Yeah, that didn't work out so well.
• 41:09
  SPEAKER_00: Yeah, and then and they acknowledge that.
• 41:11
  SPEAKER_00: They're upfront about it.
• 41:12
  SPEAKER_01: Yeah.
• 41:13
  SPEAKER_01: Uh and so yeah, the the then the next part of the story is I started asking my wife, I said, if you can explain this lab's paper to me, I will do the dishes.
• 41:23
  SPEAKER_01: And so then I just cold applied to a whole bunch of labs, and you know, then I you know knew some people and and um started meeting people and got a job at the broad and had a fantastic time.
• 41:36
  SPEAKER_01: Uh they they even gave me a pipette and a lab coat.
• 41:39
  SPEAKER_01: They taught me some of that stuff, and um learned a ton of ML, you know, did my own transformers kind of all the way from the ground up.
• 41:46
  SPEAKER_00: Really, really interesting field.
• 41:48
  SPEAKER_00: Well, that that was kind of my my my my next question is like what role was ML serving in that world?
• 41:56
  SPEAKER_01: So basically that that lab, uh what they do is gene delivery.
• 42:01
  SPEAKER_01: So uh in in kind of quick terms, they have a protein soccer ball, and inside that ball is whatever DNA you want.
• 42:10
  SPEAKER_01: And so for people who have a problem with their DNA, they need to deliver that to the correct cells.
• 42:16
  SPEAKER_01: So basically those balls have if target targeted delivery.
• 42:21
  SPEAKER_01: Targeted delivery.
• 42:22
  SPEAKER_01: It's effectively velcro, and you gotta get the right brand of velcro on the ball and the right brand of Velcro in the target cell, right?
• 42:28
  SPEAKER_01: And so the question is, how do you pick the Velcro on the ball?
• 42:30
  SPEAKER_01: And of course, instead of Velcro, it's actually what sequence of protein you use.
• 42:34
  SPEAKER_01: Well, that lab had figured out how to do a hundred thousand different options at once.
• 42:38
  SPEAKER_01: And now you have a bunch of runs with 100,000 different options, and you say, well, we should do some pattern recognition.
• 42:43
  SPEAKER_01: And then when you go do pattern recognition, you say, boy, this starts to look like ML.
• 42:47
  SPEAKER_01: And then you say, wait a second, there's all these foundation models in biology.
• 42:52
  SPEAKER_01: Can we use those?
• 42:53
  SPEAKER_01: I don't know.
• 42:54
  SPEAKER_01: And so that's exactly the kinds of things we were doing.
• 42:57
  SPEAKER_00: Yeah, I'm I'm curious that that sort of, you know, because obviously, you know, it's not it's not a straight path.
• 43:02
  SPEAKER_00: Like I said, you know, again, it would make sense to go straight from boss dynamics.
• 43:05
  SPEAKER_00: And I'm I'm curious at taking that some almost kind of like diversion, whether any of those learnings were really useful for what you're doing now.
• 43:12
  SPEAKER_01: Aaron Powell Oh, they're incredibly useful because it's it's all the same, right?
• 43:15
  SPEAKER_01: It's all giant transformer models.
• 43:18
  SPEAKER_01: And so we did that there, and you know, in that context, and the models look the same, just like they do for language.
• 43:24
  SPEAKER_01: Yeah.
• 43:25
  SPEAKER_01: The data is very different, right?
• 43:27
  SPEAKER_01: But um yeah, the models are the same.
• 43:30
  SPEAKER_00: Great.
• 43:30
  SPEAKER_00: Well, uh we think we covered all the grounds.
• 43:33
  SPEAKER_00: I I appreciate your taking the time.
• 43:34
  SPEAKER_00: Yeah.
• 43:34
  SPEAKER_00: Thank you.
• 43:35
  SPEAKER_00: This was fun.
• 43:36
  SPEAKER_00: Thanks to Andy and Jenner List.
• 43:38
  SPEAKER_00: It was a great conversation and a really fascinating insight into the latest physical AI breakthroughs.
• 43:44
  SPEAKER_00: Thanks to you as ever for watching.
• 43:47
  SPEAKER_00: If you've been enjoying the show, please like and subscribe to it and our newsletter of the same name over at automated.fm.
• 43:55
  SPEAKER_00: And we will see you next week for another episode of Automated.