Latest / Automated with Brian Heater / Ken Goldberg: Solving Robotics’ Hardest Problems: Dexterity, Data, and Design

Transcript

Ken Goldberg: Solving Robotics’ Hardest Problems: Dexterity, Data, and Design · Automated with Brian Heater

• 0:00
  SPEAKER_01: The famous creativity test is, you know, think of all the how many uses can you find for an object, right?
• 0:06
  SPEAKER_01: And the example I gave it was how many uses can you find for a guitar pick?
• 0:11
  SPEAKER_01: And it generated all the hundred uses like instantly.
• 0:15
  SPEAKER_01: And then one of them was a miniature sail for a toy boat.
• 0:20
  SPEAKER_01: And when I saw that, I was like, oh, that's amazing.
• 0:24
  SPEAKER_01: Like, that's such a good example.
• 0:26
  SPEAKER_01: I would never have thought of it.
• 0:27
  SPEAKER_01: And I was like, that's a great, that's creative.
• 0:30
  SPEAKER_01: Okay.
• 0:30
  SPEAKER_01: I cannot deny that.
• 0:40
  SPEAKER_00: Hey everyone, welcome back to another episode of Automated.
• 0:43
  SPEAKER_00: I'm Brian Heener, the managing editor of A3, and we have a very engaging interview coming up with Ken Goldberg of UC Berkeley and Ambi Robotics.
• 0:52
  SPEAKER_00: Ken is somebody I've spoken with a number of times over the years, and it's always a good time.
• 0:58
  SPEAKER_00: In fact, we actually uh chatted the other week for the automated newsletter about Morovic's Paradox and why humanoid robots can say do backflips but can't button a shirt.
• 1:10
  SPEAKER_00: I know you're gonna enjoy this one.
• 1:11
  SPEAKER_00: Uh chock full of really interesting information.
• 1:15
  SPEAKER_00: Uh and if you enjoy the show, please like and subscribe, and we will catch you on the other side.
• 1:21
  SPEAKER_00: So, as advised, I I watched the GTC keynote you gave last night.
• 1:27
  SPEAKER_00: We'll have a lot of things to discuss around it, but but the first thing that jumped out at me was that you opened with the video of Rosie the Robot.
• 1:35
  SPEAKER_00: And the reason why I bring that up specifically is because I was looking into this.
• 1:40
  SPEAKER_00: The Jetsons and its initial run ran uh three seasons, 62 through 63.
• 1:49
  SPEAKER_00: Why, like 50 years later, why is that still our primary reference for home robots?
• 1:56
  SPEAKER_01: Great question.
• 1:57
  SPEAKER_01: I don't know.
• 1:58
  SPEAKER_01: You know, it's interesting because when I was watching it, it must have been a few years after that, but it was certainly in regular reruns.
• 2:05
  SPEAKER_01: And it was, given that date, it's really interesting because that was right around the time that, you know, the post-Sputnik period in the U.S.
• 2:13
  SPEAKER_01: was ramping up its whole rocket Apollo program.
• 2:17
  SPEAKER_01: So there was a lot of interest in technology, in particular rockets.
• 2:20
  SPEAKER_01: So that makes sense.
• 2:21
  SPEAKER_01: But then the robots are also part of it, right?
• 2:23
  SPEAKER_01: So all that is part of an era of the 50s where there was a lot of excitement and optimism about new technologies.
• 2:33
  SPEAKER_01: And that rolls into the 60s when it's really like a national craze where everybody's getting into it.
• 2:39
  SPEAKER_01: And there's all kinds of programs that get kids excited about technology.
• 2:42
  SPEAKER_01: All these kids went into engineering, including me from that period.
• 2:47
  SPEAKER_01: But I think there was a real sense that this was a you know national effort, and we wanted to build things, and there was a race on to beat the Soviets.
• 2:56
  SPEAKER_01: There was a lot of excitement.
• 2:58
  SPEAKER_01: Computers were somewhat new, but very exciting at that time.
• 3:01
  SPEAKER_01: Nobody was really talking about AI per se, but robots have been around since the word robot comes from the 20s, as you know.
• 3:08
  SPEAKER_00: So that explains sort of the the you know the potential origins of Rosie and the uh cultural context.
• 3:16
  SPEAKER_00: But I'm wondering if it speaks to what's happened in the intervening years or almost like a lack of innovation or a lack of public-facing innovation on that front, that we're still using that as our reference all these years later.
• 3:30
  SPEAKER_01: I think part of it is that there's been a lot of things about robots.
• 3:34
  SPEAKER_01: Uh, and robots have been in science fiction and all kinds of movies and TV shows, Westworld, all kinds of, and of course, all the 60s shows too, like uh Lost in Space, right?
• 3:45
  SPEAKER_01: There were all kinds of robots there.
• 3:47
  SPEAKER_01: What I think was really interesting about Rosie is that she is, if you remember, she the the robot is kind of always breaking down.
• 3:57
  SPEAKER_01: And I really like that the portrayal, because it's kind of a joke, it's a running joke in the show that the robot doesn't quite work and she kind of malfunctions.
• 4:07
  SPEAKER_01: And that's actually, if you also watch Iron Man, there's a robot in Iron Man that very similar, that's kind of always messing up.
• 4:15
  SPEAKER_01: And it's true in Lost in Space, right?
• 4:17
  SPEAKER_01: There's a robot that kind of doesn't always work.
• 4:20
  SPEAKER_01: So I think that is a kind of very astute insight that the writers had that these robots were a great promise, but they weren't necessarily going to work flawlessly.
• 4:32
  SPEAKER_00: Also good for comedic effect.
• 4:34
  SPEAKER_00: But what's funny is as you're saying that I remember it now, you know, I grew up watching the Jetsons interuns, but I I'd forgotten it because uh it's I and I don't know if this is just like how we you know view like uh pop culture or history, but it's using Rosie as a reference point is like this is what we want to get to.
• 4:55
  SPEAKER_00: This is the ideal, this is what everybody's hoping for in the home.
• 4:59
  SPEAKER_01: Right.
• 5:00
  SPEAKER_01: Well, I think the the dream of the robot butler or maid is is very many people want that.
• 5:09
  SPEAKER_01: And it's you know that that something that will you know tidy up and basically take care of the you around the house.
• 5:18
  SPEAKER_01: Remember in the Jetsons, Rosie also played with Elroy and the kids, right?
• 5:22
  SPEAKER_01: She wasn't just a worker.
• 5:24
  SPEAKER_01: So I think that there's this idea that this could be, you know, sort of a companion as well.
• 5:29
  SPEAKER_01: And I think that is very much still alive.
• 5:32
  SPEAKER_01: I I think especially for people who are getting older, and I can relate to this now increasingly, that is uh many senior citizens are gonna retire, and it's increasing.
• 5:43
  SPEAKER_01: We know this is the demographics.
• 5:46
  SPEAKER_01: And we're, you know, it's very hard to get good help and people who can help you if you're at home and you need a lot of care, health care, if you're older.
• 5:55
  SPEAKER_01: So we've gone through this with parents, right, grandparents, and you can hire somebody, but often it's very expensive.
• 6:02
  SPEAKER_01: And second of all, if they're good, they get hired away.
• 6:05
  SPEAKER_01: So it's hard to keep them.
• 6:07
  SPEAKER_01: And also there's a danger if they're not so good.
• 6:09
  SPEAKER_01: You know, they might get grumpy or they might be, you know, neglectful.
• 6:14
  SPEAKER_01: And I can say my my own feeling when I project ahead, if I'm imagining myself somehow alone, let's say, I don't know, whatever happens um to Tiffany, my wife, but let's say I'm I'm alone and I have to be taken care of, I I would really like to have a robot that rather than a person that I may not trust or may not, you know, may not want to be there with me.
• 6:38
  SPEAKER_01: A robot would be great, especially I always think about it where it could do a lot of certain things that I can't do, but also maybe keep me company in the sense that it would have access to all my my past photos and emails and memories, and it could sort of help me keep track of things.
• 6:57
  SPEAKER_01: And, you know, and now with language models, you know, we could have conversations.
• 7:01
  SPEAKER_01: It would be that would be kind of preferable.
• 7:03
  SPEAKER_00: Maybe the first or second time I visited you at at your labs, you showed me some of the research you were doing with uh with with pick in place with robot arms, and a lot of it was based around food.
• 7:14
  SPEAKER_00: You know, like you had like all these cans of uh groceries and everything set up around the office.
• 7:19
  SPEAKER_00: Um and and it strikes me that that's in part because of how much more difficult it is in terms of the setting, but also in terms of like the things that they're actually picking up.
• 7:31
  SPEAKER_01: Aaron Ross Powell People think that picking things up is trivial, right?
• 7:34
  SPEAKER_01: It's it's easy for us.
• 7:35
  SPEAKER_01: And as Hans Morebeck pointed out, 35 years ago, what's what's what's uh easy for robots is hard for us, and vice versa, like grasping are still hard for robots.
• 7:47
  SPEAKER_01: We're getting closer, but being able to do it really reliably for all kinds of objects is still a challenge.
• 7:53
  SPEAKER_01: And so the home where you have many things that are unstructured, right?
• 7:59
  SPEAKER_01: You if they're not in packages, then you have all kinds of things that have to be put together.
• 8:04
  SPEAKER_01: And clothing is a good example, right?
• 8:06
  SPEAKER_01: You know, you have clothes all over the floor with laundry, right?
• 8:10
  SPEAKER_01: You have you when you're cleaning up after a meal, you have all kinds of half-open packages and and things you have to pick up.
• 8:18
  SPEAKER_01: For example, look at just like the you you have some crackers and they're in a package that's usually transparent, and you open that up and you sort of have what's left is a transparent wrapper.
• 8:29
  SPEAKER_01: That's very hard for a robot to pick up, partly because it's very hard to perceive.
• 8:33
  SPEAKER_01: If it's transparent, the typical methods of depth sensors or cameras can often overlook the object.
• 8:40
  SPEAKER_01: Humans are very good at compensating, and we can see it instantly, right?
• 8:44
  SPEAKER_01: It's trivially obvious to us, but that gives robots uh a sense of hesitation.
• 8:50
  SPEAKER_01: Also that we see that that thing is an empty wrapper, so it's okay to sort of squeeze it.
• 8:53
  SPEAKER_01: But a robot will look at that as sort of an odd shape and it's not sure what to do with it.
• 8:58
  SPEAKER_01: So a lot of the standard approaches will fail on just an empty wrapper.
• 9:03
  SPEAKER_01: Another example I like to use is eyeglasses, right?
• 9:06
  SPEAKER_01: We know how to pick them up and it's no problem.
• 9:09
  SPEAKER_01: But a robot is it doesn't, you know, it's very complicated because they're delicate.
• 9:13
  SPEAKER_01: You don't want to put your grippers on the on the glass part, right?
• 9:17
  SPEAKER_01: So there's a lot of nuances to grasping.
• 9:19
  SPEAKER_01: That's why we study those.
• 9:21
  SPEAKER_01: In fact, at the lab, we're often very interested in these adversarial objects, objects that are very difficult to grasp, because we want to study those, those in particular.
• 9:31
  SPEAKER_00: Aaron Powell I wonder what people's thresholds for potential mistakes is.
• 9:36
  SPEAKER_00: Obviously, it comes down to what it's accidentally picking up.
• 9:39
  SPEAKER_00: If you know it damages a soup cam, no big deal.
• 9:43
  SPEAKER_00: But if it breaks your glasses, that's another question entirely.
• 9:46
  SPEAKER_00: But as we're getting to this point, you know, and I know this is still a ways off, but as we're getting to this point where we're talking about like humanoids in the home, for example, um, does it have to be right 100% of the time in order for people to actually adopt it?
• 10:00
  SPEAKER_01: Good question.
• 10:01
  SPEAKER_01: Okay, great question.
• 10:02
  SPEAKER_01: Let's think about that.
• 10:03
  SPEAKER_01: So if we separate out the the factory, and it really depends on where what you're talking about, because lately I've been thinking about the evolution of robots moving, going from moving things, which is like warehouses and logistics, to making things, which is all the manufacturing, to maintaining things, which is the third level, which is what we're talking about coming back to the home care.
• 10:28
  SPEAKER_01: Let's start with the moving things.
• 10:29
  SPEAKER_01: If you're in a warehouse and you drop something, it's no big deal.
• 10:32
  SPEAKER_01: And that happens all the time, right?
• 10:34
  SPEAKER_01: So it's just normal that you at the end of the day you kind of look for all the boxes that got dropped around, right, or anywhere and they get picked up often by humans and then dealt with.
• 10:43
  SPEAKER_01: So that's not an issue.
• 10:44
  SPEAKER_01: I mean, it's really very minor fact.
• 10:47
  SPEAKER_01: And so you can have several percent of failure on dry on on grasping in a factory or in a warehouse, and that's not a problem.
• 10:54
  SPEAKER_01: Factory is different because factory, especially if you're producing something valuable, you you know, that could really be a loss, like a circuit board or a a wafer, right?
• 11:03
  SPEAKER_01: A silicon wafer could be, you know, possibly tens of thousands of dollars if you drop it.
• 11:08
  SPEAKER_01: And now let's look at the high and in the service.
• 11:13
  SPEAKER_01: I mean, main when I say maintain, I like it because it's an alliteration, but also because it really is about a different level of interaction where you're maintaining things, you're actually taking care of.
• 11:23
  SPEAKER_01: This actually does apply for many environments where, you know, like service where you're fixing cars or or are debugging circuit boards or even gardening is a category of maintenance.
• 11:37
  SPEAKER_01: And home care is the big one, right?
• 11:39
  SPEAKER_01: That's the one that you know is going to be increasingly huge and and just is a huge uh giant problem hanging over society.
• 11:48
  SPEAKER_01: Now, there you're probably okay if you're just cleaning up the the the dining room or the living room, right?
• 11:54
  SPEAKER_01: You're just picking things up and you drop it, no big deal.
• 11:56
  SPEAKER_01: You just reach down and pick it up again, right?
• 11:58
  SPEAKER_01: We do that all the time.
• 11:59
  SPEAKER_01: But it matters if you know it's a tray of hot soup, right?
• 12:04
  SPEAKER_01: And you're bringing it over to me and I and that soup just tilts down and lands all over me.
• 12:09
  SPEAKER_01: Uh that's that's that's bad, right?
• 12:12
  SPEAKER_01: So, or similarly, some some valuable, there's all kinds of examples, right?
• 12:18
  SPEAKER_01: Where you can let's say you're it's a thermometer and you're putting a thermometer in my mouth and you break that, and then all of a sudden a bunch of people, you know, mercury.
• 12:26
  SPEAKER_01: I don't know.
• 12:26
  SPEAKER_00: Mercury, you know, I would I would add to that, yeah.
• 12:29
  SPEAKER_00: I would add to that because you know, you brought up, I think aged tech is what people are calling it now.
• 12:34
  SPEAKER_00: Um, you know, aged tech?
• 12:36
  SPEAKER_00: Age tech, what they what they until recently were calling elder care.
• 12:40
  SPEAKER_00: This has been a big thing in Japan for decades now because their population is aging even more uh rapidly than ours is.
• 12:47
  SPEAKER_00: But you're also talking about bringing a robot into the home to care for the most vulnerable individuals.
• 12:53
  SPEAKER_01: Yes, absolutely.
• 12:55
  SPEAKER_01: And that's right.
• 12:56
  SPEAKER_01: So is either probably you want kid maybe take care of your kids.
• 12:59
  SPEAKER_01: That would be another one, right?
• 13:00
  SPEAKER_01: They're pretty vulnerable.
• 13:02
  SPEAKER_01: I can imagine.
• 13:04
  SPEAKER_01: Yeah.
• 13:04
  SPEAKER_01: Well, imagine like feeding the baby, right?
• 13:07
  SPEAKER_01: You don't really want a robot doing that.
• 13:08
  SPEAKER_01: The errors can be very, very consequential.
• 13:11
  SPEAKER_01: And so, but you're right, vulnerable people.
• 13:13
  SPEAKER_01: And that's someone maybe, you know, as you get older, someone who doesn't, you know, has Alzheimer's or can't see anymore, who is, you know, really vulnerable, can't even communicate that something's wrong.
• 13:27
  SPEAKER_01: And you know, you have the robot, imagine you know, someone's at home alone with the robot, the robot is doing something malfunctioning, but it's the person can't even tell.
• 13:35
  SPEAKER_01: I mean, that's a nightmare, right?
• 13:37
  SPEAKER_01: And you know, it's it's it's hurting the human in some way, and they were the human is sort of so advanced um in age that they can't really even speak up or stop it.
• 13:48
  SPEAKER_01: So we really want to be thinking about that.
• 13:50
  SPEAKER_01: There is an element of vulnerability, and safety is such a huge factor in robotics.
• 13:58
  SPEAKER_01: People don't really uh talk about it, but in industry, you know, it's a huge OSHA in the United States, has all these rules about if you want to put a robot in place, if it's fast, and that means it's got big motors, so it's gonna be dangerous.
• 14:13
  SPEAKER_01: And so therefore it has to have isolation, it has to have light curtains around it and bars and things, so you couldn't get hit by accident.
• 14:20
  SPEAKER_01: But that adds a huge layer of complexity and cost to setting up a robot.
• 14:25
  SPEAKER_01: Now, the hope is that these new generation of collaborative robots are much safer, right?
• 14:30
  SPEAKER_01: They but they move much slower, so they're not, you know, in many cases, they're they're not uh as appealing to industry.
• 14:39
  SPEAKER_00: I don't know if you're somebody who's comfortable forecasting.
• 14:41
  SPEAKER_00: I find it's about 50-50 in the robotics world, but GTC was a huge part of this, especially with OneX being there.
• 14:48
  SPEAKER_00: But we're very seriously now at this place where people are talking about humanoids in the home.
• 14:55
  SPEAKER_00: But I also suspect that as kind of a robot evangelist, it's your role to temper expectations in terms of what is and isn't uh a reality and what is and isn't doable right now.
• 15:07
  SPEAKER_00: Like broadly speaking, do you have a concept of how long it might take to get there?
• 15:14
  SPEAKER_01: Okay.
• 15:15
  SPEAKER_01: So first of all, I want to be careful because I'm I'm a robot enthusiast, but I wouldn't say I'm an evangelist, right?
• 15:21
  SPEAKER_01: I'm very nervous about that particular word because I actually want I feel like that's a big danger right now.
• 15:26
  SPEAKER_01: I think that humanoids already that look very human.
• 15:30
  SPEAKER_01: We've actually had those since Disney and in the 50s, but we've also now have robots that can move, that can walk, and even sit, for example, and do backflips, right, extremely well.
• 15:43
  SPEAKER_01: And that is an enormous amount of progress in the last 10 years, right, with between dogs and now humanoids.
• 15:50
  SPEAKER_01: And a lot of this is because of the ability to take simulation, learn in sim, and then transfer that to real.
• 15:58
  SPEAKER_01: So sim to real works very nicely for two categories.
• 16:01
  SPEAKER_01: One is for flying things like drones, you can simulate and then you can learn in simulation, and also works really well for locomotion for legs.
• 16:11
  SPEAKER_01: And so even over rough terrain, you can simulate and then you can do amazing maneuvers, walking over upstairs and um and uphills, backflips.
• 16:22
  SPEAKER_01: That all seems to be uh very well matched to simulation.
• 16:26
  SPEAKER_01: What simulation doesn't help you with, at least so far, is manipulation.
• 16:31
  SPEAKER_01: It's very hard to manipulate in sim.
• 16:33
  SPEAKER_01: And that's we can go down that rabbit hole, Brian, but it's very a whole bunch of reasons why that is still an open challenge in the field is how to how to model and simulate the complex mechanics and interactions that happen in um when you do something even as simple as just like rotating this pencil, right?
• 16:53
  SPEAKER_01: Pen.
• 16:54
  SPEAKER_01: So there's where I think we have a big gap.
• 16:57
  SPEAKER_01: And I'll be happy to talk more if you want.
• 17:00
  SPEAKER_01: I can give you my reasons for that, but there's there's a a story I can describe where, you know, it may be we've made progress, and but the progress, if you look at comparing it to large language models, is minuscule.
• 17:18
  SPEAKER_01: Language models have a huge, vast collection of data.
• 17:21
  SPEAKER_01: Do you want me to explain this, um the story?
• 17:24
  SPEAKER_00: It was something that I wanted to get at because this is something that comes up in your GTC talk, and clearly it's something that you have to talk about a lot.
• 17:33
  SPEAKER_00: But why you talk a little bit about the move from two dimensions to three dimensions and why that's so difficult.
• 17:41
  SPEAKER_01: Yes.
• 17:42
  SPEAKER_01: Okay, well let's start there.
• 17:43
  SPEAKER_01: So we let I want to lay this out in that the argument is that data solved vision, data solved language.
• 17:52
  SPEAKER_01: Therefore, data will solve robotics.
• 17:55
  SPEAKER_00: And in terms of you know, these large these large models that we have, like Dexnet is a good example, um, huge databases of images, huge databases of text uh of text, and that's being crawled to create this output.
• 18:11
  SPEAKER_01: Right.
• 18:11
  SPEAKER_01: Although let's bracket off Dexnet for a second, because that's actually a special case.
• 18:15
  SPEAKER_01: But let's say large language solved vision in the sense that you know that was the big breakthrough in 2012 with uh ImageDET.
• 18:24
  SPEAKER_01: And it was by mining a vast collection of data, of photos, images with labels, and thanks to Feifei Lee for that, that Jeff Hinton and his colleagues uh were able to train a convolutional neural network to be able to generalize, right?
• 18:41
  SPEAKER_01: Had a critical mass of data, and then we could start generalizing to new images and find objects in them with high reliability.
• 18:48
  SPEAKER_01: Right?
• 18:48
  SPEAKER_01: That was a big breakthrough.
• 18:50
  SPEAKER_01: People say solved vision, but to some degree it's made huge, massive progress in vision.
• 18:55
  SPEAKER_01: And that's actually fueled a lot of the excitement about AI in general.
• 18:58
  SPEAKER_01: What the the the next step, which happens surprisingly a few years later, but it's when language, there's a breakthrough, and it's largely due to Google's innovation of the uh the the transformer architecture, which extends deep learning and adds a temporal sequence to it, and that solves sequences of words, strings of words, to the point where it can do translation, but also generation.
• 19:25
  SPEAKER_01: So you can ask a question, and this these large systems, ChatGPT being the poster child, does you know, starts answering in amazing ways.
• 19:34
  SPEAKER_01: I mean, surprising to everyone.
• 19:36
  SPEAKER_01: And, you know, essentially, to some extent, has solved has has passed the Turing test, right?
• 19:42
  SPEAKER_01: And we all know this.
• 19:43
  SPEAKER_01: We've we've played with this, and it's a it's astounding, and it's getting better and better.
• 19:48
  SPEAKER_01: And it's it's it's amazingly good on so many fronts in just being able to synthesize vast amounts of information and then come up with really interesting answers.
• 19:56
  SPEAKER_01: And by the way, it's definitely not just regurgitating.
• 19:59
  SPEAKER_01: I think they originally people thought of them as sort of stochastic parrots, but they're definitely beyond that now.
• 20:04
  SPEAKER_01: These are these systems are are coming up with original ideas.
• 20:08
  SPEAKER_01: And if they're guided properly with the right prompts, they're able to do really amazing things.
• 20:13
  SPEAKER_01: So data solved language, right?
• 20:16
  SPEAKER_01: So then the question is well, all this progress, it seems obvious that robot robotics is next.
• 20:22
  SPEAKER_01: Now, I want to say to your point about the three dimensions, you can think of language as a one-dimensional signal.
• 20:31
  SPEAKER_01: It's just a sequence of words.
• 20:33
  SPEAKER_01: Now, it's I'm oversimplifying because it's it's complex, but it's also discrete.
• 20:37
  SPEAKER_01: You have these discrete sets of words.
• 20:39
  SPEAKER_01: Yeah.
• 20:40
  SPEAKER_01: Right, right, right.
• 20:41
  SPEAKER_01: Yeah, but vision is two-dimensional.
• 20:44
  SPEAKER_01: You have two-dimensional images that you're you're analyzing, right?
• 20:49
  SPEAKER_01: So that's those are extremely complex and high-dimensional, I mean two-dimensional, but still there's a lot of uh a huge state space of potential images out there.
• 20:59
  SPEAKER_01: But when you get into robotics, you actually have m even more complications because you have images and essentially stream of images, so you have video essentially, and what you're trying to generate is not just the next word, but actually the a command for the robot, which may be a whole bunch of joint motions, that is higher dimensional, even higher dimensional.
• 21:20
  SPEAKER_01: And the robots move in in you think of three-dimensional space, but actually, if you also include rotations, that's six dimensions.
• 21:29
  SPEAKER_01: And then you have multiple arms and fingers and legs, right?
• 21:34
  SPEAKER_01: There's a lot of complexity that's going on there.
• 21:37
  SPEAKER_01: So if you want to transform an image into emotion, there's you know it's it's far more tricky than than transforming a sentence into the next word.
• 21:48
  SPEAKER_01: So my point is that that's that's one of the big problems that you actually have a much more complex challenge to start with.
• 21:56
  SPEAKER_01: But the bigger problem is you don't have data.
• 21:58
  SPEAKER_01: And in the first Two examples, vision, you have data.
• 22:03
  SPEAKER_01: Uh, lots and lots of images have been collected over the internet, and so there's a vast repository.
• 22:08
  SPEAKER_01: And same for language.
• 22:10
  SPEAKER_01: If you all the data that's all the words that have been written and scanned, right, is vast.
• 22:17
  SPEAKER_01: Now, recently we did a little study, and it was inspired by a guy named Michael Black, who's at Physical Intelligence, the new robot company.
• 22:25
  SPEAKER_01: He made an analogy.
• 22:26
  SPEAKER_01: He said, Well, let's think of everything in terms of hours, hours of time that you put in in terms of that's how we'll measure this data size.
• 22:33
  SPEAKER_01: So he said, you know, with a robot, how many hours have we teleoperated the robot to get it to do things?
• 22:41
  SPEAKER_01: And that we can just measure how many hours that was, right?
• 22:44
  SPEAKER_01: And he said that, you know, this um effort that had been going on at Google and a number of universities, which I was a part of, Berkeley was was a participant, had collected some, you know, 3,000 hours of data.
• 22:59
  SPEAKER_01: And then he announced that physical intelligence, his company, completed 10,000 hours of data collection.
• 23:06
  SPEAKER_01: And um they do that with many people working different shifts, teleoperating the robot.
• 23:12
  SPEAKER_01: But 10,000 hours, if you do the math, that's one year.
• 23:16
  SPEAKER_01: Okay, but they did it in less than a year, but they collected a year's worth of data.
• 23:20
  SPEAKER_01: Now, compare that to the large language models.
• 23:24
  SPEAKER_01: And he said, okay, well, we'll do that by how long would it take you to read a book, right?
• 23:29
  SPEAKER_01: That would be maybe and two hours.
• 23:32
  SPEAKER_01: And so that amount of data, right, those amount of words.
• 23:35
  SPEAKER_01: So he, if you run that all the way out to like Quinn, the largest language model out there, large VL VLM, vision language model, well, that turns out to be about 1.2 million hours of data.
• 23:50
  SPEAKER_01: Um, or sorry, 1.2 billion hours of data.
• 23:53
  SPEAKER_01: That is 120,000 years of data.
• 23:58
  SPEAKER_01: That's in other words, if you just compared how long it would take a person to sit down and read all the text that's been used to make those um those systems.
• 24:07
  SPEAKER_01: So now you think, okay, well, well, the robot system so far we've got one year.
• 24:12
  SPEAKER_01: But uh to get equivalent to what these large language models are are are using, that's well, it might take a hundred thousand years.
• 24:22
  SPEAKER_01: That's what I call the robot data gap.
• 24:25
  SPEAKER_01: Okay, it's enormous when you really look at it.
• 24:28
  SPEAKER_01: So, and it's it's just that we we don't have it's not that we can mine that the same way that uh large language models can just pull more data off the internet.
• 24:36
  SPEAKER_01: It doesn't exist.
• 24:37
  SPEAKER_01: We have to create it.
• 24:39
  SPEAKER_01: And this is the big challenge that robots are facing, right?
• 24:42
  SPEAKER_01: So that's why I know we will get to humanoids.
• 24:45
  SPEAKER_01: I will I am certain it's gonna happen.
• 24:48
  SPEAKER_01: But the question is when.
• 24:50
  SPEAKER_01: And if if anyone who thinks it's gonna happen in the next year or even two years, I is is is extremely optimistic.
• 25:00
  SPEAKER_01: That way, I am I will be very willing to bet we're not gonna see that in in in two years.
• 25:07
  SPEAKER_01: And I'm not sure we'll see it in five.
• 25:09
  SPEAKER_01: I'm not even sure we'll see it in ten.
• 25:12
  SPEAKER_01: It's it's it's going to take time.
• 25:14
  SPEAKER_01: We will get there.
• 25:15
  SPEAKER_01: I'm very confident of that.
• 25:16
  SPEAKER_01: I'm not doubting that we will get there.
• 25:20
  SPEAKER_01: That let me be clear on that.
• 25:21
  SPEAKER_01: I'm I'm not uh sort of a uh you know, a sort of um Luddite who believes this will never happen.
• 25:28
  SPEAKER_01: It's we're gonna get there, but it's it's not around the corner.
• 25:32
  SPEAKER_00: Aaron Powell It strikes me as you're describing the these you know troves of information that in a certain sense the creation of the databases hasn't changed very much since you know Faye Feeling started doing it with images.
• 25:47
  SPEAKER_00: And it it's very much a a brute forced approach to learning, it seems.
• 25:52
  SPEAKER_00: Um you know, I I suspect that the the evolution of robots and the evolution of AI is also going to involve um new ways of training and new ways of learning.
• 26:03
  SPEAKER_01: Yeah, that's right.
• 26:04
  SPEAKER_01: And you're right.
• 26:05
  SPEAKER_01: So the what I just described is kind of a sort of somewhat naive way of looking at things, right?
• 26:11
  SPEAKER_01: There's ways we're gonna shrink that down for it's not gonna take 100,000 years.
• 26:14
  SPEAKER_01: Okay, I don't believe that either, but I just putting that into perspective.
• 26:17
  SPEAKER_01: But there's a number of ways we can increase it.
• 26:19
  SPEAKER_01: Actually, there's four.
• 26:20
  SPEAKER_01: The first one is simulation.
• 26:22
  SPEAKER_01: Simulation is actually really, um, as we talked about, that works really well for flying and for walking and for acrobatics, right?
• 26:30
  SPEAKER_01: For moving limbs around and and and and basically moving over terrain without falling.
• 26:36
  SPEAKER_01: That's been amazing.
• 26:38
  SPEAKER_01: It doesn't seem to work well for for manipulation.
• 26:41
  SPEAKER_01: So the second one is you is YouTube, basically, is we actually have a big repository of videos of people moving things around with their hands.
• 26:51
  SPEAKER_01: If you just sort of mine that, you couldn't come up with millions of hours of video of just people manipulating things.
• 26:58
  SPEAKER_01: That's a very valuable source.
• 27:00
  SPEAKER_01: But the problem is these are two-dimensional videos.
• 27:03
  SPEAKER_01: And it's very hard to infer what's the three-dimensional activity that's going on.
• 27:08
  SPEAKER_01: And that is actually a very active area of research and computer vision, is how to take those two-dimensional videos and transform them into three-dimensional models of what's really happening.
• 27:19
  SPEAKER_01: But then you also have to model all the little complex forces and deformations and collisions and everything else.
• 27:24
  SPEAKER_01: And that turns out to be extremely hard.
• 27:27
  SPEAKER_01: So people are working on it, they're working on all of these, but um, but that one is also not around the corner.
• 27:32
  SPEAKER_01: So that's the second one.
• 27:34
  SPEAKER_01: And the third one is what uh we just described, which is teleoperating the robot and having a person basically operated like a puppet, and you're controlling it, and then you're having it do things like uh chop vegetables and fold laundry and make coffee and all that.
• 27:49
  SPEAKER_01: And that's what physical intelligence and others are doing, and that's where the the this one year of data comes in.
• 27:56
  SPEAKER_01: Now, there's a fourth one that hasn't been widely recognized, but I'm actually very excited about, and that is real production data.
• 28:04
  SPEAKER_01: And that is where you have robots that are actually out there in the field, let's say in a warehouse, collecting data as they work.
• 28:12
  SPEAKER_01: Now, that's what AMBI robotics is doing.
• 28:16
  SPEAKER_01: For the last seven years, we've been building robots that are actually capable of doing real work, that is, sorting packages.
• 28:25
  SPEAKER_01: And it took, we had to bootstrap that.
• 28:28
  SPEAKER_01: And a lot of this is due to Jeff Mahler, PhD student at Berkeley, who it was his dissertation, but he used the first generation of convolutional neural nets combined with a big data set that we collected that was analogous to ImageNet.
• 28:44
  SPEAKER_01: We called it Dexnet.
• 28:45
  SPEAKER_01: That allowed us to train, bootstrap a model that was actually capable of being fairly good at grasping.
• 28:51
  SPEAKER_01: And it was somewhat of a breakthrough at the time.
• 28:54
  SPEAKER_01: Now, that's what you referred to earlier.
• 28:56
  SPEAKER_01: What it was able to do was to get above 90% success rate, picking up packages.
• 29:02
  SPEAKER_01: That was enough to make it economically viable.
• 29:05
  SPEAKER_01: In other words, companies would see it as a way to um, it was cost effective for them to buy these machines.
• 29:12
  SPEAKER_01: So Ambirobotics, which was formed in 2018, and then spent time commercializing and making these systems, worrying about all the things I mentioned earlier, like safety and reliability and interface and all of those things, and making it something that was really useful in this case for e-commerce, which was growing in parallel.
• 29:29
  SPEAKER_01: But that resulted in many robots in industry actually performing sorting.
• 29:34
  SPEAKER_01: And so at this point, AMBI has sorted 100 million packages.
• 29:39
  SPEAKER_01: And what's interesting is that we collected data on every single one of them.
• 29:44
  SPEAKER_01: That was originally intended somewhat as a to help us with maintenance to sort of maintain all these machines, but it turned out that it was also, now we realized that that was a gold mine of data that we can actually use to train the next generation of machines.
• 29:57
  SPEAKER_01: So that data collection, we actually just recently ran the numbers, that turns out to be 22 years.
• 30:03
  SPEAKER_01: It was collected over four years, but it was lots of robots working many shifts.
• 30:07
  SPEAKER_01: But that's 22 years of data in the bank.
• 30:10
  SPEAKER_01: It's a big resource, extremely valuable resource.
• 30:13
  SPEAKER_01: We've taken just 1% of it and used it to train a large model and this uh transformer model, right?
• 30:22
  SPEAKER_01: So the most recent generation, and it is outperforming our past method that was trained with sim to reel in terms of picking accuracy, raising those numbers up, and that is a big deal.
• 30:35
  SPEAKER_01: So we're essentially going to be doing more processing of this data, using it to generate bigger and better models, and then putting that into practice.
• 30:44
  SPEAKER_01: But then you see, we call it a flywheel.
• 30:46
  SPEAKER_01: So once you get a system that's good enough to start going into production, then people adopt it, they start using it.
• 30:53
  SPEAKER_01: But as it's collecting data, that's real data in real environments, right?
• 30:58
  SPEAKER_01: It's very valuable.
• 30:59
  SPEAKER_01: It's the best kind of data because you're dealing with real packages, not artificial.
• 31:03
  SPEAKER_01: And then you're really collecting this over time, and that then you're essentially reprocessing that, analyzing that, using that to train even better models.
• 31:13
  SPEAKER_01: And so then these systems get better and it kind of iterates.
• 31:17
  SPEAKER_01: So AMBI has also recently used that to generate a new product line that we call instead of AMBI sort, which is what we've been focusing on for the first four years, we now have AMBI stack.
• 31:28
  SPEAKER_01: So we can talk more about that later.
• 31:30
  SPEAKER_01: But coming back to what is the data story, there's these four approaches: simulation, YouTube, teleoperation, and then production.
• 31:40
  SPEAKER_01: And production is one that is extremely viable, and AMBI is really demonstrating that it works.
• 31:47
  SPEAKER_00: Maybe not talked about as much because you like you actually have to get the robots out into the world.
• 31:52
  SPEAKER_00: That first generation has to come to start collecting data.
• 31:55
  SPEAKER_00: But so so so years ago, people would say this thing about Google or Facebook.
• 32:01
  SPEAKER_00: They would say, you know, uh Google's not an email company, it's a data company.
• 32:06
  SPEAKER_00: Facebook isn't a social media company, it's a data company.
• 32:09
  SPEAKER_00: Ultimately, is Ambi a data company?
• 32:13
  SPEAKER_01: Great question.
• 32:14
  SPEAKER_01: Uh I like that a lot.
• 32:15
  SPEAKER_01: I I would say yes and no, in the sense that it's it's both, right?
• 32:20
  SPEAKER_01: Google is is certainly a a search company, an email company, uh a documents company, as well as a data company.
• 32:29
  SPEAKER_01: Because all the data they're collecting is helping them do all those things better.
• 32:33
  SPEAKER_01: That's true for Meta as well, although we might debate whether having more compelling social media is is is better, in what sense what that's better.
• 32:42
  SPEAKER_01: But um it's better for them.
• 32:43
  SPEAKER_01: It's better for their advertisers and their stock uh value, although I don't know what it is lately.
• 32:49
  SPEAKER_01: But I think that the question of companies starting to look at the data as an incredibly valuable resource that is somewhat new because I mean, always I think there was data that you could mine customer data and patterns to better serve your customers, right?
• 33:08
  SPEAKER_01: To develop products that were tailored to them or would and identify where their needs were and anticipate almost what customers want.
• 33:19
  SPEAKER_01: Amazon is a perfect example, right?
• 33:20
  SPEAKER_01: Amazon is constantly modeling and using the data, every order you make, everything you look at on Amazon's website is being, you know, combined to figure out a bit better and better model of what you're gonna want next and tailor their their distribution system, what they put in their warehouse to what are people gonna order, right?
• 33:38
  SPEAKER_01: And that's gonna shrink the time down between the time you click and the time it arrives on your front porch.
• 33:44
  SPEAKER_01: So the the idea of data is is very extremely powerful.
• 33:50
  SPEAKER_01: And it's it's increasingly interesting.
• 33:53
  SPEAKER_01: And just as I mentioned with AMBI is that you know, we we the company spends a fair amount of time developing the hardware systems, designing those and getting the right motors and putting them together and and working on on all the motion planning.
• 34:08
  SPEAKER_01: There's a lot of other aspects that go into a um uh a robot system.
• 34:14
  SPEAKER_01: It's not just data.
• 34:16
  SPEAKER_01: But the I mean, everything down to like suction cups.
• 34:20
  SPEAKER_01: What are the choice of suction cups, the design of the suction cups?
• 34:23
  SPEAKER_01: What is it, what is the right um cameras to use?
• 34:27
  SPEAKER_01: And right, there's a host of nuances, even lighting, you know, where where how are the wires routed?
• 34:34
  SPEAKER_01: Millions of as of little problems like that that are constantly having to be solved.
• 34:39
  SPEAKER_01: So data, but data is now increasingly giving you a big advantage in terms of being able to essentially process future environments and future inputs.
• 34:53
  SPEAKER_01: So maybe this is a good time to talk about the new AMBI stack system.
• 34:57
  SPEAKER_01: So the first problem what AMBI was founded on was picking.
• 35:01
  SPEAKER_01: That's what we have been working on.
• 35:02
  SPEAKER_01: And it's actually a problem I've been studying for, I don't hate to say 40 years, and since I was an undergrad, we still spend a lot of time on the picking problem.
• 35:12
  SPEAKER_01: But there's a dual to that, which is the placing problem.
• 35:15
  SPEAKER_01: Once you pick something up, how do you put it down?
• 35:17
  SPEAKER_01: In the past, we've just dropped it into a bin.
• 35:19
  SPEAKER_01: That's that's one thing.
• 35:21
  SPEAKER_01: But what if you want to what if you want to pack it into a palette that you're gonna then ship somewhere else?
• 35:27
  SPEAKER_01: So that packing problem or stacking problem is actually very nuanced.
• 35:33
  SPEAKER_01: And you see it when you go to the grocery store and you want to pack your your your um shopping bag, right?
• 35:40
  SPEAKER_01: There some people are very good at it, right?
• 35:43
  SPEAKER_01: You know, somebody will just fill that up perfectly, and most people like me are not, and you just sort of, you know, it comes out that's half empty, and you're like, you can't fit the last thing in.
• 35:53
  SPEAKER_01: So knowing how to do that well is an incredible skill.
• 35:57
  SPEAKER_01: And it turns out to be it's it's it's a variant of playing Tetris.
• 36:02
  SPEAKER_01: And you know, Tetris is uh even in two-dimensionals, right?
• 36:07
  SPEAKER_01: You're just getting the packing these little tiles as tightly as you can.
• 36:10
  SPEAKER_01: But it turns out that's actually NP hard.
• 36:13
  SPEAKER_01: It's one of the hardest, it's equivalent to one of the hardest problems in computer science, right?
• 36:17
  SPEAKER_01: It's it's there's no efficient solution available.
• 36:20
  SPEAKER_01: In 3D, it's even harder.
• 36:23
  SPEAKER_01: So doing this job, the placing problem, the packing problem, the stacking problem, that's very, very an open, interesting problem right now.
• 36:31
  SPEAKER_01: And so AMBI has started to address that.
• 36:33
  SPEAKER_01: And what we've done is we've used the data that we've collected, which is images of objects and warehouses.
• 36:38
  SPEAKER_01: We've used that and multiple images, by the way, which allows us to build three-dimensional models of where things are in space.
• 36:44
  SPEAKER_01: That's been the big benefit of this gold mine.
• 36:47
  SPEAKER_01: But it also is a part of how helping us be able to estimate how to stack things.
• 36:52
  SPEAKER_01: And that's where we built a new system.
• 36:55
  SPEAKER_01: There's an entirely new market out there, which is it's just how do you create these pallets?
• 37:01
  SPEAKER_01: And almost every industry needs to build pallets because it's gonna ship things and store things.
• 37:07
  SPEAKER_01: And that's often done manually right now.
• 37:10
  SPEAKER_01: It's just humans.
• 37:10
  SPEAKER_01: Things come down to conveyor belt, humans stacks them up and tries to fill that pallet as tightly as they can.
• 37:16
  SPEAKER_01: So we have a new system, Ambi Stack, and that's what it does.
• 37:20
  SPEAKER_01: It has a conveyor belt and then a gantry, crane-like robot.
• 37:24
  SPEAKER_01: We just announced this in uh in January, and it then attempts to build these stacks as efficiently as possible.
• 37:32
  SPEAKER_01: It turns out that's another branch of artificial intelligence called reinforcement learning that is a perfect match for that problem.
• 37:40
  SPEAKER_01: And reinforcement learning has to be done in simulation because you want to be able to do many, many trials over time.
• 37:48
  SPEAKER_01: This is, by the way, how the um the system Alpha Go, which beat the world champion in the game of Go, uh worked because it essentially trained itself using reinforcement learning to play the game.
• 38:01
  SPEAKER_01: But it was able to generate many games and over time, right?
• 38:04
  SPEAKER_01: Much faster than humans can play, right?
• 38:06
  SPEAKER_01: Millions of games per day.
• 38:08
  SPEAKER_01: So we can do something analogous, which is that we have packages, we can simulate different shapes of packages coming down a conveyor belt.
• 38:14
  SPEAKER_01: And the system will try to pack them as tightly as it can.
• 38:17
  SPEAKER_01: You know, it's over time, it's trying different strategies to learn how to do this better.
• 38:22
  SPEAKER_01: Now, if you give it a totally random stream of objects, there's a limit.
• 38:26
  SPEAKER_01: It really can't do that well because it just doesn't know what's coming, and it's just trying to basically, you know, is uh is is making certain guesses.
• 38:34
  SPEAKER_01: But if it turns out that if there's some pattern to the packages, and there almost always is, in other words, the packages aren't truly random.
• 38:42
  SPEAKER_01: They're from some set of shapes, and there's often a stream of similar shapes coming down together.
• 38:48
  SPEAKER_01: The system learns how to take advantage of that and uses it to build these very dense packages.
• 38:54
  SPEAKER_01: So our system has been trained using reinforcement learning, Ambiack can reduce 30% of the air, the empty space that is used by a competing system, commercial system that's out there.
• 39:08
  SPEAKER_01: That has been really exciting because it shows that now AI can also solve this very hard problem.
• 39:13
  SPEAKER_01: When I say solve, I'm not claiming that it's doing it perfectly, because that would be extremely difficult.
• 39:18
  SPEAKER_01: But I'm saying that it's getting closer.
• 39:20
  SPEAKER_01: So it's an approximation algorithm.
• 39:22
  SPEAKER_01: But it's actually learned to do that better, we think, than humans can do it.
• 39:27
  SPEAKER_01: And so that's where this new product, this new direction.
• 39:30
  SPEAKER_01: So we're we're good at picking, but now we can combine that with packing.
• 39:34
  SPEAKER_01: And so that's what we're very excited about is this new generation of products, Ampystack, that we're just rolling out this spring.
• 39:41
  SPEAKER_00: Part of the reason why I asked the data question is because, you know, as you said, you're also invested in the creation of the hardware.
• 39:49
  SPEAKER_00: I think it's clear to everyone why it makes sense to specifically tailor hardware to software and and vice versa.
• 39:58
  SPEAKER_00: Ultimately, though, it seems to me that the end goal is to have some kind of system that in some way is uh is hardware agnostic.
• 40:09
  SPEAKER_00: Is that something that you feel like we are moving toward?
• 40:12
  SPEAKER_01: Yes.
• 40:12
  SPEAKER_01: Okay, good.
• 40:12
  SPEAKER_01: So you're this is great, Brian, because you're bringing it back to where are we going with all this, right?
• 40:17
  SPEAKER_01: So one thing I didn't explain carefully was that I see that we're gonna get to the humanoid dream, the general, generalist robot.
• 40:26
  SPEAKER_01: But the way to get there, in my view, is to is it is one step at a time.
• 40:30
  SPEAKER_01: So that is that we actually try to solve these individual problems very well, that you get a robot to do something well, well enough that it's commercially of interest, and then you can start selling those robots to do that task.
• 40:44
  SPEAKER_01: That allows you to collect lots more data, and then you can start branching out to adjacent tasks and build no those robots and then start selling those, and those start collecting more data, right?
• 40:56
  SPEAKER_01: But you're overall amassing this very, very large and growing data set that allows you to expand it to different types of jobs, work.
• 41:05
  SPEAKER_01: Again, I want to be careful back to the stealing jobs thing, because in my view, we're not stealing jobs with these systems.
• 41:12
  SPEAKER_01: They're often moving the jobs into things that are actually more productive and transforming the workers into supervisors.
• 41:22
  SPEAKER_01: And so they actually really uh like these jobs, they're getting raises because they're being more productive.
• 41:28
  SPEAKER_01: But when I coming back to the idea of the stepping stones, that I think we're gonna get to the humanoid by learning these specific tasks and getting competency and skill level for these robots.
• 41:40
  SPEAKER_01: And then over time, they're gonna then be able to start doing other things.
• 41:44
  SPEAKER_01: Like in in in as I mentioned earlier, move we started with moving things, which is sorting and packing, but now think about making things.
• 41:55
  SPEAKER_01: And that's where you're using robots to now put things in machines, take them out, assemble objects, put things together.
• 42:03
  SPEAKER_01: That's that's the making things, and that's the next level.
• 42:06
  SPEAKER_01: And I think with all the data we're collecting, it's gonna lead to that.
• 42:10
  SPEAKER_01: And then the next step is once you're making things and you're outside, you get these skills in assembly and complex manipulation of objects, that's gonna lead to the maintaining things, where you are gonna get the sufficient skill level that you can start going into homes and hospitals and healthcare facilities, retirement homes, and actually start doing productive work there.
• 42:32
  SPEAKER_01: So, you know, it's a it's a process of rather than jumping directly into homes, which is, as we said, very difficult.
• 42:39
  SPEAKER_01: I believe that there's a path, but we go through factories, go through warehouses to factories, hospitals, and then homes.
• 42:46
  SPEAKER_01: Right?
• 42:47
  SPEAKER_01: That path seems much more viable to me.
• 42:49
  SPEAKER_01: And it's a matter of collecting data along the way to build yourself up to that full generality that we all want.
• 42:56
  SPEAKER_00: So prior to this interview, you sent me a number of the papers that your group has been working on, and one that really jumped out to me, and I think is relevant in a lot of ways to this conversation was the the sculpture, generative AI and representational sculpture.
• 43:10
  SPEAKER_00: So, you know, like really top level, you you feed in giraffe and it takes a bunch of blocks and it builds like an approximation of of a giraffe.
• 43:19
  SPEAKER_00: So so that I mean that that fits into the making part of the equation.
• 43:25
  SPEAKER_01: Very good.
• 43:25
  SPEAKER_01: Okay.
• 43:26
  SPEAKER_01: So let me do a quick shout-out to my group because I I have such a great group of students in the lab.
• 43:32
  SPEAKER_01: The name is autolab.berkley.edu.
• 43:34
  SPEAKER_01: If you want to see us or learn what we're up to, just go there.
• 43:37
  SPEAKER_01: But this is an amazing group of students.
• 43:39
  SPEAKER_01: I just adore them because they work so hard, they're absolutely brilliant, and it's a mix of postdocs, PhDs, master's students, and undergrads who are all working together and you know, shoulder to shoulder, working on all kinds of interesting problems in the lab, even as we speak.
• 43:54
  SPEAKER_01: This project that you just mentioned was something we did over the summer where we just started asking.
• 44:00
  SPEAKER_01: Asking, could a robot, could a could a large language model, again, a VLM vision language model, these are like ChatGPT, could it actually assist with robot and could they work together to build something?
• 44:13
  SPEAKER_01: And we really weren't sure if that could happen, but we we we started by just having it start up stack some blocks and put things blocks into some arrangements.
• 44:23
  SPEAKER_01: And then we started asking, could the blocks look like certain shapes?
• 44:26
  SPEAKER_01: So the result, what we call blocks net, which we'll be presenting at a conference in May, you can find it online.
• 44:34
  SPEAKER_01: But what it is is you can just say one word and show it a group of blocks, and then it will do its best to assemble that group of blocks into that shape.
• 44:43
  SPEAKER_01: So as you mentioned, giraffe is a great example.
• 44:45
  SPEAKER_01: You just give it the word giraffe and a picture of blocks.
• 44:49
  SPEAKER_01: Then it basically asks itself this prompt, okay, what are the significant features of a giraffe?
• 44:56
  SPEAKER_01: It has a long neck, it has these legs, tail.
• 44:59
  SPEAKER_01: Then it says, okay, of these blocks, which blocks would go well with these different features?
• 45:04
  SPEAKER_01: Then it starts to basically go through a stepwise process of figuring out how to generate a three-dimensional model using those blocks.
• 45:11
  SPEAKER_01: Then what it does is it generates 10 examples, and then it basically compares them itself to say which one would be the best, and it's able to do that.
• 45:22
  SPEAKER_01: Then it says, okay, is this physically stable?
• 45:25
  SPEAKER_01: So it goes over to a simulator that basically checks if it won't fall down or not.
• 45:31
  SPEAKER_01: And if it does, it basically modifies it a little bit so it is stable.
• 45:34
  SPEAKER_01: And next step is it automatically commands a robot to go and pick up those blocks and put that thing together.
• 45:41
  SPEAKER_01: And once it does that, it also it's on a little table and then it tilts the table down and the blocks fall back into the uh bin and it does it again.
• 45:51
  SPEAKER_01: So it has to do it 10 times, and over each time it's basically trying to fine-tune the design so that at the end it has a design that's very reliable.
• 46:00
  SPEAKER_01: And this has worked far better than we ever thought possible.
• 46:04
  SPEAKER_01: I have to admit, it doesn't work all the time, right?
• 46:07
  SPEAKER_01: There's some things that it can't generate a design for uh or a reasonable design for various reasons, but a lot of things, like the giraffe, like a bridge, like a shelf, it it works remarkably well.
• 46:21
  SPEAKER_01: And this is astounding to me that you can actually use these systems.
• 46:26
  SPEAKER_01: Again, this is different than the robot model we've been talking about.
• 46:29
  SPEAKER_01: This is just using the VLM, ChatGPT model that are already out there off the shelf, but combining that with some clever prompting and simulator and a robot, that that combination is able to actually build something interesting.
• 46:45
  SPEAKER_00: It's always fascinating to me as an outsider to hear that you're surprised with the out if positively surprised with the outcome of your own work.
• 46:56
  SPEAKER_00: And you know, I we we talk about this quite a bit when it comes to the LLMs, like the degree to which they are a black box, right?
• 47:04
  SPEAKER_00: We're getting a better understanding of what's going on with them, but still even the people who make them aren't entirely sure.
• 47:11
  SPEAKER_00: Like how big of a part is that kind of like that mystery and that surprise to your job as a roboticist?
• 47:18
  SPEAKER_01: Huge.
• 47:18
  SPEAKER_01: Huge, Brian.
• 47:20
  SPEAKER_01: Here's an analogy I would I would say is like the reason it's not, if you think about it, let's say you're you're cooking a new recipe, right?
• 47:26
  SPEAKER_01: You're you put it together, you do all these things, and then you at the end you serve it or you taste it, and you're surprised if it's good, right?
• 47:32
  SPEAKER_01: So it's kind of like that.
• 47:34
  SPEAKER_01: You know, it's very much new, yes, you put it all together, but that doesn't mean you know how it's going to turn out.
• 47:39
  SPEAKER_01: And that's really where research is often so fascinating.
• 47:43
  SPEAKER_01: And technology in general, like technology, we know all the pieces of the computer, but we're surprised by how this system, whether it's software or hardware or robot, will actually perform in practice.
• 47:54
  SPEAKER_01: That they the the combination is is very different than the sum of its parts.
• 47:59
  SPEAKER_01: And so that is one of the things I live for because I love those surprises when you know a system works works better than expected.
• 48:07
  SPEAKER_01: Of course, we often get the other the other kind of surprise where it's a surprise it doesn't work.
• 48:11
  SPEAKER_01: And then we're like, what wrong here?
• 48:14
  SPEAKER_01: So but that's what research is, is you're constantly being surprised because you're trying new things, you're trying things that you don't know, right, what the outcome is going to be.
• 48:21
  SPEAKER_01: Otherwise, it really wouldn't be researched.
• 48:23
  SPEAKER_01: So, you know, that's the element of surprise is very, very part of a central aspect.
• 48:29
  SPEAKER_01: And it's it, and and if you're really paying attention, that's what you're seeking out.
• 48:35
  SPEAKER_01: I think of it sometimes like fishing.
• 48:36
  SPEAKER_01: You know, you're kind of like feeling out where where do I put my effort, you know, where do I drop my line that I might pull up something, and just like in fishing, you get surprised when you pull up something, right?
• 48:46
  SPEAKER_01: It's always a surprise, and you never know what it's gonna be.
• 48:51
  SPEAKER_00: I always like to end my conversations on like big, big, broad, unanswerable, impossible questions.
• 48:57
  SPEAKER_00: You're an artist.
• 48:58
  SPEAKER_00: Well, we've spoken before, so you're probably aware of that.
• 49:00
  SPEAKER_00: But you you're you're an artist, and a lot of what the work, a lot of the work that you do kind of explores that intersection between art and technology.
• 49:09
  SPEAKER_00: I was reading an interview you did recently where you were talking about um image generators, right?
• 49:17
  SPEAKER_00: And you had previously thought that uh robots aren't capable of creativity, and maybe you're um you know may maybe that idea has since evolved for you.
• 49:32
  SPEAKER_00: So when we what does it mean when we talk about robots being creative?
• 49:38
  SPEAKER_01: I think of research, creativity, right uh research as being a very creative process and very analogous, in my view, to making artwork.
• 49:48
  SPEAKER_01: They're very different languages, but the product, in other words, an original work of art and original piece of research are very analogous because they both have to be new, novel, and coming back to our earlier point, surprising essentially to the viewer or to the reviewer.
• 50:06
  SPEAKER_01: If it doesn't have a little bit of surprise, then it's probably not going to be published or exhibited.
• 50:11
  SPEAKER_01: So both those things, again, it goes back to this phishing process, right?
• 50:15
  SPEAKER_01: That you're always trying to just intuit where is those where are those interesting things that are beneath the surface.
• 50:22
  SPEAKER_01: Now, to your to your point about creativity in machines, it's very interesting because I I didn't see any evidence of that for many years.
• 50:32
  SPEAKER_01: And it seemed to me that machine really couldn't do that.
• 50:36
  SPEAKER_01: You'd need humans to be creative, to identify what was new, what was interesting.
• 50:42
  SPEAKER_01: But I am changing my view.
• 50:44
  SPEAKER_01: And I I have admitted that I'm wrong in the terms of images, certainly, right?
• 50:50
  SPEAKER_01: We now have systems that you can describe an image and it will generate an image that will often surprise you.
• 50:55
  SPEAKER_01: And that I did not predict.
• 50:57
  SPEAKER_01: That's been amazing.
• 50:59
  SPEAKER_01: And the latest versions of, as you know, of these VLMs that have just come out are just really amazingly good at that, that skill.
• 51:09
  SPEAKER_01: And it's so that's one.
• 51:12
  SPEAKER_01: I I think that there's been other things we I the famous creativity test is you know, think of all the how many uses can you find for an object, right?
• 51:21
  SPEAKER_01: And the example I gave it was how many uses can you find for a guitar pick?
• 51:26
  SPEAKER_01: And it generated all the 100 uses like instantly.
• 51:30
  SPEAKER_01: And then one of them was a miniature sail for a toy boat.
• 51:34
  SPEAKER_01: And when I saw that, I was like, oh, that's amazing.
• 51:39
  SPEAKER_01: Like, that's such a good example.
• 51:41
  SPEAKER_01: I would never have thought of it.
• 51:42
  SPEAKER_01: And I was like, that's a great, that's creative.
• 51:44
  SPEAKER_01: Okay, I cannot deny that.
• 51:47
  SPEAKER_01: But now, you know, it's it's doing lots of things, and I think that that ability to be creative is is really exciting.
• 51:55
  SPEAKER_01: That's a frontier.
• 51:57
  SPEAKER_01: And I think that it's being used.
• 51:58
  SPEAKER_01: We're using it in the lab, we're using it to generate ideas, but also, as you said, this idea of generating three-dimensional objects.
• 52:05
  SPEAKER_01: Hopefully, it's going to generate you know new new chip designs, it's going to generate new proteins.
• 52:11
  SPEAKER_01: Uh super excited about what what what it's going to come up with.
• 52:15
  SPEAKER_00: To a certain extent, would it be fair to say that what creativity means in machines is the ability to surprise?
• 52:23
  SPEAKER_01: I think that's a good way to that's perfect.
• 52:26
  SPEAKER_01: Yeah, I think that's right.
• 52:27
  SPEAKER_01: Ability to surprise.
• 52:28
  SPEAKER_01: Now, here's one I haven't seen yet, which is I haven't seen a an AI system come up with a great joke.
• 52:36
  SPEAKER_01: I'm waiting.
• 52:37
  SPEAKER_01: I'm waiting for that.
• 52:39
  SPEAKER_01: Um, I don't think I think it'll happen, but it seems like that's really hard.
• 52:44
  SPEAKER_01: And if you ask, if you ask for good jokes from ChatGPT and other tools, they're not really there yet.
• 52:50
  SPEAKER_01: But uh boy, it would be great if they were.
• 52:52
  SPEAKER_01: We could just have an endless stream of of really hilarious jokes.
• 52:55
  SPEAKER_01: That wouldn't that be great?
• 52:57
  SPEAKER_00: There you go.
• 52:58
  SPEAKER_00: Great, as always, chatting with Ken.
• 53:00
  SPEAKER_00: Thanks so much to him for uh taking the time to do that.
• 53:04
  SPEAKER_00: Thanks to you, as always, for sticking around.
• 53:06
  SPEAKER_00: If you learn something new, please like and subscribe.
• 53:09
  SPEAKER_00: And don't forget to check out the automated newsletter.
• 53:11
  SPEAKER_00: You can find that and more over at automated.fm.
• 53:15
  SPEAKER_00: Thank you so much for joining, and we will catch you next week for another episode of Automated.