Latest / The Robotics Podcast with Fexingo: Autonomous Systems, Industrial Robots, and Hardware / What Robot Hands Still Can't Do After 100 Episodes
Transcript
- Lucas: So, episode one hundred. A hundred episodes of watching robot arms fail at things a toddler can do. And I thought, instead of adding another item to that list, maybe we step back and ask: what have we actually learned? Luna: Right. We've seen them struggle with grapes, toothpicks, wet paper towels, feathers… it's a long list. Lucas: And each time we say 'still can't.' But the interesting thing is that the failures aren't really about the robot's hardware anymore. The grippers are better than ever. The sensors are insane. The bottleneck is something else. Luna: Let me guess — it's the software. The planning and control. Lucas: Exactly. And I think a great way to see this is the so-called 'Flying Pizza' experiment from Berkeley. A robot arm flips a piece of pizza dough in the air and catches it on a spinning platform. It's beautiful. It's dynamic. It requires millisecond timing. Luna: And yet that same robot probably can't pick up a single grape off a table without crushing it. Lucas: Exactly. And that's the puzzle. The flying pizza is a 'dynamic' task — ballistic motion, predictable physics, one object, one trajectory. But picking up a grape is a 'contact-rich' task. The robot has to feel the grape's shape, its squishiness, the friction at the fingertips. And it has to decide how much force to apply in real time. Luna: So the grape is harder than the pizza. That's counterintuitive. Lucas: It is. And it's why, after a hundred episodes, the fundamental problem hasn't changed: robots are terrible at 'in-hand manipulation.' That's the ability to reposition an object inside the gripper without dropping it. Humans do it constantly — you pick up a pen and adjust it to write. Robots? Not so much. Luna: So what's the hold-up? We have deep learning, we have simulation, we have all this compute. Why can't we train a model to do in-hand manipulation? Lucas: Part of it is simulation to reality gap. You can train a neural network in simulation to do thousands of flips and regrasps. But the moment you put it on a real robot, the friction is a little different, the sensor noise is a little different, and the policy falls apart. Luna: And you can't just brute-force it with more simulation because the real world has too many variables. Lucas: Right. And that's why some researchers are going back to control theory — trying to build models that capture the physics of contact. But those models are incredibly hard to solve analytically. So we're stuck between two worlds: black-box learning that doesn't generalize, and white-box physics that's too complex. Luna: It sounds like a fundamental research challenge. And yet, there are robots that do something useful. Industrial arms weld cars, pack boxes, pick and place thousands of times a day. Lucas: Because those tasks are structured. The parts are the same every time. The lighting is controlled. There's no real uncertainty. But the dream of a general-purpose robot that can walk into your kitchen and cook you a meal? That's a different story. Luna: And that's what we've been tracking for a hundred episodes — every time we think we're close, we find another edge case. Lucas: Exactly. And I think it's worth saying: the fact that we can list so many failures is actually a sign of progress. We're finding the hard problems. Ten years ago, nobody was even trying to pick up a single grape. Now we know exactly why it's hard. Luna: That's a good framing. So what's the next frontier? What do you think will change in the next five years? Lucas: I think the real breakthrough will come from better 'tactile sensing.' We need robot fingers that can feel texture, temperature, and shear force — like human skin. And we need algorithms that can use that data in real time. A lot of labs are working on soft, sensorized grippers that can deform around objects. Luna: And then the control problem becomes: how do you use that stream of touch data? That's a whole new research area. Lucas: It is. And it's one of the reasons I keep coming back to this show. Every episode, we learn something new about what intelligence really requires. It's not just vision and language. It's also the feel of a grape. Luna: Speaking of coming back — and I know you don't like to do these — but we've hit episode one hundred, and we only exist because people like you listen and sometimes support. If today's tech conversation gave you something usable, a couple of dollars a month is genuinely what keeps this thing going — buy me a coffee dot com slash fexingo. Lucas: Yeah, it really is that small. And it means we don't run ads, we don't chase clicks, we just chase the questions that interest us. So if you've gotten something out of a hundred episodes, that's the place. Luna: Alright. Let's get back to the robot hands. You were saying about tactile sensing? Lucas: Right. So one promising direction is 'gel-based' tactile sensors — like the GelSight sensor from MIT. It uses a soft gel pad with a camera behind it, so the robot can literally see the deformation of its 'skin' at high resolution. That gives you a rich tactile image. Luna: And then you can train a neural network to map those tactile images to actions? Lucas: Exactly. And there's been some impressive results. For example, using tactile feedback, a robot can slide a battery into a tight slot — something that's incredibly hard with vision alone because the tolerances are sub-millimeter. Luna: But that's still a narrow task. What about general manipulation? Lucas: That's where the field is heading. There's a concept called 'dexterous manipulation' — using a multi-fingered hand to reposition an object in the palm, like rotating a screwdriver in your fingers. A few labs have shown it works with tactile feedback and reinforcement learning, but it's still very slow and fragile. Luna: So we're back to: robots can't do what a toddler does. But we've made progress? Lucas: We've made huge progress — but we've also learned how deep the problem is. And that's actually what I love about this show. It's not about hyping the next breakthrough. It's about understanding the real, hard constraints. Luna: A hundred episodes of constraint appreciation. That's a good tagline. Lucas: I'll take it. And I think the listener who's been with us this whole time has probably developed a really nuanced view of what AI can and can't do. That's valuable. Luna: So where do we go from here? Episode 101? Another failure? Lucas: Probably. But I think we'll also start looking at solutions more. Now that we've mapped the failure space, we can talk about what actually works. Tactile sensing, compliance, sim to real transfer — the things that might finally get us past the grape. Luna: I'm in. And thanks to everyone who's stuck around for a hundred episodes. It's weird to say it, but this show has become a real reference point for me. Lucas: Same. Alright, let's keep going. Next time: can we teach a robot to use scissors? That should be fun.