Latest / The Robotics Podcast with Fexingo: Autonomous Systems, Industrial Robots, and Hardware / Why Robot Hands Still Can't Pick Up an Apple
Transcript
- Lucas: So robots can build a car from scratch, they can fly a drone through a burning building, they can even perform certain surgical procedures with sub-millimeter precision. But ask a robot to reach into a bowl of fruit, pick out a single ripe peach, and hand it to you without crushing it — and you've just handed it one of the hardest unsolved problems in robotics. Luna: That's the manipulation problem. And it's been a bottleneck for decades. Why is something so basic for a human so incredibly hard for a machine? Lucas: Because, at its core, manipulation is about sensing and control in an uncertain world. A human hand has something like 27 degrees of freedom, plus thousands of tactile nerve endings. You don't think about how much force to apply picking up an apple — your brain just does it, using feedback from your skin and your joints. Luna: Right. Whereas a typical industrial gripper is basically two rigid fingers that just squeeze until they hit a pre-set torque. That works great for a metal engine block. Not so much for a tomato. Lucas: Exactly. And the industry knows this. The benchmark everyone talks about is something called the yale cmu berkeley Grasping Dataset — it's a standardized set of objects used to test how well a robot can pick up novel items. The current state of the art in controlled lab conditions? Around 85 to 90 percent success for simple grasps. Luna: But in an unstructured environment — say, a cluttered bin at a warehouse — that number drops below 50 percent. I've seen figures from Amazon's 2024 picking challenge where the best robot was still failing on roughly a third of the objects. Lucas: Yeah, the 2024 Amazon Picking Challenge — the top team hit about 67 percent on dense bin picking. That's in a controlled competition with known objects. In a real warehouse with unknown items, it's even lower. So the gap between where we are and where we need to be is still enormous. Luna: What's the main obstacle? Is it the hardware — the hand itself — or the software and sensing that controls it? Lucas: It's both, but I'd argue the sensing and control loop is the harder piece right now. There are some incredibly impressive hands out there. Shadow Robot's Dexterous Hand has 20 joints, 24 movements — it's basically a human hand in size and dexterity. Costs about $100,000. But even that hand struggles with tasks a toddler can do, because the tactile feedback is nowhere near as rich as human skin. Luna: So you can have a great hand, but if you can't feel what you're touching, you're essentially operating blind. That's where tactile sensors come in. Lucas: Precisely. There's been a lot of work on sensor skins. MIT's Computer Science and Artificial Intelligence Lab developed something called GelSight — it's essentially a gel-based sensor that uses a camera to capture the deformation of the surface when it contacts an object. It can detect texture, shape, even slip. The resolution is remarkable — they can read a penny's ridges. Luna: But that's a sensor, not a hand. And integrating it into a functional gripper that's also robust and affordable — that's the engineering challenge. Lucas: Right, and that integration is where start-ups like Robotiq come in. They make adaptive grippers that aren't trying to mimic the human hand. Instead, they use a simpler mechanism — three fingers that can conform to objects. It's a compromise between the rigid two-finger gripper and the full anthropomorphic hand. They're used in a lot of collaborative robot arms. Luna: So the approach seems to be: don't build a human hand, build something that's good enough for specific tasks. That's pragmatic — but does it limit the dream of a general-purpose robot? Lucas: It does, but maybe that's okay. For most industrial and warehouse applications, you don't need a hand that can thread a needle. You need a hand that can pick up a shoe box, or a bottle, or a phone — objects that come in a limited range of shapes. The real challenge is when you want a robot that can walk into a home and pick up anything — a cup, a fork, a piece of paper, a remote control. That's the holy grail. Luna: And that's where simulation becomes crucial. Because training a manipulation policy in the real world is slow and dangerous — you break things. So people use simulators like MuJoCo or Isaac Gym to train reinforcement learning models in virtual environments. Lucas: Yeah, MuJoCo — which stands for Multi-Joint dynamics with Contact — is basically the standard for simulating contact-rich manipulation. It's open source, fast, and it handles the physics of objects sliding, rolling, and colliding. Researchers can train a neural network to control a hand in MuJoCo, and then transfer that policy to a real robot — although the transfer still has a big gap because simulation can't perfectly model friction, deformation, or sensor noise. Luna: There was a big paper from OpenAI in 2018 where they trained a Dactyl hand to manipulate a cube — purely in simulation — and then it worked in reality. But that took massive compute, and the task was relatively simple: turn the cube to a target orientation. Still, it was a proof of concept. Lucas: Right, and that hand — the Dactyl — was a modified Shadow Hand. The policy was trained using a technique called domain randomization, where you vary the simulation parameters so the policy learns to be robust to differences between sim and reality. That approach has become standard. Luna: But here's the thing that's often lost in the hype: even with the best sim to real transfer, the success rate for dexterous manipulation tasks is still well below 90 percent for most tasks. And when you scale to dozens of objects, it drops fast. Lucas: That's why you hear about robots in warehouses that still rely on suction cups or vacuum grippers for picking. It's simpler and more reliable. Amazon uses a mix: suction for smooth items, pinch grippers for irregular ones. They've optimized their entire supply chain to make items robot-friendly — standardized packaging, barcode placement, everything. Luna: So the world is being reshaped to accommodate the limitations of robots, rather than the other way around. That's interesting — it's not just a hardware problem, it's a systems design problem. Lucas: Exactly. And in many ways, that's the smarter path. Instead of solving general manipulation in all its glory, companies are engineering the environment to simplify the task. That's why you see more robots in structured factories than in homes. Luna: Let's talk economics. How much does it cost to put a dexterous hand on a robot arm, and when does it make financial sense? Lucas: A typical collaborative robot arm from Universal Robots or Fanuc might cost $25,000 to $50,000. Adding a simple two-finger gripper is a few thousand dollars. Adding a Shadow Hand? You're looking at $100,000 just for the hand. Plus the control system and integration. So a full dexterous manipulator is easily $150,000 or more. Luna: And what does a human worker cost? In the US, a warehouse picker might earn $35,000 to $45,000 a year. So the robot has to last multiple years and have a very high uptime just to break even. And right now, it doesn't — it fails too often on unfamiliar objects. Lucas: That's the rub. The total cost of ownership includes maintenance, programming, and the cost of errors — damaged goods, system downtime. For now, humans are still cheaper and more flexible for most picking tasks. But as labor costs rise and sensor costs drop, the equation shifts. Luna: So where do we see progress happening fastest? Is it in sensor technology, in compliant hardware, or in ai based control? Lucas: I'd say all three, but the most exciting recent work is in compliant, soft robotics. There are grippers made from silicone rubber that can deform around objects. They're essentially pneumatic — you inflate them to grip, deflate to release. They're cheap, safe, and can handle irregular shapes. But they lack precision and speed. Luna: There's also work on underactuated hands — hands with fewer motors than degrees of freedom, so they passively conform to objects. The Robotiq gripper is an example. Another is the Yale OpenHand project, which provides open-source designs for underactuated hands. Lucas: That's a great point. The OpenHand project has been influential because it lowers the barrier to entry. Researchers can 3D-print a functional hand for a few hundred dollars. And because it's underactuated, it doesn't need complex control — just one motor to close the fingers, and they wrap around whatever they contact. Luna: So democratizing the hardware is one path. But do you think we'll see a breakthrough in the next five years that makes general-purpose manipulation viable? Lucas: I'm cautiously optimistic. The combination of better tactile sensors, cheap compliant hardware, and AI models trained in simulation is already producing rapid improvements. But I think we'll see specialized manipulation first — robots that can handle a specific class of objects very well — before we see a robot that can pick up anything. Luna: So the apple test — a robot picking a ripe apple from a bowl — might still be a few years away from being reliable. But for a warehouse that only handles shoeboxes, the solution is already here. Lucas: Exactly. And that's the story of robotics in general: we overestimate what can be done in a year, and underestimate what can be done in a decade. The grasping problem is hard, but every year, the success rates inch up. And eventually, the robot will pick that apple without crushing it — it just might not look like a human hand doing it. Luna: That's a good place to leave it. Next time, maybe we'll talk about the flip side — how human hands are adapting to work with robots. Lucas: I like that. Until then, keep your grip.