Latest / Embodied AI 101 / DragMesh-2: A Contact-Driven Framework for Dexterous Hand–Object Interaction
Transcript
- DragMesh-2: A Contact-Driven Framework for Dexterous Hand–Object Interaction. Manipulating ordinary household objects like cabinet doors and drawers is deceptively hard for robots. A human hand simply reaches out, makes contact with the handle, and the object moves. For a robot, however, this involves reasoning about complex contacts, friction, and the dynamics of articulated parts. The new DragMesh-2 framework (Zhang et al., 2026) tackles exactly this problem. It teaches a highly articulated robot hand – in their case, an amazing 51 degrees-of-freedom anthropomorphic hand – to open drawers and doors solely through contact-driven motion, without any tactile sensors. In other words, the policy does not see force or pressure readings from the fingers: it has to figure out how to push and pull purely from the consequences of contact. This approach addresses a crucial gap. Most prior work on dexterous manipulation assumes either rigid sticking to objects (like simple parallel-jaw grippers) or relies on explicit sensing of force and touch. The DragMesh-2 paper points out that multi-fingered hands offer “compliant contact patterns beyond parallel-jaw grasping”, which is precisely what you need for subtle tasks like pulling open a drawer on a hinge or sliding a cabinet door. But it also stresses that articulated-object manipulation is fundamentally different from grasping a static object. As the authors put it, “the target part cannot be directly actuated, and its motion must emerge through sustained physical hand–handle contact”. You can’t simply snap your fingers and have the drawer move; instead the robot must push and maintain contact against resistance. And here’s the catch: if you just pre-compute a movement for the handle (say from some motion planner) and then command the hand to follow it, you’ll fail, because those open-loop trajectories “do not model the contact dynamics required to move the articulated part”. The system has to continuously react to whether the object is actually budging under its force. DragMesh-2 takes a clever approach to this. Rather than commanding the door or drawer directly or relying on vision or tactile feedback alone, it trains a full dexterous hand controller in simulation where contact physics is native. In effect, it extends the idea of articulated object generation to a “hand-driven” setting. The original DragMesh system (Zhang et al., 2025) was about generating plausible object motions from simple drag actions; DragMesh-2 flips the script and generates hand motions that produce those object motions through contact. Zhang et al. call this a contact-driven framework – the articulated object only moves because the hand pushes on it. A key innovation in DragMesh-2 is their training method, which they dub PICA (Physically-Informed, Contact-Aware training). PICA is essentially a way to bake physics knowledge into the learning process without giving the policy any direct force or tactile input. In their words, PICA “injects physical signals into policy learning without tactile or force feedback”. In practice, this means the training algorithm somehow cues the network about the contact forces – for example by randomizing the weight, friction, and damping of the object joints during training, or by shaping the reward based on contact stability – so that the learned policy does not overfit to a single, idealized contact scenario. (The exact implementation details are in the paper, but the intuition is like a curriculum or domain randomization for contact loads.) The result is a policy that knows, for instance, “if this door feels heavier or tighter to move, I should push harder or adjust my finger placement,” even though it never reads a force sensor; it learned to anticipate those changes from experience. Importantly, this makes the policy far more robust. As Zhang et al. report, across seven complex objects (drawers, doors, etc. from the GAPartNet dataset), DragMesh-2 “achieves stronger robustness under contact-load variation than the compared methods”, all while keeping a high success rate. In other words, even when the hinge friction or weight of the drawer is altered, the approach still opens it successfully in most trials. Under the hood, the DragMesh-2 system is a full simulated training pipeline:. Tasks and Environment: They set up six or seven kitchen-scale articulated objects (things like cabinet doors on hinges, slide-out drawers, and other mainspring-loaded parts) using models from PartNet/GAPartNet. Each object is simulated in a physics engine (the paper likely uses SAPIEN or a similar platform). There is a single 51-DoF anthropomorphic robot hand (with palm, wrist, and five fingers, all highly articulated). The only thing the hand “feels” is its own joint angles and velocities (no force or touch readings). The goal is to open each object part, usually by pushing a handle or edge. A trial is successful if the door/drawer reaches its target open position. Contact-Driven Control: Instead of using vision, the policy is purely reactive to its own movements. For example, if a finger is pushing and the contact is sliding, the result of that contact is seen indirectly: the finger’s movement slows or the joint torque spikes (though no explicit sensor, the simulator physically couples the hand and object). The policy learns from its proprioception (joint states) and possibly the simulated pose of the object (some works assume known object pose even in simulation). In any case, the core idea is the object only moves if the hand pushes hard enough at the right spot. PICA Training: During learning, the authors randomize or perturb the physics a lot. For example, they vary the damping and friction of each articulated joint (simulating a heavier door, a sticky drawer, etc.) across episodes. This way the policy doesn’t just memorize a single “ideal” push. They also structure the reward and loss so that maintaining contact yields better rewards – effectively giving the policy a hint that contact and force matter. These are examples of the “physical signals” in PICA. Crucially, the policy network never gets direct force vectors; it only sees, say, that the door isn't moving as much as expected. The PICA mechanism ensures the network internalizes this implicitly. Policy Architecture: While the paper’s summary doesn’t detail the exact neural network, it’s probably a deep RL setup (e.g. a transformer or MLP) that takes the hand’s joint positions and velocities (and maybe object position or task goal) and outputs joint torque commands. The learning algorithm could be PPO or SAC. The training is done in simulation long enough (millions of steps) to master each task. Because the hand has 51 DoF and the tasks are contact-rich, this is a significant achievement in compute. Baselines: DragMesh-2 is directly compared against more naive approaches. For instance, a baseline might be “trajectory replay”: if you had an oracle that tells the object handle how to move (for example from DragMesh-1’s motion planner), you could try to follow that trajectory with the hand. But as the authors note, that fails to adapt to contact forces. Another likley baseline is an RL-trained policy without PICA (trained only in fixed physics). Without PICA’s randomization, those policies degrade quickly when the joint becomes heavier or if they lose grip. Possibly they also compare to a more direct force-control method (like an impedance controller with idealized force sensing), though the snippet doesn’t say. When tested, DragMesh-2’s policies handle wide variations. Zhang et al. say that “DragMesh-2 achieves stronger robustness under contact-load variation than the compared methods”. In practice, this means that if you make the door twice as heavy, or suddenly put glue in the drawer slide, the hand still slowly but reliably opens it, whereas the baselines may slip off or stall. Even under varying “damping conditions” (i.e. joint friction or servo damping), DragMesh-2 maintains its success rate. This robustness is exactly what you want in a household setting, where friction and weights are never exactly known ahead of time. The paper reports success rates near the top performance for all seven objects, a notable improvement over prior methods. (In contrast, a baseline might succeed 80% of the time under default conditions but drop to 50% when the parameters change; DragMesh-2 stays high.). Another practical point is that Zhang et al. open-source their entire simulation stack (object models, tasks, and trained policies) as a resource for future research. They call it a “pure-geometry dexterous interaction resource,” meaning they provide the geometry of the objects and tasks so others can build on their work. For example, a research lab could plug these 3D models and tasks into the SAPIEN simulator and try new dexterity algorithms. This transparency will be a boon: dexterous manipulation with articulated objects is still a niche area, and having a shared benchmark and code helps comparisons and progress. (The summary hints that all code and training scripts will be available, though we rely on the paper text for details.). In summary, DragMesh-2 represents a state-of-the-art in simulated dexterous manipulation. Its contact-driven philosophy means the policies literally learn to feel the objects through physics, rather than being told exactly how the handle should move. The PICA training ensures that this feeling is robust to real-world variability, even without real tactile sensors. And the focus on challenging everyday tasks (open those stubborn drawers!) makes the results concrete and compelling. For roboticists, this suggests a clear takeaway: if you can simulate your task well, it is possible to train extremely high-DOF hand policies that behave sensibly in contact-rich settings, by embedding contact awareness directly into the learning algorithm. Key takeaways:. DragMesh-2 tackles articulated-object manipulation (doors, drawers) with a fully dexterous hand. Such tasks require forces to be applied at the right points so that the object moves; the policy must coordinate all fingers in contact. A naïve approach (replaying trajectories or acting open-loop) fails because it ignores the actual hand–object contact dynamics. Even standard RL trained on one physics setting can overfit and fail if joint friction or loads change. The core contribution is PICA: a training mechanism that uses physics knowledge (e.g., contact forces, varying loads) to shape the policy’s learning, but does so without assuming any tactile/force sensors on the hand. This ground-up approach – learning by contact – is what gives robustness. In large-scale tests across seven different objects, DragMesh-2 consistently outperforms baselines under varying conditions. The paper reports its policies keep a high success rate even when the contact load changes, whereas other policies degrade. Lastly, the authors share their code and object assets (“a pure-geometry dexterous interaction resource”) so the community can extend this work. This makes it easier for others to build on DragMesh-2, test new algorithms on the same tasks, and ultimately bring these simulated advances closer to real robots. For ML and robotics researchers, DragMesh-2 is an inspiring example of how to combine a high-fidelity simulator, a clever training scheme, and a powerful policy model to solve a previously intractable problem. It shows that even without touch sensors, a robot hand can learn to open drawers and doors reliably, simply by experiencing contact physics during training. This approach could be adapted to other contact-rich tasks (e.g., pushing, sliding, assembly) and might inform how we train real robot hands when we only have vision and proprioception on board. In short, DragMesh-2 pushes the boundary of dexterous manipulation and offers a new framework for “feeling” objects through physics.