Latest / Embodied AI 101 / DexMachina: RL for Long-Horizon Bimanual Dexterous Policies
Transcript
- DexMachina: RL for Long-Horizon Bimanual Dexterous Policies. Modern robotic hands hold great promise for replicating the dexterity of human hands, but they also bring huge complexity. In this work, the authors tackle one of the hardest problems in dexterous control: functional retargeting of a human bimanual manipulation into a robot. Concretely, given a single demonstration of a human using two hands to manipulate an object, DexMachina trains a robot policy (via reinforcement learning) that drives its own hands to make the object follow the same trajectory. This is ambitious because the tasks are long and involved (often requiring opening lids, flipping objects, and coordinating both hands in mid-air), the action space is huge, and the mapping from human hands to robot hands is mismatched. Yet DexMachina shows that a carefully designed curriculum and reward structure can bridge this “embodiment gap” and produce feasible, high-performance policies across many robot hand designs and objects. In essence, DexMachina learns long-horizon bimanual dexterous policies from one human demo for arbitrary robotic hands, objects, and tasks. It does not try to copy the human’s precise finger motions; instead it only cares that the object ends up where the demonstration says it should. (The authors call this functional retargeting to distinguish it from naïve kinematic retargeting. Kinematic retargeting would simply clone the human’s joint angles, which often fails in practice when the robot can’t physically achieve them.) By focusing on object-level goals and using RL in simulation, DexMachina lets the robot find its own feasible way to accomplish the task. This approach leverages one human demonstration as a form of “oracle” guidance, then gradually handcuffs the oracle away so the robot policy must learn to take over. The result is a single RL algorithm that can retarget any one demo to multiple robot hands and objects – a first step toward truly general dexterous manipulation. The Problem: Functional Retargeting of Dexterous Tasks. Bimanual manipulation tasks—we mean things like opening a blender lid, closing a notebook, flipping a waffle iron—are central to daily life. The humans in our dataset perform these with smooth coordination and ease. Getting a robot to do the same is far harder: the action space is enormous (multi-fingered hands on two arms!), contact timing and forces are delicate, and one must plan a long sequence of steps without reward shaping. Collecting real robot data is slow and expensive, and even imitation learning from humans is tricky: capturing precise high-dimensional hand actions through teleoperation is nontrivial, and the human’s motions might not even be possible for the robot due to different range of motion and strength. This mismatch – the embodiment gap – means naive imitation can fail spectacularly. DexMachina addresses this by recasting the problem: we have one video or motion-capture sequence of a human manipulating an articulated object. We extract the object’s trajectory (its position, orientation, and joint angles) over time from that demonstration. Then we ask: Can a robot control its own hands so that the object follows that trajectory? The human’s hand poses themselves are only secondary guidance. In other words, instead of copying how the human moved, we just want the object to end up in the right places at the right times. This distinction is key. As the authors explain, functional retargeting “denotes the problem as: given a human demonstration, the goal is to learn dexterous hand policies that can manipulate the object to follow the demonstrated trajectory”. It differs from kinematic retargeting in that feasibility and outcome matter more than geometry. The goal is not human-like arms but a successful object manipulation. In practice, this means the robot could adopt a completely different strategy than the human did, as long as it achieves the same object motion. However, achieving this is extremely challenging for complex, long-horizon tasks with two hands. Without guidance, RL can flounder. Early in training the robot often fails catastrophically (for example, simply dropping a held object), and then it never even gets to try the later steps like opening a lid. Standard deep RL struggles in such settings. That’s why DexMachina introduces a curriculum and auxiliary guidance to make the learning path smoother, especially for these long, intricate sequences. The DexMachina Algorithm. At a high level, DexMachina combines reinforcement learning with a clever curriculum and auxiliary objectives. We have:. Task Reward: a reward that scores how well the robot’s manipulation matches the demonstration’s object path. If the object is in the right place with the right joint angles (for example, the blender lid is at the opened angle), it gets high reward. In practice, DexMachina defines the task reward as the product of three terms (positional error, rotational error, and articulated angle error) between the current object state and the demonstration state for each timestep. In math, $$r_{\text{task}} = \exp(-\beta_{pos} , d_{pos}) ,\exp(-\beta_{rot} , d_{rot}) ,\exp(-\beta_{ang}, d_{ang}),$$ where $d_{pos}$ is the distance between current and target object positions, $d_{rot}$ is the angular difference, and $d_{ang}$ is the difference in object articulation angles. This reward encourages the robot to follow the demo, but by itself it’s sparse and undirected; it doesn’t say how to achieve the goal. Auxiliary Guidance: To help training, DexMachina extracts extra signals from the one demonstration. First, it runs a kinematic retargeting of the human’s hand poses into the robot’s hand space. This provides a reference trajectory of robot joint angles $Q$ and corresponding keypoint positions $X$ for what the human-like motion would look like. Then, during training, the policy is encouraged to imitate these references softly. Concretely, there are two imitation rewards:. A motion imitation reward that encourages the robot’s hand keypoints to stay close to the retargeted keypoints (using an exponential of squared distance). A behavior cloning reward that penalizes deviation of the robot’s joint angles from the retargeted joint values (again using an exponential on the joint-angle error). These let the policy “follow along” the human motion, helping it to keep the hands oriented sensibly around the object while learning. Additionally, the authors compute an approximate contact map from the demo: basically, they identify which parts of the hand touched the object where, at each time step. During policy training, a contact reward is given that encourages the robot’s fingers to be in contact with the same object surface points as in the demo. For example, if in the demonstration the human’s thumbs pressed on the top of a blender lid at a certain frame, DexMachina tries to guide the robot to do the same. This contact reward is computed by matching the robot’s current contact points to the demonstrated contact positions (using a masked L2 distance). All together these imitation and contact rewards shape behavior, but they are weighted smaller than the main task reward so that at convergence the policy optimizes object-tracking, not just mimicry. Hybrid Action Space: Another trick is how the robot’s action space is parameterized. Instead of letting the policy output arbitrary joint commands, DexMachina uses the retargeted joints $Q$ as a baseline. The robot’s 6-DOF wrist joints are treated as incremental residual commands: the policy outputs small adjustments on top of the retargeted wrist motion. The finger joints, however, are commanded in an absolute normalized coordinate system. In practice the policy’s action vector consists of residual offsets for the wrists and normalized angles for the fingers. This “hybrid” scheme constraints the search space to around the demonstrated motion, making RL significantly easier. Virtual Object Controllers (Curriculum): Perhaps the most novel piece of DexMachina is its curriculum using virtual object controllers (VOC). Early in training, the policy is very weak and likely to fail immediately (for instance, the robot might drop the object at the first step). To prevent this, DexMachina temporarily gives the robot a “helper” that moves the object along the demo path. Concretely, it introduces artificial control forces on the object that push it toward the demonstrated trajectory. At the start of training this VOC is strong, meaning the object pretty much follows the demo almost independently of the robot’s actions. Over time, the strength of this VOC is gradually decayed (hence the “curriculum”), so that the robot policy must take over control itself. Early on, the robot policy can relax – the object is moving correctly thanks to the VOC, so it just needs to practice being in the right place and making contacts (“the object does itself the heavy tasks”), but later the VOC influence fades and the policy must fully execute the manipulation. In practice this means that in initial episodes the task is much easier (the robot rarely fails catastrophically), so the agent can learn useful hand strategies under the wing of guidance. As the VOC weakens, the policy has already learned most of the steps and just needs to refine them to hold the object on its own. This auto-curriculum is a form of guided exploration: it prevents the agent from getting stuck in early failures by partially solving the task for it, then handing responsibility to the policy bit by bit. The authors call it DexMachina – a nod to the term “Deus ex machina”, meaning an external force solves a problem until the hero can stand on its own. In their words, the VOCs “move the object on its own to follow the demonstration, and are then decayed over the course of RL training as the policy learns to take over”. Putting it all together, DexMachina trains a single RL policy (using PPO) that takes in the full state (the positions and velocities of both hands and the object) and outputs the two hands’ actions simultaneously. The reward maximizes how well the object matches the demo, plus small bonuses for imitation of hand motion and contacts. Early on the VOCs do most of the object manipulation, and later the policy does it itself. Notably, the same procedure works across different robot hands without changing the algorithm or rewards – the only difference is the kinematic model. The robot simply uses its own retargeted version of the demo as guidance. This is how DexMachina can generalize “a human demonstration to a variety of existing dexterous hand embodiments”. Experiment Setup: Tasks and Robots. To evaluate DexMachina, the authors created a simulation benchmark drawing on the ARCTIC dataset of human bimanual demonstrations. ARCTIC (CVPR 2023) contains motion-captured videos of people manipulating articulated objects (like scissors, laptops, mixers) with both hands. From ARCTIC they selected a subset covering 5 articulated objects and 7 demonstration clips with diverse motions, including both short tasks and very long, complex sequences. Examples include opening a laptop/notebook cover, manipulating a box, and interacting with kitchen devices like a mixer and a waffle iron. (For instance, the “Waffleiron-300” demo requires picking up a waffle iron, opening and closing its lid, flipping it in mid-air, then opening and closing it again.). They paired these task demonstrations with 6 different robot hand models (all furnished with 6-DoF virtual wrists). The hands include popular open-source designs (Inspire, Allegro, XHand, Schunk, Ability, DexHand) with a variety of finger counts, sizes, and actuation levels. Importantly, no per-hand tuning or custom policies were used beyond specifying the retarget mapping; DexMachina’s algorithm is identical for all. The whole training was done in simulation using NVIDIA’s new Isaac Sim Genesis physics engine for speed (enabling thousands of parallel environments). They trained each hand-task pairing from scratch with PPO (via the RL Games framework), using about 12,000 parallel environments on GPU for efficiency. Each policy controls both hands simultaneously – the observation state includes both left and right hand configuration and the object state, and the single policy outputs the combined 2-hand action vector. All training hyperparameters were kept the same across experiments. This unified setup thus serves as a common benchmark: one human demonstration per task, multiple robot hands, and a single algorithm that has to learn how to perform the manipulation. Baseline Comparisons. DexMachina was compared against several baselines to demonstrate the benefit of its curriculum and guidance:. Kinematic-only (No Learning): The simplest “policy” is just to replay the retargeted trajectory. That is, take the human motion retargeted to the robot hands and feed it open-loop in simulation. This yields plausible-looking hand motions, but crucially it often fails to actually complete the task. As the authors note, playing back the raw kinematic retarget is typically not feasible for task success – the robot might penetrate objects or never apply enough force to open a lid, for example. This baseline tests how much learning is needed beyond naive imitation. ObjDex (Task-Reward-Only RL): This baseline, introduced in previous work, trains an RL policy that only uses the task reward on the object (with hybrid action space as in DexMachina) and ignores the imitation contacts. In effect it is DexMachina minus the auxiliary rewards and curriculum. The paper calls this “ObjDex”, a policy that plans wrist motions to move the object without guided imitation. In our reimplementation we also include the same hybrid action formulation but no decaying VOC or imitation rewards. This tests whether raw sparse object guidance is sufficient. Task + Aux, No Curriculum: This is essentially DexMachina without the VOC curriculum. It includes the imitation and contact rewards but keeps the object controllers turned off (or fixed at zero). This shows the impact of the VOC strategy itself: does just adding mimicry rewards help long-horizon tasks, or do we really need the object-driving curriculum?. ManipTrans (Concurrent Work): A recent method that fine-tunes a learned hand motion model with RL using contact forces and a curriculum on imitation error thresholds. Like DexMachina, it uses demonstrations and gradually incorporates them, but it was originally tested on much simpler single-object, short-horizon tasks. For fairness, the authors reimplemented ManipTrans’s ideas in their setup (same simulator, same hybrid actions). Including it checks whether DexMachina’s VOC is uniquely helpful. Each baseline was carefully reimplemented in the same Genesis simulation framework to ensure a fair comparison. All had access to the same demonstration data and training compute. Results: Performance on Long-Horizon Tasks. Across the board, DexMachina outperformed the baselines on these bimanual tasks. Figure 3 in the paper (reproduced conceptually here) shows averaged success rates for each hand-task. Several clear trends emerged:. Direct Retargeting Fails: The kinematic-only baseline almost always fails to solve the task. As expected, simply playing back the human motion in simulation often yields no success (the object never makes it to the final state). The physics simply doesn’t cooperate without control feedback. Thus learning is essential. Task-Only RL Struggles on Long Tasks: The ObjDex baseline (RL with only the object-tracking reward) can sometimes limp through very short tasks, but it breaks on the longer sequences. For example, on multi-step tasks like the waffle iron demo or flipping mixer demo, the vanilla RL policy almost always fails early (e.g. dropping the object). Without additional guidance, the policy finds only trivial or unsafe strategies (like not lifting the object properly) and gives up. In contrast, DexMachina’s policy, with its VOC and imitation shaping, manages to navigate these sequences without the “early failure” collapse. Auxiliary Imitation Helps a Bit: The version with task+aux rewards but no curriculum did better than task-only on some tasks. The imitation clues do give the policy some foothold (it roughly follows human-like motions). However, as the authors point out, the gains are inconsistent. In many long-horizon cases it still falls short. Only when the VOC curriculum is added does training reliably succeed on the hard tasks. DexMachina Excels, Especially on Hard Tasks: With the full curriculum, DexMachina achieved the highest success rate in virtually every hand-task pairing. On short tasks, all methods get modest scores (e.g. 70–90% object-tracking AUC). But on the 4 long-horizon tasks, DexMachina’s advantage is dramatic. The figure highlights those columns: tasks like Waffleiron-300, Notebook cover opening, Mixer lid flipping, etc. In these, DexMachina policies (blue bars) far exceed any baseline. The authors report that in one case a task that ObjDex almost never solved was achieved nearly 80% by DexMachina. The “Waffleiron-300” example is illustrative: that clip requires picking up a waffle iron, opening the lid, flipping it, and reopening the lid—all while in the air. Task-only RL simply gave up before even opening the lid. With DexMachina’s help, the learned policy performs every step correctly. Statistics: Averaging across all tasks and hands, DexMachina’s success rate was by far the best. The performance gap was especially pronounced on the right half of the chart (long tasks): where ObjDex and no-curriculum variants often left the task undone, DexMachina completed the sequence. This confirms that the curriculum extension is the crucial ingredient for these hard scenarios: with identical reward terms but a curriculum that temporarily moves the object, the policy avoids getting stuck early and steadily learns the full sequence. To summarize the quantitative results: DexMachina consistently improved performance on all hands and tasks, with especially clear gains on long-horizon, complex-motion tasks. The improvements are robust (across multiple random seeds) and not beholden to hyper-tweaks. It learned each task efficiently, whereas the baseline policies often failed to make progress. Policy Behavior and Hardware Adaptation. Beyond raw success numbers, DexMachina produces sensible behaviors that adapt to the robot’s capabilities. Qualitatively, policies learned to achieve the goal in human-recognizable ways where possible, and find creative workarounds when a direct replay of the demo isn’t feasible. For example, consider the Notebook Cover task. The human demo showed opening a laptop cover by having the left hand lift one side and the right hand push down. The XHand robot (a fairly dexterous hand) learned a similar strategy: one hand supports the notebook while the other closes the lid. But for the smaller Inspire hand (with fewer actuated joints), the same approach fails (it has less reach and less grip strength). DexMachina’s policy found a different strategy: it uses both hands to stabilize and gradually close the cover, rather than imitating the left-right split. In other words, with the same demonstration reference, the agent learned a hand-specific solution. This showcases the approach: the object (Notebook) ended up closed in both cases, but the policy’s chosen hand motions differ sensibly to match hardware limits. A similar phenomenon appears in the Mixer task. The Allegro hand has a long, flexible thumb, so its policy grasps the mixing bowl and simply uses the thumb to push down and shut the lid. The Schunk hand, however, has stubbier fingers; it instead shifts its whole palm and wrist to close the lid. Again, DexMachina allowed each robot to exploit its strengths while still achieving the same end result. These examples underline one of the paper’s motivations: a single algorithm and demo can span a variety of hardware. The learned strategy naturally adapts. The authors take this further by doing a hardware comparison analysis. Using DexMachina as a level playing field, they evaluated all six hands on the long tasks to see which hand designs perform best under the same conditions. The outcome was intuitive but instructive: the largest, most fully-actuated hands (DexHand, XHand) reached the highest success and learned fastest. In fact, the key factor was degrees of freedom more than sheer hand size: both the Schunk and XHand have more finger joints and they significantly outperformed the Inspire and Ability hands of similar overall size but with fewer actuated digits. This suggests that compliant, highly-articulated hands truly shine on these tasks. The evaluation framework they built allows designers to quantify such differences using a common reference task set – a valuable tool for guiding future hardware development. Finally, on the simpler, short-horizon tasks (for example, just closing a box or sliding a lid), all hands eventually reached near-ceiling performance (70–90% on the tracking metric) under DexMachina. This indicates the algorithm is not over-specialized to only the hardest tasks. The learning curves of those shorter tasks show that even the least capable hand could succeed when the task demands were lower; in those cases, having the VOC is helpful but not strictly necessary. Discussion and Conclusion. DexMachina represents an interesting fusion of ideas in RL, imitation learning, and curriculum learning. Rather than requiring many demonstrations or hand-engineered rewards, it extracts all the guidance it needs from one human video and then tackles the rest with RL. The core insight – the virtual object controller – is a relatively simple trick, but it makes a huge practical difference for these difficult tasks. By allowing the object to “move itself” initially, the policy avoids burning valuable exploration on first-step failures. As the object controller fades, the policy gradually shoulders the workload. The name DexMachina is thus apt: like a deus ex machina, an external force steps in to make a problem solvable, then withdraws to let the hero (the policy) finish the task. The work also provides a valuable new benchmark. By combining ARCTIC demos with multiple robot models, the authors created a testbed that measures both algorithm performance and hardware capability. DexMachina significantly outperforms several prior approaches (e.g. naive RL, direct imitation, and concurrent methods like ManipTrans) across this benchmark. This demonstrates the method’s strength. Importantly, the code and environments are released open-source, so others can build on this foundation. For future work, one could imagine applying DexMachina to real robot hands by sim-to-real transfer. The policies here were learned in simulation, but the strategy of virtual controllers could potentially be adapted to curricula in the real world (for example using gravity compensation or guided motion springs during initial phases of training). Since the framework handles any hand, it could also test new robotic hand prototypes against human skill. In summary, DexMachina shows that a single human demonstration can bootstrap dexterous RL for long, complex tasks. It generalizes across hands and objects, handles intricate contact sequences, and significantly lowers the barrier to training dexterous policies. As robotic hands continue to evolve, algorithms like DexMachina will be crucial for harnessing their potential in a flexible and scalable way. References: DexMachina and related background were drawn from Zhao et al., "DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation" (ICML 2026) (and sources therein), and the ARCTIC dataset of bimanual hand-object manipulation. The DexMachina code and benchmark have been released online by the authors. All citations are drawn from these primary sources as indicated.