Latest / Embodied AI 101 / DexWM: Learning Dexterous Object Manipulation from Human Videos

Transcript

DexWM: Learning Dexterous Object Manipulation from Human Videos · Embodied AI 101

• DexWM: Learning Dexterous Object Manipulation from Human Videos. Dexterous manipulation – controlling a multi-fingered robotic hand to pick up, move, and place objects – is one of the grand challenges of robotics. It requires understanding the subtle physics of contacts and hand-object interactions, and demands very fine-grained control that typical coarse action spaces (like a gripper position or language command) struggle to provide. Even state-of-the-art methods falter when trying to generalize to new objects or goals, especially because of the lack of large datasets of human–robot manipulation. DexWM (Dexterous Manipulation World Model) is a new approach tackling this bottleneck by leveraging large-scale human video data to train a world model for dexterous tasks. Introduced in a 2025 preprint by Goswami et al., DexWM is a latent-space predictive model that learns how the scene evolves under multi-finger hand motions. Crucially, the authors pre-train on over 900 hours of human and non-dexterous robot video – including hundreds of hours of richly annotated egocentric human manipulation footage – and then fine-tune on a small amount of robot data. This combination turns out to be surprisingly powerful: even though the model only sees unlabelled camera video during training, it learns to predict not only how the object will move, but also how the hand should re-configure – enabling it to plan and control a real robot hand zero-shot on unseen tasks. In practice, a DexWM model coupled to a simple planner consistently outperforms baselines like diffusion-based imitation, achieving up to 83% success on a variety of real grasping, placing, and reaching tasks without any task-specific demonstrations. Below we’ll dig into how DexWM works, its training recipe and data, and its performance on challenging tests. We’ll see how this system brings advances in world models, leveraging modern vision and pose encoders, clever loss functions, and the treasure trove of passive human egocentric video to push the frontier of dexterous robotics. The Opportunity of Human Video for Robots. Robots traditionally learn to manipulate objects via either model-based control or policy learning from real robot rollouts or scripted demonstrators. Collecting such data – while robots physically grasp and move objects – is slow, expensive, and limited in diversity. Humans, by contrast, are extremely good at complex in-hand dexterity (like tying shoelaces, folding laundry, handling utensils), and they perform billions of manipulation operations in everyday life. If a robot could “learn from watching” humans, it might bootstrap its understanding of hand-object physics far beyond what limited robot datasets allow. This idea has been floating around for a while – e.g. imitation learning from video or tele-operation – but DexWM takes it into a new regime by learning a world model. Instead of trying to directly map human motions to robot joint commands, DexWM uses hundreds of hours of human-centric video to train a model of how the world changes given hand motions. In particular, DexWM predicts changes in a latent space of visual features conditioned on hand-keypoint actions. This is a powerful twist: once you have a good dynamics model in a latent space, you can do image-based planning with any embodiment (like a multi-fingered robot hand) by optimizing actions to reach desired latent outcomes. To make this concrete, the DexWM authors collect two main sources of data:. EgoDex (Human Egocentric Videos). This is a new large-scale dataset of humans performing everyday manipulations from a first-person camera. Using Vision Pro headsets, Apple collected 829 hours of video spanning 194 kitchen and household tasks (tying shoes, opening containers, etc.), with full 3D hand pose tracking for each frame. This dataset gives DexWM a wealth of forward dynamics: how do objects (and the hand itself) move in response to high-dimensional hand motions. DROID (Robot Manipulation Videos). Complementing the human data, DexWM also uses the DROID dataset – a 100+ hour benchmark of real robot manipulation with a Franka Panda arm and workspace camera (feat. human demonstrations and teleoperated episodes). The key is DROID’s diversity of grasping and placement in varied home environments. While the Panda in DROID has only a parallel-jaw gripper (not a full hand), the DexWM framework treats its gripper motions as “dummy” hand keypoints (explained below) to incorporate robot data into the same latent action space. By pre-training on ~900 hours of such video, DexWM effectively learns the physics of hand–object scenarios in the latent DINOv2 feature space (see below). As the authors note, “training on both human video and even a small amount of robot footage substantially improves prediction accuracy and keypoint tracking”. Intuitively, the human data injects finesse (multi-finger detail) and scene context, while the robot data helps ground the model in the kinematics of the robotic arm. The idea of world models – learning predictors of future sensory observations – is not new. But most work in vision and robotics has either used synthetic simulation (for example “soccer ball” simulations) or coarse control (like goal images or language guiding motions). DexWM stands out by combining real-world visual data with fine-grained hand joints. As described by its authors: a DexWM model “predicts the next latent state of the environment conditioned on past states and dexterous actions”. It explicitly uses action-conditioned latent transitions, a technique from recent world-model literature, but in a context where the actions are hand gestures. Since the action space is high-dimensional (42 hand keypoints plus camera motion), this demands a powerful sequence model; DexWM’s answer is a transformer-based “Conditional Diffusion Transformer” (CDiT). We’ll look at architecture next. DexWM Architecture: Latents, Transformers, and Hand Keypoints. At its core, DexWM is a latent-space dynamics model. Each video frame (from an egocentric RGB camera) is passed through a frozen DINOv2 encoder, a self-supervised model that produces patch-level image embeddings. Concretely, an input image (I) is encoded to a latent (s = E_\phi(I)\in\mathbb{R}^{P\times d}), where (P) is the number of equal-area patches (e.g. 448 patches) and (d) the feature dimension per patch. These DINOv2 features have been shown to generalize well across scenes, and provide a semantically rich representation of the camera view. By stacking a sequence of past latents ((s_{t-k}, \ldots, s_t)), DexWM’s predictor can learn how (s_{t+1}) should evolve given a chosen action. The action representation is the novelty that turns DexWM into a dexterous world model. Instead of commanding the robot base or end-effector position, DexWM’s actions (a_t) are formed from 3D hand key-points in space. Specifically, it uses the MANO hand model to represent the pose of both the left and right hands in 3D (21 joints each, 42 points total). Between two successive frames, the change in these 42 keypoints (\Delta H = H_{t+1}-H_t) (expressed in a common coordinate frame) encodes the detailed dexterous motion of the hands (finger flexing, wrist rotation, etc.). DexWM concatenates (\Delta H) with the change in camera translation/rotation to form its action vector. The authors emphasize that even the parallel-jaw gripper data from DROID can be fit into this scheme by using dummy fingertip points based on the gripper’s pose. This way all data – human or robot – can be treated as “hand motion” in the same latent dynamics model. The predictor (f_\theta) in DexWM is essentially a Conditional Diffusion Transformer (CDiT) – a transformer model adapted from recent image diffusion architectures. During training it learns to regress the next latent state (\hat s_{t+1}) given (s_{t-k:t}) and (a_t). In other words, it tries to make (\hat{s}{t+1} = f\theta(s_{t-k:t}, a_t)) match the actual encoded next latent (s_{t+1}). Crucially, the ‘conditioning’ of the transformer is done via adaptive layer norms (AdaLN) on the action vector, so the hand motion influences each transformer block. During inference (planning), DexWM rolls out many steps autoregressively: it feeds back its own predicted latent to generate further predicted frames. To ensure the latent predictions capture the fine details of the fingers (which occupy a small image area), DexWM also uses an auxiliary hand-consistency loss. Besides the usual MSE on the latent output, it attaches a small transformer “head” (g_\theta) predicting heatmaps of fingertip/wrist locations from the predicted latent. These predicted heatmaps (\hat{V}) are compared against ground-truth hand keypoint images (V), with a loss (L_\text{HC} = |V - \hat{V}|^2). As the authors found, this guided training dramatically improves hand/finger accuracy (by ~34% in PCK@20) over training on latents alone. Without this term, the world model could predict coarse object motion but often miss recoordinating the fingers precisely. In summary, DexWM’s “mechanics” are: encode images with DINOv2 → extract compact latents; represent actions as detailed 3D hand pose differences; and train a diffusion-transformer to predict future latents, with an extra heatmap supervision to preserve finger detail. The resulting model is truly hand-centric. As the authors note, “fine-grained hand conditioning yields much more precise goal achievement than text-conditioned or navigation-conditioned world models” (e.g. prior systems that used language goals or astronaut-like full-body motion). In their tests, DexWM’s direct hand-motion control produced sharper and more accurate manipulations than those other world-model baselines. (Figure 7 in the paper shows, for example, that text-conditioned models maintain only gross object layouts, whereas DexWM precisely tracks the required fingertip contacts.). Training Data: EgoDex, DROID, and Simulated Exploration. A critical ingredient for DexWM is scaling up the training data. Prior work on robotic world models was often limited to tens of hours, typically from simple tasks or simulation. DexWM breaks that mold by assembling hundreds of hours of new data. Thanks to the availability of EgoDex and DROID, pre-training is done on roughly 829 hours of human video + 100 hours of robot video, for a total near 900 hours (the exact number varies slightly by source, but this is the cited scale). EgoDex (829h): As introduced by Hoque et al. (Apple Research), EgoDex is currently the largest available dataset of natural human manipulation. The data is egocentric video (camera on the head/helmet), covering 194 distinct tasks with everyday objects. Critically, it includes the tracked 3D pose of each hand at filming time, captured via multiple cameras and SLAM on device. Thus each frame comes with ground-truth MANO parameters (and fingertip coordinates). DexWM simply uses the keypoint outputs, not the full mesh or velocities. But EgoDex’s diversity – tying shoes to cooking to laundry – gives the model a rich library of how objects and hands co-evolve under various interactions. DROID (100h): To anchor the learned model to actually controlling a robot, DexWM incorporates the DROID dataset from Stanford et al. DROID (“Distributed Robot Interaction Dataset”) consists of tele-operated episodes and demonstrations across many tasks (kitchen, bathroom, etc.), collected on a mobile setup with a Franka Panda arm. The dataset includes the robot’s joint angles and stereo camera observations, but for DexWM’s purposes only the end-effector (gripper) pose and wrist camera views are used. The authors ingeniously convert the Panda’s parallel-jaw gripper movements into synthetic “hand keypoints”: imagine two virtual fingers at the grip edges that open/close and move with the gripper. These approximated keypoints let the DROID data blend into DexWM’s MANO-based framework. The result is that DexWM sees hundreds of hours of robot-centric manipulation after learning general skills from humans. RoboCasa (4h): Finally, after this massive video pre-training, DexWM is fine-tuned on a small amount of robot-specific data in simulation. The authors use RoboCasa (a kitchen-domain simulator with a Panda + Allegro gripper model) to collect ~4 hours of random “exploratory” motion – the robot moving around without any specific tasks. This tuning step helps align the latent predictions with the precise kinematics of the Panda/Allegro system, smoothing the final sim-to-real transfer. Crucially, it still doesn’t involve any labeled tasks or demonstrations – just random joint motions in the robot environment (the DexWM code can back-project these joint angles into the same action space of 3D keypoints). In training, all data is mixed. DexWM’s loss treats human and robot sequences equivalently (besides using dummy keypoints for non-human data). The human and robot videos are all sampled in short overlapping clips (e.g. 4-10 frames) where a single action is performed. The model sees, for example, a person moving their fingers to push an object, or a teleoperated Panda opening its gripper on an object. It learns to predict the very next frame’s latent from the last few frames and the hand motion in between. This “digital twin” of the physics, learned offline, then becomes the heart of DexWM’s planning. According to the ablations reported, including human video made a significant difference. A DexWM trained without EgoDex (using only DROID) still learned something about object dynamics, but its accuracy on both latent L2 error and keypoint pose prediction (PCK) dropped sharply. In contrast, the full model (human+robot) achieved much better tracking of the finger positions and state prediction. Moreover, scaling the transformer size from tens of millions to 450M parameters consistently improved performance, indicating that the problem benefits from more capacity. The auxiliary hand consistency loss was also shown to be “crucial” – when removed, fingertip accuracy plunged, highlighting that some supervision on the actual hand was needed beyond the latent autoregression (even though the latent features encode much of the scene, the hand detail requires explicit encouragement). In the end, DexWM emerges as a world model that has seen a thousand hours of manipulation – both human-intuitive motions and grounded robot kinematics. This wide “experience” allows it to make plausible predictions about unseen tasks, as we now discuss. Planning and Control with DexWM. Once trained, the DexWM model can be leveraged for control and planning. The key idea is to use it as a forward dynamics model within a model-predictive control (MPC) loop. Concretely, given a start image and a goal, the system searches for a sequence of future hand actions that will make the latent reach the goal latent. Here’s roughly how it works:. Encode initial and goal: Say we have an image of the current state (I_\text{start}) and a target image (I_\text{goal}). We encode both into latents (s_\text{start}) and (s_\text{goal}) via the DINOv2 encoder. MPC search: We use the Cross-Entropy Method (CEM) over action sequences. Initially sample many random sequences of joint-angle trajectories (\Theta_{0:T-1}) (the Panda’s 23 joint angles before each timestep). For each candidate, we simulate it in DexWM by converting joint angles to the equivalent hand-keypoint change (a_t) (via forward kinematics, doing the dummy keypoints). Then we roll out the DexWM predictor over (T) steps, getting a predicted latent (\hat s_T). Cost evaluation: Each rollout is scored by how close (\hat s_T) is to the goal. The authors minimize a cost (C = | \hat s_T - s_\text{goal} |^2 + \lambda |\text{Keypoints}(\hat s_T) - \text{Keypoints}(s_\text{goal})|^2), i.e. distance in latent space plus a term for matching keypoint poses. We don’t see the exact equation in the paper text we have, but the idea is latent distance plus a small penalty forcing the fingertips to goal positions (with weight (\mu=10^{-3}) for example in [51]). Iterate: The top-performing action sequences (lowest cost) are kept to fit a Gaussian, and then new sequences are sampled around that distribution (standard CEM). This repeats for several iterations, narrowing in on the best plan. Execute: Finally, the best segment of joint angles from the search is sent to the real robot controllers. Then (optionally) the process repeats from the new state, in an MPC fashion. Notably, this is an image-based planning loop. The DexWM model unrolls latents (decoded to images for debugging) but the actual decision criterion is in latent feature space, not pixel error. By planning in the learned latent, DexWM can “dream” plausible futures under hypothetical hand motions (indeed, the authors show examples of counterfactual simulations: “what if I move my hand up instead of down?”). Because DexWM is trained on real physics, its simulated futures tend to reflect real contact dynamics: if the hand touches the cup in the model, the cup moves similarly to reality (see DexWM’s open-loop rollout figures). Contrast this approach with action-synthesis policies like Diffusion Policy or Reinforcement Learning. DexWM doesn’t directly output a policy π. Instead it provides a predictive model, and planning solves for the best actions on-the-fly. This has pros and cons: it’s flexible (you can replan to any new goal without retraining), but it can be slower at runtime (sampling many trajectories). The authors explicitly note that the planning can be computationally intensive (hundreds of CEM samples), and might not scale to extremely long horizons without improvements. But for the short-horizon tasks studied (few seconds of motion), model-predictive sampling works fine. Baselines and Competitors. It’s instructive to compare DexWM’s approach with other recent methods. Many prior world-models or video-prediction frameworks in robotics used coarse action representations, like moving the robot base or end-effector, or even language/text commands describing goals. For example, the Cosmos/Predict2 world models condition on a high-level text action (e.g. “place the mug on the table”), or models like PEVA use full-body human poses to drive motion. Such methods can capture broad motions, but they lack fine hand detail. As the authors show, DexWM vastly outperforms those for dexterous tasks. In experiments where a reference human grasping trajectory is transferred to a robot, models like PEVA (using only upper-body pose) “produce plausible but less physically grounded hand-object interactions” and fail to precisely track the fingertips. In contrast, DexWM’s direct hand conditioning yields much tighter fingertip-level tracking (Figure 8 in the paper). In other words, giving the model exactly the desired finger motions lets it mimic the desired grasp much more accurately. Diffusion Policy is another relevant baseline. This method trains a diffusion model to sample robot action sequences conditioned on video observations. Diffusion Policy has been very successful in imitation-learning tasks (outperforming many prior RL methods). DexWM compares to it in the real-world tests: in grasping tasks, DexWM achieves 83% success versus Diffusion Policy’s roughly 30% (i.e. DexWM outperforms by ~50 percentage points). Importantly, DexWM had no task-specific training – it just used the start & goal image – whereas Diffusion Policy was trained on relevant demonstrations. So DexWM’s latent model plus CEM planning ended up more robust than a fully-trained diffusion policy in these tests. In summary, DexWM offers an alternative to pure imitation or policy learning: it learns a dynamics engine from vision and then plans within it. This aligns with a growing trend that sees world models (learned from large offline data) as a flexible substrate for many tasks, including zero-shot or few-shot generalization. DexWM is the first to show this works for unconstrained dexterous hand manipulation with real physical objects, using essentially only vision and hand/motion priors. Results: Qualitative and Quantitative Demonstrations. The paper provides a range of simulation and real-robot experiments to validate DexWM’s capabilities. Here are the highlights:. Open-loop rollouts. In simulation, DexWM can predict entire sequences of future frames given an initial image and a fixed action sequence. The authors show side-by-side comparisons of predicted images vs ground truth (from DROID or RoboCasa data). The predicted frames – including object positions and hand shape – stay remarkably close to the real frames over several seconds (see figures). This is a sanity check that DexWM did learn something about physics. It even correctly predicts collisions: if the model is told to “move hand forward into cup”, the simulated cup moves realistically, whereas “move hand upward” leaves the cup in place. Such counterfactual scenarios demonstrate the learned causality in the model. Action transfer. One key test was: “take a reference human grasp trajectory, and ask the robot to do the analogous action on a new object or scene.” DexWM plus planning could transfer a fine grasp from a recorded human demonstration to a novel setup. Qualitatively, it maintained the subtle finger posture. In contrast, a comparator world-model (like PEVA) that had only seen a high-level body pose had trouble. The DexWM results looked much crisper in terms of end-effector and fingertip motion. Goal-conditioned tasks. The authors set up simple tasks (in simulation) like “reach this position”, “grasp this object”, or “place the object at this target” – clearly unrelated to the training tasks. They provided a start and goal image, and let CEM DexWM planning run. DexWM succeeded in the majority of these trials, finding reasonable joint trajectories. The paper reports notably higher success rates compared to text- or coarse-model baselines. (Exact numbers are not provided for each task, but the qualitative demos show robust behavior across new object shapes and positions). Sim-to-real grasping. The biggest splash is the real robot test. With minimal additional tuning (just 4 hours of robot sim data, as mentioned), the DexWM model was deployed on a Franka Panda arm with an Allegro 4-finger hand. The task was generic grasping: given an arbitrary mug on a table, plan a grasp from a camera view image. On a set of unseen objects, DexWM achieved an 83% grasp success rate. By contrast, a Diffusion Policy method (trained on grasping examples) managed only ~30% on the same tests. The DexWM also beat its own variant that had been trained only on the robot data (no human), underscoring the value of the human pre-training. In summary, the experimental results tell a consistent story: DexWM’s visual predictions are accurate, and when harnessed for planning, they translate into real-world successes. It handles variations in object geometry and position robustly, even though it never saw those particular objects during training. This is strong evidence of zero-shot generalization from human video to robotic manipulation. The most telling metric is the jump in success rate on real tasks (+50 points over Diffusion Policy) – in practical terms, DexWM does half again as well at grasping than a previous method, without using any hand-crafted heuristics or extra labels. Limitations and Future Directions. As with any new approach, DexWM has limitations to acknowledge. The authors themselves note that the planning step is computationally expensive, since sampling thousands of rollouts through a 450M-parameter transformer is not light-weight. For real-time or very long-horizon tasks, this could be a bottleneck. Potential remedies include distilled policies, gradient-based planning, or learned dynamics with smaller networks. Another constraint is scope of behaviors. DexWM currently deals with single-arm, single-hand tasks in tabletop settings. Bimanual tasks (using both hands) or mobile manipulation (with wheels) are not addressed yet. The world model is also “blind” to forces – it only sees visual cues. Pushing objects, twisting lids, or activities that require force sensing might need extensions (perhaps grounding the model with contact physics or tactile data). And all training video was indoor/manipulation scenarios; none of DexWM’s latent model is guaranteed to generalize to, say, disassembly tasks or dexterous cloth folding without more data in that domain. On the research side, it raises interesting questions: could we do away with online planning entirely, by distilling DexWM+MPC into a single-shot policy network? Could language or vision-language models define higher-level goals for DexWM planning? The authors suggest combining DexWM with hierarchical planning or goal-conditioned VLMs. Also, would training on even more human video (or videos of skilled robot hands, e.g. from animation or VR) push performance further? DexWM demonstrates that large-scale passive data is valuable – likely future work will tap into internet-scale vidéo of hands (if annotated) or synthetic demonstrations to scale up. Finally, DexWM blurs the line between “digital twin” simulators and data-driven learning. Rather than hand-crafting a physics engine for finger contacts, it learns the dynamics from data. This paradigm of learned world models may soon compete with or replace classical simulators in certain domains. The DexWM results suggest that with enough data, and the right architecture, learned similers can rival handcrafted ones for at least tensile tasks. Given how successful this world model is, we might see more robotics shifting to this “learned physics” approach. Conclusion. In short, DexWM is a compelling step towards robots that can learn general-purpose dexterous skills from watching people. By training a large latent dynamics model on hundreds of hours of human video (augmented with some robot data), it achieves nuanced hand-object control with zero task-specific training. The work underscores that “world models” can indeed leverage human demonstrations at scale, filling in the data gap that has long stalled progress in robotic in-hand manipulation. For researchers, DexWM offers a dataset of 829h hands+100h robot on Hugging Face and an architectural recipe combining vision transformers, diffusion models, and keypoint conditioning. For the future of robotics, it paints a picture where everyday videos on YouTube or AR/VR glasses could be turned into training ground for next-generation robot hands. References: The DexWM approach and results are documented in Goswami et al., “World Models Can Leverage Human Videos for Dexterous Manipulation” (2025). Its training data includes EhterNet’s EgoDex (829h human manipulation فيديو with 3D hand poses) and the DROID robot dataset (100h robot manipulation). The model uses a DINOv2 encoder and learns fine hand details via a hand-consistency heatmap loss. We refer readers to that article for full experimental details; here we have summarized the key ideas and outcomes relevant to embodied AI and world modeling.