Latest / Embodied AI 101 / DexWM: Leveraging Human Videos for Dexterous Robot World Models
Transcript
- DexWM: Leveraging Human Videos for Dexterous Robot World Models. Dexterous manipulation—getting a robot hand to deftly pick up, orient, and place objects with millimeter precision—has long been a coveted goal in robotics. Unlike coarse tasks (grasping with a simple gripper or moving an arm), fine finger motions require understanding subtle contact dynamics and complex coordination. The bottleneck is often data: recording enough diverse examples of multi-fingered manipulation to cover all the edge cases takes tremendous effort. Meanwhile, humans perform such tasks effortlessly every day. The new DexWM approach bridges this gap by training a world model on massive human video datasets, then using it to plan dexterous robot actions without task-specific robot data. In the recent paper “World Models Can Leverage Human Videos for Dexterous Manipulation” (Goswami et al., arXiv 2025), the authors assemble a world model, DexWM (Dexterous Manipulation World Model), that is pre-trained on over 900 hours of human and robot videos. This latent-space model predicts future states conditioned on high-fidelity hand actions, and when plugged into a model-predictive planner it enables a Franka Panda arm with an Allegro five-finger hand to grasp and manipulate novel objects in a zero-shot manner. In experiments, DexWM greatly outperforms prior imagery-based baselines—achieving roughly 83% success on a suite of real-world grasping tasks with no extra robot training. In short, DexWM demonstrates that existing human manipulation video can be repurposed to train a robot world model for dexterity, avoiding the need to record a huge new robotic dataset. In this article, we delve into the DexWM dataset and method. We explain how the authors built training data from human egocentric video and robot recordings, how they represent states and actions in a shared latent space, the specialized neural architecture they use, and how the model is deployed for planning robot motions. We also survey the reported results and implications. Throughout, we emphasize the core insight: rather than collecting hundreds of hours of robot hand demonstrations from scratch, we can learn the physics of manipulation from watching people and then transfer that understanding to a robot hand. This approach relies on a powerful latent world model and careful action/state alignment, but if it holds up it could open a new path for data-efficient dexterous learning. Data-Driven World Models for Dexterity. World models are neural predictors of how an environment changes in response to actions. In robotics, a world model is essentially a learned simulator: given the current state and a proposed action, the model forecasts the next state of the world. Once trained, such a model can be plugged into planning or controller design (e.g. by search or optimization) rather than learning a policy end-to-end. World models have been used for navigation and simple manipulation, but dexterous manipulation adds a twist: the actions are high-dimensional (many finger joints) and the interactions are subtle and contact-rich. The DexWM team recognized two core challenges in applying world models to dexterity. First, high-quality data of multi-fingered hand-object interactions is scarce: few labs have robots with 20+ DoF hands collecting large datasets. Second, typical world-model approaches use coarse action commands (like “move hand to position (x, y, z)” or “hand open/close”), which can miss the nuance of fingers curling, rotating, or grasping. DexWM’s key idea is to leverage human videos as a much larger source of data, and to align their action representation with what a robot hand would do. Humans interact with objects constantly, and large egocentric video datasets can be harvested at scale. For example, the recently released EgoDex dataset provides 829 hours of first-person video of people manipulating household objects, with every frame annotated with 3D hand and finger positions. By tapping into EgoDex, DexWM gets orders of magnitude more demonstration data for fine hand motion than any existing robot dataset. Complementing that, the authors also employ the DROID dataset, a recent large-scale archive of robot manipulation. DROID (“Distributed Robot Interaction Dataset”) contains 76,000 demonstration trajectories (≈350 hours) of a Franka and parallel-jaw gripper performing everyday tasks across hundreds of scenes. For DexWM, about 100 of those hours are used to help align the model to robotic gripper embodiments. Finally, the model is fine-tuned on a few hours of synthetic data in the RoboCasa simulator to familiarize it with the robot’s geometry. This hybrid data mix—hundreds of hours of human video plus robot data—forms the training ground for the world model. The results of this data strategy are striking. DexWM is pre-trained on roughly 900 hours of mixed video. That includes ∼829h of EgoDex human video and ∼100h of DROID robot video. In this way, the model first learns general principles of hand-object physics from the human data, then gets oriented to the robot’s “body” from the robot videos. Because EgoDex uses a fully-announced hand model (21 joints per hand) with a stereo camera rig and on-device SLAM, it gives dense, accurate 3D keypoint tracking. Each EgoDex frame shows a hand interacting with objects (tying shoes, folding laundry, grabbing cups, etc.), often with rich surroundings. Besides the raw visual frames, DexWM leverages the associated 3D hand poses: these form the “actions” in training (as we will explain). DROID’s footage, while using a simpler gripper, provides robot-specific contexts (chair scenes, drawers, appliances) that help mitigate the gap between human and robot perspectives. Even a few hours of randomized robot motion in simulation (RoboCasa) help “calibrate” the model to the Panda/Allegro body. In total, DexWM sees a massive diversity of hand-object interactions—far beyond the scale of any conventional robot demo collection—and uses this to train a single latent-space predictor of future states. State and Action Representation. At the core of DexWM is a latent state-space model. The raw inputs are egocentric RGB frames from the robot’s point of view (e.g. a camera on the Panda arm). Instead of predicting raw pixels, DexWM encodes each image into a structured latent representation that captures semantic content while being more compact than pixels. Concretely, each image $I$ is passed through a frozen DINOv2 encoder (a state-of-the-art vision transformer). DINOv2 processes the image into a grid of patch embeddings: for example, a $H\times W$ image might be divided into $P$ patches, each mapped to a 1024-dimensional feature vector. The result is a latent state $s \in \mathbb{R}^{P\times d}$ (with $d=1024$) that summarizes the scene. This encoder was not trained on robot data; it’s a large self-supervised model known to generalize well to scene understanding. Using it provides DexWM with semantic features of the world rather than raw pixels. When training the world model, DexWM must predict how $s_{t+1}$ follows $s_t$ given an action $a_t$. What is an “action” here? DexWM opts for an extremely detailed action representation: it encodes motion of the hand (or gripper) in 3D space. Specifically, for each time step, the action is defined as the difference in 3D keypoint positions of the hand between two frames. EgoDex provides 42 hand keypoints (21 per hand) in camera coordinates at each time. If $H_t \in \mathbb{R}^{42\times3}$ is the set of 3D points at time $t$, then the raw hand action is $H_{t+1} - H_t$ (a 42×3 vector). This single vector captures exactly how each fingertip and wrist moved in 3D from one frame to the next. In addition, the action includes the change in camera (i.e. head) pose: a 3-vector for translation and 3-values for Euler-angle rotation $\delta q$. All together, the action vector is $$a_{t}= \bigl(\Delta H_t,; \delta t_t,; \delta q_t\bigr) ;\in; \mathbb{R}^{(42+2)\times3} \approx \mathbb{R}^{(44\times 3)} \approx \mathbb{R}^{132},$$ where $\Delta H_t = H_{t+1}-H_t$. In plain terms, DexWM’s “controls” tell exactly how each of 42 joints moves, plus the camera movement, between frames. This fine-grained representation is orders of magnitude richer than a single “gripper open/close” flag; it can encode curling fingers, wrist rotations, finger splay, and so on. One challenge: DROID robot data uses a parallel-jaw gripper, not a human hand. To reconcile this, the authors map the gripper’s two jaws onto “dummy” fingers in the same keypoint scheme. In practice, they place two artificial keypoints on the gripper’s jaws so that a closing motion becomes two fingertips moving symmetrically. Thus all actions—whether from human video or gripper video—look uniform: a set of keypoint displacements plus camera change. This unified view bridges the embodiment gap: the model treats robot gripper motion (with dummy points) and human finger motion in the same framework. In effect, the dataset becomes tokenized as (latent state, action vector) pairs for both humans and robot clips. To summarize, DexWM’s training data consists of sequences of latent states $s_t = E_\phi(I_t)$ (DINOv2 embeddings of images) and actions $a_t$ (3D keypoint deltas plus camera motion). Each training example might be a short trajectory segment $(s_t, a_t, s_{t+1})$. The world model learns to predict $s_{t+1}$ from past latents and the current action. Crucially, because the action includes every individual finger movement, the model must account for the local object interactions micro-detail. This representation strategy provides the precision needed for true dexterity. Model Architecture: Conditional Diffusion Transformer. With data in hand, the next question is how to build the model $f_\theta(s_t, a_t)$ that does the prediction. The DexWM paper opts for a transformer-based backbone borrowed from recent diffusion models for images. Concretely, they describe using a Conditional Diffusion Transformer (CDiT): a deep transformer that models transitions in latent space conditioned on actions. The model has the following structure:. Sequence input: The model is given a history of recent latent states $s_{t-n},\dots,s_t$ (each of shape $P\times d$ flattened to a sequence of patch tokens) along with the current action $a_t$. Action conditioning: Actions are injected into the transformer via AdaLN layers (adaptive layer norm). Essentially, the keys and queries in each transformer layer are modulated by the action vector. This makes the future state prediction explicitly dependent on $a_t$. Prediction output: Unlike image diffusion models that produce a noise map, here the transformer directly regresses to the next latent $s_{t+1}$. In practice, DexWM’s model outputs one predicted patch-embedding vector per patch. That is, it outputs a matrix $\hat{s}_{t+1} \in \mathbb{R}^{P\times d}$. Notably, although they call it a “conditional diffusion transformer,” the inference is not an iterative denoising process. Instead the model is trained with a mean-squared-error loss on the latent vectors, and at test time it does one feed-forward pass. (Calling it “diffusion” likely refers to deriving from diffusion-style architectures, even if the diffusion sampling is skipped for efficiency.) The transformer is large – one variant “DexWM-XL” has 456 million parameters – reflecting the scale of the problem. In short, $f_\theta$ is a heavy transformer that learns to carry forward latent scene features via multi-head self-attention, with explicit conditioning on high-resolution hand actions. Because it operates on a rich semantic latent (rather than raw pixels), it can focus on key features of the environment and hand. Training Loss: Hand-Consistency Loss. Training such a model requires a loss that emphasizes the right things. The obvious base loss is to make the predicted latent match the DINOv2 embedding of the real next image: $$\mathcal{L}{\rm state} = \frac{1}{Pd} |s{t+1} - \hat{s}_{t+1}|^2.$$This latent L2 loss ensures the world model captures the broad scene change. However, the DexWM authors found that optimizing only this global visual loss can still let the model “ignore” fine details of the hand. Since a human’s hand might occupy, say, 10% of the image patches, minimizing overall error can tolerate large mistakes at fingertips if the scene is otherwise predicted well. But for dexterity, finger position matters a lot. To remedy this, DexWM introduces an auxiliary hand-consistency (HC) loss. They freeze a small network $g_\theta$ on top of (or parallel to) the world model which takes the predicted latent $\hat{s}{t+1}$ and outputs heatmaps for each keypoint (12 channels: 5 fingertips+5 on other hand + 2 wrists) in the image pixel space. They then compare this prediction $\hat{V}{t+1}$ to a ground-truth heatmap $V_{t+1}$ computed from the true 3D hand pose, using an $L_2$ loss: $$\mathcal{L}{\rm HC} = \frac{1}{12HW}|V{t+1} - \hat{V}{t+1}|^2.$$Intuitively, this forces the model to encode enough information to localize each fingertip in the image plane. The total training loss is $$\mathcal{L} = \mathcal{L}{\rm state} + \lambda,\mathcal{L}_{\rm HC},$$ with a large weight (e.g. $\lambda=100$) on the hand term. Ablation studies reported by the authors show this matters: adding the HC loss increased correct fingertip-location accuracy by ~34% in long rollouts. In other words, DexWM truly “learned fine hand control” rather than just modeling the overall scene. Conceptually, the HC loss tethers the high-dimensional latent prediction back to the physical hand configuration it came from, closing the loop on geometry. Together, these components yield a world model that is both semantically grounded and hand-aware. During training it sees example transitions $(I_t, a_t, I_{t+1})$ and tries to make $\hat{s}_{t+1}$ and predicted hand heatmaps match reality. Importantly, no ground-truth robot actions are used—all actions come from video keypoint data. The training is fully supervised by vision and two auxiliary tasks (latent matching, keypoint localization). Deploying the World Model: MPC Planning. Once trained, DexWM provides a fast way to predict future frames in latent space. How do we use this as a robot controller? The authors employ a goal-conditioned planning strategy. Given a starting scene and a desired goal scene (for example, “object at position X” or “hand grasping object”), the system optimizes an action sequence to reach it. Practically, the robot is controlled in joint space. At each decision step, DexWM receives the current observed image, encodes it to $s_t=E_\phi(I_t)$, and also has a goal latent $s_g$ (from a goal image). We now want a short sequence of future joint configurations $(\Theta_0, \Theta_1,\dots,\Theta_{T-1})$ such that, if the robot were to execute those joints in open loop, the predicted world model state $s_T$ would be close to $s_g$. Concretely, let $G(\Theta)$ be a function that converts joint angles to an action vector $a_t$ (using forward kinematics to compute the implied fingertip motion and camera motion). Then DexWM can simulate one step: $$\hat{s}{t+1} = f\theta\bigl(s_t,\dots,s_0,;a_t\bigr),$$ and roll forward $T$ steps given an action plan. We define a cost $$C = |s_T - s_g|^2 + \mu,|\text{KP}(\hat{s}T) - \text{KP}(s_g)|^2,$$ which is the latent distance at the final step plus a small penalty on predicted hand keypoint distance (KP denotes taking the predicted keypoint pixel locations). The planning problem is $$\min{\Theta_0,\dots,\Theta_{T-1}} C$$ subject to $a_t = G(\Theta_t)$ and the DexWM predictions but no direct environment feedback. This nonconvex optimization is solved via the Cross-Entropy Method (CEM) as in many model-predictive control frameworks. Briefly, CEM samples candidate action sequences (joint angle trajectories), uses DexWM to simulate forward the latent predictions, computes the cost to goal for each sequence, and then refines the sampling distribution to focus on low-cost sequences. This iterative loop is done fresh at each real-world step (MPC), so deviations can be corrected on-the-fly. The key is that DexWM quickly provides latent evolution without running a physics sim. In practice, this lets the robot try to “imagine” many possible motions with DexWM and pick one that best reaches the goal. In summary, DexWM is not a policy network at all but a dynamics model. The action decisions come from a search in latent space, grounded by DexWM’s prediction. This makes DexWM more flexible (we can change goals arbitrarily) but also depends on the model’s accuracy. The small weight $\mu$ on keypoint cost ensures the planner slightly favors endpoints with correct hand pose, but the dominant term is matching scene features. Experiments and Results. The DexWM paper evaluates the approach in both simulation and on a real robot. In simulation, the model is used to solve tasks like grasp an object, place an object at a target, and reach a goal point. The authors compare DexWM to a strong baseline: the Diffusion Policy of Shi et al. (2023), an offline generative policy trained on video data (which directly predicts actions rather than modeling the world). DexWM beats the Diffusion Policy by a large margin. Across all tasks, DexWM’s success rates are roughly 50% higher on average. This gain comes from DexWM’s ability to plan a consistent multi-step strategy, rather than sampling from a retrained distribution. The authors note that DexWM’s strength is especially pronounced when the task at test-time is not exactly seen in training. Beyond simulation, DexWM is tested on a physical Frank a Panda arm with an Allegro five-finger hand. The setup is straightforward: the robot’s eye-in-hand camera captures the scene, an initial image and goal image are provided, and DexWM + CEM are used for open-loop planning. On a set of 12 grasping demonstrations, DexWM attained 10 successes out of 12 trials (83%). In contrast, a variant of the system without any human video pre-training performed catastrophically; it only succeeded in about 1 or 2 trials out of 12 (roughly 14%), as reported by the authors. This ablation underscores a main point: human video data was critical. The model trained with EgoDex understood the fine-grained physics of grasps, whereas the one trained only on DROID and RoboCasa (robot-only data) had not seen enough hand-object nuance to plan reliably. In fact, one striking quote from the work is that “DexWM achieves 83% success in real-world object grasping, significantly outperforming Diffusion Policy and a DexWM variant trained without human data”. Concretely, DexWM was able to grasp objects not seen during training (novel cubes, a stapler, etc.), place them at target locations, and even reach desired poses with the hand. The model used only visual goals; no explicit object pose or dynamics information was given. For the baseline Diffusion Policy, its success rates tended to be far lower (often <30%) because it couldn’t systematically reach the goal. Interestingly, DexWM also compared favorably to more basic policies like a scripted prehensile controller: its performance was on par or better, thanks to the model-based planning even though it had no robot-specific finetuning. The authors also performed several ablation studies to probe what gives DexWM its power. One key finding is the benefit of scale: enlarging the transformer to 456M parameters (“DexWM-XL”) improved accuracy, implying the model can continue to absorb more data. Swapping out the visual encoder (they tried SigLIP2, V-JEPA2, etc.) only required slight retuning, suggesting the approach is not tied to DINOv2. Most importantly, ablating the hand-consistency loss degraded fingertip accuracy significantly (as mentioned, ~34% drop in PCK@20). Removing human data from pre-training also crippled performance. In short, the combination of massive human video and the finger-aware loss seems crucial. We highlight a few quantitative takeaways from the paper and follow-ups:. Latent Prediction Accuracy: In one test (4-second open-loop prediction), DexWM’s latent next-state error was substantially lower than prior models. For example, PCK@20 (percent correct keypoints within 20px) was ~68% for DexWM vs. ~63% for other transformers or <50% for older approaches. This shows DexWM keeps track of the hand nicely. Planning Performance: On simulated grasp/place/reach tasks, DexWM solved them on 50–100% of attempts, whereas the Diffusion Policy baseline averaged ~10–30% success in the same settings. The graphs in the paper show DexWM outstripping the baseline by dozens of percentage points. Zero-Shot Sim2Real: DexWM was never trained on real Allegro hand images, yet it succeeded 83% of trials. This zero-shot transfer is remarkable — it implies the latent dynamics learned were robust enough to cross the sim-real gap with only minimal RoboCasa tuning. For context, the authors note that the version trained only on RoboCasa (with no human data) solved only ~14% of tasks, so the human video really gave generalization. Baseline Comparisons: Aside from Diffusion, the authors compare to a variant world model without hand keypoints (i.e. using only coarse actions or no HC loss). These alternatives had much lower fine control, highlighting that DexWM’s design features matter. Collectively, the empirical results suggest DexWM can unlock dexterous skills from human priors. By learning a forward model of the environment in a rich latent, the system can plan detailed finger motions that achieve grasping and manipulation tasks it has never explicitly seen in robot form. Discussion and Future Directions. DexWM is an exciting step toward bridging human data and robot dexterity. Its main innovation is showing that passive video of people doing tasks can teach a robot model the essence of manipulation. This leverages one of the most abundant data sources available: people handle tools, phones, household objects thousands of times a day. If a world model can absorb that knowledge, we gloom a shortcut past slow robot data collection. However, there are important caveats. The current DexWM experiments, while impressive, are still relatively narrow. The real-robot tests were on a small set of 12 trial objects with clear placements, and the training strongly relied on high-quality EgoDex video (which has clean backgrounds and only tabletop chores). Generalizing to busy scenes or very different tasks (assembling parts, heavy tools, deformable objects) remains to be seen. The world model approach itself has limitations: it predicts in a learned latent, which may not accurately capture physical constraints like friction or object mass. In theory, a planning loop could exploit any model error. That said, the consistency loss helps anchor the model to geometry, at least locally. From a research standpoint, many avenues beckon. One is to expand the DexWM dataset further. EgoDex is 829h, but one could imagine scraping public video or controlling headsets to collect even more diverse manipulations. Another is to improve the model architecture: the current diffusion transformer is powerful but could be augmented with object-centric reasoning or dynamic simulation layers. Third, combining DexWM with active learning could be fruitful: a robot could attempt new manipulations and incorporate the failures into training. Finally, it will be interesting to compare DexWM to other recent “unified” manipulation models. For instance, the DexSim2Real² approach builds explicit 3D world models of articulated objects, while DexWM stays in a latent space. In future, these paradigms might be combined—using DexWM-like compression for hand/object forecasts alongside physics-informed modules. For now, DexWM stands as a compelling proof-of-concept: that the barrier of dexterous data can be partly overcome by standing on humanity’s shoulders — i.e., learning from the hands we watch in everyday life. Its reported success suggests a new workflow for dexterous robotics: gather large passive video of human tasks, train a unified world model with fine-grained hand actions, then use that model as a pluggable planner for any dexterous robot. If this scale-up holds true, we may see robot hands learning from human videos at a rate and breadth never before possible. References: DexWM – “World Models Can Leverage Human Videos for Dexterous Manipulation,” Goswami et al. 2025 (arXiv 2512.13644). EgoDex dataset (829h egocentric video with 3D hand pose). DROID dataset (76k trajectories, ~350h).