Latest / Embodied AI 101 / DexWM: A Dexterous Manipulation World Model from Human Videos
Transcript
- DexWM: A Dexterous Manipulation World Model from Human Videos. Dexterous robotic manipulation – getting robot hands to handle objects with the agility and subtlety of a human grasp – is notoriously hard. Unlike a simple parallel-jaw gripper that only needs to open and close, a full multi‐fingered hand has dozens of degrees of freedom. Tiny fingertip motions can completely change the outcome of a task (think of gently turning a doorknob versus breaking it). In recent years, world models – internal “mental simulators” that predict future states from current observation and actions – have become a powerful paradigm in robotics and control. But most prior world models have dealt with coarse actions (text instructions, body motions, or simple gripper commands) rather than the fine‐grained finger dynamics of dexterous hands. A new approach called DexWM (Dexterous Manipulation World Model) tackles this gap by leveraging two key ingredients: scale and fine detail. First, it taps a massive amount of visual data – over 900 hours of manipulation videos – drawn largely from human demonstrations. Second, it uses a detailed action representation, encoding the precise 3D motion of each finger, and a special loss to ensure the model really tracks those fingers. The result is a latent-space world model that can predict how objects will move in response to any given hand motion. As a result, DexWM can be used in model-predictive control on a robot arm to plan complex tasks – and remarkably, after pretraining on human video, it can even generalize zero-shot to a real robot doing new tasks. In this episode we’ll break down the ingredients of DexWM. We’ll explain the data sources it uses, how it encodes states and actions, what model architecture it uses, and the tricks (like a “hand consistency” loss) that make it work. Finally, we’ll see how it performs in simulation and on a real Franka Panda robot with an Allegro multi-finger hand, and why this success points to exciting possibilities for learning manipulation from unstructured video. The Challenge of Dexterous Manipulation. To set the stage, consider how people manipulate objects. We pick up a pen with just enough force, twist our wrists, and use individual fingertips to adjust the grip – all based on an intuitive sense of physics and contact. Getting a robot to make these kinds of subtle movements is a next frontier in robotics. Simple tasks like opening a bottle cap or assembling parts often require in-hand dexterity. Yet collecting large datasets of such skills on robots is extremely costly. Traditionally, robots have learned manipulation either through reinforcement learning in simulation or through imitation on the real arm. Both approaches struggle. Simulation often doesn’t capture the high-fidelity contact physics of fingertips, and transferring policies to the real robot is hard. On the other hand, collecting millions of hours of real-world demonstrations (as we have for images and language on the internet) is impractical. But people do move their hands around all the time, often recording it on cellphone cameras. A typing person, someone making a salad, or a craftsperson assembling a model – these all generate raw video of complex finger motions and object interactions. The insight behind DexWM is to tap into that human data. Modern egocentric video datasets (from a camera worn by a human) have begun to approach a scale where their pre-training value becomes real. For example, the recently released EgoDex dataset contains 829 hours of first-person video of people doing diverse household tasks with strong 3D hand pose annotations. DexWM uses this kind of footage as its training data. However, learning from human video raises a classic problem: the robot’s “body” is different from a human’s. Human hands are highly articulated, whereas many robots use simpler grippers or very differently shaped hands. DexWM addresses this by using a unified action representation (detailed below) and a bit of fine-tuning on robot data. Because the base model has absorbed general hand–object interaction knowledge from humans, it can adapt to the robot embodiment with very little extra tuning. In short, DexWM’s goal is: use world models trained on huge human video to give a robot fine finger-level object control. We’ll see how it does this, how well it works, and why it matters for building more general robot hands. Learning from Human and Robot Videos. The first step is gathering data. DexWM is not trained from scratch on the target robot alone. Instead, its pretraining pool includes over 900 hours of manipulation videos. This consists of:. Human Egocentric Video (EgoDex): 829 hours of first-person video showing people manipulating household objects (tying shoelaces, gripping tools, etc.), with 3D hand pose tracking. Because the footage is egocentric (from a head-mounted camera) and has rich annotations of every finger joint in 3D, it is ideal for learning how hands interact with objects. Robot Manipulation Video (DROID): On the order of 100 hours of robot demonstration video, collected from public datasets like DROID (Distributed Robot Interaction Dataset). DROID contains thousands of real robot trajectories (mostly with simpler grippers) across many tasks. These videos cover a variety of scenes and objects, grounding the model in robotic motion and viewpoints. Together this is on the order of over 900 hours of data. This is millions of video frames, orders of magnitude more than any single robot could experience through RL. Crucially, DexWM uses the human data to learn general manipulation dynamics, even though the ultimate deployment will be on a robot. The model is first pretrained on this large mix of egocentric human and robot videos. Later, a very light fine-tuning step is done: about 4 hours of random-motion data in simulation (from a Franka Panda arm with Allegro hand acting randomly in an environment, using the RoboCasa simulator) is used to bridge any remaining gap. But no task-specific demonstrations are given to the model – it only sees random grasping motions (no goals, no teleoperation) during fine-tuning. In practice this means DexWM learns its dexterity almost entirely from watching humans, with only a tiny bit of robot “warm-up” in sim. Importantly, using human video demands a few design choices. The human data has multi-finger hands with 21 joints each, while the robot’s Allegro hand also has many fingers but is mechanically different. DexWM addresses this by representing all actions in a unified way. It extracts 3D keypoints for every finger joint (using a hand-model like MANO) in the human video. Each “action” is encoded as the change in those 3D keypoints between frames. For the robot gripper data, DexWM does something clever: it creates “dummy” finger keypoints on the gripper so that the parallel-jaw behaves as if it had fingers. Concentric circle keypoints are placed at the ends of each jaw, so opening/closing the gripper maps to spreading/closing those dummy fingers. In this way, the same neural network can handle both human and robot actions, treating them all as finger motions. By training on this scale of video, DexWM learns the physics of object-hand contact in a way that would be impossible from small robot datasets alone. As the authors note, “to overcome the scarcity of dexterous manipulation datasets, DexWM is trained on over 900 hours of human and non-dexterous robot data”. The human priors let DexWM generalize to many contexts, and the robot data (and light fine-tuning) ensure it works on the actual hardware. Representing States and Actions. At the heart of DexWM is a latent-world-model. Rather than operating directly on raw pixel images (which are high-dimensional and contain extraneous details), the system first encodes each camera frame into a compact latent state. Concretely, every input image is passed through a frozen DINOv2 vision encoder. This produces a grid of patch embeddings that capture the semantic layout of the scene – the shapes of objects and hands – while discarding irrelevant details like texture or lighting. Each patch embedding is a vector (e.g. 384-dimensional), and the collection of all patch vectors forms the latent state of that frame. Using DINOv2 (a self-supervised ViT) makes the state representation stable and rich; the model doesn’t have to learn vision from scratch. The action representation is equally detailed. Instead of a simple vector of joint torques or a high-level language command, DexWM’s actions are hand-centric. In each timestep, DexWM extracts the 3D coordinates of around 21 keypoints per hand (based on the MANO hand model) from the current frame, and likewise from the next frame. The action is defined as the difference in these keypoints: how each fingertip and joint moved in 3D. We can denote the right-hand keypoints as $H_k$ and left-hand keypoints as $H_l$; then the action vector is something like $(H_k(t+1)-H_k(t),; H_l(t+1)-H_l(t))$, along with any change in camera pose. All together this ends up being a high-dimensional vector (on the order of 120+ dimensions) representing exactly how each finger moved. By encoding actions this way, DexWM has a unified view of finger motions that works for both human and robot hands. The camera motion is also included in the action vector. Since the video is egocentric, the person wearing the camera might move. DexWM accounts for this by concatenating the estimated change in camera pose (translation and rotation) between frames with the hand-keypoint differences. Therefore each action tells you “the person’s head moved this much and the hand keypoints moved that much.”. Finally, for the robot arm data (like from DROID), the Allegro or similar gripper doesn’t inherently have 21 keypoints per se. To make the robot action look like a hand action, DexWM synthesizes dummy keypoints on the gripper’s jaws. Imagine two little imaginary fingers drawn on each jaw, whose positions expand or contract as the gripper opens or closes. This is exactly what DexWM does: it places dummy keypoints on each jaw (in a circle formation) whose distance varies with the gripper aperture. In effect, opening the gripper is like spreading fingers, and closing is like pinching. This trick lets the same model “think” of the gripper as a kind of stubby hand, so it can apply the hand-learned dynamics to the robot. In summary: each frame is a latent $s_t$, each action is a high-dimensional vector $a_t$ of finger movements (plus camera). DexWM’s task is to predict the next latent state $s_{t+1}$ given $(s_t, s_{t-1}, \dots, a_t)$. The Conditional Diffusion Transformer. How does DexWM predict the future latent? The core of the model is a Conditional Diffusion Transformer (CDiT). Conceptually, this is a transformer-based neural network that takes as input the past latent states and the current action vector, and outputs the latent for the next frame. The “diffusion” in the name might invoke the idea of iterative denoising networks (as in image diffusion models), but here DexWM uses a simpler approach: it directly regresses the future latent in one shot. (This design choice trades off the iterative sampling of diffusion models for faster prediction during control.) Essentially, the CDiT is an action-conditioned sequence model for latents. In practice, the past few latent states (e.g. the previous two frames’ latents) are concatenated and passed through the transformer. The action vector is injected into every layer via Adaptive Layer Normalization (AdaLN). That means in each transformer block, the normalization is modulated by the action vector, so the network is explicitly aware of “what action is being taken” when processing the latent. This is analogous to how some language models might condition on a prompt; here the prompt is the finger motion. The result is that the transformer learns to output the latent of the subsequent frame given the context and action. A key detail is multi-step training. During training, DexWM is not only taught to predict 1 frame ahead; it’s regularly trained to predict several frames ahead by repeating its own outputs. One trick the authors used is to randomly skip timesteps during training. For example, the model might be asked to predict the latent 5 timesteps in the future, even though it sees only every second frame as input. This “skip-frame” training means the model learns to handle variable action frequencies and make longer-horizon forecasts. In essence, when optimizing the loss, the model rolls out its predictions autoregressively for a few steps and compares to the ground truth latents. This training helps the model generalize to plans that span several actions. Even though we call this a diffusion transformer, DexWM’s implementation could be thought of as a Transformer with specialized conditioning. (The paper cites prior work by Bar et al. on CDiT networks.) The important point is that it is a high-capacity sequence model (hundreds of millions of parameters, as we’ll see) that explicitly conditions on fine-grained action vectors. Hand Consistency Loss: Keeping Fingers in Check. One major insight in DexWM is that training the model to predict only the global latent can miss crucial finger details. After all, the human/robot hand often occupies a small fraction of the image. A model could predict the background objects fairly well but be sloppy about one or two fingertip locations, and still score decent pixel/latent loss. For dexterous tasks, the fingers' positions are everything. So how does DexWM ensure those fine details are learned?. The answer is an auxiliary hand consistency loss. Alongside the main latent-prediction network, the model includes a secondary output head: a small transformer that takes the predicted next latent state and tries to predict the 2D image-space heatmaps of key joints (wrists and fingertips) on that latent. Because the latent came from the DINOv2 encoder of the image, the idea is that if the latent faithfully encodes the hand’s configuration, a small decoder can map it to where the fingers are. The network is trained so that the predicted heatmap of each fingertip/wrist matches the ground-truth heatmap from the actual next image. In practice, this means the model is penalized if the latent it predicted would imply, say, a thumb floating in midair instead of grasping an object. The transformer’s latent trunk must encode the hand pose accurately enough for the heatmap head to get it right. This consistency loss greatly sharpens the model’s attention to finger details. As the authors note, recovering the whole image’s features alone “was insufficient” for fine dexterity; the hand-consistency term forces the latent to carry precise joint information. With this loss, DexWM’s training objective becomes twofold: (1) a latent-space regression loss (L2 or similar) between predicted and true next-latents, and (2) a keypoint loss on the predicted hand heatmaps. Together they ensure global scene coherence and finger accuracy. Model Predictive Control with DexWM. DexWM itself is not a policy network that takes images and outputs actions. Instead, it is a predictive model – a simulator in the loop. To use DexWM to perform tasks, the authors wrap it inside a Model-Predictive Control (MPC) framework. Here is how it works: suppose we have a goal image (or goal latent) showing the desired final state – for example, the object grasped or placed– and we are currently at a start image. We want to find a sequence of joint-angle actions that bring the real robot from start to goal. With DexWM, we do the following: generate many candidate sequences of low-level robot actions, use DexWM to roll out each sequence in imagination, and pick the sequence whose predicted outcome is closest to the goal. More concretely, the process uses Cross Entropy Method (CEM) optimization over action sequences. The algorithm samples a batch of candidate action trajectories (one per timestep), runs each trajectory through DexWM (starting from the current latent state) to predict where it ends up, and evaluates a cost for each. The cost measures how close the predicted final latent is to the goal latent, plus how well the predicted keypoints match any target configuration. (Often the goal is just specified by a goal image; DexWM can encode that image to a goal latent and goal keypoints.) The best trajectories are kept, and the sampling distribution is updated, iterating a few times. In short, DexWM acts as a fast simulator: it can roll out “open-loop” given any hypothesized action sequence. The planning objective explicitly includes both the latent distance to goal and a distance on the finger/keypoint configurations, ensuring the chosen plan not only looks right globally but also has the fingers in the correct places. The output of this optimization is a joint-angle trajectory. The first action is executed on the real robot, then the state is updated and planning repeated (hence “model-predictive”). Because DexWM’s prediction is done in a latent space, this matching to a goal is perceptual – the robot tries to make the final view look like the goal view. This allows entirely image-based goals without explicit reward shaping. In practice the paper shows DexWM achieving tasks like reaching a target position, grasping an object, or placing it at a goal location, all specified by demonstration images or keypoint poses. Experimental Results. The authors evaluated DexWM both in simulation and on a real robot. In simulation, they set up various basic manipulation tasks (reach, grasp, place) with a Franka Panda arm holding an Allegro hand. None of these particular tasks were seen during training. DexWM, combined with CEM planning, was able to solve these long-horizon tasks by nearly open-loop planning. In other words, it found sequences of finger motions that correctly accomplished the goals purely by running DexWM in the loop. Quantitatively, DexWM was compared to a range of baselines. These included earlier world models that were conditioned on less detailed inputs – for example, models that took a text command or a whole-body action as input (Cosmos-Predict2 and PEVA, for instance). Because DexWM conditions on exact hand motions, it far surpassed these baselines. The authors report that text- or navigation-conditioned models often swiped the object plausibly but missed the fingernail-on-fingertip precision needed to really grasp or hold. In yawns of keypoint accuracy (PCK), DexWM with full hand conditioning did much better. However, the real highlight was zero-shot sim-to-real transfer. After the same pretraining on EgoDex and DROID, the model was deployed on an actual Franka+Allegro robot. Apart from the 4-hours random sim fine-tuning, DexWM had never seen the real objects or tasks before. Even so, it achieved an 83% success rate on various real-world grasping trials. For context, a state-of-the-art behavioral baseline called Diffusion Policy (which is another diffusion-model-based controller) achieved around 30% success. In other words, DexWM beat Diffusion Policy by over 50 percentage points (e.g. 83% vs ~30%) across these tasks. This is a very large margin in robotics experiments. The DexWM-trained robot could pick up and place objects in ways it had never been explicitly taught, by leveraging the world understanding from human videos. The results suggest that DexWM really learned underlying physical dynamics – e.g. if the goal is to grasp a mug, the model knows how the fingers should move to produce that outcome, even if it only saw humans pour water or flip lids with their fingers in the training videos. The ESCR research group also ablated various factors. They found that combining human+robot data was critical: a model trained only on robot videos (without human Egodex) performed significantly worse in both prediction accuracy and tip precision. Increasing model capacity (going from ~30M to 450M parameters) consistently improved predictions and keypoint accuracy, suggesting that the dynamics are complex and benefit from a big model. They also tested different image encoders – DINOv2 turned out to work best for ultimate task success, though others like V-JEPA also gave reasonable performance. The experiments also demonstrated DexWM’s ability to “imagine” counterfactuals. In one simulation test, they took a starting state and applied three different hand motions (move right/up/forward) and showed that DexWM predicted distinct object trajectories in each case, matching intuition. In another, they started with a reference motion sequence from one scene (like a grasp in one scenario) and applied it to a new scene. DexWM was much more faithful in transferring hand positions than a baseline model (PEVA) that wasn’t hand-conditioned. These open-loop rollouts illustrated that DexWM had learned a consistent model of physics under finger actions. On the real robot, deployment was done purely with vision (the robot was not given object labels or force feedback). The planner simply provided a goal image and DexWM planned the finger motions to match that image. In all trials, no additional real-robot data (like teleoperated demonstrations) was used except the brief random fine-tune. Still, DexWM achieved robust grasping and placing behaviors in the real world. The authors note that coarse grasping tasks succeed almost invariably, while very high-precision tries (like balancing a delicate object) remain hard – but those are understandable given hardware limits. In sum, the experiments show DexWM as a powerful proof of concept: a world model that has seen only video (mostly of humans) can control a real robot hand in complex tasks without retraining. It even surpasses specialized methods trained on robot interactions. As the authors put it, DexWM “demonstrates strong zero-shot generalization to unseen manipulation skills… outperforming [the] Diffusion Policy by over 50% on average in grasping, placing, and reaching tasks”. Why DexWM Matters. DexWM’s success is exciting for several reasons. First, it shows that scale matters: using nearly 1000 hours of video grants the model a broad “intuition” about hand-object physics that was previously out of reach. In the same way that large language models leverage vast text corpora, DexWM suggests a path toward “foundation models” for manipulation that are pretrained on internet-scale or curated video data. EgoDex is just the beginning – we can imagine tapping into even larger egocentric datasets (like over 1000 hours of Ego4D data with manipulation) or YouTube videos to go beyond kitchen tasks to, say, woodworking or surgery. Second, DexWM demonstrates cross-embodiment learning. The knowledge of manipulation dynamics learned from human hands transferred surprisingly well to a robot hand with a different shape. The key was in the representation: by abstracting actions as 3D keypoint motions, the model learned a body-independent dynamics law of “if the fingertip pushes here, the cup moves that way.” In effect, DexWM is learning physics and control in a human body frame, then reinterpreting that physics in a robot’s frame. This opens the door to using all sorts of data from different agents (humans, animals, robots) to build unified world models. Third, the architecture and losses illustrate good principles. Conditioning on exact keypoint deltas is what gave DexWM its fine detail. The use of an auxiliary loss to grid-fix the hands can be applied to other problems with small but crucial objects. We can imagine similar techniques for, say, learning to manipulate tools, by supervising the pose of the tool’s end. The idea of multi-step autoregressive training is also a solid one for robustness. In principle, next steps could layer this with stochasticity (the environment is often nondeterministic) or with latent uncertainty estimates. Finally, the high-level planning approach shows how world models can guide behavior. DexWM isn’t an end-to-end policy; it’s a model that can be plugged into planners or higher-level controllers. In future, one could combine DexWM with learning-based planners, or integrate it with reactive feedback loops (the current work plans a full trajectory at once). Also, DexWM only used image-goals; future versions might incorporate language goals (“put the apple in the bowl”) by aligning language embeddings to its latent space. In terms of limitations and open questions: one is computation. DexWM relies on a large transformer and CEM planning, which can be slower than a feedforward policy. The paper doesn’t report inference times, but one can guess planning a few seconds ahead with a big model takes noticeable GPU cycles. Real-time use on a robot might require more efficient solver or smaller models. Another challenge is generalization to very different scenes: the model was trained on tabletop tasks, so a heavily cluttered environment or a different camera viewpoint could be problematic. The authors also point out that they didn’t integrate force feedback or object state estimation – purely vision-based control can only go so far with, say, deformable objects or unseen physics. There’s also the question of dependency on the hand keypoints: DexWM currently relies on an external tracker or ground-truth keypoints for training and planning. In a real deployment, you’d need a reliable way to sense the 3D hand pose (and object pose) – that could be done with an overhead motion capture system or onboard sensing, but it’s nontrivial. However, as vision systems get better, one could imagine DexWM absorbing its own perception model. Regardless of these caveats, DexWM is a compelling demonstration of how “everyday human data” can bootstrap robot dexterity. It suggests a future where the best way to teach a robot is to first let it watch people, then fine-tune. For roboticists, the lesson is: gather big egocentric video datasets of manipulation and pump them into your models. For ML researchers, it’s a win for world models and diffusion-like architectures in control. For embodiment, it shows the power of representation: by thinking in keypoint space rather than raw pixels or tasks, the model generalized across bodies. To adapt or extend DexWM yourself, here are a few ideas:. If you have access to a different human dataset (e.g. non-egocentric videos), one could try augmenting DexWM with that too. The core framework is general. The CDiT architecture could be replaced with other sequence models – even simpler recurrent nets – to test how important the Transformer is. You could incorporate language: combine DexWM with a vision-language model so you could say “grasp the blue mug” and have a goal image retrieved accordingly. Or embed DexWM into an RL agent as a learned model for planning, blending classic MPC with learning-based exploration. In essence, DexWM is a step toward universal manipulation intelligence, where robots learn from the rich tapestry of human activity. It’s an exciting convergence of computer vision, world modeling, and robotics – and one that will surely inspire further experiments in embodied AI. Sources: The description above is based on the DexWM project (Goswami et al., 2025) and related materials. The DexWM team pre-trained their model on over 800 hours of the EgoDex dataset (human egocentric videos with 3D hand annotations) and 100 hours from the DROID robot dataset. DexWM uses latent embeddings from a frozen DINOv2 encoder and encodes actions as 3D hand-keypoint displacements. A key innovation is the auxiliary “hand consistency” loss that forces the model to correctly predict fingertip/wrist positions. Planning is done via model-predictive control with CEM over joint trajectories. In experiments, DexWM achieved ~83% success on real robot grasping tasks, far outperforming a diffusion-policy baseline. (Citations in brackets refer to the DexWM paper and project page.).