Latest / Embodied AI 101 / Motion-Focused Latent Action for Cross-Embodiment VLA from Human Videos
Transcript
- Motion-Focused Latent Action for Cross-Embodiment VLA from Human Videos. Vision-Language-Action (VLA) models – essentially robotics “foundation models” that map visual observations and high-level language instructions into action commands – have made great strides recently. However, they usually rely on vast, annotated robot demonstration datasets (often teleoperated trajectories) for training. This is a bottleneck: collecting millions of labeled robot actions is expensive, time-consuming, and often limited in diversity. Meanwhile, we have an abundance of human manipulation videos (from wearable cameras, YouTube tutorials, Egocentric datasets, etc.) that span rich, real-world scenarios. The obvious question is: can we leverage these human videos to train a robot control policy? But human videos lack explicit robot action labels, and humans and robots have very different bodies (“cross-embodiment”). Simply treating a person pushing a cup as equivalent to a robot pushing a cup can fail because the kinematics and affordances differ. In this context, Xu et al. (IROS 2026) propose an elegant unsupervised framework that extracts a latent action representation from unlabeled human videos and uses it to bootstrap a VLA model. Their key idea is to explicitly focus the representation on motion itself – what they call a “Motion-Focused” approach – so that the learned action primitives become embodiment-agnostic. In practice, they train a hybrid Vector-Quantized VAE (VQ-VAE) that disentangles motion from static context in the video, producing a cross-embodiment action codebook. These discrete latent actions serve as pseudo-labels: the vision-language model is pretrained to predict them. Finally, when adapting to a real robot, they decouple “intent” and “perception” – letting a frozen visual encoder (like DINOv2) handle the exact scene details, while the VLA model focuses on the high-level action intent. This strategy dramatically cuts down the needed robot data: with only ~50 fine-tuning trajectories on the target robot, their system matches or exceeds the performance of baseline VLA models trained on orders-of-magnitude more data. Below we unpack the pipeline piece by piece: how they learn the latent codebook from human video, how the VLA model is pretrained on it, and how the final robotic policy is adapted. Along the way we compare to recent works (e.g. LAPA (2024) and others) and highlight what makes the motion-focused approach unique. By the end, you should see how this method cleverly closes the gap between human footage and robot control. The Challenge: Bridging Human Videos and Robot Actions. Modern VLA models typically start with a large frozen vision-language backbone (like a large transformer pretrained on images+text) and then add a small action head. This works well for generalization, but the action head itself still needs to learn physical motion. For example, even if the model “knows” what “grasp the cube” means in semantic space, it must still learn the joint velocities or end-effector velocities to actually execute that grasp. As Jing et al. note, most VLA systems attach an action network to a vision-language model but have the network learn dynamics “almost from scratch.” The result is that the policy has no explicit motion prior, which slows training and hurts performance, especially when changing embodiments. One way to build an action prior is through demonstrations. But human-teleoperated robot demos are scarce. A tantalizing alternative is to look at human videos, which abound on the web. For instance, egocentric datasets (Ego4D, AVA, etc.) capture people performing all sorts of manipulation tasks (pouring water, turning knobs, picking objects). The crux is: these videos don’t come with robot control labels, and humans and robots have different bodies. How can we extract from a video of a person opening a drawer the essence of the motion so that a robot (with two arms or a simple gripper) can imitate it?. Recent works have begun to tackle similar ideas. For example, Latent Action Pretraining (LAPA) (Ye et al., ICLR 2025) showed that it’s possible to take internet videos, train a VQ-VAE to encode motions between frames, and then train a VLA model to predict those latents. In LAPA’s approach, each consecutive frame pair is encoded into a discrete action code, and the vision-language model is trained in a self-supervised way to output that code from the images (and task description). They then fine-tune this model on a small robot dataset. This yielded significant gains: LAPA outperformed prior methods that tried to learn from videos (even matching or improving on VLA models trained with actual robot labels in some tasks). The key idea is that the latent codebook provides a pseudo-action label, overcoming the need for real robot action annotations. Xu et al. push this paradigm further by focusing the latent representation explicitly on motion and not on scene details. Their intuition is that a good cross-embodiment prior should capture what happens (the trajectory of hand and object) and ignore who or where. To this end they introduce a “Hybrid Disentangled VQ-VAE” that separates the dynamic content from the static background in the video frames (via physical motion masks). This yields discrete action codes that generalize across different environments and bodies. When the power of this motion-focused codebook is combined with a high-capacity VLA model, the result is a policy that can be rapidly adapted to any robot. Before diving into their solution, note the impressive results they report: pretraining on only human videos and then fine-tuning with about 50 real robot trajectories per task, they achieve performance on par with or superior to full-fledged VLA baselines. In LIBERO benchmark tasks they reach ~92% success, and on a realistic RoboTwin setup about 67% (both beating comparable state-of-art models) – all with orders of magnitude less robot data. Let’s see how. Hybrid VQ-VAE to Isolate Motion. At the heart of the method is a specialized autoencoder that learns a discrete “action codebook” from human videos. Recall that a standard vector-quantized VAE (VQ-VAE) compresses inputs into a finite set of learned codewords. When applied to video, one can try to encode either individual frames or transitions. Xu et al. do something clever: they build a disentangled VQ-VAE with a “motion focus”. Conceptually, their network splits each video frame into two streams: static context and dynamic content. The static stream encodes things like background, non-moving objects, scene layout – everything that doesn’t change from frame to frame. The dynamic stream isolates the pixels that are actually moving (e.g. a hand reaching, a door swinging). How to achieve that? They use physical motion masks. Concretely, one approach is: compute optical flow or use object/human segmentation to create a binary mask of moving regions, then feed the masked frames through different branches of the encoder. Xu et al. call their model “Hybrid Disentangled VQ-VAE,” implying it likely has two encoder branches (one for motion and one for context) with a shared bottleneck, many grid-based methods in video do this. The “hybrid” suggests one branch might output a continuous latent (for background) and the other outputs a discrete latent (the VQ-latent for motion). The result is a codebook of motion primitives: each discrete code stands for a chunk of motion (like “lift forward 10cm” or “rotate wrist clockwise”). Because the VQ-VAE only quantizes the dynamic stream, these codes are agnostic to static factors (lighting, textures, backgrounds) and crucially to specific embodiment. In other words, code #17 could arise from either a human hand pushing a bowl or a robot gripper pushing a bowl, as long as the motion is similar. This is the “cross-embodiment action codebook”. By design, it encodes what the action is, not who does it. We don’t have the exact network architecture details, but the abstract hints at a few things: there is a “Hybrid Disentangled VQ-VAE” and it produces a cross-embodiment codebook. A common way is to have the VQ-VAE reconstruct one frame given the previous frame plus a code (the action), or vice versa. Training this VQ-VAE on human egocentric videos (which the authors likely collected or reused from Ego4D or similar) teaches it a dictionary of possible motions. For example, if many videos show people grasping and moving items on a table, the codebook will include grasp-and-lift motions. This contrasts with a vanilla VQ-VAE on full images: that might waste capacity encoding backgrounds. By masking out everything but moving pixels, the model’s quantization budget is entirely devoted to action. It’s still a VAE though, so background can be encoded in a continuous latent or separate channel, but that part isn’t discretized. The terminologies “disentangled” and “hybrid” signal we have a two-part latent: one discrete, one maybe continuous. This separation is the core novelty: it explicitly encourages each code to capture motion intent alone, making it transferable across embodiments. Once trained, the VQ-VAE defines a codebook (C={c_1, c_2, \dots, c_K}) of action tokens. Every (masked) frame transition in the human videos maps to one of these discrete codes. The authors can then treat these codes as pseudo action labels. From here on, the system can learn a mapping from vision/language to these codes, without ever using actual robot commands in pretraining. Pretraining a Vision-Language Model on Latent Actions. With the action codebook in hand, Xu et al. proceed to train a vision-language model (VLM) to predict them. This follows the paradigm of LAPA (Ye et al. 2024), but here with the motion-focused twist. In more detail, each training sample is something like: (video frames, [optional] language, next frame’s action code) → (predict code). For example, they might take an observed initial frame (or short history), and the target code that corresponds to the motion leading to a future frame, and train the model to output that code. Because the code determines what we want the robot to do (move the arm in a certain way), this teaches the network action intent. In practice, they likely chunk videos into segments: given current observations (o_t) (images from head camera and maybe hand camera) and a language specification (L) (e.g. “move object to goal”), the model predicts the appropriate discrete action code (one of (C)). The network backbone is a vision-language transformer: its inputs include image embeddings and the language prompt, and its output is a distribution over latent action tokens. Ye et al.’s LAPA did exactly this: they trained a latent VLA model to predict these discrete actions from the current state and a textual instruction. They found that even without ground-truth labels, a transformer can learn to map from vision+language to the latent code, and that fine-tuning this model on a small robot dataset then yields good performance. Xu et al. adopt the same high-level procedure, but crucially their latent codes are motion-focused, and their VLM training uses the disentangled representations. Another subtlety: the summary mentions “the VLM backbone learns deep representations of action intent”. In practice, this means the pretrained transformer now has an internal embedding space shaped by how actions in video are coded. It has never seen a robot, just images of humans doing things. Yet by predicting the discrete token, it has learned, for instance, the embedding for “push forward on the object” as a concept. This high-level intent can later be interpreted by the robot. We don’t know if Xu et al. feed any language in at this stage. If the human videos come with narrations or subtitles, they might align them; otherwise it could be purely vision (with a dummy “task” token). Either way, the result is a VLA encoder that can take a visual observation (maybe plus a language instruction) and produce an “action intent” code. You can think of these discrete actions as analogous to verbs or motions. Through this unsupervised pretraining, the model effectively transfers knowledge from video to control. Because the codebook was learned on many different backgrounds and camera angles, the model’s intent representations are robust - it doesn’t care if the table in one video is cluttered or plain, or if the arm is a human hand or a strange camera angle, it should encode “moving the red block” the same way. The motion disentanglement plus language context lets it generalize widely. Intent–Perception Decoupling for Robot Adaptation. After pretraining on human videos, we need to turn this model into an actual robot policy for a specific robot embodiment. This is the “finetuning” or “chain-of-thought to action” stage. Here Xu et al. make another key design choice: decouple the prediction of high-level intent from the extraction of state features. Concretely, they freeze the pretrained VLM’s visual encoder and rely on a separate, fixed vision model (they mention DINOv2) to encode the current scene. This is sometimes called using “frozen perception encoders”. The idea is that the pretrained VLM already encodes what action should happen, while it doesn’t need to re-learn parsing the image into features. Specifically, at this stage the system has three main components:. VLM (Intent Predictor): The vision-language model (e.g. a transformer) takes in the robot’s current camera views and the language instruction, and outputs a distribution over the latent action tokens. This model was mostly pretrained on human video, so it has learned to interpret scenes and language in terms of action codes. During adaptation, this VLM is often frozen (or lightly fine-tuned) because it already encodes intent. Frozen Visual Encoder (DINOv2): They use a strong pretrained vision encoder (DINOv2 is mentioned) to process the raw camera images. DINOv2 outputs rich feature maps of the scene that capture geometry, object locations, etc. Importantly, this encoder is not trained further; it simply provides accurate state information. Action Expert (Flow-Matching Decoder): This is the new part that is learned with the robot data. It takes as input (a) the latent action “intent” vector from the VLM, (b) the frozen vision features (from DINOv2), and (c) robot proprioception (joint angles, etc.), and produces actual robot action commands (like joint velocities). The paper frames this as a flow-matching transformer: essentially a generative model that maps the intent and state into an action sequence. The intent-perception decoupling means that we trust the language model to handle the semantics (“what should be done”), while the perception encoder provides the concrete context (“where we are and what’s around us”). In practice, during training on robot examples, they freeze the weights of the VLM and the DINOv2 encoder, and only teach the action expert how to behave. The action expert is a small transformer network (often called a diffusion or flow model) that is trained from scratch on the few demonstration trajectories available. Why do this? If they instead let the VLM handle raw images to actions end-to-end, it might ignore fine geometric cues. For example, imagine a VLM that saw a person open a door: it might learn “turn knob clockwise” as an action, but if the robot’s camera sees the knob is on the left side, it could hallucinate turning in the wrong direction. By freezing perception, they ensure the action expert always sees the actual knob position and such. The VLM only outputs the intent code, which is abstract. This separation greatly reduces so-called “action hallucinations” – actions that the model thinks are reasonable in some generic sense, but which are actually impossible or wrong given the real scene. In the language of the paper, the VLM predicts the action intent code, while a separate frozen encoder provides state-specific features to the action expert. (The ArXiv summary stresses exactly this: “the VLM predicts action intent, [and] a separate frozen visual encoder provides state-specific features to the action expert, thereby reducing action hallucinations”.) In implementation, they concatenate the VLM’s intent embedding, the DINOv2 visual features, and any proprioceptive sensor input, and feed that into the action module. The action expert itself is trained with a flow-matching loss, inspired by recent ideas in generative modeling. In simple terms, flow matching means that during training each action example is perturbed with noise, and the model learns to predict the “velocity” or direction that would correct that perturbation towards the true action. Over many steps this becomes a diffusion-like process. Xu et al. use this to make the action expert robust and easy to train with few examples. (The upshot is that even with only ~50 imitation trajectories, the expert can learn to map intents to actual robot motions effectively.). After training, at test time the policy works as follows: given a language instruction, the system runs the VLM on the current image(s) + text to get a predicted latent action code. It uses the frozen encoder to get current state features. The action expert then takes this combined context and outputs the actual motor commands (or velocity field) to execute. Because the VLM was trained on video, it will produce the latent that corresponds to the intended motion (e.g. the code for “lift the cube”), and the expert will realize that intent around the actual cube position. This design is reminiscent of other “modular” VLA approaches (DIAL, etc.) that decouple task and perception. But here the twist is that the whole system was bootstrapped on unlabeled human video, and the decoupling is explicitly justified as a way to avoid misaligning intent and state. The locking of DINOv2 means the robot only needs a very small number of fine-tuning examples to ground the latent intent in its own kinematics. Indeed, Xu et al. report that only on the order of 50 demonstration trajectories are needed to adapt to a new robot embodiment. That’s extraordinarily low – typical fine-tuning might require hundreds or thousands of samples. Results: Strong Performance with Minimal Robot Data. Xu et al. evaluate their system on both simulated and real robot tasks. Two main benchmarks are reported:. LIBERO: This is a recent benchmark of multi-step, long-horizon tasks (e.g. spatial, object-directed tasks) in simulation, with language goals. It stresses generalization. Xu et al. report achieving about 91.8% average success on LIBERO tasks. This matches or exceeds state-of-the-art VLA models that were trained with many more robot labels. RoboTwin 2.0: This is a harder evaluation suite involving real robot execution (apparently on dual-arm platforms). On RoboTwin, the method scores about 67.7% average success. Again, this outperforms comparable baselines and approaches the levels of much larger-data models. Crucially, they emphasize that all of this was achieved by pretraining only on unlabeled human videos and then fine-tuning with a very small robot dataset (~50 trajectories) per task. In other words, the vision-language backbone never saw paired robot observations and actions until that last stage, yet the transfer from human motions gave it a huge headstart. As one result quote puts it, their model “pre-trained exclusively on unlabeled human videos” is “competitive with state-of-the-art VLA models trained on massive annotated datasets, requiring only 50 trajectories for downstream adaptation”. That encapsulates the key takeaway: with motion-focused latent pretraining, we can get SOTA-level control from only a 0.1%–1% fraction of the data normally needed. For comparison, recall how LAPA performed: Ye et al. (2024) showed that pretraining on videos (with a VQ-VAE codebook) outperformed previous video-based methods and even matched some fully-supervised baselines. By adding the motion disentanglement and decoupling, Xu et al. improve on that by further reducing data needs and boosting performance on cross-embodiment tasks. Another recent work (Jing et al. 2026) took a different angle: they pretrain an action module on random robot motions to capture general dynamics, then plug it into VLA training. Xu et al.’s method is complementary: their priors come not from random robot motions but from actual human manipulation videos, which apparently carries richer structure for real tasks. Qualitatively, the system shows robust generalization. For instance, it can interpret a command that was seen only in the context of human bodies and successfully apply it on a completely different robot body. The learned codebook also captures diverse motions – pushing, pulling, rotating, lifting – in a way that doesn’t depend on the exact arm shape. The few experiments on a real robot suggest reasonable sim-to-real transfer, thanks to the frozen DINOv2 providing real camera features. All told, the reported experiments illustrate the power of leveraging human video at scale. Instead of collecting millions of teleoperated steps, one can reap the motion knowledge embedded in videos with humans (which are freely available and diverse). Then, a small set of labeled robot examples suffices to ground that knowledge. Relation to Prior Work. It’s worth situating this in context. As noted, LAPA (Ye et al. 2024) was the first to show that a purely unsupervised video-pretraining could rival conventional VLA models. Xu et al. build directly on this idea of latent action pretraining, but add two key ingredients: focusing the latent on motion (via masked VQ-VAE), and decoupling intent/perception during finetuning. These changes are motivated specifically by the cross-embodiment transfer problem. Jing et al. (2026) previously argued that VLA architectures often lack an explicit motion prior, forcing the action head to learn everything from scratch. They solved this by a two-stage process: first train an action module on raw motion trajectories (using flow matching) without vision, then plug it into the VLA pipeline. This is somewhat similar philosophically: train motions independently, then align with vision-language. The difference is that Xu et al. learn motion priors from vision (videos) rather than from unconditioned robot data. Their hybrid VQ-VAE can be seen as a world-model that learns plausible dynamics by watching humans move. Both methods use flow matching for training the action network and both emphasize motion structure, but one uses human-labeled priors and the other uses random robot priors. Another line of work tries to adapt VLA to new robots via techniques like soft prompts or latent diffusion heads [e.g. X-VLA, XL-VLA 2025-26], but those still typically require some robot data or do not leverage human videos. Xu et al.’s approach is unique in explicitly pretraining on human data. It effectively treats the human-demonstrated motions as if they were “zero-shot demonstrations” for the robot. Finally, one could compare to classic imitation learning from third-person videos (e.g. via inverse RL or keypoint editing), but those usually require additional supervision or careful calibration. Here, the latent codebook obviates the need to manually align joints or viewpoints; the code acts as the bridge. Discussion and Future Directions. This work highlights a promising strategy: use the deluge of human video as a foundation for robot action. The motion-focused latent representation is the key novelty. By masking out static context, the model concentrates on transferable dynamics, making the codebook robust across different scenes and agents. The intent-perception split further allows the system to keep track of language-driven intent separately from raw state, which seems crucial for reducing errors. There are, of course, limitations and open questions. One assumption is that we can obtain reliable “physical masks” of motion in the video. The paper does not detail exactly how these masks are computed – possibly via optical flow or pretrained segmentation. In practice, good motion segmentation might require some tuning or additional labels. If these masks are inaccurate, the disentanglement could suffer. Also, the approach is tested on manipulation tasks; it’s not clear how easily it would extend to locomotion or highly deformable objects. The codebook size and VQ-VAE architecture are not given, but these hyperparameters can be important – too few codes and you lose nuance, too many and training becomes hard. Moreover, while requiring only ~50 shots on the robot is a clear win, that still means someone has to collect those demonstrations. Future work might try to reduce even that or to make the fine-tuning more automated (e.g. reinforcement learning from language reward). It would also be interesting to see if the approach supports zero-shot transfer to new tasks – for example, can the pretrained model follow a brand-new instruction without any finetuning at all? The current pipeline still needs adaptation with some data. Another area for improvement is the handling of language. The description is vague on how language is used during pretraining. If the intention code can be aligned to an arbitrary instruction (or a description of the human video), that would close the loop on full vision-language-action pretraining. Alternatively, one could add a small supervised alignment step to map codes to phrases. Finally, combining this with other robot learning advances (like better simulation environments, more human video data, or larger vision-language backbones) could multiply its impact. The results already show that the gap between human video knowledge and robot performance can be largely bridged. One can imagine extending to multi-agent demonstrations, household videos, or even using large language models to textualize the latent actions. Conclusion. Xu et al.’s IROS 2026 paper demonstrates a powerful new twist on unsupervised pretraining for robotics. By crafting a motion-focused latent action codebook from human egocentric videos, they teach a VLA model general action priors without any robot data. Then, by decoupling intent from perception, they adapt this model to actual robots with very few examples. The result is a policy that performs competitively with state-of-the-art VLA models that needed orders-of-magnitude more annotated data. This work suggests a future where robots learn not just from robot demos, but by watching the world — leveraging the vast library of human action videos to inform their control policies. For robotics researchers, the takeaway is clear: if you can represent motion in an embodiment-agnostic way, you can mine rich latent actions from any video source and drastically reduce the amount of expensive robot training needed.