Latest / Embodied AI 101 / Video-Action Models for Robot Learning

Transcript

Video-Action Models for Robot Learning · Embodied AI 101

• Video-Action Models for Robot Learning. mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs • Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, Elvis Nava • MIMIC Robotics, Microsoft Zurich, ETH Zurich/AI Center, UC Berkeley • RSS 2026. Robotic control has increasingly leaned on “foundation” models pretrained on massive image-text datasets: we see vision-language-action (VLA) pipelines that tap internet-scale vision-language backbones to give robots semantic priors for tasks (e.g. pick up the red block). But these static backbones lack any built-in concept of physical dynamics. In practice this means the robot’s policy must learn all the time dependencies and causal effects from scratch via its own experience. As the authors of mimic-video explain, this is highly inefficient – it’s like asking a robot to rediscover physics through trial-and-error every time. The result is that “large-scale expert data collection” becomes unavoidable just to compensate for the pretrained model’s blind spot in dynamics. In short, semantic priors from images are great, but without an intuition for how the world moves, robot learning is painfully data-hungry. The core insight of mimic-video is to give robots a head start by grounding their policies in a video foundation model rather than just static images. Intuitively, a video model has already learned vast visual dynamics from watching movies and internet videos – motions, collisions, object affordances, etc. By leveraging that knowledge, the robot can offload most of the planning and physical reasoning into the backbone, leaving only low-level control to be learned from demonstrations. Concretely, mimic-video introduces a Video-Action Model (VAM): a frozen, pretrained deep video generation network is used to propose high-level action plans in latent space, and a lightweight action decoder (an inverse dynamics model) then turns those latent video plans into real robot motor commands. This decoupling of planning (in latent video space) from control (motor actions) yields dramatic efficiency gains. Indeed, the authors report state-of-the-art performance with roughly 10× fewer demonstrations than a comparable VLA approach. In short, by “jointly captur[ing] semantics and visual dynamics” in a pretrained video model, mimic-video isolates the hard part (learning physics) within the backbone, making the remaining control problem far simpler. Let’s unpack how this works. The pipeline has two main pieces:. A video model pretrained on internet-scale data (in this case NVIDIA’s Cosmos-Predict2, a 2-billion parameter latent diffusion video model). This model takes as input a short observed video context (e.g. 5 frames of the robot’s workspace and possibly a language instruction) and generates a latent sequence for the future frames of that video. Critically, the video model encapsulates knowledge of real-world physical behavior (how objects tend to move, actions unfold over time, etc.). An action decoder (a small diffusion transformer) that reads the video model’s intermediate latent output and produces robot actions. In effect this decoder is an inverse dynamics model: it learns, given an imagined latent video of “what to do”, how to compute the underlying joint torques or end-effector movements to realize that plan with the actual robot. During inference, mimic-video runs the video model to produce a latent action plan (in video space) and immediately conditions the action decoder on that plan, all without generating pixel-perfect video. Because both models use diffusion-style latent flows, inference can run by stopping mid-denoising: the video model is only run up to a certain “flow time” (noise level) τ_v, yielding a latent feature h^{τ_v} (the partial denoised video representation). This feature then conditions the action decoder, which predicts the joint action trajectory (as a sequence of flow-matched action vectors) to execute. Crucially, the video model stay frozen (after light finetuning) and the heavy lifting of learning physics has already been done in its pretraining; only the small action decoder (which has an independent noise schedule τ_a) is trained on robot data. In implementation, the authors use separate flow-time schedules for video and actions, which means each training or inference pass needs just one forward through the video model per action-chunk – making it surprisingly efficient. This two-stage, partial-denoising approach is rooted in conditional flow matching. In essence, mimic-video learns the joint distribution of video predictions and robot actions: the video model’s latent provides a high-level plan, and the decoder learns the “flow field” of actions that realize that plan. During training, the video backbone is first finetuned via lightweight LoRA on relevant robot video data to adapt it to the robotic domain, preserving its temporal reasoning. Then, freezing the video model, the action decoder is trained from scratch to regress the action flow-conditioned on the video latents. By sampling the video’s flow time τ_v and the decoder’s flow time τ_a independently (from logit-normal and square-root distributions respectively), the model is robust to noise at various levels. This decoupled training drastically speeds up convergence: in their experiments the action decoder converges in about half the steps needed by a comparable VLA learner, even though the backbone is never updated. In practice, they implemented the action decoder as a smaller transformer that processes proprioceptive inputs and masked-out future actions. It uses cross-attention to the Cosmos video features h^{τ_v} at each timestep. The decoder layers include a two-layer MLP and residual paths, with adaptive layer norms modulated by τ_v and τ_a. In training, the robot’s current state embedding is replaced by a learned mask token (forcing the model to predict actions without peeking) and the decoder’s output is trained to match the “action flow” obtained from differencing ground-truth action sequences. We won’t drown in all the architecture details here, but the key point is this: the decoder is effectively a conditional generator over action trajectories, conditioned on the latent video plan. Compared to a standard diffusion policy (which models p(actions | image context) directly), this approach factorizes the problem. The video generator handles “what will happen next visually,” and the decoder just translates that into motor commands. This matters because video data is rich and abundant: millions of YouTube videos encode physics and causality. By grounding robot learning in that latent knowledge, mimic-video Avoids re-learning the physics from scratch. Data and Benchmarks. To train and evaluate their VAM, the authors mixed data from several sources. They use BridgeDataV2 – a cleaned subset of household manipulation demonstrations (3,046 episodes after removing trivial first steps) – and LIBERO – a benchmark of multi-step construction tasks (preprocessed to only include successful rollouts). They also collect a new in-house dataset (“mimic”) from a bimanual Franka setup, logging end-effector poses, 3D relative positions of objects, joint states, and low-level actions (end-effector deltas plus hand joint commands) under human teleoperation. All visual frames are 480×640. The combined dataset is balanced across tasks via mixing ratios, and all actions are represented in a coordinate-agnostic way (relative motions and 6D orientation vectors) for better generality. Using this data, mimic-video is tested on both simulated benchmarks and on real-world robot tasks. In sim, they use the SIMPLER-Bridge benchmark (various pick-and-place and block-stacking tasks derived from Bridge Data) and LIBERO tasks. For the real robot, they run a bimanual Franka + dexterous-in-hand system on two tasks: package sorting (sort small parcels on a table) and tape stowing (pick up a tape measure and place it into a box). Both tasks involve rich contact and dexterity, making them challenging. They compare mimic-video to strong baselines: especially to task-specific Diffusion-Transformer (DiT) policies (often called “Diffusion Policies”) trained from scratch on each task, and to existing VLA methods trained on much larger data. Results. The gains of mimic-video are striking. In simulation, it achieves state-of-the-art success rates on both Bridge and LIBERO benchmarks. On SIMPLER-Bridge it “achieves the highest average success rate” of any method, even outperforming baselines that had massive data or different architectures. On LIBERO, mimic-video outperforms several recent state-of-the-art policies that were trained with much more demonstrations – remarkably, mimic-video was trained essentially from scratch on minimal data yet reached higher success. The pattern holds on real robot experiments too: in bimanual tests, mimic-video significantly outperforms a single-task DiT-Block-policy baseline. Even though the DiT baseline sees two camera feeds (one per arm) and mimic-video only sees one, the video-conditioned policy does better, handling occlusions and complexity that stymie the baseline. Perhaps most impressively, the sample-efficiency advantage is huge. The authors report that mimic-video needs only about 10% of the action data of a comparable VLA policy to reach the same performance. In one figure, a conventional VLA policy needs 10× more trajectory examples to converge to the same success rate. In fact, even if train on literally one demonstration per task, mimic-video still yields ∼77% success on average, whereas the standard baselines fail catastrophically under such scarcity. The action decoder not only converges with far fewer examples, it also asymptotes to a higher success rate than the baseline (even if the baseline is warmed up with extra pretraining). In short: linking to a rich video prior can cut down the need for robot data by an order of magnitude. There are several interesting nuanced findings. One counterintuitive result is that the best performance arises when the video model is left at maximum noise (τ_v=1) rather than fully denoising it. In practice, this means the decoder often bases its actions on a noisy latent representation instead of a fully generated video frame. Why? The authors hypothesize that the intermediate latent features of the video model (before full denoising) are actually more general and robust, and trying to force a perfectly realistic video can mislead the decoder or introduce distribution mismatch. In effect, conditioning on a variety of noisy latents during training acts like a data augmentation, boosting robustness. The results show that as τ_v decreases (toward a clean image), the action reconstruction error goes up and policy performance actually worsens. They also find best hyperparameters: using about a 5-frame context for the video model, and extracting features from an intermediate transformer layer (layer 19 out of 24) gave the highest success. These choices likely reflect trade-offs between how far ahead the video model has to plan and how abstract the features are. But the takeaway is that the policy can leverage quite raw mid-latent features – again sidestepping the need for pixel-perfect future generation. Qualitatively, mimic-video shows robust behavior. In the real-world tests, for example, the robot could sort packages and stow tape even under partial occlusion, because the video backbone already “knew” roughly how arms and objects should move. When the action decoder predicts, it effectively says “move the end-effectors in this way to realize the latent motion plan,” and it does so smoothly. In comparison, the diffusion-policy baseline often failed to pick up objects or dropped trajectories under camera noise or occlusion. Why This Matters. The mimic-video work illustrates a powerful new paradigm in robotics: leveraging Internet-scale video-pretrained models as foundations for control. Vision-language models gave us open-vocabulary semantics (e.g. “find the red cup”), but lacked physics; now video-language (or video-only) models supply that missing piece. In effect, the robot inherits a built-in world model from hours of online videos: it already “knows” how a block tends to fall or how cloth drapes, without ever seeing a single robot demo of that event. This bridges the gap between two strands of research. On one hand, there are world-model approaches (latent dynamics models, model-based RL, etc.) that learn a predictive model of the environment to guide planning. Those often require tons of self-play or environment exploration. On the other hand, there are foundation models for robotics (PaLM-E, Gato, DiffusionPolicy transformers, etc.) that embed vision and language priors into policy learning. Mimic-video effectively uses a pretrained world-model – trained not on real images but on imaginary (internet) videos – plus a lightweight inverse dynamics learner. It’s a cheat code: planning is largely done by a model that’s been trained on vastly more data than we could ever collect on a robot. For practitioners, this suggests a clear recipe: start with the best video generation backbone you can get (NVIDIA’s Cosmos is one example, but we may soon see more video diffusion models from OpenAI, Meta, etc.), fine-tune it gently on your domain visuals, then train a small action-command network on your robot data. The result may be orders of magnitude more efficient learning. Especially in domains where obtaining demonstrations is expensive, this could be transformational. The authors show that even for dexterous, contact-rich tasks, the approach is not just academic: it works on a real dual-arm setup. Of course, there are caveats and future questions. The Cosmos backbone is large (2B parameters), which means heavy inference. In the experiments, careful engineering was needed to make each “forward” efficient, but it still requires substantial GPU power. Also, the approach as described is specialized to visio-motor tasks with fixed observation dimensions; integrating textual goals or interactive feedback might need extra machinery. It would also be interesting to see how this combines with tactile or proprioceptive prompts (indeed, follow-up work might fuse video models with touch models for even richer priors). Finally, the authors only tested on a handful of tasks; generalization to completely new tasks or environments remains to be seen. Nevertheless, the sample-efficiency leap is undeniable. By effectively “mimicking” the video backbone’s imagined outcome, the robot sidesteps much of the hardest learning. It’s as if the robot has watched millions of hours of demo videos before it even starts training – hence the name mimic-video. We can expect to see more work along these lines: turning large pretrained video and language models into world models for robots. After this, teaching a robot with just a few hours on your task might become routine instead of monumental. References. Jonas Pai et al., “mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs”, RSS 2026 (to appear). (Abstract and technical report retrieved from arXiv).