Latest / Embodied AI 101 / World Models in Robotics
Transcript
- World Models in Robotics. Modern robotics is increasingly turning to world models – learned predictive models of the environment – as a way to cope with the diversity and unpredictability of the real world. Instead of hand-coding every possible environment, a robot with a world model can imagine outcomes of its actions, effectively “dreaming” robot experiences from data. For example, a recent industrial-robotics article describes a world model as “a predictive AI engine that allows a robot to build an internal representation of its environment, predict the future state based on potential actions, and select the optimal path without needing to physically test every option”. In practice, these learned models can take many forms – some explicitly predict future images or states given actions, others invert this process to infer actions from imagined videos, and yet others simultaneously produce world predictions and action plans in one unified network. In this article we survey three broad categories of emerging robotic world models, illustrating each with recent work: action-conditioned world models (video-based predictors of future observations), video-from-actions (inverse-dynamics) models (using predicted videos to derive actions), and unified world-action models (jointly modeling dynamics and policy). We then discuss how these trends help robots leverage massive unlabeled video data and vastly reduce the need for costly robot-specific training data. Action-Conditioned World Models. The simplest flavor of a robotic world model is an action-conditioned predictor: a generative model that takes as input the current image(s) and a sequence of future actions, and outputs a dream of the corresponding future images. In other words, the world model learns to answer “if the robot does these next few moves, how will the scene look?” Such models effectively close the loop between control and perception. They can be used for model-based planning (simulating outcomes of candidate action sequences) or for policy evaluation. Building them has recently been made much easier by leveraging large, pre-trained video synthesis models. For instance, the Egocentric World Model (EgoWM) approach starts from a video diffusion model pretrained on internet movies, and then fine-tunes a tiny action-conditioning module on a small amount of robot data. By writing the robot’s motor commands into the diffusion network, EgoWM creates a predictor that “follows actions faithfully while preserving generalization and realism”. In practice, EgoWM can take a first-person view of a robot and a short sequence of joint-angle targets, and generate a coherent video of the robot carrying out those moves. On tasks ranging from simple navigation (3-DoF) to full-body humanoid control (25-DoF), EgoWM significantly improves over prior baselines: it achieves up to an 80% higher structural consistency score than traditional methods, while being several times faster (latency reduced by factors of 3–6). Crucially, this is done with only modest fine-tuning: EgoWM’s authors emphasize that by “repurposing the rich world priors of Internet-scale video models” and adding lightweight conditioning layers for actions, the system needs very little robot data to learn accurate dynamics. More generally, action-conditioned world models are being built with architectures like video diffusion or autoregressive transformers. These take as input not just images but also the sequence of actions (often embedded into the network), and generate the entire future trajectory of images. Bagchi et al. (2026) summarize many such techniques: they describe training a conditional distribution $p(x_{1:T} \mid x_0, a_{0:T-1})$ over future frames given past frames $x_0$ and actions $a_t$. In practice this is implemented by inserting an action encoder (a small neural net) into a pretrained video model. For example, an egocentric diffusion model can have the robot’s velocity or joint-torque commands fed into each time-step of the generator, biasing the predicted frames. At training time one typically fine-tunes only these action encoders (at high learning rates) while keeping the massive pretrained weights mostly fixed. This “mine the internet videos, then adapt with few robot clips” recipe works well: the pretraining captures general visual and physical plausibility, and the action modules teach the network how your robot moves. Empirically, such action-conditioned models have shown strong results. Beyond the EgoWM example, works like Vid2World and Hu et al. have similarly adapted diffusion models for control. These approaches report very high fidelity of predicted videos, low error accumulation over long rollouts, and better “action-following” (ensuring static background stays fixed when the robot moves) than older video predictors. For instance, action-conditioned diffusion models have been shown to cut long-horizon planning error in half and reduce drift dramatically compared to non-action-aware models. Quality metrics like LPIPS or SSIM are competitive with pure video generation models, while a task-focused metric (called Structural Consistency Score) improves by tens of percent. The bottom line is that by using powerful image/video priors, action-conditioned world models can yield photorealistic and controllable future predictions, often generalizing to new environments and new robot embodiments (mobile robots, manipulators, humanoids) with minimal extra training. In practical terms, a robotics team could adopt this paradigm by starting with a public video model (e.g. Stable Video Diffusion, Meta’s “Cosmos”, Google “Veo”, or OpenAI’s “Sora”) and fine-tune it on a bit of task-specific data. The inference can be optimized for speed (batch rollouts, fewer denoising steps, etc.) so that the robot can simulate hundreds of future frames in real time. Once such a model is learned, it can drive planning: the robot can try many candidate action sequences in the model’s “imagination” and pick the one that looks best (evaluated by a cost function or language model reward). This closes a loop where the learned world model directly informs control. Inverse-Video World Models (Video→Actions). A second major trend flips the perspective: instead of inputting actions and predicting images, these methods generate video illusions and then infer what actions must have produced them. In practice, one boils this down to a two-stage pipeline: (1) use a generative video model to imagine a goal trajectory, often conditioned on a text or image prompt, and (2) use an inverse-dynamics model (a learned controller or flow predictor) to convert that imagined video back into actual robot commands. If done carefully, this yields a way to leverage unconstrained video synthesis for robotics. Notably, it allows one to train on pure video data (often of humans) and get out robot behavior, bypassing most of the need for robot teleoperation data. A flagship example of this approach is NVIDIA’s DreamGen pipeline. In DreamGen, the team first fine-tunes a text-conditioned video diffusion model on just a few hours of teleoperated robot data. This aligns the model with the robot’s specific embodiment. Then, they feed initial frames and language instructions into the model to generate synthetic robot videos – including not only the behaviors seen in training, but also many novel actions in new environments. Finally, for each generated video, they extract a pseudo action trajectory. This extraction can be done with a separate inverse dynamics network (trained on real trajectories) or by a learned latent-action model. The result is a set of “neural trajectories”: sampled videos of the robot performing tasks, paired with inferred actions. These synthetic datasets can then train a normal policy or controller, or be evaluated directly in a model-predictive loop. DreamGen’s results are striking. With just one pick-and-place task’s worth of real data, the pipeline generates robot behaviors for 22 completely new verbs in 10 novel environments. That is, the humanoid learned to “open a microwave, pour water, light a candle,” etc., in contexts it had never seen, and could execute them on the real robot after training only on DreamGen’s synthetic data. In other words, by scaling up video generation instead of robot demos, DreamGen achieves strong behavior generalization from a tiny data budget. The key insight is that internet video already “knows” a lot about physics and object affordances, so a robot consuming those videos via a world model can inherit that knowledge. NVIDIA’s follow-up demos on contact-rich tasks (like folding cloth or tool use) further show that DreamGen's synthetic data covers even challenging skills. Another illustration comes from a recent humanoid robotics project (the “NEO” robot). The team here trained a 14-billion-parameter video model entirely on internet video (mid-training on human egocentric footage, then a smaller fine-tune on the robot’s own viewing perspective). At zero-shot time, the system takes a text instruction and the robot’s current camera image, rolls out a video of “what should happen,” and then runs a fast inverse dynamics model on the generated frames to produce a robot motion plan. Experimentally, this world-model policy could grasp objects in novel ways, coordinate two-handed tasks, and even perform motions involving human interaction – feats that were never in its robotic training set. Impressively, leveraging the web-scale video pretraining meant the robot learned entirely new behaviors without hours of extra robot demos. The authors note explicitly that after such video pretraining, the system can “generalize to a wide range of unseen objects, environments, and tasks,” even when its own fine-tuning data was almost exclusively standard pick-and-place. A related line of work focuses on closing the “executability gap.” Simply put, a visually plausible video rollout might still violate robot kinematics or stability. To address this, Wang et al. (2026) propose EVA (Executable Video Alignment). EVA is a post-training step: they take a pretrained video predictor and penalize it whenever its imagined motions would require impossible actions. Concretely, EVA trains an inverse-dynamics reward model on real robot trajectories, and uses it to evaluate batches of generated videos. Generated videos that induce large jerks, minor penetrations, or out-of-bound torques are given low rewards, and the video model is fine-tuned to avoid them. This means the video model learns to only imagine physically executable sequences. In tests on both simulated benchmarks and a real two-arm robot, EVA-aligned models produced much more stable action sequences and higher real-world task success than the vanilla generators. In essence, EVA makes the video predictor “aware” of physics and embodiment. Yet another twist is to incorporate 3D understanding. For instance, Mi et al. (2026) introduce a Tool-Centric Inverse Dynamics Model (TC-IDM). They observe that when a video world model depicts a robot using a tool (say, a mug or a screwdriver), one can segment out the tool and track its 3D motion. TC-IDM focuses on these tool paths: after a video is generated by the world model, the method computes the tool’s imagined trajectory (in 3D point clouds) and then uses special “decoupled action heads” to map that trajectory into the robot’s end-effector commands. This plan-and-translate approach yields a highly generalizable control: it works with different tools, different robot morphologies, and even under viewpoint changes, because it essentially follows the motion of the tool itself. The TC-IDM authors show that this significantly improves zero-shot generalization, allowing the robot to imitate tasks shown in novel videos almost out of the box. In summary, the video-inverse approach gives robotics a powerful new recipe: generate, then execute. By using video models to dream about futures (often guided by text instructions) and then converting the dreams to motion, robots can leverage the vast supply of human video on the internet. The downside is added complexity (two models instead of one) and the need to ensure the inverse model is accurate – but the promise is huge. As shown by DreamGen, EVA, TC-IDM, and others, this family of methods can greatly amplify a little robot data into a lot of robot capability, effectively “closing the loop” with imagination. Joint World-Action Models (WAMs). The third category blurs the line between world model and policy: here a single network jointly predicts both the future world state (often as video frames) and the next actions. These world-action models (WAMs) combine the notions of “world model” and “vision-language-action model” into one architecture. Instead of sequentially imagining and then planning, or separately modeling video and policy, a WAM outputs a stream of tokens representing both modalities. In practice, recent WAM implementations use huge autoregressive transformers trained end-to-end to factor the joint distribution over observations and actions. Concretely, one factorization is $$P(o_{1:T}, a_{1:T} \mid {\rm goal}) = \prod_{t=1}^T P(o_t \mid o_{