Latest / Embodied AI 101 / World Action Models: The Next Frontier in Embodied AI

Transcript

World Action Models: The Next Frontier in Embodied AI · Embodied AI 101

• World Action Models: The Next Frontier in Embodied AI. Robot learning has recently been transformed by large-scale vision-language-action (VLA) systems – think “foundation models” that map visual observations and language instructions directly to actions. These VLA policies excel at broad semantic understanding, but they remain fundamentally reactive: they map state and goals to an action without explicitly modeling how the world will evolve when those actions are taken. Meanwhile, researchers building world models in robotics and RL have built predictive simulators of environments (e.g. predicting next frames or states given actions), but those models typically don’t themselves choose actions – they only predict outcomes. A new unifying paradigm is emerging that bridges these two threads. In a recent survey paper, Wang et al. 2026 formally define World Action Models (WAMs) as embodied foundation models that jointly predict future environment states and generate the corresponding actions. In other words, a WAM learns a model of the world and a policy for acting in that world at the same time. Unlike a VLA model that only outputs an action conditioned on the current state (i.e. $p(a\mid o,l)$), or a world model that only outputs a successor state prediction $p(o'\mid o,a)$, a WAM targets the joint distribution $p(o',a\mid o,l)$ simultaneously. Concretely, “[a] model qualifies as a WAM only when it performs forward predictive modeling of future state and couples action generation to that anticipated future”. This unified approach means the model imagines the future and picks an action in harmony with that imagined future. In practice WAMs leverage generative modeling techniques (autoregressive transformers, video diffusion, latent actions, etc.) in order to predict, say, a sequence of future video frames and the robot controls that produce them. Early evidence suggests this tight coupling can dramatically improve generalization and planning in robotics. For example, NVIDIA’s DreamZero model uses a video-diffusion backbone and explicitly predicts future video frames along with actions; it achieves more than double the success rate on previously unseen tasks compared to prior reactive models. Overall, the survey by Wang et al. (readily available online) is a deep dive into this new family of models – the first systematic account of world action models. It defines WAMs precisely, contrasts them with prior paradigms, proposes a taxonomy of architectures, breaks down the diverse data sources being used to train them, and even sketches out how to evaluate these models. In this article we unpack their findings and weave in examples, to give a complete picture of what WAMs are, why they matter for robotics, and what comes next. From VLA and World Models to WAMs. To understand WAMs, it helps to first recall the landscape of embodied AI models. On one side are Vision-Language-Action (VLA) models – these are like vision-language foundation models extended for robot control. They take a current observation (often an image or video) and a language instruction, and output an action (or sequence of actions). This can be done with transformer policies, diffusion action generators, and so on. The key point is that VLA models learn rich visual and linguistic priors from massive data (often internet images and video), giving them strong semantic understanding. But they remain reactive in practice: “the model remains fundamentally reactive because future physical evolution is not explicitly represented”. In other words, a VLA policy knows what to do now, but it hasn’t necessarily imagined afterdoing what the world will look like (see Fig. 1 in Wang et al.). On the other side are traditional world models (used in model-based RL or video prediction). A world model is trained to internalize environment dynamics: given a current state $o_t$ and an action $a_t$, it predicts the next state $o_{t+1}$ (or a sequence of future frames). World models can be very powerful at capturing physics and predicting long-term outcomes. But by themselves they only predict; they don’t produce actions. A world model is “predictive rather than directly executable” – it can imagine what happens, but some policy must still choose actions. World Action Models bridge this gap. In a WAM, the forward prediction of future states is not an isolated component or side effect, but intrinsic to the policy. In fact, Wang et al. insist that “future-state prediction must be part of the policy, not just an auxiliary backbone or external simulator”. The model simultaneously optimizes an objective that includes predicting the next observation AND predicting which action (or actions) lead to that predicted observation. Concretely, a WAM is trained to model something like $p(o_{t+1}, a_t \mid o_t, l)$ or even the whole trajectory $p(o_{t:t+H}, a_{t:t+H} \mid o_{0:t}, l)$, depending on the horizon. This is different from a VLA policy (which just models $p(a\mid o,l)$) or a pure world model ($p(o'\mid o,a)$). To see why this is powerful, note that by forcing the model to explain how its own actions led to its imagined observations, it learns the causal dynamics of the scene in a grounded way. Once learned, the same model can be used for planning (generate imagined futures and actions offline) or closed-loop control (roll out by always sampling the next action). In effect, WAMs are state-of-the-art attempts at what some have called “model-based policy foundation models.”. Glossary: VLA vs. World Model vs. WAM. Putting it succinctly:. VLA models ask: “Given current sensor input $o$ and a language goal $l$, what action $a$ should I take?” They optimize $p(a \mid o, l)$. This brings powerful semantic priors into control (borrowed from large vision-language models), but without explicit physics modeling their actions can fail when the environment changes. As the survey notes, they have strong semantic grounding but remain reactive. World Models ask: “Given current state $o$ and an action $a$, what will the next state $o'$ look like?” They optimize $p(o' \mid o,a)$. Such models can simulate future states and are useful for planning, but they need a separate planner/policy to actually act. The survey distinguishes “action-conditioned”, “language-conditioned”, and “embodied” world models, but the core idea is forecasting or imagination rather than direct control. World Action Models (WAMs) combine the two: they ask “Given $o,l$, what is the next state $o'$ and action $a$?” i.e. $p(o',a \mid o,l)$. The model is trained end-to-end to do both: predict how the world will change and pick an action consistent with that change. As the survey puts it, the defining shift is “future-state synthesis and executable action are learned together inside one embodied policy framework rather than being treated as unrelated outputs”. A simple analogy is helpful: under VLA, the agent is like an actor who takes your line (the observation and goal) and immediately does something, never looking at what happens next. Under a pure world model, the agent is a toymaker who can perfectly simulate a toy car’s motion when you tell it how to push, but it never actually pushes it. A WAM is someone who both predicts the car’s motion AND pushes it in such a way as to bring about that predicted motion – a simulation and control fused. Historical Roots. WAMs didn’t emerge from nowhere. The survey traces how the growing popularity of VLA (Vision-Language-Action) policies collided with renewed interest in predictive world models in robotics. Initially, many robotics projects just focused on one or the other. Imitation learning and RL focused on reward or behavior cloning (which is similar in spirit to VLA, especially when hashtags like VLMP stand for “Vision, Language, Motion, Policy”). Separately, researchers have long built video prediction or latent-dynamics models (for planning and imagination) – think Dreamer, PlaNet, world models for Atari, or pixelRNN video. In the 2020’s, these lines began to merge. Some early work used pre-trained video or image models to help a policy (e.g. using a diffusion model to refine actions). Others used learned simulators to improve policy training. The survey highlights that recently, “a growing body of work addresses” the gap of reactive VLA by “integrating world models into the action generation pipeline”. Several groups (NVIDIA, Google, Shanghai AI labs, etc.) started building exactly the kind of joint model we now call WAM, often under names like “world model enhanced policy” or “video-prediction policy”. They use large transformer or diffusion architectures mirroring those in vision-language models, but prime them with both actions and frames. In practice, many WAMs leverage internet-scale video pretraining as well as robotics data. The intuition is: a robot that has watched millions of videos of objects moving in diverse settings develops a stronger intuition for physics, which helps it adapt when acting. The review references projects like NVIDIA’s “DreamZero” (Feb 2026) that do exactly this: they pretrain a vision-action model on web videos and robot data, and then show it can perform brand-new tasks without additional training. Similarly, other contemporaneous projects (Motus, FLAM, CoLA-World, etc.) have developed variations of the WAM idea, some focusing on latent-action spaces or factorized dynamics. Overall, the lesson is that predictive simulation and policy learning have always been natural bedfellows, and now we have the compute and data to fuse them at scale. The WAM survey authors highlight these “foundations and early integration” as the background to this new paradigm. Architectural Taxonomy: Cascaded vs Joint WAMs. One of the core contributions of the survey is a taxonomy of WAM architectures. They identify two broad families:. Cascaded WAMs: in this approach, the model is factorized into two parts. First a world model predicts or “plans” a future state or trajectory; then a second component (often called the action model or action decoder) produces the actual control commands from that plan. In other words: plan first, then act. The first stage might output a predicted video frame ( pixel rollout), a depth map, an optical flow field, or a latent embedding of future state. The second stage takes that intermediate and translates it into concrete actions. This cascade is modular: you can plug in any world predictor and any action decoder. For example, one could use a video diffusion model to “paint” the next frames, then feed those frames into a learned controller that infers the actions (perhaps by inverse dynamics). Or, one might predict a latent representation (say, a VAE embedding of the next image) and decode actions from it. The survey notes that cascaded WAMs make the coupling between prediction and action the crucial bottleneck: if the world model’s output doesn’t encode exactly the features the action model needs, performance suffers. Specifically, the survey highlights that cascaded WAMs can use explicit planning outputs (like full video frames, optical flow, depth images, 3D structures) or implicit planning outputs (latent features, future tokens, masks, hidden states). The advantage of explicit planning is interpretability and ensuring physical realism; the advantage of implicit planning is efficiency and the ability to leverage language-like token generation. In practice, many recent works combine both: e.g. they might predict a low-res video frame as an intermediate and also share latent tokens via a transformer. Joint WAMs: in this alternative approach, the model does not explicitly split into world model vs policy. Instead, a single unified model is trained to output both future states and actions jointly. There is no separate “action decoder” – both the vision prediction and the action emerge from one big network. The key question is how the network couples the two modalities of output: does it use a unified token stream, or separate heads, or a diffusion setup, etc. The survey breaks joint WAMs further along generation style. For example, some joint WAMs are autoregressive sequence models that simply treat both images and actions as tokens in a long sequence. They maintain a shared vocabulary (or embedding space) so the model can attend across modalities. Others are diffusion-based: they may have one diffusion process that jointly denoises video plus action, or factor it into multiple streams. Some hybrid designs are possible too. The slide [25†L187-L191] lists these patterns: “shared token spaces, multi-head routing, unified vocabularies” for diffusion vs autoregressive, and simply notes “single-engine or multi-engine diffusion and non-autoregressive generation” for diffusion. In short, cascaded WAMs enforce a two-step pipeline (first predict future state, then derive actions), whereas joint WAMs pull everything into one monolithic generative process. Both have trade-offs. Cascaded WAMs are often easier to interpret and debug (you can inspect the planned video, then see actions). They can also mix-and-match pre-trained components. But they may struggle if the action decoder doesn’t perfectly align with the world model’s outputs. Joint WAMs have tighter integration (the model can internalize likes and avoids inconsistent futures), but training one giant model to do everything can be more complex – they often require careful multi-task losses or scheduling. Figure 1 below (from the survey) illustrates these two families. On the left we see a cascaded system: a world model module predicts “Future Plan” (e.g. video frames), and the action module then outputs “Future Action”. On the right we see a joint setup: a single model takes in and produces simultaneously inside one network. The taxonomy doesn’t stop there. Within each family, WAM methods further vary by how they condition on context (e.g. visual prompts, language instructions, proprioceptive state), by their decoding strategy (autoregressive, diffusion, etc.), and by input modality (RGB, depth, language, etc.). But the cascaded-vs-joint split is the primary division. Building the WAM Data Ecosystem. Training a World Action Model requires data that ties together vision, dynamics, and actions. The survey emphasizes that this is not just a scale problem, but a “mixture-design” problem. That is, WAMs need diverse sources of experience: rich real robot trajectories, multi-view human videos, simulated environments, and even passive videos. Wang et al. organize the data ecosystem into four categories:. Robot-Centric Teleoperation: This includes datasets of actual robots being teleoperated to perform tasks, such as RoboNet, MOMENTS, etc. These provide high-frequency logged trajectories with aligned state-action supervision: we know exactly what joint torques or end-effector motions corresponded to each sensor reading. Often these datasets include proprioceptive signals (joint angles, forces, velocities) and can capture contact-rich behaviors. Because they come from real hardware, they avoid or minimize sim-to-real gaps. The survey highlights that teleoperation data offers “[h]igh-frequency robot trajectories [with] kinematics, proprioception, and contact-rich control” that is very valuable for learning dynamics. Portable Human Demonstration: This refers to video data captured by humans performing tasks (sometimes worn on the body or filmed from egocentric or external cameras), where the demonstrated behavior can be “retargeted” to robots. For example, a human assembly demonstration on a table could be used to generate robot actions. These datasets are cheaper to collect and cover a wide variety of contexts. The survey cites methods like UMotion or YCB tasks, where multi-view video of humans yields low-cost diversity: “retargeted human demonstrations from scenes, adding dexterous motion and multi-view interaction”. While human videos don’t come with robot joint labels out of the box, WAMs can leverage them by learning latent action embeddings or by inverse-dynamics models. The key is that these data bring real-world variety inexpensively. Simulation: Simulated environments like Habitat, Isaac Gym, Mujoco tasks, etc., provide a controlled physics-derived source of data. The advantage is “privileged supervision”: we can get perfect ground-truth depth, 6D pose, collision information, dense RAM logs, etc. And we can easily scale up by running thousands of random scenes or tasks in parallel. The survey notes that simulation supplies “scalable variation with privileged depth, pose, collision, 3D structure, and controllable physics”. Of course, sim data may diverge from real-world visuals and noise, but as part of a mixed dataset it helps the model learn clean physics. Many WAM papers first pretrain world models in sim, then fine-tune to real data. Human and Ego-Centric Video: Finally, there is the massive trove of internet video – everything from movies and YouTube clips to helmet cameras and robotics lab footages. These contain passive observations of objects and people moving in real scenarios, without any robot doing the action. Surprisingly, they are hugely useful. “Web and egocentric video supply passive dynamics, long-horizon activity, semantic diversity, and open-world priors beyond robot labs,” explain the authors. WAMs trained on such data learn a richer sense of how the world behaves in general. Of course, video alone has no action labels, so methods either ignore it or use self-supervised tricks (e.g. learn a latent action correspondence, or train an inverse dynamics head). But even without action tags, learning to predict future frames from past frames on web-scale videos gives the model an intuitive physics-embedded foundation. The bottom line: no single dataset is enough. A recent WAM paper may combine, say, robot table-top teleop logs, human cooking videos, Unity simulation rollouts, and an even broader corpus of YouTube to train its components. What matters is the diversity of signal. Teleop provides precise state-action pairs (crucial for fine motor skills), simulation provides perfect physics and scale, human demos add human-like solves and dexterity, and internet video embeds broad common-sense. The survey discusses each category in detail, but we can summarize: WAM training often involves multi-source, multi-fidelity data to capture everything from precise dynamics to semantic context. Example Architectures and Models. To ground these ideas, let’s look at a couple of concrete WAM examples from the literature:. DreamZero (NVIDIA, Feb 2026): This model was introduced as “World Action Models are Zero-shot Policies” by Ye et al. DreamZero uses a video diffusion backbone to jointly predict future frames and actions. In their abstract they say: “DreamZero is a World Action Model that leverages video diffusion to enable better generalization of physical motions across novel environments and embodiments … By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data.”. In plain terms, DreamZero starts from a large pretrained video diffusion model (trained on generic videos) and fine-tunes it on robot trajectories. When deployed, it can take in a first-person camera view plus a text instruction, and sample a future video of what should happen – along with the control sequence to make it happen. Impressively, DreamZero’s authors report over 2× improvement on unseen tasks compared to the best prior models. They attribute this to the WAM approach: while ordinary VLA models see only a single frame when choosing an action, DreamZero “learns physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves”. Motus (Shanghai Jiao Tong Univ., Dec 2025): This is a “latent action world model” that combines a video model with a separate latent action model. Motus uses a Mixture-of-Transformer (MoT) backbone with multiple experts for video understanding, generation, and action. It also introduces a “UniDiffuser” scheduling to flexibly toggle between different prediction modes (e.g. just video or video+action). Crucially, Motus learns an explicit latent action space: it uses optical flow to create “delta action” tokens and a multi-phase training pipeline for large-scale action pretraining. The effect is that Motus can ingest both videos and robot logs. In their own description, “Motus introduces a latent action world model that leverages existing pretrained models and sharable motion information,” and then “integrates three experts (understanding, video generation, and action) into one model”. In practice, Motus can predict future latent videos and decode them into actions; experiments showed it “achieves superior performance against state-of-the-art methods” on tasks spanning simulated and real robots. FLAM (Factorized Latent Action Model, Michigan, Dec 2025): Another related approach (briefly mentioned in the survey) focuses on factorizing the scene into objects or slots. FLAM explicitly represents a scene with multiple objects and learns a latent action per object. It turns a WAM into a more modular multi-agent setting: each object’s motion is predicted along with an “object-centric” control. This yields robust predictions in multi-entity environments. (For example, predicting two moving blocks with separate action intents each.) While beyond the scope here, it’s an instance of a joint WAM with additional structure, illustrating the flexibility of the paradigm. Each of these examples fits into the cascaded/joint taxonomy. DreamZero is largely a joint model – a single diffusion pipeline that outputs both frames and actions. Motus is in-between: it has a unified model architecture but explicitly handles latent actions, so it has cascaded aspects internally (latent/action flow predictions). In any case, all these methods benefit from the insight that co-training the video prediction and action prediction yields better generalization. As the DreamZero abstract summarizes, “state-of-the-art VLA models excel at semantic generalization but struggle to generalize to unseen physical motions … we introduce DreamZero, a WAM built on video diffusion… using video as a dense representation” gets better results. We can also see in these works how WAMs tie into training strategies: for example, many use mixtures of objectives (reconstruction of video frames, inverse-dynamics losses to align actions, flow-matching losses for velocities, etc.). And they regularly leverage large pretrained components (e.g. a video diffusion model trained on web videos) and then fine-tune on robot tasks. One striking trend is that the action prediction often uses an inverse dynamics signal: given the predicted change in latent or pixel space, the WAM can include a loss that the predicted action should have been the one that causes that change. This ties the two outputs together during training. Evaluating World Action Models. How do we know if a WAM is good? The survey makes a point that we need richer evaluation than usual. A WAM should pass both vision-model tests and policy tests. The authors propose two axes of evaluation: the world modeling capability and the action policy capability. In short, you can’t judge a WAM solely by looking at how good the videos are; you also can’t judge it solely by task success. You need to check consistency between the two. For world modeling, they break it into three questions:. Visual Fidelity: Do the generated future images (or frames) look realistic and consistent? This is the usual video generation metric set: PSNR/SSIM for pixel-level similarity, LPIPS or other perceptual distances, FVD (Fréchet Video Distance) for realism. If a WAM just hallucinated random but plausible-looking video, that could still be useful for some tasks, but we test it anyway to ensure basic quality. The authors mention standard metrics: “PSNR and SSIM cover low-level fidelity; LPIPS, DreamSim, and DINO test perceptual and semantic consistency; FVD measures distribution-level realism and temporal quality”. Physical Commonsense: More importantly, are the imagined worlds physically plausible? For example, objects shouldn’t teleport or pass through each other. Does motion obey continuity (objects maintain shape), conservation (energy is plausible), and correct gravity/collision? There are specialized benchmarks here: the survey lists VideoPhy, PhyGenBench, VBench-2.0, etc. These measure whether objects continue coherently, follow correct trajectories, and obey causal order. The bulletpoint summary says: “Physical commonsense asks whether generated worlds obey object continuity, material behavior, causal order, and plausible motion”. This is a high bar: a WAM must not only produce a “nice looking” video but one that an expert could not easily reject as non-physical. Action Plausibility: Finally in the modeling axis, even if the video is realistic, does it still encode the information needed to derive correct actions? Essentially, if we hand the generated future to an inverse dynamics model, would it recover the same actions? This is tested in benchmarks like WorldSimBench or the Inverse Dynamics Model (IDM) Turing Test, which check whether the imagined state trajectories contain the cues to infer the agent’s controls correctly. Concretely, they might generate a fake video of a robot arm pushing a button: a good WAM’s video should imply a pushing motion and a plausible trajectory, not like the button’s on a table floating around. For action policy evaluation, the survey returns to more familiar robotics benchmarks. They observe that WAMs should ultimately be tested as robot controllers on multi-task and long-horizon problems, possibly with language conditioning. Example suites include Meta-World, RLBench, ManiSkill (for manipulation tasks), as well as RoboCasa, GemBench, CALVIN, etc. These measure versatility, language-following, robustness, etc. They also flag specialized domains like bimanual/humanoid control (RoboTwin, BiGym, etc.), which push higher DOF coordination. In practice, any WAM paper should report how well its policy solves tasks in simulation and (ideally) on real robots, compared to baselines. The key message: Evaluate both sides. Check that the imagined futures are coherent (visually and physically) and that they actually facilitate correct action choices. The survey even argues the field needs new metrics that tie these axes together – for example, something that measures if “imagined futures are causally consistent with executed actions”. Why WAMs Matter. The real takeaway is: World Action Models promise to bring some of the breakthroughs in generative AI into robot learning. By explicitly modeling the physics of the world inside the policy, they open up a lot of possibilities:. Better Generalization: Since WAMs can train on huge, heterogeneous data (including passive video), early results show they generalize far beyond what pure imitation policies do. DreamZero, for example, doubles success on brand-new tasks after seeing only videos of those tasks (without action labels). WAMs can also transfer between embodiments – a model can watch humans or other robots and adapt to a new body because it learned the underlying physics, not just the motor commands. Planning and Imagination: A WAM can imagine multiple steps ahead. If it’s autoregressive or diffusion-based, one can sample a sequence of future frames and actions. This is like having a built-in simulator in the policy. So we can use classic model-based tricks (MPC style planning, branch-and-bound search, etc.) using the WAM as the forward model. Early work (like FLAM’s offline RL example) shows you can sample entirely synthetic trajectories from a WAM and apply offline RL, dramatically boosting data-efficiency. Policy Safety and Robustness: Since the model has an internal sense of physics, it may be better at avoiding catastrophes. For instance, if the WAM predicts that a certain action leads to a catastrophe in the video (a ball flying off?), the policy can learn to avoid it. (This is speculative but intuitive: the internal model can be probed for unsafe outcomes.). Integration with Other Modalities: WAMs naturally combine vision, text, and control. You can condition them on language and have them model the world accordingly. They can also incorporate tactile or audio inputs (with more work), going beyond just RGB images. The survey even calls out “multimodal physical state” as an open challenge (e.g. including force and tactile information). Foundation Models for Robotics: Ultimately, WAMs are a step toward a foundational paradigm in robotics much like how GPT-style LLMs transformed NLP. Instead of training a small policy per task, we can train a big WAM once and fine-tune or prompt it for new tasks. That said, WAMs are big and heavy. The survey discusses many open challenges. For example, inference time is a problem: diffusion and large transformers can be too slow for tight real-time control (noted under “Inference Latency and Efficiency”). Long-horizon prediction is also hard: even world models drift over time, and in WAMs that compounds both visual drift and action errors. Figuring out how to evaluate safety and combining modalities (vision + touch + force) are open questions. Also, as the authors point out, we still need systematic studies: how much does each data source (teleop vs simulation vs video) really contribute? Can smaller WAMs compete, or do we need 10B+ models for this? The survey encourages future work on all these fronts. Conclusion: A New Frontier. World Action Models represent a convergence of ideas. At their heart is a simple insight: if a robot is going to act in the world, it should learn to predict that world as part of doing so. This joint modeling is what the survey by Wang et al. articulates and organizes. We now have multiple demonstrations that WAMs can leapfrog the capabilities of older reactive systems – they plan in pixel space, adapt from videos, and even handle tasks purely from language prompts in new environments. For practitioners, this suggests new strategies. If you're solving a robotics control problem, consider whether adding video or world-model training can boost your policy. If you have any "model-based" intuition, push it further by integrating it with generative models or LLMs. And watch out for emerging WAM-based libraries and code (e.g., open-source DreamZero or Motus repositories) – they may become the new toolkit for high-end robot learning. The “Embodied AI” roadmap is expanding: first came reinforcement learning and imitation learning, then vision-and-language policy models, and now the survey argues the next frontier is truly generative world-action models. By jointly learning physics and control, WAMs promise robots that can literally dream up their own best moves before acting – a powerful capability that could drive the next generation of generalist robots. References: The above synopsis is based on the new survey “World Action Models: The Next Frontier in Embodied AI” by Wang et al., 2026 (Fudan University et al.), as well as related recent WAM papers like NVIDIA’s DreamZero and Motus. The survey organizes existing literature, architectures, datasets, and tests in a unified framework, providing a valuable roadmap for anyone in embodied AI.