Latest / Embodied AI 101 / DreamZero: World Action Models Are Zero-Shot Policies
Transcript
- DreamZero: World Action Models Are Zero-Shot Policies. The DreamZero paper from NVIDIA’s robotics team presents a novel approach to robot control by outright generating imagined futures in video space and aligning them with action commands. This departs from the trend of treating manipulation policies like pure vision-language problems. Instead, DreamZero defines a World Action Model (WAM) that explicitly predicts how the world will evolve (in video) and what actions make it happen. In practice, the authors train a huge 14-billion-parameter autoregressive video diffusion transformer to output short clips of future video and corresponding robot motion, conditioned on the current observation and task instruction. This jointly understanding of world dynamics and motor control pays off: in real-robot trials, DreamZero’s policies generalize to new objects, new tasks, and even new robot bodies far better than state-of-the-art Vision-Language-Action (VLA) models. Summaries of the results boast “over 2× improvement in generalization to new tasks and environments compared to state-of-the-art VLAs”. The system even runs in closed-loop on real hardware, achieving about 7 Hz control despite the model’s size. The key insight is that video carries dense, step-by-step information about physics and appearance that a model can learn from. DreamZero’s World Action Model learns physical dynamics by jointly predicting future world states (video) and actions. By contrast, VLA models (like Google’s RT-2) mainly learn to map images and language to actions, effectively treating actions as just another token in a language model. VLAs have shown impressive semantic abilities, but they often struggle with new physical motions or environments they haven’t seen during training. In DreamZero’s terms, a VLA can pick up on web-scale language knowledge (“find the red cup”) but still fails to invent a correct arm trajectory for a completely novel task or object arrangement. A pure predictive world model, however, builds an internal physics simulator. By training on “diverse, messy, real-world robot play” rather than repeated demos, DreamZero’s model internalizes the multi-step consequences of actions and can generalize to motions it never explicitly saw. World Action Models vs. Vision-Language-Action Models. To unpack this a bit, it helps to recall what a Vision-Language-Action model is. Broadly, VLA models are large pretrained transformers that are co-trained on internet vision–language data and robot driving/teleoperation data, with the clever trick that robot actions are encoded as token strings in the same vocabulary as words. In practice, a VLA like Google’s RT-2 learns to answer “What is a chair?” and “Place the cup on the coaster” using the same network. During finetuning on robotics, each (image, command) pair is paired with an “answer” which is actually the action commands, expressed as a sequence of tokenized velocities or joint moves. The result is a model that can leverage large language and image pretraining to improve semantic grounding in robotics. Such VLAs indeed excel at novel object labels or counting tasks. For instance, RT-2 can interpret a command like “pick up the smallest animal” by reasoning over its language model knowledge of animals and sizes, even if “smallest” was never in the robot data. However, VLAs have a logical blind spot: they don’t explicitly model how the world changes from one frame to the next under their actions. In contrast, DreamZero’s World Action Model (WAM) explicitly imagines the next chunk of video frames while also producing the actions that would cause that future. In the authors’ words, WAMs use video “as a dense representation of how the world evolves”. Rather than learning an implicit correspondence between visual features and actions, DreamZero’s model stands up an internal simulator. Concretely, given the current state (camera image, robot state, and an instruction), DreamZero generates a short video “chunk” of, say, 1–2 seconds, and an accompanying sequence of robot moves. By training on chunks of real trajectories, it learns to predict realistic physics. Video frames fill in the details of object motion, collisions, and subtle grasps. This video-based approach pays immediate dividends in generalization. For nominal tasks similar to those seen in training, DreamZero and VLAs both learn to place and push objects. But as soon as the test task differs (new verb or new object), DreamZero still “knows” plausible physical continuations, while VLAs often degenerate. In experiments, pretrained VLAs on common robotics benchmarks often fell back on default motions (“pick-and-place”) even when the command was wholly different, resulting in near-zero progress. By contrast, DreamZero’s predictions still contained the right semantic gist (e.g. lifting a lid or wiping a surface), so when executed the robot at least made forward progress toward the novel goal. In the AgiBot real-robot trials, on tasks completely unseen during training (like untying knots or shaking hands), DreamZero achieved about 39.5% average progress while the best pretrained VLA achieved only ~16.3% – a dramatic gap. The DreamZero Architecture (Video + Actions). How does DreamZero actually do it under the hood? The core is a video diffusion transformer with about 14 billion parameters, augmented to also output actions. We can break down the pipeline in stages (simplified):. Encoding the current world: The system first compresses recent camera frames into a latent video representation. (In practice this is done by an encoder that maps images to a “video token” sequence.) It also encodes the task instruction (in text) and the robot’s proprioceptive state (joint angles, gripper) into feature vectors, which serve as conditioning context. These become the prompt for the generative model. Formally: Inputs = [video-latents, instruction embedding, robot-state]. Autoregressive chunked prediction: DreamZero proceeds in short chunks of time – each chunk is about 1.6 seconds of future motion. For each chunk, the model will output a small video clip of that length plus a sequence of fine-grained actions (e.g. 5–10 joint-control steps at, say, 30 Hz) that realize that clip. After executing one chunk’s actions on the robot, the real new camera frames are observed and the next chunk is planned. Chunking keeps prediction errors from compounding too much, and lets the model resynchronize with reality frequently. Denoising to predict future: Training the model is inspired by diffusion-based generation. For each chunk of real-world trajectory in the dataset, we take the ground-truth future video and actions, and progressively add noise to them. The model is trained to “denoise” – that is, given a noisy partial video and actions (plus the context), predict the instantaneous velocity needed to reach the clean target. This is known as flow matching. In effect, the model learns a function that, starting from pure noise, iteratively “flows” the latents to the correct future video frames and action sequence. The benefit is that the model can flexibly imagine many possible futures in a smooth space, yet is trained to recover the actual future it saw. (In pruning terms: we distort the target with different noise levels and train the model to point towards the true future at each step.). Teacher-forcing on past context: During training, each chunk prediction is conditioned on the ground-truth history of video. In other words, the model always sees the correct frames up to the current point, so it only has to predict the next chunk. This prevents small errors from compounding through time and stabilizes learning. But at inference time, it will only ever see real observations fed back at the chunk boundaries, so it remains grounded. Closed-loop execution: At inference (robot control) time, DreamZero truly operates in feedback. It predicts a chunk’s video+actions, executes the actions on the physical robot, but then throws away the predicted frames and inserts the actual new camera frames into its memory before planning the next chunk. This closure means any discrepancy (say a bumped object or slightly different motion) is immediately corrected. Picture it like keeping the real world always in the loop every ~1.6 seconds. Classifier-free guidance: To ensure the generated video and actions strongly follow the instruction (or any conditioning), the model uses the usual diffusion trick known as classifier-free guidance. In practice this means doubling the computation: one pass of the model with the instruction given, another pass without, then blending them. The authors run these two passes on separate GPUs in parallel so the speed hit is manageable. Overall, DreamZero learns to tightly align visuals and motion. A vivid sign of this is how “autoregressive alignment looks and feels better”. Unlike some policies that latently decide on an action then hope the world changes accordingly, DreamZero explicitly predicts how it will change. Observers noted the predicted video matches the eventual action very closely: when DreamZero “fails,” it often means its imagined future was wrong, and then the robot dutifully executed that incorrect plan, rather than failing to execute the plan. In other words, the bottleneck tends to be imperfect video prediction, not mis-execution. From Vision to Actuation: Training Details. DreamZero’s training data comes from heterogeneous robot play rather than narrowly fenced task demonstrations. The core dataset is some 500 hours of teleoperated manipulations on the AgiBot robot in 22 different real environments. These include kitchens, living rooms, offices, shops – basically lots of cluttered settings. Crucially, the motions during play are diverse and unlabeled, not carefully repeated demonstrations of fixed tasks. The team also use other sources (like the public DROID dataset for a Franka arm) to increase diversity. As the authors summarize, “heterogeneous ‘play’ data improved generalization more than many repeats of the same tasks”. In practice this means they feed thousands of video-action chunks from many tasks into training, teaching DreamZero a broad low-level dynamic repertoire. No task labels or reward functions are needed; this is pure supervised learning from video-action pairs. Some scenes include human videos (via RGB-D or simulated screens) to jump-start cross-embodiment skills. During each training step, the model sees a few dozen frames of past video and the command text, and the target chunk of future frames and controls (both noised). It optimizes the flow-matching objective so that one tiny denoising step would nudge noise toward the true future. The text command input is important: DreamZero can be instructed just as VLAs can (“place the cup on the blue coaster” in the examples). This conditions the generation on what to do, rather than just predicting arbitrary valid trajectories. But compared to VLAs, the text here is only a hint; the heavy lifting of figuring out how to accomplish that instruction physically is done by video generation. Real-Time Control and “Flash” Acceleration. One might wonder: how on earth can a 14B diffusion model run fast enough to control a real robot? The answer is aggressive engineering. The model actually plans in the background while the robot executes the previous chunk. DreamZero uses asynchronous execution: as the robot carries out chunk N, the model is already computing chunk N+1, ideally finishing the heavy denoising before the robot is idle. Motion execution is smoothed slightly (averaging chunk commands) to hide any jitters. They also cache computations during diffusion: if the network’s predicted velocity field hardly changes between steps, it reuses it instead of recomputing. Most critically, the team developed a “Flash” training technique to dramatically cut sampling steps. Normally, a video diffusion model would need many denoising steps (say 8–16) to generate high-quality future frames. But DreamZero’s Flash variant teaches the model to do much of the work in one shot. Concretely, they train the model so that even with just 1 denoising step it predicts a good action, because during training those gradients force the action output to be nearly correct even if the video latent is still noisy. The result is that at inference, a single-step sampling still produces a usable plan. Empirically, this works surprisingly well. In a table-bussing example, full 4-step generation achieved ~83% task progress in about 350 ms, but naive 1-step collapsed to ~52%. Flash-trained 1-step recovered to ~74% even at only 150 ms latency. In other words, DreamZero-Flash gave a ~2.3× speedup with only modest loss in performance. Combined with dual-GPU and async tricks, DreamZero can generate and execute chunks at roughly 7 actions per second live on real robots (on state-of-the-art GPUs). By contrast, many lightweight VLA policies can run at 20 Hz on a single consumer GPU, so DreamZero is slower. The authors openly note this: “Compute hunger: A 14B diffusion transformer is heavy… action rate is ~7 Hz, slower than some lightweight VLAs (>20 Hz)”. In practice, DreamZero is a high-end system requiring new GPU hardware (the authors mention “Blackwell-class”) and careful MKL-style optimizations. But it is real-time, and the gain in generalization seems to justify the cost in these experiments. Performance on Novel Tasks and Transfer. The big question is: does all this pay off on actual robot problems? The answer appears to be yes, emphatically. We saw earlier that in unseen tasks DreamZero’s progress (~39.5%) dwarfed VLA baselines (~16%). We can flesh out a few highlights from their evaluations:. Seen demonstrations (AgiBot): On 10 tasks similar to training but in new environments (new objects, room layouts, etc.), DreamZero achieved about 62.2% task progress, compared to only ~27.4% for the best available VLA (even a version pretrained on internet data). Notably, the VLAs trained from scratch (no internet pretraining) scored near 0%. So DreamZero more than doubles the success rate of these strong baselines on everyday pick-and-place and stacking tasks. Truly unseen tasks (AgiBot): On 10 tasks never seen during training – things like untangling shoelaces, taking hats off, or hand-shaking – DreamZero still managed ~39.5% average progress. The matched VLAs essentially failed (<1% from scratch; ~16% with pretraining) because they typically just kept doing a simple pick-and-place motion regardless. In plain terms, DreamZero was actually performing many steps of the novel motion correctly, according to its video imagination, whereas the baselines were essentially clueless. Different robot (DROID/Franka): To test generality, DreamZero was also trained and evaluated on the open-source DROID dataset (Franka arm data). Here it achieved 49% progress on unseen verbs, versus 25–32% for off-the-shelf VLAs. So it wins again even on another robot platform. Post-training on specific tasks: Interestingly, DreamZero can be fine-tuned on specific downstream tasks without losing its broad abilities. The team tried teaching DreamZero specific chores (like shirt-folding or table-bussing); after a small amount of fine-tuning, DreamZero reached success rates comparable to specialized policies on those tasks while still generalizing to new objects/environments. This suggests the video-world approach isn’t brittle to specialization. Cross-embodiment transfer: One of the coolest results is that video alone can transfer knowledge. By feeding the model a few minutes of demonstration video from a different robot (or even a human) performing a task – without any action labels – DreamZero substantially improves its performance on that task. In tests, adding just 10–20 minutes of such video from another robot or a human gave over 42% relative improvement on new unseen tasks. For instance, imagine showing DreamZero a quick video of a human wiping a surface; it can then execute similar wiping motions better, even though its only tangible instruction was language. This leverages the fact that the model internally equates the video prediction with the action plan – watching a video of the motion effectively teaches the model how to adjust its world and actions. Both human-to-robot and robot-to-robot videos helped, with similar gains. Embodiment adaptation: DreamZero was also able to adapt to a new robot platform with minimal data. They transferred from AgiBot to a very different arm called YAM. By simply collecting 55 trajectories (about 30 minutes) of random play on YAM and continuing DreamZero training, the model suddenly worked zero-shot on many YAM tasks. After this tiny adaptation, YAM could do things like moving pumpkins or picking up bears (objects never seen) with high consistency. The authors claim this is “the most efficient embodiment transfer” they know of – what used to take hundreds of hours now took half an hour. The crossword is that DreamZero’s design allows it to re-anchor itself to a new body and camera quickly, thanks to its video grounding of actions. In summary, across benchmarks DreamZero consistently outperformed the comparison policies. The improvement was often large: on the unseen AgiBot tasks it was roughly double (39.5% vs ~16%), and even on seen tasks it more than doubled (62.2% vs 27.4%). What's also remarkable is how cleanly the model retained its zero-shot abilities in all these scenarios – since it models motion dynamics rather than only memorizing particular tasks, it naturally reuses that skill in new settings. Comparison to New Benchmarks and Environments. These results have been reported in the context of emerging large-scale robotics benchmarks. For example, AI2’s newly-announced MolmoSpaces provides a simulator-agnostic ecosystem of over 230,000 indoor scenes and 130,000 annotated objects for testing manipulation and navigation policies. MolmoSpaces explicitly tests models on diverse indoor tasks (static/moving manipulation, navigation, multi-room long-horizon tasks) to judge their robustness. DreamZero’s approach is well-suited to that challenge: because it learned from extremely varied real data, it should generalize across wildly different kitchens, offices, etc. In fact, MolmoSpaces’ own paper reports that “newer and stronger zero-shot policies” outperform older ones on their tasks. While DreamZero hasn’t (yet) been evaluated on MolmoSpaces, one can easily imagine it doing well given its open-ended generalization abilities. Similarly, industry projects like NVIDIA’s own RobotArena∞ (and Lightwheel’s RoboFinals) are no longer satisfied with narrow toy benchmarks. RobotArena∞, for instance, automatically creates simulated testbeds from real demonstration videos by using vision-language models and 3D reconstruction. Such efforts aim to stress-test VLA policies at scale, in synthetic twins of realistic scenes. DreamZero, in contrast, is tested in the physical world across many real scenes. In future it could also be benchmarked via these new platforms. In any case, DreamZero’s success on real robots in 22 environments suggests it has learned core skills across the kinds of variability MolmoSpaces and RoboArena care about. Discussion and Limitations. DreamZero represents a bold step toward video-centric robot control, but it has trade-offs. First, the model is very large and compute-intensive. Even with optimizations, running a 14B transformer requires multiple GPUs to keep up. The authors note that at ~7 Hz on what they call “Blackwell” hardware, DreamZero is actually slower than many VLA policies (RT-1 style controllers can run >20 Hz on a single consumer GPU). For now, ultra-fast reflex tasks (grasping falling objects, etc.) would likely prefer a lean policy. Similarly, if your robot had limited on-board compute, you’d need some model compression or distillation. The team partially addresses this via the Flash one-step technique, but it’s still a heavy approach. Second, DreamZero’s memory span is short – it only imagines a few seconds ahead. The authors acknowledge that long-horizon or multi-minute tasks (e.g. full-room tidying) would need hierarchical planning or other leaps. Additionally, the model relies solely on vision (RGB/depth) plus proprioception. Fine manipulation requiring sub-centimeter precision (like assembly or pinch grips) remains challenging without additional sensing (force, tactile) or very dense training on those tasks. The model’s errors also highlight a bottleneck: “most failures come from video prediction errors”. In effect, DreamZero executes whatever it imagines; if the visual forecast is slightly wrong, the robot will faithfully carry out that flawed plan. Improving the video prediction (e.g. via better pretraining or architectures) is thus a clear path to better policies. On the plus side, many aspects of the system are open-loop: once trained, adding more data often improves it nonlinearly. The authors note that simply scaling up the underlying video model or training longer could yield richer behaviors. Importantly, they have open-sourced their model weights and code, making DreamZero a potential foundation for others to build on. The entire system is essentially plug-and-play given enough GPU power: no task-specific reward engineering is needed, everything is learned end-to-end. Finally, DreamZero bridges to exciting applications. Because it can learn from raw video, one can imagine feeding it hours of YouTube instructional videos to acquire house-cleaning skills, or constantly improving it by letting it watch humans and robots around. The paper shows even short clips of human motion (e.g. someone ironing clothes) clock dramatic improvement in the robot’s ability to imitate that motion. As one example quote suggests, with “longer memory and richer senses (like touch), this approach could underpin truly general-purpose helpers”. In that sense DreamZero is a glimpse of a future where robots literally learn by imagining and learning from the fabric of the visual world – a true “video world model” of robotics. Conclusion. The DreamZero work introduces a powerful new paradigm: use video generative models as robot policies. By treating future observation prediction and action generation as a single diffusion problem, it bypasses many pitfalls of purely mapping language to moves. The result is a controller that not only follows verbal commands but “has physics in its bones,” as the authors demonstrate. In rigorous tests, DreamZero’s WAM policies markedly surpassed the best vision-language baselines on unseen tasks. It even enables data-efficient cross-embodiment transfer and few-shot adaptation with only minutes of video input. This approach underscores a broader trend: foundational visual world models for robots. As the DreamZero team puts it, “better imagined futures today make more capable robots tomorrow.” Indeed, by harnessing the recent advances in large-scale video diffusion and combining them with robotics, DreamZero charts a path toward robots that learn from raw video in the same way as advanced game engines – by simulating what happens next. For now the infrastructure (data and compute) is hefty, but the payoff in generality is clear. Future work will likely trim the model and extend its memory, but DreamZero already shows that world-centric thinking can become an off-the-shelf strategy for manipulation. It adds a compelling new tool to the robotics toolbox: if you can dream it in video, you can do it with your robot. Sources: The above is based on Ye et al.’s “World Action Models are Zero-shot Policies” (NVIDIA, 2026) and related robotics literature. Key findings and quotes are drawn from their project page and experimental results.