Latest / Embodied AI 101 / GigaWorld-1: Large-Scale World Model for Robot Policy Evaluation
Transcript
- GigaWorld-1: Large-Scale World Model for Robot Policy Evaluation. Robotic manipulation policies are notoriously costly to test on hardware – each trial requires physical robots, human supervision, and time. This bottleneck is what the GigaAI team addresses in GigaWorld-1, a project that systematically studies world models as surrogate evaluators of robot policies. Rather than check a new grasping or stacking algorithm only on the real robot (which might take days of human effort), a world-model evaluator simulates what the robot would see and do, letting us quickly compare policies. In this work, the authors introduce WMBench, a benchmark of paired real-robot trials and simulated rollouts, and use it to compare many world model designs. Their results lead to three key takeaways (emphasized in the abstract): first, long-horizon consistency of action effects matters far more than short-term visual realism when judging a policy; second, improvements from pretraining rely on balancing general video data with robot-specific demonstrations; and third, architectural choices (how actions are encoded, memory is maintained, and how the model is fine-tuned for evaluation) crucially determine how well the simulated outcomes align with the real robot’s behavior. In other words, a cheap simulator only helps if it faithfully preserves what happens when the robot acts over many steps, not just if each frame looks pretty. This was all made possible by assembling unprecedented data and code. The authors collected 12,980 hours of video (including internet physics videos, open robot datasets, egocentric human videos, and their own robot demos) to train GigaWorld-1, and they benchmarked models on 324,000+ simulated rollouts paired with actual robot trials. They fully open-sourced their code, datasets, and world model checkpoints, laying the groundwork for scalable evaluation of future embodied AI systems. We’ll unpack how they did this, what they discovered, and what GigaWorld-1 is. The WMBench Benchmark. To evaluate world models, the authors built WMBench – a closed-loop evaluation suite that mirrors how we deploy policies in reality. In WMBench, each trial is a short robot task (e.g. folding a box, placing objects) for which they have both the real run and many simulated runs. Critically, WMBench enforces a strict train/test split and a closed-loop setup so that a world model truly has to simulate the future as a policy acts, not just generate a video from a fixed script. The WMBench protocol proceeds in four stages (mirroring actual deployment):. Step 1: Real-World Collection. The team collects real robot rollouts in closed loop (the robot policy runs on the real machine) with initial state observations, task instructions, multi-view videos, and human-annotated success/failure labels. This provides ground truth behavior under a wide range of tasks and initial conditions. Step 2: Strict Train/Test Holdout. WMBench splits episodes so that test layouts and initial states are unseen during training, preventing the model from simply memorizing particular setups. This ensures a world model’s performance reflects generalization to new configurations. Step 3: Closed-Loop Simulation. To evaluate a world model, the policy is run in the loop with the model: at each timestep the model generates the next camera images given past history and the commanded action, and the policy chooses the next action from those simulated images. This continues until the task ends (success or failure). In effect, the robot “imagines” out the trial in simulation. Step 4: Outcome and Quality Metrics. After simulation, we compare the generated rollout with the real result. We measure low-level video quality (frame consistency, geometry, semantics, motion smoothness) and high-level policy outcome. In particular, each simulated run is scored by “World Model as Evaluator Score” (WMES), a 0–3 rating of whether the eventual task succeeds or fails consistent with reality. These WMES labels are obtained from human reviewers or an automatic vision-language model (described later). In short, WMBench treats a world model as a surrogate robot: it must drive the policy through a whole episode just like the real robot did, and only then do we judge if the simulated final outcome (and intermediate visuals) match the real world. This closed-loop, outcome-focused evaluation is new. As the authors put it, “WMBench evaluates world models as robot policy evaluators, not only as video generators,” with a protocol that “mirrors real deployment”. The WMBench Dataset. The core of WMBench is its paired data of real vs simulated trajectories. The authors collected 2,989 unique episodes spanning eight manipulation tasks. Each episode has two parts: one rollout under teleoperated control (to capture a human execution) and one under an automated policy (a GigaBrain model). These include both successes and failures, roughly balanced 1:1 between teleoperation and model rollouts. The train set contains about 82,470 seconds of video, and the test set 7,200 seconds, carefully filtered to remove corrupt or duplicate clips. Importantly, they augmented this with a huge pool of simulated trajectories. Over 100 teams in the CVPR 2026 GigaBrain Challenge contributed thousands of short rollout segments (each 20–30 seconds or so). In total, 324,000 of these segments were stitched into long-horizon episodes. Three human annotators then labeled each episode with a WMES from 0 (wrong outcome, collapsed visuals) to 3 (correct outcome, high fidelity). This provides a massive benchmark corpus for analyzing world-model reliability. Put simply, the authors have nearly 3,000 real robot trials and 324,000 model trials all evaluated side-by-side, across diverse tasks. WMBench Metrics. Beyond WMES, WMBench measures video quality in detail. It computes perceptual fidelity (image and aesthetic scores), feature-level similarity (JEPA/CLIP scores), and consistency of multi-view keypoints. It evaluates geometric and semantic accuracy (are object shapes plausible, does the model follow instructions correctly) and motion smoothness over time. Crucially, it scrutinizes long-term coherence: how much does the simulation drift or degrade when rolled out for tens of seconds. Our content will focus on the high-level findings, but the takeaway is that the authors treat WMBench as a multi-faceted rubric for policy evaluation – not just an end-of-rollout success label, but also trajectory accuracy, object semantics, etc., all oriented toward judging policy performance. Curating the Training Corpus. To train GigaWorld-1, the team assembled a massive video dataset. In total, they collected 12,980 hours of labeled video from four complementary sources:. Internet and Physics Videos (1,298h). Generic web videos (e.g., physical phenomena, dynamics) to give broad visual-physical priors. Open-Source Robot Datasets (5,377h). Public robot manipulation logs from diverse platforms (single- and dual-arm robots like AgiBot, RoboMind, Galaxea, etc.) covering many tasks. Egocentric Human-Hand Videos (2,411h). First-person videos of hands manipulating objects (e.g. EgoDexterous) to emphasize fine hand-object interactions. Giga-Collected Robot Demos (3,894h). The authors’ own logged demonstrations on humanoids and dual-arm platforms, with calibrated trajectories. This mix ensures the model learns general physics from web videos and task-specific manipulation from robot data. The corpus spans humanoids, dual-arm robots, single-arm manipulators, and dexterous hands. Each clip went through an extensive quality-and-motion filter pipeline. Videos must have consistent metadata and decodable frames; any with overlays, occlusions, or motion artifacts are rejected. Clips with too little motion (or physically implausible abrupt motions) are also pruned. In robot videos, they even verify that the observed motion matches the recorded action inputs, using a vision-language model validator, to catch sync errors. After curation, the remaining clips are balanced by source, robot type, view, task, and motion intensity, and annotated. Each frame gets semantic object masks (via Segment-Anything) and depth maps (via Depth-Anything) for training. A fast/slow vision-language captioner produces short-term action labels and long-term context descriptions to provide textual conditioning signals. In sum, GigaWorld-1 is born of hundreds of human-years of video – carefully filtered and richly annotated – so that the resulting world model has both broad visual knowledge and fine-grained task detail. GigaWorld-1: Model Design. Building on this data, GigaWorld-1 is an autoregressive video diffusion model specially tuned for evaluation. It starts from a large pretrained latent diffusion backbone (like an updated UViM or similar). Critically, the authors freeze the frozen visual encoder and text encoder from that backbone, and instead learn LoRA adapters and lightweight control pathways for the robot domain. In other words, most of the heavy visual-spatial feature extraction comes from a pretrained network, while robot-specific learning happens via small trainable modules and condition injections. At rollout time, GigaWorld-1 keeps an autoregressive memory of past frames. A window of recent generated frames (and the original initial frame) is encoded through a history memory mechanism, while the model predicts the next block of frames. It mixes together (in the diffusion transformer) the noisy latent of the next frames plus temporally-aligned conditions. Those conditions include the commanded action, a predicted depth map, semantic segmentation masks, and text captions describing the task focus. Each generated frame block is then decoded to RGB, appended to history, and re-encoded for the next step. As the paper’s figure notes, “historical frames, future noisy latents, and temporally aligned controls (actions, depth, semantic maps, captions) are fused for embodied rollout generation”. This loop keeps the simulator grounded in both visual context and action signals. To make actions fully interpretable, GigaWorld-1 uses a unified control representation. For static (overhead) views, it converts the commanded end-effector pose into a 3D heatmap of where the robot hand should go. For dynamic (wrist) views, it uses a “ray map” that encodes camera motion versus scene motion. Both maps (with others) are concatenated and fed in as a single latent control vector at each step. This keeps the model aware of exactly what the robot is doing, which the authors found crucial for faithfulness. Long-horizon consistency is enforced via a hierarchical memory. The very first frame of the episode is stored as an “anchor” to preserve initial scene geometry. Beyond that, there are separate memories for long-term, mid-term, and short-term context: scene-level configuration, task progress, and recent motion respectively. During generation, all this memory context is fed into self-attention so the model doesn’t forget earlier details. (In cross-attention, they only inject the current text prompt.) For example, picking up an object should maintain the same background layout and object identity even after many steps; this multi-scale memory helps. There are some other clever touches. To smoothly transition between subgoals (say from “reach” to “grasp”), the model interpolates text embeddings over time using spherical linear interpolation, avoiding abrupt task jumps. Training the model is done in stages. First (Stage 1) they continue training the pretrained video backbone on the curated mixed corpus with a bidirectional denoising or flow-matching objective, so it learns robotics dynamics in a general way. Second (Stage 2), they convert it into an autoregressive roll-out model: this involves adding the relative-RoPE positional encoding, injecting the memory mechanism (first-frame anchor and multi-scale memories), and training the transformer to predict future frames given past frames + actions. Now the model can simulate forward one step at a time. Stage 3 is optional adaptation to a specific environment using Low-Rank Adapters – tuning to the lighting/appearance of a new workspace while keeping dynamics fixed. Finally, Stage 4 distills the large autoregressive teacher into a smaller few-step student model for speed. They use an ODE-based warm-start plus DMD2 distillation (teacher distribution matching + adversarial loss) to get real-time rollouts in many fewer diffusion steps. Two model sizes are released: a Nano with 1.3B parameters and a Plus at 5B. Even the 1.3B Nano can run at about 20 frames-per-second on a single <24GB GPU, and it can iterate practically indefinitely through time (“infinite horizon”) by streaming memories. In short, GigaWorld-1 is a carefully engineered video diffusion transformer that tightly integrates action controls and memory, all tailored to serve as a faithful simulator for policy rollouts. Experiments and Key Findings. With WMBench and GigaWorld-1 in hand, the authors ran extensive comparisons to draw their conclusions. They evaluated seven distinct video world models (including the GigaWorld-Policy model and various general video models) under WMBench, each with four different action encodings. In total over 324,000 rollouts were generated and analyzed. They measured hundreds of metrics (e.g. trajectory keypoint error in multi-view, JEPA similarity, semantic consistency) and the primary WMES outcome scores. The results confirmed the initial hypotheses:. Long-Horizon Consistency Beats Short-Term Image Quality: Across all comparisons, what mattered most was whether the model’s multi-step action outcomes matched reality, not whether each frame was perfectly photorealistic. A model could have slightly blurrier images but still correctly move objects for 20 seconds and get the outcome right – this was preferable to one that had mismatched action consequences. As stated succinctly, “evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism”. In practice, world models that drift or hallucinate objects halfway through a task failed to serve as reliable evaluators, even if their single-frame scores (PSNR or FID) were high. We see this on the WMBench leaderboard: GigaWorld-1’s Nano model (1.3B) leads in trajectory accuracy and task consistency, even though other models might tie or exceed it on raw image metrics. Data Balance over Blind Scale: Another insight is that simply having more data isn’t enough – the type of data matters. Large-scale video pretraining is helpful, but only if it’s balanced with robot-like content. The study found that adding broadly diverse physics videos improved the model’s general knowledge but could harm robot control if it overly dilutes task-specific signals. Therefore, pretraining gains “stem not only from data scale but from balancing general world knowledge with robot-specific controllability”. In other words, the 12.9K-hour corpus was carefully mixed so that GigaWorld-1 retained both broad visual priors and the particular semantics of manipulation actions. Models trained only on generic internet videos but no robot demos performed poorly at tasks like grasping, demonstrating this need for curated balance. Action Encoding and Memory Matter: Finally, the architecture of the world model makes a big difference. Models that aligned the action representation with each camera view did better. For instance, injecting the end-effector heatmap into the overhead view and the ray-map into the wrist view (as GigaWorld-1 does) yields much more precise multi-view trajectory following. Persistent memory is also key: ablations showed that removing the first-frame anchor or long-term context causes the simulation to lose track of objects. The authors highlight that design choices like “action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior”. Indeed, on the WMBench leaderboard we see a clear split: GigaWorld-1 (a robot-oriented model) outperforms general video diffusion models (like LTX or CogVideoX) across almost every metric. Performance on WMBench. Topping the WMBench leaderboard are the GigaWorld-1 variants. The 5B (Plus) model achieves the highest average score across all metrics, and even the smaller 1.3B (Nano) model ranks second overall. For example, GigaWorld-1-Nano gets about 0.89 on JEPA Similarity and Semantic Alignment (much higher than generic models’ ~0.6). In concrete terms, if we look at closed-loop success rates on tasks like “Put bowl on plate” or “Fold the box,” GigaWorld-1’s simulated rollouts result in the same success/fail outcome as the real robot vastly more often than alternatives. One striking illustration (from the paper’s results) shows that on Fold the box, many models fail (produce ❌ where the real robot succeeded ✅), whereas GigaWorld-1 mostly matches reality. This closes the sim-to-real validation gap. Over all metrics, the robot-specialized models clearly beat the off-the-shelf video ones. The authors point out that “robot-oriented models generally outperform generic video backbones” on embodied tasks. GigaWorld-1’s Nano model, for instance, has similar image quality scores to others but much higher trajectory accuracy and subject consistency scores. The end result is that GigaWorld-1 improved the alignment of simulated outcomes with real outcomes by roughly 14.9% over the next-strongest baseline. In practice this means plans and policies evaluated in GigaWorld-1’s world are far more predictive of their real-world performance. Scaling Evaluation with a Vision-Language Model. Annotating thousands of rollouts by hand is expensive, so the authors also built a vision-language (VLM) pipeline that can predict WMES automatically. Remarkably, this AI scorer agrees with humans almost perfectly. On a test of 5,000 rollouts, the VLM’s predicted score differed by at most one point from the human score in 99.16% of cases. (Two-level errors – e.g. predicting a success when it was a failure – happened in under 0.84% of videos.) In other words, the VLM can essentially replace crowd evaluation for large-scale experiments. This makes it feasible to continually test new models in WMBench without human labels, a valuable tool for rapid iteration. (This pipeline was also used in filtering and curriculum steps, such as the motion verifier for robot videos.). Practical Impact and Future Directions. GigaWorld-1 and WMBench together present a practical roadmap for building world models tailored to policy evaluation. The paper even lists its findings as design guidelines: use multi-view action maps, maintain rich memory, and include a mix of generic and robotic training data. By open-sourcing everything (the full code, model weights, and WMBench data are available online) they invite other researchers to plug in their policies and world models. Already, GigaWorld-1 can dramatically accelerate research. Instead of running dozens of real robot trials, a lab could roll out policies in a GigaWorld-1 simulation to filter promising approaches. It also opens up offline development: RL algorithms could be debugged and tuned against the model, even chain-of-thought planning could be evaluated, all without hardware wear-and-tear. The fact that a 5B model running at 20 FPS can simulate indefinitely means we can generate virtually unlimited evaluation data. Looking ahead, there are challenges to tackle. Current world models use only vision; adding depth sensors or tactile contact would be valuable. GigaWorld-1 was trained on specific robot classes – generalizing to new robots or novel objects remains to be seen. And of course, no model is perfect: there will still be corner cases where the simulation misleads. But this study lays a strong foundation. It shows us what works: namely, leaning on faithful action dynamics over photorealism, blending broad and focused data, and carefully structuring the model for control. In summary, GigaWorld-1 represents a major step toward making embodied AI research scalable. By benchmarking world models on complex manipulation tasks, the authors have highlighted the path to better simulators. For researchers in robotics and AI, this means a new tool: a “digital twin” that we can trust to evaluate robot policies, saving time and cost. As robotics moves into the era of large models, such surrogate evaluators will be essential – and GigaWorld-1 points the way. References: The above discussion is based on the “GigaWorld-1” project materials and paper (Wang et al.), including the WMBench benchmark description and results. All figures and results cited are from that work.