Latest / Embodied AI 101 / World Models for Robots: The Next Big Leap?
Transcript
- World Models for Robots: The Next Big Leap?. Will World Models Allow Robots to.? • Agility Robotics (Chris Paxton) • Date unknown. In recent months, robotics leaders have started talking about “world models” – learned simulators of physics built from big data – as a path to more general robots. In a new blog post, Chris Paxton of Agility Robotics laid out this idea: that a robot could watch enormous video datasets and learn a predictive model of its environment, then use that model to plan actions in situations it hasn’t seen before. This would be akin to a human imagining where the ball will land after you throw it. Proponents argue the idea could let robots solve new tasks by “dreaming up” or simulating outcomes from first principles. But is this just hype, or is there real progress? We’ll break down the core concept, survey recent demonstrations, and ask what still needs to be solved. In short: world models can give robots new generalization power – as recent experiments show – but they also bring huge new challenges in data and computation. What Do We Mean by a “World Model”?. The term world model has many meanings. In classic robotics it often meant a map or state estimator. More recently, inspired by AI research, it means a predictive model of how the world changes. As one summary puts it, a world model is essentially “a (potentially latent-space) dynamical model that predicts how the state of the world evolves under actions”. In other words, the robot learns to forecast what the camera would see (and maybe its proprioception) a few moments into the future for various possible actions. This echoes Jürgen Schmidhuber’s definition from the 1990s: predicting future sensory data given current actions. In practice, modern world-model proposals use deep networks (often diffusion or transformer models) trained on video. Given a current image and a candidate action, the model generates a future frame or latent embedding, effectively running an “internal simulation” of physics and image changes. This is unlike a policy model that directly predicts an action for a task; instead the world model predicts what happens next, and an outside planner or control policy can use those predictions. The hope is that by learning from thousands of hours of video (from humans, simulations, etc.), the model implicitly captures physical laws like gravity or object affordances, without a human writing equations. How World Models Can Help Robots. World models promise a few big advantages for robotics:. Advanced planning and planning: A robot with a world model can “think ahead” by rolling out possible action sequences. Andrew Ansell describes that instead of a policy that goes from observation to action, a world model “allows the robot to not only perceive but also predict future states of the environment before taking action”. By visualizing possible futures, the robot can detect failures before they happen and replan. Generalization across tasks: Since the model learns general dynamics, it can, in principle, handle novel scenarios. A recent workshop paper showed that one learned world model could generalize to many assembly tasks. The authors trained a single visual world model on 90 different object-insertion tasks and then tested it on completely new objects. Amazingly, the world-model approach succeeded on new insertions 56% of the time, compared to only 7% for a traditional model-free policy. Crucially, adding more diverse objects to training improved zero-shot success, suggesting strong scalability. Scaling via video data: By building on advances in generative video, we can leverage internet-scale video datasets. For example, startups like Tau Robotics and 1X have begun training video diffusion models on large unlabeled video collections. One report describes training a 1-billion-parameter transformer-based video predictor on 500 million video frames (from Egocentric human video and other sources). Once trained, this model could predict the next frames of a robot camera stream and even propose presumed future actions. Ansell notes that techniques like masked autoregressive diffusion (MarDini) now allow robots to “imagine” long-horizon future videos to plan uncertain scenarios. In short, generating synthetic video of robot physics becomes feasible as video models improve. Narrowing the sim-to-real gap: Traditional robot learning either used oversimplified simulators (which fail in the real world) or policy learning directly in hardware (slow and dangerous). World models offer a hybrid: learn a simulator from real data. Instead of hand-coded friction coefficients, the network learns how cups topple or how human hands grasp objects. As one article explains, existing simulators have to simplify contact physics and sensor noise; world models can learn those details from data. DeepMind’s research on learned simulators shows that future algorithms could even extract explicit physics parameters after training. In practice, if a robot’s world model truly captured its real environment, it could train policies almost entirely in its internal simulated realm. To see these ideas in action: 3One4’s report mentions Google DeepMind’s “Genie 2” world model – from a single image it generates an interactive 3D world, modeling character animations, water, gravity, etc. Another example is 1X Technologies’ work: they trained on thousands of hours of real humanoid robot videos (their EVE robot) and built a model that can imagine multiple futures given a starting robot state. That 1X model can simulate multi-step tasks like folding laundry and even “evaluate at scale” by testing policies across millions of imagined scenarios. These demos hint that robots could learn new tasks by watching videos rather than through trial-and-error. What We’ve Seen So Far. A few early results give glimpses of world models at work:. Video prediction for planning: Ansell’s overview of recent papers notes models that integrate video generation into control. For instance, a 2023 paper uses text-and-image prompts to generate robot trajectories as video sequences, then uses an inverse-dynamics model to infer the required actions. In tests, diffusion-driven video models let a robot “preview” different action sequences and choose one that fulfills a high-level goal, enabling more robust planning. Such work is still experimental, but it shows the principle: combine generative video with dynamics inference. Generalizing manipulation: The RSS insertion result is a striking proof-of-concept. By training a single latent model of peg-in-hole for many shapes, the robot could assemble unseen parts it had never been programmed for. This is exactly the kind of generalization world models are hyped to enable. It suggests that, at least for some tasks, learning an internal simulator then planning in it can beat naive policy learning. Large-scale RL alternatives: Parallel to world models, some teams simply train big policy networks with lots of data. For example, Figure.ai’s recent Helix model (not a world simulator per se, but a vision-language-action network) showed a 7-billion-parameter controller that picks and places thousands of novel objects with a single set of weights. It achieved dexterous multi-robot tasks with just ~500 hours of supervised data. This isn’t a “world model” in the predictive sense, but it shows that large models can unify many tasks without explicit simulation. It provides a baseline: is it easier to scale a raw action model (like Helix) or to scale an internal world model plus planner? The answer will likely vary by domain. Industrial interest: Big tech and startups are betting on world-model-like ideas. DeepMind’s recent “RT-1X” and “RT-2” papers demonstrated that training one large model on diverse robot data yields surprising multi-task performance. NVIDIA’s Cosmos platform and Google’s Genie series are explicitly marketed as “AI world simulators” for robots and virtual agents. World Labs (founded by Fei-Fei Li) just raised a huge round to produce 3D environments from static images. This mounting momentum suggests mainstream attention, but it’s still uncertain how many robotics problems will truly be solved by these tools. The Hurdles and Open Questions. Despite the promise, critical challenges remain:. Data scarcity: Robots are nothing like internet images. The IEEE Spectrum recently summarized a major obstacle: we simply don’t have the data. While we have billions of web images and hours of video, we do not have comparable “visual + action” datasets for robots. A self-driving car has simple controls and tons of road video, but a humanoid robot has dozens of joints and virtually no broadcasted footage of all that movement. As the Spectrum piece notes, internet-scale robotics data doesn’t exist: people aren’t uploading action-labeled robot videos much. Even companies collecting data (Figure.ai, Redwood, etc.) have only tens of thousands of demonstrations, far below the scale of image and text corpora. Collecting truly varied robot video - especially multi-camera, multi-joint recordings - remains a bottleneck. Embodiment variety: Robots come in so many shapes (arms, legs, hands, wheels, etc.) that a one-model-fits-all seems far-fetched. The Spectrum article also cautions that Spot robots and industrial arms emit very different telemetry, so a single large model might have trouble covering all morphologies. This could limit how “general” a world model can really be, unless one training effort aggregates data from many robot platforms. Compute and latency: Even if data existed, training these models is expensive. The Tau Robotics example (in an internal report) shows the costs: they trained a 1-billion-parameter video model for 42 hours on 64 NVIDIA H100 GPUs just to pre-train on 0.5 billion frames. And the resulting model was still slow: 110 frames/s on a single GPU for latent inference. Another industry engineer quipped that their humanoid model needed about 11 seconds of internal “thinking” to plan 5 seconds of motion. That kind of latency may be unacceptable for real-time control. In short, scaling up yields capability but also huge hardware requirements (more GPUs, better batteries to power them, etc.). Realism vs hallucination: Learned simulators don’t guarantee physical accuracy. Current video models are great at visual plausibility but not at obeying exact physics. One could train a video diffusion model to roll a ball across a table, but it might not respect friction or inertia precisely. A recent commentator dubbed this the “causation gap”: models can show a candle melting, but they don’t explain why it melts, so they might fail in novel conditions. For robots interacting physically (grasping, deformable objects, etc.), this is crucial. Model errors could lead to unsafe actions. Although world models could encode constraints (like rigid-body dynamics) as inductive biases, current large generative models lack that grounding. Robotics experts worry about “hallucinations” that could cause crashes or accidents, and it’s currently hard to bound those in a learned simulator. Long-horizon complexity: Simple world models can predict the next few frames, but real tasks often span many seconds or minutes with branching outcomes. Generating coherent, collision-accurate video over long horizons is still unsolved. The Tau team remarks that their model “generates accurate predictions for simple scenes, while generating complex collisions or other humans remains challenging”. And as the Spectrum commentary points out, even 99%-success in everyday tasks might not be enough for industrial deployment due to long-tail edge cases. In other words, world models may improve average performance, but guaranteeing safety under every unusual circumstance is a different problem. How to Think About World Models vs Pure Scaling. It helps to compare world-model approaches with the prevailing trend of doing “more data, more parameters” robotics. The scaling advocates argue (reasonably) that vision and language thrived on simple architectures plus scaling, and early robot models (like DeepMind’s RT-1X) show some promise. Upsides of scaling include using commodity GPUs, existing ML frameworks, and a straightforward path to iterate (collect more data, train a bigger transformer, repeat). Those systems also get to use all the latest transformer or diffusion innovations directly. By contrast, world models are still awaiting an LLM-like breakout. They often require custom architectures (latent simulators, diffusion pipelines), and it’s unclear how to run them on hardware efficiently. We don’t have “World Model chips” yet the way we have GPUs for transformers. In terms of raw performance, scaled-up policy models have already begun to do impressive multi-task behaviors in labs and even light industrial settings. On the other hand, world models offer conceptual advantages: interpretability and potential safety. A model of the world is something we can inspect, constrain, and reason about. For example, in manipulation one could embed a collision-check in the world model or use classical optimization (MPC) on top of it. This modularity might help catch errors before they harm the robot or environment. Also, a world model might generalize better out-of-distribution, because it encodes causal structure instead of mere patterns. Early evidence (like the zero-shot insertion success) supports that world models can dramatically boost generalization on novel tasks. Most experts foresee a middle path. As one commentary puts it, the division is a spectrum, not a binary: “world models, when coupled with methods that know how to use them, are strong at generalization…; deep learning excels at parsing and summarization of visual data”. We may see hybrid systems: a big video model provides priors (for example, predicting how a scene will evolve), but a neural policy or classical controller still steers the actual robot in real time. Some ideas include top-down LLM-like reasoning feeding goals to a world model, or world-model-based planning that spawns subgoals for a fast reactive policy. What Changes If It Works?. If world models really delivered, it could transform robotics R&D. Companies would likely invest heavily in data collection (video data, human teleoperation, or simulation). Indeed, the frontier players are already building “data pipelines” like autonomous fleets of robots or virtual simulators to feed these models. A reliable world model would mean robots could be pre-trained from factory-shipped, then cheaply fine-tuned for each new task by adjusting parameters, rather than programming each scenario from scratch. In industry terms, it could lower the time-to-solution for new tasks. Instead of writing a model for every conveyor belt or warehouse layout, engineers could feed a generalist model a description (or a short demonstration video), and the robot would “figure it out” by internally simulating options. This is the promise of the “general-purpose AI robotics” narrative we’re hearing from companies and media. However, remember that previous wave of “robotics is the new AI” hype often outpaced reality (e.g. driverless cars, general-purpose agents). As one writer cautioned: synthetic sims are promising, but “how quickly will robots become more useful?” remains to be seen. In practice, any deployment will still need extensive validation: even a world model with 95% accuracy isn’t fully trustworthy. Inevitable edge cases will emerge only with real-robot testing. Simulation (even a learned one) can’t eliminate the need for actual trial runs and safety checks. For now, robotics engineers should watch these developments carefully but critically. World-model research is advancing fast: new diffusion-based video predictors and large-scale experiment papers are coming out every few months. They may solve narrow tasks impressively. But for a truly uncanny general-purpose humanoid or field robot, we probably aren’t there yet. Scale helps, as the learning-theory crowd notes, but it isn’t a free lunch. Ultimately, robotics progress has always required integrating software with better hardware: we’ll need improvements in sensors, actuators, and energy storage too, to fully leverage any world-model intelligence. Takeaways for practitioners: World models point to a potentially powerful new paradigm – imagine-and-plan via video – that is actively being pursued by startups and big labs. Initial experiments (robotic insertion, video-conditioned policies) validate that idea in limited settings. But they also highlight common limits: data hunger, compute load, and imperfect physics. In your own work, focus on parts of tasks where predictive modeling can shine (like manipulation with heavy visual dynamics) and be skeptical in areas where missing physics could be dangerous (like delicate assembly). Keep an eye on emerging datasets (robotic video corpora, egocentric recordings) and new architectures (hierarchical simulators, hybrid models). Even if your project doesn’t end up using a full generative world model, ideas like planning in a learned latent space or synthesizing training data will likely influence future robotics tools. In summary, world models are an exciting direction for robotics: they offer a way to leverage all that AI data-driven learning into the physical world. Experiments show they can improve generalization and planning, but the engineering path is steep. True deployment will require orders of magnitude more data and smarter model architectures. For now, they complement rather than replace existing methods – one more tool in the researcher's toolbox. Whether they become a silver bullet or just another incremental improvement depends on the next few years of research and engineering. Sources: Key references include Paxton’s overview (itcanthink substack) and the robotics community’s response: analysis pieces on the world model approach, recent system demos (DeepMind’s “Genie”, 1X’s humanoid model), and technical papers like the RSS workshop study. For in-depth discussion on scaling vs modeling, see the IEEE Spectrum robotics report and related blogs. Each provides data points on the potentials and pitfalls of applying large data-driven world models to embodied robots.