Latest / Embodied AI 101 / # Robotics: The Endgame

Transcript

# Robotics: The Endgame · Embodied AI 101

• Robotics: The Endgame. Robotics is entering a bold new phase. As NVIDIA’s Jim Fan argues, progress in embodied AI may mirror the trajectory of large language models (LLMs): massive pretraining on broad data followed by application-specific fine-tuning, with world models and reinforcement learning bridging the gap to real-world autonomy. This “robotics endgame” envisions a dramatic shift away from today’s language-centric approaches (so-called Vision-Language-Action, or VLA, models) toward vision-based world models that predict physical outcomes and guide actions. In this narrative we unpack that roadmap, highlighting key concepts like World Action Models (WAM), video-based world-model pretraining, new Dexterity Scaling Laws, and neural physics engines. Along the way we cite the latest research—from EgoScale to DreamZero to DreamDojo—that exemplifies these trends. From LLMs to Physical Worlds. Fan’s core analogy is simple but powerful: LLMs taught us that pretraining on enormous data can unlock broad capabilities. Early LLMs trained to predict text next tokens achieved tremendous generality; and then tuning (supervised fine-tuning, RL from human feedback, etc.) adapted them to specific tasks. Fan argues that robotics should follow that same recipe, but with one crucial twist: the “next token” of language becomes the “next state” of the physical world. In other words, instead of a model predicting words, we want a model predicting how the world will evolve given actions. This observation highlights an essential transition: robotics research has to move from a purely language/semantic paradigm to one where video and physics prediction are front and center. In practical terms, the proposed pipeline looks like this:. Massive Cross-Modal Pretraining – analogous to a language model’s unsupervised learning, but here using human videos and supervision (often language annotations) to teach an agent about objects, scenes, and motions. World-Model Pretraining – a second foundation phase where the system learns to simulate the world, typically by predicting future video frames conditioned on actions, using large-scale video. This is akin to giving the model an internal “neural physics engine.”. Task-Specific Fine-Tuning and RL – finally, the model is adapted with modest robot data (imitation demos, a bit of RL, etc.) to polish performance on real tasks. At this stage one can also run large-scale parallel RL in simulation to drive further gains. Fan’s keynote at NVIDIA’s AI Ascent summit explicitly parallels each LLM stage with a robotics analog: just as LLM research moved through unsupervised pretraining → instruction fine-tuning → RL-improvement → automated discovery, robotics can progress through large-scale video pretraining → small-scale robot fine-tuning → simulated RL loops. The aspiration of this pipeline is to shrink the “sim-to-real gap” and eventually let robots generalize across tasks and environments at human-like levels. The Limits of “Vision-Language-Action” Models. To appreciate why such a roadmap is needed, consider the state of the art in robot learning. A popular approach in recent years has been building large “Vision-Language-Action” (VLA) models. These systems typically learn from paired vision, language, and action data – for example, videos of humans demonstrating tasks, annotated with textual instructions, and with the corresponding robot actions. They can be very effective at capturing what to do in a scene when given a language command. However, Fan and colleagues critique this VLA paradigm as too language-centric. In effect, VLAs become “language-vision(-central) action” models: they allocate most of their capacity to parse semantics and sentences, with vision and action playing a supporting role. As Fan bluntly puts it, current VLA models “learn a bunch of knowledge and nouns, but physical operations and verbs are obviously lacking”. In practice, a VLA agent might excel at understanding what object you said to pick up, yet struggle to actually pick it up, especially if the physical situation is unfamiliar. Chinese press coverage of Fan’s talk dramatizes the point: NVIDIA’s past demo system GROOT and other recent growth models (like VIMA and various imitation-learning agents) use this VLA approach, but they are fundamentally limited. “This old model is all mouth, no hands,” Fan quipped – robots end up good at “words” but poor at “deeds.” The challenge is that language annotations and semantic understanding don’t fully capture the messy physics of the real world. A model might know the noun “cup” and the verb “tilt”, but without an internal model of liquids and collisions it fails when actually told to tilt an unseen cup. Fan’s response is to flip the paradigm. Instead of treating language as primary, he proposes robots learn by watching the world move. The new model is called a “World Action Model” (WAM), and its core idea is to jointly predict how the scene will change and what action will be taken. In a WAM, the input might be the current video frame (or state) plus a goal or subgoal, and the output is both (a) a prediction of the next video frames and (b) the low-level motor signals that cause that change. In essence, the agent learns both “what the world will do” and “what the robot should do” in one falling swoop. This is a big shift. Fan summarizes it as: “the robot first learns to predict what happens in the world, then it fine-tunes itself on how to act”. Concretely, his team’s approach (“DreamZero”) goes like this: (1) pretrain a video model on huge unlabeled video data so it learns to predict future scenes, (2) fine-tune that model on a smaller set of robot interaction data to align it with actual controls, and (3) optionally do a final RL pass to squeeze out remaining errors. The vision is that by swapping “language-LM” for a “video world model,” a robot can ground its understanding in real physics. The concept of predicting future scenes is not new per se – it ties back to classical world models in control (e.g. Sutton’s Dyna architectures) and to recent advances in action-conditioned video generation. What’s new here is scaling those ideas with deep diffusion or transformer models on massive video datasets, and tightly coupling the video prediction to action selection. That is, instead of treating video prediction or action prediction separately, a World Action Model decodes both simultaneously. In the DreamZero example, motor commands are rendered as if they were extra “pixels” alongside the visual scene, so the network outputs an image and a vector of joint velocities in one shot. This tight coupling helps the model learn the physics (the video) and the corresponding kinematics (the action) in lockstep. By contrast, a traditional VLA might only predict actions given language, without explicitly modeling the next images. The results of this approach are striking. In real-robot experiments, DreamZero (a 14-billion-parameter WAM built on video diffusion) was able to outperform state-of-the-art VLAs by a large margin. The team reports more than double the zero-shot generalization to new tasks and environments compared to top VLA baselines. DreamZero can carry out tasks in a novel setting with no additional training simply from its pretraining, whereas VLAs stumble on unfamiliar objects or dynamics. Moreover, thanks to model and system optimization, DreamZero runs closed-loop control in real time (about 7 Hz) despite its size. Perhaps most impressive, it can transfer across embodiments: a purely video-only demonstration from a human yields a 42% jump in performance on unseen tasks after just minutes of adaptation data. In short, the World Action Model concept combines video prediction and control into one generative model, and initial experiments show it is a powerful way to learn general skills. We have much to learn still about its tradeoffs and limitations, but it’s clear that WAMs like DreamZero are planting a flag for the “next frontier” of embodied AI. Data at Scale: Teleoperation vs. Egocentric Video. Building these large-scale models requires vast data. Fan’s talk emphasizes that data strategy and scale will break robotics open, much as they did for language. Up to now, the main source of robot data has been teleoperation: have a human operator remotely drive or kinesthetically guide the robot through tasks, and record the demonstrations. This was crucial for many early learning success stories, but it has a hard limit. A physical robot simply cannot work 24/7 without fatigue or failure. Fan notes that a real robot’s “effective working time per day” is more like 3 hours (and that’s if the robot stays in good condition). You can’t magically speed that up without incurring huge costs or breaking robots. To break through this bottleneck, Fan suggests a paradigm like Tesla’s self-driving pipeline: Make data collection seamless and ongoing. The idea is to equip humans or robots with sensors so that ordinary work or wearables generate useful training data automatically. NVIDIA’s team built a system called MANUS (a five-fingered glove with trackers) that maps human hand motion 1:1 onto a robot hand. Humans wearing this can manipulate objects naturally while the system captures joint actions. Combined with first-person cameras (much like Apple’s Vision Pro demos or point-of-view cameras), this setup can harvest egocentric video plus implicit actions. Fan calls this “FSD-like”: data collection in the background, without the operator even thinking about it. The practical result is that the training regime flips to being 99.9% human video + 0.1% robot data. In the EgoScale scheme, NVIDIA’s team pretrains their main model on 20,000 hours of human hand videos – and zero robot data at this stage – and then fine-tunes on only 4 hours of real robot demonstrations (plus 50 hours in sim), which amounts to less than 0.1% of the total data used. Remarkably, this reversed data mix is enough to learn extremely high-dexterity tasks such as card sorting, manipulating syringes, transferring liquids, etc. They even envision practitioners using such methods for at-home care robots in the future. The key enabler here is that egocentric video data is cheap and scalable. There are millions of hours of GoPro-style or helmet-cam videos on the internet of people performing manual tasks (think cooking, repairs, crafts). The challenge has been how to leverage it effectively. EgoScale is a pioneering answer: it demonstrates that simply feeding huge volumes of labeled first-person video into a vision-language-action model yields systematic improvements. The researchers trained a VLA model on 20,854 hours of action-labeled ego-video – more than 20× the largest previous efforts – and found a clear logarithmic scaling law: adding more data steadily decreases a validation loss in a predictable way. Crucially, this drop in loss translates directly to robot performance. The team found that the pretraining loss correlates strongly with how well the final model does on real robot tasks. In other words, more data reliably buys you more ability, just as in the language domain. This law of dexterity — a robotics analogue to LLM scaling laws — means engineers can plan around data size. For example, the NVIDIA team showed that by pretraining on thousands of hours, their final dexterous hand policy improved about 54% in average success rate over a no-pretraining baseline. The improvements followed a near-linear log-curve: every doubling of video hours produced a steady gain. They dubbed this relationship a “Dexterity Scaling Law”. It suggests that, given enough human video data, robot manipulation will keep getting better in a predictable way. This insight is powerful: it turns data into a software-like fixed asset, just like compute. The team even quotes NVIDIA’s Jensen Huang: “the more you buy, the more you save” (meaning more compute/data yields more efficiency in the long run). It’s worth noting that this approach requires “language+vision+action” labeling of video. EgoScale used videos annotated with a controllable trigger vocabulary (e.g. “pick up”, “pour into”, etc.) so the model had both visual frames and a compact action descriptor to learn from. But since languages can only encode so many actions in detail, EgoScale and its follow-ups also move toward deriving continuous latent actions from video itself (more on this below). Still, the broad lesson stands: robotics can piggyback on human video datasets as if those were the common crawl of our domain. The data flywheel here is: more human video → better initial models → fewer real demos needed → more focus on diverse scenarios instead of repeat data collection. World Models for Robotics: DreamZero and DreamDojo. Armed with these data-centric insights, we can look at the proposed new models. Two key examples from NVIDIA’s group are DreamZero and DreamDojo. Both leverage video-world-model ideas, but in different roles: DreamZero is a policy or action model, while DreamDojo is a simulator. Together they illustrate what you get when you train massive video-predictive models and connect them to control. DreamZero: A World Action Policy. DreamZero is the embodiment of the WAM concept. As we saw, DreamZero trains an autoregressive video diffusion model to predict the next few seconds of camera views and robot actions. The model is pretrained on a mixture of video data (human and robot) and then fine-tuned. In essence, it asks: “Given the recent camera frames, what is a plausible video sequence of the next time steps, and what controls would produce it?” Because it produces both, it can act as a closed-loop policy: at each time step it looks at camera input, samples a plausible action and then sees the predicted next frame – and those predictions feed into the loop. In one sense, DreamZero is a very large and slow neural network playing the role that classical simulators + planners would. The key advantage is generalization: DreamZero has seen huge amounts of varied data, so it can often correctly predict novel situations. For example, in one paper the authors show cases where DreamZero can move a Coke can on a picture of Taylor Swift – even though it has never been explicitly trained on that combination, it has seen enough videos of “move can” in other contexts to infer the physics and semantics. In contrast, a VLA trained mainly on language might misunderstand the physical nuance or fail if the object differs. Because DreamZero outputs predictions forward in time, it can attempt tasks even without any demonstration. After its original pretraining, you could give it a vague goal and it could simulate possible futures and choose actions accordingly. The NVIDIA summary claims over 2× zero-shot generalization improvement on new tasks compared to the best VLA baselines. It even adapts to a brand-new robot “YAM” with only 30 minutes of play data, carrying over skills from its prior training. This shows that a good WAM can function as a generalist policies engine. Two technical notes are worth highlighting. First, DreamZero runs surprisingly fast for such a big model: about 7 frames per second (7 Hz) in closed-loop inference. This was achieved through model pruning, caching, and a dedicated inference engine on NVIDIA hardware. Second, DreamZero supports cross-embodiment transfer very nicely. In one experiment a tiny 10–20 minute video of another robot performing a task was enough to dramatically improve DreamZero’s performance on that task by 42%. This suggests the world model inside DreamZero learns an abstract representation of how tasks map to videos, agnostic to the specific robot. That kind of adaptability will be crucial if a single foundation model is to serve many robot forms. In short, DreamZero exemplifies a World Action Model: it uses video as a dense encoding of world dynamics, and decodes both next frames and next actions simultaneously. This yields a policy that is both perceptually aware and physics-informed. It is the product of pretraining on massive data plus careful fine-tuning. As Fan observes, DreamZero effectively “looks at how the world moves, then learns how to move itself”. The implication is that future policy learning will revolve around such models, rather than just training policies that directly map from inputs to actions without an internal world-predictive component. DreamDojo: A Neural Simulator. While DreamZero focuses on control, DreamDojo focuses on simulation. DreamDojo is an end-to-end neural physics engine — essentially a learned simulator built on video prediction. It is trained on nearly 44,000 hours of egocentric human videos (the largest such dataset assembled for this purpose) and learns to autoregressively predict future video frames given past frames and actions. Its authors call it a “foundation world model” for robotics. DreamDojo’s innovation is in bringing world-model pretraining to the scale of human videos. Traditional robotics simulators use hand-crafted physics or engines (e.g. MuJoCo, Bullet) to simulate dynamics, but these are brittle for complex real-world contact-rich tasks. DreamDojo instead learns the dynamics from data. Because it had no explicit action labels in much of the data, the team equips DreamDojo with a latent-action oracle: during training it uses a separate action-extraction network to infer a continuous action vector between frames. In effect, every human video is augmented with a “latent action” that describes what change occurred. This serves as a self-supervised proxy: by conditioning the video predictions on these latent actions, DreamDojo internalizes the cause-effect in manipulations. After this massive pretraining, DreamDojo is fine-tuned on a small amount of real robot data. The result is impressive: the model can predict physical interactions with high fidelity. In experiments it “demonstrates a strong understanding of physics and precise action controllability”. The raw model, however, does not yet run in real-time — that’s where the distillation pipeline comes in. By distilling the giant transformer into a smaller one, they achieve about 10.8 frames per second at 640×480 resolution, with good long-horizon consistency. The distilled DreamDojo can thus simulate roughly real-time video. What can you do with this? DreamDojo can serve as a simulator for robotic tasks. You can feed it a sequence of actions (from a policy) and have it “run” the video forward. Multiple downstream uses are demonstrated: live teleoperation (the human moves the robot hand and DreamDojo predicts what would happen), policy evaluation, and even planning by imagining outcomes of candidate action sequences. Critically, because DreamDojo is learned from data, it requires no physics equations or graphics engine under the hood. The mantra is that “compute = environment, environment = data”: the environment is not coded by humans but implicitly by the millions of parameters trained on data. This approach has deep implications. In principle, one can now scale up “simulation” simply by scaling model size and data. Instead of hand-crafting new simulators for each robot or task, DreamDojo suggests we can lean on learned general simulators. In practice, this means future RL can leverage the same kind of parallel data-forwarding that LLM inference does: throw more GPUs at the world model and generate more training rollouts, all contained in neural nets. Fan envisions a “real-to-sim-to-real” loop: use DreamDojo to augment reality, run parallel RL inside it to improve policies, then deploy back in the real robots. In this vision, simulation and learning become one continuous loop, driven by data. So between DreamZero and DreamDojo, NVIDIA’s strategy covers both ends of the loop: DreamZero is the active policy learning agent, DreamDojo is the passive environment generator. Both rely on video-based world models to learn from human-scale data. Robot Learning Flywheels: From Humans to Hyperlearning. The talk emphasizes that data is the new frontier, much as with LLMs. Several strategies to fuel this “robotics data flywheel” emerge:. Wearable and Egomotion Capture (UMI): This refers to equipping humans with sensors (e.g. gloves, trackable rigs) to gather correlated video+action data. We discussed MANUS above. Such data is much easier to collect at scale than actual teleoperation of robots, because it piggybacks on human work. It does require some human effort (wearing the gear), so still a bit limited as a totally passive loop. Egocentric Human Video: The most scalable source. All the time, people are using phones or headcams; capturing their hands manipulating things (cooking, woodworking, car repair, etc.) can be a goldmine. By itself it’s just raw video, but DreamDojo and EgoScale show how to bootstrap it into robot-relevant data. “Screen Time” Telepresence: An intriguing idea alluded to by Fan was spirit of “background teleoperation” – robots collecting data behind the scenes like Tesla cars uploading drive logs without human intervention. In robotics, one might imagine any robot operating in daily life constantly streaming what it sees. That wasn’t elaborated in detail, but it fits the flywheel narrative: as sensors and connectivity improve, robots could log everyday interactions, feeding more real-world experience into the models. Simulated Data Amplification: Crucially, learned simulators like DreamDojo themselves become data generators. Once the world model is decent, you can sample countless trajectories inside it. Each rollout can then be treated as “experience” for policy training. The greater the fidelity of the DreamDojo, the more realistic the synthetic data. Thus the loop closes: policies trained in DreamDojo can be tested on real robots, small corrections made, and the cycle repeats. This resembles the “simulators as labs” model in autonomous driving (e.g. CARLA for cars), but here the simulator learns from data instead of code. Fan stressed that only a tiny sliver of real teleop data is needed after all this. In the EgoScale experiments, the final real-robot fine-tuning phase only used about 4 hours of actual demonstration (and 50 hours of simulator data). That’s under 0.1% of the entire training mix. It was enough to reach strong generalization on dexterous tasks. In effect, once you have the giant pretrained model, a few hours of ground truth is enough to anchor it to reality. This is reminiscent of how a large language model might need only a little fine-tuning data for a new domain. Another aspect is parallelism. Just as LLMs and other models use thousands of GPUs to speed training, Fan suggests using “massively parallel RL” inside learned environments to break bottlenecks. For example, one could run hundreds of instances of DreamDojo on a DGX cluster, training many policies in tandem and cross-seeding improvements. This would require new tools (we’re not there yet), but it illustrates the vision: robotics research can be automated the same way language model research is, with compute farms churning out policies. Scaling Laws and Future Frontiers. Let’s summarize the most concrete principles Fan and team have outlined:. World-Model Pretraining: Treat “future-state prediction” as the fundamental task, scaled up to video. Like token prediction in LLMs, world-model prediction over video is the self-supervised task that injects broad knowledge about object interactions and physics. In practice this means training generative video models on huge datasets, then aligning them with actions. Vision-Language-Action to World Action Models: Abandon language as primary; instead weave language, vision, and action together around the world model. DreamZero and related WAMs exemplify this, combining video diffusion backbones with action outputs to create “imagine-then-act” policies. Data as the Engine: Build pipelines that feed on human-scale data. This includes wearable capture, massive egocentric videos, and simulated rollouts from models. The core idea is to create self-reinforcing loops: more data → better models → more data (via synthetic and expanded capture) → still-better models. Dexterity Scaling Law: Anchor belief that manipulation performance will continue to improve logarithmically with more data and compute. EgoScale showed a clear log-linear relation for dexterous tasks. If this holds, then robotics becomes a curve-fitting game: gather data until you achieve your desired loss on a held-out validation set, and you can predict the outcome. Data Flywheels: Fan hints at various ways to keep the flywheel spinning. We’ve mentioned passive capture. Another concept he mentioned is that teleoperation data can be downsized to a spark, not a fire: perhaps only a few hours of high-quality demos are needed if the rest is covered by cheap data. Indeed, he notes that under 0.1% of the overall data (sim or real) contributed in experiments to mastering complex tasks. Neural Simulation (DreamDojo): By learning simulators directly from visual data, robots won’t rely solely on hand-coded physics. DreamDojo sets the stage: “compute = environment = data”. In other words, improving the simulator is a matter of adding training examples, not re-writing equations. The Long-Term Horizon: Fan even laid out a high-level vision of where this leads:. A Physical Turing Test, where it becomes hard to tell robot and human performance apart on common tasks. A Physical API, where fleets of robots are orchestrated like software processes, configurable via code. Autonomous Robot Research, where robots themselves design and assemble the next generation of robots with minimal human input. These may sound utopian, but they serve to illustrate the ultimate goals. At a practical level, the focus now is on proving out the building blocks (WAMs, ego-data, world models, etc.) and showing consistent gains on benchmarks. Putting It Together: How to Use These Ideas. For researchers and practitioners, what concrete lessons can we draw?. Focus on World Models: If you’re working on robot planning or control, consider integrating a learned predictive model of future frames. Instead of directly training a policy in one stage, break the problem: first learn a video dynamics model (even without actions), then condition it on actions to align with controls. This is what DreamZero does. Even approximating this idea (e.g. with video forecasting plus an inverse model) could yield dividends. Leverage Human Video: Don’t limit yourself to robot-collected data. Explore public egocentric video datasets (EPIC-KITCHENS, Ego4D, etc.) or record first-person videos of tasks. Use these for representation learning. Work on algorithms that can extract proxies for actions from unlabeled video (latent action discovery). The continuous “latent actions” approach used by DreamDojo is one example; others like AdaWorld or GENIE are emerging in the literature. The goal is to turn every frame of human video into training signal for manipulation. Adopt Two-Stage Training: Mirror EgoScale’s recipe. Pretrain large models on broad unlabeled or weakly-labeled data, then finetune on narrow robot data. The pretraining could be unsupervised (just predicting frames) or semi-supervised (with language or noisy labels). The finetuning is where you inject exact tasks and correct errors. Experiment with Any-Scale RL: Invest in parallel simulation. NVIDIA’s vision suggests that as soon as you have any world model, you should start using it as a playground for policies. Already frameworks like Isaac Gym, Brax, and JAX-based sims are enabling thousands of parallel rollouts. If you replace or augment the physics-based sim with a neural model like DreamDojo, you can do even more diverse scenarios with less engineering cost. Even a crude learned simulator is better than none. Measure Scaling Trends: Whenever possible, quantify how performance scales with data and model size. The log-linear trend in EgoScale came from carefully measuring validation loss as data grew. Building that curve is valuable for future extrapolation. We should gather more scaling data points on other tasks (e.g. locomotion, object rearrangement) to test the universality. Benchmark WAMs and Hello: The concept of “World Action Model” is broad; there will be many realizations. The community should develop standardized benchmarks where models must jointly predict vision and action. DreamZero reports results on some unseen tasks – others should follow with different robots and tasks, to see how reproducible the 2× generalization is. Likewise, open-sourcing data or code (NVIDIA plans a GitHub for DreamZero) will let others compare approaches. Conclusion. We are witnessing the outlines of a shift from “brains-only” AI to fully embodied intelligence that leverages the success of large models. Like LLMs, these new robotics models will require vast data and compute; unlike LLMs, their data is video and their task is dynamical. The Ro botics Endgame, as Jim Fan dubs it, is a world where robots have predictive bulks of we do, bridging the gap between seeing the world and acting in it. Key to this vision are video-based world models that learn from human behavior and from interaction data, connecting them via latent actions to actual robot motion. The Nvidia “Dream” projects are early realizations of this: DreamZero shows the power of World Action Models to produce generalist policies, and DreamDojo shows that a neural simulator can emerge from data. Meanwhile, EgoScale and related work quantify how performance grows with scale, giving us a roadmap of how much data we need. For the robotics community, this means thinking differently about both models and data. We should be ready to copy the LLM playbook where it makes sense: invest in massive foundation models and then fine-tune. But we must choose the pretraining tasks wisely: the physics of our world, not just words and dialogs. As Fan puts it: the game is now about “predicting the next state of the physical world” rather than “predicting the next token of text”. Embracing that fully — through world models, data flywheels, and scaling laws — could indeed unlock the “robotics tech tree” by 2040, much as GPUs and data unlocked vision and language before. The rise of WAMs and video simulators is a bold bet, but one grounded in concrete research results. If the scaling laws hold and the data pipelines keep spinning, these ideas may well take robotics into its long-promised endgame.