Latest / Embodied AI 101 / Cosmos 3: Omnimodal World Models for Physical AI
Transcript
- Cosmos 3: Omnimodal World Models for Physical AI. Roboticists and embodied-AI researchers have long dreamed of a unified “brain” that can both see the world and imagine it – a single model that understands language, images, video, speech, and even robot actions, all together. In practice today we tend to build separate modules: one vision-language model for captioning scenes, another model for robot planning, another for video prediction, another for policy learning, and so on. NVIDIA’s new Cosmos 3 flips that approach on its head. It is an omnimodal world model: one large transformer that can process and generate text, images, video, audio, and sequences of actions in the same framework. In NVIDIA’s words, “Cosmos 3 connects understanding, generation, simulation, and action through a shared omnimodal world model” that spans “text, images, video, audio, and actions”. In short, Cosmos 3 aims to subsume what had been separate specialist systems – vision-language models, video generators, world simulators, and world-action models – into one single model. It does this with a novel mixture-of-transformers architecture that has two “towers”: an autoregressive language-vision reasoning tower and a separate diffusion-based generation tower. The language tower handles text and video/image features for understanding and planning, while the diffusion tower handles image/video pixels, audio, and action tokens to generate futures. This makes Cosmos 3 extremely flexible. It can be prompted with any mix of modalities (say a text instruction and a video clip so far) and produce any other modality (say the next few video frames or even a robot action sequence) by arranging the appropriate tokens into these two subsequences. In practical terms, that means one model can do text-to-image, text-to-video, video prediction, video captioning, speech-to-text, speech-to-video, and even action planning tasks – all with the same backbone network, rather than separate specialist nets for each. Cosmos 3 is specifically built for Physical AI – robots and agents that must move through and affect the real world. In Physical AI, an agent constantly perceives, reasons about, and acts in a dynamic environment, and we need models that tie those together coherently. As the authors note, training a robot entirely in the real world is often too slow, dangerous, or impractical. In response, the community has assembled pipelines of separate models (for perception, simulation, policy, etc.), but stitching those fragments together introduces inconsistencies and inefficiencies. Cosmos 3’s unifying approach aims to avoid that fragmentation. By jointly training on a huge mixture of multimodal data, it learns a coherent representation that can “interpret environments, simulate physical dynamics, infer actions, and predict future observations” all within one system. In effect, Cosmos 3 is a world model in the tradition of Ha & Schmidhuber’s work, but on steroids: it is trained end-to-end on images, video, sound, text, and control data to serve as both a perception system and a simulation engine for physical tasks. Architecture: Mixture-of-Transformers. At the heart of Cosmos 3 is a unified transformer architecture that can handle any modality. The model first uses modality-specific encoders to turn raw inputs into token sequences in a shared latent space. For language it uses standard word-piece embeddings; for vision it actually uses two vision encoders simultaneously. A Vision Transformer (ViT) projects images or video frames into tokens (the ViT is trained with the model backbone), while a Vector-Quantized VAE (VQ-VAE) produces a second set of tokens for generative outputs. The ViT encoder uses 16×16 pixel patches and applies a “DeepStack” pooling to produce a rich feature token stream, whereas the VAE encoder aggressively compresses the image: reducing resolution by 32×32 spatially and 4× in time (for video). The ViT tokens feed into the model’s reasoning tower, and the VAE tokens feed into the generation (diffusion) tower. Audio is handled similarly with a frozen audio VAE that chops sounds into latent tokens. And remarkably, actions are also turned into a token stream. Cosmos 3 uses a unified action interface across different embodiments: actions from a wheeled robot, a manipulator arm, a drone, etc. are all projected into a common latent vector of shared geometry (3D translations, 6D rotations, grasp states, etc.). In other words, an action like “move end effector by (dx,dy,dz) and rotate by (drx, dry, drz)” or “drive car forward 1m and turn 15°” is encoded as values in a fixed scheme of translations and rotations. This joint action encoding means the same model can plan for many robotic platforms without hardcoding each one’s action space. Once each modality is encoded into tokens, Cosmos 3 arranges them into two interleaved subsequences before feeding them to a Mixture-of-Transformers (MoT) backbone. One subsequence is autoregressive (AR): it contains language tokens and vision tokens from the ViT encoder. The other is a diffusion (DM) subsequence: it contains VAE tokens (for images/video), audio tokens, and action tokens. The AR subsequence is processed by a set of transformer decoder layers tuned for language-like prediction: it can read a text prompt plus some video context and produce text or other symbols in an autoregressive way. The DM subsequence is processed by a separate set of transformer parameters optimized for denoising/diffusion: it can take a noisy target image sequence and gradually refine it into a coherent image or video. Crucially, the AR and DM towers share most of their weights (hence “mixture of transformers”) but have distinct projection heads, so the model can flexibly do both token generation and diffusion. For example, to do text-to-image generation, the AR subsequence contains only the language prompt tokens, while the DM subsequence contains a random-noise grid of VAE image tokens. The model’s diffusion tower then iteratively denoises those tokens into a high-quality image pixel by pixel (via the VAE). Similarly, text-to-video would put the text in AR and a noise video clip in DM. For a video-prediction task (forecasting future frames of a video), the AR subsequence could include tokens for past frames (via the ViT) and any instruction text, and the DM stream is the target future frames’ tokens to be generated. Because the model was trained on all these configurations in one go, it can gracefully handle mixed I/O setups. Internally, the ViT and text interact in the AR path to build contextual understanding (“the robot pushed the cup” or “a car is turning”), and that informs the generator’s DM path to render the next frames or sounds. Intuitively, you can think of Cosmos 3 as having a single large “engine” of transformer layers, with two smaller output mechanisms: one head that acts like an autoregressive language model (copying words/images forward step by step) and another head that acts like an image/video diffusion model (denoising pixels). By structuring the tokens carefully (text tokens plus ViT image tokens in the AR block, VAE-image/audio/action tokens in the diffusion block), the model seamlessly does both. This unified mixture-of-transformers design was key to Cosmos 3’s flexibility. Beyond vision and action, Cosmos 3 also handles audio. During training it saw millions of short audio clips (speech and environmental sounds) and learned to generate or understand sound along with the visual scene. For example, it can imagine the sound that should accompany a generated video, or interpret instructions that reference sounds. By contrast to multimodal models that only do vision+language, the addition of video, audio, and actions truly makes Cosmos 3 a world model – it’s modeling entire episodes of a physical environment across many senses and agent motions. Cosmos 3 comes in multiple sizes and variants. The paper describes a smaller Cosmos3-Nano (~16B parameters) intended for more efficient use, and a larger Cosmos3-Super (~64B parameters) for highest-quality generation. There are also specialized forks of the model: for example, Cosmos3-Super-Text2Image is fine-tuned for photorealistic image generation, and Cosmos3-Nano-Policy-DROID is a version trained for robot control in the DROID manipulation benchmark. In all cases the underlying architecture – mixture-of-transformers with AR and diffusion streams – is the same. The Nano and Super differ in depth and width (16B vs 64B params), so the Super can generate more detailed outputs at higher resolution, while the Nano is leaner for faster inference. (Indeed, experiments show the Super can achieve 3–4× higher throughput on large GPUs like the H100 compared to the Nano, at the cost of using many GPUs to run in parallel.). Data and Training. Building a model like Cosmos 3 requires mountains of data. The authors assembled a massive multimodal dataset drawn from the web and synthetic simulators, tens of billions of video frames and images in total. They split this data into two streams during training:. Reasoner Pathway: This is paired data mainly for understanding and language grounding. It includes images with captions, video clips with transcripts, and even text-only dialogues. In total the Reasoner set has about 24.2 million examples, split into 22.0M for broad pre-training and 2.2M for more focused fine-tuning. The pre-training on this path emphasizes general visual understanding (e.g. large-scale image-text pairs), while fine-tuning shifts to Physical AI domains like driving, robotics, and smart infrastructure. About half of the fine-tuning examples are video+text, so the model actually sees narrated videos. Sources for this stream include public datasets like LocateAnything, dashcam footage from Nexar, the MADS driving dataset, synthetic driving scenes from MimicGen, the BEHAVIOR-1K robot benchmark, and even proprietary internal logs. This ensures Cosmos 3 learns the language-vision correspondence and scenario descriptions that are common in robotics and autonomous vehicles. Generator Pathway: This is unannotated raw media for generative modeling. It contains about 767 million images and 347.7 million video clips filtered from web-scale collections. These are not paired with captions; they simply provide learnable targets for image/video diffusion. In mid-training, the authors also introduce a curated subset of this data: about 15.6M images and 74.7M videos that are biased towards Physical AI scenarios (e.g. traffic scenes, warehouse interiors, robot arms) and high-fidelity synthetic data. Furthermore, they include lots of audio: 12.8M unlabeled non-speech clips (ambient sounds) and 6.0M clips of speech (with lip-sync), so the model also learns natural audio. To support action modeling, the authors created and released five large synthetic interaction datasets covering domains like basic physics, robotic manipulation, driving, digital human motion, and warehouse safety. All this raw media is cleaned and deduplicated (for example by clustering embeddings to remove near-duplicates), and filtered for quality. The overall training strategy is staged: first pre-train on generic web-scale material, then fine-tune on more specialized Physical AI content. Importantly, Cosmos 3 training is joint across modalities. During training the model sees many combinations of input-output pairs. Sometimes it sees an (image, caption) pair to learn vision-language alignment. Other times it sees a short video clip and is asked to predict the next frames (video generation). Other times it sees a text instruction and is asked to produce an image (text-to-image). For actions, sometimes it sees a partial trajectory and is trained to predict the next actions (imitation learning style). The joint loss blends autoregressive cross-entropy (for language tokens) with diffusion losses (for image/video reconstruction) in the same model. The result is a single representation capable of transferring knowledge from one modality to another. In fact, ablation studies show that this joint training is crucial: when the model is also trained to predict actions, it learns a common “cross-domain prior” of physical structure that speeds up adapting to new robot embodiments or scenes. Likewise, tying the vision encoder to the reasoning tower significantly improves the physical plausibility of its generated images and video. In other words, the understanding head actually helps the generative head produce more physically coherent worlds. All of this training was done on supercomputers with hundreds or thousands of GPUs over many weeks. When the dust settled, NVIDIA released model checkpoints and pipelines under an open license (OpenMDW-1.1) so others can experiment. The final models require substantial hardware (e.g. the 64B model is typically sharded across 8+ H100 GPUs for inference), but NVIDIA also provides a “Nano” 16B variant and integration in both HuggingFace Diffusers pipelines and their VLLM serving framework. Unifying Understanding and Generation. What can one actually do with Cosmos 3? The most striking selling point is that it can understand scenes and generate them seamlessly. For example, it can answer questions about a scenario (“What will the robot do next when it picks up the red box?”), then immediately try to show that scenario as a video. It can take a language plan (“The car turns left in the intersection, then the siren turns on”) and render a short driving video clip illustrating it. It can watch a robot arm moving blocks (input video), and predict the rest of the motion (output video) or even output the corresponding joint commands. It can look at an image of a room and a text hint like “make it snowy”, and generate a transformed image. All these tasks usually require separate systems (LLMs, image-to-image models, video prediction networks, motion planners) – but with Cosmos 3 they use the same backbone. In effect, Cosmos 3 embodies the long-held AI vision that language, vision, planning, and action should live in one space. The authors note that prior work often “stitches together isolated vision-language models, video generators, and action predictors,” which is unwieldy. By contrast, Cosmos 3’s unified approach means a single memory of the world. For instance, if the model reads the text “pick up the green cup” and sees a video of a robot, it will internalize that task as part of the same token sequence that it uses for generation. This integrated memory allows it to “interpret environments, simulate physical dynamics, infer actions, and predict future observations” without any external glue. In a sense it becomes a world-language foundation model for robotics: you can query it with text or images and it produces grounded action and visual predictions. This unification shows value in practical evaluations. In reasoning tasks (like answering questions about images or planning actions from language), Cosmos 3 matches or exceeds specialized vision-language or vision-language-action systems. In generation tasks (like synthesizing high-resolution images or coherent videos), it sets new benchmarks among open models. The team reports that external watchdogs ranked Cosmos 3’s generative outputs at the top of open-source models: e.g. their text-to-image outputs were judged the best among public models at the time. Video generation is similarly impressive: in head-to-head tests the large Cosmos3-Super model substantially outperforms preceding open models like Video-GPT or MAGI-Vid, producing sharper and more consistent two-second videos given a single input image frame. On the action side, Cosmos 3 can even act as a robot controller. The authors fine-tuned a variant called Cosmos3-Nano-Policy-DROID for the DROID manipulation benchmark, and it ranked #1 on the RoboArena policy leaderboard. In other words, this same footprint that generates images and text can also map visual observations to motor commands. Of course it was tested alongside many other robot policies, but Cosmos 3 came out on top in the open-source track. This underscores the central claim: one foundation model now handles perception, imagination, and control across tasks. Its unified training also means it can adapt across domains: for example, an ablation study showed that pre-training with actions helped the model transfer more quickly to new robot hardware or environments. In short, players who work on vision, language, video, or control all get a boost from the shared knowledge in Cosmos 3. Quantitatively, on standard benchmarks Cosmos 3 is often state-of-the-art (for open models) or very competitive with closed models. In multimodal reasoning tests it matches leading vision-language LLMs; in image/video generation metrics it beats previous best open systems; and in a newly curated Physical AI benchmark it achieves the highest overall score, leading in semantic alignment and visual integrity of generated scenes. Notably, in direct comparisons Cosmos 3-Super outperforms even some cutting-edge closed-source video generators (like Veo-3.1 and Seedance-1.5) in key aspects of realism. The consistent message from these evaluations is that the integrated mix of pre-training data and double-headed architecture pays off: Cosmos 3 is not only multi-talented, but also high-performing in each talent. Using Cosmos 3. For practitioners curious how to leverage Cosmos 3, NVIDIA has provided code, models, and examples in a friendly ecosystem. The models are released on Hugging Face under the “nvidia/cosmos3” collection, and they support the familiar Diffusers API and a custom vLLM serving pipeline. For text-to-image or image/video generation, you can use directly with or the 64B “-Super” variant. The same pipeline can do text2image, text2video, or image2video just by providing different inputs. For reasoning tasks, you use a conditional generation API where you feed in video frames + text and get back text tokens or predicted actions. There are cookbooks and notebooks in the repository showing how to prompt the model for different tasks, and how to offload parts of the model across GPUs or use quantization for efficiency. In robotics applications, you would typically run Cosmos 3 as a perceptual world model that informs planning. For example, you could embed the model in a loop where the robot observes its camera (converted to tokens) and a task instruction (text), then Cosmos 3 generates the next few actions or even imagined future states. The authors demonstrate this with a Policy Server setup: the Cosmos3-Nano-Policy-DROID runs a server that streams actions to a robot (or simulator client) given camera input. Of course, real-world deployment requires care (the model’s simulations aren’t perfect), but this shows how easily Cosmos 3 can be plugged into an embodied-control pipeline. Because Cosmos 3 supports so many modalities, there are endless use cases. It has effectively turned itself into a physical scene painter: you can describe a scenario and it will paint it out in image or video form. Or conversely, you can show it a scene and it will describe and predict it. In an autonomous driving lab, one could use it to simulate traffic scenes from a high-level scenario description. In a factory, one could imagine using it to predict how a conveyor belt scene evolves given a robot arm’s command. The ability to handle audio means it could even simulate alarm sounds or background noise along with vision. And since it retains an internal representation of actions, it could in principle interact with an environment in closed loop, though the report mainly emphasizes its generative/prediction strengths. Finally, the open release (models and synthetic datasets under the OpenMDW license) means others can fine-tune or extend it. For instance, if you have a proprietary dataset for a niche robotics task, you could adapt Cosmos 3 via additional supervised or reinforcement learning to your hardware. Or you could use it as a backbone feature extractor: attach task-specific heads for visual classification or grasping that leverage the Cosmos-trained vision embeddings. NVIDIA also provides an entire Cosmos Framework with tooling for data curation, training, and evaluation, which users can employ to train on their own data streams or benchmark agents with Cosmos models. Discussion and Limitations. Cosmos 3 is an impressive step toward closing the gap between general AI and embodied robotics, but it’s not a magic bullet. The authors are clear that the model still has failure modes typical of generative systems. In long or complex scenes you often see glitches: temporal flicker, slight camera jitters, or objects that morph unexpectedly. The model’s notion of physics is still approximate, so it can predict motion that looks plausible to the eye but would violate real dynamics if you tried it on a robot. Sound-video alignment is not perfect either – sometimes the robot motions and the audio background lag or desynchronize. And crucially, Cosmos 3’s outputs are only suggestions, not guaranteed correct plans. The report cautions that for any safety-critical application, you still need extra safeguards (e.g. rule-based checks, human oversight) before letting the model control a real robot. Another practical limit is compute. The top-tier Cosmos3 models are huge, so running them at high fps requires a cluster of GPUs, specialized libraries (like FlashAttention or VLLM), and careful memory management. For research readers, this means that while you can experiment with the Nano model on a small GPU, replicating the largest Super model’s performance requires industrial-scale hardware. That said, the distilled Nano is still very capable for many tasks, and the open release encourages community work on model compression or more efficient variants. In terms of future directions, Cosmos 3 opens many avenues. Its unified multimodal framework suggests one could continue scaling or specializing: e.g. train a “Cosmos 4” with even more real-world interaction data or online RL to improve real-world consistency. One could also integrate it with physics engines: use Cosmos 3 as a fast estimator of what might happen, while consulting a simulator for precise validation. There is also room to refine specific modalities: aligning its audio with visuals better, or giving it explicit 3D awareness (right now it learns 3D implicitly). And of course, combining Cosmos 3 with the latest language models could yield an agent that can use large-scale text knowledge while grounding it in perception and action. Conclusion. Cosmos 3 is a bold leap toward truly general AI for the physical world. By training a single “omnimodal” transformer on vast amounts of text, vision, audio, and action data, NVIDIA has shown that we can build a foundation model that spans dialogue, images, video, and robotics. The results – top performance on image/video generation, visual reasoning, and even robot control benchmarks – demonstrate that a unified representation is not only feasible but powerful. For roboticists, this means we now have an off-the-shelf world model that we can query with language or vision and get back a prediction, a simulation, or even an action in the same framework. Of course, Cosmos 3 is a starting point, not the end. It still makes mistakes, and it doesn’t truly “understand” physics the way a human or a physics engine does. But it significantly raises the bar: it shows that all those pieces – vision, language, dynamics, control – can live in one model. Looking forward, we may see Cosmos3 as the brain of many future physical AI systems, just as GPT and DALL·E have become backbones for language and image apps. For now, professionals in embodied AI can explore its open models and datasets, adapt them to their own robots, and build on this new foundation. The Cosmos has truly expanded – now it feels tangible in the lab as well as in the cloud. References: The above summary is based on the NVIDIA Cosmos 3 technical report and associated materials, as well as the NVIDIA Cosmos Lab website, which detail the model’s architecture, data, and evaluation results. The models, code, and data have been publicly released under an open license for the community.