Latest / Embodied AI 101 / NVIDIA Cosmos: World Foundation Models for Physical AI

Transcript

NVIDIA Cosmos: World Foundation Models for Physical AI · Embodied AI 101

• NVIDIA Cosmos: World Foundation Models for Physical AI. In early 2025, NVIDIA unveiled Cosmos – an ambitious new platform extending the “foundation model” concept to the physical world of robotics and autonomous vehicles. Like how GPT and DALL·E demonstrated the power of giant pretrained networks in language and vision, world foundation models (WFMs) promise to revolutionize how robots learn about and interact with their environment. As NVIDIA CEO Jensen Huang put it at CES 2025, “the ChatGPT moment for robotics is coming,” and Cosmos is intended to bring that moment by providing pretrained generative models that simulate physics-rich video worlds. In practice, Cosmos offers open, customizable networks that can take in sensor data (or even text descriptions) and predict future video frames or generate entirely new synthetic videos – all rendered with realistic lighting and motion. These models are trained on truly massive video corpora (on the order of 20 million hours of driving and robot footage) so they internalize commonsense physics and visual context. The upshot is an unprecedented ability to cheaply generate data, simulate scenarios, and give AI agents foresight in tasks from warehouse picking to self-driving. In this article we explore what Cosmos actually is, how it works, and why it matters – including its novel trust and safety features (notably invisible SynthID watermarks) developed in partnership with Google DeepMind. The Need for “Physical AI” Simulation. Modern robotics and autonomous vehicles face a staggering training challenge. Unlike chatbots that learn from the wealth of internet text, robots must grapple with a three-dimensional, dynamic physical world. Training a robot with real-world trials is slow, expensive, and sometimes unsafe. Collecting enough diverse video footage of a car in snow, or a robot manipulating unfamiliar objects, is prohibitively hard. NVIDIA’s vision is to apply the same underlying idea behind large language models – pretrain a huge network once on wide-ranging data – to the “world” itself. The notion (articulated in the Cosmos paper) is that physical AI needs two digital twins: one of the agent (the robot’s policy model) and one of the world (a world model). This world model is a neural simulator: it learns the dynamics and appearance of the environment from real videos, so that it can imagine new scenarios for training. Put simply, a world foundation model is a general-purpose video generator. It takes inputs (which could be a text prompt, some initial video frames, or even raw 3D sensor data) and produces a sequence of frames depicting a plausible future. Importantly, these models are physics-aware: they aim to respect gravity, object permanence, and other real-world laws in the videos they generate. As NVIDIA’s blog explains, the Cosmos models “use input data, including text, image, video, and movement, to generate and simulate virtual worlds in a way that accurately models the spatial relationships of objects in the scene and their physical interactions”. In other words, you could feed the model a description like “a yellow car braking on a rainy highway with mountains in the background,” and it will output a realistic video of exactly that unfolding over time. Or you could give it the last few camera frames from a warehouse robot plus a command, and it will predict what the next seconds of video should look like, allowing the robot to plan ahead. NVIDIA positions these WFMs as the robotics/AV analogue to LLMs: “as fundamental as large language models” to their domains. They are pretrained once on wild internet-scale video data, then fine-tuned or “post-trained” for specific tasks. Crucially, NVIDIA is making the first wave of Cosmos models open and free under a permissive license, so any developer can experiment with them. The platform covers not just the models themselves but the whole pipeline: video curation, tokenizer tools, fine-tuning frameworks, and safety guardrails. As the Cosmos authors summarize, “to help Physical AI builders… we make our platform open-source and our models open-weight with permissive licenses”. The Cosmos Platform and Model Family. NVIDIA’s Cosmos platform consists of several key pieces: the world foundation models themselves (for video generation), data processing tools, and safety/monitoring systems. The models come in three complementary forms, each tailored to a type of task (predicting future video, transforming simulated video, or reasoning about video). Here is a brief overview of the core models:. Cosmos Predict: A multimodal video generator. Given some prompts – which might be text descriptions, static images, or short video clips – this model produces up to tens of seconds of high-fidelity video that continues or elaborates on the prompt. For example, you might give it a street scene image and it will create several seconds of traffic motion; or a text prompt describing a kitchen and it will show a robot moving in that kitchen. NVIDIA notes that “Cosmos Predict generates diverse video scenes from text, image, or video prompts – ideal for post-training on subjects like robots or self-driving cars”. In practice, this model gives an AI agent “foresight”: it can peek into possible futures. Cosmos Transfer: A video style-transfer/simulation engine. Instead of freeform generation, Transfer takes structured inputs (often from 3D simulators) and converts them into photorealistic video. For example, if a robotics engineer has a CARLA or NVIDIA Isaac Sim driver, its sensor streams (depth maps, object positions, etc.) can be fed through Cosmos Transfer to render cine-quality footage of that scene under different lighting or weather. In other words, Cosmos Transfer “applies multi-control style transfer — changing lighting and environments — on physics-based videos, often created in simulators like NVIDIA Omniverse”. This lets developers leverage existing simulators while dramatically increasing realism. You can take a digitally modeled factory floor and produce diverse, lifelike videos for training perception systems. Cosmos Reason: A vision-language reasoning model built on top of video inputs. Rather than generating video, this model is designed to understand and plan over video. It takes video frames (and optionally text queries) and applies “chain-of-thought” physics reasoning. For example, it can answer questions about what’s happening or what will happen next in a scene (one Nvidia doc calls it a planning model that “reasons what steps an embodied agent might take next”). It’s been trained on “physical common sense” data (with supervised fine-tuning and even reinforcement learning) so that it can handle unusual edge cases. As NVIDIA puts it: “Cosmos Reason enables robots and vision AI agents to reason like humans, using prior knowledge, physics understanding, and common sense to understand and act in the real world”. The model can even generate new prompts or annotate synthetic videos to improve the other models’ outputs. All three models are foundational: they are pretrained on broad data and released openly to be fine-tuned. NVIDIA actually published a technical paper (Jan 2025) describing Cosmos as a whole platform. In that paper they explicitly advocate the model-of-model approach: a general WFM can be specialized to custom use cases (e.g. “text-to-warehouse” or “video-to-factory” scenarios) by further tuning, much like how ML practice fine-tunes GPT for tasks. The paper highlights that Cosmos WFMs were trained on an astounding scale – roughly 9,000 trillion video tokens (essentially frames or video “words”) extracted from 20 million hours of footage. This covers real-world driving, robotics labs, industrial scenes, “human interactions,” etc., so the models see an enormous variety of environments and object behaviors. Even within each model type, Nvidia trained multiple sizes to balance speed vs quality. For example, the blog notes three model classes: Nano (small and fast, suited for real-time inference/edge devices), Super (moderate size for general use), and Ultra (largest models pushing maximum fidelity). In practice, a user might fine-tune a Super model quickly, or distill the Ultra model into a smaller one for deployment. NVIDIA also provided auxiliary networks: e.g., a 12-billion-parameter text-visual “upsampler” to refine prompts, and a 7B video decoder optimized for AR/VR streams. (The exact scale of parameters isn’t as important as knowing these models are huge neural networks – multi-billion-node transformers and diffusion nets – requiring GPU power to run.). An important part of Cosmos is how it interfaces with NVIDIA’s simulation tools. In particular, NVIDIA often mentions Omniverse, their real-time 3D sim platform. Unity or Gazebo users might take note: Cosmos can consume Omniverse output. In fact, NVIDIA’s docs emphasize that Omniverse provides the “simulation environment” while Cosmos provides the “foundation models to generate video data and train physical AI models”. In one workflow, a developer could put together a scene in Omniverse (say a city intersection), export it as raw simulation frames, feed it into Cosmos Transfer, and get back a clean, photorealistic video of car traffic. On the flip side, Cosmos Predict or Reason can take a fixed video from Omniverse, analyze it, and even propose new scenario variations – blurring the lines between formal simulators and learned generative models. Data Infrastructure: Tokenizers, Pipelines, and More. Under the hood, Cosmos relies on a massive data and compute infrastructure. Training on 20M hours of video is nontrivial: one minute of video at 30 fps has 1,800 frames, so 20M hours is on the order of 1.3 billion minutes or ~2 trillion frames. Each of those frames might be tokenized into 100 or more “visual tokens” for a transformer. NVIDIA tackled this with a highly optimized GPU pipeline and new tokenizers. One key tool is the NVIDIA NeMo Curator pipeline, accelerated on data-center GPUs like H100 (Hopper) or the upcoming Blackwell. According to NVIDIA’s blog, their optimized pipeline can ingest 20 million hours of video in parallel and label/curate it in about 2–6 weeks. The blog quantifies this: processing that same data with an unoptimized CPU cluster would take over three years, whereas on a GPU fleet it took as little as 14 days. This huge speedup means Cosmos could continuously update with fresh real-world footage. Developers using Cosmos can also leverage NeMo Curator on DGX Cloud to filter their own data (e.g. find all videos with snow or pedestrians) before training or fine-tuning models. Perhaps even more critical are the video tokenizers. Standard practice for video generation is to break each frame into tokens (patches or embeddings) so a Transformer can process it. NVIDIA reports that the Cosmos tokenizer architecture achieves 8× better compression and runs 12× faster than previous state-of-the-art solutions. In concrete terms, this means a given video can be tokenized with far fewer tokens (saving memory and compute) and with lower latency. This advancement was crucial: training multi-billion–parameter video models otherwise would be prohibitively slow. These tokenizers (and related vision encoders) are also released open-source under the same license, so anyone can try them out via Hugging Face or GitHub. For model training and deployment, NVIDIA leans on its own stack. The entire Cosmos training pipeline used NVIDIA DGX Cloud (managed clusters of GPUs) to crank through video data and train models. The blog notes that “data curation and the training of Cosmos models relied on thousands of NVIDIA GPUs through NVIDIA DGX Cloud”. This means the models were embarrassingly parallelized across a large GPU farm. For inference and fine-tuning, developers can similarly grab GPU instances on DGX Cloud or on any GPU cluster. NVIDIA explicitly lists pretrained Cosmos models on both the NVIDIA NGC catalog and Hugging Face (under the new NVIDIA Open Model License). They plan to also package them as fully optimized NVIDIA NIM microservices for easy deployment. In short, if your organization uses NVIDIA hardware or cloud, Cosmos slots neatly into the existing AI infrastructure (NeMo for training/fine-tuning and DGX for compute). Generating Synthetic Data for Robots and Cars. The main practical benefit of Cosmos is in synthetic data generation and scenario simulation. With these models, developers can automatically create endless variants of training footage. Consider two examples:. Robotics Training: A domestic or industrial robot typically needs thousands of hours of footage to learn tasks. Gathering real video of every possible kitchen layout or object arrangement is unrealistic. Using Cosmos Transfer, a developer could take a few 3D scenes of robot workspaces, then generate hundreds of photoreal videos under different conditions (different lighting, backgrounds, camera angles). These videos can then be labeled automatically (e.g. where objects are) and used to pretrain perception networks. NVIDIA points out that roboticists can generate synthetic grasping or navigation scenarios with Cosmos Transfer, fine-tune their policies with Cosmos Predict in a “world model”, and even post-process data with Cosmos Reason. As Agility Robotics’ CTO notes, “Data scarcity and variability are key challenges in robot environments… Cosmos’ text-, image- and video-to-world capabilities allow us to generate and augment photorealistic scenarios for a variety of tasks… without needing as much expensive real-world data capture.”. In other words, Cosmos can mock up entire worlds a robot might see so it doesn’t have to learn only from the limited real footage available. Autonomous Vehicle Simulation: For self-driving cars, edge-case scenarios (snowstorms, flash floods, unusual road layouts) are especially critical and hard to find in real datasets. Cosmos helps here too. Engineers can take a base driving dataset (from real test drives or simulators) and run it through Cosmos Transfer to alter weather, lighting, or add/deduct vehicles and pedestrians. The press release mentions that vehicle companies like Waabi and Wayve are evaluating Cosmos to enrich their training sets. For example, Waabi is considering using Cosmos to curate massive driving datasets (selecting interesting scenes), while Foretellix (a scenario testing firm) will use Cosmos plus Omniverse sensors to “evaluate and generate high-fidelity testing scenarios” at scale. Uber, which has vast driving logs, is also partnering with NVIDIA to “accelerate autonomous mobility” by combining its data with Cosmos models. In practice, this could mean automatically generating videos of a car approaching a school bus, turning corners on slippery ice, or encountering suddenly merging traffic – all in virtual scenes, allowing safe testing and retraining of the AV stack. Video Analytics and AI Agents: Beyond robotics and AVs, any AI that needs to understand video could benefit. For example, a security camera system might train an agent to recognize accidents or intrusion. Cosmos Reason can help here by being able to “answer queries” about a video feed in real time (e.g. “Is that person moving toward a forbidden zone?”). The models could also automatically summarize and label hours of factory floor video. NVIDIA’s materials highlight that industrial automation companies (like those with hundreds of surveillance cameras) can use Cosmos to ingest video and have the AI generate alerts or contextual insights. Cosmos could also be used for AR/VR: e.g. expanding a short 360° video clip into a longer animated scene for previews or content. In all these use-cases, the key is control and variety. Cosmos lets developers “condition” the generation on specific inputs. If your robot labs have a 3D model of a shelf (from simulation software like Isaac Sim), you can feed that model into Cosmos Transfer and get back a realistic video of a robot approaching that shelf under different lighting. If you have a clean dashboard camera shot of a highway, you can use Cosmos Predict to generate multiple future clips – maybe one scenario where a pedestrian steps out, another where a car brakes. By training on these several plausible futures, an AV stack can become less brittle. Cosmos even supports multi-camera generation: you could generate matching front, side, and lidar-view videos for a car, creating cohesive multi-sensor data. NVIDIA and early adopters are already building stories around this. For instance, 1X Robotics (about general-purpose humanoids) launched an open-world-drive dataset using the Cosmos tokenizer. XPENG (the Chinese EV maker) plans to use Cosmos to speed up its humanoid robot development. Skild AI (humanoid controllers) and Hillbot (industrial first responder bots) reportedly use Cosmos to boost their training pipelines. The CEO of Uber, Dara Khosrowshahi, even said that combining Uber’s rich driving data with Cosmos and DGX Cloud could “supercharge the timeline” for safe autonomous cars. These are confident real-world validations that a lot of companies see value in generative world models. Safety, Trust and Watermarking. A big concern with any generative AI, especially in critical domains, is trustworthiness. NVIDIA built several safeguards into Cosmos to ensure responsible usage. First, by policy the platform aligns with NVIDIA’s “trustworthy AI” principles: privacy, non-discrimination, safety, etc. Practically, they provide guardrail models – small networks that filter and sanitize user inputs and outputs. For example, if a user tries to prompt a racist or dangerous scene, the guardrails can block or rewrite the prompt. If a generated video appears to depict disallowed content, it can be flagged. These guardrails can also be fine-tuned or extended by developers for their niche application. In short, NVIDIA is pushing the notion that generative physical AI should automatically screen out obviously bad uses. The second major trust feature is watermarking of outputs. On the AI-for-images side, Google DeepMind has developed a system called SynthID, which embeds an invisible digital watermark into any image or video produced by AI. NVIDIA announced that it is integrating Syn​thID into Cosmos, making it the first external partner to use this technology. Technically, this means every frame output by Cosmos’s generation engines carries a hidden signature. These watermarks are designed to be imperceptible to people, yet extremely robust to editing (even cropping, color filters, recompression, etc.). As DeepMind describes, “SynthID adds an invisible digital watermark to an AI-generated image (or video segment). The watermark doesn’t change the image or video quality. It’s added the moment content is created, and [it’s] designed to stand up to modifications like cropping, adding filters… or lossy compression.”. Why does this matter? In the era of deepfakes and misinformation, being able to trace an AI-generated clip is crucial. By using SynthID, NVIDIA ensures that any video coming from Cosmos can be algorithmically identified as synthetic. If someone tries to misuse a Cosmos video – say, by editing it or reposting it as real – a SynthID detector (powered by Google’s Gemini/Ai tools) will flag it. This fosters transparency: images and videos that come from Cosmos are essentially self-annotated as “AI-made.” Nvidia’s press release explicitly notes that “videos generated with Cosmos autoregressive and diffusion models include invisible watermarks to identify AI-generated content, helping reduce misinformation and misattribution”. This watermarking is part of an industry trend (for example, OpenAI’s image services also use content credentials and SynthID) to make synthetic media provenance obvious. In addition, Cosmos models log metadata about their generation (model version, prompt, etc.), which can be retained as content credentials by data consumers. And because NVIDIA is open-sourcing everything, users can inspect the filters and even improve them. In summary, NVIDIA’s responsible-AI measures around Cosmos include standard guardrails (filtering abuses) and this cutting-edge watermarking (SynthID) to maintain trust in the ecosystem. Collaboration with Google DeepMind. The Cosmos announcement was not just a solo NVIDIA effort. In parallel, NVIDIA revealed deep collaborations with Google and others on physical AI. The most direct link is SynthID itself – Google DeepMind’s watermark tech is now part of NVIDIA’s pipeline. But the partnership goes further. At the same GTC event, NVIDIA and Google DeepMind (along with Disney Research and more) announced Newton – a new open-source, GPU-accelerated physics engine for robotics. Newton is built on NVIDIA’s Warp framework and targets exactly the kind of simulations that Cosmos ultimately learns from. It is actively co-developed by NVIDIA and DeepMind, and is managed by the Linux Foundation as an open project. (DeepMind’s involvement means that behind the scenes, the same teams working on SynthID and generative models are helping to build a next-generation simulator.). Specifically for Cosmos, the partnership’s headline is that NVIDIA is the first “external user” of SynthID. According to company statements, NVIDIA will embed SynthID’s watermarks directly into Cosmos outputs to “protect intellectual property” and maintain authenticity. In short, it’s a strong signal that NVIDIA trusts Google’s solution for watermarking AI media. Beyond that, NVIDIA noted that Google Cloud will be an early adopter of Nvidia’s new Blackwell hardware, and that NVIDIA has helped optimize Google’s open models (like the Gemma family) for its GPUs. While not directly about Cosmos, these moves highlight how NVIDIA is aligning its physical AI strategy with Google’s broader AI ecosystem. In sum, Cosmos did not emerge in isolation. It sits at the intersection of two massive trends: the new wave of multimodal AI from companies like Google, and the push for optimized simulation at NVIDIA. By partnering on SynthID, Newton, and model optimization, NVIDIA is ensuring that Cosmos will have many tools to draw on – from high-fidelity physics to AI safety technologies. Industry Adoption and Outlook. NVIDIA’s roll-out of Cosmos has already seen significant interest. The press release lists a who’s-who of early adopters: startups like 1X Robotics and Agility, industrial-robot firms like Skild AI and Hillbot, AV companies like XPENG, Waabi, Wayve, and automation giants like Uber. These organizations are integrating Cosmos into their pipelines in various ways – for example, 1X used the Cosmos tokenizer to launch a large humanoid-driving dataset, and Foretellix is using Cosmos-generated scenarios to stress-test its AV validation tools. Even non-technical stakeholders are excited: the Best of CES judges gave Cosmos awards for “Best AI” and “Best Overall,” citing the platform’s potential to democratize robotics development. Looking ahead, what does Cosmos represent for the field? On the positive side, it brings the power of generative AI to physical domains. Robotics teams that lacked data science scale can now benefit from billion-parameter simulators. Small companies can fine-tune Cosmos instead of training their own world models from scratch. The openness of the platform (code, models, data are largely public) means the entire community can iterate – potentially leading to standardized benchmarks, common synthetic datasets, and shared tools (for instance, NVIDIA already published model cards on Hugging Face). It may accelerate research on model-based RL or world-model planning, since the “simulator” is now a neural net. Of course, there are cautionary points. A learned model is not a physics engine: it can hallucinate or violate conservation laws beyond its training. The fidelity of Cosmos videos will depend on how representative the training data was – if you show it a scenario wildly outside what it saw (e.g. zero-gravity), it may fail. Developers will need to validate how realistic these synthetic scenes are. There is also the compute cost: running a 10B-parameter video model in real time is still challenging, so NVIDIA’s categorization of “Nano” models suggests they see on-device or edge use as a goal, but it may require significant engineering. Finally, as with all "foundation models," there is the risk of overtrust. Even with SynthID, someone could try to strip or alter content; governance will need to catch up. Nevertheless, NVIDIA Cosmos is a landmark in the journey toward embodied AI. It packages state-of-the-art video generation, enormous datasets, and sophisticated pipelines into a developer-friendly platform. For roboticists and ML researchers, it offers new experimental building blocks: instead of hand-designing every simulator, you can now sample worlds from a learned prior. The fact that industry players and open-source communities are rapidly embracing it suggests we will see many innovations on top of Cosmos. In practical terms, if you’re working on robot perception or policy, you can download these models right now (they’re on NVIDIA’s NGC or Hugging Face). Try conditioning Cosmos Predict on your own test scenes, or pipe your synthetic sensor outputs through Cosmos Transfer to make endless training videos. Join the Discord or browse the Cosmos GitHub cookbook for recipes. As one NVIDIA exec put it, with Cosmos they are “democratizing physical AI, putting general robotics in reach of every developer”. Whether that promise fully pans out remains to be seen, but it is clear that the tools for building smart agents in the real world are growing more powerful every day – and synthetic video is now very much part of that future. Sources: Official NVIDIA press materials and blog posts on Cosmos; the Cosmos research paper abstract; NVIDIA Cosmos documentation and GitHub pages; Google DeepMind SynthID documentation. These in turn cite various NVIDIA keynote and CES announcements.