Latest / Embodied AI 101 / Xiaomi-Robotics-U0: A 38B World-Foundation Model for Unified Embodied Perception and Synthesis

Transcript

Xiaomi-Robotics-U0: A 38B World-Foundation Model for Unified Embodied Perception and Synthesis · Embodied AI 101

• Xiaomi-Robotics-U0: A 38B World-Foundation Model for Unified Embodied Perception and Synthesis. In July 2026 Xiaomi’s robotics lab released Xiaomi-Robotics-U0, a 38-billion-parameter autoregressive transformer that unifies standard vision generation with embodied scene and video synthesis. Simply put, U0 is a giant, generalist “world model” for robots: it can generate images from text, edit images, imagine multi-view robot-centric scenes, transfer trajectories to new environments, and even roll out entire manipulation videos — all within one shared framework. Crucially, it achieves this without throwing away the broad visual knowledge learned from Internet-scale data. According to Xiaomi, U0 “treats embodied synthesis as a direct extension of world foundation modeling”, co-training on both general vision-language data and robot data in one next-token objective. This approach preserves wide-ranging generative capabilities while adding the geometric, kinematic, and temporal constraints that robot tasks demand. In practice, U0 achieves state-of-the-art results across benchmarks. It even beats specialized baselines at embodied tasks: for example, in human evaluations of multi-view scene generation and transfer it outperforms GPT-Image-2.0 (a leading multi-modal baseline), and it won first place on the WorldArena embodied video task. When used as a synthetic data engine, its impact on real robots is dramatic: policies trained with U0-generated data saw out-of-distribution manipulation success jump from 36.9% to 63.2% on hard tasks. The authors argue that these results show “foundation world models can serve both as embodied world models and [as] scalable data engines for embodied intelligence”. Below we walk through U0’s design, capabilities, and significance in detail, highlighting how it bridges the gap between general generative AI and robotic “world modeling.”. The Problem: Bridging Foundation Vision and Embodied Constraints. Modern generative models (diffusion or transformer-based) can create stunning images and videos from text. But deploying these models “as-is” in robotics runs into problems. The usual image/video generators are trained on natural, Internet images; they lack strict enforcement of multi-view and physical consistency. For a robot, a generated scene needs precise geometry: if the model generates a car from the front and another from the side, both views must align in 3D; if a robot arm is depicted interacting with an object, the contact points and motion must be physically plausible. Violating these constraints confuses any downstream control policy. In other words, solid robotics perception demands multi-view consistency, geometric coherence, and embodiment constraints that standard T2I models don’t enforce by default. Many prior approaches tried to adapt vision models to robot data by fine-tuning them on a few thousand robot images or trajectories. But there’s a catch: as one commentator pithily notes, “you gain embodiment, you lose the Internet” – when you fine-tune on robot data alone, the model forgets much of its broad-world knowledge and diversity. The generated scenes become more robot-like but often banal or unrealistic in texture, lighting, and background. Worse, collecting diverse robot demonstrations in every environment is expensive. Xiaomi-Robotics-U0 turns this trade-off upside down. Instead of sacrificing generality for embodiment, it co-trains on both large-scale Internet vision data and curated robot data under one unified objective. In effect, U0 extends a world-foundation model into the embodied domain, rather than grafting a small head onto a frozen generator. This “continued foundation training” lets U0 keep the Internet’s visual knowledge (lighting, materials, variety of objects) while learning the rules of multi-view, kinematics, and physics needed by robots. In the authors’ words, U0 is “a direct extension of world foundation modeling”. Concretely, U0 supports five tasks in one model (see Fig.1):. Text-to-Image (T2I): Generate a high-quality image from a text prompt. Anything-to-Image (X2I) Editing: Given a reference image and an instruction, produce an edited or related image. Embodied Scene Generation: Given a language description of a scenario and a robot specification, generate a set of multi-view RGB images and depth maps of that scene as a robot would see it from different camera poses. Embodied Transfer: Given an existing robot trajectory or video in one environment, translate it into a new environment (changing background, lighting, object appearances, etc.) while preserving the robot’s motion and workspace layout. Embodied Video Generation: Given an initial robot frame and an instruction (or policy “subgoal plan”), roll out a future video of the robot interacting with the environment according to the task, preserving temporal consistency. All these generation tasks—some single-shot (like T2I or one-step scene gen), some sequential (like video rollout)—are handled by the same autoregressive decoder, without bespoke transformers per task. In practice, U0 treats the entire mixed sequence of text tokens, image tokens, depth tokens, and even robot joint commands as one long token stream. Generating a scene or video then just becomes predicting the next token in this multimodal sequence, autoregressively. Model Architecture: A 38B Autoregressive Transformer. At its core, U0 is a 38-billion-parameter transformer decoder initialized from a large vision-language model. Specifically, Xiaomi started from EMU3.5, an open 38B architecture that itself is built on Qwen-3-32B (an existing large language/image model) plus a image tokenizer. In other words, U0’s backbone is a Qwen-3-32B transformer decoder, but with a shared discrete codebook for images (called ) and an expanded vocabulary to include vision and robot tokens. All inputs – text, images (encoded as IBQ tokens), robot state/action vectors, depth maps, etc. – are embedded into this unified token space. Generating any modality is just a matter of continuing the token sequence. This design choice – collapsing all modalities into one token stream – is deliberate. It means there are no separate prediction heads or specialized networks per task. Unlike some prior “embodied” models that add a new projection or adapter for each scenario, U0 uses the same transformer layers for every task. For example, generating a video frame is not done by a separate video head; it’s just more token-prediction by the decoder. This uniformity forces the model to share parameters across vision, language, and dynamics tasks, which Xiaomi argues helps the model generalize and also simplifies training. In the words of the authors, U0 “eschews task-specific prediction heads” in favor of a single next-token objective for images, scenes, edits, or videos. Essentially, text caption “” and the first few image tokens might both be in the input buffer, and the model just repeatedly predicts what token should come next (which might be an image patch token, a depth value token, or a joint-angle token, depending on context). This unified autoregressive formulation makes it easy to mix different training samples in batches, and supports tasks like chained forecasting (e.g. video rollouts). FlashAR and vLLM acceleration. A 38B transformer can be slow to generate high-resolution images in a strictly token-by-token loop. To speed this up, the Xiaomi team introduced an anti-diagonal decoding scheme called FlashAR. Roughly speaking, FlashAR breaks the image into blocks or “patch-grids” and predicts many pixels in parallel along anti-diagonals of the image grid, rather than strictly left-to-right pixel order. This lets parts of the image be generated simultaneously, enabling massive parallelism. In practice, they report something like an 83× speedup (on 1024×1024 images) versus naïve decoding. Furthermore, U0 integrates with the vLLM library to batch multiple queries and cache attention, squeezing out even more throughput in deployment. In Xiaomi’s measurements, one H20 GPU with FlashAR and vLLM can generate a 1024px image in ~5.44 seconds, about 83× faster than the same model in vanilla sequential mode. These engineering innovations make U0 practical enough for research and applied use. Despite its 38B size (which makes it a datacenter-scale model, comparable to systems like NVIDIA’s cosmic-scale world models), the FlashAR+vLLM optimizations convert it from a research curiosity into a usable “embodied generative engine.”. Training Data and Annotation: Building a Multitask World. Training U0 required assembling a huge, diverse dataset spanning both general visual content and robot-centric scenes. Xiaomi reports roughly 9.5 million single-image examples (56.4B tokens) and 2.6 million multi-frame video clips (49.6B tokens) in the corpus. Key sources include:. General vision-language data: Internet image-text pairs (like ShareGPT4V, open image-caption corpora, etc.) to retain wide visual knowledge. Embodied manipulation data: Real and simulated robot trajectories and videos from domains like AgiBotWorld, MiBot (Xiaomi’s in-house), RoboTwin 2.0, GenieSim, Open X-Embodiment, etc. These contain egocentric frames with associated taction labels capturing robot tasks. Autonomous driving: Datasets like Cosmos-Drive-Dreams for scenes with cars and traffic. Egocentric videos: Mobile first-person videos (like EgoWalk) to improve handling of wearable or egocentric viewpoints, which resemble robot cams. 3D scene reconstructions: Multi-view datasets (ScanNet++, DL3DV) providing RGB+D views of real environments, to instill geometric consistency. Synthetic environments/games: Simulated worlds (Open-P2P, others) to further diversify backgrounds and objects. Together, these domains ensure U0 sees a huge variety of scenes, while still including a healthy dose of robot-centric content. To augment and normalize this mix, the team built an automated annotation pipeline. A separate giant model (Qwen3-VL-235B) was used to label each scene with semantic descriptions, segmentations, depth maps, and to decompose a robot’s task into sub-goals. Depth-Anything was run on images to get proxy depth and edges. The result is that every training sample carries rich structure: what the objects are, where the camera is, how the robot is posed, and so on. These auxiliary annotations act as implicit supervision for geometry and controllability (for example, giving the model signal that a “table” object should appear flat across views). Training was then performed end-to-end over this mixed data. Xiaomi balanced samples carefully, oversampling long-tail or rare categories (like rare object types or unusual lighting) and trimming noisy robot sequences, so that no single domain dominated. The overall effect is something like multi-task co-training: U0 learns to predict images from captions or depth, predict depth from images, predict the next frame of a robot lift from the first frame, and so on, all with the same transformer. Critically, because the general visual tasks are in the mix, the model never forgets how to create natural-looking textures and lighting – it’s literally seeing those examples each batch, alongside the robot examples. Unified Generation Tasks: What U0 Can Do. Because U0 handles everything with one model, the user interface is remarkably simple. Xiaomi provides a single autoregressive inference script with a flag to select among the five modes. Internally, the model only needs different prompt templates and some input tokens (like text or reference images) for each task. But the underlying mechanism is always “predict next token”. To illustrate U0’s capabilities:. Text-to-Image (T2I): As with any text-conditioned image transformer, U0 can generate high-res RGB images from descriptions (e.g. "a yellow robotic arm assembling a circuit board”). On general prompts it behaves on par with other 38B image models, thanks to retaining broad knowledge. On robot-specific prompts, it injects the right geometry (the robot parts, tables, objects) in consistent 3D. Image Editing (X2I / Anything-to-Image): Given any reference image plus a text instruction (e.g. “Change the lamp’s color to blue” or “Remove the wrench from the table”), U0 will produce an edited image. Since it was co-trained on internet image-edit pairs, its edits look natural; since it was also trained on robot re-dynamics, it will faithfully adjust lighting/materials without breaking spatial layout. Xiaomi calls this “Anything-to-Image” editing: it can start from a real robot scene image and perform controlled changes. Embodied Scene Generation: This is one of U0’s signature features. The user provides a textual scene description plus a specification of which robot embodiment to use. For example: “Using a UR5 robot, generate a workspace where the robot sees a red block on a white table in a kitchen.” U0 will then generate multi-view RGB frames (and depth maps) of that entire scene from different camera angles around the robot. The key is multi-view consistency: U0 jointly generates all views in a physically plausible way, so the red block has the same size and pose from each angle. The model supports many camera angles and ensures consistent lighting and shadows. Xiaomi reports that U0 is the first model capable of high-quality multi-view embodied scene generation. In qualitative examples, U0 can imagine novel robot scenes (tables, floors, backgrounds) from scratch and produce realistic looks. Embodied Transfer: Perhaps most immediately useful: U0 can take a real robot trajectory and “transfer” it to a new visual context. You provide a recorded sequence (scene + robot motion) from one environment, plus a new scene description (e.g. “same task but now with a blue tablecloth and sunlight through a window”). U0 re-renders each frame of the trajectory in the new environment, keeping the robot’s pose and the task layout exactly the same. For instance, if a robot was stacking blocks on a brown desk, U0 can produce a new video of the same stack motion but on a glossy white table with different assembly parts, without redoing the robot commands. In technical terms, it conditions on the scene description and old observation and outputs new images and depth for each step, preserving geometry. This “structured editing” across entire sequences is a novel capability; Xiaomi highlights that U0’s embodied transfer is controllable and fine-grained (you can specify exactly what to change, like lighting, background, object textures) while preserving multi-view consistency and interaction dynamics. In head-to-head comparisons on a transfer benchmark, U0 produced much more geometrically precise transfers than baseline generative models like GPT-Image-2.0. Embodied Video Generation: Finally, U0 can simulate future frames of a robot task. You give an initial robot scene image (or a first-person robot camera frame), plus a text instruction (or a high-level plan). U0 then rolls forward a sequence of frames, effectively imagining the robot carrying out the instruction. This is done autoregressively: the model takes the last predicted frame as input and generates the next one, and so on, for multiple time steps. The result is a video of the robot performing the task with physically consistent motions. Xiaomi demonstrates “zero-shot” video generation, meaning U0 can produce a plausible rollout even in a novel scene it has never seen before. In benchmarks of such video FSM (future-state modeling) tasks, U0’s videos remained smooth and coherent across time, scene changes, and viewpoint shifts. Notably, on the new WorldArena embodied video challenge, U0 topped the leaderboard. In summary, U0 exposes all these tasks through a single interface. For example, generating a scene has the same code path as generating a T2I image. Under the hood, it’s the same model running with different prompts. This unified approach means one codebase and one model can replace multiple tools, simplifying robotics pipelines. Results: Benchmarks and Real-World Tests. Xiaomi and independent evaluators have tested U0 extensively. On standard generative image metrics (FID, CLIP-score, etc.) for general scenes, it performs competitively with other top generative models of its size. But the real gains show up when looking at embodied metrics:. Scene Generation Human Eval: In a crowd-sourced human evaluation, U0’s multi-view scene outputs were judged more consistent and plausible than those from GPT-Image-2.0 (a strong open large vision transformer) for the same robot poses and scene descriptions. Testers noted that U0’s images showed better camera alignment and object geometry. Transfer Benchmarks: On a benchmark of robot scene transfers (where one must change lighting or objects while keeping poses fixed), U0 again outperformed GPT-Image-2.0 on quantitative metrics (like depth/edge consistency and segmentation accuracy) and human preference tests. Video Quality & WorldArena: The WorldArena benchmark (a new unified test for embodied video models) is particularly telling. U0’s entry – anonymized as “UNIS” – ranked first overall in the embodied video category. WorldArena evaluates not just how “natural” the video looks, but how well it follows physical rules and given instructions. U0’s combination of multi-view understanding and smooth motion paid off. Policy Improvement (Data Engine): Perhaps the most striking evidence comes from real-world robotics tasks. Xiaomi took a fixed manipulation policy (π₀.₅) and fine-tuned it on tasks in new, out-of-distribution environments. Using only a small amount of real demonstrations, the baseline success rate was about 36.9%. Then they augmented the training set with U0-synthesized videos of the robot in those novel conditions (same tasks, different lighting/background). The policy trained on this augmented data jumped to a 63.2% success rate. In other words, synthetic worlds from U0 made the robot far more robust. In internal real-world tests, Xiaomi reports similar relative gains: e.g. a 26% average lift in task success when using U0-generated training data under unknown lighting and background changes. Taken together, these results confirm U0’s practical value. Outperforming baselines on multitask human evaluations, topping new benchmarks, and demonstrably boosting real robot learning are strong evidence. As the authors conclude, this suggests large generative “world models” like U0 can indeed act as scalable data engines and simulators for embodied AI. Inference and Deployment: FlashAR and Usability. A key advance in U0 is making inference efficient enough to be usable. As described, the team built a specialized FlashAR++ decoding head that allows partial parallel generation of image tokens. Practically, running U0 to generate a 1024×1024 image on a single NVIDIA H100 (H20 variant) takes roughly 5.4 seconds with their optimized stack. That’s 82× faster than a naive autoregressive loop, and about 3× faster than their previous FlashAR eager decoder. Multi-GPU tensor parallelism and batch serving can push throughput further for large deployments. The engineers even packaged U0 with vLLM patch-sets and a fastAPI-based demo – so researchers can download the model weights and run it on common hardware surprisingly easily. Even without full 1-shot deployment, the code and models are all open-source. Xiaomi released the U0 weights on Hugging Face (both the normal and FlashAR variant) along with a huggingface repo that contains training and inference scripts. There’s a step-by-step quickstart, and a live Gradio demo. For example, one can run:. to generate a multi-view VR image sequence. The fact that the FLASHAR version is even on Hugging Face shows how Xiaomi intends this to be a community resource, not just an internal model. Significance: A New “Unified World Model” for Robotics. Xiaomi-Robotics-U0 represents an important convergence of trends in AI. On one hand, foundation models – giant vision and language transformers – have revolutionized image/video generation. On the other hand, robotics researchers have long sought internal “world models” that let a robot simulate and plan about its environment. U0 merges these: it is a foundation model (multi-modal, pre-trained on internet-scale data) that explicitly incorporates embodied world knowledge. This is the first example, to our knowledge, of a single large model that “unifies foundation image generation with embodied world modeling” under one roof. Prior embodied “world models” usually meant training a small model just on robot trajectories (e.g. video prediction of a robot camera). Those models could be kinematically correct but visually bland and narrow. U0’s approach – co-training on general imagery – reverses that: it brings Internet-scale visual realism into robotics. In effect, U0 is the bridge between PhotoGPT and RoboGPT: it keeps the language+visual pretraining, and adds robot action understanding. Another key takeaway is the idea of world models as data engines. The success on policy training (pi₀.₅) shows a path forward: instead of collecting endless real data, we can synthesize as needed from a powerful generative model. The fact that U0 was “rank #1 on WorldArena” means the community is evaluating models in precisely this role: how good are they for robot tasks. U0 establishing a leading score suggests we’re moving towards vision-language-action models that can simulate fairly realistic scenarios for robots. Of course, U0 is not a magic bullet. It’s huge and requires datacenter GPUs, and it still inherits errors of any generative model (occasional visual glitch, no physics engine guarantee). It also uses imitation-derived controllers for dynamics, so it won’t magically discover new actions. But in terms of bridging big-VL-models and small-robot-models, it’s a critical step. As the Xiaomi team notes, this is a “key turning point” towards unified embodied generation. In practical terms, the open-sourcing is valuable. Roboticists can now experiment with a model that was previously as inaccessible as OpenAI’s DALL-E 3 – but tuned for robot scenes. Having a single codebase for multi-view generation could accelerate research in areas like:. Data augmentation: Load U0 to create varied training sets for your robot tasks. Simulation: Use U0 to complement simulators where photorealism or object variety is needed. Planning with VLMs: Combine U0’s world-predictions with vision-language planners to do closed-loop reasoning. Interactive loop: Embed U0 in a feedback loop (as some suggest with vLLM) to iteratively refine a robot’s understanding or plan. In the bigger picture of “embodied AI,” U0 hints that we might treat world modeling more like language modeling than like classical simulation. We might train even larger models on even more diverse embodiment (hands, drones, etc.), and let them generate the “realistic scenarios” our robots need. By ranking high on benchmarks and proving helpful for policy learning, U0 shows that this idea has real merit. References. Xiaomi’s paper (arXiv:2607.11643) and open-source repo provide full details. The above draws on their technical report and related blogs, as well as explanatory posts from Xiaomi’s site and independent analyses. Key takeaways cited include its 38B size, unified objective, FlashAR decoding, and empirical results (beating GPT-Image-2.0, first on WorldArena, 36.9→63.2% OOD lift). All code and weights are publicly released by Xiaomi (GitHub, HuggingFace, ModelScope) for reproducibility.