Latest / Embodied AI 101 / Xiaomi-Robotics-1: A Scalable Vision-Language-Action Foundation Model for Mobile Manipulation
Transcript
- Xiaomi-Robotics-1: A Scalable Vision-Language-Action Foundation Model for Mobile Manipulation. Robotics is entering the age of foundation models. Just as computer vision and language recently saw massive gains from large pretrained models, teams are now training “vision-language-action” (VLA) policies on huge robot data to produce generalist skills. In a new preprint, the Xiaomi Robotics team introduces Xiaomi-Robotics-1, a city-scale robot policy that exemplifies this trend. Xiaomi-R1 is a transformer-style policy trained on over 100,000 hours of real-world manipulation data. The result is a mobile-manipulation model that can follow diverse natural-language instructions on new tasks “out of the box”, and also serve as a strong base policy for rapid fine-tuning on specialized tasks. Concretely, the authors demonstrate two striking capabilities. First, without any task-specific training, Xiaomi-Robotics-1 can take a user’s natural-language command (like “open the drawer and pick up the mug”) and execute it on a robot in a previously unseen home environment. In the RoboCasa-365 benchmark, for example, the model achieves about 57–58% success rate averaged across 50 household tasks – far above prior state of the art. Second, once this foundation model is in hand, it can be efficiently fine-tuned to complicated, dexterous tasks using only a small amount of additional data. In experiments on simulation benchmarks like RoboDojo, the Xiaomi-R1 policy—after suitable adaptation—scored 20.07 (13.93% success), easily toppling the previous best (score 13.07, success 8.8%). In other words, R1 is both a general “out-of-box” controller and a highly data-efficient base for learning new skills. The key to Xiaomi-R1 is scale and data: it is pretrained on an unprecedented volume of robot trajectories, with an end-to-end training recipe that leverages vision-language models to annotate actions. The training proceeds in two stages: a pretraining phase on the raw large-scale data to learn generic action-generation, and a post-training (alignment) phase that adapts the model to real robot hardware and to human-style instructions. In the sections below, we unpack how this works, how it compares to earlier VLA efforts, and why it matters for the future of generalist robotic control. Massive Pretraining on Real-World Robot Data. In the pretraining stage, the Xiaomi team’s goal is to endow the policy with broad, generalizable motor skills. To do this, they amassed over 100,000 hours of real manipulation demonstrations collected from many environments (homes, offices, industrial settings, outdoors, etc.) and hardware configurations. Importantly, the data were captured using Xiaomi’s “Universal Manipulation Interface” (UMI) devices – portable kits with hand-held grippers that humans can use to demonstrate tasks. UMI has been proposed in prior work as a way to collect rich human demos for robots: by using a standardized gripper interface, one can gather high-quality trajectory data that are hardware-agnostic. In practice this means Xiaomi’s demonstration data cover a wide variety of object interactions, gripper motions, and even base movements, across many scene types. Rather than learning from raw, unstructured trajectories alone, the team automatically annotated every trajectory segment with language. They split each long demonstration into fixed-length clips, then applied a vision-language model to describe what happened in each clip – focusing on state changes of the gripper and objects. For example, a snippet of pushing or grasping might be labeled with a phrase like “gripper closed to grasp red ball” or “block moved to the left side of table.” These automatically generated captions become part of the training data. Concretely, the model is trained so that given a current camera image and a textual description of the desired state change, it must output the actions that achieve that change. In other words, training examples look like. This language-augmented demonstration learning has a couple of advantages. First, it provides a structured goal or context for each action sequence: instead of learning to blindly imitate motions, the model learns to associate visual conditions and text commands with actions that make the scene transition in the commanded way. Second, the annotation process is fully automated and scalable. Xiaomi reports being able to label the entire 100k hours of data in roughly two weeks using cloud compute, thanks to the speed of modern vision-language models. In this way, pretraining resembles how GPT-3 is trained on huge text corpora: Xiaomi-R1 is trained on huge robot logs with (automatically) generated captions as side information. The result is a policy backbone with very broad “common sense” about manipulation: it has seen how grippers interact with countless objects and learned to associate subtle language cues with appropriate motions. The net effect is that, after pretraining, Xiaomi-Robotics-1 has general action-generation capability. Given a visual observation of a scene and a textual prompt describing a target state (or change), it can predict a sequence of robot actions that drives the scene toward that target state. In practice, the “language prompt” during pretraining was the caption generated by the labeling pipeline. But because those captions use natural language, the policy learns a strong alignment between language concepts and action primitives. This forms a foundation which can later be steered by actual user instructions. Two-Stage Training: Pretrain, then Align. Naively applying the above pretrained policy to a real robot still has gaps. The Xiaomi team addresses this with a second post-training (alignment) stage. The alignment stage has two goals: (1) adapt the model to the specific embodiments of real robots, and (2) train it to follow human-style task instructions rather than auto-generated captions. Embodiment Alignment. The 100k hours pretraining data came from UMI devices, which deliver rich demos but are not the same as a particular robot’s joints or sensors. To bridge this sim-to-real or cross-embodiment gap, Xiaomi prepared additional training data on actual robots. They collected on the order of 10,000 hours of real-robot demonstrations across various platforms: over 7,200 hours from mobile-manipulator robots and dual-arm robots alone, as well as finer-tuned subsets of UMI trajectories. They also incorporated public multi-task datasets (such as BridgeData V2, Google’s RT-1 data, the DROID dataset, etc.), mixing in examples from different hardware setups. Training on this mixture effectively “transfers” the action-generation skills from the abstract UMI world onto concrete robot embodiments. In technical terms, it teaches the same high-level behavior but now represented in the action space of, say, a 7-DOF arm on wheels. Instruction Alignment. A second gap is between the kind of language used in pretraining captions (which describe state changes) and the kind humans use as commands (which are imperative task instructions like “open the drawer”). To handle this, the team further fine-tuned the model on data that explicitly pairs natural human commands with tasks. This data was drawn from a few sources: about 1,000 hours of the UMI data were re-labeled by humans to match instruction-style prompts, and standard robotics instruction datasets (Bridge V2, RT-1, DROID, etc.) were included. The fine-tuning objective was to take an instruction (e.g. “make the mug upright on the table”) together with the current vision, and produce the right actions. In effect, this teaches the model to interpret human instructions as descriptions of desired state changes. With these two alignment steps, Xiaomi-Robotics-1 graduates from “just a generic action generator” to a true robot policy. The embodiment alignment makes its outputs compatible with real hardware, and the instruction alignment makes its inputs human-friendly. After this, the model can be deployed on a robot to follow user commands immediately. Strong Scaling Behavior. One striking theme in the Xiaomi-R1 results is how smoothly the model improves with scale. The authors report that larger model sizes and more pretraining data consistently yield better performance, both during pretraining and on downstream tasks. This is analogous to the well-known scaling laws in language and vision models. For instance, they observe that more pretraining (or bigger networks) leads to higher success rates even before alignment, and that a stronger pretrained model leads to better out-of-the-box performance after post-training. This implies there were no obvious “blow-ups” or diminishing returns: gathering even more robot data and using larger networks just helps more. It also shows that the two-stage recipe preserves the benefits of pretraining through to the final fine-tuned policy. In practice, this scaling translated into big gains on benchmarks. Roughly speaking, Xiaomi-R1’s “epoch=100k hour” model outperforms smaller variants on every metric they tried. And when the final model is used with just the post-training data, it immediately beats competitors in unseen environments. Out-of-the-Box Performance. How well does Xiaomi-R1 actually work in practice without any on-task fine-tuning? The benchmarks say: very well, by current standards. On the RoboCasa-365 multi-task benchmark (50 kitchen-domain tasks with both atomic and composite actions), Xiaomi-R1 achieves roughly 57–58% average success rate. To put this in context, the previous leader (ABot-M0.6) was at 46.6%. Even more impressively, on RoboCasa’s composite-unseen split (complex tasks not seen during training), the model still works in about one third of attempts (32.1% success). This suggests strong generalization: the model really did learn a broad repertoire of skills, not just memorized a few scenarios. Another new testbed is RoboDojo, which scores policies on a unified set of manipulation tasks. On the RoboDojo leaderboard, Xiaomi-R1 is now in first place. It scored an average of 20.07 points with a 13.93% success rate across the benchmark. This comfortably beats the runner-up (score ~13.07, 8.80% success) and any other published policy. Notably, Xiaomi-R1 was able to do this without task-specific tuning — it’s literally the out-of-the-box policy from training. Besides overall scores, the team also highlighted performance in other domains. For instance, in the newly released VLABench benchmarking suite (100 tasks with 6 aspects of robot intelligence like spatial reasoning, long-horizon planning, etc.), Xiaomi-R1 achieved state-of-the-art results (around 59.1% success, 70.3 progress score). And on the original RoboCasa (single-scene version rather than the 365 variant), it hit 74.5% average success. In each case, it surpassed prior specialized policies by a wide margin. In short, Xiaomi-R1 is reliably at or near the top of every benchmark it’s been evaluated on, before any extra fine-tuning. Fine-Tuning on New Tasks. Beyond zero-shot performance, one of the touted benefits of XIaomi-R1 is that it can be adapted to new tasks with very little data — the hallmark of a foundation model. Imagine a scenario where you have a brand-new, very complex robot task (say, a multi-step assembly on real hardware). Instead of training from scratch, you can start from Xiaomi-R1 and fine-tune. The authors tested this idea in simulation. For example, they took sophisticated, dexterous scenarios and compared how many extra samples each method needs to reach a given success rate. The result: Xiaomi-R1 required far fewer demonstrations to achieve strong performance than training a policy from scratch or many other baselines. In effect, pretraining on 100k hours “covered the hard part” of learning good motion priors, so the robot only needs to see a bit of new data to specialize to a particular task. This kind of transferability has enormous practical implications. In robotics, collecting data on a new task is expensive. If one can just collect a few dozen episodes instead of tens of thousands, development costs plummet. Xiaomi explicitly notes that this “train general base, then adapt with little data” approach can slash both data collection and compute costs for novel tasks. As an example, the paper reports that fine-tuning Xiaomi-R1 on real-robot versions of complex tasks (in simulation) led to success rates and progress scores far above the prior state of the art. In some cases they got multiple-fold improvements in efficiency compared to the next best method. Relation to Prior VLA Models. Xiaomi-Robotics-1 builds on a growing line of work in vision-language-action models for robotics. An earlier example from the same group is Xiaomi-Robotics-0 (Feb 2026). That model similarly combined large-scale cross-embodiment data with vision-language priors. In fact, Xiaomi-R0’s paper explains that it was pre-trained on a diverse set of robot trajectories and language data, giving it broad action capabilities while still retaining the semantic grounding of an image-language model. R0 also introduced techniques for asynchronous action streaming (chunking predicted actions to minimize latency) so that the policy could run smoothly in real time. Xiaomi-Robotics-1 can be seen as the next generation: it takes the same core idea (large data, transformer policy, language conditioning) but vastly scales the dataset and model. The two-stage recipe (pretrain on UMI demos, then post-train on real robot data & instructions) is reminiscent of R0’s “pretrain/fine-tune” approach, but on a much larger scale. Other contemporaneous works also aim at multi-task VLA policies (e.g. RT-1 from Google, Cosmos, GROOT, and so on), but Xiaomi-R1’s history on benchmarks shows it currently leads the pack. Importantly, the Xiaomi work highlights the power of real demonstration data collected with the UMI interface. While many approaches rely purely on simulation or scripted data, Xiaomi’s use of tens of thousands of hours of human-demonstrated robot operation – automatically annotated with language – seems to give it an edge in realism and diversity of skills. This may be a critical ingredient for the observed generalization. How to Use Xiaomi-Robotics-1 (and Why It Matters). For robotics practitioners and ML researchers, Xiaomi-Robotics-1 is interesting both as a technical achievement and as a potential tool. On the technical side, it validates that the foundation model paradigm extends to embodied agents: with enough data and model capacity, one can build a single policy that handles very diverse tasks, much like how GPT-4 can answer many types of queries. It also shows how automated pipelines (like vision-language labeling) can unlock scaling in robotics. On the practical side, we can imagine a few ways to use or adapt this model once it’s released. The authors say they will share code and checkpoints, so researchers could fine-tune Xiaomi-R1 on their own target domains. For example, a lab could take the pretrained base and use a few hours of demonstrations to specialize it for assembly tasks, warehouse picking, or anything of interest. Because R1 already knows general heuristics — how to grasp objects, navigate around obstacles, follow instructions — the fine-tuning would likely require orders of magnitude less data than training from scratch. In other words, R1 could become the foundation policy akin to how ImageNet models became foundations in vision or GPT became a foundation in language. There are also broader implications for the field. The success of Xiaomi-R1 suggests that building even larger or more diverse robot datasets could further push the frontier. If we pool data across institutions or use ever larger models, perhaps future versions will handle even more complex tasks, longer horizons, or truly creative problem-solving. It also raises questions about evaluation: as each team releases stronger base policies, benchmarks will need to evolve (as seen with RoboDojo, VLABench, etc.) to measure subtler aspects of intelligence. Conclusion. Xiaomi-Robotics-1 represents a leap in multi-task robotic policies. By leveraging over 100,000 hours of real manipulation data and a clever two-stage training regimen, the researchers have produced a vision-language-action model that performs many tasks out-of-the-box and can quickly adapt to new ones. It strongly outperforms previous methods on benchmarks like RoboCasa-365 and RoboDojo, demonstrating that scale pays off. For roboticists, R1 is an exciting example of a “foundation model” for embodied AI — a generalist starting point that could accelerate the development of new robot applications. As the code and models become available, we’ll likely see a flurry of experiments: fine-tuning R1 for niche problems, extending it with additional sensors, or even distilling it into lighter-weight policies for deployment. Meanwhile, this work adds to a growing consensus: combining large datasets, language grounding, and flexible architectures seems to be the way forward for broader robot intelligence. Xiaomi-Robotics-1 is a milestone on that path, showing we can build generalist robot brains by simply giving them enough data and a way to understand instructions. Sources: The above is based on the Xiaomi-Robotics-1 arXiv preprint and related project descriptions, which detail the model’s training recipe, scale of data, and benchmark performance.