Latest / Embodied AI 101 / Robix: Unified Vision-Language Model for Robotic Reasoning and Planning
Transcript
- Robix: Unified Vision-Language Model for Robotic Reasoning and Planning. One of the big challenges in embodied AI is giving robots the sort of high-level reasoning and flexible planning skills humans take for granted. Traditional robots usually have separate modules for perception, planning, and dialogue, which must be carefully hand-engineered and integrated. Robix takes a radically different approach: it is a single vision-language foundation model that sits at the top of a hierarchical robot system, acting as the robot’s “brain.” From raw images and spoken instructions, Robix generates both a natural-language response and the next symbolic action to execute. In effect, it unifies robot reasoning, long-horizon task planning, and human-robot interaction into one end-to-end model. Rather than relying on distinct planning engines or dialogue managers, Robix “thinks out loud.” At each step it produces an explicit chain-of-thought (an internal reasoning paragraph), decides the next atomic command for the low-level controller, and even formulates a verbal reply to the user. This architecture enables Robix to follow complex multi-step instructions, monitor its progress, ask clarifying questions when needed, and adapt on the fly to interruptions or invalid commands. Conceptually, Robix is designed as the high-level cognitive layer of a robot. It sits above a separate motion-control module (often another learned vision-language-action, or VLA, model) that actually executes grasps or moves. Robix itself does not output joint torques or pixel-level actions. Instead, it outputs discrete commands like “place the banana in the plastic container,” or “pick up the red mug.” A human operator or a specialized low-level model then carries out these commands in the real world. This hierarchical split has advantages: Robix can focus on abstract reasoning (planning, understanding language, common sense) while relying on the low-level controller to handle precise motor execution. In practice, Robix repetitively loops as follows: it observes the latest camera image and any user utterance, then it predicts a thought sentence (its internal reasoning), an action command, and a response to speak to the user, all conditioned on the current context and the past history. To manage context, the system buffers only the most recent N visual frames explicitly, while logging the full trail of past thoughts and actions in memory. This unified architecture gives Robix some striking capabilities. In demonstrations and experiments, it exhibits proactive dialogue (it can ask clarifying questions anytime an instruction is ambiguous), real-time interruption handling (a user can cancel or modify the task mid-execution and Robix will adjust), and commonsense reasoning about the scene. For example, if told “prepare a fruit snack,” Robix might verify which fruits are available before proceeding. If the user suddenly says “actually, don’t use the strawberries,” Robix incorporates that feedback immediately. These behaviors are usually hard to implement in modular robots, but Robix’s end-to-end chain-of-thought approach makes them natural. In short, by bringing language (both understanding and generation) together with vision and goal-oriented planning in one model, the system enables more fluid and natural human–robot collaboration. Training Robix: A Three-Stage Pipeline. Building a model with such broad abilities requires carefully staged training. The Robix team extends a base vision-language model (Qwen2.5-VL) into a robotics-specialist through three phases: (1) Continued Pretraining, (2) Supervised Fine-Tuning (SFT) on synthetic interaction data, and (3) Reinforcement Learning. Each stage gradually injects more embodied and interactive training signals. (1) Continued Pretraining. The first stage is essentially a massive domain-focused pretraining. Starting from the pretrained Qwen2.5-VL model (in both 7 B and 32 B variants), Robix is “continued” on roughly 200 billion tokens of curated multimodal data. Importantly, this data is biased toward robotics-relevant tasks. It is split into multiple categories designed to instill spatial and task reasoning:. 3D Spatial Understanding: About 40B tokens come from multi-view and 3D geometry tasks. For instance, Robix sees image pairs from different camera angles and learns to predict how points correspond between views (multi-view correspondence). It is trained to detect and describe 3D bounding boxes (“this is a blue box on the shelf”), to sort objects by relative depth or estimate their absolute distance, and even to predict its own camera motion (egomotion) from visual cues. These tasks teach the model about three-dimensional scene layout and movement, foundational for navigation and planning. Visual Grounding: Another ~70B tokens come from 2D grounding and object tasks. Here Robix learns to localize objects mentioned in language. Training examples include 2D bounding box annotation (“point to the red mug”) and dense captioning. It also practices tasks like object counting and understanding visual prompts. Essentially, Robix sees text queries paired with images (often annotated with points or bounding boxes) so that it learns to tie words to specific image regions. These data improve Robix’s ability to precisely identify the right object when asked (“Where is the orange?”) or to follow a referring expression. Task-Centric Reasoning: The remaining 10B tokens target higher-level task reasoning. These are drawn from egocentric and robotic datasets (like AgiBot, DROID, Ego4D, etc.) where an agent performs tasks. The tasks include verifying task status (“is the table clear?”), assessing affordances (“can I place the plate here?”), and predicting next actions in a task sequence. Crucially, these examples come with chain-of-thought traces. That is, for each task, a model (such as a powerful vision-language model) has generated a step-by-step reasoning write-up, and Robix learns to mimic this style. This means early on Robix is already exposed to explicit logical reasoning about tasks. Beyond these, the pretraining corpus also includes more general multimodal examples (image captions, VQA, STEM questions from visual diagrams, etc) and over 1M high-quality instruction-following examples to instill basic conversation skills. The key takeaway is that after this continued pretraining, Robix has seen a wide range of scenes and subtasks: it knows what objects look like, has learned geometric and spatial concepts, and has practiced reasoning about multi-step tasks — all in a vision-language context. (2) Supervised Fine-Tuning with Synthesized Interactions. While the pretraining builds broad capabilities, it does not teach Robix how to be a polite multi-turn robot assistant. For that, the authors generated a huge amount of synthetic dialogue-style training data. This SFT stage addresses the lack of large human–robot conversation datasets by creativity. The RoboLab team assembled seven categories of instruction-following dialogues:. Multi-stage instructions: Sequences of subtasks like “First take this plate to the sink, then grab the lemon, finally put the lemon in the fridge.” These were derived from real teleoperated trajectories. Constrained instructions: Tasks with explicit constraints, e.g. “Stack only red cups on the top shelf.”. Open-ended instructions: Generic goals in random table-top scenes, such as “Find something to pack in the bag.”. Anytime interruptions: Scenarios where the user might interrupt mid-task with a new instruction or clarification, forcing the model to replan on the fly. Invalid instructions: Tasks that the robot should refuse or ask for clarification on (e.g. “Pick up the imaginary umbrella”). Ambiguous instructions: Commands that lack key information, prompting “what do you mean by that?” or similar proactive query. Chat instructions: Natural convo or social interaction snippets to make the robot’s dialogue more fluid. Each synthetic scenario is a little dialogue or episode where Robix is supposed to plan and reason. Importantly, the data synthesis includes chain-of-thought reasoning traces for every example. The system uses other top vision-language models to generate these step-by-step rationales, emphasizing scene understanding, long-term planning, and task steps. These generated thoughts (kept concise, under ~200 tokens) condition the model to output its reasoning explicitly. During fine-tuning, Robix learns to align such reasoning with chosen actions and responses: essentially it learns to think like a robot teleoperator and explain itself. This stage insures that Robix knows how to handle the full spectrum of user interactions — from following constraints to politely refusing impossible demands — within its unified output space of “thought–action–response”. (3) Reinforcement Learning. The final polish comes from a round of reinforcement learning (RL). The authors use a method called Group Relative Policy Optimization (GRPO) to fine-tune Robix’s behavior towards even more coherent planning. In lay terms, they treat the Robix model as an agent in a simulated environment (using the same synthetic tasks), and optimize it with a reward model that checks whether Robix’s internal reasoning, chosen action, and eventual outcome all line up sensibly. The RL stage explicitly penalizes “irrational” reasoning (for example, if the chain-of-thought says one thing but the action contradicts it). It also enforces consistency of formatting and answers, hoping to reduce errors like going off-topic. By alternating Robix’s decisions with graded rewards, GRPO nudges the model to place its blocks of reasoning into better alignment with the tasks and instructions. This further improves performance, especially on long-horizon tasks, by encouraging Robix not just to think hard but to follow through correctly. After RL, the final policy (called Robix-7B-RL or Robix-32B-RL, depending on model size) is used in evaluation. In summary, the three-stage training strategy first builds broad vision and reasoning skills, then specializes them for interactive dialogue tasks, then fine-tunes them under goal-directed rewards. Each phase stacks on the previous, giving Robix both a wide knowledge base and the ability to apply it coherently in the real world. Fundamental Perception and Reasoning Abilities. Before testing Robix in full closed-loop tasks, the authors evaluated its core “perception+reasoning” chops on standard benchmarks. In these tests, Robix competes against state-of-the-art vision-language models, including other robotics-focused LMs and generalist models like Gemini and GPT-4o. Across the board, Robix shows strong improvements, especially in tasks where embodied spatial understanding matters. For 3D spatial reasoning, Robix clearly outperforms its Qwen2.5-VL backbone and even larger models. On eight industry benchmarks (Fig. interpretation, spatial relationships, depth puzzles, etc.), the 7B and 32B Robix models achieve average accuracies of 73.4% and 75.8%, compared to 66.9% and 70.7% for Qwen2.5-VL of the same sizes. In other words, Robix gets roughly 6–7 points better. Even more impressively, the Robix-32B model beats Google’s Gemini 2.5 Pro on 5 out of 8 spatial tasks. This indicates that the specialized pretraining data successfully endowed Robix with a deeper intuition for 3D scenes. For visual grounding and understanding, Robix again leads on nearly every test. Across eight grounding tasks (e.g. referring expressions, object pointing, etc.), the gains are especially large. In one benchmark (LVIS-MG), Robix-7B and Robix-32B lifted F1 scores by nearly 40 and 25 points over the base Qwen2.5-VL models. This shows Robix can locate and count objects from descriptions far better. In practice, that means if you say “point at the red can next to the blue bottle,” Robix is much more likely to get it right than most open-source models. Notably, it also outperforms closed-source models (like GPT-4o) on many of these grounded tests, a testament to the targeted grounding data used in pretraining. The gains continue on task-centric reasoning benchmarks. For example, on the new AGIBot-ER suite (a set of embodied tasks requiring step-by-step execution), Robix-7B/32B improved absolute accuracy by +12.8 and +7.2 points over their Qwen backbones. They similarly beat Cosmos-Reason1 and RoboBrain baselines by large margins. Essentially, Robix has better “common sense” about how to carry out tasks in an environment. We see this especially on tasks that involve multi-step logic or verifying conditions (“Is the toaster already on?”), which are measured in these benchmarks. In summary, across 31 public multimodal benchmarks of vision and reasoning, Robix consistently places at or near the top. Its strength shines in embodied settings: spatial reasoning, grounding, and task planning. On purely general benchmarks (like VQA), the difference is smaller (large general models still lead there), but Robix remains competitive. These results confirm that the continued pretraining stage effectively taught Robix the building blocks of embodied AI. Interactive Planning and Benchmark Tests. Beyond static evaluation, the authors tested Robix on interactive, long-horizon tasks. They created new benchmarks to simulate realistic robot jobs and measured how well Robix can plan through a multi-step instruction. Two main evaluations were used:. Offline benchmarks (teacher forcing): Here Robix is given tasks and must generate a complete plan in text (think of it as a “paper robot” writing out steps). The authors built three sets: (1) An AGIBot-derived set of 16 common household tasks (none seen in training), (2) an internal out-of-distribution (OOD) set of 16 scripts covering things like table setup, grocery shopping with unusual items, etc., and (3) an in-distribution (ID) test drawn from the synthetic data categories (multi-stage, open-ended, invalid, etc.). Intersection of examples required reasoning, unconstrained planning, or dealing with tricky instructions. Metrics: For the ID/OOD sets, they measured plan accuracy (matching a reference plan) or F1 for instruction categories like refusal or replanning. Importantly, all these tests are automated: Robix sees the instructions but does not execute in reality, it just lists how it would act. Robix again excels. The 32B model with RL (Robix-32B-RL) wins all evaluations. On every test set, it outperforms all baselines, open-source and closed-source. For example, on the hard OOD benchmarks it achieved on average ~90% plan correctness, whereas Gemini 2.5 Pro and GPT-4o lagged behind by several points. The 7B version also beats other similar-size models by a wide margin. These offline results underscore that Robix truly generalizes: even though the tasks may involve objects or situations outside its training, chain-of-thought reasoning and the RL fine-tuning help it to adapt its logic. In fact, the authors perform an ablation showing that removing chain-of-thought outputs drastically lowers performance on the OOD tasks – proving that having the model explicitly reason step by step is not just for show, but crucial for robustness. One noteworthy point is how much the three-stage pipeline helped. The RL-stage models (the “-RL” variants) significantly outperform the SFT-only models on long tasks, with gains of ~8 points on the OOD test. This indicates the reinforcement component indeed made the planning more coherent across many steps. Real-World Robot Experiments. Laboratory benchmarks are one thing; the ultimate test is running an actual robot. The authors wired Robix into a real hierarchical robot system and tried it on five everyday tasks:. Table Bussing: Clearing used dishes/utensils off a table. Checkout Packing: Organizing groceries into bags at a checkout. Dietary Filtering: Selecting foods under a dietary constraint (e.g. “pack a caffeine-free breakfast”). Grocery Shopping: Recommending and picking items in a grocery aisle from instructions. Tableware Organization & Shipment: Sorting and packing tableware for shipment. These tasks involve objects, categories, user preferences, and demand a sequence of moves — exactly what Robix is built for. They ran two sets of trials. In the first (“UMI” setup), a human operator effectively acted as the low-level executor. That is, Robix generated commands, but a human with a teleoperation glove carried them out perfectly. This isolates errors to Robix’s planning and understanding, not motor failures. In the second (“GR-3”) setup, a trained vision-language-action model (GR-3) in a miniature robot (“ByteMini”) physically executed the commands, so this tested the full autonomy loop. Results (UMI): With perfect execution, the Robix-32B agent averaged about 92.6% task completion across all scenarios. Gemini-2.5-Pro (Google’s closed model) was slightly under at 91.0%, so Robix edged it out. The open-source Qwen2.5-VL-32B baseline barely managed 28%! In some tasks, Robix was the clear leader. The high score shows that Robix has learned to generate plans that a human (or perfect controller) can follow to finish a complex task nearly completely. The bullet point from the site notes that Robix ranks first in 3 out of 5 tasks and has higher average progress than Gemini. This validates that Robix’s integrated reasoning and dialogue really pay off in realistic scenarios. Results (GR-3 Robot): When connected to an actual mobile manipulator (the ByteMini with the GR-3 controller), Robix-32B still achieved remarkable results: about 92.5% average task progress. That beat Gemini-2.5-Pro by 4.3 points and GPT-4o by a whopping 28.1 points. In other words, Robix completed almost all sub-tasks correctly, whereas its competitors were measurably less successful. Notably, these real-robot trials included interruptions and human-in-the-loop dialogue as well. For instance, if the user asked mid-task about item availability, Robix adjusted its plan. The evaluation reports that Robix handled on-the-fly replanning and clarifications robustly in every test. Together, these online experiments show that Robix’s planning isn’t just a lab curiosity: it translates into real-world performance. In tasks like clearing tables or packing groceries, it can produce step-by-step plans that a robot can actually execute and reach the goal. The results reinforce that integrating the cognitive layer with end-to-end training paid off: Robix knew to ask “Do you prefer organic bread?” or “There is no red apple, would you like something else?” at appropriate times, which older models typically miss. Significance and Use Cases. Robix represents a new paradigm: treating high-level robot planning and interaction as a vision-language generation problem. Its success demonstrates that a single large model can handle what used to require complex pipelines. For robotics researchers, it suggests several practical lessons:. Unified chain-of-thought training: By having the model produce an explicit internal monologue, we can steer it towards logical, multi-step reasoning. Robix shows this works: denying chain-of-thought sharply drops performance. Other teams can similarly include CoT outputs during training to guide planning. Data synthesis for interaction: Robix’s approach to generate synthetic dialogue data—covering ambiguous cases, interruptions, invalid commands, etc.—shows a way to overcome data scarcity in HRI. Even without real conversations, we can simulate them by combining existing trajectories and powerful backbone models. This pipeline can be adapted by others to train robots for flexible dialogue. Hierarchical architecture: The idea of a “cognitive LLM” sitting above a low-level controller proved fruitful here. Practitioners could adopt similar designs: use any vision-language-action model (e.g., learned end-to-end controllers or even classical motion planners) for low-level execution, while an LLM-style model handles gist-of-task planning and communication. This decoupling leverages the strengths of both. Scale and sources: Robix was a very large model (32B) trained on an extremely large custom corpus (hundreds of billions of tokens). While this is beyond most labs, even middle-sized versions (7B) showed big gains. Their use of existing datasets (ScanNet, Ego4D, DROID, etc.) to build the corpus means that motivated teams could assemble similar pretraining data for their domain. Baseline improvements: If you are using LLM techniques for robotics already, Robix’s results set a new bar. For example, systems based on GPT-4 or Gemini might be pleasantly surprised that a well-trained open-access model can match or beat them in planning. This indicates room for open research to catch up with closed-model capabilities. In summary, Robix pushes the frontier on language-guided robotic planning. It achieves end-to-end task execution (from pixels and speech to multi-step actions) with a unified network. The work shows how combining multimodal pretraining, interactive synthetic tuning, and RL can yield a robot “brain” that generalizes strongly. For roboticists, it is a proof-of-concept: next-generation assistants might indeed be large vision-language models that seamlessly handle both dialogue and action planning. Future work may expand on Robix by scaling up model size and data further, or by integrating richer memory and world models, but the core takeaway is that the Robo-LLM approach works remarkably well. References: The Robix results above are drawn from the authors’ project description and preprint, which detail the model’s architecture, training, and performance on various benchmarks.