Latest / Embodied AI 101 / EmbodiedOneVision: Interleaved Vision-Text-Action Pretraining for General Robot Control
Transcript
- EmbodiedOneVision: Interleaved Vision-Text-Action Pretraining for General Robot Control. Modern robotics is moving toward generalist embodied agents – robots that can understand language, see the world, plan multi-step tasks, and execute complex actions. A key challenge is endowing robots with human-like flexibility: the ability to reason across vision, language, and action in a seamless loop. Recent “vision-language-action” (VLA) models try to close this gap by co-training on large text-vision corpora and robot trajectories. These systems have made impressive strides in multi-modal robot control, but often still fall short of fluid interleaving: their decisions don’t quite match the fluid give-and-take of human reasoning and acting. In this context, EmbodiedOneVision (also called EO-Robotics) emerges as a notable advance. Introduced by researchers at Shanghai AI Lab in collaboration with AgiBot, EmbodiedOneVision proposes a unified embodied foundation model (called EO-1) together with a massive curated dataset (EO-Data1.5M), trained through interleaved vision-text-action pretraining. The idea is to let a single model learn to handle images, instructions, and robot actions all together, enabling “seamless robot action generation and multimodal embodied reasoning”. EO-1 is an open-source model (3 billion parameters, MIT-licensed) built on a decoder-only transformer backbone (initialized from the Qwen2.5-VL vision-language model) that can consume images, video frames, text, and robot state, and produce both language outputs and continuous action commands. Crucially, during pretraining the model sees interwoven sequences of vision, text, and actions from the new dataset, and is trained with a combination of language modeling and a “flow matching” objective for actions. The result is a single model that can, for example, read a natural language instruction, look at an image of a scene, plan a sequence of moves, and actually output the joint-control commands to carry them out. According to the authors, EO-1 surpasses prior open VLA models on a battery of benchmarks (from embodied question-answering to long-horizon manipulation) and even outperforms large proprietary systems like GPT-4 on some tasks. In this article we’ll unpack how EmbodiedOneVision works: its architecture, its data, its training, and its performance. We’ll also discuss why it matters and how you might use or extend it in your own robotics research. Foundations: Unifying Vision, Language, and Action. EmbodiedOneVision is motivated by how humans effortlessly mingle perception, language, and physical action. When a person is told “make me a sandwich,” we parse the instruction, visually inspect the kitchen, plan sub-steps (get bread, spread butter, etc.), and physically manipulate objects – often without consciously toggling between separate “modules” of thinking and doing. To approach that level of fluidity, EmbodiedOneVision adopts a single unified architecture that treats images, words, and actions on an equal footing. At its core is a decoder-only transformer that is given an interleaved sequence of tokens drawn from all modalities: visual tokens (from images or video frames), text tokens (from instructions or questions), robot proprioceptive state, and even noisy action vectors (for the continuous flow objective). As one team summary explains, “EO-1 model is a Vision-Language-Action (VLA) model that adopts a single unified decoder-only transformer, equipping with [a] discrete language-modeling head for. reasoning and continuous flow-matching head for robot action generation”. In other words, the same neural network processes all inputs and produces both text and action outputs. Figure 0 (below) conceptually illustrates the idea: the model can take a command or question in text, observe an image or camera feed, and then output either a textual answer or a stream of real-valued joint commands. This stands in contrast to architectures that separate perception from control, or that require an external planner. Here, the transformer itself is learning to do both high-level planning and low-level control, by treating action trajectories as just another “token” sequence (albeit continuous-valued, handled by a special head). Figure 0: EO-1’s Unified Transformer processes an interleaved stream of vision, text, and (noisy) action tokens. Discrete tokens flow through an autoregressive language head, while a continuous flow-matching head predicts robot control commands. Both heads share a common backbone (initialized from Qwen2.5-VL) and are trained together. To make this work, EO-1 uses two output “heads”: a language head (just a standard autoregressive LM head) and a flow-matching head for actions. During training, the data stream typically mixes textual and visual tokens with placeholder “noisy” actions. The language head is trained to predict the next token (as in a normal language model) to handle questions or instructions, while the flow-matching head is trained to “denoise” or predict correct continuous actions from a noisy input – analogous to diffusion modeling. In practice this means the model learns to output a precise robot trajectory in response to an instruction and scene, by gradually refining noise into the correct command sequence. The combination of these objectives drives the transformer to link language understanding with motion generation, effectively learning the affordances and physics of tasks along with how to describe them in words. The authors emphasize that this unified setup leads to seamless embodied reasoning. For example, a single forward pass can generate an explanation or reasoning about the scene (via the text head) and then immediately produce the next physical move (via the action head). There is no explicit modular planning – everything emerges from the same joint model. It’s a bold design: rather than training separate policy and planner networks or grounding text to motion step-by-step, EO-1 internalizes both tasks in one giant foundation model. As the project page puts it, this “enables seamless perception, planning, reasoning, and acting in a single model”. EO-Data1.5M: A Multimodal Embodied Dataset. The success of such a unified model depends critically on the data. EmbodiedOneVision introduces EO-Data1.5M, a massive curated corpus of over 1.5 million multimodal examples designed for this interleaved learning. Each example in EO-Data1.5M contains vision, language, and (simulated) action data woven together. The dataset is not just raw robot trajectories or random videos – it is carefully crafted to emphasize the kinds of reasoning and coordination needed in robotics. EO-Data1.5M samples are organized into three broad categories:. Physical Common Sense: These samples test understanding of the physical world. For instance, given an image or video of objects in an environment, the model might be asked questions about stability, collisions, or physics (“Will the block fall if I push it?”). Such data teaches the model basic intuitive physics of household or lab scenes. Task Reasoning & Spatial QA: This part focuses on planning and spatial language. The data may present a multi-step task (“Prepare a breakfast sandwich” or “Sort these groceries”), along with a description of the environment, and ask questions about the plan or relationships (“What should be done before grilling the steak?”, “Which item goes to the fruit basket?”). These examples hone the model’s ability to link textual instructions with the spatial layout of the scene. Interleaved Manipulation Data: Possibly the most innovative component, these samples stitch together reasoning and control. A typical interleaved example might include a sequence like: a video snippet showing a partial task, followed by a textual query about the next action, then the correct robot action (e.g. joint trajectories) to continue the task, and so on. In effect, it connects a question-answer pair with an action-execution step. This explicitly teaches the model the causal relationship between high-level “why/how” questions and the low-level motions that accomplish them. According to the authors, EO-Data1.5M was collected via a scalable pipeline that combined automated vision-language generation with human oversight. First, large corpora of robotics videos were filtered for diversity. The videos were segmented into clips of sub-tasks, and a vision-language model supplied descriptive captions. Then, question-answer pairs were generated using templates and an LLM, covering temporal reasoning (e.g. “What happened in the previous step?”), spatial inference (“Where will the red block move if pushed?”), and goal-oriented tasks. Importantly, for each question the correct robot motion was also included (as an “answer” in action space). This created richly annotated sequences where language and action are aligned to the same context. The scale is impressive. In addition to the 1.5M interleaved “embodied” samples, the authors supplemented pretraining with web multimodal data and robot control data. For example, they gathered ~5.7 million web-based image-text pairs, 7.1 billion tokens of natural language, plus 1.2 million robotic episodes (roughly 127.3B tokens) from various open datasets. In total this corpus spans text, images, video, and multistep robot trajectories – all unified in one pipeline. The EO-Data1.5M portion alone contributes about 1.0 billion tokens of interleaved reasoning data. In summary, EO-Data1.5M provides dense multimodal alignment. A single data point might say “Here’s an image of a countertop cluttered with ingredients, ask the sandwich-making agent ‘What’s the next step?’ and then record the actual joint commands to pick up the bread and spread butter.” Over millions of such examples, the model learns how textual concepts map onto spatial layouts and how those in turn map to physical controls. As the authors put it, the dataset was built to emphasize “interleaved vision-text-action comprehension”. This stands in contrast to earlier approaches that often train on language-only data OR vision-only data OR separate robot trajectories. By tightly coupling all three, EmbodiedOneVision attempts to make the model’s pretraining environment mirror a rich, interactive real world. Training: Autoregression Meets Flow Matching. Given the unified architecture and data, how is EO-1 actually trained? The key is a combination of two objectives applied to the same transformer:. Autoregressive Language Modeling: Whenever the data stream includes text (e.g. questions, instructions, or answers), the model is trained like a standard decoder-only language model to predict the next token. For example, if the input contains the query “Pick up the blue cube and…” the model must predict the continuation, or complete an answer sentence. This objective grounds the model in natural language and text-based reasoning. Flow-Matching Denoising for Actions: Whenever the example includes a robot action (typically in the latter part of the sequence), the model uses a continuous denoising objective. Intuitively, the training procedure injects noise into the target action vectors and has the transformer learn to “denoise” them back to the correct commands. This is akin to diffusion models or score-matching in continuous spaces. The “flow matching” term comes from a recent technique to train diffusion-like models more efficiently by matching vector fields. Here it lets the shared transformer learn smooth, multi-step action generation. During training on EO-Data1.5M, both objectives are applied in synergy. A mini-batch might contain some sequences focusing on question-answer turns (for the language head) and others on action demonstration (for the flow head). Many sequences mix both: for instance, a question in text might immediately be followed by the corresponding robotic action tokens. Crucially, the transformer’s self-attention is causal (left-to-right in time), so that earlier vision and language embeddings can influence later action embeddings. This enforces a temporal dependency: the model learns that “given this scene and instruction, output an action.” Over millions of examples, it internalizes the correlations between visual cues, textual goals, and motor commands. By virtue of this training regime, EO-1 effectively learns to do imagination and rehearsal. If asked a purely reasoning question (“Why did the block tip over?”) it can answer in text. If tasked with a control command (“Move the block to the left”), it will output a continuous trajectory. And in mixed scenarios, it can transition between answering questions and performing moves. One striking aspect is that the same model weights handle both kinds of outputs, so the boundary between “thinking” and “doing” is blurred. In their words, the model achieves “embodied reasoning and acting” in one sweep. Benchmarks and Robotic Experiments. How does EO-1 perform? The authors ran a large suite of evaluations on both offline benchmarks and real robot tasks. The headline result is that EO-1 outperforms prior open models across the board, and even beats larger closed models on some metrics. Multi-Modal Reasoning Benchmarks. For language-and-vision reasoning, they tested on several known benchmarks:. RoboVQA (Vision-and-Text Question Answering for Robotics): This dataset requires answering long-horizon, compositional questions about simulated robotic environments. EO-1 achieves a BLEU-4 score of 58.5 on RoboVQA, markedly higher than other open models: for comparison, GPT-4o scored 47.2 and Qwen2.5-VL 3B scored 55.9 in their tests. In raw percentages, those numbers suggest EO-1 provided roughly 10–15 % more accurate textual answers than the baselines, even surpassing GPT-4o (a tuned GPT-4 variant) by a notable margin. ERQA (Embodied Reasoning QA): A related benchmark focusing on reasoning about objects and goals, scored by exact-answer metrics. EO-1 scored 45.5 versus GPT-4o’s 40.0 and Qwen2.5’s 35.3. Again it’s top among open models, indicating better semantic alignment of vision and text. EO-Bench (Spatial and Temporal Reasoning): This appears to be a new suite introduced by the authors, with split “Spatial” and “Temporal” sub-tasks. The spatial tasks (perhaps things like rearranging objects) and temporal tasks (like multi-step plans) score out of 100%. EO-1 achieved 36.4% on Spatial and 38.9% on Temporal, edging out GPT-4o (35.6% and 39.3% respectively). Notably, GPT-4o still leads slightly on the temporal subset, but overall EO-1’s average was 44.8% vs GPT-4o’s 40.5%. This implies the single-model approach is already competitive with larger systems on complex multi-step understanding. In sum, on multi-modal comprehension tasks, EO-1 leads the pack of open models and even challenges closed-source giants. Given that EO-1 is only 3B parameters (a modest size by today’s standards), this is a strong indication that the joint pretraining was effective. The results suggest that combining vision and action data in training can yield better reasoning than vision-language alone. Robotic Control Benchmarks. The authors also evaluated pure control tasks in simulated environments commonly used in research. They compared EO-1 to baselines on standardized benchmarks:. LIBERO: A benchmark of various manipulation skills (like moving and placing objects). EO-1 achieved a 0.982 success rate (98.2%) on LIBERO, outperforming the best prior model (denoted “π₀”) which was 0.942. This suggests nearly perfect performance on these curated tasks. Simpler: A set of dexterous tasks run in Google’s robotics simulator and with different robot models (an Agility robot “WidowX”). For all variants of Simpler (different embodiments), EO-1 again had the highest success rates (e.g. 0.765 vs baselines around 0.714–0.555). In terms of these numbers, the margins are smaller because worldwide the tasks are already quite saturated. But consistently, EO-1’s policy outstrips other policies learned via imitation or planning. This is especially notable given that many baselines (like π₀) are strong imitation-learning agents trained on large datasets. The embedded reasoning in EO-1 seems to give it an edge in executing the right actions. Real Robot Experiments. Beyond sims, the EO-1 team tested on real hardware. They emphasize long-horizon, everyday tasks that require fine manipulation. For example:. Multi-Step Cooking and Sorting: Four complex tasks were used, including “Make Breakfast Sandwich,” “Roast Beef Steak,” “Fold Clothes,” and “Sort Grocery Items”. Each requires coordinating vision and dexterous motion (e.g. cutting, folding, placing). The authors report that EO-1 achieved “stable and strong long-horizon dexterity” on these tasks, meaning it reliably completed the multi-stage plans without failing. By contrast, they imply that prior models struggled with such extended sequences. Visual Reasoning Control Tasks: Two particularly illustrative tasks were Visual Rearrangement and Tic-Tac-Toe. In Visual Rearrangement, the robot must, say, sort or shuffle objects on a table into a new configuration based on a spatial puzzle. In Tic-Tac-Toe, the robot interprets the game state (visual board + instructions) and uses two arms to place X’s and O’s appropriately. Both require high-level planning (game strategy or spatial reasoning) and low-level control (bimanual arm coordination). EO-1 “seamlessly integrates high-level embodied reasoning with low-level robot control, enabling smooth, correct execution” of these tasks. The implication is that EO-1 could tell from the board “block my opponent” and then actually move its arm in the right way, in one continuous process. Adaptation and Generalization: The team also tested how well EO-1 handles variation. They changed object positions, lighting, and background scenes, and gave novel linguistic instructions. The result: EO-1 maintained robust performance, “stable instruction following and open-world generalization” even under these shifts. In one phrase, it could still understand “move the red cup to the left of the bowl” even if the cup was in a somewhat different place, or if the room lighting changed. This highlights the benefit of the large-scale, diverse pretraining data. In quantitative summary, EO-1 demolished most open baselines across these tasks. For example, on a suite of pick-and-place tests (seven tasks on a Franka arm), EO-1 scored 93.5% success versus ~83% for the next best. On an AgiBot humanoid doing long tasks, it got 80.7% success over ~67%. On a WidowX niche robot assembly, 85.2% vs baselines ~70%. Finally, on the reasoning-control tasks (four of them), EO-1 was 83.1% versus 61.7% for the runner-up. These are averages, but they indicate consistently stronger performance. An important point is that EO-1 uses the same model with the same weights for all these tasks, simply with different inputs and prompts. In other words, it truly is a generalist policy. One model handles ordinary pick-place, complex cooking, AND playing tic-tac-toe. Key Innovations and Technical Details. Let’s highlight the core innovations of EmbodiedOneVision in clearer terms:. Interleaved Multimodal Pretraining: The central novelty is training on sequences that mix vision, text, and action. Think of a sequence like:. By pretraining on 1.5M such sequences, EO-1 learns the joint space of questions, answers, and moves. This is quite different from simply training on vision-language (like CLIP or Flamingo style) or on pure action trajectories. The interleaving explicitly simulates the loop of TELL→SEE→DO→TELL→DO that exemplifies an embodied interaction. Unified Transformer Architecture: Rather than have separate modules for planning and control, EO-1 uses a single decoder-only transformer for everything. Inputs are embedded text tokens (from an LLM tokenizer), visual tokens (likely from patch-embedding or a pretrained ViT), robot state (joint angles/forces encoded as numbers), and even “noisy actions” (randomized vectors). All of these share the same transformer layers. The model then forks into two heads: (1) an autoregressive head that emits the next token if the sequence is text-based, and (2) a continuous head that outputs action vectors (joint velocities or torques). Crucially, everything is left-to-right causal, so actions can depend on all previous vision+text context. The backbone was initialized from Qwen2.5-VL-3B, a Chinese vision-language model; this gives it strong initial vision-text perception capability. Flow-Matching for Actions: The training of the action head uses flow matching denoising – a diffusion-like objective. Practically, each training action (a sequence of continuous motor commands) is corrupted with noise, and the model must predict the original. This enforces smooth action outputs and helps model uncertainty. Flow matching is related to work by Song & Ermon on generating continuous trajectories. Using this together with autoregressive (discrete) training is fairly new in robotics; it lets the transformer handle continuous outputs without a separate spline-fitting or regression module. Data Creation Strategy: EO-Data1.5M itself is an innovation. The authors report using large Vision-Language Models (VLMs) and heuristics to generate diverse QA pairs, spatial reasoning problems, and physical commonsense questions from raw videos. For example, they filtered robot task videos by visual similarity to ensure variety, cut them into short subclips, autogenerated subtitles, and then used LLM templates to form question-answer pairs. This pipeline yields data that cover not just “what is this object?” but “what happens if we do X?”, capturing both deterministic planning and hypothetical reasoning. The appendix (in the Chinese summary) hints at example templates: e.g., “If I move the blue block, where will the red block end up?” or “Which object should I grasp first to make a sandwich?”. Long-Horizon and Multi-Embodiment Focus: Many robotics datasets emphasize either very short skills or a single robot. In contrast, EO-1 was explicitly tested on long-horizon, multi-stage tasks and on multiple systems (a Franka arm, an AgiBot humanoid, and a WidowX robot). This means the model has seen many time steps of action sequences and learned to replan if needed. The multi-embodiment aspect suggests the training data included trajectories from different robot kinematics, teaching the model some degree of embodiment-agnostic reasoning. Open-Source Toolkit: A practical innovation is the use of HuggingFace and Lerobot to integrate this pipeline. The EO-1 team has released a HuggingFace Transformers pipeline for their model, plus “processor” code (Lerobot) that translates camera images and state into the model’s expected input format and vice versa. In effect, anyone can load EO-1 from HuggingFace, feed in an image and instruction, and get out a robot command sequence. The model card shows sample Python code: after preparing the camera image and robot state in tensors, one calls and obtains both a text answer and an action array. The team notes this requires only ~6.5 GB of GPU memory for inference, making it accessible to many labs. Significance and Applications. Why does EmbodiedOneVision matter to robotics researchers? Several reasons stand out:. Toward Generalist Robots: EO-1 is part of a wider shift toward “foundation models” for robots – large pretrained networks that serve as general-purpose brains for many tasks. This mirrors trends in language (GPT) and vision (CLIP, DINO) where one model does many tasks. Demonstrating that a 3B model can handle diverse, long-horizon manipulation tasks suggests that scaling up is worth pursuing. Practitioners interested in generalist automation should take note: this model provides a baseline that blends perception and control in one pipeline, without task-specific tuning for each skill. Open Access Resource: Unlike many prior robotics results, EO-1 and its data are openly available under an MIT license. The GitHub repo and HuggingFace pages are public. This means researchers can directly experiment: import the pre-trained model, or fine-tune it on their specific domain, or even augment the data. The availability of EO-Data1.5M itself (and related benchmarks like EO-Bench) could jump-start new work in embodied multimodal learning. Long-Horizon Control: A common failure mode in robot learning is short-sightedness: policies that do well for one-step or few-step tasks tend to drift over longer sequences. EO-1’s success on multi-step tasks (sandwich-making, folding clothes, etc.) shows that large-scale pretraining can embed a sort of “plan-like” knowledge. Researchers working on tasks that require many moves (assembly, cooking, etc.) may find that incorporating language-model-style pretraining yields more reliable overall behavior than pure imitation or reinforcement on short clips. Integration of Reasoning: EO-1 was explicitly designed so that “thinking” and “acting” happen together. Tasks like Tic-Tac-Toe or spatial puzzles, where a purely reactive policy would struggle, become solvable because the model can plan ahead in text and then execute. This is a new paradigm: you feed in a language-description of a reasoning problem, and the same network generates the solution in action space. People building robots for knowledge-centric tasks (household chores with natural language goals, or human-robot collaboration games) may leverage this integration. Benchmark Progress: The results reported (for example, BLEU scores on RoboVQA or success rates on LIBERO) provide new state-of-the-art numbers for open models. Even though the work is very recent, one can compare these to previous reports. For instance, older open VLA models typically scored much lower on RoboVQA (GPT-4o’s number is a useful bar; EO-1 beats it). This advances the frontier for what’s expected from a 3B model. The community can use EO-1’s performance as a target for any new model, and can evaluate on the same benchmarks to measure improvement. Using and Extending EO-1. Practically speaking, how would a researcher use EO-1 in their work? Because the model is on Hugging Face with a standard Transformers interface, it’s straightforward to plug in. The authors even provide example code. A typical loop looks like:. This unified API (via the and Lerobot processing) means any compliant simulator or real robot can feed visual and state signals in, and get action outputs out, without separate vision or RL modules. The model also produces text, so one could even use it as a conversational agent: for example, ask it “What should I do now?” and read its reply, then let it do it. One can fine-tune EO-1 on a smaller, domain-specific dataset if needed. Because it is a decoder transformer, standard fine-tuning methods apply. The authors themselves present variants where the model is adapted (“bridge” and “fractal” branches on HuggingFace). Alternatively, one could add reinforcement learning on top of EO-1 policy rollouts to further polish specific tasks, following strategies like “reinforcement learning from human feedback” used in LLMs. The dataset EO-Data1.5M is also available (as a HuggingFace dataset)。 Each sample is a JSON record with fields for images, text, and action sequences. Researchers could draw inspiration from its generation pipeline: for instance, adapting their own video-based Q&A task algorithms. Limitations and Future Directions. Of course, EO-1 is not without limits. It currently focuses on manipulation-type tasks in structured environments (tables, kitchens, etc.). It does not directly cover navigation or outdoor scenarios. Its performance, while strong, is not perfect – for example the EO-Bench scores hover around 40-45% for best models, leaving room to grow. The transformer may struggle with very long sequences beyond thousands of tokens, limiting ultra long-horizon tasks or very high-resolution inputs. And 6.5 GB memory per inference suggests it’s non-trivial to run on tiny robots. The authors themselves suggest rich future work. They mention exploring more complex scene understanding (like dynamic obstacle avoidance or human-robot collaboration), improving the model architecture (maybe asynchronous pipelines or more efficient backbones), and extending to new embodiment types (humanoids, legged robots, etc.). A key direction is incorporating more diverse data – they talk about adding human manipulation videos and detailed pose annotations, to transfer human dexterity priors into robots. Another is making the pipeline faster or lighter, so that the model could someday run in real time on an onboard CPU or mobile device. Conclusion. EmbodiedOneVision aims to break down the walls between perception, language, and action in a robot’s brain. By interleaving all modalities in training and deploying a unified transformer, it demonstrates that a moderately sized model can handle rich interactive tasks. The combination of open-source release and impressive benchmark performance will surely spur new work in embodied AI. For robotics researchers, EO-1 and its dataset offer a new starting point: a prebuilt “generalist policy” that can be studied, tested, and built upon. Whether one is interested in cooking robots, warehouse automatons, or social robots that chat and help, the ideas here – of end-to-end multimodal pretraining – are likely to be influential. Sources: The key details above come from the EmbodiedOneVision paper abstract and accompanying project materials, Hugging Face model card and README, and the EO-Robotics website, all of which describe the EO-1 model, dataset, training, and results. Additional summarized insights come from a Chinese-language summary of the paper. These were cross-checked with the publicly released code, documentation, and benchmark outputs linked by the authors. All cited URL references use the format specified.