Latest / Embodied AI 101 / One Model to See, Plan, and Act: Introducing EO-1 for Embodied AI

Transcript

One Model to See, Plan, and Act: Introducing EO-1 for Embodied AI · Embodied AI 101

• One Model to See, Plan, and Act: Introducing EO-1 for Embodied AI. EmbodiedOneVision (EO-1): A Unified Decoder-Only Transformer for Embodied AI • Delin Qu, Haoming Song, Qizhi Chen, et al. (Shanghai AI Lab & AgiBot) • arXiv • 2025. The dream of a robot that can see its world, reason about it, and then reach out and manipulate it — all with one mind-like model — is a longstanding goal in embodied AI. Traditional robotics pipelines often split perception, planning, and control into separate modules. Yet humans perform these tasks seamlessly together: we look at a scene, reason about our next steps, and immediately extend our arm to act. Achieving this fluid integration of vision, language reasoning, and action control in machines has proved challenging. Recent work has begun unifying vision and language (e.g. GPT-4’s vision capabilities) and even adding low-level motor control, but still typically treats planning and action as the final “sequencing” step. The new EmbodiedOneVision (EO-1) model, developed by Shanghai AI Lab and AgiBot, takes a bolder approach: it is a single 3-billion-parameter decoder-only Transformer that natively interleaves vision inputs, text tokens (instructions/questions), and continuous action vectors all in one sequence. By training on millions of triple-modal examples that mix camera frames, language, and robot actions, EO-1 learns to perceive the world, reason about tasks, and issue motor commands in a single integrated model. In effect, EO-1 is a unified embodied foundation model for robots: one neural network that can attend to an image, answer a question about it, and simultaneously generate the next robot motion, without chaining separate planners or controllers. The EO-1 model achieves strong results across a variety of benchmarks. For example, it scores 58.5 BLEU-4 on the RoboVQA task (well above the 47.2 achieved by OpenAI’s GPT-4o) and attains a 44.8 overall on the novel EO-Bench in multimodal reasoning. In robot manipulation, EO-1 reached 98.2% success on the LIBERO benchmark and an 86.0% overall success rate on 28 real-world tasks across four different robot arms. These far exceed previous open-source models (e.g. 94.2% for the prior π₀ flow-based model on LIBERO). In short, EO-1 sets a new bar for open-source embodied AI, demonstrating that one unified transformer can robustly connect “eyes, brain, and hands” in robots. The Embodied Intelligence Challenge. Embodied AI aims to endow machines with human-like intelligence in the physical world. This means combining several faculties: perceiving rich sensory input (vision and sensorimotor feedback), understanding high-level goals expressed in language or through examples, planning a sequence of steps, and finally controlling actuators to execute those steps. Traditionally, these capabilities have been compartmentalized. A robot vision network might recognize objects; a language model might generate a text plan; and a separate motion planner or reinforcement learning policy executes actions. These modular pipelines work for narrow tasks, but often struggle with the “long tail” of real-world generalization, where perception, reasoning, and control must flexibly inform one another. In contrast, foundation model approaches attempt to put many abilities into a single giant network. For example, Large Language Models (LLMs) like GPT-4 can now take images as input and answer questions about them, showing impressively multimodal reasoning. Models like Gato (DeepMind 2022) even handled simple control tasks by including some demonstration data. However, previous embodied AI models typically treat action generation as a special endpoint: they generate all the trajectory or command tokens at the end of a reasoning sequence, rather than interleaving sensory reasoning with ongoing control. As the EO-1 authors note, this “sequential” approach ignores the genuine interplay between reasoning and acting that we see in human behavior. Several recent works have moved toward integrated models. PaLM-E (Google, 2023) inserts visual embeddings into a huge language model to guide robot actions, and SayCan (Google, 2022) uses a LLM with grounded affordances. Yet these either rely on multiple steps (language planning + separate execution) or require extremely large pre-trained modules. The π₀ model (2024) from UC Berkeley introduced a hybrid “flow” decoder to output continuous actions, and OpenVLA (an open-source project) provided initial benchmarks. EO-1 builds on these ideas and pushes them further by fully unifying vision, language, and motor control in one Transformer. In summary, EO-1 tackles a core question for embodied AI: Can one single neural model simultaneously look, think, and act? The answer, so far, seems to be a promising “yes.” By designing a decoder that can predict both words and actions, and by training on interleaved multimodal data, EO-1 demonstrates that a unified architecture can indeed capture the entwined reasoning–acting dynamics of real robots. EO-1’s Unified Architecture. At its core, EO-1 is a 3-billion-parameter decoder-only Transformer. This means it processes input and produces output in a single sequence, just like an autoregressive language model (e.g. GPT) does with text. But instead of only text tokens, EO-1’s sequence includes interleaved tokens of different types: image patches, textual instructions or questions, the robot’s state (e.g. joint positions), and pieces of the robot’s action vector. In other words, EO-1 sees the camera image as a set of tokens, reads the instruction or question as tokens, and also takes as input the previous or “noisy” action as tokens. The Transformer then outputs the next tokens, which could be either the next word of an answer or a refined action command. Crucially, EO-1 uses two different output “heads” on top of the shared transformer: a discrete language-modeling head for text, and a continuous flow-matching head for actions. The text head simply predicts the next token (like GPT), allowing the model to generate words that describe or reason about the scene. The action head predicts a continuous vector field that encodes the robot’s next motion. In effect, the model simultaneously learns a standard next-token objective (for language) and a flow-based denoising objective (for the action commands). Training then proceeds autoregressively through the entire interleaved sequence. In practice this means that when EO-1 “speaks,” it might output some words, then an action vector, then more words, and so on, all with a single forward pass of the transformer. Because the model is encoder-less (decoder-only), all inputs are turned into a single token sequence. Images are first encoded as a series of discrete “patch” tokens (for example via a visual tokenizer or encoder), and robot sensor values (like end-effector positions) are linearly projected into tokens. The positional context reflects the sequence order, but conceptually you can think of it as citing each piece of data in turn. The authors initialize EO-1’s weights from an existing multi-modal LLM (Qwen2.5-VL), giving it a rich prior in vision-language understanding, and then further train it on the specially gathered embodied AI data. One innovation in EO-1’s design is how it handles partial actions. In a sequence, the model might need to generate a block of action (a vector) in the middle of text. The “flow-matching” head treats the output as a denoising problem: during training, the true action is corrupted with noise in the middle of the sequence, and the model learns to predict the clean action. At inference, the model can output a precise action vector. To avoid disrupting the causal structure, EO-1 uses a “rectified sampling” strategy: if there are multiple action segments in a sequence, it replaces earlier noisy actions with the clean ones as it goes, so later reasoning sees the correct prior actions. The technical details are deep, but the upshot is that EO-1’s single Transformer can flexibly weave together words and motor commands. In short, EO-1’s universal transformer processes multimodal inputs indiscriminately and produces either textual or continuous outputs as needed. This unified architecture “enables seamless perception, planning, reasoning, and acting in a single model”. By jointly optimizing both text and action objectives, the model builds cross-modal knowledge transfer: for instance, textual commonsense can inform how it grips an object, and camera images can influence its planning language. As the EO-1 team puts it, their model integrates “discrete auto-regressive decoding with continuous flow-matching” to achieve a “synergistic integration of language, vision, and action modalities”. Interleaved Pretraining: Vision, Language, and Actions. A key to making the unified model work is the pretraining paradigm. The EO-1 authors intentionally use interleaved vision-text-action pretraining. That is, each training example is a sequence that mixes all three modalities. For example, one sample might start with an instruction (“Pick up the red cube and stack it on the blue one”), include a sequence of image frames of a robot arm in motion, and also contain the corresponding action vectors at each step. The ground truth “next token” could be another piece of text (e.g. an answer to a question) or the next action vector (to keep moving). By training on such sequences, the model learns the natural interplay of thinking and doing. Most prior models in this space have learned vision-language and action in separate phases. In contrast, EO-1’s training alternates between text and action targets. Concretely, the loss function is the sum of two parts: a standard next-token log-loss for the text segments ($\mathcal{L}{ar}$), and a flow-matching denoising loss for the action segments ($\mathcal{L}{fm}$). The model back-propagates through both seamlessly. In practice, this lookup makes EO-1 equally comfortable outputting fluent language or precise motor commands. This interleaved training leads to a virtuous cycle: text reasoning helps refine action generation, and vice versa. The authors observe that after training on millions of these mixed sequences, EO-1 gains “reasoning-enhanced generalization”: it can answer complex questions about the scene and produce robust, dexterous motions. In the EO-1 coding framework, this synergy is built on combining standard autoregressive training with modern diffusion-like techniques (flow matching) for continuous outputs. The result is an architecture where language and action are not siloed tasks but different “languages” spoken by the same Transformer. In sum, EO-1’s pretraining is what makes it genuinely multimodal. By never forcing the model to commit to “plan then act” or “think then move” but rather showing it examples of intertwined reasoning and acting, the model internalizes how to loop between thinking and doing. As shown in experiments below, this pays off in benchmarks that test interleaved capabilities (like answering questions while controlling a robot). Constructing the EO-Data1.5M Dataset. Of course, a model can only learn as much as the data it sees. To feed EO-1 the rich multi-modal sequences it needs, the authors assembled EO-Data1.5M, a large, diverse embodied AI dataset with about 1.5 million samples. This dataset is a cornerstone of the project, combining web-sourced multimodal data, existing robot-control logs, and newly created reasoning examples. The construction of EO-Data1.5M involved several stages:. Robot Video Filtering: First, the team collected raw robot demonstration videos from sources like simulated environments and real robots. They then used visual feature clustering and human review to filter out redundant or irrelevant footage, ensuring the remaining videos covered a wide diversity of skills (multiple robot arms, object arrangements, etc.). This created a base of robot episodes. Video Segmentation and Captioning: Next, these long robot episodes were chopped into short clips, each representing a sub-task or action sequence. Each segment is captioned or annotated with the detailed actions taken (e.g. joint angles, gripper commands) so that each segment becomes a self-contained (image sequence, action sequence) pair. This step yields roughly 1.2 million robot-control sequences. QA Pair Generation: To instill reasoning capabilities, the authors also generated question-answer tasks from the data. Specifically, they created two types of reasoning examples. Embodied temporal reasoning questions ask about task planning and physical commonsense (e.g. “What will happen if the robot fails to pick up the green block first?” or planning subgoals). Embodied spatial reasoning questions probe understanding of space and objects (e.g. “Where should the robot place the yellow cube to complete the stack?”). These QA pairs involve both the visual context and the underlying action sequence, and total over 1.5 million examples. In addition to these, EO-Data1.5M includes large-scale web multimodal data (about 5.7M examples) such as image-caption pairs and vision-language datasets, and more robot-control datasets (like AgiBotWorld, LERobot logs, RoboMIND). Overall, the full dataset blends: web vision-language data, real robot logs, and the team’s own interleaved reasoning tasks. It is carefully curated for quality, not just quantity, as the authors emphasize that “the quality and task-alignment of interleaved multimodal data” is crucial for open-world generalization. By pretraining EO-1 on this dense dataset with interleaved vision-text-action sequences, the model learns from a wide range of situations: reading instructions, watching many robot videos, and even answering questions about them. This large-scale embodied dataset is a major contribution of the EO-Robotics project, and it gives the model the grounding it needs to tie abstract reasoning to physical movement. Hands-On: How EO-1 Works in Practice. Let’s walk through what EO-1 actually does in a typical use case. Suppose you ask a robot, “Can you pick up the red cube and stack it on the blue cube?” and show it a camera image where both cubes are visible on a table. The user input here is an instruction in text and an image. These are tokenized: the words become text tokens, and the image becomes a mesh of discrete visual tokens via the model’s visual encoder. The robot’s own joint-state (e.g. current arm angles) is also included as a token sequence. These tokens are interleaved and fed into the transformer. EO-1 then starts generating the output sequence token by token. Initially it might continue the instruction answer or internal “thinking” text with something like “Initiating plan.” as a token. But at some point it will switch to outputting an action vector token – for example, a small velocity command for the arm’s motors. Internally, the Transformer’s hidden layers are attending jointly to the vision tokens and the words, so it “knows” that the red cube is the target and the blue cube is the destination. Once it outputs the action token (through the continuous-flow head), that actually causes a real or simulated movement. Then EO-1 receives the next image frame from the camera (after the arm moved) and feeds that along with the previous tokens back into the transformer (the transformer is stateless between steps, so it always sees the recent sequence). Now EO-1 might describe the new scene (“Cube lifted”) or answer a question about it, or check if more steps are needed. It can output another action command, such as moving the arm toward the blue cube, when appropriate. Thus behaviorally, the model can be thought of as producing a word/action “dialogue” that converges on completing the task. Technically, the continuous action head uses flow matching: during training, the true action is perturbed with noise, and the model learns to predict the unperturbed (clean) action vector. This allows EO-1 to produce precise continuous outputs despite the transformer’s natural discreteness. The discrete head, meanwhile, is just a normal language output head, letting the model generate sentences or labels. Both heads share the same transformer layers, ensuring the model’s reasoning can flexibly condition on and influence the action stream. From a user’s perspective, inference with EO-1 can be very straightforward. HuggingFace code examples show that one can simply load the EO-1 model into a pipeline and feed it an observation dictionary. For example, giving it an image and a text task yields text answers or action tensors right out of the model (with about 6–7GB GPU memory needed). The EO-1 team provides code on GitHub and HuggingFace, making it possible to experiment with the model on different robots or tasks. This accessibility helped them evaluate EO-1 across many platforms (discussed next). Benchmarking EO-1: Reasoning and Control. To measure EO-1’s capabilities, the authors ran it on a wide suite of benchmarks covering both reasoning tasks and control tasks. On the reasoning side, tests include emerging multimodal QA benchmarks: the newly introduced ERQA (Embodied Reasoning QA by DeepMind, 2025), RoboVQA (robotics visual question answering), and their own EO-Bench (a set of novel tasks). On the control side, they used established multi-step manipulation benchmarks like LIBERO (lifelong robot skill transfer), SimplerEnv, and real-world physical tasks on various robot arms. Across the board, EO-1 outperformed previous open models. For example, on RoboVQA, EO-1 scored 58.5 BLEU-4 (generating answers), significantly above 47.2 by GPT-4o under the same test. On EO-Bench, it achieved an overall score of 44.8. For ERQA, although specific numbers aren’t listed here, the EO team reports that EO-1 “surpasses existing open-source models” on that benchmark. In embodied QA tasks, where a model must answer questions about robot scenes, this shows EO-1’s integrated vision-language understanding is especially strong. In robot control, EO-1 also set new highs. On the LIBERO benchmark, which measures success across a suite of pick-place tasks, EO-1 reached a 98.2% average success rate. For context, recent VLA models scored in the mid-90s (e.g. 94.2% for the π₀ model and 97.1% for OpenVLA). On SimplerEnv, a benchmark with various simulated dexterous tasks, EO-1 beats π₀ by large margins (exact numbers not given, but noted by the authors). Perhaps most impressively, EO-1 was tested on 28 real-world robot tasks spanning four different robot platforms (the Franka Panda arm, WidowX 250, AgiBot G-1, and Lerobot SO100). Over these, it achieved an overall 86.0% completion score. This was significantly better than prior baselines on the same tasks (for example, the authors mention outperforming π₀-Fast models). In addition to raw success rates, EO-1 showed strong generalization. When asked to handle variations in instructions or new object configurations it hadn’t seen before, the model still scored well. The paper highlights a 79.0% success rate on language variations and 81.0% on long-horizon dexterity tasks. These figures suggest EO-1 isn’t just memorizing; it can reason about instructions and adapt its actions accordingly. The authors attribute this to the model’s interleaved training, which forces it to learn causal relationships between texts, visuals, and motions, much like humans do when they talk themselves through a task. In summary, EO-1 achieved state-of-the-art performance on multiple fronts: detailed question-answering about physical scenes, real-world manipulation benchmarks, and out-of-distribution generalization. Across these tests it consistently “surpasses existing open-source models”. Notably, it does so with a relatively moderate size (3B parameters) by modern standards, underscoring the power of its architecture and data over just brute model scale. In many cases, EO-1 even beat larger models on reasoning benchmarks, illustrating that the training strategy and multimodal design were critical. Discussion: Why It Matters and What’s Next. EO-1 represents a significant step toward truly generalist robots. By showing that one unified model can handle perception, reasoning, and control, it suggests a shift away from hand-engineering each part of the robotics stack. In practice, this could simplify developing new robot behaviors: instead of designing a separate policy for each task, we might simply “prompt” the model with a new instruction and let its learned experience carry it out. There are, of course, limitations and open questions. The current model was trained on curated data and benchmarks aimed at everyday manipulation tasks. It could struggle with time-critical control (fast dynamics), very high-dimensional sensory streams (like raw video at high frame rates), or tasks requiring long planning under uncertainty. Also, as an autoregressive model, inference can be slower and less reactive than a hand-tuned controller. The authors acknowledge that tuning (like turning off the built-in safety “assistant” dialogue mode) may be needed to maximize real-time performance. But the core takeaway is powerful: a single Transformer can serve as both “brain” and partial “body” of a robot. On the research side, EO-1 opens many avenues. One could imagine combining it with real-time control loops (e.g. model predictive control on top of predicted velocities), or scaling up its interactive ability with richer sensors (touch, proprioception). The “foundation model” framework also invites continual learning: future work could fine-tune EO-1 on new robot embodiments or tasks with minimal data, leveraging its broad prior. The team mentions plans to release the pretraining checkpoints, the dataset, and an EO-Bench environment, which would let the community further probe and improve the model. Finally, EO-1 highlights a broader trend: language models are moving out of the keyboard and into the physical world. Just as GPT-4 unified vision and language under text-space, EO-1 unifies language and action under the same neural roof. The hope is that as these models mature, they will make it far easier to program robots: you could simply tell a robot what to do in natural language and let the model handle perception, planning, and control seamlessly. In human terms, it’s like handing over all our knowledge of reasoning and motor skill to an AI teammate. In conclusion, EmbodiedOneVision (EO-1) is an ambitious unified model that lives up to its name. It embodies vision, text, and action in one Transformer, showing that with the right training data and architecture, a single neural network can do everything from answering questions about a scene to actually moving a robot arm. The success of EO-1 (3B parameters, trained on 1.5M multimodal samples) suggests that unified architectures are a promising direction for general-purpose embodied AI. For researchers and roboticists, EO-1 offers a compelling new tool: an open-source, one-stop model for perception, reasoning, and control. Key Takeaways: EO-1 uses a single decoder transformers to process images, text, states, and actions together. It combines autoregressive next-token training with continuous flow-based action training. A new large dataset (EO-Data1.5M) provides interleaved vision-language-action examples for training. In experiments, EO-1 exceeds previous models on both reasoning benchmarks (e.g. RoboVQA, ERQA) and robot control benchmarks (LIBERO, Simpler, real tasks). This integrated approach suggests a future where one foundation model can serve as the core of embodied intelligence.