Latest / Embodied AI 101 / EmbodiedOneVision (EO-1): A Unified Decoder-Only Transformer for General Robot Control

Transcript

EmbodiedOneVision (EO-1): A Unified Decoder-Only Transformer for General Robot Control · Embodied AI 101

• EmbodiedOneVision (EO-1): A Unified Decoder-Only Transformer for General Robot Control. The quest for truly general-purpose robot intelligence – a system that sees, thinks, and acts seamlessly – has long driven advances in embodied AI. Traditional robot policies often chain together distinct modules for vision, language, planning, and control. For example, some systems use a vision encoder to process camera images, a language model to interpret instructions, a separate planning algorithm to break down tasks, and finally a low-level controller to execute motor commands. But such “pipeline” approaches can be brittle: information does not flow fluidly between modules, and the system lacks holistic situational awareness. A human, by contrast, can look at a scene, read an instruction, reason about multi-step plans, and physically manipulate objects all within one mind. Building an artificial analog of this human-like “vision-text-action” integration is therefore a major goal of embodied intelligence research. The new EO-1 model (also called EmbodiedOneVision) tackles this problem with a bold design. EO-1 is a single, unified transformer policy – a decoder-only network – that processes images, text, and even state/action data all together. It is trained end-to-end so that perception, reasoning, and control emerge within one model. In practice, EO-1 was built by Shanghai AI Lab and AgiBot: a 3-billion-parameter decoder-only transformer preinitialized from a vision-language foundation (Qwen2.5-VL) and fine-tuned on a massive interleaved dataset for embodied tasks. Its training combines two objectives: standard autoregressive token prediction for text, and a continuous “flow-matching” denoising objective for action outputs. In other words, EO-1 treats robot actions as continuous signals to be generated through a diffusion-style process, while treating language and vision tokens with usual next-token prediction. Crucially, both modalities share the same transformer weights. The result is a model that can look at images, articulate goals in words, and immediately produce motor action sequences – all from one neural stack. EO-1 is not purely theoretical. The authors publicly released the full model, code, and dataset under open-source licenses. On their demo site and in technical reports, they demonstrate that this single transformer can outperform previous open-­source VLA (vision-language-action) policies on a wide range of tasks. Benchmarks for embodied reasoning (like ERQA and the new EO-Bench) and for low-level robotic control (LIBERO, SimplerEnv) all see EO-1 once again setting new state-of-the-art marks among open models. Real-robot evaluations – on Franka, AgiBot, WidowX, etc. – likewise show robust generalization to novel objects, lighting, and instructions, with completion rates far above prior models. The story so far suggests that a single model can indeed learn an embodied “multimodal brain” that ties perception, planning, reasoning, and action into one unit. In this article we dive deep into EO-1. We’ll unpack its architecture, explore the novel interleaved pretraining scheme and dataset that powered it, and survey the experimental results. Throughout, we reference the authors’ materials and related literature to explain how EO-1 works and why it matters. Architecture: A Unified Decoder for Vision, Language, and Action. At the heart of EO-1 is a decoder-only transformer network reminiscent of a large language model – but extended to handle images and “tokens” for robot states and actions. The underlying weights are inherited from Qwen2.5-VL (the vision-language 3B model), so EO-1 starts with a powerful capability for multi-view image understanding and language understanding. On top of that backbone, EO-1 repurposes some of its capacity for action generation. Concretely, the EO-1 policy is set up within the open-source LeRobot training/evaluation framework as follows. Each training example is treated like a multimodal “conversation” or sequence. First, one or more camera images are encoded (via Qwen2.5-VL’s vision encoder) and fed into the transformer as visual tokens. Along with the images, the current robot state (e.g. joint angles or end-effector pose) and the task instruction are also fed in as tokens. Finally, the network is asked to output a sequence of future actions for the robot, represented in continuous or discrete form. In practice EO-1 predicts a short chunk of future actions at once (for example, an 8-step sequence of joint velocities). Two learning objectives run in parallel on such sequences. For the language and vision portion, EO-1 simply uses autoregressive next-token prediction (like a traditional language model). This means it learns to predict text descriptions or question-answers in a chain, and also implicitly predicts the masked segments of conversations that comprise images and state readings. In contrast, for the action outputs the model uses a continuous diffusion-like method known as flow matching. Essentially, random noise is added to the target action values, and the transformer learns to denoise from that noise. The “flow-matching head” predicts how to transform the noisy actions back toward the ground-truth controls. This combination – supervising text tokens with cross-entropy and supervising actions with flow denoising – allows the model to handle both discrete and continuous data through the same weights. This shared-weights approach means EO-1 truly has a single set of parameters for vision, language, and action. For example, object recognition ability learned in the vision part can directly influence how actions are generated. Similarly, complex reasoning gleaned from language instructions can guide the sequence of actions the robot should take. As the authors note, “when processing visual information. object recognition ability can directly guide action generation; [and] causal reasoning learned in language understanding helps predict the plausibility of action sequences”. This deep fusion is enabled because all modalities flow through Qwen2.5-VL’s transformer. During inference, EO-1 generates actions in chunks: it samples a predicted action sequence, executes some initial steps (e.g. 8 control steps), sees the new sensor feedback, and then samples the next chunk. Throughout, the model never splits perception, planning, or control into separate components – it is one autoregressive process (for text and images) interwoven with a flow-based generation for actions. The EO-1 authors highlight this unified architecture as a key pillar of their design. A single decoder-only transformer processes “multimodal inputs indiscriminately (image, text, video, and action)”, rather than using separate vision encoders versus control decoders. The image and language tokens get processed exactly as text tokens would (via the same queries/keys/values). Then a special continuous flow-matching head is attached to certain output positions to emit motor commands. In practice, after feeding in one context (e.g. “image of table + instruction: ‘pick up the red block’ + current robot state”), the model’s next output tokens represent a short vector of target controls for the end-effector or joints, which it has learned to produce via diffusion training. In sum, the network has one brain – not a separate “eyes”, “brain”, and “hands” – and can reason and then immediately act in a moment-by-moment loop. Interleaved Vision–Text–Action Pretraining. A novel part of EO-1 is how it was trained. Drawing inspiration from how humans interleave thinking and acting, the team developed an interleaved pretraining approach. Instead of processing all visual and textual context first and then outputting a final action, EO-1 is trained on sequences where images, language, and actions are interwoven in time. The core idea is that during pretraining, EO-1 sees data streams that mix perception, question-answering, and control steps in one sequence. For example, one training snippet might consist of: (image of workspace) – “Which cube should be picked first?” – (image of robot hovering) – “Answer: The green cube” – (image) – “Action: Move 20cm right” – (image of new state) – “Success? Yes” – etc. This interleaving means the model learns the “plan-execute-verify” loop: it must reason about why an action is taken, then see the outcome, then ask or answer questions based on that outcome, and so on. In practice, the authors describe at least three interleaving formats:. Temporal planning loops: These emulate a plan-execute-verify cycle. The model is trained on sequences like, “Before you grasp the object, ask why it should be grasped?” Then after the model reads an answer, it executes the grasp action, then asks “did it succeed?”. This forces the model to learn a full logical chain – from the question, to the goal, to execution, to result verification. Spatial reasoning interleaving: Here the model alternates between moving and explaining why. For example, to avoid an obstacle, the data might say “Explain why moving left avoids the obstacle.” The model learns not just to move, but to articulate the spatial rationale: “Need to go around, so first shift 20cm left”. Free-form dialog interleaving: Some sequences are open conversational Q&A that run in parallel with controls. If the robot encounters an unexpected situation (e.g. a blocked view), the data injects a side-question like “What now?” The model must generate an answer (e.g. “remove the obstacle”), then proceed with the corresponding action. This teaches reactive reasoning. Each of these styles shows up in the dataset through a “re-weighted or alternate sampling strategy” built into the data pipeline. In essence, instead of giving the model a single fixed trajectory, the training pipeline randomly interleaves language queries and answers into the middle of robot control episodes. The result is that EO-1 sees examples of mixed reasoning and action during learning, not just (observation, action) pairs. As one diagram explains, the model might get an observation-frame, then a sequence of “image→language→action→image→question→answer→action→…” all in one go. This “interleaved” formatting encourages the model to build causal links across modalities – for instance, realizing that the answer “because it’s red” should alter the action of picking the red object. To manage this interleaving, the data processing uses a special schema where each conversation includes a view of images and a set of actions. The continuous actions are noisified for the flow-matching objective, but the network still sees clean trajectories as context for subsequent text tokens. In practice the authors solved a tricky “causality dilemma”: the actions are noised for training, but the follow-on images and questions in a sequence should treat the denoised result as true. The solution was to structure each example so that any linguistic or visual context after an action is based on the true final state, while the action itself is learned via denoising. This dual-objective training on exactly this data was key to fusing reasoning with control. Put simply, EO-1 learns to answer “why” questions about what it just did, and to perform “what next” queries based on what it saw, all as one continuous prediction problem. Training also includes standard web-scale multimodal data and curated robot logs alongside the interleaved episodes. In fact, the full pretraining corpus is stratified into three parts：. Web multimodal data (public image/text/video corpora) for general grounding and language. Robot control data (raw action sequences and state logs from expert demonstrations or prior policies). Interleaved embodied data (the new 1.5M dataset described next) that ties reasoning with control. By pretraining on all three, EO-1 gains both broad knowledge and specific embodied skills. In sum, the “interleaved training” is the novel strategy here: the transformer sees a blended sequence of vision, language, and noisy action targets and learns them together. As the authors argue, this lets a single policy seamlessly incorporate both high-level planning and low-level motor knowledge. EO-Data1.5M: Building a Billion-Token Embodied Dataset. The ambitious training scheme above relies on a massive dataset of annotated embodied sequences. To this end, the authors constructed EO-Data1.5M, a custom dataset of over 1.5 million samples designed for embodied reasoning. It is one of the largest publicly released datasets of its kind, and it is fully open for research. EO-Data1.5M is organized to unify varied data sources. Broadly, it contains:. Physical common-sense QA: These are vision-language QA examples about physical environments. For instance, questions like “Can the cup pass under the table?” or “Why will the block fall if the support is removed?” Each example has an image (or video frame) and a question about objects/spatial relations. The model must output a text answer. This trains general reasoning about physics and 3D space. Task planning QA: These focus on multi-step manipulations. For example, “What is the correct order to make a sandwich?” accompanied by images of ingredients. Or “How should the robot arrange all dishes from one side of the table to the other?” These teach the model high-level planning and sequential logic in embodied contexts. Spatio-temporal reasoning tasks: These combine trajectory predictions with questions. For instance, an image of two chairs might lead to, “Draw a path that avoids the chairs,” then the model predicts a geometry or action sequence. The idea is to anchor planning in space. Control commands: These are actual robot action sequences (as produced by simulation frameworks like LeRobot) for tasks like pick-and-place, drawer opening, object sorting, etc. They often accompany questions or instructions. For example, an entry might show a “place red block on green block” instruction, an image of the setup, and the action numbers needed to execute that. This portion grounds the model in real control data. Interleaved samples: Many datapoints are true interleavings: a snippet like “Ask 'why pick this up?', answer, then 5 robot control steps, then 'was it correct?', and so on.” These multi-step dialogues and controls were synthesized or curated to explicitly link reasoning and action. In practice, the dataset was assembled via a "scalable data construction pipeline" using the open-source LeRobot environment. LeRobot provides simulators for many robots and tasks; the authors used it to generate action logs. They also scraped or fetched vision-and-text data for the QA parts. All samples are stored in a standardized format: each sample includes an easy conversation transcript (vision frames, instruction text, QA, etc.) plus a parallel record of robot states and actions. Specifically, each entry ties and to via a common field. This means a model like ours can attend seamlessly from a language question token to the corresponding robot joint token. The upshot is a unified dataset that “comprises 1.5 million high-quality interleaved samples covering physical interactions, reasoning tasks, spatial understanding, and control commands”. It is nearly 6 GB in compressed form, and – just like the model – publicly available under a permissive license. By pretraining on EO-Data1.5M alongside other data, EO-1 ingests a vast range of embodied situations. The use of this data for training is a major innovation: it explicitly teaches the model how images, words, and actions causally relate. Training Recipe and Implementation. EO-1’s training proceeds in phases. Initially, the Qwen2.5-VL backbone is loaded (no extra computer vision or language pretraining is needed from scratch). Then the new flow-matching head for actions is grafted on. The team mixes together mini-batches of the different data categories: some batches from web image/text corpora, some from robot-only logs, and many from the EO-Data1.5M interleaved subset. Within each interleaved sample, the input tokens are chained as described so that some token positions correspond to images or text (trained with softmax/CE loss) and others correspond to actions (trained with flow loss). The transformer parameters are updated end-to-end to handle both tasks simultaneously. In the flow-matching objective, each “action chunk” is predicted to cover a fixed number of timesteps (8 in practice), and noise is gradually denoised over 10 steps when sampling. The model is trained for hundreds of thousands of steps on large GPU clusters (the authors mention using up to 3 GPUs with mixed precision). Significantly, the EO-1 release includes full training scripts and instructions. The LeRobot documentation details how to configure and train the model. For instance, one sets and points the policy to the Qwen2.5-VL base (the exact 3B checkpoint). Then, one supplies LeRobot-format datasets (which can include EO-Data1.5M or other robot datasets) for. The model uses standard transformer optimizers, and transformers libraries under the hood. A key trick: because the action head uses diffusion, the authors keep that part in 32-bit precision even as the backbone runs in 16-bit. This stable setup can be run on consumer GPUs (the policy uses about 6.5 GB of memory at inference, per SourcePulse) but training was done on powerful hardware like H100s with optimized libraries (e.g. FlashAttention for speed). Because EO-1 is fully open-source, one can reproduce it via the public GitHub (EO-Robotics/EO-1) and dataset repo (IPEC-Community/EO-Data1.5M). The repositories even include example config files and sample code for loading the model in PyTorch. Students and researchers can therefore train from scratch (compute permitting) or fine-tune the provided checkpoint on their own data. This democratization is part of a trend: the authors explicitly hope that “EO-1 will serve as a solid foundation for developing general-purpose autonomous robots”. As a result, the project has already garnered interest and stars on GitHub (over 290 ⭐) and been made easily usable through Hugging Face interfaces and the LeRobot ecosystem. Benchmarking Embodied Reasoning. How well does EO-1 understand a task scenario? To probe this, the authors tested it on various embodied reasoning benchmarks, where a vision-language model must answer questions about a scene or planning problem. One such test is RoboVQA, a long-horizon visual question-answering task. On this (not yet widely publicized) benchmark, EO-1 achieved a BLEU-4 score of 58.5, far above closed-source baselines (OpenAI’s GPT-4o scored 47.2, Google’s Gemini 1.5 Flash scored 46.0). This shows that EO-1 can not only recognize objects but also reason about multi-step tasks and spatial relations (e.g. “the robot should first clean up its workspace by. because.”). Another demanding benchmark is ERQA (Embodied Reasoning Question Answering), released by Google DeepMind in early 2025. ERQA challenges models to answer true/false or multiple-choice queries involving spatial and physical reasoning in robot scenes. On ERQA, EO-1 reached 45.5% accuracy, slightly beating InternVL2.5-8B (45.2%) and significantly outperforming Qwen2.5-VL-7B (39.3%). (For context, these accuracies are tough; most vision-language models hover below 50% on ERQA.) In particular, EO-1 showed a deep understanding of 3D spatial relations – for example, correctly determining whether one object could physically fit through a maze of obstacles. This suggests that the joint flow-based training helped EO-1 internalize physics and geometry rather than just map pixels to words. The authors also built their own EO-Bench for evaluation-focused reasoning. EO-Bench includes tasks like Multi-View Commonsense Inference (looking at a scene from different angles), Trajectory Prediction (predicting where a moving object will go), Visual Localization, and Temporal Planning (assembling a plan and verifying success). On these, EO-1 achieved average scores around 36–39 (on a 0–100 scale) whereas other leading VLMs averaged only the low 30s. For instance, on multi-view spatial puzzles, EO-1 scored 36.4 vs ~32 for others, and on step-by-step planning questions it scored 38.9. While these numbers need context, the key is that EO-1 consistently outperformed standard vision-language models, demonstrating its specialized embodied awareness. In summary, EO-1’s embodied reasoning is superior to prior open and even some closed models. By training vision and language together with action, the model has built a richer world model. As one commentator puts it, EO-1 “seamlessly integrates perception, planning, and reasoning” and its comprehension depth “exceeds that of typical VLMs”. This is foundational: a robot system that “understands” its tasks will in principle make fewer inexplicable mistakes and adapt better to novel instructions. Robot Control Benchmarks. Perhaps even more striking are EO-1’s gains in actual robotic control tasks. The authors evaluated the model on several standard benchmarks for vision-guided manipulation, comparing against the best open-source baselines. First, LIBERO (from the Lifelong Robot Learning benchmark) was used. LIBERO consists of manipulation tasks divided into subsets: Spatial (generalize to new object positions), Object (generalize to new object categories), and Long (long-horizon tasks requiring >15 steps). Here EO-1 was essentially unbeatable. On LIBERO-Spatial, it achieved 99.7% success rate, compared to 96.8% for the strongest prior model (referred to as “O1” in the report). On LIBERO-Object, it got 99.8%. Even on LIBERO-Long (path lengths >15), where many models crumble, EO-1 still achieved 94.8% success, whereas a prior policy (a fast-token version of the baseline) only managed 60.2%. Averaged overall, EO-1’s LIBERO success was 98.2%, a new state-of-the-art. In plain terms, EO-1 can follow complex language instructions in cluttered environments, generalizing to new layouts and objects with very high reliability. Second, the team used SimplerEnv, a Google-maintained simulated benchmark that wraps standard real-robot tasks (e.g. on WidowX arms) into controlled tests. SimplerEnv has two variants: Visual Matching (VM) where the robot sees objects in the same position but new visuals, and Visual Aggregate (VA) which adds even more visual variety. On SimplerEnv’s WidowX tasks, EO-1 averaged 72.7% success over the suite of tasks, outperforming the baseline (~69.2%) and every other open model. The improvement was especially dramatic on one task, “Stack Blocks”, where EO-1 succeeded 81.8% of the time versus only 52.5% for the best prior model. This suggests EO-1’s robustness to visual changes: under VA conditions (more distractors and lighting shifts), it still managed 63.0% success, showing strong visual generalization. On Google's in-sim "Google Robot" tasks (another SimplerEnv set), EO-1 similarly excelled. In the VM (Visual Matching) category it averaged 76.5% success; notably, on the hardest subtask “Drawer Apple” it got 52.8% vs only 46.6% for the baseline. Even on the scarier Visual Aggregate setting (lots of clutter), it held 63.0%. These tasks are known to be challenging: they involve fine-grained reasoning about how to insert objects or open drawers. EO-1’s lead over prior methods was clearly bigger here than in simpler cases. Finally, the team conducted real robot tests on 28 physical tasks spanning multiple platforms (Franka arm pick-place, WidowX button pressing, AgiBot advanced manipulation, etc.). In this suite, EO-1 achieved an overall 86% completion rate. By contrast the prior models completed only 68–71% of tasks (and one model only 43%). The real-world gap is significant: EO-1 succeeded on far more tasks out of 10 trials. Notably, these included long tasks like making a breakfast sandwich or organizing groceries (described below) where long-term consistency and adaptability are required. In short, across all these robot-control benchmarks, EO-1 sets a new bar. It shows that the theoretical benefits of unification carry over to concrete performance: the model not only “understands” scenes better, it actually produces correct actions more often. The combined gains on LIBERO and SimplerEnv translate to robots that can follow instructions and manipulate objects with unprecedented reliability for an open-source system. (For context, many prior open-source policies like VIMA, RT-1, or GGWP achieved in the 50–80% range on these tasks. A 95–99% success is basically closing in on a human-level policy in simulation. Of course, a few failures still happen, but EO-1 raised the floor dramatically no doubt.). Real-World Long-Horizon Tasks. Beyond benchmarks, the EO-1 team tested the model on long-horizon real tasks requiring dexterous, multi-step manipulation. Four challenging tasks were examined on actual robots:. Make a Breakfast Sandwich: The robot must assemble a sandwich from bread, cheese, and ham, involving picking, placing, unfolding, etc. Roast Beef Steak: The robot has to process a raw meat (on a board) through multiple steps (spread seasoning, place in oven, etc). Fold Clothes: Given a pile of cloth, fold it neatly. Sort Grocery Items: Sort various mixed items into categories. Each task demands planning across 5–10 substeps, often with delicate grasps or bimanual moves. On all of these, EO-1 performed consistently: for example, whether it was stacking bread and cheese or neatly folding a towel, the model “showed stable and strong long-horizon dexterity”. In trials, EO-1 successfully completed each task in the majority of attempts. These results highlight that the model’s competence isn’t limited to toy problems – it can handle realistic interactive tasks in real environments, relying on its integrated reasoning to sequence multi-step plans correctly. Relatedly, the team assessed open-world generalization with changing conditions. The robot was given instructions with synonyms or more detail, objects were moved slightly from expected positions, lights were altered, and backgrounds were changed. Remarkably, EO-1 “demonstrated stable instruction following and open-world generalization capabilities”. In one example, telling the robot “scoop up the lettuce” versus “pick the leaf” did not confuse it. Shifting the cup 10 cm to the side still let it pour correctly, because the model actually recognized the cup’s new location from vision (not just fixed coordinates). This robustness likely stems from the vision-language backbone’s strong grounding: EO-1 truly sees with Qwen2.5-VL and thus adapts to new visuals. The unified model also unlocks pure reasoning tasks that blend perception with planning. In a Visual Rearrangement task, EO-1 was asked to physically rearrange objects into a specified pattern without direct programming (for instance, organizing colored cubes into a tic-tac-toe formation). And in a Tic-Tac-Toe game, the robot must place pegs on a board to win or block the opponent. These tasks require simultaneous spatial reasoning, planning, and fine control. EO-1 “seamlessly integrates high-level reasoning with low-level control”, achieving correct play and assembly even under real dynamics. In short, the robot wasn’t just randomly moving pieces, it understood the game logic or target pattern and executed plans to achieve them, much like a human would. These real-world evaluations reinforce a main point: EO-1’s unification of modalities lets it handle truly open-ended embodied tasks. It doesn’t reduce everything to a fixed pipeline like “parse language → generate plan → control program”. Instead, it learns to form its own intertwined strategy. When faced with, say, folding clothes, the vision and language pieces concurrently inform its motion primitives. When rearranging objects, its model of space (gained via training and vision) guides each grasp. In practice, the reported success rates (86% over 28 tasks) are quite high for such open situations, and far above what typical single-task controllers achieve. Open Source and Accessibility. An important aspect of EO-1 is its “full openness”. The team released the model weights, the training code, and the entire 1.5M-sample dataset to the community. They published the GitHub repo (EO-Robotics/EO-1) under an Apache 2.0 license, and the data under MIT, ensuring anyone can use or build upon this foundation. This contrasts with many industrial or commercial embodied agents, which often remain proprietary. By open-sourcing everything, EO-1 lowers the barrier for labs to experiment with an advanced multimodal policy. For example, anyone can now clone the repository, follow the instructions to install LeRobot and dependencies, and immediately use the pretrained EO-1 checkpoint for their own tasks. Indeed, the model has already been integrated into Hugging Face’s datasets and model hubs. Practically, using EO-1 is straightforward. One specifies in a LeRobot config, points to the Qwen2.5-VL base checkpoint (which is also publicly available), and supplies any LeRobot-formatted dataset (even custom real data). The policy automatically tokenizes images, states, and text, then loads or fine-tunes the transformer. People can fine-tune EO-1 on new robot setups or tasks, or even add new languages or camera angles, thanks to this unified interface. In short, the open release means future embodied-AI research can start from a powerful pretrained core instead of retraining from scratch. The docs and press highlight this point: EO-1 is intended as a “foundation model” – a general-purpose starting point for robots. By analogy, GPT-like models transformed NLP by being open-basis engines for downstream tasks. EO-1 aspires to do the same for physical tasks. That said, the authors also see room for growth: they mention EO-1 as the first in an “EO-Series” of embodied models. Researchers might extend it to larger scales, add new modalities (e.g. audio), or refine the action predictors. But as it stands, EO-1 provides a concrete foundation that others can build on. Discussion and Outlook. EO-1’s success suggests several important takeaways for embodied AI. First, a single multimodal architecture can indeed learn both perception and control, provided it is trained on appropriate data. The results challenge the old assumption that robotics must use separate perception or planning modules. With enough data and the right learning objectives, the transformer learns to embed even low-level continuous control in its predictions. Second, the temporal fusion of training data appears crucial. Earlier vision-language-action work often took a more serial approach: feed in an instruction and image, then output one action at a time (or plan via some translator). EO-1’s interleaving is more dynamic and human-like. By “asking questions” in the data, the model was encouraged to learn cause-effect between steps. It is plausible this is what gave EO-1 better generalization: it doesn’t just know “this is a red block, so do x”, it has seen examples of “why red matters” (and similar Q&As while acting) during training. Third, the combination of autoregressive language modeling and flow-based action prediction is elegant. The field had explored diffusion and flow for images and text, and separately worked on transformer for discrete actions (e.g. GPT for robotics), but here they are joined. Technically, this was enabled by Qwen2.5’s decoder-only form (which can accept extra heads) and by recent advances in continuous generative modeling (the flow-matching literature). The result shows future embodied models can freely mix discrete symbols and continuous values. In principle, one could extend this further – e.g. have Gaussian diffusion for position and autoregression for other sensors – or explore alternatives like equivariant networks. But EO-1’s recipe provides a concrete blueprint. Finally, EO-1 shows that training data quality matters enormously. The authors invested in a 1.5M-sample dataset with clever interleaving. It dwarfs many robot learning datasets and emphasizes reasoning connections. The question for the community is whether other teams can replicate or surpass these results. For instance, could we build an open multi-agent foundation model by merging EO-Dataset with language corpora? How about adding unstructured video of humans performing tasks? These are exciting avenues. It’s also worth noting where EO-1 might have limits. Although 3B parameters is large for a robot policy, it’s small compared to 100B+ LLMs. The model’s skill ceiling is likely bounded by the amount of data and compute used. Moreover, EO-1 currently works in simulation tasks or controlled real environments; fully open-ended real-world integration (with all messy sensor noise) may still pose challenges. The interleaved pretraining is sophisticated, but turbulence in raw sensor input (like slippage, physics surprises) might not be fully captured by the data. Lastly, as a single model, it may not be trivially interpretable – debugging why it failed on a task could be hard. These are general concerns for large unified models. Nonetheless, EO-1 marks a significant advance. It matches or exceeds other models on every benchmark the authors tried, and it does so without any proprietary data or hidden training tricks. In effect, the model demonstrates that the vision-language revolution in AI can extend into the realm of physical action. For practitioners, the takeaway is that building a “multimodal brain” for robots is now feasible using transformer architectures and large-scale pretraining. Given that EO-1 is open-sourced, readers can directly experiment with it. You might try fine-tuning EO-1 on a new robot arm, or integrating it with more sensors. On the research side, this work invites comparisons: How does EO-1 stack up against other emerging embodied foundations such as Elon Musk’s xAI efforts (e.g. unicorn models), Nvidia’s Gigatron, Microsoft’s VMT+, or Meta’s Perceiver variants? The code and data let us test these interactions. Moreover, one can analyze EO-1’s structure: which attention layers attend to vision vs action? Does the model use multi-step planning or is it strictly myopic? The unified nature of EO-1 will surely inspire papers and tools to dissect exactly why it succeeds. In closing, EO-1 – EmbodiedOneVision – shows that a finely tuned transformer can indeed serve as a general robot controller. It breaks from the tradition of separate “language vs. vision vs. control” modules and demonstrates that, with large data and careful design, a single model can do all three. For the field of embodied AI, this is a compelling result: the dream of a unified robot “multimodal brain” has taken a concrete step toward reality, with fully open research resources to back it up. References: Core information on EO-1 is drawn from the authors’ documentation and publications. See also the EO-1 GitHub and dataset page for implementation details.