Latest / Embodied AI 101 / MolmoAct2: An Open Foundation Model for Real-World Robotics
Transcript
- MolmoAct2: An Open Foundation Model for Real-World Robotics. Robotics has long yearned for “foundation models” – general-purpose controllers that can translate high-level commands into real-world actions across many tasks and embodiments. In recent years, Vision-Language-Action (VLA) models have emerged as a promising path. These models take in visual observations and language instructions and aim to output robot controls, unifying perception, reasoning, and actuation in one giant transformer-style net. The idea is akin to giving a robot a “large model” brain wired to both cameras and motors. In principle this could let us ditch per-task training and scale up to open-ended capabilities. In practice, however, most VLA systems today fall frustratingly short of that goal. Closed commercial models like RT-2 or big in-house efforts (e.g. Google’s Gemini-Robotics) are often opaque and tied to specific hardware. Open research models have shown progress, but still struggle with generalization, speed, or ease of use. As one survey notes, robotics lags behind language and vision largely because robotic tasks demand fine-grained, embodied interaction data – the kind that is hard to collect at scale and hard to label consistently. Against this backdrop, MolmoAct2 (from the Allen Institute for AI, May 2026) stakes out a clear vision: build a fully open robotics foundation model that can actually run on real robots in real time, and let the community tinker with every piece. In the authors’ words, MolmoAct2 is “a fully open action reasoning model built for practical deployment”. Unlike many recent papers that release just a closed demo or a handful of weights, the MolmoAct2 team has opened up everything: code, model weights, training pipelines, and massive new datasets. They also introduce several architectural and data advances specifically designed to make a VLA model faster, more reliable, and capable of explicit spatial reasoning. In short, MolmoAct2 refines the idea of an “action reasoning model” (MolmoAct1 from 2025) and turns it into a polished open platform for real-world robot manipulation. In this article we’ll unpack what MolmoAct2 is, why each of its components matters, and how it fits into the growing ecosystem of robotics foundation models. We’ll see that MolmoAct2 stands out by its emphasis on structured reasoning (predicting 3D depth and visual plans before actions), its new multi-arm data, an open tokenizer for actions, and an adaptive geometric reasoning trick to cut latency. We’ll also survey its performance: across simulations and zero-shot trials on physical robots, MolmoAct2 markedly outperforms prior open models. Most importantly, its fully open release means any lab can take it, drop it on their robot (it’s even integrated with Hugging Face’s LeRobot toolkit), and continue improving it. From VLA to Action Reasoning: The MolmoAct Lineage. To appreciate MolmoAct2, it helps to recall its predecessor, MolmoAct (2025). MolmoAct introduced the idea of an Action Reasoning Model (ARM): rather than mapping instructions straight to low-level motor commands, the model first reasoned about space. In particular, MolmoAct was designed as a three-stage autoregressive pipeline. Given a camera view and a language goal, the model first outputs a series of depth-perception tokens that essentially reconstruct the scene’s 3D geometry in a latent form. Next it produces visual reasoning trace tokens that represent a 2D planned trajectory through the scene. Finally it predicts action tokens for the robot’s control commands. Each token stage can be decoded on its own: you can render the depth map from the first stage, overlay the planned 2D trace from the second, and then convert the third into executable motor moves. This gives the model a form of explicit spatial chain-of-thought. In other words, MolmoAct doesn’t just blindly spit out torques; it “thinks” by building intermediate visual plans. Even with this structure and surprisingly little data, MolmoAct already demonstrated that structured reasoning pays off. It was trained on only about 10.6K in-house trajectories (around 22 hours of robot video across kitchens, bedrooms, living rooms, etc.) plus some filtered shared data, and yet it could beat much larger closed models on standard manipulation benchmarks. As the team notes, “MolmoAct served to prove that an open, reasoning-based architecture could beat much larger closed models on industry-standard benchmarks”. In other words, even a small open model with depth- and plan-prediction tokens outperformed closed models with billions of parameters. This exciting result validated the ARM idea: adding geometric and planning structure inside a foundation model can boost generalization. However, MolmoAct1 was more of a proof of concept than a deployable system. It needed careful fine-tuning for new tasks, and was quite slow at inference (each action took on the order of seconds). Its 10.6K trajectory dataset, while substantial, was tiny compared to what one might want for generalization. MolmoAct2 is all about filling these gaps. The goal is a real foundation model for robotics — one that works on real hardware “out of the box” — and is fully published so others can improve it. In the words of the authors: “MolmoAct2 is built for practical deployment, advancing its predecessor along five axes.” These axes are: (1) a stronger embodied VLM backbone (“MolmoER”), (2) new datasets (especially a massive bimanual dataset), (3) an open action tokenizer (“OpenFAST”), (4) an improved model architecture that fuses continuous control, and (5) an adaptive depth-reasoning mode (“MolmoThink”) to cut latency. We’ll examine each in turn. MolmoER: An Embodied-Reasoning Vision-Language Backbone. At the heart of MolmoAct2 is MolmoER, a vision-language model backbone refined for spatial reasoning. This builds on Molmo2, a family of video-language models AllenAI introduced in early 2026. Molmo2 was already state-of-the-art among open VLMs (for grounding, counting, etc.), but MolmoAct2 needed more emphasis on embodied tasks. Thus the authors took Molmo2 and further fine-tuned it on roughly 3.3 million “embodied reasoning” examples. These examples included image-based pointing, object detection, abstract spatial questions, multi-image reasoning, and video-based spatial QA. The process is described as “specialize-then-rehearse”: first train on standard vision-language tasks, then refine on this large synthetic/collected corpus tailored to spatial understanding. Why does this matter? The idea is to endow the model with stronger commonsense about objects and space. For a robot, knowing which pixel corresponds to the handle of a mug or how two viewpoints align can be crucial. MolmoER attains this with explicit datasets for pointing/grasping. In fact, the team reports that on a suite of 13 tough “embodied reasoning” benchmarks (studies of multi-frame alignment, pointing, spatial counting, etc.), MolmoER achieved an average score of 63.8 out of 100. This beats recent heavyweights: GPT-5, Google’s Gemini-2.5 Pro, Meta’s Qwen3-VL-8B, and even NVIDIA’s GR-ER 1.5. In their own words, “MolmoER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks”. In short, MolmoER is a cutting-edge open VLM for spatial tasks. Integrating it means MolmoAct2 starts from a strong point in vision-language understanding. In the MolmoAct2 system, MolmoER takes in the current image (and maybe prior state) and tokenizes it, conditioning the action generation. It still predicts depth and plan tokens as before, but these predictions ride on a more capable “brain”. Concretely, MolmoAct2 uses Molmo2-ER up to some layers, then attaches the action modules (see below). But crediting MolmoER: by focusing on embodied scenarios, the backbone of MolmoAct2 is unusually attuned to 3D geometry and sequential reasoning. Scale and Diversity: New Robot Datasets. Alongside the model, MolmoAct2 brings massive new data. The release includes three large datasets designed to cover a spectrum of real robots and tasks. The centerpiece is the MolmoAct2-Bimanual YAM dataset. This is a 720-hour collection of teleoperated demonstrations involving two robot arms working together on tabletop tasks. Think of things like folding towels, placing groceries, charging a phone, bussing a table, etc. According to the team, this is “the largest open-source bimanual manipulation robotics dataset ever published”. In fact, it contains over 30× more robot data than the entire MolmoAct1 training set. The demo breadth is impressive: 700+ hours of coordinated two-arm actions. This breadth is crucial because many useful tasks (e.g. handling a large object) inherently require two hands. Importantly, MolmoAct2 is trained on this data as part of its base mix, so bimanual coordination is baked into the model rather than added by fine-tuning. In the blog they note, “MolmoAct was capable of bimanual via per-task fine-tuning; MolmoAct2 has bimanual capabilities baked into the base model”. But the release isn’t just one big dataset. The authors supplemented Bimanual YAM with a diverse mixture from other robots and sources. This includes the SO-100 and SO-101 datasets (16,000+ demos each collected from low-cost open-source arms), a filtered subset of the DROID Franka data (real Franka robot log trajectories), Google’s BC-Z and FRGT (Fractal) data from the Open X-Embodiment consortium, Bridge WidowX demos, and finally MolmoAct1’s original 10.6K kitchen/bathroom demos. The goal was broad coverage: different arm types, camera heights, control schemes, and styles. They also cleaned up the language annotations. For example, many robotics archives use terse task labels (“move block to cup”), so MolmoAct2 re-annotated all demos with an open vision-language model to increase instruction diversity. This doubled the unique natural language commands in their mix to ~146K. Together these data bring billions of timesteps of training. Importantly, most of this comes from accessible sources: teleoperated logs or community datasets. The authors emphasize that all of it is being released as part of the package: “full code and training data…every dataset used during training” is on public repos. In a field where proprietary robots often lock away data, MolmoAct2 gives the community a treasure trove of real-world manipulations at different scales. An Open Action Tokenizer: OpenFAST. A subtle but crucial piece in many modern VLA systems is action tokenization: how to discretize continuous control space. MolmoAct2 follows the recent trend of using a learned tokenizer (originally FAST from Physical Intelligence) to turn joint trajectories into tokens that the transformer can predict. However, the original FAST tokenizer was trained on private data. To keep MolmoAct2 fully open, the team trained its own variant, OpenFAST. OpenFAST is “an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments”. In practice this means they collected or reused millions of robot motion segments (from the datasets above) and ran an algorithm (similar to FAST) to build a vocabulary of atomic action tokens. Each token might encode, say, a small segment of joint-space motion or gripper command. At inference time, the model generates a sequence of these tokens, which OpenFAST then decodes into actual commanded trajectories. Why does this matter? By releasing OpenFAST, MolmoAct2 ensures anyone can experiment with token-based control. You can look inside the tokenizer, retrain it on new data, or swap it out. The authors explicitly note: “the data used to train [FAST] hasn’t been openly released. We’ve built MolmoAct2-FAST Tokenizer, a fully open-source reimplementation trained on our data, and published it”. This means more transparency and adaptability: if a lab has a different robot or range, they can augment OpenFAST’s vocabulary accordingly. In summary, MolmoAct2’s control pipeline now has an open continuous-to-discrete interface from end to end. Language vision → transformer reasoning → OpenFAST tokens → actual joint commands. All pieces are available for inspection and improvement. Hybrid Architecture: Discrete Tokens Meet Continuous Control. MolmoAct2’s architecture reflects a smart hybrid of discrete reasoning and continuous control. In many VLA models, the output is discrete actions (via token vocabularies) or continuous control (via regression networks). MolmoAct2 simultaneously supports both. Concretely, the authors “graft a flow-matching continuous-action expert onto a discrete-token VLM via per-layer KV-cache conditioning”. In plainer terms: the core MolmoER transformer still works with token representations (for vision, language, etc.). On top of this, at each layer the network also passes through a flow-based continuous-action branch. Flow matching (a type of diffusion model for trajectories) is used to train a module that can sample smooth continuous action sequences conditioned on the same context. These two streams – token and continuous – are tied together by a clever trick: at each transformer layer, the hidden states (the “key-value caches”) are fed into the continuous expert. So the action predictor sees the same context as the token predictor. The result is a single integrated policy: the vision-language layers produce hidden features, which then inform the flow-action expert to output final robot moves. This design lets MolmoAct2 handle raw joint motion more directly. Instead of predicting every joint angle token by token, the flow model can generate the entire N-dimensional action segment at once, which is often smoother and more stable. At the same time, the discrete tokens can still represent planned contact points and mid-level trajectories (via the visual trace tokens). An analogy: imagine the model planning out a path on paper (discrete tokens) while also having a separate smooth steering wheel (continuous flow) for actual drive – and both are computed in lock-step. This hybrid strategy seems effective: according to the authors, it allows “closed-loop manipulation” of higher fidelity. While the paper goes deep into implementation, the takeaway is that MolmoAct2 does not solely treat actions as text tokens. It explicitly models the geometry of robot motion via a probabilistic flow, guided by the transformer’s reasoning states. MolmoThink: Adaptive Depth Reasoning. One of the original insights of MolmoAct-style models is to predict depth maps as an intermediate step. Dense depth estimation helps ground the 3D structure explicitly. But predicting a full depth map every timestep can be slow. MolmoAct2 addresses this with MolmoThink, an adaptive depth mechanism. The idea: only recompute depth where the scene has actually changed. In many manipulation tasks, large portions of the camera view remain static from one step to the next (especially the background or distant objects). MolmoThink checks which regions of the image have dynamic content (e.g., a moved block or a new grasp) and limits the depth-prediction tokens to those patches. In effect, it superimposes incremental depth reasoning on an otherwise static background. This saves computation. The team reports that this selective strategy retains nearly the same geometric grounding power – the model still “knows” the 3D scene where it matters – but at a fraction of the cost. In practice, MolmoAct2-Think achieves about 17% faster inference than brute-force full-depth decoding. In concrete terms, a single action call with MolmoAct2-Think took about 790 ms (on an NVIDIA H100) versus 6,700 ms in the original MolmoAct. Even the base MolmoAct2 (without depth tokens) took around 180 ms per action, so MolmoThink trades a bit of that off (~180 ms → 790 ms) to get the extra accuracy of 3D planning, still far better than the old model. MolmoThink thus embodies a key principle: explicit reasoning, but only as needed. It underlines the MolmoAct philosophy that a model should “think” about 3D space when it will help, rather than blindly always computing heavy updates. The result is a model that can maintain geometric consistency on-the-fly, without grinding to a crawl. Performance: Simulation and Real-World Benchmarks. What does all this yield in practice? The MolmoAct2 paper includes a comprehensive experiment suite – arguably the most extensive for any open VLA model to date. It tests on standard simulations, zero-shot transfers to real robots, and even real-world fine-tuning. Across the board, MolmoAct2 sets new bars for open models. Simulation (MolmoBot and RoboEval): On the authors’ MolmoBot benchmark (a difficult household manipulation sim), MolmoAct2 averages 20.6% success across all tasks. This is roughly double the score of Physical Intelligence’s π0.5 baseline model (10.3%). (MolmoBot is intentionally hard – many baselines score single digits.) Even on RoboEval, a multi-step bimanual simulation benchmark, MolmoAct2 scored 0.443 versus 0.405 for π0.5 (higher is better). In short, MolmoAct2 is substantially more capable than previous open models on these standard suites. Real mania – Franka Zero-Shot: The team also ran zero-shot tests on actual robots. On a Franka single-arm setup, MolmoAct2 handled a range of tasks out-of-the-box, without any task-specific tuning. The results are striking. Over 15 trials of each task, MolmoAct2 succeeded 100% at placing an apple on a plate, 86.7% at inserting a pipette into a rack, 93.3% at putting a small cube into a tape roll, and 93.3% at placing a knife into a box. Even a longer task of moving several objects into a bowl saw 62% success. Aggregating, MolmoAct2 averaged 87.1% success across tasks, compared to just 48.4% for MolmoAct1 and 45.2% for π0.5. In other words, the new model almost doubles the reliable success rate of prior methods on these basic manipulation tasks. Fine-tuning & Adaptation: They also tested how well MolmoAct2 can learn new tasks via modest additional training. On household routines (setting tables, cleaning, folding towels, etc.), the model showed more than adequate performance (especially on towel folding and bowl placement). On the LIBERO lifelong-learning benchmark, after some fine-tuning MolmoAct2 reached 97.2% average success, with the depth-enabled MolmoAct2-Think variant hitting 98.1%. This is roughly 10–11 percentage points above the original MolmoAct1’s performance on the same suite, underscoring that everything from data to model improvements paid off. Third-Party Benchmark: To double-check these claims, the team hired Cortex AI (an independent robotics testing firm) to evaluate MolmoAct2 on multiple bimanual manipulation tasks. In that systematic trial, MolmoAct2 came out on top. It scored an average of 0.51 (out of 1.0) across the tasks, ahead of OpenVLA (0.36) and π0.5 (0.32) and far above other policies. MolmoAct2 had the best task-level record, ranking first in 7 out of 8 tasks. In short, across simulation and reality, MolmoAct2 proved significantly stronger than prior open models. The gains are attributed to every piece of its redesign – the richer MolmoER reasoner, the larger datasets (especially the new bimanual data), the improved architecture, and the depth reasoning. A Fully Open Ecosystem. Crucially, the MolmoAct2 team has made every component available to the community. As they proudly announce, the full codebase and training data are public. The GitHub repository includes the transformer training code, the policy fine-tuning scripts, and even the Docker environment to replicate experiments. The model weights for all checkpoints (base and fine-tuned) are on Hugging Face, under Apache-2.0 licenses. Every dataset mentioned – MolmoAct2-BimanualYAM, the Franka and SO100/101 subsets, etc. – is also published on Hugging Face Datasets. Even the OpenFAST tokenizer code and training recipe are released. For users, this means you can literally clone their GitHub, download the datasets, and reproduce or adapt MolmoAct2 without guesswork. The authors highlight that “we’re opening the full code and training data behind the model: fine-tuning scripts, every dataset used, all evaluation rollouts, and the training recipe for the open source MolmoAct2 tokenizer”. In practice, one can either use the provided fine-tuned checkpoints on common platforms (Franka, WidowX, dual-YAM arms) or load the pre-trained checkpoint and continue training on new tasks or hardware. To ease adoption, they also integrated MolmoAct2 into Hugging Face’s LeRobot platform. LeRobot is a library for embodied AI, and MolmoAct2 is now a plug-in model within it. That means labs already using LeRobot can drop MolmoAct2 into their pipeline with minimal hassle. In short, the release is as frictionless as it gets for robotics software. The authors note over 400K downloads of MolmoAct2 artifacts within weeks of the announcement, a testament to the demand. The philosophy is clear: open everything so the community can experiment, adapt, and improve. This contrasts with many industrial demos or closed-source models where only inference is shown. The MolmoAct2 team even collaborated with I2RT Robotics and Cortex AI, but assures that these partners had no role in model design – emphasizing that the work is fully driven by the researchers and is independent of any specific vendor. Relation to Other VLA Models. MolmoAct2 joins a rapidly growing field of robotic foundation models. It’s worth quickly situating it among recent efforts. For example, OpenVLA (2024) from Berkeley and Stanford was an early open VLA: a 7B-parameter model trained on about 970K real-world demos. OpenVLA outperformed Google’s RT-2-X (55B) by a substantial margin on multi-robot tasks. Similarly, the SmolVLA project (2025) prioritized small, efficient models that run on single GPUs and even CPUs; SmolVLA managed to match the performance of models 10× its size while using only community-collected demos. Another example is ABot-M0 (2026), which constructed a massive “UniACT” dataset (9,500 hours) and introduced ideas like an action manifold projection for smooth trajectories. Each of these contributes pieces to the puzzle: more data, new architectures, or efficiency tricks. MolmoAct2’s niche is its emphasis on explicit spatial reasoning and accessibility. The idea of chaining depth → plan → action has parallels to other structured approaches, but MolmoAct2 wraps this pipeline into a single transformer and pairs it with plug-and-play continuous control. It demonstrates that even relative “mid-sized” models (the VLM is 7–8B scale, plus expert) coupled with smart data and structure can beat heavier models. It also pioneers the norm that all research code and data should be open. In many ways, MolmoAct2 and OpenVLA represent a trend of transparent benchmarks where academic teams openly compare to commercial giants. Implications and Use Cases. For practitioners, what does MolmoAct2 mean? First, it provides a baseline generalist policy that one can test on one’s own robot. Suppose you have a standard manipulator (Franka, WidowX, Yaskawa, etc) with a camera on it. You can fine-tune the released MolmoAct2 model on a handful of demonstrations for your specific setup, or even try zero-shot. The Hugging Face integration and provided fine-tuned policies (e.g. MolmoAct2-DROID or BimanualYAM) give you a starting point. Compared to building a policy from scratch, this could save months of work. Second, researchers can use MolmoAct2 as a research platform. For example, if you want to test a new idea for LLM-to-robot adaptation, you can plug in MolmoAct2’s architecture and data. Or if you have a novel robot or simulator, you can fine-tune MolmoAct2 there and compare performance. The open tokenizer means you could explore different action vocabularies. The depth tokens mean you can study how explicit geometry helps (or replace them with your own perception module). In essence, MolmoAct2 lowers the barrier to experimenting with a state-of-the-art VLA model under the hood. Finally, the release of the large bimanual dataset alone is huge. Up until now, most robotics learning has focused on single-arm tasks. With 720+ hours of two-arm data, new research on coordination, manipulation synergies, or even biomechanical transfer (e.g. to dual-human teleop) becomes more feasible. Other teams can fine-tune on just bimanual tasks or combine it with their own to teach new two-arm behaviors. Concluding Thoughts. MolmoAct2 represents a milestone in the pursuit of generalist robotics. It doesn’t magically solve every problem – no current model does – but it makes a significant leap toward closing the gap between lab demos and real deployment. By carefully engineering the model (MolmoER and MolmoThink), scaling up relevant data (especially bimanual), and releasing all the pieces, the authors have created a practical platform. The reported gains – up to double the success rate of baselines in the real world – are non-trivial. Importantly, the system behaves like a real robot controller: it perceives the scene, “thinks” in multi-step fashion, and acts smoothly without long computation pauses. For the robotics community, the open nature of MolmoAct2 is as important as its raw performance. It may set a new norm where state-of-the-art means state-of-open-source. Future papers can now benchmark against MolmoAct2 instead of proprietary models. Labs can integrate it and stand on its shoulders, rather than reinventing the wheel. The hope is that more teams will follow this model: publish not just results, but code, data, and even fine-tuning recipes. The era of black-box robot AIs may be waning. In the immediate term, teams should try out MolmoAct2 on their own machines and robots. With the code on GitHub (Apache-2.0) and Hugging Face, it is literally a away in some cases. Whether one is working on a new household robot, a factory arm, or a simulator benchmark, MolmoAct2 provides a strong generalist starting point. And since every piece is open, there’s no barrier to customizing it – from swapping out the FLAN model for a future LLaMA condition, to adding different per-task heads. In sum, MolmoAct2 is a heavyweight entry in the robotics foundation model space, distinguished by openness and 3D reasoning. It doesn’t just promise generality; it provides a concrete, usable pipeline right now. For anyone building the next generation of robots, MolmoAct2 is both a powerful tool and a rich research artifact. References: The claims above are drawn from the MolmoAct2 preprint and blog by Duan et al. (AllenAI, 2026), as well as related VLA literature. Key details (model design, datasets, and results) are explicitly cited from that source, among others. Designers were careful to release full artifact packages for reproducibility. In context, note that other open models (e.g. OpenVLA, SmolVLA) also make different trade-offs, but MolmoAct2’s emphasis on spatial reasoning and real-world deployment marks it as a new benchmark.