Latest / Embodied AI 101 / MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation
Transcript
- MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation. The Allen Institute’s new MolmoSpaces platform marks a major step toward training robots that truly generalize. The guiding insight is that robots trained for the real world need to see the real world’s diversity in simulation. As the MolmoSpaces team puts it, deploying robots at scale “demands robustness to the long tail of everyday situations. The countless variations in scene layout, object geometry, and task specifications that characterize real environments are vast and underrepresented in existing robot benchmarks”. In practice, that means never before has the field had training and test data spanning hundreds of thousands of homes, offices, and schools, packed with hundreds of thousands of objects and millions of ways to grasp them – all under full physics simulation. MolmoSpaces delivers exactly this scale and variety. It is an “open ecosystem” for embodied AI that unifies vast 3D assets, simulation scenes, and benchmarks in one place. The core statistics are striking: over 230,000 distinct indoor environments, 130,000 curated object models, and 42 million high-quality 6-DoF grasps on roughly 48,000 manipulable objects. All of these assets are physically grounded. They run on real dynamics engines like MuJoCo (rather than simplified “magic” physics), with carefully validated parameters for weight, density, friction, and joints. The result is an ecosystem spanning the full spectrum of common embodied tasks – from simple pick-and-place in a single room to multi-room navigation and long-horizon interaction in a whole house. Importantly, everything in MolmoSpaces is open and cross-platform. The object and scene definitions are released as MJCF models (for MuJoCo) and also converted to USD format, so you can load them in MuJoCo, ManiSkill, NVIDIA Isaac Lab/Sim, or other simulators. The team provides tools to swap robots, controllers, and sensors in and out, and even to collect new teleoperation data. In short, MolmoSpaces is meant to be a common playground for the community: inspect the XML, tweak physics, regenerate grasps, and run the same scene in your favorite simulator. In the sections below we’ll take a tour of MolmoSpaces’ key ingredients: the assets and scenes, the physics fidelity, the enormous grasp dataset, the MolmoSpaces-Bench evaluation suite, and the open tools. Along the way we’ll see how this platform enables a new, simulation-first approach to robot learning – one that is already yielding promising zero-shot transfer results in real robots. The Challenge: Generalist Robots Need Diverse Simulation. The intuition behind MolmoSpaces is simple: if you want robots that can handle the messy real world, you must train them on massively diverse simulated worlds. In computer vision or NLP we already train on billions of images or text examples; for robotics, the analogous training “infrastructure” has been missing. Most prior benchmarks feature only modest variation – a few dozen rooms or objects – and often simplify the physics to explore algorithmic ideas. But when a real robot encounters a new household or an unfamiliar object, it faces distribution shifts that small-scale simulators can’t capture. MolmoSpaces founders Ali Farhadi and Ranjay Krishna frame this by noting that “the next wave of AI will act in the physical world, but realizing this potential requires robots that generalize across new spaces”. Historically, simulation was merely a stepping stone: you’d train roughly in sim, then collect a few batches of real data to fine-tune. MolmoSpaces flips that approach on its head. It was explicitly built to test whether richly diverse simulation alone can yield real-world robustness. By providing orders of magnitude more variety than any previous platform, it asks: if you shuffle rooms, assets, lighting, weights, and even the task instructions themselves, can we get robots that truly understand what to do rather than just memorizing narrow patterns?. This vision isn’t just wishful thinking. The creators argue that with “sufficient diversity across scenes, objects, lighting, physics, and task definitions, zero-shot transfer from simulation alone is not just possible, but practical”. Their own experiments bear this out: policies trained only in MolmoSpaces — with no real-world demonstrations — are already solving complex manipulation and navigation tasks on physical robots. In other words, by scaling up the virtual world, they’ve begun closing the “sim-to-real” gap in a fundamentally new way. For context, consider the lineage. AI2’s earlier platforms like AI2-THOR enabled exactly the idea that open, photo-realistic 3D worlds could speed up research in navigation. But THOR’s physics were crude (it used Unity with “magic” grasp spheres rather than real contact forces), and it covered only a few dozen scenes. MolmoSpaces, in contrast, targets fully physics-based interaction across hundreds of thousands of scenes. It also adds a strong emphasis on manipulation: tens of thousands of draggable, openable, holdable objects with annotated articulations and stable grasps. All this bakes in the variety our robots will need to see in reality. Scenes and Environments at Unprecedented Scale. At the heart of MolmoSpaces are its environments. Right now the library boasts over 230,000 indoor 3D scenes spanning homes, offices, schools, hospitals, and more. These scenes come from several sources and generation methods, including curated sets and procedurally generated houses. Notably, the team leveraged existing datasets where possible: they converted AI2-THOR’s 120 hand-crafted scenes (the iTHOR-120 set), procedural THOR environments, and a large Holodeck dataset. But they didn’t stop there — they used procedural generation at huge scale, sometimes even with the help of language models to arrange furniture and clutter with realistic variability. Each scene is annotated with metadata (room types, object positions, etc.) so researchers can slice and analyze performance by environment type if needed. Crucially, all these scenes are fully physics-enabled. Before accepting a scene, the team rigorously tests it: e.g. they apply small forces to every rigid object and reject any object that doesn’t budge at least 2 cm (otherwise it might be “stuck” in the simulation). For articulated objects (drawers, doors, sliders), they test moving them through their range and discard those that jam before 60% travel. They simulate each scene briefly to catch any drifting or intersection glitches, achieving over 95% success rate in these checks. Finally they compute occupancy maps to place the robot in collision-free start poses. All of these steps ensure that when a policy navigates or manipulates in MolmoSpaces, it’s dealing with plausible physics and stable setups. The composition of objects into scenes also underwent careful curation. The pool of objects is enormous – more than 130,000 models in total – so strict filters were applied. All 3D models come from two major sources: a custom collection from AI2-THOR and an even larger set drawn from Objaverse (an open 3D repository). From THOR, they extracted and converted 1,600+ rigid graspable objects across 134 categories (dishes, books, lamps, etc.), and dramatically expanded it with common household articulates like fridges, microwaves, ovens, dishwashers, doors, cabinets, and more. Each articulated model has its hinge or slider defined (axis, position, range) so that opening a fridge is no longer a special case hack but just another joint to actuate. From Objaverse the team started with 625,000 online models but applied heavy filtering to get high quality assets. They required each model to have complete metadata, represent a single object, and be properly scaled. Textures had to pass a quality threshold, and the 3D geometry had to be reasonably complex (under 1.5 MB) yet friendly to real-time sim. They even ran CLIP to check that models look similar under different renderers (“cross-renderer fidelity”) and sprayed a variety of test points on each model (receptacle test) to ensure it could actually hold other objects. After all that, they ended up with about 129,000 curated models covering roughly 3,000 WordNet synsets (categories). These are split into train/val/test sets of objects, enabling controlled experiments on held-out shapes. For procedural scene generation, they further select about 92,000 of the easiest-to-place models. Once objects are vetted, they populate scenes in bulk. Some scenes are hand-assembled by artists; others are digitizations of real homes (digital twins). But many are generated automatically. For instance, they ran “heuristic procedural generation” to stack rooms into multi-floor buildings, furnished them with objects in semantically plausible places (e.g. a TV on a TV stand, books on shelves), and even experimented with LLM-assisted layouts to get more natural variety. This blend of hand-crafted and procedurally-generated scenes means MolmoSpaces covers everything from very clean test environments to richly cluttered, unpredictable spaces. Experiments can thus probe a model’s limits not only on fixed layouts but also on broad variations. In summary, MolmoSpaces offers researchers literally hundreds of thousands of physically plausible indoor worlds. A robot can practice in thousands of different kitchens or messy living rooms drawn from this collection. Because the scenes span a wide range of building sizes and styles (single-room apartments up to 10+ room office/commercial spaces), researchers can examine generalization across layout complexity and scale. Best of all, these scenes are open and scriptable: you can pick a floor plan, swap out furniture presets, and even spawn new variants via the provided tools. Assets and High-Fidelity Physics. A platform is only as good as its objects and physics detail. MolmoSpaces pays special attention here. For every item and robot in the world, the physics parameters (mass, density, collisions, joints) were systematically tuned. This is a big change from earlier toy simulations. Recall that AI2-THOR treated a grasp as happening “magically” whenever the gripper neared an object, without any real force or weight. MolmoSpaces abandons that shortcut. Instead, it uses real physics engines (like MuJoCo) and painstaking calibration so that a successful grasp really feels real. To achieve this, the team annotated or measured key quantities. For rigid objects, they first estimated the true mass via large language models (LLMs) or known data. They then checked the simulated mass (mass = density × volume in MuJoCo) and manually adjusted densities when needed so that each object’s weight is plausible. For articulated items (like a drawer), they built a teleoperation rig in simulation to physically open/close it and tuned the joint friction and inertial parameters until it moved like a real-world counterpart. Even the standard robot arm (a Franka Emika FR3) was calibrated this way: by comparing its behavior to actual recorded trajectories of a real FR3 moving blocks, they identified friction and torque limits to ensure the sim robot is a faithful stand-in. They didn’t stop at weights. Collision handling is also critical. Every model got careful collider meshes, often generated automatically with the CoACD toolkit for convex decomposition. The designers manually reviewed these as well. Large receptacle items (tables, cupboards) use simple primitives (boxes, cylinders) for stability, whereas more complex manipulable items (figurines, tools, gadgets) are fitted with multiple convex parts for fidelity. Every small or thin object fell back to a primitive to avoid simulation blow-ups. The result is that cloth-like objects or very irregular shapes are mostly absent, favoring robust mesh definitions. Combined with the calibration above, this means when a policy picks up or bumps into something, the physics behavior is consistent and believable. All of these efforts yield a foundation of high-fidelity simulation. Agents trained here must contend with slipping, inertia, and joint limits. A “stable grasp” means lift an object under force and torque perturbations in sim and have it stay in hand. Doors and drawers open with realistic feel. Even lighting conditions and sensor viewpoints are part of the variation control. Bottom line: in MolmoSpaces, your robot programs solve realistic tasks — they can’t rely on a magic snap-to-grasp or invisible supports. Huge Grasps Dataset for Scalable Manipulation. Beyond static scenes and objects, MolmoSpaces equips every manipulable object with thousands of candidate grasps. In total there are 42 million 6-DoF grasp annotations across about 48,000 objects. These grasps aren’t guesses: each object’s entire geometry was sampled using a simulated Robotiq 2F-85 gripper to find stable grasp poses. Concretely, the pipeline generates up to ~1,000 grasp candidates per object by “sampling” contacts on the object’s mesh (favoring mid-fingerpad or fingertip contacts on thin parts). For articulated objects, it samples grips on each movable piece (like a drawer handle) but rejects any grasp that collides with other parts. Then the grasps are clustered in the 6D space and pruned so that only a representative set of diverse, non-overlapping grasps remains. The quality control doesn’t end there. Every grasp is stress-tested. Rigid-object grasps are shaken with small linear and rotational perturbations to check stability. Grasp candidates for hinged or sliding parts must allow the object to actuate (e.g. open) through at least 70% of its range while maintaining contact. Finally, a simulated hand (the Robotiq fingers) is placed at each candidate and commanded to pick up or open the object. If the gripper can reliably lift and open it without collisions or drops, the grasp is marked as valid. This rigorous pipeline weeds out flimsy or “accidental” grasps. Why so many grasps? The idea is to support diverse manipulation strategies. A robot policy can choose from a menu of precomputed grasps rather than learning them from scratch, speeding up training. There’s even a provided “grasp loader” to embed these poses directly into any scene, and a trajectory-generation tool that can roll out a picking or opening demonstration given a chosen grasp pose. This enables scaling up imitation learning or dataset creation: you could generate millions of pick-and-place demonstrations in sim using these grasps, then train a model on them. In practice, the 42M grasps cover nearly every combination of object and grasp type you’d care about. If an agent needs to reorganize a kitchen, it can choose among dozens of stable ways to hold a plate or open a dishwasher. If it needs to navigate with a tool, it can sample the safe grasps for the tool’s handle. Combined with the objects and scenes, this grasp bank makes MolmoSpaces a truly large-scale playground for manipulation. MolmoSpaces-Bench: Systematic Generalization Tests. Rich assets are only useful if you can measure improvement. That’s where MolmoSpaces-Bench comes in. This is a set of benchmark tasks and evaluation tools designed specifically to stress generalization under controlled variation. Instead of reporting a single success rate, MolmoSpaces-Bench lets you sweep across multiple axes of difficulty and see how an agent’s performance changes. For example, it divides factors such as:. Object properties: Vary the shape, size, weight, and articulation of objects. Does your policy still succeed if the target is heavier, or attaches to a hinge rather than a drawer?. Layout complexity: Test in different room layouts — from a packed studio to a sprawling office or multi-floor house. Task complexity: Go from a trivial grasp-in-place step to a two-step task (e.g. pick up a bottle and then open a door with it) or even longer sequences. Sensory conditions: Change lighting, camera angle, or partial occlusion. Can the agent still find and identify objects under shadow, or if the gripper obscures part of the view?. Dynamics: Adjust friction, mass, or gravity slightly. This simulates subtle sim-to-real variations. Semantic conditions: Even the phrasing of instructions can be varied syntactically (different prompts) to test language robustness. By systematically holding most variables constant while varying just one, MolmoSpaces-Bench reveals specific weaknesses. Does a learned policy suddenly fail when objects are 2× heavier, or when the room is rotated? You can spot those out-of-distribution “kinks” that would be invisible if you only measured an overall pass rate on a mix of scenes. As the MolmoSpaces blog notes, this “gives researchers something the field has long lacked: the ability to systematically vary one factor at a time while holding others fixed across thousands of realistic scenes”. Concretely, MolmoSpaces-Bench defines 8 benchmark tasks spanning mobile and static manipulation. They include atomic skills like Pick+Place, Open (door/drawer), Close, and combinations plus navigation. For instance, one task might be “Navigate to the kitchen, pick up the red mug, and place it on the table.” Another might involve moving a heavy object vs. a light one. Crucially, these tasks are evaluated not just once but on many variants of environment and object parameters. So your model is scored on its distributional performance rather than a single scenario. This controlled evaluation framework has already yielded insights. In their experiments the team found that performance on MolmoSpaces-Bench correlates extremely well with transfer to the real world (Pearson R ≈ 0.96). They also used the bench to compare different policy versions: newer zero-shot models (trained on MolmoSpaces data alone) outperformed earlier simulators-trained agents, and specific failure analyses emerged. For example, policies were sensitive to initial arm configurations and camera viewpoint (suggesting a need for more viewpoint augmentation), and they sometimes hallucinated objects based on prompt style nuances. These kinds of findings demonstrate MolmoSpaces-Bench as a quantitative microscope for embodied AI. In practice, a researcher can use MolmoSpaces-Bench to probe their algorithm in rich ways. Want to know if adding texture randomization helps? Train in one setting and test on varying light. Curious if your grasp planner struggles with large weights? Test a version of Pick+Place with heavy vs. light boxes. In all cases you keep other conditions identical. Having all scenes and data in a single ecosystem makes such fair comparisons tractable. The benchmark tools and logging let you collect detailed performance spectra. Open Data and Tools: Reproducibility by Design. One of the most exciting aspects of MolmoSpaces is that everything is released open-source. The dataset, code, and tools we’ve described are publicly available under permissive research licenses. The team emphasized that not only are assets open, but the infrastructure is too: all MJCF scene files, physics parameters, collision meshes, and even the grasp generation scripts are provided for inspection and extension. You can take their base and plug in a new robot or sensor suite and it all just works. The release comes in stages. The initial launch includes the core modules of MolmoSpaces in MuJoCo MJCF (and provisional USD) form:. Assets: 130K+ object models (USD/MJCF), with meshes, materials, physical parameters, and rich metadata (names, categories, scale, mass). (These include the 129k from Objaverse plus ~1.6k from THOR.). Environments: 230K+ scene definitions (floorplans, object placements, lighting, etc.), ranging from single-room apartments to procedurally generated 10+ room spaces. Grasps: over 42 million annotated 6-DoF grasp poses on 48,000+ objects (both rigid and articulated). Tools: Python APIs and utilities for loading any of the above into multiple simulators. For example, they include a ManiSkill-compatible loader and scripts to convert the MJCF files into USD for NVIDIA’s Isaac Sim/Lab. There is also beta support for teleoperated data collection: you can hook up a smartphone or tablet (using the “Teledex” system) to drive the simulated robot remotely and record demonstrations. This works with their existing setups (DROID, CAP) with no extra calibration, so collecting a few dozen human demos is effortless. The emphasis on openness means any researcher can tweak MolmoSpaces for new needs: add more scenes, adjust physics, or regenerate grasps under new constraints. It truly is a platform rather than a proprietary product. To simplify access, the data lives on HuggingFace and GitHub. There’s a handy Python script that will fetch subsets of the data. The licenses are also clear and friendly: the Objaverse-derived subset is ODC-BY (common for 3D libraries) and everything else is CC BY 4.0. Just cite the papers if you use them, but otherwise you’re free to build on this foundation. Importantly, because MolmoSpaces is simulator-agnostic, it can slot right into existing workflows. Let’s say you have a stable RL setup in Isaac Sim; you can load MolmoSpaces scenes (via USD) and policies trained in those will behave identically to the MuJoCo trials. Or if you use ManiSkill’s environments (based on the SAPIEN engine), you can compare them side-by-side on the same underlying scenes and assets. The developers explicitly made MolmoSpaces work across MuJoCo, NVIDIA Isaac Lab/Sim, and ManiSkill. This is a big deal: it means community innovations in one framework (for example, GPU acceleration in Isaac or differentiable contacts in ManiSkill) can be directly applied to the same standardized testbeds. Results and Outlook: From Simulation to Reality. Along with MolmoSpaces the Allen Institute announced MolmoBot – a suite of robot manipulation models trained entirely in this virtual playground. MolmoBot is essentially a collection of neural policies (based on their OLMo vision-language-action models) that are capable of doing tasks like picking, placing, and opening objects, even in new environments. Crucially, these models were not tuned with any real robot data. Remarkably, MolmoBot transfers to physical robots zero-shot. In lab tests it performed pick-and-place with unseen mugs, opened drawers and doors, and executed multi-step tasks on actual furniture it had never encountered during training. The Ai2 team notes this is achieved without any real demonstrations, without photorealistic rendering, and without task-specific engineering. Their analysis found that simulation diversity was key: training on many different houses and objects had more impact on real-world success than training longer in a single virtual scene. In short, these results suggest that by investing in MolmoSpaces’ expansive, varied worlds, we can effectively narrow the sim-2-real gap. It’s worth stressing – these are early results. But with the platform open to all, we should see much more to come. Already, MolmoSpaces has more diversity than most previous embodied AI benchmarks by orders of magnitude. For example, the Habitat static navigation sim had on the order of a few thousand scenes; older manipulation suites like ManiSkill had hundreds. MolmoSpaces sets the bar at hundreds of thousands. Even Google’s robotics research hasn’t released anything nearly this big; most private labs rely on proprietary data or scaling up one scene at a time. AI2’s philosophy is explicit: let simulation be the common training ground. By making these environments public, they are inviting the community to collectively “design richer virtual worlds” together. Of course, MolmoSpaces has limits too. The objects are mostly rigid non-deformable solids, so tasks like cloth folding or pouring are not yet covered. The physics, while carefully tuned, is still as accurate as the models and parameters provided. Real robots and sensors have noise types that may still be missing. But compared to where we were, it is a strenuous leap forward. And it’s only the first step: the infrastructure allows anyone to extend it, whether by adding new articulated models, simulating outdoor scenes next, or plugging in new multi-robot setups. For practitioners, the immediate takeaway is: if you’re developing a navigation or manipulation agent, MolmoSpaces provides a hugely richer testbed than before. You can use it for additional pretraining, systematic evaluation, or generative data collection. If your research focuses on generalization, MolmoSpaces-Bench will help pinpoint failure modes. And if you want to prototype a robot behavior quickly, you can teleoperate in MolmoSpaces on a phone UI, then refine with self-play. The ecosystem approach means you spend more time on algorithms and less time on building environments or collecting real demos. In summary, MolmoSpaces is a massive open library of 3D worlds and tasks – the most ambitious yet assembled for embodied AI. It marks a shift from “each lab collects its own data” to “we all share one giant scientific training ground.” By combining diverse assets, high-fidelity physics, and systematic benchmarks, the platform aims to unlock robust, general-purpose robot learning. Early signs with MolmoBot are promising, and it will be exciting to see how others leverage this resource. For any researcher working on robot navigation or manipulation, MolmoSpaces should be on the radar as an enabling resource — one that finally lets us ask questions about generalization at the scale real intelligence demands. References: MolmoSpaces is described in a recent Allen AI technical report and blog posts. Key results on benchmark correlations and sim-to-real transfer are reported in that work. The Ai2 blog also details MolmoBot and open infrastructure, all of which underpin the above discussion.