Latest / Embodied AI 101 / NeoWorld-3: Can Executable 3D Programs Improve Sim-to-Real Robotics?
Transcript
- NeoWorld-3: Can Executable 3D Programs Improve Sim-to-Real Robotics?. NeoWorld-3: From Pixels to Code-Defined Physical Worlds • Eachen Soong et al. • Shanghai Jiao Tong University and Doxi AI • arXiv preprint • October 20, 2026. Cold Open. Picture a robot reaching for a microwave door. Before the arm moves, you want answers to several concrete questions. Where is the handle? Which parts move together? What volume will the door sweep through? Where can the other arm hold a bowl while that happens?. Now imagine answering those questions with an editable program: a cabinet body, a door, a hinge, dimensions, and relationships between them. Change a dimension, and the dependent geometry changes with it. Move a component, and the system can inspect the resulting configuration. That is the proposition behind NeoWorld-3: reconstruct an environment as something a machine can execute, inspect, and revise. Its design connects program authoring with numerical geometry refinement and interaction checks. The interesting question is what this representation buys a roboticist. Does source code give perception a more useful output—a world model that planning, simulation, and control can operate on directly?. The Idea. NeoWorld-3’s adaptation happens during reconstruction. A vision-language agent examines observations, writes a construction program, renders it, and revises it. The representation retains source code, named parameters, and object placements, so structural changes and numerical fitting operate on the same artifact. Take a hypothetical cupboard. Its walls, shelves, and door could share a width parameter. Adjusting that parameter changes related surfaces coherently. The program expresses which dimensions belong together, preserving relationships during fitting. This makes the representation a geometric prior: optimization searches through meaningful variations of an object rather than treating every surface coordinate as an unrelated decision. The computational layer is NeoSDK. It supports native construction in Blender and a tensor-based implementation of supported geometry operations. Numerical search runs through that tensor representation; selected candidates are materialized natively and checked for consistency. The useful distinction is between choosing the construction and fitting its measurements. An agent can decide that an object needs another component, while an optimizer adjusts the dimensions of components already represented. NeoMCP exposes observation, editing, rendering, and refinement operations as tools. For scenes, a coordinator delegates individual objects to workers, then reviews their outputs before integration. Each worker receives relevant observations and editable state. This divides a kitchen-sized reconstruction into local modeling problems while retaining a shared scene against which updates can be checked. The toolkit also separates visual geometry from physical solids. Inspection examines boundaries, intersections, orientation, and containment. Additional checks address articulation, while configured physics feedback can propose bounded rigid-body corrections. These are explicit operations on geometry and structure, giving the reconstruction process concrete things to inspect and revise. Real2Code provides useful context for why code belongs here. That earlier system reconstructs object parts from images, summarizes their geometry with oriented bounding boxes, and uses a fine-tuned language model to predict articulation code. The geometric front end supplies parts; the code describes their relationships. A variable-length program naturally accommodates objects with different numbers of components and joints. The important inheritance is compositionality: a robot object becomes an assembly whose connections can be represented explicitly. Consider an illustrative validation question: does a drawer remain seated in its rails throughout its travel? Answering it requires identifiable parts, a declared motion, and a relationship that can be checked across configurations. Articraft formalizes this style of asset-specific testing through a domain-specific SDK and validation interface. That helps explain the appeal of executable representations: geometry, articulation, and tests can refer to the same named components. For NeoWorld-3, the central design choice is therefore the place where adaptation happens. The system keeps a structured construction available for revision, then combines agent decisions with numerical fitting. That gives semantic reasoning and geometric optimization a shared object to work on. The Result. The headline is approximately 46% lower reported Chamfer distance than VIGA, the strongest comparison method on that object-reconstruction metric. That percentage is calculated from the authors’ published scores. Evaluation uses 212 3DCodeBench objects, 77 selected PartNet-Mobility objects, and ten RoboCasa365 scenes. Comparisons hold the language model fixed within each setting. The articulated-object results also improve joint estimation and geometry across motion states. Why focus on Chamfer distance? It compares sampled points from reconstructed and reference geometry. In this setting, it asks a direct shape question. The broader 3DCodeBench evaluation combines geometric measures with rendered-image similarity and program executability. Its references come from procedural object factories adapted from Infinigen, giving evaluators known geometry and construction code. That makes it a relevant benchmark for a system whose output is itself a modeling program. Those measurements serve different engineering purposes. Executability concerns whether the generated construction runs. Image comparison examines the rendered result. Geometry comparison evaluates the resulting shape. Reading them together gives a more useful picture of an asset-generation system than treating successful code execution as the final objective. That distinction is built into 3DCodeBench’s evaluation design. VIGA is a meaningful baseline because it already uses an iterative code–render–inspect loop. Its agent synthesizes programs, examines the resulting visual state, and uses discrepancies to guide edits, supported by memory and semantic tools. The comparison therefore goes beyond contrasting a sophisticated pipeline with a single uncorrected model response. The tooling ablation strengthens the systems story. Replacing NeoSDK and NeoMCP with BlenderMCP while retaining the NeoWorld-3 framework worsens visual similarity and leaves substantially more interpenetrating parts. The custom geometry stack contributes measurable value alongside agent orchestration. The result supports investing in the operations and feedback available to the agent, not merely its prompt. On hardware, two AgileX Piper arms use wrist RGB-D cameras and a MuJoCo twin. The demonstration combines cube-in-bowl placement, microwave loading, and door operation through geometric planning, visual pose updates, and contact feedback. For a practitioner, that progression is useful: a measurable reconstruction improvement, an intervention on the tool stack, and an integrated example connecting the reconstructed environment to manipulation. The Biggest Weakness. The biggest weakness is the missing causal link between reconstruction quality and robust real-robot transfer. The appendix labels the hardware experiments development demonstrations rather than a large-scale task-success study. Mass and friction are configured; door opening uses compliant feedback; bowl placement and door closing replay kinesthetic demonstrations recorded for that layout. “Zero-shot” therefore cannot mean a wholly demonstration-free manipulation pipeline. Consider a counterfactual experiment: replace the reconstructed microwave with a coarser geometric model while leaving the execution stack unchanged. If the task still succeeds, the successful demonstration says little about the value of detailed reconstruction. If it fails consistently, the reconstructed world has earned a stronger causal claim. The distinction matters because a functioning combined system and an effective individual component are different experimental findings. RialTo provides a useful evaluation precedent. It constructs digital twins, transfers demonstrations into simulation, improves policies through reinforcement learning, and evaluates real-world manipulation under changes in object placement, visual distractors, and physical disturbances. It also examines the importance of transferring the scene into simulation. Its relevance here is methodological: downstream behavior becomes an explicit experimental endpoint. For NeoWorld-3, I would hold the controller, observations, calibration procedure, and demonstration access fixed, then vary only the reconstructed environment. Compare its full output against simpler geometry and another reconstruction pipeline. Evaluate repeated trials across changed layouts and objects, using task completion and contact-related failures as outcomes. That experiment would answer the question a deployment team actually faces: does paying for a better world reconstruction produce more reliable robot behavior? The current evidence establishes a promising reconstruction-and-control integration; it does not isolate that deployment benefit. This is the strongest objection because it concerns the bridge from the paper’s measured reconstruction gains to its embodied-AI significance. Verdict. Simulation infrastructure teams should pay attention first. The valuable architectural idea is to retain construction logic and meaningful controls throughout reconstruction. That creates a shared interface for authoring, numerical refinement, inspection, and downstream consumption. NeoSDK’s separation between native construction and numerical search is especially relevant to teams building tools around existing graphics and simulation software. Articraft helps locate the contribution within the field. Its agent already writes against a specialized interface for parts, geometry, joints, and tests, while a surrounding execution system returns structured feedback. It shows how much of an agentic modeling system lives in the interface surrounding the language model. NeoWorld-3 belongs to that engineering tradition: the representation and the available operations are first-class design choices. VIGA supplies the complementary lesson. Executing a program produces evidence that can guide the next program edit. The output becomes part of an ongoing reasoning process. For researchers studying embodied agents, that is an appealing pattern: keep decisions grounded in inspectable consequences, and give the agent concrete ways to revise them. My practical takeaway is to design world representations around the questions their consumers will ask. A planner may need a named obstacle. An articulation module may need a parent–child relationship. An optimizer may need a dimension it can change without destroying the assembly. Programmatic representations offer a natural vocabulary for those interfaces, as Real2Code’s explicit treatment of parts and joints illustrates. For policy-learning researchers, a natural next integration is a programmatic reconstruction front end feeding a workflow like RialTo’s simulator-based policy improvement. That is a research direction, rather than a reported NeoWorld-3 experiment. It connects editable environment construction with an established way to use reconstructed environments for learning. The one-line takeaway: treat reconstruction as an editable engineering artifact that perception, optimization, and control can share. Outro. The next thing to watch is how world programs become persistent: revised from new observations, reused across tasks, and queried by different robot subsystems. Imagine a robot returning to the same kitchen tomorrow. A bowl has moved. A door is open. The robot updates the relevant state, checks the consequences, and continues working with a representation it can still inspect. That is the opportunity this episode points toward: continuity between what a robot sees, what its internal model expresses, and what its control system can use. A useful world model should remain useful after the first reconstruction is finished.