Latest / Embodied AI 101 / Before the Robot Touches Anything: Physics-Grounded Deliberation with Embodied Tree of Thoughts

Transcript

Before the Robot Touches Anything: Physics-Grounded Deliberation with Embodied Tree of Thoughts · Embodied AI 101

• Before the Robot Touches Anything: Physics-Grounded Deliberation with Embodied Tree of Thoughts. Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model • Wenjiang Xu, Cindy Wang, Rui Fang, Mingkang Zhang, Lusong Li, Jing Xu, Jiayuan Gu, Zecui Zeng, and Rui Chen • University of Chinese Academy of Sciences, Nanjing University, Tsinghua University, JD Explore Academy, and ShanghaiTech University • IEEE Robotics and Automation Letters • 2026. The Core Idea. A robot can produce a perfectly reasonable manipulation plan and still fail for reasons that never appear in the task instruction. Open a microwave door, and the door may sweep a nearby ball off the table. Put an apple into a holder before moving the holder, and the apple may block the only viable top-down grasp. Open a drawer before picking up an object beneath its front panel, and the drawer itself may make the object unreachable. These are not primarily failures of language understanding. They are failures to predict how a scene will change under contact, gravity, kinematic constraints, and action ordering. Embodied Tree of Thoughts, or EToT, addresses this problem by separating semantic proposal from physical verification. A vision-language model proposes several high-level plans. Rather than immediately sending one to the robot, the system reconstructs the scene inside a physics simulator, executes the candidate plans in that digital twin, observes what goes wrong, and asks the VLM to revise the planning tree. Only a plan that survives this simulated pre-mortem is transferred back to the physical robot. Real-world visual feedback then closes the loop and triggers reconstruction and replanning if execution diverges from expectation. This is not a new foundation model, nor is it an end-to-end visuomotor policy. EToT is a planning architecture assembled from pretrained perception systems, GPT-4o, a physics simulator, grasp estimation, and a small library of semantic robot skills. Its contribution is the way those components are organized: language proposes alternatives, simulation supplies consequences, and tree search turns failures into new branches. That organization is the most important part of the paper. Why Language-Based Planning Needs an Embodied World Model. Modern VLMs are quite good at identifying objects, interpreting natural-language goals, and producing plausible task decompositions. What they generally do not possess is a calibrated forward model of a specific physical scene. A static image can tell a model that a tennis ball is next to a microwave. It does not directly reveal the microwave door’s swept volume, whether the door will collide with the ball, how far the ball will roll, or whether it will leave the robot’s workspace. Similarly, geometric keypoints may specify where a pen should end up without predicting whether the holder will tip, whether an obstruction will cause the pen to rebound, or whether a grasped object will slip because of friction and gripper-width limits. One response has been to use video-generation models as world models: condition a generator on the current image and proposed action, then inspect the generated future. The attraction is obvious. Generative video can represent rich appearance and does not require an explicit object model. But visual plausibility and physical validity are different objectives. A generated video can smoothly depict a pen entering a holder even when an apple is already occupying the opening. EToT instead uses a reconstructed, interactive simulation. The simulator supplies rigid-body dynamics, collision checking, gravity, articulated joints, and persistent state across multiple actions. The VLM no longer has to imagine all physical consequences internally. It proposes what to try and interprets the resulting rollout, while the simulator determines how objects actually move under its physics model. The “Tree of Thoughts” component is equally important. A single chain-of-thought plan still commits early to one object, one receptacle, one grasp orientation, and one action order. EToT represents these alternatives as branches. Some branches are proposed before simulation because the instruction is ambiguous. Others are introduced after simulation reveals a previously hidden constraint. The paper therefore makes three closely coupled moves:. It represents the current scene as an executable digital twin rather than only an image or scene graph. It searches over multiple semantic action sequences rather than accepting one VLM decomposition. It uses simulated failures to create new high-level plans, including preparatory actions that were never stated in the instruction. What the System Actually Controls. A deliberately small skill interface. EToT operates over five semantic action primitives:. an object. a supporting surface. a container. an articulated object. an articulated object. A pose-qualified version of is available when an object has meaningfully different grasp orientations, such as horizontal and vertical grasps. AnyGrasp estimates grasp poses and gripper widths from RGB-D observations. Placement and container insertion are exposed through dedicated APIs, while opening and closing articulated objects rely on manually scripted control primitives. This restricted interface is not incidental. It converts an unconstrained language-planning problem into search over a typed action grammar. The VLM selects objects, destinations, grasp variants, and ordering, but it does not synthesize joint trajectories or arbitrary contact strategies. The released implementation makes this structure especially clear. Priori Branching is implemented with few-shot GPT-4o prompts containing domain rules and examples. A second prompt checks the generated plan’s format and logical validity, rejecting outputs until they satisfy constraints such as “a pick must be followed by a placement or insertion” and “a holder is not an articulated object.” Another query detects whether the instruction explicitly requests pose diversity and, if so, expands the relevant pick action into pose-specific branches. That is sensible systems engineering, but it changes how the result should be interpreted. The planning competence does not emerge from an unconstrained VLM alone. It comes from the combination of a hand-designed skill vocabulary, explicit symbolic rules, few-shot examples, iterative format checking, simulation, and VLM-based visual diagnosis. Reconstructing the embodied world model. The Real2Sim stage begins with RGB-D perception. For most rigid scenes, the authors aim to construct the digital twin from one observation. First, SAM 3 extracts object masks, optionally assisted by user-specified points when stacked or visually complicated objects are segmented incorrectly. SAM-3D-Objects then converts each masked RGB crop into a textured mesh. Because the generated mesh does not preserve metric scale, the system uses the size-estimation component from DexSim2Real² to align the mesh dimensions with the RGB-D measurements. FoundationPose estimates each object’s six-degree-of-freedom pose relative to the camera, after which the scaled and posed meshes are imported into OmniGibson. Articulated objects need additional information. The pipeline uses two RGB-D observations, taken from different viewpoints or with the object in different kinematic states, to recover components, internal geometry, joint placement, and motion limits through DexSim2Real². This matters for objects such as drawers and microwaves: a visually complete external shell is not enough if the simulator must open the door or place something inside. Collision geometry also receives explicit attention. OmniGibson’s default convex hull can incorrectly seal the opening of a non-convex object such as a pen holder. The authors therefore use convex decomposition with CoACD before importing these objects. This seemingly mundane detail is central to the method. If the collision model says a holder has no opening, no amount of high-level reasoning will make insertion feasible. The reconstruction is not fully automatic. The appendix acknowledges that generated shape deformation, noisy depth, camera distortion, stacked objects, and pose-estimation errors may require minor manual corrections. The resulting “world model” is thus best understood as a task-specific simulation scene assembled with foundation-model assistance, not an autonomously learned representation guaranteed to match every physical parameter. Priori Branching: enumerate decisions before acting. Given the task instruction and an image, the VLM first parses object instances, spatial relations, and articulated states. Relations may say that a pen is on a drawer or that the drawer is closed. From this description, the model generates several complete candidate action sequences. The authors distinguish two forms of initial branching. Instance-level branching handles alternative objects or destinations. If a pen may be placed into either of two holders, EToT creates a branch for each holder. If one of several balls may satisfy a task, each ball can become a separate branch. Manipulation-parameter branching handles distinct interaction modes. A holder might be picked from the side or from above. Because these imply different contact and gripper conditions, they become separate actions in separate branches. Sequences sharing a prefix are merged. Two plans that both begin by picking up the pen share that node and diverge only at the destination. The result is a compact action tree rather than a disconnected list of complete plans. Priori Branching is particularly useful when the feasible alternative is already implicit in the instruction or visible scene. In Task 2, it creates branches for the black and white holders. In Task 3, it creates horizontal and vertical grasp branches. Simulation can then select the stable holder or grasp without first damaging the real scene. But initial branching cannot anticipate every relevant intervention. An instruction to open a microwave does not naturally imply moving an unrelated tennis ball. That is where reflection enters. How the Tree Searches, Fails, and Repairs Itself. EToT traverses the planning tree with breadth-first search. At each node, it executes that node’s action in the simulator and renders the resulting observation sequence. A VLM-based judge evaluates the rollout. The judge is not checking only whether the commanded object reached its immediate target. It compares the state of all visible objects before and after execution. An action is considered unsafe if it creates an undesirable change that cannot easily be repaired using the available skill set—for example, knocking a ball off the table or sending a pen outside the reachable workspace. If the action succeeds and the node has children, the search continues. If it succeeds at a leaf, EToT extracts that root-to-leaf path as the executable task plan. In other words, the system seeks a feasible branch rather than optimizing a continuous numerical reward. If the action fails, the associated subtree is removed and the rollout is passed into Reflective Branching. The VLM receives the failure images, the failed action, and the existing path. It diagnoses the cause and proposes one or more corrected sequences, which are merged back into the tree. Search resumes from the first new node. The paper focuses on two correction patterns. The first is collision-induced disturbance. If opening a microwave pushes a tennis ball toward the table edge, reflection inserts actions that pick up the ball and place it somewhere safe before retrying the door. The preparatory relocation is not part of the original request, but it becomes necessary once the simulator exposes the interaction. The second is ordering-related conflict. In the task where an apple must end up inside a holder and both must be placed on a drawer, a semantically natural plan is to insert the apple first and then move the combined arrangement. Simulation reveals that the apple blocks the holder’s required top-down grasp. Reflective Branching reverses the dependency: move the empty holder first, then insert the apple. Task 7 illustrates how all the components interact. The robot must distribute an apple and a tennis ball between a holder and a drawer while leaving the drawer closed. The holder cannot fit both objects. The apple is too tall for the drawer to close. And if the drawer is opened too early, its front blocks access to the ball. Solving the task requires choosing the right object-to-container allocation, moving the ball before opening the drawer, and maintaining a valid open-insert-close sequence. It is not just a longer plan; it combines allocation, geometry, dynamics, and ordering. After a simulated plan is selected, execution remains closed-loop. The physical camera observes every action, and the same type of VLM evaluation checks whether it succeeded. If execution fails or the environment changes, EToT treats the current physical scene as a new root state, reconstructs it, and runs planning again. This is how it handles human intervention in the disturbance experiment. The Experimental Testbed. The experiments use an xArm6 six-degree-of-freedom manipulator, a one-degree-of-freedom parallel gripper, and an Azure Kinect DK mounted at a fixed oblique angle. GPT-4o serves as the VLM for EToT and all baselines. Each experimental condition is repeated ten times, and success requires completing the stated task without harmful environmental changes. The paper describes seven nominal tasks plus a separate disturbance condition—eight evaluated scenarios in total:. | Scenario | Hidden constraint | |---|---| | Open a microwave | The door will push a ball off the desk unless the ball is moved | | Put a pen into a holder | One holder is dynamically unstable | | Pick up a holder | The side grasp slips; the top grasp succeeds | | Close a drawer | A tall toy blocks closure | | Disturbance: pick up a ball | A human makes the selected ball ungraspable | | Long Task 5 | An apple obstructs the viable pen holder | | Long Task 6 | Inserting the apple first blocks the holder grasp | | Long Task 7 | Capacity, drawer height, access, and ordering constraints interact |. These are carefully constructed diagnostic tasks rather than a broad manipulation benchmark. They test four properties: manipulability awareness, three-dimensional spatial reasoning, physical consequence prediction, and recovery from disturbances. Tasks requiring at most three actions are classified as short-horizon, while Tasks 5 through 7 require more than three. Three baselines are considered. ReKep generates relational keypoint constraints for manipulation. ReKep with chain-of-thought adds prompts intended to make the VLM analyze dependencies and likely failure modes more explicitly. The strongest baseline, Reflect*, augments the latter with the same outcome evaluator and an oracle recovery mechanism: when a recoverable physical failure occurs, a valid recovery plan is manually supplied. The star is important. Reflect* is not merely an off-the-shelf reflective planner; it is an unusually strong, manually assisted baseline. All methods also receive manually annotated, high-precision keypoints, and generated semantic plans are manually translated into the paper’s unified action API when necessary. These choices reduce perception and interface differences, focusing the comparison on high-level planning, but they also make the experiment less autonomous than the high-level architecture diagram might suggest. Results: Where Simulation Pays Off. The main success-rate results are:. | Method | T1 | T2 | T3 | T4 | Disturbance | T5 | T6 | T7 | Average | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | ReKep | 0% | 50% | 40% | 0% | 0% | 0% | 40% | 0% | 16.3% | | ReKep + CoT | 80% | 70% | 0% | 80% | 0% | 10% | 50% | 20% | 38.8% | | Reflect* | 80% | 70% | 80% | 100% | 100% | 20% | 50% | 30% | 66.3% | | EToT | 90% | 90% | 90% | 100% | 100% | 80% | 90% | 70% | 88.8% |. EToT succeeds in 71 of 80 trials. Its average is 22.5 percentage points above the oracle-assisted Reflect* baseline and 50 points above ReKep with chain-of-thought. The long-horizon results are the most informative. Reflect* scores 20%, 50%, and 30% on Tasks 5 through 7. EToT reaches 80%, 90%, and 70%. Reflection after a real failure helps when the failure leaves the world recoverable, as in retrying a dropped or slipped grasp. It cannot help when a mistaken action creates an irreversible state under the available skills. The simulator gives EToT an opportunity to reject those plans before they reach the robot. The ablations reinforce this interpretation:. | Variant | Average success | |---|---:| | Full EToT | 88.8% | | Without Priori Branching | 61.3% | | Without Reflective Branching | 35.0% | | Without real-world replanning | 75.0% | | Video-generation model instead of simulator | 21.3% |. Removing Priori Branching particularly hurts tasks where the system must choose among objects, containers, or grasp modes. Removing Reflective Branching is more damaging: the planner can no longer introduce obstacle relocation or repair incorrect action ordering, and it fails completely on all three long-horizon tasks. Turning off real-world replanning leaves most nominal tasks intact but reduces the disturbance task from ten successes to zero. The video-model ablation is striking but should be read carefully. The authors condition the Jimeng video generator on each action and reinitialize it from a real image at every step, specifically avoiding compounding generated-frame errors. Even so, it depicts the pen entering a holder already obstructed by an apple. The resulting average is 21.3%, with zero successes on all long-horizon tasks. This supports the paper’s claim that appearance-level prediction is insufficient for these contact-sensitive scenarios. It does not establish that every possible learned generative world model will underperform simulation; only one general-purpose video generator and prompting configuration are evaluated. Among EToT’s nine unsuccessful trials, the reported failure distribution is 44.44% execution error, 44.44% depth-estimation error, and 11.11% world-model planning error—apparently four, four, and one failures respectively. Execution problems include accidental contact with the drawer and the apple becoming wedged against the holder. Depth corruption shifts otherwise valid grasps away from the target. The remaining simulator error occurs when an unrealistic contact with the gripper stabilizes a pen that should tip the holder. The simulator reduces physical hallucination; it does not eliminate the reality gap. Finally, the authors acknowledge that tree-search cost grows with depth and branching factor. Their initial response is multi-world parallelism: sibling actions start from copies of their parent’s terminal state and are simulated in a batch. This reduces Task 7 planning time by roughly 50%, although the paper does not report absolute latency, node counts, token usage, or hardware-normalized throughput. Why This Work Matters. EToT’s deepest contribution is its definition of a world model. In much of contemporary embodied AI, “world model” implies a learned latent transition model, generative video predictor, or neural 3D representation. Here, the world model is an executable hypothesis about the scene: reconstructed geometry placed in a conventional physics engine. That choice yields a clean division of labor. The VLM handles semantic interpretation, open-ended hypothesis generation, and visual failure explanation. The simulator handles state transition. Neither component is sufficient alone. A simulator cannot infer from language that either of two containers satisfies a goal, while a VLM cannot reliably calculate the consequences of frictional contact from pixels. Relative to ReKep and related spatial-constraint methods, EToT adds forward physical prediction. Relative to video-based reflective planning, it exchanges visual generativity for explicit contact and collision structure. Relative to Tree-Planner, STEP Planner, and language-guided geometric search, it evaluates nodes by executing them in an embodied environment rather than relying only on semantic or geometric plausibility. The closest comparison identified by the authors is Prompting with the Future, which also uses an interactive digital twin. That approach applies the twin primarily to low-level model-predictive control after accepting a fixed high-level task decomposition. EToT promotes simulator feedback to the task-planning level: a rollout can cause the system to relocate an obstacle, change a receptacle, or reorder the entire sequence. Viewed through a task-and-motion-planning lens, EToT resembles a learned proposal mechanism wrapped around an expensive but informative feasibility oracle. It is less formal than classical TAMP: there is no claim of completeness, no explicit symbolic domain model with proven preconditions and effects, and no optimization over path length or execution cost. But it can use open-vocabulary visual reasoning to discover constraints that would be laborious to encode by hand. Perhaps most importantly, the method demonstrates the difference between reactive recovery and proactive avoidance. A robot that reflects after failure can recover only if the failed action leaves the scene salvageable. A robot that simulates first can avoid unrecoverable branches altogether. Important Limitations. The evaluation remains small. Ten trials on eight handcrafted scenarios are enough to reveal large qualitative differences, but each individual failure changes a task score by ten percentage points. The paper reports no confidence intervals or hypothesis tests, and it does not evaluate broad distributions of objects, instructions, clutter configurations, lighting conditions, or physical parameters. The experiments show that EToT handles the designed failure mechanisms, not yet that it generalizes across an open household domain. The system also controls task planning more than perception or dexterous execution. It uses a fixed-base arm, a fixed camera, five discrete primitives, manually scripted articulated-object skills, manually annotated high-precision keypoints, and manual translation of some baseline plans. Pushing, pressing, tool use, bimanual operations, deformable objects, liquids, and mobile manipulation are outside the demonstrated scope. The authors explicitly identify richer skills and mobile manipulation as future work. Scene construction remains a substantial engineering procedure. Mesh scale and pose may require adjustment, articulated objects need extra views, and non-convex collision geometry requires preprocessing. The paper does not provide a similarly detailed account of how object mass, friction, compliance, and joint dynamics are identified from the real scene. Yet some evaluated phenomena—especially grasp slippage and tipping—are highly sensitive to precisely those parameters. The released code improves transparency but is not yet a push-button reproduction. The simulator stack requires separate environments for OmniGibson and AnyGrasp, additional assets, Isaac Sim, local path modifications, and licensed AnyGrasp components. The real-world implementation is described as under refinement and asks the operator to select task-specific keypoints before execution. The planning logic is available as notebooks rather than a fully integrated autonomous planning server. The VLM is both proposer and evaluator. Although simulation grounds the state transition, the judgment of whether a rollout is safe and successful remains linguistic and visual. Correlated mistakes are therefore possible: the same model family can overlook a relevant branch, misread the rollout, and accept its own flawed correction. Replacing some judgments with geometric predicates, contact events, task-specific state estimators, or independently trained critics would make the architecture less dependent on a single model’s interpretation. Search scalability is another open issue. Breadth-first expansion treats actions largely as feasible or infeasible and returns the first successful leaf. It does not explicitly trade off execution time, risk, energy, information gain, or reconstruction uncertainty. Complex scenes with dozens of objects, many destination choices, and continuous grasp parameters will need stronger pruning than simple prefix merging and batched simulation. Finally, the comparison to video generation is narrower than the paper’s broader rhetoric may suggest. A general-purpose video generator is not the strongest conceivable learned physical model. Future comparisons should include action-conditioned 3D dynamics models, object-centric simulators learned from interaction, uncertainty-aware neural simulators, and hybrids that use generative models to fill gaps in reconstructed geometry while preserving explicit collision constraints. How Roboticists Could Build on It. EToT is most immediately useful as a supervisory planner for structured workcells or household setups where the object count is moderate, a reusable skill library exists, and executing a bad action is more costly than spending additional compute on simulation. A robust extension would represent reconstruction and physics parameters as distributions rather than point estimates. Instead of testing each branch in one digital twin, the planner could run it across an ensemble of worlds with perturbed object poses, friction coefficients, masses, depth estimates, and grasp transforms. A plan would be accepted only if it remained safe across most plausible worlds. The search itself could be upgraded with risk-aware A*, Monte Carlo tree search, progressive widening for continuous grasp parameters, and transposition tables for physically equivalent states. Active-perception actions could become first-class branches: if two plans differ only because the holder’s interior or drawer clearance is uncertain, the robot could move the camera before committing. Finally, EToT need not remain tied to scripted primitives. VLA policies, diffusion policies, or learned visuomotor controllers could serve as leaf-level skill executors while the digital twin remains responsible for high-level counterfactual planning. The crucial interface is not the particular controller. It is the ability to simulate a skill, inspect its consequences, and revise the task plan before physical execution. Final Assessment. Embodied Tree of Thoughts is compelling because it makes a modest but consequential argument: better verbal reasoning is not a substitute for interacting with a physical model. Its headline result—88.8% success versus 66.3% for an oracle-assisted reflective baseline—is supported by tasks specifically designed around hidden dynamics, irreversible mistakes, and long-range action dependencies. The ablations tell a coherent story: initial diversity handles visible alternatives, simulation reveals latent constraints, reflective branching repairs the plan, and real-world replanning absorbs disturbances. The work is still a prototype. Its task domain is narrow, its digital twins require assistance, and its search will not scale unchanged to open-world mobile manipulation. But as an architectural direction, it is important. It shows how VLM reasoning can become more reliable without asking the VLM to become a perfect internal physics engine. The robot does not merely think of several plans. It tries them in an embodied counterfactual world—and learns what not to do before touching the real one.