Latest / Embodied AI 101 / Reason First, Move Second: A Modular Alternative to End-to-End Robot Policies

Transcript

Reason First, Move Second: A Modular Alternative to End-to-End Robot Policies · Embodied AI 101

• Reason First, Move Second: A Modular Alternative to End-to-End Robot Policies. Explicit task reasoning empowering robotic manipulation • Shun Gui, Kai Gui, and Yan Luximon • The Hong Kong Polytechnic University, Southeast University, and the Laboratory for Artificial Intelligence in Design • npj Artificial Intelligence • 2026. Why this paper is worth attention. Modern vision-language-action systems usually treat manipulation as a representation-learning problem. Feed a model enough images, language instructions, robot states, and demonstration trajectories, and it should eventually learn a mapping from perception to action. RT-X, OpenVLA, SpatialVLA, and π0 differ substantially in architecture and scale, but they broadly share this statistical-learning premise. The paper we are examining takes the opposite position. Instead of training a policy to imitate how robots have acted in the past, Shun Gui, Kai Gui, and Yan Luximon ask whether a robot can inspect a scene, express that scene in physically meaningful numbers, reason over those numbers with a general-purpose large language model, and then construct an executable manipulation program. Their answer is a modular system that separates vision, language, reasoning, and action. The interfaces between those modules are not latent embeddings. They are explicit textual and numerical representations: object names, three-dimensional bounding boxes, Cartesian positions, target coordinates, orientations, and release commands. No manipulation demonstrations are used to train the system, and no task-specific action policy is learned. On a Ufactory xArm 7, the resulting system completes tasks ranging from metric object rearrangement to semantic sorting, vague-request resolution, water pouring, and operating a drawer. Across 240 real-world trials, the authors report 91.67% success, corresponding to 220 successful executions. The paper was published on August 10, 2026, as an unedited early-access article in npj Artificial Intelligence. The important idea is not that LLMs can produce plans; robotics has already seen years of work on language-model planning. Nor is the key idea simply modularity. The more specific contribution is an attempt to replace the learned vision-to-action interface of a VLA model with an inspectable, metric protocol that an off-the-shelf LLM can manipulate directly. That makes this paper less a new foundation model and more a challenge to the assumptions behind foundation-model robotics. The argument against a monolithic VLA. End-to-end VLA models have an attractive property: if the training distribution is broad enough, a single network can absorb perception, semantics, embodiment, and control. But those capabilities are entangled. When a task fails, it can be difficult to determine whether the cause was object recognition, instruction grounding, task decomposition, action decoding, or low-level execution. They also require robot data. Open X-Embodiment and related collections have made cross-dataset training possible, but physical demonstrations remain expensive, unevenly distributed across embodiments, and concentrated around familiar pick-and-place behaviors. A policy trained for one action convention or arm may not produce meaningful actions for another without adaptation. The authors argue that this is partly an interface problem. Conventional VLAs communicate between modules using high-dimensional features whose physical meanings are not directly specified. The action decoder must therefore learn how visual and linguistic features correspond to end-effector motions. In the proposed framework, that mapping is not learned from demonstrations. The vision system emits explicit geometry, the LLM transforms goals and geometry into a metric plan, and conventional robot software executes that plan. This is why “explicit” matters in the paper’s terminology. Given an instruction such as “move all apples forward by 100 millimetres,” the system does not merely output an unexplained action token. It identifies the apples, extracts their current coordinates, applies the requested displacement, generates intermediate lifting and transfer motions, and emits a structured sequence with start points, destinations, optional orientation commands, and release decisions. There is still a black box in the system—the internal computation of the LLM—but the representations entering and leaving that black box have identifiable semantics. That is a meaningful, if narrower, form of interpretability. The system from perception to motion. The architecture contains four conceptually independent modules:. Three-dimensional object perception. Natural-language instruction input. LLM-based task reasoning and action planning. Grasp generation and robot execution. Their connection is the paper’s central engineering contribution. Turning a camera observation into a numerical scene. The robot observes the workspace using a Microsoft Azure Kinect, which supplies both RGB images and point clouds. The manipulation platform is a seven-degree-of-freedom xArm 7 with a Ufactory parallel gripper, controlled from a laptop with an NVIDIA RTX 4050 GPU. The experiments use 25 rigid, everyday objects on a relatively clear tabletop. For three-dimensional object perception, the framework reuses the authors’ earlier Recursive Cross-View, or RCV, detector. RCV reframes 3D detection as a set of 2D detection operations across views. It therefore avoids dedicated 3D box annotations, although it still requires a relatively small set of ordinary 2D bounding-box labels when adapting the detector to new object classes. For every detected instance, RCV returns an object identity, its eight-corner 3D bounding box, and its center in the robot’s coordinate system. These outputs are serialized as a Python-style dictionary. The LLM receives records resembling an object-centered scene graph, except that the nodes carry metric geometry rather than only symbolic relations. From these values, the reasoner can derive object dimensions, heights, orientations, volumes, pairwise distances, and relative placement. In one example involving two apples, the system reports both center coordinates and computes a center-to-center distance of approximately 225 units. The important distinction from a conventional VLM description is that “the apples are far apart” can be accompanied by an actual distance that can later be used in motion generation. This object-centric bottleneck deliberately discards most of the original pixels and points. The LLM does not reason over dense visual tokens. It reasons over a compact world model containing the entities that the downstream task is expected to require. That design is efficient and auditable, but it also creates a strong dependency: anything RCV fails to represent effectively does not exist for the planner. Missed objects, incorrect class labels, and inaccurate boxes propagate directly into the generated action strategy. Language as a task specification. The language side is intentionally uncomplicated. The user’s instruction is passed to the LLM together with the serialized scene. There is no separately trained language grounding module. The prompts establish the coordinate convention, explain the object dictionary, enumerate the available output fields, and impose task and safety rules. The supplementary material is particularly revealing here. It contains full prompts for seven representative cases, including metric translation, object sorting, ambiguous requests, pouring, and drawer operation. These prompts are not minimal. They tell the model how to interpret axes, how to choose safe transit heights, how to preserve or alter orientation, how to format operations, and how to verify that a plan satisfies the requested outcome. The drawer prompt, for example, explicitly asks the model to infer the opening direction, calculate the newly exposed interior region, choose collision-free placement points, and remember to close the drawer afterward. That prompt engineering should be considered part of the method. The framework removes task demonstrations and policy optimization, but it does not remove task specification. Instead, some of the prior knowledge that would ordinarily reside in a policy or skill graph appears in natural-language rules and a structured action grammar. Claude as a metric task planner. The principal reasoning model is Claude Sonnet 3.7. It receives the user instruction and scene dictionary, writes a scene analysis, decomposes the task, performs the necessary geometric or commonsense reasoning, and generates an action strategy. Consider the instruction “classify all items.” In the paper’s example, the scene contains an apple, a peach, a tennis ball, a doll, and two baskets. The LLM reasons that two available baskets imply a two-way classification. It groups the apple and peach as fruit and the tennis ball and doll as toys. It then uses the basket locations to construct the placement sequence. The classification itself is semantic, but the execution is numerical. For each item, the model must determine a safe lifting height, a point above the chosen basket, and a release position. The resulting twelve-step program follows the familiar lift-transfer-lower pattern four times. The model also performs quantitative reasoning. For equal-spacing tasks, it calculates desired object locations. For stacking, it estimates cube volumes and tries to put smaller cubes above larger ones. For pouring, it calculates which bowl is closer to the bottle and services that bowl first. In the reported example, the system computes three pairwise distances and orders the two pouring operations to reduce transport distance. The drawer example is the most ambitious. Based on the handle and drawer geometry, the LLM infers the direction of opening, chooses a 150-unit pull, estimates the newly accessible volume, and selects two placement coordinates for a cube and doll. The generated reasoning leaves roughly 31 units of edge-to-edge clearance between them before commanding the drawer to close. This is the heart of the paper’s claim: an LLM can transform semantic goals and metric object descriptions into a concrete robot program without learning that transformation from trajectories. From an action program to executable grasps. The LLM’s output is constrained to a Python-style operation sequence. Each operation identifies a target object and provides a start coordinate, target coordinate, optional orientation information, and a Boolean release decision. The supplementary examples show plans broken into explicit primitives such as lifting an apple to a safe height, translating it above a destination, and lowering it before opening the gripper. Grasp poses come from GoalGrasp, another earlier method from Gui and Luximon. GoalGrasp combines object-level 3D detection with engineered human grasp priors to generate six-degree-of-freedom grasp candidates without grasp annotations or grasp-policy training. Its object-level formulation allows the controller to request a grasp for a particular named instance rather than selecting an unassociated grasp from a scene-wide candidate set. In the earlier GoalGrasp evaluation, the method achieved 94% success for user-specified grasping and 92% under partial occlusion. The robot then combines the selected grasp, the LLM’s target states, and its existing path-planning and control functionality. In effect, the LLM chooses what geometrically meaningful state should come next, while GoalGrasp and the controller determine how the gripper should acquire and move the object. This division is sensible. Asking an LLM to emit joint torques would waste the mature machinery of inverse kinematics, grasp planning, and trajectory generation. At the same time, the paper still lets the LLM generate relatively low-level Cartesian waypoints. A more conservative implementation might use the LLM only to propose symbolic or geometric goals and leave all exact coordinate calculations to verified software tools. One perception pass, one plan, open-loop execution. The complete scene is perceived once, at the start of the task. The LLM then creates the entire plan in a single reasoning pass, after which the robot executes without updating its visual world model. This keeps execution simple and makes the original plan easy to inspect. It also means that the robot cannot react if an object shifts, a grasp changes its pose, a human enters the workspace, or a preceding action produces an unexpected state. The authors explicitly identify this open-loop design as a limitation and propose closed-loop visual feedback as future work. For a paper about reasoning, this is an important boundary. The system demonstrates explicit planning, but only limited deliberative autonomy. It does not yet perform the repeated observe, evaluate, replan, and recover cycle needed for robust long-horizon behavior. What the experiments test. The evaluation is organized around six capability families. The first is qualitative and quantitative scene understanding. The objective is to establish that object labels and 3D boxes can be converted into descriptions containing metric positions, dimensions, distances, and orientations. The remaining experiments require physical execution. Quantitative manipulation tests instructions such as moving apples forward by 100 millimetres or shifting all objects 80 millimetres to the right. These tasks require the model to map linguistic directions onto the defined robot coordinate frame and modify the relevant object centers. Rearrangement includes aligning objects with equal gaps, stacking cubes, placing objects into containers, and tidying the table. These tasks add geometric relationships between multiple objects. Retrieving items from vague requests tests commonsense grounding. A user might say that they are hungry while only a bowl is within reach; the robot must infer that food should be placed in the bowl. Another example asks the robot to help a bored child, prompting it to select toys rather than fruit. Object classification requires the system to infer useful categories and sort all detected objects. The number of containers constrains the number of categories, rather than the prompt spelling out every object-to-container assignment. Finally, long-horizon manipulation is represented by pouring water into two bowls and placing objects inside a drawer. The latter requires opening the articulated receptacle, calculating usable interior space, transferring multiple objects, and closing it again. Illustrated plans range from six steps for cube stacking to twelve steps for tidying or classification and eight high-level steps for drawer manipulation. These are materially longer than one-shot pick-and-place, although still modest compared with open-ended household activity. The headline results. The physical evaluation reports 40 trials for each of six execution categories, with randomized object choices, positions, and backgrounds:. | Execution category | Success rate | Successful trials | |---|---:|---:| | Quantitative manipulation | 95.0% | 38 of 40 | | Rearrangement | 97.5% | 39 of 40 | | Retrieving items from vague requests | 100% | 40 of 40 | | Object classification | 95.0% | 38 of 40 | | Pouring water | 85.0% | 34 of 40 | | Placing objects in a drawer | 77.5% | 31 of 40 | | Overall | 91.67% | 220 of 240 |. The pattern is more informative than the average. Semantic selection is not the bottleneck: vague requests achieve perfect success in the reported trials. Performance decreases as contact, articulation, and sequential geometric dependence increase. Pouring loses six trials, while the drawer task loses nine. The failure analysis attributes the 20 unsuccessful trials to five sources. Reading Table 5, there are three visual-detection failures, four grasp failures, six collisions, four out-of-workspace cases, and three reasoning failures. Only a minority are classified as reasoning errors. Most arise at the interface with geometry and physical execution. This supports one of the paper’s strongest practical arguments: modularity makes failures localizable. A missed detection can be separated from an invalid grasp, an unreachable coordinate, or a logically incorrect plan. But it also demonstrates that successful high-level reasoning does not remove the hard parts of robotics. It merely reveals where they are. Comparing the reasoning models. The authors separately compare GPT-4, Claude Sonnet 3.7, and DeepSeek V3 over 240 recorded reasoning cases. Human annotators judge task analysis and action-strategy generation rather than running the complete robot experiment for every model. Claude performs best, reaching 99.17% for task analysis and 98.75% for action-strategy generation. GPT-4 reaches 95.42% and 91.67%, while DeepSeek V3 reaches 89.58% and 83.75%. These results motivate the choice of Claude for the physical experiments. The distinction between reasoning accuracy and robot success is revealing. Claude’s action strategies are judged correct almost 99% of the time, while end-to-end robot success is 91.67%. The roughly seven-point gap is where perception errors, reachability, grasping, collision avoidance, calibration, and open-loop execution enter. Comparisons with existing VLAs. The paper reports that OpenVLA and Octo both achieve 0% when deployed zero-shot in the authors’ exact xArm environment without demonstrations or adaptation. Neither produces effective approach, grasp, or manipulation behavior. That result should be interpreted carefully. OpenVLA and Octo are learned policies with embodiment- and action-space assumptions. Dropping them unchanged into a novel arm and workspace is not the scenario in which they are normally expected to perform best. The experiment demonstrates an important deployment advantage of the modular system, but it is not an apples-to-apples ranking of manipulation competence. The paper also presents success rates copied from publications for roughly analogous tasks, including OpenVLA, SpatialVLA, CoT-VLA, π0, and ReKep. The authors explicitly acknowledge that the environments, robots, tasks, and data regimes differ, making a strict comparison impossible. Those numbers are evidence that the proposed system is competitive enough to be interesting, not proof that it has surpassed trained VLAs on a common benchmark. Where this sits in the broader robotics landscape. Although the authors describe the method as a new VLA paradigm, conceptually it occupies the boundary between VLA policies and LLM-driven task-and-motion planning. OpenVLA directly maps images and language to actions using a model trained on diverse robot trajectories. Its strength is the possibility of acquiring reusable visuomotor behavior from data. Its weakness is that a new embodiment or precise new behavior often requires adaptation. Embodied Chain-of-Thought, or ECoT, adds grounded reasoning traces to OpenVLA and reports a 28-point absolute improvement on difficult generalization evaluations without collecting additional robot data. Yet the underlying controller remains a learned VLA policy. Reasoning helps select and explain actions, but the grounding from model state to robot action still comes from prior policy learning. SpatialVLA attacks a related problem by injecting explicit 3D position encoding and adaptive spatial action grids into a VLA trained on approximately 1.1 million robot episodes. It represents a learned, geometry-aware route. Gui and colleagues instead compress geometry into text and ask an external LLM to operate over it. ReKep is probably the closest conceptual neighbor. It represents tasks using relational keypoint constraints and employs hierarchical optimization to convert those constraints into end-effector poses. Importantly, ReKep operates with a perception-action loop, while the present paper commits to a complete open-loop plan. ReKep is more optimization-centric; explicit task reasoning is more object- and language-centric. Inner Monologue provides another useful contrast. It allows an LLM planner to update its reasoning using environmental feedback. The new paper deliberately avoids that feedback during execution. A natural synthesis would retain the numerical scene and action language proposed here while adding Inner Monologue-style state updates and recovery. What the paper genuinely contributes. The strongest contribution is the numerical interface between semantic reasoning and robot geometry. Object names and metric boxes are expressive enough for an LLM to connect common sense with physical calculations, yet simple enough for an engineer to inspect. The second contribution is a strong demonstration of task-level generalization without task demonstrations. New instructions do not require collecting trajectories and fine-tuning an action head, provided the required objects can be detected and the task can be expressed using the existing action grammar. Third, the framework offers useful failure attribution. In an end-to-end policy, a collision can be difficult to diagnose. Here, an engineer can inspect whether the detector’s box was wrong, the LLM selected an unsafe waypoint, GoalGrasp proposed a poor grasp, or the controller failed to realize an otherwise valid command. Finally, the paper demonstrates that LLM reasoning can extend below symbolic task decomposition. The model does not stop at “put fruit in basket.” It computes metric transfer targets, safe heights, object clearances, opening directions, and ordering choices. That bridge from semantics to quantitative action is the most interesting empirical result. The caveats behind “training-free” and “explicit”. The framework is training-free only in a particular and useful sense: it uses no task-specific robot demonstrations and performs no manipulation-policy training for the evaluated tasks. It is not free of learned components or data. The LLM is pretrained on Internet-scale text. RCV depends on a 2D detector and requires bounding-box annotations for new categories. GoalGrasp contains engineered grasp assumptions. The camera must be calibrated, the robot controller must be functional, and detailed prompts define the action protocol. Likewise, “explicit reasoning” describes the visible intermediate representation and generated plan, not complete transparency into the LLM’s internal causal process. A fluent explanation can be inspected, but it is not guaranteed to be a faithful account of why a language model produced a particular output. The supplementary prompts also reveal considerable procedural scaffolding. The drawer prompt already instructs the model to calculate safe heights and exposed interior space and to close the drawer. This is not demonstration learning, but neither is it entirely spontaneous discovery. A fair description is program synthesis from a structured task specification. What the evidence does not yet establish. The experiments use one robot, one gripper, one camera arrangement, 25 rigid objects, and a clear tabletop. Randomizing positions, object selections, and backgrounds is valuable, but it is not the same as generalization across embodiments, sensor placements, clutter levels, deformable objects, or contact regimes. The object-centric representation is also both a strength and a ceiling. Bounding boxes are sufficient for transporting apples or cubes, but not for many precision tasks. Insertion, tool use, cable handling, cloth manipulation, scooping, wiping, and fine assembly require local surfaces, contact states, force constraints, and often temporal feedback. Binary success rates hide additional questions. A quantitative-manipulation benchmark would be stronger with measured translation errors and spacing deviations. Pouring could report transferred volume, spill rate, and container-pose error. Drawer manipulation could report contact force, clearance, and recovery from partial opening. The current evaluation establishes feasibility, but not yet precision or safety envelopes. The lack of closed-loop perception is the most immediate technical weakness. A twelve-step plan compounds uncertainty from calibration, grasp displacement, object motion, and controller error. The lower pouring and drawer scores are consistent with that accumulated geometric risk. The comparisons also need restraint. Zero-shot failure by an embodiment-mismatched policy is unsurprising, while cross-paper success rates are not controlled experiments. A stronger evaluation would compare against adapted VLAs, LLM skill planners, task-and-motion planners, and constraint-based approaches on the same hardware and task definitions. Finally, the code and generated datasets were not public at publication time, although the authors say they may be requested. The full prompts are available in the supplement, but reproducing the system still requires RCV, GoalGrasp, calibration details, parsing logic, and the execution stack. How roboticists might build on it. The paper suggests a practical architecture for low-volume, high-mix manipulation: keep semantic reasoning explicit, keep metric state structured, and avoid training a new end-to-end policy when the task can be constructed reliably from existing robot capabilities. A production-oriented version should probably use a receding-horizon loop:. Perceive the current scene. Ask the LLM for the next goal or short sequence. Validate that proposal with deterministic geometry. Execute one primitive. Reobserve and compare the expected and actual states. Replan or recover when they differ. The LLM should not be trusted as the sole calculator or safety checker. Distances, free-space regions, reachability, inverse kinematics, collision constraints, and stability margins should be exposed as tools. The model can decide which computations are relevant, while conventional software performs them exactly. The action representation should also become a typed domain-specific language. Every operation could carry preconditions, postconditions, uncertainty estimates, force limits, and recovery branches. An object-placement command would be accepted only after grasp feasibility, path feasibility, support stability, and workspace constraints had been verified. Perception uncertainty deserves equal attention. Instead of passing a single 3D box, the system could pass confidence intervals or pose distributions. Low-confidence objects could trigger an alternative viewpoint or a clarification request rather than a potentially unsafe manipulation. Most importantly, this architecture need not compete with learned VLAs. A hybrid system may be more compelling. Explicit task reasoning can perform semantic decomposition, metric goal construction, verification, and failure diagnosis. A learned visuomotor policy can execute contact-rich local skills such as insertion, deformable handling, or compliant wiping. ECoT already shows the value of adding grounded reasoning to a learned policy; this paper shows the complementary value of making the external planning interface metric and inspectable. The bottom line. Explicit task reasoning empowering robotic manipulation is best understood not as evidence that LLM planning can replace robot learning, but as evidence that we may be learning some parts of the stack that do not need to be learned. For structured tabletop manipulation, an explicit object model, a capable language reasoner, a grasp generator, and a conventional controller can cover a surprisingly broad task set without robot demonstrations. The reported 91.67% real-world success rate makes that argument difficult to dismiss. At the same time, the experiment succeeds partly because the environment is object-centric, calibrated, static, and well matched to the available primitives. The next step is not simply a larger reasoner. It is a closed-loop, uncertainty-aware, constraint-verified system that combines explicit planning with learned physical skills. The enduring lesson is architectural: give reasoning modules information with physical meaning, and require them to return actions that engineers—and other software components—can inspect.