Latest / Embodied AI 101 / The LLM Proposes, the Planner Disposes: VLA-SP for Verifiable Robot Manipulation

Transcript

The LLM Proposes, the Planner Disposes: VLA-SP for Verifiable Robot Manipulation · Embodied AI 101

• The LLM Proposes, the Planner Disposes: VLA-SP for Verifiable Robot Manipulation. Large Language Model-Driven Symbolic Planning for Long-Horizon Robotic Manipulation Tasks • Hanzhuo Zhang, Jiahao Xu, Yicheng Xu, Vincent Chau, and Wanyuan Wang • Southeast University and the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences • Tsinghua Science and Technology, Just Accepted • 2026. As of August 19, 2026, this article is extremely new. The journal record says it was received on December 18, 2025, revised on August 2, 2026, accepted on August 10, and made available online on August 13 as a “Just Accepted” manuscript. A Symbolic Bottleneck Between Foundation Models and Robots. The basic proposition behind this paper is simple and important: do not let the language model directly control the robot, and do not even let it own the final task sequence. Instead, use the model as a compiler that translates perception and language into a formal planning problem. Then hand that problem to a conventional symbolic planner, which verifies the representation and computes the plan. Only after that should the plan be grounded into robot poses and executed through trusted skills. The authors call the resulting system VLA-SP, short for Vision-Language-Action via Symbolic Planning. It receives camera observations and spoken instructions, detects task-relevant objects with YOLO, transcribes speech with Faster-Whisper, and asks an LLM to fuse those observations into a file. The problem is interpreted under a reusable, predefined , checked and solved with Fast Downward, converted into six-degree-of-freedom target poses, and finally executed through a library of robot skills. That architecture makes VLA-SP less a new foundation model than a carefully partitioned robotic software stack. Its novelty lies in where it places the boundary between probabilistic foundation-model reasoning and deterministic machinery. Language models handle semantic interpretation. PDDL captures task state and action logic. Fast Downward handles combinatorial planning. Robot-specific adapters handle geometry and control. This is a pragmatic answer to a recurring problem in embodied AI. Modern multimodal models are good at recognizing that a user means, roughly, “put the red block in the box after moving the cup out of the way.” But roughly is not enough for manipulation. A robot needs object identities, predicates, goals, legal action orderings, collision-free poses, grasp configurations, and controller calls. The further we move from the original utterance toward actuator commands, the less tolerance there is for linguistic improvisation. VLA-SP’s central bet is that symbolic planning should serve as a narrow waist between open-ended semantic reasoning and tightly constrained physical execution. If that waist is designed correctly, we can swap language models, perception systems, planners, robots, or skill implementations without rebuilding the entire stack. The paper reports evaluation across seven simulated planning domains and six physical manipulation tasks, using both X-Arm and Franka platforms. The complete system obtains an average real-world task success rate of 82 percent. The Hard Part Is the Sequence of Representations. At first glance, instruction-conditioned manipulation appears to be one problem: observe a scene, hear a command, and act. In practice, it is a chain of distinct representation problems. The visual system begins with pixels but the planner needs typed objects and relational facts. The speech recognizer produces words, while the planner needs a goal state. The symbolic planner returns operators such as , , or , whereas the robot needs poses, grasp widths, approach directions, force settings, and trajectories. Each interface can preserve the user’s intent—or quietly corrupt it. Long-horizon tasks amplify these interface errors. A slightly incorrect grasp pose may fail immediately, which is easy to diagnose. An incorrect symbolic assumption can survive for several actions before making the goal unreachable. An object omitted from the initial state may cause the planner to reject a perfectly feasible task. A goal predicate that captures the wrong interpretation of an ambiguous instruction may lead to a formally valid but behaviorally incorrect plan. End-to-end VLAs try to absorb many of these mappings into one learned policy. That can be attractive when sufficient demonstrations exist and when fast reactive behavior matters. But it also makes failures difficult to localize. If an end-to-end policy puts the wrong object in a drawer, was the instruction misunderstood? Was the object misidentified? Did the temporal policy lose track of progress? Did the action decoder drift? The policy’s internal representation may offer few usable answers. A modular symbolic system exposes the intermediate artifacts. VLA-SP can, at least in principle, retain the transcript, detected objects, generated PDDL instance, planner diagnostics, final symbolic plan, grounded poses, and skill execution log. That is an unusually useful property for robotics engineering. It turns “the model failed” into a sequence of testable hypotheses. Of course, explicit interfaces do not eliminate errors. They merely make the errors inspectable. That distinction is central to understanding both the contribution and the limitations of the work. Walking Through VLA-SP. The authors describe VLA-SP as a two-stage embodied VLA framework. The first stage is a Multimodal Information Processing Module. The second is a PDDL-based Vision-Language-Action Module. Together, they automate the path from speech and vision to physical manipulation. From Cameras and Speech to an LLM Context. The first stage performs two familiar preprocessing operations. A YOLO detector identifies task-relevant objects in the visual observation, and Faster-Whisper converts a spoken instruction into text. Those outputs become the model’s description of what exists and what the user wants. This appears straightforward, but it defines the system’s perceptual contract. The planner never sees the world directly. It sees whatever symbolic candidates survive the detector and whatever linguistic intent survives transcription. A conventional object detector gives the system relatively stable names and locations, but its usable vocabulary is tied to its configured or trained categories. Even when detection is accurate, planning may require more than labels and bounding boxes. Predicates such as , , , , or must either be inferred from geometry, supplied by the prompt, or computed by downstream modules. The abstract does not specify how all such predicates are obtained. Speech introduces a different family of errors. Automatic transcription can lose object attributes, spatial prepositions, negation, or ordering words. “Put the cup beside the box, not in it” is brittle if a single token disappears. Language may also be perfectly transcribed yet remain ambiguous. If two red blocks are visible, a faithful transcription does not tell the system which one the speaker meant. An important extension would therefore be to preserve provenance. Each symbolic object and predicate should carry information about where it came from: detector output, geometric calculation, language inference, or a fixed domain rule. Confidence should travel with that provenance. Classical PDDL will not consume uncertainty directly, but the surrounding agent can use it to decide whether to plan, ask a question, or reacquire an observation. Why Generate Rather Than the Whole Planner?. After multimodal processing, the LLM produces a instance under a reusable. This is arguably the paper’s most consequential design choice. A PDDL domain defines the system’s symbolic physics: object types, legal predicates, available actions, and the preconditions and effects of those actions. A problem instance supplies the objects currently present, the initial state, and the desired goal. The planner combines the two to find an action sequence. Suppose, as an illustrative example, a domain defines that a robot can pick an object only if the gripper is empty, the object is clear, and the object is reachable. Picking makes the robot hold the object and makes the gripper nonempty. Placing reverses those facts and changes the object’s spatial relation. The LLM need not rediscover these rules every time. It only has to determine that the scene contains, say, a robot, two blocks, a tray, and an obstacle; that the robot’s gripper is initially empty; that one block is on the table; and that the goal is for that block to be in the tray. That division sharply reduces the model’s freedom. Generating arbitrary robot code would require the LLM to reason simultaneously about task semantics, action legality, APIs, ordering, and error handling. Generating an entire PDDL domain would allow it to alter the robot’s assumed capabilities or weaken critical preconditions. Generating only the problem instance keeps the action semantics under human control. This does not remove hand engineering. It relocates it into the domain model and skill library. Someone must still decide what predicates exist, what every action means, which effects are deterministic, and how symbolic operators correspond to physical capabilities. But that engineering is reusable and reviewable rather than regenerated for each instruction. The distinction also improves portability. A domain describing , , and can remain stable while the underlying implementation changes from one robot arm to another. The symbolic task does not need to know whether the arm is an X-Arm or a Franka. It needs to know whether the chosen platform exposes a skill satisfying the operator’s contract. Fast Downward as an External Verifier. The generated instance is submitted to Fast Downward. If it parses and describes a solvable problem under the domain, Fast Downward produces a multi-step plan. If the planner detects an issue, its feedback is routed back into an iterative correction process. The paper emphasizes that this gives VLA-SP traceable failure diagnosis rather than relying solely on unconstrained LLM self-reflection. This is a meaningful use of tool feedback. An LLM asked “Is this PDDL correct?” is still a probabilistic model judging its own probabilistic output. A real planner can provide much stronger signals. It can reject malformed syntax, unknown predicates, invalid types, inconsistent objects, unsatisfied requirements, or an instance for which no plan exists. The correction loop changes the LLM’s job from one-shot planning to constrained program repair. The model can be shown a specific failure and asked to modify a specific artifact. That is generally more tractable than telling it only that the robot failed. The planner also takes ownership of action ordering. The LLM identifies the state and goal, but the symbolic search algorithm determines whether an obstacle must be moved before an object is picked, whether a temporary placement is required, and which legal sequence reaches the goal. However, planner verification is only as meaningful as the supplied model. Fast Downward can establish that a plan is valid relative to and. It cannot establish that the detector saw the correct object, that the generated initial state matches reality, that a nominally reachable object has a feasible grasp, or that a domain effect accurately describes the physical world. In other words, VLA-SP can catch formally invalid representations, but a formally consistent hallucination may still pass. If the LLM declares that a container is open when it is actually closed, the planner may generate an entirely valid plan that cannot succeed. From Symbolic Actions to Six-Degree-of-Freedom Execution. Once Fast Downward returns a plan, VLA-SP grounds its symbolic actions into six-degree-of-freedom poses and invokes predefined robot skills. This is the point where semantic planning becomes physical manipulation. The phrase “predefined robot skills” carries substantial architectural weight. It implies that the planner is composing tested capabilities rather than producing joint commands. A symbolic presumably invokes a procedure that obtains a target pose, computes an approach, commands the gripper, executes a motion, and reports completion or failure. A skill performs the corresponding release sequence. This skill boundary is what makes cross-platform operation credible. The task planner can produce the same abstract action on two robots while platform-specific adapters implement the motion. The paper’s use of both X-Arm and Franka systems therefore tests more than hardware variety: it tests whether the symbolic interface is genuinely decoupled from embodiment. Yet geometric grounding remains one of the least forgiving parts of the stack. Symbolic feasibility does not imply inverse-kinematic feasibility, collision-free motion, stable grasping, or safe placement. The domain may say that an object is reachable while a particular grasp orientation is not. It may say that a placement surface is clear while the proposed end-effector path intersects another object. A production implementation should consequently use a hierarchy of validators. PDDL checks task logic. A geometric module checks grasp and pose feasibility. A motion planner checks paths and collisions. The controller checks execution tolerances. Perception then checks whether the expected symbolic effect actually occurred. The available abstract describes planner-driven correction before execution, but it does not clearly state whether VLA-SP re-observes the scene after every physical skill and replans when execution deviates from the predicted state. That distinction matters. Long-horizon reliability usually demands not just plan, then act, but plan, act, verify, update, and replan. What Does “VLA” Mean in This Paper?. The term vision-language-action now covers several rather different architectures. In much of the current literature, a VLA is a learned policy that consumes images, language, and often proprioception, then directly predicts discrete action tokens, continuous controls, or action chunks. VLA-SP is different. Based on the disclosed architecture, its central model does not directly decode continuous robot actions. It interprets multimodal inputs and generates a symbolic planning instance. Deterministic planning and conventional robot skills complete the route to action. Calling this a VLA is defensible at the system level: the complete pipeline connects vision and language to robot action. But it is more precise to think of VLA-SP as a neuro-symbolic VLA architecture, not an end-to-end VLA policy. That distinction affects expectations. An end-to-end VLA may acquire subtle visuomotor behaviors from data, but it can be difficult to verify. VLA-SP offers task-level interpretability and compositionality, but its capabilities are bounded by its predicates, domain operators, perception stack, pose-grounding machinery, and skill library. This is not a weakness so much as a different point in the design space. For many industrial, laboratory, and service-robot settings, explicit capability boundaries may be preferable to nominally open-ended behavior. Reading the Experimental Result Carefully. The paper reports simulated experiments across seven planning domains and real-world evaluation on six manipulation tasks. The hardware experiments use two different robot platforms, and the complete framework averages 82 percent success across the six tasks. The authors frame the results as evidence for reliability, interpretability, executability, and cross-platform applicability. The cross-platform component is especially valuable. A single-arm demonstration can accidentally conflate task-planning performance with one well-tuned hardware stack. Repeating the architecture on X-Arm and Franka provides evidence that the symbolic plan and skill interface are not entirely embodiment-specific. The combination of simulated planning domains and physical tasks also tests two different propositions. Simulation can isolate whether the LLM produces planner-compatible problem instances across varying symbolic structures. Hardware tests whether those plans survive perception, pose grounding, motion, grasping, and control. An 82 percent average is encouraging for a full pipeline because every module contributes potential failure probability. If transcription, detection, symbolic construction, planning, pose estimation, grasping, and placement each succeed 97 percent of the time, a six-stage pipeline already falls to roughly 83 percent end-to-end reliability. Long action sequences compound that effect further. Still, an average across six tasks leaves important questions unanswered. For a mature assessment, we would want per-task success, the number of trials, plan-length distributions, confidence intervals, failure categories, recovery rates, planning latency, and performance as horizon grows. We would also want to know whether failures are concentrated in perception, PDDL generation, geometric grounding, or skill execution. The most revealing ablation would separate the contribution of planner feedback from ordinary prompting. Other useful comparisons would include direct LLM plan generation, LLM-generated structured action lists without PDDL, PDDL generation without correction, oracle scene state, typed versus untyped domains, and open-loop versus action-by-action verification. Because this is presently a just-accepted article rather than a volume-assigned final issue record, its abstract provides the headline evaluation but not all of that experimental granularity. The Closest Neighboring Approaches. VLA-SP belongs to a rapidly growing family of systems that place symbolic or programmatic structures between foundation models and robot controllers. A particularly close 2026 comparison is Symbolic Wrappers Enable Precise Human in the Loop Operation for Large Action Models. That system also combines speech, foundation-model perception, an LLM, PDDL, and deterministic robot control. In its neuro-symbolic branch, the LLM is restricted to generating dynamic problem information while static action capabilities remain hard-coded. It reports that planning-layer failures are contained before unsafe robot commands are issued. VLA-SP’s distinctive emphasis is its explicit two-stage formulation, iterative Fast Downward feedback, seven-domain simulation evaluation, and transfer across two physical platforms. PLanAR takes the symbolic interface further into runtime execution. It represents states with predicates and skills with action schemas, but after every action it uses onboard observations to check whether the expected symbolic effects occurred. If reality diverges from the plan, the state is updated and the system replans. That action-by-action verification is a natural complement to VLA-SP’s pre-execution planner checking. Lang2Manip chooses a looser symbolic representation. Instead of generating PDDL, its LLM outputs a robot-independent JSON plan containing operations such as picking, placing, moving, or pushing. Those operations are subsequently grounded through grasp planning, inverse kinematics, and motion planning. The approach reduces formal domain engineering and may be easier to extend, but it gives up some of PDDL’s explicit precondition-and-effect semantics and solver-based guarantees. Work on deformable-object assembly demonstrates another hybrid. There, vision-language models identify subgoals, an LLM supplies symbolic structure, and a learned 3D diffusion policy executes low-level manipulation. Relative to VLA-SP, that approach uses a more expressive learned executor for contact-rich behavior, while retaining symbolic decomposition to manage horizon and recovery. A separate line of work targets the domain model itself. Bridging the Semantic Gap: Trajectory-Guided Domain Repair for Reliable Planning begins from the observation that syntactically valid PDDL can contain incorrect preconditions or effects. It uses execution trajectories and failure attribution to repair the domain, reporting a 71.2 percent execution success rate across twelve benchmarks while reducing the number of environment interactions relative to exploration-heavy repair. That work highlights VLA-SP’s most important residual risk: a fixed domain is safe only when its symbolic semantics remain aligned with the physical system. Finally, a June 2026 preprint sharing Jiahao Xu, Hanzhuo Zhang, and Wanyuan Wang as authors proposes GTP-FA, or Grasp-Then-Plan with Failure Attribution. It focuses on selecting grasps in light of downstream motion-planning feasibility and attributing failures between stages. Conceptually, that geometric failure analysis could sit immediately below VLA-SP’s symbolic planner. Together, these systems suggest a broader convergence: foundation models are increasingly being used not as monolithic robot brains, but as semantic components inside architectures with planners, skill libraries, geometric solvers, and explicit validators. What VLA-SP Gets Right. The strongest choice in VLA-SP is that the LLM does not own the definition of a legal action. The domain file and robot skill library establish a capability envelope. Inside that envelope, the model can interpret diverse instructions and scenes without being trusted to invent new low-level behavior. Second, planner feedback is a genuine external signal. The correction mechanism is not merely a larger prompt asking the model to reconsider. It uses a formal tool that can identify concrete representational failures. Third, the symbolic plan is an excellent debugging artifact. It can be inspected by a developer, compared with the user’s instruction, replayed in simulation, checked against domain invariants, and logged for failure analysis. Fourth, separating symbolic intent from robot-specific execution offers a plausible path to platform reuse. The X-Arm and Franka evaluations are therefore central to the paper’s case, not incidental hardware demonstrations. Finally, the architecture is deployable with familiar robotics infrastructure. It does not require that every manipulation skill be relearned inside one giant policy. Laboratories with calibrated workcells, established controllers, and reliable primitives can add natural-language task specification without discarding their existing stack. Where the Difficult Problems Move. The first unresolved problem is the domain model. A reusable reduces per-instruction generation errors, but it creates a substantial specification burden. Every operator must accurately capture both symbolic prerequisites and the effects exposed by the corresponding physical skill. If an action model is too permissive, the planner may approve physically invalid sequences. If it is too restrictive, feasible tasks appear unsolvable. If the physical skill changes but the domain does not, symbolic and real dynamics diverge. Domain versioning and automated conformance tests should therefore be treated as part of the robot’s safety case. The second problem is state abstraction. YOLO can say that an object is present, but task planning may need to know whether it is clear, contained, supported, occluded, open, reachable, or already grasped. These predicates are often where semantic hallucination enters the system. The third problem is uncertainty. Classical planning usually assumes a known initial state and deterministic action effects. Real manipulation is partially observed. Grasp success may be uncertain, objects may shift, and the camera may not see inside a container. VLA-SP’s symbolic layer would become more robust if surrounded by explicit belief management, active perception, or contingent planning. The fourth problem is runtime verification. Pre-execution PDDL validation prevents many bad plans, but it cannot prevent a good plan from becoming obsolete after a failed grasp. The most important practical extension is to verify the expected effect of every skill, update the symbolic state, and replan from the new observation. The fifth problem is ambiguity. An LLM may be capable of choosing a plausible interpretation, but a robot should not silently resolve every ambiguity. A clarification question is often cheaper and safer than executing the most likely plan. The symbolic representation can help here: missing object bindings or mutually incompatible goals provide precise triggers for dialogue. The sixth problem is geometric and physical safety. Logical validity does not establish collision avoidance, force safety, stability, or human awareness. Those properties remain the responsibility of motion planning, control, monitoring, and hardware-level protections. Finally, task diversity should not be confused with open-world competence. A system can accept varied language while still being limited to a fixed ontology and skill set. VLA-SP’s real generalization boundary is the intersection of what its detector can recognize, what its PDDL domain can express, and what its robot skills can execute. How I Would Adapt This Architecture. For teams interested in building on VLA-SP, the first step should be designing the capability model—not choosing the LLM. Start with a small skill library whose contracts are measurable. For each skill, specify required objects, geometric inputs, preconditions, expected symbolic effects, timeout behavior, and failure codes. Only then encode those contracts as PDDL operators. Next, treat the LLM as an untrusted compiler. Give it the domain, a typed inventory of perceived objects, allowed predicates, and a strict output schema. Reject undeclared objects and predicates before invoking the planner. If possible, use a constrained intermediate representation before rendering PDDL so that basic structural errors are caught without another model call. The validation stack should have several layers:. Schema validation for syntax, identifiers, and types. Symbolic validation through Fast Downward or another planner. Geometric validation for grasps, inverse kinematics, placements, and collisions. Execution validation that checks whether each skill met its postconditions. Task validation that confirms the final physical state satisfies the user’s actual intent. Every layer should return structured feedback. “Planning failed” is too vague. The correction agent should know whether an object was undeclared, an initial predicate was inconsistent, no symbolic plan existed, no feasible grasp was found, or an executed skill failed to establish its expected effect. Preserve stable object identities across perception and planning. A symbolic plan that refers to is useful only if that identifier remains attached to the same physical object after the camera moves or the scene changes. Object-centric tracking and scene memory are therefore as important as the initial detector. Add clarification as a first-class action. If two objects satisfy the same linguistic description, the planner should be allowed to produce something equivalent to rather than forcing the LLM to guess. For cross-platform transfer, maintain separate skill adapters beneath one symbolic interface. Then test contract conformance automatically: the same symbolic unit test should run against simulation, X-Arm, Franka, or any subsequent embodiment. Evaluation should stress each interface independently. Vary paraphrases and accents for speech; introduce false positives and missing detections; perturb object arrangements; increase symbolic horizon; swap LLM backends; alter domain predicates; inject grasp failures; and move the same task between robots. Report not just final success, but where each failure entered the pipeline and whether recovery succeeded. This architecture is particularly appropriate when tasks are compositional, skill semantics are relatively stable, and verification matters more than unconstrained dexterity. It is less natural when behavior depends on rich contact dynamics that are difficult to express symbolically, unless the symbolic planner delegates those phases to learned visuomotor skills. Final Take. VLA-SP’s contribution is not that an LLM can write PDDL. We already know language models can emit structured programs. The important contribution is the system boundary: the LLM interprets, the symbolic planner verifies and sequences, and robot skills execute. That boundary offers something end-to-end VLAs still struggle to provide: a readable account of what the robot believes, what goal it is pursuing, why each action is legal, and where a failure occurred. The reported 82 percent average across six physical tasks and two robot platforms suggests that this division of labor is practical, not merely conceptual. But its long-term value will depend on closing the loop after every action, handling uncertain symbolic states, and continuously checking that the domain model matches physical reality. For roboticists, the broader message is clear. We do not have to choose between foundation models and classical planning. The more productive question is which parts of the autonomy stack should remain probabilistic—and which parts should be explicit enough to inspect, verify, and trust.