Latest / Embodied AI 101 / Fast Plans Are Not Enough: Does the Robot Actually Use Its Plan?

Transcript

Fast Plans Are Not Enough: Does the Robot Actually Use Its Plan? · Embodied AI 101

• Fast Plans Are Not Enough: Does the Robot Actually Use Its Plan?. Fast Plans, Faithful Actions: Closing the Planning-Execution Gap in Hierarchical Vision-Language-Action Models • Chuanliang Xie, Boyu Ma et al. • MARS Lab, Nanyang Technological University • arXiv preprint • 2026. A successful rollout can hide a broken hierarchy. Consider a thought experiment. Two robots complete the same tabletop task. Both display an intermediate plan, approach the correct object, grasp it, and place it in the designated container. Their videos look equally convincing. Now imagine changing the plan while keeping the scene and task instruction fixed. The first robot changes its approach appropriately. The second executes almost exactly the same motion as before. The original videos did not distinguish these systems. Yet they support very different engineering claims. In the first, the planner supplies an actionable decision. In the second, the displayed plan might be largely incidental to the policy that actually controls the robot. For a hierarchical system, that distinction matters. We may want to replace the planner, correct a subgoal, insert a constraint, or let a human edit the next step. Those capabilities require more than a readable intermediate representation. They require an executor whose behavior changes appropriately when that representation changes. Welcome to Embodied AI 101. This episode uses a compact empirical study to examine a larger design question: how should we build, measure, and trust the handoff between a vision-language planner and a learned action expert?. The empirical anchor. Submitted September 25, 2026, the paper studies an author-built π0.5-derived hierarchy—not unmodified π0.5. A PaliGemma planner supplies configuration, gripper, and duration waypoints to a flow-matching expert. Block-AR decodes whole waypoints, shifting inputs blockwise with within-block bidirectionality and cross-block causality. Normalized goal modulation, or NGM, combines normalized displacement, layerwise modulation, denoising-phase gating, endpoint noise, and dropout. | Measure | Token-AR | Block-AR | Block-AR + NGM | |---|---:|---:|---:| | Maximum LIBERO planner passes | 57 | 8 | 8 | | Robot planning latency | 1,094 ms | 125 ms | — | | Robot request latency | 1,266 ms | 293 ms | — | | LIBERO-Long success | 92.2% | 91.0% | 96.2% | | Four-suite mean success | 96.70% | 95.85% | 98.45% | | Robot successes | 48/60 | 46/60 | 51/60 |. Dashes indicate timings not separately reported here. Replacing endpoints with current states leaves suffix-conditioned success unchanged; under NGM, LIBERO-Long success falls 7.4 percentage points. Unconstrained goal injection, using Token-AR, achieves only 15% success. Those observations provide the starting point. The rest of this episode is an engineering interpretation, with thought experiments and proposed evaluations distinguished from the reported experiments. What a hierarchy buys—and what it owes you. Before discussing implementation, it helps to separate several meanings of “high-level planning.”. In the original π0.5 work, the model predicts semantic subtasks and then generates low-level actions conditioned on them. Its training recipe combines heterogeneous sources, including robot experience and semantic supervision. The high-level representation therefore participates in a broader strategy for transferring knowledge between kinds of data, rather than simply compressing a motor trajectory. HAMSTER makes a different choice. Its high-level VLM predicts a coarse two-dimensional end-effector path from an image and task description. A separate, three-dimensional-aware control policy uses that path as guidance. The intended advantage is that the high-level component can learn from cheaper off-domain information without having to predict embodiment-specific motor commands directly. GAE provides another useful comparison. It places sparse three-dimensional trajectories between a VLM and a generalizable action expert, emphasizing geometric refinement at the lower level rather than requiring the expert to recover the entire task’s semantics. Its separation between planning and execution is consequently different from a jointly adapted model sharing internal representations. These interfaces are not interchangeable. A semantic subtask leaves many geometric decisions unresolved. An image-space path specifies more geometry but introduces questions about depth and viewpoint. A robot-configuration target is closer to execution, but it also ties the representation more tightly to a particular embodiment. My design preference is to choose the interface backward from the responsibilities we want the executor to retain. Should the executor decide which side of an obstacle to pass? Should it select a grasp? May it revise the target when the object moves? Must it obey a specified contact sequence? Different answers imply different intermediate representations. There is also a basic accounting question: what work has genuinely been delegated? If both modules can independently infer the entire task from the same observation and instruction, the apparent hierarchy may contain substantial duplication. That duplication might improve robustness, but I would not assume it creates a useful command boundary. The practical requirement is a documented division of responsibility. The planner should communicate decisions the executor needs, while leaving enough freedom for the executor to handle geometry, uncertainty, and feedback at its own timescale. A waypoint should be an execution contract. Automatic Waypoint Extraction, introduced in Waypoint-Based Imitation Learning for Robotic Manipulation, offers a useful foundation for thinking about sparse supervision. AWE selects endpoints whose piecewise-linear interpolation approximates a demonstration within an error tolerance. It turns waypoint construction into a preprocessing problem rather than requiring a person to annotate every intermediate target. But reconstruction quality and execution quality answer different questions. Consider a hypothetical demonstration that lifts a mug, carries it across a table, and lowers it onto a coaster. A compressed representation might reproduce the recorded end-effector path accurately. Now move an obstacle into the transport corridor. The old endpoints can remain numerically well defined while the old route becomes inappropriate. For my own system, I would therefore treat a waypoint as an execution contract, not merely a coordinate. The contract should identify the coordinate frame, the target interpretation, and the completion criterion. “Reach this pose” is underspecified if one component interprets orientation in the world frame and another interprets it relative to the wrist. “Close the gripper” is underspecified if it does not distinguish a commanded aperture from evidence that an object was actually grasped. Timing deserves the same care. I would explicitly distinguish a desired arrival time from an execution budget. A budget permits the controller to stop and request help. An arrival-time command might instead encourage it to complete a motion more aggressively. Those are different control semantics, even if both happen to be represented by an integer. I would also specify what happens when the contract cannot be satisfied. Does the expert request a new plan, return a failure code, hold position, or choose an alternative? Without such a policy, a convenient learning interface can become an ambiguous runtime interface. Finally, I would log commanded and achieved states separately. The next segment should not quietly inherit the assumption that the previous target was reached. In a deployment review, I would want to see the measured state, the outstanding target error, and the reason execution advanced to the next segment. This is not an argument for making every waypoint complicated. It is an argument for making its meaning explicit. A small interface with clear semantics is more useful than a large interface whose fields are only loosely connected to behavior. Block decoding: optimize dependencies, not just token count. Two neighboring approaches help clarify the efficiency question. FAST changes the representation of robot actions. It uses compression based on the discrete cosine transform instead of independently binning every action dimension at every timestep. Its central lesson is that the choice of tokenization affects both sequence length and what an autoregressive model must learn. PD-VLA attacks the decoding procedure. It reformulates autoregressive decoding as a nonlinear system and uses parallel fixed-point iterations, aiming to accelerate action-chunk generation without requiring a new trained architecture. This is a different intervention from deciding that a semantic group should be the model’s prediction unit. My interpretation is that an efficiency audit should distinguish three quantities: how much information is represented, how much computation processes it, and how many computations must occur sequentially. Here is a deliberately simplified example, not a measurement from the paper. Suppose a plan contains five waypoints with ten value fields each. Scalar decoding requires fifty sequential value-generation steps. Predicting complete waypoints requires five sequential groups. Assume prefix processing costs forty milliseconds. A scalar decoding step costs ten milliseconds, while a wider waypoint step costs fifteen. The scalar version takes roughly 540 milliseconds; the grouped version takes roughly 115. The grouped operation is individually more expensive, yet the shorter dependency chain dominates. That example also shows why counting passes is not enough. The cost of a pass can change with width, caching, kernel implementation, vocabulary projection, and hardware utilization. I would benchmark the actual serving path rather than extrapolate directly from a complexity diagram. For a custom block decoder, my first correctness test would concern information leakage. During training, could any current target value enter the representation used to predict another value in the same output block? A model can look excellent under teacher forcing while failing when only previously generated blocks are available. My second test would concern multimodality. Suppose a target can be reached using two distinct arm configurations. If output fields are sampled independently, could the decoder combine components from incompatible solutions? Shared hidden representations do not, by themselves, make every possible sampled combination coherent. I would therefore inspect complete decoded waypoints, not just per-field accuracy. Useful diagnostics would include joint-limit violations, invalid orientation representations, inconsistent gripper timing, and disagreements between predicted duration and plausible motion. The attractive optimization is not “parallelize everything.” It is “remove sequential dependencies that the interface does not actually need, then test what dependencies were lost.”. Faster inference is not the same as faster control. For deployment, I would insist on three clocks. The first is model latency: when an observation enters the inference service and when a prediction leaves it. The second is observation-to-action age: how old the relevant observation is when the first corresponding command reaches the robot. The third is task time: how long the complete physical operation takes. Real-Time Chunking, from Kevin Black, Manuel Galliker, and Sergey Levine, illustrates why those distinctions matter. RTC generates the next action chunk while the current chunk executes. It treats actions already committed to execution as fixed and fills in a compatible continuation. The problem is not merely producing a prediction sooner; it is producing a prediction that remains usable when it arrives. Consider another hypothetical system. Its action buffer covers four hundred milliseconds. Planning takes three hundred milliseconds, and action generation takes another two hundred. Starting both stages only when the buffer begins executing leaves a hundred-millisecond deficit, even before transport overhead. Reducing planner latency could eliminate that deficit. But so could changing scheduling, moving the inference trigger earlier, or increasing the committed execution window. Those alternatives have different implications for reactivity, so I would not judge them using a single throughput number. In a profiling trace, I would separate image acquisition, preprocessing, prefix encoding, plan decoding, action generation, transport, and command execution. I would also record whether each stage overlaps another stage or blocks it. Then I would inspect the slow tail. A service that usually finishes on time but occasionally misses the buffer deadline may require a different execution policy from one with the same mean and tightly bounded latency. For a hierarchical stack, I would additionally log the age of the planner’s observation at each executed segment. A low-level expert may receive updated proprioception while still acting under an old semantic or geometric decision. That combination deserves its own test. My deployment question would be: under the expected disturbances and inference delays, can the system continually supply compatible actions based on sufficiently recent information? A fast decoder is valuable insofar as it helps answer that question. Faithfulness is a causal-identification problem. The surrounding literature gives a reason to be skeptical of conditioning interfaces that are evaluated only through ordinary task success. In Causal Confusion in Imitation Learning, de Haan, Jayaraman, and Levine show that behavior cloning can identify the wrong relationships in observational data. More available information can even produce worse control, because predictive correlations in demonstrations need not remain reliable under the learner’s own behavior. Their proposed remedy uses targeted interventions to identify relevant causal structure. CAST addresses a related issue for language following. It argues that datasets may associate a particular observation with only one likely behavior, giving a policy little reason to consult the instruction. Its counterfactual augmentation creates alternative language-and-action labels for otherwise similar observations, with experiments in visual navigation. For a planner–executor interface, I would build an evaluation around deliberately ambiguous situations. Imagine a scene with two equally reachable containers. The task permits either one, but the planner explicitly selects the left container. Now replace that command with an equally valid command selecting the right. This is a stronger test of controllability than asking whether an arbitrary perturbation changes the action vector. I would separate three outcomes. The executor might ignore the change. It might respond, but move in an irrelevant or unstable way. Or it might carry out the new valid command successfully. Only the third establishes the behavior we actually want from the interface. The next test would hold the command fixed while changing the environment. Move an obstacle into the chosen route, or shift the destination slightly within the controller’s intended correction range. The executor should retain the decision while adapting its implementation. Together, those experiments ask for selective dependence: change behavior when the command changes, and also change behavior when the world makes the old motion inappropriate. I would use matched initial conditions and controlled generative randomness for local action comparisons. For closed-loop evaluation, I would then let trajectories diverge naturally and score the consequences. The local test isolates the immediate response; the rollout tests whether that response remains useful. I would also avoid treating every response to an image perturbation as desirable visual grounding. Swapping in an unrelated image can cause a large action change without demonstrating obstacle avoidance or grasp correction. My preferred visual interventions would have a known physical interpretation. Finally, I would publish examples where following the command is the wrong response. An unreachable target should not be rewarded merely because it causes a large motor reaction. Faithfulness needs an admissibility boundary: comply with valid commands, preserve appropriate feedback, and reject or revise invalid requests. Strong conditioning creates capacity, not good judgment. To understand the appeal of layerwise modulation, it helps to recall FiLM. FiLM conditions a network by applying feature-wise affine transformations based on an external input. Instead of merely placing a condition beside other inputs, it lets that condition alter intermediate computation throughout the model. That is a useful architectural lever. But my design interpretation is that it changes how easily a condition can affect the answer, not whether affecting the answer will be beneficial. Consider a controller tasked with reaching a pose behind an obstacle. A strong target signal could help it maintain the intended destination. It could also encourage direct motion toward that destination when a detour is required. The target is informative about the desired outcome but incomplete as a description of the appropriate trajectory. For a generative action expert, there is an additional axis of computation. Flow matching learns a vector field that transforms samples along a probability path between noise and data; sampling numerically integrates that learned field. Its internal generation time is distinct from the physical time covered by the resulting action sequence. I would make that distinction explicit in every implementation review. “Early” could mean early in the generated robot motion or early in the numerical generation process. A schedule that changes conditioning during generation does not automatically constitute a feedback schedule during execution. When adapting such conditioning, I would start by documenting units and scales. A displacement representation that mixes translations, rotations, joint angles, and gripper states needs a deliberate normalization convention. Otherwise, the apparent importance of a dimension can be entangled with its numerical representation. Next, I would evaluate condition corruption by semantic distance, not only numerical magnitude. A small positional perturbation might preserve the intended grasp region. The same-sized perturbation near an object boundary might identify a different contact. I would not assume that one noise scale is benign across both cases. This becomes particularly important if the training action is left unchanged while the condition is perturbed. My acceptance criterion would be that the perturbed condition still describes an approximately compatible task. Where it does not, I would either change the action target as well or label the sample as an explicit uncertainty-handling case. Condition dropout deserves a similarly precise interpretation. Ho and Salimans’s classifier-free guidance jointly trains conditional and unconditional generation and combines their predictions at sampling time. Training with missing conditions and performing that inference-time combination are distinct operations; one does not imply the other. For my own system, I would test missing, noisy, stale, and contradictory goals separately. “No goal was supplied” is not equivalent to “a confident but wrong goal was supplied.” A robust fallback under the first condition gives little assurance about the second. The objective is therefore not maximum goal dependence. I would seek calibrated influence: enough authority to preserve the planner’s decision, but not so much that the executor stops checking whether its proposed motion still makes sense. Read benchmark gains at the right scope. Each variant was trained once. Real-robot evaluation used three tasks with twenty trials per method per task; other-backbone transfer remains untested. Those limitations determine the questions I would ask next, rather than making the reported results uninteresting. LIBERO was introduced to study knowledge transfer in lifelong robot learning, with procedurally constructed manipulation tasks and human demonstrations. Its original design separates several research questions about representations, task ordering, architectures, and transferred knowledge. A score on selected suites should therefore be interpreted together with the training and evaluation protocol. For reproduction, I would insist on identifying the exact dataset, checkpoint, action normalization, observation processing, and rollout configuration. Physical Intelligence’s OpenPI documentation provides a specific π0.5 LIBERO checkpoint and its reported evaluation results, which is useful provenance for an upstream reference. It does not remove the need to match experimental conditions when comparing a modified system. I would also separate several kinds of uncertainty. Repeating rollouts from different resets probes execution variability. Repeating training probes sensitivity to optimization and data ordering. Expanding the task set probes a different question: whether the result survives a broader range of behaviors. For the interface itself, I would want the baseline and modified model evaluated on the same valid-command counterfactuals. An improvement in ordinary success could arise from stronger perception, better timing, more favorable regularization, or improved command use. A targeted evaluation should distinguish these explanations rather than selecting one from the architecture diagram. On hardware, my preferred headline would include completion time and failure categories alongside success. Did a method reduce waiting? Did it recover more often after a missed grasp? Did it fail less often at a particular transition? Did it merely change which trials timed out?. I would also retain unsuccessful runs in the timing analysis. A system that terminates early or stops moving after a mistake can otherwise look deceptively efficient. Finally, I would avoid treating a benchmark improvement as evidence that the hierarchy is universally preferable to a strong flat policy. My comparison would include the simplest viable alternative under the same latency, training-data, and evaluation budgets. A practical adaptation and reproduction plan. As of October 2, 2026, I could not verify an official method-specific repository. OpenPI does provide upstream π0.5 models, checkpoints, and examples for inference and adaptation, making it a concrete starting point rather than a reproduction of this particular method. I would organize an implementation effort around observable milestones. First, establish a reliable reference pipeline before changing the model. Save complete inference inputs, action outputs, environment seeds, timestamps, and termination reasons. Make a small replay set that includes both successful and failed segments. If the reference changes while the interface is being modified, attribution becomes unnecessarily difficult. Second, define and test the waypoint data representation independently of learning. I would round-trip configurations through normalization, discretization, and decoding; verify coordinate conventions; and inspect gripper events visually. I would make malformed fields fail loudly rather than silently coercing them into plausible commands. Third, implement the planner change without simultaneously changing the executor’s conditioning. My first questions would concern decoded structure, complete-waypoint consistency, and actual serving latency. I would use the same observation prefix and execution policy while comparing decoders, so scheduling changes do not get mistaken for model improvements. Fourth, add the conditioning route behind a switch that permits controlled evaluation. I would test it on the frozen replay set before committing to a full hardware campaign. The most useful early plots would show responses to valid alternative targets, small physically meaningful scene changes, and missing or stale commands. Training-gradient routing also deserves an explicit decision. Knowledge Insulating Vision-Language-Action Models studies how introducing a continuous action expert can affect training speed and transferred VLM knowledge, and proposes insulating the backbone during training. That work makes gradient flow an architectural choice worth recording, rather than an incidental implementation detail. For an adaptation, I would compare whether the backbone is frozen, receives only planner supervision, or also receives action-expert gradients. I would not initially change all three alongside the decoder and conditioning scheme. A clean sequence of interventions is more useful than a single heavily modified model whose gains are difficult to explain. I would then move to a small hardware matrix with alternating methods, predefined reset procedures, and a fixed success rubric. Before scaling the task set, I would inspect every failure and verify that timing logs correspond to actual commanded motion rather than just returned predictions. The final reproduction artifact should include more than weights. I would want extraction settings, normalization statistics, vocabulary assignments, attention masks, conditioning schedules, serving configuration, and evaluation scripts. These are precisely the kinds of details an interface study must make auditable, because an apparently minor mismatch can change what the planner’s command means. The next experiment: test whether the modules are genuinely reusable. My highest-priority follow-on experiment would be a planner-swap study. Train one executor, then evaluate it with several planners whose target quality is independently characterized. Include demonstration-derived targets, a strong learned planner, a weaker planner, and systematically perturbed but still feasible commands. The question would be whether improvements in planner quality translate into improvements in execution without retraining the lower level. If they do, we have evidence for useful modularity. If the executor works only with the particular planner it was trained beside, I would describe the system as a coupled policy with an intermediate representation, not assume that the modules are independently replaceable. A second experiment would collect branching demonstrations. From the same scene and robot state, demonstrate multiple valid targets or routes. This is motivated by CAST’s counterfactual-data perspective, but a manipulation study would need its own physically grounded branching procedure rather than assuming navigation results transfer directly. I would combine that with recovery data gathered from the system’s own execution states. The motivation is the familiar imitation-learning problem formalized by Ross, Gordon, and Bagnell: the learner’s actions influence the observations it later encounters, so evaluation cannot be reduced to prediction on expert-distribution snapshots. Finally, I would test confidence-aware execution. Let the expert request replanning when the target is inconsistent with current perception, and measure whether that request improves outcomes. This would turn the interface from a one-way command channel into a negotiated handoff with an observable failure response. These are proposed extensions, not capabilities established by the current experiments. The takeaway: make the handoff earn its place. My standard for a useful hierarchy would be demanding but concrete. The planner should produce decisions quickly enough to matter. The executor should respond appropriately to changes in those decisions. It should retain the ability to correct motion from perception. And the system should expose what happens when a command is stale, ambiguous, or infeasible. For a research team, the most actionable move is to add interface tests before adding more planning sophistication. Create valid alternative commands. Introduce interpretable environmental changes. Inspect complete timing traces. Compare against the simplest system that could solve the task. That shifts the discussion away from whether a robot can display an impressive plan and toward whether the plan provides a reliable means of controlling behavior. A hierarchy earns its place when its intermediate decisions are timely, usable, and correctable—not merely visible.