Latest / Embodied AI 101 / Can Rephrasing Fix VLA Language Sensitivity Without Retraining?
Transcript
- Can Rephrasing Fix VLA Language Sensitivity Without Retraining?. Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models • Mikey Watts, independent researcher; Yuchen Cui, UCLA • arXiv preprint • 2026. Can you make a frozen robot policy more reliable by changing its instructions rather than its weights? Yes, in the simulated manipulation settings studied here—but that is not the same as making the policy understand language robustly. The attention-grabbing example is substantial: pi-zero-point-five succeeds at “switch on the stove” 100 percent of the time, but at “switch on the hot plate” only 2 percent. Watts and Cui turn that brittleness into an opportunity. They collect evidence about which phrasings work, distill rewriting rules, and put a rephraser in front of the unchanged policy. The reported gains transfer to held-out tasks. The important qualification is that the headline improvement is 16 to 27 percent relative, not 16 to 27 percentage points. This is a useful result about calibrating a robot’s language interface—not evidence that a small rulebook solves general-purpose instruction following. What the Paper Did. There is no new action decoder here. For architectural orientation, pi-zero combines a vision-language backbone with a flow-matching action expert that produces continuous action chunks. The public INT-ACT checkpoint relevant to this study accepts an image, an instruction, and robot state; its model card specifies four-action chunks and BridgeV2 fine-tuning with paraphrased instructions. The baseline therefore already has a training-time defense against phrasing sensitivity. That choice matters. Comparing a rephraser against a policy trained on exactly one sentence per task would leave an obvious objection: perhaps ordinary language augmentation solves the problem. Here, the practical question is what to do about sensitivity that remains after augmentation. It does not establish that augmentation is ineffective. It establishes a reason to investigate a complementary intervention. The main experiment uses pi-zero on SIMPLER’s WidowX-style Bridge manipulation scenes: eight evidence tasks and twelve sealed evaluation tasks, with twenty-four layouts per evaluation task. The second model-and-benchmark pairing is pi-zero-point-five on LIBERO. “Sealed” needs careful interpretation. It means a task contributes no evidence to learning the rewriting rules. It does not necessarily mean the underlying robot policy has never encountered that task during its own training. Those are different generalization questions: transferring a language adapter to a new task, and transferring a motor policy to a new task. Likewise, the Bridge evaluation’s out-of-distribution distinction concerns object nouns absent from the policy’s fine-tuning language. That is a meaningful vocabulary shift, but it is not equivalent to entering an unfamiliar building, operating a different robot, or acquiring a new contact-rich skill. A noun missing from one fine-tuning corpus also need not be absent from the backbone’s earlier pretraining. The intervention has three stages. First, generate alternative instructions and measure how the frozen policy performs under them. Second, give that evidence to a language model, which produces roughly ten to twenty explicit rewriting rules. Third, apply those rules to incoming instructions before passing the resulting text to the policy. No gradient update to the robot controller is involved. Conceptually, this is a small, empirical specification of the executor’s preferred language. Instead of assuming that every reasonable synonym should work, the system learns which vocabulary and sentence forms are compatible with this particular policy. The rules are an operational artifact: something an engineer can inspect, version, and challenge. The authors also compare different evidence sources: simulation rollouts, demonstration-based action-matching proxies, and their combination. Rollout-derived rulebooks perform best on the non-canonical instruction sets. I would not interpret that as proof that out-of-fine-tuning tasks inherently produce superior rules. Task distribution and scoring method change together. A proxy can reward a prediction that resembles an expert action without measuring whether the complete episode succeeds. For example, matching a demonstration’s gripper behavior need not establish that the robot selected the correct object or reached the correct destination. The cleaner follow-up would hold the tasks fixed and vary only the scoring signal. Can action similarity rank useful paraphrases as well as actual rollout success? If not, how much rollout evidence is necessary to calibrate it? That experiment would tell practitioners whether they can cheaply reuse demonstration datasets or need environment interaction. Deployment also has a subtle architectural detail: a scene-reading model produces a textual trace, and a separate applier receives that trace, the instruction, and the rules—not the image itself. So this is not merely a synonym replacement function. There is an interpretation stage before the policy-specific rewriting stage. That separation could help a rephraser resolve what an indirect request refers to, but it also creates another possible failure point. If the scene interpretation identifies the wrong object, perfectly following the rulebook can still produce the wrong command. The useful design constraint is that rewriting happens before execution. The low-level controller remains unchanged and continues using its normal observations. This is not a new online recovery policy that watches a failed grasp and decides what to say next. How Large Are the Gains?. Start with a concrete conversion rather than the relative-improvement headline. For human-written instructions with the Claude applier, the paper reports a 23 percent baseline and a 21.7 percent relative improvement. Multiplying those numbers gives approximately 28 percent success: an increase of five percentage points. In a hypothetical hundred attempts drawn from that same evaluation distribution, that corresponds to about five additional completions. It is not twenty-two additional completions. This arithmetic does not diminish the result; it tells us what kind of engineering improvement it represents. Five points can be valuable when the controller is expensive to retrain, its weights are inaccessible, or the intervention can be integrated independently. But a system succeeding roughly twenty-eight times out of a hundred remains unsuitable for unattended operation on that workload. An inexpensive improvement and an adequate deployed system are different achievements. The most important baseline is therefore not just the unchanged instruction. It is generic rephrasing without the learned rules. Otherwise, the experiment would conflate the value of a competent language model simplifying the request with the value of policy-specific evidence. That distinction changes the conclusion in at least one condition. For Claude on VLM-generated instructions, adding the rules gives a reported 4.7 percent relative gain over generic rephrasing, with a p-value of 0.0924. This comparison does not clear the conventional five-percent threshold. It is not proof that the rules do nothing; it is a reason not to claim that their incremental benefit is established in every setting. For an engineering team, the implication is straightforward: evaluate the cheap baseline first. If a generic scene-aware rephraser captures most of the benefit on your instruction distribution, the additional evidence-collection pipeline has to justify itself through better reliability, fewer regressions, or lower downstream cost. The LIBERO replication provides a different operating regime. On its held-out, in-fine-tuning tasks, success rises from 93.6 to 97.8 percent. The rulebooks do not measurably improve the out-of-fine-tuning tasks. Here, a 4.2-point improvement is more consequential than its size initially suggests. The failure rate falls from 6.4 percent to 2.2 percent—about a two-thirds reduction, calculated from the reported averages. That is worth taking seriously. But the scope remains essential. The strongest replication is repairing instruction variation around tasks the policy was fine-tuned to perform. It does not demonstrate a corresponding increase in its ability to execute unfamiliar tasks. Again, held out from the rule learner does not mean held out from the robot policy. The released artifacts include three draws of each Bridge rulebook type and three LIBERO rulebooks. Repeating the distillation is preferable to showing one fortunate prompt. Still, a deployment will usually use a particular rulebook, not the average of several experimental variants. I would want to see its individual regression profile before replacing the production instruction path. What the Oracle Search Actually Establishes. The oracle experiment asks a separate question: how much performance is available if we can search for a favorable instruction for each task?. The project describes three adaptive rounds of sixteen candidates, followed by remeasurement of the selected wording. Under that search, the Bridge vocabulary-strata gap contracts from twenty-one percentage points to three. This is powerful diagnostic evidence. Suppose a robot fails when asked to manipulate an unfamiliar object. You might conclude that its visual representation or motor skill does not generalize. If changing only the wording substantially improves performance, then that conclusion was incomplete. At least some usable behavior was already accessible through a different language input. However, the searched instruction is selected using feedback from the task being evaluated. The deployed rulebook does not get that privilege. It must choose a rewrite without trying dozens of alternatives on the new task. That separates two propositions:. There exists a substantially better instruction, and we can predict that instruction without interacting with the task. The first motivates the second. It does not establish it. The held-out rulebook experiments—not the oracle bars—are the relevant evidence for deployment. There is another technical nuance. “Oracle” here is a practical search result, not a mathematical upper bound over every meaning-preserving sentence. A finite search can miss better instructions. Conversely, selecting a winner on the same noisy measurements used to judge it can exaggerate performance. Independent remeasurement is important precisely because it separates discovery from confirmation. For a practitioner, I would use this style of search as a debugging instrument. Before collecting another large batch of demonstrations, test whether failures are highly sensitive to vocabulary or sentence form. If they are, language-interface work may be a productive intervention. If they are not, a rephraser is less likely to address the main bottleneck. But do not use that diagnostic to assign a mechanism prematurely. A language change can affect object selection, grasp timing, action generation, or behavior after contact. Success-rate differences alone do not locate the failure inside the architecture. The earlier From Intention to Execution study makes this distinction useful. INT-ACT separately measures whether the gripper approaches the correct source object and whether the task finishes successfully. Its experiments show that apparently correct object-directed behavior can coexist with poor execution. Even that “intention” measure is behavioral, not direct access to an internal plan. My reading is that these studies jointly argue for decomposed diagnostics. Rephrasing sensitivity demonstrates a functional problem in the complete system. It does not, by itself, prove that fine-tuning erased a particular semantic representation in the VLM. How Convincing Is the Sensitivity Evidence?. The dramatic examples and the catalogue of smaller effects deserve different levels of confidence. The single-edit search evaluates 426 candidate pairs for pi-zero-point-five, uses uncorrected significance tests, and adds an eighteen-point minimum-gap requirement. At a nominal five-percent threshold alone, a complete-null thought experiment would produce about twenty-one false flags on average. That arithmetic is not an estimate of the number of false discoveries in the published examples. The effect-size filter and selection procedure matter. It does explain why a collection of individually significant comparisons is not automatically a calibrated statement about how frequently arbitrary edits cause failures. I would distinguish an existence claim from a prevalence claim. The stove example is compelling evidence that a severe wording failure exists in the tested setting. A searched collection of failures is much weaker evidence about what percentage of ordinary user requests will encounter such a failure. The authors also manually screen scene ambiguity. That is necessary because semantic equivalence depends on the scene. Replacing a specific object name with a broader category may preserve meaning when there is one candidate and introduce ambiguity when there are several. For a stronger audit, I would label equivalence before looking at rollout outcomes. The evaluator should verify the moved object, destination, relation, ordering, and any exclusions. Otherwise, there is a risk of accepting a successful rewrite because it works while scrutinizing a failed rewrite because it might have changed the task. This matters especially for seemingly innocuous cleanup rules. Removing an adjective can eliminate redundancy, or it can erase the only distinction between two objects. Making an instruction shorter is not inherently making it more faithful. The main intervention tests use paired base instructions rather than treating every rollout as an independent sample. That is a sensible distinction: more repetitions improve the estimate for an instruction, but they do not magically create more distinct requests. There is nevertheless a second level of uncertainty. Many instructions refer to the same small set of tasks. Strong evidence about phrasing variation within those tasks is not equivalent to strong evidence about a large population of unseen tasks. I would therefore ask for task-level resampling, leave-one-task-out analyses, and per-task effects alongside instruction-level significance. A method can improve the pooled average because it repairs a recurring lexical failure on several heavily represented tasks. That is useful, but it is a different result from consistently improving every task family. The human-written evaluation strengthens the paper: thirty-seven people supplied 363 unique instructions after viewing demonstrations, without seeing the canonical wording. It reduces the concern that the rephraser merely understands the stylistic habits of another language model. Still, these are written descriptions of visible demonstrations—not a live human-robot interaction study. I would not extrapolate directly to spoken disfluencies, pointing, conversational corrections, or requests whose intended outcome has never been demonstrated. A deeper issue is that insensitivity to language is not necessarily robustness. LIBERO-Plus tests this explicitly: some policies retain substantial performance when instructions are removed, yet fail when the requested target changes within the scene. Its analysis shows why apparent paraphrase robustness can reflect weak use of language rather than good semantic grounding. A trustworthy interface needs both properties: similar behavior when meaning is preserved, and appropriately different behavior when meaning changes. Those should be tested together. That also tempers the interpretation of pi-zero-point-five outside its fine-tuning set. Similar averages across ordinary phrasing styles do not prove complete language blindness. The project’s oracle results rise from roughly 52 to 66 percent in that regime. My inference is narrower: the effectiveness of ordinary rewriting depends strongly on the task regime, while favorable instructions can still exist. What Is New Compared With Other Language Wrappers?. Putting a language model above a robot policy is not the novelty. Hi Robot already uses a hierarchical VLM-to-VLA interface to interpret complex requests, generate lower-level commands, and incorporate user feedback. It evaluates that arrangement on physical robotic platforms. The distinction here is the target of adaptation. A hierarchical planner asks which command should come next. This paper asks which surface form of an already selected command the executor follows reliably. Those functions could coexist. A planner might correctly decide that the robot should place an object on a plate, while a policy-specific rewriting layer chooses a formulation less likely to destabilize execution. The interesting contribution is learning that compatibility layer from behavioral evidence and expressing it as reusable rules. CoVer is the more direct comparator. It generates alternative instructions, samples action candidates, and uses a contrastively trained verifier to select among them at action-chunk boundaries. Its evaluated configuration uses eight rephrasings and five action samples per rephrasing. Importantly, it already caches rephrases at boot time; it does not need a new language-model generation at every control step. So the clean computational contrast is one committed rewrite versus repeated candidate generation and verification, not “one language call versus endless language calls.”. Nor should forty candidates be casually translated into forty times the wall-clock latency. CoVer’s implementation uses batching and parallel computation. Its original paper also reports gains on two real-world WidowX tasks. Watts and Cui find that CoVer trails their baselines in this evaluation; their alternate success-metric audit does not reverse that conclusion. That makes CoVer a relevant, substantive comparison rather than an obviously weak strawman. But it does not establish that rewriting generally dominates verification. My interpretation is that the two methods spend computation on different uncertainty. A rulebook tries to choose a good language input before execution. A verifier tries to choose a good action among candidates as execution proceeds. Their usefulness depends on where errors occur and how accurately the selection mechanism predicts those errors. A combined system is an interesting hypothesis: normalize known lexical failures first, then spend a smaller verification budget on remaining action uncertainty. This paper does not test that combination, so it should remain a proposed experiment rather than a claimed result. Reproducibility and the Actual Deployment Cost. The public release deserves a differentiated assessment. The paper-linked artifact repository contains prompts, distilled rulebooks, and per-phrase sensitivity records. Project-page links also lead to a larger development repository with source directories, scripts, experiment records, and historical design documents. That is more useful than a video-only release. It gives another team material with which to inspect the rules and reconstruct the approach. It is also important not to confuse the curated artifact package with a verified, one-command reproduction of every table. I verified the presence of those public materials, not an end-to-end rerun. The development repository includes earlier research designs, so a reproducer needs an explicit mapping from the paper’s final experiment to commits, configurations, and result artifacts. The inherited checkpoint is publicly available, and its model card reports training configuration, including batch size, learning rate, and checkpoint-selection procedure. Since the intervention freezes that checkpoint, reproducing the controller’s original training is not the immediate barrier. Loading exactly the right weights and preserving its action normalization and inference settings are more relevant. For the language pipeline, I would insist on saving generated instructions, scene traces, rulebooks, model identifiers, and decoding settings. The authors’ generation documentation names a floating model alias and records multiple generations of the phrase-building procedure. That makes provenance particularly important: a prompt file alone cannot guarantee the same output from a hosted model months later. The latency is also not zero. The paper reports about 2.5 seconds for scene-trace generation, followed by approximately one to two seconds for the local Qwen applier, ten seconds for Claude, or seventeen seconds for Gemini. Adding those stages gives roughly three-and-a-half to four-and-a-half seconds for the Qwen path, and around twelve-and-a-half to nineteen-and-a-half seconds for the other reported paths. These are startup delays, not additional delays at every policy step. That can be an excellent trade for a long manipulation episode. It can be a poor trade for a short cycle or a command that must take effect immediately. A deployment evaluation should measure completed tasks per unit time, not just success after allowing whatever setup latency the wrapper needs. The local applier also does not make the evaluated system wholly local: it still depends on the upstream scene trace. An offline deployment would need to replace or localize that component and evaluate the resulting system anew. Then there is simulation. SIMPLER is a serious evaluation instrument, not an arbitrary toy environment. Its original work addresses visual and control discrepancies and validates simulated evaluation through more than 1,500 paired sim-and-real trials across two embodiments and eight task families. That supports using it to discover and compare failure modes. It does not automatically validate every subsequent intervention on real hardware. For this method, I would specifically test whether the preferred rewrites survive changes in real camera appearance, occlusion, object labeling, and physical execution. A vocabulary repair that helps in a rendered scene might interact differently with the visual evidence available on a real table. The right conclusion is not “simulation results do not count.” It is that simulation establishes a promising mechanism and a controlled comparison. Real-robot transfer remains an additional empirical claim. What This Means for Practitioners. My assessment is that this is a credible argument for policy-specific instruction calibration, especially when the controller must remain frozen. It is not yet an argument for deploying an unconstrained language rewriter in front of every robot. If I were adapting the idea, I would begin with an audit rather than with a complicated distillation pipeline. Take a representative set of tasks, write multiple genuinely equivalent requests, and measure whether performance changes materially. Include identity-preserving edits such as syntax and politeness, but also carefully verified synonyms and indirect descriptions. Then add counterfactual requests that deliberately change the goal. The same scene should support more than one valid task. If the robot performs the same behavior regardless, paraphrase stability is giving you false reassurance. For intervention comparisons, I would test unchanged instructions, generic rephrasing, the learned rulebook, and a small deterministic normalization baseline. That last baseline is worth including when prominent gains appear tied to a few noun substitutions. If a short dictionary captures much of the benefit, it may be easier to inspect and maintain. If the LLM rulebook does better, the difference helps identify what contextual reasoning contributes. I would also separate the scene-reading component from rule application. Hold the trace fixed when comparing rewriting strategies, then evaluate the full pipeline with fresh scene interpretation. Otherwise, a gain or regression can be incorrectly attributed to the rules when it actually originates in perception. For generalization, leave out more than task identifiers. Hold out object families, lexical substitutions, and scene configurations. Transferring a known substitution into a new pick-and-place combination is valuable, but it is an easier test than learning how to handle a previously unobserved kind of linguistic ambiguity. Report the distribution of effects, not just the average. How many instructions improve? How many previously successful instructions fail? Are failures harmless stalls, incorrect object selections, or unwanted completions? A gain in mean success is insufficient if the wrapper introduces a small but unacceptable class of semantic errors. I would treat meaning preservation as a deployment constraint. Keep the original request alongside the rewrite. Check whether object identities, destinations, relations, and exclusions have changed. When the reference is genuinely ambiguous, clarification is preferable to confidently rewriting the request into something easier for the policy but potentially different from what the user wanted. Finally, version the rulebook with the exact controller checkpoint. A rule is evidence about a particular executor under particular conditions—not a universal law of robot language. Updating the policy should trigger a regression suite for the language adapter, even if the natural-language API appears unchanged. The strongest lesson of this paper is that a robot’s instruction string is an active control input. Benchmark wording can hide capability, expose brittleness, or reward a policy that barely uses language. Watts and Cui show that some of this interface behavior can be measured and improved without retraining. That is a practical contribution. The next step is to establish how broadly the repairs transfer, how reliably they preserve intent, and whether the added language pipeline pays for itself on physical robots. Before concluding that a failed task requires a larger policy, it is worth checking whether the current policy is being asked in a language it can actually use.