Latest / Embodied AI 101 / LIBERO-Para: Paraphrase Robustness in Robotic Manipulation

Transcript

LIBERO-Para: Paraphrase Robustness in Robotic Manipulation · Embodied AI 101

• LIBERO-Para: Paraphrase Robustness in Robotic Manipulation. Modern robots increasingly rely on large vision-and-language models to interpret human instructions. These Vision-Language-Action (VLA) systems combine visual perception with language understanding to carry out tasks like fetching objects or flipping switches. In principle, a robust VLA agent should handle the many ways we say the same thing. In practice, however, HAI-Lab’s new work shows that even minor rephrasings can break these systems. In the proposed LIBERO-Para benchmark, simply substituting synonyms or changing sentence structure can slash task success rates by 22–52 percentage points. In other words, when you tell the robot “Turn on the stove” vs. “Fire up the cooker,” the agent often does entirely different things – or nothing at all. This dramatic paraphrase fragility suggests that today’s fine-tuned vision-language policies rely on surface pattern matching rather than true meaning. To diagnose this issue, the authors systematically created thousands of paraphrased instructions on a standard robotics benchmark (LIBERO) and measured how popular VLA models respond. They find that most failures under paraphrase aren’t due to clumsy control or physics, but simply task misidentification – the model misunderstands what it should do. One striking result: 80–96% of failures occur at the planning level, not execution, meaning the wrong plan is generated at the outset whenever the phrasing deviates from what the model “expects.” In response, the team introduces a new evaluation metric called PRIDE (a Paraphrase Robustness metric) that weights each success by how hard its paraphrase is, since treating all failures as equal hides these nuances. In this episode, we’ll explore how LIBERO-Para works, what it reveals about VLA models, and why paraphrase robustness matters for real-world robots. We’ll break down the two key axes of language variation the benchmark tests, walk through the main findings (including the sharp drops for simple synonyms), and discuss how PRIDE tries to capture “difficulty.” Along the way, we’ll tie this into broader questions of how to ground semantics in embodied agents, and how future models might be made more language-flexible. By the end, you’ll understand why a robot that seems to work well on fixed instructions can abruptly fail when presented with familiar tasks in disguise – and what the robotics community can do about it. Vision-Language Agents in Robotics. Imagine you have a mobile robot assistant in the kitchen. You want it to “place the cup on the counter,” so you give it that exact phrase. But a friend in the same room might say “put the mug up on the table” or “hand me the mug.” Ideally, the robot should recognize these as the same request. In recent years, roboticists have tried to achieve this kind of understanding by leveraging large vision-and-language models. These models (like CLIP, Visual BERTs, or extensions of large language models with vision) are pre-trained on massive image-text datasets and can encode concepts like objects and attributes. The hope is that by fine-tuning such a model on a specific manipulation dataset, the robot can ground natural language commands into actions. In practice, a popular approach is to take a pre-trained vision-language backbone and fine-tune it on a task-specific robot dataset, as in the LIBERO framework and others. This can yield strong performance when the test instructions closely match the training language style. For example, HAI-Lab (the same group behind LIBERO) and others have built VLA models that perform multi-step tasks (opening drawers, pushing plates, flipping switches) by ingesting language goals and camera images. These models indeed can accomplish impressive things on the benchmarks they are trained on. However, when the data is limited and the language in training is uniform, agents may learn to rely on the exact wording of instructions. The recent LIBERO-Para study points out that “in downstream robotic settings, [VLA models are] typically fine-tuned with limited data, leading to overfitting to specific instruction formulations”. In other words, if a task was only ever described as “open the top drawer,” the model might never see alternatives like “open the upper drawer” or “get the bowl out of the top compartment.” As a result, even though the model’s backbone has seen varied language, the fine-tuned policy may forget that synonyms mean the same thing. This problem – the lack of robustness to paraphrasing – has actually been underexplored in embodied AI. Many benchmarks simply measure whether a model can succeed on a fixed set of instructions, without testing if it truly understands the meaning beyond the words. LIBERO-Para confronts this gap head-on by systematically creating paraphrases of each instruction and measuring performance. Their findings reveal exactly how brittle current agents can be when the language strays from the script. The LIBERO-Para Benchmark. The LIBERO-Para project builds on the existing LIBERO-Goal tasks, a subset of the LIBERO lifelong learning benchmark designed for language-guided object manipulation. In LIBERO-Goal, the robot starts from a fixed initial scene configuration for each task, so the only cue of which task to do is the instruction text. This setup is ideal for our purposes: by holding the physical scene constant and only changing the wording, we can isolate the effect of language variation. LIBERO-Para takes each of the 10 base tasks from LIBERO-Goal and generates many paraphrased versions of the instruction, while keeping everything else fixed. The original instructions include tasks like “Turn on the stove,” “Put the cream cheese in the bowl,” and “Open the middle drawer of the cabinet”. In total, LIBERO-Para contains 4,092 paraphrased instructions covering all 10 tasks. On average, there are about 100 paraphrases per variation type. Each paraphrase is held out for evaluation – none of these reworded commands are used in training the models. That means a model that succeeds on the original phrasing of “Turn on the stove” might face versions like “Carefully switch the stove on” or “Could you please activate the burner?” at test time. Because the underlying scene is identical, any change in success rate can be pinned to the linguistic shift in the instruction. Two Axes of Variation. The key insight in creating this dataset is to control two main axes of linguistic change, rather than throwing random rewordings. The authors use an object axis and an action axis, inspired by linguistic taxonomies of paraphrasing. Object axis (3 types): This axis varies how the target object is referred to. For example, take the word “stove” in the command. The dataset considers:. Addition: adding a functional descriptor, e.g. “stove” → “gas stove”. Contextual substitution: a contextual synonym, e.g. “stove” → “cooktop”. Habitual synonym: a common alternate name, e.g. “stove” → “cooker”. So even sticking with the same sentence structure, the object can be called “stove,” “gas stove,” “cooktop,” or “cooker.”. Action axis (10 types): This axis varies how the action part of the sentence is phrased. For instance, instead of just “Turn on the stove,” one could say “Carefully turn on the stove,” or frame it as a question: “Could you turn on the stove?” or embed it in a conditional: “Turn on the stove so that it heats up.” The action axis includes:. Lexical changes: Adding adverbs or using synonyms (e.g. “carefully turn on” or “fire up the stove”). Structural changes: Rewriting grammar (e.g. coordinate clauses like “Go to the stove and turn it on” or subordinate clauses “Turn on the stove so it becomes hot”). Pragmatic changes: Changing the speech act (e.g. “I need the stove on.” as a need-statement, or “Please activate the stove.”, or framing as a hint or permission request). Each instruction is paraphrased along both axes independently. For example, you might see only the object changed (keeping the exact same turn-on phrasing), or only the action phrasing changed (using the same noun “stove”), or both changed at once (e.g. “Could you put the bowl in the cooker?” switching object and phrasing). Across all combinations, this yields 43 distinct paraphrase type combinations (since there are 3 object types × 10 action types plus the “no change” baseline). In practice the data includes all of those plus many examples: roughly 4,092 paraphrase examples in total. The beauty of this design is that researchers can analyze performance at a fine granularity. One can ask: does a model fail mostly when only the object name changes? Or when the sentence structure changes? And how do combined changes compare? Crucially, because each original task has the same initial state across all paraphrases, the visual scene and goal configuration are fixed. The only thing different is the instruction text. Any drop in the robot’s success can thus be attributed to how the model interpreted the new phrasing. Experiments: VLA Models vs Paraphrased Instructions. Armed with LIBERO-Para, the authors evaluated a collection of state-of-the-art VLA agents. In their experiments, they tried seven different VLA configurations ranging from 0.6 billion to 7.5 billion parameters. These include current open vision-language-action models of various sizes (for example, one might correspond to a smaller model like PaLM-E 0.6B, and another to a mid-size backbone around 7B, though the exact model names aren’t central to the story). The point is that they tested both relatively compact and very large models that are representative of the kinds of agents used in robotics research. For each model, they measure success rate on each task under two conditions: (1) the original LIBERO instructions, and (2) the paraphrased versions. If an agent reaches the task goal (e.g. successfully placing the object as asked), it’s marked successful. The key metric is how much the success rate drops when the instruction is reworded. And the result is startling: all models suffer a severe performance hit under paraphrasing. Concretely, success rates drop by around 22 to 52 percentage points across these evaluations. In plain terms, a model that was 80% successful on the original phrasing might fall to as low as 28% on the paraphrased instructions (a 52-pt drop). Even a smaller drop of 22 points could mean going from, say, 90% success to 68%. These are not minor blips — they are the difference between a reliable system and a mostly broken one. The worse drop (52 points) came in some cases with larger paraphrase changes, but even with simpler paraphrases the hit was substantial (on the low end, 22 points). Figure-like (hypothetical) example: if “Turn on the stove” had original success 95%, a 50-point drop might mean the agent now only succeeds ~45% of the time on “Fire up the cooker”. In practical terms, half of the time the robot would misinterpret that alternate phrasing and do something wrong. That’s a crushing failure rate for what we’d consider an obvious restatement. Importantly, this degradation was consistent across all model sizes. The 7.5B model (likely the best-performing in the original scenario) still crashed hard under paraphrasing. In other words, simply scaling up the model didn’t magically fix the sensitivity to language variation. All configurations – from relatively small to quite large – showed significant drops in their success rates when faced with paraphrased instructions. What Goes Wrong? Task Misidentification. To understand why these failures are happening, the authors dug into the nature of the errors. They classify failures into two types: planning-level errors vs. execution-level errors. A planning error means that the agent formed the wrong plan from the start, e.g. it went to or moved the wrong object. An execution error means the plan was correct but the robot bungled the mechanics (e.g. it tried to pick up the right object but dropped it or collided). After analyzing the trajectories, they found that somewhere between 80% and 96% of the paraphrase-induced failures were planning errors. In other words, the agent was almost always heading towards the wrong goal when the instruction was paraphrased. It wasn’t struggling to grasp a bowl or nudge a plate; it was simply "reading" the instruction incorrectly and deciding on the wrong action sequence. This directly speaks to task misidentification. The robot effectively hears a paraphrased command and thinks it's a different command entirely. Concretely, suppose the original task is “Put the wine bottle on the plate.” If paraphrased to “Place the wine bottle onto the dish,” maybe the model correctly understands "wine bottle" but misinterprets "dish" as some other object (say, the stove). The solution it plans is therefore completely different from the intended one. By the end, its trajectory diverges from the expected goal – it might wander toward the stove instead of the plate, or try to grab something else – which the analysis flags as a planning failure. In contrast, if it had gone to the correct place but then fumbled, that would have been an execution error, which was rare. This finding tells us exactly the crux of the weakness: paraphrasing mainly throws off the semantic understanding of the instruction. The agent’s controller works fine once it interprets the goal; it’s the interpretation that breaks. The paper phrases it succinctly: “paraphrasing disrupts task identification”. Put another way, the robot doesn’t know which task to do when the words change, so it picks a plan that doesn’t even match the original intent. Lexical vs. Structural Shifts. Digging deeper, the authors identify that the biggest drops seem tied to object-level lexical variation. That is, simply swapping object names (like “stove” → “cooker”) caused huge performance hits. Even though “stove” and “cooker” obviously refer to the same item in context, many VLA models failed to associate them correctly. The paper notes that “even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding”. In essence, the model was probably expecting to see the exact word “stove” or whatever brand names the data contained, and didn’t fully connect it with “cooktop.”. Action paraphrasing (changing sentence structure or formality) also hurt performance, but to a slightly lesser extent. For example, adding a polite prefix or reordering clauses might confuse the tokenizer or the language input in a way that muddles the essential action phrase. The data suggests that the model might handle simple structural changes somewhat better than object synonyms, but both axes cause problems. It’s worth stressing that the object vocabulary mismatches seem to drive most of the trouble. If the robot hears “on the sink” vs “in the bowl,” it definitely acts differently; similarly “glass of wine” vs “wine glass” might trick it. All told, this means today’s VLA systems are overfit to the exact phrasing seen in training. They do not, in practice, “understand” that different words can mean the same thing in context. Instead, they lean on lexical cues. This is a classic failure of generalization. The robot never saw “cooker” in training, so it doesn’t know to map “cooker”→“stove.” In a sense, the groundings it learned are brittle: if you change the signal, the network doesn’t see the same vector. The bottom line from this analysis is sobering: the success gaps on paraphrases suggest that many VLA models are merely matching keywords, not truly reasoning about object identities. They largely ignore synonyms and contextual clues. This is precisely what the authors mean by “surface-level matching rather than semantic grounding”. The Pitfall of Binary Metrics and the PRIDE Score. So far we’ve talked about raw success rates dropping a lot. But the authors also point out a more nuanced evaluation issue. Standard evaluation would simply compute a binary success rate (1 for success, 0 for failure) averaged across all paraphrase cases. That treats an easy paraphrase the same as a hard one. But not all paraphrases are equally challenging. Gluing the word “please” in front might not change much, whereas turning an imperative into a question form might break the model. Giving all failures lumped together hides this nuance. To address this, HAI-Lab introduces PRIDE, a metric intended to reflect paraphrase difficulty. In essence, PRIDE weights each instruction by how linguistically different it is from the original. A paraphrase that is nearly identical to the original command gets a low difficulty score; one that is syntactically or semantically far away gets a higher score. Then the agent’s success is measured in a weighted way, so failing on a “hard” paraphrase counts more against it than failing on an “easy” one. The hope is that PRIDE will give a more informative picture of robustness: does a model only win on the trivial rephrasings or can it handle the tough ones?. According to the authors, PRIDE quantifies difficulty using both semantic and syntactic factors. While the paper doesn’t provide the exact formula in the parts we’ve read, we can infer the idea: they likely assess how similar the paraphrase is to the original in meaning (perhaps via word embeddings or a synonym database for objects) and how complex the sentence transformation is (for instance, comparing parse trees or counting structural changes). For example, adding an adjective (“turn on the stove” → “turn on the gas stove”) is mostly a minor semantic shift, whereas restructuring it (“turn on the stove” → “going to the stove and turn it on”) is more syntactically complex. PRIDE would rate the latter as a more difficult paraphrase. When computing the overall score, each example’s success would be weighted by this difficulty. The paper argues that using PRIDE uncovers patterns that raw success misses. A model could have, say, a 50% success under paraphrasing, but that might be because it aced all the easy “please do X” cases and failed the “X if Y then do Z” cases. PRIDE would reveal that discrepancy by penalizing the model more for failing the latter. In practice, the work shows that with a plain success metric, we might falsely think a model is reasonably robust, hiding the fact that it only does well on superficial paraphrases and not on more creative rewordings. PRIDE is their tool to make this explicit. In summary, the authors recognize that simply counting successes treats all instructions equally, which can obscure whether a model is genuinely understanding language or just lucking out on easy variants. The PRIDE metric is their proposal to weight each case by how challenging the paraphrase is according to linguistic measures. Key Findings. Putting it all together, the LIBERO-Para experiments yield several headline takeaways:. Dramatic Performance Drop: Success rates plummet for paraphrased instructions. Across seven different VLA models, every one saw its success rate fall by 22–52 percentage points under paraphrase. This confirms quantitatively that paraphrase vulnerability is not a minor bug but a major weakness. Surface Matching, Not Semantics: The primary cause is that models lean on literal wording rather than deep semantics. Even swapping a single word for a close synonym can derail the plan. This suggests the agents do not truly ground the meaning of object names, but rather trigger behaviors on specific keywords. Task Misidentification (Planning Errors): Most errors occur because the model generates the wrong plan, not because it executes poorly. In fact 80–96% of paraphrase-induced failures are planning-level errors. In practice this means the agent literally misunderstands which task to do when instructions are phrased differently, rather than stumbling in execution. Model Scale Is Not a Panacea: Larger models did not solve this problem. Even the 7.5B-parameter models suffered similarly to the smaller ones. This implies that the issue isn’t simply about representational capacity; it lies in the fine-tuning and the nature of the training data. Hiding of Failures by Aggregate Metrics: If we only look at raw success rates, we miss how those failures are distributed among easy vs. hard paraphrases. A model might look OK on average while actually failing systematically on certain types. The authors’ PRIDE metric is designed to expose these differences. Overall, the result is clear: VLA agents today are not robust to natural language variation. We might naively expect that because these are neural models trained on language, they would at least handle common synonyms or polite remodelings of commands. But in the robotics fine-tuning case, that generalization doesn’t hold. The agents can do the job if you speak their precise training dialect, but freezing that phrasing ruins their performance. One might wonder: is this just a quirk of the LIBERO tasks, or a more general phenomenon? Many robotics tasks fix both the initial state and the way goals are described. In the wild, users will not stick to the same template. The LIBERO-Para results strongly suggest that until we address this brittleness, real-world application could suffer. For example, a disabled user who is helped by a home robot might naturally rephrase instructions (especially over time), but the robot might fail them because it never learned those synonyms. Using LIBERO-Para and Addressing the Issue. To make this diagnostic work useful to others, the authors have released LIBERO-Para and associated code under an open-source license. The paraphrased instructions are provided as BDDL files (the domain description language used by LIBERO) that can plug into the same evaluation pipeline. Researchers can download the paraphrase files and run them through the LIBERO simulator environment using their own models. The expectation is that people working on VLA will start including LIBERO-Para in their test suite as a stress test. As the authors note on Huggingface, you simply place the provided files into your LIBERO setup and run the evaluation scripts from their repository. This should make it straightforward to reproduce their findings or compare new models. What can model developers do with this information? The obvious answer is to train with more linguistic variety. If the root cause is that fine-tuning data was too limited in phrasing, then augmenting the dataset with synonyms and restructured sentences could help. For instance, an automatic paraphrase generator or crowd-sourced data collection might expand each instruction to include many variants, so the model sees all these mappings early. Data augmentation with paraphrases is a well-known trick in NLP for robustness; it makes sense to apply it here. Once we know synonyms get missed, explicitly training on those synonyms should reduce the drop. In other words, the model needs kaleidoscopic examples of how tasks can be described. Another approach is to leverage the semantic capabilities of the underlying language model more directly. Some VLA architectures use a separate language model (like a pretrained LLM) that generates action plans token-by-token. Potentially, one could incorporate semantics-aware components: for example, during instruction understanding, use a contextual embedding model that aligns synonyms. If a system knows that “cooktop” is closely related to “stove” in embedding space (from its pretraining), it could map these to the same object class. That would make the policy robust to synonym swaps. Perhaps the fine-tuning process should include a semantic similarity loss or use prompts to check consistency. However, the paper is largely diagnostic, so it stops short of prescribing a one-size-fits-all fix. The main contribution is revealing the failure and providing the benchmark. Still, the authors make clear that any fair evaluation of VLA models should include diverse language. Without that, we don’t really know if a high success rate reflects true language understanding or just memorization of phrasing. One caution they note (via an emergent mind commentary) is that all these experiments were done in simulation. Simulation environments, however detailed, are still simpler than the real world. Things like sensor noise, lighting changes, or unforeseen object arrangements can further confuse a robot. The authors themselves acknowledge that real-world paraphrase robustness is an open question. It may be that in real robot deployment, the problem is even worse: in addition to paraphrase variance, the robot has to cope with camera noise and imperfect actuators. So while LIBERO-Para is controlled, it highlights a vulnerability – one that almost certainly will carry over or even amplify on a physical robot. The bottom line: letting robots hear different phrased commands in real-world tests is an urgent next step. Broader Context and Future Directions. This work on LIBERO-Para sits at the intersection of robotics, language understanding, and benchmarks. A similar spirit underlies other diagnostic studies in robotics (e.g. varied lighting or simulated shifts to test vision robustness) and in NLP (paraphrase evaluation). But in robotics, language variation hasn’t been systematically tested before. The finding here – that state-of-the-art VLA agents are not robust to paraphrase – is likely to resonate with practitioners who may have seen odd failures in informal trials. It also dovetails with emerging concerns about overfitting in embodied tasks. For example, some prior observations hinted that training on only fixed initial scenes or limited objects can hide gaps in generalization. This new study shows an analogous gap in language. Looking ahead, a few paths seem clear:. Incorporate Paraphrase Data in Training: Robotic datasets should include linguistic diversity. Future benchmarks or data collections could explicitly demand multiple ways to ask for each goal. One might even create adversarial paraphrases during training to harden the model (a bit like adversarial training in vision, but with language). Semantic Grounding Enhancements: Algorithms could combine symbol grounding with vector semantics. For instance, recognizing that “cream cheese” and “cheese” are related or that different modifiers refer to the same object. Techniques from grounded language learning, where objects have attribute labels, might help. Better Language Models: Perhaps hooking up a larger pre-trained LLM that has seen paraphrases could help. If the policy network queries an LLM to interpret instructions, the LLM might correctly paraphrase the command into a canonical form for the policy. Essentially, use a second language system as a translator. We saw some works like SayCan that leverage LLMs as planners; here, an LLM might normalize language first. Evaluation Standards: On the evaluation side, the idea of PRIDE is a nice reminder that we need richer metrics. Other aspects of language (e.g. negation, disjunction) could be tested in a similar fashion, and new benchmarks may extend LIBERO-Para’s approach. Real-world Paraphrase Testing: Finally, of course, putting these models on a real robot with real human users would be the ultimate test. The good news is that LIBERO-Para provides a structured way to sanity-check models before physical deployment. Developers can run their agent on the paraphrase set and fix glaring holes early. Conclusion. The LIBERO-Para study shines a spotlight on a hidden weakness in current robotic language-understanding systems. By systematically paraphrasing instructions in a robotic manipulation benchmark, the authors reveal that vision-language-action models often collapse in performance – especially when object names change – because they reliance on lexical cues. In their tests, success rates fell by up to half simply by rewording commands, and most errors were due to misunderstanding the task rather than execution blunders. This work underscores that true linguistic generalization remains a big challenge for embodied AI. It reminds us that a robot that passes a fixed-instruction benchmark isn’t guaranteed to handle the variability of natural language. As one takeaway: if your home robot only understands one way to ask, it’s not ready for everyday use. The PRIDE metric they propose is a step toward better evaluation, but the larger lesson is that training and model design need to prioritize semantic robustness. The authors have provided the community with an open benchmark (and code) to measure paraphrase robustness. We encourage practitioners to include LIBERO-Para in their evaluation pipelines. With these tools, future VLA models can be tested – and (hopefully) strengthened – for true understanding. Ultimately, enabling robots to follow any of the many ways we might phrase the same goal is essential for fluent human-robot interaction. This paper is a timely wake-up call that we still have a lot of work to do on that front. References: The figures and statistics here are drawn from “LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models” by Kim et al., 2026; details of the benchmark design appear on their Huggingface dataset page. The EmergentMind analysis also highlights the key findings and notes the need for real-world follow-up. All cited quotes and data are from these sources.