Latest / Embodied AI 101 / A Patch Only Needs a Glimpse: Breaking VLA Robots Under Partial Observability
Transcript
- A Patch Only Needs a Glimpse: Breaking VLA Robots Under Partial Observability. Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics • Xiaofei Wang, Mingliang Han, Tianyu Hao, Yi Yang, Yun-Bo Zhao, and Keke Tang • University of Science and Technology of China, SmartMore Corporation, Guangzhou University, Chinese PLA General Hospital, and Hefei Comprehensive National Science Center • IEEE Robotics and Automation Letters • 2026. The paper in one sentence. This paper asks whether an attacker who sees only the beginning of a robot’s execution—not the complete rollout—can construct one static visual patch that derails everything that happens afterward. Its answer is yes: use the observed prefix to identify where the VLA is looking, corrupt its grounding of instruction nouns, and steer its actions toward a direction that creates an abrupt bend in the end-effector trajectory. That is a meaningful change from the first generation of adversarial-patch studies for vision-language-action models. Earlier attacks demonstrated that visual patches could badly degrade robotic performance, but commonly optimized those patches using the full sequence of observations. That is a useful robustness stress test, yet an unusually powerful adversary: in a real deployment, future images depend on actions that have not happened yet. Wang and colleagues replace this with an attacker-observability model in which only the first (K) frames, actions, and states are available. The learned patch must then generalize to the unseen remainder of the episode. The result is not a black-box sticker attack that anyone can perform after casually watching a robot. The adversary still has substantial model and system access. But the paper shows that complete future knowledge is unnecessary. A short trajectory prefix, combined with internal VLA signals and a control-aware objective, is enough to produce persistent closed-loop failures. Why partial observability changes the security question. The phrase “partially observable” can be slightly misleading to a robotics audience. This is not primarily a new POMDP formulation of the robot’s policy. It refers to what the attacker can observe. The robot continues to receive its normal camera image and language instruction, while the attacker sees only an initial segment of the execution. Formally, the victim VLA maps the current image and language instruction into an action. Those actions change the environment state, which produces the next image. Under full observability, an attacker gets the entire clean trajectory in advance and optimizes a patch over every frame. Under the new setting, the attacker sees only the first (K) state-action-observation tuples, optimizes a patch from those samples, freezes it, and applies it to frames (K+1) through the end of the rollout. Evaluation is performed on that unseen suffix. This formulation captures an important property of embodied attacks: the perturbation need not predict every future image if it can exploit persistent weaknesses in the policy. A patch that consistently attracts semantic attention or biases action direction becomes part of the feedback loop. The first bad action alters the scene. The changed scene produces a different next action, and the state distribution begins moving away from anything represented in the original prefix. The threat model is described as partial-access gray-box. The adversary can query the VLA and use model-side information required for optimization, but cannot alter the model parameters, directly command the robot, or modify the low-level controller. It can place one static physical patch in the visible workspace. In practice, the method also needs cross-modal attention, gradients through the model, instruction parsing, and end-effector state information. That is weaker than controlling the robot, but considerably stronger than ordinary inference-only API access. The most plausible actors are therefore an insider, compromised system integrator, malicious model-service operator, or red team with access to the robot stack. The formulation is less representative of a completely external attacker who knows neither the architecture nor the robot telemetry. From a short prefix to a long-horizon failure. The method has two stages. The first chooses where the patch should appear. The second chooses what the patch should contain. Its central design decision is to attack both sides of the perception-action interface rather than merely maximizing generic output error. Stage one: place the patch where language meets vision. Each image is represented as a grid of visual tokens. Using the final frame of the observed prefix, the attack extracts cross-modal attention between the instruction tokens and those visual tokens. Attention is summed across heads, reshaped into the two-dimensional visual-token grid, and aggregated by taking the maximum response over the instruction tokens at each location. The algorithm then slides a square window over that saliency map and chooses the contiguous region with the greatest total attention. In the default configuration, the selected window covers three by three visual tokens, corresponding to a 42-by-42-pixel image patch. All subsequent gradient updates are confined to this mask. Using the maximum across words rather than an average means that a strong response to any instruction token can make a region eligible. This avoids diluting the attention assigned to one critical object, although it may also make localization sensitive to a single noisy token or attention head. The decision to use only the last prefix frame is empirically motivated. Averaging attention over all prefix frames produces a temporally smoother estimate, but it also mixes observations from different robot and object configurations. At (K=30), last-frame localization achieves an average attack success rate of 81.0%, compared with 77.7% for prefix-mean localization. The corresponding normalized attack scores are 89.3% and 86.6%. The difference is smaller at (K=10), but the final frame remains better overall. This result is intuitively reasonable. The latest image gives the best estimate of what will remain visible when the patch takes effect. It also suggests that temporal freshness matters more than reducing the variance of the attention estimate. Stage two, part one: displace semantic grounding. The first content objective targets object grounding. The instruction is parsed into task-relevant nouns, and each noun is mapped to the corresponding subword tokens. For every frame in the prefix, the attack constructs a clean attention map and a patched attention map for each noun. A soft argmax converts each map into a focus location, and the optimization tries to move the patched focus away from its clean position. For an instruction such as “put the bowl on the stove,” the attack is not directly asking the policy to predict a particular wrong action. It is attempting to change where the visual-language stack believes “bowl” and “stove” are grounded. Once that association shifts toward the patch, the background, or an adjacent object, the action head receives a corrupted semantic representation. The authors are appropriately cautious about interpretation. They describe attention displacement as an operational proxy for altered grounding, not proof that attention is a causal explanation of the policy’s decision. Their visualizations nevertheless show noun-related peaks moving away from bowls, books, drawers, and other intended referents after the patch is introduced. Ablations also show that removing this semantic term generally weakens the attack, especially at the more effective prefix length of 30 frames. There is an important embodied insight here. A standard image attack might seek a large feature-space deviation, but a VLA does not need to misidentify everything. Confusing one task noun can be sufficient. If a policy grounds “book” to the wrong region during a long-horizon instruction, every subsequent reach, grasp, and placement decision can be compromised. Stage two, part two: manufacture an abrupt trajectory bend. The second objective is control-oriented. From the observed states, the attack extracts the prefix of end-effector positions. It then samples candidate future directions uniformly on the unit sphere, with direction magnitudes normalized to the robot’s typical action scale. For each candidate, the method appends a hypothetical straight-line continuation to the observed trajectory and fits a cubic spline through the combined path. It computes the maximum curvature of that spline and compares it with the maximum curvature of the prefix alone. The candidate producing the largest increase becomes the adversarial target direction. The patch is then optimized so that the VLA’s patched action directions align with this target. Although the candidate suffix is straight, it can be sharply incompatible with the tangent of the existing path. The spline smooths the join, creating a high-curvature transition near the end of the prefix. In effect, the method searches for a direction that forces the robot to turn as abruptly as possible relative to what it was just doing. This is computationally much cheaper than simulating every candidate through the environment. It requires no learned dynamics model and no access to the true future trajectory. Curvature is only a surrogate—it is not equivalent to failure—but it offers a task-agnostic way to turn limited historical motion into a disruptive future direction. Once that direction is injected into a closed-loop policy, several behaviors can emerge. The robot may overshoot, attempt to correct itself, see a changed image, and then be pushed away again by the persistent patch. The paper’s qualitative rollouts show oscillations, detours, sharp bends, and repeated failed corrections rather than a single isolated action error. On LIBERO-Long, most first-step angular deviations after patch insertion fall between 30 and 60 degrees. Combining perception and control. The final objective is a weighted combination of semantic displacement and trajectory-direction alignment. The default semantic weight is 1, while the trajectory weight is 12. The patch is optimized for 50 SGD iterations with an initial learning rate of 20, decayed by 5% per iteration. The authors sample 100 candidate trajectory directions by default. The combination matters because the two losses create different failure pressures. Semantic corruption makes the robot act on an incorrect interpretation of the scene. The trajectory term makes the resulting control output geometrically difficult to recover from. At (K=30), the complete objective outperforms either single-component variant across all four suites in attack success rate. The advantage is particularly large on LIBERO-Long, where the full attack reaches 86.6% versus 58.5% without the semantic term and 73.0% without the trajectory term. The ablation is less uniform at (K=10). For example, removing the trajectory term slightly improves Spatial ASR, and removing the semantic term slightly improves Goal ASR. The correct reading is therefore not that every loss is beneficial in every regime. Rather, once the prefix contains enough information, the combined attack becomes consistently stronger. Experimental design. The principal simulation benchmark is LIBERO, using its Spatial, Object, Goal, and Long suites. Each suite contains ten tasks. The paper reports 100 trials per task and maximum horizons of 220, 280, 300, and 520 steps for the four suites. The primary victims are four suite-specific OpenVLA models, with Hume checkpoints used to test another VLA architecture. The baselines come from the earlier ICCV study of VLA adversarial vulnerabilities:. UADA, or Untargeted Action Discrepancy Attack, maximizes disagreement between normal and attacked actions. UPA, or Untargeted Position-aware Attack, targets spatial movement direction. TMA, or Targeted Manipulation Attack, pushes action dimensions toward an attacker-selected target. All baselines are given the same prefix length and number of optimization iterations as the new method. The comparison therefore asks how well each objective adapts when future observations are removed, rather than reproducing the baselines’ original full-data training schedules. The first metric, ASR, is simply one minus the attacked task success rate. The second, normalized ASR, also credits slowdowns. A failed task receives a score of one. If the robot still succeeds, it receives a fractional score based on how much of the available time budget the attack consumes beyond the benign completion time. This is useful because an industrial robot that eventually succeeds after a lengthy detour may still represent a serious availability or throughput failure. Main results: moderate prefixes are enough. At a prefix length of 30, the method is substantially stronger than all three baselines:. | LIBERO suite | Proposed ASR | Best baseline ASR | Improvement | Proposed nASR | |---|---:|---:|---:|---:| | Spatial | 73.8% | 59.1% — TMA | +14.7 points | 87.5% | | Object | 90.7% | 63.8% — UADA | +26.9 points | 96.0% | | Goal | 72.8% | 59.5% — UADA | +13.3 points | 79.6% | | Long | 86.6% | 57.2% — UPA | +29.4 points | 93.9% | | Average | 81.0% | 58.9% — UADA average | +22.1 points | 89.3% |. These values are computed from the paper’s Tables I and II. The strongest gains occur on Object and Long. Object-centric tasks are naturally exposed to semantic-grounding corruption, while long-horizon tasks provide time for small directional errors and failed corrections to accumulate. Spatial tasks are harder to disrupt because their motions are short and tightly constrained; the policy can sometimes correct a deviation before task completion. The relationship between prefix length and attack strength is not monotonic. Averaged across suites, ASR rises from approximately 58.7% at (K=10) to 81.0% at (K=30), but it falls slightly at (K=40). A longer prefix provides better localization and trajectory context, yet leaves fewer future steps to attack. LIBERO-Long is the exception: its ASR continues rising to 89.1% at (K=40), because even after observing 40 frames the attack still has a long suffix in which to compound errors. The high normalized scores at short prefixes are also revealing. At (K=10), average ASR is roughly 58.7%, but nASR is 80.1%. Even when the attack does not force outright failure, it frequently causes enough detours and corrective motion to consume a significant part of the rollout budget. Ablations and mechanism checks. Attention-guided placement consistently beats random placement across the ten LIBERO-Goal tasks. This is evidence that the localization stage contributes something beyond simply giving the attacker a colored square somewhere in the image. A stronger control would still be valuable, however: for example, a neutral or random-texture patch placed in the same high-attention region would help distinguish adversarial optimization from ordinary occlusion. The number of sampled trajectory directions has a clear saturation point. Increasing the count from 50 to 100 raises average ASR from 44.6% to 80.1%. Increasing it further to 200 yields only 80.7%, while adding computation. The authors therefore choose 100 directions. Their spline-based ranking also reports more than 90% top-five agreement across several prefix lengths, although the paper provides limited explanation of the exact reference used for that agreement statistic. On Hume, the full method reaches an average ASR of about 84.2% at (K=30), with individual suite results between 82.2% and 86.3%. This is useful evidence that the attack is not an OpenVLA-only artifact. It is not, however, a transfer attack: the attacker still optimizes against the Hume model and its accessible internal signals. Patch generation takes between 47 and 53 seconds on a single H100 at (K=30). The paper compares this with 351 to 783 seconds for UADA-style optimization over complete rollouts, reporting a 6.9-to-16.7-times speedup. The geometric surrogate is therefore not only more compatible with partial observability but much cheaper than optimizing across every future frame. Two simple preprocessing defenses accomplish very little. At (K=30), average ASR is 81.0% without defense, 79.2% after JPEG compression, and 80.2% after Gaussian blur. In some suites, preprocessing actually produces a slightly higher attack rate. This is a narrow defense evaluation, but it shows that the patch is not merely exploiting fragile high-frequency pixels that disappear under basic filtering. The physical-robot result. The physical evaluation uses a ROKAE xMate ER7 Pro arm, a Robotiq 2F-85 gripper, an overhead Intel RealSense D455 camera, and OpenVLA-7B. The optimized patch is printed as an 18-by-18-centimeter square and mounted on a rigid support, approximately facing the camera. Two manipulation tasks are evaluated for 50 benign and 50 attacked trials each. For “put the ranch bottle into the pot,” success falls from 72% to 12%, a 60-percentage-point drop. For “move the tiger from the red bowl to the gray bowl,” it falls from 58% to 8%, a 50-point drop. The attacked failure rates are therefore 88% and 92%. These are large effects, and the repeated trials make the result more convincing than a handful of selected videos. At the same time, the experiment tests one robot, one camera configuration, two tasks, a conspicuous patch, and a deliberately favorable front-facing orientation. It does not characterize robustness to large viewpoint changes, lighting variation, partial occlusion, print-to-print variation, or moving cameras. Another important distinction is that the physical experiment validates the physical realizability of the optimized pattern, not necessarily the complete logistics of the formal online threat. The demonstrated patch is printed and mounted before execution. The paper does not show an attacker watching 30 frames of a live episode, spending roughly 50 seconds optimizing, printing the result, and inserting it into the same rollout. A more plausible interpretation is that a prefix is obtained from a prior or repeated execution and used to attack later runs of the same task. What the paper genuinely contributes. The strongest contribution is the threat-model shift. The paper turns “we can break a VLA if we know the whole movie” into “we can break the unseen future after watching only the opening scene.” This is much closer to how an attacker, evaluator, or compromised service would actually acquire data. Second, the attack is designed around embodied failure rather than generic neural-network error. Semantic displacement addresses the cross-modal grounding layer, while the curvature objective addresses physical execution. Neither requires knowing the exact future goal state. Together they exploit the fact that a VLA’s outputs modify its own future inputs. Third, the work demonstrates that long-horizon robustness cannot be inferred from one-step action stability. A patch may produce an action that looks recoverable in isolation, yet repeated exposure can keep pulling the controller away during every correction. The high nASR values make this temporal degradation visible even when the robot eventually succeeds. Finally, the physical results show that the vulnerability is not eliminated by camera capture, printing, and robot dynamics. The evaluation is controlled and limited, but the size of the observed performance drops warrants serious attention from anyone deploying VLA policies in unattended or semi-public workspaces. A critical reading of the threat model and evidence. The partial-observation assumption is more realistic than full-rollout access, but the remaining adversary is powerful. It needs the instruction, prefix images, end-effector trajectory, internal attention maps, and differentiable access for patch optimization. The attack therefore does not establish vulnerability to an unknown, remotely hosted, inference-only VLA. The authors explicitly acknowledge that fully black-box policies remain outside the method’s scope. Several implementation details are also underspecified. The paper does not explain how task nouns are extracted, which transformer layer supplies the cross-modal attention, how the method is adapted to different attention architectures, or exactly how pixel feasibility and clipping are enforced. No code or dedicated project page is identified in the version reviewed. There is also an apparent sign-convention inconsistency. The general attack problem is written as maximizing the adversarial objective. Yet the semantic component is expressed as negative grounding displacement, and the trajectory component as negative cosine alignment. Those expressions produce the stated behavior if minimized, which is consistent with conventional PyTorch SGD, but not if literally maximized. The intended algorithm is understandable, but a reimplementation needs to resolve this discrepancy from the prose or future code release. The simulation metric deserves care. ASR is defined as one minus attacked success rate, rather than the fraction of initially successful episodes converted into failures. It therefore includes any failures the benign VLA would already have produced. The paper does not place clean simulation success rates beside the attack tables, making it difficult to separate attack-induced failure from baseline unreliability. The physical experiment is stronger in this respect because it directly reports benign and attacked success under the same tasks. A better primary simulation metric would condition on paired episodes that succeed without attack and then report how many fail when patched. Reporting the paired change in completion time, collisions, workspace violations, peak velocity, jerk, contact force, and minimum human distance would also connect task failure more directly to robotic safety. The present experiments establish a reliability and availability vulnerability, but they do not by themselves demonstrate injury or hardware damage. The reported rollout count is slightly ambiguous as well. Ten tasks per suite and 100 trials per task imply 1,000 rollouts per suite, or 4,000 across four suites, while the text says this setup yields a total of 1,000 rollouts. Confidence intervals or per-seed variance are not reported for the principal tables. Finally, the defense study is intentionally basic. JPEG compression and blur are useful sanity checks, not adaptive defenses. They say little about adversarial training, patch segmentation and inpainting, multi-view perception, temporal consistency checking, action shielding, or an attacker who explicitly optimizes through the defense. What robotic security teams should take from it. The first lesson is that workspace integrity has become part of the VLA security perimeter. Traditional robot security focuses heavily on network access, ROS nodes, controller permissions, and command authentication. A VLA also treats every visible object as input to a learned control program. Unrecognized signs, stickers, screens, packaging, and tools should therefore be handled as potentially untrusted data. A practical system could maintain a model of the expected scene and flag newly introduced high-saliency regions. Fixed workcells could use authenticated fiducials or calibrated object inventories. Mobile and service robots would need more flexible novelty detection, but even simple temporal change detection could identify a patch that suddenly appears near a task-relevant region. Second, defenses should test semantic consistency rather than relying only on image preprocessing. The attack is constructed to move noun-related grounding, so one possible monitor is disagreement among multiple grounding mechanisms: VLA attention, an independent open-vocabulary detector, depth or segmentation, and a second camera. If “bowl” jumps to a background patch in one model but not the others, the controller should pause rather than act. Third, the action stream needs an independent safety layer. The paper reports immediate direction changes commonly in the 30-to-60-degree range and deliberately seeks high-curvature motion. A model-based shield could reject sudden direction reversals, enforce curvature and jerk limits, collision-check proposed actions, or compare the VLA output against a conservative local planner. Such a shield will not defeat every adaptive attack, but it prevents semantic errors from directly becoming unrestricted motion. Fourth, temporal evaluation should become standard. Robustness testing cannot stop at perturbed-image accuracy or one-step action distance. Patches must be evaluated over closed-loop rollouts, across task horizons, camera trajectories, corrective behavior, and repeated exposure. Metrics should distinguish clean policy failure, attack-induced failure, delay, unsafe contact, and recovery. Finally, training-time defenses should expose VLAs to semantically placed patches across entire trajectories. Random Cutout or generic image augmentation is unlikely to be sufficient. The difficult cases are patches positioned near high-attention objects, optimized to remain effective during state-distribution shift, and paired with objectives that attack both grounding and control. Where this sits in the growing VLA attack landscape. The preceding ICCV work by Taowen Wang and colleagues established UADA, UPA, and TMA and showed that patch attacks could cause severe failures in simulation and on physical robots. The new RA-L paper makes that line of attack more observation-efficient by replacing full-rollout objectives with semantic and geometric surrogates learned from a prefix. Eliot Jones and colleagues attack a different surface: the language channel. Their work adapts LLM jailbreaking methods to find text inputs that provide broad access to a VLA’s action space, with effects that can persist after a single instruction-level attack. This demonstrates that visual patches are only one member of a multimodal threat family. BadVLA moves to the supply chain. Instead of optimizing a test-time perturbation, it implants a training-time backdoor that preserves clean performance but triggers control deviations when a visual object or pattern is present. That threat is especially relevant when fine-tuning and training are outsourced. The CVPR 2026 paper When Robots Obey the Patch studies yet another axis: universal transfer under unknown architectures, fine-tuned variants, camera changes, and sim-to-real shift. Its UPA-RFAS method targets features, attention, and image-text alignment to obtain black-box transfer. The key distinction is useful: the partial-observability paper asks how little trajectory information an attacker needs against an accessible model, while UPA-RFAS asks how little model access is required when one patch must transfer broadly. A realistic worst-case attack would combine both properties. Bottom line. This paper does not show that any stranger can place a tiny sticker near any robot and seize control. Its attacker has internal model access, robot telemetry, a stable camera view, and the opportunity to optimize a task-aware patch. What it does show is still important: full future knowledge is not the barrier one might hope it is. A VLA exposes structural signals—language-conditioned attention and recent motion geometry—that allow an attacker to convert a short observation prefix into a persistent visual control disturbance. On LIBERO, the result is an average attacked failure rate of roughly 81% at a 30-frame prefix. On the physical robot, two tasks lose 50 to 60 percentage points of success. For VLA deployment, visual robustness, semantic grounding, and control safety can no longer be treated as separate engineering concerns. A weakness at their intersection is precisely what allows one static patch to turn a momentary perceptual error into a long-horizon robotic failure.