Latest / Embodied AI 101 / Restore the Sensor Contract: Training-Free Robustness for Brittle VLA Policies
Transcript
- Restore the Sensor Contract: Training-Free Robustness for Brittle VLA Policies. Restoring the Right Stream: Training-Free OOD Robustness for Vision–Language–Action Policies • Zhiyin Yan and Gang Shen • Entropy, Volume 28, Issue 9, Article 975 • 2026. When a 95% Robot Becomes a 20% Robot. A manipulation policy sits in front of a familiar tabletop. The same robot is present, the same objects are available, and the language instruction requests the same skill seen during evaluation yesterday. But the external camera has moved slightly to one side. Or the table texture has changed. Nothing about the underlying task has become fundamentally harder. The policy nevertheless collapses. That is the failure mode behind Zhiyin Yan and Gang Shen’s Restoring the Right Stream. The paper starts from an increasingly uncomfortable observation in vision–language–action research: benchmark competence and operational robustness are not the same property. A policy can nearly saturate the original LIBERO suites while remaining acutely dependent on a fixed camera pose, a familiar background, or a narrow range of robot initial states. LIBERO-Plus made this problem difficult to dismiss. It extends LIBERO with controlled perturbations along seven dimensions: object layout, camera viewpoint, robot initialization, language, lighting, background appearance, and visual noise. Across more than 56,000 generated scenarios, the benchmark finds that camera and robot-state changes are especially damaging. Models that approach 95% success in familiar conditions can fall below 30% after modest perturbations. It also uncovers less obvious pathologies, including positional shortcuts, weak sensitivity to language, trajectory memorization, and negative interactions between simultaneously applied shifts. Yan and Shen ask a deliberately pragmatic question. Suppose we already have a VLA checkpoint that performs well in distribution. Suppose retraining is expensive, the original data mixture is unavailable, or the deployment team is not permitted to update the model. Is there still anything useful we can do?. Their answer is yes—but only if we stop treating a VLA’s visual input as one indivisible observation. The key is to recognize that many manipulation policies receive multiple camera streams, most commonly a static scene view and an eye-in-hand or wrist view. Distribution shift does not necessarily corrupt those streams equally. If one remains informative while the policy is being distracted by the other, robustness becomes partly a problem of runtime sensor arbitration. The paper’s training-free intervention probes that asymmetry, estimates which visual path is still functionally reliable, and restores the input configuration that lets the frozen policy use the right stream. LIBERO-Plus Is Diagnosing More Than Bad Computer Vision. It is tempting to describe these failures as ordinary visual domain shift. The camera moved, the pixels changed, and the vision encoder failed. That explanation is incomplete. A VLA does not simply classify the current image. It maps an observation into a sequence of physical commands, and the resulting motion changes the next observation. A small perceptual error can therefore compound through the closed loop. An incorrect lateral motion changes the gripper’s apparent position. That new image produces a different action. The wrist camera then approaches the wrong part of the scene. By the time the policy closes the gripper, its internal trajectory may already be unrecoverable. What would have been a tolerable one-frame recognition error in a static vision task becomes a task-level failure in robotics. The official LIBERO-Plus repository illustrates the scale of the problem. The exact values depend on the model version and evaluation release, but the current leaderboard shows enormous variation under visual perturbations:. | Model | Camera | Background | Overall LIBERO-Plus | |---|---:|---:|---:| | OpenVLA | 0.8% | 34.8% | 15.6% | | OpenVLA-OFT | 56.4% | 93.3% | 69.6% | | NORA | 2.2% | 58.6% | 39.0% | | WorldVLA | 0.1% | 17.1% | 25.0% | | π₀ | 13.8% | 81.4% | 53.6% | | π₀-FAST | 65.1% | 73.2% | 61.6% |. These are not uniformly weak policies. Several perform very well on the original LIBERO evaluation. Their OOD results instead reveal highly specific dependencies on model architecture, training recipe, observation interface, and camera use. That last point is central. Two policies may consume nominally identical observations but use them very differently. One may localize objects mainly from the external camera and consult the wrist image only near contact. Another may rely almost entirely on the wrist camera once motion begins. A third may concatenate both sets of visual tokens while functionally attending to only one. Token concatenation is not synonymous with sensor fusion. A transformer can learn a brittle hierarchy between camera slots. The first image may become the de facto primary observation because of token order, positional embeddings, data preprocessing, or consistent correlations in the demonstrations. The second stream may carry useful information but contribute only weakly to the action head. Under an external-camera shift, the policy can therefore continue trusting the most corrupted source. This is a shortcut-learning problem expressed through embodiment. The Overlooked Asymmetry Between Scene and Wrist Cameras. A static scene camera and an eye-in-hand camera have very different invariances. The static view offers global context. It can show every object, the target receptacle, and the robot’s approximate configuration before the wrist has entered the workspace. But it is tied to a particular extrinsic calibration. Translate the camera, alter its yaw, or change its field of view, and every pixel-to-action correlation moves. The wrist camera has the opposite profile. Its global coverage is limited, and target objects may initially be small or absent. Yet its pose is mechanically coupled to the end effector. Once an object enters view, the relationship among the gripper, object, and contact geometry can remain relatively stable even when the external camera changes. Background perturbations are similarly asymmetric. A scene camera may devote most of its pixels to the table, walls, fixtures, and distractors. The wrist image often contains a much larger proportion of the manipulated object and gripper. Changing the room texture can therefore move the scene stream much farther from its training distribution than the wrist stream. In principle, a sufficiently well-trained policy should exploit these complementary properties automatically. In practice, standard behavioral cloning gives it little incentive to learn calibrated stream reliability. During training, both cameras usually work. There is no explicit penalty for trusting the wrong one, because “wrong” and “right” streams remain correlated on the demonstration distribution. This creates a useful opening at inference time. If the architecture already contains redundant evidence, we may not need to teach the policy an entirely new skill. We may only need to keep a corrupted stream from overpowering the usable one. That is the distinction Yan and Shen exploit. The Method: Probe, Score, and Restore. The paper’s intervention is best understood as a wrapper around a frozen VLA. It does not update the vision encoder, language model, action head, or policy parameters. Instead, it changes how the existing visual streams are presented and selected at inference time. Conceptually, the procedure has three stages. Probe the streams separately. Construct controlled alternatives to the native multi-camera observation so that the policy’s dependence on each view becomes observable. Score their functional reliability. Compare the action predictions induced by the alternatives using signals available from the frozen policy, such as predictive confidence and output consistency. Restore the useful path. Promote, retain, or substitute the stream that produces the more reliable policy behavior, rather than allowing the shifted stream to dominate the action. The important word here is functional. The objective is not necessarily to reconstruct a photorealistic version of the training scene. A robot does not receive credit for generating a beautiful canonical image. It receives credit for taking the correct action. A stream is therefore “restored” when the policy once again receives information through an input path it can use reliably. Suppose the native input is a pair containing a scene image and a wrist image. The wrapper can compare the original pair against counterfactual configurations that isolate or prioritize one view. Depending on the policy interface, that may involve suppressing one stream, substituting the healthier stream into a high-leverage slot, or repeating a stream rather than introducing an unnatural blank image. Each candidate configuration is passed through the same frozen policy. The resulting action chunks become a diagnostic signal. For an autoregressive VLA, reliability can be associated with the sharpness of its action-token distributions. A candidate that produces diffuse probability across incompatible action bins is less trustworthy than one yielding a coherent, low-uncertainty sequence. But low entropy alone is not sufficient: neural networks are famously capable of being confidently wrong outside their training support. Consistency is therefore equally important. If mild, task-preserving changes to the same stream produce radically different commands, the policy is operating on an unstable representation. Conversely, if one stream produces similar end-effector motions across those probes while the other produces conflicting trajectories, the stable stream is the better runtime choice. For continuous or flow-matching action policies, the same principle can be expressed without categorical token entropy. One can compare sample dispersion, endpoint variance, vector-field stability, or agreement among action chunks generated from nearby input variants. The general question remains the same: which observation path induces a locally stable policy?. The final routing decision should not switch freely at every control tick. If the wrapper alternates between scene-dominant and wrist-dominant inputs at camera rate, it can inject discontinuities into the action sequence. Temporal smoothing or hysteresis is therefore important. A stream should take control only after accumulating sufficient evidence, and it should remain selected until the reliability margin changes meaningfully. This is still training-free, but it is not computation-free. The wrapper may require several policy evaluations for one executed action. That trade-off is reasonable in simulation and in relatively slow manipulation, though it becomes an important engineering constraint for high-frequency control. Why Stream Restoration Can Work Without Learning Anything New. At first glance, inference-time input rearrangement sounds too simple to recover a failed robot. Its effectiveness rests on three structural properties of current VLAs. The first is latent competence. The policy may already know the task and possess useful object-centric features. Its failure can arise because shifted scene tokens overwhelm those features, not because the required behavior is absent from the model. The second is partial redundancy. Multiple views often contain overlapping information about object identity, gripper state, relative pose, and task progress. The redundancy may be poorly fused, but it is still present. The third is asymmetric shift. Camera, background, and some forms of noise do not destroy every observation channel equally. An intervention can help precisely because one path remains closer to the sensory relationships seen during training. Consider a pick-and-place task. Early in the episode, the scene camera may be useful for global target selection. As the gripper approaches, the wrist view becomes increasingly informative about local alignment. A camera-viewpoint perturbation can make the scene-conditioned action point consistently to the wrong image-relative location, while the wrist-conditioned action remains approximately centered on the target. The wrapper does not need to infer the camera extrinsics or rebuild a three-dimensional world model. It only needs to discover that the wrist-conditioned command is more stable. This is a form of test-time causal probing. By altering which stream is available while holding the instruction and robot state fixed, the system obtains evidence about which input currently controls the action. It asks not merely, “Is this image unusual?” but, “Does relying on this image destabilize the policy?”. That is a much more task-relevant OOD question. A generic visual anomaly detector might flag a new background even when the policy remains perfectly competent. Conversely, it might accept a small camera transformation that is visually ordinary but catastrophic for the learned controller. Stream-conditioned action consistency measures the effect of the shift at the level we ultimately care about. Reading the Results: Recovery, Not Immunity. The paper’s empirical claim should be interpreted carefully. Stream restoration does not convert a brittle behavioral-cloning policy into a universally invariant robot foundation model. It recovers a meaningful part of the OOD gap when the failure is caused by misallocation of trust between visual inputs. The strongest case is a stream-localized perturbation: the external view changes substantially while the wrist stream remains usable. Background changes can have a similar structure because the close-range camera is often less dominated by scene appearance. In these conditions, routing the frozen policy toward the healthier stream can produce a large task-level gain with no additional demonstrations and no optimization. The intervention is less naturally suited to perturbations that change the physical state itself. A new robot initialization, displaced target object, unseen distractor arrangement, or altered task geometry may affect every camera consistently. There is no uncorrupted stream to recover; the world really is different. This distinction helps explain why robustness averages can be misleading. A method targeting camera and background shifts may make a valuable deployment contribution even if its seven-category average remains below a model trained on a broad augmentation dataset. Conversely, a high aggregate score can conceal a near-total failure on one operationally critical axis. Clean performance matters just as much. A runtime defense that improves OOD success by constantly replacing informative views can damage the capabilities for which the original model was selected. A good restoration rule therefore retains the native route when the streams agree and intervenes only when there is evidence of asymmetric failure. The most revealing comparisons are not simply “base policy versus proposed method.” They are:. the native multi-view policy;. scene-only and wrist-only execution;. fixed stream substitution;. random stream selection;. adaptive restoration without temporal smoothing;. and the full reliability-gated method. Fixed wrist routing may look excellent under external-camera perturbations but fail when global scene context is necessary. Fixed scene routing may preserve early target localization while remaining brittle near contact. Random routing tests whether gains come merely from perturbing the input. The adaptive version is convincing only if it captures the advantages of each stream without inheriting all of their weaknesses. This is also why success rate should be accompanied by route-usage statistics and phase-conditioned analysis. We want to know when the wrapper switches, how long it remains on each stream, whether its confidence margin predicts success, and which manipulation phases benefit. How This Differs from Training for Robustness. The most direct solution to LIBERO-Plus is to train on more diverse data. The benchmark authors generated more than 20,000 successful trajectories under expanded conditions and mixed them into OpenVLA-OFT fine-tuning. The resulting OpenVLA-OFT+ model reaches 79.6% overall success in the official repository, with 92.8% under camera perturbations—up from 69.6% overall and 56.4% on camera shifts for the corresponding OpenVLA-OFT baseline. That is a stronger form of robustness because it changes the representation and policy. It can teach the model that the same physical action should remain valid across different renderings and viewpoints. But it has costs. Diverse successful trajectories must be generated or collected. The model must be fine-tuned without destroying existing skills. The deployment team must store, validate, and distribute a new checkpoint. And the training distribution can never enumerate every future combination of camera, appearance, hardware, and task variation. Stream restoration occupies a different point in the design space. It is an inexpensive patch for a frozen policy and a diagnostic tool for understanding failures. It can be installed outside the model, disabled without rollback, and combined with later fine-tuning. Representation-anchoring approaches such as Anchor-Align tackle the problem during training by preserving broadly useful pretrained features while improving language–action alignment. Stream restoration instead assumes the checkpoint already exists and asks how its available features can be orchestrated more safely at runtime. The two approaches are complementary. Training should make each view more invariant and improve fusion. Runtime arbitration should still monitor whether a sensor has moved outside its expected operating regime. How It Differs from Runtime Failure Recovery. Another nearby line of work intervenes after the policy begins to fail. ProbeAct, for example, monitors object positions, end-effector kinematics, gripper signals, and repeated failure locations. A state machine identifies grasp, transport, and placement failures, while a control-barrier-function layer modifies actions around unsafe or repeatedly unsuccessful regions. On LIBERO-Plus, it improves OpenVLA-OFT from 69.6% to 74.1%. The distinction is where each method acts. Stream restoration operates upstream. It attempts to prevent a perceptual routing error from becoming an incorrect action. Probe-guided recovery operates downstream. It observes the consequences of the policy’s behavior and tries to recover or constrain it. A practical robot should probably use both. The first layer would monitor visual-stream health and choose the best available perceptual path. A second layer would detect whether the resulting motion is making progress. A third would enforce geometric or safety constraints. If all three disagree, the robot should stop rather than convert model confidence into mechanical persistence. This layered view is more realistic than expecting one VLA checkpoint to handle perception, uncertainty, recovery, and safety implicitly. A Different Competitor: World Action Models. Recent world action models offer another explanation for improved robustness. Rather than mapping the current observation directly to actions, these systems inherit predictive priors from large-scale video or world-model training. Their ability to represent how scenes evolve may reduce dependence on static appearance cues. A 2026 comparative study reports that Cosmos-Policy reaches 82.2% success on LIBERO-Plus, while LingBot-VA reaches 74.2% on RoboTwin 2.0-Plus. The authors argue that explicit dynamics prediction and video pretraining can offer stronger robustness than many conventional VLA recipes, although some heavily trained VLAs remain competitive. That does not make stream arbitration obsolete. World models still receive sensors, and multi-camera predictive models can still become overconfident in a shifted view. Inference-time reliability routing is largely orthogonal to whether the action generator is autoregressive, diffusive, flow-based, or world-model-driven. The larger lesson is that robustness can enter at several levels:. through diverse training data;. through representations that preserve geometry and dynamics;. through explicit cross-view consistency;. through runtime stream arbitration;. and through downstream failure recovery. Yan and Shen focus on the cheapest of these levels, but cheap does not mean trivial. A small wrapper that reveals and exploits hidden competence can be operationally valuable, particularly when a seven-billion-parameter checkpoint is already deployed. Engineering the Idea into an Existing VLA Stack. For practitioners, the method suggests a fairly concrete integration path. First, expose the actual boundaries between visual streams. This is easy in a modular OpenVLA-style implementation but can be awkward when images are packed into a single multimodal prompt. The wrapper must know which visual tokens originate from the scene camera and which originate from the wrist. Second, define candidate stream configurations that preserve the model’s expected tensor shapes. Blank images are not necessarily neutral; they introduce a new and potentially severe distribution shift. Repeating, substituting, or selectively attenuating a valid stream is often safer than inserting zeros. Third, choose a policy-relevant reliability statistic. For discrete action-token models, normalized token entropy is available almost for free. For deterministic regression policies, use action consistency under mild augmentations. For stochastic flow or diffusion policies, compare several sampled action chunks and examine endpoint dispersion. Translation, rotation, and gripper dimensions should be normalized before their disagreements are combined. Fourth, avoid making routing decisions at the full control frequency. Visual reliability generally evolves more slowly than motor commands. Probe the streams periodically, cache the decision for several control steps, and require a margin before switching. Fifth, preserve shared computation. The language instruction and much of the robot state do not change between candidate evaluations. A naive implementation may rerun the entire VLA several times, but a production system should cache text embeddings, reusable visual features, and static key–value states where the architecture permits. Sixth, add an abstention state. If every stream configuration produces unstable or mutually inconsistent actions, the correct conclusion is not that the least bad one must be safe. The wrapper should be able to pause, request another camera, move to an observation pose, or hand control to a recovery controller. Finally, log the causal probes. For every intervention, record the original streams, candidate actions, reliability scores, selected route, and eventual outcome. These logs are excellent data for subsequent fine-tuning because they identify precisely where the policy’s implicit sensor fusion failed. Training-free deployment can therefore become a data-collection strategy for the next training cycle. What the Method Cannot Fix. The first limitation is obvious but fundamental: the method needs redundancy. A single-camera policy has no alternative stream to restore. A multi-camera policy whose views are simultaneously occluded or shifted gains little from routing. Second, confidence under distribution shift is unreliable. A corrupted scene view can produce a sharp but systematically wrong action distribution. Agreement and temporal evidence help, but no internal uncertainty score guarantees correctness. Third, camera streams are not semantically interchangeable. A wrist image and an external image encode different scales, occlusions, and geometries. Substitution works only to the extent that the shared visual backbone and downstream policy tolerate those differences. Fourth, stream selection does not create global information that the chosen camera cannot see. A wrist-dominant policy may align a gripper accurately once the target is visible but remain unable to choose among distant objects. Fifth, the additional inference can violate a robot’s latency budget. A three-pass wrapper around a large VLA may be acceptable at a few hertz but unsuitable for fast contact-rich control without caching, distillation, or a lightweight proxy score. Sixth, the intervention does not address language neglect. LIBERO-Plus finds that some models remain surprisingly insensitive to instruction perturbations and can execute familiar trajectories even when language is altered. Restoring a camera stream cannot repair a policy that never grounded the instruction properly. Finally, combined perturbations remain dangerous. LIBERO-Plus reports negative compositional gaps: success under two simultaneous shifts is often worse than one would predict by multiplying their individual effects. A reliable wrist stream under a camera perturbation may cease to be reliable once noise, robot-state variation, and object displacement occur together. The method should consequently be understood as structured robustness recovery, not universal OOD immunity. The Deeper Lesson: Multi-View Tokens Need a Reliability Model. The most important contribution of Restoring the Right Stream may be diagnostic rather than algorithmic. Current VLAs are frequently described as multimodal because they concatenate language, robot state, and one or more image streams. But true multimodal control requires more than a common transformer. It requires knowing when a modality is informative, when it conflicts with another, and when it has left the support of the learned policy. Classical sensor fusion systems make this explicit. Measurements have noise models, innovation tests, observability conditions, and mechanisms for rejecting faulty sensors. Modern VLAs largely ask gradient descent to discover all of that implicitly from successful demonstrations. The LIBERO-Plus results suggest that it often does not. A future VLA architecture should treat cameras less like an undifferentiated block of tokens and more like conditionally reliable experts. It should maintain stream-specific uncertainty, reason about camera geometry, test cross-view agreement, and degrade gracefully when one input disappears. Training should include view dropout, camera perturbations, cross-view action consistency, and examples where one sensor is misleading. Inference should include health monitoring and abstention. Evaluation should report single-stream ablations, not only full-input success. There may also be a fundamental limit to how much robustness can be obtained without sacrificing some clean capability. Recent information-theoretic work argues that, for discrete-action VLAs under adversarial channels, capability and preserved task-relevant information share a finite budget. That result concerns a different and more adversarial setting, but it reinforces an important point: universal invariance is not literally free. Yan and Shen’s method avoids claiming otherwise. It obtains a cheaper gain by exploiting structure already present in the observation system. It does not make the policy robust to every possible world. It stops the policy from ignoring a usable sensor merely because a familiar but corrupted one occupies the privileged input slot. Closing Perspective. The VLA community has spent enormous effort building larger policies, collecting broader datasets, and improving action representations. All of that remains necessary. But Restoring the Right Stream highlights a more mundane systems problem: sometimes the model knows enough to succeed and is simply listening to the wrong camera. That is encouraging because it means some robustness failures are recoverable without another training run. It is also concerning because it reveals how little explicit reliability reasoning exists inside today’s supposedly multimodal policies. The right deployment lesson is not to choose between robust training and inference-time restoration. It is to build a stack. Train on meaningful diversity. Preserve geometric and semantic representations. Monitor each observation stream independently. Route around asymmetric failures. Detect lack of progress. Enforce control constraints. Stop when no available sensor supports a trustworthy action. A training-free wrapper cannot turn benchmark success into general intelligence. But it can restore the sensory contract on which the policy’s existing competence depends. And for a robot already in the field, that can be the difference between a shifted camera being a minor nuisance and an otherwise capable policy becoming completely unusable. Publication note — September 2, 2026: The DOI had been assigned to Entropy Volume 28, Issue 9, Article 975, but author-affiliation and code-repository metadata were not yet exposed by the public indexes available during research for this episode.