Latest / Embodied AI 101 / Anchor the Semantics, Align the Actions: A Better Way to Fine-Tune VLAs

Transcript

Anchor the Semantics, Align the Actions: A Better Way to Fine-Tune VLAs · Embodied AI 101

• Anchor the Semantics, Align the Actions: A Better Way to Fine-Tune VLAs. Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment • Dwip Dalal, Shivansh Patel, Chahit Jain, Jeonghwan Kim, Utkarsh Mishra, Alex Baratian, Hyeonjeong Ha, Heng Ji, Svetlana Lazebnik, and Unnat Jain • University of Illinois Urbana-Champaign, Texas A&M University, and University of California, Irvine • arXiv preprint • 2026. The uncomfortable premise behind VLA fine-tuning. The standard pitch for vision-language-action models is compelling: start with a vision-language model trained on internet-scale data, attach an action pathway, and fine-tune on robot demonstrations. The policy should inherit the VLM’s knowledge of objects, colors, spatial relations, and natural-language instructions while learning the final mapping into continuous control. Anchor-Align begins with evidence that this transfer story is often wrong in practice. Behavior cloning does not merely add control capability to a pretrained representation. It can progressively overwrite the representation that made the VLM attractive in the first place. The paper’s motivating example is deliberately simple. Train a robot only on “pick up the green mug and place it on the plate,” with both a green and a pink mug visible. Then, at test time, ask it to pick up the pink mug. A useful VLA should combine its pretrained understanding of color with its learned mug-grasping behavior. Instead, the behavior-cloned policy repeatedly selects the green mug—the object associated with the training trajectory—regardless of the new instruction. On the physical system, the baseline reaches for the green mug in 90% of these semantic-perturbation trials. This is not simply ordinary overfitting in the action head. The authors argue for two distinct failures:. Representation drift: action fine-tuning erases or deforms pretrained visual-language concepts. Language-action misalignment: even when a model retains some language capability, its language output and action output may encode contradictory interpretations of the same observation. Anchor-Align addresses both, using supervision already present in robot demonstrations. Its first objective keeps the trainable VLA close to a frozen copy of the original VLM at every decoder layer. Its second turns each continuous action chunk into an interpretable direction word—such as left, up, or forward—and trains the language pathway to predict that word from the same observation used for action prediction. The result is a relatively small change to the fine-tuning recipe, but a large change in behavior: real-robot mean success rises from 28.3% to 54.2% with one VLA architecture, and from 36.7% to 60.0% with another. The more important contribution, however, is the paper’s framing. VLA adaptation should not be treated as “learn actions and hope the semantics survive.” It should explicitly preserve the useful prior and explicitly connect that prior to control. Why ordinary behavior cloning loses the plot. Consider the gradients seen by a conventional VLA backbone. Robot data are narrow compared with web-scale VLM pretraining. The camera setup is repetitive, the vocabulary is limited, task descriptions are templated, and many observations can be mapped to acceptable actions using scene-specific visual shortcuts. If every training demonstration places the green mug in roughly the same region, the policy can reduce action loss without representing “green,” “mug,” or even the current instruction in a compositional way. It can associate a familiar image pattern with a familiar trajectory. From the optimizer’s perspective, that is a perfectly valid solution. LoRA does not automatically prevent this. Anchor-Align’s default baseline already fine-tunes the backbone with rank-64 LoRA across all linear layers, yet its visual-language representations still degrade sharply. On GQA, a benchmark probing object, attribute, and spatial reasoning, standard behavior cloning loses 94% of the pretrained model’s accuracy within 10,000 steps. The issue is therefore not restricted to unconstrained full-parameter fine-tuning. A common response is to co-train the VLA on captioning, VQA, or spatial-reasoning examples alongside the robot data. This supplies a continuing language-modeling signal and, in principle, protects general-purpose knowledge. Anchor-Align argues that this remedy is indirect in two ways. First, a generic VQA loss does not directly constrain the model’s hidden states on the robot observations that matter. The model may preserve enough capability to answer questions about web images while still deforming its representation of wrist-camera and tabletop scenes. Second, the language head and action head receive supervision on different samples. The language head answers a question about one image while the action head controls a robot in another. Sharing a backbone does not guarantee that they will form the same interpretation of any particular robot state. The authors demonstrate the latter problem by asking each head what the robot will do next. They convert the policy’s continuous action into a direction category and separately prompt the language head for a direction word. In the standard VLA-Adapter baseline, the two agree on only 16.8% of evaluated LIBERO-PRO observations. In the authors’ co-training-plus-Knowledge-Insulation implementation, agreement is even lower, at 14.3%. This distinction matters. A model can have a competent action head and a competent language head in aggregate while the two remain inconsistent frame by frame. Generic multimodal competence is not the same thing as an action-grounded multimodal representation. The Anchor-Align recipe. Anchor-Align operates on VLAs with a pretrained VLM backbone and a dedicated continuous action head. Images and the language instruction pass through the multimodal backbone. Robot proprioception is encoded separately. Learned action queries then retrieve task-relevant features from the backbone, and the action module predicts a fixed-horizon chunk of seven-degree-of-freedom actions: translation, rotation, and gripper control. The action module may use direct regression or conditional flow matching. For regression, behavior cloning minimizes an absolute-error action loss. For flow matching, it trains the velocity field used to generate the action chunk. Anchor-Align leaves that primary control objective intact and adds two weighted losses. Vision-Language Anchoring. The first addition is a frozen copy of the original pretrained VLM. At every training step, both the frozen anchor and the trainable VLA backbone process the same robot images and instruction. For every decoder layer, the method compares the hidden states of all vision and text tokens in the trainable backbone with the corresponding states in the frozen VLM. The squared differences are averaged across layers to form the anchoring loss. Action-specific positions and proprioceptive inputs are not what is being preserved; the target is the multimodal representation inherited from pretraining. This is stronger than merely penalizing parameter movement. Two networks can remain close in weight space while producing meaningfully different features, and different modules may respond very differently to the same weight-space constraint. Anchor-Align directly constrains the computation performed on the current robot observation. It is also more comprehensive than aligning only the vision encoder. The teacher is the original VLM itself, and supervision is applied to both visual and textual positions throughout all decoder layers. In the default 24-layer Qwen2.5-based backbone, every decoder layer contributes to the loss. The action loss is still free to move the student away from the anchor when control demands it. The anchoring term is not a hard freeze. It creates a soft optimization pressure: acquire action-relevant information without arbitrarily reorganizing the pretrained visual-language geometry. That difference is visible in the frozen-backbone baseline. Freezing preserves the VLM, but produces only 43.1% mean success on LIBERO-PRO. Standard fine-tuning reaches 61.0%, but destroys much of the pretrained reasoning capability. Anchor-Align reaches 71.9%, suggesting that preservation and action specialization need not form a zero-sum trade-off. Language-Action Alignment. Anchoring alone says what the model should retain. It does not tell the retained representation how to encode the robot’s next action. The second objective supplies that bridge. For each non-stationary ground-truth action chunk, the authors average its translational motion. They identify the axis with the largest absolute displacement and use its sign to select one of six labels:. forward or backward,. left or right,. up or down. Chunks whose average translational magnitude falls below a threshold are omitted from the alignment loss. This avoids assigning unstable direction words to nearly stationary behavior. The language prediction is read from the final-layer representation of the last instruction token—the state immediately before action prediction begins. That representation has already attended to the image and instruction, so it acts as a compact summary of what the model believes should happen next. A learned square projection maps this state back into the space expected by the original language-modeling head. The LM head itself is frozen. Cross-entropy then trains the system to emit the direction word derived from the ground-truth action. For the 0.5-billion-parameter Qwen2.5 backbone, the projection is 896 by 896 and adds roughly 804,000 parameters, or about 1.5 megabytes in bfloat16. The direction is predicted over the model’s full 151,646-token vocabulary rather than through a disconnected six-class classifier. Gradients flow through the projection and the backbone’s LoRA parameters, but not through the language head. That frozen language head is an important design choice. The model is not allowed to redefine the output embedding for left until it means whatever is convenient for the robot task. Instead, its pre-action state must remain interpretable through the pretrained vocabulary geometry. There is a subtle but important qualification here: the method does not directly compare the model’s predicted continuous action with its predicted word during training. Both are supervised from the same ground-truth action. It is therefore best understood as shared-target, same-observation co-supervision rather than a direct consistency penalty between the two outputs. The empirical question is whether this shared supervision is enough to make them agree at inference time. The paper’s diagnostic suggests that it is. The complete objective. Training jointly optimizes the ordinary action loss, the layer-wise anchor loss, and the direction-word alignment loss. In the default simulation configuration, the anchor weight is 0.1, the alignment weight is 0.02, and the stationarity threshold is 0.15. Training uses a batch size of 32, a learning rate of (2 \times 10^{-4}), random crops and color jitter, and 10,000 gradient steps on four NVIDIA GH200 GPUs. The default policy uses a Prismatic-Qwen2.5-0.5B backbone. It receives third-person and wrist-camera images plus proprioception. DINOv2 and SigLIP features are concatenated per visual patch, producing a 512-patch visual prefix across the two cameras. The action pathway uses VLA-Adapter’s bridge-attention regression head. The authors also instantiate Anchor-Align inside StarVLA with a 3-billion-parameter Qwen2.5-VL backbone and a GR00T flow-matching diffusion transformer, providing a materially different test of the recipe. The frozen teacher does add training cost, but it is removed for deployment. The authors discard the teacher’s large LM head because only intermediate states are required, leaving a reported 0.7-gigabyte memory overhead. Iteration time increases from 1.28 to 1.64 seconds, a 28% penalty. Their co-training baseline takes 2.50 seconds per iteration, adds five gigabytes, and requires 25,000 external VQA examples. Thus, “no architectural change” should be interpreted as no substantial deployed-policy change. Training does require a second forward pass and the alignment projection, but inference retains only the fine-tuned backbone and ordinary action pathway. An evaluation designed to expose shortcuts. The paper emphasizes benchmarks that perturb the relationship between observations, instructions, and trajectories. LIBERO-PRO tests language rephrasing, object replacement, and position swaps. Its position-swap setting is particularly diagnostic: objects move to locations not seen during training, so a policy that replays a memorized trajectory should fail. LIBERO-Plus covers seven perceptual shifts: language wording, background texture, robot initialization, camera viewpoint, object layout, illumination, and sensor noise. CALVIN ABC-to-D tests long-horizon transfer by training in environments A, B, and C and evaluating chains of five instructions in held-out environment D. Standard, unperturbed LIBERO is included in the appendix to ensure that robustness gains do not come at the cost of ordinary task success. The comparisons span OpenVLA-OFT, VLA-Adapter, a frozen-backbone VLA-Adapter, MolmoAct, the authors’ Knowledge-Insulation adaptation, and several CALVIN-specific systems. The real-robot experiments additionally compare the regression-based VLA-Adapter setup with the flow-matching StarVLA configuration. Simulation results: the big gain is re-grounding. On LIBERO-PRO, standard VLA-Adapter scores 91.1% on language rephrasing and 89.6% on object swaps, but only 2.3% when object positions are exchanged. OpenVLA-OFT, MolmoAct, and the co-training baseline score zero on position swaps. Anchor-Align reaches 97.0% on rephrasing, 96.2% on object swaps, and 22.6% on position swaps, increasing the three-axis mean from 61.0% to 71.9%. The position-swap result is simultaneously the paper’s strongest evidence and an important reminder of the remaining gap. Moving from roughly 2% to roughly 23% is a dramatic relative improvement and supports the anti-memorization argument. But 22.6% remains low in absolute terms. Anchor-Align reduces shortcut dependence; it does not solve spatial generalization. On LIBERO-Plus, mean success increases from 85.1% to 90.3%. The largest gains appear under background changes, sensor noise, shifted robot initialization, and illumination. Anchor-Align reaches 99.6% under background-texture changes, 99.0% under lighting changes, and 96.9% under sensor noise. The method does not sacrifice in-distribution performance. On standard LIBERO Spatial, Object, Goal, and Long, it records 98.4%, 100.0%, 97.2%, and 90.8%, respectively, modestly improving upon VLA-Adapter’s already strong 96.0%, 99.8%, 96.0%, and 89.0%. Five-seed experiments show standard deviations no larger than 0.7 percentage points, with the method’s mean improvements comfortably exceeding seed-to-seed variability. The ablations are clean. Alignment by itself raises LIBERO-PRO mean success from 61.0% to 65.9%. Anchoring alone reaches 68.1%. Together they reach 71.9%. On LIBERO-Plus, the corresponding sequence is 85.1%, 88.6%, 87.3%, and 90.3%. Both components help, and neither subsumes the other. The authors also test whether the direction objective is merely generic auxiliary regularization. In the Shuffle control, each true direction is consistently mapped to the wrong direction word. In Scatter, directions map to semantically unrelated tokens such as purple, seven, and music. Both retain essentially the same classification structure and update frequency as the real objective. Yet Shuffle scores 61.4% on LIBERO-PRO and Scatter 63.3%, close to the 61.0% baseline and far below Anchor-Align’s 71.9%. Correct motion semantics, not just another loss term, appear to drive the gain. On CALVIN, Anchor-Align completes all five tasks in 77.9% of instruction chains, versus 73.1% for VLA-Adapter and 66.5% for OpenVLA-OFT. Its average completed chain length rises from 4.3 to 4.5. The improvement widens at deeper horizons, where small grounding mistakes compound across tasks. A separate StarVLA simulation experiment uses a larger Qwen2.5 3B backbone and an OFT-style MLP action head. Anchor-Align increases language-rephrase success from 73.6% to 89.0% and object-swap success from 89.8% to 91.2%. This does not constitute exhaustive architecture coverage, but it helps show that the result is not peculiar to the default 0.5B regression model. Real-robot results: semantics survive contact with hardware. The physical experiments use a UFactory xArm7 with wrist and front-view RGB cameras. The team collects 150 demonstrations through a 3D-printed GELLO teleoperation device at 30 hertz. Both methods train for 15,000 steps. Evaluation covers seven held-out tasks with 20 rollouts each, for 140 total trials. The low-level controller runs at 100 hertz while policy actions are produced at 30 hertz and interpolated with velocity clipping. The evaluation regimes move target objects, destinations, and distractors; add clutter; and jointly alter object position and orientation. The semantic-perturbation test changes both the layout and instruction, asking a model trained on a green mug to pick up a pink one. With VLA-Adapter, mean success rises from 28.3% under standard fine-tuning to 54.2% with Anchor-Align. With StarVLA, compositional object-layout success rises from 35% to 50%, spatial rearrangement from 30% to 60%, and cluttered-scene success from 45% to 70%. The mean increases from 36.7% to 60.0%. On the pink-mug test, standard StarVLA picks its familiar green mug in 90% of trials. Anchor-Align selects the requested pink mug in every trial. Full placement success, however, reaches only 40%. This is revealing: the method largely repairs target selection and semantic grounding, but downstream grasp-and-place execution remains a separate bottleneck. Manual failure analysis reinforces that interpretation. Anchor-Align reduces incorrect-object approaches from ten to one and semantic confusions from seven to zero. Memorized-trajectory failures fall from fifteen to nine, and grasp failures from fifteen to eight. Grasp-and-drop errors rise slightly, from twelve to thirteen, because the improved policy reaches and grasps more objects, creating more opportunities to fail during transport or release. Successful Anchor-Align trials are also reported to finish 1.7 times faster. The authors observe larger, more decisive actions around the grasp point, while standard BC tends to issue small corrective motions. That explanation is plausible but should be treated as a hypothesis: the paper reports the behavioral pattern, not a controlled causal analysis of why alignment changes action magnitude. What changes inside the representation?. The strongest part of the paper may be its attempt to connect task success to internal representation behavior. First, there is the GQA result: standard fine-tuning retains only about 6% of the backbone’s original GQA capability after 10,000 steps, whereas Anchor-Align retains 70%. Because Anchor-Align simultaneously improves robot performance, this argues against the assumption that action learning necessarily requires discarding general visual-language reasoning. Second, the authors measure representation similarity using centered kernel alignment. CKA compares the geometry of activations rather than requiring identical activation magnitudes. Standard BC’s text-token representation drops to 0.34 CKA in the final decoder layer relative to the pretrained VLM. Anchor-only training maintains an average CKA of 0.95 across layers, while the complete Anchor-Align model achieves 0.91 overall. Preservation alone would not be enough if the backbone remained oblivious to action. The authors therefore fit linear probes from each layer’s pooled hidden state to the ground-truth action. Anchor-Align reaches a peak action-decoding (R^2) of 0.60 at layer 22, the highest of the compared methods, while maintaining the high CKA score. The intended picture is not a frozen representation with a separate motor module bolted on. It is a preserved representation enriched with linearly accessible control information. A direction-specific probe tells the same story. Direction classification improves from 86.0% to 94.4% on the in-distribution spatial split and from 21.4% to 30.3% on the out-of-distribution object split. Most strikingly, language-action agreement rises from 16.8% to 78.4%. For standard VLA-Adapter, rollout-level agreement has essentially no relationship with success: Pearson correlation is −0.03. With Anchor-Align it becomes +0.51, meaning rollouts with more agreement are substantially more likely to succeed. Agreement itself is a lenient metric—it counts cases where both heads make the same mistake—but the magnitude of the change is difficult to dismiss. The paper extends this diagnostic to ChatVLA, MolmoAct, and Magma using real robot images from MolmoAct’s mid-training dataset. Across direction and orientation, agreement ranges from only 7.4% to 20.9%. MolmoAct, for example, predicts action direction with 41.3% accuracy but language direction with 22.6%, and the heads agree on only 15.5% of frames. Its action pathway appears to contain control information that its language channel cannot express. The diagnostic framework also constructs labels for orientation, gripper state, and task completion. These are not used by the main Anchor-Align training objective, but they point toward a broader research program: derive semantically meaningful abstractions from trajectories, then test whether every output pathway tells a coherent story about the same physical state. How Anchor-Align differs from nearby approaches. Anchor-Align sits within a growing line of work questioning whether VLA fine-tuning destroys useful pretrained structure. MAPS constrains modules in weight space using different proximity schedules. Don’t Blind Your VLA focuses on preserving visual features, aligning patch representations at an intermediate layer with a frozen visual foundation model. Anchor-Align instead matches both vision and text hidden states at every VLM decoder layer, then adds the same-observation language-action objective absent from those preservation-only methods. Knowledge Insulation addresses harmful gradients arising from newly initialized continuous action experts, protecting the VLM backbone during training. Anchor-Align’s evidence suggests that completely shielding the backbone can also limit action adaptation: its adapted co-training-plus-insulation baseline preserves little useful action information and underperforms standard BC in this fine-tuning regime. That result should be scoped to the authors’ implementation rather than interpreted as a universal refutation of Knowledge Insulation. Actions as Language, also called VLM2VLA, resolves the modality mismatch more directly by expressing low-level actions as natural-language sequences and training through LoRA. Anchor-Align keeps a conventional continuous regression or flow-matching action head; language is an auxiliary semantic description of the action rather than the deployed action representation itself. That makes Anchor-Align easier to graft onto existing continuous-control stacks, while VLM2VLA offers a more unified but more consequential reformulation of action generation. Other systems preserve capability through architecture or data: dual visual encoders, string-based action tokenization, mixtures of robot and VQA data, or large instruction-tuning corpora. Anchor-Align’s attractive property is that it needs neither internet co-training data nor newly annotated robot reasoning traces. It manufactures its auxiliary labels from the demonstrations already used for behavior cloning. VLM4VLA and Rethinking VLM Representation for VLA Initialization add an important qualification. A strong generic VLM is not automatically a strong VLA initializer, and improving isolated VQA abilities does not reliably improve control. Yet both find that pretrained representations and conservative update strategies remain valuable. The emerging position is therefore more nuanced than “preserve everything”: select a representation whose biases are relevant, inject action-side information, and avoid uncontrolled drift. Limitations and open questions. The alignment label is intentionally coarse. Averaging an action chunk and selecting one dominant translation axis throws away curved motion, coupled-axis movement, rotation, gripper state, and phase structure. A chunk that begins by moving down and ends by moving sideways may receive whichever label dominates the average. A chunk spanning opposite motions may appear stationary and be discarded. This coarseness may be beneficial as an information bottleneck: the target describes high-level motion intent rather than duplicating the continuous action. But it also means the method’s gains do not show that arbitrary low-level action detail can be translated into language this cleanly. Coordinate conventions are another practical concern. Forward, left, and up are meaningful only relative to a specified frame. A label construction that works for a fixed tabletop and camera may become ambiguous across mobile manipulators, changing base poses, mirrored embodiments, or multi-camera datasets. Deployments across embodiments may need robot-centric, camera-centric, and world-centric labels, or explicit frame tokens. The frozen teacher can preserve both useful and unhelpful priors. If the base VLM systematically misrepresents transparent objects, deformables, tool contact, or wrist-camera views, anchoring too strongly could resist the very adaptation robot data should induce. The reported anchor weight works well in these experiments, but the paper does not yet provide a general procedure for choosing the preservation-specialization frontier. The default simulation evidence is strongest for one small backbone and one regression architecture. The StarVLA results broaden the case, but the larger-backbone simulation study reports only selected LIBERO-PRO axes, while the semantic real-robot test is performed only with StarVLA. The authors reasonably exclude deterministic L1 regression there because it poorly represents multimodal action distributions, but this also means the most compelling color-generalization test is not demonstrated across both action heads. The real-world improvement is large, but the study uses 150 demonstrations, seven held-out tasks, and the best-performing checkpoint from each 15,000-step run. The paper does not present real-robot multi-seed confidence intervals comparable to its simulation analysis. Larger task suites, multiple robot instances, and a checkpoint-selection protocol isolated from evaluation would make the result more conclusive. Finally, agreement is not synonymous with correctness. Two heads can agree because they share the same shortcut. The authors recognize this and report per-head accuracy, ground-truth comparisons, corruption controls, and a success correlation. Still, future work should move beyond categorical agreement toward calibrated consistency: does the linguistic description predict the full action distribution, the resulting state transition, and progress toward the instruction?. A practical recipe for VLA builders. For teams adapting an existing VLM-plus-action-expert system, Anchor-Align suggests a straightforward workflow:. Keep the exact pre-robot VLM as a frozen teacher. Process the same robot images and instructions through teacher and student, and remove unused teacher heads to control memory. Anchor representations rather than only weights. Begin with multimodal token states across several layers; use all layers when feasible. Derive semantic labels from trajectories. Direction is a good starting point, but orientation, gripper state, contact phase, and task completion may be equally useful. Make labels readable through the pretrained language interface. A small projection into a frozen LM head preserves the meaning of target tokens better than an isolated classifier. Measure both retention and adaptation. Track a VLM capability benchmark or representation-similarity metric alongside action decodability and task success. Evaluate perturbations that break trajectory shortcuts. Position swaps, changed instructions, distractors, and camera shifts reveal problems that saturated in-distribution suites hide. The code repository and pretrained checkpoints are publicly linked from the project materials, including training, evaluation, and reproduction scripts. The more general lesson is to treat robot demonstrations as containing multiple forms of supervision. A trajectory is not only a sequence of continuous targets. It also implicitly specifies direction, contact state, progress, and often causal state change. Extracting those abstractions programmatically can create an interface between foundation-model semantics and low-level control without paying for a separate annotation pipeline. The bottom line. Anchor-Align makes a persuasive case that VLA fine-tuning has two obligations: preserve the pretrained representation that supports open-world interpretation, and ensure that this representation describes the same behavior the action head executes. Its frozen-teacher anchoring limits destructive drift. Its direction-word objective pushes control information through a language-readable state. Together, these changes improve robustness, long-horizon success, and real-robot generalization while using no additional labeled data and adding no teacher-side cost at deployment. The method does not eliminate the hard parts of manipulation. Position-swap success is still modest, placement remains difficult after correct semantic selection, and six direction words capture only a small slice of action semantics. But the conceptual shift is valuable: a VLA should not have to forget how to understand the world in order to act in it. The better objective is to preserve that understanding—and make action an explicit part of it.