Latest / Embodied AI 101 / VLAct: The VLA Backbone Is Not Just Plumbing

Transcript

VLAct: The VLA Backbone Is Not Just Plumbing · Embodied AI 101

• VLAct: The VLA Backbone Is Not Just Plumbing. Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models • Senqiao Yang, Chengyao Wang, Yuxin Chen, Zixuan Wang, Longxiang Tang, Haokun Gui, Jinhui Ye, Changsheng Lu, Xiaoyang Wu, Mingkang Zhu, Pengguang Chen, Shu Liu, Zhuotao Tian, Hengshuang Zhao, Bei Yu, and Jiaya Jia • StarVLA research team • arXiv • 2026. The headline—and one important correction. VLAct is built around a deceptively simple claim: when robot data is limited, the quality of the representation learned from each trajectory matters independently of model size and dataset scale. Instead of treating vision-language-action continued pre-training as supervised action fitting, the authors treat it as representation learning for physical interaction. Their target artifact is not a finished policy but a reusable, action-aware vision-language backbone that can later support different robots, tasks, and action decoders. The first arXiv version was submitted on August 27, 2026. Before going further, it is worth separating two numbers that are easy to conflate. VLAct’s 92.5% success rate is on the RoboTwin 2.0 Data Scaling setting. The 20% data result belongs to a different experiment: adaptation to the previously unseen GR-1 humanoid in RoboCasa-GR1. There, VLAct reaches 49.5% success using 20% of the downstream trajectories, exceeding the reported full-data Qwen3VL-OFT result of 48.8% and GR00T-N1.6 result of 47.6%. With all RoboCasa-GR1 fine-tuning data, VLAct reaches 54.0%. So the paper does not claim 92.5% RoboTwin success with 20% of RoboTwin data. That clarification actually strengthens the conceptual story. RoboTwin demonstrates high performance on a familiar embodiment and benchmark. RoboCasa-GR1 tests the more interesting question: whether a backbone trained on Franka and AgileX trajectories carries useful physical knowledge into a humanoid action space that was absent during continued pre-training. The real question: what enters the backbone?. The dominant scaling narrative in robot learning has been productive. Open X-Embodiment assembled data from 22 robots and showed that cross-robot training can generate positive transfer. Octo turned a large slice of that data into an adaptable generalist policy initialization. OpenVLA trained a seven-billion-parameter VLA on 970,000 robot demonstrations. Systems such as GR00T N1 combine heterogeneous robot trajectories, human video, simulation, and a dual-system architecture. These projects ask how much capability can emerge from larger and more diverse embodied datasets. VLAct accepts that premise but points to a structural difference between web and robot data. Image-text data can be collected across an enormous range of scenes and concepts. Robot trajectories require physical execution, teleoperation, simulation, or procedural generation. They remain sparse relative to the combinatorial space of objects, scenes, goals, contact modes, camera arrangements, and embodiments. Even a large robot dataset is therefore likely to provide dense repetition in some regions and almost no coverage in others. The consequence is that continued pre-training has two jobs. It must fit the observed actions, but it must also compress those observations into features that remain useful outside the training distribution. A trajectory showing a gripper approaching a cup should ideally teach more than a dataset-specific action vector. It should refine features for object identity, relative pose, grasp affordance, task phase, contact anticipation, and gripper state. VLAct therefore treats the VLM backbone as a first-order design variable. In the controlled downstream comparisons, the task data, optimizer, training budget, freshly initialized action head, and head architecture are held fixed. The principal experimental variable is whether that downstream model starts from generic Qwen3-VL weights or the VLAct continued-pretrained backbone. The paper attributes gains of 7.6 to 21.4 percentage points in these matched comparisons to the backbone initialization. The most distinctive part of this framing is what happens to the pre-training heads: they are thrown away. They are scaffolding used to shape the backbone, not components that must be retained in the final policy. During downstream adaptation, the full backbone is unfrozen and a new action head is initialized for the target problem. The pilot study: action heads are not neutral readouts. Why should the pre-training head matter if it is later discarded? VLAct’s pilot study argues that an action decoder does not merely read a fixed representation. Its gradients determine what information the backbone makes easy to extract. The paper considers four representative interfaces. FAST compresses continuous action chunks and predicts discrete tokens autoregressively. OFT appends action-query tokens and uses a small MLP to regress the complete continuous chunk in parallel. A PI-style head conditions a flow-matching motor module on the VLM features and iteratively transports noise into an action trajectory. The GR00T-style head also uses flow matching but more explicitly separates a semantic vision-language system from a DiT-like motor generator. These interfaces ask different questions of the shared latent. OFT rewards features from which a point estimate is directly recoverable. The flow heads need conditioning features that remain useful across noise levels and iterative generation steps. FAST asks the language-model stack to support sequential prediction in a compressed discrete vocabulary. The discrete pilot result is nuanced. A FAST-pretrained backbone improves when transferred to a continuous GR00T head, suggesting that token supervision does inject action structure. But keeping the FAST decoder reaches only 45.2 on the reported LIBERO-Plus probe, whereas attaching GR00T to the FAST-pretrained backbone reaches 76.7. The authors interpret this as evidence that the backbone has learned coarse action organization while the discrete decoding interface loses fine temporal or amplitude information needed for control. This should not be read as a universal indictment of FAST. The original FAST work showed that frequency-space tokenization can make autoregressive VLAs practical for high-frequency and dexterous data, and reported substantial training-speed benefits at scale. VLAct establishes a narrower result: in its controlled transfer probe, discrete supervision alone did not produce the strongest continuous-control interface. The more consequential pilot concerns continuous heads. Fine-tuning PI from scratch obtains 60.5 on the RoboTwin probe. Initializing the backbone from OFT-only pre-training reduces that result to 55.1. But adding GR00T as a second pre-training head raises the unseen PI result to 63.1, even though PI itself was not used during that pre-training run. Once PI is included in the three-head mixture, downstream PI reaches 77.0. The paper calls the failure mode decoder lock-in or head-specific representation collapse. The language is stronger than the direct measurement—the authors do not geometrically demonstrate a collapse—but the transfer matrix is persuasive. Strong performance when reusing the same head does not imply that the backbone contains generally accessible action information. It may contain information arranged around the exact decoder that generated its gradients. The VLAct recipe. Preserve the original VLM prior. VLAct begins with Qwen3-VL-4B. Qwen3-VL supplies broad visual, linguistic, spatial, and multimodal features, but robot continued pre-training exposes the model to a much narrower distribution: repeated workspaces, a small number of camera families, recurring manipulators, and action-correlated visual patterns. Updating everything end to end risks overwriting features that are not immediately useful for minimizing the robot loss but would be valuable during downstream generalization. The first countermeasure is shallow-layer protection. During continued pre-training, VLAct freezes the vision encoder and the lower half of the language-model layers while training the upper layers and action modules. The released configuration identifies the protected language layers as 0 through 17. During downstream adaptation, the entire model is unfrozen. The ablation is straightforward. Updating the full backbone produces 78.9 on LIBERO-Plus and 77.1 on RoboTwin. Freezing only the vision encoder raises those to 81.3 and 79.3. Protecting both the vision encoder and the lower half of the LLM reaches 82.6 and 80.5. This is consistent with the hypothesis that some plasticity should be reserved for high-level action conditioning while early perceptual alignment is protected from the narrow robot distribution. The second countermeasure is auxiliary vision-language co-training. Caption examples are mixed into robot minibatches so that trainable layers continue receiving dense supervision about objects, attributes, spatial relations, and scene context. The paper’s reported objective weights the auxiliary caption loss at 0.5 relative to the action objective. Grounding, spatial question answering, and even text-only instruction data are explored as alternatives. Every tested auxiliary source improves over action-only pre-training under a fixed budget, with image captions providing the strongest single-source result. The text-only result is especially informative. Since it supplies no direct visual or motor knowledge, its gain is difficult to explain as task transfer. A more plausible interpretation is regularization against domain collapse: periodically training on a broader foundation-model objective prevents every trainable direction from being organized around the correlations of the robot corpus. Use multiple continuous decoders as representation teachers. VLAct attaches OFT, PI, and GR00T heads to one shared backbone latent. All three predict the same future action chunk, and their losses are summed with equal weights. The backbone forward pass is shared, so the additional cost lies mainly in the action modules rather than three repetitions of the four-billion-parameter VLM. A useful interpretation is that the heads act as different views of the same control target. To satisfy OFT, the latent must make continuous actions directly recoverable. To satisfy PI, it must condition a flow field over noisy action trajectories. To satisfy GR00T, it must support a more explicitly separated semantic-and-motor stack. The information common to these demands is more likely to describe task phase, target geometry, contact state, and intended motion than quirks of one decoder. The same-head results suggest this is not merely a compromise that trades peak performance for portability. OFT fine-tuning improves from 78.8 after OFT-only pre-training to 80.5 after head-diverse pre-training. PI rises from 75.4 to 77.0, and GR00T from 71.7 to 76.0. Multi-head supervision therefore improves both transfer across heads and adaptation back into a familiar head. There is no additional ground-truth information in those three losses; each decoder sees the same action chunk. The benefit must come from optimization geometry and inductive-bias diversity. In that sense, VLAct resembles multi-view representation learning: the model is pushed toward features that remain useful under several parameterizations of the output problem. Share only the action semantics that genuinely align. Cross-embodiment training introduces a second specialization hazard. If every robot has an independent output projector, physically comparable signals can be isolated behind robot-specific parameters. But forcing all robots into one naively shared coordinate system is also wrong: the seventh joint of one arm need not correspond to anything on another robot. VLAct uses a 20-dimensional padded action layout. The first six dimensions represent the left AgileX arm’s absolute joint angles, and the next six represent its right arm. Dimensions 13 through 18 are reserved for the Franka’s delta end-effector pose. Dimension 19 is shared between the Franka gripper and AgileX’s left gripper, while dimension 20 represents the right AgileX gripper. Inactive dimensions are masked from the loss. This is only partially unified, and that restraint is important. The method preserves each dataset’s native controller convention rather than converting every trajectory through inverse kinematics or retargeting. Franka remains in delta end-effector space; AgileX remains in absolute joint space. The main explicit semantic alignment is the normalized open-close gripper coordinate, while the shared head and backbone still couple learning across embodiments. Separate embodiment heads obtain 78.5 on RoboTwin and 81.1 on LIBERO-Plus. Moving to a shared head without the full action alignment gives 79.5 and 81.4. The partially unified representation reaches 80.5 and 82.6. The effect is not as dramatic as multi-head supervision, but it is consistent across both embodiments. Respect the topology of joint angles. Absolute revolute-joint angles are periodic. Numerically, positive 179 degrees and negative 179 degrees look almost a full revolution apart, although physically they differ by only two degrees. VLAct first wraps joint targets into the canonical interval from negative pi to positive pi. It then wraps the prediction residual before applying the loss, so errors are measured along the shorter path around the circle. On RoboTwin Base Clean, raw joint regression obtains 75.5. Canonicalizing the targets raises success to 78.6, and adding the wrap-aware residual loss reaches 80.5. This is a reminder that representation learning includes the numerical geometry of the action targets, not just transformer architecture. The cleaning pipeline similarly normalizes differences that should not become dataset identifiers. Franka delta actions are rescaled by control frequency into a per-second representation. Implausibly large steps are masked rather than immediately causing an entire trajectory to be discarded, and gripper values are mapped to a common interval from zero to one. What was actually trained and released?. The main continued-pre-training mixture combines Franka data from DROID and MolmoAct with AgileX data from InternData-A1 and RoboCOIN, plus caption supervision. The public checkpoint card separates out an additional DROID-100 dataset group. The backbone predicts 50-step action chunks in the released configuration, using a 20-dimensional padded action and a 14-dimensional state representation. The reusable checkpoint was trained for 100,000 steps with AdamW, a 5,000-step warmup, cosine decay, and a per-GPU VLA batch size of 16. The model card lists learning rates of (10^{-5}) for the Qwen-VL interface and (10^{-4}) for action and base modules. The paper reports a 16-GPU continued-pre-training setup. There is, however, a reproducibility wrinkle. The repository explicitly distinguishes the paper protocol from the packaged source run. The paper uses an auxiliary caption weight of 0.5, whereas the checked-in launcher and artifact configuration record 0.2. The repository also notes a difference between the paper’s 16-GPU description and a packaged multi-node configuration. Anyone reproducing headline tables should follow the paper protocol; anyone reproducing the downloadable 100,000-step artifact should inspect its resolved configuration. The release is otherwise unusually complete. The code is under the MIT License, while the backbone checkpoint is under Apache 2.0, subject to the licenses of Qwen3-VL and the training datasets. The repository includes pre-training, cleaning, benchmark, fine-tuning, and deployment infrastructure. The checkpoint is not a directly deployable robot policy or a drop-in. Users must define camera ordering, image preprocessing, proprioceptive state, action parameterization, normalization, control rate, and safety constraints. For a new action space, the intended procedure is to load the shared backbone and randomly initialize the downstream action module. There is also a checkpoint-selection trap. The 92.5% RoboTwin headline uses OFT, while the released RoboTwin policy card uses GR00T. The public LIBERO-Plus and VLA-Arena policies use PI even though the paper’s main comparison tables report OFT. The raw continued-pretraining backbone—not a benchmark checkpoint—is therefore the correct starting point for a new embodiment. Reading the results carefully. The cleanest evidence comes from matched Qwen3-VL-OFT comparisons in which only the backbone initialization changes. | Benchmark | VLAct | Matched Qwen3-VL-OFT baseline | |---|---:|---:| | LIBERO-Plus | 82.6% | 75.0% | | VLA-Arena | 54.8% | 33.4% | | RoboTwin 2.0 Base, Clean | 80.5% | 61.7% | | RoboTwin Data Scaling, Clean / Random | 92.5% / 90.8% | 88.2% / 88.3% | | DOMINO, Success / Manipulation Score | 18.50 / 34.20 | 10.86 / 30.49 |. Franka simulation. LIBERO-Plus evaluates perturbations in camera viewpoint, robot state, language, lighting, background, image noise, and object layout. VLAct’s largest improvements over the matched baseline occur on camera changes, robot variation, noise, and layout. Camera success rises from 47.0 to 73.9, while noise robustness rises from 73.1 to 86.0. That pattern supports the claim that continued pre-training improves visual-spatial robustness. The improvement is not universal. On the language perturbation axis, VLAct scores 81.5 versus 87.0 for the matched baseline. The total still rises substantially, but caption mixing and prior preservation do not automatically improve every language-conditioned generalization measure. VLA-Arena gives the largest aggregate matched gain: 54.8 versus 33.4. VLAct reaches 63.2 on safety tasks, 64.0 with distractors, 36.9 on extrapolation, and 50.0 on long-horizon suites. The corresponding baseline values are 36.1, 40.0, 24.7, and 32.7. Bimanual and dynamic simulation. RoboTwin separates a Base setting, with 50 clean trajectories per task, from a Data Scaling setting that adds 500 domain-randomized expert demonstrations per task. Across 50 tasks, the latter corresponds to 2,500 clean and 25,000 randomized demonstrations. These randomized demonstrations are successful expert rollouts collected under varied backgrounds, clutter, lighting, and table geometry—not random actions. In the low-data Base setting, VLAct-OFT produces the most striking gains: 80.5 versus 61.7 on Clean and 41.5 versus 10.5 on Random. Once large downstream randomized data are added, the gap narrows to 92.5 versus 88.2 on Clean and 90.8 versus 88.3 on Random. This is the expected shape for a good initialization: its relative value is greatest when downstream coverage is limited, although it continues to help after scaling. Head choice remains consequential. In the Data Scaling setting, PI reaches a slightly higher 93.0 on Clean, while OFT gives the stronger Random result of 90.8. No single decoder dominates every condition, reinforcing the paper’s premise that the backbone should not be tied to one action interface. On DOMINO’s 35 dynamic manipulation tasks, VLAct raises success from 10.86 to 18.50 and manipulation score from 30.49 to 34.20. That is a large relative improvement, but the absolute success rate remains below 20%. Representation-centric pre-training helps with moving objects and evolving scenes; it does not solve dynamic bimanual manipulation. Physical Franka experiments. The physical evaluation uses table-mounted Franka Research 3 arms, a fixed external RealSense D435, and wrist-mounted D405 cameras. Single-arm tasks receive 50 demonstrations each, while dual-arm tasks receive 100. The authors train separate single-arm and dual-arm models for 50,000 steps on eight H800 GPUs. Each task is evaluated from the same ten fixed robot and object configurations for both VLAct and the baseline. VLAct averages 92.5% on in-domain short-horizon tasks, compared with 77.5% for Qwen3VL-OFT. Novel-object extraction and placement both reach 90%. On longer tasks, table cleaning reaches 86.6% and scooping beans reaches 80.0%, compared with 73.3% and 33.3%. Extended and fully substituted out-of-domain table-cleaning variants reach 82.5% and 83.3%, versus 47.5% and 46.6%. Dual-arm coordination averages 72.0%, compared with 44.0%. These results cover meaningful contact-rich behaviors, including folding, handover, unplugging, and tool use. But they remain small-sample experiments: ten fixed rollouts per task, with partial-credit scoring for some long-horizon tasks. They demonstrate a promising initialization effect rather than broad real-world reliability. Previously unseen embodiments. The GR-1 and ARX X5 robots do not appear in VLAct continued pre-training. On RoboCasa-GR1, 20% of downstream data gives 49.5% success, 50% gives 51.0%, and all data gives 54.0%. Because fine-tuning is still required, this is cross-embodiment transfer rather than zero-shot control. On the official RoboDojo snapshot dated August 24, 2026, VLAct records a 10.66 partial-progress score and 7.60% success on ARX X5. It ranks eighth of 35 by score and sixth by success, outperforming all entries explicitly categorized as world-action models in that snapshot. The absolute success rate is nevertheless low, and its Memory category is particularly weak. RoboDojo should therefore be read as evidence of competitive transfer under a difficult benchmark, not near-solved generality. An additional experiment adds 20,000 UMI-style trajectories from a different collection pipeline. LIBERO-Plus rises from 82.6 to 83.7 without other changes. This modest gain suggests that the recipe can absorb another action-data source without destabilizing the learned representation. Where VLAct fits in the VLA landscape. Open X-Embodiment, RT-X, and Octo made cross-robot data diversity the central mechanism of transfer. VLAct is complementary: it asks how the optimization recipe should shape a reusable VLM when the data budget is held fixed. Octo emphasizes architectural flexibility across observation and action spaces; VLAct emphasizes decoder-agnostic information inside a vision-language backbone. OpenVLA and OpenVLA-OFT provide the most direct intellectual precursor. OpenVLA showed that an open VLM-derived action model can scale across hundreds of thousands of demonstrations. OpenVLA-OFT then demonstrated that parallel decoding, continuous actions, chunking, and a simple regression objective can dramatically improve adaptation speed and success. VLAct moves the decoder-design question upstream: instead of selecting one good head and allowing it to shape pre-training, it uses several heads to make the resulting backbone less dependent on any one of them. GR00T N1 also separates vision-language understanding from a diffusion-transformer motor module. VLAct borrows that style as one of several supervisory views rather than committing to it as the final architecture. This is a subtle but consequential distinction: the motor architecture becomes a temporary teacher of representation structure. There is also a connection to earlier representation-first robotics work such as Voltron. Voltron combined visual reconstruction and language generation because neither low-level spatial structure nor high-level semantics was sufficient across the full range of robotics tasks. VLAct extends a similar balancing principle into action learning: captions preserve broad semantic grounding, while heterogeneous continuous decoders pressure the representation to expose fine motor information in more than one computational form. What the paper establishes—and what remains open. The strongest contribution is methodological rather than architectural. Freezing layers, mixing captions, predicting action chunks, masking padded dimensions, and wrapping periodic angles are individually familiar tools. VLAct’s novelty is to organize them around a clear target: the transferable action representation inside the backbone, evaluated by discarding the pre-training heads and controlling the downstream recipe. The head-transfer experiment is particularly valuable. VLA papers often report only adaptation with the same action interface used during pre-training. VLAct shows why that can be misleading. A better evaluation standard would include a matrix in which rows denote pre-training heads and columns denote downstream heads, alongside sample-efficiency curves and held-out embodiments. Still, “representation collapse” is inferred through downstream optimization rather than measured directly. The paper does not provide a detailed VLM-retention evaluation, feature-similarity analysis, causal probing of affordances, or a linear-probe suite for task phase and geometry. Such diagnostics could distinguish among several explanations: genuinely richer physical representations, better conditioning of the optimization landscape, implicit ensembling of gradient signals, or simple regularization. The cross-embodiment claim is also bounded by the manually designed schema. Only two main pre-training embodiment families are represented, and most arm dimensions remain disjoint. Sharing a normalized gripper scalar is useful, but scaling the approach to mobile manipulation, hands, torque control, locomotion, or humanoid whole-body actions may require a more principled semantic action ontology. Compute should not be ignored either. Sharing the VLM forward pass makes three-head training cheaper than running three complete VLAs, but two flow-based motor modules still add memory and computation. The paper does not deeply study whether two heads are enough, whether equal loss weights are optimal, or how gradient conflict evolves as more decoders and embodiments are added. Finally, only matched in-house comparisons cleanly isolate the backbone. Comparisons against published industrial systems involve different data, compute, checkpoint selection, and optimization recipes. The authors generally acknowledge this distinction, but readers should resist collapsing every benchmark table into a single model ranking. The paper itself identifies model scale as a limitation: experiments are restricted to a four-billion-parameter VLM. A larger backbone may retain its prior differently, require less freezing, or respond differently to multi-head gradients. How I would use VLAct in a robotics lab. First, I would begin with the raw continued-pretraining backbone, not one of the benchmark-specific policies. I would explicitly define the target robot’s camera contract, proprioceptive inputs, control frequency, action horizon, normalization, and safety envelope, then initialize a new head unless the existing action contract matches exactly. Second, I would reproduce the core idea rather than blindly preserve the 20-dimensional layout. Canonicalize dimensions only when their physical meanings genuinely agree. Mask missing dimensions, preserve native controller spaces when retargeting would add artifacts, and use topology-aware losses for angles, rotations, and other constrained quantities. Third, I would choose action heads that are structurally different. A one-shot L1 regression head, a conditional flow head, and a more decoupled motor transformer offer meaningfully different gradients. Three superficially different MLPs probably would not. The goal is not an ensemble at deployment; it is to identify the intersection of information required by different decoder families. Fourth, I would monitor preservation directly. Caption loss is useful, but I would add frozen evaluation suites for object recognition, spatial grounding, instruction following, and possibly point-affordance prediction. Layer freezing should be selected from those retention curves and downstream transfer, not assumed to generalize unchanged across VLM scales. Finally, I would make the backbone-head transfer matrix part of model selection. Evaluate every continued-pretraining checkpoint with at least one unseen decoder, a low-data target embodiment, and a visual-distribution shift. If pre-training improves only the head that generated its gradients, it has probably produced a specialized policy initialization rather than a reusable action backbone. The public implementation makes these experiments accessible. The key controls expose protected modules, auxiliary VLM loss scaling, the action-head list, masked padded dimensions, and wrap-aware losses. Just keep the paper-versus-artifact configuration differences in view when reporting reproductions. Closing perspective. VLAct’s central message is not that data scaling has stopped mattering. It is that trajectories do not convert automatically into transferable physical intelligence. The decoder, action parameterization, frozen layers, auxiliary objectives, and cross-embodiment layout determine what the backbone learns to retain. The most useful mental model is that continued-pretraining heads are temporary instruments. Each one pulls the latent space toward a different notion of what makes an action predictable. By forcing several of them to agree while protecting the original VLM prior, VLAct tries to leave behind a representation that knows more about acting than any single decoder needs. That is a compelling direction for VLA research: not merely asking how many robot trajectories we can collect, but how much reusable physical structure we can make each one teach.