Latest / Embodied AI 101 / Expand Before You Compress: TEAM-VLA's Training-Free Shortcut to Faster Robot Policies
Transcript
- Expand Before You Compress: TEAM-VLA’s Training-Free Shortcut to Faster Robot Policies. Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models • Yifan Ye, Jiaqi Ma, Jun Cen, and Zhihe Lu • Hamad Bin Khalifa University, Mohamed bin Zayed University of Artificial Intelligence, and Zhejiang University • IEEE Robotics and Automation Letters • 2026. Vision-language-action models inherit an awkward systems property from the large vision-language models beneath them: the model may contain billions of parameters, but a large part of the runtime cost comes from repeatedly processing a long sequence of visual tokens. For a chatbot, an extra few dozen milliseconds might be tolerable. For a robot waiting to correct its gripper pose, react to object motion, or recover from a partial grasp, that delay is part of the control loop. TEAM-VLA asks a focused question: How much of the image does the language backbone really need to process at every layer?. Its answer is not simply “keep the highest-attention patches.” In robotic manipulation, those patches tend to be far too sparse. A language-image similarity map might place one token on the gripper, one on a cup handle, and another on the target receptacle, while omitting the geometry connecting them. Pruning from that sparse map is fast but brittle. The paper’s central move is therefore deliberately counterintuitive: expand before compressing. TEAM-VLA first turns sparse task-relevant patches into more complete spatial regions, prunes the rest before the expensive language backbone, and then merges the remaining representations at a deeper layer once the model has developed action-aware semantics. On OpenVLA-OFT and the four standard LIBERO evaluation suites, this reduces average CUDA inference latency from 109 to 72.1 milliseconds and relative FLOPs from 100% to 39%, while matching the uncompressed model’s 96.6% average success rate. That is roughly a 1.51-times speedup and a 61% reduction in theoretical computation without retraining the policy. The accessible manuscript is arXiv version 1, submitted on December 10, 2025, and associated with the 2026 RA-L article. The discussion here follows that version, including several implementation ambiguities that matter for reproduction. Why Token Compression Is Harder for Robots. OpenVLA is a 7-billion-parameter VLA built from a Llama 2 language backbone and a visual encoder combining DINOv2 and SigLIP features. It was pretrained using 970,000 real-world robot demonstrations. OpenVLA-OFT subsequently improved its adaptation recipe with continuous actions, parallel action decoding, and action chunking, raising average LIBERO success from 76.5% to 97.1% while dramatically increasing action-generation throughput. TEAM-VLA starts from this already optimized OpenVLA-OFT policy and attacks a different bottleneck: the token sequence moving through the multimodal backbone. The attraction of token compression is straightforward. Transformer attention grows approximately quadratically with sequence length, and multimodal models may inject hundreds of image-patch tokens into every language-model layer. Removing a token before the first backbone layer avoids its cost throughout almost the entire network. Removing it near the end is much safer, because the representation is already semantically mature, but most of the computation has already been paid. This creates an efficiency-versus-information dilemma:. Prune early, and the runtime benefit is large but the relevance signal is weak. Prune late, and relevance is easier to estimate but the speedup is small. Prune aggressively, and the robot can lose object boundaries, gripper geometry, or spatial context. Retain generous context, and much of the computational benefit disappears. The problem is intensified by the form of robotic language. General VLM prompts can contain rich questions and descriptions that create strong cross-modal attention. A manipulation instruction may be only “put both moka pots on the stove.” Some tokens denote objects; others, such as “put,” “the,” and “on,” are functional terms. If every language token is forced to select its most similar image patch, non-object words can generate isolated and essentially arbitrary matches. Even the meaningful noun phrase may activate only one or two patches on a much larger physical object. Other efficient-VLA methods often compensate with trainable token selectors or temporal evidence. Comparing the current frame with the previous frame is a useful way to discover moving arms and manipulated objects, but it requires image history and assumes frame-to-frame continuity. That assumption becomes less reliable after camera motion, episode resets, occlusion, abrupt viewpoint changes, or dropped frames. TEAM-VLA instead uses only the current observation and does not learn any additional parameters. The TEAM-VLA Pipeline in One Pass. TEAM-VLA modifies neither the pretrained weights nor the policy’s training objective. It inserts two operations into an existing VLA forward pass:. Foreground-oriented pruning before the language backbone. Sparse image-language matches are expanded spatially and combined with sampled scene context. Visual tokens outside this mask are discarded. Action-guided merging inside the language backbone. At a middle layer, a small set of action-relevant visual tokens is retained as anchors, and the remaining visual representations are softly aggregated into them. The first stage creates most of the speedup because it shortens the sequence before expensive backbone processing. The second stage reduces the remaining sequence more conservatively, using representations that have already interacted with language and action tokens. One important clarification is that “token expanding” does not create more tokens than the original image encoder produced. It expands the retention mask around sparse matches. The method temporarily keeps more neighboring patches than a naive similarity rule would select, but it still removes most of the original visual sequence. Stage One: Recovering Objects from Sparse Similarity Anchors. Starting with high-precision, low-recall matches. After the image encoder has produced patch embeddings, TEAM-VLA compares every task-language embedding with the image tokens using cosine similarity. For each language token, it selects the single image patch with the highest similarity. This gives a small collection of foreground anchors. Some language tokens may select the same patch, so the number of unique anchors can be smaller than the instruction length. The selected patches are projected back onto the image grid to form a binary mask. This is extremely cheap, but it is not a segmentation map. It behaves more like a set of noisy visual keypoints. A token associated with “moka” may land on the pot body; “pots” may land near a handle; “stove” may mark one burner. Large parts of all three objects—and perhaps most of the arm—remain unselected. TEAM-VLA treats these initial matches as evidence that a useful region is nearby, not as a final decision about which exact patches matter. Turning anchor density into spatial expansion. The binary mask is scanned with a small all-ones convolutional kernel, normally described as a three-by-three window. The resulting density map records how many selected anchors occur in each patch’s local neighborhood. Dense clusters are interpreted as strong evidence of a meaningful task region. TEAM-VLA expands these clusters across their local neighborhoods, producing an operation similar to conditional morphological dilation. If several language tokens point to nearby patches, the method assumes that the surrounding object surface or robot structure is also relevant. Isolated activations are treated more cautiously. Rather than expanding an entire neighborhood, the method adds one randomly chosen neighboring patch. The goal is to prevent a potentially useful cue from remaining a single disconnected point without allowing every noisy language match to grow into a large retained region. The conceptual logic is sensible for manipulation scenes:. Real task objects and robot links generally occupy contiguous image regions. Several semantically related tokens may produce nearby anchors. Function words are more likely to produce scattered, uncorrelated activations. Local density therefore offers a cheap proxy for distinguishing object-related clusters from linguistic noise. The paper reports that this foreground-identification stage takes roughly one to two milliseconds even with both agent-view and wrist-view images, because the masks can be processed as a batch by standard convolution operations. Its first figure contrasts this with approximately 980 milliseconds for a Grounded-SAM pipeline and roughly 0.5 milliseconds for temporal patch differencing. The expansion approach is somewhat more expensive than differencing but remains buffer-free. Sampling the background instead of deleting it all. A manipulation policy cannot operate using object appearance alone. It may need the relationship between an object and a receptacle, available free space around the gripper, the edge of a table, or a landmark that helps interpret depth and orientation. TEAM-VLA therefore retains a small fraction of otherwise unselected patches through what the paper calls context sampling. The manuscript describes this as a lightweight interval-based strategy over the token sequence, controlled by a retention ratio (u). In practical terms, it preserves a sparse, roughly uniform sketch of the wider scene. The final early-retention mask is the union of the expanded task regions and these context patches. All other visual tokens are removed before entering the language backbone, while language, robot-state, and action-related tokens continue normally. This component is more important than it may first appear. A purely object-centric mask can identify what should be grasped while losing where it must go. Context sampling gives the policy a low-resolution scene scaffold without paying to process every background patch. Stage Two: Action-Guided Soft Token Merging. Early pruning removes obvious visual redundancy, but the remaining tokens are disproportionately valuable. Applying another hard pruning step risks deleting precisely the fine-grained information that survived the foreground-selection stage. TEAM-VLA therefore switches from deletion to merging. At an intermediate transformer layer, the method ranks the remaining image tokens according to their relevance to task and action representations. The top (M) tokens become a protected source set. All other visual tokens become targets that will be absorbed into those sources. Unlike random or alternating source selection in generic token-merging algorithms, this makes the surviving token positions explicitly action-oriented. The source and target representations are RMS-normalized, and their pairwise similarities are calculated. A softmax then converts those similarities into assignment weights: each target token contributes primarily to the source tokens that are most semantically similar to it. The weighted target features are accumulated at each source, after which the source representation is normalized by the amount of assignment mass it received. The result is a sequence containing only the (M) merged visual tokens. The remaining transformer layers process these compressed representations. This is not lossless compression, but it avoids the all-or-nothing behavior of pruning. A patch on the edge of a cup may not deserve its own token, yet its feature can still contribute to a retained token representing the cup body. Background geometry can similarly be folded into a nearby action-relevant representation rather than disappearing completely. The timing is essential. In the earliest layers, token features mostly describe local appearance, making representational averaging hazardous. By the middle of the backbone, multimodal interaction has made the visual features more semantic and action-conditioned. Similarity in that space is more likely to correspond to functional redundancy rather than superficial visual resemblance. Experimental Design. TEAM-VLA is evaluated on OpenVLA-OFT using the LightVLA codebase. The experiments run on a single NVIDIA A100 with 40 gigabytes of memory. The benchmark is LIBERO, using the Spatial, Object, Goal, and Long suites. These cover spatial relationships, object-level generalization, varying goals, and longer-horizon manipulation procedures. The original LIBERO benchmark contains 130 language-conditioned manipulation tasks and was designed to test the transfer of declarative and procedural knowledge. The reported metrics are task success rate, relative FLOPs, and CUDA latency. Latency is averaged over the ten tasks in each evaluated suite. The compression budget is not fixed across suites. The authors use:. A context ratio of 0.1 and 50 final source tokens for LIBERO-Spatial. A context ratio of 0.35 and 130 source tokens for LIBERO-Object. A context ratio of 0.25 and 80 source tokens for both LIBERO-Goal and LIBERO-Long. This task-dependent configuration reflects different visual-complexity requirements, but it also means TEAM-VLA is training-free rather than tuning-free. A practitioner must still select token budgets and context ratios for the deployment domain. The comparison includes SparseVLM, VLA-Cache, EfficientVLA, and SpecPrune-VLA. These represent several efficiency strategies: generic multimodal token sparsification, reuse of unchanged visual tokens across frames, broader training-free model compression, and history-aware action-conditioned pruning. Main Results: Matching Success at Two-Thirds the Latency. The aggregate result is the paper’s strongest claim:. | Method | Average success | CUDA latency | Relative FLOPs | Uses prior frames | |---|---:|---:|---:|---:| | OpenVLA-OFT | 96.6% | 109.0 ms | 100% | No | | SparseVLM | 95.5% | 85.3 ms | 77% | No | | VLA-Cache | 96.9% | 101.7 ms | 83% | Yes | | EfficientVLA | 88.9% | 70.6 ms | 35% | No | | SpecPrune-VLA | 96.5% | 75.1 ms | 43% | Yes | | TEAM-VLA | 96.6% | 72.1 ms | 39% | No |. TEAM-VLA exactly matches the baseline’s average success rate while reducing latency by 36.9 milliseconds, or about 33.9%. The theoretical FLOP count falls by 61%. The per-suite results reveal that “no average degradation” does not mean every suite is unchanged. Relative to OpenVLA-OFT:. Spatial improves from 97.6% to 99.2%. Object remains effectively unchanged at 96.5%. Goal falls from 97.9% to 97.0%. Long falls from 94.5% to 93.8%. The gains and losses happen to average to the same overall 96.6%. This is still an impressive trade-off, but it is more accurate to describe TEAM-VLA as preserving aggregate performance than as uniformly improving the full policy. Against EfficientVLA, TEAM-VLA gives up only 1.5 milliseconds while recovering 7.7 percentage points of average success. Against SpecPrune-VLA, it is approximately three milliseconds faster, uses slightly fewer FLOPs, and avoids historical observations. VLA-Cache produces the highest average success at 96.9%, but its latency remains close to the uncompressed model. The paper’s token-count visualization reports that TEAM-VLA retains only about 16.6% of the baseline visual tokens while maintaining the same success rate. This percentage refers to visual tokens after merging, not the complete multimodal sequence and certainly not the entire model computation. The vision encoder still processes the original images, and the first part of the language backbone processes the tokens surviving stage one. That is why reducing the final visual-token count by more than 80% translates into a 61% FLOP reduction and a 34% latency reduction rather than an equivalent wall-clock improvement. Interpreted as policy-call throughput, 109 milliseconds corresponds to about 9.2 model evaluations per second, while 72.1 milliseconds corresponds to roughly 13.9. These are not full robot-control frequencies—the figures exclude sensing, preprocessing, communication, and actuation—and OpenVLA-OFT uses action chunking, which decouples action rate from policy-refresh rate. Nevertheless, a shorter refresh delay can improve responsiveness to deviations from the predicted chunk. What the Ablations Tell Us. The component ablations support the paper’s core argument that sparse language similarity is not sufficient for early pruning. On the ten-task ablation suite, retaining context alone reaches 92.1% success at a relatively generous context ratio of 0.4. Using a smaller 0.25 context ratio plus three-by-three token expansion reaches 93.8% while pruning a similar number of tokens. Keeping only the most similar language-related patches performs substantially worse, at 89.5%. This is evidence that the gain comes from reconstructing spatial neighborhoods rather than merely adding a few semantic anchors. A larger five-by-five expansion kernel raises success only slightly, from 93.8% to 94.1%, but increases latency from 76.8 to 82.5 milliseconds. Larger expansion preserves more visual context, but it weakens the compression advantage. The final merging budget shows a clear knee. Compressing to 50 visual tokens gives only 88.3% success. Raising the budget to 80 recovers 93.8%. Increasing it to 110 or 130 provides no consistent further gain. This suggests that the useful information has a finite compact representation, but the minimum viable budget is not extremely small. The layer ablation is particularly informative. At layers 1, 4, and 8, direct pruning actually performs better than merging. At layer 16, the pattern reverses: merging obtains 93.8% success, compared with 92.1% for pruning. In other words, merging becomes advantageous only after the representations are sufficiently mature. This reinforces the design principle of pruning obvious redundancy early and reserving aggregation for a deeper layer. The expansion threshold exposes the expected Pareto frontier. Aggressive expansion retains an average of 342 patches, reaches 97.2% success, and takes 86 milliseconds. A middle setting retains 196 patches, reaches 96.6%, and takes 75.3 milliseconds. The most restrictive setting retains 151 patches and lowers latency to 71.8 milliseconds, but success falls to 94.6%. How TEAM-VLA Fits into the Efficiency Landscape. TEAM-VLA’s merging stage descends conceptually from Token Merging, or ToMe, which showed that similar vision-transformer tokens can be combined without retraining and often with less accuracy loss than hard pruning. TEAM-VLA’s embodied-AI contribution is to choose merge anchors using task and action relevance, and to pair merging with a separate foreground-reconstruction stage before the language backbone. Its contrast with recent VLA methods is equally important. VLA-Cache exploits temporal redundancy: if a patch has barely changed since the previous observation, its computation can be reused through a key-value cache. That can be highly effective in stable scenes, but it introduces state and a dependency on temporal continuity. TEAM-VLA gives up reuse in exchange for a stateless, current-frame operation. SpecPrune-VLA also uses past context and classifies actions as coarse or fine-grained to change pruning aggressiveness. TEAM-VLA avoids that controller and instead tries to make first-frame visual selection reliable enough that historical evidence is unnecessary. EfficientVLA is broader in scope. It removes redundant language layers, performs task-aware visual-token selection, and caches features in a diffusion action head. Its published result is on CogACT and SIMPLER, so its headline numbers are not directly comparable with TEAM-VLA’s OpenVLA-OFT and LIBERO evaluation. Conceptually, however, the two approaches may be complementary: TEAM-VLA concentrates on spatial token organization, while EfficientVLA addresses redundancies across several architectural modules. The deeper significance is that TEAM-VLA treats VLA compression as an embodied problem rather than importing a generic VLM pruning rule unchanged. A robot needs more than semantic recognition. It needs object extent, arm configuration, free space, contact geometry, and the relation between source and destination. The expansion stage protects spatial continuity; the merging stage protects action-conditioned semantics. Limitations and Reproducibility Concerns. The experimental evidence is promising but narrow. TEAM-VLA is evaluated on one VLA family, one simulation benchmark, and one high-end data-center GPU. There are no real-robot experiments, no embedded or edge-GPU measurements, and no evaluation on flow-matching or diffusion-heavy VLAs where the action head may dominate runtime. There are also no reported measurements of memory consumption, energy, trajectory smoothness, collision rate, recovery behavior, or action-distribution drift. The vision encoder remains uncompressed. All image patches must first be encoded before TEAM-VLA can compare them with language. If a deployment is dominated by high-resolution visual encoding rather than language-backbone attention, the achievable end-to-end speedup may be smaller. The main comparison is not fully controlled. The authors state that their own reruns of OpenVLA-OFT and the baselines produced lower success rates than earlier reports. They therefore use baseline numbers reported by SpecPrune-VLA and other papers, arguing that those experiments used similar hardware and latency conditions. This makes the table informative but less conclusive than running every method in the same repository, environment build, checkpoint, and evaluation harness. The manuscript also contains several specification ambiguities. The implementation section says the expansion threshold is set to two, while the threshold ablation later states that one is the default for the main experiments. In addition, the density map is formed by summing a binary mask and is therefore integer-valued, yet the written rules use strict inequalities that leave values exactly equal to the threshold unassigned. With a threshold of one, the stated interval for “sparse” neighborhoods contains no integer values. The intended implementation may use non-strict boundaries, but the paper does not clarify this. The soft-merging equations have a related notation issue. The manuscript computes the total matching mass received by each source token, but the following displayed normalization appears to divide by one plus the source-set size rather than one plus that matching mass. The stated similarity-matrix dimensions and multiplication order also appear transposed. The surrounding prose makes the likely intended weighted-average operation understandable, but an exact reproduction requires resolving these details. Unfortunately, the code does not currently resolve them. As of August 1, 2026, the official repository contains a short README and an architecture image, but no PyTorch implementation. An open issue posted on March 9, 2026 asks why the code has not been released and has no visible author response. This conflicts with the paper’s statement that the code is publicly available and materially limits reproducibility. Finally, both sparse-region expansion and context selection include random choices, yet the paper does not provide seed-sensitivity or variance analysis for the compressed policy. In a closed-loop setting, small representational changes can compound over a trajectory. Reporting only average task success makes it difficult to tell whether compression changes failure consistency or creates rare but severe outliers. The Broader Lesson. TEAM-VLA’s most useful contribution may be its decomposition of token compression into two qualitatively different decisions. Before the language backbone, token selection should prioritize recall and spatial completeness. Sparse semantic anchors are expanded, and a little global context is retained. Deeper in the backbone, compression should prioritize semantic preservation. Tokens are merged into action-relevant representatives rather than simply deleted. That pattern—seed, expand, prune, and then merge—offers a practical design template beyond this particular OpenVLA-OFT experiment. It could potentially be combined with quantization, action caching, layer removal, or architecture-specific acceleration, although those combinations are not evaluated here. The result is not yet a turnkey deployment package: the benchmark scope is limited, parameters are task-dependent, the written algorithm needs clarification, and the promised implementation is missing. But the underlying insight is strong. For embodied models, the best token is not always the patch with the highest isolated attention score. Sometimes the information the robot needs lies in the neighborhood around that score—and the safest way to compress it is to preserve the neighborhood first.