Latest / Embodied AI 101 / TurboVLA and the Case for Taking the LLM Out of the Servo Loop

Transcript

TurboVLA and the Case for Taking the LLM Out of the Servo Loop · Embodied AI 101

• TurboVLA and the Case for Taking the LLM Out of the Servo Loop. TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM • Hengyi Xie, Chenfei Yao, Xianjin Wu, Xuanyang Xi, Yiping Tang, Di Xu, Yingying Zhu, Dingkang Liang, Xiang Bai, and Han Ding • Huazhong University of Science and Technology and Huawei Technologies • arXiv preprint • 2026. TurboVLA arrives with an unusually concrete headline for a vision-language-action paper: 0.2 billion parameters, 0.9 gigabytes of inference VRAM, and 31.2 milliseconds from multimodal input to a continuous action chunk on an RTX 4090. That works out to roughly 32 complete policy invocations per second. On LIBERO, the same model reports a 97.7 percent average success rate, placing it alongside—and in some cases above—systems with several billion parameters. The paper’s first version was submitted on July 29, 2026, with training code and checkpoints released immediately afterward. Those numbers are impressive, but the most interesting contribution is not really the frame rate. TurboVLA is making an architectural argument. The authors claim that many robot execution policies are paying for a large language model at every control decision even when the language problem has already been solved. Once the command is something like “put the red bowl on the plate,” the policy does not necessarily need open-ended generation, chain-of-thought reasoning, or autonomous task decomposition on every observation. It needs to identify the relevant objects, relate them to the command, and produce an appropriate motor trajectory. TurboVLA therefore replaces the now-common vision-to-language-to-action pipeline with a direct vision-plus-language-to-action pipeline. Vision and language are encoded separately, fused through a compact grounding-style interaction module, and decoded into an action chunk in one pass. The broader question raised by the paper is consequential: should a general-purpose language model be the representational center of a low-level robot policy, or should reasoning and execution be separated into different computational loops?. There is one qualification worth establishing at the outset. The headline’s 32 hertz refers to the rate at which the model can generate whole action chunks. It is not, by itself, evidence of a 32-hertz visual-feedback control loop. The released LIBERO evaluation executes a 12-step chunk for 12 open-loop environment steps before requesting another prediction. We will return to that distinction, because it matters when translating benchmark latency into real robot responsiveness. The Bottleneck TurboVLA Is Actually Attacking. The dominant VLA recipe has inherited its topology from vision-language models. An image encoder converts observations into visual tokens, those tokens are projected into a language model’s embedding space, and the language model jointly processes the image representation and instruction. Robot actions are then generated from that language-centered representation. There are two major versions of this design. In autoregressive VLAs such as the original OpenVLA, actions are discretized and emitted as tokens. This incurs the sequential decoding cost familiar from text generation. More recent models avoid that particular bottleneck by attaching continuous action experts, flow models, or parallel action heads. Yet the image and instruction still pass through a multi-billion-parameter multimodal backbone before the action head can run. TurboVLA argues that action-side acceleration leaves a large upstream cost untouched. This is an important distinction. A continuous action head can make action generation faster without making the complete policy lightweight. If the backbone still has to transform hundreds of visual tokens through dozens of large language-model layers at every replan, both activation memory and latency remain substantial. TurboVLA’s target is therefore not action tokenization alone. It is the use of a language model as the mandatory bridge between perception and control. The paper’s alternative depends on a specific decomposition of robot intelligence. High-level reasoning decides what skill should be executed. An execution policy determines how the current scene should move the robot toward that skill. For the latter, language can function as a structured conditioning signal rather than as a generative workspace. Objects, attributes, verbs, and spatial relations still matter, but a compact encoder may represent them adequately. That assumption is plausible for benchmark-style manipulation commands and many industrial skills. It is less plausible when the instruction is ambiguous, requires commonsense inference, or describes a long sequence that must be decomposed online. TurboVLA is explicit about this boundary: it is designed for concrete execution-level instructions, not as a replacement for an LLM planner. The authors ultimately suggest a hierarchical system in which expensive language reasoning occurs at a lower frequency and TurboVLA handles efficient execution. The Architecture: Ground First, Then Act. TurboVLA is less a new primitive than a carefully chosen composition of existing ones. It combines a strong pretrained visual encoder, a conventional bidirectional text encoder, Grounding DINO-style cross-modal interaction, and an ACT-style action-chunk decoder. The novelty lies in how these parts are connected—and in what has been removed. Separate visual and language streams. For LIBERO, the visual backbone is DINOv3 ViT-B. The policy receives two RGB camera views. For RoboTwin 2.0, it scales to DINOv3 ViT-L with three 224-by-224 views. Each view is converted into spatial patch tokens, projected into a shared 256-dimensional policy space, and augmented with positional and camera-view embeddings. The separate view identifiers are important because simple token concatenation would otherwise leave the policy to infer which camera produced each patch. The public implementation uses more than a bare linear projection. DINO features pass through normalization, a two-layer nonlinear projection, and a learned skip projection before entering the shared feature space. This preserves a direct route from pretrained visual features while still allowing the representation to be reshaped for control. The released LIBERO training recipe fully unfreezes the DINOv3 backbone, so its visual representation is not merely used as a fixed feature extractor. Language is handled by BERT-base rather than a generative decoder. Crucially, TurboVLA retains the complete sequence of text tokens instead of pooling the instruction into one vector. This preserves correspondences among words such as object names, attributes, and relations. “Place the red bowl behind the plate,” for example, should not be compressed so aggressively that red, bowl, behind, and plate become inseparable. The public code defaults to a frozen BERT backbone, followed by a trainable projection into the 256-dimensional policy space. BERT nevertheless runs online during every published forward pass; the repository explicitly states that no text-feature cache is required or used. That makes the latency number relatively conservative for settings where the command remains fixed throughout an episode: the frozen BERT output could be computed once and reused, while the subsequent vision-conditioned language updates would still run each cycle. Robot proprioception follows a third path. Joint or end-effector state is projected into a small set of state tokens and passed directly to the action decoder. It does not participate in the vision-language interaction module. Conceptually, the architecture first asks which scene content is relevant to the command, and only afterward combines that task-conditioned scene representation with the robot’s current configuration. That separation is sensible. Proprioception is necessary to convert a target-relative visual decision into an executable motion, but it is not generally needed to decide whether the instruction refers to the red bowl or the blue cup. Keeping state out of the grounding module also reduces the temptation for the policy to solve task identification through embodiment-specific correlations. Six layers of bidirectional grounding. The central module contains six bidirectional vision-language interaction layers. In one direction, language tokens attend to visual patches, producing instruction features adapted to the current scene. In the other direction, visual patches attend to language tokens, making the visual representation task selective. The resulting streams are concatenated and supplied to the action decoder. All interaction occurs in the relatively narrow 256-dimensional policy space. This bidirectionality is more meaningful than it might first appear. Language-to-vision attention can emphasize the pixels relevant to “the mug beside the bowl.” Vision-to-language attention can resolve what those words mean in the current image—perhaps which of several mugs is actually beside a bowl. A one-way language-conditioned visual encoder captures the first process but not necessarily the second. The interaction weights are initialized from the feature-enhancement components of Grounding DINO. This is an important part of the pretraining story. TurboVLA does not use embodied pretraining beyond the target robot datasets in its main LIBERO and RoboTwin comparisons, but neither is it trained from scratch. It inherits visual priors from DINOv3, linguistic priors from BERT, and cross-modal alignment priors from a visual grounding model. “No embodied pretraining” should therefore be read as “no additional robot-trajectory pretraining,” not “no pretrained knowledge.”. The grounding initialization also helps explain why removing the generative language model does not remove visual-language semantics. Much of what an execution policy needs from language is referential: identify a noun phrase, discriminate an attribute, or interpret a spatial relation. Object-grounding pretraining is arguably more directly aligned with that requirement than next-token prediction is. Parallel continuous action chunks. The fused visual and language tokens, together with state tokens, become memory for an ACT-style transformer decoder. The decoder contains one learned query for each future action step. All queries attend to the multimodal memory in parallel and produce the entire chunk in a single forward pass. For LIBERO, TurboVLA predicts 12 seven-dimensional actions. For RoboTwin, it predicts 50 fourteen-dimensional absolute joint-position actions. There is no action vocabulary, autoregressive token loop, diffusion denoising trajectory, or iterative flow-sampling procedure. Training is ordinary behavior cloning with an L1 loss against expert action chunks. Architecturally, that is the source of much of the model’s simplicity. The action decoder does not have to model a distribution through repeated sampling, and the text encoder does not have to support language generation. Each component is specialized for the information transformation it actually performs. The resulting LIBERO model has approximately 216 million parameters, rounded to 0.2 billion in the headline. At BF16 precision, the raw parameter payload is only on the order of 0.43 gigabytes. A reported peak of 0.9 gigabytes is therefore plausible once visual tokens, intermediate activations, attention workspaces, and framework overhead are included. The model is small by contemporary VLA standards, although 216 million parameters is still substantial compared with classical visuomotor policies. How the Evaluation Was Constructed. The main LIBERO experiment jointly trains over all four standard suites: Object, Spatial, Goal, and Long. Together they contain 40 language-conditioned tasks. TurboVLA uses the modified “no-noops” RLDS datasets associated with OpenVLA, trains for 80,000 optimizer steps with 10,000 warm-up steps, and uses an effective batch size of 256 across four RTX 4090 GPUs. Evaluation consists of 50 rollouts per task, or 2,000 trials in total. The checkpoint release contains suite-specific evaluation checkpoints originating from this mixed-suite training setup. RoboTwin tests whether the same basic architecture scales beyond seven-degree-of-freedom, single-arm manipulation. Here one policy is trained across 50 bimanual tasks, using three views and a larger DINOv3 ViT-L backbone. Training is restricted to the benchmark’s clean demonstrations; the authors explicitly exclude randomized-scene data because of their compute budget. The resulting configuration has about 0.4 billion parameters and predicts a 50-step dual-arm action chunk. The real-world study uses an AgileX Piper arm and four tasks: grabbing a roller, moving a playing card away, pressing a stapler, and stacking three bowls. The authors initialize from the LIBERO-trained TurboVLA checkpoint and fine-tune on 65 teleoperated demonstrations per task—260 demonstrations overall. Each task is evaluated for 40 trials and compared with π0.5 using the same platform, data, and protocol. Efficiency measurements are made at batch size one on an RTX 4090 using official implementations and checkpoints for runnable baselines. Latency begins when the policy receives its multimodal input and ends when it produces either an action chunk or an equivalent set of autoregressive action tokens. Peak inference VRAM covers the complete online policy. Sensor exposure, camera transport, robot middleware, and actuator communication are not part of that definition. The Results: A New Point on the Pareto Frontier. The headline LIBERO comparison is unusually strong because TurboVLA does not merely trade some success for speed. It reports the highest average in the paper’s table while also being the smallest and fastest high-performing policy. The authors’ standardized measurements on an RTX 4090 are summarized below. | Policy | Parameters | Peak VRAM | Chunk latency | LIBERO average | |---|---:|---:|---:|---:| | TurboVLA | 0.2B | 0.9 GB | 31.2 ms | 97.7% | | VLA-Adapter | 1.5B | 4.3 GB | 87.3 ms | 97.3% | | π0.5 | 3.4B | 12.8 GB | 93.6 ms | 96.9% | | OpenVLA-OFT | 7.7B | 15.7 GB | 112.2 ms | 97.1% | | SmolVLA | 2.3B | 7.1 GB | 203.1 ms | 88.8% |. Relative to π0.5, TurboVLA is approximately three times faster, uses about one-fifteenth of the parameters, and consumes roughly one-fourteenth of the inference memory in this setup. Relative to OpenVLA-OFT, it is around 3.6 times faster while slightly improving the reported average success. The four suite scores are 99.2 on Spatial, 99.8 on Object, 97.4 on Goal, and 94.2 on Long. The RoboTwin result suggests that the architecture is not confined to saturated LIBERO tasks. The 0.4-billion-parameter model reaches 60.2 percent across 50 clean bimanual tasks, compared with 57.0 percent for π0.5 and 50.3 percent for StarVLA-α. Its 43.4-millisecond latency is less than half the 95.6 milliseconds reported for π0.5 under the authors’ measurement setup. On the four real-world tasks, TurboVLA obtains success rates of 92.5, 80, 90, and 87.5 percent. The paper reports that it consistently exceeds π0.5 under the matched protocol. This is not a broad real-world generalization study, but it is useful evidence that the direct fusion architecture survives camera noise, viewpoint changes, and physical execution errors rather than succeeding only inside LIBERO. The appropriate conclusion is not that a 216-million-parameter model has matched the full generality of a foundation VLA. The experiment measures adaptation and execution on specified task distributions. Within that regime, however, the results undermine the assumption that a billion-scale generative model is required inside every policy invocation. What the Ablations Reveal. The language ablation is especially informative. Removing language drops average LIBERO success from 97.7 to 70.8 percent. LIBERO-Goal collapses from 97.4 to 11.6 percent, showing that visual observations alone do not determine which valid behavior the robot should execute. A learned task-ID embedding recovers 95.4 percent, but natural-language instructions retain a 2.3-point advantage. Language is therefore doing more than selecting one of 40 arbitrary policy modes, although much of the benchmark can still be represented through closed-set task identity. BERT is not uniquely necessary. T5-small reaches 97.1 percent, only 0.6 points below the full model, while reducing the complete parameter count from 216.1 million to 141.9 million. SigLIP’s text encoder obtains 95.5 percent. One of the paper’s most actionable results may therefore be hidden in this table: a roughly 142-million-parameter TurboVLA variant already retains nearly all of the reported LIBERO performance. Directly concatenating independent vision and language features reaches 95.2 percent. One-way cross-attention increases that to 96.1 or 96.5, depending on direction. Bidirectional interaction reaches 97.7. The largest difference appears on Goal, where bidirectional fusion improves from 90.8 to 97.4 relative to simple concatenation. This supports the authors’ claim that scene-conditioned language and language-conditioned vision contribute complementary information. Depth and chunk horizon both have interior optima. Increasing interaction depth from two to six layers improves average success from 93.5 to 97.7 percent, but eight layers fall to 96.6. Likewise, an eight-step chunk scores 96.4, a 12-step chunk reaches 97.7, and a 15-step chunk drops to 95.6. More look-ahead is not automatically better: long chunks increase temporal coverage but make multimodal behavior cloning harder and can leave the policy committed to stale decisions. How TurboVLA Relates to Other Efficiency Strategies. OpenVLA-OFT attacks the action side of the problem. It replaces autoregressive action token generation with continuous parallel decoding, action chunking, and an L1 objective, producing a major throughput improvement over the original OpenVLA. TurboVLA adopts essentially the same high-level lesson about continuous chunks, but then asks why the 7-billion-scale OpenVLA backbone must remain upstream of that decoder. VLA-Adapter is conceptually closer. It studies how vision-language representations should be bridged into an action policy and introduces Bridge Attention around a relatively small 0.5-billion-parameter backbone, without robot-data pretraining. TurboVLA goes further in architectural separation: instead of retaining a compact generative VLM as the representational center, it uses independent DINOv3 and BERT encoders plus a narrow, explicitly bidirectional grounding module. SmolVLA combines a smaller VLM with an asynchronous inference stack that decouples action execution from expensive policy updates. Realtime-VLA FLASH retains a large diffusion VLA but uses a lightweight draft model, parallel verification, and phase-aware fallback. FLASH reports task-level average latency of 19.1 milliseconds, with speculative rounds as short as 7.8 milliseconds, but this is a different measurement from TurboVLA’s complete single-policy forward pass. These methods operate on complementary axes. TurboVLA simplifies the base multimodal pathway. OpenVLA-OFT improves the decoder. SmolVLA changes the execution scheduler. FLASH avoids unnecessary full-model calls. A practical system could combine all four ideas: a compact direct-fusion policy, parallel continuous chunks, asynchronous receding-horizon execution, and selective verification or fallback. Reading the “32 Hz” Claim Correctly. TurboVLA’s latency claim is clearly defined and appears to include the online BERT and DINOv3 forward passes. But a 31.2-millisecond action-chunk prediction should not be equated automatically with a 31.2-millisecond closed-loop control period. In the released LIBERO command, both the action chunk and the number of open-loop execution steps are set to 12. The policy observes the environment, predicts 12 actions, and executes all 12 before replanning. It can generate chunks at 32 hertz, but the demonstrated observation-to-replan frequency is governed by the environment’s action rate divided by the number of open-loop steps. The low-level controller may still stream setpoints at its native rate; it simply does so from a previously generated chunk. This does not negate the latency result. Fast chunk generation creates headroom for more frequent replanning, temporal ensembling, safety checks, or multiple candidate predictions. But a true real-time systems evaluation should separately report model latency, sensor-to-policy latency, replan frequency, actuator command frequency, jitter, and the age of the observation underlying each executed action. The public implementation also appears to be effectively single-frame. If it receives a tensor with a temporal dimension, it selects the latest observation rather than constructing an explicit visual history. That is efficient, but it leaves motion estimation, occlusion recovery, and contact-state inference mostly to the instantaneous visual representation and proprioception. Dynamic manipulation may require temporal tokens, recurrent state, optical flow, or short-horizon prediction. What the Paper Does Not Yet Establish. LIBERO is approaching saturation for well-tuned policies. TurboVLA’s 97.7 percent is valuable evidence that efficiency need not destroy in-distribution performance, but it says relatively little about unseen linguistic composition, novel objects, novel camera layouts, or open-world task planning. The released text-layout metadata is built around the 40 standard instructions, and the experiments do not present a dedicated language-generalization split. RoboTwin broadens the action and task spaces, but the study is limited to its clean setting. The authors do not train on the randomized-scene data, so robustness to the benchmark’s stronger domain variation remains unanswered. Similarly, the real-world evaluation covers four tasks with 65 demonstrations each on one platform. It validates physical execution, not general-purpose robotic transfer. The consumer-hardware framing also deserves precision. An RTX 4090 is a consumer product, but it is a high-end desktop GPU with a substantial power envelope. The 0.9-gigabyte memory footprint suggests that TurboVLA could fit on much smaller accelerators, but the paper does not demonstrate 31.2-millisecond latency on a laptop GPU, Jetson-class module, integrated accelerator, or CPU. Memory capacity and compute throughput are separate constraints. Moving from a 4090 to an embedded GPU may preserve fit while dramatically reducing rate. Measuring TensorRT compilation, INT8 or FP8 quantization, CPU preprocessing, and power consumption would be a natural next step. Finally, TurboVLA’s success should not be interpreted as evidence that semantics or pretrained foundation models are unnecessary. The model depends on powerful visual, language, and grounding pretraining. What it removes is the repeated use of a generative language-model core during execution. That is a much narrower—and more defensible—claim. How I Would Adapt TurboVLA. For a robotics team with a finite skill repertoire and stable natural-language commands, TurboVLA offers a strong default architecture. It is particularly attractive when policies must run locally, GPU memory is shared with perception or mapping systems, and a task planner already exists elsewhere in the stack. The first adaptation I would test is caching the frozen BERT outputs. The instruction usually remains constant for many seconds, while the released policy recomputes BERT on every forward pass. Caching only the base text tokens would preserve all six layers of scene-dependent bidirectional interaction. It should reduce latency without changing the policy’s learned multimodal computation. The second is to reconsider the text encoder. The T5-small ablation gives up only 0.6 LIBERO points while reducing the complete model to about 142 million parameters. For a deployment constrained more by memory than by the last fraction of benchmark success, that is a compelling trade. Distilling the grounding module and visual backbone could move the model further toward embedded hardware. The third is to retune action horizon around the actual robot and scheduler. Twelve steps worked best on LIBERO, while RoboTwin uses 50. Neither value should be treated as universal. Contact-rich assembly, dynamic interception, and deformable manipulation generally demand shorter effective commitments than free-space reaching. A receding-horizon implementation could predict a long chunk but execute only its early portion before incorporating a new observation. I would also combine TurboVLA’s base architecture with asynchronous execution. While one chunk is being consumed, the system can process a newer observation and prepare a replacement trajectory. Temporal aggregation or smooth chunk handoff would then be required to prevent discontinuities. This would convert the model’s fast full forward pass into genuinely low action age, rather than merely high offline throughput. For dynamic scenes, explicit short-term history is likely the most important architectural extension. Multi-frame DINO tokens could be fused with temporal position embeddings, or a compact world-state module could maintain object motion and contact estimates. The guiding principle should remain TurboVLA’s: add specialized capacity where the execution problem requires it rather than routing everything through a general-purpose LLM. From an engineering standpoint, the release is unusually usable. Training and evaluation code, LIBERO and RoboTwin checkpoints, and normalization metadata are public. The source code is Apache-2.0, although the released checkpoints inherit DINOv3 licensing terms, and the model is a custom PyTorch implementation rather than something directly loadable through. The Takeaway. TurboVLA does not show that language models have no role in robotics. It shows that a language model may not belong inside every 30-millisecond execution cycle. For concrete manipulation instructions, a strong visual encoder, a lightweight text encoder, several layers of bidirectional grounding, and a parallel action-chunk head can be enough. On the evaluated distributions, that design reaches the performance ceiling of much larger VLAs while using less than one gigabyte of inference memory. The lasting contribution may therefore be architectural rather than numerical. TurboVLA gives the field a clean separation to test: let expensive models reason, plan, and clarify when necessary; let compact, task-conditioned policies handle the continuous work of seeing and acting. If that separation continues to hold beyond benchmark manipulation, the future VLA stack may look less like one monolithic foundation model and more like a hierarchy of models operating at the timescales where their capabilities are actually needed.