Latest / Embodied AI 101 / One Policy, Many Bodies: Qwen-VLA's Bid to Unify Embodied AI

Transcript

One Policy, Many Bodies: Qwen-VLA's Bid to Unify Embodied AI · Embodied AI 101

• One Policy, Many Bodies: Qwen-VLA’s Bid to Unify Embodied AI. Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments • Qiuyue Wang, Mingsheng Li, Jian Guan, et al. (40 authors) • Qwen Team • arXiv preprint, v2 • 2026. Qwen-VLA starts from a familiar diagnosis: embodied AI has accumulated increasingly capable specialists, but not yet a coherent generalist. We train one model for tabletop manipulation, another for bimanual control, another for continuous navigation, and perhaps yet another to predict human hand motion. Each model has its own action representation, control rate, observation stack, training code, and assumptions about the body it controls. The paper asks whether those distinctions are as fundamental as our current software stacks make them appear. Its answer is Qwen-VLA, a roughly five-billion-class system combining a Qwen3.5-4B multimodal backbone with a 1.15-billion-parameter flow-matching action decoder. The model is trained across manipulation, navigation, human egocentric motion, synthetic trajectories, and conventional vision-language data, while textual prompts tell it which body and control convention it is currently operating. The report was first submitted on May 28, 2026, and revised as version two on June 1, 2026. The result is not a universal action language in which every robot literally means the same thing by every output dimension. It is something more pragmatic: one conditional model, one continuous trajectory-generation interface, and one set of large shared weights that can multiplex between heterogeneous action spaces. That distinction is essential for understanding both what Qwen-VLA accomplishes and what remains unresolved. The Interface Fragmentation Problem. Robot learning datasets disagree about almost everything below the level of task intent. A WidowX policy may predict delta end-effector poses and a gripper command. A humanoid may require absolute positions for dozens of joints. A dexterous platform may add finger configurations. A navigation agent instead emits planar displacement and heading changes, while human egocentric datasets provide wrist transforms and hand poses rather than executable robot commands. These sources also run at different frequencies, expose different camera layouts, and use different prediction horizons. Even seemingly compatible Cartesian actions may be expressed in different reference frames or normalized according to different statistics. Historically, researchers have addressed this heterogeneity either by converting datasets into a preferred canonical representation or by attaching embodiment-specific input and output modules to a shared trunk. Qwen-VLA takes a third route. It observes that, at a sufficiently abstract computational level, all these tasks have the same form: condition on visual evidence, a language-specified objective, and information about the acting body; then produce a sequence describing what should happen next. For manipulation that sequence contains motor commands. For navigation it contains waypoints. For human data it contains future wrist and hand motion. For trajectory-centric domains it can represent future spatial paths. This framing is attractive because visual grounding and spatial reasoning should, in principle, transfer between domains. The ability to identify “the red bowl behind the plate” is useful whether the output is an arm trajectory or a navigation waypoint. Long-horizon navigation may teach temporal instruction alignment that benefits manipulation, while robot demonstrations can ground otherwise abstract language into physical motion. The central hypothesis is therefore not merely that heterogeneous data can coexist in one batch. It is that shared perception, language, and trajectory structure can create positive transfer. Unifying the Tensor Interface, Not the Laws of Physics. The model represents every continuous target as a fixed-size matrix: a prediction horizon by a maximum number of action channels. An embodiment that uses fewer channels occupies the leading portion of that matrix, while the rest is padded with zeros. A binary mask prevents padded dimensions and unused time steps from contributing to the loss. This sounds simple, but it makes a consequential architectural choice possible. Qwen-VLA does not need a separate decoder for the WidowX, ALOHA, a humanoid, human hands, and navigation. A single DiT action expert emits the maximum-size tensor, and the current task reads only its relevant prefix. Crucially, the channels do not acquire one universal physical meaning. The paper explicitly preserves each dataset’s native control convention. A leading channel might describe Cartesian translation in one dataset and a joint angle in another. Dataset-specific quantile normalization maps each active dimension into approximately the same numerical range, using the first and ninety-ninth percentiles to reduce the effect of outliers. The model must infer the intended interpretation from its conditioning context. The right analogy is typed polymorphism rather than an embodied Esperanto. The shared decoder accepts the same tensor shape, but a textual schema tells it what the numbers mean. That schema is the embodiment-aware prompt. Each example identifies the robot platform, whether it has one or two arms, optional components such as a waist or mobile base, its control frequency, its output horizon, and the task instruction. Navigation examples receive an analogous declaration describing the waypoint convention. The prompt is processed by the vision-language backbone, and its hidden states condition the action decoder. Switching platforms therefore does not require selecting another policy head; it changes the model’s linguistic context. Qwen-VLA applies the same idea to cameras. Images are wrapped in tokens identifying their provenance, such as an egocentric camera or a left or right wrist camera. The transformer can then learn which view is informative without requiring hard-coded camera-specific encoders. This is another example of moving system metadata into a tokenized interface rather than baking it into the architecture. There is an important deployment caveat. A prompt can specify that a body is different, but it cannot by itself provide the normalization statistics, low-level controller, camera calibration, safety constraints, or action semantics needed by a completely new robot. In practice, embodiment prompting is best understood as a routing and conditioning mechanism over control conventions represented during training—not yet proof that arbitrary hardware can be added by writing a sentence. Architecture: A Multimodal Brain and a Generative Motor System. The cognitive component is Qwen3.5-4B, a natively multimodal model in which visual tokens are inserted directly into the language sequence. Its architecture uses gated linear attention through most layers, interspersed with grouped-query softmax attention. The intended combination is efficient processing of long image, video, and language contexts while retaining periodic layers capable of global attention. On top of this backbone, the authors attach a single-stream diffusion-transformer-style action expert. It contains 16 DiT blocks, accounting for about 1.13 billion of the decoder’s 1.15 billion parameters. Projection layers map raw actions into the DiT representation, while another projection adapts Qwen hidden states to the action decoder’s channel width. The visual-language states and a noisy candidate action trajectory are concatenated and processed together through self-attention. The action head uses flow matching rather than autoregressive token generation. During training, a clean trajectory is blended with Gaussian noise at a randomly selected corruption level. The network learns a velocity field: given the corrupted trajectory and its multimodal context, it predicts the direction that moves the sample toward a plausible clean action sequence. During inference, the process begins with noise and follows the predicted field through several Euler integration steps. For robotics, this offers two useful properties. First, outputs remain continuous, avoiding the quantization introduced when joint or Cartesian commands are converted into language-like tokens. Second, a generative trajectory model can represent multiple valid modes. A cup might admit several grasps and approach paths; the policy need not regress all of them into an unsafe average. The separation between backbone and action expert also reflects a useful division of labor. The backbone carries object recognition, reference resolution, spatial language, and world knowledge. The DiT specializes in high-frequency, temporally coherent continuous signals. Unlike a very small policy head, however, Qwen-VLA’s motor system is itself substantial: its 1.15 billion parameters are larger than many complete imitation-learning policies. The paper describes the few-step flow process as suitable for real-time control, but it does not provide a detailed end-to-end latency table in the main report. Actual control rate will depend on image resolution, camera count, context length, integration steps, batching, and inference hardware. The Data Mixture Is the Real Foundation Model. Qwen-VLA’s architecture is relatively legible. Its data mixture is where much of the claimed generality comes from. Robot manipulation trajectories make up 74.2% of pretraining. Human egocentric trajectories account for 6%, navigation trajectories for 7.5%, and the team’s synthetic simulation trajectories for 3.7%. The remaining 8.5% consists of general vision-language data, two-dimensional spatial grounding, autonomous-driving VQA, and fine-grained embodied action captions. The robot component combines public corpora such as DROID, BridgeData V2, RH20T, RT-1, RDT-1B, AgiBot World, RoboMIND, and several other multi-platform datasets. The authors report more than 10,000 hours of public interaction data, supplemented by more than 1,000 hours of internally collected real-robot trajectories. Representative bodies include WidowX and Google single-arm robots, single- and dual-arm Frankas, Mobile ALOHA, Fourier GR-1, Galaxea R1, AIRBOT MMK2, TienKung, and AgiBot platforms using grippers or dexterous hands. Human data is converted into the same broad trajectory interface. The corpus draws from processed Ego4D and EPIC-KITCHENS videos, EgoDex, EgoVerse, and Xperience. For each hand, wrist motion is represented by relative translation and axis-angle rotation. Hand articulation is compressed from a 45-dimensional pose into ten principal components, or eigengrasps. Two wrists plus two sets of eigengrasp coefficients yield a 32-dimensional target per time step. This does not solve human-to-robot retargeting in a strict kinematic sense. Instead, it allows the shared decoder and backbone to learn statistical regularities about reaching, grasping, bimanual coordination, and object interaction from a broader visual world than robot teleoperation can economically cover. Synthetic data comes in two forms. A vision-conditioned pipeline built with IsaacLab, cuRobo, and an early version of RoboInF generates executable manipulation programs, verifies success, and rolls them out under variations in lighting, camera placement, textures, object pose, initial robot state, and controller dynamics. The reported dataset contains 359,848 successful full and subtask trajectories across short- and long-horizon instructions. A second, rendering-free pipeline creates text-action pairs for six manipulation families across six single-arm robots. It samples reachable goals and uses accelerated motion planning to produce collision-free joint trajectories. This yields roughly 7.2 million trajectories and more than 14,000 hours of simulated motion. Because images are unnecessary, it provides cheap coverage of verbs such as moving, grouping, rotating, swapping, pushing, and pulling. The navigation data adds instruction following, object search, and moving-target tracking. These episodes use video sampled at two frames per second and represent future motion as planar displacement plus heading change. Auxiliary driving and spatial VQA data is included not primarily to teach direct vehicle control, but to reinforce temporal interpretation, multi-camera geometry, localization, and planning-aware visual reasoning. Finally, the team creates about 48,000 fine-grained action-caption pairs. A Qwen model first summarizes a manipulation video and then revisits densely sampled frames to describe contact points, object identity, trajectory, orientation, gripper state, and body motion. Human annotators review the result. Although this is only 0.2% of the mixture, it addresses a real ambiguity in robot datasets: a label such as “rotate the bowl” omits the direction, number of rotations, grasp side, and final location that distinguish one trajectory from another. Four Stages from Language to Closed-Loop Control. The training recipe addresses a common asymmetry in modern VLAs. The visual-language backbone arrives with billions of tokens’ worth of useful structure, while the continuous action decoder begins randomly initialized. Training both together immediately forces the new decoder to learn action statistics, flow dynamics, embodiment conditioning, temporal structure, and visual grounding at once. Its noisy gradients can also damage the pretrained backbone. Qwen-VLA therefore begins with text-to-action pretraining, or T2A. The backbone is frozen, images are removed, and only the DiT learns to reconstruct trajectories from task language and embodiment prompts. At first this sounds ill-posed. “Pick up the red cup” cannot determine a precise trajectory without seeing the cup. But T2A is not intended to produce an executable scene-conditioned path. It learns a distribution over the temporal shape of relevant behavior: reaching before grasping, transporting after contact, or coordinating two arms during folding. Language selects a region of action space; later stages determine which member of that region fits the observed scene. The ablations make this stage unusually informative. Training on complete trajectories works better than training on isolated chunks, because a chunk conditioned only on language is ambiguous about where it occurs within the task. A mixture of about 80% real and 20% synthetic T2A data produces 71.1% downstream success on Simpler-WidowX, versus 60.9% without T2A. Synthetic data broadens language-action coverage, while real trajectories anchor the decoder in less idealized motion. Images are actively counterproductive during this stage: they raise cost and allow the model to bypass the intended language-action prior. The best configuration trains T2A for only about 2,000 steps. Extending it to 40,000 steps degrades downstream performance, suggesting that the useful structural prior is learned quickly and then begins to overfit. The authors also find that emphasizing intermediate noise levels during T2A works better, whereas the later visually conditioned stages benefit from the more conventional Beta-distributed flow timesteps. In continued pretraining, or CPT, both modules are unfrozen and trained on the complete heterogeneous mixture. This is where the language-indexed action prior becomes grounded in actual images, multi-camera observations, and mixed real-simulation visual distributions. CPT produces Qwen-VLA-Base. Supervised fine-tuning then branches in two directions. A multi-task track jointly trains vision-language tasks, spatial grounding, navigation, and simulated manipulation, balancing tasks and embodiments. A separate physical-robot track starts from the same base checkpoint and fine-tunes on internal teleoperation data. Manipulation examples use 16-step chunks, while navigation uses eight-waypoint chunks. The vision-language loss receives a weight of 0.1, compared with 1.0 for action prediction, preserving perception without allowing language generation to dominate policy learning. The final Qwen-VLA-Instruct checkpoint adds reinforcement learning using PPO and sparse simulator success rewards. Applying PPO to a flow policy is technically nontrivial because the deterministic denoising process does not directly expose action log-probabilities. The authors turn each denoising transition into a Gaussian stochastic step, store intermediate states during rollout, and recompute a selected transition under the updated policy. This supplies an approximate likelihood ratio with one additional DiT evaluation. Rewards and advantages are assigned per 16-step action chunk. A lightweight value head is attached to stop-gradient backbone features, while 128 parallel environments generate 8,192 transition chunks per training iteration. All RL rollouts come from SimplerEnv, making transfer outside that simulator a useful test of whether closed-loop optimization improves general execution rather than merely memorizing one benchmark. What the Experiments Actually Establish. A strong multi-benchmark generalist. In simulation, one Qwen-VLA-Instruct checkpoint scores 97.9% on LIBERO, 56.7% on RoboCasa-GR1, 73.7% on Simpler-WidowX, and 86.1% and 87.2% on the easy and hard RoboTwin splits. The comparison models are independently specialized for their respective benchmarks, whereas Qwen-VLA is jointly trained and then evaluated across platforms without selecting a per-benchmark policy. The generalist does not win every column. ABot-M0 reports 98.6% on LIBERO and 58.3% on RoboCasa-GR1, slightly above Qwen-VLA. But Qwen-VLA leads the reported Simpler-WidowX and RoboTwin results and remains competitive everywhere else. That is credible evidence that multi-embodiment co-training need not impose the large specialist penalty often associated with multitask policies. It is not, however, zero-shot task evaluation. The Instruct model receives multi-task SFT demonstrations spanning the evaluated classes of simulated environments. The appropriate claim is that a single jointly trained policy can replace several benchmark-specific policies while preserving specialist-level performance. Pretraining matters on physical ALOHA. For real-world evaluation, the authors fine-tune a unified checkpoint across six categories on a bimanual ALOHA system. The version initialized from Qwen-VLA-Base reaches 83.6% average in-domain success, compared with 48.5% for the same architecture trained without embodied pretraining. Reported averages for separately fine-tuned π0.5 and GR00T N1.6 baselines are 71.6% and 28.6%. The pretrained model is particularly strong on pick-and-place, table cleaning, bowl stacking, and fine-grained manipulation. It is not uniformly superior: π0.5 achieves 80.8% on towel folding, versus Qwen-VLA’s 65.4%. Deformable manipulation therefore remains an area where broad pretraining does not erase task-specific difficulty. The OOD comparison is more striking. Under unseen colors, object instances, positions, backgrounds, and instruction variations, pretrained Qwen-VLA averages 76.9%. The non-pretrained version reaches 36.2%, π0.5 reaches 41.5%, and GR00T N1.6 reaches 25.4%. This strongly supports the value of the pretraining mixture, although these tests vary conditions around an ALOHA task distribution rather than transferring without adaptation to a previously unseen morphology. Manipulation and navigation can coexist. On continuous vision-language navigation, Qwen-VLA-Instruct obtains a 57.5% success rate on R2R Val-Unseen and 59.6% on RxR Val-Unseen. It exceeds the reported open navigation specialists on success rate and reaches an oracle success rate of 69% on R2R. The finer metrics provide a more nuanced picture. StreamVLN has slightly higher R2R SPL—51.9 versus 51.2—and a substantially higher RxR nDTW score, 61.9 versus 57.1. Qwen-VLA is therefore good at ultimately reaching the goal, but its path can be less faithful to the reference route. The sliding-window trajectory interface appears sufficient for competent navigation, but it does not yet dominate specialist models on every measure of efficiency and instruction alignment. OOD behavior is promising, but far from solved. On SimplerEnv-OOD, all models are fine-tuned only on basic Bridge pick-and-place data and then tested on new spatial instructions. Qwen-VLA-Instruct averages 32%, compared with 12.6% for π0.5. It generalizes to relations such as moving an object farther away or placing it on the right half of a region. Yet the absolute number matters: roughly two-thirds of trials still fail. On “put the carrot in front of the plate,” Qwen-VLA-Instruct scores only 4.2%, below π0.5’s 13%. The model has broader spatial transfer, not a solved compositional geometry system. DOMINO provides the most provocative result. Without dynamic-manipulation training and using only the current observation rather than explicit motion history, Qwen-VLA-Instruct achieves 26.6% success and a 39.5 manipulation score. This exceeds zero-shot LingBot-VA and, in the reported table, also exceeds PUMA after DOMINO-specific dynamic training. The authors argue that coherent action chunks help the policy commit within short interception windows, while navigation and trajectory pretraining supply transferable spatial-to-kinematic structure. Again, the correct interpretation combines relative and absolute performance. Twenty-six percent is impressive for zero-shot moving-object manipulation, but most episodes still fail. DOMINO shows that Qwen-VLA has acquired a useful dynamic prior despite static observations; it does not show robust general dynamic control. Most of the Gain Comes Before Reinforcement Learning. The cumulative ablation separates CPT, SFT, and RL. SFT provides the major jump: Simpler rises from 64.3% to 70.8%, RoboCasa from 40.4% to 56%, and RoboTwin-Easy from 64.3% to 86.3%. RL then raises Simpler by another 2.9 points, but changes most held-out benchmarks by less than one point. That is still a positive result. RL in one simulator does not catastrophically erase other capabilities and slightly improves zero-shot DOMINO success from 25.7% to 26.6%. But Qwen-VLA is principally a large-scale pretraining and multitask imitation-learning story. PPO is a final refinement, not the primary source of generality. Two other ablations reinforce the paper’s core thesis. Retaining vision-language examples during action training improves RoboCasa by 4.9 points while introducing little interference on simpler benchmarks. Meanwhile, private embodiment-specific projection MLPs, concatenated action segments, and shared zero-padded projections all perform similarly. Once conditioning and masking are in place, a lightweight shared projection is sufficient. Explicit proprioception also produces only small gains on RoboTwin: at most 0.7 points on Easy and 1.3 on Hard. The authors therefore omit state from the default model, reasoning that multi-view images already expose the robot configuration. That conclusion should be treated as benchmark-specific. Under occlusion, latency, contact, or camera failure, joint and force state may become substantially more important. Qwen-VLA in the Generalist-Policy Lineage. Qwen-VLA is best viewed as a synthesis and expansion of several preceding directions. Octo established an open, adaptable generalist policy trained on 800,000 Open X-Embodiment trajectories. Its emphasis is a versatile manipulation initialization that can be fine-tuned to new sensors and action spaces. OpenVLA more directly transfers Internet-scale vision-language representations into robot control, using a seven-billion-parameter model trained on 970,000 demonstrations and providing public checkpoints and training infrastructure. Both helped normalize the idea that a broad robot policy should be pretrained once and adapted rather than rebuilt for every task. The closest architectural ancestor is Physical Intelligence’s π0, which also places a flow-matching action model on a pretrained VLM and trains across single-arm, dual-arm, and mobile-manipulator data. π0.5 extends that approach through heterogeneous co-training with web data, object detections, semantic subtask prediction, and low-level actions, demonstrating long-horizon manipulation in unseen homes. Qwen-VLA’s distinctive contribution is therefore not the invention of VLM-conditioned flow control. It is the breadth and explicitness of the unification: manipulation, navigation waypoints, human hand trajectories, trajectory supervision, and textual responses share one backbone and one continuous action expert, with embodiment metadata functioning as the control schema. The T2A stage is also a notable training contribution, separating acquisition of a language-indexed motor prior from expensive visual grounding. What Remains Unproven. The first open question is genuine zero-shot embodiment transfer. Qwen-VLA demonstrates that known embodiments and action conventions can share weights without substantial interference. The report does not isolate a rigorous test in which a wholly new morphology, kinematic structure, and controller are introduced through text alone. Because outputs require dataset-specific normalization and interpretation, some platform integration remains outside the learned policy. Second, the paper’s trajectory-centric scope is broader in training than in evaluation. Autonomous-driving and forecasting data appear primarily as auxiliary reasoning and trajectory supervision, while the main closed-loop results cover manipulation and navigation. A direct driving or general motion-forecasting benchmark would make the action-and-trajectory claim more complete. Third, current-frame success on DOMINO should motivate explicit temporal modeling rather than suggest it is unnecessary. A history-aware version could estimate object velocity, distinguish temporary occlusion from disappearance, and adapt chunk execution online. Likewise, the navigation results point toward persistent memory, while contact-rich manipulation calls for force and tactile observations. The authors acknowledge related limitations: embodied data remains small relative to ordinary vision-language corpora; joint training creates objective-balancing trade-offs; and evaluation is still dominated by short-horizon benchmarks. They identify episodic memory, world modeling, failure recovery, richer physical feedback, and larger-scale RL as future directions. Reproducibility is another substantial constraint. As of September 8, 2026, the official GitHub repository contains the README and media assets but no model implementation, checkpoints, releases, or packages. More than 1,000 hours of the robot corpus is proprietary, as is much of the generated-data infrastructure. The published report is detailed enough to communicate the design, but not currently sufficient for an independent group to reproduce Qwen-VLA at full scale. This contrasts with OpenVLA and Octo, whose openness was part of their original contribution. What Roboticists Should Take from It. Even without released weights, several ideas are immediately reusable. The first is to treat embodiment metadata as a formal policy input. Robot identity, camera provenance, action convention, frequency, and horizon should be represented explicitly rather than left for the network to infer from accidental dataset correlations. The second is to separate motor-prior learning from visual grounding. Large text-action or state-action corpora can initialize a generative decoder cheaply, provided the objective is understood as learning trajectory structure rather than executable open-loop plans. Qwen-VLA’s finding that full trajectories beat isolated chunks during this stage is especially relevant: long sequences teach the motor grammar, while chunks are better reserved for visually conditioned closed-loop execution. The third is to retain perceptual rehearsal during policy training. Spatial VQA, grounding, and dense action descriptions are not decorative auxiliary tasks. On cluttered and compositionally demanding benchmarks, they appear to preserve precisely the object discrimination and instruction parsing that action-only fine-tuning tends to forget. Finally, researchers adopting the shared zero-padded action interface should test the boundary conditions that Qwen-VLA leaves open: held-out robot bodies, changed prompt wording, new normalization statistics, camera occlusion, delayed observations, and contact-rich tasks where vision cannot reconstruct the true state. A unified tensor is a scalable engineering abstraction, but it should not be confused with a universal dynamics model. Final Take. Qwen-VLA’s strongest contribution is a concrete argument that manipulation and navigation need not remain separate model families. A single multimodal backbone and a single continuous trajectory decoder can absorb radically heterogeneous supervision, provided the model is told what body it controls and is trained through a carefully staged curriculum. The work does not eliminate embodiment-specific engineering, solve long-horizon autonomy, or deliver reliable zero-shot control of arbitrary hardware. Its OOD results remain uneven, and the absence of released weights limits immediate validation. But the direction is compelling. Qwen-VLA reframes embodiment not as a reason to build another isolated policy, but as context supplied to a shared one. If that abstraction continues to scale—especially with memory, dynamics prediction, tactile feedback, and truly held-out robots—the boundary between manipulation model, navigation model, and motion model may become an implementation detail rather than a defining division of embodied AI.