Latest / Embodied AI 101 / InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action

Transcript

InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action · Embodied AI 101

• InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action. Roboticists have long dreamed of a unified policy that can both understand rich language instructions and visual scenes, and also predict future dynamics in order to generate continuous actions. The latest InternRobotics model, InternVLA-A1.5, makes dramatic strides toward this goal by integrating a pretrained vision-language model (VLM) with world-model-style foresight. Developed by Shanghai AI Lab’s Physical Intelligence Team, InternVLA-A1.5 extends their earlier InternVLA-A1 work, but with a novel twist: instead of trying to predict raw images from scratch, it uses a small set of learnable “foresight tokens” to query the future from a frozen, large-scale video generation model. In practice this means the policy retains the semantic language grounding of the VLM while absorbing spatiotemporal priors from a state-of-the-art video model, yielding remarkable generalization and efficiency in real-time control. At a high level, InternVLA-A1.5 is a Mixture-of-Transformers policy built on a 2-billion-parameter Qwen3.5 VLM backbone. The backbone processes the multi-view camera images and language instruction together with a tokenized robot state, preserving its pretrained semantic and instructional knowledge. Attached to this backbone is a lightweight unified expert that generates the robot’s actions. Crucially, InternVLA-A1.5 also injects a latent foresight mechanism: during training a few special “foresight” tokens attend jointly to the visual and language context and are trained to predict future latent dynamics. This is done by supervising them with a frozen “WAN2.2-5B” video diffusion model. In other words, InternVLA-A1.5 distills the expansive spatiotemporal priors of an existing video generator into its own small foresight module. At inference those foresight tokens and the video branch are dropped, so the resulting policy remains fast and lightweight. This architecture carefully balances semantic understanding and action planning. By design, the original VLM continues to train on vision-language tasks (like visual QA and subtask prediction) so its instruction-following and grounding stay sharp. Meanwhile, the unified expert learns to turn the backbone’s representation and the learned foresight into continuous control signals (via a flow-matching generative head). InternVLA-A1.5’s authors stress that many earlier vision-language-action (VLA) models suffered from “eroding” their VLM’s semantics when forced to predict pixels or handle heterogeneous objectives. By contrast, InternVLA-A1.5 keeps the full semantics of the VLM and explicitly decouples the purposes: the backbone handles understanding, the foresight tokens tap into a video model for future cues, and the action expert handles control. As the paper notes, existing designs “tend to erode the semantics of the pretrained backbone” and learn future prediction “from scratch in pixel space, leaving the dynamics priors of pretrained video generators unexploited”. InternVLA-A1.5 sidesteps this by inheriting world-model dynamics via the frozen video teacher, rather than relearning them pixel-by-pixel. Architecture: Vision-Language Backbone + Unified Expert. Under the hood, InternVLA-A1.5 resembles a standard multimodal transformer with some custom heads. The base of the network is a native Qwen3.5-2B vision-language model. Inputs are tokenized as follows: multi-view camera images and the language instruction are tokenized like any VLM lexems, and importantly the current robot state (joint angles and velocities) is discretized into tokens and appended. A special “control mode” token is included to tell the model which action domain (e.g. position vs velocity control) is active. All these tokens flow through the shared transformer layers. At various depths, a small “unified action expert” branch reads the transformer outputs. This expert is very lightweight compared to the 2B backbone – in effect a few layers that take shared multimodal context and output an action distribution. The model description highlights three core components of InternVLA-A1.5:. The Core: The Qwen3.5-2B vision-language backbone processes images, text, and robot state via full self-attention. It retains its original gating mechanisms (modality-specific “DeltaNet” layers) so that it can still solve vision-language tasks effectively. The Foresight: A fixed set of foresight tokens are inserted and attend to the same context as the action expert. These tokens effectively query the network for information about future states. During training they are supervised by a frozen WAN2.2-5B video diffusion model – meaning the network learns to produce representations that match what the advanced video generator would output for future frames. The README explains: “Uses learnable foresight tokens to query task-relevant future dynamics from the shared multimodal context, supervised by a frozen WAN2.2-5B video generation model during training”. The Output: At inference time, the policy discards the entire video branch. Instead of producing images, the unified expert directly predicts continuous action chunks via flow-matching. Each forward pass outputs a short sequence of future control signals (chunk size 50 steps) that the robot can execute. This design choice is pragmatic: the README notes the video branch is dropped for efficiency, and the output is continuous actions generated with a diffusion-like flow model so that the policy can run in real time. Stated more plainly: during training the network is actually multi-headed – one head (the frozen video model) generates latent future frames, another head (the unified expert) learns to produce the robot actions. But at test time we throw away the video head and just use the unified action head. This keeps inference fast. The authors emphasize that the output head “predicts continuous action chunks through flow matching, keeping deployment latency practical”. This Mixture-of-Transformers approach builds on the team’s previous internVLA-A1 work. In A1, they also used an understanding expert (scene/text encoder) plus a generation expert (future image predictor) and an action expert. But A1 trained the generation expert directly to predict images. InternVLA-A1.5 replaces that step with the latent-token approach. In A1.5, forward modeling of vision is instead passed to the big pre-trained Wan video model. The small foresight tokens essentially ask the Wan model for the future, under supervision. In practice, future frames never need to be generated pixel-wise by the policy; only the small latent vectors for those frames are distilled. This is why the authors say they “recast future prediction as a latent-querying problem” – the small foresight token sequence becomes a compact code for the future, learned under Wan’s guidance. Multi-Stage Training and Data. Training InternVLA-A1.5 is a deliberate multi-stage process. The goal is to transfer the semantics of the VLM while adding dynamics knowledge gradually. In the early stage, the model is trained on vision-language tasks: it continues to answer questions about the scene and predict subtask progress tokens so that it doesn’t forget the language understanding. The name InternVLA hints at this: the backbone is pretrained on vision-language QA and task grounding data (InternData-A1 and an internal “InternVLA-M1” multimodal set). In the next stage, the unified expert and flow-matching head are turned on so the model learns to imitate robot trajectories. Finally, the foresight tokens and Wan2.2 supervision are introduced. At each stage, other objectives are partially frozen or weighed down to avoid catastrophic forgetting, as the paper notes: “existing unified models. suffer interference among heterogeneous objectives”, so the authors carefully isolate training phases to avoid that. The training data is truly massive and diverse. InternVLA-A1.5 is pretrained on about 1.2 million robot episodes (over 861 million frames) from six sources, plus 3 million multimodal QA samples for grounding. Key data sources include:. InternData-A1 (simulated): ~588K episodes, 396M frames (weight 0.20). This is a large synthetic manipulation dataset. AgiBotWorld (real): ~113K epis, 206M frames (0.25 weight). Real robot trials collected by Shanghai AI Lab. UMI (real): 377K epis, 201M frames (0.10). A public real-world kitchen manipulation dataset. DROID (real): 96K epis, 27.6M frames (0.15). Another robotic dataset. Galaxea (real): 19K epis, 25.0M frames (0.20). RoboMind 1.0 (real): 8.6K epis, 5.4M frames (0.10). The table on the project page summarizes this mixture, showing that they essentially pool everything from large sim to diverse real-world streams. Alongside this, they co-train on an InternVLA-M1 multimodal corpus: ~3M examples of QA-style tasks (captioning, OCR, referring, spatial QA, trajectory QA, etc.) drawn mostly from vision-language sources. This second stream ensures the VLM backbone keeps its semantic anchors. The authors explicitly mention pretraining on this scale: “Pretrained on 1.2M robot episodes and 3M multimodal samples”. Ablations on the earlier InternVLA-A1 confirm that without such large-scale, heterogeneous pretraining, performance “degrades dramatically”. Likewise, InternVLA-A1.5 ablations (in the paper) show that each component (pretraining, multi-stage pipeline, foresight tokens, subtask prediction) significantly contributes to success. Training also injects coordinate prediction and world awareness. InternVLA-A1.5 uses “subtask tokens” explicitly: at each moment the VLM is asked not only to model the current step, but to predict what subtask the robot is on. These discrete subtask tokens (for example, describing “pick up orange tube” or “place blue tube in right box”) help the model track progress. The Alphaxiv site notes that explicitly predicting subtasks in language form grants much better long-horizon control. Indeed, during inference the policy’s autoregressive token generation can yield either an answer to a visual question, a subtask description, or an action command – though only the action tokens are actually executed. This hybrid token set trains the network to maintain scene grounding at all times. Latent Foresight via Frozen Video Generator. The centerpiece of InternVLA-A1.5 is how it gains foresight of the future. Instead of learning a vision module to predict frames, it leverages an existing video diffusion model, Wan2.2-5B, as a frozen teacher. During training, the small foresight token vector is fed into Wan2.2’s architecture (as if prompting it for a continuation of the scene); the policy then compares its own internal predicted latent state to what Wan2.2 outputs and backpropagates the error through only the new tokens and upstream layers. In effect, the network is learning to emulate Wan2.2’s world-model using a few extra degrees of freedom (the tokens), rather than rebuilding that huge model from scratch. As the authors put it: “Future prediction is recast as a latent-querying problem, where a small set of learnable foresight tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained video generation model”. In simpler terms, the policy never sees raw future images – it only learns the latent code that a giant video model would produce for those images. This distillation approach gives two big advantages: it inherits the spatiotemporal priors of Wan2.2, and it avoids the high compute of pixel-generation. The paper emphasizes “the policy inherits world-model dynamics priors without ever learning pixel-level generation”. Once trained, the foresight tokens contain a condensed summary of what the next few frames should look like, as inferred by Wan2.2. The action expert can then use that latent summary along with the current scene to decide what to do next. During inference, after training, the foresight mechanism is kept in the model (the tokens themselves still work to refine representation) but the Wan video model is not run (only the learned weights matter). The policy thus remains fast: only a feedforward pass of the small model is needed, and there’s no need to generate videos. The README notes succinctly: “The video branch is discarded at inference and predicts continuous action chunks through flow matching”. In practice, this means that InternVLA-A1.5’s policy operates just like a VLM-powered transformer at runtime – it predicts one action vector (or a short chunk of them) at a time – but it does so having been guided by a video model during training. Results: Strong Performance Across Benchmarks. InternVLA-A1.5 delivers state-of-the-art (and often state-of-the-art-by-large-margins) performance on a wide range of robotic benchmarks. According to the authors’ report, the base A1.5 model (without fine-tuning on any particular task) already outperforms all previous methods on all of the six major simulation benchmarks they tried: RoboTwin 2.0 (bimanual tasks), LIBERO, LIBERO-Plus, SimplerEnv, DOMINO, and EBench. For example it scores 98.9% average success on LIBERO, and 84.8% on LIBERO-Plus (which tests robustness to visual/layout shifts). On RoboTwin 2.0 it achieves 93.2%, and even on the challenging DOMINO suite (long-horizon tasks) it gets roughly 27.7% success rate (compared to much lower baselines). These numbers top any open model to the authors’ knowledge. Below is a summary table from the project page showing some key results:. Benchmark | Metric | InternVLA-A1.5---|---|--- SimplerEnv (WidowX robot) | Avg. Success Rate | 80.8% RoboTwin 2.0 | Avg. Success Rate | 93.2% DOMINO | SR / MS | 27.7% / 39.8% LIBERO | Avg. Success Rate | 98.9% LIBERO-Plus | Avg. Success Rate | 84.8% EBench | SR / Score | 35.2% / 49.5%. (SR = success rate, MS = median success, other metrics as published.). These results are all achieved by the base checkpoint before any task-specific finetuning. The team also provides specialized fine-tuned versions (InternVLA-A1.5-RoboTwin, -Libero, -DOMINO, etc.) that presumably push the scores even higher on each benchmark. In comparative experiments, InternVLA-A1.5 consistently beats strong baselines, including earlier InternVLA-A1 and other generalist controllers. For instance, in the RoboTwin 2.0 suite InternVLA-A1.5 is said to outperform a π₀.₅ baseline by 26.7% on highly dynamic tasks. (π₀.₅ is a previously reported generalist policy; InternVLA-A1.5’s authors reference it by symbol as a strong prior method.). What stands out is how well InternVLA-A1.5 handles generalization. The LIBERO and LIBERO-Plus benchmarks include a variety of object manipulation tasks, and A1.5 lands at the top. In DOMINO, which emphasizes robustness and dynamic scenarios, it matches or exceeds the state of the art. In EBench (which mixes languages and requires visuo-lingual feats) it is also leading. The authors summarize that A1.5 is “ranked highest or highly competitive across six simulation benchmarks” including robustness tests. In short, combining the semantic VLM training with the latent foresight gives a policy that competes with the best specialized models across the board. One reason for its success is that InternVLA-A1.5 doesn’t just memorize a fixed distribution. Its training on diverse data plus the preserved VLM means it can adapt to novel instructions and appearances. For example, in the LIBERO-Plus visual robustness tests, InternVLA-A1.5 still managed 84.8% average success zero-shot, a remarkable result given the suite’s introduced lighting and camera shifts. The authors credit this to “the preserved semantics” from the VLM – by not overriding the vision understanding, the policy handles new visual conditions gracefully. Real-World Tasks: Compositional Generalization and Long Horizons. Beyond simulation, InternVLA-A1.5 was also tested in the real world on an instruction-following chemistry task suite that the authors call their “4-task real-world suite”. The tasks involve manipulating colored tubes and conducting a multi-step mixing procedure (denoted “MOF”). Critically, these tasks were designed to test compositional generalization: for instance, the agent must sort colored tubes into boxes or holes, but during evaluation the color assignments are swapped to held-out combinations the model hasn’t seen. In these experiments InternVLA-A1.5 showed superior generalization. For the Sort Tubes, Insert Tubes, and Move Tubes tasks, it handled the held-out color-to-box assignments reliably. Specifically, it achieved 80.5% success on the “Insert Tubes” task and 76.4% on “Move Tubes” even on the unseen combinations. In contrast, earlier policies often failed to bind the color and location correctly when these were rearranged. A visualization on the project page shows the difference: the policy correctly follows an instruction like “Place the blue tube in the right box” even if “blue-right” was never paired during training. (Video rollouts of seen vs. out-of-distribution instructions confirm the compositional grounding.). The fourth task, MOF, is a long-horizon chemistry assembly requiring a sequence of 14 steps. InternVLA-A1.5 managed this with 76.4% overall success, far above the ~29% achieved by a baseline π₀.₅ policy. The paper attributes this dramatic improvement to two factors: the latent foresight giving an internal model of how the liquids will flow, and the explicit subtask prediction keeping track of which steps remain. Indeed, the authors report that “the model’s ability to track task progress through explicit subtask prediction and its internal foresight” accounts for the higher success on this 14-step task. Overall, the real-world experiments highlight that InternVLA-A1.5 can not only manipulate dexterously but also understand complex instructions on unseen arrangements and complete long multi-step plans. The preserved language grounding (from Qwen3.5) ensures it follows the instructions literally, while the learned world model (from Wan2.2) and subtask tokens guide it through the sequence without derailing. It’s worth noting that the success rates on these physical tasks are still under review (the paper draft says final numbers are forthcoming), but the presented results clearly exceed prior methods. Comparison to Prior and Concurrent Work. InternVLA-A1.5 is part of a wave of new Vision-Language-Action models pushing the boundaries of generalist manipulation. Its own predecessor, InternVLA-A1 (Cai et al. 2026), introduced the MoT framework for robotics, combining understanding, generation, and action. A1’s generation expert predicted future visual states explicitly, whereas A1.5’s approach is more implicit. The two model cards are similar in spirit: both leverage large-scale pretraining on mixed synthetic/real data, and both use flow-matching for continuous control. Where A1 needed huge compute to learn video prediction from scratch, A1.5 smartly offloads that work to a specialized model. Other concurrent works tackle complementary issues in VLA. For instance, ActiveVLA (Liu et al. 2026) empowers robots with active perception – planning camera viewpoints and zoom – for fine-grained 3D manipulation. Meanwhile FocusVLA (Zhang et al. 2026) studies how to make the policy attend only to task-relevant image regions, reducing visual noise. These papers do not directly overlap with InternVLA-A1.5’s latent foresight idea, but they similarly recognize that naive VLA models have blindspots: ActiveVLA notes most models “overlook visual details” by relying on a fixed camera train, and FocusVLA quantifies how too many irrelevant tokens ruin performance. The InternVLA team’s innovation is orthogonal: instead of focusing on perception, they focus on prediction by leveraging generative priors. It’s also worth situating InternVLA-A1.5 among other generalist robotics efforts, such as PiWorld’s policies (π-models) and open-source stacks like OpenVLA. InternVLA distinguishes itself with its complete self-supervised approach (no online RL) and integration of extremely large models (2B Qwen3.5, 5B Wan2.2). The publicly released model collection (Hugging Face InternRobotics/InternVLA-A1.5-*) and source code promise easy replication and comparison. Using and Extending InternVLA-A1.5. For practitioners, the good news is that InternRobotics has released the model weights and code. On Hugging Face, the “InternVLA-A1.5-base” checkpoint is available (3B parameters), alongside task-specific fine-tuned variants for RoboTwin, LIBERO, and DOMINO. The license is CC BY-NC-SA 4.0, allowing academic and non-commercial use. The official GitHub repo includes training and inference scripts. For example, one snippet shows how to fine-tune the base model on a Push8 task via “internvla_a15_finetune.sh” under their LeRobot setup. Anyone looking to adapt the model can start from the provided base checkpoint and their datasets (InternData-A1 and others are also public). Key practical points:. Input format: To use the policy, you must supply multi-view RGB images (224×224), a language instruction tokenized by Qwen, and the current robot state (joint angles and a one-hot control-mode token). InternVLA-A1.5 tokenizes all of this and runs it through the VLM. Output format: The policy predicts a sequence of future actions (chunk size 50 steps). In control, you can execute these actions sequentially (like model-predict-control). The continuous actions have up to 32 dimensions (e.g. joint torques or poses). Fine-tuning: The “InternVLA-A-series” repo supports fine-tuning on new tasks or accelerations. If you have task demonstrations, you can train the unified expert (and possibly the foresight tokens) on them. The released benchmarks code shows how to do this (they have fine-tuning workflows for RoboTwin, LIBERO, SimplerEnv, etc, in the repo). Because InternVLA-A1.5 was trained on an extremely broad set of tasks and conditions, it may already perform well zero-shot on many scene-understanding manipulation tasks. However, for specific scenarios (unseen objects, kinematics, etc.), fine-tuning or adaptation will likely be needed. The architecture is modular enough to allow that: one could, for instance, keep the VLM backbone frozen, re-train only the unified expert on new trajectories, or even learn new subtask tokens for different task vocabularies. Why InternVLA-A1.5 Matters. InternVLA-A1.5 represents an important step toward truly generalist robot policies. By unifying vision-language comprehension with latent-world-model reasoning, it addresses a core tension in embodied AI: how to benefit from large pretrained language/vision models without throwing away their knowledge when learning dynamics. The solution—“latent querying” of a massive video generator—could be widely applicable. One can imagine future VLA policies that consult not just video models but also physics or geometric priors in their latent space. Moreover, InternVLA-A1.5 is a public benchmark of what is achievable with scale and smart design. It shows that with 1.2M episodes of data and open-source LLM/CV backbones, you can surpass closed-source giants on manipulation. Its excellent compositional generalization is particularly noteworthy: the policy didn’t just memorize color placements, it learned the language grounding and could rebind symbols in novel ways. This hints at a new level of reasoning in robot policies. Of course, InternVLA-A1.5 is not the final answer. It requires large compute and data, and its real-world tests are still limited (only a handful of tasks on a fixed hardware setup). The policy might still struggle with truly out-of-distribution scenes (completely new objects, lighting, etc.) or adversarial setups. Also, running even a 2B-parameter model on robot hardware is non-trivial, though InternRobotics designed it for real-time inference. Finally, the reliance on a frozen video model means it inherits whatever biases Wan2.2 has (e.g. if Wan2.2 wasn’t trained on cluttered tabletop scenes, the foresight might falter there). Nevertheless, InternVLA-A1.5 will likely inspire many follow-up ideas. One could try replacing Wan2.2 with other generative priors (text-to-action, physics engines, etc.). The “latent foresight tokens” concept itself is reusable: it’s similar in spirit to how some vision-language models use “query tokens” to retrieve knowledge. For example, future work might introduce multiple sets of foresight tokens (one for near future, one for long-term planning) or integrate them into a planning loop. In summary, InternVLA-A1.5 is a compelling new Vision-Language-Action model that advances the state of robotic manipulation. It pushes toward a unified policy that “inherits world-model dynamics priors” while staying grounded in language and perception. For researchers and practitioners, it serves both as a strong baseline to compete with and as an open framework to build upon. Sources: The descriptions above are based on the InternRobotics paper and model release (arXiv:2607.04988), the project webpage and README, and related documentation on Hugging Face. These detail the architecture, training data, and reported results of InternVLA-A1.5. Aligning with their findings, the model clearly demonstrates the benefits of training-only latent foresight for embodied control.