Latest / Embodied AI 101 / ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation
Transcript
- ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation. Embodied AI is rapidly advancing through Vision-Language-Action (VLA) models – large models that take in a robot’s camera view and a language instruction, and output control actions. By training on massive datasets of demonstrations, open-source VLA models (like OpenVLA) can follow natural-language commands to manipulate objects. However, the very factors that make VLAs powerful also make reasoning hard: they must simultaneously ground instructions in visual context and emit smooth high-dimensional actions. Early VLA policies usually learned this mapping end-to-end, hiding complex planning in latent states. As a result, they often struggle with long-horizon tasks or ambiguous instructions. Recent work has begun to equip VLA actors with explicit reasoning – for example generating intermediate “chain-of-thought” steps – to bridge this gap. ThinkingVLA sits squarely in this line of research. It introduces an interleaved vision–language thinking mechanism for robot control, letting the agent alternate between interpreting language subgoals and processing visual feedback during planning. To see why this matters, recall that a VLA policy maps the current observation (camera image + proprioceptive state) plus a language command into the robot’s next action. In prior work, models like Qwen3-VL or OpenVLA were trained end-to-end on hundreds of thousands of episodes. These learn powerful multi-modal representations, but when faced with a multi-step instruction or a cluttered environment, they often have trouble breaking the problem into steps. Research has shown that integrating an explicit planning step – a kind of chain-of-thought – can dramatically improve performance on such tasks. For example, DeepThinkVLA restructured a VLA policy so that it first writes out a textual reasoning trace before emitting the action chunk. This hybrid approach (combining causal and parallel decoding) and a two-stage training regimen (supervised “thinking” followed by RL fine-tuning) boosted success rates on the LIBERO benchmark from ~80% to ~97%. In other words, letting the robot “think out loud” in text about what it’s going to do proved extremely effective. While DeepThinkVLA focused on textual reasoning, other work has shown that visual reasoning traces can also help. For instance, NVIDIA’s CoT-VLA system explicitly generates subgoal images as intermediate goals. Given the initial scene and language instruction, CoT-VLA prompts the model to imagine a future keyframe (e.g. “How should the scene look after step 1?”) before planning actions to reach it. This image-conditioned chain-of-thought provides geometric grounding (where to move, what objects to focus on) that pure text might miss. Indeed, Zhao et al. found that augmenting a 7B VLA model with a “visual chain-of-thought” – predicting future frames – yields strong gains in manipulation tasks. They train a model that switches from causal attention (for generating the subgoal image) to full attention when outputting actions, achieving robust planning in both simulation and real robots. Building on these ideas, it becomes natural to combine vision and language reasoning. Models like VLA-Thinker argue that relying only on text-chain-of-thought can be limiting if the visuals aren’t re-integrated during planning. Wang et al. note that treating image inputs as static context “limits the ability of the model to actively revisit the environment and resolve ambiguities during long-horizon tasks”. In other words, the robot should be able to “look again” and re-evaluate vision when reasoning. This motivates an interleaved approach: instead of planning purely in language or purely in pixel space, the agent’s internal reasoning trace alternates between processing language subgoals and visual information. The recent ThinkingVLA paper (Lu et al., ’26) takes this step. It proposes a VLA framework in which the robot’s policy explicitly interleaves reasoning in the visual and linguistic domains. Concretely, their model is built on a vision-language transformer backbone (like Qwen3-VL) that can process image+text inputs. At each time step, the policy can alternate between two modes of “thinking”:. Linguistic reasoning steps: The model writes out a brief textual plan or question, refining the robot’s understanding of the instruction. For example, after parsing “plug the red cable into the outlet,” the agent might generate an internal subgoal like “Look for the red cable next to the wall.” This uses the language model component to dissect tasks into sub-steps. Visual reasoning steps: The model focuses on the camera image (potentially along with a latent map of the scene). It attends to different parts of the image or generates latent visuals representing the current or predicted state. For instance, after having the subgoal from the previous step, the visual step might “zoom in” on the left side of the workbench to verify the cable’s location. This uses the vision encoder to extract spatial details and resolve any visual ambiguity. These reasoning steps are interleaved in sequence. A simple way to imagine it is: the policy maintains a hidden “trace” of reasoning tokens. One token might be a word (generated by the language head), and the next might be a special visual token or latent vector (produced by the vision head). By alternating text and image reasoning, the agent builds a richer internal plan that is both semantically coherent (via language) and spatially precise (via vision). At the end of this multimodal reasoning chain, the policy then decodes an action sequence conditioned on both the original instruction and the entire reasoning trace. Importantly, ThinkingVLA learns this interplay end-to-end. The authors capitalized on large pretrained VLAs (like a “Thinking” edition of Qwen3-VL) as the starting point – these models already understand language and images well. They then fine-tune with a combination of imitation learning and reinforcement learning, encouraging the policy to use the interleaved trace format. In practice, they generate pseudo-labels to bootstrap training: for each demonstration (instruction + expert trajectory), the data is segmented into steps, and each segment is paired with a caption or sub-instruction using a visual-language model (e.g. Gemini or Qwen3-VL-2B). This provides supervised signals for the intermediate reasoning steps. During RL fine-tuning, the model is rewarded not only for completing the task but also implicitly for maintaining a useful reasoning chain. The net effect is that ThinkingVLA develops integrated multimodal reasoning. The language steps ensure the plan stays consistent with the instruction’s logic, while the visual steps ground that plan in the actual scene. For example, if the robot is asked to “assemble the blocks in a pyramid,” a pure language chain-of-thought might outline “pick up green block, place on red block, then blue on top.” But with visual validation, the agent can check if the green block is on the left or right, reposition if needed, or even realize an obstacle is in the way. In an experiment, Lu et al. demonstrate this on a suite of manipulation tasks (e.g. arranging objects, routing a cable through holes, or inserting a peg into a slot). Compared to a standard VLA baseline, ThinkingVLA notably improves success on long-horizon instructions and ambiguous scenarios. For instance, on a table-setting task where the instructions omit specifying order, the interleaved model achieves roughly 15–25% higher success than a text-only chain model. While exact numbers can vary by task, these gains mirror what other “thinking” models have seen – DeepThinkVLA saw a 15.5-point boost just from its hybrid decoder, and VLA-Thinker reported similar leaps when adding visual reasoning steps. In summary, ThinkingVLA represents a new direction in VLA research: embodied, multimodal thinking. By formalizing an interleaved, chain-of-thought-like process that alternates between vision and language, it allows robots to better interpret instructions and adapt to their surroundings. This is part of a broader trend toward more elaborate reasoning in embodied agents. Other concurrent work, for example, explicitly stitches together text subgoals with image keyframes as a global plan trace. ThinkingVLA complements this by letting the model self-generate its reasoning on the fly, directly conditioned on current observations. Together, these approaches illustrate that guiding VLA models with internal reasoning – whether linguistic, visual, or both – makes them more effective. As one implementer of VLA summarised: most robot policies “have shown promising capabilities, but … text-based chain-of-thought reasoning [alone] treats the visual inputs as static”. ThinkingVLA breaks out of that limitation by enabling back-and-forth dialogue between what the robot reads in the instruction and what it sees in the world. Key citations: This work builds on the vision of chain-of-thought for robots (see DeepThinkVLA for language-based reasoning, CoT-VLA for visual subgoal planning, and VLA-Thinker for combined image reasoning). TorchRL’s definition of a VLA policy clarifies the basic problem setting. ThinkingVLA brings these threads together in a unified framework for interleaved vision–language reasoning to push the limits of multimodal robot instruction-following performance.