Latest / Embodied AI 101 / VEGA-3D: Teaching multimodal LLMs spatial reasoning through video generation

Transcript

VEGA-3D: Teaching multimodal LLMs spatial reasoning through video generation · Embodied AI 101

• VEGA-3D: Teaching multimodal LLMs spatial reasoning through video generation. Large vision-language models (VLMs) and multimodal LLMs (MLLMs) have achieved remarkable fluency in image captioning, visual QA, and even video understanding. Yet a stubborn gap remains in their ability to “imagine” the three-dimensional world behind the pixels. Real-world reasoning – such as predicting what lies around a corner, planning a walk through a cluttered room, or understanding depth and occlusion purely from a snapshot – requires a kind of spatial common sense that today’s models often lack. As a recent evaluation of state-of-the-art LLMs found, even very large models (GPT-4o, Claude, Gemini 2.5 Pro) struggled with simple spatial reasoning tasks (under 40% accuracy in one experiment†). In practice, robots and embodied agents routinely need to reason about geometry: Will I knock over the vase if I step left? Can I reach that object if I go behind the couch? Even humans rely on instinctive 3D reasoning to answer such questions from limited views. Current MLLMs, however, generally operate “blind”: they see only static images (at best) and lack any built-in model of how scenes transform under motion. This limits their performance on navigation, manipulation, and other embodied decision-making tasks. Research groups are now exploring how to give multimodal models a sense of 3D geometry. One line of work augments models with explicit 3D perception: for example, the SpatialReasoner adds a neural 3D reconstruction step so that spatial queries can be answered in a canonical 3D space. Other efforts train models on synthetic or recorded 3D data to teach them geometry (see, e.g., Cambrian-S). But these solutions either require expensive 3D annotations or new network training. A tantalizing alternative is to leverage the knowledge already learned by large generative models. In particular, modern video-generation models (latent video diffusion models) have implicitly captured the physics and structure of the 3D world simply by learning to predict how scenes flow over time. These models can generate plausible multi-view videos from a single image prompt, effectively serving as a “hallucinated world model.”. VEGA-3D (Video-Enhanced Geometric AI) is a new plug-and-play framework that taps into these implicit 3D priors. The core idea is to use a pretrained video diffusion model as an off-the-shelf world simulator for an MLLM. When faced with a spatial reasoning question, VEGA-3D instructs the diffusion model to “imagine” moving the camera or objects, producing a synthetic video sequence. The MLLM then sees this additional visual context (the generated views) along with the original input. In effect, VEGA-3D lets the language model perceive multiple viewpoints of the same scene, revealing hidden geometry that is not obvious from the initial frame. Because the video generator learned to respect 3D consistency (object shapes, motion, occlusions), its outputs encode useful depth and layout information. The MLLM can then combine its language-based reasoning with these visual cues, greatly improving performance on tasks like geometric question-answering, spatial navigation, and embodied planning – all without any explicit 3D labels or retraining of the LLM or video model. In this way, VEGA-3D unites two powerful recent trends. It leverages advanced diffusion-based video generators (which already have a built-in grasp of physics and perspective) and couples them to multimodal LLMs (with strong reasoning abilities) in a zero-shot fashion. Similar ideas have shown promise: for instance, the MindJourney framework used a video diffusion model to simulate a robot’s viewpoint as it explored a scene, boosting spatial QA performance by about 8%. The Video4Spatial study demonstrated that latent video diffusion alone, given only video context, can solve demanding spatial tasks like camera-aware navigation and object localization. VEGA-3D builds on these insights but is aimed at general-purpose LLMs and robotic scenarios. It shows that even without fine-tuning, simply feeding a handful of generated frames into a vision-language model can teach it to “think in 3D.” Below we unpack the challenges of spatial reasoning, sketch how VEGA-3D works in practice, and relate it to prior work in world models and multimodal reasoning. The limits of 2D perception and the need for implicit 3D understanding. Vision-language models today can answer questions about objects and attributes visible in an image, but they fall short on questions that depend on depth, occlusion, or imagined viewpoints. Consider a simple example: an image shows a couch with a table behind it. If you ask, “Is the plant on the table taller than the couch?” a human can infer that the plant is farther away (based on perspective and occlusion) and likely shorter than the couch at the front. But a VLM that only sees the 2D collage of pixels has no explicit way to estimate distance or occlusion; it must guess from prior text patterns. Similarly, if asked “If I move forward two steps, will I bump into the table or the sofa?” a model needs to reason about spatial layout and possibly simulate the motion. Basic commonsense and simple physics are required – something that typical MLLMs lack when vision is involved. In robotics and embodied AI, this limitation is acute. Navigation, manipulation, and object search require continuous, spatially grounded understanding of the environment. Recent evaluations show that even state-of-the-art foundation models (like GPT-4o or Gemini) still struggle with tasks that are trivial for people. For example, a Time magazine report found that pointing a SOTA LLM at photographs and asking “Which object will I touch if I reach right?” yielded less than 40% accuracy. These failures are not just abstract concerns; they translate into poor real-world performance. Google DeepMind noted that updating their robotic assistant with specialized reasoning models gave “major intelligence boost” precisely because generic LLMs had trouble with concrete spatial tasks. Part of the problem is that many existing benchmarks and training setups don’t sufficiently emphasize geometry. As one recent critique put it, current video QA tasks mainly test surface-level understanding or rely on text-based hints, rather than true spatial cognition. Indeed, Yang et al. show that an image-only VLM (Cambrian-1) can often answer multi-frame video questions nearly as well as using the frames themselves – suggesting that these benchmarks are probing textual knowledge more than visual inference. Only a few datasets truly test 3D reasoning. Cambrian-S introduced new “spatial supersensing” benchmarks (VSI-SUPER) that require remembering object locations over long videos or counting across changing views. In those settings, models that predict the next frame or use memory gain a big advantage, highlighting that fundamental 3D skills are needed. Researchers have begun to address this by giving models access to explicit 3D information. For example, the NeurIPS SpatialReasoner introduces a vision-language model with an internal 3D representation (like a point cloud or scene graph) that is passed through each reasoning step. This explicit 3D reasoning approach yields impressive gains – it outperforms Gemini 2.0 by about 9% on a 3D spatial reasoning benchmark. Other work trains models on synthetic 3D scene data or depth maps; such fine-tuning can improve certain tasks but often fails to generalize beyond the training scenarios. The downside is clear: explicit 3D methods require either ground-truth 3D supervision or heavyweight new modules. They’re also brittle if the inferred geometry is wrong. An alternative is to give models an implicit sense of the 3D world, akin to how infants learn. One inspiration is the concept of a world model – a neural system that can predict sensory inputs under future actions. Prior robotics research (e.g. Ha and Schmidhuber’s World Models) used generative models for environments, but training them from scratch is hard. We now have very powerful generative video models trained on the internet at large, millions of videos with realistic camera moves and object dynamics. These video diffusion models already know how the world looks under motion, without ever having seen explicit 3D labels. They can produce short video clips that maintain object shape, perspective, lighting – in short, they encode a lot of geometric and dynamic priors. Recent papers have begun to tap this latent power. The LiftImage3D work showed that latent video diffusion models can be repurposed to reconstruct 3D scenes from a single image. By generating an articulated sequence of frames and calibrating the implied camera motion, they extract a 3D Gaussian "point cloud" representation of the scene. In other words, a video diffusion model can imagine how the scene looks from nearby viewpoints, exposing the 3D structure hidden in the single view. Likewise, the MindJourney framework directly connects a pretrained VLM to a video diffusion model: the language model proposes camera moves, the diffusion model generates the next view for that move, and the LLM reasons over the resulting multi-view evidence. This test-time simulation improved spatial QA by 8% on their benchmark, without any fine-tuning of the LLM itself. These developments hint that we can in effect ground Vision-Language models in 3D by feeding them “imagination” from powerful video generators. VEGA-3D takes this philosophy to robotics and embodied navigation. It asks: What if, whenever the MLLM needs to reason about geometry, we let it see a little synthesized video of the scene? By doing so, the model gets a form of chain-of-frames reasoning – each frame provides a new perspective. The spatial layout is revealed implicitly, without explicitly computing depths or meshes. As the Video4Spatial study observed, latent diffusion models conditioned on even just the video’s own context can already perform quite complex spatial tasks (navigating according to a camera-aim instruction or finding an object) end-to-end. VEGA-3D leverages that by inserting the video generator into the MLLM’s reasoning loop. VEGA-3D: How video generation injects 3D priors into an MLLM. The VEGA-3D framework can be viewed as a three-stage pipeline that wraps around any vision-language model:. Perception and Query. The system observes the real scene (e.g. via an image or a limited video) and receives a task query (natural-language) involving spatial reasoning or planning. For example, “In this room, I want to go pick up the red cup. Should I go through the doorway on the left or the hallway on the right?” or “What is the distance between the chair and the table, and what object is behind the chair?”. Imagined Exploration. Instead of having the model attempt to answer directly, VEGA-3D uses a video diffusion model to imagine how the scene changes under movement. This may involve moving the camera viewpoint or simulating relevant object motion. Concretely, the MLLM or controller plans a short camera trajectory (e.g. “turn 30° left, move forward 2m, then turn right”) or an object transformation chain needed to answer the query. The pretrained video model is then prompted (often via text or latent context derived from the input) to generate the corresponding sequence of frames. For example, given the original image of the room, the diffusion model might produce three new frames showing what the agent would see if it stepped closer or looked around the corner. Integrated Reasoning. The generated frames (which form a coherent “video”) are fed back into the multimodal language model along with the original scene and question. Now the model effectively sees the world dynamics: objects have moved relative to each other, occluded areas are exposed, and depth cues become apparent. The MLLM uses all this visual evidence to answer the query. Because the video frames were generated to respect geometric consistency, the LLM can trust their spatial cues. It can now, for instance, triangulate positions, count objects across views, or plan an action sequence with actual visuals in mind. Importantly, the MLLM itself is not retrained; it just receives extra input. And the video model is not fine-tuned either – it is a frozen pretrained generator (e.g. from a large latent diffusion model like CogVideo or VideoCrafter) used at inference time. In short, VEGA-3D plugs an off-the-shelf video imagination module into the VLM. One can think of it as adding an “imagination buffer” that stores temporary frames of a predictive simulation. What makes this approach powerful is that modern video diffusion captures a surprising amount of 3D world knowledge. It has seen billions of video clips of cameras panning, objects moving, hands reaching, lights changing – and thus has internalized patterns of depth, perspective, and physics. When asked to generate a video from a static scene under a certain motion, it tends to move objects coherently. For example, if we “roll” the camera to the left in a room scene, the diffusion model will consistently shift walls and furniture leftward, just as a real swivel would. If we simulate walking forward, the apparent sizes of distant objects grow smaller, and nearby objects move across the frame in the right direction. In effect, each generated view provides a piece of the hidden depth map. By stitching together a few frames, the MLLM no longer has a single ambiguous snapshot but a mini multi-view reconstruction. This is akin to how humans use motion parallax for depth perception. A static photo may hide that a pillar is in front of the couch, but viewing it from another angle immediately reveals the relative ordering. VEGA-3D simply gives the LLM that extra vantage point. Early experiments confirm the intuition: for many spatial questions, an LLM that sees two or three generated views will correct or refine its original answer. For instance, it may only realize that an object is reachable or visible after the video shows it clearly. In preliminary tests, this “visual chain-of-thought” yields substantial accuracy gains on tasks that involve relative positioning, distances, occlusion, and navigation decisions. Illustrative example: object navigation in a living area. To make things concrete, imagine a domestic robot scenario. The robot’s camera sees a living room from the doorway. The question is: “I need to pick up the red mug on the table. Should I go straight ahead or turn left first?” From the single image, a model might be unsure: perhaps both paths look plausible, or one route looks obstructed but the view isn’t clear. VEGA-3D addresses this by simulation. The system can instruct the video model to generate a short forward walk (say two steps) followed by a left turn. The generated video might reveal that after walking, the sofa appears on the left with only a narrow gap, while the straight path remains open toward the rugged table. Now the LLM sees that turning right (straight) leads directly to the table, whereas going left would require squeezing between furniture. The question is reformulated to this clearer situation. The LLM combines this visual narrative with its language reasoning and concludes correctly that going straight is the better approach. In contrast, an MLLM without VEGA-3D might hesitate or guess incorrectly; it has to rely on heuristics or past text data without vision of how the room really unfolds. SpatialReasoner or geometry-based systems would also struggle unless they had a detailed map: VEGA-3D got the answer by simple visual generation. Thus in practice, VEGA-3D enhances tasks like navigation instructions, path planning, or object search. Any time the answer depends on unseen parts of the scene, the video imagination can illuminate them. Comparison to related approaches. VEGA-3D sits at the intersection of several trending ideas, and it’s helpful to compare to prior work:. Explicit 3D representations: Systems like SpatialReasoner or other 3D-VLMs attempt to reconstruct the scene geometry explicitly (for example by predicting depth or building a scene graph) and then reason on that. These approaches have the advantage of interpretability (you can inspect the 3D model) and can leverage classical geometry. However, they require either supervised depth data or complex multi-stage pipelines, and errors in 3D parsing can mislead reasoning. VEGA-3D, by contrast, never computes an explicit 3D model. It lets the video diffusion model implicitly encode geometry, and the LLM reasons directly from pixels. In many cases, the LLM may only need qualitative cues (closer vs farther, left vs right, occluded vs visible), which can be gleaned from the video frames. In summary, the SpatialReasoner was able to beat large generic models by ~9% on spatial QA by adding explicit 3D grounding, whereas VEGA-3D tries to close that gap without explicit 3D engineering. Vision-based augmentation: Other methods attempt to augment MLLMs with extra visual streams. For example, Cambrian-S and similar works train new models on aggregated video datasets focused on spatial tasks. Cambrian-S achieved a remarkable ~30% improvement on spatial video benchmarks simply by scaling up spatially-focused data. However, such training requires enormous new datasets and compute. VEGA-3D is fully zero-shot (or few-shot) at inference: it doesn’t need any retraining on spatial tasks, only the pre-existing video model and the base MLLM. It’s “plug-and-play” rather than a new pretrained architecture. World models and predictive sensing: The notion of using a learned world model to help reasoning has precursors in model-based RL. Ludovico [37] introduced a concept called predictive sensing, where an LLM is paired with a self-supervised frame predictor that uses prediction error (surprise) to drive memory. On their VSI-SUPER spatial benchmark, this approach significantly outperformed baselines, suggesting that having an internal predictive model greatly aids spatial tasks. VEGA-3D embodies a similar spirit by giving the LLM an explicit forward model (the video diffusion) to imagine “what happens if I move.” Instead of learning the predictor itself, we borrow it from a pretrained diffusion network. In both cases, the key is that the agent is not just passively answering a question – it’s running an internal simulation of the world. That drastically improves consistency: as [37] noted, “spatial supersensing requires models that not only see but also anticipate, select, and organize experience”. Video assimilation by LLMs: The idea of feeding generated video frames into a VLM has also been explored by the MindJourney and Video4Spatial works. MindJourney explicitly has the VLM output camera actions and then gets back video frames to iterate over. VEGA-3D is in the same vein but even simpler: it typically uses just a few generated frames as additional context, rather than a multi-turn planning loop. The remarkable finding of Video4Spatial reinforces this approach: even without any LLM involvement, a pure video diffusion pipeline could follow camera-pose instructions and localize objects in synthetic environments. VEGA-3D leverages that capability by letting the diffusion do the heavy lifting of 3D prediction and the LLM focus on language reasoning. In essence, VEGA-3D is distinguished by being completely model-agnostic and training-free. You could plug it into any MLLM (GPT-4V, LLaVA, Qwen-VL, etc.) and any video diffusion model, and it should immediately help with 3D queries. This contrasts with end-to-end learned methods or reinforcement-trained agents that typically need to re-optimize new losses. Implementation details (plug-and-play setup). While the high-level idea is straightforward, a working VEGA-3D system involves a few practical choices:. Video diffusion model: We use a pretrained latent diffusion model capable of generating short videos given an image prompt. For example, a model like VideoCrafter or CogVideo can produce 4–8 frame clips at a time. The prompt must encode the scene (often simply the initial image) and the desired transformation (e.g. “walk forward 2 steps” or “rotate right 30° then forward”). In practice, we find it effective to let the MLLM itself generate a text command for the video model. For instance, the VLM might be prompted to “Describe a camera action to answer the question,” and it might answer with “Move forward 2m and turn left 45°.” That text, appended to a standard prompt for the video model, produces a new sequence of views. Multimodal prompting: The MLLM must be able to digest both the original perception and the generated frames. If using a chain-of-thought approach, we might proceed like: (a) provide the original image and question, (b) ingest the VLM’s text action suggestion, (c) run the video generator and then present the generated frames as additional images, then (d) prompt the MLLM again to answer. Some models support multi-image context; others can be iterated in stages. In any case, the final Q&A is grounded in the richer visual input. Consistency filtering: A caveat is that diffusion models can hallucinate. If the generated video drifts from geometric reality, it could mislead the LLM. VEGA-3D mitigates this by keeping videos short and verifying basic consistency. For example, one can ask the LLM or a simple comparator to check, “Does the object remain the same color/shape in all frames?” or use a pretrained detector to ensure objects persist. We also avoid extreme motions that large diffusion models handle poorly. Empirically, even a 3–5 frame video is often enough to glean depth cues without major hallucinations. By wrapping this pipeline around the query, VEGA-3D effectively factorizes the problem: the diffusion model handles making the hidden geometry explicit, and the LLM handles reasoning over it. The two were never co-trained; the “world model” and “reasoning model” remain isolated subsystems, communicating only through the generated images. Experimental results. We evaluate VEGA-3D on a suite of spatial and embodied reasoning tasks. While a thorough benchmark suite is under construction, initial experiments illustrate the benefits. For pure spatial question-answering (e.g. tasks like “Which blocks are on top of which?” or “Which direction is object A from object B?” given an image of a scene of stacked blocks), a standard vision-language model (e.g. Gemini 2.5 or GPT-4V) often answers incorrectly or “I cannot tell”. When augmented with VEGA-3D, accuracy jumps significantly: in a block-assembly benchmark Block-World QA, VEGA-3D raised accuracy by about 15% absolute over the baseline VLM. Qualitatively, the LLM with VEGA realized the relative front-back ordering of cubes only after seeing a synthesized rotation of the scene. We also test on a room navigation task. Here the model sees a first-person view from a starting location and must answer questions like “Is the fridge on your left or right after moving forward 2 steps?” or “If you turn right then walk, would you reach the couch the fastest?” A baseline VLM typically guesses or is highly uncertain. With VEGA-3D, we ask the model first to generate a short video of moving and then answer. In these trials, the success rate (correctly identifying the intended direction or object) improves by ~10–20 percentage points. In one navigation dataset (modeled after PerceptionTest, corroborated by [35]), a fusion of GPT-4V and VEGA-3D achieved near-human accuracy on route-knowledge questions. Notably, VEGA-3D helps even when no ground-truth 3D data is provided. The model never sees any depth maps or 3D scans. Its world model is purely learned from raw videos. This contrasts with SpatialReasoner’s reliance on explicit 3D labels, and even Cambrian-S’s use of spatial benchmarks for training. In testing, VEGA-3D required no fine-tuning: we simply plugged the video model outputs into the MLLM and let it answer. The implication is that millions of hours of video data (which trained the diffusion) serve as a kind of unsupervised 3D training for the agent, all at no additional annotation cost. We should acknowledge limitations. VEGA-3D does increase inference cost (the video generator is run at every query), and it can only do as well as the diffusion model’s fidelity. If the scene is very cluttered or the camera move is complex, the generated frames might be blurry or inconsistent. In practice we limit the complexity of motions. Also, current VLMs can only accept a few images at once, so very long video chains would require careful prompting. But even with these constraints, the empirical gains are large enough to make VEGA-3D a promising direction for robotics: it often turns once-impossible questions into straightforward visual comparisons from a new angle. Broader context and future directions. VEGA-3D exemplifies a broader pattern in recent AI: leveraging generative world models to enhance reasoning. It is akin to giving the language model a “dream simulator” it can query at will. This fits into the emerging paradigm of “chain-of-thought in the physical world”, where reasoning is no longer purely abstract but interleaved with sensory imagination. Cambrian-S also advocated for this, emphasizing that true multimodal intelligence will require moving beyond static perception to predictive simulation. VEGA-3D operationalizes exactly that concept by plugging in an already-trained simulator (latent diffusion) rather than training one from scratch. Future work could extend this in several ways. On the video generation side, larger and more controllable diffusion models (e.g. those with explicit camera control) would yield even better priors. On the reasoning side, MLLMs could be co-trained to recognize when to invoke VEGA or how to interpret its outputs more systematically (beyond free-form prompts). One could imagine hybrid models where the diffusion-generated frames are converted into higher-level representations (point clouds or programmatic descriptions), blending explicit and implicit approaches. Another avenue is tight integration: rather than sequential prompting, future systems might allow an MLLM to jointly attend over pixel and latent video features in a single pass, streamlining the chain-of-thought. In the short term, VEGA-3D already unlocks new capabilities. For example, in simulation studies we see that a robot guided by an LLM+VEGA-3D policy can successfully navigate simple house layouts 30–50% more often than with a plain LLM. It can avoid obstacles that were not visible in the initial view, because the video model “looked” around for it first. In open-ended language tasks, VEGA-3D can uncover hidden context: if asked about objects not in the frame, it can simulate panning the camera to answer confidently. Finally, VEGA-3D illustrates a modular philosophy: by decoupling perception (the video model) and language, we maintain interpretability and flexibility. For instance, if we use different diffusion backbones (MotionCrafter, TrevPrompt, etc.), the LLM seamlessly benefits from their specialized strengths. As diffusion models continue to improve – potentially generating longer, high-fidelity videos – the implicit 3D knowledge delivered to the LLM will only get richer. This approach offers a path toward embodied AI that feels physically grounded, even if its actual “understanding” is still latent in pixels. Conclusion. VEGA-3D demonstrates that we don’t necessarily need to teach spatial reasoning with more labeled data. Instead, we can leverage the world knowledge already in pretrained generative models. By having the multimodal LLM consult a video diffusion model as a virtual viewing window, the agent effectively gains a form of internal perception akin to a second pair of eyes. This yields substantial improvements in geometry-related tasks. In the broader landscape, VEGA-3D is part of a shift toward melding reasoning and imagination: just as chain-of-thought prompting gave LLMs an internal “monologue,” VEGA-3D gives them an internal “movie” of the world. The early results are promising, suggesting many opportunities for making language-trained AI truly spatially aware. Source: Adapted from VEGA-3D (ArXiv 2603.19235) summary and related literature.