Latest / Embodied AI 101 / SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Diffusion Transformers
Transcript
- SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Diffusion Transformers. Transparent, controllable world models are an exciting frontier of generative research: given one image and a camera-control trajectory, generate a realistic video of that scene under motion. Earlier “world model” systems could produce only very short clips (a few seconds) or needed massive compute, but NVIDIA’s new SANA-WM changes the game. It is an open-source 2.6B-parameter video diffusion model that, starting from a single image and a 6-DoF camera path, generates one-minute long, 720p videos – all on a single GPU. In other words, SANA-WM turns “one image + camera plan → 720p, 60s controllable video on one GPU” into reality. Despite its modest size, it matches the visual quality of much larger industrial systems (for example, it reaches the fidelity of Tencent’s LingBot-World and HY-WorldPlay) while being orders of magnitude more efficient. The key is a series of clever design choices that trade off model size for smart architecture. The SANA-WM paper identifies four core ideas that together unlock this scalability: (1) a hybrid linear attention scheme, (2) a dual-branch camera control mechanism, (3) a two-stage generation pipeline (with a refinement stage), and (4) a robust data/annotation pipeline. We’ll unpack each of these ideas. In the end, SANA-WM achieves 36× higher throughput than a fully-softmax-based world model, while fully supporting precise 6-DoF control. After training on about 213K public video clips in two weeks on 64 H100 GPUs, a single H100 can generate a full 60s clip, and even a distilled 4-bit version can run on a consumer RTX 5090 in about 34 seconds. This marks the first time a minute-long world-model video can be produced on a single GPU and even on consumer hardware. Why Long and Controllable Videos Matter. First, a quick reminder of what a “world model” is. Unlike a generic text- or image-to-video model, a world model takes an initial scene and a specific camera motion plan as input, and must generate the video that follows exactly from moving the camera through the scene. For example, given a snapshot of a forest glade and instructions like “move forward 3 meters, then pan 30° to the right,” the model should render the sequence of frames that a real camera would see. Because the camera motion commands are precise (e.g. 6-DoF paths), the model’s output must obey 3D geometry and temporal consistency. Such controllable simulators are highly desirable: they could augment training data for autonomous vehicles, let filmmakers explore a virtual scene as if with a physical camera, or allow robotics researchers to “roll out” plausible future scenes under different actions. Unfortunately, generating long, realistic videos, especially with precise continuity, has been very expensive. Prior published world models like HY-WorldPlay (8B params) or LingBot-World (28B params, from Ant Group) needed large GPU clusters: HY-WorldPlay used 8 A100s (hundreds of GB) to barely produce <480p clips, and LingBot-World required two 14B models on 8 GPUs (over 450 GB memory) to yield about 1 minute of video per hour. In contrast, SANA-WM runs on a single H100 (80GB). It’s roughly 10× smaller in parameter count (2.6B vs tens of B) but designed to “punch above its weight” through architecture. The fact that SANA-WM with 2.6B parameters matches or beats the output quality of 8–28B models, while generating 60-second clips at high resolution, is a testament to these specialized techniques. Hybrid Linear Attention: Tractable Long Videos. The single most important trick in SANA-WM is how it handles attention over many video frames. A 60-second video at 30 fps is 1800 frames; even in a compressed latent space, that’s on the order of 1000+ tokens – too many for standard quadratic attention. SANA-WM’s solution is a hybrid attention transformer. It interleaves linearized attention blocks with ordinary softmax-attention blocks. Concretely, out of 20 transformer layers it uses, only 5 (e.g. layers 3, 7, 11, 15, 19) use full Softmax attention; the other 15 layers use a special “Gated DeltaNet” (GDN) linear attention. How does this help? The Gated DeltaNet is essentially a recurrent linear-attention mechanism: it maintains a fixed-size state that gets updated each frame, rather than keeping all past key/value vectors in memory. This means memory is constant in the number of frames. As the authors note, using plain Softmax at 60s-long context would overflow GPU memory, whereas with GDN the model can handle 961 latent frames with only ≈51 GB peak memory (and importantly avoids OOM). At the same time, the intermittent Softmax layers ensure that long-range relationships can still be captured precisely. In ablations the hybrid design proved far better than either all-Softmax (OOM) or all-linear (lower quality) for the trade-off of speed and fidelity. Intuitively, one can think of GDN layers as carrying forward the gist of the scene through time (like a memory that grows or decays via learned gates), while occasional full-attention layers refocus the model on exact details. According to the authors, this mix yields a roughly 36× higher throughput than a pure-Softmax model – a huge jump without obvious loss of quality. Dual-Branch Camera Control: Coarse + Fine. World models must follow the camera path exactly, or else the result isn’t useful. SANA-WM introduces a novel “dual-branch” design to handle 6-DoF control with high precision. In one branch, called Unified Camera Positional Encoding (UCPE), the global trajectory is encoded at the latent-frame rate. Each latent token is told the camera’s 3D pose (position and orientation) relative to the scene’s coordinate frame. This situates each frame’s viewpoint correctly. However, there’s a catch: the video encoder compresses multiple real frames into each latent. In SANA-WM’s setup every 8 actual frames compress to 1 latent frame. That coarse compression means some fine motion detail could be lost. The second “Plücker” branch remedies this. Every group of 8 real frames’s worth of camera-rays is converted into Plücker coordinates and aggregated into a 48-channel feature. These features capture sub-frame ray directions of the moving camera. By injecting both the UCPE (coarse pose per latent step) and the Plücker (fine-grained ray details per pixel region) into the model, SANA-WM ensures that both the broad trajectory and the subtle inter-frame movements are honored. In practice this pays off: SANA-WM’s camera-following error is extremely low. For easy trajectories it reports only 4.50° rotation error (versus 12.96° for Matrix-Game 3.0, a contemporary competitor). Even on hard camera paths the error is well under 10°. These are impressive numbers, especially given SANA-WM’s small model size. In short, the dual-branch control encodes the camera path at two scales: a macro pose sequence and a micro coordinate embedding. The model’s transformer can attend to both signals (fed into the rigid-attention query/key streams) so that the generated frames move the camera exactly as specified. This double encoding ensures precise adherence to any user-supplied trajectory. Two-Stage Generation with a Video Refiner. Generating a one-minute video in a single pass is also challenging in terms of consistency. Even with the best diffusion model, slight artifacts or “drift” accumulate over time. SANA-WM deals with this by splitting generation into two stages. The first stage is a vanilla diffusion rollout (the 20-block hybrid transformer) that produces a raw latent sequence. The second stage is a refiner network that polishes this output for long-range consistency. Concretely, the refiner is built on a large pretrained video model (an LTX-2 architecture, about 17B parameters) with LoRA adapters. It is trained not from scratch, but in a special “truncated σ-flow matching” mode: it takes the Stage-1 latent video, adds a bit of noise, and then trains to undo that noise towards the ground-truth frames. During this fine-tuning, reference frames (e.g. the initial image) are used as anchors (keys/values) to keep the appearance consistent. The result is that only a few (3–4) denoising steps in Stage 2 are needed to elevate quality, rather than a full multi-step diffusion. The paper reports that adding the refiner dramatically improves results: a benchmark metric called VBench overall score jumps from about 74.7 (no refiner) to 80.6 (with refiner), essentially matching the level of 28B-parameter baselines. In practice one can think of Stage 1 as producing a plausible 60s video that roughly follows the scene, and Stage 2 “sharpening” and fixing any slow wander or blurring so that the final video stays crisp and coherent even at the end. Building a High-Quality Dataset. One of SANA-WM’s pillars is not just model design but the data it trains on. The team collected about 213,000 video clips from public sources, each annotated with metric-scale 6-DoF camera poses. These clips come from datasets like SpatialVID-HQ, DL3DV, MiraData, etc., covering indoor, outdoor, game, and nature scenes. Importantly, the camera poses are estimated very carefully (using depth-based pose regression models like Pi3X and MoG-2 and filtering for consistency) so that the supervision signal is accurate. They also synthetically augment data by rendering trajectories in 3D scenes (using 3D Gaussian Splatting, for example) to expose the model to more varied motions. Having accurate metric poses is crucial: it allows the diffusion model to learn the physics of how pixels should shift under 3D motion. Thanks to this pipeline, SANA-WM can be trained with only publicly-available videos (about 213K clips of various lengths) and still learns high-fidelity geometry. In effect, the annotation process yields input-output pairs (image, camera path)→video where the “correct” video is known. This different labeling strategy (compared to a text prompt, say) focuses training on spatio-temporal geometry. Training and Runtime Efficiency. Training SANA-WM is nontrivial but still within reach of a well-equipped lab. On 64 H100 GPUs, they train for about 15 days total. The training is staged: first fine-tune a VAE to compress video frames, then train shorter sequences, and finally expand to the full 60s with camera conditioning. Notably, to fit very long contexts, they use a “context-parallel” trick: the 961-frame latent sequence is split across two GPUs during training, and a careful affine-state composition is used to exactly reconstruct the GDN memory state without loss. This lets them train with long-horizon sequences despite memory limits. At inference time, the gains are dramatic. A single H100 can generate a 60-second 720p clip in two ways: with the full model it takes some minutes (depending on denoising steps), but with a distilled “sprint” version (using consistency models and NVFP4 4-bit quantization) it can generate in about 34 seconds on an RTX 5090 GPU. (This is a huge leap: previous state-of-the-art needed around $240,000 worth of H100 GPUs to generate one such clip; SANA-WM can do it on a $2,000 consumer GPU.) Even without distillation, a raw single-H100 run is still much faster per clip than earlier systems. In fact, on the paper’s benchmark SANA-WM (with refiner) manages 22.0 clips per hour per GPU, whereas LingBot-World only did 0.6 per hour on 8 GPUs – meaning in per-GPU terms SANA-WM is ∼293× more efficient. This improvement is essentially entirely due to its streamlined architecture (hybrid attention, etc.), not a gigantic model size. In summary, the efficiency story is that nothing was free–they invested work in model design and training tricks so that at inference the world model is very lightweight. As the authors emphasize, the breakthrough is not “bigger is better” but “better architecture is better.” The hybrid attention alone gives 36× throughput vs all-Softmax. The distillation and quantization bring it down enough to run on a laptop-class card. Everything is open-source too, so others can build on it. Results: Quality, Control, and Comparison. How good are the videos? By multiple metrics, SANA-WM holds up against the best. In the paper’s one-minute world-model benchmark (80 diverse initial frames, easy and hard camera paths), SANA-WM + refiner achieved a VBench score around 80.6 on easy paths (and 81.9 on hard paths). This matches or beats LingBot-World (81.8) and far outpaces smaller models like Matrix-Game 3.0 (78.5) or HY-WorldPlay (68.8). Recall that LingBot-World achieved its ~81.8 using 28+28B parameters at 480p, whereas SANA-WM hits similar numbers at 720p with only 2.6B parameters. Visually the footage is crisp and stable: one significant win is temporal consistency. Models without a refiner (or without a careful design) tend to “drift” so later frames blur or deviate. SANA-WM’s drift is almost zero – the VBench quality drop from start to end of a clip is only ~0.3 (refined) or 1.2 (without refiner) out of 100, comparable to LingBot-World. In contrast, HY-WorldPlay showed large degradation. For camera adherence, the numbers speak clearly: SANA-WM’s rotation error is 4.50° on simple paths and 8.34° on complex paths. By comparison, LingBot-World had ~10° on simple, and HY-WorldPlay had >18°. The translation error is likewise tiny (about ~0.12 m on 3–5 m moves). In short, the model follows the instructions more closely than any open alternative. The efficiency gains were already noted: SANA-WM (with refiner) yields 22 videos/hour per H100 versus LingBot’s 0.6. Even ignoring throughput, one can run a 60s inference at 720p with <75 GB of GPU RAM (versus 450+GB needed by LingBot on 480p). SANA-WM also runs elegantly on consumer hardware (34s per clip on an RTX 5090 with quantization) – a first for long world models. These results underline that SANA-WM is not just a technical demo but a practical system: a decade ago, such an effect on one GPU was science-fiction. Discussion and Future Directions. SANA-WM is an impressive engineering achievement, but it also points to future research directions. On the positive side, it shows that carefully combining efficient attention (GDN) with a few full-attention blocks can handle extremely long video contexts. The dual-branch control shows how to bake geometry into a transformer model so it never forgets the viewpoint. And the two-stage approach reminds us that it’s okay to refine coarse outputs rather than insisting on perfect generation in one pass. All code and weights being public means the community can extend SANA-WM to new domains (e.g. adding objects, interactive control, or other forms of action beyond camera motion). The flipside is to note the remaining limitations. SANA-WM still lacks any explicit 3D scene memory – if you move the camera all the way around and come back, small details may shift (its PSNR on revisits is only ~14–15 dB). Fast-moving people or cars can be inconsistent frame-to-frame over a long clip. And beyond ~60 seconds, even SANA-WM’s latent memory will strain. Overcoming those probably requires adding a structured memory (e.g. an explicit 3D scene representation) or coupling the model more tightly with a task (like integrating with an RL loop). In other words, SANA-WM is a big step forward, but one can imagine the next steps being “SANA-WM with world maps” or “SANA-WM for fine-grained object physics”. For now though, SANA-WM provides a powerful building block. Robotics researchers can use it to simulate camera trajectories cheaply. VR/AR developers can generate background worlds for minutes at a time. And video-generation researchers have a new baseline to compare against. The era of “long, controllable AI videos on one GPU” has begun – thanks to hybrid attention and some clever engineering. Sources: The above draws on the SANA-WM paper and project release (NVIDIA Research, 2026), along with summaries of their results. These show how the authors built and evaluated an open 2.6B world-model that achieves minute-long 720p controllable video generation on a single GPU.