Latest / Embodied AI 101 / HyDRA: Hybrid Memory for Dynamic Video World Models
Transcript
- HyDRA: Hybrid Memory for Dynamic Video World Models. Video world models aim to give AI agents the ability to imagine future scenes and make decisions by simulating what happens next. This capability is crucial for robotics and autonomous systems, from planning a route around moving obstacles to modeling how objects will move under force. However, most existing video world models suffer a critical blind spot: they implicitly assume that nothing changes unless the camera sees it. In practice this “static canvas” assumption leads to glaring failures. When a moving subject – say a car or a walking person – disappears behind an obstacle or the camera pans away, the model simply forgets its trajectory. On reappearing, the subject often comes back as a frozen statue or a blurry ghost (or vanishes altogether) instead of resuming its motion seamlessly. In other words, the model has not “remembered” the object’s motion while it was hidden. This is not just a visual glitch; it breaks the physical realism needed in embodied settings. As Chen et al. note, video world models are “critical for applications like autonomous driving and embodied intelligence” and yet current memory schemes cause “hidden characters to vanish, freeze, or distort upon re-emergence”. For any agent that must predict occluded objects (e.g. a robot holding a tool behind a cabinet, or a self-driving car tracking a pedestrian around a corner), this kind of failure is unacceptable. The new paper “Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models” by Chen et al. (2026) from the Kuaishou Kling Team directly tackles this blind spot. Their key idea is a Hybrid Memory paradigm: the model must simultaneously act as an archivist that remembers static parts of the scene, and as a tracker that actively predicts and recalls moving subjects even when they leave view. Intuitively, the world can be split into two parts – the static background (buildings, environment geometry) and the dynamic actors (cars, people, animals) – and each part requires a different memory strategy. The static background should be stored lavishly and unchangingly (so that the ground and walls don’t warp), while each dynamic object needs its own “thread of memory” that keeps its identity and motion alive off-screen. By design, the Hybrid Memory approach forces a future model to never let a moving object “go on holiday” unnoticed. To realize this idea, the authors build two pillars: (1) HM-World, a new synthetic video dataset that contains deliberately engineered occlusion events to train and test this capability, and (2) HyDRA, a specialized video-diffusion model with a novel architecture for hybrid memory. HM-World provides the practice regime (a gym) where the network must learn to track hidden objects, while HyDRA is the tool that actually implements the memory. In experiments, HyDRA dramatically reduces the “freeze-and-drop” failure modes of prior models, producing ongoing, physically plausible motion even through long occlusions. The Challenge: Occlusions and the “Static Canvas” Fallacy. To appreciate HyDRA’s contribution, let’s unpack the problem in more detail. Traditional video generation or prediction models (whether autoregressive networks or diffusion transformers) are typically trained on video clips where everything in view matters at each time step. Their memory mechanisms (if they have any) tend to treat the past as just a static backdrop. For example, many recent video diffusion models simply append past frames as input context or use a fixed-size attention window. This works fine if the camera is panning smoothly or the scene is mostly static. But it fails spectacularly whenever something moves out of sight. Once a subject walks behind a pole or out the door, the model loses track of it. It has no explicit notion of an “object continuing off-screen.” So when that subject comes back in frame, the model has effectively reset it as a new object – often rendering it as a blurry shape, a statue, or not at all. This is exactly because current video world models are built on a static-canvas assumption. They excel at remembering static backgrounds – a building stays a building, a road stays in place – but any motion that disappears is treated like a one-off event, not a continuing process. In the words of Chen et al., current memory schemes “primarily treat environments as static canvases”. In effect, the model assumes the physics of moving objects pauses when they leave view. When reality requires the physics to carry on, the prediction is physically implausible. For roboticists, this manifests as broken predictive physics: the model has no idea where that car or ball will be because it quit simulating its motion while it was out of sight. Some prior work has attempted to patch this in limited ways. For example, Yu et al. (SIGGRAPH Asia 2025) introduce a “Context-as-Memory” approach where all past frames are stored as memory and a retrieval step selects which past frames to use. Their memory is still the raw frames, and retrieval is based on geometric overlap. It can help with static consistency, but it does not really understand “moving subjects” as entities. Likewise, memory-augmented video models have appeared for editing tasks (e.g. Memory-V2V which caches previous edits to maintain consistency). Those methods show the value of explicit memory, but they either target editing scenarios or use very general caches. None of them explicitly separate static scene memory from independent moving objects. HyDRA takes a different tack: explicitly decouple static background from dynamic objects in the memory architecture. The goal is to ensure that, even if a subject is completely out of view for many frames, its identity (appearance) and trajectory (motion) remain intact behind the scenes. When it re-enters, the model “remembers” exactly who it is and where it was heading. In short, HyDRA’s Hybrid Memory paradigm forces the model to never treat a hidden actor as just “no longer of concern.” It keeps the hidden actor “in mind” the whole time. HM-World: A Dataset for Out-of-View Dynamics. A big part of HyDRA’s contribution is simply pointing out that there was a missing benchmark. Few public video datasets contain enough occlusions to rigorously test this issue, since naturally occurring videos rarely align with a camera pan that exactly hides and then re-reveals a subject. To fill this gap, the authors built HM-World (Hybrid Memory World) – a large corpus of synthetic videos specifically designed for out-of-sight challenges. HM-World consists of 59,225 high-fidelity video clips generated in Unreal Engine 5. Each clip features a carefully choreographed “exit-and-re-entry” event: a camera (with independent pan/tilt) moves in a scene containing moving subjects (people, animals, vehicles). The key design is that the camera trajectory and the subject trajectory are decoupled. In many clips, a subject will walk behind an obstacle or literally out of the frame, then reappear after some time – all while the camera may also be moving. The dataset spans 17 diverse scenes (indoor and outdoor), and includes 49 different subjects with various motion patterns. By construction, each scene contains multiple such exit-entry events per video. This synthetic dataset lets the model practice exactly the core skill: when you “lose sight” of an object, predict where it goes and be ready when it comes back. As Chen et al. write, HM-World is a “large-scale training resource” that enables the model to learn spatiotemporal decoupling, “simultaneously anchoring static backgrounds and tracking dynamic subjects that disappear and reappear”. In a sense, HM-World is the first “video puzzle” set for hidden-subject consistency in world models. By using synthetic data, the authors unleash creative control: camera angles, scene complexity, subject types and motions can be varied systematically. This means HyDRA can be evaluated on scenarios far more challenging than typical natural videos. (The hope is that insights will transfer to real-world tasks – but for now HM-World stresses the model in the exact way designers intend.). The HyDRA Architecture. With the Hybrid Memory paradigm and dataset in hand, the second pillar is the HyDRA model itself. At its core, HyDRA is a video diffusion-based transformer (a “video diffusion model”) that has been outfitted with a special memory system. The underlying generation engine is similar in spirit to recent video prediction models: it takes as input some history of frames and camera poses, and it generates the next frames (often in a diffusion denoising manner). The novelty is how it implements memory. In simple terms, HyDRA inserts a Memory Module into the diffusion transformer. This module has three main components: (1) Memory Tokenization, (2) Dynamic Retrieval Attention, and (3) Camera Injection. Together, they ensure the model can compress and recall relevant history for moving objects without drowning in irrelevant details. Memory Tokenization. Instead of naively storing all past latents or pixels (which would be huge and noisy), HyDRA first compresses the recent history into a small set of “memory tokens.” A 3D convolutional network processes the stack of past latent frames and produces a sparse set of learned tokens. The key is that this tokenizer uses a large 3D receptive field (across space and time) to summarize motion patterns. As the authors describe, the 3D tokenizer “expands the receptive field to capture long-duration motion information,” filtering out static clutter while preserving the essence of moving subjects. In effect, each token becomes a compact summary of where something in the scene has been moving. Because these tokens are low-dimensional, the model can hold many of them without exploding memory costs. This compression is crucial – it retains the motion cues needed to reconstruct hidden objects, but tosses away redundant background detail. Dynamic Retrieval Attention. During generation, for each new frame, HyDRA does not simply attend to all past tokens. Instead, it performs a relevance search: it takes the current query (the noisy latent of the target frame being decoded) and compares it to each memory token to compute a spatiotemporal affinity. This measures how likely each token is to contain information about the actors relevant to the current frame. HyDRA then selects the top-$K$ tokens (those with highest affinity) to actually use. These top tokens are concatenated with the recent short-term context (say a sliding window of previous frames) as the input to the attention layer. This “dynamic retrieval attention” effectively means the model only pulls in those memory chunks that seem related to the subject at hand, rather than re-attending to irrelevant history. By focusing on top-$K$ relevant tokens, the model keeps hidden subjects alive: if a token remembers the predator’s left turn before it disappeared, that token will be chosen once the predator reappears heading in that direction. Meanwhile, local denoising stability is preserved by still attending to the immediate past frames. Camera Injection. Finally, HyDRA explicitly feeds the camera motion cues into the network. The full 6-DoF camera trajectory (rotation plus translation) is encoded and injected as a condition in the transformer layers. This allows the model to decouple ego-motion from object motion. In realistic scenes, the camera often moves while tracking a subject; knowing exactly how the camera moved helps the model compute where the subject should be in world coordinates. By encoding camera pose changes, HyDRA can, for example, recognize that a stationary object in the world should appear moving in the new frame because the camera turned. This explicit conditioning was found to improve spatial accuracy and prevent the model from double-counting motions. Importantly, world coordinates of moving objects remain consistent regardless of camera shifts. All together, HyDRA’s architecture explicitly separates “what to remember” (the memory tokens for dynamic actors) from “what to hold static” (the background encoded via local frame-to-frame modeling). The combination of tokenization and retrieval is the core innovation: it is an end-to-end differentiable memory read/write system tailored for video diffusion. In the authors’ words, HyDRA “compresses memory into tokens and utilizes a spatiotemporal relevance-driven retrieval” mechanism to “preserve the identity and motion of hidden subjects”. Training and Results. Chen et al. train HyDRA on HM-World and compare it to several strong baselines. These include Wan2.1 (a state-of-the-art open video diffusion model), the Context-as-Memory technique, and even a commercial real-time model (often referred to as “WorldPlay”). They evaluate on held-out HM-World clips with occlusions. The improvements are striking. Quantitatively, HyDRA produces much higher fidelity when formerly-hidden subjects re-enter view. For example, HyDRA’s PSNR on the predicted frames is about 20.36 dB, compared to ~18.7–18.9 dB for the best baselines. In other words, it cuts the reconstruction error nearly in half (in pixel terms) after occlusions. Its SSIM metric likewise exceeds the baselines, indicating crisper details. But more importantly, HyDRA scores far better on a custom “Dynamic Subject Consistency (DSC)” measure, which quantifies how well the model maintains an object’s identity and motion across the occlusion gap. (DSC is essentially the IoU or tracking consistency between ground truth and predicted objects.) HyDRA achieves an exceptionally high DSC score (~0.85) compared to far lower scores for other methods. This means that when an object reappears, HyDRA almost always recognizes it as the same object and places it in the correct motion context. Qualitatively, the difference is obvious. In example results, baseline models often depict the re-entering subject as stopped in place or distorted (sometimes two faded outlines), whereas HyDRA shows it moving smoothly through the occlusion gap. In side-by-side videos, one can see the subject “slide” out of view and then pick up the same velocity when coming back – just as if a human kept track of it mentally. Even the commercial WorldPlay model, which can generate long videos, tended to preserve appearance but would frequently “stutter” the motion when a subject returned. HyDRA, by contrast, produces fluid, continuous trajectories with the correct pose and appearance. On absolute numbers, HyDRA even outperforms the commercial model in this synthetic testbed. The PSNR gap was about 5.5 dB in favor of HyDRA, a substantial margin. The authors note that the commercial model “suffers from stuttering motion,” whereas HyDRA achieves what you’d expect from a consistent physics simulation – the hidden object obeys its prior velocity and trajectory precisely. The bottom line is clear: enforcing hybrid memory yields a qualitatively and quantitatively more coherent world model under occlusion. Ablation tests further confirm the importance of each ingredient. If HyDRA’s memory tokenizer is replaced with a trivial temporal average (no 3D conv), performance drops dramatically – showing that capturing motion cues over time is essential. Likewise, if the retrieval instead uses a naive rule (e.g. only tokens from known field-of-view overlap) rather than this affinity ranking, the DSC score falls, indicating the fine-grained token matching is needed. In short, the experiments demonstrate that it’s not just more data or bigger model: the specialized memory design is the critical factor. Implications and Outlook for Embodied AI. The HyDRA work is significant because it highlights and begins to fill a crucial gap in video world modeling. Many tasks in robotics, simulation, and autonomous agents involve temporally extended interactions with objects that can leave the frame. A warehouse robot, for instance, might watch a box roll across the floor, temporarily lose sight of it behind a pillar, and then reacquire it. To plan grasps or predictions of where that box will end up, the agent’s world model must carry forward the box’s momentum even during the transient blind spot. Likewise, in autonomous driving, pedestrians and cars often occlude behind other vehicles. A truly reliable driving world model must not treat them as coming to a halt when obscured by a truck. By explicitly tackling this “forgetfulness,” HyDRA moves us closer to world models that respect continuity and predictive physics of dynamic entities. The hybrid memory paradigm advocated here could be a building block for many future systems. For example, one could integrate HyDRA-like memory into reinforcement learning: an RL agent with such a world model could imagine multi-agent scenarios more realistically. In simulation-based training, using HM-World or similar curricula could better train models to “remember” off-screen. Furthermore, the idea of token-based memory with relevance retrieval might inspire other modalities (e.g. 3D LiDAR or multimodal world models) to maintain separate memory streams for static map vs. moving obstacles. With all that said, HyDRA is just a first step. The current results are strong but situations remain where memory breaks: scenes with many (>2) moving objects or very long occlusions still degrade performance. Extending the approach to handle multiple simultaneous occlusions robustly is an open challenge. Also, HM-World itself is synthetic; a next phase would be to close the sim-to-real gap, either by fine-tuning on real video or by creating more photorealistic benchmarks. Finally, the computational cost of retrieval and tokenization is nontrivial; future work could streamline this or combine it with object-centric representations. Despite these caveats, the core lesson is clear: dynamic world modeling requires dynamic memory. HyDRA’s tactics – a split memory for static versus dynamic, and an explicit retrieval step – are a substantial advance toward that goal. As the authors put it metaphorically, HyDRA makes the model an archivist for the scenery and a security guard for the actors. In more grounded terms, it ensures our video “magicians” never let the rabbit disappear without a mental note of its motion. Summary: Chen et al. unveil a powerful idea (Hybrid Memory) and backing dataset (HM-World) to press video world models beyond static assumptions. Their HyDRA architecture shows that with the right memory mechanisms – tokenized history and smart retrieval – a model can preserve object identity and momentum through long occlusions. For practitioners in robotics and embodied AI, this suggests that investing in memory design is crucial for any application requiring consistent physical reasoning in video predictions. The code and models are publicly released, so one can start experimenting with hybrid memory directly in their own simulation or planning pipelines. The real world is rarely static, and now our world models are one step closer to catching up. References:. Chen et al. present the Hybrid Memory paradigm and HyDRA model, together with the HM-World dataset. Context-as-Memory and Memory-V2V are examples of prior work using explicit memory in video models. Their results show large gains in PSNR/SSIM and a new “Dynamic Subject Consistency” metric when using tokenized spatiotemporal retrieval (as described in) during occlusions. These findings point the way to more physically-plausible, long-horizon world simulators for robots and agents.