Latest / Embodied AI 101 / Stop Making the Video Model Remember the World
Transcript
- Stop Making the Video Model Remember the World. Programmable World Model • Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang, Ruihan Yu, Muyao Niu, Siqi Yang, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, and Zhixiang Wang • Alaya Lab, National Taiwan University, and National Yang Ming Chiao Tung University • arXiv preprint • 2026. A plausible frame is not a persistent world. Imagine an interactive video model renders an enemy being shot, falling to the ground, and disappearing behind the camera. The player turns away, walks around for ten seconds, and then returns. The enemy is standing again—or has vanished entirely. The individual frames may look convincing. The event itself may have been visually dramatic. But the system failed at something more fundamental: remembering that a particular entity underwent an irreversible state transition. That gap between plausible observations and persistent facts is the target of Alaya Lab’s Programmable World Model, released as an arXiv preprint on September 9, 2026. Its central claim is simple: a generative video model should not simultaneously be the database, rules engine, geometric simulator, memory system, and renderer for an interactive world. Instead, world state should live in an explicit engine, state transitions should follow executable rules, and the video model should primarily render the consequences. This sounds almost obvious to anyone who has built a game engine, a POMDP stack, or a model-based robotics system. An observation is not the environment state. A camera image is a partial, view-dependent projection of something larger. Yet much of the recent interactive-video literature effectively treats visual history as the world’s recurrent state. If an entity is not in the context window, not visible in the current image, or not recoverable from a compressed visual memory, its continued existence becomes a matter of generative plausibility rather than an enforced fact. Programmable World Model, or PWM, puts an authoritative symbolic and geometric layer back into the loop. A coding agent writes the world specification. A lightweight engine executes it. A deterministic compiler turns the resulting state into spatial conditioning maps. A pretrained video model paints the final observation. The state executor is the law, the compiler is the camera-aware interface, and the video model is the renderer. The result is best understood not as a purely learned dynamics model, but as a hybrid generative simulator: explicit state and programmed transitions underneath, neural visual realization on top. The architecture as a compiler stack. At every interaction step, PWM performs three conceptually separate operations. First, the engine receives the current canonical state and a player action. It validates the action, applies the relevant rules, resolves triggered events, and produces the next canonical state. Second, a state compiler projects the visually relevant portions of that state through the requested camera trajectory. Third, the generative renderer turns those projected controls, the camera path, and visual history into the next video chunk. The canonical state is divided into four broad components:. persistent entities and their three-dimensional poses;. semantic and functional attributes;. relations between entities;. executable rules governing transitions. A character might therefore have a stable identity, a three-dimensional oriented box, health, faction membership, inventory, and a status such as active or down. Relations can encode properties such as hostility or ownership. Rules determine which actions are legal and what follows from them—for example, subtracting damage after a valid hit, changing status when health reaches zero, or marking an objective complete after all hostile entities are defeated. Initialization begins with a reference image. An off-the-shelf 3D detector estimates the visible entities and their geometric layout, yielding initial oriented bounding boxes and identifiers. A vision-language-based coding agent combines that recovered layout with the user’s natural-language description and writes an engine-readable world program. The program supplies the attributes, relations, supported actions, event triggers, objectives, and transition rules that cannot be inferred from geometry alone. The important architectural choice is that the engine—not the video model—owns the resulting state. Health and inventory do not have to remain recoverable from rendered pixels. An entity does not stop existing because it leaves the camera frustum. A death event does not need to be rediscovered from visual history every time the camera returns. In principle, the complete state can be inspected, edited, replayed, tested against invariants, or synchronized among multiple clients. That is a meaningful change in failure mode. A learned visual model can still render the state incorrectly, but it can no longer silently rewrite the canonical facts. The engine may say that entity 17 is down even if the renderer accidentally depicts it standing. That disagreement becomes detectable rather than being absorbed into an opaque latent history. What “programmable” actually means. The coding agent is responsible for translating open-ended language into a world program grounded to persistent entity identifiers. Users can subsequently instruct the agent to revise the behavior of the world, changing rules and interactions without retraining the renderer. The paper’s examples revolve around combat logic, but the state format can in principle express doors, vehicles, objectives, ownership, inventories, event schedules, or other discrete mechanics. This gives PWM a different notion of generalization from that of a learned transition model. The system does not have to infer the rule “a vehicle enters after the alarm is triggered” from training trajectories. The agent can write that rule directly. Once written correctly, the engine will execute it consistently every time the trigger occurs. But there is an important caveat: determinism is not the same as correctness. An executable but wrong rule is perfectly consistent. The paper does not specify the programming language or domain-specific language used by the coding agent, the model powering the agent, its sandboxing strategy, its error-recovery loop, or any static and dynamic checks applied to the generated program. It also does not quantitatively evaluate natural-language-to-program accuracy. CombatStateBench evaluates whether the renderer follows an already constructed engine state; it does not test whether the coding agent correctly understood a user request and implemented the intended mechanics. For a production system, this part of the stack would need far more machinery: typed schemas, constrained code generation, deterministic replay, unit tests derived from language, invariant checking, resource limits, and perhaps formal verification for critical rules. PWM establishes that executable state is a useful interface to generative rendering. It does not yet establish that open-ended agent-written world programs are reliably correct. Why the world is made of boxes. The paper’s main representational contribution is the state-augmented 3D oriented bounding box, or OBB. This is the intermediate representation connecting the engine to the video model. A text description is easy to produce but spatially weak. Saying “the red car is behind the player” does not precisely determine its world-space position, scale, or orientation. A 2D box or segmentation mask gives stronger image-space control, but it is tied to one viewpoint. Move the camera, and the control itself must somehow be regenerated. At the other extreme, PWM could maintain complete meshes, articulated skeletons, materials, surface geometry, animation tracks, and G-buffers. That would provide strong control, but it would also recreate much of a conventional graphics and simulation stack. It would require much richer training supervision and force the explicit engine to synthesize detailed motion at inference time. The authors identify a particularly important training–inference asymmetry. During dataset construction, the motion has already happened: one observes a video and extracts its geometry. At inference time, the direction is reversed. The system receives a high-level event—say, a character falling—and must first construct an appropriate structural trajectory before the video model can render it. A highly detailed representation therefore makes inference surprisingly difficult. Extracting a skeleton trajectory from an existing video is not equivalent to generating a physically and visually appropriate skeleton trajectory from the symbolic state transition. As the explicit representation becomes richer, PWM would need its own animation model, motion planner, or physics simulator to fill in those details. An OBB is the proposed compromise. It records an entity’s center, extent, and orientation in shared world coordinates. It can be moved, rotated, resized, projected into arbitrary cameras, and associated with identity, category, state, and other attributes. A falling character can be represented by a box changing orientation and spatial occupancy, while the video prior invents the limb motion, clothing dynamics, body articulation, lighting, and contact details. This is a compelling intermediate representation for coarse interaction. It is also the architecture’s clearest boundary. Boxes are not contact geometry. They do not encode an object’s topology, articulated joints, grasp affordances, support surface, deformability, or mass distribution. PWM deliberately assigns those details to the renderer, which is reasonable when the objective is a visually convincing game event. It is much less sufficient when a robot must predict whether a gripper will clear a handle by five millimeters. Compiling canonical state into video controls. Once the engine has produced the next state, the compiler projects each relevant OBB through the target camera. Depth-aware rasterization resolves overlapping entities, and the resulting projected regions are populated with three kinds of features: identity, semantics, and motion direction. The identity map assigns each persistent entity a stable slot from a bank of learnable embeddings. This gives the renderer a spatial key saying that the object occupying this region is the same instance that appeared previously, even after occlusion or a temporary departure from the view. The semantic map embeds the entity’s category with a pretrained text encoder and rasterizes that embedding over the visible box projection. Two characters can share the same semantic category while remaining distinct through their identity slots. Semantic conditioning is particularly important for entities that were not visible in the initial reference image and therefore have no direct appearance evidence. The direction map represents the entity’s own movement in one of seven quantized states: static, left, right, up, down, forward, or backward. Crucially, direction is computed from world-space velocity and then transformed into the camera coordinate system. It is not inferred from image displacement. A stationary object therefore remains marked as stationary even when a panning camera causes it to sweep across the frame. Camera motion travels through a separate conditioning pathway inherited from the video backbone. This separation helps distinguish “the camera moved past the car” from “the car drove past the camera,” an ambiguity that is difficult to resolve from optical motion alone. One subtle point is that the methods section defines identity, semantic, and direction maps, but not a dedicated raster channel for every symbolic variable. Health, inventory, faction, and task progress remain in the engine. Their observable consequences must be compiled into geometry, motion, or renderer-facing semantics. In the death examples, for instance, the transition is communicated largely through the changed orientation and occupancy of the fallen entity’s OBB. A health value of 37 does not automatically become visible unless the world program maps it to an observable change. The compiler is deterministic, but the final rendering is not. It guarantees where controls are placed and which state produced them. It cannot guarantee that the generative model will perfectly obey those controls. That residual conformance problem is exactly what the benchmark attempts to measure. Two kinds of memory: facts and appearance. PWM builds its renderer on LingBot-World-v1, retaining the pretrained camera-control pathway and freezing the main video backbone while training a Structured Spatial ControlNet. The compiled identity, semantic, and direction maps are encoded at the video latent’s spatial resolution. Control features are then injected into corresponding blocks of the main network, allowing the pretrained prior to fill in everything the boxes leave unspecified. Long-horizon generation proceeds autoregressively in chunks. Denoising can be bidirectional within a chunk, but information only moves forward across chunk boundaries. Later chunks receive a bounded temporal history organized at multiple resolutions: recent observations preserve detail, while older observations are compressed more aggressively. The initial image remains available as a persistent appearance anchor. During training, past latents are deliberately corrupted or dropped so that the model learns to survive imperfect self-generated history. PWM also adopts the geometry-aligned spatial memory used by AlayaWorld. Previously completed RGB frames are paired with estimated depth and camera pose, lifted into a world-space cache, and reprojected into upcoming target views. Temporal memory says what happened recently; spatial memory provides evidence about what a previously observed location looked like. This creates a useful two-memory architecture. The canonical engine state remembers facts: identity, position, health, ownership, status, rules, and event history. The visual memory remembers appearance: textures, background structure, lighting, and the observed look of an entity from earlier viewpoints. Those should not be conflated. A visual cache is useful for recreating a wall when the camera returns, but it should not be responsible for deciding whether a door is locked. Conversely, a symbolic state can say that the door remains locked without containing enough information to reproduce its scratches, reflections, and material appearance. PWM assigns each responsibility to a different subsystem. The project page demonstrates an 897-frame, roughly 30-second rollout with many NPCs entering and undergoing events. It also shows large camera rotations that reveal entities maintained behind the initial view. These are qualitative demonstrations rather than long-horizon state-drift measurements, but they illustrate the intended division between global state and local visual context. Building supervision from ordinary gameplay video. Training the structured control branch requires camera trajectories, metric depth, persistent tracks, semantic categories, and three-dimensional object trajectories. Those annotations rarely coexist in ordinary video, so the authors construct them with an automated data engine. ViPE estimates camera intrinsics, poses, and metric depth. Qwen3-VL describes the video and proposes countable semantic categories. SAM3 segments and tracks instances over time. WildDet3D uses the RGB frame, tracked 2D box, depth, and camera intrinsics to estimate a 3D OBB for each visible object. The boxes are transformed into a shared world coordinate system, and consecutive centers yield object velocities from which the seven motion classes are derived. Finally, the system projects the boxes back into each camera to create the renderer’s training controls. The reported training sources are HUD-free gameplay videos from Cyberpunk 2077, Forza Horizon 6, and Grand Theft Auto V, spanning first- and third-person viewpoints. The paper does not report the number of videos or frames, the balance among the games, or the distribution of entity categories and events. This pipeline is scalable in the sense that it avoids manual annotation, but its errors are compositional. Camera error affects world coordinates. Tracking errors corrupt identity slots. Depth error affects box scale and projection. Poor semantic discovery changes segmentation prompts. The choice of a coarse representation probably makes the renderer more tolerant to this noise, but it does not remove it. It also only reduces the training–inference mismatch. Training controls are extracted from motions that actually occurred in a game engine. Inference controls are constructed from agent-written rules and box trajectories. Those two distributions can still differ substantially, especially for unusual interactions. What CombatStateBench establishes. CombatStateBench contains 50 controlled clips. For each sequence, the data engine reconstructs an initial layout, and an AI agent evolves a short box-based combat scenario. The benchmark includes camera motion, entity motion, off-screen entities, and persistent death transitions. An automatic verifier checks properties including initial reprojection, depth consistency, box geometry, ground contact, temporal continuity, prescribed movement, and the persistence of transitions before accepting a sequence. PWM is compared with LingBot-World-V2 and YUME. Because those systems lack an external instance-level state interface, the authors communicate events to them by changing the text prompt—for example, adding a description that a particular NPC has been killed. All systems receive the same initial observation and benchmark transition schedule, but they receive very different control bandwidth: PWM gets projected instance-level geometry, while the baselines get language through their native interfaces. Evaluation uses Qwen3.6-27B as a vision-language judge. It sees only the generated RGB frames, with no ground-truth boxes or identities. The headline results are:. | Method | Imaging quality | Subject consistency | Background consistency | Temporal stability | Count accuracy | State accuracy | |---|---:|---:|---:|---:|---:|---:| | LingBot-World-V2 | 67.46 | 81.87 | 91.89 | 96.85 | 40.75 | 8.00 | | YUME | 64.10 | 92.35 | 93.63 | 98.76 | 32.00 | 58.00 | | Programmable World Model | 67.62 | 94.74 | 96.98 | 99.00 | 94.00 | 98.00 |. Count accuracy is measured on 400 sampled frames—eight per clip—and requires the VLM’s count of visibly alive characters to match the engine’s count. State accuracy covers 50 death events. Three post-transition frames are sampled for each event, and the event counts as successful if at least one frame contains a visually dead character. The gains are large. PWM improves count accuracy by 53.25 percentage points over LingBot-World-V2 and 62 points over YUME. It improves state accuracy by 90 points and 40 points respectively. Just as importantly, adding structured controls does not appear to damage conventional video quality: PWM leads all three systems on the four reported VBench dimensions, although its imaging-quality margin over LingBot is only 0.16 points. The conservative interpretation is that a renderer follows explicit spatial state much more reliably than it follows a textual request to produce an event. That is useful evidence, and the qualitative comparisons show the characteristic baseline failures: entities disappear, surviving characters are lost, or supposedly dead characters continue moving. The benchmark does not yet establish broader simulation correctness. State accuracy does not require the correct entity to die, nor the death to occur at the correct location. Count accuracy does not test persistent identity. Neither metric checks inventories, ownership, trigger conditions, collision outcomes, program correctness, or whether an off-screen entity returns with its previous appearance. There is also no reported ablation isolating identity, semantic, and direction controls; no same-backbone comparison with and without the explicit engine; and no comparison against a baseline receiving oracle OBBs. The experiment deliberately compares a structured interface against prompt switching, so it demonstrates the value of that interface more directly than it demonstrates the superiority of any particular video architecture. From visual memory to authoritative state. PWM belongs to a rapidly developing sequence of approaches that progressively externalize world state. YUME and LingBot-World emphasize interactive, autoregressive video generation, camera control, rich actions, and long visual horizons. Their recurrent carrier remains predominantly visual context and memory. AlayaWorld improves that carrier with compressed temporal history and a geometry-aligned cache, enabling a model to revisit previously observed regions more consistently. But those memories are optimized to preserve observations, not to serve as an executable ledger of world facts. StatePlay moves internal state into the learned model. It jointly predicts frames and variables such as health, skill meters, and timers, allowing predicted state to guide frame generation. That improves mechanics consistency, but state evolution is still learned and recurrent. Prediction errors can therefore compound. Its evaluation is also centered on one fighting game and a synchronized state–frame–action dataset. MASS goes further by making a typed global state the authoritative synchronization object for a multiplayer world. A learned Logic Engine advances that state from joint actions, and a Rendering Engine generates any requested client view. MASS demonstrates impressive scaling—up to 1,024 players and 10,000 recurrent steps—but its transition model remains learned rather than rule-executed. PWM takes the final step in this progression: its authoritative state is advanced by executable programs. That trades learned open-ended dynamics for controllability and verifiability. The architecture is consequently close to a conventional networked game engine, except that an agent authors the logic and a video model replaces much of the asset, animation, and rasterization pipeline. That framing clarifies both its strength and its limitation. PWM is not trying to learn every law of its environment. It is trying to make a programmed environment look like an open-domain, generated world. What embodied-AI researchers should take from it. For roboticists, the most valuable contribution is not the combat demo. It is the insistence on separating task-relevant truth from sensor appearance. Many embodied world models are trained end to end to predict pixels or latent video from actions. That can be useful for representation learning and policy optimization, but it makes several semantically important questions difficult to answer. Does object A still exist behind the robot? Which container owns item B? Has a safety interlock been triggered? Was the drawer already opened earlier in the episode? A visual latent may contain enough evidence to answer these questions, but it is rarely inspectable or authoritative. A PWM-like architecture would allow planning to operate directly over explicit states. The system could cheaply branch thousands of symbolic or geometric futures, then invoke the expensive neural renderer only for trajectories where observation-level prediction matters—for example, when training a vision policy, estimating occlusion, or evaluating how a human operator would perceive a situation. The renderer would become an observation model rather than the sole dynamics model. A robotics-oriented implementation would probably need at least five extensions:. Belief state rather than point state. Single-image OBB detection should produce uncertainty over object identity, pose, dimensions, and existence. Treating an uncertain perception result as canonical truth merely converts perception error into deterministic simulation error. Hybrid transition dynamics. Discrete task rules can remain executable, while rigid-body contacts, articulated mechanisms, deformable objects, and human motion use physics models or learned probabilistic residuals. Known safety constraints should not be delegated to a video prior. A richer compiler. In addition to identity and semantic maps, the renderer could receive depth, surface normals, optical flow, articulated keypoints, contact regions, affordance fields, material classes, and expected force events. Different tasks could request different levels of structural detail. Multimodal rendering. An embodied observation model may need RGB, depth, event cameras, tactile signals, audio, or proprioceptive side effects—not just cinematic video. Continuous conformance checking. Generated observations should be parsed back into estimated state and compared with the canonical state. If the renderer depicts the wrong object, misses a contact, or changes a count, the system should reject or regenerate the chunk. This last point is crucial. Explicit state makes disagreement detectable, but only if someone checks. The current benchmark performs a coarse offline version of that check with a VLM judge. A deployed embodied simulator would need it continuously and at the instance level. PWM could be valuable for synthetic-data generation, particularly for long-horizon tasks involving rare events and persistent semantics. A developer could program that a tool is moved, a container is locked, a person leaves the room, or an alarm changes the allowed action set. The renderer could then create diverse appearances without requiring a handcrafted asset for every scene. It could also support multi-camera and multi-agent scenarios naturally. Every agent would query the same canonical state through a different sensor pose. This avoids maintaining multiple potentially inconsistent visual histories as independent versions of the world. But PWM as presented should not be mistaken for a robotics-grade predictive model. Its engine executes specified rules rather than discovering unknown physical dynamics. Its boxes do not resolve manipulation geometry. Its renderer is rewarded for visually realizing coarse events, not for metrically accurate contacts or calibrated sensor prediction. At present, it is better viewed as a template for combining symbolic simulation with generative observations than as a replacement for a physics engine or learned control model. Reproducibility and the experiments still needed. As of September 11, 2026, the public repository contains a short README and teaser assets, while inference code and pretrained weights remain unchecked items on the release roadmap. The technical report and project page are available, but the implementation needed to reproduce the reported system is not. The report also omits several practical details: training-set scale, renderer and control-branch parameter counts, optimization settings, hardware requirements, inference speed, action-to-frame latency, chunk length, the identity-bank capacity, and the exact coding-agent configuration. There is no reported FPS or latency figure for PWM itself, despite its presentation as a playable interactive system. The most informative next experiments would test:. natural-language program synthesis against executable unit tests;. instance-specific state conformance rather than global counts;. quantitative off-screen persistence over much longer rollouts;. ablations of identity, semantics, motion direction, temporal memory, and spatial memory;. scaling with entity count and complex relation graphs;. uncertain or contradictory initial observations;. interactions requiring collision, articulation, and continuous physics;. closed-loop agents acting on PWM’s rendered observations. Those experiments would separate three sources of error that the current evaluation largely combines: program error, state-transition error, and rendering error. PWM’s architecture makes that decomposition possible. Future benchmarks should take advantage of it. The bottom line. Programmable World Model’s strongest idea is also its simplest: do not ask a generative renderer to be the sole keeper of reality. Alaya Lab externalizes identities, attributes, relations, rules, and off-screen entities into an executable canonical state. It then uses state-augmented 3D boxes as a compact intermediate representation, compiling them into camera-aligned controls for a long-horizon video model. On a narrow combat benchmark, this produces dramatically better entity counts and death-state realization than asking existing video world models to follow changing prompts. The evidence is preliminary, the benchmark is permissive, and the implementation is not yet public. Nevertheless, the architectural direction is persuasive. For embodied AI, the enduring lesson is not that every world should be made of bounding boxes. It is that facts, dynamics, geometry, memory, and appearance are different computational responsibilities—and forcing one video model to perform all of them is neither necessary nor desirable.