Latest / Embodied AI 101 / # WildWorld: Dynamic World Modeling with Actions and Explicit State
Transcript
- WildWorld: Dynamic World Modeling with Actions and Explicit State. Imagine learning to predict a game world like Monster Hunter: Wilds. Every shot fired not only flashes a muzzle crack on-screen, but also secretly deducts ammo from your inventory. An agent that only sees pixels will miss this hidden decrement of “ammunition” state, even though it’s vital for planning future actions. This disconnect – where actions induce latent state changes that aren’t visible – is a fundamental challenge for video-based world models. In most video datasets, an action is just a visual pattern (e.g. a figure slamming a sword) without any explicit bookkeeping of what that action did behind the scenes. As the authors observe, “existing datasets typically lack diverse and semantically meaningful action spaces, and actions are directly tied to visual observations rather than mediated by underlying states”. In such settings, actions become entangled with pixels and a model can easily forget critical state transitions over time. The WildWorld project tackles this issue head-on by providing a huge, richly annotated video dataset specifically designed for action-conditioned world modeling. WildWorld is automatically collected from the AAA game Monster Hunter: Wilds, a photorealistic open-world action RPG. It spans over 108 million game frames and covers 450+ unique actions, ranging from movements and attacks to skill activations. Crucially, each frame is augmented with explicit state and action labels – think player health, ammo count, skeleton poses, camera parameters, animation IDs, etc. – allowing models to observe not just the pixels but the underlying state of the game world. In short, WildWorld is a mammoth dataset of synchronized RGB video, depth maps, camera poses, character skeletons, and fine-grained game-state variables, all neatly time-aligned. In this write-up we unpack WildWorld: why it was created, how it’s constructed, and what early experiments on it reveal. We discuss the WildBench evaluation suite built atop the dataset and how state-conditioned video generators fare on these tasks. We also draw connections to cross-embodiment learning in robotics, where such state-aware models could enable transfer across different bodies. The upshot is that WildWorld offers a new platform to train and test dynamic, action-conditioned world models that think beyond pixels – a step toward agents that truly understand hidden states driving the visual world. Why Explicit State Matters. In classical control and modern RL, the world is often modeled by hidden state variables evolving under actions. A dynamical view sees the environment as a set of latent factors (player stats, enemy health, ammo count, etc.) whose transitions are driven by actions. Raw video frames provide only partial glimpses into that state. For example, firing a gun may leave a flash on-screen, but the critical effect – decrementing the ammunition count – is not directly visible in the RGB. If a model only tracks pixels, it might recall the flash but forget that ammo was used. Over many steps this leads to drifting predictions: the agent runs out of ammo prematurely and mispredicts future dynamics. Previous work in video prediction or generation often sidesteps this by focusing only on observable changes. Datasets like Something-Something or Kinetics label actions for recognition, but don’t provide the underlying state changes. Procedurally-generated video (e.g., Car-Racing or Maze environments) might have actions, but rarely annotate the world state. The result is that existing action-labeled video corpora “fail to capture critical internal variables like ammunition counts” and instead entangle every action with immediate visual effects. This makes learning long-horizon, stateful dynamics very hard. For instance, even if a model sees a character starting with full health, it cannot infer future stamina just from one shot of them running. But in a real system, that stamina is part of the world state and does change (perhaps slowly or after many steps) as actions accumulate. WildWorld explicitly bridges this gap: by providing the true with each frame, a model can learn that “shoot” always trips an internal clock or counter, even if the pixels of the gun stay on screen or fade out. In the WildWorld authors’ words, actions often “manifest through implicit state transitions rather than explicit visual changes”, and capturing these implicit changes in the data is key. Another motivation comes from robotic learning. In cross-embodiment robotics, one seeks policies or models that transfer across different bodies or environments. A core insight there is that having abstract, body-agnostic states and actions vastly improves generalization. For example, shooting a gun depletes bullets whether the shooter is a human or a robot arm – an abstract “ammunition” variable tracks this across embodiments. WildWorld’s approach of annotating a universal state (health, ammo, position, etc.) for each scenario dovetails with this idea: it gives algorithm designers a clean state-space that could, in principle, be re-used if the “player” embodiment changes (say from one character model to another, or even from virtual agent to a robot in a similar environment). As one emergent AI perspective elaborates, effective cross-embodiment learning essentially “construct[s] unified action and observation spaces” so that a single high-level policy or model can apply to many morphologies. By making state explicit, WildWorld takes a step toward that vision in the context of world-model learning. In short, WildWorld’s philosophy is that to train the next generation of world models or action-conditioned generators, we should explicitly include the hidden layers of the environment. That means not just giving the model video, but also subtitles telling it what each action did to the underlying state. With this in hand, models can learn to “imagine” sequences that stay consistent with the true physics of the game, instead of drifting away in pixel space. The WildWorld Dataset: Composition and Content. The WildWorld dataset collects massive, multi-modal data from Monster Hunter: Wilds, an ultra-realistic single-player RPG. In the dataset’s own description: it “contains over 108 million frames” with more than 450 distinct action classes. These actions range from walking and running (traversal) to attacking, shooting, dodging, and casting skills (combat) – covering both common locomotion and the long tail of an action RPG. The captured gameplay spans 29 different monster species, four customizable player characters, four weapon types, and five distinct outdoor stages. In practice this means WildWorld has data from day-night changes, varied terrain, many enemy designs and behaviors, and multiple player stances. It’s a richly diverse visual domain, yet all games are rendered at high fidelity (4K-quality textures, realistic lighting) because it is a AAA title. Each frame in WildWorld is accompanied by a suite of observations and state labels:. Observations (per-frame): The raw RGB image (game screen) and a lossless depth map (from the engine), plus the exact camera pose (intrinsic + extrinsic parameters) used that frame. In other words, we know exactly where the virtual camera was in 3D, both its position and orientation. This comes in handy for tasks like reconstructing camera motions or ensuring consistency of perspective in generation. State and Action Labels: Crucially, WildWorld logs a rich state for the player character and environment at every frame. This includes the 3D skeleton keypoints of the player character, their absolute location and rotation in the world, velocity, current animation ID (like “swing axe” or “draw bow”), health/stamina bars, remaining ammunition, and other gameplay attributes. If the player is shooting a gun, the ammo count is explicitly recorded changing. If a monster roars or runs, its animation ID and health are written down. All actions taken are labeled at the fine-grained level too. In this way, the game state is fully observed by us, even if some aspects (ammo, stun meters, etc.) are not directly visible. Essentially, WildWorld provides “explicit ground truth” for both the actions (what move was executed) and the game state variables. Hierarchical Captions: Beyond raw data, WildWorld adds natural-language annotations. Each short clip (a contiguous sequence of frames) is given both action-level captions and a sample-level summary. The process involves segmenting each sample into action segments (e.g. “monster swings tail”, “player dodges left”), sampling keyframes at 1 FPS, and then using a vision-language model to generate descriptive text grounded in the state. There are 119 annotation columns per frame in total, and these captions include references to actions and state (e.g. “hunter draws bow, arrow count decreases by one”). The bottom line: you can query the dataset with English descriptions or parse them as supervision in multi-modal models. Altogether, this means WildWorld is orders of magnitude larger and more richly annotated than any prior video dataset for world modeling. For context, most action video datasets have tens of thousands of clips, or a few million frames, and only high-level labels. WildWorld’s 108 million frames is on par with something like ImageNet in sheer scale, but every “image” is a frame in an interactive scene, and it comes with game-engine bookkeeping. Because of this comprehensiveness, the authors envision a spectrum of uses: any model that needs to predict how actions change the world over time can be trained or tested on WildWorld, with the possibility to explicitly check its internal “beliefs” against ground truth. Data Collection and Curation Pipeline. Producing 108 million coherent frames in a consistent format is no small feat. The WildWorld team built a fully automated capture system to run the game, record streams, and filter data. In brief:. Automated Gameplay: The researchers scripted in-game simulations by programmatically navigating menus and using a rule-based AI companion system. In practice, a custom bot would launch quests, spawn different monsters, and have the player character and an AI ally run through combat and traversal scenarios. This avoids any human-in-loop. By covering a variety of behaviors (e.g. hunting monster A with weapon X in forest stage, then monster B with weapon Y in winter stage), the pipeline achieves the broad coverage of enemies, weapons, and environments. Synchronized Multi-Stream Recording: While the game runs, the system captured multiple data streams simultaneously. Using tools like OBS Studio paired with a ReShade plugin, the pipeline recorded the main RGB video output and the depth-buffer in lockstep. Crucially, each frame was tagged with a precise timestamp so that later we could align the RGB, depth, camera pose, and state data with perfect 1-to-1 correspondence. This ensures that a model learning from WildWorld never has to guess if two modalities are off by a frame. Quality Filtering: Being automatic doesn’t mean infallible, so the team applied several filters to weed out bad data. Clips shorter than 81 frames (roughly 5 seconds) were dropped to focus on meaningful sequences. Any segment with a temporal gap longer than 50ms (perhaps a dropped frame) or with extremely over/under-exposed lighting was also culled. They even detect camera occlusion events (e.g. when the in-game camera rig snaps to third-person and occludes the view) by monitoring the spring-arm contraction, and remove any clip where overlap of characters is excessive (>30% of screen). Finally, the developers disabled all HUD and UI overlays during capture, so that every frame is a clean view of the game world. The end result: virtually every saved clip is free of glitches, switch-cuts, or UI distractions. Hierarchical Caption Generation: After filtering, the remaining clips were annotated with captions (as mentioned). The pipeline splits each clip into semantic action chunks, picks frames at 1 FPS, and feeds them to a vision-language model (likely a transformer-based captioner). The model is prompted to describe what’s happening, explicitly leveraging the known actions and state. For example, it might produce “Hunter swings sword at Rathalos, Rathalos’s health bar drops slightly”, using the true state variables to enrich the text. These captions are not used to filter the data – they are optional annotations meant to make the dataset multi-modal and easier to use with language-driven models. This engineering effort means WildWorld is neatly packaged: each sample is a timeseries of frames, accompanied by depth, poses, skeletons, and a table of state variables, all synchronized and cleaned. The full dataset is so large it’s distributed via a custom Hugging Face datapack (currently listed as coming soon); the code to drive the capture and processing is open-sourced on GitHub. In summary, the authors have gone to great lengths to ensure that WildWorld is high-quality and richly labeled – as if every video frame came with a backstage pass to all internal variables. WildBench: A Benchmark for Interactive World Models. Collecting data is half the battle. The WildWorld authors also constructed WildBench, an evaluation suite to test interactive video models on this data. WildBench consists of 200 curated test samples chosen to be representative (including one-on-one duels and two-player hunts), with ground-truth annotations held out for evaluation. Each sample provides an initial few frames and a sequence of future “action commands” or state inputs, and the model has to generate the subsequent video frames. The key is that the model’s output is judged not just by visual fidelity but by how well it follows the intended actions and preserves true state. WildBench measurements fall into four categories:. Video Quality (VBench): Standard video metrics such as Motion Smoothness, Dynamic Degree, Aesthetic Quality, and Image Quality are computed on the generated sequences. These measure how smoothly and cleanly the video plays out, judging blurriness or frame artifacts. The authors note, however, that in this interactive setting these vanilla metrics quickly saturate – most models get them near-perfect – and thus fail to distinguish models that really understand the dynamics. That’s a key insight: good-looking video is necessary but not sufficient for world modeling. Camera Control: They evaluate how accurately generated camera motions match the ground truth. Specifically, using Structure-from-Motion (SfM) techniques they compute the Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) between predicted camera positions/angles and the true trajectory. This checks if the model placed the virtual camera in the right place at each timestep, a proxy for appreciating player movements. Action Following: This metric tests whether the generated video correctly reflects the requested actions. For instance, if the input says “player uses heavy attack then dodges”, does the video show those sub-actions in the right order? The authors propose a quantitative score for how well the generated frames align with the ground truth action labels over time, and they also validate it aligns with human judgment (about 85% agreement). In practice, a higher Action-Following score means the model is reliably executing the scripted moves. State Alignment: Finally, since WildWorld knows the true world state sequence, we can check how closely the model’s imagined state stays on track. They do this via keypoint tracking and other means: for example, if the true data says the monster lost 30% health by frame 50, the generated video should reflect a similar monster reaction. The state alignment metric quantitatively measures the divergence between the ground-truth state variables and those inferred from the video. This directly gauges long-horizon consistency. WildBench thus puts interactive video models through a suite of tests that require understanding the action semantics, not just generating pretty frames. It is one of the first benchmarks to explicitly quantify action fidelity and consistent state propagation in generated video. The creation of WildBench underscores that the authors don’t just dump data – they also want to drive development of models that can meaningfully use the state info. Models and Methods Evaluated. The WildWorld paper evaluates several approaches for generating videos conditioned on actions (and other signals). All are built on modern diffusion-based video models (the paper uses a variant called Wan2.2-Fun, a 5-billion parameter Transformer-based video diffusion model). The key variable is what conditions the model receives:. Baseline (No Control): This is a vanilla model (Wan2.2-TI2V-5B) that takes only the initial frame (or a short context) and the textual prompt, but no explicit action or state signals. It tries to hallucinate a plausible continuation. As expected, it can produce visuals that “look like” Monster Hunter, but it has no way of knowing, say, that the player should shoot at frame 10 or dodge at frame 20. It’s our reference point for comparing how much explicit control helps. CamCtrl (Camera-Conditioned): In this variant, the model is feed the ground-truth camera poses for each frame. That is, at each timestep the model knows exactly where the virtual camera should be. To implement this, the team fine-tunes the Wan2.2 model with an added camera-conditioning control layer – specifically, they use a “Wan2.2-Fun-5B-Control-Camera” model that injects camera pose embeddings into the diffusion network. Conceptually, this should help the model get movements of the viewpoint correct (e.g. turning to the left when the camera does) even if it still doesn’t know anything about the player’s intended actions. CamCtrl thus isolates the effect of geometric control. SkelCtrl (Skeleton-Conditioned): Here the model receives a rendering of the 3D skeleton of the player character (projected to screen coordinates using the true camera pose). The skeleton is drawn as a colored stick figure moving on screen, and the video model sees this alongside the original image input. The idea is that if the model can see the exact pose of the character each frame, it might better predict how to render the next frame. Again, Wan2.2 is fine-tuned with this new condition (SkelCtrl). This provides body motion information but in a way that is still relatively low-bandwidth compared to full state. StateCtrl (State-Conditioned): This is the most complete conditioning. The StateCtrl model is injected with structured game state variables in addition to having access to camera and skeleton. The authors organize state into discrete categories (monster type, weapon type) and continuous values (player coordinates, health/time). These are embedded via learned MLPs and a Transformer that fuses all entity states into one “state embedding” per frame. This embedding is then injected into the diffusion network’s intermediate layers, so the model has direct knowledge of every aspect of the world state at each step. Importantly, the StateCtrl training also includes a decoder and predictor: a small network tries to recover the state from the embedding, and another predicts the next frame’s state. Losses on those decoders enforce that the embedding truly captures the state changes. Thus, StateCtrl aims to bake the full underlying dynamics into the generation process. StateCtrl-AR (Autoregressive): Finally, they experiment with an autoregressive variant. In this version, the model is only given the initial state in the first frame, and then it must predict both future frames and future states step by step. After generating each frame, it feeds its own predicted state back as the input for the next frame. This simulates a true “world model” that keeps track of state internally. In practice, this proved harder: small errors in state prediction accumulate over time, but it demonstrates how one could potentially use the dataset for fully predictive modeling. All these models are trained on short clips of length 81 frames (about 5 seconds at 16 FPS) at a resolution of 544×960 pixels. Training uses standard diffusion settings (Adam optimizer, 250k gradient steps, etc.). Once trained, each model can generate 81-frame videos conditioned on whatever signals it received during training. Experimental Findings on WildBench. When evaluated on WildBench, several clear patterns emerged:. Explicit Control Improves Action Fidelity. Models that received camera and/or skeleton inputs consistently did a better job at following the specified actions than the uncontrolled baseline. In other words, if we tell a model “the camera should pan right” or “the character skeleton is grabbing a sword,” it makes the generated video more responsive to those commands. The paper notes that diverse approaches on WildWorld outperform baselines on interaction-related metrics, showing that having these cues helps learning how to actually enact moves. For instance, SkelCtrl and StateCtrl achieved noticeably higher Action Following scores (their videos contained the requested sub-actions in the correct order) than the plain baseline. This is not surprising: providing a skeleton or state narrows the model’s possibilities, so it can more confidently generate the intended moves. State-Conditioning Yields Consistency. By explicitly feeding the full world state to StateCtrl, the model gained an advantage in maintaining long-term consistency. StateCtrl’s embeddings let it know exactly, say, the monster’s remaining health at each frame. The metric State Alignment confirmed that StateCtrl’s generated sequences stayed truer to the ground truth state evolution than others. This highlights the benefit of structured state: it bridges the “hidden ammo” or “hidden health” gap. Interestingly, because the model was also trained to predict future states, StateCtrl could often correct small visual aberrations by aligning them to plausible states. Standard Quality Metrics Saturate Quickly. Perhaps counterintuitively, most models achieved near-ceiling scores on basic video-quality measures like VBench scores. The authors point out that these metrics “saturate and fail to capture nuanced differences in dynamic motion”. In practice, even the baseline model (with no controls) could generate fairly smooth, sharp video if not forced to do specific actions. Thus, simply looking at image fidelity or blur doesn’t tell you if the model truly understood the task. This underlines why the new metrics (Action Following, State Alignment) are necessary: they reveal trade-offs that VBench misses. Trade-off Between Fidelity and Control. Adding explicit signals helps with dynamics but can hurt image sharpness. Qualitatively, models like CamCtrl/SkelCtrl sometimes produced frames that were less aesthetically pleasing (e.g. more blur or distortion) than the unconstrained baseline. The paper notes a clear trade-off: when models use strong control inputs, they tend to emphasize correct movement over visual polish. For example, a model might sacrifice some color consistency to ensure the sword swing is timely. This is a known phenomenon: enforcing exact alignment can sometimes bias the generation away from the best raw image. However, for world modeling, that trade-off is often worth it since the priority is “does the scene evolve correctly?” rather than “does it look most beautiful?”. Metrics Validation – 85% Human Agreement. To ensure the action and state metrics are meaningful, the authors had humans judge a subset of generated sequences. They report about 85% agreement between humans and the proposed Action Following score. Similarly, the State Alignment score was validated as an effective measure of consistency. This extra validation gives confidence that the metrics are capturing what a person would check: are the right actions happening, and are the states plausible?. Autoregressive Evaluation. The StateCtrl-AR model (predicting state forward) showed the expected “promise but caveat”: it could in principle simulate arbitrary long rollouts, but in practice small errors snowballed. By frame 50 or so, the AR model’s video often drifted off from the true action sequence (if we had provided it) because its internal state had gradually diverged. This manifests as a lower action-following score for StateCtrl-AR vs. the fully supervised StateCtrl. It’s an important demonstration: building a fully predictive model that maintains state is possible, but non-trivial. The fact that the metric detected this drop-out shows the benefit of having the ground-truth state for evaluation. In summary, WildBench experiments validate the dataset’s utility. Models trained with access to the extra state signals indeed learn more accurate and controllable dynamics. At the same time, the findings emphasize that achieving photorealistic coherence and perfect adherence to actions simultaneously is still an open challenge. The authors conclude that we need state-aware video generation – exactly what WildWorld was designed to encourage – to make further progress. Implications and Future Directions. The WildWorld dataset opens up several research avenues. First, it provides a thorough testbed for video world models with actions. The inclusion of multiple player characters and 450+ actions already introduces an element of cross-embodiment: a single model sees different heights, body plans, and weapons, but all share an abstract state space (health, ammo, etc.). This reflects one pillar of cross-embodiment learning: that diverse morphologies can still map onto shared state abstractions. In robotics, this hints at training a universal policy or world model: for instance, both a humanoid and a robotic arm could learn from the same “hit enemy reduces health” transition rule if their data looks like WildWorld (with different skeleton skews but consistent effect on health). Second, WildWorld can be seen as a bridge between game AI and real robotics. While it’s a synthetic dataset, the game’s AAA graphics make it a reasonable proxy for a visually complex world. A robot operating in a physically similar environment (forests, caves) could conceivably reuse models pre-trained on WildWorld imagery, then fine-tune on a small real sample. More importantly, the methodology of WildWorld – using a simulator to generate labeled state trajectories – is exactly how many robotics datasets are made (e.g., manipulating simulated objects with known physics). By publishing this massive example, the authors encourage others to treat rich game engines as data sources (as already happens in driving with CARLA, but now extended to arbitrary collection tasks). Furthermore, the explicit annotations support research in vision-language-action integration. The provided captions mean you could train a model that takes a textual instruction plus initial scene and generates the action sequence or video. This ties into recent interest in multi-modal planning (e.g. generating video from commands). WildWorld could serve as a playground for text-driven game agents: given “monster A is here and the hunter has a bow”, what sequence of visual outputs does a model predict?. Of course, WildWorld has limitations. It is still limited to a fantasy game domain – it doesn’t contain kitchen scenes, street driving, or real-world variations. The specific mechanics (monster AI, weapon physics) may not directly transfer to other domains. Also, while the data is huge, it’s not annotated for everything (no audio; only one human avatar). But as an action-conditioned generative benchmark, it is unprecedented in scale. The paper and project emphasize that this is a generative ARPG environment – literally a video game turned into a dataset. Researchers interested in even broader applicability might consider similar pipelines for other games or simulators (e.g., sports games, driving sims, or robotics sims) to cover new domains. Finally, the WildWorld work is a call-to-arms for the community to think more holistically about world modeling. It's not enough to have smart vision backbones or fancy GANs; model architectures and training must explicitly incorporate the structure of the environment. WildWorld provides the training material to do exactly that. It also highlights evaluation: without metrics like Action Following and State Alignment, one could miss the boat on actual performance. Future work will likely build on these ideas – perhaps developing new architectures (like graph neural world models) that directly consume the state data, or exploring unsupervised ways to infer states from raw video and then validating against WildWorld’s ground truth. In conclusion, WildWorld is an ambitious step toward richer world models. It trades on a commodity (game simulation) to give AI researchers a resource of indulgent size and detail. The hope is that this dataset will enable models that can think not just in pixels, but in proper states, leading to more reliable long-term predictions. Who knows, maybe next we’ll see an embodied AI that learned to hunt monsters with the same clarity as this game’s code – all thanks to the explicit states provided by WildWorld.