Latest / Embodied AI 101 / In-Context World Modeling for Robotic Control

Transcript

In-Context World Modeling for Robotic Control · Embodied AI 101

• In-Context World Modeling for Robotic Control. Contemporary robotic policies that combine vision, language, and action (often called VLA models) have achieved remarkable capability in fixed environments – think a robot arm following a spoken or textual instruction to pick up an object in a simulator or a lab setting. However, these models are notoriously brittle when something as simple as the camera angle or robot configuration changes. In practice, every time you move the camera, replace a gripper, or tweak the robot’s kinematics, the policy can fail. Until now, the standard solution has been laborious: collect new data and fine-tune the model for each new setup. The new In-Context World Modeling (ICWM) approach (Wang et al., arXiv:2606.26025) proposes a very different solution. Rather than retraining, a single pre-trained policy calibrates itself on the fly by observing a few random interactions, all within the transformer’s context window. In other words, ICWM uses in-context adaptation to perform system identification – feeding the model snippets of how the world behaves so it can infer camera angles, robot morphologies, or other hidden variables without any parameter updates. This is reminiscent of how large language models do “in-context learning” by reading examples of tasks in the prompt, but here the example is not what to do but how the world works. As the authors describe it, ICWM “treats system identification as an in-context adaptation problem” rather than a parameter-update problem. By design, the robot first executes a short, task-agnostic probing sequence – essentially taking random actions and recording the resulting observations – and then immediately receives its actual instruction. The transformer policy consumes both the probing sequence and the instruction (plus the current observation) in one forward pass. The hidden activations induced by the probing sequence implicitly encode the underlying world dynamics. In effect, the policy “internally models” the new camera view or kinematic setup, allowing it to succeed on the real task without any additional fine-tuning. Broadly speaking, ICWM answers the need for zero-shot domain generalization in robotics. Standard VLA models typically condition only on the instantaneous observation and language goal, implicitly assuming a fixed camera and robot they saw during training. As Wang et al. note, this means these policies “implicitly assume a fixed execution context … necessitating data-intensive fine-tuning for any new environment”. ICWM, by contrast, explicitly acknowledges that each deployment may have a different hidden configuration (camera pose, arm geometry, lighting, etc.) and uses the context window as a slot to infer those hidden variables from interaction data. The Key Idea: System Identification in the Context Window. At its core, ICWM is about turning the transformer’s context window into a temporary world model. Classical robotics has the notion of system identification – adapting a model of the robot and sensor to match reality. Typically this might involve calibrated camera parameters or identifying an unknown payload. ICWM achieves a similar goal implicitly, via the transformer’s attention mechanisms, and crucially does it zero-shot at runtime. The published description frames the robot’s environment as a partially observable system: each episode’s hidden state has two components, a time-varying scene state (like object positions) and a time-invariant configuration ψ — things like the camera extrinsics or link lengths that remain fixed during that trial but differ between trials. In ICWM, the policy is formally written as πθ(a_t | o_t, l, Ψ(T)), where o_t is the current observation (e.g. image), l is the language instruction, and Ψ(T) is an implicit representation of the world dynamics derived from a short history of interactions T. Here, T = {(o^s_i, a_i, o^e_i)} for i=1…N denotes N snippets of the form: “starting observation o^s_i, robot does action a_i, leads to new observation o^e_i”. In practice, these snippets come from a brief exploratory sequence (see below). The transformer's job is to look at T and build an internal hidden state that captures ψ – for instance, deducing that “the camera is high and to the left, and the gripper is slightly longer than before.” Once that hidden state is formed, the usual policy head conditions on it to choose actions for the actual task. In effect, the interaction history disambiguates the “world configuration,” so the policy can act as if it has made its own internal calibration. This stands in contrast to standard in-context learning. Normally, we give a language model some example QA pairs or a few demonstrations of the task, to define what we want it to do. ICWM uses the context window not to specify the task, but to characterize the environment itself. As one summary put it: traditional prompts teach the model “what task to perform”; ICWM’s prompt (the interaction history) tells the model how the system operates. In the authors’ words, the calibration steps are “evidence of how the current system operates,” enabling the model to have “implicitly modeled its own sensory and physical context” by the time the real instruction arrives. Because the system variables are inferred in this way, ICWM requires no fine-tuning or new parameters at deployment. The model learned how to interpret context during training, so at test time it simply reads off the context. In practice the authors note that this adds almost no overhead: one can pre-compute keys/values for the context frames (as in any transformer) so that running the probe sequence doesn’t slow down the later task execution significantly. In short, a single pretrained policy can handle new cameras or even new robot arms without ever running gradient descent – it just looks around for a few seconds and then does the job. Active Probing and Context Integration. How is this done in practice? According to the ICWM framework, each trial is divided into two phases. First, the Active Probing Phase: before the robot has any goal or language prompt, it performs a short burst of essentially random exploration in its workspace. For example, the arm might move its joints through a few random configurations or attempt random grasps. Crucially, these actions and the resulting images are not task-directed – they are purely “ego-probing” to reveal the geometry. The system records sequences of visual transitions (start image → action → end image) for a handful of randomized moves. The moonlight.ai literature review describes it this way: “It executes N random movements within the workspace, recording visual transitions (o_i^s —[a_i]→ o_i^e). These transitions do not contain task-relevant information but provide enough data to identify the current system configuration”. In other words, even though the robot isn’t trying to solve the intended task yet, these few seconds of data tell the model what it’s working with (camera tilt, table distance, gripper offset, etc.). Immediately after this probing phase comes the In-Context Execution Phase. Now the actual task instruction (a language command) and the current visual observation are fed through the transformer policy along with the entire sequence of probing data, as part of the input context. The model processes it in one forward pass. Effectively, the hidden state after absorbing the probing sequence acts as a learned summary of the world. The literature review says: “These N clips are fed into the model. using the induced hidden representations to modulate its task actions. This allows the policy to adapt its behavior to the specific geometry and kinematics of the current test environment without further training”. To give a concrete sense, imagine the camera has been moved higher above the table than during training. The probing images will consistently show quick downward movements yielding small object displacement on the image and possibly different apparent gravity direction. The transformer interprets this pattern and adjusts its internal expectation accordingly. Now when the instruction (“move the red block to the left”) arrives, the policy’s visual encoder and attention have been biased by the probing data; it implicitly “knows” the camera angle and translates the instruction to actions that make sense in that perspective. Critically, none of the transformer's weights change – the knowledge is carried entirely in the activations built by the context. This is why one can call ICWM a kind of implicit world model: it doesn’t learn an explicit parametric map of the dynamics for each new camera, but its hidden activations embody the relevant dynamics. As the authors put it, by “processing these interactions before task execution, the model implicitly captures the world dynamics of the current system”. The power of this method hinges on training the model to do exactly that: to attend to the probing interactions and use them to predict correct task actions, rather than ignoring them. Training the ICWM Policy. Of course, for this to work, the transformer policy must be trained in a special way. During training, ICWM must see many different configurations so it can learn to pay attention to them when they appear. The paper’s approach is roughly as follows (details like exact architecture are in the paper):. Dataset Construction: Collect or synthesize a rich dataset of tasks under many variations. This includes varying camera viewpoints, lighting, object textures, and even robot morphologies if possible. Each training episode looks like: choose a random configuration ψ from some distribution, simulate a short random interaction sequence in that configuration, and then simulate a language-guided task episode (with ground-truth actions). The probing and task segments are concatenated as the input. Behavior Cloning Objective: The policy is trained with supervised learning (behavior cloning). In each training sample, the model is given the context (probing frames plus instruction plus current frame) and must predict the correct next action at each task time step. Formally, if the probing history is T, observation is o_t, and instruction is l, the model’s output distribution is πθ(a|Ψ(T), o_t, l). The loss is simply the cross-entropy (negative log-likelihood) of the true action under πθ. Since Ψ(T) is not an explicit variable but an induced hidden state, the same network parameters σ must learn both to encode context T and to map to actions. Attention Learning: Because the model learns end-to-end, it will learn to attend to parts of T that are predictive of the hidden variables ψ. For example, it might notice that the relative image shifts in the probing sequence correspond to a certain camera angle. Over time, the transformer figures out patterns in T that correlate with required action adjustments. Importantly, if the configuration truly changes the task dynamics, then ignoring T would worsen training loss on out-of-distribution cases. The model is forced to use T if it improves its action predictions. They point out that no special architectural changes are needed – ICWM simply leverages the existing Transformer backbone. In fact, the authors highlight that no extra parameters are added: the same transformer that handles the vision-language inputs also handles the context. It’s effectively training the same network to be a kind of implicit meta-learner. One practical note: to speed up inference, one can store the transformer's key/value cache for the probing context, so that only the final portion of the sequence (the actual task) needs to be passed through the network anew. In this way the latency of adding a few context frames is kept very low, almost “to near-baseline levels” according to the authors. Experimental Results: Adapting to New Views and Bodies. Wang et al. validate ICWM in both simulated benchmarks and real robot experiments, focusing on changes in camera viewpoint (and to some extent robot morphology). The main takeaway is that ICWM often dramatically outperforms conventional VLA models when the environment is altered, all without any finetuning at test time. In simulation, they use the LIBERO benchmark (a suite of diverse robotic manipulation tasks) with multiple camera settings. They train their ICWM policy on a range of views, and then test on held-out (novel) viewpoints. Compared to a standard VLA model (behavior-cloned from multiple views but without context) their method shows significantly higher success rates. For example, they report up to +13% absolute success rate on out-of-distribution camera views. In concrete terms, if the conventional policy succeeded on, say, 50% of tasks after a large viewpoint change, the ICWM policy might succeed on 63%. Anecdotally, many tasks that the baseline policy got completely wrong (because it misinterprets the scene due to the shifted camera) become solvable when ICWM has seen a quick probe. On a real-world UR5e robot, they perform manipulation tasks (such as object placing or cabinet opening) where the camera is angled differently than in the training data. The real-robot results mirror simulation: ICWM policies outperform the baselines in these novel-view scenarios by a comfortable margin. While precise numbers for the real experiment aren’t quoted in the abstract, the authors stress “substantial resilience in real-world manipulation tasks”. (They also claim robustness to changes in the robot’s morphology, like putting different end-effectors on the arm, though most reported results seem to emphasize the camera angle case.). Reviewer summaries highlight that ICWM’s benefits are clearest on vision changes: “Experiments in both simulation and on real-world robot platforms show ICWM significantly outperforming standard VLA baselines on novel camera viewpoints”. Crucially, all of this is done zero-shot: once the policy is trained, you can hand it a new camera angle and it will just work after the brief context probe, with no extra training or parameter updates. Another perspective is that ICWM beats a multi-view baseline: one way to try handling multiple cameras is to train on data from many different angles (essentially domain randomization). Even so, a standard transformer without context still tends to “implicity assume a fixed execution context” and thus underperforms when really novel setups appear. ICWM’s use of context gives it an edge by effectively performing per-instance adaptation. Other Approaches and the ICWM Advantage. ICWM is not the only idea for dealing with new camera poses or robot variants, but it brings a novel twist. For contrast, consider a contemporaneous line of work: one paper explicitly feeds the camera’s pose parameters (the extrinsic calibration) into the policy network. In “Do You Know Where Your Camera Is?”, researchers condition the imitation model on known camera extrinsics (encoded via plume embedding of worldview) and show that this greatly improves viewpoint generalization. In effect, that method hard-codes extra input for “where is the camera relative to the robot”. It does work – the paper finds that without extrinsics, policies must cheat by looking at background cues (and then fail if the background changes), whereas adding the true extrinsic data “restores performance and yields robust RGB-only control”. The drawback, however, is that you must measure or know the new camera pose. ICWM avoids that requirement. It never explicitly sees the camera’s rotation matrix; instead, it infers it from image evidence. In other words, ICWM discovers the extrinsics from trial movements, rather than being told them. This makes ICWM broadly applicable: it can adapt even if the robot hardware moved slightly or if full calibration information isn’t available. Another recent method, AnyCamVLA, takes a completely different tack. It uses a neural renderer at test time to “rephotograph” the scene: given an image from the new viewpoint, it synthesizes what that view would look like from the old (training) viewpoint. Then the original policy (trained on the old view) can operate on the re-rendered image. A feed-forward view-synthesis model does the heavy lifting of handling both extrinsic and intrinsic changes. The AnyCamVLA approach indeed yields improved robustness: on LIBERO tasks it consistently outperforms baselines that rely on fine-tuning data augmentation or even adding 3D features. Comparing to ICWM: view-synthesis is elegant but requires a good neural renderer and may introduce its own artifacts. It also processes every frame in a possibly expensive way. ICWM invece is agnostic to how the image got there; it simply adapts the decision policy directly. In practice, one could even combine ideas (e.g. run ICWM on the synthesized image if desired). The key difference is that AnyCamVLA still uses a separate image-adjustment step, whereas ICWM embeds the adjustment process inside the policy network via context. More broadly, ICWM fits in a class of techniques that use contextual adaptation. It’s reminiscent of meta-learning for RL (where an initial “fast” context adaptation happens), but here it’s purely done through network hidden state and illustration data, not through gradients or external memory. In the space of vision-language-action models, it stands out by leveraging the transformer’s context in a novel way: the probe experiences are a prompt that says “here’s how this world behaves.” That’s a relatively unexplored use of prompts in robotics. Practical Considerations. For teams building robotic systems, ICWM has some appealing advantages. One model can potentially serve multiple robot variants and camera setups – imagine a warehouse where all robots use the same AI stack, and whenever a camera jogs or you swap to a longer gripper, you just press a “recalibrate” button (let the robot wave around randomly for a few seconds) and continue. The alternative would be retraining or manually calibrating each time. In terms of integration, nothing fancy is required if you already use a transformer policy: you just have to prepend that random interaction segment as part of the input sequence. The training loop needs to sample random interactions (which can be done in sim or with any off-policy data), but no supervision on those actions is needed other than the fact that they produce state transitions. During inference, it does mean you need a brief probing step. How short is “short”? The paper doesn’t insist on a long exploration – just enough bits to disambiguate. In practice this might be on the order of a few seconds or a handful of random pushes. The overhead is there, but in many applications a few seconds of calibration is fine if it buys zero-shot adaptation. The authors also note that because the context is fixed-length, one could even cache it if multiple tasks follow the same probe (though they do it once per task in their experiments). One might worry about scenarios with severe real-time constraints: can a mobile robot afford to wander before every task? That’s an open question that the authors themselves acknowledge. They note that evaluating the computational and time overhead of the active probing is important in real deployments. However, in settings like industrial manipulators or even household robots doing intermittent tasks, this trade-off could be well worth it to avoid complex fine-tuning pipelines. Accuracy-wise, it’s also interesting that ICWM seems to handle more than just crude camera shifts. The paper and summary mention robustness even to some morphological changes (e.g. “varying link lengths or gripper offsets”) and to semantic distractors (new textures, backgrounds). This suggests the context probe is solving a fairly general system-ID problem. Of course, how far this extends is up for further research: e.g. can it adapt to a completely new gripper with very different dynamics, or to changes in tool compliance? These would be natural next tests. Conclusion. In summary, In-Context World Modeling (ICWM) offers a promising new paradigm for robotic policy adaptation. By cleverly using a sequence of brief, random interactions as a prompt, it lets a pretrained vision-language-action Transformer infer the current environment’s hidden configuration and adapt behaviors accordingly. The result is impressive zero-shot generalization: the same model can work with new camera angles or (to some extent) altered robot morphologies, without any retraining or manual recalibration. As Wang et al. convincingly show, this yields significantly higher success rates on out-of-distribution tasks (e.g. up to ~13% absolute gain in novel camera-view tasks) compared to conventional baselines. This approach may reshape how we think about deploying robotic “foundation models.” Rather than viewing every change in hardware or viewpoint as a domain for expensive re-collection, ICWM suggests we can handle many changes on the fly, treating environment variation itself as another modality in the context. It fuses ideas from in-context learning in NLP with classical system identification in robotics. While questions remain – such as how to efficiently probe under tight time budgets, or how well it scales to entirely new robot kinematics – the core idea is both elegant and practical. For roboticists and ML engineers, ICWM offers a concrete strategy: when training your next vision-language policy, include a phase of task-agnostic self-play in each episode and feed it into the model’s input. The policy will learn to “look at itself” and calibrate. In doing so, it gains a kind of built-in world model that can tackle novel setups instantly. This could greatly simplify the lifecycle of robot learning systems, enabling one codebase to cover more real-world variability out of the box. It’s worth keeping an eye on this line of work, and on further evaluations of how broadly the idea can be applied – for instance, to locomotion or other dynamic tasks, or in combination with explicit world-modeling or planning methods. But for now, ICWM delivers a powerful new tool in the embodied AI toolkit: using the context window to learn the world, not just the task. References: The above summary is based on In-Context World Modeling for Robotic Control (Wang et al., 2026) and commentary from AI Weekly, as well as a recent moonlight.io literature review. We also contrast with related work on viewpoint adaptation. All findings and quotes are drawn from these sources as cited.