Latest / Embodied AI 101 / MCP-Cosmos: Bring Your Own World Model

Transcript

MCP-Cosmos: Bring Your Own World Model · Embodied AI 101

• MCP-Cosmos: Bring Your Own World Model. Large language model (LLM) agents can now talk to APIs and tools via the Model Context Protocol (MCP), but they still often stumble when planning multi-step tasks. A core reason is that today’s agents lack a real model of the world they are interacting with. Standard approaches either plan abstractly (ignoring execution hiccups) or act one step at a time (suffering from “horizon myopia” and rampant trial-and-error). IBM’s new MCP-Cosmos framework aims to close this gap by injecting a latent-space world model into the agent’s workflow. In MCP-Cosmos, when the agent considers making a tool call, it first “imagines” the outcome using a world model. It only executes a plan in the real world once it has simulated and refined that plan internally. In other words, agents get to test drive potential action sequences in a virtual space before actually spending API calls. This simple idea – a “Bring Your Own World Model” (BYOWM) strategy – led to significantly higher task success rates and more accurate tool use on complex benchmarks. The Planning vs. Reacting Trade-off. To appreciate what MCP-Cosmos fixes, it helps to contrast two extremes of LLM agent behavior. In a planning-centric approach, the agent (or LLM) tries to reason out an entire plan of tool calls up front, typically by chaining thoughts (“chain-of-thought”) and composing APIs in a single prompt. The problem is that such planning assumes the agent knows exactly how each API call will turn out. In reality, APIs and environments can be stochastic, fail, or produce unexpected results. Planning-only agents often commit to sequences that fail in practice, with little recourse at runtime. On the other hand, a reactive agent – take the classic ReAct pattern – alternates between thinking and calling a tool, observing the real response, then thinking again. This derived more robust behavior but suffers from short sight: it doesn’t look far ahead, so it tends to hammer the environment with repeated calls until something works. For instance, a reactive agent might query an unfamiliar API repeatedly or try many irrelevant tool calls just to cover its bases, leading to wasted work and slow performance. MCP-Cosmos argues that the missing element is a predictive model of the environment. If the agent had a way to simulate the effect of a tool call before actually making it, it could avoid both pitfalls: it would neither blindly plan in the dark nor resort to brute force. As Ganapavarapu and Patel (2026) put it, a BYOWM strategy “allows agents to simulate state transitions and refine plans in a latent space before execution”. In practice, this means giving the agent a world model – a generative model that can predict how the world (here, the software environment exposed by MCP servers) evolves in response to actions. The rest of the agent’s machinery (the LLM planner and the MCP tool interfaces) stay the same; only now the agent can introspect on consequences. Figure: An agent uses an internal “dream” of the world to test out actions before acting. In MCP-Cosmos, a latent-space world model lets the agent simulate API calls (like this robot querying code) and adjust its plan without touching the real environment. How a Latent World Model Works. At the heart of MCP-Cosmos is a simple two-phase loop: imagined planning and real execution. In the planning phase, the agent generates a sequence of candidate actions (tool calls) using only the world model as its environment. Concretely, we implement a interface with a function. Whenever the planner (an LLM acting as the agent’s decision maker) proposes calling a tool with certain arguments, instead of immediately invoking the real tool, the agent feeds this request into the world model. The world model then returns a simulated observation – for example, a summary of what the tool would return, or a mock data result. This pseudo-observation is indistinguishable from an actual API result from the agent’s perspective, so the planner updates its internal state as if that call had been executed. In code, it looks like this: the planner calls , and gets back a predicted output. One can think of this as the agent “hallucinating” the tool’s response. As the agent continues planning, each action-simulation pair is added to a world-model trajectory. In other words, the agent builds an imaginary timeline of pairs without ever touching real APIs. If the planner suggests a second action, the world model is invoked again, returning another simulated observation, and so on. Essentially, the agent can roll out entire sequences of actions in the latent world. This lets it explore multiple alternative plans cheaply. At some point the planning loop reaches a stopping condition (for example, an LLM-generated signal that a final answer has been found, or a fixed depth limit). The result of this phase is one or more candidate plans – sequences of tool calls together with their simulated results. Once the agent has such a plan (or a set of candidate plans), it commits to executing it in the real environment. In the execution phase, the agent goes through the planned tool calls one by one, now actually calling the APIs via MCP. The real observations (real API results) are collected into an execution trajectory. If any action fails during real execution, the system could try to adjust the remaining plan, but in our evaluation this step was typically disabled to keep the comparison fair. After all steps are done, the agent has a complete actual trajectory of actions and real results. It then synthesizes a final solution or answer from these results (for example, summarizing gathered data). Throughout, the planner and decision logic are exactly the same LLM prompts or reasoning used in a reactive setup – the only difference is that now those prompts receive simulated observations during the planning phase. In pseudocode, the key idea is:. In their description, the authors note that the world-model trajectory and execution trajectory are both returned for transparency, along with the final answer. In practice, the ReAct or SPIRAL-style prompts used for the planner don’t change; they just see either real or simulated results depending on the phase. Agents and World Models Evaluated. To show that this BYOWM framework isn’t tied to a specific model, MCP-Cosmos is designed to be modular. The agent code that plans and executes remains the same; we simply plug in different world models as needed. In the paper, three representative world models were tested:. GPT-OSS-120B: A large open-source LLM (roughly 120B parameters) used in a generative mode as a world model. Given an API call and current context, it tries to predict the outcome textually (e.g., “calling tool X with args Y would return Z”). Claude Sonnet 4.6: Use of a proprietary LLM (Anthropic’s Claude) as the world model. Claude has multi-turn reasoning, but in this case it is used simply to simulate environment steps. Arctic-AWM-4B: A specialized agentic world model developed by Snowflake. This is a 4B-parameter model built on the Qwen architecture, which was trained via reinforcement learning on fully synthetic tool-use environments (“AgentWorldModel-1K”). In effect, Arctic-AWM-4B is already tuned to act like an agent interacting with tools; it can serve as a light-weight simulator of the MCP servers. Each of these world models stands in for the “imagination” engine. Importantly, Arctic-AWM-4B is not understood to be inherently superior – it’s simply a different approach (task-specific RL training) compared to general-purpose LLMs. In fact, the experiments found that the open and Claude models often outperformed the specialized Arctic model. (We’ll see that in the results.). For planning policies, MCP-Cosmos tested two strategies:. ReAct-Plan-Exec: A reactive-style planner (like original ReAct) is used during the planning phase, except it samples actions and lets the world model predict their outcomes. This is essentially the baseline ReAct method enhanced with a simulation loop (i.e. ReAct agent “imagines” each step before executing the plan). SPIRAL-Exec: A more systematic planner combining symbolic search with LLM thinking. SPIRAL (Symbolic Planning via Grounded Reflective Search) embeds a planning LLM into an MCTS loop. In MCP-Cosmos, the SPIRAL planner also calls into the world model during its search rollouts. This means SPIRAL can look ahead systematically by expanding search nodes in the latent environment provided by the world model. (The details of SPIRAL are complex, but for our purposes think of it as an LLM-based MCTS.). Thus we have evaluation configurations like: “ReAct with GPT-WM” vs “ReAct with no WM (baseline)”, and “SPIRAL with GPT-WM”, “SPIRAL with Claude-WM”, etc. The baseline is just a pure ReAct agent interacting directly with the real environment (no world model simulation). Benchmarks and Metrics. MCP-Cosmos was tested on tasks drawn from MCP-Bench, which is a suite of realistic multi-tool tasks using actual MCP servers. The authors focused on tasks requiring multiple tools (2-server or 3-server tasks) over 24 selected tasks (covering about 12 types and ~300 example runs). These tasks use a variety of real APIs (like NASA, maps, databases, etc.), so solving them requires correct tool use and coordination of data between calls. The tasks were not used to train anything—the world models and planners were fixed models, and MCP-Bench served purely as held-out evaluation. Performance is measured with a hierarchy of metrics. Traditional bottom-line metrics include task completion (did the agent solve the task?) and tool usage scores (did it call the right tools with the right arguments?). Two key measures are tool call success rate (percentage of tool calls that returned the correct type of result) and tool parameter accuracy (how well the agent filled in the arguments or parameters of each tool). These capture how effectively the agent is interacting with the APIs. However, the authors realized that raw success scores can be misleading on multi-step tasks: an agent might brute-force a solution with many extra calls, inflating its success. So they introduced a new metric called Execution Quality. Intuitively, Execution Quality rewards agents that solve tasks efficiently (few calls) as well as correctly. It is computed as the average of the tool success rate and an inverted, normalized call count. In formula form:. [ \text{Execution Quality} = \frac{\text{ToolSuccessRate} + \text{(Normalized Inverted AvgToolCalls)}}{2}, ]. where “normalized inverted avg tool calls” decreases as the agent’s average number of calls increases. In practice, higher Execution Quality means you solve the task while making fewer unnecessary API calls. Results: More Success, Fewer Mistakes. The experimental results firmly support the promise of MCP-Cosmos. Across the board, world-model-augmented agents achieved significantly higher API success and accuracy than the reactive baseline. For example, the authors report that MCP-Cosmos runs “observed improvements in … tool success rate and tool parameter accuracy” compared to the baseline agent. In other words, agents with a world model made fewer mistakes in which tools they called and how they called them. Qualitatively, we can say that by “thinking ahead”, the agent avoided blind alleys where a tool call would fail or return irrelevant data. One concrete way to see this is to look at the ReAct agents. The vanilla ReAct baseline had a tool call success rate of only around 77.7% on average – meaning almost a quarter of its calls were incorrect or unhelpful. By contrast, ReAct augmented with a world model (or SPIRAL with a world model) typically achieved near-perfect success rates. Similarly, parameter accuracy (how exactly the agent filled arguments into each tool) was much higher with world modeling. The paper’s overall “accuracy”-type scores were substantially improved when using a world model. Another noteworthy highlight: the best performing configuration was SPIRAL planning with the GPT-OSS-120B world model, which achieved roughly 44–45% on the benchmark tasks under the standard scoring rubric. That might sound low, but recall these tasks are quite complex (especially the 3-server ones). Under the Execution Quality metric, that same configuration scored about 56.5%, while the baseline ReAct agent only got 36.8%. In other words, the world-model agent didn’t just solve more tasks correctly, it did so with far fewer wasted calls. Importantly, the new Execution Quality metric highlights the efficiency benefit. The pure ReAct agents tended to use many more calls per task. In one comparison, the baseline ReAct used about 7.04 calls on average per task, whereas SPIRAL+WM (GPT-OSS) might only use 4–5 calls. By the Normalized-Inverted-Calls formula, this efficiency dramatically boosts the score. The Execution Quality penalty hit the baseline hard, explaining why its score dropped to 36.8%. In qualitative terms, benign but “dumb” agents that guess repeatedly often squeak by on success, but at high cost. The world-model agents “waste” far fewer calls. For example, with the world model a strong planner no longer fishes randomly; it commits to a vetted sequence. As the authors note, world models “constrain powerful planners, reducing exploration and improving efficiency by focusing execution on vetted plans”. This means the agent isn’t frantically querying APIs in all directions; it concentrates on the most promising plan carved out in the simulation phase. Interestingly, having a strong planner alone is not enough without the world model. The authors tried an ablation where a much more powerful LLM was used in the baseline agent (Anthropic’s Claude Sonnet-4.6). Indeed, the Claude-powered planner found more accurate solutions when allowed, but it also went on a wild exploration. It made an average of 29.78 calls per task – quadruple the 7.04 calls of the weaker baseline. This confirms that a smarter reasoning engine will aggressively check lots of options unless constrained. When a world model was added, even the strong planner was forced to stick to a vetted plan, vastly cutting down the unnecessary calls and raising the execution quality. In short: a stronger LLM alone multiplied the “shotgun searching,” whereas the explicit world model kept the search focused on what really matters. One more trade-off emerged: computational cost. Running these world-model simulations is not free. Every time the agent imagines a call, it invokes a large model. In practice this means much higher API and token usage during planning. The paper notes that some of the world-model setups used several times the tokens of the baseline (in one case 5× more). This is an expected overhead: to “think ahead,” the agent must do more thinking. In their report, the world-model agents consumed substantially more compute, reflecting the extra LLM queries. However, this cost may be acceptable compared to the real-world expense (in time and potential errors) of blind trial-and-error in the environment. Key Takeaways and Insights. Putting the pieces together, MCP-Cosmos demonstrates that coupling LLM agents with a predictive world model yields tangible gains in multi-turn tool use tasks. The core idea – simulate before acting – is straightforward, but its effect is clear: more accurate tool calls and less wasted exploration. We can summarize a few key points:. Predictive Planning Improves Reliability: By simulating API calls in latent space, the agent avoids many execution-time mismatches. The benchmarks show improvements in tool-success metrics when the agent uses a world model. This means it learned to call the right tool with the right parameters more often, because it had “seen” the result in advance. Efficiency (Execution Quality) Jumps: Agents with world models solved tasks with far fewer extraneous calls. Even if the baseline eventually finds an answer (by exhaustive search), the world-model agent tends to reach it more directly. The novel Execution Quality metric makes this evident: world-model agents scored much higher, reflecting their leaner execution footprint. In practice this means they spend fewer API quota and less time fiddling with irrelevant calls. Strategy Matters (SPIRAL vs ReAct): The benefit held for both simple reactive agents and more complex planners. In the experiments, the SPIRAL planner combined with a world model was the top performer under both raw and efficiency metrics. This suggests that systematic search (guided by LLM reasoning) meshes well with having a simulator. However, even a plain ReAct agent saw gains when augmented with a world model, so the approach is broadly applicable. Strong Planner Alone Isn’t Enough: A very capable LLM (e.g. Claude) does improve accuracy, but it also over-explores without a world model. This underlines that explicit modeling of the environment is not redundant; it complements the planner by pruning bad branches early. Specialized vs. General World Models: The mortgage test between Arctic-AWM (specialized RL-trained model) and general LLMs (GPT or Claude) found the surprise that the LLMs generally did better on this suite. The Arctic model was “less effective than general-purpose LLM-based WMs” in most cases. This hints that for these kinds of web/API tasks, large LLMs already encode a lot of world-like knowledge that can serve as a simulator. It doesn’t rule out the promise of learned environment models, but it shows off-the-shelf LLMs can already do a lot. New Evaluation Metrics: Perhaps the most interesting methodological contribution is pointing out the gap in how we judge agents. Previous metrics emphasized success and task completion, but missed inefficiency. By introducing Execution Quality – essentially combining success with call economy – the authors highlight the exploration-cost trade-off. As they note, this reveals hidden costs: a high-achieving agent that brute-forces too much will score lower than a more clever one that does it in fewer steps. This encourages future work to focus not just on “can it solve it?” but “how gracefully does it solve it?”. Overall, MCP-Cosmos provides strong evidence that explicit world models belong in the LLM agent toolbox. It shows that predictive simulation and planning are not just abstract concepts but deliver measurable benefits when integrated with real tasks and tools. By modularizing the framework (“You can bring any world model you like”), it opens the door to further experimentation: one could try learned neural world models, hybrid symbolic models, or even fine-tuned simulators for specific domains. Caveats and Future Directions. It’s worth noting the context and limitations. The benchmarks used static software environments; nothing was physically changing unpredictably in real time. All interactions were with fixed APIs, so the world model only had to learn those stable dynamics. If you move to, say, robotics or interactive games, you would need a richer simulation. Also, the current world models (especially LLM-based ones) can hallucinate, and there is no feedback loop to correct the world model itself here. The world model is assumed to be “good enough” for planning but it isn’t learned online. So in its present form, MCP-Cosmos requires that the world model already roughly knows the domain. Another point is compute cost. The multi-turn simulation uses multiple LLM calls. In practice, one must ensure this cost does not outweigh the savings from fewer real calls. The paper observed that some configurations used 2–3× more compute during planning. As LLMs get cheaper or as smaller world models improve, this trade-off will shift. Future work might train compact dedicated simulators (versus calling a full LLM each step) or only simulate the most uncertain calls. Finally, the new Execution Quality metric is a useful step, but more nuanced evaluation can still help. For instance, capturing why an agent chose a tool (e.g. its reasoning traces) or how robust it is to model errors would give deeper insight. The current metrics focus on end results and counts. There is room to better quantify partial planning finesse and reasoning quality. Conclusion. MCP-Cosmos marks a step toward truly predictive LLM agents. Instead of treating the digital environment as a black box, these agents build an explicit internal mirror of it. This mirror need not be perfect – even an LLM predicting text working as a pseudo-simulator helped significantly – but it grounds the agent’s planning. The upshot is that when the agent finally acts in the real world, it has much better odds of getting things right first time. In short: MCP-Cosmos shows that “thinking twice before acting” (or rather, dozens of times in simulation) pays off. As our community builds ever more capable agents, incorporating world models like this could become standard practice, moving us closer to dependable agentic systems that reason and act as a coherent whole. References: MCP-Cosmos paper summary and results (Ganapavarapu et al., 2026).