Latest / Embodied AI 101 / FutureSim: Replaying Real-World Events to Evaluate AI Forecasting Agents
Transcript
- FutureSim: Replaying Real-World Events to Evaluate AI Forecasting Agents. Modern AI benchmarks often freeze the world in a snapshot – a static dataset of images, text, or puzzles. But the real world doesn’t stand still. News breaks, events happen, and intelligent systems in the field must continually update their beliefs as fresh data arrives. FutureSim is a new benchmark that explicitly tests this adaptive ability. Instead of asking “can the model answer this question with its old knowledge?”, FutureSim spins up a three-month simulation of January–March 2026, feeds an AI agent real news each day, and tasks it with forecasting the unfolding future. As Alan Hou’s review puts it, traditional benchmarks “don’t measure what happens when an agent operates in the real world over time: new information arrives, beliefs need updating, predictions must adapt”. FutureSim fills that gap by replaying historical events in order and scoring how well agents stay in sync with reality. In practical terms, FutureSim dumps an agent into December 2025 with only history up to that point. Then, day by day, real news articles from January–March 2026 stream in. At each simulated day the agent can decide which pending forecasting questions to answer or update. Each question has multiple possible outcomes (e.g. “Who will be the next prime minister of Nepal?”) and resolves when the historical answer becomes known. The agent assigns probabilities to all outcomes, is scored (via Brier skill score), and then moves on. Crucially, once the agent calls , it cannot look ahead – it must commit to predictions and updates using only available information. Over 88 days of the simulation, about 244,000 news articles (from 141 sources, 7.36M deduplicated total) become available. The agent must sift through this surfeit of time-stamped data – using tools like time-gated search – to refine its beliefs. In effect, FutureSim is a flight simulator for forecasters: you aren’t just landing a plane at one end, you’re navigating a dynamic world of changing weather and traffic. As Alan Hou summarizes the analogy, FutureSim “throws … real time [events] at [the agent]”: the news feed is the instrument panel, the questions are the flight plan, and there’s simply no cheating by looking ahead. The Gap: From Static Datasets to Real-World Forecasting. Why is this sequential setup important? Most AI evaluations today focus on static tasks: can a model classify images it’s already seen during training? Can it answer a quiz question based on text in its context window? But real-world applications like financial trading, research assistants, or advisory bots run continually. New information trickles or floods in, and a robust AI must update its beliefs and plans. Existing forecasting benchmarks – say, predicting a sports outcome or economic indicator at one point in time – ignore this. They test prediction at a single moment (Metaculus-style or event-by-event), not the process of prediction over time. Likewise, most continual learning labs introduce synthetic data shifts, not the messy, correlated real shifts of news and events. In short, “no benchmark tests whether agents can maintain and update beliefs as real-world events unfold chronologically, the way a human analyst would read news daily and revise forecasts”. FutureSim’s key insight is to ground an evaluation in genuine chronology: the world is streamed in its natural order. This shift from static to sequential evaluation means we’re testing something fundamentally different: test-time adaptation. Instead of freezing the model after training, we allow it to adapt its internal state as it encounters new contexts. In FutureSim, that manifests as repeatedly revising forecasts with each wave of news. The agent isn’t merely retrieving memorized facts – it is actively reasoning about how today’s headlines change the likelihood of tomorrow’s outcomes. Succinctly,. Before: question → agent → frozen answer, score.After: real-world timeline → agent → evolving forecasts, ever-growing context, score trajectory. In the authors’ words, we’re moving “from testing knowledge retrieval to testing belief maintenance”. That requires new capabilities: long-term memory, uncertainty tracking, and continuous search. FutureSim is designed to probe exactly these. The FutureSim Environment. FutureSim’s setup is deceptively simple to describe, but rich in practice. The core components are:. Chronological News Feed: A database of real news articles dated and time-stamped. Agents can only read articles up to the current simulation date, ensuring no leakage of future information. Over the 88-day run, ~244,000 articles are added from sources like CommonCrawl News. Forecasting Questions: 330 open-ended questions (short answer style) about real-world events, all resolved by late March 2026. These questions are not cherry-picked hypothetical scenarios – they’re derived from actual Al Jazeera news articles (Jan–Mar 2026). The ground truth for each question materializes on schedule, after the knowledge cutoffs of all tested models. Agent Interface: At each day, the agent can perform actions. It can for any question (providing probabilities for all listed outcomes) and then when it’s done with current context. Meanwhile, the agent has access to tools within the sandbox: for example, a command-line search over the news corpus, or a more advanced query tool (LanceDB) that lets it filter by date range. The agent also has memory storage where it can jot notes or reminders. Scoring Mechanism: When a question resolves, the agent’s forecast is scored using the Brier skill score, a proper scoring rule that rewards well-calibrated probabilities. (Intuitively, Brier punishes being too confident about wrong answers or too timid about right ones.) We compare against a simple baseline (e.g. always 50/50) to get a skill score from –1 (hot mess) to +1 (perfect). For interpretability, we also track accuracy (the chance that the agent’s single highest-probability guess was correct), but the Brier score is the principal metric. In practice, here’s how a simulation proceeds: Agents start on Dec 24, 2025 (just before the new year) with no knowledge of 2026 events. Day by day, news arrives. On January 1, 2026, for example, the agent might see a dozen new articles. It can search them, write to memory, and for any open questions (the dataset tells it what questions are pending on Jan 1) issue a call. Each forecast call is free-form: the agent lists possible outcomes (usually obtained from question templates) and assigns probabilities. For instance, for a Nepal election question it might output:. This snapshot was actually made by a GPT-5.5 agent in the simulation. Over the next days, as more news comes in, the agent can call again to update its earlier predictions on the Nepal PM question. When March 28, 2026 arrives (the cut-off), the true outcome (say “Balendra Shah” if he won) is revealed and the agent receives Brier credit or debit for each of its daily probability tables. In between, an agent might ignore some questions entirely or focus on those it thinks are important – there’s no requirement to answer everything. Crucially, once it moves past a day (by calling ), it can’t go back or see beyond; the future is locked until we reach it. This design embeds several important features: 1) Chronology: Events unfold in the real order they happened – not shuffled. Each day’s context reflects the real world on that date. 2) Real Data: All predictions concern actual occurrences. The context is drawn from CommonCrawl news snapshots, which are immutable. This avoids the artificiality of random or synthetic scenarios – the world is messy, and its causal webs are complex. 3) Partial Observability: There are millions of documents (news for every day). An agent cannot simply ingest all of history at once. It must wield search tools to fish out relevant facts, and even then must decide what to store in internal memory versus what to discard. 4) Active Updating: The agent itself controls when to update predictions. It might wait until there’s enough evidence, or guess early to maximize timely reward. In other words, the agent embodies “epistemic humility”: it should recognize what it doesn’t know and update its confidence as news arrives. 5) Economic Stakes: Some questions correspond to real prediction markets with millions of dollars traded (Super Bowl results, elections, etc.). This means the benchmark isn’t just contrived – people bet real money on some of these outcomes. In fact, FutureSim can compare an AI’s forecast trajectory to the actual market trajectory (see below). By playing out this world-in-miniature, FutureSim forces AI agents to maintain situational awareness over a long horizon. The agent’s “context window” grows each day, and it must choose what to remember and what to let fade. This is a big departure from a one-shot evaluation: as one reviewer writes, we’re now “evaluating agents as adaptive systems rather than static functions”. Forecasting Tasks and Metrics. The benchmark’s challenge comes from the forecasting questions. Unlike multiple-choice or classification, these are open-ended probability assessments. Each question is phrased as a short query (e.g. “Who will win the Turkish presidential election?”) and comes with a set of plausible categorical outcomes (candidates, milestones, etc.). The agent must assign a probability to every possible outcome. Importantly, the probabilities do not have to sum to 1 – this allows the agent to express ignorance by summing to less than 1 as well (it can effectively leave some mass “unassigned” if unsure). However, scoring is still straightforward: when the true outcome is revealed, we compute the Brier skill score using the standard formula, which compares the squared error of the agent’s probabilities to that of a reference (e.g. uniform guessing). We also track accuracy as the more intuitive top-probability hit-rate, but a model can game accuracy by being very overconfident, so Brier (a proper scoring rule) is the deeper measure. In practice, to organize this humanly, the authors collected 330 such questions, all resolvable between Jan 1 and Mar 28, 2026. They come from Al Jazeera news so that each question is grounded in an actual news story. Because all questions resolve after typical model knowledge cutoffs (January 2026 is future relative to models), no evaluated agent has seen the answers prior. In the simulation, December 24-31, 2025 is effectively warm-up with no forecasts yet; on Jan 1, the first batch of questions (e.g. events anticipated later in January) becomes active. A question remains “open” until its resolution date. Agents can submit a probability forecast any day up until resolution; the final forecast before resolution is the one that counts for scoring (though intermediate updates can be scored cumulatively over time to compute overall skill). One example illustrates the process. Consider the Nepal prime minister question. On January 1, 2026, the agent knows from news that Nepal is electing a new PM and who the candidates are. Our GPT-5.5 agent might initially forecast: 34% for Balendra Shah, 18% for Khagendra Sunar, 12% for Sushila Karki, etc. As January news rolls in (e.g. primary results, parliamentary votes), it may adjust those probabilities day by day. When the actual winner is declared, say Balendra Shah, each day’s forecast receives a partial credit by Brier: if the agent consistently gave Shah around 0.34, it would get some points, but if it had predicted only 0.10 and he won, it would be heavily penalized. The agent’s Brier skill score for that question might be, for example, 0.2 (moderate) or even negative (worse than a dumb 50/50 guess if it was miscalibrated). Overall, FutureSim aggregates these scores. It reports the average Brier skill over all questions and also plots time-series trajectories showing how accuracy or skill evolves. A baseline “no prediction” system (basically predicting 50%/50% for everything until truth) scores a Brier skill of 0. Scores above 0 are better than random guessing, below 0 are worse. The authors note that several agents scored negative overall – meaning they were systematically misleading. By contrast, allocating probabilities in a calibrated way (for instance, recognizing uncertainty) tends to yield a positive skill. Evaluating Frontier Agents. The FutureSim study ran a suite of state-of-the-art language models in this environment. They used each model in its native “agent harness” (usually a code-oriented interface like Codex or Claude Code) and gave it the same tools (search, memory) and questions. The models included GPT-5.5 (in Codex), Opus 4.6 (Claude Code), and some top open-weight models – DeepSeek V4 Pro, GLM 5.1, Qwen 3.6 Plus (in an “OpenCode” harness). (These model names are placeholders for the latest LLMs demo’ed at the time of writing.). The headline result was stark: even the best model got only about 25% accuracy on top-1, and looked shaky on calibration. GPT-5.5 led with ~25% accuracy, but its Brier skill was never very high. Other closed-source leaders (Opus 4.6, etc.) hovered around 20% or less top-1 accuracy, and several of the open models were down in the 10–13% range. Critically, many models had negative Brier skill: they were worse than random. As one commentator summarized, “Frontier models achieve 25% accuracy, [with] many worse than baseline”. In concrete terms, repeatedly doing worse than 50/50 on a task suggests the model was badly miscalibrated or froze on mistakes. Even GPT-5.5, while best, often showed overconfidence on wrong answers – a classic forecasting pitfall. Polymarket comparison charts (from the project site) give color to these numbers. Several of the FutureSim questions coincided with real prediction markets. For example, the agent’s probability for the 2026 Super Bowl winner was tracked against the actual market odds (which saw $700M traded). Strikingly, there were moments where GPT-5.5’s curve outpaced the crowd: it shifted probability faster than the human aggregate. On the Nepal PM example, GPT’s forecast trajectory roughly mirrored the market’s. But on other events like US elections or the Grammys, GPT’s curve lagged badly – in one case it was “dramatically worse than the human aggregate”. These glimpses show that large LMs can sometimes uncover real signal in news, but often fail to track it or give too much weight to noise. In summary, the absolute numbers were not encouraging: a 25% accuracy means 75% of the time its most confident guess was wrong. FutureSim’s designers consider this a lower-bound view: models had only a limited context (7.3M articles) and modest tool support. In theory, better search, bigger memory, or extra compute could lift performance. Nevertheless, as-is, the frontier models were far from reliable forecasters. Adaptation, Memory, and Search: Ablation Studies. One of FutureSim’s strengths is its modularity: you can ablate or tweak components to probe what matters. The authors ran such experiments focusing on adaptation, memory, search, and compute. Test-Time Adaptation: In a key variant, all models were initialized with identical poor forecasts (taken from a weak baseline model) to give room for improvement. Then the simulation ran, and models could revise. The result: models that could adapt did improve over time, but none solved the anchoring problem. DeepSeek V4 Pro, for instance, showed “strong adaptation”, boosting its scores to match GPT-5.5 by the end. Opus 4.6 started worse but improved significantly as it updated. However, no agent fully broke out of its initial bias: even after weeks of news, no one ever reached the Brier baseline of 0 (random guessing). In plain language, once an agent locked into a bad forecast, getting out of it was hard. This “anchoring effect” echoes human forecasting literature: without strong beliefs about uncertainty, LMs tended to cling to early priors. Memory Ablation: The environment provides a memory where agents can store notes, and a retrieval interface. In the ablation, agents were forbidden from writing any memory during the sim. Performance predictably fell. The helpful memories tended to be things like “key evidence citations”, “how likely was X at last update?”, “calibration notes”, even meta-notes on source reliability. Models without memory lost track of resolved questions and contextual clues, hurting accuracy and Brier. The takeaway: having just a big context window isn’t enough – explicitly logged insights help navigate a long horizon. Search Ablation: They compared a full dynamic setup to cruel handicaps. In one worst-case, all news context was frozen on Day 1 (no new articles after starting). In another, the agent was only allowed a single search query per question (like a static non-iterative lookup). Both changes slashed performance. By contrast, letting the agent do daily context updates and unlimited sequential queries roughly doubled accuracy. In other words, the ability to continuously gather fresh evidence was crucial. An agent that could only query once (or not query at all) simply didn’t pick up on evolving signals. FutureSim quantifies that: going from one static query to query-as-needed roughly doubled forecasting accuracy, confirming that sequential search is a core skill. Inference Scaling: Even with everything else fixed, throwing more compute at the reasoning made a modest difference. Running GPT-5.5 at five “reasoning-effort” levels (more back-and-forth or larger responses) showed a clear trend: more compute → higher accuracy, with diminishing returns. The highest effort beat the low ones, though beyond a certain point extra reasoning flattens out. So part of the gap could, in principle, be closed by simply larger models or deeper prompting (if cost is no object). But calibration and update strategy are still the bigger issues. Multi-Agent Dynamics: In a fun exploratory test, three copies of an open model (DeepSeek v3.2) were run simultaneously on the same questions, but each could see the others’ aggregate forecasts. The idea was to simulate something like market feedback or expert panel discussion. Unsurprisingly, the agents’ individual predictions converged toward the crowd consensus over time. This “negotiation” effect improved calibration: the group aggregate stayed near the true outcome probability more often than any one agent would alone. It suggests that incorporating a crowd signal (even other LMs) can hedge the strong initial priors that single agents get stuck on. It’s an intriguing area for future study. Key Insights and Future Directions. FutureSim has already yielded some sobering takeaways. As one commentator notes, “current agents are shockingly bad at maintaining beliefs over time”. In practice, this means that if you’re designing an AI assistant that needs to work week-to-week, you can’t rely on a fixed knowledge of January 1. You need explicit memory and board-aids for belief tracking, and a principled strategy for when to update — otherwise your model will simply freeze or wander as news arrives. The benchmark highlights several research thrusts:. Memory & Belief State: Large models must be augmented by external memory or state variables if they are to carry knowledge forward day by day. The fact that all agents did worse without memory (in the ablation) points to the need for structured long-term memory and retrieval (perhaps specialized knowledge graphs or vector stores) to recall old inferences and evidence. Uncertainty Calibration: Negative Brier scores showed many models were overconfident or underconfident. Training paradigms like Bayesian fine-tuning or ensembles to better capture uncertainty might help. The notion of “epistemic humility” is built into FutureSim: the agent should know when it doesn’t have enough info and spread its bets accordingly. Adaptive Reasoning: FutureSim is a test-bed for test-time adaptation methods. Techniques like reinforcement learning online, continual fine-tuning, or even simple heuristics like “if new evidence strongly contradicts my prediction, reset it” can be studied here. The fact that adding a shared small bias lift (starting from the weakest model) gave room to improve indicates compounding dynamics are key. Researchers might use FutureSim to systematically tweak how often models self-update or to develop “adapters” that can be trained on partial streams of evidence. Search and Tools: The dramatic drop in performance without agentic search shows that retrieval-augmented methods and crowdsourcing tools are not just optional; they’re necessary. FutureSim could drive better search algorithms that are date-aware, multi-hop, and context-sensitive. It also suggests investing in agents that decide when to search. In FutureSim, an agent could waste days by flailing or could effectively gather evidence only when needed – that strategy is worth optimizing. Human Benchmarking: One open question is how well humans would do on the same setup. The authors compare against prediction market aggregates as a rough baseline, but a formal human-forecaster leaderboard would be informative. How much of GPT’s limitations are model flaws versus an intrinsically hard problem? FutureSim noted that market responses had sometimes outpaced the models (or vice versa), but a controlled human experiment could set a target to strive for. Extending Horizons: Three months is just a start. The authors themselves remark that three months might be too short to see truly “long-horizon” drift – errors didn’t compound through year-long stretches. Extending the timeline to 6 or 12 months would stress beliefs even further. On the other hand, even this 3-month test already unearthed weaknesses. In sum, FutureSim shifts the paradigm from “can the model answer this question right now?” to “can the model learn over time?” The paper shows that even cutting-edge models, given full access to daily news, lag well behind trivial forecasting benchmarks. As one review bluntly puts it, these results demonstrate that having a powerful static language model is not sufficient for real-world forecasting – agentic orchestration is needed. Specialized pipelines, additional mechanisms for memory and uncertainty, and possibly multi-agent collaboration are required to make progress. Conclusion. FutureSim presents a valuable new litmus test for AI agents: their ability to anticipate the future under realistic conditions. By replaying a slice of history, it shows us exactly where models trip up – lack of memory, miscalibration, or insufficient search – in a way that a single-snapshot quiz never could. The benchmark and code are openly available, inviting the community to try new tactics. For anyone building long-term assistants, planning algorithms, or integrated AI systems, FutureSim is a wake-up call: the world is changeable, and our agents must change with it. While today’s models scored poorly, the path is clear. Developer and researcher notes rightly stress that “you need explicit memory systems, belief tracking, and mechanisms to decide when to update vs. stick with prior beliefs”. Treat FutureSim as a simulation sandbox – not of robots moving through mazes, but of reasoners marching through history. Only when our agents can keep pace on these timelines can we consider them truly adaptive.