Latest / Embodied AI 101 / WildClawBench: A Real-World, Long-Horizon Benchmark for AI Agents
Transcript
- WildClawBench: A Real-World, Long-Horizon Benchmark for AI Agents. As large language models (LLMs) and vision-language models (VLMs) become the “brains” behind software agents — systems that autonomously carry out tasks via code, command-line tools, or APIs — there is growing interest in how to evaluate these agents on realistic jobs. Traditional benchmarks often test only isolated skills (e.g. solving a math problem, answering trivia, or calling a single API function). WildClawBench, a new benchmark introduced by InternLM (a Chinese AI research group), aims to change that. Instead of short puzzles, it asks: Can a model-based agent actually do “real work,” end-to-end, in a realistic environment, without hand-holding?. In WildClawBench, the agents are dropped into a full-featured, live software environment (OpenClaw) and given complex, multi-step tasks. These tasks are drawn from real human activities: summarizing technical papers, negotiating meeting times by email, analyzing video clips, writing and debugging code, and even detecting security threats hidden in documents. Each task takes on the order of 8–20 minutes of total run time and requires chaining dozens of model-driven actions. The benchmark is built to stress-test the full stack of an LLM/VLM agent: planning, tool-use, multimodal understanding, long-term context management, and more. In this article we unpack WildClawBench: what it is, how it works, what tasks it contains, and what it reveals about current AI agents. We will see that WildClawBench is not actually about physical robots or manipulation (despite its claw-themed name), but about software agents performing real-world information-processing jobs. Nevertheless, many of its design ideas — containerized evaluation, rigorous grading, and a public leaderboard — mirror what robotics researchers do for embodied tasks. As we’ll describe, WildClawBench provides a turnkey platform (Docker images, task data, scoring code, etc.) for running benchmark experiments on state-of-the-art LLM agents, and the initial results show that even top models solve only a fraction of these real-world tasks. Why WildClawBench: The Need for “In-the-Wild” Agent Evaluation. Modern LLM agents (e.g. AutoGPT-style systems, LangChain applications, etc.) are being deployed to handle multi-step tasks that resemble real jobs: summarizing data, generating reports, debugging code, browsing the Web to answer complex queries, automating workflows, and so on. Evaluating these agents, however, has lagged behind. Many benchmarks limit themselves to toy settings or final-answer checks. For example, an agent might be asked to call a fixed API sequence and then judges only whether the last API output is correct, rather than examining every step of its execution. This leaves open the question: If I drop GPT-like agents into a realistic, unscripted environment, can they actually achieve the intended goal?. WildClawBench is explicitly designed to answer that question. In the authors’ own words: “WildClawBench tests the full picture.” Unlike benchmarks that check one capability at a time (a single instruction, one tool call, etc.), WildClawBench forces agents to plan and act autonomously over long horizons. Agents must figure out what sequence of actions to take — using algorithms, web searches, code tools, or anything at their disposal — to accomplish tasks like synthesizing information from multiple sources, negotiating with human-like text interfaces, processing video and images, or writing nontrivial programs. Concretely, WildClawBench presents 60 hand-crafted tasks, each authored by humans specifically to be challenging and realistic (not just paraphrased from existing benchmarks). The authors note that it spans six thematic categories (see below) and is bilingual, with tasks in both English and Chinese. In aggregate the tasks average about 8 minutes of wall-clock execution and involve on the order of 20 model-driven calls or tool uses each. Importantly, these tasks are run in a native runtime environment — not in a sandboxed simulator — meaning the agents use real tools (a browser, file system, shell, email client, etc.) in a Docker container to complete objectives. This end-to-end setup is what makes WildClawBench a “real-world” benchmark. Roboticists might think of this as analogous to putting a robot in a new physical environment and giving it a series of chores, except here the “environment” is digital (the OpenClaw AI assistant platform) and the “chores” involve information tasks. Still, the philosophy is similar: measure whether your system can actually complete complex, multi-step tasks under realistic conditions, and say goodbye to artificial shortcuts. The OpenClaw Environment and Execution Pipeline. WildClawBench tasks run inside OpenClaw, an open-source personal AI assistant framework. OpenClaw provides the agent with a suite of real-world tools and interfaces: for example, a file system shell, a web browser interface for searching or visiting websites, email and calendar access, and internal utilities. (In effect, it’s like giving the LLM agent an entire computer desktop to work on.) Each task is given a unique Docker container image, which is an unmodified Linux system with OpenClaw installed and preconfigured. The agent under test is then plugged into this container. The tasks and environment are orchestrated via CLI agent harnesses. WildClawBench includes support for multiple agent frameworks (listening to OpenAI, Anthropic, or other LLM APIs, or even local models) but the interface is the command line. During a run, all of the agent’s actions (for instance, “open a browser and perform this search query”, or “run this shell command”, or “send this email”) happen inside the container, and the environment state changes accordingly. Because the containers are standardized, runs can be reproduced exactly on different machines. In fact, the benchmark authors go to great lengths to avoid data leakage: ground truth answers and grading scripts are only loaded after the agent finishes its run. The agent never sees the “solution” data during execution, so it cannot simply memorize or overfit to it. Setting up to run WildClawBench is relatively straightforward (though computationally heavy). The HuggingFace repository for WildClawBench contains the Docker image (a tar archive of roughly 10–20 GB) plus all the input data and grading tools. A typical workflow is: (1) pull the Docker image from HuggingFace, (2) launch a docker container, (3) use the provided Python evaluation scripts to iterate through the 60 tasks in each category with your agent, and (4) collect the scores. Each task run can take several minutes to complete — the top-performing agent needed on the order of 500 minutes (≈8½ h) of wall-clock time to process all tasks in one run. This is analogous to some long robotics benchmarks (e.g. weeks-long trials), but for agents it reflects the heavy computation of many large LLM API calls. In short, WildClawBench provides a containerized, reproducible framework for agent evaluation. Every participant (agent model) uses the same underlying environment for each task, and all grading is deterministic and scripted (or uses a separate LLM judge after execution). This is a level of rigor that many previous agent benchmarks did not have, and it sets the stage for fair comparisons. Task Categories and Examples. WildClawBench’s 60 tasks are organized into six broad categories, chosen to cover diverse realistic activities. The authors summarize each category with example tasks and core challenges. We highlight some here:. Productivity Flow (10 tasks) – These tasks simulate typical office or research work that requires gathering and organizing information. For example, one task might ask the agent to digest an entire ArXiv paper and produce a concise summary. Another might involve processing a batch of PDFs: classifying them by content or extracting specific tables. There are tasks around calendar scheduling (e.g. parsing meeting constraints and creating schedules) and web research (e.g. compiling a concise biography from Wikipedia). The core skills tested include information synthesis, handling multi-document sources, and outputting structured results. Code Intelligence (12 tasks) – These tasks require the agent to understand and write code or solve algorithmic puzzles. Some tasks start with SATcode of an undocumented codebase: the agent must figure out what the code does and then write a script (in Python or a similar language) that uses it. Other tasks are visual puzzles akin to “jigsaw” or “connect-the-dots”: the agent is given an image puzzle and must generate code or an exact image output to solve it. For example, one challenge might ask the agent to reproduce a particular plot from input data. These tasks test the agent’s abilities in code comprehension, end-to-end generation of working programs, and even pixel-level reasoning (when puzzles have precise visual answers). Social Interaction (6 tasks) – These involve multi-turn, dialogue-like scenarios. For instance, there might be a task to negotiate meeting times via email back-and-forth, or to handle a simulated customer-service conversation where the agent must extract certain actions or intents. The key here is that the agent must maintain context across multiple messages, route information appropriately (e.g. escalate issues to the right department), and orchestrate APIs (like sending calendar invites) as part of the workflow. Search & Retrieval (11 tasks) – These tasks blend web search with local data. An example could be: retrieve conflicting information from different sources about a topic and reconcile it into a single answer. Or, pull financial data scattered across several documents and web pages and summarize trends. Another is “fuzzy repository search,” where the agent must find relevant code examples or text snippets in a large messy codebase. These tasks test the agent’s ability to combine web search results with internal knowledge, satisfy multi-constraint queries, and verify the trustworthiness of sources. Creative Synthesis (11 tasks) – These tasks involve generating or analyzing multimedia content. For example, one task asks the agent to watch a 45-minute football (soccer) match video, identify key events, and clip the video into highlights or write a match report. Others involve images: assembling styled outfits from example photos, then generating full-body model images for each outfit (testing cross-modal image generation), or taking a written announcement of a product launch and creating a short video storyboard or poster. Some even involve audio processing, like dubbing video from English to Chinese. These require the agent to do advanced video/audio understanding, apply generative models (text-to-image, etc.), and engage in non-trivial design and layout tasks. Safety Alignment (10 tasks) – These tasks test robustness and safety. The idea is to hide “bad stuff” in tricky ways. For example, the agent might have to scan a long document that contains malicious instructions buried deep within normal text (prompt injection) or detect leaked API keys hidden in a mock Git history. Another might simulate a user installing new “skills” (plugins) that could be malicious. The agent must refuse harmful content or flag security issues. The challenges here are adversarial awareness: finding needle-in-a-haystack problems like hidden credentials, malicious code snippets, or misinformation that should be rejected. Together, these categories cover a broad spectrum: from pure information processing and productivity (like a digital assistant) to coding, communication, rich media processing, and security. All tasks were originally created by the WildClawBench team, not borrowed from earlier benchmarks, ensuring they truly reflect real planning and tool-use scenarios. Also, since tasks are bilingual, some tasks specifically test multilingual understanding (e.g. translation tasks from English to Chinese in the Creative category). Each task has a public ID and comes with all the necessary data: documents to analyze, scripts to run, etc. The tasks are non-trivial: most involve dozens of subtasks. For illustration, consider a couple of concrete examples (paraphrasing the authors’ descriptions):. Example 1 (Productivity): ArXiv Paper Digest. The agent is given the text of a new academic paper and must extract and present the key findings in a structured format (e.g., listing problem, approach, and results). It must navigate through PDF figures, references, and possibly LaTeX math. Example 2 (Creative): Football Match Highlight Generator. The agent receives as input a 45-minute video file of a sports match plus commentary. It must analyze the video frames and commentary to identify goals or important events, cut out those video clips, and write short text captions for each highlight. This uses vision, audio processing, and generation in tandem. Example 3 (Safety): API Key Leak Scan. The agent is given an archive of a code repository, including many text files and commit histories. Among these, some files secretly contain fake “API keys” or credentials. The task is to find all the leaked keys and alert the user, without being tricked by irrelevant text. These are the types of tasks where even a human might have to switch contexts and call on different tools. WildClawBench tracks everything with deterministic rules when possible (e.g. did the agent actually isolate the correct clips?) and with semantic verification when needed (using an LLM or VLM to judge if the output makes sense). Grading and Leaderboard. For each task, WildClawBench provides a ground-truth answer, grading scripts, and a scoring rubric. The grading process is hybrid: some checks are fully deterministic (e.g. a file or value produced by the agent must exactly match the expected answer), while others are judged by a separate AI evaluator. For example, if a task’s goal is a free-form answer, a pre-trained LLM/VLM can compare the agent’s answer to the expected answer to see if it’s semantically correct. Importantly, all evaluation is done after the agent has finished the task. The system never feeds the answers or grading code into the agent while running, which prevents any subtle forms of "cheating" or overfitting. Scores for each task are normalized between 0 and 1, and then combined to give per-category and overall scores. The benchmark is designed to be cumulative: an agent must solve all sub-parts of a task to score 1.0 on that task. In effect, partial credit is rare; an agent either completes the pipeline or it doesn’t (with some allowances for partial subtask success). InternLM has been running this benchmark on leading AI models and maintaining a public leaderboard. As of early 2026, the results show that no model is close to mastering WildClawBench. In one evaluation (with models using the OpenClaw harness), the best-performing system was Anthropic’s Claude Opus (v4.7 series) which only achieved about 62% success overall. Every other model fell below 60%. Even more recent figures (according to the HuggingFace repo leaderboard updated March 2026) show Claude Opus 4.6 at ~51.6% and GPT-5.4 at ~50.3%, with most other systems in the 30–40% range. For perspective, these scores are strikingly low compared to typical benchmarks: basically half the tasks are failing for the top models. Why so tough? The combination of required skills is unprecedented. Agents must do sustained planning and recovery (if a search fails, try a different strategy), handle both vision and text inputs, manage multilingual content, intimately understand code and documents, and even exhibit common-sense safety. The leaderboard also shows that agent implementation details matter: simply swapping the agent framework (for example using OpenAI’s CLI vs Anthropic’s) can change scores by up to 18 percentage points, reflecting how sensitive these tasks are to tooling and execution environment. The bottom line: WildClawBench remains a very hard test. In their own words, the authors conclude that “long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models”. This is a call to the community that we are only at the beginning of training truly general agentic systems. What Makes WildClawBench Stand Out. Several design features distinguish WildClawBench from earlier agent or NLP benchmarks:. Realistic Tools and Environment: Unlike constrained contest sandboxes, WildClawBench runs agents in a real computing environment with actual tools. Agents use a real browser, file system, email/calendar, and other utilities to get tasks done. This mirrors the “realness” of robotics simulators or real robot tests. For instance, rather than providing an agent with a simulated file-write function, the benchmark gives a Linux container where the agent’s “vim” edits or “git clone” commands are literally executed. This means the tasks better reflect deployment — if it works in WildClawBench, it likely works in the real world. Multi-modal and Bilingual Content: The tasks are not just text-only. They include images, video, and audio processing steps (e.g. the video highlight task). They also appear in two languages (English and Chinese), forcing agents to handle multilingual inputs as part of their workflows. This multimodal breadth is ahead of most benchmarks, which remain text-centered. Multi-step Planning (“Agency”): As summarized by the authors, WildClawBench explicitly tests “agency” — whether an agent can do multi-step orchestration and autonomous planning. The table of “What We Test / Why It’s Hard” makes clear that agents must chain 10–60+ tool calls per task and adapt on the fly if something fails. This is very different from GPT-style question-answering tasks: it’s like telling the model, “Go do X” and expecting it to break X into steps, retry subtasks on failure, and decide on its own how to proceed. Long Horizons: Each task can occupy 10–20 minutes of actual runtime. To compare, many robot tasks or dialog tasks last only seconds or are factored. Here agents do rich workflows (e.g. negotiating meetings over multiple emails, or crawling and summarizing 50+ academic papers). That requires managing long histories and stateful reasoning. Reproducible, Containerized Execution: Every task is isolated in its own Docker container, with the exact same image for all agents. This means researchers can run the tasks on their own hardware and, in principle, get identical results to those reported. There is a strict separation of ground-truth answers from the agent’s view, eliminating sneak-peeks. Such reproducibility is more commonly seen in RL or robotics benchmarks (with fixed simulation seeds) but is less common in language-agent tasks — WildClawBench sets a new standard by providing all the machinery (Docker images, data, grading code) publicly. Public Leaderboard and Standardization: InternLM provides an open leaderboard of results, including average time and even API cost. For example, one leaderboard entry shows that Claude Opus 4.6 took 508 minutes and about $80 of API calls to score 51.6%. By quantifying time and money spent, the benchmark pushes for efficiency as well as accuracy. This mirrors robotics competitions that often log computation and energy usage. Overall, WildClawBench encourages a level of rigor and transparency that is akin to the best robotics and RL benchmarks. For ML researchers building general agents, it offers a clear, standardized way to compare methods. For roboticists, it’s a reminder that high-level planning and tool use (beyond physical motion) are also essential frontiers in embodied AI. Insights from Early Results. The initial evaluations on WildClawBench have already provided several insights about current AI agents:. Huge gap to human-level: Even leading models only solve about half of the tasks. In a static tabletop of tasks, this might have been acceptable, but in a real deployment, you’d expect a successful agent to do almost everything reliably. The fact that the best scores hover around ~60% means there are major ongoing challenges. Variability across tasks: Some tasks prove much harder than others. Anecdotally, agreement with the authors’ analysis, agents often fail at the long-horizon or safety tasks. For example, catching a hidden malicious instruction can be devastating: if the model is simply focusing on positive instructions, it might cheerfully follow a hidden trap. On the other hand, some productivity tasks (like basic summarization or search) are easier — but still error-prone under multitasking conditions. Detailed breakdowns by category (not publicly posted yet) likely show major variance. Multi-modality is tricky: Tasks that combine video and language (e.g. creating video highlights or generating images from descriptions) remain challenging. The benchmark results confirm that even with the best vision-capable LLMs, aligning modalities is error-prone. Agents might spot the right moments in a video, but generating a faithful text description is not guaranteed. Similarly, tasks that require precise visual output (like drawing an exact figure) often fail due to subtle model errors. Tool-use augmentation matters: Because agents here directly call tools (such as an actual code execution environment), failures often come from misunderstanding how to use the tools, not from “the environment is different.” For instance, if an agent misuses a CLI command or misinterprets a file path, it can cause the whole chain to fail. The benchmark has revealed that agents might need better integration between their in-language planning and the real code-tool environment. Cross-domain generality is weak: An agent that does well on code tasks may do poorly on social tasks or vice versa. This suggests that generalist models still have domain gaps. One striking finding the authors note is that switching harnesses (the underlying agent framework) shifts performance by up to 18 points. This means it’s not just the model, but how the model interfaces with the environment that matters greatly. In summary, the WildClawBench results paint a picture not of an occasional challenge, but of systemic difficulty. In some sense the most important result is: “there is a lot of work to do.” It sends a clear signal to the research community that building truly general, robust agents requires fresh ideas. How to Use or Extend WildClawBench. Given its open design, research groups interested in long-horizon or embodied AI can directly use WildClawBench as-is, or as inspiration. Here are some practical takeaways:. Reproducing the benchmark: All WildClawBench resources are publicly available. InternLM hosts the tasks and Docker image on HuggingFace (as a dataset) and GitHub. To run it yourself, you would download the Docker image ( etc.), load it, and then run the script provided in the repository. You supply a model (via an API wrapper or local binary) and a config, and the script will sequentially run each task and record scores. The output is an easily parsed JSON report of success rates. This means any new model or method can be benchmarked under identical conditions to the published results. Custom Agents: WildClawBench doesn’t lock you into one agent architecture. You can plug in any agent that can interact with the container command-line. For example, you could use LangChain, AutoGPT, or your own prompt-engineering loops. A simple setup is to let the LLM decide which tool to call next (e.g. “openclaw run browse google.com" or “openclaw run email send.”), and parse the response to continue. By comparing the scores, you can quickly see if your improvements (say, a better planning algorithm or a fine-tuned model) actually help. Building on the idea: Even if your interest is not in digital assistants or email, the WildClawBench methodology suggests how to create analogous benchmarks. For instance, a team working on physical robots might containerize a robotics simulator and script analogous tasks (like kitchen chores or warehouse sorting) with a grading suite. The key lessons — isolate episodes, hide ground truth, combine rule-based/outcome checks with semantic scoring, maintain a leaderboard — all apply. WildClawBench could serve as a template for those who want to stress-test more physically embodied AI with the same rigor. Cross-Embodiment Caution: One remark is that the term “cross-embodiment” is not really addressed by WildClawBench. That phrase usually refers to policies that transfer across different robot forms (like a policy learned on a 3-finger gripper arm working on a 5-finger hand). In the context of this benchmark, there is no physical embodiment at all. Thus, if your interest is cross-embodiment in robotics, WildClawBench is not directly applicable. However, the broader idea of building agents that generalize across contexts is related. In fact, the bilingual and multi-modal nature of WildClawBench does test a bit of “generalization” (e.g. language transfer), which could inspire similar cross-domain tests in robotics (like transferring between simulators, or between manipulation and navigation tasks). Leaderboards and Fairness: Researchers using WildClawBench might contribute their results to the leaderboard. The InternLM team seems to encourage adding new model entries. If doing so, remember that the reported scores can vary with random seeds and run conditions, so it’s best to average over several runs. Also, the benchmark currently focuses on closed-set tasks (with fixed data). One could imagine a dynamic variant where tasks change over time, but that would get tricky to standardize. Conclusion. WildClawBench is a timely new challenge at the frontiers of embodied AI. It recognizes that LLM/VLM agents — especially those intended to “do work” on behalf of users — must be tested in realistic, multi-step, real-world-like settings. By providing a comprehensive suite of long-horizon tasks (some quite sophisticated, like video clipping and secure file audits) in a reproducible Docker framework, it sets a high bar for current and future agents. The initial results are sobering: even the largest models fail on a large fraction of tasks. For robotics and ML researchers, WildClawBench is interesting both as a resource and a model. It shows how to structure complex evaluation tasks, and highlights what skills generalist agents still lack. Even though its specific tasks are digital, the principles — test across modalities, require error recovery, keep evaluation honest — are ones that resonate with evaluating physical robots as well. The “in the wild” spirit here is similar to taking robots outside the lab: unspecified dynamics, real tooling constraints, and unpredictable edge cases. In practical terms, the availability of this benchmark means any team with a new large model or agent policy can immediately try it out on these tasks. It provides a quick litmus test: if your new system can’t beat ~60% success, there’s work to do on planning and robustness. If it does beat that (for example, if a 2027 model scores 80%), that would be a major breakthrough announced in this same leaderboard. In the broader sweep of embodied AI research, WildClawBench may be seen as a harbinger. It pushes us to think of agents not just as question-answer or single-dialogue machines, but as autonomous workers in an ecosystem of tasks and tools. Perhaps future work will integrate even more realistic complexities (uncertain user feedback, adversarial environments, or even physical robots). For now, WildClawBench gives us a concrete foundation: a set of “wild” tasks and tools we can all agree on. Solving them will require advances in reasoning, memory, and cross-modal grounding. It’s a worthy, open-ended challenge — and a stimulant to innovation in the embodied AI era. References (from WildClawBench documentation): InternLM team, WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation, 2026 (available via HuggingFace and arXiv).