Latest / Embodied AI 101 / Claw-Eval: Toward Trustworthy and Transparent Evaluation of Autonomous Agents
Transcript
- Claw-Eval: Toward Trustworthy and Transparent Evaluation of Autonomous Agents. As large-scale language models (LLMs) move from static question-answering tasks into real-world agentic roles, we need a radically different way to measure their performance. No longer are we satisfied with one-shot question answering or code generation; modern AI agents must do things in sequential, multi-step workflows, often in dynamic software environments. Claw-Eval is a new benchmark suite designed precisely for this setting. Developed by Ye et al., it targets the challenge of evaluating autonomous agents end-to-end – not just checking whether the final answer is correct, but whether the entire process was reliable, safe, and robust across repeated trials. In traditional LLM benchmarks (MMLU, HumanEval, etc.), we typically verify that the model “knows” the right answer or code snippet. But agentic use cases are fundamentally different. A simple email-sorting task isn’t just about generating a list of labels in English; it requires integration with a mail client, parsing individual messages, handling edge cases (e.g. spam or urgent notices), and possibly making multiple API calls. The key questions are planning, action execution, and consistency – each of which can go wrong in subtle ways that a naive final-output check would miss. Claw-Eval explicitly tackles this by creating realistic, human-verified multi-step tasks and by recording every agent action so we can grade the trajectory, not just the outcome. In this analysis, we’ll unpack the design of Claw-Eval and why it represents a significant step forward for agent evaluation. We’ll explain its task suite, its scoring metrics (including a novel Pass^k stability score), and the major insights from evaluating state-of-the-art agents on it. Along the way, we’ll compare to other agent benchmarks and argue that Claw-Eval’s emphasis on transparency, safety, and reproducibility highlights the path to truly trustworthy AI agents. Why Robust Agent Evaluation Matters. Imagine an AI agent living inside your computer. It can open your calendar, read emails, browse the web for information, and even execute code on your behalf. Over twenty sequential steps, it might fetch attachments, analyze PDFs, write a summary document, and send an email. At the end, we only see the final outputs (say the email it sends, or the file it produces). But what if halfway through the workflow the agent went off-script? Perhaps it temporarily accessed the wrong file, or it mishandled an exception, or it even took an unsafe action like deleting some data and then recovered. A traditional benchmark that only checks the final answer would not notice those missteps. Ye et al. argue that most existing agent benchmarks suffer from three serious shortcomings:. Trajectory-opaque grading: Benchmarks typically verify only the final output of a workflow, paying no attention to how the agent arrived there. If the agent took shortcuts, made mistakes, or even violated constraints along the way, those errors can pass undetected as long as the endpoint looks right. Limited safety and robustness testing: Most tests simply label a run “done” or “not done.” They do not explicitly check for unsafe behavior (like writing to unauthorized files, leaking data, or other policy violations) nor measure how sensitive the agent is to small perturbations. An agent might randomly stumble on the right answer once, but fail dramatically on a slightly different version of the task. Narrow scope of tasks and modalities: Many benchmarks focus on text-only question answering, code puzzles, or academic puzzles (math problems, logic quizzes, etc.). These do not capture the richness of real-world agent tasks, which often involve multimodal inputs (web pages, images, documents, video) and tools (APIs, interfaces, databases). To paraphrase Ye et al., the key question is shifting from “Does the model know the answer?” to “Can the model actually do the job reliably in a live environment?”. Claw-Eval was built to answer exactly that second question. The Claw-Eval Benchmark Suite. Claw-Eval (short for Computer-Logging Agent Workflows Evaluation) is an end-to-end testbed featuring 300 tasks spread across 9 categories in three broad groups:. General Service Orchestration (≈104 tasks): These are standard office and productivity tasks – e.g. scheduling meetings, processing emails, handling to-do lists, making calculations, or managing files. The agents interact with simulated versions of real services (like a calendar API, email client, file system, spreadsheets, and so on). Multimodal Perception & Generation (≈35 tasks): These tasks require the agent to use tools on images, web documents, or even ask questions about short videos. For example, a task might involve generating a formatted web page from a sketch, answering questions about the content of an image or a video clip, or extracting structured data from a PDF report. Multi-turn Professional Dialogue (≈161 tasks): In this group, the agent engages in extended dialogues or consultations – such as playing the role of a customer support assistant, a medical scribe, or a project manager in a meeting. These tasks go beyond single-turn Q&A, instead evaluating how well an agent can maintain context, ask clarifying questions, and follow through on instructions over many dialogue turns. Each task is human-verified. Rather than trusting another LLM to judge correctness, the Claw-Eval designers have hired people to write the tasks and rubrics and to inspect the results. This avoids the kind of biases that “LLM-as-judge” setups can introduce. In practice, tasks were drawn from realistic scenarios (many inspired by internal enterprise use-cases) and carefully vetted. Recorded Evidence and Rubrics. A hallmark of Claw-Eval is its insistence on full transparency of the agent’s execution. Every action the agent takes is recorded through three independent channels:. Execution Traces: A log of every tool call or code execution command the agent issued. This shows exactly what it tried to do and in what order. Audit Logs: Records of any protected or sensitive operations (e.g. system-level accesses), serving as a safety checkpoint for unauthorized actions. Environment Snapshots: Periodic snapshots of the simulated environment or filesystem so that the grader can see intermediate states (for example, the contents of a folder after each file operation). These “flight recorder” logs allow graders to perform trajectory-aware grading. Instead of only judging the final answer, Claw-Eval applies detailed rubrics to the agent’s entire reasoning trace. In total there are 2,159 fine-grained rubric items across all tasks. Each rubric breaks a task into pieces: primary objectives and secondary considerations. For instance, a meeting-scheduling task might have separate rubric points for (a) listing conflicts in participants’ calendars, (b) identifying free time slots, (c) selecting an optimal time and explaining why, and (d) giving the calendar event a sensible title. An agent might only partially satisfy these criteria, and the rubric allows partial credit or targeted feedback. This approach contrasts sharply with end-only checks. To illustrate: a naive check might simply verify that an event was created on Monday, 14:30. But without trajectory logs and rubrics, it would miss that the agent never actually checked anyone’s calendar conflicts or misunderstood a participant’s availability. Claw-Eval’s human-designed rubrics explicitly penalize such omissions, ensuring that the agent’s process is as important as its product. (A sample rubric excerpt from the data: for scoring a scheduling analysis, a perfect score requires “complete analysis (conflicts, free slots, rationale) and an appropriate title,” whereas anything less is graded down accordingly.). In general, Claw-Eval’s grading engine computes scores along three dimensions for each task:. Completion: How well did the agent achieve the stated task goals? This essentially measures correctness and completeness of the outcome (based on rubric items). Safety: Did the agent stay within allowed actions and avoid any disallowed or dangerous behaviors? If any safety violation occurred, it can zero out the score entirely, since unsafe execution is a blackout failure. Robustness: How stable is the agent’s performance under slight perturbations or “edge case” inputs? This captures consistency. For example, an agent that fails completely if the date is changed by 1 day is less robust (and thus scored lower) than one that adjusts gracefully. These dimensions are combined into an overall task score, typically computed as something like (0.8\times\text{Completion} + 0.2\times\text{Robustness}) (the exact weights come from the rubric design), all multiplied by the Safety indicator (which could be 0 or 1). In the Claw-Eval technical report, they describe it succinctly as. meaning the agent must score at least 75% on that weighted rubric to “pass” a task. The priority here is clear: even a high completion score is rendered moot if safety = 0, enforcing that agents must be safe above all. Multi-Run Consistency: Pass^k Scoring. Perhaps the most distinctive feature of Claw-Eval is its Pass^k metric, especially the default. Traditional benchmarks often evaluate a model by running it once (or a few times) and averaging success rates. Claw-Eval instead demands strict stability. A task is counted as passed under only if the agent successfully completes the task in three independent consecutive trials. In other words, an agent needs to “prove it” by doing the job three times in a row, each with a new random seed or slightly perturbed environment. The motto is: “one success might be luck; three in a row shows true capability”. By comparing Pass^k to the more lenient Pass@k metric (the percentage of tasks passed in at least one of k tries), Claw-Eval explicitly distinguishes raw ability from reliability. If an agent sometimes fails a task (say due to picking the wrong hyper-parameter in one try, or a timing glitch), that fragility will hurt its Pass^3 score even if it can occasionally succeed. Focusing on Pass^k reflects the benchmark’s goal: we care about deployable reliability, not just trivia skill. In robotics terms, it’s like running your grasping algorithm on the real robot several times and requiring it to succeed each time before waving the green flag. This emphasis on “stability through triple consistency” is, as the authors say, “the most critical hurdle an agent must clear to move from a prototype to production”. Creating Realistic Agent Environments. Building Claw-Eval meant more than just writing prompts. Ye et al. took care to simulate realistic software environments where agents can take actions, just as a deployed agent would. They implemented a set of mock services and APIs (about 15 of them) to stand in for things like calendars, email servers, web browsers, file management, financial calculators, and more. These services have clearly defined inputs and outputs so that an agent can “call” a function (for example, a API) and see concrete effects (the calendar is updated). Critically, each test is run in a sandboxed Docker container to isolate it. This prevents, say, one task’s modifications from affecting the next task. Each trial starts from a clean state, ensuring that results are reproducible and consistent across runs. The sandbox also means agents can be safely allowed to run code (for searching or data processing) without touching the real host system. Essentially, Claw-Eval provides a trusted playpen for the agent to dutifully execute steps. Every task and outcome in Claw-Eval has been manually verified by humans, both to avoid bugs and to ensure the rubrics are correct. In practice, this means the task creators themselves or human graders have run through the scenarios to confirm that scoring rules accurately reflect good and bad behavior. By contrast, some benchmarks rely on LLMs to self-critique (automated grading), which can introduce circular errors or bias. Claw-Eval deliberately avoids that pitfall: the price paid is more upfront effort, but the payoff is trust in the evaluation. Finally, to promote transparency and ease of use, the creators have openly released the tasks, rubrics, and code under a permissive license. There is even a public leaderboard and GitHub repository where anyone can submit agent results. This open approach makes Claw-Eval not just a one-off study but a community resource for developing and comparing agents. Comparing to Other Agent Benchmarks. Claw-Eval joins a growing field of benchmarks for AI agents, but with a distinct philosophy. Let’s contrast it briefly with some notable predecessors:. AgentBench (Liu et al., ICLR 2024) provides interactive environments like a simulated OS or game to test an LLM’s reasoning. It focuses on multi-turn reasoning puzzles, but its scoring is mostly end-goal oriented (did the agent solve the puzzle?). It does not emphasize recording the full action trace or repeated trials by default. WebArena (Zhou et al., 2023) is a work-memory for evaluating web tasks (shopping site navigation, forums, code wikis). It simulates a web environment with templates and checks whether the final goal is met, such as finding a product or updating a post. Again, the primary metric is success or failure, not explicit safety checks or multi-run stability. GAIA (Mialon et al., 2023) is a large set of human annotations for assistant-like tasks that include images and files. It mixes text and multimodal questions and is more of a question-answering challenge (with increasingly complex steps). GAIA highlights broad skills (reasoning + tool use) but does not put agents into a live environment; you answer questions about attached contexts, not actually calling tools or modifying files. MINT and ToolBench are frameworks focusing on LLMs calling tools (often via Python code execution). They emphasize chaining reasoning steps but also typically judge final outputs. In short, many existing benchmarks are valuable but differ in scope. Claw-Eval is unique in blending a realistic execution environment with rigorous safety/robustness scoring and emphasizing repeatability. It takes the view that an “agentic agent” is more like a small software system than a simple function: we should record its system logs and failure modes just as we would for a distributed software service. Results: What Claw-Eval Revealed. The authors ran a range of “frontier” language models through Claw-Eval – both open models and strong commercial ones – to see how they fare. Some of their headline findings are enlightening:. Opaque Evaluation Misses Failures: If one only checked final answers (as in traditional benchmarks), a large fraction of failures go unnoticed. By comparing Claw-Eval’s full-pipeline grading to a naive final-check, they found that about 44% of safety violations and 13% of robustness failures that Claw-Eval caught would have been missed by an end-only check. In practice, this means many agents could “cheat” the test by stumbling on the right answer once while violating policies or performing shoddily behind the scenes. The richer logging in Claw-Eval exposes these hidden issues. Reliability vs. Raw Capability: When the authors deliberately injected minor errors or randomness into the tasks (for example, perturbing task inputs or environment variables), they saw that agents’ average performance scores remained largely unchanged, but their stability dropped significantly. Concretely, the Pass^3 score fell by as much as 24% under these perturbations, while the one-run Pass@3 stayed about the same. This confirms the intuition: slight noise can make a roughly 75% agent succeed sometimes and fail other times. Only the triple-run metric exposes that unreliability. In other words, some agents had a high ceiling but inconsistent execution – Claw-Eval clearly penalizes that. Variation Across Modalities: No single model dominated all task categories. In particular, most models did much worse on video-based tasks than on document or image tasks. This isn’t surprising: video understanding is a known weakness of current vision-language models. But it underlines that an agent’s score can be very modality-dependent. The top performer on text scheduling tasks might flounder on a multimodal webpage-generation task. The dataset revealed these weaknesses quantitatively. Some specific numbers (from their leaderboard) exemplify these trends. For instance, among open models, GLM-5 and Step 3.5 Flash were neck-and-neck on one-run success (Pass@3 around 70%), but GLM-5 slightly outpaced Step on triple-run stability (Pass^3 57.7% vs 56.7%). Notably, Step 3.5 Flash had zero variance in safety scores over runs – it consistently avoided violations – whereas others had slight variance. The Claw-Eval leaderboard even charts pass rate versus speed and cost, highlighting how an efficient model with decent reliability (like Step 3.5 Flash) might be preferable in practice to a slower model with marginally higher peak accuracy. In qualitative terms, Claw-Eval’s authors point out that today’s agents are often “good enough once,” but not yet good enough every time. For example, an agent that chains tools together might overfit to one particular phrasing of a request but shrug if the wording changes slightly. Claw-Eval’s multi-trial regime identifies exactly those brittlenesses. The takeaway is that “reliable capability” – doing the task dependably – is not the same as maximum capability in an ideal case. Another insight: The scores on Completion (being capable of finishing tasks) were generally high for top models, but scores on Robustness and Safety were where the gaps appeared. Many systems could be pushed into a wrong answer by stressful conditions (low robustness) or step off-limits under certain circumstances (safety). Finally, having the rubric outputs allows a fine-grained breakdown of where agents fail. The authors note anecdotes like agents that consistently got the logic right but ignored a minor rubric item, or agents that hallucinated confidence but actually misunderstood a step. All these patterns are visible once you have the action logs and detailed scores organized. Implications for Agent Development. Claw-Eval is itself a tool, but it’s also a mirror held up to the field. What lessons can we draw for those building or deploying AI agents?. Design for Transparency. In any critical application, you want visibility into what your agent did, not just whether it produced a result. Claw-Eval shows that tracking execution traces and environment state is feasible and immensely valuable. Practically, this means agent frameworks (the orchestration code you write) should log intermediate actions and check for policy violations at each step, much like how microservice logs are monitored in distributed systems. Test for Edge Cases and Variance. It’s not enough to benchmark once. When developing an agent, you should randomly vary some inputs and test repeatedly, looking for fragile points. Claw-Eval formalizes this via Pass^k, but the general idea is applicable anywhere: try your agent in slightly different simulated conditions and ensure it still works. If not, that’s a cue to improve its error-handling or reasoning robustness. Prioritize Safety Filters. The fact that Claw-Eval effectively throws out any trial with a safety violation (no matter how well it otherwise did) is a strong statement: we should do the same in real systems. It’s better for an agent to gracefully refuse a questionable action than to carry it out just for the sake of “completion.” For example, if an agent is automating your bank spreadsheets and it tries an illicit API endpoint, the process should immediately abort and reset. The authors’ data suggest many agents still make unsafe moves occasionally, so a strict safeguard layer is critical. Embrace Human-in-the-Loop Checks. Claw-Eval’s rubrics underline that judged tasks often involve subjective or context-dependent criteria (like style, completeness of reasoning, or making a polite tone). Fully automating such judgment is hard. In practice, agents in high-stakes settings might need human oversight or at least human-vetted reward signals. The rubric construction process used in Claw-Eval can inspire development of better automated evaluators (including other LLMs used as “critics”), but it also acknowledges that humans are often the gold standard. Diversify Training and Validation. The multimodal results remind us that training only on text data will leave agents weak on images or video. A modern agent should, if possible, draw on specialized vision models or incorporate vision-language datasets during training. Likewise, localization (here, Claw-Eval uses both English and Chinese tasks) is important. Models that excel in English text but lack Chinese capabilities would drop in bilingual evaluation. Real-world agents often need to handle whatever language the user prefers. Set Realistic Benchmarks for Deployment. Ultimately, when a company says “our agent passed X% of tests,” Claw-Eval suggests they should clarify what that really means. A 90% one-run pass rate might translate to only 70% pass in production if anomalies occur. Stakeholders need to understand the difference between “works if conditions are ideal” and “works dependably.”. Access and Future Directions. Claw-Eval itself is available for the community. The full task set, service implementations, and scoring pipeline are open-sourced (with MIT license) on GitHub and via Hugging Face datasets. Code and instructions show how to hook up an agent (via an API or sandbox) to this benchmark. A public leaderboard lets researchers submit results from new models. (At the time of writing, open-source models like GLM-5, Qwen, Xiaomo, and others have already been evaluated with Claw-Eval, and you can compare your agent to those baselines.). Beyond benchmarking, the Claw-Eval study articulates some promising research directions. For example, the sharp drop in Pass^3 relative to Pass@3 underlines a need for agents that self-correct mid-stream. Mechanisms like maintaining a memory of what went wrong last trial, or verifying each action with internal critics, could boost consistency. The safety violation rate suggests incorporating stronger in-built guardrails, such as model-based verification agents that veto dangerous plans. And the modality gaps indicate a hybrid approach – using best-of-breed vision models for image/video questions and LLMs for text – may be required in future systems. Moreover, one could imagine extending Claw-Eval in various ways: adding more categories of real-world tasks (say, simulation-based tasks in physical robotics or IoT device control), stress-testing adversarial scenarios (what if the environment degrades?), or scaling to even longer workflows. Its core idea – full trace auditing combined with robust scoring – will be applicable in those expansions. In the broader context of agent research, Claw-Eval aligns with a trend toward “embodied” or “bridged” AI evaluation, where models are judged by their effects in an environment (be it virtual or physical). Roboticists will recognize the parallel: just as we benchmark a robot arm by letting it run in a physics simulator and checking for collisions, Claw-Eval lets an AI “arm” loose in a software simulator and checks for missteps. Both emphasize that a successful deployment requires more than a good final snapshot – it requires safety and accuracy at every step of the journey. Conclusion. Claw-Eval represents a major step forward in how we validate autonomous AI agents. By providing transparent, human-verified tasks and demanding consistent, safe performance across trials, it pushes models to be not just smart, but dependable. Early results show that current “state-of-the-art” agents still have a ways to go on these fronts. Yet the clarity of failure modes that Claw-Eval provides will help guide improvements. As AI systems continue to integrate deeper into real tasks, benchmarks like Claw-Eval will be indispensable. They remind us that success on paper is not enough; trustworthiness in action requires a new kind of rigorous testing. Future agent developers can take heart: a benchmark that catches hidden flaws is the first step toward closing those gaps. Claw-Eval thus highlights the path to agents that are not only capable but reliably deployable – which must be our ultimate goal. Sources: The description above is based on Ye et al., “Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents”, arXiv:2604.06132 (2026). Their work is complemented by community write-ups [15] and detailed task data [22], as well as related benchmark references [43][44]. Key points are drawn from these references to highlight Claw-Eval’s design and findings.