Latest / Embodied AI 101 / AgentFloor: A Benchmark for Long-Horizon Agent Planning

Transcript

AgentFloor: A Benchmark for Long-Horizon Agent Planning · Embodied AI 101

• AgentFloor: A Benchmark for Long-Horizon Agent Planning. Modern AI “agents” – systems that carry out multi-step tasks by invoking language models and tools – often rely on massive proprietary models for everything, even routine actions. But is that really necessary? A new study introduces AgentFloor, a 30-task benchmark designed precisely to answer this. In AgentFloor, tasks are arranged on a six-tier “capability ladder” from simple instruction following up to complex long-horizon planning under constraints. The benchmark was used to compare 16 open-weight models (from 0.27B to 32B parameters) against a frontier model (OpenAI’s GPT-5) in over 16,500 trials. A clear pattern emerged: small and midsize open models handle the bulk of routine tasks just as well as GPT-5, but the long-horizon planning tasks still favor the largest models. In fact, the best open model (Gemma 4’s 26B parameter “gemma4:26B”) achieved the same overall success rate as GPT-5 while running many times faster and cheaper. By contrast, even GPT-5 only solved about 10% of the hardest planning problems, leaving a big reliability gap. The upshot is a practical design vision: use tiny models for routine steps and reserve the heavyweight models for the rare, deep-planning cases. In this episode, we’ll dive into what AgentFloor does, why it was needed, and what it tells us about using small “self-hostable” models versus giant frontier models in real-world agent systems. We’ll explain the benchmark’s six tiers of tasks, how the experiments were run, and exactly where the performance line falls. We’ll also explore the cost and latency advantages of open models, the taxonomy of their failure modes, and what all this implies for building practical AI assistants. Along the way, we’ll connect AgentFloor to prior benchmarks (like AgentBench and AgentIF) that target related questions of planning and instruction-following. Why Agentic Planning Matters. Imagine a typical AI assistant pipeline: a user asks a complex request, the system calls an LLM (or several LLMs) to break it into steps, fetch information, apply rules, and write out an answer. In practice, most of those LLM calls are short, routine jobs – things like looking up a record, extracting a field, or doing a quick calculation. Yet many systems today feed every call to the same large model. That could be like using a supercomputer to turn on a lightbulb: wasteful and slow. What if we had evidence showing exactly which tasks do require full “frontier” intelligence and which can be offloaded to smaller models? That would let system designers save a fortune by routing the easy chores to lightweight models and only deploying the big ones when really needed. This is the gap that AgentFloor aims to fill. AgentFloor is specifically crafted to expose the boundary between “small-model-able” and “big-model-needed” tasks in an agent workflow. It features deterministic, abstract tasks (no randomness or web noise) so that the differences really come down to reasoning power and planning capability. The authors (Karmakar & Chatterjee, 2026) summarize it this way: AgentFloor “introduces a 30-task benchmark organized as a six-tier capability ladder” spanning instruction following, tool use, multi-step coordination, all the way up to “long-horizon planning under persistent constraints”. By running a slew of experiments – over 16,500 runs in total – they show a “clear boundary of model necessity”. In plainer terms, smaller open models can do the bulk of short-horizon, structured tool-usage work, but complex planning still gives the edge to the biggest models. We’ll unpack exactly what those tiers look like next. The Six-Tier Capability Ladder. At the heart of AgentFloor is the idea of a capability ladder. There are six tiers (A0 through E), each adding one new cognitive demand. Each task is played in a simulated “agent environment” with an in-memory database and a fixed set of abstract tools. This eliminates variability: there are no real web APIs, no changing world state, just clean, repeatable problems. Concretely, AgentFloor uses eight abstract tools – imagine functions like , , , , etc. – that operate deterministically on data fixtures. The tiers are as follows:. Tier A0 (Instruction Following): The simplest rung. The model gets a plain instruction (no tools needed) and must compute or retrieve the answer with a single step. This checks basic understanding of language instructions, for example parsing a query like “What is the capital of France?” or a simple multi-piece computation that doesn’t need external tools. Tier A (Single-Tool Use): Now one abstract tool is allowed. A typical task might say “Use the search_records tool to find a record matching X, then report one of its attributes.” Here the model has to successfully call a tool once and then finish. Tier B (Two-Tool Chain): This layer requires using two tools in sequence. For example, “search for a customer record with search_records, then feed that result into get_attribute to compute a final answer.” The output of the first tool becomes the input of the second. This tests whether the model can plan a mini workflow of two steps. Tier C (Branching): In Tier C the model must make a conditional decision mid-task. A task might say “Use search_records to find all orders. If any order is marked ‘urgent’, then use lookup_record on the highest-priority one; else, use lookup_record on the earliest one.” The model has to take an intermediate result and decide which tool to invoke next. Hence “branching” – a simple if/then structure embedded in the task. Tier D (Multi-Source Synthesis): These tasks involve gathering information from multiple sources (or records) and combining them correctly, possibly under some consistency constraints. For instance, it might say “Find the three cheapest flights on two different airline databases, then reconcile any differences and pick the overall cheapest.” The model needs to call multiple tools and fuse the outputs properly, including resolving conflicts or constraints. Tier E (Long-Horizon Planning): Finally, Tier E tasks require sustained planning over many steps, often 8–12 tool calls. A Tier E example could be something like managing a schedule under constraints: “Plan a 3-day conference track subject to constraints X, Y, Z on rooms and timeslots.” The model must make a sequence of decisions, keep track of constraints on which resources have been used, and not violate the rules, all without forgetting earlier parts of the plan. This is truly a “long-horizon” problem within this abstract environment. Each tier has 5 tasks, making 30 tasks total. Every task is deterministic and fixed so we get consistent scoring. The entire environment is in-memory: no web queries, no external APIs, no timers. This ensures that the only thing that varies is the model’s reasoning, planning, and tool-calling ability. The simplicity means we can run many evaluations consistently: 16,542 runs ended up being scored in this study. Why this structure? The idea was to isolate computational demands one at a time. Tier A0 tests pure language instruction following; A tests basic tool use; B adds multi-step chaining; C adds conditionals; D adds merging sources; and E adds multi-step planning with constraints. By ordering them from easiest (A0) to hardest (E), AgentFloor can pinpoint at which level each model starts to struggle. As one of the key takeaways puts it, this “capability ladder” reveals how far up the tool-use hierarchy each small model can go before it needs frontier power. Experimental Setup. The experiments behind AgentFloor were massive and meticulously controlled. In all, 16 open-weight language models were evaluated, ranging in size from 0.27 billion parameters up to 32 billion. These models include recent open-source releases from multiple labs (e.g. variants like Gemma-4, Ménal, Quartz, Qwen, and others). They were all run locally (via the LLM server Ollama) at temperature=0 and with identical system prompts, to ensure a fair zero-shot evaluation (except for deliberate ablations like toggling reasoning). Alongside them, the authors also tested a frontier model, which they label as GPT-5. Presumably this stands for a hypothetical next-generation GPT (the exact details aren’t confirmed), but it serves as a stand-in for a top-tier proprietary model like GPT-4 or beyond. GPT-5 was accessed via OpenAI’s infrastructure. So we have 16 “small” or mid-sized open models vs 1 big commercial-scale model. Each model was asked to attempt each task many times (with any permissible non-determinism disabled, to keep it repeatable). In total there were 12,000 runs in the main sweep (that presumably means 16 models × 30 tasks × multiple trials each) plus additional runs for ablation variants, giving 16,542 scored runs. Scoring was strict. Each run is judged as Pass/Fail by a Task Completion Rate (TCR) metric. To earn a pass, a model must produce exactly the correct final answer, follow any submission format rules, use exactly the allowed tool sequence, and not do anything forbidden (like hallucinating an extra tool or making an unsupported action). For example, if the task says “use tool A then B, then give the number,” failing to call tool B or calling a wrong tool would immediate fail. Many checks are purely symbolic (did it call tools in the right order?) and some semantic checks (like whether it truly satisfied all constraints) are done by a small deterministic judge LLM. The point is that AgentFloor doesn’t give partial credit – either you solved the task precisely or you didn’t. Importantly, they pre-registered a statistical analysis plan. They compare models in a paired fashion across runs, compute bootstrap confidence intervals, and even do equivalence testing (Two One-Sided Tests, TOST) with a ±10 percentage-point margin. This means, for instance, they can say “these two models are statistically equivalent up to 10 points difference” or “one is significantly better than the other”. They also correct for multiple comparisons (Holm–Bonferroni) when checking many models or tiers. In short, this isn’t just eyeballing the numbers – it’s a rigorous bench. Finally, models were also evaluated for efficiency. The team measured cost-per-passed-task and latency on typical hardware. For example, on a Mac Mini or GPU, how many dollars per successful solution? Or how many seconds does it take? These operational metrics let them say not just who is better, but who is cheaper or faster when matched at the same accuracy. Results: Where Small Models Can Reach. The headline result from AgentFloor is simple: small and midsize open models already nail most of the “easy” agent tasks, and the gap to GPT-5 only shows up on the hardest tasks. Across all thirty tasks, the best open-weight model (Gemma-4 26B) scored statistically the same as GPT-5. In raw terms, Gemma4:26B got about the same overall Task Completion Rate as GPT-5. Their paired difference was +0.4 percentage points in favor of Gemma, with a 90% confidence interval entirely within ±5 points. In other words, there is no significant difference on overall accuracy. Recall this is with strict pass/fail – Gemma just as often gave the right final answer as GPT-5 did, when averaged over every task. However, that overall parity masks where the differences really lie by tier. Let’s break it down tier by tier:. Tier A0 (Instruction following): Here Gemma4:26B actually beat GPT-5. It aced the tier with 100% correct, while GPT-5 scored only 80%. The authors note that GPT-5’s misses were typically trivial issues like rephrasing the instruction instead of following it exactly, which failed the strict checker. In any case, on pure instruction comprehension tasks, even modest models suffice. Tier A (Single-Tool Use): At this level, Gemma4:26B and GPT-5 were statistically equivalent. Gemma scored about 2.2 points higher (e.g. 97% vs 95%), but within the ±10pp equivalence margin. In other words, a ~26B open model did just as well as the frontier model when it could use exactly one tool. Tiers B/C/D (Chains, Branches, Multi-source): For the intermediate tiers, the results are also very close. Gemma4 was within about 9 percentage points of GPT-5 in every case, though the sample sizes (n=45 per tier comparison) weren’t large enough to prove formal equivalence at the 10% margin. The takeaway is that small models fall off a bit as the logic gets more complex, but not dramatically. Many of the open models stayed in roughly the same ballpark as GPT-5. No model was perfect, but the hardest failures were still rare. Tier E (Long-Horizon Planning): Here the frontier advantage emerges starkly. On the four planning tasks, GPT-5 managed about a 10% pass rate, whereas Gemma4:26B got 0%. (No open model scored above 0% on E.) In fact, nobody got more than 10% on any E task. These tasks turned out to be extremely challenging: the vast majority of chains of 8–12 tool calls ended in failure, usually because the model lost track of a constraint or prematurely gave up. Although GPT-5’s off-10% performance is far from reliable, it was still statistically superior to the best open model by over 10 points. So the pattern is clear: Up through tier D, open models handle the work, but at tier E you see a gulf. The highest tiers exposed a need for long-run planning stamina that small models just don’t (yet) have. Even GPT-5 struggled, highlighting how tough those tasks are. To put some numbers on reliability, the authors also looked at “smallest model to hit 80%”. In Tier A0 and A, pretty modest models made it. A 4B-parameter model (Nemotron-3-nano:4B) cleared 80% on the no-tool tasks, and a 3B (Ménal-3:3B) cleared 80% on the single-tool tasks. But for Tier B, the best a 2B model (Qwen3.5:2B) managed was about 70%, and no open model even got 60% on tiers C, D or E. In short, the cognitive “knee” seems to fall off after Tier B: routine single/double-step jobs are covered by even small models, but once you need complex coordination (C/D) or extended planning (E), open models’ accuracy drops dramatically. (The specific model names are interesting: “Gemma” comes from Google DeepMind’s recent 26B-model, “Qwen” is from Alibaba, “Mistral” is a Franco-German open model, and so on. The broader story is that a wide range of open academic and community models were tested, and all roughly followed the same pattern. The authors mention names like Gemma4, Qwen-32B, Mistral 3B, Granite4, etc., but the key is consistency of the trend.). Best Open Model: Cheaper and Faster. Beyond pure accuracy, AgentFloor lets us compare efficiency and cost. This matters a lot for production: if a model is a bit worse but orders of magnitude cheaper, that could tilt design decisions. The clear winner here is Gemma4-26B. Being a mixture-of-experts model, it’s relatively lightweight to run considering its 26B parameter “size” (it has only ~4B active weights at a time on A4B format), which makes it very efficient. The authors report that on a Mac Mini, it cost about $0.0022 per successful task, versus $0.0327 for GPT-5 – roughly a 15× cost advantage. Even on a beefy GPU instance, Gemma4 was about 3× cheaper than GPT-5 for the same accuracy point. And in latency, it wins too: a passed task takes ~16.1 seconds on Gemma4, vs 40.8s for GPT-5. In short, you get roughly the same success rate but at a fraction of the cost and time. Other smaller models were even cheaper per task (just not as accurate). For example, a tiny 3B model (“granite4-3B”) cost only $0.00046 per pass on a Mac – 71× cheaper than GPT-5 – though with lower accuracy (~40% overall TCR). This underscores a theme: on these tasks, accuracy and cost trade off smoothly. The cheapest model with any real accuracy will solve, say, enough Tier A0/A quickly and cheaply, while the pricier models pick off the harder cases. The authors found that in aggregated performance vs cost, the open models form a Pareto frontier containing all points; GPT-5 is just dominated by a faster open model at every accuracy threshold. This implies a big economic insight: A hybrid system using an open model for routine calls and calling GPT-5 only for the toughest problems could be orders of magnitude cheaper than naively sending everything to GPT-5. If 80–90% of daily queries are short tasks (as the paper suggests), companies could massively cut costs by deploying self-hosted models like Gemma or Qwen for those, and invoking the expensive API only on the rare, complex jobs. Failure Modes and Attempts to Fix Them. When models did fail, the study categorized the reasons in a neat taxonomy. The dominant mode (F1) was hallucinated-tool calls: the model would mention a tool that wasn’t actually in the allowed list. (For instance, saying “I will now check inventory” when no tool exists would fail.) Next was F2, malformed tool calls (wrong format), then F4 step-budget exhausted (ran out of allowed steps), F5 early resignation or splitting plan without execution, F6 wrong tool chosen, F7 partial completion, and finally F3 residual miscellaneous errors. Interestingly, hallucinating a tool (just being incorrect about what tools are available) was by far the most common failure mode, so controlling that is key. Faced with the big failure on Tier E, the authors also tried some targeted interventions (ablation tests) to see if they could raise model performance. For example, they gave GPT-5 extra time (doubling its step budget for one task) or tried a more explicit prompt style. The results were uneven and very model-specific. For instance, appending an “explicit submission” instruction helped a Mistral-8B model jump from 0% to 100% on one E-task (E1) by simply clarifying the expected format, but that manipulation did nothing for other models that were failing earlier in the process. Giving GPT-5 more steps only helped on E1 by about +33 points, because E1 was genuinely tight on steps; other E tasks still failed for different reasons (like confusions or tool errors). Turning off a special “reasoning mode” in Qwen3-32B helped that one model on Tier B but actually hurt its 8B and 14B relatives on the same tier. In short, there was no silver bullet – no simple prompt hack that universally closed the gap. Each model had its own quirks. The intervention experiments led to a key insight: the boundary is not just about scale. It’s not that “just make the model bigger and everything works.” Instead, different models break differently, and no single tweak fixes all of them. That underlines how brittle long-horizon planning remains even for top LLMs: they struggle in distinct ways, and you need substantial work (or maybe different architectures) to solve those. Implications for Designing Agent Systems. What does all this mean in practice? The authors emphasize it as a cost-and-performance map for real agent design. Their results suggest a primal design principle: route each agentic sub-task to the smallest model that can do it well. In other words, send the short, structured jobs (instruction following, single-tool lookups, etc.) to open-source models, and reserve calls to big APIs only for the narrow cases that demand them (long-horizon planning under multiple constraints). In such a hybrid pipeline, the bulk of work (say 70–90% of calls) could be handled by tiny local models. The benefits would be huge: much lower dollar costs, much faster latencies (imagine half-second responses instead of multi-second on routine queries), and even reduced server load on paywalled APIs. The authors quantify this: matching GPT-5 accuracy with Gemma happens at roughly one-fifteenth the API cost. They point out that even comparing moderate open models (like an 8B or 26B) to GPT-5 yields orders-of-magnitude cheaper cost per successful task. In practice, that translates to material savings for companies running high-throughput agents. Of course, a hybrid system needs to know when to escalate to the big model. The failures in Tier E aren’t absolute proof that small models can’t do planning, just that they didn’t on these 0-shot tasks. One could imagine having a classifier or rule that if a job looks like “multi-step constrained plan”, forward it to GPT-style agent. The kicker is that even GPT-5 isn’t great at those E tasks, so you’d likely still need human review or some other fallback for critical planning tasks. The paper also highlights an economic flip side: if companies only needed GPT-size power for a small fraction of tasks, the value of “frontier” APIs might shift. Right now many charge a premium that assumes heavy usage. If usage patterns shift to hybrid calls, pricing models might change to reflect that the marginal value of the API calls is limited to hard planning tasks. Similarly, it implies growing demand for high-quality open checkpoints and orchestration tools – things like better pipelines for self-hosting, local GPU inference stacks, etc. The AI ecosystem could tilt more towards serving multiple model sizes, not just one monolithic service. Finally, AgentFloor suggests research directions. The glaring failure of open models on Tier E tasks identifies a sweet spot for improvement: if future small models (or multi-model pipelines) could crack robust long-horizon planning, the returns (in capability and cost savings) would be immense. For example, integrating planning modules, memory mechanisms, or recursive decomposition ideas might help. Moreover, the benchmark itself – being open-source – acts as a testbed for such innovation. The authors have released the full benchmark, code harness, and results so others can reproduce and extend this analysis. Teams could try fine-tuning a model just on AgentFloor tasks, or plugging in a chunking strategy, to see if they can push down the Tier E gap. Context: Other Agent Benchmarks. AgentFloor joins a growing family of “LLM-as-agent” benchmarks, but with its own niche. For context, consider a few others:. AgentBench (ICLR 2024) was one of the first large efforts in this vein. It created a multi-dimensional suite of tasks across 8 environments (like game simulations, database puzzles, web navigation) to test LLM agents. AgentBench found that top commercial models could often act capably, but many smaller open models lagged far behind on complicated tasks. Crucially, it identified long-term reasoning and multi-step planning as common failure points in LLM agents. These findings aligned with AgentFloor: indeed, AgentBench noted “poor long-term reasoning … decision-making” were majors hurdles. However, AgentBench was more about diversity of environment types (games, operating systems, knowledge graphs, etc.) and used many tasks, whereas AgentFloor zeroes in on a gradient of planning difficulty in a single, abstracted environment. AgentFloor’s value is in precisely gauging where the tipping point is between “small model OK” and “big model needed” in that ladder. AgentIF (ArXiV 2024) is another benchmark (from Tsinghua-AAAI) focusing on LLM instruction following. It assembled 50 real-world agentic instructions (say, business processes) that were extremely long (thousands of words) with multiple constraints. It measures, can the LLM take a complex instruction and produce a valid plan or answers satisfying all constraints? AgentIF’s angle is more on constraint-following (with metrics like “Constraint Success Rate”) rather than sequential tool calls. Still, both AgentIF and AgentFloor are concerned with how well LLMs follow detailed procedures. One could imagine combining them: e.g. after a model generates a plan via AgentIF, check it by simulating in a tool-based environment like AgentFloor. They’re complementary. Others: There are smaller projects like ALFWorld tasks (grid worlds) or benchmarks for web agents or thing, but many focus on environments with randomness or real interfaces. AgentFloor’s innovation is the controlled determinism and focus on cost/capability trade-offs. It is reminiscent of classic “algorithmic reasoning” challenges, but recast as an agent problem. The picture is this: Across many benchmarks, long-horizon planning and multi-step coordination repeatedly emerge as the hardest nut to crack. AgentFloor confirms that pattern in a systematic way. It adds new insight in its economic/cost framing: not only do all models struggle with planning, but smaller ones struggle trivial tasks even less than we thought, so the cost/benefit calculus strongly favors small models except in the planning regime. Limitations and Caveats. It’s important to note where AgentFloor is not telling the whole story. The authors themselves stress that this is a highly controlled benchmark on synthetic tasks. Real-world agent pipelines involve messy factors not modeled here:. No web or GUI: AgentFloor has no Internet, no web pages, no unstructured text. It’s all structured database records. In reality, agents often have to parse emails or websites or images, and such grounding could change which model is needed. No API drift or latency: The tools are always available and behave consistently. Actual APIs can fail or change. Also, the overhead of orchestrating many small calls might erase some of the latency/memory advantage. No prompt exploitation or safety filters: In a real system, some models might hallucinate or require content filtering. The benchmark is optimistic about reliability, except for its strict checks. Fixed step budgets: The researchers limited Tier E to 10 steps, for instance. If a system could iterate more or do backtracking, performance might change. (They did try doubling steps on GPT-5, which helped a bit, but there might be other ways to extend planning.). English language constraints: It appears all tasks assume fluent English instruction. Models might vary more with other languages or multimodal data. One frontier model: The study only used GPT-5 as “the big shark”. Perhaps other big models (Claude, Llama3 70B, etc.) might behave differently. The absolute gap might change, though the general trend likely holds. Zero-shot only: All runs were zero-shot aside from system prompts. Finetuning on these tasks or giving a few examples might change a model’s success rates. For instance, maybe fine-tuning a small model on similar tasks could let it scale farther up the ladder. Given these caveats, AgentFloor doesn’t prove how open models will actually behave in the wild, but it sheds a lot of light on intrinsic capability. It’s a starting point. As the authors put it, it’s a capability-and-cost map that system builders can use for rough decisions. In practice, one should validate any routing strategy in situ. Looking Ahead. The AgentFloor benchmark with its open repository is now available as a tool for the community. It not only diagnoses the status quo but can serve as a testbed for improvements. For example:. Improving Tier E performance: Since even GPT-5 tops out low, there is plenty of room to innovate. One could try chain-of-thought prompts, planning-specific fine-tuning, modular planning (decompose a problem), or hybrid architectures (LLM plus symbolic planner). Getting even one open model up to 50% on an E-tier task would be a big breakthrough. Rerouting research: With evidence that Tier A/B tasks don’t need huge models, we might invest less in scaling up those capabilities and more in niche areas like complex reasoning. As [the authors suggest], reducing small-model failures on long-horizon plans “could unlock large economic returns”. Extended benchmarks: The AgentFloor concept could be expanded. For example, adding some real-world noise (varying data, partial observability), or linking multiple tasks in sequence (model outputs feeding into later tasks, mimicking dialogues). Or benchmarking under stressed compute constraints. Routing systems design: Practical AI products might embed the findings into their architecture: e.g. building a “model router” that chooses which LLM to call based on the query complexity. This is similar to techniques in ensemble modeling or multi-expert systems, but now with direct empirical data guiding it. New models: Finally, as the frontier moves (GPT-5, 6, or others), one could rerun the benchmark to see if the gap closes. If a future model solves 90% of Tier E, how would that shift the conclusions? Conversely, what about multi-model systems (LLM+vision, LLM+symbolic)?. In summary, AgentFloor provides a granular look at long-horizon planning ability. It confirms that today’s big LLMs still hold the edge on complex plans, but also that all-stars among the new open models can match them on everything else. For AI architects, it means there’s a large operational win in using open models wherever they work. The hard problems stay hard for everyone, but they also occupy a smaller slice of the workload. Charting out which tasks sit in which slice – that’s the map that AgentFloor paints. As one industry commentary put it, the benchmark “addresses a practical production problem: most agent workflow calls are short and routine, so optimized routing could significantly reduce cost and latency without sacrificing performance”. For engineers and researchers building the next wave of AI agents, that insight could be a game-changer. Sources: The above is based on Karmakar & Chatterjee’s AgentFloor (arXiv:2605.00334), as summarized in sources and related commentary.