Latest / Embodied AI 101 / EBench: A Diagnostic Benchmark for Generalist Manipulation Policies
Transcript
- EBench: A Diagnostic Benchmark for Generalist Manipulation Policies. Modern robotics increasingly explores generalist manipulation policies – large vision-language-action (VLA) models trained on broad, diverse datasets – with the hope of robots that can handle a wide variety of tasks in unstructured environments. Recent papers like π₀.₅ (a multi-modal “foundation” policy) and commercial efforts like Qwen-RobotManip have pushed overall success rates on benchmark tasks higher and higher. But as Chris Paxton notes, the era of reporting a single success‐rate “score” for a robot policy is coming to an end. Two policies can achieve the same overall success percentage and still behave utterly differently – one might excel at navigation and coarse manipulation, another at dexterous tasks, etc. In short, a single number hides as much as it reveals. Recognizing this, EBench proposes a “CAT-scan” style benchmark that peels back the layers: it evaluates robot policies across many axes of difficulty and generalization, yielding a detailed capability profile rather than one summary metric. At its core, EBench is a large-scale simulation benchmark for embodied manipulation. It spans 26 tasks drawn from everyday scenarios, ranging from kitchen chores to light industrial settings: for example, sorting bottles into baskets, loading a dishwasher, making a sandwich, or performing a shop checkout scan. These tasks are grouped into three families: multi-stage long-horizon sequences on a mobile robot base, mobile pick-and-place tasks across rooms and shelves, and fixed-table dexterous tasks requiring fine control. (All 26 task motions are rendered in photorealistic physics via NVIDIA’s Isaac Sim.) The key innovation is that every task is annotated along five orthogonal dimensions of capability and evaluated under four axes of distribution shift. This multi-dimensional taxonomy turns one benchmark into a dense grid of signals about what a policy can – and can’t – do. For instance, tasks are labeled by Operating Mode (mobile robot vs. fixed table-top), Horizon (short vs. long sequences), Precision requirement (low/medium/high tolerance), Atomic Skill (verbs like “pick”, “insert”, “pour”, etc.), and Scene (different indoor environments). Cross these with four generalization axes – novel Objects, novel Backgrounds (environments), paraphrased Instructions, and novel Compositions of sub-tasks – and you get a powerful diagnostic matrix. Because of strict object-level train/test splits, EBench forces models to encounter new instances at test time, revealing who truly generalizes and who merely memorizes. For example, evaluations are partitioned into Validation-Train (seen scenes and tasks), Validation-Unseen (familiar tasks in new configurations), and a Test-Mini set that isolates each type of out-of-distribution challenge. Plotting a model’s performance on validation vs. held-out test immediately highlights its overfitting: EBench can draw “val/test migration” curves showing exactly how much success drops when objects or contexts change. Crucially, the benchmark does not hand out just one score. As Paxton puts it: “Every robotics benchmark gives you a number. EBench gives you a profile.”. After running, the evaluation produces a rich report: a five-dimensional “ability fingerprint” (performance broken down by operating mode, horizon length, precision, etc.), transfer curves from train to test, radar charts over generalization shifts, and even a per-task heat map of success rates. This avoids the “overfitting game” where one model might look great by concentrating on easy tasks. For example, Paxton quips that EBench is “not a leaderboard, but a CAT scan for your policy” – it won’t just tell you who’s #1, it will tell you why one model beats another in some aspects and lags in others. Behind EBench is a client–server setup in Isaac Sim. The heavy simulation runs on a server, while each policy communicates through a lightweight client. This means any robot learning code (PyTorch, TensorFlow, etc.) can plug in without conflict. All policies share identical physics and sensor rendering, ensuring fair comparisons. The benchmark is open-source, with released task assets and an online evaluation platform – the designers even allow you to submit your model for automated scoring. Profiling Five Capability Dimensions. Let’s unpack the five capability dimensions that EBench designers chose:. Operating Mode (Mobile vs. Fixed): Some tasks require navigating a wheeled base and coordinating whole-body motion (mobile tasks), while others keep the robot fixed at a table (like flipping a pancake or inserting a peg). Mobile tasks test navigation plus manipulation; fixed tasks isolate fine-arm control. Horizon (Short vs. Long): Short-horizon tasks might be a single pick-and-place or flip motion. Long-horizon tasks are multi-step chains – e.g. “sort all bottles into the basket” (which might involve opening cabinets, perceiving multiple objects, etc.). Long tasks tax planning and persistence. Precision (Low / Medium / High): How tolerantly aligned must the robot be? A low-precision task might be throwing a toy into a box (coarse tolerance), whereas high-precision could be inserting a key into a narrow slot (tight clearance). EBench explicitly labels tasks by their precision demands. Atomic Skill (Verbs): Each task breaks down into primitive actions like pick, place, move, pour, flip, push, pull, insert, etc. By tallying performance per atomic skill, EBench diagnoses which sub-skills a policy can really do. Scene (Environment): Tasks occur in different indoor scenes (kitchen, office, industrial lab, etc.). Some models may have been tuned to “home” environments; EBench tests across multiple settings to see if performance generalizes. Because every task has these labels, one can slice results across any axis. For example, we can ask: “How well does a model do on all high-precision tasks?” or “How does it transfer from familiar to new objects?” These slices reveal, say, that two models with the same overall success rate might excel in different modes. Benchmark Suite and Splits. EBench’s 26 tasks include both everyday and specialized chores. A few examples make the idea concrete:. Long-horizon tasks (9 total): Chains like “Sort all the bottles into the basket”, “Put all the dishes into the dishwasher”, or “Make a fruit milkshake in a cup”. Each involves multiple steps (inspect, plan, execute several moves). Mobile pick-and-place (10 tasks): Navigation plus pick-place, e.g. “Pick up the apple from the shelf and place it on the fruit bowl.” or “Scan every item at the checkout counter.” These often mix perception, navigation, and manipulation. Table-top dexterous tasks (7 tasks): Fixed-base tasks with fine control, such as “Put the bookmark on the top of the book.” or inserting pegs or collecting small objects precisely. Each task’s reward or success criteria is also multi-faceted. Instead of a simple binary (success/fail), many tasks score partial credit. For instance, in “bottle pack,” each correctly placed bottle adds to the score. This yields both a success rate (did it meet some threshold?) and a continuous score (how many sub-goals met). The EBench reports track both: “mean success rate (%)” and a normalized “score” (see InternRobotics Ebench leaderboard). Train/test splits are carefully controlled. Objects are partitioned so that test-time tasks contain new objects not seen during training. Likewise, instructions may be paraphrased. EBench separates Validation-Train (in-distribution) from Validation-Unseen (familiar tasks with new scenes or objects) and a Test-Mini set explicitly built by perturbing task factors (new backgrounds, new object instances, mixed combinations). This strict isolation means a model can’t cheat by recognizing a specific object or context from training; it must genuinely generalize. The designers point out that this level of rigor makes pretraining benefits dramatically visible – while earlier benchmarks like LIBERO or RoboTwin2.0 didn’t magnify the gap, EBench clearly shows which models learned general reasoning versus those just replaying training demos. Key Findings: Different Profiles, Same Success. EBench’s evaluations of state-of-the-art models immediately reveal: models with the same overall success can hide very different strengths and weaknesses. As the authors put it, “two policies can share a success rate and have nothing else in common”. The benchmark paper evaluates four “paper baselines” and a couple of community models. These include π₀ and π₀.₅ (a pair of VLA models from a recent paper), XVLA, InternVLA-A1 (the authors’ prior model), plus two unsanctioned submissions: LingBot-VA and Qwen-RobotManip (the latter from the recent Qwen report). On the topside summary (averaged across tasks), the new Qwen-RobotManip policy leads with about 45.6% test success rate and a 60.8% test score, putting it well above all others. Among the original baselines, π₀.₅ still tops the pack. (As shown on the EBench leaderboard, “Qwen-RobotManip sets the ceiling; π₀.₅ leads the four paper baselines”.) Notably, the paper’s own analysis found that π₀.₅ achieved “the highest test success rate and the best train–test retention of the four baselines”, consistent with it being the most robust of the old guards. However, that single-stat comparison is only the beginning. EBench then drills down to see where each model shines or falters. The benchmark authors summarize their own findings: InternVLA-A1 “dominates mobile manipulation” but “collapses on dexterous tasks”, while XVLA is strong on a different set of atomic skills than any other policy. In other words, InternVLA-A1 is great at tasks involving a wheeled base and coarse movement, but really struggles when tasks demand fine precision or delicate grasps. XVLA seems proficient at some verbs (say, insert or sweep) that the others aren’t. Even between π₀ and π₀.₅ (which share similar architecture and data), EBench finds subtle differences. Most strikingly, val-to-test analysis reveals that π₀.₅ generalizes the best among baselines – it has the smallest drop between seen and unseen conditions. The authors note “π₀.₅ has the tightest val–test gap”, which matches the community’s anecdote that π₀.₅ often fine-tunes well to new situations. In short: it isn’t just the highest score that matters, but how it was achieved. Consider now the champion model, Qwen-RobotManip. EBench’s five-dimensional analysis shows that Qwen’s performance “is not a single spike but a shape”. Qwen leads overall, but its successes are spread across categories. For example (quoting the EBench breakdown):. Mobile tasks: 43.8% success. Fixed/tabletop (dexterous) tasks: 50.0%. Short-horizon tasks: 50.2%. Long-horizon tasks: 33.1%. Low-precision tasks: 50.6%. High-precision tasks: 18.8%. These numbers (from Paxton’s summary) paint a picture: Qwen is strong on everything except high-precision manipulations. It performs almost equally well on mobile and tabletop tasks (43.8% vs 50.0%), indicating a broadly capable profile rather than a one-dimensional Rascal. Its success is balanced across short and long sequences. However, the bottleneck is clear: just 18.8% on high-precision tasks. (Even LingBot-VA nudges up to 27.5% on high-precision, showing that this is a consistent weakness.) In essence, EBench tells us: Qwen’s “shape” is overall higher, but we now know exactly where it should improve next (fine dexterity). The community-written report adds further color (albeit in Chinese, but informative). It notes that Qwen’s strength partly comes from covering many daily skills: it wins or ties on actions like flip, pour, press, handover, grasp, place, move – basic everyday maneuvers – often by noticeable margins. By contrast, it lags on infrequent skills like pull, push, sweep, and insert, where some older model was better. Scene-wise, Qwen dominates in “home” environments (bathroom, living room, etc.) but all models flounder in industrial or logistics settings. So even at the top, the benchmark reveals clear gaps in the “long tail” of tasks. For example, “shop checkout” and “bottle sort” tasks still elicit near-0% success from everyone – they’re evidently too complicated or underrepresented for current models. Generalization: Beyond the Training Distribution. Another major aspect of EBench is its focus on generalization tests. After training on one set of environments and objects, a true generalist policy should still work when the setting changes. To probe this, EBench perturbs tasks along four axes:. Object Shift: Swap an object in the scene (e.g. a novel detergent bottle). Background Shift: Change the room or scene textures. Instruction Shift: Rephrase or rewrite the task command. Composition Shift: Combine elements in new ways (e.g. two-step instructions vs single-step). Even though these seem subtle, EBench measures them explicitly. The results show, for instance, that Qwen-RobotManip is remarkably stable across these shifts: its success rates only hover around 44–47% in all four categories. In contrast, LingBot-VA (another high-performing model) did well under background or instruction changes (~38–39%) but its success plunged to only ~21% under object replacement or mixed perturbations. Plainly, LingBot-VA is more “brittle” – it works if the room changes or the phrasing is different, but it fails badly if the key object is replaced. Qwen’s flat curve suggests it has learned more robust abstractions. π₀.₅ also showed a fairly uniform (though lower) performance across shifts. This sensitivity analysis is possible only because EBench enforces strong train-test separation: if a policy simply memorizes the objects or scenes from its demonstrations, it would collapse under these tests. EBench even highlights the familiar “overfitting game”: plotting each model’s train-vs-test points shows who truly learns to generalize (points near the $y=x$ line) versus who merely “memorized” (points far below). By these metrics, Qwen and π₀.₅ remain closest to the diagonal, while others drop farther. A New Standard for Robot Benchmarks. What does all this tell us? The rise of EBench marks a shift in how we evaluate robot learning. Instead of crowning a single winner by aggregate score, it demands we inspect the structure of a model’s performance. In this spirit, the developers stress: “This isn’t a leaderboard – it’s a CAT scan.”. The goal is to make research progress by diagnosing concrete weaknesses. For instance, EBench immediately identifies that high-precision grasping remains unsolved by any generalist model (all have very low SR there) – so future work might focus on improving fine contact control. It shows that scaling up models (as with Qwen) does raise scores broadly, but even the giant model still struggles with the long-tail and precise tasks. It also makes pretraining benefits visible: in previous multi-task benchmarks the difference between a scaled-up model and a smaller one was murky, whereas in EBench the gap is stark. For practitioners, the multi-dimensional report is actionable. The designers even built an online evaluation platform and open-source code. You can plug in your robot policy, run all 26 tasks with randomized seeds, and instantly get a structured report showing where your model stands (just as shown in the EBench papers). As one such report charges, you’ll see whether your policy looks like a “genius on one slice and a toddler on another”. In other words, EBench gives the community a standard way to test new “robot foundation models” on equal footing – not just by one percent figure, but by a full diagnostic profile. In summary, EBench provides the field’s first detailed X-ray of generalist manipulation policies. Its creators convincingly demonstrate that high success rates alone can be misleading. By slicing performance along operating modes, horizon lengths, precision, individual skills, and out-of-distribution shifts, the benchmark teases apart what each model actually knows. The initial findings are instructive: the latest models are strong on common mobile and tabletop tasks (especially if they’ve seen lots of data), but still fragile on fine control and rarer skills. Going forward, researchers can use EBench to pinpoint precisely how to improve: “find the bottleneck and fix it,” as Paxton says of Qwen’s low high-precision score. In this way, EBench should help accelerate truly general robotic competence by guiding model builders to the weak spots in today’s seemingly impressive multi-task policies. References: The above discussion is based on the EBench benchmark description and technical report, as well as community analyses by Chris Paxton and others. The Qwen-RobotManip report and related press provide additional context (notably, Qwen’s 45.6% success on EBench and its balanced profile). All figures and numbers come from these sources.