Latest / Embodied AI 101 / PhAIL: Benchmarking Vision-Language-Action Models on Real-World Bin-Picking

Transcript

PhAIL: Benchmarking Vision-Language-Action Models on Real-World Bin-Picking · Embodied AI 101

• PhAIL: Benchmarking Vision-Language-Action Models on Real-World Bin-Picking. Roboticists and machine-learning engineers have made remarkable progress in teaching robots to perform tasks like folding towels or sorting fruit. But a nagging question remains unanswered: how well would these models actually hold up on a real factory or warehouse line? In the lab and in simulations we often celebrate high success rates, but industrial operations care about continuous throughput and reliability over hundreds of repetitive runs. The new PhAIL benchmark (Physical AI Leaderboard) was built to answer exactly that question for “vision-language-action” (VLA) robot models. PhAIL puts industry-relevant metrics on the table – measuring picks per hour and time-to-failure – by having real robot arms run hundreds of blind bin-to-bin picking episodes. The results are startling: even the best AI policies manage only a tiny fraction of human throughput, and they require human intervention about every few minutes. In short, today’s “foundation models” for robotics are still far from production-ready in a work setting. PhAIL makes this crystal clear by open-sourcing all the data, videos, and code, so everyone can see exactly where these models succeed and fail. Vision-Language-Action Models: The Hype and the Hope. Before diving into PhAIL’s experiments, let’s briefly recall what a “vision-language-action” (VLA) model is. Essentially, a VLA is a unified robot policy that takes raw visual input and a language instruction as input, and directly outputs the robot’s actions. These models leverage large vision-language neural networks (similar in spirit to image+text models) together with an action-generating module, aiming to generalize across tasks, objects, and environments without per-task retraining. The slogan is: give a robot a free-form instruction like “pick up the blue mug and place it on the table,” and out comes a continuous arm motion to do it. In theory, VLAs promise flexible, scalable robot intelligence – analogous to how large language models have generalized to many text tasks. Research in VLA models has surged recently. Examples include Google’s RT-2 system (which uses PaLI-X for vision-language encoding) and open-source efforts like Physical Intelligence’s π₀ (“OpenPi”), Hugging Face’s SmolVLA, and NVIDIA’s GR00T series. All these systems are pre-trained as general “foundations” and then fine-tuned (or used out-of-the-box) for tasks from pouring and scooping to drawer opening or block stacking. In simulation, VLA models have shown promise on benchmarks like VIMA, Bridge-CO, or SO-RA datasets. But those benchmarks typically measure raw success on single tasks or demonstrations, not sustained operation speed or % uptime. Everyone assumed if success rates are okay, models can be ramped up to real use – but nobody had put that to a stringent test. Meanwhile, industrial automation has very different demands from the latest hot research demo. A warehouse picking task is repeated thousands of times a day, and what matters to a factory manager is: How many units per hour can the robot do? How often will it stall or break? PhAIL was created by the Positronic Robotics team to answer exactly that: it’s the first real-hardware benchmark for VLA models on an industrially-relevant task. The goal is to measure throughput and reliability on physical robots, not just a handful of successes in simulation. As the PhAIL team puts it: “not success rates in simulation, real results on real hardware”. In practical terms, PhAIL measures Units Per Hour (UPH) and Mean Time Between Failures/Assists (MTBF/A) – metrics that managers use to evaluate automation. The rule of thumb is, if a robot needs human help every few minutes, it’s not saving anyone any labor. The Bin-to-Bin Picking Task. For the inaugural PhAIL challenge, the task is simple but ubiquitous: bin-to-bin object picking. Imagine a worker line where items come in one bin (the source tote) and need to be picked one by one and placed into another bin (the destination tote). This kind of picking is extremely common in fulfillment centers and manufacturing lines, anywhere from late-stage assembly to packaging. It’s not science-fiction – it’s real business. PhAIL’s designers chose it because it’s commercially important and physically straightforward. If models can’t handle this reliably, they won’t be ready for harder assembly jobs. Each PhAIL trial (episode) starts with an operator loading a random assortment of identical objects into the source tote, up to a known count. The robot (a Franka Research 3 (FR3) 7-DoF arm with a Robotiq 2F-85 parallel gripper) then runs autonomously under the control of the tested model. A wrist-mounted camera and an external camera provide visual input. The human operator doesn’t intervene unless something goes wrong. The policy runs until it has tried to move each item or 30 seconds per item have elapsed. At the end of the episode, the operator records how many objects were successfully transferred to the destination tote. If the robot behaves dangerously (say, colliding totes or falling objects), the operator triggers an emergency stop. Crucially, the evaluation is blind: the operator isn’t told which model is running on any given trial, eliminating bias. The task variants include four types of objects: wooden spoons, towels, scissors, and small batteries. These were chosen to represent different shapes and manipulation challenges (bendy vs rigid, thick vs thin items). All of these object types were included in the training dataset; nobody is testing novel objects yet. Typical episodes involve about 8 items in one tote, picking them one by one, with a 4-minute limit per episode (30 seconds per item with margin). The entire bench operates under the “DROID” platform configuration (originally introduced in the DROID dataset, FR3 arm plus two stereo rigs – one external Zed camera and one wrist Zed Mini camera). In short, everything is standard and reproducible: off-the-shelf hardware, open teleoperation interface, and open code (the team also offers the Positronic toolkit that unifies model interfaces and robot I/O). The choice of bin-picking is deliberate. This is exactly the kind of repetitive, high-volume task where automation should shine and where partial failures really hurt ROI. If a robot handling 8 items a run constantly drops or misses half the objects, it churns through shifts of workers “just to babysit the robot,” as the PhAIL developers candidly note. In warehouse jargon, you wouldn’t buy a $10k robot only to need a human hand on the joystick every 4 minutes. PhAIL exposes precisely those failure modes at scale. Models Evaluated and Baselines. In the first PhAIL release, four leading VLA models were evaluated (along with two human baselines). All four models were given identical training data – a large demonstration dataset collected on the same robot under teleoperation in various kitchen and workshop scenarios. In particular:. OpenPI (π₀.5): This is a fine-tuned version of Physical Intelligence’s π₀ model (with a 0.5 variant) trained on the PhAIL dataset. The base π₀ is a generalist flow-based VLA transformer that was open-sourced early 2025, pre-trained on many tasks. For PhAIL, the team took π₀, adapted it to the Franka+DROID setup, and fine-tuned on their bin-picking demos. OpenPI represents a state-of-the-art “foundation” model for robotics. GR00T N1.6: From NVIDIA’s robotics group, the GR00T models are large VLA policies (using an Eagle VLM + diffusion trunk) designed initially for humanoid robots and manipulation. N1.6 is the 1.6-billion parameter variant. NVIDIA graciously fine-tuned GR00T on the PhAIL dataset for this benchmark. It relies on both cameras to perceive the scene. ACT (LeRobot): Unlike the above, ACT is a smaller VLA policy built by the LeRobot project (a community robotics framework). Here ACT was trained from scratch on the PhAIL demonstrations rather than starting from a huge pre-trained model. It serves as a kind of strong “from-zero” baseline on this data. SmolVLA (Hugging Face): SmolVLA is a compact 450M-parameter VLA model by the Hugging Face community (the lerobot project) meant to be efficient enough to run on consumer hardware. They pre-trained it on various open datasets and provided it as a drop-in VLA. For PhAIL, a SmolVLA variant fine-tuned on relevant data was submitted by the Hugging Face team. Each model ran hundreds of episodes under blind evaluation. To calibrate, PhAIL also ran two baselines on the same hardware: (1) Human teleoperation – a skilled operator using a VR/leader arm interface commanded the robot to do the same pick-and-place sequence; and (2) Human hands – essentially ignoring the robot, a person just manually picked items with their hands at full speed. These human baselines measure the physical limit of the setup and act as targets for throughput. (After all, if the human hands only does 10 picks/h on this bench, 64 might not look so bad.). All models use the same observation and action interface via the Positronic SDK. The hardware stack is identical in every test. Importantly, the only differences from one model to the next are the policy network itself and how it was fine-tuned. This ensures a fair apples-to-apples comparison of policies on exactly the same task, hardware, lighting, and object distributions. Throughput and Failure Metrics. PhAIL chose metrics with industrial eyes. Instead of just reporting “80% success” or “average reward,” they report Units Per Hour (UPH) and Mean Time Between Failures/Assists (MTBF/A). UPH is straightforward: how many items the system successfully moves per hour of continuous operation. This directly translates to productivity (and ROI) for deployment. MTBF/A measures how long (on average) the robot runs before requiring human help – either to assist a stuck object or reboot after a failure. The idea is: if MTBF is too low, the robot needs constant babysitting. In practice, PhAIL defined a “failure” or “assist” as any episode where the robot couldn’t complete the run without human aid. That includes dropping an object, failing to grasp after a few tries, or any unsafe action. After each such event, the operator would intervene (either finish the job manually or reset). Then the clock for MTBF resets. The combination of these metrics paints a clear picture: throughput and continuous uptime. Metric-wise, these make PhAIL more like a factory KPI than a research score. As the developers note, industry people care much less about, say, “success on 10 picks in a row” and much more about “can this run thousands of picks per shift reliably?” High success on pilot episodes means little if the robot stops every three picks. By focusing on UPH and MTBF/A, PhAIL forces the conversation in operational terms. The evaluation protocol recorded every detail. Each episode was video-recorded from multiple cameras and logged joint/belief states. All that data (video+telemetry) was made public for every run. This transparency means anyone can review failure cases. Did the robot hesitate because it was confused? Did it knock the tote? You can watch each episode side by side. The PhAIL team even released their entire fine-tuning dataset (352 teleoperation episodes, ~12 GB of data) and training scripts. In short, they exposed the whole process so the community can audit and improve it. Results: Models Versus Humans. The headline results are grim for VLA models. In PhAIL’s bin-picking experiments, the best-performing AI policy achieved only about 5–20% of an expert human’s pick rate, and needed human rescue roughly every four minutes. The table below (excerpted from the PhAIL report) shows the key numbers:. | Method | UPH | Completion | MTBF/A | |------------------------------|-----:|------------:|-------------:| | Human (hands) | 1,331 | 100% | – | | Human (teleoperating robot) | 330 | 100% | – | | OpenPI (π₀.5) | 64 | 48.5% | 4.0 min | | GR00T N1.6 (NVIDIA) | 60 | 47.7% | 3.9 min | | ACT (LeRobot) | 44 | 35.0% | 3.6 min | | SmolVLA (Hugging Face) | 18 | 10.8% | 2.7 min |. (Table: PhAIL bin-picking benchmark results). Let’s unpack those. First, the human benchmarks: a person manually picking items can do 1331 items per hour by hand on that setup (100% success by definition). The same person using the robot arm (teleoperation) managed 330 UPH at 100% completion – a reasonable baseline since the robot’s motion speed is limited by safety. That means the robot is physically capable of a few hundred picks per hour if someone steers it optimally. By contrast, the best AI policy (OpenPI) only moved 64 items per hour – about 5% of a human’s hand-pick speed, or about 19% of the teleop speed. All three of the fine-tuned foundation models (OpenPI, GR00T, SmolVLA) hovered in the 60–18 UPH range, with completion rates well under 50%. The self-trained ACT policy did even worse. In plain terms: even carefully fine-tuned VLA systems picked less than half the items in the allotted time, dropping the rest. Notably, the frequent failures are alarming. The “Mean Time Between Failures/Assists” for the best model was only around 4 minutes. That means on average, the robot got stuck or wrecked the episode every 4 minutes, prompting a human to step in. The PhAIL team emphasizes that “at 4 minutes between failures, ‘autonomous operation’ means a full-time babysitter”. If your robot workforce needs a human within every ten minutes of operation, what have you gained efficiency-wise?. Put differently, in a typical 8-item run with a 4-minute cap the best AI were only moving about half the objects before timing out (see text after the table). Every episode was a gamble: sometimes the model would fail after 1–2 picks, sometimes 4–5, but rarely all 8. This is why completion rates are under 50%. The result is that the yield (items moved per attempt) of the AI arms is very low. Even though all models were given the same gear and training, none even reached 50% success across episodes. It’s worth emphasizing that all models were tested fairly: same data pipeline, same real robot, same randomization of items and start conditions. A scoreboard approach would go crazy if some model had a secret hint of which objects were present; here everything was blind and uniform. The fact that OpenPI and GR00T both plateaued around 60 UPH (with statistically identical results at this sample size) suggests we’re hitting some fundamental challenge, not just bad luck. Another perspective: compared to the teleoperated human baseline (UPH 330), the best model is 5-to-6 times slower. Since the hardware could physically move at 300+ picks/h when commanded, this large gap is “almost entirely policy quality” rather than a robot speed limitation. In essence, our arms can move fast – it’s the vision/planning that lags way behind a human. Warehouse operators would look at the “1331 hands” number as the true gold standard: at that rate, a single person hand-picking matches 4 full robot arms in theory. The current AI matches barely one-twentieth of that. All of these figures underscore the key takeaway: today’s Vision-Language-Action agents are far from industrial-strength. There’s a palpable gap between “looks good in demo” and “works at production speeds with reliability.” Anything under a minute to failure is not autonomous in any meaningful sense. These results are not cherry-picked – PhAIL averaged many runs per model (OpenPI and GR00T are still accumulating more for confidence). The conclusion from the data is bluntly put by the team: “the best model runs at 5% of human throughput… and needs operator help every few minutes”. What Explains the Failures?. Seeing these numbers, one might ask: what went wrong? The PhAIL team reports several insights drawn from analyzing the data and videos. Two patterns stood out particularly:. 1. Data volume matters, but only up to a point. Even the largest VLA models are surprisingly data-hungry. PhAIL included four different object types in this task, and they collected varying amounts of teleoperation data for each (spoons ~340 min, towels 178 min, scissors 160 min, batteries 134 min). As it turned out, more training data directly correlated with better performance. Wooden spoons (6 hours of demos) saw the highest success – about 50% average completion and ~71 UPH across models – while batteries (2.2 hours) averaged only ~27% completion and ~35 UPH. The correlation was strikingly linear (r = 0.96 for UPH vs. data time). In short, the more demos a model saw, the better it did. That suggests models were simply starved for examples. But even the spoons case, with nearly 6 hours of multi-camera demos, only managed ~50% completion at best. Collecting “more data” could help, but the results hint that scale alone won’t make them flawless. The policies seem to have learned something, but not enough nuance to reliably handle every grasp scenario. The team warns that “collecting more demonstrations may not be enough to close the gap.”. 2. Small setup changes break the models. In hundreds of runs, PhAIL introduced mild variations in the environment to test robustness. Namely, they swapped the left/right placement of the outbound tote and also switched the side on which the external camera was mounted. These are trivial changes from a human’s perspective (just mirror the workspace), but the models “notice” them deeply. For example, placing the tote on the right side (requiring the arm to reach further) systematically lowered success – likely due to slight differences in reachability and collision risk. The bigger finding was about the camera. When the external camera was on the opposite side of the outbound tote, the inbound tote would block the view, and the model’s vision dropped. GR00T’s completion rate, for instance, plummeted by 22 percentage points under this occluded configuration. OpenPI was more robust, but still affected. Essentially, if the main camera couldn’t see the items clearly, the policies couldn’t compensate just with the wrist camera. A human operator, with common-sense reasoning and 3D vision, wouldn’t bat an eye when the camera moves – we’d just look from another angle. But current VLA models fell apart. The PhAIL report remarks poignantly: “A human operator wouldn’t notice these changes. For current VLA models, they're the difference between half-working and barely working.”. In other words, the evaluated agents are brittle. They rely heavily on the precise viewpoint they were trained on. Move the tote a foot or swap sides, and the learned policy’s confidence drops catastrophically. This emphasizes that beyond raw pick success, these VLAs have low tolerance for variation – exactly the kind of variation a real factory environment might produce day to day. A third lesson, hinted by the raw results: existing VLA models still struggle with basic error recovery and safety handling. Several times PhAIL videos show the arm knocking over the destination bin or pushing objects off the table when confused, actions a human would never do. Each such incident triggered a safety fallback. The fact that every run was recorded and is public lets us see exactly how these failures unfold – sometimes the policy jitters, sometimes it completely loses track of the object mid-grasp. All in all, PhAIL’s open dataset (videos and logs) is a gold mine for diagnosing these blind spots. Open Data and Reproducibility. What makes PhAIL particularly valuable to the community is its transparency. The entire benchmark – not just final numbers – is open. The PhAIL website hosts a live leaderboard with every model result, and links each number to the corresponding episode video and telemetry data. Researchers can watch exactly how often and why each model failed. Did it drop a spoon? Did it try an impossible grasp? The synchronized multi-view videos and robot state logs show all that. Moreover, PhAIL published its fine-tuning dataset and code. The 12 GB dataset (352 episodes of teleoperated bin-picking) is on phail.ai, along with the exact scripts used to train each model. The Positronic toolkit (GitHub) includes interfaces for any policy model. In effect, anyone can reproduce PhAIL’s setup: hook up a Franka+Robotiq like DROID, load the data, and fine-tune a new model to try beating the leaderboard. The benchmark even has a submission portal for new models and checkpoints. This level of openness is unusual in robotics, and it’s a deliberate strategy to drive improvement. As the PhAIL team puts it, they want anyone who thinks “my model is better” to simply upload it and put it to the test under identical conditions. In short, PhAIL borders on being an open-source infrastructure project as much as a benchmark. The Positronic Robotics group sees it as an investment in building reliable robotics systems: one codebase to integrate any model, any hardware, and an evaluation engine that works out-of-the-box. By ironing out all the engineer-work of plugging models into robots, they ensure the comparison is fair and the barrier to entry low. This “infrastructure layer” is as important as the benchmark results themselves. It means the community can seamlessly swap in new VLA architectures, or even fully different paradigms, and see how they fare on the same physical rig. Related Context and Benchmarks. PhAIL is not the first benchmark in robotics, but it is unique in its scope. Previous benchmarks have typically focused on perception accuracy or success on a handful of attempts. For example, the Amazon Robotics Challenge (bin-picking) and Intel’s Future Networks Challenge have tested object identification and grasping accuracy using custom hardware, but with counted points or success percentages on tens of attempts. Simulated benchmarks like VIMA or Bridge-CO evaluate policy generalization on clean, synthetic tasks. Even large real datasets like D4RL or the recent DROID dataset have accelerated learning of policies, but they mostly measure offline or on-policy success in variety of tasks, not throughput. The DARPA Robotics Challenge likewise was performance-based but multi-faceted (stairs, debris, etc.). What none of those do is measure sustained industrial throughput on repetitive operations with real metrics. To our knowledge, PhAIL is the first to ask: “Given the best current VLA models and a realistic bench, how fast and how long can they actually go?” In doing so, it highlights the divide between toy-problem success and production demands. Some recent work has hinted at these issues. For example, the SmolVLA paper notes that while it matches larger models on small-scale tests (like block stacking or cube counting), real-world tests like the SO-101 pick-and-place bring out weaknesses. Likewise, many VLA papers report success rates after tens of fine-tuning episodes, but don’t consider if the policy can run uninterrupted for hours. The PhAIL results serve as a reality check on all these claims. We should note, though, that this is just one task. PhAIL is explicitly designed to grow. Bin-picking is a starting point because it’s clear and common, but the goal is to add other tasks that matter to industry – think sorting, lifting heavier objects, tiny part assembly, or mobile manipulation. The PhAIL consortium (including cloud provider Nebius and crowdsourcing partner Toloka) is already discussing new challenges, and they explicitly invite the robotics community to propose tasks beyond pick-and-place. For now, we can reflect on what bin-picking taught us. Implications and Takeaways. The sobering conclusion from PhAIL is that even the most advanced VLA-models today are fragile under production conditions. A 5% throughput of human hands is essentially a science experiment, not a working solution. It forces us to ask: where do we go from here?. One immediate implication is data and training: VLA models clearly need a lot more robust demonstrations, and probably better learning techniques (like adversarial training, domain randomization, or combining vision with other sensors). The spoon vs battery gap shows that multi-hour teleoperation helps, but the plateau hints that sheer scale won’t magically get them to 95%+ success. We might need complementary approaches – maybe explicit collision reasoning, feedback loops, or hybrid planning controllers. In any case, it’s a warning that “fine-tuning on 5 hours of data” was not enough. Second, model design: the sensitivity to viewpoint suggests current architectures aren’t handling partial observability well. Future VLAs might need built-in invariances or active perception abilities. In PhAIL we saw that having multiple cameras didn’t solve the occlusion problem for GR00T or OpenPI. A model that could, say, decide to move the tote or telescope the camera, or tilt its wrist camera, might cope better – but our tested models couldn’t. More robust visual encodings or some semblance of “memory” could help. Third, metrics. PhAIL underlines the need to change how we judge robotics research. Instead of reporting a handful of success runs, papers should include throughput and uptime metrics. Requiring long-run tests or worst-case analysis (like did the policy ever slam into something?) would expose brittleness earlier. A positive aspect of PhAIL is that it shows researchers the exact kinds of metrics industry cares about, potentially shifting priorities. Finally, this may influence deployment strategy. If even advanced AI models aren’t ready for fully autonomous service, hybrid approaches (keep humans in the loop, or use simpler algorithms for “easier” picks) might be necessary in the near term. For instance, one could imagine a warehouse system where VLAs handle the easy cases and flag the tricky items for human pickers. Or using traditional vision-based grasp planners as a fallback. Such hybrid designs might emerge as more pragmatic. Ultimately, PhAIL encourages honesty about progress. The fact that all results, including video evidence, are out in the open, means spin is hard. The community now has to grapple with the cold, hard truth of “5% throughput.” That may feel like a setback, but it’s actually valuable. It grounds our expectations and gives clear targets. If someone proposes a new VLA or imitation method, they can test it on PhAIL and instantly see if it’s any closer to human-level. Conclusion: Closing the Reality Gap. The era of “we have a Roomba and some language model, so robots will triage your inbox and fridge” has run ahead of the facts. PhAIL is a much-needed reality check – a quantitative map of how far we are from true physical autonomy. Rather than blame the hardware or the operators, it reveals that the policies themselves are the bottleneck. The best we have today still requires constant oversight to do a simple picking job. This isn’t entirely surprising: building truly robust, general robot controllers is hard. But before PhAIL, we didn’t have the numbers to say exactly how hard, or which pitfalls to fix first. PhAIL’s message is clear: real robots on real tasks behave differently from their lab demos. Slight viewpoint changes break them. Anything short of near-perfect reliability means any “automation” will end up as a glorified baby-sitting job. If the path forward is to improve these models, researchers should pay heed to what PhAIL found. Invest in more diverse data, better scene-variation training, and architectures that can generalize beyond the narrow conditions they were trained under. Or invent new mechanisms for error detection and recovery. The silver lining is that the framework is now in place to track this progress objectively. The PhAIL leaderboard is live, inviting new entrants. The infrastructure (Positronic Robotics’ toolkit) can plug in any model and let us judge it fairly. DreamZero (NVIDIA’s next model) is already slated for testing, and more partners are joining. In coming months we should watch for updates: can new entrants break 100 UPH? Can the MTBF be pushed to hours instead of minutes? And crucially, can these policies start to generalize to new object shapes and orientations not seen in training (a planned follow-up PhAIL challenge)?. For now, PhAIL stands as a milestone: it turned the spotlight on the most fundamental question of embodied AI – does it actually work at scale? – and gave a resounding, data-backed answer: Not yet. But with the game board laid out, researchers know exactly what “defeat” looks like and can aim higher. In the war of robots vs. repetitive work, we’ve seen who’s losing badly. It’s now up to the community to close the gap. Sources: Full details and raw data are in the PhAIL whitepaper and website, with commentary in Positronic Robotics’ blog post. The above analysis draws directly on PhAIL’s reported metrics and observations from those sources. All episode videos and data are publicly available at phail.ai for further inspection.