Latest / Embodied AI 101 / Bimanual Pegboard Manipulation: A Benchmark for Vision-Language-Action Models

Transcript

Bimanual Pegboard Manipulation: A Benchmark for Vision-Language-Action Models · Embodied AI 101

• Bimanual Pegboard Manipulation: A Benchmark for Vision-Language-Action Models. The field of robot learning is rapidly moving toward Vision-Language-Action (VLA) models – large, unified policies that take raw images (and possibly language instructions) as input and output control actions for robots. These models, inspired by the success of large language and vision models, promise to fuse semantic understanding with low-level motor control in a single architecture. The LeRobot (“Large-scale Embodied Robot-learning”) framework on Hugging Face provides a standardized ecosystem for such models, complete with datasets and simulators. Just as foundational language models rely on benchmarks and test suites (e.g. GLUE for NLP), VLA models need common evaluation benchmarks to measure progress. Allen Institute for AI (AllenAI) and Hugging Face have begun assembling these benchmarks. Today we discuss one of their latest contributions: the dataset, a LeRobot-formatted collection of bimanual pegboard manipulation episodes. This dataset (52 episodes, ~30,000 frames, 3 camera views, two 7-DOF arms) is designed as a standardized benchmark for testing VLA policies on a classic dual-arm dexterity task. In this article we will explain the motivation for such a benchmark, describe the LeRobot infrastructure and dataset format, and detail what the pegboard dataset contains and how it can be used to evaluate a vision-language-action policy. Vision-Language-Action (VLA) Models: Combining Perception, Language, and Control. Vision-language-action (VLA) models are a newly emerging paradigm in robotics. The hope is to build a single generalist policy that can handle many robots, tasks, and environments by leveraging large-scale representations. Recent works (e.g. X-VLA, OpenVLA, Pi0 variants, StarVLA, GR00T, etc.) extend the transformer architectures of large language and vision models by adding an action head and training on robot data. For example, the X-VLA model from Hugging Face (0.9B parameters) uses soft “embodiment prompts” to absorb differences between robot platforms, fusing visual and language inputs into actions. These VLA models are attractive because they could, in principle, unify all robotic skills under one roof: by pretraining on hundreds of thousands of diverse robot demonstrations and using language tokens to identify robots and tasks, a single transformer backbone could learn to imitate or plan actions for many robots. However, this promise comes with challenges. VLA models must handle heterogeneous data: different body morphologies, sensors, camera placements, action formats, etc. In practice, each model is usually evaluated on a handful of standard benchmarks (e.g. LIBERO or Meta-World tasks) to compare progress. AllenAI’s recent work on MolmoAct and X-VLA has emphasized the need for more benchmarks and bench-testing so that different VLA models can be compared head-to-head in fair, reproducible ways. This is where standardized benchmark datasets come in. Just as ImageNet or COCO are standard datasets for vision, and SQuAD or GLUE for language, the robotics community needs fixed task suites for vision-and-language conditioned control. AllenAI is addressing this by releasing several “eval” datasets on the Hugging Face Hub, each targeting a particular manipulation domain (cup stacking, pipette-based tasks, etc.). The dataset is one such benchmark, focusing on bimanual pegboard insertion tasks (i.e. closely spaced holes/pegs on a board). It serves to test how well a VLA model can understand a manipulation instruction (possibly given in language) and coordinate two arms to reach, grasp, and insert pegs. In short, VLA models blur the line between vision, language, and action. To measure their real-world utility, we need evaluation tasks that are end-to-end: from camera image + instruction → sequence of joint commands. The pegboard dataset is a step in that direction, providing a common testing ground. LeRobot: A Unified Robot Framework and Data Format. This new pegboard benchmark is built on LeRobot, the open-source robotics framework hosted by Hugging Face. LeRobot provides a Python toolkit and standardized data format for a wide range of robot platforms and tasks. The core idea is to unify disparate robot datasets into a common schema: synchronized videos (or image streams) plus tabular state/action logs. Each dataset is stored in Hugging Face (HF) as a LeRobotDataset, which consists of:. MP4 video(s) or image streams for vision data (often from multiple cameras). Parquet table for timestamps, robot state, actions, and metadata. As the LeRobot README explains: “We utilize the LeRobotDataset format. Structure: Synchronized MP4 videos (or images) for vision and Parquet files for state/action data.”. This makes it easy to stream data, slice episodes, and integrate with HF tools. Importantly, anyone can load a LeRobot dataset in Python (with or the class) and get a dictionary for each frame, including the image arrays and state/action vectors. LeRobot supports many robot embodiments (single-arm, dual-arm, wheeled, etc.) with a hardware-agnostic interface. It also provides standard policies (behavior cloning, RL, diffusion) and analysis tools. For VLA, LeRobot includes implementations of the latest models (Pi0, Pi0.5, GR00T, VQ-BeT, X-VLA, SmolVLA, etc.). The framework even has a CLI (, ) for training and evaluating models on tasks. AllenAI (along with partners) has been a major contributor of LeRobot datasets. For example, the MolmoAct2 project (MolmoAct2: Action Reasoning Models for Real-world Deployment) collected a MolmoAct2-BimanualYAM dataset of over 720 hours of dual-arm demonstrations on tabletop tasks. That massive collection was built using LeRobot and served as training data for their foundation model. Now, AllenAI is also releasing evaluation subsets of these tasks – smaller curated sets (like the pegboard one) intended for benchmarking. These eval datasets are published on the HF Hub under the AllenAI organization and follow the same LeRobot format. As one dataset card notes: “This repository is one subset of the MolmoAct2-BimanualYAM Dataset, a large-scale collection of bimanual robot manipulation demonstrations. Across the full collection, MolmoAct2-BimanualYAM contains more than 720 hours of training demonstrations.”. In other words, the pegboard benchmark draws from that same data-gathering effort, but focuses on a specific skill. It inherits the LeRobot conventions (Parquet tables, video streams, episode indexing, and even language annotations) so that anyone familiar with LeRobot can immediately use it. The Pegboard Manipulation Task. The pegboard insertion task is a classic dual-arm manipulation challenge. Imagine a flat board with several holes and some pegs that fit into those holes. The robot (with two arms, 7 degrees of freedom each for a total of 14 joints) must pick up pegs and insert them into designated holes. This could involve one arm holding the peg while the other arm maneuvers the board, or coordinating both arms to align and insert pegs from different angles. In many benchmarks, pegs may need to be sorted by color or shape before insertion, adding a semantic element. Such tasks require fine motor control, proprioceptive awareness, and often some bimanual planning (e.g. handing one peg from the left arm to the right arm, or stabilizing the board while inserting). They are small enough to fit on a tabletop and repeatable, but complex enough to test a robot’s precision and coordination. Humans often do pegboard puzzles, and in robotics they test algorithms for insertion, grasping, and bimanual dexterity. In this dataset, the robot platform appears to be an Aloha-AgileX dual-arm with 7-DOF arms (the same base robot used in RoboTwin 2.0), although the precise simulator isn’t documented in the card. The mention of “14-DOF arms” in the summary and the style of LeRobot hints that each arm has 7 joints. The environment likely includes multiple RGB cameras observing the workspace from different viewpoints – typically three cameras (e.g. left arm camera, right arm camera, and a static overhead or third-person camera). Indeed, the summary specifies “3 camera views” per frame. By “in-distribution,” AllenAI means this pegboard setup is similar to what the model might have seen during training. (They might also release an “out-of-distribution” pegboard set with novel hole layouts or unseen peg colors.) But for evaluation, the in-distribution version ensures a fair baseline: the tasks are drawn from the same domain as the training data used by models like X-VLA or MolmoAct2. Researchers can thus report how well their fine-tuned VLA agent handles known pegboard tasks. Each episode in the dataset is a complete demonstration from start to finish. With 52 episodes in total, the collection covers a variety of initial configurations and instructions. If the LeRobot conventions are followed (as in other AllenAI datasets ), each episode should also have an associated language instruction. For example, an episode might be labeled with a textual goal like “Pick up the red peg on the left and insert it into the top-right hole.” These instructions (though not explicitly displayed in the brief summary) are implied by the “Vision-Language-Action” context. In LeRobot data, a frame’s field can be resolved to a string instruction via a separate metadata file. Thus a VLA model evaluated on this data would get an image from the cameras plus the target instruction, and must produce the correct sequence of joint actions to complete the insertion. The dataset totals about 30,000 frames across 52 episodes, or roughly 580 frames per episode. At 10 frames per second (a common rate in LeRobot datasets) that’s nearly a minute of motion per demo. Each frame record includes:. The three synchronized camera images (likely stored as one combined RGB frame or as separate video streams). The robot’s proprioceptive state (joint positions, gripper states, etc.). The last action taken (often the torque or delta-joint commands). A timestamp and indices (, , ) to coordinate logs. Because it’s on the HF Hub, the data can be accessed easily with the library. For example:. Behind the scenes, the code handles pulling from HF and decoding any videos. The LeRobot docs emphasize that you can treat each frame like a dict of arrays, so nothing special is needed to parse it. Using the Pegboard Benchmark. AllenAI intends this dataset as a benchmark: a fixed test suite for comparing VLA models. Here’s how one would use it. Suppose you have trained or obtained a pretrained policy (for example, an X-VLA checkpoint or MolmoAct2 fine-tuned model) and want to measure its performance on pegboard tasks. Within LeRobot, evaluation is done with the (formerly ) command. All evaluation datasets are named with the prefix. For instance, if your policy repository on HF is , you can run:. This will launch the simulator (or use a virtual environment) and run 10 episodes from the evaluation set, recording the actions the policy takes. As the model interacts with each episode, LeRobot logs the results (much like a human teleoperator’s performance). The key is that uses the eval dataset, and points to your policy. The instructions on AllenAI’s model card are explicit: “Prefix the dataset repo with and supply. A local or hub checkpoint can be used”. After running , you can compare the recorded actions with the ground-truth demonstration or success criteria. For simple tasks, a command-line metric might be simply whether the peg was correctly inserted (success rate). More sophisticated metrics (distance to hole over time, smoothness, etc.) could also be computed. The tool can optionally validate success according to the task’s defined condition. In summary, the workflow is exactly analogous to evaluating on LIBERO or other benchmarks: you point the evaluation script to the required eval dataset and to your model. This standardized protocol means any VLA policy (X-VLA, Pi0.5, etc.) that supports the same action space (absolute joint commands, presumably) can be plugged in and measured. For reference, one of the AllenAI example invocations (for a feeding task) was:. followed by “Prefix the dataset repo with eval_ and supply ”. The pegboard case is identical in form. Why is This Benchmark Important?. Why focus on pegboard insertion? There are a few reasons:. Precision and Coordination. Inserting pegs into holes is a high-precision task; it tests the model’s ability to align objects and avoid collisions. In a bimanual setting, it also tests coordination: one arm may hold or present a peg while the other places it. It’s qualitatively different from tossing objects or pushing them, as in many benchmarks; it requires fine control. Semantic Complexity. Pegboards often come with instructions like “put the red peg here, the blue peg there.” Handling color and target location means the model must correctly process both vision (to recognize pegs and holes) and language (to understand the goal). This makes it a true vision-and-language task, not just vision-to-action. Standardization. Unlike vague real-world scenes, a pegboard setup is highly structured. All episodes share the same environment geometry (the board), and only vary in peg arrangements and goals. This makes the dataset controlled and “in-distribution” – great for fair comparisons. Complements Other Tasks. AllenAI’s eval suite already includes tasks like test-tube pouring, cup stacking, and pipette manipulation (as seen by their other dataset names). Pegboard insertion adds variety: it is clearly a manipulation (placement) task, whereas the others include pouring/ordering tasks. A well-rounded VLA benchmark suite requires multiple different manipulation challenges. The AllenAI dataset card (as summarized by our prompt) emphasizes that this is an “open, standardized benchmark for vision-language-action model assessment.” Standard benchmarks are crucial for progress – they let researchers focus on model ideas rather than recreating environments, and they ensure apples-to-apples comparisons. The Pegboard benchmark fills a niche: testing dexterous, bimanual coordination under language guidance. Dataset Details and Format. Let’s spell out exactly what’s in. All information here is drawn from the official dataset card and LeRobot conventions:. Episodes: 52 total episodes. Each episode is a trajectory where the robot carries out one pegboard task from start to finish. Episodes are indexed 0–51. Frames: ~30,000 frames in total, or on average ~577 frames per episode. LeRobot usually records at 10 Hz, so each episode is roughly 1 minute long. Every frame record includes synchronized data across modalities. Robot: A bimanual platform with two 7-DOF arms (14 degrees of freedom total). The configuration is similar to the “Aloha-AgileX” robot in RoboTwin 2.0 (14 DOF). We can assume standard joint-angle control. Cameras: 3 camera views. Likely configurations: one per arm and one static overhead or third-person view. Each camera provides an RGB image per frame. In the LeRobot dataset, these might be stored as three parallel video streams or as one combined multi-view video – the exact naming depends on the dataset schema. But the user will get something like , , in each frame dict. Action and State: On each frame, the dataset logs the current robot state and the ACTION that the demonstrator executed at that step. The “action” is probably 14 continuous values (e.g. joint-angle velocity commands or target joint angles). The state includes joint positions, maybe gripper open/close, etc. These are in the Parquet table fields like and. Task/Instruction: Each episode has an associated task instruction. In LeRobot format, frames carry a that can be looked up in a “meta/tasks.parquet” file for a textual instruction. While we did not inspect the content, it is likely something like “insert peg A into hole 1, peg B into hole 2,” or perhaps shorter goals. This makes the tasks language-conditioned. Physics/Simulator: The dataset is recorded in simulation (as far as we know, likely on IsaacSim or similar via LeRobot’s pipeline). AllenAI’s press release (March 2026) mentions “simulation-first stack” for transferring to real robots, but here we focus on the data: it’s simulated so it can be easily re-run if needed. LeRobot Format: The data is packaged per LeRobot’s spec – a directory on HF containing Parquet files and MP4 video files. We saw in other AllenAI dataset cards that the viewer shows an “Auto-converted to Parquet” section with splits. Here there is just one split (“train”) of 52 episodes. All the snippets above (like [71] and [78]) show how LeRobot datasets appear on HF. They have the usual columns: , , , , , etc. Because it’s a LeRobot dataset, you can explore it with the Hugging Face dataset viewer (like in [71] and [78]). There you would see something like:. and so on, until. If you wanted to peek at the images, there is a LeRobot visualizer app (like the ) that can display episodes frame-by-frame with the images and state. But conceptually, each episode in Python gives you:. and so on. Because the frames are synchronized, one can also re-run an episode: the images can be passed to a vision model, the actions to ground-truth controllers, etc. In fact, one could even do behavioral cloning: use this dataset to train a policy. (But given its small size, it’s likely only intended as test data.). How to Evaluate Your VLA Model on Pegboard. To use this dataset as a benchmark, you follow the LeRobot evaluation protocol. That typically involves (or in older versions). The general idea is:. Load or deploy your policy (e.g. an X-VLA checkpoint) into the LeRobot server. Run each episode from the dataset through the policy, step by step. At each frame, the policy observes the current camera images and maybe the last action, and then it produces the next action. LeRobot applies that action in simulation and moves to the next frame. Log the outcome. Usually there is a success condition per task. For peg insertion, success might be detected when all pegs are correctly placed. LeRobot can track this if the environment defines it, or one can compute it from the final state. Concretely, the CLI might look like:. (This assumes is a HF repo ID for a trained model, and that supports.) The flags:. is your policy’s checkpoint (could be local or HF). is the eval dataset ID (note the prefix). means run the first 10 episodes. You can increase it to cover all 52 if desired (though some experiments chunk into smaller runs). After running this, LeRobot will produce a log (usually a JSON or CSV) of results for those episodes. It records for each episode whether the goal was achieved and possibly intermediate info. You can then compute the success rate (e.g. “8 out of 10 episodes solved”). Different models can be compared on this metric. As a hint of how formal this process is, the AllenAI model card we saw gave an example of usage and said explicitly: “Prefix the dataset repo with eval_ and supply ”. This pegboard dataset follows that same convention: by naming it with , LeRobot’s tooling knows it’s for evaluation. Although this dataset only has 52 episodes, small for training, it’s enough to get a statistical sense of performance. Often benchmarks use 10–20 episodes for a quick check. Because the tasks involve placement, the metric is usually percentage of fully correct insertions. One could also measure partial credit (e.g. distance of closest peg to hole), but that requires more setup. In many LeRobot benchmarks, success is binary per episode. Broader Impact: Towards Replicable VLA Research. The release of is part of a broader push for reproducibility and standardization in robotics research. Previously, many papers introduced novel tasks or custom simulators that were hard for others to replicate. LeRobot’s philosophy (and AllenAI’s projects) is to make everything open and on the HF Hub – datasets, models, code. By providing a crisp, public benchmark, they enable anyone to test their method under the exact same conditions that others use. This pegboard benchmark is particularly relevant for teams working on large VLA models: for example, if you have an OpenVLA or StarVLA, you can now directly evaluate on peg insertion and compare to the MolmoAct2 (AllenAI) results. Because all is on HF, you can even share results on an HF Leaderboard if one is set up. Moreover, pegboard tasks are common in robotic skill evaluation. In human intelligence tests, the Pegboard Dexterity test is well-known. Giving it to robots (and testing vision-language variants) is a natural step. A high score here means the model can translate instructions into precise, multi-step arm motions. That’s a stretch goal for VLA research, so it’s fitting as a benchmark. Finally, AllenAI’s release signals a pattern: they likely plan similar eval datasets for other domains. Indeed, their Hugging Face profile already lists things like , , and 【48†58-L58】. The pegboard is the in-distribution companion to those tasks. By covering a diversity of manipulation problems (pouring, stacking, pipetting, peg insertion, etc.), the community gets a richer picture of where VLA models work or fail. Conclusion. The dataset is a new LeRobot benchmark for bimanual pegboard manipulation. It provides a concise, standardized collection of dual-arm insertion tasks (52 episodes, 30k frames, 3 cameras, 14-DOF robot) that can be used to evaluate vision-language-action models. Hosted on the Hugging Face Hub, it integrates seamlessly with the LeRobot ecosystem: policies can be evaluated interactively via , and the data itself can be loaded for analysis or even imitation learning. In broader context, this benchmark exemplifies the push toward large-scale, unified robotic models and the importance of open benchmarks in driving progress. As one LeRobot document puts it, the goal is to unify “a wide range of robot and teleoperation platforms” under one framework, sharing data and models to lower barriers. The pegboard dataset is a building block in that vision: a concrete test that any VLA policy can face, making research comparable across labs. In summary, if you are developing a VLA policy (whether it’s X-VLA, OpenVLA, Pi0-based, or something else), this pegboard benchmark is now a ready-to-use challenge. By running your model on and reporting the success rate, you contribute to a growing body of evaluation results and help gauge the true capabilities of these ambitious models. Key References: The LeRobot documentation and AllenAI’s MolmoAct2 paper supply background on VLA and dataset standards. The LeRobot GitHub README details the dataset format【60†L119-L127】.odel card illustrates the evaluation CLI usage. All of these affirm that – while the pegboard dataset itself is new – it fits into a well-designed ecosystem aiming to standardize robot learning research.