Latest / Embodied AI 101 / ABC: An Open Behavior Cloning Stack for Bimanual Manipulation

Transcript

ABC: An Open Behavior Cloning Stack for Bimanual Manipulation · Embodied AI 101

• ABC: An Open Behavior Cloning Stack for Bimanual Manipulation. In the quest to make robot learning more reproducible and scalable, ABC (Action, Behavior, Cloning) provides a complete open-source toolkit for real-world robotic manipulation via behavior cloning. At its core is ABC-130K, a massive bimanual manipulation dataset collected on a low-cost YAM robotic workstation. This dataset – the largest of its kind to date – contains roughly 3,500 hours of teleoperated robot data (about 130,000 episodes across 195 tasks). These tasks span a rich variety of manipulation skills – pick-and-place, folding, handovers, insertion, tool use, assembly, and more – organized into seven “primitive” categories. Within each task the objects, initial configurations, and subtasks are varied to cover realistic diversity. Notably, 42,980 of the episodes (about 1,552 hours) include subtask annotations: contiguous sub-trajectories are labeled with human-readable task segments (e.g. “pick object,” “insert card,” etc.) and teleoperator IDs, providing rich context for supervised learning. ABC-130K was collected on a bimanual YAM station – two open-source 6-DOF YAM arms mounted in parallel on a table, enclosed by white walls to reduce visual clutter. Each arm is fitted with a parallel-jaw gripper (a FlexPoint Dexterous Gripper in some sequences) and two Intel RealSense D405 cameras (one wrist camera on each arm) is combined with an overhead third-person camera, all streaming RGBD at 30 Hz. This configuration yields synchronized multi-view video along with full proprioceptive state (joint angles, end-effector poses, and gripper openness) for each arm. The total data comprises 134,806 episodes and 3,553 hours of manipulation, covering everything from delicate peg-in-hole insertions to dexterous re-grasping. Crucially, ABC also releases the entire data collection and training stack. The hardware design for the YAM station – including 3D-printed parts, off-the-shelf motors, low-cost encoders, and cabling – is documented. The teleoperation interface uses a wearable VR/AR headset (Meta Quest or Apple Vision Pro) through pass-through video (to avoid motion sickness) combined with a pair of GELLO leader arms that map human hand motions to the YAM “follower” arms. The team collected data by continuously teleoperating the robot to perform randomized task sequences; separate operators managed different tasks and shifts, which are tracked anonymously. All datasets are stored in the MCAP format (timestamped, synchronized streams of video, robot state, and metadata) and published on Hugging Face. Code is provided to download and parse episodes easily. The entire pipeline – from robot firmware and leader-arm interface to Python training scripts – is open-sourced for the community. In parallel with real-world data, ABC includes a simulation suite (ABC-Sim). A MuJoCo simulator faithfully reproduces the YAM hardware and environments. All ten representative tasks (e.g. throwing bottles into a bin, sweeping scraps off a table, turning mugs right-side up, loading dishes, hanging mugs, setting up chess pieces, spelling “ABC” with blocks, in-hand handovers, etc.) were built as interactive MuJoCo scenes. Collected on these in sim are more than 400 hours of VR teleoperation data: operators wear the same VR headset and control the simulated YAM arms with leader arms, providing a “cheap” high-fidelity proxy for testing models. A Blender‐based renderer can re-render saved trajectories for photoreal visuals. The authors show that performance in simulation correlates highly with real-robot performance (for example, across 12 checkpoints they observed an $r\approx0.85$ correlation in strict success rate). This lets researchers iterate on architectures or hyperparameters in sim before expensive real trials – a significant boon when access to hardware is limited. In short, ABC provides everything needed to reproduce the research: hardware schematics, robot-code, sim scenes, training infrastructure, and evaluation scripts. ABC-130K: The Data Backbone. The ABC-130K dataset is the foundation of the ABC stack. It contains 195 distinct tasks, totaling 134,806 episodes and 3,553 hours of data. These tasks cover diverse manipulation domains: for example, the dataset includes over a dozen folding tasks (paper airplanes, shirt piles, paper boxes), dozens of building tasks (stacking blocks, assembling objects), and various precision tasks like peg insertions, key locking, or retrieving credit cards from a wallet. In fact, the authors highlight that the resulting learned policies can perform “dexterous tasks such as box folding and extracting credit cards from wallets”. Some episode lengths reach a few minutes (e.g. multi-step folding and squeeze operations), while others are quick (~7 s for simple pick/place). Each episode’s diverse context is emphasized: objects and starting configurations are randomized within each task. For instance, the sorting pills task might involve different pill colors and bin locations, and block tower tasks use random block sizes and tower shapes. The tasks are grouped into seven categories by contact mode (sliding, lifting, plugging, etc.) and control style, reflecting the key skills in contact-rich manipulation. This structure is documented in the appendix. Overall, the scale and variety here far exceed prior open datasets. For comparison, the MolmoAct2 bimanual YAM dataset had only ~720 hours over 28 tasks, while AgiBot-World has about 3,000 hours over ~200 tasks – and both used more expensive setups. ABC’s single $8,000 YAM station produced 3,500+ hours on 195 tasks, making it an order-of-magnitude larger or more varied in most dimensions. The teleoperation interface logs detailed data. Three RGBD streams (60 Hz stitched per eye for VR, but logged at 30 Hz each) are captured along with the full 7-DOF state for each arm (6 joints + gripper aperture) at 10–20 Hz. Proprioception (joint torques, velocities, etc.) and relevant metadata (e.g. current task ID, operator ID, timestamps) are also recorded. A subset of episodes includes annotation tracks: workers segment each episode into subgoals (subtasks) with natural language labels. These annotations (42,980 episodes, ~1,552 hours total) allow training policies that condition on subgoal context or segment actions by type. The entire dataset is publicly available. Accessing it is straightforward: a single command can download the data and accompanying metadata (task lists, video codecs, etc.). The dataset page even provides a meta-file listing the hours and episode counts per task, making it easy to select or prune data. The license is Apache 2.0, encouraging reuse even in commercial or downstream research. In short, ABC-130K provides an unprecedented shared resource for manipulation learning. Hardware and Teleoperation Setup. A key enabler of ABC-130K is the low-cost YAM workstation. The platform uses two I2RT YAM arms – 6-DoF serial robots co-developed by UC Berkeley and Stanford (YAM stands for “Yet Another Manipulator”). These arms are open-source (model files and firmware on GitHub) and designed to be affordable; indeed, the authors stress that their entire dual-arm station cost only about $8,000. Both arms mount parallel on a table and are shielded on three sides by white panels. The white walls intentionally “narrow background diversity” to simplify vision without overly constraining the task variations. Each arm is equipped with a parallel-jaw gripper (the “FlexPoint” gripper in many trials) and two cameras – one on the wrist, one at an overhead vantage. In practice, each episode is recorded with a third-person camera looking down on the workspace (mounted above or opposite an arm). All cameras capture color+depth at 30 Hz. This combination of two egocentric and one exocentric viewpoint provides rich observation data for learning. Cameras are hardware-synchronized and the calibration (intrinsics/poses) for each is saved in each episode file. In terms of teleoperation, the system uses GELLO leader arms – passive 7-DoF arms that the human operator holds to specify motion. When the operator moves their hands via GELLO, the YAM arms mirror their motions in scaled space. At the same time the operator wears a VR headset (Meta Quest or Vision Pro) streaming the camera views at 60 Hz. A key feature is “pass-through” rendering: the VR headset lets the operator see the real world around them overlaid with the robot scene, greatly reducing motion sickness for long demos. This setup lets users naturally and accurately teleoperate the real robot as if in mixed reality. Importantly, the same VR+leader-arm rig is re-used in simulation: operators control the virtual YAM arms in MuJoCo using the GELLO arms, enabling seamless data collection in sim with identical interface. This design consistency means skills learned in sim have identical inputs as on the real robot. The data collection procedure was streamlined for large scale. Operators would follow scripted procedures to cover each task variation systematically. For tasks with known failure modes (like box folding), multiple operators or intervention tools (button-press pause, etc.) ensured data quality. The total burden of collecting 3,500 hours would have been intractable without this VR leader-arm approach – the authors estimate that teleoperators could comfortably operate the system for hours at a time thanks to the ergonomic pass-through setup. All collected data – joint positions, gripper states, camera frames, etc. – are time-stamped and saved as MCAP logs. Calibration and environment information (object meshes, table contours, etc.) are also stored. Critically, all this hardware and code is documented and released. The YAM arm open-source kit is referenced, assembly instructions are given, and the teleop software stack (ROS drivers, VR interface code, data logger) is published. Thus, any lab can replicate the ABC data collection if desired. When the paper calls for “placing researchers on an equal footing,” they mean it – you can literally build the same robot and collect new data with the same tools. ABC-Sim and Co-Training. Real robots are expensive and fragile, so ABC provides a simulation pipeline as a proxy evaluation and development environment. The authors outfitted a MuJoCo scene to match the YAM station and several tasks (toppling a mug rack, pouring beads, handing off objects, etc.). The choice of tasks in sim covers the domain well: in total, ten multi-step manipulation tasks were implemented for closed-loop testing. Each sim task was carefully calibrated so that images from the MuJoCo renderer resemble the real workbench. By default they used fast internal rendering in MuJoCo, but they also published a blender-based re-render pipeline for photo-realistic replay. To evaluate how well sim can stand in for reality, the team ran both sim and real tests on matched tasks. They trained policies (from different model checkpoints) using mixed real+sim training recipes and logged rollout success in simulation and on the actual robot. They found a high correlation: across 12 checkpoints spanning several model variants, the Pearson correlation was about 0.85 for strict success rate and 0.91 for progress percentage. This strong correlation validates the idea that “sim performance is a faithful predictor of real performance”. In other words, researchers can freely iterate on architecture design, hyperparameters, or loss functions in the much faster simulation environment without losing touch with the real-world outcome. To support this, ABC-Sim includes simulated VR teleoperation data. Using the same VR+GELLO interface described above, over 400 hours of demonstration were collected on the ten sim tasks. These virtual demonstrations are available alongside the real data. One can for example hold out a subset of real data, train on a mix of real and sim (fast to collect) using a “FAST co-training” regime (more on this below), and validate in sim before deploying the model on the actual robot. The authors specifically note: “We release over 400 hours of VR teleoperation data in simulation on all of these tasks, as well as the VR teleoperation stack to collect teleop data”. By open-sourcing the entire sim suite (MuJoCo model files, task scripts, Blender textures), even groups without any hardware can use the sim data and code. They can also create custom tasks by dropping new object meshes and recording teleop in the sim. The ABC pipeline is therefore not just a fixed dataset, but an extensible toolkit. Figure 10 of the paper (reproduced online) shows renderings of the ten sim tasks, from throwing bottles into a bin to spelling “ABC” with blocks. Policy Model Architectures. Given this wealth of data, the next question is how to train policies on it. The ABC project focuses on two modern classes of policy architectures: Diffusion Transformers (DiT) and Vision-Language-Action (VLA) models. Both are variants of behavior cloning (supervised learning to imitate the teleop actions given the observations) but with different structural biases. Diffusion Transformers (DiT). A Diffusion Transformer policy models the robot’s trajectory as a sequence (or chunk) of continuous actions, generated via a diffusion process conditioned on the current visual context. Concretely, each training example provides the recent history of states (joint angles, images) and a short block of expert action commands (e.g. 30 joint deltas). The DiT then learns to denoise a randomly noised version of the expert action trajectory, akin to diffusion models in image generation. Architecturally, the DiT used here consists of a large Transformer that attends over the observations and the action sequence. Crucially, the vision processing is externalized: each camera image is first encoded by a frozen pre-trained encoder (DINOv3 [36], a self-supervised vision transformer) to produce a compact token embedding. These vision tokens are then fed into the DiT. Interestingly, the ABC implementation chose a “lopsided” parameter split: the DiT action-decoder head is extremely large (up to 1.93 B parameters), while the vision encoder is relatively small (DINOv3-B with ~86 M parameters). The rationale is that in multi-view vision, each image encoder pays high FLOPs per parameter, so it’s more efficient to enlarge the action head. In experiments they swept four sizes of DiT head: 153 M, 290 M, 746 M, and 1.93 B parameters. They found that bigger is better: at fixed training steps or fixed total compute, the largest DiT consistently achieved lower training loss. For example, after 300K training steps, the DiT-xL model had an MSE loss around 0.040 whereas the smallest DiT-S was around 0.059. In other words, scaling the diffusion-transformer head yields better fit to the teleoperation data. (This is consistent with trends in other domains: large models slowly reduce residual error as compute grows.) There is a cost trade-off: at inference the 1.93 B-step Diffusion head must be run at each diffusion step. Still, they point out a neat fact: because VLA models put most computation in a big vision-language network that runs only once, the DiT spends its compute on the iterative head. For training specifics, they used mean squared error logits to match joint velocities/gripper speeds, with dropout, learned 30-step chunks of actions, etc. The DiTs predict raw action deltas (position and quaternion increments) which the robot executes. During sampling, they iterate the diffusion denoising for 10 steps to produce each action chunk. The encoder sees the last few frames (they used 3 images) per arm to capture context. In summary, the ABC-DiT is essentially a very large sequence model specialized to produce text-like “action instructions” (modeled as real-valued tokens) for the robot. Vision–Language–Action (VLA) Models. The second class, Vision-Language-Action (VLA) models, takes a different approach. A VLA is essentially a large vision-language model (VLM) fine-tuned end-to-end to output robot actions. In ABC’s implementation, they use Gemma 3 (a 4.3 B param VLM) as the visual and optionally linguistic backbone, followed by a tiny diffusion-style action head (only 45 M parameters) that generates actions. They call their model ABC-VLA. Concretely, each input (image sequence) is fed through SigLIP (a CLIP-style image encoder) and Gemma to produce a contextual embedding. Then, instead of outputting text, they process this embedding through a small two-layer Transformer (with AdaLayerNorm conditioning on the Gemma features) to generate a 30-step action sequence, again using diffusion. In effect, the large Gemma VLM provides high-level perception, and the diffusor head provides dexterous motion prediction. One advantage of this design is inference speed: the expensive Gemma forward pass is done once per sample, and then the small head (45 M) is run 10 times for the 10 diffusion steps. By contrast, the ABC-DiT must run its ~1.9 B diffusion head at every step, though it never runs a VLM at test time. The [45] excerpt makes this clear:. “In ABC-DiT, the diffusion head has roughly 1.93B parameters and must be evaluated at every diffusion step. In ABC-VLA, the VLM is large (4.3B), but is only run once to produce cached features; the diffusion head itself has only 45M parameters and is then run for the 10 diffusion steps”. To optimize this architecture, the authors also experimented with how to inject the visual context into the decoder. The base ABC-VLA used an AdaLayerNorm approach, where Gemma’s output is used to adaptively shift and scale the normalization layers of the action Transformer. They tried alternatives like direct token-wise cross-attention (as in CLIP-Trans or RT-2 style). In ablations on real evaluation, they found that using pooled cross-attention (attending tokens at each diffusion step) outperformed the simpler global AdaLN: swapping to pooled token attention improved mean task progress noticeably. The paper reports, e.g., a jump from ~47% progress to ~61% progress on a set of test tasks. They also found that replacing CLIP with the DINOv3 encoder (the same image model used for DiT) boosted performance further, likely because DINO is more specialized to spatial details. Additionally, they applied a “FAST co-training” trick: augmenting the loss with a CLIP-like cross-entropy on the visual features. This improved a vanilla cross-attention baseline from ~12% to ~33% progress, though their final AdaLN+DINO model did even better. In sum, ABC provides one example of a large-scale VLA policy: Gemma-3 with AdaLN diffusion head (the released checkpoints are ~4.3B for the VLM plus 45M head). This puts ABC-VLA in the same paradigm as RT-2/π-VLA, but open-sourced and trained on a purely robotic dataset. Because it leverages a generalist vision-language backbone, it could potentially take language prompts (indeed, the code allows a text instruction channel, though the main paper focuses on vision-only inputs). Training Insights and Scale Effects. With dataset and models in hand, the authors ran an extensive sweep of training regimes. All training was standard imitation learning (minimizing action prediction error), but with some modern twists. For example, they used a staggered protocol mixing fully on-policy demos, random noise injection, and the aforementioned FAST re-weighting. They also trained on mixed real+sim data in some runs to see if that helped. One clear finding is scale is king. Larger models and bigger batches consistently yielded better results. For DiT, as noted, the 1.93B head achieved the lowest loss. On a fixed compute budget, the bigger model still had lower final loss – the large model simply learned more from the 130K episodes than a smaller one could. Similarly, for VLA, upgrading Gemma to a larger variant gave a noticeable lift, though in ABC they stuck with Gemma-3 (4.3B) as a good sweet spot of performance vs. resource. They also explored conditioning horizon. Because each block is 30 steps, they tried re-prompting the model with shorter vs longer segments. Generally longer horizons (and letting the model “plan” a bigger chunk) seemed to help consistency, though this is a minor detail. Notably, when given less data per task, the policies performed worse – unsurprisingly. There was no magic of “few-shot generalization”: the models needed enough coverage that each task was adequately represented. Thus the payoff of 130K demos is clear: tasks that might take days for a single robot to master were learned by imitation across many episodes. It’s worth pointing out a practical outcome: even with the best off-the-shelf training, some very challenging tasks still require additional data. The team explicitly discusses using DAgger (dataset aggregation) for box-folding: the naive behavior-cloning policy couldn’t fold the final flap reliably. They used an iterative teleop correction procedure (gathering more data where the policy fails) to patch up these shortcomings. This underscores that ABC is a foundation – further fine-tuning or correction may be needed for peak performance on the hardest maneuvers. Real-World Evaluation. Ultimately, the goal is robust real-robot performance. The paper reports autonomous policy runs (no human in the loop) on a suite of test tasks. These include some tasks held out from training, and some complicated multi-step chores. Quantitatively, they track two metrics: strict success (did the robot fully complete the task within a time limit?) and progress (a normalized measure of how far along the subgoals the robot got). The results show promising capabilities. For instance, on a set of six evaluation tasks – throwing plastic bottles into a bin, loading plates into a dishrack, turning mugs upright, etc. – the top models achieve respectable success rates (on the order of tens of percent) and much higher partial progress. For example, one best checkpoint might have ~60% mean progress across tasks. The gap between progress and full success indicates that the policies often get most of the way there but fail on the final precise alignment (a common issue in end-to-end control). One highlight is the credit-card-from-wallet task. This is a highly dexterous insertion/removal: the robot must open a zippered or flapped wallet, slide out a card, and place it on the table – something that requires fine finger control and force sensing. The ABC-130K dataset explicitly includes many variations of this task, and the learned policy can execute it largely autonomously. In the demo video they show the robot methodically drawing cards out – one of the key examples promised in the abstract. Similarly, the trained agents can fold things like paper or cloth. The video examples include folding a t-shirt and a paper airplane. Full boxes are folded at partially (think in half but not perfectly flattened). These show non-trivial compliance and multi-contact control. Alongside simulated benchmarks, these real-world trials validate that the methods learned do generalize beyond trivial scripts. It’s important to note, however, that performance is still below human level. Many tasks see only moderate success rates. The contribution here is not that ABC policies solve all tasks perfectly, but that they provide a high starting point and a unified benchmark. The authors emphasize that “by providing a reproducible toolkit, we aim to place researchers on an equal footing”. In other words, anyone with the dataset and code can reproduce these results and push them further. The logs of all evaluation trials are even slated for release, allowing future analysis of failure modes. For practitioners interested in using the ABC stack, training recipes are provided. For example, the paper notes that the final policies run in real time (~20–30 Hz control loops) and use only onboard processing (the DiTs can be large but were run on a GPU or even TPU). Checkpoints of both ABC-DiT and ABC-VLA are publicly released. There are scripts to convert an MCAP episode into training examples, and PyTorch code to finetune or use the models. Combined with the robot’s URDF/MuJoCo model and calibration data, a lab could pick up the trained networks and deploy them on a similar YAM dual-arm setup. Putting It in Context. ABC builds on a growing trend of large-scale offline learning for robotics. It draws inspiration from prior open efforts (e.g. RoboNet, KIT, OpenX-Embodiment) but scales up dramatically. Perhaps the closest precursor is the Open X-Embodiment (OXE) datasets for pick-and-place – which combined data across many platforms but mostly single arm tasks. ABC’s novelty is bimanual focus, open infrastructure, and breadth of tasks. The concurrent OXE efforts like RT-2 focus on VLA but often rely on larger industrial arms and proprietary datastores; ABC shows what can be done with consumer-grade hardware. Technically, the use of diffusion policies echoes recent works like DiffusionPolicy and π-VLA. ABC’s DiTs are similar in spirit, and its VLA (Gemma + diffusion head) is akin to π_0, though ABC is purely trained on its own demonstrations not on web video. In terms of model size, ABC-DiT at ~2B parameters sits between smaller academic diffusion policies and huge models at DeepMind or Google. The ABC-VLA at 4.3B is in line with other “few-billion-parameter” robot VLAs. The fact that all of this is open-source – hardware and software – is what ultimately defines the contribution. Conclusion. The ABC stack marks an important milestone in embodied AI. By providing an open, well-documented dataset and training pipeline for dexterous, bimanual manipulation, it lowers the bar for academic groups without fleets of robots. Researchers can now evaluate new architectures or imitation learning methods on a common large-scale benchmark, rather than struggling to gather their own data. The emphasis on reproducibility (open models, code, data, even evaluation logs) is refreshing and in line with modern ML culture. For roboticists, ABC offers a practical toolkit: one can deploy the trained models, adapt them to new objects via finetuning, or even plug in language instructions through the VLA head with some engineering. For example, since Gemma can understand text, a natural next step is to extend ABC-VLA to handle language commands like “fold the blue cloth” by fine-tuning on same images but different text prompts. Similarly, the recorded subtask labels enable training hierarchical or subgoal-conditioned policies. In summary, ABC (Open Behavior Cloning) brings the power of big data and big models into the real-robot manipulation arena, with a community spirit. Its slogan might be: “Learn the ABCs of Behavior Cloning together.” With all the pieces released, the field now has a solid foundation on which to build more capable robot learning systems – whether via scaling up models, adding reinforcement fine-tuning, or cross-embodiment generalization. The next generation of roboticists will benefit from this shared starting point. Sources: The above is based on the ABC project paper (Allshire et al., 2026) and its accompanying materials, as well as background on VLA models.