Latest / Embodied AI 101 / ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

Transcript

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining · Embodied AI 101

• ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining. Vision-language-action (VLA) models have emerged as a powerful paradigm for robot manipulation: they learn to map from visual inputs and language instructions directly to action sequences. But training such models requires immense, diverse embodied data. Collecting large-scale robot trajectories is prohibitively expensive and slow. One promising alternative is to leverage the ever-growing trove of human egocentric videos taken from head-mounted cameras in kitchens, workshops, etc. These videos contain naturalistic manipulation behaviors, but naively mixing them with real robot data is challenging – humans and robots move differently and the camera viewpoints differ. ACE-Ego-0, introduced by Hao Li et al. (2026), tackles this challenge by unifying heterogeneous data in a single pretraining framework. The core idea is to put everything into a common representation before training. Specifically, ACE-Ego-0 converts raw first-person human videos into “pseudo-action” trajectories that mimic what a robot would do, and then trains one big VLA model on both these pseudo-robot actions and real robot demonstrations. Crucially, both human-derived actions and robot actions are expressed in the same camera-centric action space, and the model is explicitly conditioned on the robot’s morphology (e.g. its joint structure) so that the same policy can control many different robot bodies. The training also uses a reliability-aware loss that downweights noisy human-derived labels, so the model learns primarily from high-quality signals while still benefiting from the breadth of human data. In practice, the authors assemble an enormous pretraining corpus – about 6.0K hours of mixed data (4.53K from real robot and simulator logs, plus ~1.48K from filtered human videos) – and show that ACE-Ego-0 achieves state-of-the-art results on standard benchmarks. For example, it attains 72.8% success on the RoboCasa GR1 TableTop suite and over 91% on both the “easy” and “hard” settings of RoboTwin 2.0, significantly outperforming prior VLA models. It also transfers effectively to a real dual-arm robot platform (with 78.3% success on a set of retail-style tasks), demonstrating that this mixed pretraining yields a robust manipulation policy. In what follows we’ll unpack the key pieces of ACE-Ego-0: how it represents actions, the video-to-action pipeline, the training procedure, and the results that underscore its impact on embodied AI and multi-robot generalization. The VLA Data Bottleneck: Humans vs. Robots. Modern VLA models (like PaLM-E or Pi0.5) can indeed generalize impressively in manipulation, but they rely on very large training datasets. Real robot demonstrations—where a single robot executes an action sequence—are precise but costly to gather. In contrast, there is a wealth of egocentric human video data (Ego4D, EPIC-KITCHENS, HOI4D, EgoDex, etc.) that captures hours upon hours of people interacting with objects in kitchens, workshops, and stores. These provide rich visual and behavioral diversity at almost no cost. However, mixing these sources is nontrivial. Humans typically act under different kinematic constraints (two human hands vs. a robot arm, different ranges of motion), and the cameras themselves have different perspectives. For example, a first-person camera on a human sits near the eyes or forehead, whereas a robot might have an eye on its head or chest. Thus, the same high-level command like “reach and grasp the mug” translates into very different coordinate frames and action spaces for a human vs. a robot. Past work (e.g. EgoVLA by Yang et al.) tried aligning human and robot by learning shared MANO hand parameters and retargeting, but this mainly handled single humanoid or hand-based robots and did not fully solve the multi-embodiment problem. ACE-Ego-0’s key insight is to solve these alignment issues before joint training by using a unified action representation. Instead of learning to transform between different coordinate frames during training, they re-express both human actions and robot actions in the camera’s coordinate frame. In practice, this means that all end-effector positions and motions are given relative to the head-mounted camera, regardless of whether they came from a human or a robot demonstration. As a result, one model can be taught with both kinds of data without confusing “left arm vs. right arm vs. different camera angles,” because everything is in a shared camera-centric view. In algorithmic terms:. Camera-space actions: Both real robot trajectories and reconstructed human actions are converted into the observation-centric camera frame. Concretely, if a robot’s end-effector pose was originally given in the robot’s base frame, the system uses calibrated extrinsic parameters to transform it into the head-camera frame. Similarly, reconstructed human hand/arm trajectories (from egocentric video) are computed in the same camera coordinate system. At training time, the network always predicts desired end-effector movements relative to the camera, so it need not learn arbitrary coordinate transforms on its own. Morphology conditioning: ACE-Ego-0 still must know which body it’s controlling. To handle different robot types (single-arm vs. humanoid vs. mobile base), the action head is conditioned on a vector encoding of the robot’s morphology. The authors embed each robot’s URDF graph (its joint structure and link parameters) and also introduce a learned “surrogate token” for humans. These condition vectors are fed into the model’s action prediction module, enabling a single policy to adapt its output to the appropriate joint space. Unlike retraining separate models for UR5 vs. Franka vs. a mobile manipulator, the same ACE-Ego-0 weights can be reused, with only the morphology code informing the difference. Time-aligned chunking: To further align data from different sources, ACE-Ego-0 concatenates actions into consistent time chunks. Both robot logs and human clips are segmented into short sequences of unified length, so the model always sees fixed-duration action segments. This keeps the temporal scale consistent across datasets (e.g. a 2-second grab followed by lift should look similar whether from a human video or a robot trajectory). By architecting these representations up front, ACE-Ego-0 “resolves spatial, structural, temporal, and label-quality mismatches between human egocentric video and robot trajectories”. After this alignment step, the model sees a much more homogeneous training signal: sequences of 6D end-effector motions (plus gripper states) in the camera frame, annotated by whether they came from human or robot. This unified view makes joint pretraining possible without confusing the network. Video-to-Action Pipeline: Extracting Robot Pseudo-Actions. A central component of ACE-Ego-0 is the pipeline that transforms raw human video into robot-style action trajectories. This subsystem makes it possible to mine supervision from human footage at scale. Here’s how it works:. Clip curation: We start with a huge corpus of ego-centric videos (sources like Ego4D, EgoExo, EPIC-Kitchens, HOI4D, EgoDex, Xperience-10M). The pipeline first filters these videos. Clips must be between 4–30 seconds (to ensure a complete manipulation primitive) and the camera must be body-mounted (discard any non-ego, e.g. static set-ups). Heuristic filters remove segments without visible hands or with low confidence face-detections, ensuring we only keep first-person clips. A language captioning filter also verifies each clip has at least one manipulation verb and one object noun, to focus on relevant interactions. 3D hand/arm reconstruction: For each remaining clip, the system reconstructs the 3D motion of the human’s hands (and potentially arms). Using state-of-the-art vision tools, each frame is fed through a MANO hand model estimator. Specifically, the authors employ SAM3 (a segment-anything tracker) to get hand silhouettes and HaMeR (Hand Mesh Recovery) to estimate a detailed hand pose and shape in 3D. This yields a sequence of estimated hand poses in camera space. To reduce noise, they then perform a global trajectory optimization: essentially smoothing the joint trajectories to minimize reprojection error frame-by-frame. This optimization step, which uses camera pose estimates from the VIPE system, cuts jitter and ensures smoother physical hand motions. Action parameterization: The 3D hand trajectories are then converted into an action format usable by robots. The authors define a unified 22-dimensional action vector per arm. For each arm (left/right), this contains the 3D position (x,y,z) of the end-effector, a continuous 6D orientation (represented as a rotation vector or equivalent), the gripper opening distance, and an “activity flag” to indicate whether the hand is actively manipulating or idle. (Since human hand data is continuous, they threshold gripper motions into open/closed.) The result is a time-aligned sequence of these action vectors, all expressed in the head-camera coordinate frame. (Robot demonstrations are processed similarly: real sensor-recorded end-effector poses are also expressed in the camera frame.). Quality filtering: Not every clip yields high-quality pseudo-actions. The pipeline applies a series of checks to discard or down-weight bad segments. Trajectories with NaN values, extremely low motion energy, implausible correlations between the two hands, or spikes in velocity get thrown out. Frames where the depth sensor saw no positive depth or projected points fall outside the image are masked. This filters out instances where the vision estimate was unreliable. After this processing, ACE-Ego-0 recovers a large set of pseudo-labeled manipulation trajectories. In total, this pipeline curation yields ∼1.48K hours of useful human egocentric clips paired with estimated robot-like action sequences. By construction, these pseudo-actions “mirror” what a robot end-effector would have done to cause the observed human motion. They aren’t perfect – there is inevitable noise from vision inference – but they provide a rich, large-scale training signal that would otherwise be unattainable. For example, in one clip of a person packing a shoe into a box, the pipeline reconstructs the 3D motion of the person’s hands: first reaching down, grasping the shoe, lifting it, and putting it into the box. All that is encoded as a short sequence of 3D positions, orientations, and gripper states (open/close) for the “left-hand” and “right-hand” agents, expressed relative to the head camera. A robot watching this can learn “when I see this context, I should first move to this 6D position and close my gripper” – even though the robot’s arm has a different kinematic chain. Crucially, ACE-Ego-0 has a built-in notion of which “arm” is which: a morphology token is set so the model knows it’s controlling a humanoid with two arms, or a single-arm, etc. In summary, the video-to-action pipeline scales human data into robot-compatible labels. It automatically converts massive egocentric datasets into roughly 1.5K hours of action-annotated sequences, enabling the VLA model to see orders of magnitude more diverse behaviors than purely robot-collected data ever could. Unified Pretraining and Reliability-Aware Training. With both real robot trajectories and pseudo-human actions now in a common format, ACE-Ego-0 trains a single VLA model across them jointly. The model architecture is a vision-language backbone (e.g. a multimodal transformer) feeding into an action head that predicts the next chunk of actions. But the training strategy is carefully designed to respect the differences in data quality. The primary training objective is standard action prediction accuracy on the robot data: when given an RGB-D frame sequence plus an instruction, the model should output the correct end-effector motion (position, orientation, gripper) at each step. For the human-derived pseudo-actions, however, the supervision is noisier, so the model treats it as an auxiliary learning signal. The total loss is split:. Main (robot) loss: The model is trained with a primary supervised loss (e.g. mean squared error / Huber loss on continuous actions) using only the real robot demonstrations. This ensures that high-fidelity sensor-recorded actions drive the core learning. Human auxiliary loss: Pseudo-action labels from human videos contribute an auxiliary loss branch. However, this loss is modulated by reliability weights. In effect, each time step and channel of the human data is given an estimated confidence score. ACE-Ego-0 uses reliability-aware weighting to focus the human loss on the cleanest parts of the data. Concretely, the authors estimate two sets of quality measures: one at the step level and one at the dataset level. If a particular frame of a video had unstable depth or the tracker barely saw the hand, its step-level weight will be low. If an entire dataset (say, one video source) is known to be less reliable, it might get a lower dataset-level prior. These factors multiply to scale the auxiliary loss at each timestep. The result is that unreliable pseudo-actions (which might mislead the model) are down-weighted, while good signals (where the hand motion was clearly visible) still contribute useful gradients. In addition, they introduce channel-wise priors for human data. They found that some components of the action vector (e.g. 3D position) can be recovered more accurately than others (e.g. precise wrist rotation). So they manually set higher target importance to position channels and lower to things like rotation or gripper distance when the supervision is coming from video. This biases the model to trust what can be reliably inferred from the video (like “move forward to grasp the cup”) and to ignore wobble in the fine details. In practice, they train by sampling mini-batches that mix robot- and human-sourced trajectories. Robot examples receive the full action loss, whereas human examples go through the weighted auxiliary loss path. The total objective can be seen as:. L = Lrobot-action + α × whuman ∘ Lhuman-pseudo-action,. where whuman are the per-step confidence weights and α might be a tunable scalar. This scheme ensures ACE-Ego-0 “consistently improves both unified joint pretraining and supervised fine-tuning” by leveraging the human videos without letting them corrupt the policy. Empirically, the paper reports that including the large-scale human supervision under reliability weighting yields better performance than training on robot data alone, especially when later fine-tuning on new robot tasks. Pretraining Data: Multi-Embodiment and Scale. ACE-Ego-0’s pretraining pool is truly massive and heterogeneous. The authors aggregate roughly 6000 total hours of data from many sources, deliberately covering a wide spectrum of tasks and embodiments:. Robot demos (4.53K hrs): They collect data from multiple physical robots and simulated environments. This includes two models of AgiBot (Alpha/Beta series) demonstrations, data from a humanoid “AgiBot DigitalWorld” sim, Galaxea R1Lite single-arm logs, plus the RoboCasa Tabletop simulator (24 diverse tasks, 1000 episodes each), and an extensive 1800 hours of self-collected Galbot footage. These span vertical, wheeled, and mobile dual-arm robots, at control frequencies 10–30Hz. All robot end-effector poses in these logs are transformed into the head-camera frame using extrinsics, ensuring consistency with the human data. Human egocentric videos (1.48K hrs): They draw from a kaleidoscope of first-person video datasets: Ego4D and its EgoExo4D subset, EPIC-KITCHENS-100 (cooking tasks), HOI4D, the EgoDex daily activity dataset, and a gargantuan 10 million-click Xperience dataset. Together these cover kitchens, garages, workshops, retail scenarios, etc. After the filtering pipeline, they retain ~1478 hours of clips as described earlier. This ensures the model sees a broad “long-tail” of manipulations: opening drawers, cutting vegetables, folding laundry, assembling IKEA furniture – many things no robot might otherwise try. When training, the data is mixed in controlled proportions. The authors sample from each contest of robot vs. human video so one does not overwhelm the other. Crucially, robot data drives the main loss, while human data goes through the auxiliary path. This way, the robot demonstrations (high-precision but limited variety) anchor the learning, whereas the large human corpus (lower-fidelity but extremely varied) supplements it with novel scenarios. Think of it like giving the model a small number of perfect “axioms” (robot logs) and a vast heap of noisy but diverse “examples” (human clips), and teaching it to leverage them both sensibly. The pretraining data standardization is also important. As noted above, the final action representation per arm is a 22-dimensional vector (x,y,z position; 6D rotation; gripper open distance; on/off flag). They normalize gripper distances to a small range and assign a neutral gripper state if the person barely moved their fingers. All data goes through quality checks, throwing out anything with NaNs, zero motion, spikes, or impossible bilateral motion. Architecture Details. At its core, ACE-Ego-0 uses a transformer-based VLM (Vision-Language Model) as the backbone to fuse visual and language inputs, with a separate action prediction “expert” head. The vision-language trunk can be any standard multitask model (e.g. a CLIP variant or video-language model) that encodes the image and instruction. On top of that, an Action Head network takes the language-vision features and outputs continuous action sequences. Key details include:. Action chunks: The model predicts chunks of actions in one go. Specifically, each output is a short trajectory of end-effector targets (rather than just a single step prediction). This is trained with a criterion akin to diffusion or “flow matching”, encouraging the sequence predictions to match the ground truth trajectories over time. The effect is to model smooth, long-horizon motions rather than jerk-by-jerk commands. Morphology tokens: As mentioned, each batch example includes a code that describes the embodiment. For a human-video example, they might use a learned “human-surrogate” embedding. For a robot example, they insert embeddings derived from the robot’s URDF graph. These embeddings are concatenated (or injected via cross-attention) into the action head, telling it “you are controlling this body”. Importantly, the VLM trunk is unchanged; only the action expert sees these condition vectors. Channel conditioning: The action head outputs all 22 dimensions together, but during training the losses can be weighted. For human-derived examples, the loss on each channel is scaled by priors (e.g. X/Y/Z get higher weight than rotation or gripper) to reflect confidence. Overall, ACE-Ego-0’s architecture is quite novel in this field by explicitly disentangling spatial frame alignment and morphology. Many prior approaches would have expected the model to learn these implicitly. Here, by enforcing a canonical camera view and feeding in the robot description, the learning task is much simpler: “predict the right movement in camera space, given this robot shape.”. Experimental Results. The authors evaluate ACE-Ego-0 on a suite of challenging manipulation benchmarks, comparing against state-of-the-art VLA models and baselines. The results are impressive and consistent across settings:. RoboCasa GR1 TableTop: This is a 24-task benchmark of tabletop manipulations (pick-and-place, opening/closing objects, etc.) in simulation. ACE-Ego-0 achieved an average success rate of 72.8%, significantly higher than any published competitor. For context, a previous best model (Pi0.5) scored around 37%, and NVIDIA’s GR00T-N1.6 managed ~47.6%. ACE-Ego-0 not only wins overall, but it improves on nearly every task (grasping, pushing, placing, etc.). The authors highlight that the camera-centric action interface allows the policy to generalize broadly: it even handles tasks with weird camera angles or tall grasp targets that other models struggle with, without any manual retuning. RoboTwin 2.0 (Bimanual): This is a new dual-arm benchmark with both “easy” (simpler objects) and “hard” (articulated, deformable objects) settings. ACE-Ego-0 tops the leaderboard at 91.12% success on easy and 90.62% on hard tasks. It outperforms other multi-arm policies like JoyAI-RA and LingBot-VLA. Importantly, ACE-Ego-0 can coordinate two hands naturally by virtue of the unified action output: it simply outputs two 22D vectors per time step, one per arm, and has learned what bimanual joint motions look like from the human videos (which often involved both hands). The improvement is especially stark on tasks requiring precise coordination (e.g. opening a large box with both arms): here ACE-Ego-0’s visual training from human videos pays off by covering scenarios not seen in limited robot demos. Real ARX Robot (Bimanual): To test real-world transfer, the team deployed ACE-Ego-0 on a dual-arm ARX hardware platform (a retail-automation style setup with Two Hands and a head camera). They evaluated on six complex tasks (pack a shampoo bottle, fold a plastic bag, organize objects, etc.). ACE-Ego-0 achieved about 78.3% average success across these, which is far above baselines. In ablations they report that without human video pretraining the success drops noticeably, even with the same fine-tuning. The reliability-aware training was also crucial for avoiding dangerous failures: by filtering out noisy actions, the policy rarely attempted obviously wrong motions in real tests. Qualitatively, the robot could follow verbal instructions like “put the shoes in the box” and execute multi-step routines robustly. In all cases, ablation studies confirmed the importance of ACE-Ego-0’s components. Removing camera-space alignment (i.e. training in robot frame) caused large performance degradation – the model got confused by conflicting coordinate systems. Removing morphology conditioning again collapsed performance on multi-embodiment tests (the model could no longer handle different robots with one policy). Using a naively equal-weight human loss (no reliability weighting) actually hurt performance, as the policy tried to emulate the overlooked noise in the pseudo-actions. The authors also find that adding human video data especially helps fine-tuning on novel tasks: in low-data regimes, the broad coverage from human data fills in gaps, so the policy adapts faster to new objects or tasks. Overall, the numbers speak for themselves: ACE-Ego-0 is the new state-of-the-art on these unified benchmarks. It sets a new bar (72.8% vs 47% on RoboCasa; ~91% vs ~80% on RoboTwin) and demonstrates that “harnessing 6K hours of mixed human/robot data” is a winning strategy. Significance and Future Directions. ACE-Ego-0 represents an important advance in embodied AI and robot learning:. Scale and Diversity via Human Data: It validates the idea that human egocentric video can serve as training fodder for robots. By creative alignment, we can turn people’s natural behaviors into something a robot can learn from. This dramatically reduces the reliance on expensive robot data. The fact that adding almost 1500 hours of human data improved performance so reliably suggests that foundation-scale embodied pretraining is not just a dream – it can be done today. One Policy for Many Robots: The architecture effectively realizes a “one brain, many bodies” vision. Through camera-space actions and morphology vectors, ACE-Ego-0 shows that a single VLA model can control different robots without retraining. This is a big deal for industries like retail or warehousing, where different locations might use different robot hardware. Deploying one generalized model (with a new URDF input) is far cheaper than training a new model per robot. Bridging Modalities: ACE-Ego-0’s techniques are broadly useful. Any scenario where you have two domains with different coordinate frames or embodiments – e.g. simulation vs reality, one robot vs another, or human vs animal – could potentially benefit from similar “camera-centric” representations. It’s a step toward more universal robot-learning frameworks. Open-Source and Pretrained Weight: Importantly for practitioners, the authors have open-sourced ACE-Ego-0 under an Apache 2.0 license. The project page indicates that pretrained weights, code, and even inference scripts will be available. This makes it easy for researchers and robot integrators to try out ACE-Ego-0 “out of the box.” For someone building a robotic system, the promise is: gather a handful (even a few dozen) of example trajectories on your own robot, feed them into this existing ACE-Ego-0 model (which already understands general manipulation from humans), and get a working controller with minimal new data. Bridging to Real-World Complexity: Finally, ACE-Ego-0 pushes closer to handling the messy, contact-rich tasks humans do. The success on challenging picks, bag folding, and shoe-box assembly suggests these techniques can handle real friction, cloth, and hard-to-simulate interactions. Combined with its robustness measures, ACE-Ego-0 is a promising direction for robots that help in unstructured environments. Looking ahead, there are exciting possibilities. The current model focuses on end-effector actions (6D plus gripper). Future work could integrate full-body poses or dynamic stability (e.g. for humanoid walking tasks). The reliability-aware loss scheme could be extended to more sophisticated uncertainty models. And of course, as more raw video data becomes available (e.g. more Ego4D), this pipeline can scale further. In summary, ACE-Ego-0 demonstrates a practical and effective recipe: “take everything, align it, weigh it by trust, and feed it to one big model.” The fact that this yields a policy outperforming all others on dual-arm benchmarks is a strong endorsement. It’s a prime example of leveraging Internet-scale human behavior to make real robots better – a milestone in the embodied AI story. References: The above draws from Hao Li et al., ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining (arXiv 2026), its accompanying project page, and summary articles from HyperAI. (For brevity we have paraphrased; see the paper for details.).