Latest / Embodied AI 101 / EgoHumanoid: Turning Human First-Person Experience into Humanoid Loco-Manipulation
Transcript
- EgoHumanoid: Turning Human First-Person Experience into Humanoid Loco-Manipulation. EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration • Modi Shi, Shijia Peng, Jin Chen, Haoran Jiang, Tianyu Li, Di Huang, Ping Luo, Hongyang Li, Li Chen • The University of Hong Kong, Shanghai Innovation Institute, Beihang University • Robotics: Science and Systems (RSS) • 2026. The headline, stated precisely. Humanoid learning has an awkward data problem. The environments where we want humanoids to work—homes, shops, gardens, offices—are exactly the environments where collecting humanoid demonstrations is most difficult. Transporting the machine is inconvenient. Teleoperation equipment is cumbersome. Falls are costly. Hardware reliability declines outside a controlled laboratory. And every minute spent collecting data ties up an expensive robot, a trained operator, and usually several support engineers. EgoHumanoid asks whether the robot needs to travel with the data collector at all. Instead, a person wears a VR headset, body trackers, and an egocentric camera, performs the desired task in natural environments, and returns with synchronized video and motion traces. An alignment pipeline then converts those demonstrations into observations and actions that can be mixed with conventional robot demonstrations when fine-tuning a vision-language-action policy. The resulting policy is deployed on a 1.3-meter Unitree G1 and evaluated on tasks combining one-to-five meters of locomotion with object interaction. There are three qualifications worth making immediately. First, “robot-free” describes the human demonstrations, not the complete training recipe. The headline EgoHumanoid model is co-trained with a limited set of humanoid demonstrations collected in the laboratory. The major contribution is avoiding in-the-wild robot data collection, rather than eliminating robot data entirely. Second, the main “generalization” environments are absent from the robot dataset but represented in the human demonstrations. This is therefore cross-embodiment generalization into robot-unseen scenes, not zero-shot generalization to scenes unseen by both data sources. Third, the reported 51-point gain—from 31% to 82%—uses an average stage-completion score. A four-stage trial that completes three stages receives 75%, even if the overall task ultimately fails. The result is still substantial, but it should not be read as an 82% binary end-to-end success rate or as a 51% relative improvement. With those distinctions established, EgoHumanoid becomes more interesting, not less. It is a concrete demonstration that diverse human experience can supply the environmental and behavioral coverage missing from robot datasets, while a much smaller quantity of robot data anchors that experience to the machine’s contact mechanics and kinematics. Why loco-manipulation changes the transfer problem. Human-video transfer is already difficult for a fixed manipulator. Human hands and robot grippers look different, have different workspaces, and follow different dynamics. Humanoid loco-manipulation compounds that gap. The first issue is visual. A typical demonstrator is between 1.6 and 1.8 meters tall, while the G1 used here is about 1.3 meters tall. Even if human and robot wear nominally identical head cameras, tables, shelves, handles, and containers appear at different elevations and under different perspective distortion. Locomotion causes those differences to evolve throughout the episode rather than remain a fixed camera calibration error. The second issue is morphological. Directly copying human joint angles is inappropriate when limb lengths, joint limits, balance strategies, and hand structure differ. Human walking also contains torso sway and head stabilization behaviors that are not desirable commands for a bipedal controller. The third issue is observability. Fixed-base policies often see objects from repeatable camera poses. A walking humanoid rarely stops at precisely the same location or orientation. Small navigation errors become large viewpoint and reachability changes during manipulation. EgoHumanoid’s authors estimate that comparable loco-manipulation behaviors required roughly two to three times as many demonstrations as fixed-base tasks of similar manipulation difficulty. The paper’s central hypothesis is that low-level motion does not need to transfer directly. What transfers is the structure above it: where to walk, how to approach an object, when to turn, which hand should move, when to close the gripper, and how a long task decomposes into phases. View and action alignment are designed to expose this shared structure while discarding the most embodiment-specific details. A full-stack alignment system. EgoHumanoid is less a novel policy architecture than a full-stack data interface around an existing VLA. Its pipeline has four parts:. Collect synchronized human egocentric video and tracked body motion. Collect a smaller set of robot-native demonstrations through VR teleoperation. transform both sources into a common visual and action format. Fine-tune a single policy on balanced mixtures of the two datasets. The human and robot episodes contain the same broad semantic event—carry the pillow to the bed, dispose of the trash, transfer the toy, or stow an item in a cart—but originate from different embodiments and environments. Human data contributes scene diversity and natural high-level behavior. Robot data contributes embodiment-correct manipulation and establishes what the low-level controller can physically execute. The policy itself is adapted from Physical Intelligence’s π0.5. It consumes a language instruction and a head-mounted RGB observation, then emits a chunk of commands in a deliberately restricted whole-body action space. A separate whole-body controller handles joint-level execution and balance. In the paper’s terminology, “whole-body” therefore refers to coordination at the task level, not direct prediction of every humanoid joint. Collecting the two halves of the dataset. For human collection, the demonstrator wears a PICO headset with five motion trackers and a head-mounted ZED X Mini. The system records 24 body keypoints, 26 keypoints for each hand, controller poses, and synchronized egocentric video. Motion is initially captured at approximately 100 hertz and later reduced to the policy’s 20-hertz rate. The current open-source setup substitutes the more accessible ZED Mini for the ZED X Mini used in the experiments. This is robot-free but not sensor-free, and it is not arbitrary internet video. Reliable labels depend on tracked wrists, hands, and pelvis motion. Demonstrators are instructed to maintain stable torso posture, consistent wrist orientation, and hand visibility. In other words, EgoHumanoid occupies a middle ground between highly constrained robot teleoperation and passive video learning: collection is portable and far easier to move between scenes, but demonstrations remain deliberately structured for transfer. Robot demonstrations use the same family of PICO hardware. The operator issues discrete lower-body commands with the controllers and supplies wrist targets from controller-to-headset poses. Inverse kinematics and NVIDIA’s GR00T WholeBodyControl stack convert those commands into joint-level motion on the G1. The Dex3 hands receive binary grasp commands from the controller triggers. This symmetry in sensing and operator interface reduces some accidental differences before learning even begins. The main experiments use 100 robot episodes per task and add 300 human episodes per task for co-training. That is not foundation-model scale, but it is enough to investigate whether environmental diversity can cross the embodiment boundary. The release includes the collection and processing code plus small human and robot example datasets on Hugging Face; the repository explicitly labels these as samples, so the public artifact should not be mistaken for the full experimental corpus. View alignment: move the camera before learning the policy. The most visually distinctive part of EgoHumanoid is its attempt to synthesize what a lower robot-mounted camera would have seen from the human recording. The pipeline begins with MoGe, a monocular geometry estimator that produces a per-pixel 3D point map and scale-invariant depth. Those points are transformed into a target camera frame corresponding to the humanoid’s lower head position and then projected back into an image. The paper simulates a downward camera translation of 25 centimeters, with random perturbations of plus or minus five centimeters to prevent the policy from depending on a single exact camera height. Reprojection creates holes. Some surfaces become visible from the lower virtual camera even though they were hidden in the original human frame. Other pixels lack valid geometry because monocular depth estimation is imperfect. EgoHumanoid fills these regions with Stable Diffusion 2 inpainting, using 20 denoising steps and a guidance scale of 7.5. The final training image is therefore neither a simple crop nor a geometrically pure render: it combines observed pixels, estimated 3D geometry, and generated content. This is an important design choice. Rather than demand that the visual encoder become invariant to a large systematic viewpoint shift, EgoHumanoid modifies the data so that human observations occupy a more robot-like visual distribution. The ablation reports improvements across all four tasks, with the largest effect on Toy Transfer and Cart Stowing, where objects appear at varying elevations. There is also a practical lesson in the code release. The paper reports a 25-centimeter shift, while repository examples use seven centimeters and the command-line default is ten centimeters. These values should be treated as embodiment- and mounting-specific calibration parameters, not universal constants to copy unchanged. The released implementation supports arbitrary translations, perturbation magnitudes, batch processing, and replacement of the source image within HDF5 episodes. The risk, of course, is label-preserving hallucination. An inpainting model can produce plausible pixels without preserving the exact geometry relevant to contact. EgoHumanoid does not separately quantify geometric errors introduced by inpainting. My reading is that this is acceptable primarily because the transformation is intended to improve broad viewpoint compatibility, while precise contact behavior remains anchored by robot demonstrations. Action alignment: share intent, not joints. Action alignment carries most of the conceptual weight. The authors do not attempt to make a human move like a G1. Instead, they define a command interface that both embodiments can produce. For the upper body, each arm is represented by the change in its six-degree-of-freedom end-effector pose between consecutive frames: three translational changes and three rotational changes. Human wrists are expressed relative to the pelvis, smoothed, and downsampled from 100 to 20 hertz. Rotations are smoothed in the tangent space of three-dimensional rotations rather than by naïvely averaging quaternions. Robot demonstrations are converted to the same delta representation. Relative actions are a sensible compromise. Absolute wrist positions would preserve large systematic differences caused by human and robot arm lengths. Joint-space retargeting would require solving for a robot configuration and could distort the hand-object geometry that matters to manipulation. Delta poses instead say, approximately, “move the left hand slightly forward and rotate it this way,” independent of the absolute distance between head, torso, and wrist. For locomotion, the human pelvis trajectory is converted into the same primitive interface used during robot teleoperation. After smoothing, the pipeline estimates heading, projects world-frame displacement into a pelvis-local frame, computes forward and lateral speed, and derives yaw rate. Each of these motion dimensions is quantized into three bins. Vertical pelvis change supplies a separate height command for standing or squatting. The lower-body policy therefore does not imitate human footsteps. It imitates a semantic navigation command sequence. A stable locomotion controller decides how the G1 should realize “move forward,” “translate left,” “turn right,” or “lower the body.” This abstraction is one reason navigation transfers much better than fine manipulation: the representation intentionally removes most human-specific walking dynamics. Hand articulation is compressed even more aggressively. Each hand becomes a binary open-or-closed command. The paper derives this label from smoothed finger curvature across the 26 tracked joints. The open-source processing pipeline operationalizes the estimate with a weighted combination of fingertip distances and curvature, followed by square-wave fitting and optional manual transition adjustment. That engineering detail reveals how difficult it can be to turn noisy human tracking into clean imitation targets. Combined, the policy outputs an 18-dimensional action: 12 dimensions for two end-effectors, three locomotion commands, two binary hand commands, and one height-change command. What is transferred is consequently a compact sequence of navigation, wrist, and grasp intentions—not a full human motion trajectory. The VLA sits above a whole-body controller. After alignment, a single π0.5-based policy can be trained on human and robot episodes without separate output heads. The input RGB image is resized to 224 by 224 pixels. Proprioceptive state is intentionally omitted because human and robot joint-state distributions are incompatible. Language provides the task instruction, and the policy predicts chunks of 50 future actions. At 20 hertz, 50 actions represent 2.5 seconds of motion, but deployment is receding-horizon rather than fully open loop. A forward pass takes approximately 130 milliseconds on an RTX 4090. When a new chunk arrives, already elapsed actions are removed and the overlapping portions of old and new chunks are linearly blended. This compensates for inference latency and suppresses discontinuities before the command stream reaches the whole-body controller. Training is not lightweight: the reported setup uses eight A100 GPUs, an effective batch size of 256, bfloat16 precision, and 20,000 optimization steps. The public repository supports both full and LoRA fine-tuning, estimates more than 70 gigabytes of GPU memory for the former and roughly 22.5 gigabytes for the latter, and packages data in LeRobot format. It also supplies the collection, view transformation, training-server, and G1 deployment paths rather than only publishing model code. The absence of proprioception is simultaneously what makes co-training clean and what limits precision. From one RGB frame, the model may not know the robot’s exact wrist orientation, accumulated action error, or joint-limit proximity. The authors explicitly identify orientation ambiguity as a central failure mode for rotation-sensitive manipulation. Evaluation: four tasks and two kinds of generalization. EgoHumanoid evaluates four behaviors. In Pillow Placement, the robot carries a bulky pillow, approaches a bed, squats, and places it near the head of the bed. Trash Disposal requires carrying an object and inserting it horizontally through a container opening. Toy Transfer adds approach, bimanual grasping, turning, carrying, and placement. Cart Stowing requires pushing a cart under sustained contact, reaching for a product, placing it into the cart, and resuming cart motion. Each task includes between one and five meters of locomotion, and each evaluation setting contains 20 trials with position perturbations. The in-domain scenes resemble the laboratory robot data. The cross-embodiment generalization scenes include a conventional bedroom, an outdoor garden, home furniture, and a real convenience store—environments represented in the human demonstrations but absent from robot collection. Tasks are decomposed into ordered locomotion- and manipulation-dominant stages. The paper records whether every stage succeeds and defines a trial as fully successful only when all stages complete. Its main plots, however, average the binary stage outcomes. This gives useful credit for partial progress and enables subskill analysis, but it softens the compounding difficulty of long-horizon execution. What the results actually show. In laboratory-like environments, the robot-only model obtains a 59% average stage score, while co-training reaches 78%. In environments represented only by the human demonstrations, the difference expands from 31% to 82%. The headline “51% improvement” is therefore a 51-percentage-point increase in the stage-average metric. The most important result is not merely that human data helps. It is where it helps. A human-only policy reaches 100% on the first navigation-dominant stage of all four tasks. It also performs well on coarse manipulation such as pillow placement and trash disposal. Human locomotion traces clearly teach the policy how scenes unfold: how quickly targets expand in the image, when an approach is complete, how to turn while carrying an object, and what visual context predicts the next phase. Performance falls as contact precision increases. On the cart task’s initial toy-grasping stage, human-only training reaches 5%, robot-only reaches 15%, and co-training reaches 60%. That result is more revealing than either source in isolation. Human data appears to provide environmental and phase-level priors, while robot data supplies the embodiment-specific correction that turns those priors into a workable grasp. The combination is not simply averaging two policies; it lets one data source make the other more useful. Failure analysis supports that interpretation. The human-only policy produces roughly three times as many manipulation failures as locomotion failures. Robot-only failures are more evenly divided because the model lacks scene coverage for navigation but has seen robot-native contacts. Co-training reduces both categories. Adding human demonstrations from zero to 300 consistently improves performance across the tested sampling ratios. The best robot-to-human ratio is task dependent. Navigation-heavy or coarse-contact tasks prefer more human data, with a one-to-two robot-human ratio. Precision-heavy Toy Transfer and Cart Stowing prefer two robot samples for every human sample. A universal 50-50 mixture is therefore unlikely to be optimal; sampling should reflect where embodiment-specific supervision is most valuable. The scene-diversity experiment is especially relevant for dataset builders. With robot data fixed, Trash Disposal’s score in a genuinely unseen test scene rises from 57.5% with robot data alone to 75% with human data from one scene and 82.5% with human data from three scenes. The trend is non-decreasing even though the one- and two-scene conditions tie. This suggests that the value of human data is not only faster motion collection; it is access to environmental variation that would otherwise never enter the dataset. The efficiency claim deserves similar precision. Robot demonstrations take 62.1 seconds per episode on average, compared with 39.7 seconds for human demonstrations. That is about 36% less time per episode, or roughly 1.56 times the throughput—not quite two times overall. The simpler pillow and trash tasks do approach or exceed a twofold difference, while Cart Stowing remains time-consuming for humans because most of its duration is intrinsic to the task. The lineage: from UMI to whole-body human transfer. EgoHumanoid belongs to a rapidly converging family of systems that make human demonstrations directly actionable. Universal Manipulation Interface, or UMI, established a highly influential recipe: use portable handheld grippers with onboard cameras and tracking, represent actions as relative end-effector trajectories, and carefully match deployment latency. Its demonstrations are robot-free and can produce precise, dynamic, bimanual policies, but the collection hardware intentionally resembles the target gripper and focuses on manipulation rather than bipedal navigation. EgoHumanoid trades some of that contact fidelity for a more wearable setup and explicit locomotion supervision. EgoMimic is the clearest precursor to EgoHumanoid’s co-training philosophy. It records video and 3D hand tracks with Project Aria glasses, visually masks embodiment-specific arms, and trains shared human-robot manipulation policies. Human Policy ~ Human Policy similarly introduces a unified human-humanoid action space and co-trains a Human Action Transformer on egocentric human and humanoid manipulation. EgoHumanoid’s distinguishing step is extending this logic to multi-meter loco-manipulation on a biped, with an explicit lower-body command interface and camera-height transformation. EMMA brings human full-body data into wheeled mobile manipulation, combining it with static robot demonstrations so that mobile teleoperation is unnecessary. EgoMI instead emphasizes active vision: it records coordinated human head and hand motion and uses memory to handle rapid viewpoint changes on a semi-humanoid robot with an actuated head. These systems expose two alternatives to EgoHumanoid’s fixed-head, image-warping approach—model locomotion phases explicitly, or make head motion part of the policy. The most revealing contemporary comparison is HoMMI, also presented at RSS 2026. HoMMI extends UMI with a head-mounted phone, embodiment-agnostic 3D visual features, a relaxed head-action representation, and a constraint-aware controller. It learns whole-body mobile manipulation on a wheeled RB-Y1 directly from robot-free human demonstrations, without teleoperation data. EgoHumanoid tackles the harder balance problem of a biped and uses a generalist VLA, but retains robot demonstrations for manipulation fidelity. HoMMI eliminates those robot demonstrations by investing more heavily in specialized collection hardware, 3D representation, and controller structure. The broader design space is becoming clear. One can minimize the embodiment gap in hardware, as UMI does; edit observations and actions into alignment, as EgoHumanoid does; learn a shared latent embodiment, as HAT-style approaches do; or push more physical reconciliation into a whole-body controller, as HoMMI does. Future systems will probably combine all four. Limitations that matter in deployment. EgoHumanoid’s largest limitation is also the source of its scalability: its action abstraction discards information. Binary grippers cannot express dexterous preshaping or graded force. Three-bin locomotion commands cannot describe precise foot placement. Delta wrist actions lose absolute configuration information. The policy coordinates whole-body behavior, but the expressive motion repertoire is bounded by the downstream controller. Omitting proprioception prevents a direct human-robot state mismatch, yet creates partial observability. The robot may visually encounter two similar frames while its wrists occupy different orientations or its controller has accumulated different tracking errors. The authors report that rotation-sensitive behavior is particularly difficult for this reason. An embodiment-specific state adapter, state-token masking, or modality dropout could retain robot proprioception without requiring humans to provide an equivalent signal. The visual alignment pipeline also assumes that camera-height mismatch is the dominant visual problem. It does not remove human hands, synthesize robot arms, or explicitly preserve contact geometry through inpainting. The paper intentionally avoids hand replacement because detection and rendering errors could introduce another failure source. That minimalist decision is reasonable, but future policies may need a more principled 3D representation when manipulated objects are heavily occluded by the demonstrator. Collection is portable rather than effortless. Demonstrators must avoid excessive torso sway, keep their hands visible, and follow consistent wrist conventions. High-speed movements, hand occlusion, or arbitrary human behavior would require better SLAM, tactile sensing, and tracking. Scaling this setup to hundreds of people also raises calibration, privacy, consent, and contributor-attribution issues. Finally, the evidence covers four carefully designed task families. The policy is language-conditioned, but the experiments do not demonstrate broad open-vocabulary instruction following or spontaneous task composition. The central result is a data-transfer result: a VLA can absorb human and robot trajectories in one aligned interface and use human scene coverage during robot deployment. How I would adapt EgoHumanoid. For a research group building on this work, the first decision is not which VLA to use. It is which parts of the task are embodiment-agnostic enough to learn from humans. Use human demonstrations aggressively for navigation routes, object search, approach strategy, scene variation, and coarse placement. Concentrate scarce robot demonstrations around contact transitions: final approach, grasp establishment, insertion, rotation, release, and sustained force. EgoHumanoid’s sampling experiments suggest that manipulation-heavy episodes should appear more frequently even when the raw human corpus is much larger. Second, design the common action space around capabilities already supported by a reliable low-level controller. Delta end-effector targets are useful only if inverse kinematics and whole-body control can track them under base motion. Discrete locomotion is a defensible starting point for stable deployment, but tasks involving stairs, pedals, narrow gaps, or stepping around obstacles will require richer motion targets and probably human-motion retargeting. Third, treat view alignment as calibration rather than generic augmentation. Measure the actual human and robot camera extrinsics, object-height distribution, and expected stopping-pose variance. Inspect the reprojected geometry around handles, rims, and object boundaries rather than evaluating only image realism. Where inpainting frequently covers task-critical regions, consider 3D point features, multi-view observations, or explicit validity masks. Fourth, retain some robot-specific feedback. A practical extension could provide RGB and language to a shared cross-embodiment backbone while injecting robot proprioception only into a small execution head. Training-time modality dropout could force the policy to function without that channel on human episodes while exploiting it during robot execution. This directly targets EgoHumanoid’s orientation ambiguity without requiring human joint states to resemble robot joint states. Finally, evaluate three splits separately: scenes seen by both data sources, scenes seen only in human data, and scenes seen by neither. Report binary full-task success alongside stage completion, recovery behavior, intervention count, and failure severity. EgoHumanoid convincingly demonstrates human-to-robot scene transfer; a stricter split would reveal how much of that benefit becomes genuine environmental generalization. The open repository makes these experiments unusually approachable. It provides human and robot collection tools, HDF5 processing, viewpoint transformation, LeRobot conversion, π0.5 fine-tuning, policy serving, and G1 deployment. Researchers should nevertheless reconcile paper and repository settings—particularly camera-translation distances, sensor models, gripper-label processing, and generic OpenPI examples—before treating the release as a one-command reproduction. Final takeaway. EgoHumanoid does not show that humanoid learning has become robot-free. It shows something more immediately useful: robot experience and environmental experience no longer have to come from the same embodiment. A few robot demonstrations can describe what the machine can physically execute. A larger human corpus can describe where, when, and why those actions should occur across varied environments. View alignment makes the scenes visually compatible; action alignment expresses shared intent; and co-training lets the VLA combine coverage from humans with precision from the robot. The most convincing evidence is the asymmetry in the results. Human data transfers navigation almost directly. Precise manipulation still needs embodiment-specific grounding. That boundary is not a failure of the method—it is a map of where the field should invest next. If future systems can preserve EgoHumanoid’s portable scene coverage while closing the remaining contact, orientation, and active-perception gaps, human activity may become one of the largest practical data sources for general-purpose humanoid control.