Latest / Embodied AI 101 / HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations
Transcript
- HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations. The HoMMI project (Whole-Body Mobile Manipulation Interface) from Stanford/TPR introduces a new way to teach robots complex mobile manipulation skills by watching humans perform the tasks, without any teleoperation. In HoMMI, a person simply demonstrates tasks like picking laundry or setting a table while wearing DIY sensors (smartphones) on their head and hands. This “robot-free” data is collected at scale, then used to train a policy that can control a an actual bimanual mobile robot over long-horizon tasks. Crucially, HoMMI explicitly bridges the gap between the human demonstrator’s body and the robot’s body – what the authors call the embodiment gap – so that the robot can follow the human video trajectories even though our bodies differ. In practice, the HoMMI researchers mounted two iPhone cameras on lightweight hand-held grippers (acting as “mock robot hands”) and a third iPhone on a hat. As a human operator walked around performing tasks, these devices streamed synchronized RGB/depth and 6-DoF pose data (at 60 Hz) into Apple’s ARKit framework to fuse it into one global frame. This setup is portable and intuitive: unlike VR-style telemetry it avoids motion sickness by giving the person direct visual and even haptic feedback. The result is robot-free demonstration data that includes an egocentric head‐camera point cloud plus wrist camera images, all aligned to the world. From this raw stream, the HoMMI pipeline constructs an “embodiment-agnostic” representation of the scene. Intuitively, they lift the viewed scene into 3D and anchor everything to the moving gripper frame, masking out the human’s arms so the network doesn’t learn meaningless body differences. Behind the scenes, each head-camera frame is converted into a 3D point map via the depth sensor, broken into small 16×16 patches, and each patch is encoded by a pretrained DINO-V3 vision transformer (ViT) feature. A sinusoidal 3D positional encoding is also added, tying the appearance features to geometric location. Attention pooling then compresses these geometry-aware tokens into a single embedding of the current head view. In parallel, each wrist camera image is fed through the same DINO-V3 encoder (resized to 224×224) to produce a grasping-relevant embedding. All of these visual embeddings (head and wrists), together with the current robot-like joint angles and gripper widths from the mock grippers, form the observation that the policy network sees. Critically, every observation and action is expressed in the gripper’s coordinate frame. In practice, HoMMI fixes the left-hand frame as the origin: the head point clouds and wrist images are transformed so that the left gripper is at the origin. This gripper-centric reference prevents drift between the human head and the robot head – by reasoning relative to the manipulator rather than human height, the policy automatically handles different torso lengths and camera mounts. This 3D representation also lets HoMMI “hide” the human embodiment. The authors mask out any points belonging to the person’s arms by removing points behind the grippers (all points with z<0 in the gripper frame are discarded). This way, the policy never sees a human limb, only the objects and the environment, making it effectively agnostic to the particular demonstrator’s arm shape. The same idea applies to head position differences: instead of recording the human’s exact head orientation, HoMMI represents gaze as a 3D look-at point in space. During training, the system computes where the human is “looking” by intersecting the head camera ray with the scene. The learned policy then simply outputs a target 3D point to look at. At execution time on the robot, this abstract point is converted into a feasible neck angle – the robot’s 2-DOF neck is commanded so that its camera centers on the point. In other words, the policy learns what to look at, not how to twist the neck joint. This decouples the vision intent from the robot’s physical constraints, solving the kinematic mismatch between a human’s flexible neck and the robot’s limited head joints. The authors show that this “relaxed” head control retains all the benefits of active perception, without forcing bizarre or impossible head motions to literally mimic the human demo. Once all observations are embedded into this gripper-centric, 3D-aware format, HoMMI uses a diffusion-based policy network to predict actions over a short future horizon. Concretely, they adopt a transformer-based diffusion policy (sometimes called DiT) that takes the recent sequence of observations and outputs a sequence of actions for the next ~1.5 seconds. Each action in HoMMI’s output is a 23-dimensional vector: it contains two 9-dimensional gripper poses (each pose is 3D position plus a 6D rotation matrix for orientation), one 3D look-at point for the head, and two gripper opening widths. At training time, they use a DDIM diffusion scheduler to teach the model to denoise these multi-step action trajectories end-to-end. The network is fairly deep – about 10 transformer layers with 768-dimensional tokens and 12 attention heads – but it ingests all views simultaneously, so in effect it learns to coordinate vision from the head, vision from the hands, and proprioception all at once. This design contrasts sharply with previous “handheld robot teaching” systems. For example, the original UMI interface (Universal Manipulation Interface) used hand-held grippers but no head camera, so demonstrators had no global view. Those methods were limited to short-range or fixed-base tasks. In HoMMI, the third headset camera adds global context, at the cost of introducing a big human-to-robot gap in the data. The key novelty of HoMMI is that it embraces the egocentric view but then theoretically closes the gap. The combination of a 3D latent scene and the look-at action exactly does this: the policy no longer cares whether the point cloud came from a 1.7m-tall human or a 1.5m robot, nor does it try to imitate impossible neck rotations. In effect, the learned “hand-eye” controller sees only the essential geometry and contact cues, agnostic to human shape, and focuses on what needs to be done rather than how the human happened to move their limbs. At execution time, the trained HoMMI policy runs on a real mobile manipulator: the team used a Rainbow Robotics RB-Y1 platform. This robot is a dolly with two 7-DOF arms mounted on an omnidirectional base, plus a 2-DOF neck and a 6-DOF torso lift. To match the demonstrations, the robot was fitted with custom fin-ray fingers similar to the grippers the demo had used, and it carried a stereo head camera and RGB wrist cameras on the wrists – just like the smartphone setup. An external computer streamed camera images (over GigE/PoE) into the policy, which output whole-body trajectories. Finally, a constraint-aware whole-body controller took the desired end-effector and look-at trajectories and computed velocities for the base, torso, arms, and neck simultaneously, respecting joint limits and collision avoidance. In short, the controller acts like a full-body inverse kinematics: it makes the robot’s hands and gaze follow the plan as closely as possible, by making the arms move, base drive, and torso lift together. The authors note that this coordinated control is essential for their long-horizon tasks, since many will require, say, moving the base while keeping a hand on an object – all under the robot’s physical limits. The HoMMI team evaluated their approach on a range of complex tasks that combine navigation, manipulation, and perception. These included things like tidying laundry, delivering objects across the room, and setting a tablescape – tasks where the robot often must travel to a cupboard or bin, look around, grasp items with both hands, and place them correctly. Eschewing any expensive teleoperation, they collected dozens of human demos in a mock home environment using the smartphone interface. According to their reported results, the HoMMI-trained policies transferred directly to the RB-Y1 robot, allowing it to perform these bimanual, navigation-rich tasks with high success. In fact, the project homepage stresses that “HoMMI successfully transfers policies from human demonstrations to a robot with different embodiment” and achieves “precise, long-horizon manipulation tasks in large, real-world environments”. Quantitatively, HoMMI outperformed several baselines on every evaluated task. Those baselines help clarify why HoMMI’s design matters. For example, one baseline removed the head camera completely and relied only on the wrist cameras (essentially UMI-style). This wrist-only system could not reliably find targets that were out of the immediate hand-view, so it either wandered aimlessly or bumped into obstacles. In a laundry task, for instance, a wrist-only policy would fail to locate the laundry bin or align properly with it, because its field of view was too narrow. Conversely, a “head-only” baseline (ignoring the wrist cameras) had the global perspective but not the fine detail needed to grasp; it would navigate roughly to the object but then perform wildly inaccurate grasps and alignments, often missing the target. HoMMI’s two-stream approach solved both problems at once: the head camera provided the bird’s-eye context so that the robot could find doors, bins, or tables and align broadly, while the wrist cameras provided the local detail for pinch grips and close placement. In the authors’ words, using both views “combines local contact cues with global scene context” and dramatically boosts robustness. Another baseline they tried was simply appending the head RGB image to the input and regressing a full 6-DOF head pose (like trying to imitate the human head exactly). This “naive egocentric” version did worse than HoMMI. In practice, the regressors struggled to accurately reproduce the subtle head orientations from the noisy data, leading to erratic robot neck motions and poor tracking. By contrast, HoMMI’s abstract 3D-look-at representation avoided that instability. The backend experiments found that when they disabled active head control entirely (freezing the head camera), performance on search/alignment steps plummeted. In short, active perception – moving the head to gather information – was crucial for success, but it must be done intelligently rather than as uninformed mimicry. To highlight the effect of these design choices, the authors even provided attention visualizations. The full HoMMI policy consistently focused its “attention” (in the transformer sense) on task-relevant objects: the graspable handles, the bins, the tableware, etc. Baseline policies without the full multimodal setup often had diffuse or irrelevant attention maps, fixating on empty floor space or their own gripper. In their write-up they note, “Our method learns informative attention on task-relevant objects, while baselines attend to irrelevant regions”. This speaks to the careful cross-embodiment tuning: by filtering out the demonstrator’s body and training the network on the meaningful parts of the scene, HoMMI ended up with more consistent, useful perceptual features. Overall, the reported success rates and qualitative trials underscore that the combination of techniques was more than the sum of its parts. The trained robot was reportedly able to pick up objects, reorient them, navigate between rooms, and even cooperate bimanually (e.g. one hand holding and one hand moving an item) in tasks that would defeat simpler approaches. In fact, the press surrounding HoMMI emphasizes the “scalable, robot-free, in-the-wild data collection” enabled by this interface, as well as the removal of reliance on teleoperated training data. They argue this could pave the way for far larger datasets (imagine people everywhere collecting demos with phones) that were previously impractical. HoMMI joins a growing interest in using egocentric human data to train robots. For instance, the recent EMMA system (by Zhu et al. at Georgia Tech/MIT) similarly pursues mobile manipulation via human full-body demonstrations, though EMMA co-trains those with static-robot data to improve generalization. Unlike HoMMI which needed no teleoperation data at all, EMMA still uses a small amount of robot data as an anchor. There are also works like DexPilot that use vision-based human demonstration for dexterous tasks, but those typically operate in simulation or static-arm settings. What sets HoMMI apart is the hardware integration and full-stack policy: it demonstrates that an end-to-end diffusion model can directly drive a real mobile hardware setup from purely human video. Of course, there are practical caveats. HoMMI relies on a relatively expensive setup (multiple iPhones and ARKit syncing) and on a specific robot platform. Its performance in truly novel environments (with layouts far from the ones seen in demos) wasn’t fully explored in the public materials. And while the authors handle height differences to some extent – indeed, they point out that their 3D embedding lets people of different heights demo interchangeably – extremely different morphologies (e.g., a very short operator for a tall robot) might still pose edge cases. The diffusion policy is also fairly heavy-weight (many layers, running on a GPU) which might be a limitation for low-latency on-board control in the future. Nonetheless, the results here suggest those challenges can be addressed. In summary, HoMMI is an impressive synthesis of ideas: it combines a clever data interface, a perceptual trick (3D egocentric embedding), a novel action representation (look-at point), and modern sequence modeling (diffusion transformers) to achieve the goal. By successfully “closing the embodiment gap” between human demonstrators and mobile robots, it enables complicated long-horizon tasks to be taught without any robot in the loop at demonstration time. As Xu et al. report, this approach “enables long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination, navigation, and active perception”. For practitioners, the practical takeaway is that one can gather portable human demonstrations (using phones or other wearable cameras and gripper proxies), then use tools like HoMMI’s policy architecture to train robots – without ever needing the robot’s remote or simulator. This could greatly scale up the kinds of training data we can collect. Future work might include generalizing to more robot types, incorporating language or semantic constraints, or combining this with online fine-tuning. But as it stands, HoMMI represents a noteworthy step toward more natural, scalable robot learning from people’s actions. Sources: Details above are drawn from the HoMMI project site and preprint (Xu et al. 2026), which describe the interface, network architecture, and experimental findings. Where helpful, content has been paraphrased from their publicly available documentation and key contributions summaries.