Latest / Embodied AI 101 / EgoNav: Diffusion-Based Humanoid Navigation from Human Egocentric Video
Transcript
- EgoNav: Diffusion-Based Humanoid Navigation from Human Egocentric Video. Robots that can confidently navigate the real world have usually relied on explicit motion planning, sensor fusion, or reinforcement learning with heavy engineering. A new approach from Stanford’s AI Lab—EgoNav—challenges this paradigm by training entirely on a few hours of humans walking around with cameras. EgoNav uses a diffusion generative model to predict distributions of future paths, conditioning on a fused 360° view of the environment and a few seconds of past motion. Remarkably, it requires no robot-specific demonstrations or fine-tuning. After training on just 5 hours of egocentric human walking data, the system can be deployed zero-shot on a Unitree G1 humanoid robot. In real-world tests, EgoNav naturally handles challenging cases like closed and glass doors, crowds of people, and dynamic obstacles, simply by sampling from its learned navigation prior. This article explains how EgoNav works, why it matters, and what it might inspire in embodied AI. EgoNav’s key insight is to treat navigation as one big multi-modal prediction problem: given where the robot has been and what it sees, output entire future trajectories that humans would plausibly take. The approach is embodiment-agnostic – it learns a prior over walks from people, then applies that distribution to the robot’s motion. A frozen pretrained vision model (DINOv3) and semantic segmentation provide rich perceptual context, capturing things like the outlines of glass walls or walking pedestrians. A novel 360° visual memory encodes the surrounding scene in a compact form. A diffusion-based trajectory generator then produces many candidate paths, and a simple receding-horizon controller picks one that avoids collisions. By combining these elements, EgoNav achieves real-time performance (≈1.7 Hz planning on an onboard Jetson) and demonstrates robust zero-shot navigation. The results are striking: without ever seeing a robot or a robot’s dynamics, EgoNav negotiates corridors, open spaces, doorways and even moving crowds just by analogy to how humans walk. In one trial, the robot stopped at closed doors rather than walking into them; in another, it patiently paused for pedestrians to pass. Glass walls – invisible to a regular depth sensor – were naturally avoided because the vision backbone recognized them from appearance. EgoNav’s creators validated its safety offline (outperforming baselines in fewer collisions) and then ran a 37.5-minute real-world test covering over 1,100 meters, achieving 96% autonomy. All of this comes from a diffusion planner trained on five hours of wrist-mounted human video. In the sections below, we unpack how EgoNav was built and why it works. We start with the dataset and the core components (visual memory, diffusion model), then explain the hybrid sampling strategy and control loop. Finally, we review the experimental results – both in simulation and on the real robot – and discuss how this line of work fits into the broader trend of learning navigation from passive human data. Throughout, we cite from the authors’ technical report to highlight key claims. The references from EgoNav’s paper and related literature provide context for the innovations introduced here. Learning a Navigation Prior from People. Most robot navigation systems train on robot-specific data or hand-coded maps. EgoNav flips this on its head: it learns only from humans. The Stanford team collected a small dataset of humans walking with wearable RGB-D cameras around a university campus. In total they put together 300 minutes (5 hours) of walking footage, covering 25+ kilometers in varied environments (buildings, outdoor paths, autumn/spring weather, crowds, etc.). The data is at 20 Hz and is richly annotated: at each timestep the system holds the person’s full 6-DoF pose, a color image, a depth map, and an 8-class semantic segmentation of the scene. Critically, as the human moves, the past frames are accumulated into a 360° visual memory (more on that below). This dataset is quite small by machine learning standards, but it is dense and multimodal. Why learn from humans? Humans are exceptional at navigating novel scenes, implicitly integrating visual cues and social rules. By watching them move, EgoNav bypasses having to manually label obstacles or tune path planners. To ease the configuration gap, the authors only use inputs that a real robot would have: egocentric camera (RGB-D) and odometry of where the person walked. There is no mocap system or external tracker. Because the training is fully self-supervised, no human-labelled “correct path” is needed; the system simply learns the distribution of human paths through the environment. As the authors note, EgoNav is “human-native (no robot data), scene-aware (egocentric observations only), distributional (multi-modal predictions), and robot-ready (real-time…)”. In other words, it only sees what a human head-mounted camera sees, and it predicts where humans would likely walk next. Then the same distribution is used to drive the robot’s motion, after the fact. This idea of an embodiment-agnostic navigation prior is powerful. The model never learns anything about the robot’s shape or gait – only that “paths that humans take in these visual contexts tend to avoid obstacles and follow logical corridors.” In practice, EgoNav can do this because the Unitree G1 humanoid has a fairly human-like height and movement style. The robot’s sensors and speeds are not wildly different from a walking person’s, so the transfer isn’t impossibly biased. The key is that the model isn’t trying to imitate low-level dynamics (like pushing feet to accelerate). It simply generates nominal waypoints or trajectories; the robot’s controller handles the physical execution. In sum, the EgoNav dataset and training setup provide a walking prior: a probabilistic map from camera+motion history to entire future trajectories. We can think of it as a big human path database distilled into a learned network. Next we explore how the visual observations and past motion are encoded into this navigation prior. Visual Memory: A 360° Egocentric Map. A unique feature of EgoNav is its visual memory module. Unlike a conventional robot with LIDAR or SLAM, here the system only has a wearable camera. At each time step, EgoNav collects the recent RGB-D frames and merges them into a spherical panorama centered on the agent. In practice, the pipeline does the following: when a new RGB-D frame arrives, it is converted into a point cloud (using the depth), filtered by range, and then transformed into the current egocentric frame. In other words, as the camera moves, past frames are projected into the current coordinate system. Over a few seconds, this generates a wide-span point cloud of the surroundings. The points are painted by the stored colors (RGB) and semantic labels (the semantic segmenter from the images). Then this 3D map is reprojected into a 360° equirectangular image (180×360 by 5 channels). The five channels are (R,G,B, depth, semantic). Because adjacent frames overlap a lot, this fused panorama accumulates a nearly full 3D context around the wearer. The end result is a fixed-size “image” of shape 180×360×5 that captures the entire surrounding scene in front of, behind, and to the sides of the agent. (For reference, 180×360 is a standard full-sphere view, 2×360 degrees horizontally and ~180 vertically.) Intuitively, this visual memory serves as a compact map of where obstacles, doors, etc. are relative to the person. By contrast, a single forward-facing camera only sees ~90° of the world and misses things behind or to the side. The memory includes semantic information (e.g. floor, wall, door, vegetation, people) which helps categorize different obstacles and affordances. To actually feed this memory into the prediction model, EgoNav uses a small autoencoder. A pretrained spatial VAE (convolutional variational autoencoder) compresses the 180×360×5 panorama into a 64-dimensional latent vector. This encoding is very lightweight and fast to compute: on the robot’s Jetson Orin NX computer the encoding takes only ~30 ms per update. Thus, at every planning step the system has a current 64D memory embedding plus the raw segmentation labels for the latest frame. Feeding a 64D vector instead of a 5-channel image makes the diffusion model easier to train. Yet the latent still carries rich geometric and semantic context about the environment. This visual memory is one of EgoNav’s main strengths. By fusing multi-frame data, it makes the system “scene-aware” in the abstract sense. For instance, if the human has walked down a hallway and turned around, the memory retains anything left in the backpack or the sides of the hall. In the real robot, this means it won’t accidentally walk into a piece of furniture that is just behind it, because the memory “remembers” it exists. In technical terms, the paper notes that the memory “captures far more scene context in one compact representation” than a single camera frame. We will see in the experiments that removing the memory (using only a normal forward camera) markedly degrades EgoNav’s ability to avoid collisions. In summary, EgoNav builds a full 360° egocentric map from egocentric vision during both training and deployment. This map is key to enabling the diffusion model to consider obstacles in all directions when predicting trajectories. In the next section, we turn to how the model produces those future trajectories based on the memory and past motion. Diffusion-Based Trajectory Prior. At the core of EgoNav is a diffusion generative model that takes as input the current scene and motion history and outputs a probabilistic distribution of future trajectories. Let’s break down its inputs and outputs:. Past trajectory: EgoNav keeps track of where the person (or robot) has been over the last few seconds, typically as a sequence of horizontal positions. This “waypoint history” provides momentum and direction information. For instance, if the person was walking north, the model knows to continue in that direction unless something causes a turn. Visual context: The 360° memory embedding (64D) and latest DINOv3 features both serve as conditions. The memory provides geometric and semantic layout, while the DINOv3 features come from the raw RGB images and inject appearance cues. DINOv3 is a state-of-the-art self-supervised vision transformer that can extract high-level features from images without requiring task-specific labels. EgoNav keys DINOv3 to recognize things invisible to depth: the shimmering reflection of a glass wall or the body of a moving pedestrian might be captured. Thus, the diffusion model sees a combination of static semantic layout (from memory) and dynamic visual cues (from DINO). Trajectory output: The model predicts a full sequence of future waypoints. Specifically, it is trained to produce 100 future points, corresponding to 5 seconds of motion at 20 Hz. These are planar (2D) positions of the agent’s center at future timestamps. Because walking is somewhat constrained, 5 seconds is a good balance: it lets the robot plan through intersections or around obstacles without being too far ahead to be irrelevant. Together, the past waypoints and these two vision-based embeddings form the conditioning input c for the diffusion model. Concretely, EgoNav’s model is a U-Net with about 46 million parameters that implements a denoising diffusion process. Training the Diffusion Model. Training uses the collected human walking dataset. Each training example consists of:. A segment of a human trajectory in the dataset (past 1s of motion plus next 5s of motion). The corresponding visual context: the memory embedding and DINO features at the end of the past segment. EgoNav treats each trajectory as a “sequence image” of 100 points. It then uses a diffusion objective to learn to add noise and then remove it. More precisely, the model learns the standard DDPM denoising objective on trajectories: at each diffusion timestep, Gaussian noise is added to the true future trajectory, and the network must predict the noise (or an updated trajectory) to denoise it back toward the original path. This is analogous to how image diffusion models work on pictures, except here the “image” is a planar path. All conditioning inputs (scene and past motion) are provided as context to the network (using techniques like classifier-free guidance). The diffusion model thus learns the distribution of trajectories that people took in the past given similar scenes and motion histories. Crucially, because diffusion is inherently probabilistic, when it comes time to sample from this model we can generate many different plausible paths. This multimodal capability means EgoNav can represent options like “go straight” vs “turn left” if both were common in the data. By contrast, a deterministic model would only pick a single path. During training, EgoNav uses the entire human dataset. Importantly, note that none of this involves a robot. The model never sees how the robot walks, only how the humans did. This makes EgoNav a pure case of learning from video/demonstration without any robot-specific supervision. It is akin to learning a prior over sensible motions just by watching life, and then imposing that prior onto a robot. Generating Future Paths via Diffusion. Once trained, the diffusion model can be sampled to produce full trajectories. EgoNav implements a clever hybrid sampling scheme to make this fast enough for real-time use. A naïve DDPM can require dozens or hundreds of denoising steps to produce high-quality outputs, which would be too slow. Instead, EgoNav uses a 5+5 step hybrid of DDIM (deterministic) and DDPM (non-deterministic) steps. In practice, the system runs 5 DDIM steps followed by 5 DDPM steps (for a total of 10 denoising steps). The authors report that this gives a 100× speedup over a full 50-100 step DDPM, while basically preserving the multi-modal quality of the output. On the onboard NVIDIA Jetson Thor computer (a powerful embedded GPU), this scheme can generate about 110 trajectories per second, which translates to around 1.7 Hz planning frequency. In other words, every ~0.6 seconds the system can refresh a new set of candidate paths in the current scene. Because diffusion sampling is stochastic, each run yields a slightly different set of trajectories. EgoNav takes advantage of this by generating many candidate trajectories in parallel (up to 110 per second) and then evaluating them with filters. In offline evaluation, they looked at metrics like Best-of-N (minADE), measuring how well the sampled modes cover actual human paths. In deployment, the robot’s controller inspects each candidate for collisions and smoothness, then picks one. We will say more about that in the next section. The result of this diffusion process is basically a probabilistic funnel of future movement. If the environment is clear ahead, most samples will head straight. If there’s a corner, some will turn; if it’s open, some may take weird side steps. Because it is trained on varied campus data with doors, stairs, benches, other walkers, etc., the diffusion model learns which patterns are likely safe. Crucially, the model uses classifier-free guidance to tilt the distribution toward the observed conditions. In diffusion-speak, this means sometimes the conditioning (memory and past) is dropped at random during training, so that the network can smoothly interpolate between “unconditioned” (free) and “fully conditioned” generation at inference by linear interpolation. This trick lets EgoNav bias the sampling toward obeying the scene context. For example, if during sampling a potential trajectory strays into a wall region of the memory map, the guidance nudges it away. This is a key part of modern diffusion planners. To summarize, egoNav’s diffusion model provides a rich, high-dimensional landscape of possible future trajectories given the visual memory and recent motion. The next step is to actually pick a safe path and execute it on the robot. Receding-Horizon Control and Collision Avoidance. The diffusion model alone produces many candidate paths, but EgoNav still needs to choose one to follow. The system uses a receding-horizon controller. In each planning cycle (every ~0.6s), EgoNav:. Samples a set of candidate trajectories from the diffusion model (using the 5+5 step scheme). Each candidate is a sequence of future (x,y) waypoints. For each candidate, it checks for collisions or feasibility. This involves simulating a simple footprint of the robot moving along the first few steps of the trajectory through the semantic/depth memory. Any trajectory that would hit an obstacle (wall, person, etc.) is rejected. Among the collision-free candidates, it scores them (e.g. by smoothness or closeness to a goal). The current prototype just uses a simple smoothness criterion, but more complex scoring (like distance to goal or reward) could be plugged in. It selects the “best” trajectory, or if there are none safe, it may halt. Actually, in practice, the reported autonomy metric (96% successful) implies safe execution almost always. It then executes only the first short segment of the selected trajectory (e.g. the next half-second or so). Because new sensory information will arrive, EgoNav will re-plan repeating this pipeline in a loop. This is the usual receding-horizon model predictive control scheme. This design means EgoNav is model-predictive: it constantly replans as it collects new observations. It also means it is robust to the inevitable drift between human frames and robot motion – by re-evaluating frequently, small discrepancies are corrected. The controller is relatively light – it’s not doing full physics or trajectory optimization. It just trusts the diffusion prior to give plausible motions. But by adding the collision check filter, EgoNav ensures safety even if the diffusion model occasionally suggests a path into something. In the experiments, having this loop allowed the system to avoid dead-ends or moving targets like people. For example, if a path sample intersects a currently visible person, that sample is pruned. Because the model was trained only on human data, one might worry about domain shift. Yet because the diffusion model only cares about high-level trajectories, the robot can basically follow them with its onboard locomotion stack. EgoNav decouples high-level planning from low-level control. For these tests, the Unitree G1’s built-in gait generator was used to track the selected waypoints, along with a basic velocity controller. The authors don’t elaborate on it, but presumably the robot runs in a locomotion mode where it holds direction and just moves forward/back by small amounts as needed. Two details are worth highlighting: latency compensation and multi-modality. Because generating and filtering 110 trajectories takes some time (~0.6 seconds), the controller knows that by the time a path is chosen, the real robot will have moved a bit. The system compensates by effectively predicting a little further ahead (accounting for its own compute time in the planning horizon). Also, by sampling many trajectories and only following the first portion, EgoNav ensures it still acts smoothly. If it sampled a single long path and then executed it all, it could make mistakes; as it stands, each short segment is validated fresh by the diffusion prior + collision-check at the next cycle. This approach earned the descriptor “real-time with latency compensation” in the paper. Implementation Details and Compute. Putting all this together yields a robot navigation pipeline with some heavy parts (deep networks) and some simpler parts (point clouds, collision checks). The Stanford team implemented the system on a Jetson Orin Xavier NX (for encoding the memory via the VAE and running the DINOv3 model) and a Jetson Thor (for the diffusion U-Net sampling). These are embedded AI computers suitable for onboard robot use. Key timing numbers:. Visual memory: updating the 360° point cloud and encoding to a 64D vector takes about 30 ms per frame on the Orin NX. Given a 20 Hz camera, this comfortably fits in each cycle. Diffusion sampling: each planning iteration generates up to 110 trajectories. The 46M U-Net model running 10 denoising steps took about 0.6 s per cycle, i.e. ≈1.7 Hz throughput. This is slower than the camera rate, but fine because the robot moves slowly and the receding-horizon checks only a short horizon each time. Trajectory filtering: checking collisions for 110 paths is very cheap (point lookups or simple simulation) relative to the diffusion sampling, so it does not add significant delay. So overall EgoNav runs in real time on modest hardware, a testament to the hybrid sampling design. Running 10 steps per sample is far faster than typical diffusion samplers. The paper notes this hybrid approach yields “100× acceleration over full DDPM” while preserving quality. They actually did an experiment varying the composition of deterministic (DDIM) vs stochastic (DDPM) steps and found that 5+5 strikes a good balance (see Fig. above). Another detail: to train the diffusion network, the authors likely used more powerful GPUs offboard. We don’t have specifics, but it’s clear they trained on only 5 hours of data, so training time was probably reasonable (compared to RL or capture). They also leveraged a frozen DINOv3 backbone: this means they did not fine-tune any vision networks on navigation data. The DINOv3 features were computed once and then used as fixed input for training the trajectory network. This reduces training complexity and also prevents catastrophic forgetting of the strong perceptual features. The segmentation into 8 classes presumably used a pretrained semantic model (e.g. a ResNet or a transformer segmentation network). The classes likely include things like {ground, road, building, vegetation, people, vehicle, sky, other} or similar. Having semantic labels gives Egonav knowledge of “door vs wall” vs “ground” etc. One ablation they mention is that without semantics, doors and walls look the same and navigation fails (more collisions). We’ll revisit that in results. With this setup, EgoNav offers a relatively lightweight alternative to map-building or occupancy grids. Instead of building a metric map, it builds only the 360° view needed for planning at each step, then discards it as the scene changes. It doesn’t need loop closure or global consistency; it constantly forgets the older past beyond what’s encoded by the visual memory embedding. Experimental Results. Let’s survey what EgoNav can actually do. The authors evaluate both in offline metrics and in fully live robot trials. Offline evaluation: Using held-out segments of the human-walking data, they measure how well EgoNav’s diffusion model predicts the actual future paths. Because diffusion is probabilistic, the right metric is something like “best-of-N” minimum error. They define three metrics: Collision-Free Score (CFS, how often a trajectory avoids obstacles in hindsight), Smoothness (low jerk), and Best-of-N minADE (similar to minADE-K in trajectory prediction – how close the best sample is to the ground truth path). The diffusion prior is compared to some baselines (not detailed in the summary, but likely an RNN predictor or similar). EgoNav significantly outperforms baselines on collision avoidance and covers more modes of the actual trajectory distribution. In other words, by training generatively, it tends to produce at least one path that was very close to what the human actually did, and it almost never wanders into obstacles. They also ablate components. In one study they remove certain inputs and see how performance degrades (lower CFS or higher error). Key findings:. Semantic channel: Removing semantic labels causes a 5.1% drop in collision performance. This is logical: without knowing which surfaces are doors or ground, the model gets more confused about walkable areas. A quoted note says “can't distinguish doors from walls” when semantics are stripped, hence it collides more. Attention (visual memory): Turning off the panoramic memory (i.e. using only a single camera view) also hurts performance by 2.6% (in CFS presumably). This confirms the memory’s value in giving context. Without it, the model is more "short-sighted" and more collisions occur. DINOv3 features: Interestingly, removing the DINO-based appearance features has only a modest impact on the offline metrics. This suggests that on the curated human data, depth+semantics was already capturing much of the signal. However, in real-world tests (on the robot), DINO features turn out to be critical. The write-up notes: “DINOv3 features prove critical in real-world glass/dynamic scenes despite modest offline gains”. This makes sense: if the training data happens to have few or no transparent/glass walls, then semantics and depth might have sufficed offline. But once you deploy on real hardware, those unseen cases pop up. The DINO model, being trained on a broad internet corpus, recognizes glass reflections or adult bodies as salient. Real-world deployment: The crowning proof of EgoNav is running it on an actual robot with no additional training. The authors did a continuous run lasting 37.5 minutes, totaling 1,137 meters traversed. Environments included indoor corridors, glass-paneled hallways, and outdoor paths with pedestrians. The robot achieved 96% autonomy (meaning almost all of its motion came from EgoNav’s planning rather than emergency stops). The behaviors seen include exactly the desired ones: it stops at closed doors (i.e. it waits rather than blindly walking into something), it navigates around walking people, and it avoids glass walls (which are obstacles to humans but transparent in depth maps). These “emerged naturally from the learned prior”, according to the report. Figure snapshots (from the project page) show EgoNav in action. In one image, the robot approaches a closed door: its diffusion samples fan out just short of the door until it opens, then some samples continue. In another, pedestrians are walking past the robot, and EgoNav’s selected path detours smoothly. No specialized door-detection or behavior was coded – the diffusion model simply had learned from videos that people don’t walk through closed doors or crowds, so it prefers circumnavigating them or waiting. The fact that this works zero-shot is remarkable and a strong testament to the generality of the learned prior. Comparatively, had the robot used a standard planner with its onboard depth camera only, it might have blindly walked into the glass (since lidar/depth sees nothing) or bumped into people (since they move). EgoNav, by contrast, knows these are “no-go” because similar silhouettes appear in the training data. Autonomy and safety: The 96% autonomy figure implies that only ~4% of the distance required human intervention or the robot stop. In appendix videos the authors note a few cases where EgoNav was overly conservative or got stuck, but most of the time it worked. There is no guarantee of absolute safety (anything learned from data can err), but the evidence shows EgoNav handled typical hazards well. The offline collisions metric (CFS) gave confidence that most predicted trajectories were safe, and the receding horizon allowed failing safe if something unexpected happened. One more note: EgoNav’s output trajectories are fairly short-term (5 seconds ahead). This means it does not plan a full path to a far-away goal. Rather, it achieves local navigation: walking down hallways, turning corners, etc. For longer tasks (like going to a particular room), one would combine EgoNav with some high-level planner or waypoint system. In that sense it is most analogous to an advanced local planner with learned intuition. Integration and Usage. For roboticists and ML practitioners, EgoNav offers an intriguing new tool. Some practical takeaways:. Data collection is relatively cheap: Five hours of video is nothing compared to months of robot-driving data. This hints that many navigation tasks might be approached by simply collecting relevant human-worn camera footage. In principle, one could even mine online first-person videos for training, though alignment and calibration would be needed. Preprocessing pipeline is important: The 360° memory is a big engineering win. It shows that moderate robotics SLAM-like processing (build a rough 3D cloud and reproject) can yield a compact coordinate-free representation that diffusion models can digest. It’s a clever middle ground between raw images and a full metric map. Foundational vision models can help: EgoNav uses a frozen DINOv3 network (on RGB images) to capture aspects of the scene that geometry misses. This is an example of a “foundation model” in robotics: a visual model trained on vastly different data (web images) can enhance perception (detecting glass, people, patterns) without retraining. We expect to see more methods doing this reuse of vision transformers or self-supervised nets. Inference efficiency via hybrid diffusion: For real-time, EgoNav uses only 10 denoising steps. More steps would improve fidelity, but 10 was enough to capture routes. This hybrid scheme (DDIM+DDPM) is a practical contribution. Teams building diffusion planners should note that a partial deterministic sampling followed by only a few stochastic steps is a sweet spot. Receding horizon makes it robust: Because EgoNav never commits to a single long plan, it can handle dynamic changes. This pattern – sample many futures, pick the best short bit, then replan – is a general recipe for diffusion planners in mobile robotics. How might other researchers use or extend EgoNav? One could imagine swapping in a different vision frontend (e.g. use an onboard lidar scan instead of depth, or a different segmentation net) transparently, as long as the memory snapshot is fed in. If someone had a different legged robot, they could similarly use EgoNav zero-shot, though gait differences might require small tuning. Because the core is just waypoints, any low-level controller could follow them (e.g. a differential drive robot might not follow exactly the same trajectories, but could try). Extensions could involve adding more conditions. Right now EgoNav predicts where the human would walk, but it has no sense of specific goals like “go to the exit” or “follow the person”. One could combine the diffusion prior with a planner that sets a rough goal direction. For instance, steer the diffusion sampling via a reward on a global map to bias certain modes (this is similar to classifier gradient techniques in diffusion). That would turn the short-horizon distribution into a purposeful long-horizon plan. Also, EgoNav currently avoids people and glass implicitly by learning they block paths. One might contrast this with methods that explicitly segment humans and force distance constraints. Each has pros and cons: explicit tracking is robust but brittle (occlusions, etc.), while EgoNav’s learned approach is graceful but might be surprised by very unusual scenes. For now, EgoNav seems to have learned the major obstacles well. Finally, note that EgoNav is specifically a navigation system. It doesn’t handle obstacles that require complex locomotion (like climbing stairs or squeezing through narrow gaps) – for those tasks one needs whole-body control. However, it could be combined with such controllers: for example, if the diffusion path leads up stairs, a separate stair-climbing gait on the G1 would activate. In other words, treat the diffusion planner as the “what to do”, and let specialized controllers handle “how to walk” when needed. Comparison to Related Work. EgoNav sits at the intersection of several trends in robotics and ML:. Learning from demonstration (LfD): Classic LfD uses human demonstrations to teach tasks. EgoNav’s twist is that the demonstrations come as raw egocentric video, not annotated trajectories. This is similar in spirit to projects like VideoMimic (which reconstructed humans from video to train whole-body skills). But EgoNav only cares about foot trajectories, not full pose. It bypasses the need for motion capture or explicit retargeting. Vision-based navigation: Many systems use CNNs or RL to navigate in sim; some use Sentinel or RPlab stuff. But most of those need robot-centric data or simulators. EgoNav instead narrows the gap by using real scenes and real humans as teachers. This is closer to “learning from YouTube” or “internet-scale video for navigation” – although EgoNav’s dataset is relatively small, it demonstrates the promise of such an approach. Diffusion models in planning: There is a growing body of work applying diffusion to robotics. For example, the Diffuser/Diffusion Planning papers (Janner et al. 2022 and successors) showed how to plan with diffusion on high-dimensional tasks by conditioning on goals. SceneDiffuser (Huang et al., 2023) conditioned on point clouds. EgoNav follows this general pattern: treat planning as conditional generation. Its specific contributions are (1) applying diffusion to human-to-robot transfer, (2) using egocentric vision and a spatial memory for conditioning, and (3) a careful sampling scheme for real-time speed. Foundation vision models: DINOv3 is an example of a self-supervised ViT that was not trained for navigation, yet EgoNav borrows its power. This echoes trends in robotics where CLIP or other pre-trained nets are used for perception. The lesson is: even if you train on only 5 hours of data, you can stand on the shoulders of giant vision efforts to disambiguate tricky obstacles. In fact, chart [32] shows DINO features didn’t help much in offline (human data) metrics, but became crucial in the wild – underscoring that foundation models build knowledge humans might take for granted (like “glass is present”). Simulation vs Reality: EgoNav completely skips simulation. Many navigation approaches use sim2real or structured maps. EgoNav does neither. This could inspire simpler pipelines: just record people, learn models, deploy. It’s close to end-to-end in spirit, though the perception encoding and controller are modular. Some users might worry this lacks formal verification; indeed, it’s more heuristic, but the experiments show it works surprisingly well in practice. Conclusion and Outlook. EgoNav demonstrates a surprising fact: with the right model, a small quantity of human video can teach a robot to navigate unseen environments. The diffusion-based prior encapsulates human common sense about walking: don’t bump into people, wait if necessary, skirt around glass. And all of this emerges with minimal engineering. For roboticists, EgoNav suggests an exciting path forward. Rather than hand-designing everything from scratch, we can harness the wealth of naturalistic human data. This is in line with other embodied AI trends (e.g. using movie data to teach motion, or crowdsourced cameras to map environments). Of course, there are limits – EgoNav doesn’t know physics of forward falls, or how to recover from trips, but for pure path planning it sets a new bar. Future work could extend EgoNav by incorporating language (e.g. high-level commands combined with this visual prior), or by blending it with other sensors. One could imagine EgoNav-360 planners on vehicles (watching dashcam footage) or in aerial robots (following human hikers). The core idea – train a multi-modal distribution of actions from first-person human data – is broadly applicable. In summary, EgoNav is a compelling proof-of-concept for egocentric video-driven navigation. By marrying diffusion models with smart representation of past vision, it achieves zero-shot, real-time navigation on a real humanoid. It outperforms simpler planners on collision avoidance and gives rise to natural pedestrian-like behavior. The work opens a door to more research on how much robots can learn from just observing humans move. References: The description above is based on Weizhuo “Ken” Wang et al., Learning Humanoid Navigation from Human Data (2026) and the associated project page. Background on diffusion-based planning comes from recent surveys and tutorials, and related video-driven humanoid learning is discussed by recent works such as VideoMimic [Anon].