Latest / Embodied AI 101 / ReaDy-Go: Teaching Robots to Dodge Photorealistic People in Gaussian-Splat Digital Twins
Transcript
- ReaDy-Go: Teaching Robots to Dodge Photorealistic People in Gaussian-Splat Digital Twins. ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles • Seungyeon Yoo, Youngseok Jang, Dabin Kim, Youngsoo Han, Seungwoo Jung, and H. Jin Kim • Seoul National University and the KAIST InnoCORE AI-Transformed Aerospace Research Center • IEEE Robotics and Automation Letters • 2026. The central idea. Most real-to-sim navigation pipelines reconstruct an empty hallway, lobby, or factory floor and then freeze it. The result may look remarkably similar to the deployment site, but it does not capture the part of navigation that is often most safety-critical: people moving through the robot’s path. ReaDy-Go is an attempt to make these reconstructed environments move. Starting from a monocular video of a target site, the system builds a static 3D Gaussian Splatting scene, inserts animatable human avatars that are themselves represented by Gaussian splats, generates interaction-rich human trajectories, and uses a dynamic expert planner to produce robot demonstrations. A small RGB navigation policy is then trained by imitation learning and deployed on a differential-drive robot without requiring the Gaussian map or expert planner at runtime. The paper’s most important contribution is therefore not a new foundation policy or a fundamentally new dynamic-scene representation. It is a data-generation system: a way to convert a short recording of a particular deployment site into tens of thousands of photorealistic training samples in which realistic-looking people cross, parallel, and obstruct a robot’s path. The strongest experimental argument is a controlled comparison against Vid2Sim. ReaDy-Go and Vid2Sim use the same navigation policy, human trajectories, and expert planner; the key change is whether a moving person is rendered as a photorealistic human Gaussian avatar or as a conventional simulator asset. ReaDy-Go’s substantial advantage in dynamic tasks suggests that the visual realism of moving agents is not merely cosmetic. It changes what the deployed policy learns to recognize and avoid. Why static real-to-sim is not enough. RGB-only navigation remains attractive because a monocular camera is inexpensive, light, and already useful for many other perception tasks. But monocular visuomotor learning is also vulnerable to depth ambiguity, visual distribution shift, and the high cost of collecting collision-rich real-world demonstrations. Simulation solves the data-volume problem while creating a visual transfer problem of its own. One response has been to train general navigation models such as GNM, ViNT, and NoMaD on broad real-world datasets. These systems aim to transfer across robots and environments, usually with goal images or topological context. ReaDy-Go explores the complementary hypothesis: if a robot will repeatedly work in one household, restaurant, office, or factory, it may be more effective to specialize a compact policy using a reconstruction of that exact site. Gaussian splatting is particularly appealing for this kind of specialization. A 3DGS reconstruction stores a scene as explicit anisotropic Gaussian primitives with learned positions, orientations, scales, opacities, and colors. These primitives can be projected and alpha-composited efficiently, giving both high-fidelity novel-view rendering and enough geometric structure to support downstream planning. Earlier GS-based robotics systems demonstrated static-map planning or policy training in reconstructed deployment environments. Splat-Nav and GaussNav plan directly through Gaussian maps, while systems such as SOUS VIDE and EmbodiedSplat use reconstructed environments for navigation-policy adaptation. EmbodiedSplat, for example, reconstructs mobile-device captures, extracts simulator-compatible geometry, and fine-tunes ImageNav policies for the target environment. These methods establish the value of environment personalization, but largely treat the reconstructed site as static. Vid2Sim moves further toward interactive simulation. It converts monocular videos into photorealistic, physically interactable urban environments by combining neural scene reconstruction with mesh extraction and a conventional simulation engine. ReaDy-Go’s distinction is that both the background and the visible human obstacle are rendered through Gaussian representations, avoiding a mixed appearance pipeline in which the environment is a neural reconstruction but the pedestrian is a recognizable game-engine asset. This is also a useful terminology caveat. “Dynamic 3D Gaussian Splatting” can refer to jointly reconstructing a time-varying 4D scene from dynamic observations. ReaDy-Go instead builds a compositional dynamic simulator: a static Gaussian background plus separately reconstructed, independently animated Gaussian humans. That is a narrower formulation, but one that is unusually practical for generating controlled navigation scenarios. The complete pipeline. The pipeline has three broad stages. First, ReaDy-Go reconstructs the deployment site and prepares it for both rendering and planning. Second, it adds Gaussian human avatars whose full-body motion is generated from planned 2D paths. Third, it runs robot and human planners together, rendering the robot’s observations and recording expert commands for imitation learning. A major architectural decision is that the expensive 3D machinery is only needed during dataset generation. The deployed policy receives three RGB frames, a local relative-goal vector, and its previous action. It does not carry the Gaussian map, human body model, motion diffusion model, or Hybrid A* planner onboard. That division is central to the work: privileged geometry and simulator state supervise a lightweight policy that must later react from pixels. Reconstructing the target site. ReaDy-Go begins with images from a monocular video of a nominally static environment. COLMAP estimates camera poses, while an ArUco marker visible in the initial frames supplies metric scale and world-frame alignment. This small calibration step matters because monocular structure from motion otherwise recovers geometry only up to an unknown scale—unacceptable when robot radii, human dimensions, velocities, and safety margins must be expressed in meters. The authors use PGSR, or Planar-based Gaussian Splatting, rather than vanilla 3DGS. PGSR encourages Gaussian primitives to behave more like locally planar surface elements and adds geometric regularization alongside the photometric objective. ReaDy-Go needs this because a reconstruction that looks good from camera viewpoints may still contain geometrically damaging floaters and poorly localized primitives. Those artifacts may be nearly invisible in rendered RGB but can falsely close a corridor once the map is voxelized. For planning, the static Gaussians are converted into an occupancy representation. A voxel is initially considered occupied if it intersects part of a one-standard-deviation Gaussian ellipsoid. The system then examines the accumulated opacity inside each voxel. If the summed opacity does not cross a threshold, the voxel is treated as free space. This filters weak, semitransparent Gaussian noise, particularly near floors and in weakly textured regions. The resulting volume is projected into separate 2D maps. The robot map accumulates occupancy from near the floor up to robot height; the human map uses a taller vertical range. Occupied regions are then inflated by the corresponding robot or human radius. This is an understated but important part of the paper. ReaDy-Go does not assume that a radiance representation is automatically a valid collision map. It adds an explicit conversion layer between the representation optimized for image synthesis and the representation required for safe planning. In the experiments, each of the three sites—Outside, Lobby, and Library—was reconstructed from approximately six minutes of video containing about 1,000 to 1,500 images. Making the people Gaussian—and animatable. The human component starts with HUGS, or Human Gaussian Splats. HUGS disentangles a person and a background from monocular video, initializes the person around an SMPL body model, and learns Gaussian appearance parameters together with linear-blend-skinning weights. The resulting avatar can be rendered from new viewpoints and deformed into novel body poses. ReaDy-Go extracts six such avatars from sequences in the NeuMan dataset. These are not people reconstructed from the target-site capture. They are reusable Gaussian assets that can be placed in a newly reconstructed lobby, library, or outdoor environment, then transformed and animated independently. A geometric path alone is not enough to animate a human convincingly. Simply translating an upright avatar along a spline would produce sliding feet, fixed limbs, implausible turns, and weak visual motion cues. ReaDy-Go therefore adopts PriorMDM, which uses a pretrained human-motion diffusion model as a generative prior for fine-grained trajectory control. Given a planned 2D human path, the system converts it into body-root linear and rotational velocities and normalizes those controls into the representation expected by the HumanML3D-trained model. PriorMDM predicts a three-dimensional sequence of body joints. SMPLify then fits SMPL parameters to those joints, after which the sequence is transformed into the coordinate frame of the target environment and used to deform the HUGS avatar. The result is a useful chain of abstractions:. The human planner decides where the person should go. PriorMDM decides how the body should move along that route. SMPL provides the articulated skeleton and body surface. HUGS turns that articulated motion into a photorealistic Gaussian avatar. The Gaussian renderer composites the animated person with the reconstructed site. The system does not simulate forces, contact, balance, or reciprocal social behavior. The motion is generated kinematically from a prescribed path. But for visual navigation training, it provides coherent locomotion, turning, limb movement, appearance, and viewpoint changes—exactly the cues that a temporal RGB policy can use to distinguish a person from static clutter. This all-Gaussian composition is also the basis of the paper’s photorealism claim. Rather than rendering one representation for the scene and another for the human, ReaDy-Go combines the primitives before rendering observations. Qualitatively, the paper and project demonstrations show consistent novel views of animated avatars across the three target environments. The paper does not, however, report a quantitative rendering benchmark such as PSNR, perceptual similarity, or a human realism study. Turning a renderer into a navigation-data compiler. Photorealistic humans are useful only if the generated episodes contain meaningful interactions. ReaDy-Go therefore includes separate planners for the human and robot. The human planner samples start and goal positions so that the pedestrian’s route either crosses or runs approximately parallel to the line connecting the robot’s start and goal. This deliberately creates episodes in which robot and human trajectories are likely to interact. A graph search such as A* first produces a collision-free seed path on the human walkable map, after which spline optimization smooths the trajectory. This is targeted scenario generation rather than unconstrained pedestrian simulation. It biases the dataset toward crossing, overtaking, and detouring cases that teach more about dynamic avoidance than randomly walking a person through an irrelevant corner of the scene. The robot expert is based on Hybrid A*, augmented with a motion-primitive library suitable for a nonholonomic ground vehicle. Each expansion considers nine basic commands formed by combining three velocity magnitudes—one third, two thirds, and full speed—with left, straight, and right steering choices. Candidate transitions are scored using path length, a steering penalty, and a heuristic estimate of remaining cost to the goal. Dynamic collision handling begins from the human Gaussian representation. When a human is inside the camera field of view and sufficiently close to the robot’s planned trajectory, the simulator extracts the avatar’s occupied region. High-uncertainty human Gaussians, defined using covariance trace, are removed above the 75th percentile. The remaining Gaussian centers are voxel-downsampled and projected into the robot’s 2D navigation map. The projected human footprint is then expanded using a constant-velocity prediction over a two-second horizon. Hybrid A* plans around this predicted region, but the system does not trust the initial prediction for the entire episode. It replans periodically and can trigger an immediate replan if a dynamic obstacle approaches the look-ahead portion of the current path. There is an interesting compromise here. The expert uses exact simulator geometry rather than estimating people directly from rendered images, so it is privileged. But it only updates the dynamic map when the human lies inside the camera’s field of view and threatens the planned route. That encourages demonstrations whose reactions are at least broadly compatible with what the future RGB policy could have observed. During each successful rollout, ReaDy-Go records the rendered RGB image, the expert’s linear and angular velocities, and the goal position expressed in the robot’s local frame. Across sufficient trials, sampled start-goal pairs cover the target environment while the curated human routes create static avoidance, crossing, and detour examples. For each environment, the paper uses 400 training episodes, corresponding to roughly 80,000–120,000 samples, plus 50 validation episodes for checkpoint selection. The released implementation notes that only successful scenarios are retained and recommends generating more candidate episodes than ultimately required; it estimates that full dataset generation can take one to two days, depending on parallelism and hardware. The policy is intentionally small. The navigation policy is deliberately conventional. This isolates the effect of the simulator rather than conflating it with a large transformer, recurrent world model, or diffusion policy. A shared residual convolutional encoder processes three RGB images sampled at half-second intervals. Each frame produces a 20-dimensional latent vector. The three vectors are concatenated with the previous linear and angular velocity and a two-dimensional relative-goal position, producing a 64-dimensional feature vector. A shallow three-layer MLP predicts the next linear and angular velocity. The policy is trained by behavioral cloning with mean-squared error against expert actions. Images are resized to 144 by 256 pixels, and Adam uses a learning rate of (10^{-4}). The released code defaults to 30 epochs, batch size 64, and randomly applied color jitter and Gaussian blur. The policy has about two million parameters. At deployment, inference takes approximately 18 milliseconds on a Jetson Orin NX, equivalent to 55 hertz, while the physical camera operates at 20 hertz. The robot still receives a relative goal derived from wheel odometry. Thus, “RGB-only” describes obstacle perception, not the complete absence of proprioception or geometric goal information. Experimental protocol. The task is PointGoal navigation. In static trials, the robot must avoid the fixed environment and reach a sampled goal. Dynamic trials add one or two moving human obstacles. A run succeeds if the robot reaches within one meter of the goal without collision and before the 50-second limit. The authors evaluate 100 simulation episodes for every environment-task pair and ten real-world episodes for each pair. Start-goal configurations and dynamic-obstacle trajectories are matched across methods. The physical platform is a differential-drive robot with a fixed forward-facing ZED2 camera, Jetson Orin NX, and wheel odometry for relative-goal estimation. Real tests use six humans not seen among the Gaussian training avatars. Performance is measured with success rate and average reaching time. Failed trials are assigned the full 50 seconds when computing reaching time. Consequently, ART is not just a speed metric: it combines path efficiency with failure frequency. A method can lower ART by moving faster, by taking shorter detours, or simply by failing less often. The general navigation baselines receive goal images captured within ten meters of the start, oriented along the start-to-goal direction. ReaDy-Go instead receives a relative-goal vector. This is a reasonable attempt to use each model in its intended interface, but it is not a perfectly modality-matched comparison. The results therefore support environment specialization under the paper’s deployment protocol; they should not be read as a universal ranking of PointGoal and image-goal navigation systems. Simulation results: realism matters primarily when things move. The following values average the three reported environments; the paper itself reports the individual environment results. | Method | Parameters | Simulation static: SR / ART | Simulation dynamic: SR / ART | Real static: SR / ART | Real dynamic: SR / ART | |---|---:|---:|---:|---:|---:| | GNM | 9M | 58.0% / 24.61 s | 50.3% / 28.01 s | — | — | | NoMaD | 19M | 42.3% / 31.22 s | 35.0% / 34.35 s | — | — | | ViNT | 30M | 61.7% / 23.29 s | 51.0% / 28.01 s | 63.3% / 27.32 s | 30.0% / 38.84 s | | Vid2Sim | 2M | 92.7% / 11.54 s | 65.0% / 22.62 s | 83.3% / 19.50 s | 53.3% / 31.34 s | | ReaDy-Go | 2M | 91.3% / 11.15 s | 78.7% / 16.75 s | 96.7% / 15.83 s | 80.0% / 22.25 s |. These averages are computed from the paper’s reported per-environment tables. The static simulation results are revealing because ReaDy-Go does not clearly beat Vid2Sim on success rate. Vid2Sim averages 92.7%, while ReaDy-Go averages 91.3%. Their average reaching times are also close. That is what we should expect: the two systems share the same architecture and static scene, and their meaningful difference is the representation of moving people. The dynamic results separate them. ReaDy-Go obtains success rates of 78%, 78%, and 80% in Outside, Lobby, and Library. Vid2Sim reaches 57%, 70%, and 68%. Averaged across the three sites, replacing the simulator human with a Gaussian human improves success by 13.7 percentage points and reduces ART by about 5.9 seconds. Another way to see the result is to compare degradation from static to dynamic tasks. Vid2Sim falls by about 27.7 percentage points on average, while ReaDy-Go falls by about 12.7 points. The photorealistic-human pipeline roughly halves the performance loss associated with introducing moving obstacles. Both environment-specific systems also substantially outperform GNM, ViNT, and NoMaD within this evaluation. The result supports the practical value of target-site training: a two-million-parameter policy can exploit the visual layout and obstacle statistics of one known environment more effectively than a much larger general model deployed zero-shot. It does not show that specialization should replace general navigation; it shows that specialization remains a powerful final adaptation stage. The expert itself reaches only 91%, 83%, and 78% success in the three dynamic simulation settings. This is an important reality check. Constant-velocity prediction and reactive replanning do not solve every interaction, and the imitation target is not an infallible oracle. In Library, the learned ReaDy-Go policy even slightly exceeds the reported expert success rate, illustrating that the planner row is not a strict empirical ceiling for the closed-loop policy. Real-world transfer. The real-world results preserve and amplify the dynamic advantage. ReaDy-Go succeeds in 29 of 30 static trials and 24 of 30 dynamic trials across the three environments. Vid2Sim succeeds in 25 static and 16 dynamic trials, while ViNT succeeds in 19 and nine, respectively. Because there are only ten trials per environment-task pair, every individual table step represents ten percentage points, so these numbers should be interpreted as strong initial evidence rather than high-precision estimates. In the dynamic task, ReaDy-Go achieves 90% success Outside, 70% in the Lobby, and 80% in the Library. Vid2Sim reaches 60%, 40%, and 60%. The corresponding ReaDy-Go reaching times are 20.68, 26.49, and 19.57 seconds, versus 28.98, 36.50, and 28.54 seconds for Vid2Sim. The physical environments had minor changes in floor appearance, lighting, and background details between capture and deployment, and the test participants were not the six people used to create the Gaussian avatars. Despite these shifts, ReaDy-Go’s real-world aggregate success remained close to its simulation performance. The failure analysis points in the same direction. ViNT suffers more static collisions, indicating difficulty exploiting the exact geometry of the target sites. ReaDy-Go and Vid2Sim have more similar numbers of failures unrelated to moving people, but ReaDy-Go produces fewer dynamic-obstacle collisions and fewer static collisions caused while detouring around a person. For zero-shot generalization, the authors combine all 1,200 training episodes from Outside, Lobby, and Library and deploy the resulting policy in a fourth, unseen real environment. It achieves 70% success in static trials and exactly 50% in dynamic trials, with ARTs of 30.60 and 35.45 seconds. This is weaker than performance in the reconstructed sites but still demonstrates that the policy has learned some transferable notions of goal seeking and obstacle detouring. There is no unseen-environment baseline in this experiment, so it establishes potential rather than comparative state of the art. What ReaDy-Go contributes to embodied AI. The first significant idea is that dynamic-agent appearance is part of the sim-to-real problem. Simulation work often treats the background as the main visual domain and moving obstacles as abstract geometry. ReaDy-Go suggests that this division is wrong for end-to-end visual policies. A robot must learn not merely that some occupancy blob moves, but what a real person looks like while changing scale, articulating limbs, turning, becoming partially occluded, and crossing the camera’s field of view. Second, ReaDy-Go treats a Gaussian reconstruction as a scenario compiler, not simply a renderer or map. The system extracts planning geometry, composes reusable object-level assets, generates motions, renders observations, and emits action labels. This is a more consequential use of 3DGS for robotics than novel-view synthesis alone. Third, the paper shows an effective separation between training-time complexity and deployment-time simplicity. During generation, the system has Gaussian geometry, body models, motion diffusion, exact avatar trajectories, occupancy maps, and a search-based expert. At deployment, all of that has been distilled into a compact feed-forward policy. This is an attractive pattern for resource-constrained platforms operating in fixed facilities. Fourth, ReaDy-Go provides a useful bridge between site-specific navigation and general-purpose policy learning. A future system need not choose between a large navigation foundation model and a tiny environment-specific model. A general visual encoder or visuomotor policy could be initialized from diverse real-world data and then adapted with ReaDy-Go-style episodes generated from the final deployment site. Finally, the controlled Vid2Sim comparison is stronger than a generic “our simulator works” demonstration. By holding the planner, trajectories, and policy architecture fixed, the experiment isolates a concrete representation choice. Within the limits of the evaluation, it supports the claim that photorealistic Gaussian humans are more useful training stimuli than conventional simulator humans. Limitations and open questions. The largest limitation is that this is not yet a general dynamic-world simulator. The static environment is frozen, while only one or two humans move. The people follow prescribed trajectories and do not react to the robot, negotiate right of way, stop unexpectedly, form groups, open doors, move objects, or interact physically with the scene. The motion model also emphasizes visual plausibility over physical or social fidelity. PriorMDM generates articulated locomotion along a specified path, but there is no contact-aware dynamics engine or explicit model of pedestrian intent. The expert assumes constant velocity over a two-second horizon, then compensates through replanning. That is reasonable for generating demonstrations but insufficient for crowded or adversarial social navigation. Appearance diversity is limited to six Gaussian avatars. Testing with six previously unseen real people is encouraging, but a more demanding evaluation would vary clothing, body shape, gait, carried objects, mobility aids, children, groups, partial visibility, and dense bidirectional traffic. The real-world sample size is small: ten episodes per environment and task, on one robot platform, with no reported confidence intervals. The one-meter success radius is also fairly permissive for indoor service robotics. Additional metrics such as minimum time to collision, personal-space violations, intervention count, path-length efficiency, and speed-conditioned safety would provide a more complete picture than success and ART alone. ART must be interpreted carefully because every failure is assigned 50 seconds. It is useful as a combined measure of reliability and efficiency, but it cannot distinguish a cautious success from a fast collision without consulting success rate and failure categories. The comparison against general navigation models is informative but not fully controlled. ReaDy-Go receives an odometry-derived goal vector, while GNM, ViNT, and NoMaD receive goal images and were not adapted to the three target scenes. The experiments convincingly show the value of environment-specific training, but not that the ReaDy-Go policy architecture is intrinsically superior to modern general navigation architectures. Ablation coverage is another open issue. The central ReaDy-Go-versus-Vid2Sim comparison is valuable, but the paper does not separately quantify the contributions of PGSR geometry, opacity filtering, covariance-based human filtering, motion diffusion, temporal frame spacing, avatar diversity, dataset size, or reactive replanning. Nor does it measure whether higher rendering fidelity correlates continuously with better policy transfer. The data pipeline still has nontrivial setup cost. It requires a static site video, an ArUco marker for metric alignment, structure from motion, Gaussian reconstruction, pretrained human assets, motion-generation dependencies, and substantial offline rollout time. The released code is valuable and MIT-licensed, but the repository also depends on several external projects and datasets with their own setup and licensing requirements. Finally, behavioral cloning from successful expert rollouts does not explicitly teach recovery from unseen states or offer a safety guarantee. DAgger-style correction, offline reinforcement learning, uncertainty-aware control, or a runtime safety shield could complement the photorealistic dataset. The authors similarly identify safe reinforcement learning, denser dynamic scenes, aggressive motions, and agents beyond humans as future directions. How roboticists might build on it. For a practitioner, the most reusable element is the pipeline abstraction rather than the exact two-million-parameter policy. A service-robot team could reconstruct each deployment floor, replace the six human avatars with a broader library, and generate site-specific edge cases before installation. Industrial users could substitute articulated workers, rigid Gaussian forklifts, carts, pallets, or other facility-specific agents. The motion module need not always be a human diffusion model; rigid objects could follow trajectory predictors, while articulated machinery could use known kinematics. A stronger learning stack could retain ReaDy-Go’s data engine while replacing pure behavioral cloning with closed-loop dataset aggregation. Scenarios could be selected adversarially based on current policy failures, concentrating rendering and planning compute on near-collisions, blind corners, and ambiguous passing situations rather than uniformly sampled episodes. The same system could also serve as an adaptation layer for a general navigation model. Instead of training a compact encoder from scratch, one could fine-tune a pretrained visual representation on target-site Gaussian rollouts, preserving broad semantics while acquiring the site’s exact geometry and dynamic-agent appearance. The released implementation makes this line of work unusually actionable. It exposes the scene preparation, human planning, expert planning, trajectory rendering, dataset collection, imitation-learning, validation, and experiment stages rather than publishing only a renderer or a trained checkpoint. Final assessment. ReaDy-Go’s key message is simple: a digital twin of a navigation environment is incomplete if it reproduces the walls but not the people. Technically, the paper succeeds by integrating several mature components—PGSR, HUGS, SMPL, PriorMDM, occupancy mapping, Hybrid A*, and imitation learning—into a coherent real-to-sim-to-real pipeline. None of those pieces alone is the central novelty. The novelty is making them work together as a scalable generator of dynamic, site-specific visuomotor data. The evidence is strongest where the experimental design is most controlled. ReaDy-Go and Vid2Sim are comparable in static simulation, but diverge once people move. That divergence appears again, more strongly, on the physical robot. It supports a useful conclusion for embodied-AI researchers: the realism of dynamic entities can matter at least as much as the realism of the static background. ReaDy-Go does not solve crowded social navigation, dynamic mapping, or general-purpose visual navigation. What it provides is a credible next step for Gaussian-splat robotics: from static photorealistic replicas toward dynamic training grounds in which compact policies can learn the particular hazards of the places where they will actually operate.