Latest / Embodied AI 101 / HSImul3R: Physics-Driven Reconstruction of Human–Scene Interactions

Transcript

HSImul3R: Physics-Driven Reconstruction of Human–Scene Interactions · Embodied AI 101

• HSImul3R: Physics-Driven Reconstruction of Human–Scene Interactions. Imagine taking a casual video of someone lounging on a sofa or picking up an object, and turning it into a fully 3D simulable environment – complete with the person and all nearby furniture, all arranged in a physically consistent way so a robot could “play it back” in a physics engine. This is exactly the goal of HSImul3R [Cao et al.†2025], a new framework for turning sparse, uncalibrated images (or video frames) into a simulation-ready 3D human–scene interaction (HSI). Most existing 3D reconstruction methods handle either the human or the scene but rarely both together; HSImul3R is the first to jointly reconstruct a coherent environment and the human pose with all interactions intact, and then even refine the result so that it will stay stable under gravity and contact in a simulator. In short, HSImul3R weaves physics-in-the-loop into the perception pipeline: it alternates between vision-based reconstruction and physics-based optimization to produce outputs free of floating limbs or phasing-through-walls. Ultimately this pipeline yields a fully consistent 3D scene (including the human), ready to feed directly into a humanoid simulator for downstream tasks like planning or learning. In this episode we’ll break down how HSImul3R works, why its contributions matter, and how it compares to other recent work on human–scene reconstruction and imitation. We’ll discuss its three main pieces – contact-aware modeling, reinforcement-learning–based refinement, and simulation-guided training – as well as the new HSIBench dataset it introduces. Along the way we’ll also situate it in context: related efforts like PROX and PhySIC have enforced contact constraints for single images, and pipelines like LEMO and more recent motion synthesis methods aim to incorporate scene affordances into human motion. HSImul3R builds on that foundation but goes further – integrating modern 3D generative priors and reinforcement learning to produce fully simulated human–scene interactions from everyday videos. The Problem: Bridging Vision and Physics. Why is physics so crucial in reconstructing human–scene interactions? Consider a typical reconstruction pipeline: perhaps one system uses multi-view photogrammetry or a depth sensor to recover the room geometry, and another estimates the human’s 3D pose (e.g. via a neural mesh model like SMPL-X). If we simply stitch those together, we often get absurdities: the person might float a few inches above the chair, or a hand might intersect solid objects. For example, the landmark PROX approach (ICCV 2019) penalizes body–scene interpenetration and encourages plausible contact surfaces. But that work assumes a known dense scene and focuses just on pose estimation. Another SIGGRAPH‘25 result, PhySIC, similarly optimizes a single image pair of human+room to avoid collisions and fill in missing geometry. However, these still solve a mostly vision-alignment problem with a bit of post-hoc physics. What HSImul3R emphasizes is truly simulation-ready output: the entire reconstruction should survive falling under gravity in a physics engine, with contacts holding up the person realistically. This means HSImul3R must close the loop between perception and simulation. On one hand, it uses state-of-the-art vision techniques (like a recent pressing of Structured-from-Motion with human priors) to get an initial 3D world. On the other hand, it puts that world into, say, a MuJoCo humanoid simulation to check if the human pose remains stable amid the recovered objects. If not, the system iteratively nudges the solution to enforce contact and fix penetrations through three key mechanisms:. Contact-aware interaction modeling – During vision reconstruction, HSImul3R leverages learned generative priors and explicit contact losses so that the human mesh and scene geometry “snug together” properly. Scene-targeted reinforcement learning – Once the rough static scene is built, a humanoid controller is used to adjust the pose so that, under gravity, the person stays sitting or leaning as in the video without drifting or jerking unnaturally. This is guided by both staying close to the original pose and respecting object proximity. Direct simulation reward optimization (DSRO) – Finally, the system fine-tunes its underlying generative model itself based on physics feedback. By running many samples through the simulator, the model learns (via a reward signal) to prefer outputs that remain stable with contact. Together these components form a bi-directional pipeline: image-based priors pull the reconstruction toward plausibility, while simulation rewards push the generators toward producing physically valid interactions. HSImul3R’s authors report that this yields much higher stability and far less interpenetration than baselines like HSfM alone. We’ll now unpack each piece in more detail. From Images to Scene: Contact-Aware 3D Modeling. The first step is to take the raw video or images and produce an initial 3D reconstruction of both the scene and the human. HSImul3R builds on the recent Humans-and-Structure-from-Motion (HSfM) framework (CVPR 2025) which jointly recovers 3D human meshes, camera parameters, and a scene point cloud from multiple uncalibrated views. HSfM itself combined a neural human mesh fitter with a data-driven structure-from-motion engine (DUSt3R) to resolve scale and position so that people and room are in one consistent metric frame. HSImul3R inherits this multi-view pipeline to get the “skeleton” of the reconstruction: a scene model plus a human body mesh (typically SMPL-X) in 3D. However, HSfM and its kin do not reason about physics or contact beyond basic geometry. HSImul3R’s contact-aware modeling layer injects physical plausibility into this stage. Concretely, the system uses a state-of-the-art 3D generative model (called MIDI, Multi-Instance Diffusion) that can produce detailed 3D shapes from images or segments. During reconstruction, MIDI provides a prior for how each object (like a chair or sofa) should look and where limbs ought to meet surfaces. HSImul3R then adds loss terms to the optimization: contact losses encourage human bones (e.g. buttocks, knees, hands) to lie flush with object surfaces when the video suggests touching, and non-contact losses punish any mesh vertices penetrating through the scene. For example, if the person is sitting, the sitting bone (pelvis) should lie flat on the sofa; if their hand brushes a table top, that mesh region should be slightly above the table surface. By aligning these generative priors with the reconstructed geometry, the method explicitly enforces “realistic human–scene coupling.” In practice this means jointly optimizing the human pose, the scene object positions, and even camera poses so that contact points match and collisions disappear. The OpenReview synopsis notes that this module “enforces realistic human–scene coupling within the unified 3D world”. In effect, the vision network is being told, “keep the hand resting on the table, not intersecting it.”. This contact-aware stage yields a better starting point: the human mesh is already pressed against the correct surfaces, and obvious intersections are (mostly) fixed. However, it’s still a purely geometric step – gravity isn’t applied yet. The scene might look fine as a snapshot, but will it hold up under physics? That’s where the next stage comes in. Stabilizing Poses: Scene-Targeted Reinforcement Learning. After contact-aware fitting, HSImul3R takes its static reconstruction and drops it into a physics simulator with a humanoid avatar. The initial pose from reconstruction (sitting, leaning, etc.) is used as a target, but we let a controller try to maintain it under dynamics. Specifically, the team adapts the Perpetual Humanoid Control (PHC) framework (ICCV 2023) for this task. PHC is an RL-based full-body controller originally designed to track motion data tightly; here it is used to refine static poses so they aren’t lost to gravity. The twist is that the controller is trained not just to match the original pose, but also to respect closeness to objects. HSImul3R adds a scene-proximity loss: if a knee or foot is supposed to touch the floor or sofa, the RL loss encourages it to stay near that surface. In other words, the policy must stabilize the pose by "leaning" on the scene objects in exactly the places identified by the contact modeling. This dual supervision—on both motion fidelity (stay close to the reconstructed pose) and object proximity (keep contact points in contact)—helps prevent odd drifting or jitter when the sim is stepped with gravity. Conceptually, this stage is akin to running the body through a few frames of simulated time and doing bound-constrained optimization via RL. It nudges each joint so that, for example, a seated person will sit firmly on the chair rather than slowly sink through it. The result is a static configuration that is dynamically stable (the pose stays put) and still visually matches the input images. In practice, the authors note the refined poses “align human contact points with nearby object surfaces for stable contact under gravity”. This physics-augmented RL refinement is one of HSImul3R’s clever hacks: by leveraging existing humanoid control policies, it effectively “muscles up” the reconstruction. Of course, focusing on static frames has its limits (we’ll revisit that later), but for posture-heavy interactions it already beats doing nothing. After this RL step, the scene and human are together in a self-consistent equilibrium ready for outright simulation, but HSImul3R doesn’t stop here either. DSRO: Learning from Simulation Rewards. The final, most novel piece of HSImul3R is Direct Simulation Reward Optimization (DSRO). Here, the system uses the simulator not just to refine a single pose but to improve its 3D generator itself. The idea is inspired by RL from human motion videos (e.g. SFV and recent imitation projects) but applied in a new way. In DSRO, HSImul3R repeatedly samples human–scene configurations from its 3D generative model (the diffusion-based GDimodel). Each sample is run in the physics engine with gravity and contacts. If the scene remains stable (no falling through, finished pose holds), it is given a positive reward; if the person penetrates objects or collapses, it is penalized. Crucially, this reward signal is used to fine-tune the generative model, biasing it toward outputs that survive simulation. Concretely, the authors mention “flipping the diffusion loss sign based on a binary stable/unstable label” to push the model away from bad configurations. The result is that the diffusion prior “learns” a rough physical prior: future samples from the model are less likely to violate stability. One can think of it as aligning the model’s latent space so that modes correspond to physically plausible interactions. DSRO closes the loop: simulation outcomes are fed back to perception. It is a form of direct (even if heuristic) reward conditioning on a generative network. In practice, the authors show that after DSRO training, the reconstructions become noticeably more robust in the physics checks. The reviewer summary notes: “scenes that remain stable under contact are rewarded, while unstable ones are penalized, thus biasing generation toward physically valid geometries.”. To our knowledge, this is quite an innovative twist. Past systems either separately refine pose or do standard GAN/VAEs; HSImul3R uses physics as a teacher. It echoes work like SFV (SIGGRAPH Asia 2018) where deep RL controllers learn from video clips, but here the goal is not animating motion but producing a geometry distribution that is itself stability-aware. The review does flag that DSRO is somewhat heuristic and binary, but as a first proof-of-concept it’s compelling: the final HSImul3R pipeline produces reconstructions that largely behave well under gravity, barreling past simpler methods. HSIBench: Benchmarking Stability. To quantify how well these strategies work, Cao et al. also introduce a small benchmark HSIBench for human–scene reconstruction. It consists of 300 multi-view recordings of humans interacting with various household objects (3 subjects, 19 object types). Importantly, each recording has accurate camera calibrations and a rough scene scan, enabling evaluation of the 3D output. But HSIBench is special in focusing on physical consistency: the reconstructions can be dropped into a simulator to see how often they hold up. In their experiments, HSImul3R is compared against baselines like HSfM without physics. The headline results are that HSImul3R achieves much higher stability (percentage of poses that remain upright without collapsing) and far lower penetration rates than the baselines. For example, a seated pose produced by plain vision methods might slip partway through the chair under simulation, whereas HSImul3R’s refined pose stays put. Qualitatively the authors show fewer limbs poking through objects and generally more natural contact behavior in the final reconstructions. HSIBench itself is a useful byproduct: it highlights that even state-of-the-art image-based methods need this extra physics-aware step. By releasing HSImul3R and HSIBench, the authors hope to encourage further research on “physically consistent” 3D perception. Indeed, this dataset helps evaluate not just reconstruction accuracy but physically plausible interaction, a niche that standard 3D benchmarks ignore. How It Compares to Related Work. HSImul3R sits at the intersection of 3D vision, generative modeling, and embodied simulation, so there’s quite a bit of context. On the vision side, consider:. HSfM (CVPR 2025): Co-authored by some of the same group, HSfM merges classic SfM with learned human mesh priors to reconstruct people and environments together. It solves scale ambiguity by using the human as a metric reference and does bundle adjustment over body joints. However, HSfM stops at geometry; it doesn’t enforce contact beyond reprojection consistency. HSImul3R builds on HSfM’s output but adds the physics cascade to ensure the result won’t fall apart in simulation. PROX (ICCV 2019) and PhySIC (SIGASIA 2025): These methods tackled image-to-3D pose with scene constraints. PROX introduces interpenetration and contact losses in SMPL optimization, and PhySIC jointly optimizes human mesh and scene mesh from a single image to avoid impossible poses. In spirit they echo HSImul3R’s contact objectives, but HSImul3R goes further by handling multiple views, reconstructing the full scene, and explicitly using simulation stability as feedback. LEMO (ICCV 2021): This sequence-oriented method learns motion priors from large MoCap data (AMASS) and incorporates a contact term to smooth out 4D body capture in scenes. LEMO shows that modeling contact and motion smoothness yields more realistic tracked motion. HSImul3R similarly cares about contact, but applies RL instead of hand-crafted priors to enforce it in static frames. On the motion synthesis / imitation side, there’s also a connection:. Synthesizing Long-Term Motion (Wang et al.): That 2020 arXiv paper generates long-range 3D human motion conditioned on scene affordances. It highlights that motion generation methods often ignore objects unless they enforce geometry constraints (walking on floor, sitting on chairs, etc.). HSImul3R isn’t a motion generator, but its philosophy is similar: integrate scene geometry into the human modeling, and “optimize” (via RL) for feasible interaction. In fact, that work uses an optimization pass to improve affordances, analogous to HSImul3R’s simulation checks. VideoMimic (CoRL 2025): This very recent system reconstructs 3D scenes and human motion from a single video, then does imitation learning on a humanoid. It also produces simulation-ready avatars from video. The big difference is scope: VideoMimic focuses on single-camera videos and on learning controllers for locomotion tasks (like walking, climbing), whereas HSImul3R addresses multi-view images and static interaction scenes. Both use “real-to-sim” pipelines, but HSImul3R goes deeper into the geometry – videoMimic uses traditional structure-from-motion plus a learned motion retargeter, while HSImul3R invents losses and RL stages specifically for pose stability. Finally, from the robotics perspective, HSImul3R’s output is tailor-made for humanoid simulation. The paper even demonstrates deploying the reconstructions into a simulated robot (e.g. a humanoid model) for manipulation or world modeling. A stable, physically plausible reconstruction means the robot’s controllers won’t glitch out due to collisions or drift. In that sense, HSImul3R is a step toward closing the gap in the “visual sim-to-real” pipeline: instead of only using synthetic 3D worlds, we can derive the world and human interactions directly from video, and feed that into a physics-based learning algorithm. Putting It All Together: Pipeline Overview. To summarize the full HSImul3R workflow in one shot:. Input: Uncalibrated color images (e.g. a few frames of a video) capturing a person interacting with objects. (No depth or stereo needed, though multi-view helps.). Initial reconstruction: Run an adapted HSfM pipeline. Detect 2D keypoints and fits a parametric human mesh in each view (via HMR or similar). Simultaneously use a learned SfM (DUSt3R) to get a rough point cloud + camera poses, then align human meshes into that space. The result is a 3D scene point cloud plus a posed SMPL-X human. Contact-aware refinement: Apply the MIDI generative prior to refine the shapes (chairs, tables) in multi-view. Add losses that enforce contact (human body parts near surfaces) and interpenetration avoidance. Jointly optimize human pose, object geometry, and camera until no obvious collision remains. Physic-based polishing: Embed the reconstruction into a physics sim with a humanoid avatar. Use a Perpetual Humanoid Control policy to track the static pose while adding a loss to keep contact points against surfaces. This yields a refined pose and scene configuration that is a true simulated equilibrium. Model fine-tuning (DSRO) (optional): Draw samples from the generative model, run them under physics, and use the result as a binary reward to tweak the diffusion model’s weights. Iterate so the model learns to prefer stable interactions. Output: A final 3D human mesh and object meshes in one scene, all consistent. Every contact labeled in the mesh should hold under gravity, so the person doesn’t clip or fall through. At this point you have a “simulation-ready” scene. Each step is bidirectionally linked: the vision losses use simulated contact points, and the RL controller uses vision-based pose clues. It’s a complex pipeline, but the authors argue each part is necessary to push the output into simulator viability. Empirical Results and Limitations. On HSIBench, HSImul3R shows clear gains in key metrics. For example, the percentage of human poses that remained stable during a simulated fall-test was much higher (often doubling the baseline), and the average penetration depth (how far bones poked into objects) was far smaller. Qualitative examples depict nicely seated people whose hands truly rest on table edges, vs. baselines where one small error causes jitter or slip in the sim. The report states “substantially higher physical stability and lower penetration compared to HSfM and other baselines”. That said, some constraints are inherent. All experiments in the paper involve posture-centric scenes – sitting, leaning, standing – essentially before-and-after snapshots. There is no long motion sequence or dynamic activity (no walking or jumping). The authors note that their “static-interaction assumption” limits the method: for fully dynamic actions one would need a much richer simulation (and likely model full whole-body motions, which could be a future extension). Also, DSRO is still heuristic: it treats stability as a binary reward without telling the model how to fix near-misses. The reviewers pointed out that flipping a sign on the diffusion loss is a blunt instrument. Indeed, one could imagine fancier loss shaping that actually penalizes the amount of penetration or adds learned physical priors. Finally, the pipeline is quite stitched together from many ingredients (diffusion models, SFm, RL controllers). It may be complex to implement end-to-end. The authors cite several components: DUSt3R for SfM, MIDI for generative shape, PHC for control, plus their new losses. This means there's not a single elegant algorithm, but rather a powerful engineering effort combining existing tools with new physics-centric constraints. But from a practical standpoint that can be fine – if it works, it can be used. And the integration itself is a valuable proof-of-concept: it shows that these pieces can be made to talk to each other and yield a robot-ready model from video. Significance and Future Directions. HSImul3R is a step toward truly embodied perception: converting everyday video data into robot-ready simulations. For roboticists interested in “learning from videos,” it offers a way to generate plausible 3D environments and human-interaction templates out of thin air (of RGB frames). Imagine a robot that watches someone load a dishwasher on YouTube and then gets a simulation of the whole kitchen with the person and looks of where to grip the plate – that’s the kind of end-goal HSImul3R hints at. More broadly, this work highlights a key insight: perceptual models should be tested in the loop with physics. A reconstruction is not truly “correct” if it falls apart under gravity. By embedding simulation feedback into the training loop (even to the level of fine-tuning a generative model), HSImul3R points to a new paradigm where perception and control co-evolve. We expect to see more of this in future: for instance, one could imagine adding tactile priors for soft objects, or extending to multi-human interactions, or integrating with vision-language instructions. On that note, HSImul3R currently handles humans and static objects. What about articulated objects (like opening a fridge) or fluid dynamics or deformables? Those would require even more sophisticated simulation. And the current setup assumes fairly well-behaved input videos (multiple views, static poses). The next challenge is generalizing to truly in-the-wild video: unknown camera path, moving subjects, occlusions, unknown object classes, etc. In the era of large-scale vision-language-action models, one could even imagine using HSImul3R-style reconstruction as a world model for a VLA agent to plan multi-step tasks with physical plausibility. In summary, HSImul3R tries to bring embodied intelligence full circle: from pixels to physics and back again. By combining generative priors, pose tracking, reinforcement learning, and physics simulations, it shows a concrete way that “thinking” (simulation) can improve “seeing” (3D reconstruction). It’s still early days – the ICLR reviewers found the work somewhat incremental and niche – but as a technical achievement it demonstrates that robotics and vision can finally shake hands over the 3D world. For practitioners, the code and models (once released) might become useful tools for any project that needs a simulation of human–object scenes from vision data. References: HSImul3R is described in Cao et al., “HSImul3R: Reconstructing Simulation-Ready Human-Scene-Interaction from Sparse Views,” 2026 (arXiv preprint). For background on related ideas see PROX (ICCV 2019), PhySIC (SIGGRAPH Asia 2025), LEMO (ICCV 2021), and “Synthesizing Long-Term 3D Human Motion…” (arXiv 2020). The VideoMimic project (CoRL 2025) also tackles video-to-humanoid-sim pipelines.