Latest / Embodied AI 101 / Imagined Motion, Grounded Geometry: How Rest2Art Makes Closed Objects Articulate

Transcript

Imagined Motion, Grounded Geometry: How Rest2Art Makes Closed Objects Articulate · Embodied AI 101

• Imagined Motion, Grounded Geometry: How Rest2Art Makes Closed Objects Articulate. Articulated Object Reconstruction from Rest-State Observation • Daeun Lee, Jaeah Lee, Woosung Kim, Haebeom Jung, and Jaesik Park • Seoul National University • ECCV 2026. A closed cabinet is visually quiet. Nothing is moving, the hinges may be hidden, the drawer fronts may sit flush with the frame, and several mechanically different designs can produce almost the same image. Yet a person can usually look at that cabinet and make a reasonable guess about which panels are doors, which are drawers, and how each one should move. Rest2Art asks whether a reconstruction system can do the same thing—and then go further by producing an articulated 3D asset that can be imported into a simulator. The paper’s central idea is unusually clean: when the observation contains no motion, generate plausible motion as a hypothesis, but do not trust the generated pixels. Instead, use an explicit 3D mesh to determine which hypotheses are geometrically consistent. The video model supplies a counterfactual observation of what might happen; classical rigid-body fitting and collision reasoning turn that proposal into joint parameters. That division of labor is more important than any individual model in the pipeline. Rest2Art uses vision-language models, segmentation foundation models, image-to-video diffusion, point tracking, mesh processing, and nonlinear optimization. But each learned component is treated as noisy. The mesh acts as a common workspace where those predictions can be compared, corrected, and fused. There are two qualifications worth stating immediately. First, although the project is naturally described as single-image articulated reconstruction, the core algorithm formally begins with a rest-state mesh. A single RGB image is supported by placing an external image-to-3D generator in front of Rest2Art. Second, “no motion data” means that the target object never needs to be observed moving. The system still imports a substantial learned motion prior from a pretrained video diffusion model. Rest2Art eliminates object-specific dynamic capture, not motion knowledge in general. The Inverse Problem Hidden Inside a Closed Cabinet. A simulation asset needs more than a visually convincing surface. It needs a decomposition into rigid links, a parent-child structure, and a parameterized joint for every movable part. For a drawer, that means a translation direction and an allowable displacement. For a door, it means a rotation axis, a point through which that axis passes, and some estimate of its angular range. If multiple articulation states are available, much of this structure becomes observable. Points on a drawer translate together. Points on a door trace arcs. The static body remains fixed while moving parts separate from it. Systems such as Ditto, PARIS, ArtGS, and the REArtGS family exploit precisely these changes, using paired states, interaction sequences, or video to disentangle rigid components and fit their joints. The rest-state setting removes that evidence. A rectangular front panel could be a drawer translating toward the camera, a door rotating around a vertical hinge, or a decorative panel that does not move at all. A hidden hinge can lie on either edge. Two adjacent fronts can be separate drawers or one larger door with decorative seams. Even perfect 3D geometry cannot always resolve these alternatives because the decisive mechanism may be completely occluded. This makes Rest2Art an abductive system rather than a conventional reconstruction system. It is not recovering a uniquely observable answer. It is combining appearance, object semantics, common mechanical patterns, and geometric constraints to select a plausible explanation. That distinction matters for how we should interpret the output. Rest2Art cannot prove that a cabinet opens in a particular way. It estimates the articulation most compatible with its learned priors and visible geometry. For simulation generation, that can be enough. For safety-critical manipulation, the prediction should still be treated as a hypothesis to verify through perception or interaction. The focus of the paper is correspondingly constrained: mostly openable furniture and related rigid objects, represented with revolute or prismatic joints. These cover doors, lids, drawers, knobs, laptop screens, and many familiar mechanisms, but not deformable parts, screw joints, coupled linkages, folding chains, or general non-rigid motion. What Rest2Art Actually Takes In—and What It Produces. The formal input is a mesh showing an object in a single configuration, ideally its closed or resting state. The output is a set of part-level meshes and a kinematic tree connecting them. The number of parts is not supplied in advance. That mesh can come from several sources. In the main quantitative experiments, the authors reconstruct it from multiview images using 2D Gaussian Splatting, chosen in part because it retains metric scale. Existing scanned or artist-created meshes can be used directly. For a single image, the mesh is first generated with an off-the-shelf image-to-3D model such as SAM 3D or a structured-latent 3D generator. Rest2Art then runs unchanged on that generated surface. So the single-image result is compositional:. RGB image → generated static mesh → Rest2Art segmentation and articulation → completed links and URDF. This is not merely a semantic distinction. The upstream 3D model determines the quality of the back surface, the separation between adjacent panels, the object’s canonical orientation, and potentially its scale. Rest2Art cannot recover a correct hinge from a mesh in which the door has already fused into the cabinet body. Its strongest metric-scale claims therefore apply to calibrated multiview reconstruction or scans, not automatically to arbitrary single-image-generated meshes. Before reasoning about parts, the system canonically aligns the object and rerenders it from 12 views concentrated near the front, where furniture articulation is usually most visible. Standalone meshes are rendered at 800 by 800 pixels with a fixed camera arrangement; reconstruction-based inputs can reuse their original renderer and camera calibration. From there, the pipeline has three substantive stages:. Discover and segment the movable parts. Infer their joint types and parameters using generated motion. turn the resulting surface patches into closed volumes suitable for simulation. Part Discovery by Making Foundation Models Disagree Productively. The first stage addresses what is arguably the hardest subproblem: deciding which portions of a closed object are independently movable. Rest2Art begins by asking a vision-language model to describe the object as a hierarchy. A cabinet prediction might include a fixed base, two doors, seven drawers, and a collection of handles. In the reported implementation, GPT-5.2 processes the rerendered views using six in-context examples and a temperature of 0.4. The system favors the candidate hierarchy containing the greatest number of plausible parts, reducing the chance of immediately collapsing several subtle panels into one component. A hierarchy from a VLM is semantically rich but spatially imprecise. It may recognize that cabinets contain drawers while miscounting the particular drawers in front of it. Rest2Art therefore turns each predicted part category into a segmentation prompt and queries SAM 3 on every view. The key operation is not simply combining the two outputs. It is comparing their implied instance counts. If the hierarchy predicts three drawers and the segmenter consistently finds three, those masks become validated evidence. If the segmenter finds more drawers than the VLM predicted, Rest2Art treats the hierarchy as incomplete. It overlays the detected masks in vivid colors and sends that annotated image back to the VLM, which can now see boundaries that were nearly invisible in the original texture. The reverse disagreement is handled more conservatively. If the VLM predicts a part that the segmenter fails to find in one view, the system does not immediately delete it. That failure could be caused by occlusion or a poor viewing angle. Rest2Art instead waits for evidence from the remaining views. This asymmetry is sensible. Extra high-confidence segmented instances suggest that the semantic description undercounted the object. A missing mask in one image says relatively little because segmentation quality varies sharply with visibility. The loop runs for at most two rounds in the default configuration. The mesh also helps enforce consistency across viewpoints. High-confidence masks are projected onto the 3D surface and then reprojected into other cameras as point prompts. A newly prompted mask is accepted only when it agrees with previously cached evidence for the same part. This allows a reliable observation in one view to rescue a weak observation elsewhere without freely propagating segmentation errors. Once the masks have survived this verification process, they must be converted into one coherent 3D labeling. Rest2Art does this separately for each part. Views are ranked using detection confidence and mask coverage, with an additional preference for views that expose the children of hierarchical parts. Evidence from the best views is accumulated on individual mesh faces through z-buffer rendering rather than indiscriminately averaging every mask. That per-part weighting is an important design decision. The best image for identifying a left door may be a poor image for a recessed lower drawer. A global set of “good views” would force all parts to inherit the same visibility assumptions. The initial face labels are still noisy, so Rest2Art uses mesh connectivity as a regularizer. It keeps the dominant connected component for each label, temporarily unassigns isolated fragments, and then propagates labels through shortest paths on the surface graph. The propagation follows mesh topology rather than raw Euclidean distance, reducing the chance that a label jumps across a narrow air gap. Every propagated label is finally projected back into the strongest corresponding 2D view and accepted only if it lands inside that part’s mask. Conceptually, the mesh is functioning as a geometric blackboard. The VLM contributes semantic structure, SAM contributes spatial masks, and neither is allowed to write directly to the final answer. Their evidence must first become consistent with a common 3D surface. The ablations support this interpretation. GPT-5.2 alone obtains 62.3 percent hierarchy accuracy on the Articulated Containers Dataset. Co-refinement with SAM 3 raises that to 72.5 percent. The effect is larger for Qwen3-VL: hierarchy accuracy rises from 52.2 to 72.5 percent, while part-count accuracy rises from 47.8 to 63.8 percent. Under the default two-round limit, 84.1 percent of scenes reach agreement; extending the loop all the way to 20 rounds only raises convergence to 92.8 percent, with the remaining cases often oscillating between two plausible decompositions. That oscillation is informative. It is not merely an optimization failure. It exposes genuine representational ambiguity: there are objects for which the semantic and spatial models settle on different, internally coherent explanations. A future version of this system should arguably preserve both explanations rather than forcing a single tree. Using Video Diffusion as a Counterfactual Motion Sensor. After segmentation, Rest2Art knows where the candidate parts are but not how they move. This is where the paper introduces its most distinctive idea. The system selects the view with the best average visibility and blends every movable-part mask onto it using distinct colors. A vision-language model writes a prompt describing the expected articulation, and the prompt is explicitly suffixed with instructions to keep the camera locked. The colored overlay and prompt condition Wan2.2-I2V-A14B, augmented with a VBVR LoRA intended to improve physical and temporal reasoning. The implementation generates 81 frames at 960 by 960 resolution and samples two random seeds by default. The mask overlay is not cosmetic. A video generator presented with a textureless cabinet may fail to distinguish a drawer front from the surrounding frame. Coloring the parts externalizes the result of the segmentation stage and tells the generator exactly which regions are candidates for movement. In the authors’ ablation, adding the physical-reasoning LoRA and mask conditioning reduces the rate of hallucinated motion from 23.1 to 11.5 percent on the ACD-HSSD split. The fraction of videos judged to contain accurate articulation rises from 69.2 to 80.8 percent. The video is still far from guaranteed to be correct, but the conditioning moves it into a regime where it can serve as a useful proposal mechanism. Dense point trajectories are then extracted with CoTracker3. Tracking points are initialized inside an eroded version of each part mask so that uncertain boundary pixels do not dominate. The first video frame corresponds to the real rest-state rendering, so Rest2Art can read metric depth from the mesh and unproject each tracked pixel into a 3D anchor point. Only the initial positions are geometrically trusted. Their future locations come from the generated 2D trajectory. Generated videos and trackers both degrade over time, so Rest2Art stops using a sequence when fewer than half of the original tracks remain visible. This visibility gate acts as a joint detector for tracker drift, occlusion, and the characteristic late-frame breakdown of image-to-video generation. Before fitting a joint, the method classifies the motion as static, prismatic, or revolute. It uses two simple but physically meaningful cues. For translation, every point on a rigid part should move by roughly the same distance and in roughly the same direction. Under rotation, points farther from the hinge travel farther, so displacement magnitudes vary across the part. Rotational tracks also bend over time, whereas translational tracks are approximately straight. Rest2Art measures both the variation in point displacement and the curvature of trajectories. If those cues disagree, it fits both motion models and chooses the one with lower reprojection error. The actual joint fitting happens by repeatedly asking: if these known 3D anchor points moved according to a candidate rigid joint, where would they appear in every video frame?. For a drawer, the optimizer searches for one 3D translation direction and a scalar displacement at each time step. For a door, it searches for a 3D rotation axis and a frame-specific angle. The resulting 3D points are projected through the known camera, and robust optimization minimizes their distance from the generated 2D tracks. Outliers are down-weighted and then trimmed before a final fit. This is the point at which Rest2Art stops treating the generated video as an image and starts treating it as a noisy measurement device. A monocular trajectory still leaves severe ambiguity, especially for revolute joints. Several differently oriented 3D axes can produce almost identical 2D arcs. Rest2Art resolves part of this ambiguity by rotating the segmented child mesh around each candidate axis and penalizing interpenetration with the parent. It also nudges the axis toward the dominant direction of the boundary between the two parts, since hinges often align with that interface. Multiple random video seeds provide another source of robustness. Fits are produced independently, scored using displacement and reprojection quality, and combined when they agree. If every initial sample says a supposedly movable part is static, the system can generate two additional videos as a fallback. Why not ask the VLM to output a hinge axis directly? Articulate AnyMesh does something close to that, using labeled candidate points and geometry-aware visual prompts. Rest2Art argues that direct continuous 3D regression from a VLM is overly dependent on canonical viewpoints. In a yaw-perturbation experiment, VLM predictions degrade as the object rotates away from familiar axes, while the motion-fitting pipeline remains more stable. Video models do not need to name the axis numerically; they only need to produce enough temporal structure for geometry to recover it. The larger lesson is that a generative model can be useful without being geometrically correct. The video need not preserve every texture or maintain perfect rigidity. It only has to make the correct part move in approximately the correct manner. A rigid model, a robust loss, and the original mesh can reject much of the remaining deformation. This reframes video diffusion as a kind of counterfactual sensor. When the desired physical experiment was never recorded, the model imagines several possible experiments, and a constrained estimator extracts whichever physical signal survives. Completing Surface Patches for Simulation. Segmentation produces open surface shells rather than independent solids. That is insufficient for collision checking, inertia computation, or reliable URDF import. Rest2Art therefore offsets each part inward by a nominal thickness of 1.5 centimeters and connects the original and offset boundaries with side walls. Normals near segmentation cuts are smoothed before offsetting, and thickness is reduced near neighboring parts to avoid introducing new interpenetration. Already watertight components are left unchanged. Drawers receive additional treatment because their interiors are completely hidden in a closed observation. Rather than pretending to reconstruct the true drawer box, Rest2Art inserts a parametric tray whose dimensions are 75 percent of the inferred drawer extent and whose depth is limited by the available space inside the parent body. This is best understood as functional completion, not hidden-surface reconstruction. The output has a plausible volume that can contain objects and participate in collisions, but there is no evidence that it matches the manufacturer’s real internal geometry. The distinction extends to the phrase “simulation-ready.” Rest2Art clearly produces usable link meshes, a kinematic tree, and joint parameters, and the paper demonstrates URDF export. But it does not systematically identify mass distributions, friction, damping, compliance, contact materials, or actuator dynamics. For robotics, I would treat the asset as kinematically deployable rather than as a calibrated dynamics twin. Reading the Experimental Results Carefully. Rest2Art evaluates segmentation and joint estimation primarily on the Articulated Containers Dataset, introduced by the S2O work as a more realistic alternative to the relatively repetitive PartNet-Mobility objects. Rest2Art reports results on ACD subsets derived from HSSD and Amazon Berkeley Objects, with all inputs rendered in closed configurations. It also uses MultiScan for real RGB-D scene data and presents qualitative tests on web images, generated meshes, and scanned environments. The segmentation results are striking. Rest2Art reports an IoU of 0.88 and mAP of 0.89, compared with IoUs of 0.20 for Find3D and 0.21 for PartField. Find3D reaches a relatively high mAP of 0.70 despite its low overlap because many queried parts are under-segmented while the large base region dominates evaluation. PartField also receives the ground-truth part count in one comparison setting, whereas Rest2Art discovers the count. Those numbers suggest that semantic-spatial co-refinement is doing real work, although they should not be read as a universal verdict on open-vocabulary 3D segmentation. Rest2Art is specialized around openable object hierarchies, carefully selected renderings, and repeated model queries. The feed-forward baselines solve broader segmentation problems under different computational assumptions. For joint estimation on ACD-HSSD, Rest2Art obtains 74.49 percent joint-type accuracy, an angular error of 11.35 degrees, and a positional error of 0.85 decimeters—about 8.5 centimeters. On ACD-ABO, it reaches 73.38 percent type accuracy, 4.78 degrees of angular error, and 0.20 decimeters, or about 2 centimeters, of positional error. These are the strongest type-accuracy results among the methods directly applicable to the rest-state input in the table, while other systems remain better on selected axis metrics. One metric detail is especially important: angular and positional errors are computed only when the predicted joint type is correct. A method can therefore show a low axis error on a small subset of easy, correctly classified parts while misclassifying many others. Joint-type accuracy and conditional axis accuracy must be read together. The comparison against motion-observing methods also requires care. ArtGS, REArtGS, REArtGS++, and Articulation in Motion cannot run directly from a closed state. To construct their required inputs, the authors select a plausible late frame from Rest2Art’s generated video, synthesize additional viewpoints with Qwen-Image-Edit, and estimate their cameras using MASt3R-SfM. Those baselines are therefore being tested on synthetic multistate observations rather than the geometrically consistent real observations for which they were designed. Small cross-state errors can make a static base appear to move and cause their reconstructions to collapse. This is a useful stress test, but it does not establish that Rest2Art is generally better than two-state reconstruction when genuine motion capture is available. It establishes that converting a generated video into complete, mutually registered multiview states is brittle—and that fitting a low-dimensional motion model directly to generated trajectories can be more robust in the no-observation regime. On MultiScan, Rest2Art reports 17.23 degrees of angular error, 0.26 decimeters of positional error, and 65.78 percent type accuracy over 48.94 percent of evaluable scenes. Articulate AnyMesh covers more scenes, at 61.70 percent, but has substantially lower joint-type accuracy and higher axis errors. Here, however, both single-state methods are run on the two available scans and allowed to report the result with lower reprojection error. The experiment demonstrates resilience to real scanning noise, but it is not a strict one-closed-image protocol. The qualitative scope is broader than the quantitative benchmark. Through a meta-prompt that rewrites the object vocabulary, the method is shown on eyeglasses, laptops, scissors, buckets, moka pots, and mechanical robots. It is also demonstrated on raw ScanNet++ geometry and on single casually captured images reconstructed with SAM 3D. These examples support modularity, although they do not yet establish category-general joint accuracy. Runtime is currently closer to offline asset preparation than interactive perception. On one NVIDIA H200, the reported Rest2Art stages take roughly 277 seconds per object, excluding upstream mesh reconstruction. Video generation alone averages 172 seconds, followed by about 40 seconds for mesh completion and 30 seconds for co-refinement. Tracking and joint fitting are comparatively cheap, taking six and seven seconds. How Rest2Art Fits into the Articulated-Asset Landscape. The closest task precedent is S2O, or Static to Openable Enhancement. S2O starts with a static triangle mesh and decomposes the problem into openable-part segmentation, motion prediction, and interior completion. It also introduced ACD and showed that segmentation quality is the dominant bottleneck on realistic assets. Rest2Art inherits essentially the same functional objective but replaces task-specific learned segmentation and motion heuristics with foundation-model co-refinement and generated motion hypotheses. The closest methodological baseline is Articulate AnyMesh. It also accepts a static mesh, predicts movable-part labels using a VLM-assisted segmentation pipeline, and estimates prismatic or revolute joints from geometric clues. For hinges, it projects candidate connection points into annotated renderings and asks a VLM to select the relevant axis. Rest2Art’s main departure is to avoid treating the VLM as the final continuous joint regressor. It generates an actual trajectory and solves for the joint that explains that trajectory while remaining compatible with the mesh. SINGAPO approaches the ambiguity more explicitly as generation. From one rest-state image, it predicts a connectivity graph and samples part attributes with a diffusion model, then assembles an asset by retrieving meshes from a part library. This allows several plausible objects to be generated from an ambiguous view. The tradeoff is identity and metric fidelity: the result is a plausible member of the distribution, not necessarily a reconstruction of the observed object’s precise surface. Rest2Art instead tries to preserve the given mesh and lets only the missing articulation come from a generative prior. URDFormer learns a much more direct image-to-URDF inverse model from procedurally generated scenes translated toward realistic appearance. Its strength is scalable, fast generation of simulation environments from ordinary images, including full kitchens. Its structure and geometry are more strongly tied to the synthetic asset families and representations used during training. Rest2Art is slower and more modular, but it can retain the geometry of a separately reconstructed object. Real2Code occupies another useful point in the design space. It reconstructs segmented part geometry from several unposed images, represents each part with an oriented bounding box, and asks a fine-tuned CodeLlama model to generate an executable articulated description. It scales to objects with as many as ten moving parts, but its strongest setting contains visible articulated geometry across multiple views. Rest2Art targets the harder case where those displacements never appear. Seen together, these systems offer three strategies for missing articulation:. Learn a direct inverse map from images to a simulator representation. Generate an entire plausible articulated object from a learned object distribution. Preserve the observed geometry while sampling only the missing motion evidence. Rest2Art is the clearest example of the third strategy. What Roboticists Should Borrow—and What Still Needs Work. The most transferable idea is not the particular combination of GPT-5.2, SAM 3, and Wan2.2. Those components will change quickly. The enduring pattern is to use foundation models as proposal distributions and explicit geometry as a verification layer. That pattern could apply beyond articulation. A video model could propose object-flow hypotheses for manipulation; a VLM could propose support relationships; an image generator could expose likely occluded states. The result should then be tested against camera geometry, rigid-body constraints, collision conditions, force closure, or task feasibility. Rest2Art shows that generative errors become more manageable when the downstream estimator asks for a small physical quantity rather than accepting generated pixels wholesale. The explicit mesh is equally important. End-to-end latent systems can be elegant, but modular robotics pipelines need a representation through which models with different interfaces can communicate. Rest2Art’s mesh supports visibility testing, label fusion, adjacency-aware propagation, metric unprojection, collision penalties, boundary analysis, mesh completion, and simulator export. It is not merely a rendering output; it is the pipeline’s shared state. There are, however, several clear opportunities for improvement. The first is uncertainty. Rest2Art samples multiple videos but ultimately emits one articulated asset. For a visually ambiguous cabinet, a better interface would preserve a ranked set of kinematic hypotheses: left-hinged door, right-hinged door, or drawer. A downstream robot could then select an information-gathering action that discriminates among them. The second is active verification. Rest2Art reconstructs what a robot might test next but does not close that loop. Even a few millimeters of observed motion after a low-force push could collapse much of the posterior over joint type and axis. The natural extension is rest-state inference followed by uncertainty-aware interaction. The third is upstream geometry dependence. The paper reports that different mesh backends significantly change segmentation performance, and its failure analysis notes that poor part boundaries directly degrade collision-based axis refinement. Single-image meshes add another source of hallucination, especially on backs, interiors, and thin parts. A joint segmentation-remeshing stage could make the surface representation better suited to articulation rather than merely appearance. The fourth is the video prior itself. Despite the overlay and LoRA, two of 69 analyzed videos contain geometry corruption severe enough to prevent meaningful tracking. Short-range or subtle motions are another failure mode because the resulting displacement is too small for reliable fitting. More samples would help, but a better solution may be a motion generator trained to output rigid object flow, articulated scene graphs, or 3D trajectories rather than photorealistic RGB. Finally, deploying these assets for policy learning requires additional engineering. I would simplify collision meshes, identify or randomize inertial properties, add conservative joint limits, validate self-collision behavior, and domain-randomize the uncertain axes and interior dimensions. A visually compelling URDF should not be assumed to have sufficiently accurate contact dynamics for manipulation training. Reproducibility is also not yet turnkey. As of August 12, 2026, the public GitHub repository contains the project README and states that code is coming soon. The method also depends on several large pretrained systems and was benchmarked on an H200-class GPU. The Bottom Line. Rest2Art does not make the ambiguity of rest-state articulation disappear. Instead, it builds a disciplined pipeline for reasoning through that ambiguity. A semantic model proposes which parts exist. A segmentation model tries to locate them. Their disagreements trigger correction rather than being silently averaged. A video model imagines how the object could move. Point tracking converts that imagined motion into trajectories. Rigid-body fitting and mesh geometry decide which joint explains those trajectories without violating the object’s shape. Finally, geometric completion turns the surviving explanation into a usable articulated asset. The paper’s strongest claim is therefore not that a single closed image uniquely reveals an object’s mechanics. It is that sufficiently rich learned priors can compensate for missing observations when their outputs are treated as hypotheses and subjected to explicit physical constraints. For embodied AI, that is a compelling recipe: let generative models imagine what the sensors did not see, but let geometry decide what the robot should believe.