Latest / Embodied AI 101 / VS-Splat: Learning Where 3D Detail Belongs

Transcript

VS-Splat: Learning Where 3D Detail Belongs · Embodied AI 101

• VS-Splat: Learning Where 3D Detail Belongs. VS-Splat: Voxel-Selective feed-forward Gaussian Splatting for end-to-end 3D object reconstruction from sparse-views • Yunsu Jeong, Hyuk Heo, Youngsang Kwak, Jaehwa Kwak, and Il Yong Chun • Sungkyunkwan University and AiM Future • arXiv • 2026. Imagine a tabletop robot inspecting a mug. Most of the space around the mug is empty. The handle occupies a small fraction of the object’s bounding volume, but its shape might determine whether a grasp succeeds. A reconstruction system therefore faces an allocation problem: how much computation should it spend establishing that empty space is empty, and how much should it spend resolving the handle?. Increasing resolution everywhere is one answer. It is also an expensive answer—and potentially the wrong one. The interesting alternative is to learn where additional representational capacity will actually help. That is the central idea behind VS-Splat. The authors propose an object-centric, feed-forward Gaussian reconstruction system that learns spatial allocation from rendered-image supervision rather than explicit 3D occupancy labels. Instead of distributing fine-detail primitives uniformly, it concentrates them in selected regions. This episode covers the September 11, 2026 preprint: object reconstruction experiments, not a demonstrated closed-loop robot perception system. For Embodied AI 101, the useful question is therefore twofold. What makes learned voxel selection work as a reconstruction mechanism? And what additional evidence would turn that mechanism into something we should trust inside an embodied perception stack?. The allocation problem is not the rendering problem. First, separate two achievements that are often bundled together. The original 3D Gaussian Splatting work by Kerbl and colleagues made an explicit radiance representation practical for fast rendering. It also used iterative optimization and density control to adapt that representation to an individual scene. Fast rendering did not eliminate the reconstruction process; it made the resulting representation—and its optimization loop—efficient to evaluate. A Gaussian primitive carries a position, spatial extent and orientation, opacity, and appearance information. A rasterizer projects these primitives into image space, sorts their contributions, and blends them. Consequently, the representation has several different costs: predicting primitive parameters, storing them, projecting them, and compositing their overlapping footprints. Primitive count matters, but it is not the entire performance model. Feed-forward reconstruction changes where the expensive learning happens. GS-LRM, for example, processes tokens from posed input images and predicts per-pixel Gaussian parameters. Its reconstruction prior is learned across a dataset, rather than rediscovered through a long optimization for every new object or scene. Importantly, pixel-indexed prediction is not uniform sampling of a 3D bounding box: the image grid provides the prediction slots, while geometry determines where those slots land in space. That distinction prevents an overly broad reading of VS-Splat’s motivation. Different feed-forward methods waste capacity in different ways. A voxel-based method may allocate slots throughout a mostly empty volume. A pixel-based method may allocate slots to redundant observations, backgrounds, or multiple views of the same surface. Neither situation is adequately described by saying that all existing methods distribute Gaussians uniformly in 3D. Consider a simple allocation exercise. You have one thousand spatial cells and enough budget for sixteen detailed predictions at every cell. That produces sixteen thousand predictions. If only one hundred cells require detailed treatment, the same local capacity could be supplied with sixteen hundred predictions. But notice the qualification: local capacity. You still need to identify those hundred cells. A cheap global pass followed by expensive selective processing can be attractive; an expensive global pass followed by a tiny selective head may save little. VS-Splat’s public configuration uses a grid with 48 cells per axis and sixteen fine predictions per selected location. That grid contains 110,592 candidate locations. Sixteen fine predictions everywhere would mean 1,769,472 fine primitives. At an illustrative ten-percent selection rate, the corresponding count would be roughly 177,000. These are arithmetic examples, not measured output counts. The important design question is not simply whether the output is sparse. It is where sparsity enters the computation, and what information is available when the allocation decision is made. A coarse reconstruction establishes spatial addresses. The coarse stage lifts pose-conditioned features, aggregates their mean and variance, compresses them through a lightweight 3D U-Net, and decodes one Gaussian per voxel. The architectural lineage is useful here. LaRa builds image-feature and embedding volumes, combines local attention with spatial processing, and decodes a Gaussian volume. Its original representation uses 2D Gaussian primitives, and its base configuration produces two primitives at each location in a 64-cubed Gaussian volume. It is therefore a relevant architectural baseline, but not an identical representation with a different selector attached. A shared volume gives the network a common addressing scheme. Instead of asking which feature in image one corresponds to which feature in image two, the system can ask what each image says about a particular candidate location. That does not solve correspondence by itself. Projection tells you where a location would appear, not whether that location contains a visible surface. Nevertheless, it gives the subsequent network a geometrically organized collection of evidence. Here is a small thought experiment for understanding feature aggregation. Suppose three views assign scalar descriptors of two, two, and two to one candidate location. Another location receives zero, two, and four. Both locations have the same average descriptor, but only the second exhibits disagreement. An average alone would collapse that distinction. An average together with a dispersion statistic preserves at least some evidence about consistency. It still does not preserve everything: the summary forgets which particular view disagreed, and disagreement could reflect occlusion, appearance changes, or an incorrect geometric hypothesis. MVSplat provides a useful comparison. It constructs plane-swept cost volumes whose cross-view feature similarities inform depth prediction, then predicts Gaussian attributes using photometric supervision. That is a more explicitly correspondence-oriented use of multiple views. My architectural reading is that these designs make different compromises between retaining detailed matching evidence and compressing it into a manageable spatial representation. Neither choice removes the need to reason about visibility. For a roboticist, the practical lesson is to inspect the intermediate representation, not just the output splats. Ask whether the model retains enough information to distinguish “the cameras disagree,” “the surface is occluded,” and “there is probably nothing here.” Those are different conditions, even when they produce similarly weak reconstruction evidence. The key mechanism: connect selection to a rendering gradient. A Gumbel-Sigmoid score multiplies each coarse Gaussian’s opacity; scores above 0.5 select voxels for refinement. That short description contains the central learning mechanism. To understand it, we need to separate the continuous training signal from the discrete allocation decision. Jang, Gu, and Poole’s Gumbel-Softmax work introduced a differentiable relaxation of categorical sampling. The basic idea is to perturb learned preferences with random noise and produce a smooth approximation to a discrete choice. A temperature controls how soft that approximation is. With a lower temperature, outputs become more concentrated near discrete alternatives; at a higher temperature, intermediate values are more common. The closely related Concrete distribution, developed by Maddison, Mnih, and Teh, supplies the same general perspective: replace a difficult discrete stochastic node with a continuous, reparameterized one through which automatic differentiation can operate. This introduces a surrogate objective and biased gradients relative to the original discrete problem; it does not make the underlying discontinuity disappear. That last point matters. A hard decision about whether an element belongs in a set is still discontinuous. Calling a model end-to-end trainable does not mean every discrete indexing operation has acquired an exact derivative. The engineering task is to provide a useful learning path for the decision-making parameters. Opacity modulation supplies an intuitive path because opacity already affects rendered images. Consider a toy example of our own. A white primitive appears against a black background, with everything except its effective opacity held fixed. Its native opacity is 0.8, and its gate is 0.25. Their product is 0.2, so its contribution is much weaker than it would be with an open gate. If the target pixel is substantially brighter, increasing the gate could reduce the error. If the primitive introduces an unwanted bright mark, reducing the gate could help instead. The decision variable is no longer trained only through an abstract classification objective; it participates in the image-forming computation. Real Gaussian rendering is more complicated because multiple primitives overlap and contributions depend on visibility and blending order. A primitive hidden behind other contributions may receive little useful signal from that view. That complication follows from the renderer, not from the choice of Gumbel relaxation. There is also an identifiability issue worth keeping in mind. In our toy example, native opacity 0.8 with gate 0.25 gives the same product as native opacity 0.4 with gate 0.5. A rendering objective may care about that product without uniquely determining how responsibility should be divided between its factors. This is why I would not interpret the selector’s score as a calibrated probability of physical occupancy. It is trained to support reconstruction. A useful rendering contribution and a geometrically occupied cell are related concepts, but they are not interchangeable. The public training wrapper passes coarse and fine outputs to the loss computation. The configuration enables squared-image-error and SSIM terms, while assigning zero weight to LPIPS. For implementation, I would instrument the selector throughout training: score histograms, selected-location counts, gradient magnitudes, and the evolution of thin structures. A gate can become numerically decisive before it becomes semantically useful. Monitoring only the final image loss would make that failure mode harder to diagnose. Fine detail requires better evidence, not just more primitives. PAP resamples image features at predicted Gaussian centers using inverse-distance aggregation. SFR combines local and coarse features through sparse 3D convolutions; retained parents receive opacity and appearance updates. Inference omits coarse rendering. The interesting idea is to revisit image evidence after obtaining a more specific spatial hypothesis. Imagine that the center of a coarse spatial cell lies just behind a curved object surface. Sampling image features at the cell center and sampling them at an estimated surface-related point are not equivalent operations. Near a silhouette, a small displacement can move the projected lookup from background to foreground. Near a texture boundary, it can change which local appearance information the decoder receives. This gives us two separate opportunities for improvement: allocate more capacity to an informative region, and feed that capacity more accurately aligned evidence. GeoLRM is a relevant neighboring approach. It first estimates sparse occupied anchors and then uses a reconstruction transformer to refine geometry and retrieve image information through 3D-to-2D interactions. Its released training pipeline includes explicit occupancy-ground-truth generation and separate proposal and reconstruction training. It demonstrates that sparse spatial proposals and geometry-aware feature retrieval predate VS-Splat; the supervision and coupling of the stages are important distinctions. There is a caution about feature weighting, however. Camera distance is not the same thing as visibility. In a hypothetical tabletop capture, the closest camera might see a gripper occluding the object. A more distant camera could provide the cleanest observation. A distance-based weighting rule would need the learned features or downstream processing to compensate for that situation; distance alone cannot establish which observation should be trusted. I would therefore test view aggregation separately from allocation. Hold the selected locations fixed, then compare simple distance weighting with visibility-aware weighting or learned cross-view aggregation. That experiment could reveal whether the bottleneck is deciding where detail belongs or obtaining reliable evidence for the detail already selected. More primitives are only useful when their conditioning information can support a better prediction. What is new relative to other adaptive approaches?. Selective refinement is not new in isolation. LaRa already uses a coarse-to-fine appearance mechanism: primitive centers are projected into image and rendering information, and the fine stage predicts appearance refinements. Its coarse-to-fine process is therefore not equivalent to uniformly decoding a finished representation in one undifferentiated step. Generative Densification, by Nam and colleagues, is an even closer comparison. It selects coarse Gaussians using view-space positional gradients and predicts finer Gaussians by upsampling learned features. Its hierarchy also uses learned masking to decide which elements need further densification. The important shift is from repeatedly splitting raw parameters during per-scene optimization to learning how to generate additional detail. VS-Splat’s project description emphasizes a different decision point: learn which spatial regions should receive fine primitives using image-rendering supervision, without requiring structural 3D labels. The authors also report its usefulness as a backbone for an existing densification method. I find it helpful to organize the design space around three questions. First, what is being selected: image-indexed prediction slots, spatial cells, or already instantiated primitives? Second, what evidence drives selection: geometry labels, reconstruction gradients, learned features, or some combination? Third, what changes after selection: appearance, primitive count, spatial resolution, or all three?. Those questions are more informative than treating “adaptive Gaussian reconstruction” as a single category. They also suggest combinations rather than only competitors. A spatial proposal mechanism could determine where a first detailed representation is worthwhile, while a later densification mechanism could identify residual under-modeling within those regions. For someone adapting the idea, this is the transferable contribution: make allocation an explicit learned responsibility, with a training signal connected to the output you actually care about. Reading the evidence without overreading the headline. The paper reports the following PSNR results; higher is better. | Dataset | LaRa | VS-Splat | |---|---:|---:| | GObjaverse | 27.51 | 28.77 | | Google Scanned Objects | 29.81 | 30.84 | | CO3D | 21.18 | 21.62 |. CO3D uses 139 hydrant and teddy-bear objects. Its headline includes a pose-robustness extension; unextended PSNR is 21.31. The extension learns camera-ray embeddings with pose-derived supervision. That qualification changes how I would design a reproduction. I would create an experiment ledger that explicitly records the feature encoder, camera-conditioning variant, training recipe, view-selection policy, and evaluation subset. Otherwise, “the model” becomes a moving target: a result can accidentally be attributed to allocation when another component changed alongside it. The thirty-epoch ablation reports coarse, selected-coarse, and full-model PSNR of 27.80, 27.81, and 28.40 respectively. My reading is that the selector is best understood as enabling subsequent refinement, rather than as an image-quality improvement merely from deleting candidates. But the next experiment I would want is a matched-budget comparison. Keep the encoder, coarse backbone, training duration, and total fine-primitive budget fixed. Compare learned selection against random selection, opacity-based selection, and a dense allocation with fewer primitives per cell. That would help distinguish three hypotheses: better placement, a better feature-processing architecture, and simply a more favorable distribution of model capacity. I would also separate image reconstruction from geometric utility. The 2D Gaussian Splatting work explicitly motivates its surface-oriented representation by limitations of volumetric Gaussians for geometrically consistent surface recovery. Its use of oriented disks and geometric regularization illustrates why appearance quality does not settle the geometry question. For a manipulation-oriented benchmark, I would add surface error around handles, rims, contact patches, and narrow clearances. These regions need not occupy many pixels to dominate a downstream decision. Finally, I would inspect distributions, not just averages. Does the method help almost every object a little, or a subset a great deal? Does selection occasionally miss a thin component entirely? Are failures correlated with view coverage? Those questions determine whether an average reconstruction gain translates into a dependable component. What the timing numbers do—and do not—mean. For GObjaverse, four-input reconstruction plus eight renders takes 114 milliseconds, versus LaRa’s 146 milliseconds. That is a useful systems result. It is not, by itself, a camera-frame-rate measurement or a demonstration of onboard real-time perception. Consider a hypothetical pipeline that reconstructs an object in 120 milliseconds and then renders each requested view in one millisecond. Ten views would cost about 130 milliseconds altogether. Reporting the average time per output image would obscure the latency before a newly observed object becomes available. Conversely, reporting only reconstruction latency could hide the advantage of reusing one explicit representation for many subsequent views. The right measurement depends on the application. A digital-twin service may value reconstruction amortized over hundreds of renders. A wrist-camera perception loop may care about the delay between receiving new evidence and updating the object estimate. There is also a straightforward Amdahl’s-law issue. Suppose eighty percent of runtime occurs before selective refinement. Even making the remaining twenty percent ten times faster reduces total runtime only to eighty-two percent of its original value—a speedup of about 1.22 times. For deployment, I would therefore profile image encoding, volumetric processing, selection and indexing, fine decoding, and rasterization separately. Then I would include the surrounding stack: image transfer, object extraction, camera estimation, coordinate conversion, and any representation fusion. Mean latency is only the beginning. Variable selection counts could produce variable memory use and tail latency. A useful benchmark would show how runtime changes with object complexity, view count, image resolution, and the amount of selected space. The deployment question is not “Is the renderer fast?” It is “How quickly and predictably can the entire system incorporate new evidence?”. Where this belongs in an embodied stack. My first integration target would be an object-centric appearance model, not a replacement for a robot’s geometric map. There is an instructive contrast with SplaTAM. SplaTAM explicitly addresses online tracking and mapping from an unposed RGB-D stream, including estimating camera motion and expanding a Gaussian map as new areas are observed. Those responsibilities are part of the task definition, rather than consequences automatically supplied by choosing Gaussians as the representation. A practical integration proposal would keep those responsibilities separate. The robot’s existing state-estimation machinery would provide camera poses and an object frame. A reconstruction module would build a renderable object representation from selected observations. A separate geometric layer would remain responsible for collision queries, free-space reasoning, and metric uncertainty. The normalization boundary deserves particular attention. The released inference configuration specifies an axis-aligned box spanning minus 0.5 to plus 0.5 on each axis. For an application, I would explicitly record the transform between that reconstruction frame and the robot’s metric frame. What object scale was used? Where is the origin? Which axis convention applies? How are camera translations transformed? These are mundane questions, but a visually convincing reconstruction in the wrong coordinate system is not a useful robotic estimate. The distinction between a renderable object and a usable surface is equally important. SuGaR addresses mesh extraction by adding surface-alignment regularization and exploiting that alignment during reconstruction. Its existence is a reminder that obtaining an explicit mesh from splats is a substantive modeling step, not merely changing the output file extension. There is also a temporal question that I would test before any streaming integration. Suppose one observation set produces a selection score of 0.49 at a location and the next produces 0.51. A hard allocation boundary can change the local representation even when the evidence changes only slightly. That is a hypothetical failure mechanism, not a reported VS-Splat result. The appropriate test would use nearby camera trajectories and track selected locations, rendered appearance, and reconstructed surfaces over time. Possible remedies—persistent spatial identities, hysteresis, or temporal consistency losses—would be extensions to evaluate, not assumed properties of the model. Active perception offers another promising but unproven connection. A learned allocation map might help identify where the model spends representational effort. It should not automatically be treated as a map of uncertainty or expected information gain. For that purpose, I would train and evaluate a separate quantity: how much would a feasible additional observation improve a task-relevant estimate? High reconstruction importance and high value of another observation are different ideas. A richly textured visible surface might deserve many primitives but little additional sensing. A poorly observed grasp clearance might deserve the next camera motion despite contributing almost nothing to current rendering loss. That distinction is where a reconstruction system begins to become an embodied perception system. Reproduction: inspect the release before investing in training. As checked on September 20, 2026, the authors provide a public implementation in. Its README describes training and evaluation entry points, but the inspected Weights section contains no download entry. I verified the materials, not execution of the model. The public training configuration specifies four views, 512-by-512 images, fifty epochs, and both coarse and fine stages. The README recommends four 80 GB A100 GPUs with batch size sixteen for reproduction. There are details worth auditing before launching a run. The training entry point enables mixed bfloat16 precision and gradient accumulation over two batches. Combined with four devices and a configured per-device batch size of four, that would ordinarily imply an effective optimizer batch of thirty-two. I would resolve that against the intended recipe rather than silently equating batch size, microbatch size, and effective batch size. I would also check setup instructions against the repository tree: the README’s environment filename differs from the checked-in filename. This is a reason to inspect the release carefully, not evidence that the method itself fails. The evaluation implementation constructs both VGG- and AlexNet-based LPIPS evaluators. For reproduction, the specific variant should be recorded explicitly; a metric name alone is not a complete experimental specification. My reproduction plan would proceed in three stages. First, establish a frozen inference artifact. Pin a commit, resolve the checkpoint source, record preprocessing and camera conventions, and reproduce a small set of known examples. Do not begin with custom robot data and several simultaneous architecture modifications. Second, reproduce allocation behavior. Log selected-cell counts and inspect where fine capacity is spent. Construct a small diagnostic collection containing thin handles, holes, repeated textures, weak texture, and partial occlusion. Include cases where the visually important region differs from the geometrically important region. Third, evaluate the intended downstream use. If the goal is view synthesis, test held-out viewpoints and trajectory consistency. If the goal is manipulation, test metric geometry and action-relevant errors. If the goal is embedded deployment, test the actual hardware and include the upstream processing budget. I would keep a dense or simple-threshold baseline throughout. A sophisticated learned selector should earn its complexity under a controlled budget, not merely outperform a differently configured system. The decisive artifact would be a quality-versus-latency curve accompanied by failure cases and geometric checks—not one attractive reconstruction and one average timing number. The takeaway: learn where detail is worth computing. The idea I would carry back to the lab is not that voxels or Gaussians are inherently the right representation for embodied intelligence. It is that representational allocation can be part of the learned prediction problem. VS-Splat connects that allocation to image reconstruction, directing detailed Gaussian prediction toward selected spatial regions without relying on explicit structural labels. The broader engineering lesson is to ask what happens before and after the selection decision. Before it, does the model have enough evidence to recognize important structure? During training, does the decision receive a useful signal? Afterward, can the selected branch actually recover detail that the coarse representation could not?. For robotics, add two more questions: is that detail geometrically trustworthy, and does it improve a decision under the system’s real latency budget?. Those questions keep the promise and the evidence aligned. Selective computation is compelling when it makes limited resources more useful. The next step is demonstrating that the regions worth reconstructing beautifully are also the regions worth understanding accurately enough to act.