Latest / Embodied AI 101 / Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization
Transcript
- Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization. Modern 3D reconstruction methods often treat every small piece of geometry as a separate “primitive” – a point, voxel, or Gaussian. But humans think of scenes as collections of objects, not raw primitives. Yoo et al. take this insight to heart. They build a feed-forward model that directly parses a scene into object-centric “token groups,” without relying on 3D supervision. In their framework, a scene is not a bag of points but a compact set of object instances, each represented by one instance token (encoding the object’s identity and extent) and several anchor tokens (encoding local geometry and appearance). These two levels of tokens are then decoded into standard 3D primitives (ellipsoidal Gaussians) for rendering. Crucially, this factorization “decouples object identity from local appearance, making object instances a native interface of the representation rather than a derived product”. In practical terms, the network is trained end-to-end (via differentiable rendering) to both reconstruct the input images and align the tokens with object masks, all without any 3D ground-truth. The result is a single forward pass from a set of (unposed) images to a collection of grouped 3D Gaussians grouped by object, unlocking direct instance-level reconstruction, segmentation, and even scene editing. Motivation: Objects Instead of Primitives. The motivation is simple: to enable object-level understanding and manipulation of 3D scenes, the scene representation itself should be object-centered. Existing feed-forward 3D reconstruction pipelines (NeRFs, point cloud nets, Gaussian splatting models, etc.) typically output a large unstructured cloud of points or Gaussians. Even if one later attaches semantic features to each primitive (by lifting 2D features), the representation still lacks persistent object handles. In such systems, any notion of “this is all one sofa” emerges only by grouping many primitives in post-processing, which is brittle and inefficient. Yoo et al. argue that this is a “representation mismatch”. The raw geometric primitive – say a small Gaussian blob – has no memory of its neighbors, so the model never learns what objects are internally or how they persist. To fix this, they re-architect the output: token groups become the fundamental units. In their words, “a 3D scene is understood through its objects, not the primitives” composing them. Instead of outputting 100K Gaussians with separate semantics, their network emits a small number of object tokens (one per object) each paired with a handful of “anchor” tokens inside it. This makes objects “first-class units” in the representation. Once this object-centric decomposition is learned, the same tokens can be used for rendering, masking, editing, and retrieval, without any separate post-processing. In short, object instances become native interfaces of the scene rather than hacky afterthoughts. Method. Overview. The input to the model is a collection of unposed RGB images (for example, frames of a ScanNet room with unknown camera poses). A frozen 3D "foundation" network (e.g. a multi-view geometry model like “Dust3r” or another reconstruction backbone) first processes each image independently. From each view it extracts a feature map and a point map (a rough depth or 3D coordinate for each pixel). These per-view outputs are merged into a set of multi-view context tokens. Essentially, every image patch contributes a context vector carrying appearance and initial geometry. From these context tokens, the novel part of the model constructs a fixed number of anchor tokens and group tokens. Anchors are intended to represent local parts in 3D space; group tokens will represent whole objects. The pipeline then refines these tokens together via Transformer attention layers and finally decodes the anchors into Gaussians. Figure 2 of the paper (and the project page) illustrates this three-stage process. Anchor and Group Tokens. First, a set of $K$ anchor tokens is initialized from the context tokens. Concretely, the model performs farthest point sampling on the pooled 3D coordinates (from the stacked point maps) to pick $K$ distinct locations $a_k$ in space. For each chosen anchor location, we take the context feature $x_{a_k}$ corresponding to that patch and add a learned positional embedding $\phi_{\mathrm{pos}}(a_k)$ of the 3D coordinate. In equations: $$A_k^{(0)} = x_{a_k} + \phi_{\mathrm{pos}}(a_k).$$ This gives an initial anchor token $A_k^{(0)}$ that encodes local appearance and an approximate 3D position. At the same time, the model creates $L$ group tokens $G^{(0)}_\ell$ as learnable embeddings (one per potential object). The number $L$ is a design choice (roughly an upper bound on object count). With anchors and groups initialized, the network proceeds to jointly refine them. Cross-Attention Decoding. The core of the method is a two-step cross-attention decoding (see Figure 2 on the project page). Image-anchor decoder. In this step, each anchor token $A_k$ “reads” from the multi-view context so as to gather the detailed appearance and geometry from all images. Formally, the $K$ anchors query the large set of context tokens via a Transformer cross-attention layer. This produces updated decoded anchors $A_k = \mathcal{D}_{anchor}(A_k^{(0)}, X)$ grounded in the images (here $X$ denotes all context token features). In effect, each anchor now carries a summary feature of whichever object part it is in, informed by all views. Anchor-group decoder. Next, the group tokens similarly attend to the decoded anchors. Each group token $G_\ell$ queries the set ${A_k}$ via another cross-attention Transformer, producing updated group embeddings $G_\ell = \mathcal{D}{group}(G\ell^{(0)}, {A_k})$. After this step, each group token has pooled information from many anchors – ideally all anchors belonging to one object. Soft Assignment. How do anchors get assigned to groups (i.e. which anchors belong to which object)? The model uses a soft assignment achieved by dot-product similarity. After decoding, compute scores $$s_{k,\ell} = \langle A_k, G_\ell \rangle$$ between each anchor token and each group token. A softmax across $\ell$ yields a probability distribution $\pi_{k,\ell}$ for anchor $k$ over groups. In practice this makes group tokens compete for anchors: each anchor “belongs” mostly to the one group $\ell$ with the highest dot-product. This soft-competition is analogous to slot-based attention (like in Object-Centric models), but implemented here with Transformers. In short, each anchor token has a learned affinity to exactly one group token. Gaussian Reconstruction. Finally, each anchor token is decoded into a small cluster of 3D ellipsoidal Gaussians, which are standard renderable primitives. Concretely, a small MLP takes $A_k$ and produces $N_g$ Gaussians (e.g. $N_g = 8$) with parameters: center offset, covariance (scale and rotation), color (via spherical harmonics), and opacity. Each generated Gaussian inherits its parent anchor’s soft group-score $\pi_{k,\ell}$. Thus in the final scene, we often have something like $K \times N_g$ Gaussians scattered in 3D, but each Gaussian carries a probability of belonging to each object instance. In evaluation and rendering, one can assign each Gaussian to the group with highest probability (or use the soft probabilities directly to render instance masks). Altogether, this pipeline realizes a feed-forward mapping: multi-view images $\rightarrow$ anchor tokens + group tokens $\rightarrow$ grouped Gaussians. Because everything is done in one network pass (no per-scene optimization), inference is very fast once the model is trained. Training without 3D Supervision. The network is trained end-to-end using only 2D supervision. Two loss components guide it:. Reconstruction loss. The generated 3D Gaussians are rendered into novel views (via alpha-blended Gaussian splatting). The rendered RGB images are compared to held-out ground-truth images from those views using a standard photometric loss (MSE plus a perceptual LPIPS term). This ensures that the anchor tokens capture the scene’s geometry and appearance well enough to reconstruct images. Grouping (instance) loss. To make the group tokens align with real object instances, the authors turn the anchor-group assignments into a 2D instance segmentation problem. Here’s how it works: since each Gaussian has inherited its anchor’s assignment probabilities, one can render an “instance probability map” for each group $\ell$ by compositing the Gaussians using $\pi_{k,\ell}$ as their alpha weights. These probability maps can then be matched to the ground-truth 2D object masks in each view (ScanNet provides instance masks). During training, a Hungarian matching finds the best correspondence between predicted groups and true object masks. Then a pixel-wise binary cross-entropy plus Dice loss is applied to each matched pair. In effect, this encourages each anchor group to form a coherent object mask in the images. Notably, no 3D annotation is ever used. All supervision comes from the 2D images and their 2D masks across views. Still, the 3D tokens are encouraged to carve consistent objects out of space. The authors even apply a warm-up on the segmentation loss so that geometry emerges first and grouping influences the tokens only after initial object structure is formed. As a result, the network automatically learns to carve up the scene into objects during training. (For completeness, the model also supports a “semantic distillation” extension: the instance tokens serve as a natural basis to incorporate 2D semantic features. The authors briefly describe reusing the cross-attention to lift 2D vision-language embeddings into the group tokens, storing a shared semantic vector per object and a small residual per anchor. This yields coherent semantic labeling at inference and enables open-vocabulary queries. However, we focus here on the core geometry and instance grouping.). Results. Yoo et al. evaluate on the ScanNet dataset of indoor scenes. They compare to recent feed-forward 3D reconstruction methods and to per-scene optimized baselines. The key metrics are novel-view synthesis quality (PSNR/SSIM/LPIPS), semantic feature fidelity, and class-agnostic instance segmentation (average precision) in 3D. They also demonstrate qualitative scene editing and retrieval. Reconstruction and Feature Lifting. In Table 1 (project page), their model is competitive in novel-view image quality while using far fewer tokens. For instance, they compare to two previous feed-forward 3D Gaussian models (Uni3R and C3G) and to a semantic 3D model (LSM). Their method achieves a PSNR of ~25.3 dB on new views (almost the same as Uni3R’s 25.5 dB) and similar SSIM/LPIPS. Crucially, it does this with under 100 semantic units (instance tokens) instead of tens of thousands of primitives. Because each of its ~32k Gaussians is grouped, the model stores semantics in only ~66K parameters (for anchor+group embeddings) versus 8–67 million in the baselines. This 1000x compression still yields a higher semantic fidelity: the feature-lifting mIoU on both source and novel views is 0.661/0.657 for their model, substantially above ~0.54 for prior methods. In short, they achieve the best semantic correspondence while being vastly more compact. Qualitatively, Figure 3 on the project page shows an example ScanNet scene rendered from a new viewpoint. The ground truth and baseline reconstructions (dense Gaussians) are matched pretty well by the token-based approach. In other words, explicitly grouping Gaussians into objects did not noticeably harm rendering quality. As the authors note, the anchor tokens still preserve fine detail for image synthesis even while their parent token groups provide object identity. Instance Segmentation. Perhaps the most striking result is class-agnostic instance segmentation in 3D. Table 2 compares their feed-forward model against several prior approaches on AP metrics. Older methods like Gaussian Grouping and ObjectGS rely on per-scene optimization (e.g. running a separate grouping step after reconstruction). Another baseline (IGGT+LUDVIG) is a hybrid with some additional processing. In contrast, the new model produces instance masks in one shot. Impressively, it outperforms all baselines: their AP (all-threshold) is ~0.235, versus ~0.139 (Gaussian Grouping) or 0.178 (ObjectGS). At AP50 the gap is even larger (0.438 vs ~0.29 for Gaussian Grouping). In plain terms, their objects tokens yield more accurate segmentation than methods that painstakingly fit objects after the fact. This underscores the benefit of baking object structure directly into the network. Open-Vocabulary Retrieval and Editing. Another advantage of instance tokens is that high-level tasks become trivial. For example, each token group naturally has an associated feature vector. The authors show that if you derive semantic features from a vision-language model (e.g. CLIP or LSeg) and attach them to each token, you can perform text-based queries over objects. Because you have one feature per object, retrieving “the sofa” just means comparing one or two object vectors to the text embedding – a far smaller search than scanning millions of pixels or Gaussians. The project page even includes qualitative examples of open-vocabulary segmentation in a novel view (Figure 4) where each object is cleanly labeled by a text query. Scene editing is similarly simplified. Since each object lives in one token group, one can remove or move an object by editing that token. The authors provide an interactive demo: their rendered scene can be manipulated in the browser by toggling tokens on/off, which instantaneously deletes the corresponding object from view. You can also translate or insert an object by moving or copying its token before re-rendering. No additional mask or reprojection step is needed – the model’s output is already object-indexed. In short, token manipulation (as they call it) lets you treat digital 3D scenes like a set of LEGO blocks. Significance and Comparison. By making objects the unit of representation, this work bridges a gap between 3D reconstruction and higher-level scene understanding. It clarifies how one can leverage powerful 3D “foundation models” for geometry (e.g. Dust3r and related approaches) but go further by folding object-centric learning into the model rather than after. The result is a unified system: the 3D output can be used for rendering, like a NeRF; for segmentation, like a 3D mask processor; for manipulation, like a slot-based world model; and for retrieval, like a CLIP encoder over objects. In the recent landscape of 3D vision research, this stands out as one of the first to achieve object decomposition in a single feed-forward pass. Prior object-centric 3D methods (e.g. SlotLifter in radiance fields or GOCL for Gaussians) still required iterative fitting per scene. By contrast, Yoo et al.’s model generalizes across scenes: once trained on ScanNet it can parse new scenes without per-scene optimization. It also contrasts with dense semantic fields (like the NeurIPS 2024 LSM model) by explicitly binding tokens into objects. Compared to purely geometric feed-forward Gaussians (AnySplat, Uni3R, etc.), it adds a strong compositional prior that appears to improve downstream tasks. For robotics and embodied AI, this is promising. A robot tasked with understanding an environment would benefit greatly from having a 3D map broken into named objects. It could plan to pick up “the small brown cushion” by finding the cushion’s token, rather than clustering raw 3D points. The fact that the model can ingest uncalibrated images and still produce object groups means it could run on visual SLAM outputs or multi-view captures. Moreover, the ability to edit scenes by object tokens could enable simulation and planning: for example, moving furniture in the reconstructed scene to plan a navigation path, or inserting virtual objects. The open-vocabulary retrieval suggests integration with language instructions (“find the vase”) becomes more direct. In summary, Yoo et al. have shown that layering a slot-like, object-aware architecture on top of modern 3D Gaussian reconstruction yields a powerful hybrid. Their model “treats whole objects, not individual primitives, as first-class units” of the 3D scene. The gains in instance segmentation AP (more than 2× over old methods) and in semantic coherence (mIoU) speak to the effectiveness of this design. Perhaps most importantly, the same tokenized output can do all the usual tasks – novel-view synthesis, segmentation, instance editing, retrieval – without any extra specialized modules. Cited Works: The ideas here build on a rich literature of 3D reconstruction and object-centric learning. In particular, Gaussian Splatting (Kerbl et al. 2023) and its feed-forward descendants (Uni3R, C3G, pixelSplat, etc.) focused on geometry. Instance segmentation in 3D has seen techniques like Gaussian Grouping and ObjectGS, but these typically optimize per scene. Slot-based models (SlotLifter, GOCL) show the value of object reps but were often per-scene as well. Yoo et al. combine these strands: they take the feed-forward Gaussian idea and slot decomposition and merge them. The resulting feed-forward “instance-structured 3D token groups” is a novel representation that answers the call for explicit objects in 3D AI.