Latest / Embodied AI 101 / Gaussians as State, Futures as Supervision: Inside GSP-3D

Transcript

Gaussians as State, Futures as Supervision: Inside GSP-3D · Embodied AI 101

• Gaussians as State, Futures as Supervision: Inside GSP-3D. GSP-3D: generalizable 3D Gaussian Splatting with diffusion policy and world model for long-horizon bimanual manipulation • Xukun Liu, Junhua Huang, Shenggang Wei, Liguo Hou, Junmou Qiao, Aifeng Liu, Hongsheng Huang, Fengjuan Xie • Northwest Institute of Mechanical and Electrical Engineering; Jiangxi Guoke Defence Group Co., Ltd. • Frontiers in Neurorobotics, Volume 20 • 2026. The central idea. Consider a bimanual handover. One arm must present an object at the right pose, the second arm must enter the shared workspace without colliding, both grippers must coordinate their timing, and the policy must remain stable as the object becomes partially occluded. A single visually plausible action is not enough. The entire short sequence has to produce a plausible future scene. That is the motivation behind GSP-3D, published on August 31, 2026. The paper combines three ideas that have each become influential in robot learning: diffusion-based action generation, explicit 3D Gaussian scene representations, and learned world models. Its thesis is that bimanual policies will generalize better if they are conditioned on richer 3D primitives than ordinary point clouds, and will behave more coherently if their action predictions are constrained by a model of future scene evolution. The cleanest mental model is DP3 plus two additions. First, the sparse point cloud used by 3D Diffusion Policy is replaced by attributed, anisotropic Gaussian primitives. Second, a training-only dynamics model predicts how those primitives should evolve under an action chunk. At deployment, the world model is discarded; only the observation encoder and diffusion policy remain. That distinction is important. Despite the “world model” language, GSP-3D is not doing model-predictive control, online imagination, or search over Gaussian rollouts. Its world model acts more like a learned differentiable teacher: it supplies an auxiliary training signal intended to shape the action distribution toward scene transitions resembling those in successful demonstrations. Why bimanual manipulation stresses policy representations. Action chunking already addresses one part of the bimanual problem. Instead of predicting one motor command at a time, methods such as ACT and Diffusion Policy generate a short sequence, reducing high-frequency jitter and helping the policy express coordinated phases of motion. ACT established this strategy on real ALOHA tasks, while diffusion policies offered a flexible way to model multiple valid trajectories rather than collapsing them into a single averaged behavior. But action generation is only as good as its conditioning state. RGB encoders may recognize objects while remaining surprisingly weak at metric reasoning. Depth and point clouds improve spatial grounding, which is why DP3’s deliberately simple point-cloud encoder performed so well across simulated and real manipulation tasks. DP3’s result was not that robot perception needs a gigantic 3D backbone. It was that even a compact, colorless point cloud can give diffusion policies useful invariance to viewpoint, appearance, and object instance. GSP-3D asks whether that representation can be enriched without making the controller prohibitively expensive. A point records a location and perhaps color or a learned feature. A 3D Gaussian records a location but also an anisotropic extent and orientation, opacity, and appearance. Geometrically, we can picture each primitive as a soft ellipsoid rather than an infinitesimal sample. A collection of these ellipsoids can express local surface orientation, thickness, uncertainty, and occupancy in ways that a similarly sized set of bare points cannot. The proposed advantage is especially relevant when two robot arms occupy the same scene. Their relative positions, the volume swept by a gripper, and the orientation of an object surface all matter. A representation that says “there is mass distributed along this direction” could, in principle, be more useful than one saying only “there is a sample here.”. The second challenge is temporal. Standard behavior cloning rewards the policy for matching expert actions, but does not directly ask whether those actions imply sensible future observations. GSP-3D therefore trains another network to predict future Gaussian states. If an action would make the scene evolve in a way inconsistent with the demonstrations, the resulting prediction error should push the policy away from that action. This is an attractive decomposition: use explicit geometry for spatial reasoning and predictive supervision for temporal reasoning. The key question is whether the implementation and evidence fully support that story. The GSP-3D pipeline. GSP-3D is best understood as a staged system rather than one monolithic network. An RGB-D observation first passes through the Generalizable Gaussian Regressor, or GGR. The resulting Gaussian set is compressed by a multiresolution voxel encoder and cross-attention module. That compact code, together with robot proprioception and a task embedding, conditions a denoising U-Net that generates a chunk of joint commands. Separately, a Gaussian world model is trained on demonstration trajectories. It receives recent Gaussian states and an action chunk, and predicts future Gaussian states. The world model is then frozen. During policy training, its future-state error is added to the ordinary diffusion denoising objective. At test time, the world model and future ground truth disappear entirely. Although the paper describes the framework as end-to-end trainable, its implementation is operationally staged. GGR is trained with rendering supervision, the Gaussian world model is trained separately and frozen, and the diffusion policy is subsequently optimized using those components. That is not a defect—modular training may be considerably more stable—but it matters for anyone attempting to reproduce the system. From one RGB-D frame to a Gaussian scene. GGR begins with a U-Net-style RGB-D encoder. Camera intrinsics and the depth image are then used to back-project image pixels into 3D. A lightweight point decoder takes the resulting positions and visual features and predicts the remaining attributes of each Gaussian: scale, rotation, opacity, and color. Covariance is represented through a scale and quaternion, ensuring that each primitive behaves as a rotated ellipsoid rather than an unconstrained matrix. The regressor is supervised through differentiable rendering. Its predicted Gaussians are projected back into the camera, alpha-composited, and compared with the target RGB and depth images. In effect, GGR must generate a set of blobs that can re-render the observed scene. Color jitter, blur, synthetic depth noise, and domain-randomized scenes are used to improve robustness, while median filtering is applied to depth measurements. This construction eliminates the scene-by-scene optimization normally associated with Gaussian splatting. Conventional 3DGS optimizes a Gaussian cloud for a particular captured scene. GGR amortizes that process: after training, one forward pass predicts a new Gaussian representation for each observation. Once the Gaussians have been predicted, however, directly passing hundreds or thousands of primitive tokens to a diffusion U-Net would be expensive. GSP-3D therefore partitions 3D space into voxels at multiple resolutions. An MLP maps each primitive’s attributes into a feature, and a pooling operation combines the Gaussians that fall into the same voxel. The multiscale voxel features are flattened and consumed by a transformer with learnable queries, producing a fixed-size latent code. Conceptually, this is less a renderer than an attributed point-set encoder. The Gaussian renderer is critical while learning the primitives, but the control policy ultimately consumes their centers, covariances, opacities, colors, and task-related attributes through sparse spatial aggregation. The Gaussian-conditioned diffusion policy. The policy follows the familiar action-diffusion recipe. During training, noise is added to an expert action chunk. The denoising network receives that corrupted chunk, the diffusion timestep, the Gaussian scene code, and the robot’s proprioceptive state. It learns to estimate the injected noise, allowing the reverse process to transform random noise into a coherent action sequence during inference. Actions are fourteen-dimensional joint-position targets: seven dimensions for each arm. The model observes three timesteps and predicts the next six. Proprioception is concatenated with the noisy action sequence, while the Gaussian code is introduced through cross-attention layers inside the denoising U-Net. The paper also introduces task and affordance information. RoboTwin objects have annotated candidate grasp poses covering different directions and grasp axes. These affordances are described as additional Gaussian attributes. A learnable task embedding tells the policy whether it is expected to lift, hand over, stack, insert, or perform another coordination pattern. In principle, this division is sensible. Geometry can remain broadly task-agnostic, while a compact task condition changes how the two arms use that geometry. A handover and an insertion may involve the same object and scene but demand different arm roles, approach directions, and timing. The Gaussian world model. The GS-WM operates directly on Gaussian primitives. Each Gaussian’s position, covariance, opacity, and color are projected into a token embedding. A causal transformer consumes recent Gaussian sets and the associated action chunk, then predicts residual changes to Gaussian parameters over future timesteps. Opacity is handled through a bounded transformation, while covariance and position are trained with geometric reconstruction losses. Training targets are produced offline. GGR is applied to the later frames in each expert trajectory, giving the system “ground-truth” future Gaussian sets without requiring manually labeled 3D states. GS-WM is trained to reproduce those states, frozen, and then inserted into policy training. The intended gradient story is straightforward. Suppose the policy proposes an action chunk slightly different from the demonstrated one. The frozen world model predicts the scene resulting from that chunk. If the prediction diverges from the demonstrated future scene, the auxiliary error penalizes the policy. Over training, the denoiser should prefer action chunks whose predicted consequences follow the temporal regularities of successful demonstrations. That makes GS-WM a representation-space consistency model, not a simulator. The authors explicitly acknowledge that 3DGS is fundamentally a radiance-field representation and that GS-WM does not enforce contact, collision, conservation laws, or rigid-body dynamics. It learns empirical regularities such as continuous object motion and smoothly changing occlusion from demonstrations. Its removal at inference is a practical advantage. The policy does not need to roll out a transformer world model every control cycle. It is also a scientific limitation: the system cannot use the model online to detect failure, compare alternative plans, or replan around unexpected dynamics. The RoboTwin 2.0 evaluation. RoboTwin 2.0 is an appropriate test bed for this hypothesis. The broader platform contains 50 dual-arm tasks, five robot embodiments, and a RoboTwin-OD library with 731 objects in 147 categories. Its data generator randomizes clutter, illumination, background, tabletop height, and language, making it useful for studying whether policies learn geometry rather than brittle image correlations. GSP-3D uses an Aloha-AgileX configuration with 640-by-480 RGB-D observations. Simulation runs internally at 250 hertz, but observations and actions are logged every fifteen steps, giving an effective demonstration rate of roughly 16.7 hertz. A trajectory contains approximately 100 to 250 logged timesteps, or seven to fifteen seconds of manipulation. Fifty expert demonstrations are used per task. The main table covers eight tasks: clicking an alarm clock, handing over a block, moving a can and pot, placing an empty cup, rotating a QR code, scanning an object, placing a fan, and pressing a stapler. The comparison includes RDT, Pi0, ACT, image-conditioned Diffusion Policy, and point-cloud-conditioned DP3. Every method receives the same demonstrations and predicts the same fourteen-dimensional joint targets. Evaluation uses 100 trials per task. The paper separately compares GSP-3D and DP3 under clean and randomized conditions, although randomized results are presented primarily as a bar figure rather than a complete task-by-perturbation numerical table. Training curves and ablations are averaged over three random seeds. Experiments were conducted on a laptop with an RTX 4060 GPU. What the results say. Across the eight main tasks, GSP-3D reports an average success rate of 69.0 percent, compared with 61.25 percent for DP3—a gain of 7.75 percentage points. The improvement is consistent rather than being dominated by one task: GSP-3D beats DP3 by six to nine points on every entry. Its strongest results are 84 percent for the alarm-clock task and 82 percent for rotating the QR code. The absolute scores are also informative. Scanning an object reaches only 40 percent, and placing the fan reaches 42 percent. GSP-3D therefore improves the baseline without solving the benchmark. Those harder tasks may involve tighter perception-action coupling, occlusion, or precise interaction phases that remain difficult even with a Gaussian representation. The ablations are unusually tidy. Removing both 3DGS and the world model exactly reproduces the DP3 row, with a 61.25 percent average. Retaining the Gaussian representation but removing the world model yields 65.125 percent. Retaining the world-model variant while removing 3DGS yields 62.5 percent. The full system reaches 69 percent. Read literally, the Gaussian representation contributes the larger share, while the temporal auxiliary objective provides another roughly four points when added on top. Computational overhead is modest in parameter count and memory. GSP-3D has about 265.7 million parameters versus DP3’s 262.4 million, and peak memory rises from 2.356 to 2.384 gigabytes. That is approximately a 1.2 percent increase in each case. Average inference time, however, rises from 1.3981 to 1.5214 seconds—closer to a nine-percent latency increase. The cleanest supported conclusion is therefore not that Gaussian splatting transforms long-horizon control. It is that, under this simulation protocol, richer Gaussian attributes and short-horizon predictive supervision consistently improve a strong point-cloud diffusion baseline without substantially increasing model size or GPU memory. A critical reading of the evidence. A Gaussian cloud is not automatically a complete 3D scene. The paper repeatedly contrasts sparse point clouds with “structurally complete” Gaussian geometry. That characterization deserves restraint. GGR begins by back-projecting a single depth image and is trained to reconstruct RGB and depth from the same observed viewpoint. The Gaussian covariance can provide useful local surface extent and orientation, making the result richer than raw points. But this process does not inherently recover the backs of objects, geometry behind occluders, or surfaces never observed by the camera. Same-view reconstruction can be satisfied by a soft surface shell close to the measured depth. The manuscript also says the rendering loss enforces multiview photometric and geometric consistency, yet the described GGR sample contains one RGB-D input and a target from the same viewpoint. No multiview target, camera-baseline experiment, or novel-view reconstruction metric is specified. Consequently, the safest interpretation is that GGR produces an attributed representation of the visible scene, not a complete reconstruction. A decisive experiment would compare centers only against centers plus scale and rotation, then add opacity and color. That would reveal whether the control gain comes from genuine Gaussian geometry, the larger feature vector, the sparse voxel-transformer encoder, or simply additional network capacity. The world model is short-horizon and not physically grounded. At the effective logging frequency, six predicted actions span about 0.36 seconds. The three-frame observation history covers roughly 0.18 seconds. The full tasks last seven to fifteen seconds, so “long-horizon” refers to closed-loop repetition of short action chunks, not to a world model that imagines an entire manipulation episode. That is a normal receding-horizon design, but the terminology matters. GS-WM is not tested on long autoregressive rollouts, and the paper does not report prediction accuracy as a function of horizon. Nor does it measure the properties it claims the world model improves: action jerk, inter-arm timing error, self-collision rate, object-motion continuity, or frequency of physically impossible transitions. The evidence for temporal coherence comes indirectly from task success and the world-model ablation. There is also an underspecified set-correspondence problem. The paper defines a Gaussian state as a variable-sized collection and predicts residual updates to individual primitives. Yet independently generated Gaussian sets are permutation-invariant, and pixels can correspond to different surfaces as objects move or become occluded. The manuscript does not describe Gaussian tracking, matching, canonical ordering, Hungarian assignment, padding, or a permutation-invariant set distance. Without one of those mechanisms, a direct parameterwise loss between predicted and future Gaussian sets is difficult to define robustly. The gradient path into the diffusion policy is similarly important. For the auxiliary error to train the policy, GS-WM must consume a differentiable estimate of the policy’s clean action chunk—not merely the expert action. Standard diffusion training, however, observes a randomly noised chunk and predicts noise. The paper does not fully explain how it converts that intermediate denoising prediction into the clean action passed to GS-WM. That missing detail could materially affect stability and computational cost. “Real time” is not demonstrated for the complete controller. GGR is described as enabling real-time Gaussian reconstruction, but the paper does not report its latency independently. The full policy’s stated average inference time is 1.5214 seconds. If that number denotes one policy call, as the table heading suggests, it corresponds to approximately 0.66 hertz and is more than four times the 0.36-second duration of a six-action chunk at the dataset rate. The simulator might execute buffered chunks asynchronously, or the reported timing might include setup and evaluation overhead rather than steady-state control. But those possibilities are not documented. Until the timing protocol is clarified, “minimal computational overhead” is justified for memory and parameter count, while “real-time deployment” remains unproven for the end-to-end policy. Generalization is still simulation generalization. RoboTwin’s randomization is valuable, and GSP-3D reportedly remains ahead of DP3 under altered lighting, backgrounds, and tabletop clutter. But there is no real-robot evaluation or sim-to-real transfer experiment. The authors appropriately list unvalidated sim-to-real transfer, reflective surfaces, extreme lighting, and occlusion among their limitations. The randomized evaluation would also be stronger with exact per-task values, confidence intervals, and isolated perturbation tests. At present, lighting, clutter, background, and distractors are grouped together. We cannot determine whether the gain comes from geometric invariance, training augmentation, depth robustness, or one particularly favorable randomization regime. Affordance supervision may be a hidden confound. Section 3.4 says RoboTwin-OD’s annotated candidate grasp poses are incorporated as Gaussian attributes. It is unclear whether GGR predicts those affordances from RGB-D, whether they are retrieved from simulator annotations, or whether they are directly supplied to the policy. The baselines are described as receiving the same observation-action pairs, but no equivalent affordance channel is mentioned. That distinction is crucial. Predicted affordances would be a legitimate learned component. Ground-truth simulator affordances would be privileged information and would make the comparison not merely cross-modal but cross-information. Similarly, the learnable task embedding is not clearly tied to the language instruction, leaving open whether the policy supports language variation or simply uses a closed-set task identifier. Reproduction will require filling substantial gaps. The paper provides learning rate, batch size, several U-Net dimensions, observation and prediction lengths, and simulator settings. It does not give the auxiliary-loss weight, despite identifying sensitivity to that weight as a limitation. Also absent are the Gaussian token budget, voxel resolutions, transformer depth and head count, set-matching method, diffusion sampling steps, detailed training schedules, and an unambiguous statement about joint multitask versus per-task training. As of the August 31, 2026 publication, the article contains no code or project-page link. Its data-availability statement says raw supporting data will be made available by the authors, rather than pointing to a released artifact. Given the number of implementation-sensitive components, an open repository will be necessary for independent validation. The closely related work changes the novelty story. GSP-3D is presented as opening largely unexplored territory at the intersection of Gaussian splatting, policy learning, and world models. The broader literature makes the defensible contribution narrower—but still potentially useful. GSRL introduced a generalizable Gaussian-splatting representation for reinforcement learning in 2024, using amortized Gaussian prediction rather than scene-specific optimization. That work already argued that Gaussian primitives can serve as interpretable, geometry-aware policy state. More importantly, ManiGaussian, published at ECCV 2024, combined dynamic Gaussian splatting with a Gaussian world model and future-scene reconstruction for multitask robotic manipulation. Its world model supplied scene-dynamics supervision in Gaussian space, closely matching GSP-3D’s high-level motivation. DP4, published at ICCV 2025, is closer still. It uses the exact term Generalizable Gaussian Regressor, constructs a current Gaussian scene from single-view RGB-D, predicts future Gaussian scenes conditioned on policy-generated trajectories, and uses current- and future-scene reconstruction as training-only supervision for a diffusion policy. DP4 reports experiments across Adroit, DexArt, RLBench, and three real-world tasks. Neither ManiGaussian nor DP4 appears in GSP-3D’s published reference list. That is a significant omission because those papers predate GSP-3D’s April 28, 2026 submission and overlap with its central combination of Gaussian representation, future-scene prediction, and policy supervision. This does not make the systems identical, but it means GSP-3D should not be read as the first work to combine those ideas. The architectural differences are worth preserving. DP4 conditions its policy mainly on local and global point-cloud representations and uses its Gaussian model for supervision. GSP-3D feeds a compressed Gaussian latent directly into the diffusion policy, adopts sparse multiresolution voxel aggregation and cross-attention, uses a transformer dynamics predictor, and targets RoboTwin bimanual tasks. Its contribution is therefore best framed as a particular Gaussian-conditioned diffusion architecture and RoboTwin evaluation, rather than the invention of Gaussian world-model regularization. Several other approaches clarify possible next steps. KStar Diffuser incorporates the physical bimanual kinematic graph and differentiable forward kinematics into diffusion training. Where GSP-3D regularizes visual scene evolution, KStar regularizes robot structure, reachability, and collision-related behavior. The two ideas are complementary. Action-Geometry Prediction, published at CVPR 2026, jointly predicts an action chunk and a future dense 3D latent derived from a pretrained geometric foundation model. It uses RGB rather than requiring depth and evaluates in both RoboTwin simulation and real-world executions. This is a particularly relevant comparison for the claim that forecasting future geometry improves bimanual coordination. DREMA takes a more physically explicit route. It constructs object-centric Gaussian scenes, embeds them in a physics simulator, and uses the resulting digital twin to transform and validate demonstrations. Unlike GSP-3D’s statistical regularizer, DREMA can execute policies in an imagined, physics-constrained environment and synthesize new training data. Together, these works reveal a useful design spectrum. At one end are lightweight training-only representation losses. At the other are object-centric, physics-enabled worlds suitable for planning and data generation. GSP-3D sits closer to the lightweight end. How a robotics team might adapt the idea. The most sensible replication strategy would begin with DP3 rather than implementing every GSP-3D component at once. Establish a point-cloud diffusion baseline, cache geometric observations offline, and verify the eight-task evaluation protocol. Then replace the point encoder with a Gaussian encoder while keeping the denoiser, data, action normalization, and random seeds fixed. The first ablation should test which Gaussian attributes actually matter. Compare centers alone; centers with scale and rotation; centers with opacity; and the complete colored representation. If scale and rotation provide most of the gain, the result supports the geometric hypothesis. If color or a larger transformer accounts for it, full differentiable splatting may not be necessary. For the world model, canonicalization is essential. An object-centric Gaussian representation, persistent primitive IDs, scene-flow-based correspondence, or a permutation-invariant matching loss would make future prediction more defensible. A cheaper alternative is to predict a fixed sparse voxel latent or rendered future depth map rather than raw Gaussian identities. That sacrifices some explicitness but avoids unstable token matching. Bimanual safety should be regularized directly. Forward-kinematic consistency, joint-limit penalties, signed-distance self-collision losses, and gripper-object contact constraints would complement the visual world-model error. A Gaussian scene can tell the controller where surfaces appear to be, but it cannot by itself guarantee that two high-dimensional manipulators follow reachable and collision-free trajectories. For real hardware, wrist cameras or multiview fusion would address the largest weakness of the single-view GGR. Reflective tools, transparent containers, thin cables, and severe hand-object occlusion are exactly where raw RGB-D back-projection becomes unreliable. Uncertainty-aware opacity and covariance could help, but only if the policy is trained to distinguish perceptual uncertainty from physical extent. Finally, latency must be treated as a first-class objective. Practical options include pruning low-opacity Gaussians, using object-centric tokens, distilling the diffusion policy to fewer denoising steps, or executing chunks asynchronously while the next prediction is computed. If the world model is ever moved into the test-time loop, it will need an even more aggressive latent representation. The bottom line. GSP-3D reports a consistent and meaningful improvement over DP3 on its selected RoboTwin tasks. Its strongest idea is not Gaussian rendering by itself, but the use of explicit geometric attributes as policy state, paired with future-state prediction as a training regularizer. The ablations support both components, and the memory overhead is small. The evidence is nevertheless narrower than the title suggests. The system is simulation-only, predicts less than half a second into the future, does not directly measure temporal coherence, leaves Gaussian correspondence and the policy-to-world-model gradient path unclear, and lacks a public implementation. Its novelty also needs to be read alongside ManiGaussian and especially DP4. The durable research direction is compelling: action policies should not merely imitate what experts commanded; they should learn whether those commands imply sensible geometric futures. GSP-3D is one more indication that future-scene supervision can improve visuomotor control. The next convincing step is a correspondence-aware, physically regularized, real-time version evaluated on real bimanual hardware.