Latest / Embodied AI 101 / Make Geometry the Policy: GAM's Bet on 3D-Native Robot Control

Transcript

Make Geometry the Policy: GAM's Bet on 3D-Native Robot Control · Embodied AI 101

• Make Geometry the Policy: GAM’s Bet on 3D-Native Robot Control. Geometric Action Model for Robot Policy Learning • Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, and Sunghwan Hong • KAIST AI, ETH Zurich, and ETH AI Center • arXiv preprint • 2026. The core idea. Most recent generalist robot policies inherit their worldview from one of two places. Vision-language-action models inherit semantic abstractions from large vision-language models. Video world-action models inherit temporal priors from generative video models. Both are reasonable choices, but both begin with representations developed primarily for two-dimensional images. The Geometric Action Model, or GAM, makes a different bet: if the central challenge in manipulation is reasoning about surfaces, distances, occlusions, camera motion, and contact, then the policy backbone should be a model pretrained to reconstruct three-dimensional geometry. Rather than attaching geometric features to a conventional VLA, GAM turns the geometric foundation model itself into the policy. The architectural move is surprisingly direct. The authors split a pretrained geometric transformer at an intermediate layer. Its shallow blocks encode the current multi-camera observation. A newly inserted causal transformer predicts the latent representation of the next geometric state while also producing an action token. Both the predicted geometry and the action token then travel through the remaining pretrained geometric blocks. At the end, one head predicts an executable action chunk and the original geometry head can decode a future depth map. Perception, prediction, and control therefore share one geometric computation path. That shared path is the paper’s real contribution. The work is not simply saying that depth helps robotics, nor is it another instance of using a frozen 3D encoder. It asks whether the internal hierarchy of a geometric foundation model can become the substrate on which a robot predicts both what will happen and what it should do. Why the choice of backbone matters. A VLA can learn a great deal from a two-dimensional vision-language backbone. Large-scale image-text pretraining gives it object categories, attributes, linguistic concepts, and considerable commonsense knowledge. But contact-rich manipulation demands distinctions that are not always central to image-text learning: whether the gripper is in front of or behind an object, whether two surfaces are millimeters apart, how a camera displacement changes pixel coordinates without changing the underlying scene, or where an occluded object is likely to continue in three-dimensional space. A sufficiently large VLA can, in principle, recover such information implicitly. The issue is whether that is a good use of robot data and model capacity. If the visual representation is organized primarily around semantic discrimination, the action decoder must extract metric geometry from whatever monocular cues survive in that representation. Camera shifts are an especially revealing stress test because they alter the two-dimensional appearance of the whole scene while leaving the physical task essentially unchanged. Video world-action models address a related weakness by learning how scenes evolve. Cosmos Policy, for example, adapts a pretrained video diffusion model to generate actions as latent frames and can also produce future images and value estimates for planning. This provides a powerful temporal prior, but its native representational currency is still appearance over time. Depth, scale, and cross-view consistency remain properties that the video model must infer rather than the organizing principle of its backbone. Earlier geometry-aware VLAs occupy a middle ground. Spatial Forcing aligns intermediate VLA embeddings with representations from a pretrained 3D model, encouraging a conventional VLA to develop better spatial features without requiring depth sensors at deployment. That retains the semantic advantages of the VLA while using geometry as a teacher. GAM is more radical: it removes the VLM from the center of the architecture and makes the geometry model responsible for the policy’s core computation. This distinction is important. A frozen geometry encoder followed by an action MLP says, “geometry is useful input.” Feature alignment says, “the VLA should resemble a geometric model.” GAM says, “the geometric model should perform the action reasoning.”. The geometric substrate: Depth Anything 3. The implementation uses DA3-Giant—Depth Anything 3—fine-tuned on Track4World. DA3 is designed to recover spatially consistent geometry from an arbitrary number of input views, whether or not camera poses are supplied. It uses a comparatively plain transformer architecture and predicts a depth-and-ray representation from which scene geometry and cameras can be recovered. Track4World adds world-centric dense 3D tracking, so GAM’s actual initialization contains both strong multi-view geometry and tracking-oriented adaptation rather than merely monocular depth features. At a high level, DA3 divides each image into patches and adds a camera token for every view. Some transformer blocks process views independently, while global-attention blocks fuse information across cameras. A dense prediction head, based on the DPT design, taps features from several depths in the network and converts them into pixel-aligned geometry. GAM splits this 40-block ViT-Giant backbone after block 12, near the point where alternating cross-view attention begins. This location satisfies two requirements. The first 13 blocks have already transformed raw RGB patches into useful spatial features. At the same time, enough of the pretrained network remains after the split to turn forecasted intermediate tokens into coherent geometry. The original depth decoder also needs access to appropriate intermediate layers, so inserting the predictor too late would make future-depth decoding impossible or ineffective. This idea of finding a useful “seam” inside a foundation model is one of the most transferable lessons in the paper. GAM does not treat the backbone as an indivisible encoder. It interprets the early and late layers as performing meaningfully different functions: early layers construct a geometric state, while later layers propagate and refine that state into dense three-dimensional outputs. The new temporal model is inserted between those roles. Walking through one policy step. At each policy step, GAM receives two RGB images: an external view and a wrist-mounted view. It also receives a seven-dimensional proprioceptive state, the previously executed action, and a fixed natural-language instruction. The reported downstream policies use a single observed timestep and predict an eight-step action chunk in a seven-dimensional end-effector action space. Each chunk consists of Cartesian translation, axis-angle rotation, and a gripper command. The eight low-level commands are executed open-loop before the next observation is processed. The two images first pass through the frozen shallow DA3 blocks. The output is not an explicit point cloud. It is a collection of patch and camera tokens whose organization has been shaped by geometric pretraining. This is an important nuance: GAM does not require an RGB-D camera or construct a point cloud as a preprocessing stage. It receives RGB and carries geometry in its latent representation, with depth decoding used as both supervision and a diagnostic output. A frozen T5 encoder converts the instruction into language tokens. Lightweight projections convert proprioception and the previous action into tokens with compatible dimensions. The model concatenates the language tokens with one block per timestep containing robot-state, action-history, and geometric tokens. A 12-layer causal transformer with width 1,024 then processes the sequence using block-causal attention, allowing the current prediction to use language and all available past or present information without seeing future observations. The predictor produces two linked outputs. The hidden states in the geometric slots are interpreted as the latent representation of the next visual state. A designated action-history slot is transformed into a new action token, analogous to predicting the next token in a causal language model. Geometry and action are therefore generated by the same temporal computation rather than by two isolated modules. GAM then does something less obvious but central to the method. It copies the predicted action token once for each camera view, appends those copies to their corresponding future-geometry token sequences, and sends everything through DA3 blocks 13 through 39. Global-attention layers are given a causal mask so they do not introduce temporal leakage. The action token can now interact repeatedly with spatial features as the pretrained geometric hierarchy refines them. At the end, a lightweight action head aggregates the resulting action tokens and regresses the full eight-step action chunk. In parallel, the frozen DA3 depth head can decode the propagated geometric tokens into a future depth image. In other words, the model does not predict an action from the geometry encoder’s final pooled feature. It makes the action participate in the same deep feature propagation that produces three-dimensional structure. That detail turns out to matter empirically. In an ablation where the action loss is applied directly to the causal predictor’s output—bypassing the remaining DA3 blocks—success on the LIBERO-Plus Object suite falls from 89.7% to 84.1%. The late geometric blocks are not merely an expensive route to the action head; they appear to refine the action representation, with the largest benefit under environmental perturbations. What the model is actually learning. GAM combines three objectives. The action loss is ordinary behavioral-cloning regression: an L1 distance between the predicted action chunk and the expert action. A future-feature loss asks the causal predictor’s geometry tokens to match the tokens obtained by running the actual next observation through a frozen geometric encoder. Finally, a depth loss asks the propagated future tokens to decode into the next observation’s depth map, using scale-invariant and gradient-based penalties inherited from geometric depth estimation. The reported weights are three for action, one for feature prediction, and three for depth. The feature objective constrains the forecast in a rich latent space, while the depth objective prevents that space from drifting toward arbitrary features that happen to help action regression. Together, they encourage the predictor to forecast a state that remains legible to the pretrained geometry decoder. Depth labels come from different sources. Simulator depth is available for MimicGen, RoboCasa365, LIBERO, and related simulated data. Real robot data and Open X-Embodiment trajectories instead use pseudo-depth predicted by the geometric teacher. Real-world post-training therefore performs a form of geometric self-distillation: the model learns actions while preserving the future geometry implied by its pretrained perception system. The training recipe begins with 784,000 single-arm trajectories. Open X-Embodiment contributes 72% of the mixture, MimicGen 18%, and RoboCasa365 10%. The authors retain the original language annotations rather than generating synthetic instructions and map compatible datasets into a common seven-dimensional action interface. The public data documentation illustrates how much engineering hides behind that sentence: rotations appear as Euler residuals, axis-angle vectors, velocities, or partially missing dimensions; gripper conventions must be inverted or rescaled; and unsupported action dimensions are masked from the loss. The resulting model has roughly 1.405 billion parameters, of which about 983 million are trainable. DA3 blocks zero through 12 and the 50-million-parameter depth head remain frozen. Blocks 13 through 39 contribute roughly 765 million trainable parameters, the causal predictor contributes 210 million, and the action head adds another eight million. “Minimal architectural modification” should therefore not be confused with parameter-efficient adaptation: the design reuses a single backbone, but still updates close to a billion parameters. Pretraining is also compute-intensive. The paper reports approximately 96 hours on 64 NVIDIA GH200 GPUs for the first stage, followed by benchmark-specific post-training. GAM is lighter than many competing foundation-model policies at inference, but it is not a small-lab training recipe from scratch. Its practical accessibility depends heavily on the released initialization checkpoint. The simulation results: robustness is the real story. On the original LIBERO benchmark, GAM reaches 97.6% average success. That is competitive but not the highest reported number: Cosmos Policy reaches 98.5%, and several systems cluster around 97% or above. LIBERO is sufficiently saturated that small nominal differences are not especially informative. The more revealing evaluation is LIBERO-Plus, which applies controlled perturbations to camera position, robot initialization, language, lighting, backgrounds, image noise, and scene layout. Models are post-trained on ordinary LIBERO and transferred to these perturbations without additional training. GAM obtains 85.5% overall, compared with 84.6% for π0.5 and 82.4% for Cosmos Policy. Its drop from original LIBERO is 12.1 percentage points, the smallest among the reported systems. The headline result is the camera split. GAM reaches 83.1%, while the next-best baseline, Cosmos Policy, reaches 73.4%. π0.5 obtains 72.0%, OpenVLA-OFT 56.4%, and Fast-WAM 16.4%. As viewpoint perturbations increase from the easiest to the hardest LIBERO-Plus level, GAM reportedly remains ahead throughout the sweep. This is the evidence most directly aligned with the paper’s geometric thesis. The broader table deserves a careful reading, however. GAM is not best under every perturbation. π0.5 does better on robot-state and layout variation; Cosmos Policy is stronger on language and lighting; and π0.5 slightly leads on background changes. GAM’s overall advantage is only 0.9 percentage points over π0.5, whereas its camera advantage is 9.7 points. The strongest defensible conclusion is therefore specific: the geometric backbone substantially improves robustness to viewpoint changes, rather than universally solving out-of-distribution manipulation. On RoboCasa-Kitchen, GAM reaches 69.4% over 24 tasks, ahead of Cosmos Policy’s 67.1%. For this benchmark, the authors adapt the policy to three camera views and increase the action chunk from eight to 16 steps. Per-task performance remains uneven: articulated-object operations such as closing doors and drawers are often strong, while several object transfers and coffee setup tasks remain difficult. Real robots and a deliberately displaced camera. The physical evaluation uses a wrist-mounted ZED camera and an external RealSense camera. Four tasks range from straightforward cube pick-and-place to longer sequences: stacking two milk cartons and a cube, placing both a pot and a pan on an induction cooktop, and opening a pot, inserting a cube, and closing the lid. The dataset contains 284, 202, 184, and 169 demonstrations for the four tasks, respectively, and the tasks are trained jointly. Each task is evaluated in 20 trials: ten under the nominal camera configuration and ten after translating the external camera by 85 centimeters and rotating it by 45 degrees. GAM outperforms π0.5 and Spatial Forcing on every task in both conditions, with the largest qualitative separation under the displaced camera. This is encouraging, particularly because the perturbation is physically substantial. But it is still a compact evaluation: ten trials per task and condition, one robot setup, two camera configurations, and no reported confidence intervals. The experiment demonstrates that the simulation trend survives on hardware; it does not yet establish general camera equivariance across arbitrary sensors, calibrations, workspaces, or embodiments. The speed claim, with the important footnote. GAM’s single-pass action decoder avoids the iterative denoising used by diffusion and flow-matching policies. The headline latency is 6.9 milliseconds per model forward on one GH200 GPU, corresponding to roughly 145 forward passes per second. The reported alternatives require 29.2 milliseconds for π0.5, about 70 milliseconds for OpenVLA-OFT, and 382.4 milliseconds for Cosmos Policy. There is a deployment detail worth emphasizing. The 6.9-millisecond GAM result uses CUDA Graphs, whereas the main baseline numbers do not. Under the matched configuration where all systems use BF16 and Torch Compile but not CUDA Graphs, GAM takes 17.5 milliseconds. That is still about 1.7 times faster than π0.5, four times faster than OpenVLA-OFT, and almost 22 times faster than Cosmos Policy. CUDA Graphs then reduce GAM’s static execution path from 17.5 to 6.9 milliseconds, producing the advertised 55-fold maximum speedup. The architectural speed advantage is therefore real, but the largest multiplier combines architecture with a GAM-specific runtime optimization. The numbers are also model-only latency: they exclude image acquisition, preprocessing, robot communication, and control-system overhead. Moreover, GAM predicts eight commands that are executed open-loop, so 145 model forwards per second should not be interpreted as a demonstrated 145-hertz closed-loop robot controller. What the ablations reveal. The most consequential ablation is pretraining. On LIBERO-Plus Object, the full pretrained system reaches 89.7%. Removing the large robot-data pretraining stage while retaining both future-feature and depth losses reduces success to 73.4%. Removing pretraining and all predictive losses drops it to 50%. Much of GAM’s robustness therefore comes from the combination of geometric initialization and large-scale robot trajectory pretraining, not just the architecture used during benchmark fine-tuning. The predictive losses show a more nuanced pattern. Once the pretrained GAM initialization is present, removing the depth loss yields 89.0%, removing the feature loss yields 89.7%, and removing both yields 89.5%—almost unchanged from the full model. Without pretraining, however, those losses greatly improve robustness. My reading is that future geometry is particularly important for constructing the transferable initialization, while downstream action adaptation can rely on the geometric dynamics already embedded in the weights. Longer observation histories do not help. A single-step context reaches 89.7% on the Object perturbation suite, while histories of two and four steps reach 84.4% and 85.1%. The authors connect this to causal confusion in imitation learning: an extended history can let a policy exploit correlations that do not remain reliable under changed environments. The split layer is also critical. Layer 12 works well, layer 19 remains viable but weaker, and inserting the predictor at the input or near the end of the network causes performance to collapse. Forecasting raw patch embeddings is too difficult, while forecasting after most geometric processing has already occurred leaves insufficient pretrained computation to convert the predicted state into useful geometry and actions. GAM is thus not a generic recipe where a temporal transformer can be inserted anywhere; it depends on locating the functional boundary inside the backbone. Is GAM really a world model?. The paper appropriately describes GAM as a world-action model, but its meaning of “world model” is narrower than model-predictive control or counterfactual planning. The causal predictor forecasts the next geometric latent and next action under the demonstrated behavior distribution. It conditions on previous actions, not on a proposed set of alternative future action sequences, and the reported controller does not roll out multiple candidates and choose between them. The predicted geometry is nevertheless useful. It forces the policy to represent what the scene should look like after progress toward the instruction, and it couples that representation to action generation. But GAM is best understood as a behavior-conditioned, one-step latent forecaster used as a policy—not yet as a dynamics model that can answer, “What happens if I take action A rather than action B?”. That distinction separates it from Cosmos Policy’s optional planning mode, where generated futures and values can rank action trajectories. GAM gives up test-time search and multimodal diffusion in exchange for fast deterministic inference. For reactive manipulation and deployment-constrained systems, that may be exactly the right trade. For tasks with highly multimodal strategies, long-delayed consequences, or explicit risk evaluation, a counterfactual planner may still have an advantage. The deterministic L1 action head introduces another familiar behavioral-cloning limitation. When demonstrations contain several equally valid motions, a feed-forward regressor may average between them. The experiments do not isolate action multimodality, so it remains unclear how GAM would perform on tasks where successful trajectories divide into distinct homotopy classes or contact strategies. Geometry is present, but physics is still incomplete. GAM’s representation is more natively geometric than those of ordinary VLAs or video policies, but “geometry” should not be mistaken for a complete physical state. Future depth maps capture surface location, silhouette, occlusion, and camera-relative structure. They do not directly specify friction, compliance, mass, force closure, tactile contact, or the state of fully occluded mechanisms. The robot also receives no depth sensor at runtime. Its geometry is inferred from RGB by DA3, and real-world depth supervision comes from the same family of models. Any systematic geometric error in the teacher can propagate into the policy. For many tabletop tasks, that prior is evidently good enough to be useful. Precision insertion under reflective surfaces, deformable manipulation, transparent objects, or severe hand-object occlusion may expose different limits. Nor is the model formally equivariant to camera transformations. The camera-perturbation results show empirical robustness, probably arising from DA3’s multi-view pretraining, Track4World adaptation, robot pretraining, depth objectives, and data augmentation. The experiments do not attribute the gain cleanly among those ingredients or guarantee behavior under arbitrary SE(3) transformations. Finally, “generalist” requires qualification. GAM is pretrained across 784,000 trajectories, but the LIBERO experiments fine-tune separate policies for each suite, and RoboCasa receives its own three-view adaptation. The four real tasks are trained together, yet they still represent one setup and a small instruction vocabulary. This is a generalist initialization and architecture followed by benchmark-specific post-training, not one zero-shot policy controlling every experiment. Language is another deliberate limitation. Instructions pass through a frozen T5 encoder rather than a large reasoning model. The authors explicitly identify commonsense and language reasoning as bounded by this component. GAM is strongest where the task language is known and the principal distribution shift is geometric; it is not positioned as a replacement for the open-world semantic capabilities of large VLAs. Where GAM sits among related approaches. Spatial Forcing and ROCKET preserve the VLA as the primary policy and align its representations with a geometric teacher. That is appealing when open-vocabulary semantics are central and geometry is a regularizer. GAM sacrifices the large VLM backbone, but lets geometric processing influence every stage from observation encoding to final action refinement. A concurrent approach, Vision-Geometry-Action, makes a closely related argument that manipulation should be built on native three-dimensional representations rather than language or video backbones. It uses a pretrained 3D world model, progressive volumetric modulation, and joint training for action generation. GAM’s distinguishing mechanism is the explicit split inside the geometric backbone, next-state prediction at that seam, and reuse of the remaining geometric blocks as the action decoder. Explicit 3D policies such as 3D Diffusion Policy instead consume point clouds or comparable spatial observations and learn task-oriented encoders. Those systems can offer clearer metric grounding but depend on depth sensing, reconstruction quality, or embodiment-specific processing. GAM tries to obtain similar geometric advantages from RGB while transferring the priors of a large, pretrained reconstruction model. The emerging design space is therefore not simply VLA versus world model. It includes semantic backbones with geometric alignment, video backbones with temporal generation, explicit point-cloud policies, and native geometry backbones that generate actions. GAM is strong evidence that the fourth category deserves serious attention. What adopting GAM would involve. As of August 6, 2026, the authors have released the LIBERO and LIBERO-Plus implementation, training configurations, rollout scripts, a pretrained initialization, and suite-specific checkpoints. The repository includes Docker and Conda environments, DeepSpeed training paths, and scripts for reproducing perturbation evaluation. The source code uses the MIT License, although the underlying geometric weights and robot datasets retain their own licensing and citation requirements and should be audited independently. For a new robot, the first challenge is likely to be action unification rather than architecture. GAM expects a consistent delta-pose representation, proprioceptive convention, gripper meaning, chunk duration, and control frequency. The public pretraining utilities make these conversions explicit, and teams should treat them as part of the model rather than incidental data plumbing. The second challenge is geometric supervision. Simulation users can render metric future depth. Real-robot users can generate pseudo-depth with the pretrained GFM, avoiding additional sensors. The ablations suggest that, given the released pretrained GAM initialization, future-depth losses may be less critical during small downstream fine-tunes. They remain useful diagnostics, however, because decoded futures reveal whether the policy is attending to the intended object and predicting physically coherent progress. The third challenge is choosing where and how much to fine-tune. The reported configuration freezes the initial geometric blocks and depth head but updates the deeper backbone, predictor, and action head. Freezing the entire GFM would be cheaper but would turn the method back into a static feature extractor. Moving the split or heavily modifying the backbone should be approached cautiously because the layer ablations show a narrow functional seam. For research extensions, the most promising directions are clear. An LLM or VLM could provide high-level task decomposition while GAM handles geometric control. A stochastic action decoder could preserve multiple valid trajectories. Force and tactile tokens could augment depth with contact state. Most ambitiously, the predictor could be conditioned on candidate future actions and unrolled as a genuine counterfactual dynamics model, combining GAM’s geometric latent space with model-predictive planning. The bottom line. GAM’s most important message is not that every robot policy should predict depth. It is that foundation-model selection defines the inductive bias of the entire controller. A VLM asks the robot to reinterpret semantic features as geometry. A video model asks it to extract three-dimensional structure from evolving pixels. GAM starts with a model whose representation was built to explain where surfaces and cameras exist in space. The results support that choice most convincingly under camera displacement: a setting where image coordinates change dramatically while physical geometry does not. The method is fast, comparatively compact, and backed by unusually revealing ablations. At the same time, it remains compute-heavy, benchmark-post-trained, deterministic, and limited in language and counterfactual reasoning. Even with those qualifications, the architectural principle is compelling. A geometric foundation model does not have to remain a frozen perception module sitting in front of the policy. If it is split at the right layer, taught to predict its own future latents, and allowed to refine action tokens through its deep geometry blocks, the geometry model can become the policy itself.