Latest / Embodied AI 101 / Geometric Action Model for Robot Policies

Transcript

Geometric Action Model for Robot Policies · Embodied AI 101

• Geometric Action Model for Robot Policies. In autonomous manipulation, it’s not enough for a robot to recognize and categorize objects – it must reason about the full 3D geometry of the scene to predict how things will move under its actions. Yet most recent language-conditioned policies and “world models” operate largely on 2D images or low-dimensional embeddings. They short-change the very spatial structure (depth, scale, occlusion) that humans effortlessly use to manipulate objects. The Geometric Action Model (GAM) approach, introduced by Jisang Han and colleagues, takes a different tack: it makes 3D geometry the central substrate of the policy itself. In GAM, a single pretrained 3D perception model – called a Geometric Foundation Model – is repurposed as the unified backbone for perception and action. By splitting this geometric model mid-way and inserting a causal transformer to forecast the future, GAM jointly predicts both the next 3D scene structure and the robot’s action in one efficient forward pass. Intuitively, GAM says: “Instead of tacking on 3D features to a 2D policy or running separate pipelines for vision and action, why not directly make a rich 3D geometry engine the policy network itself?” The result is a policy that inherently understands spatial layouts and contact relationships. In practice, this pays off in higher accuracy, much better generalization to new viewpoints or lighting, and lightning-fast inference – all with fewer parameters than prior foundation-scale methods. In a broad suite of simulated and real-world benchmarks, GAM matches or exceeds the success rates of state-of-the-art baselines (many of which have several billion parameters), yet runs over 100Hz and is robust to camera shifts. In this chapter we’ll unpack how GAM works, why its geometry-first philosophy brings benefits, and what it might mean for future robot learning. We’ll compare GAM to previous vision-language-action and video-prediction policies, dive into the model architecture, and highlight the key experiments and results that show its strengths. Our goal is to give you both the big picture and enough technical detail to see how you might adapt these ideas for your own robot tasks. Geometry vs. 2D Latents: The Motivation. Robotic manipulation is inherently a 3D game. When you grasp a mug or align a peg in a hole, you must reason about how objects extend and move through the third dimension. A tall stack of blocks looks very different from the side than head-on, and depth cues tell you when your gripper will actually touch the object. However, most recent generalist manipulation policies, especially language-conditioned ones, haven’t explicitly represented 3D geometry. For example, modern vision-language-action (VLA) policies typically fine-tune large vision-language models (like CLIP or DINO) to predict actions from image frames and text commands. They inherit rich semantic understanding (e.g., of "block" or "cup") but only see the world in flat images. Similarly, video world-action (WAM) models use large video prediction networks to jointly hallucinate future frames and actions, which captures temporal priors but still treats everything as pixels. This 2D-centric view forces the policy to implicitly learn depth, scale, and occlusions on its own. In practice, that leads to brittleness. A model might do well in the camera views it was trained on, but fail when the camera or lighting changes slightly. Indeed, past work has observed that when you throw small viewpoint or object-appearance shifts at these policies, their success rates often plummet. It’s as if they never quite learned “how far away” things are or when contact happens. The key insight driving GAM is that 3D geometry should not be an afterthought – it should be the policy’s native language. If you have a network already designed to build a 3D scene model from images (what GANs and NeRFs do, but in this case a transformer-based “geometric foundation model”), why not just make that network generate the next scene and the robot action? That way, the geometry priors baked into the network (from its pretraining on multi-view images) are fully leveraged. Rather than passively extracting some latent features to feed to an action head, GAM actively rendezvouses perception and action in the same 3D space. In practical terms, the authors take a pretrained geometric model (the “GFM”) which normally maps multi-view images to a 3D latent code or depth map. They split it into two parts at an intermediate layer. The early layers act as an encoder that digests the current camera images (and robot state) into a spatial latent representation. Then, at the split point, they insert a causal transformer – a time-series predictor – whose job is to forecast the future latent 3D state one step ahead, conditioned on the instruction language, proprioceptive inputs, and the action that the robot took in the last step. The predicted latent state then flows through the remaining deeper layers of the geometric model, which decode it back into both an estimated future depth map (representing the next scene) and an actual robot action. In other words, GAM simultaneously outputs “what the world will look like next” and “what the robot should do” all in one pass. By doing this, GAM turns the 3D model into a shared substrate for world modeling and control. The shallow layers serve as visual feature extractors, the transformer predicts how those features move over time, and the deep layers decode them into consequences (future geometry) and decisions (actions). Because it uses a pretrained 3D model, GAM inherits strong spatial priors. And because it uses a single backbone network (“one backbone to rule them all”), it avoids expensive multi-step sampling like diffusion or training multiple networks for perception vs. control. Architecture of GAM: Turning a 3D Model into a Policy. Let’s walk through GAM’s architecture step by step. Imagine you have a pretrained Geometric Foundation Model, which we’ll call GFM. In effect, GFM is a big transformer (or transformer-like) network that has been trained on lots of multi-view image data to infer 3D structure. For example, it may take as input a set of RGB images from different cameras (or different viewpoints over time), process them through many layers of spatial reasoning, and output a depth map or 3D voxel grid. Crucially, it has multiple layers of processing, from low-level image filters up through high-level geometry features. 1. Encoding observations with shallow GFM layers. At time t, the robot observes the scene via one or more RGB camera images. These images (and the current language instruction) go into the bottom part of the GFM. Concretely, GAM maps multi-view RGB frames into an intermediate latent token tensor. This is done by simply running the initial (say) L_s layers of GFM forward on the images. These shallow layers act as an encoder $E$ that extracts spatial–geometric features from the raw pixels. No new networks are trained for this part; it’s simply reusing the frozen or fine-tuned GFM layers as-is. Intuitively, at this point the GFM has converted the camera views into a structured representation (tokens) of the scene’s geometry – something like a 3D-aware latent map. 2. Causal future predictor (the time transformer). Next comes the novel insertion: a causal transformer module responsible for time prediction. This transformer operates on a sequence of latent states to predict the next state. Specifically, at each time step $t$ within an episodic context window, we feed into the transformer a sequence of “past” latent tokens $\mathbf{Z}{t-k}^{(L_s)}, \dots, \mathbf{Z}{t}^{(L_s)}$ where $\mathbf{Z}{t'}^{(L_s)} = E{\le L_s}(\text{images at }t')$ are the tokens from the GFM encoder at earlier times. Along with those latent tokens, the transformer also receives tokens representing the robot’s proprioceptive state (like joint angles, gripper pose) and the previous action $a_{t'}$ at each step, as well as the language instruction (encoded via a frozen language model). All this multimodal information is concatenated and processed by the transformer under block-causal self-attention, so that it can only attend to current and past steps. The output of this transformer has two parts: it produces (a) a predicted future geometric token $\widetilde{\mathbf{Z}}{t+1}^{(L_s)}$ and (b) a predicted next action token $\widetilde{\mathbf{a}}{t}$. Think of the action token as an abstract encoding of “what the robot will do” at that step. By jointly predicting the latent 3D state and the next action in one sequence, the model keeps the action and geometry reasoning tightly coupled. 3. Decoding predicted state and actions through deep GFM layers. Once we have the transformer’s output, GAM sends the predicted future tokens back through the rest of the GFM – the deeper layers $D$ that were set aside. Specifically, both the predicted geometry token $\widetilde{\mathbf{Z}}{t+1}^{(L_s)}$ and the predicted action token $\widetilde{\mathbf{a}}{t}$ are fed into $D_{>L_s}$. For simplicity, you can imagine that the action token is duplicated for each camera viewpoint (if there are multiple cameras) and concatenated with the geometry token. This combined token then flows through the remaining M − L_s layers of the original geometric model. Because the deep GFM is trained to fuse multi-view features and decode them into spatial outputs, it now treats the predicted features as if they were a real future scene. At the end of the GFM decoder, two “heads” produce the final outputs:. Depth (geometry) head: This head decodes the tokens into an estimated depth map (or 3D occupancy) corresponding to the future scene at $t+1$. Essentially it is asking, “If the predicted latent tokens were correct, what would the scene look like in terms of geometry?” Because this is just running the original GFM’s depth output on predicted latent, it yields a dense prediction of where surfaces and objects are. Action head: In parallel, a lightweight regression head takes as input the sequence of predicted action tokens (over the window) and outputs an executable action chunk $\hat{a}_t$. Concretely, it can treat the set of action tokens in the context window as features and linearly (or via a small MLP) map them to the continuous control commands the robot should execute (e.g., gripper velocity/force, joint torques, etc.). This output is exactly the next action for the robot. In summary, a single forward pass does all of the following:. Perceives the current multi-view images (through $E_{\le L_s}$). From past latents and commands, predicts the next latent ($\widetilde{\mathbf{Z}}_{t+1}^{(L_s)}$) and next action token. Decodes that latent into future geometry (depth maps) and decodes the action token into an actual motor command. Importantly, the GFM’s layers are used as-is (mostly frozen): the shallow layers encode the images, the transformer predicts the next tokens, and the deep layers propagate that prediction to outputs. The only new trainable modules are the causal transformer and the two small output heads. This architectural design “turns a static 3D perception model into a language-conditioned world-action model” with minimal modifications. Comparison to Prior Paradigms. To appreciate GAM’s novelty, it helps to contrast with two recent families of approaches:. Video World-Action Models (WAMs): These leverage large generative video models (often diffusion networks) to predict future frames from past frames and robot actions. Then an action decoder picks actions that lead to desired future frames. For example, a model like VideoFusion or diffusion-based policies fall here. By nature, they get very strong temporal priors (they can hallucinate realistic video futures), but all reasoning is in pixel space. Depth, scale and occlusion are never explicitly represented – the network must “blurrily imagine” contact events. Moreover, diffusion inference is slow, requiring dozens of iterative denoising steps at high cost. So while WAMs have shown impressive single-task performance, they struggle with multi-view 3D consistency and are computationally heavy. Vision-Language-Action (VLA) Models: These take pretrained vision-language backbones (like CLIP or DINO) and adapt them to policy learning. They exploit strong object and scene priors from huge image-text corpora, so they understand “what” and “where” in a semantic sense. Typically, the chosen vision model is frozen or partially fine-tuned, and its features are fed to some action network. Some recent works attempt to graft geometry into these pipelines (e.g. via depth augmentation or featurized 3D cues), but usually the geometric model is just a sidekick – not the policy itself. The policy still maps 2D embeddings to actions, often with a light action head on top. As a result, the policy may not fully leverage the geometry beyond whatever hints the learned features carry. This can again cause limited generalization under scene shifts. The Geometric Action Model differs in a key way from both: it makes the geometry network the star of the show. Instead of a diffusion network conditioned on images or a CLIP backbone, GAM directly embeds a 3D model and uses it for everything. The authors summarize it as turning the GFM “into the policy”. This means preserving the GFM’s “rich 3D priors” throughout and grounding every action in an explicit predicted 3D state. Another advantage is unity: perception, prediction and action all happen in one forward pass of the same network. No second stage or looping is needed. The entire architecture is effectively one autoregressive transformer over geometry/action tokens. Thus GAM avoids expensive sampling loops (like diffusion) and only needs a single pass to output both what happens next and what the robot should do. In contrast to approaches that passively “inject” geometry signals (e.g. concatenating point clouds or distilling depth into features), GAM integrates geometry end-to-end in the decision loop. The action is no longer “blind” to the geometry – the predicted future 3D scene directly influences the choice of motor commands via the shared tokens. This tight coupling is what gives GAM robustness. For example, if the camera view changes, the encoder layers will produce different latent tokens reflecting the new viewpoint. But because the deeper layers are grounded in actual 3D structure, the policy can still infer “object in front has moved depth-wise” and recover. Empirically, this leads to much smoother generalization across visual shifts than purely 2D policies. Training GAM: Losses and Tokens. The GAM network is trained end-to-end with a combination of loss terms that enforce both correct action and correct geometry prediction. The training data consists of episodes of robot interaction (in simulation or real), each given with multi-view video, robot proprioceptive state, past actions, and a language instruction. At each time step the model predicts a continuation, and we have ground truth data for the next action and the next multi-view depth map. Concretely, the losses include:. Action loss ($\mathcal{L}_{\text{act}}$): This is typically an $L_1$ (or similar) regression loss between the predicted action chunk $\hat{a}_t$ and the true next action $a_t$. It ensures the policy output stays on track with expert actions (from demonstrations). Feature (latent) loss ($\mathcal{L}_{\text{feat}}$): Here, the idea is to align the predicted latent tokens with the actual latent that the frozen GFM would produce from the real next view. Concretely, after running the real next RGB frame through the first $L_s$ layers of GFM, we obtain $\mathbf{Z}{\text{real}}^{(L_s)}(t+1)$. The transformer’s predicted $\widetilde{\mathbf{Z}}{t+1}^{(L_s)}$ should match that. So $\mathcal{L}_{\text{feat}}$ is typically a squared error (or similar) between these two latent tensors. This forces the time-predictor to actually learn the correct dynamics in the geometry representation space (not just ignore geometry and hallucinate arbitrary). Depth reconstruction loss ($\mathcal{L}_{\text{depth}}$): After decoding the tokens, GAM produces a depth map. This is compared to the ground-truth depth image from the next time step (from simulator or a depth sensor). A scale-invariant loss (plus gradient consistency) is used to test if the predicted topology and shape match reality. In effect, this ensures that the deep GFM sees the predicted tokens as a realistic future scene. Putting it together, the total loss is a weighted sum: $$ \mathcal{L} = \lambda_{\text{act}},\mathcal{L}{\text{act}} + \lambda{\text{feat}},\mathcal{L}{\text{feat}} + \lambda{\text{depth}},\mathcal{L}_{\text{depth}},. $$ With this, the network must simultaneously learn to execute the right actions, predict where things will be, and maintain consistency between latent and geometry predictions. The authors found that this multi-task training is crucial: it prevents the policy from ignoring geometry (e.g. by collapse the depth to trivial values) and really enforces a 3D-capable internal model. Ablations showed that without the feature and depth losses, the policy quickly overfits to camera views and fails under perturbations; indeed, dropping pre-training or geometric supervision can make success on “LIBERO-Plus” tanks from ~90% to ~50%. It is worth noting that GAM adds relatively few new weights. The GFM itself (with M layers total) has millions of parameters pretrained from vision-data. GAM splits between layer $L_s$ and $L_s+1$. The transformer inserted at that point (often with similar hidden width) is on the order of tens to a few hundred million parameters. And the action/depth heads are very small. Overall, this entire network can be on the order of 1–2 billion parameters. But since the GFM can be large (say 1–2B) and is reused, GAM’s trainable add-ons are actually modest in size. The authors specifically report a 1.4B-parameter GAM instance (including GFM) which already outperforms larger baselines. Experiments and Results. The authors evaluated GAM extensively on both simulated benchmarks and real-robot tasks, always in a language-conditional setup. Their main simulation testbed is the LIBERO benchmark suite, which includes diverse multi-stage manipulation tasks (like object relocation, stacking, tool use, etc.) with natural language instructions. Importantly, they use both the standard LIBERO and a harder variant called LIBERO-Plus where the test-time cameras, lighting, backgrounds, and even object poses are more varied than in training (to assess robustness). In addition, they test GAM on several real robot setups (e.g. simulated vs real pick-and-place, block stacking, insertion tasks). Across the board, GAM shows state-of-the-art performance. Here are some highlights:. Success Rates: On the original LIBERO benchmark (in-distribution conditions), GAM matches or slightly exceeds the best prior methods. For example, GAM achieves about 97.6% average success rate on LIBERO, comparable to the best prior results reported. The real breakthrough is under perturbations: on LIBERO-Plus (camera varied), GAM scored 85.5% overall success, which is about 9.7 percentage points higher than the next-best baseline. In the hardest camera-perturbation split, GAM reached 83.1% (compared to around 73.4% for the best baseline). That’s a very strong indication that using explicit geometry really cushions against visual shifts. Comparison with Baselines: Prior baselines included large VLA and WAM policies with up to 7–8 billion parameters. For instance, an “OpenVLA” 7B policy and a “π_0.5” 3.3B policy were among the top contenders, along with a 2B “Cosmos-Policy”. Despite being smaller (GAM is only ~1.4B params vs baselines’ 2–8B), GAM outperforms them. In one table, GAM’s LIBERO success was 97.6% (essentially saturating), and its LIBERO-Plus 85.5% far surpassed OpenVLA’s 69.6% and π_0.5’s 84.6%. Even compared to the diffusion-based WAM “Fast-WAM” of 6B params (which fell to only 50% under perturbations), GAM stayed robust at 85.5%. In short, the geometry-aware GAM achieved higher accuracy than any method of comparable or larger size, especially under domain shift. Inference Speed: Perhaps surprisingly, GAM is not just more accurate but dramatically faster to run. The entire forward pass (RGB→action) takes only about 6.9 milliseconds per step (around 145 Hz) on a modern GPU. This is 55× faster than a diffusion-based video-policy baseline. It’s faster than other autoregressive models too, largely because there’s no iterative sampling – just one pass through the transformer. At over 100Hz, GAM is well within real-time control requirements for most manipulators, unlike some heavy VLM-based policies that run tens of FPS at best. Parameter Efficiency: The authors emphasize that GAM’s compactness goes hand-in-hand with speed. The 1.4B parameter GAM (including the 3D model) outpaces baselines that use 2–8.5B params. They point out: “GAM achieves a smaller model size and lower latency than all compared methods”. (For instance, a 1.4B GAM vs 3.3B π_0.5 took 95% less inference time.) This underscores the value of architecture design over brute force scaling in robotics. Real Robot Demonstrations: In physical robot tests (on tasks like stack-blks, pick-place with disturbances), GAM consistently outperformed baseline policies under both nominal and perturbed conditions. It wasn’t just a simulation fluke: explicit geometry reasoning translated into real-world robustness. The GPU speed also means you could potentially run GAM on-board (with some optimization) for live control. Ablations: The paper includes several ablation studies. Notably, if they remove the GFM pretraining (training GAM from scratch), performance on perturbed LIBERO-Plus plummets (from ~90% to ~50%). That confirms the prelearned geometry is crucial for generalization. They also experimented with different split points $L_s$. If the causal transformer is inserted too early or too late in the network, performance degrades, indicating a sweet spot (the authors found about halfway through the model was best). This suggests the model needs enough “visual processing” before predicting, but also enough “decoding capacity” after to make use of the future tokens. The big takeaway from the experiments is that GAM strikes an excellent balance: it fully harnesses 3D awareness to boost generalization, without sacrificing efficiency. As the authors note, across both simulation and real benchmarks, GAM was “more accurate, more robust, faster, and lighter” than all prior foundation-scale competitors. Why Structured Actions and Geometry Help. You might wonder: why exactly does GAM generalize so well? We’ve hinted at it already, but let’s elaborate. Traditional policies see images and language, and produce actions. In doing so, they have to solve two problems in one shot: figure out the state of the world (perception & prediction) and decide on control (planning). GAM explicitly breaks this problem out: it maintains an internal 3D model of the world’s state and its likely evolution. This structured latent space disambiguates many issues. For instance:. Occlusion and Contact: In 2D pixel space, it’s hard to know when the gripper will make contact with a hidden object. In a 3D geometry model, that contact moment is explicit as surfaces touching. By carrying along a latent geometry, GAM inherently knows depth. If an occluded block suddenly becomes visible, the predicted depth tokens capture that “burst of information,” alerting the policy to, say, stop moving and grasp. Camera View Changes: With a 3D-aware model, the underlying physical relationships remain consistent even if your viewpoint changes. So a small camera perturbation won't utterly confuse the depth tokens. The policy effectively reasons in world coordinates, not camera coordinates. This is evidenced by the ablation where even moderate camera shifts gave little drop in GAM’s success, unlike pure 2D models. Long-Term Prediction: In multi-step tasks (stacking, multi-object goals), understanding how objects move over time is crucial. The latent predictor aims to forecast future 3D states step by step, so the network really learns dynamics. This is akin to having a built-in physics model. Empirically, GAM did very well on the “long” sequence tasks in LIBERO, suggesting it indeed can chain multiple steps accurately. The structured latent (3D) helps it remember object permanence and handle interactions better than a frame-by-frame approach. Structured Action Representation: GAM doesn’t just predict a single control command; it emits an “action token” alongside the geometry. You can interpret this as a structured summary of the robot’s intended sub-action. Feeding this token into the decoder ensures that the model learns how actions and world states co-evolve. It’s like saying, “the robot’s plan for the next step is part of the state we predict.” This tight coupling probably helps the model learn, for example, that moving the gripper right will cause corresponding changes in depth features (closer or farther objects). The policy is never asked to produce an action divorced from the world; actions come from the same latent stream. Single-pass Efficiency: Because GAM predicts geometry and action together, it avoids the “iterative planning” loop some methods use. For example, a diffusion policy might repeatedly sample candidate action sequences and see which yields a good predicted video. GAM, by contrast, treats the joint prediction as one forward sequence. This means it can run at 145Hz, and crucially it can leverage temporal credit assignment in training: an action that leads to a bad predicted geometry will be penalized immediately in the depth loss. Essentially, every weight update flows through both the geometry and action heads, tying them together. In summary, GAM embeds a form of structured action representation. The “action token” is not an uninterpretable vector; it interacts with the 3D features inside a spatial transformer. This likely imposes a constraint that the model learns a smoother, more physically-plausible control mapping, which pays off when things don’t go exactly as in training. It’s part of why the authors could still keep good performance even with a relatively small number of parameters – the architecture itself imposes useful biases. Limitations and Future Directions. While GAM shows great promise, it’s worth noting some limitations and open questions. Dependence on Pretrained Geometry: GAM relies on having a strong pretrained GFM. This model was presumably trained on plenty of multi-view or depth data for generic scenes. If you want to apply GAM to very different environments, you’d need a suitable GFM for that domain. The authors find pretraining crucial: without it, the learned model doesn’t generalize. So GAM is complementary to – not replacing – the effort to build universal 3D vision models. Language Understanding: In the current GAM, the language instruction is encoded by a frozen T5 model, which feeds tokens into the transformer. So GAM’s language capabilities are limited by T5’s pretrained knowledge, and it doesn’t actively plan out language. In the discussion, the authors suggest that plugging in a full large language model or reasoning module might be a next step, to allow more complex instructions or symbolic reasoning. Right now, instructions likely need to be relatively straightforward (e.g. “stack the red block on the blue block”). One-Step Prediction: GAM predicts a single next step at each time. The overall policy is autoregressive - it takes one action, observes the result, then predicts again. This works fine but could be extended. One could imagine unrolling the transformer multiple steps into the future (like a multi-step rollout) to plan longer-horizon actions in one shot. Alternatively, integrating an actor-critic or model-predictive control mechanism on top of GAM’s latent predictions could add robustness. Scaling to More Agents or Scenes: Currently GAM is focused on one robot and its scene. Scaling to multiple robots or very cluttered environments might require modifications (e.g. larger latents, different attention biases). Similarly, interactions with deformable objects or fluids (less amenable to short-term geometry prediction) might be challenging. Nonetheless, the success of GAM suggests a powerful architectural principle: take a good world model and reuse its internals for control. One can imagine similar ideas beyond manipulation: for example, in autonomous driving, repurposing a 3D scene understanding model as a driving policy. Or in aerial robotics, coupling a terrain reconstruction network with action planning. Anywhere you have a pretrained spatial model, inserting a temporal action predictor could yield stronger policies. Conclusions. The Geometric Action Model represents a compelling new paradigm for embodied artificial intelligence. Instead of treating geometry as a sideline, it makes geometry the canvas on which the entire policy is drawn. By splitting a pretrained 3D vision-transformer and inserting a simple future-predictor, GAM achieves a tight integration of perception, prediction, and control. This pays big dividends: GAM matches the highest baseline success rates on standard tasks while sharply outperforming them under visual perturbations, all with lower latency and parameter count. For roboticists and RL researchers, GAM offers both insights and practical ideas. Architecturally, it shows the power of modular reuse: leverage what pretrained networks know (3D structure) and add only the minimal mechanism needed for the task (a causal transformer). Conceptually, it emphasizes that even a robust foundation model’s layers can be re-purposed – the geometric model’s deep layers aren’t just a decoder for inputs, they can also decode imagined futures and actions. Finally, it points the way toward policies that truly understand physical space – a step closer to robots that can reason about their environments in human-like geometric terms. References: The ideas and results here are drawn from the paper “Geometric Action Model for Robot Policy Learning” by Han et al. (CVLab KAIST / ETH Zurich, 2026), which provides full details. Their code and model information are expected to be released alongside the paper.