Latest / Embodied AI 101 / GAM: Ground the Target Before You Learn the Action

Transcript

GAM: Ground the Target Before You Learn the Action · Embodied AI 101

• GAM: Ground the Target Before You Learn the Action. Grounded Action Model: 3D Grounding as a Foundation for Robotics • Gehao Zhang, Weikai Huang, Shailesh Shailesh, Yiyan Peng, Jiafei Duan, Ranjay Krishna • Northwestern University, University of Washington, National University of Singapore, Microsoft • arXiv preprint • 2026. The failure hiding behind a good rollout. Imagine a robot trained to move a mug into a bowl. Its demonstrations are consistent, its validation videos look convincing, and its success rate is high. Now move the mug to the other side of the table. Alternatively, leave the scene untouched and ask for a different mug. These interventions ask different questions. The first asks whether the controller responds to geometry. The second asks whether the instruction actually determines the target. A policy can look competent in the original setting while answering neither question reliably. For this episode of Embodied AI 101, that is the lens through which we will examine Grounded Action Model, or GAM. The identifiable paper is the arXiv preprint submitted on September 20, 2026. The useful question is not simply whether three-dimensional representations beat two-dimensional representations. It is where we should resolve the relationship between an instruction and the physical objects involved in an action. Should the action network discover that relationship implicitly, while also learning motion? Or should a separate pretrained system establish a target-specific spatial representation before motor learning begins?. Those alternatives create different engineering responsibilities. The first leaves more flexibility inside the policy. The second creates an interface that can be inspected, corrected, and manipulated—but also introduces a bottleneck that can discard necessary information. We will work through that interface, the action generator behind it, and the evidence supporting the design. Then we will separate what the reported experiments establish from the experiments I would want before adopting this approach in a new manipulation stack. What the policy is actually given. First, a correction to the shorthand description: GAM’s prompts can be language, 2D clicks, or 2D boxes; its point clouds are derived internally. It is not simply a policy requiring a supplied, sensor-measured 3D scene. The relevant perception foundation is WildDet3D. Its public interface accepts an RGB image and flexible prompts, then predicts metric object boxes. Those estimates include physical position, size, and orientation. Additional depth measurements can be incorporated when available, but monocular operation is supported. This distinction matters: an explicitly three-dimensional representation need not originate from a depth camera. WildDet3D itself combines appearance-oriented and geometry-oriented processing. Its architecture uses a segmentation-pretrained image encoder alongside a depth-oriented encoder adopted from LingBot-Depth, then combines their information for detection. The associated data effort extends supervision to roughly a million images and thousands of categories. The relevant prior is therefore learned object localization and geometry, rather than knowledge acquired solely from robot trajectories. GAM uses WildDet3D, frozen during policy training. Its visual stream is a masked 16-by-16 feature grid retaining targets and arm. Each object supplies eight geometry tokens from 512 reconstructed points plus box attributes; current and previous joint positions supply state history. Geometry is expressed in the robot-base frame. That compact specification suggests an important way to think about the representation: appearance and geometry answer overlapping, but not identical, questions. Consider two objects with similar bounding boxes but different usable surfaces. Their position and extent may be almost interchangeable for reaching, yet inadequate for choosing a grasp. Conversely, appearance alone may distinguish the objects without exposing the metric displacement required to reach either one. A useful precedent for compact geometry encoding is PointNet. Its central construction processes individual points with shared functions and aggregates them through a symmetric operation, making the representation independent of point ordering. This gives a straightforward route from an unordered set of spatial samples to a fixed-size descriptor without first constructing a dense voxel volume. For a manipulation implementation, however, compression raises a question that classification accuracy does not settle: which local details must survive? A representation sufficient to recognize a bottle might still discard the lip, handle, or narrow contact surface that determines successful interaction. I would evaluate geometric descriptors against those task-dependent requirements rather than assume that a good object representation is automatically a good control representation. There is also a fundamental distinction between metric output and accurate metric output. WildDet3D explicitly identifies monocular depth ambiguity, calibration accuracy, and difficult orientations as limitations. Coordinates expressed in meters remain estimates, not guaranteed measurements. My practical interpretation is that explicit geometry makes errors more inspectable. It does not make them disappear. If an estimated object center is wrong, the debugging question becomes concrete: was the error in identification, depth, calibration, or the learned response to an otherwise correct observation?. A familiar generator behind an unfamiliar observation. Action generation uses a 12-block multi-stream flow-matching transformer, language-conditioned normalization, and absolute joint-position/gripper chunks. The multi-stream architecture belongs to a familiar family. In Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, Patrick Esser and colleagues use modality-specific processing while allowing information exchange through joint attention. Separate streams need not mean isolated streams: each can maintain its own transformations while attending to representations from the others. For robotics, that is a sensible architectural vocabulary. A visual feature, an object descriptor, and a robot configuration are not interchangeable measurements. Giving them different processing parameters does not prevent the model from learning relationships among them. It simply avoids requiring every modality to enter an identical processing path immediately. The flow-matching objective is also worth stating precisely, because it is easy to confuse the generator’s dynamics with the robot’s dynamics. For a straight-path conditional flow construction, training takes a data sample and a noise sample, chooses an intermediate location between them, and learns the direction that carries the noisy sample toward the data. Generation begins with noise and numerically integrates the learned vector field. The foundational work by Yaron Lipman and colleagues formulates this as regression onto conditional vector fields, avoiding the need to simulate the full generative trajectory during each training example. Applied to an action sequence, the sample being transformed is an entire candidate chunk. The intermediate states of that transformation are not physical robot states. That gives us two clocks to keep separate. One clock indexes the future commands in a chunk. The other indexes the numerical refinement of that chunk during generation. A predicted flow velocity should not be interpreted as a commanded joint velocity. This separation also clarifies the deployment trade-off I would investigate. Executing more of a generated chunk could amortize expensive perception and generation. Replanning sooner could provide more opportunities to respond to a slipping object or a changing scene. Neither choice follows automatically from the quality of the representation. I would therefore treat chunk length, executed prefix length, numerical solver steps, and observation latency as a coupled design problem. A controller with excellent geometric conditioning can still behave poorly if the geometry is stale by the time the commands reach the robot. The broader lesson is that a new observation interface does not eliminate ordinary closed-loop control questions. It changes what the policy knows when it answers them. The important inductive bias is selection. Here is the distinction I find most useful when reasoning about object-centric control: some scene changes should leave the action unchanged, while others should change it immediately. Replace a decorative background and the intended grasp should remain stable. Move the selected object and the grasp must move. Keep every pixel fixed but select a different object and the action should change again. These are not three examples of generic robustness. They require different responses to different interventions. An effective representation must suppress some variation while preserving other variation with high fidelity. This concern predates GAM. Selective Visual Representations Improve Convergence and Generalization for Embodied AI, by Ainaz Eftekhar and colleagues, investigates a task-conditioned representational bottleneck built with a learned codebook. Its purpose is to reduce irrelevant visual information rather than pass every feature from a general-purpose backbone directly into the embodied policy. The broader idea is that selecting information can be as consequential as increasing representational capacity. For manipulation, I would frame the corresponding hypothesis as a division of learning responsibilities. A perception system identifies the relevant entities. A controller learns how to interact with their estimated configurations. That separation might reduce the number of nuisance correlations the controller can exploit. But “might” is important. Masking is not a mathematical guarantee of invariance. If an upstream feature already mixes information across spatial locations, retaining that feature can retain contextual information too. Likewise, a bounding box can contain background, neighboring objects, and occluders. Selection changes access to information; it does not necessarily create perfectly isolated object variables. The ten-task ablation reports 46.8% mean success for full GAM, 22.3% without image masking, 12.8% without point cropping, and 10.3% without both; point budgets remain fixed. Detection-only and image-only variants achieve 16.0% and 20.3%. The next experiment I would want separates semantic filtering from effective geometric resolution. When a fixed sample budget covers a larger scene, the target may receive fewer samples. A deterioration could therefore reflect both additional distractors and reduced target detail. I would compare equal total point budgets and equal target-point budgets. I would also retain background geometry in a separate stream rather than mixing it into the target descriptor. Those controls could reveal whether the decisive benefit comes from exclusion, object-specific organization, or simply spending more representational capacity on the contact-relevant region. That distinction matters when adapting the method. The right conclusion might be “discard irrelevant context.” It might instead be “keep context, but stop making it compete with the target for the same small representation.”. Reading the simulation results without flattening the story. RoboTwin’s official evaluation setting uses 50 clean demonstrations for each of 50 tasks, followed by 100 evaluation trials per task in clean and randomized conditions. Its leaderboard combines single-task and jointly trained entries, and the default aggregate averages clean-to-clean and clean-to-random performance. These protocol details are essential when interpreting a headline ranking. RoboTwin uses task-specific GAM policies; LIBERO uses one policy across four suites. For RoboTwin, the grounding backbone is first adapted with simulator-provided boxes and depth. The reported RoboTwin scores are 63.0% clean, 47.6% randomized, and 55.3% average. Spatial Forcing averages 52.0%; the next-highest randomized result listed is Abot-M0 at 30.4%. The single-task π0.5 comparison averages 45.0%. I would read those numbers through two separate questions. How good is the controller under its familiar conditions? And how much of that ability survives the specified distribution shift?. Averaging those questions into one number is convenient, but it does not make them equivalent. A deployment with tightly controlled fixtures may value peak familiar-scene performance differently from a deployment with frequent changes in background and clutter. The ranking is useful only after deciding which regime resembles the intended application. The detector adaptation also changes how I would account for data efficiency. Fifty robot demonstrations do not describe the complete information budget of a system that inherits pretrained perception and receives additional geometric supervision. That is not an objection to using those resources. It is a reason to list them separately. In a reproduction report, I would distinguish robot trajectories, perception pretraining, simulator-derived annotations, and human target specification. Otherwise, “few-shot” can obscure which component already knows most of the difficult spatial mapping. LIBERO-PRO asks whether the target actually controls the policy. LIBERO-PRO was introduced specifically to probe generalization beyond familiar evaluation configurations. Its evaluation tools support changes to object properties, positions, language, task targets, and environments. The benchmark authors report examples where strong standard-task performance coexists with insensitivity to altered instructions or manipulated scenes. For understanding the evaluation, it helps to imagine two tests. In the first, the correct object stays where it was, but its appearance changes. In the second, the original scene stays intact, but the instruction designates another object. The first tests tolerance to altered evidence. The second tests whether the system can change the binding between a task and a physical entity. Across 16 LIBERO-PRO settings, GAM averages 61%, versus 53% for π0.5. In Spatial, relocation scores 60% versus 20%, and target reassignment 88% versus 1%. Yet Goal relocation favors π0.5, 38% versus 19%; GAM’s LIBERO-10 relocation and reassignment scores are only 5% and 8%. Object perturbations also trail π0.5 in all four suites. That unevenness is more informative than an unqualified claim of generalization. I would organize the remaining problem into three stages: identify the requested entity, estimate its state, and synthesize an appropriate interaction. A system can improve the first two stages while still having a narrow motor repertoire. Consider an object that becomes substantially larger. Correctly locating its center does not specify how to change finger placement, approach clearance, or wrist orientation. A policy might receive accurate new geometry and still lack experience converting that geometry into a successful grasp. Likewise, explicit coordinates do not guarantee action equivariance. Translating an object does not imply a unique translation of joint commands. Workspace boundaries, alternative inverse-kinematic solutions, and contact constraints can change the required motion qualitatively. For that reason, I would not ask only whether the selected object remains localized. I would ask whether localization accuracy predicts control success, and where that relationship breaks. Such a breakdown would identify whether the next investment belongs in perception, the action representation, demonstration diversity, or recovery behavior. Real robots: compare the complete systems. On bimanual YAM, both GAM and π0.5 achieve 19/20 in-distribution successes. Under visual shift, GAM achieves 17/20 versus 4/20, using clicks or boxes where π0.5 uses language. The interface difference should stay attached to the result. A visual selection and a language description do not necessarily provide equivalent information about which instance is intended. That does not make the comparison useless. If an operator can reliably select the intended object, that interface may be exactly what an application needs. But an experiment evaluating the entire operator-to-action system answers a different question from an experiment isolating the low-level controller. For the latter, I would match target information across systems. For the former, I would additionally measure operator effort, ambiguity, correction frequency, and the time required to communicate a new target. Successful interaction design is a legitimate robotics contribution; it simply deserves its own accounting. On Franka, GAM with a Molmo2 planner averages 64.7% ID and 49.8% OOD step completion, versus 42.4%/24.0% for π0.5 and 35.3%/17.1% for MolmoAct2. Evaluation uses 20 ID and 10 OOD trials per task. Molmo2 supplies point targets using episode history. The Franka baselines receive no external planner. Molmo2 is relevant because its capabilities include point-based grounding across images and video, alongside visual-language understanding. Its training work explicitly addresses pointing and tracking rather than relying only on textual descriptions of visual content. For an engineer, a point-based handoff creates an attractive boundary: the planner can communicate a selected entity without producing an entire motor trajectory. I would want that boundary to carry more than pixel coordinates, however. A robust implementation could attach identity, confidence, timestamp, operation, and expected completion conditions. Two interpretation cautions follow. First, step completion is not episode success. A system that completes early operations but consistently fails the final operation can receive substantial partial credit. For an application requiring every stage, I would report complete episodes and the distribution of failure stages alongside the step metric. Second, memory can reside in several places: tracked identity, robot-state history, planner context, or an explicit task-state store. I would diagnose them separately. A failure to choose the previously observed object is not the same failure as losing its track during occlusion, even if both eventually produce the wrong grasp. Where GAM sits among other approaches. A few neighboring approaches help clarify the design space without turning it into a competition between acronyms. 3D Diffusion Policy, by Yanjie Ze and colleagues, demonstrates the value of compact point-cloud representations for visuomotor imitation. Its combination of an efficient point encoder and a conditional action generator is an important reference for anyone considering geometric policy inputs. The experimental question I would carry forward is not whether a point-cloud policy belongs to an older category. It is what happens when target selection, sensing, demonstrations, and action architecture are matched. A strong geometry-based baseline with identical object crops could reveal how much benefit remains attributable to the pretrained grounding system itself. SpatialVLA, by Delin Qu and colleagues, injects three-dimensional information through Ego3D position encoding and represents motion with adaptive action grids. It retains a vision-language-action foundation while explicitly strengthening spatial representation. This is a useful reminder that geometry and VLA pretraining are not mutually exclusive choices. In a new system, I would treat semantic pretraining, metric supervision, object selection, and action parameterization as separate design axes. A comparison between named model families can conceal those independently adjustable components. HAMSTER, by Yi Li and colleagues, places a coarse 2D end-effector path between a high-level vision-language model and a lower-level, three-dimensional controller. Its hierarchy assigns task reasoning and precise motor execution to different components. A path interface and an object interface expose different degrees of freedom. A path can constrain how an interaction unfolds, but it also commits the planner to a motion sketch. Selecting an object leaves more motion decisions downstream. I would choose between them based on whether the high-level difficulty is target identification or the structure of the maneuver. VoxPoser, by Wenlong Huang and colleagues, connects language reasoning to control through composable 3D value maps and model-based trajectory planning. That suggests another useful comparison: a spatial interface can encode entities, desired regions, constraints, or trajectories. Before selecting an architecture, decide which information must cross the planner-controller boundary—and which decisions should remain revisable during execution. What I would build—and measure—first. As checked on September 26, 2026, the GAM repository still marks its implementation as forthcoming, including training and inference code, checkpoints, and the real-robot setup. WildDet3D’s repository, separately, provides code and model artifacts. An available perception component should not be mistaken for a released, reproducible GAM controller. Until the complete release is available, I would treat an implementation effort as an adaptation of the research design, not an exact reproduction. My first deliverable would be an observation inspector, not a training run. For every policy query, it would show the selected instances, reconstructed geometry, coordinate transforms, retained visual regions, and the age of the measurements. I would want to inspect a failed grasp without inferring the internal observation from an attention heatmap. The second deliverable would be a precise prompt contract. WildDet3D’s inference documentation distinguishes category detection from visual-example matching and one-to-one geometric prompting. A box can therefore mean “find similar instances” or “lift this selected instance,” depending on the mode. That distinction should be explicit in any integration. I would write tests using repeated, visually similar objects. Select one instance, move another, introduce an occluder, and verify which identity survives. These tests should run before motor learning, because otherwise the action model can absorb perception inconsistencies that later become difficult to diagnose. GAM uses CoTracker3 for maintaining target associations. CoTracker3 provides online and offline variants; its online formulation processes video sequentially, while the offline formulation can exploit information across a complete video window. It also estimates tracking confidence and visibility. For deployment, I would explicitly audit causality and buffering. Does the tracker use only observations available at the decision time? How long does it wait for a window? What happens when confidence drops? A visually smooth track is not sufficient evidence that the information was timely enough for control. Next comes a geometric error budget. I would perturb calibration deliberately, compare estimated surfaces with independent measurements where available, and evaluate errors near actual contact locations. A box-center metric can look acceptable while the surface that matters for a grasp is misplaced. I would also avoid treating agreement between appearance and reconstructed geometry as agreement between independent sensors. If both originate from the same image, some failures can be correlated. An apparently coherent representation may still be coherently wrong. For timing, I would record capture-to-command latency, not just transformer runtime. The log should distinguish image acquisition, tracking, grounding, representation construction, action generation, planner calls, and command transmission. I would inspect tail latency and observation age during failures, especially around target changes. For learning, I would begin with a small, controlled task family and preserve intermediate observations alongside demonstrations. That would allow action-learning changes to be compared against a fixed perception trace, then reevaluated with live perception. The two evaluations would help separate motor-learning improvements from changes in upstream stability. Finally, I would retain a separate representation for collision-relevant context. A non-target object can be irrelevant to task identity yet crucial to execution. My initial adaptation would pair detailed target geometry with a coarser workspace representation, then test whether the additional context improves obstacle handling without reintroducing dependence on decorative appearance. Three experiments that would change my confidence. The first is a matched-interface experiment. Give several controllers the same selected objects, the same perception outputs, and the same demonstrations. Compare full-scene input, object-organized input, and target-only input. Then vary the generator while holding the observation fixed. This would separate improvements due to selection, pretrained perception, and action architecture more cleanly than a comparison between complete systems with different interfaces. I would include deliberately imperfect prompts, not just correct ones. Small localization errors, ambiguous selections, and occasional missed detections are part of the operating regime an interactive controller must handle. The second is a matched-planner experiment. Use the same planner, history, and subtask decisions across controllers wherever their interfaces permit it. Also evaluate an oracle-target condition and a no-history condition. The goal would be to identify whether failures arise from remembering the task, selecting the object, grounding it metrically, or executing the interaction. This is especially important when interpreting multi-step performance. Improving the component responsible for choosing the next operation can produce a large system gain without changing the motor policy’s competence at any individual operation. The third is a representation-stress experiment. Hold target identity fixed while independently varying scale, pose, contact geometry, obstacle placement, and calibration. Measure perception error and manipulation outcome together. I would include cases where the target is accurately localized but successful motion requires a qualitatively different grasp or approach. The desired result is not merely a higher aggregate success rate. It is a failure map: conditions under which better grounding helps, conditions under which the learned motor repertoire becomes the bottleneck, and conditions under which selective perception removes information the robot needed. That map would tell me where to spend the next month of engineering. The takeaway. The question I would take from GAM is not whether every robot policy should replace its visual backbone with a detector. It is whether the relationship between task intent and physical entities should become an explicit, testable interface before action prediction. That is an attractive direction because it creates concrete intervention points. An operator can correct a selection. A planner can change a target. An engineer can inspect geometry. An evaluator can distinguish failures of reference, perception, and execution. But an inspectable interface also exposes responsibilities: uncertainty, identity, calibration, timing, missing context, and recovery. For professional roboticists, that is the productive reading. Treat grounded observations as a design hypothesis with measurable boundaries—not as a declaration that three-dimensional understanding has been solved. The opportunity is to make motor learning’s inputs more deliberate, and then determine exactly which forms of generalization that buys.