Latest / Embodied AI 101 / SAGA: Ground the Affordance, Then Learn the Motion

Transcript

SAGA: Ground the Affordance, Then Learn the Motion · Embodied AI 101

• SAGA: Ground the Affordance, Then Learn the Motion. SAGA: Open-World Mobile Manipulation via Structured Affordance Grounding • Kuan Fang, Yuxin Chen, Xinghao Zhu, Farzad Niroui, Lingfeng Sun, Jiuguang Wang • RAI Institute • IEEE Robotics and Automation Letters, Volume 11, Issue 9 • 2026. A structured middle ground for generalist robots. Imagine asking a mobile manipulator to retrieve a snack from a shelf using a brush or duster. The robot may need to approach the shelf, step onto a low platform, grasp the tool by its handle, orient the functional end toward the snack, sweep the snack into a basket, and avoid knocking over nearby objects. The language instruction may mention only some of those requirements. The objects may be unfamiliar. Their colors, textures, positions, and geometry may differ from anything in the robot’s training set. This example captures the problem SAGA is trying to solve. Generalist mobile manipulation is not just a matter of recognizing nouns or replaying a motor skill. It requires binding multiple objects to different interaction roles, grounding those roles in three-dimensional geometry, and coordinating locomotion and manipulation as the scene changes. SAGA—Structured Affordance Grounding for Action—argues that the representation connecting semantics to control is the crucial design choice. Rather than feed a language embedding directly into a visuomotor policy, it first converts the task into a structured collection of affordance–entity pairs. It then grounds each pair spatially as a separate 3D heatmap and gives those maps to a learned diffusion policy. The resulting system occupies a useful middle ground between large end-to-end vision-language-action models and modular systems built around hand-engineered skills or motion planners. A bibliographic note is useful here. The public arXiv manuscript appeared on December 14, 2025. As of September 3, 2026, the DOI is assigned to the September 2026 issue of IEEE Robotics and Automation Letters, with pages 10361–10368. The first three authors are marked as equal contributors, and the work was conducted at the RAI Institute. Why language-to-action remains difficult. A language-conditioned policy must ordinarily solve several problems inside one model. It has to interpret the command, identify the referenced objects, determine the role of each object, understand their geometry, choose an appropriate interaction strategy, and finally produce embodiment-specific controls. That coupling is particularly problematic in mobile manipulation. The instruction “put the toy in the basket” may require an ordinary top-down pick. If the toy is on a shelf, the same semantic command may require horizontal grasping. If it is across the room, the robot must move its body before reaching. An end-to-end policy is therefore expected to learn that linguistic variation, object appearance, scene geometry, and embodiment dynamics are partly independent factors—even though its training data generally presents them as entangled trajectories. Scale-first systems such as π₀ attempt to cover this variation with large, heterogeneous robot datasets and a pre-trained vision-language backbone. That is a credible route to generality, but adaptation can remain difficult when the target embodiment or control interface is poorly represented in pretraining. At the opposite extreme, systems such as VoxPoser use language and vision models to build explicit spatial objectives for a model-based planner. These systems expose their reasoning more clearly but often depend on carefully designed perception APIs, cost functions, motion primitives, and geometric assumptions. SAGA’s proposal is to move the semantic-geometric boundary. Foundation models handle open-vocabulary recognition and instruction parsing. A learned robot policy handles contact-rich control and embodiment dynamics. Between them sits an explicit, typed spatial representation that says both where the relevant entities are and how each entity participates in the task. The authors organize the intended generalization along three axes: new environments and object instances, new task objectives or combinations of behaviors, and new forms of user specification. The representation is intended to absorb variation in the last two axes before that variation reaches the low-level policy. Tasks as affordance–entity pairs. SAGA represents a task using eight predefined affordance types. Each active affordance is paired with an embedding describing the associated entity:. | Affordance | Interaction role | |---|---| | | Entity held by the gripper | | | Destination for a grasped object | | | Functional part of a grasped tool | | | Entity contacted directly by the gripper | | | Entity contacted through a tool | | | Entity the robot should neither contact nor traverse | | | Entity the robot should approach | | | Entity on which the robot should place a foot |. These are not eight independent skills. They are typed slots that can be activated concurrently or sequentially. The distinction matters. A brush handle might occupy the channel, the brush head the channel, the snack the channel, and a basket the channel. A nearby laptop could simultaneously occupy the channel. This gives SAGA a kind of lightweight role binding. It does not merely tell the policy that a brush, snack, and basket are present. It tells the policy that the brush handle is to be grasped, the brush head is the active tool surface, the snack is the thing to be contacted through that surface, and the basket is the destination. That distinction is central to the method. A single binary mask covering every task-relevant object would erase the relationships among them. By keeping one heatmap channel per interaction role, SAGA can present geometrically similar objects to the policy while asking for different actions toward each one. Every task uses a fixed-size representation. If an affordance is irrelevant, its entity embedding is set to zero. SAGA also includes a compact motion embedding for information that cannot be expressed through object roles alone—for example, whether a sweeping motion should proceed from right to left. For a long-horizon instruction, a vision-language model decomposes the command into a sequence of subtasks, each with its own structured representation. It is worth being precise about the word “affordance” here. SAGA is not discovering an unlimited set of physical action possibilities from first principles. Its affordances are a manually selected ontology of interaction roles. Open-world variation primarily enters through the entity associated with each role: an unseen mug, brush, stair, drawer, vegetable, or basket can fill a familiar slot. SAGA therefore has an open object vocabulary but a largely closed interaction vocabulary. That constraint is also what gives the representation its power. The policy does not have to infer from an unconstrained text vector whether “red towel,” “cleaning cloth,” and “thing to wipe with” imply the same low-level relationship. Once grounding succeeds, those descriptions can collapse to approximately the same spatial pattern in the , , or contact channels. From semantic roles to 3D heatmaps. SAGA obtains entity embeddings from language or image inputs using a pre-trained multimodal encoder based on the CLIP family. It applies the same encoder densely to the robot’s visual observation, producing a semantic feature vector at each image location. The entity embedding is compared with every visual feature using cosine similarity. Locations whose visual features resemble the requested entity receive higher values. Repeating this process for each affordance produces a multi-channel image-space heatmap. The channel might highlight a tool handle, while highlights a basket and highlights a fragile object. Depth measurements are then used to lift the heatmaps into the robot’s point cloud, attaching an eight-dimensional affordance feature to each 3D point. This representation is continuous rather than a hard segmentation. That has two advantages. First, it can represent uncertainty and spatial extent rather than committing to one pixel or one object mask. Second, the downstream policy can become tolerant to the exact calibration of the semantic model. SAGA reinforces that tolerance during training by randomly rescaling and sharpening individual heatmap channels. The policy is thereby encouraged to use the topology and relative spatial structure of a map, rather than overfit to a particular similarity magnitude. Notice the division of labor. The visual encoder localizes the entity named in each slot, but it does not itself predict the final grasp pose or contact motion. A high score in the channel does not mean that every highlighted point is equally graspable. It means, roughly, “this is the thing whose role is to be grasped.” The learned visuomotor policy combines that semantic role with point-cloud geometry and prior demonstrations to decide where and how to act. That decomposition helps explain SAGA’s hoped-for data efficiency. If two language descriptions ground to similar heatmaps, the control policy sees them as nearly the same task. Conversely, the same object geometry placed in a different affordance channel carries a different operational meaning. The representation is invariant to some nuisance variation while preserving the distinctions that matter for control. Anchoring intent while the scene changes. A major design choice is that SAGA computes the affordance heatmaps only from the initial observation of a subtask. It does not rerun the heavy multimodal encoder at every control step. The motivation is partly computational, but also perceptual. At the beginning of an interaction, the scene often offers the cleanest view of the target entities. During a grasp or tool-use motion, the arm and gripper may occlude them. Recomputing semantic maps online could therefore cause the grounding to jump, disappear, or lock onto a distractor just when stable control is most important. SAGA instead uses a two-stream policy. One PointNet stream encodes the initial point cloud augmented with the fixed affordance heatmaps. This is the semantic anchor: what the task is about and where the relevant entities began. A second PointNet stream processes the current point cloud without the initial semantic markings. It captures the geometry now visible to the robot. The two streams are fused with the robot’s proprioceptive state and the task’s motion embedding. A diffusion head predicts a chunk of future actions rather than one action at a time. Chunking supplies short-horizon temporal consistency, while repeated inference keeps the system closed loop. The policy is expected to learn correspondence between the grounded initial scene and the evolving current scene through their shared geometry and common coordinate frame. This is an elegant compromise. The semantically expensive computation is performed once, while the faster geometric stream remains reactive. But it also moves a difficult problem into the policy: tracking task-relevant entities through motion and occlusion without an explicit object tracker, scene-flow model, or online heatmap update. Robot, observations, actions, and training data. The system is deployed on a Boston Dynamics Spot equipped with a six-degree-of-freedom arm and parallel-jaw gripper. One wrist camera and two forward-facing cameras provide calibrated RGB-D observations. Their point clouds and affordance maps are fused, cropped to a two-meter workspace, and uniformly downsampled to 1,024 points for real-time inference. The policy is shown operating at 5 hertz. The proprioceptive input is 19-dimensional, covering the torso pose, end-effector pose, and finger position. Actions are 21-dimensional task-space commands containing target poses for the torso and end effector, plus binary control flags. This means “whole-body control” should be understood at the level of coordinated torso and arm targets passed through Spot’s underlying control stack. SAGA is not directly learning raw torque commands for every arm and leg joint. Training uses conditional imitation learning on 2,410 teleoperated trajectories, amounting to roughly 1.3 million state–action pairs. Individual trajectories contain as many as 600 steps. Object instances, layouts, and configurations are varied during collection. Tasks include pulling a cart, poking with a shovel, closing a laptop, grasping and placing, and other combinations of the eight affordances. The data distribution is uneven. About 71.5% of trajectories contain , 61.1% contain , and 32.5% contain. Direct contact appears in 19%, indirect contact and tool function each in 18.7%, while and occur in only 8.3% and 7.9%. This imbalance makes the results on stepping and novel avoidance combinations particularly interesting, although the paper does not isolate how much each data subset contributes. Every training trajectory is associated with annotated affordances and text descriptions from which the structured representation is constructed. SAGA therefore reduces the quantity of robot data required by injecting more supervision into each trajectory. It should not be interpreted as annotation-free learning: the representation transfers some burden from raw trajectory scale to ontology design and structured labeling. One policy, three interfaces. A strong aspect of SAGA is that the control policy does not care how its affordance heatmaps were specified. Language, points, and demonstrations all feed the same internal interface. For language commands, a GPT-4-based vision-language model decomposes a long instruction into subtasks and extracts descriptions of the target entities and desired motion. The multimodal encoder converts those descriptions into the entity and motion embeddings used for grounding. This gives SAGA open-vocabulary recognition without requiring the robot policy itself to parse arbitrary language. For point input, the user selects an image location for each relevant affordance. SAGA takes the visual embedding around that point—averaged over a three-by-three neighborhood—as the entity embedding. The user does not need to click the exact future grasp or contact pose. A click merely identifies the entity or region that should fill a particular role; the policy still determines the detailed motion. The demonstration interface is more novel. Given a few trajectories but no instruction, SAGA freezes the entire visuomotor policy and optimizes only the latent task representation. It searches for entity and motion embeddings that make the demonstrated action chunks likely under the frozen policy. Because the embedding-to-heatmap-to-action pipeline is differentiable, gradients can flow all the way back into those latent task vectors. The authors call this heatmap tuning, drawing an analogy to soft-prompt tuning in language models. Instead of modifying millions of policy parameters, adaptation changes the compact conditioning representation until it highlights scene regions that elicit the demonstrated behavior. For the reported experiments, this optimization uses a relatively high learning rate of (10^{-3}) to converge quickly. Conceptually, this may be SAGA’s most reusable contribution. It treats a robot task representation not merely as an input supplied by a user, but as a small, optimizable program for a frozen policy. What the real-robot experiments show. The zero-shot evaluation contains nine subtasks arranged into three nominally long-horizon scenarios. The environments resemble household, office, and retail spaces and contain unseen furniture and object instances. The first scenario requires walking to a table, pushing a basket, and sweeping a snack into it with a brush. The second requires approaching a shelf, stepping onto a stair, horizontally picking an object from the shelf and placing it in a basket, and then lifting that basket. The third tests mobile pick-and-place, tabletop pick-and-place, and closing a drawer. SAGA is compared with three zero-shot baselines. DP3 is converted into a multitask policy conditioned on a language embedding. CodeDiffuser receives VLM-extracted entity descriptions and binary point-cloud masks. π₀ is fine-tuned on the SAGA dataset, with depth added to its input. The authors use the same training and evaluation protocols across methods. The paper reports ten trials per subtask through a bar plot rather than a numerical table. Reading those ten-trial bars gives the following results:. | Subtask | SAGA–Language | SAGA–Point | Best baseline | |---|---:|---:|---:| | Walk to table | 100% | 100% | 0% | | Push basket | 90% | 100% | 30% | | Sweep snack | 80% | 100% | 20% | | Step onto stair | 100% | 100% | 80% | | Horizontal pick-and-place | 80% | 80% | 40% | | Lift basket | 90% | 100% | 0% | | Mobile pick-and-place | 70% | 70% | 0% | | Tabletop pick-and-place | 70% | 100% | 30% | | Close drawer | 100% | 100% | 30% |. A simple unweighted average of the plotted rates is approximately 86.7% for SAGA–Language and 94.4% for SAGA–Point. The same calculation gives about 18.9% for DP3, 16.7% for CodeDiffuser, and 4.4% for π₀. Those aggregate values are my calculation from the plot, not an additional metric tabulated by the authors. Point conditioning performs somewhat better because it removes semantic ambiguity. If several objects plausibly match a description, a click directly selects the intended region. Language grounding is nevertheless strong across tasks with unseen object instances, tool use, mobile reaching, and multiple simultaneous affordance types. The shared policy also changes its geometry-dependent behavior without changing its high-level affordance set. The same and channels support top-down tabletop picking, horizontal shelf picking, and picking that first requires body motion. This is an important demonstration that the affordance vocabulary does not simply select one memorized primitive per channel. The baselines exhibit different weaknesses. DP3 and CodeDiffuser often approach the wrong region or make unstable contact. The authors argue that CodeDiffuser’s binary-mask representation, as implemented in this comparison, identifies relevant objects but does not preserve the distinct role assigned to each object. π₀ occasionally succeeds on pushing and closing but otherwise collapses toward a small set of action modes. The paper attributes this to the Spot embodiment being absent from π₀’s pretraining and the SAGA fine-tuning data representing less than 0.2% of π₀’s original training corpus. That explanation is plausible, but the experiment is better viewed as evidence about low-data adaptation to this particular embodiment than as a general verdict on large VLAs. Few-shot heatmap tuning. The few-shot tests use ten demonstrations for two new problems. “Sort vegetables” is an object-level adaptation: familiar and roles are applied to unseen entities. “Clean-avoid” is a task-level adaptation requiring the previously unseen combination of grasping a towel, contacting a table through it, and avoiding a plate. Heatmap tuning reaches 60% success on both tasks. The zero-shot SAGA policy supplied with the ground-truth language instruction reaches 20% on vegetable sorting and 40% on clean-avoid. DP3 and π₀ remain at zero in the plotted results; SKIL reaches zero on vegetable sorting and 30% on clean-avoid. The notable result is not just that ten demonstrations help. It is that adapting the conditioning representation outperforms modifying or retraining the action policy under the same low-data regime. The learned latent vectors produce heatmaps resembling those generated from language or point prompts while encoding task details that the instruction-conditioned representation missed. Qualitatively, the adapted maps highlight the vegetables and destination basket in one task, and distinguish the towel, table, and plate roles in the other. How SAGA relates to adjacent approaches. One useful way to interpret SAGA is as an evolution of ideas seen in MOKA, an earlier project involving Kuan Fang. MOKA asks a vision-language model to reason about marked images and produce point-based affordances and waypoints for open-world tabletop manipulation. SAGA retains the idea that affordances form a useful bridge between VLM reasoning and robot action, but replaces sparse, planner-oriented keypoints with dense role-specific heatmaps consumed by a learned whole-body policy. VoxPoser similarly grounds language as composable 3D value maps, then uses those maps inside a model-based trajectory optimizer. SAGA’s heatmaps resemble value maps visually, but their function is different. They are policy-conditioning features rather than complete reward or cost fields. The diffusion policy learns how a , , or contact map should translate into embodiment-specific motion. F3RM, or Feature Fields for Robotic Manipulation, also combines CLIP-style semantics with 3D geometry. Its principal focus is few-shot 6-DoF grasping and placing through distilled 3D feature fields. SAGA broadens that design space from semantic object designation to a typed collection of mobile-manipulation roles, including tool function, indirect contact, collision avoidance, approach, and stepping. At the policy level, SAGA inherits the strongest lesson of DP3: sparse point clouds can be an efficient and highly generalizable observation representation for diffusion-based control. SAGA’s contribution is not a replacement for that 3D policy machinery. It adds a second, semantically grounded stream that tells the geometric policy which parts of the cloud matter and in what capacity. CodeDiffuser is perhaps the closest conceptual neighbor. It also separates language interpretation from low-level control and uses VLM-generated code to create 3D attention maps. The difference is representational emphasis. CodeDiffuser builds task-specific attention through executable perception programs, while SAGA standardizes the interface around a fixed set of role-separated affordance channels. The former favors programmable semantic disambiguation; the latter favors reuse across specification modalities and policy-level composition. SKIL uses foundation-model-derived semantic keypoints to obtain data-efficient imitation and cross-object generalization. SAGA replaces a sparse keypoint abstraction with dense heatmaps and, in its adaptation mode, tunes the task representation rather than the policy. The comparison suggests a broader emerging pattern: data-efficient robot adaptation may be easier in a compact spatial conditioning space than in the full parameter space of the controller. What the paper does not yet establish. The first limitation is the fixed ontology. Eight affordances cover a surprising amount of single-arm mobile manipulation, but they do not obviously capture bimanual coordination, dexterous in-hand manipulation, articulated constraints, pouring, wrapping, handovers, or force-sensitive assembly. The authors explicitly identify bimanual and humanoid interaction as cases that will require richer, possibly learned affordance types. Second, semantic similarity is not physical understanding. A CLIP-like encoder may localize a towel or drawer handle, but it does not necessarily know friction, compliance, mass distribution, grasp stability, or whether a surface is wet or fragile. SAGA’s policy must learn those properties indirectly from demonstrations and geometry. The authors note that deformable objects and tasks depending on subtle material properties may require retaining additional visual cues beyond the affordance-informed point cloud. Third, the decision to freeze the initial heatmap creates a known failure mode. If objects move far from their initial positions, leave the field of view, become confused with similar objects, or enter the scene after grounding, the current geometry stream must recover correspondence without renewed semantics. The authors identify online affordance updating and better correspondence estimation as future work. The project page’s qualitative failure section also points to inaccurate motions and cases where the policy ignores a requested affordance. Fourth, “open-world mobile manipulation” should not be confused with unrestricted autonomous operation in a building. The perception input is cropped to a local two-meter workspace, and the evaluation consists of controlled household-, office-, and retail-like scenes. There is no demonstrated global semantic mapping, exploration, lifelong memory, or room-scale object search. Long-horizon instructions are decomposed into three-stage scenarios, but the quantitative graph reports success separately for each subtask rather than full-chain completion rates. Fifth, the statistical evaluation is small. Ten trials per zero-shot task make the large margins easy to see, but one success corresponds to ten percentage points. The plot gives no confidence intervals. The paper also contains no extensive controlled ablation separating the benefits of role-specific channels, heatmap augmentation, the initial-versus-current two-stream design, motion embeddings, and diffusion action chunking. The full system clearly works better than the chosen baselines, but the individual causal contribution of each design choice remains less certain. The baseline results also require context. DP3 was originally presented as a task-specific, data-efficient policy, not as a general language-conditioned controller. CodeDiffuser was designed around language ambiguity in mostly arm-centric manipulation. π₀’s principal advantage comes from cross-robot pretraining, yet the target Spot control scheme was outside its pretraining distribution. SAGA’s comparison convincingly shows that structure is valuable in this low-data Spot setting; it does not prove that the approach will dominate sufficiently scaled and appropriately pretrained VLAs across embodiments. Finally, the data-efficiency claim has a real but qualified meaning. A corpus of 2,410 teleoperated trajectories is far smaller than the datasets behind generalist VLAs, but it is still a substantial real-robot collection. It also comes with structured affordance and entity annotations. SAGA reduces the need to observe every combination of wording, appearance, and behavior by providing the right factorization; it does not eliminate the need for varied physical interaction data. The broader significance. SAGA’s strongest idea is not simply “use heatmaps.” Robotics has used masks, keypoints, value maps, descriptor fields, and attention volumes for years. The stronger contribution is a typed, spatial, optimizable interface between foundation-model semantics and learned control. That interface has several desirable properties. It canonicalizes paraphrases and object appearances before they reach the policy. It preserves the different roles of multiple entities. It can be inspected visually when the system fails. It permits a human to correct language grounding with a point. And it can be inferred from demonstrations by optimizing a small latent representation instead of retraining the robot’s motor system. For practitioners, the approach is most attractive when a domain contains substantial geometric variation but its tasks can be described with a modest vocabulary of recurring interaction roles. A warehouse robot might use slots for grasping, placement, pushing, restricted regions, tool surfaces, and approach targets. A service robot might add opening, support, handover, and human-contact constraints. The important design question would be whether those roles form a stable abstraction across the demonstrations one wants to reuse. A practical SAGA-style system would require calibrated RGB-D sensing, dense semantic visual features, an affordance ontology, structured annotations for a diverse demonstration corpus, and a reliable low-level controller beneath the learned task-space policy. Extensions could replace the fixed ontology with learned or hierarchical roles, add uncertainty-aware online heatmap updates, incorporate tactile and force features, and condition one policy on both affordances and embodiment. A scene-graph or symbolic planner could sit above the representation to manage longer task sequences, while SAGA-like control handles each physically grounded stage. Final assessment. SAGA presents a compelling argument that generalist robot learning should not be framed solely as a competition between larger end-to-end models and increasingly elaborate modular planners. There is a third option: deliberately design the intermediate representation so that semantics and control meet in a form both sides can use. Its empirical case is strong within the scope tested. One policy controls a legged mobile manipulator across eleven real-world tasks, supports language and point prompts without task-specific fine-tuning, and adapts from ten unlabeled demonstrations by tuning only its latent task representation. On the nine zero-shot subtasks, its plotted success rates are dramatically higher than the DP3, CodeDiffuser, and π₀ variants used for comparison. The remaining questions concern scale: whether eight manually chosen affordances can grow into a richer interaction language, whether initial grounding can remain reliable in dynamic scenes, whether the method transfers across embodiments, and whether the gains survive larger and less curated task distributions. Even with those caveats, SAGA offers a valuable design principle for embodied AI: before asking a policy to learn every relationship between words, pixels, geometry, and actions, give it a spatial language of intent.