Latest / Embodied AI 101 / SIRModel and the Missing Geometry Layer in Vision–Language Manipulation

Transcript

SIRModel and the Missing Geometry Layer in Vision–Language Manipulation · Embodied AI 101

• SIRModel and the Missing Geometry Layer in Vision–Language Manipulation. SIRModel: Learning Spatial Intermediate Representation to Parameter-Efficiently Fine-Tune a Vision Language Model for Manipulation • Li Lin, Minghao Shi, and Tenglong Wang • Southeast University • AI (MDPI), Volume 7, Issue 9 • 2026. The Real Problem Is the Interface. A vision–language model can tell you that the red cup belongs on the upper shelf. A manipulation controller can move an end effector toward a target pose. The difficult part is everything between those two statements. The high-level instruction leaves most of the control problem unspecified. Which red cup? Where is its graspable surface in three-dimensional space? From which direction should the gripper approach? How much clearance is available? What pose will preserve stability during transport? And after the cup has been grasped, what visual evidence should convince the policy that it is time to transition from reaching to lifting?. This is the interface problem at the center of long-horizon manipulation. Semantic models operate comfortably with categories, relations, and intentions. Robots ultimately operate with coordinate frames, distances, orientations, contacts, and forces. A generalist manipulation policy must translate between these regimes repeatedly and in closed loop. SIRModel approaches that translation by inserting a learned spatial intermediate representation between vision–language reasoning and action prediction. Rather than requiring a large vision–language model, or VLM, to discover manipulation geometry implicitly while being fine-tuned end to end, the method gives spatial reasoning a dedicated representational role. At the same time, it uses parameter-efficient adaptation rather than updating the full pretrained model. That combination is the important idea. SIRModel is not simply arguing that robots need more three-dimensional information. Explicit 3D robot policies have made that case for years. Nor is it simply arguing for parameter-efficient fine-tuning, which is now routine in large-model adaptation. The paper’s more interesting claim is that the choice of intermediate representation determines whether a pretrained semantic model can be adapted efficiently for geometric control. A research note is necessary here. As of August 31, 2026, the DOI resolves to the article record, but the publisher’s machine-readable interface did not expose the full tables and implementation details reliably during research for this episode. I also did not find an indexed SIRModel code release among the authors’ public repositories. I will therefore not invent exact success rates, parameter counts, tensor dimensions, or hyperparameters. The useful focus is the paper’s verifiable architectural claim, the mechanism that could make it work, and the experiments needed to distinguish that mechanism from a larger or better-regularized baseline. Why Visual Tokens Are Not Already a Spatial Representation. It is tempting to argue that a modern VLM already contains spatial information. After all, its vision encoder processes patches with positional embeddings. Attention layers preserve enough image layout to answer questions such as whether an object is left of another object. Large multimodal models can localize objects, describe scenes, and sometimes emit bounding boxes or points. But this is not the kind of spatial understanding that manipulation requires. Image position is not metric position. A patch token at the center of an image does not tell the controller whether the corresponding surface is fifteen centimeters or one meter away. Apparent size is entangled with physical size and depth. Perspective projection makes identical displacements in the image correspond to very different Cartesian motions. Occlusion hides precisely the surfaces that frequently matter for grasping and placement. More fundamentally, vision–language pretraining rewards semantic invariance. A cup remains a cup under changes of viewpoint, scale, lighting, and background. That invariance is useful for recognition, but manipulation often needs the information that recognition is encouraged to ignore. A robot must care that the handle is rotated by thirty degrees, that the rim is partially occluded, or that the target lies just beyond the gripper’s collision-free workspace. The coordinate system also matters. A VLM sees image-relative relations. A policy acts in a robot base frame, end-effector frame, or camera frame. A successful system must either know the transformation among those frames or learn an equivalent mapping from demonstrations. If camera placement, robot embodiment, or workspace scale changes, an implicitly learned mapping may fail even while object recognition remains perfect. Long-horizon operation magnifies these mismatches. A modest geometric error during one atomic action can create the initial condition for the next action. A slightly unstable grasp changes the object’s pose during transport. A placement near the edge of a container makes a later retrieval harder. The resulting failures are not independent; geometry errors alter the state distribution encountered by the rest of the policy. The word spatial must therefore be used carefully. There are at least three levels of spatial competence:. Semantic spatial reasoning captures relations such as left, inside, above, and behind. Image-grounded localization identifies pixels, regions, masks, or image-plane trajectories. Interaction geometry expresses task-relevant positions, orientations, approach directions, clearances, and contacts in a form that can drive action. VLMs are increasingly strong at the first level and useful at the second. Manipulation needs the third. SIRModel’s intermediate representation is intended to make that third level a first-class part of the policy rather than an accidental property of generic hidden states. What SIRModel Changes. The most useful way to understand SIRModel is as a three-part system: a semantic substrate, a spatial bottleneck, and a motor interface. The semantic substrate is the pretrained VLM. It contributes object concepts, linguistic compositionality, visual recognition, and broad task priors. These are expensive capabilities to acquire from robot demonstrations, which are comparatively scarce and narrow. SIRModel therefore preserves the VLM as much as possible rather than relearning its representation through unrestricted behavioral cloning. The second component is the learned spatial intermediate representation, or SIR. Functionally, this representation must answer a question like: What task-relevant geometry should the action generator know at this moment?. That qualification—task-relevant—is crucial. A complete geometric reconstruction of the scene is unnecessary for many manipulation tasks and may even be counterproductive. To open a drawer, the policy needs the handle pose, a viable approach direction, the drawer’s motion axis, and enough obstacle context to avoid collision. It does not need a uniformly accurate model of every textureless wall and background object. A learned intermediate can concentrate capacity on precisely those relationships that predict successful interaction. Language conditions which geometry matters. “Place the block in the bowl” should emphasize the block, the bowl interior, the gripper, and the free path between them. “Move the bowl to the left of the plate” changes both the manipulated object and the relevant target relation, even if the image remains unchanged. The third part is the motor interface. Action prediction is conditioned on the spatial representation rather than being derived directly from undifferentiated VLM states. The intermediate therefore becomes a contract between perception and control. The upstream model must express actionable geometry; the downstream controller must convert that geometry, together with robot state and temporal context, into commands. This decomposition changes gradient flow. In a direct VLA policy, the action loss must simultaneously teach the model which objects matter, how language resolves references, how perspective maps into the robot frame, and how those estimates map into actions. Those signals are weakly separated. Because multiple visual interpretations can produce similar actions in the training set, the model may learn brittle shortcuts—camera-specific pixel coordinates, background correlations, or memorized trajectories. A spatial bottleneck makes the intended factorization more explicit. The VLM is responsible for semantics and visual context. The spatial pathway is responsible for task-conditioned geometry. The controller is responsible for embodiment-specific execution. The modules can still be trained jointly, but the architecture places a useful constraint on what information should cross their boundaries. This is also what differentiates a learned intermediate from a fixed one. Hand-designed intermediates can include object labels, subtask text, bounding boxes, segmentation masks, keypoints, affordance maps, or Cartesian waypoints. Their advantage is interpretability. Their disadvantage is that the designer must decide in advance what is sufficient. A bounding box is useful for identifying an object but poor at representing grasp orientation. A single keypoint may locate a handle but omit its axis. A textual subtask such as “approach the cup” specifies intention without distance or direction. A single deterministic waypoint sequence can suppress meaningful ambiguity: several grasps may be valid, and which one is safest may depend on uncertainty not represented by the waypoint. A learned representation can encode combinations of these variables without requiring a complete annotation schema. It can, in principle, preserve multiple geometric hypotheses or encode confidence. It can also adapt its content to the task. The price is reduced interpretability: calling a latent spatial does not prove that it is metrically calibrated. That makes representation validation central. A convincing SIR should support simple probes for quantities such as relative object position, gripper-to-target displacement, approach orientation, and task phase. It should transform predictably when the camera or scene is transformed. Intervening on the representation should create a corresponding change in behavior. Without these tests, a latent might improve performance while actually encoding task identity, demonstration index, or another shortcut. The conceptual success of SIRModel therefore depends on two properties being true at once:. The intermediate must be constrained enough to acquire interaction geometry. It must remain flexible enough to encode geometry that fixed labels would omit. That is a narrow design target, but it is exactly where many VLA architectures struggle. Why Parameter-Efficient Fine-Tuning Belongs in the Same Story. Parameter-efficient fine-tuning is sometimes described primarily as a cost-saving technique. That is only part of its relevance here. Robot datasets are small relative to the corpora used to pretrain VLMs. Full fine-tuning exposes billions of semantic and visual parameters to a comparatively narrow behavioral-cloning objective. The optimizer may improve action loss while degrading broad recognition, language grounding, or visual robustness. It may also overfit to a specific camera, workspace, embodiment, or set of object instances. A parameter-efficient update restricts how the model can change. The large backbone remains a stable semantic prior, while a compact set of trainable parameters learns the manipulation-specific transformation. In SIRModel, the learned spatial pathway gives that limited adaptation capacity somewhere productive to go. Rather than asking a small low-rank update to rewrite spatial reasoning throughout the entire VLM, the architecture offers a localized interface in which geometric information can be organized. This creates a useful form of inductive bias. The trainable delta does not need to turn a general VLM into a complete controller. It needs to expose the right information to the spatial intermediate and align that representation with the action generator. There is also a systems benefit. Freezing most of a VLM reduces optimizer-state memory, gradient storage, and communication during distributed training. It makes per-robot or per-workcell adaptation more plausible because one shared backbone can be paired with compact specialization modules. Different embodiments could share language and vision while maintaining separate spatial or control adapters. But parameter-efficient does not mean computationally small. The frozen backbone must still run during training and inference. Its activations still consume memory, and its latency may dominate the control loop. A large VLM with one percent trainable parameters is not equivalent to a small policy. SIRModel’s efficiency claim should therefore be interpreted as adaptation efficiency, not automatically as real-time or energy efficiency. The restriction can also become a liability. If the source VLM has discarded information essential for control, no small adapter can recover it. Severe domain shift—unusual sensors, fisheye optics, transparent objects, deformables, or industrial lighting—may require deeper visual adaptation. PEFT is most attractive when the backbone already sees the world well and the missing capability is the mapping from semantic vision into actionable geometry. How to Read the Empirical Claim. The key experimental question is not merely whether SIRModel completes more tasks than a baseline. The question is whether the learned spatial intermediate explains the improvement. A strong evaluation needs a matched direct-policy baseline using the same VLM, observations, action decoder, data, optimizer, and trainable-parameter budget. Otherwise, the result could come from extra capacity or a more favorable optimization path. A second baseline should use a fixed intermediate representation. That comparison asks whether learning the representation adds value beyond inserting any structured bottleneck. The most informative ablation ladder is therefore:. frozen or lightly adapted VLM followed directly by the policy head;. the same system with a fixed spatial target;. the same system with the learned SIR;. full fine-tuning, if computationally feasible. The comparison with full fine-tuning is particularly important. If SIRModel matches it with far fewer trainable parameters, the intermediate is functioning as an efficient interface. If it surpasses full fine-tuning, the likely explanation is regularization and factorization rather than raw representational capacity. If full fine-tuning remains substantially better, SIRModel may still be useful operationally, but the claim becomes a cost–accuracy trade-off. Long-horizon metrics should also be interpreted differently from single-skill success. Suppose a policy succeeds at each stage most of the time. Requiring five stages in sequence compounds the failure probability, even before accounting for distribution shift caused by imperfect earlier actions. A small gain in per-stage robustness can therefore produce a much larger gain in completed sequences. Conversely, an average sequence score does not reveal the mechanism. SIRModel might improve initial object grounding but leave contact-rich execution unchanged. It might reduce catastrophic early failures while offering little benefit to later stages. Per-skill results, failure taxonomies, and transition-level analysis are needed to see whether the spatial representation helps during reference resolution, approach, grasping, transport, placement, or recovery. Generalization tests are even more diagnostic. If the SIR truly captures interaction geometry, gains should persist under changes to object location, camera viewpoint, distractor arrangement, target receptacle, and instruction phrasing. Camera and embodiment changes are especially revealing because they challenge the distinction between image coordinates and metric relationships. One should also inspect spatial probes independently of action success. Can a lightweight decoder recover task-relevant relative poses from the SIR better than from the original VLM features? Does the representation respond consistently to controlled object translations? Does its geometry remain stable when language paraphrases preserve the same task? Those tests would directly support the paper’s causal story. Where SIRModel Sits Among Spatial Robot Policies. SIRModel occupies an interesting middle ground between end-to-end VLA policies and explicitly geometric robot-learning systems. At one end are policies that consume visual and language embeddings and learn the rest implicitly from demonstrations. These systems are architecturally simple and can scale well with data, but their geometric competence is difficult to inspect. Their success may depend heavily on consistent cameras and workspaces. At the other end are policies built directly around metric 3D observations. RISE, for example, demonstrates how point-cloud perception can simplify real-world imitation learning. Its public implementation emphasizes camera calibration, camera-to-robot transformations, and careful normalization of translation and workspace ranges. Those details illustrate the benefit and burden of explicit geometry: the representation has a clear metric meaning, but the sensing and calibration pipeline must be correct. SIRModel’s proposition is different. It aims to retain the semantic strength of a VLM while learning a manipulation-oriented spatial interface. This is attractive when instructions, object identity, and relational reasoning are as important as geometric precision. Semantic keypoint methods provide another nearby comparison. SKIL uses semantic keypoints to support data-efficient and cross-embodiment imitation, including long-horizon real-world tasks with relatively few demonstrations. Keypoints form a sparse, interpretable bridge between visual correspondence and control. The contrast with SIRModel is instructive. A keypoint says where a meaningful part is, but not necessarily everything the controller needs to know about free space, orientation, uncertainty, or contact. A learned SIR can be richer. On the other hand, a keypoint is auditable and can often be transferred across embodiments. A latent representation may silently bind itself to camera or robot-specific regularities. The best design may combine the two: keypoint or pose supervision can anchor a learned latent without requiring that the entire intermediate be hand specified. A contemporaneous, supervision-rich direction is RoboInter. Its public project describes more than 230,000 episodes across hundreds of scenes and multiple robot-arm types, with intermediate annotations spanning subtasks, primitive skills, segmentation, bounding boxes, placement proposals, affordances, grasp poses, motion traces, and contact points. It also explores both modular and end-to-end plan-then-execute policies. RoboInter represents the “annotate the missing middle” strategy. It seeks to expose many possible interfaces between planning and execution and to train models explicitly on them. SIRModel represents a more compact “learn the missing middle” strategy. The former offers interpretability and controllable supervision at substantial data-engineering cost. The latter potentially avoids deciding on ten different annotation types but must demonstrate that the learned representation contains the desired geometry. Predictive latent models form a fourth category. SLIM, for example, first learns action-grounded representations through inverse- and future-dynamics objectives, then trains a flow-matching control policy. Its latent is valuable because it predicts interaction-relevant temporal structure rather than merely reconstructing visual appearance. SIRModel and predictive-latent approaches address complementary dimensions. SIRModel emphasizes the semantic-to-spatial interface; predictive models emphasize how actions transform the scene. A natural extension would make the SIR both spatial and predictive: it should encode not only where objects and affordances are, but how candidate actions will change those relationships. That extension matters for contact. Static geometry can tell the robot how to approach a drawer handle, but opening the drawer requires anticipating constrained motion. Pouring, pushing, insertion, tool use, and deformable manipulation all depend on dynamics. Spatial representation is the right foundation, but long-horizon competence ultimately needs spatial-temporal interaction models. How I Would Adapt the SIRModel Principle. For a lab already running a VLA stack, the safest adaptation is to treat SIRModel as an interface-design pattern rather than a drop-in architecture. Start by deciding what failure you are trying to remove. If the policy confuses objects or instructions, the problem may still be semantic. If it reaches the correct object but misses the grasp, a spatial intermediate is promising. If it grasps correctly but fails during contact, the missing representation may be dynamic or tactile rather than purely spatial. Next, choose the weakest spatial supervision that rules out obvious shortcuts. Depending on the platform, that might be relative end-effector-to-object pose, object-centric keypoints, an SE(3) subgoal, task-space motion traces, contact locations, affordance fields, or future object displacement. The purpose is not necessarily to expose that target directly at inference. It is to force the intermediate to organize around interaction geometry during training. Coordinate frames must be explicit in the implementation, even if the representation itself is learned. Log camera intrinsics and extrinsics. Record which actions are expressed in the base, tool, or camera frame. Normalize translation and rotation separately. Apply spatial augmentation in a way that transforms observations, targets, and actions consistently. An augmentation that moves an object in the image without applying the corresponding change to the action target teaches precisely the wrong invariance. Likewise, ordinary image crops can alter the relationship between pixels and camera rays. Spatially grounded policies require geometry-aware augmentation rather than generic vision augmentation. The trainable architecture can remain compact. Keep the VLM largely frozen, insert a small set of spatial query tokens or a lightweight spatial encoder, and let those tokens attend to task-conditioned visual features. Add robot proprioception before action decoding rather than forcing the VLM to infer gripper pose from pixels. Adapt a few upper backbone layers only if the frozen features fail spatial probes. Training should proceed in stages. First, verify that the spatial pathway learns the intended variables without action prediction. Second, train the control head using the intermediate. Finally, allow limited joint adaptation so the VLM can expose information that the spatial pathway needs. This staging is not mandatory, but it makes representation failure easier to distinguish from policy-optimization failure. Diagnostics should be designed before the headline benchmark. Visualize nearest neighbors in SIR space. Decode relative poses. Perturb one object while holding language fixed. Paraphrase the instruction while holding geometry fixed. Swap backgrounds. Move the camera. Mask the target briefly. Each test asks whether the representation tracks the causal variables of interaction rather than superficial correlations. The controller should also be tested with corrupted intermediates. If small perturbations produce smooth changes in the trajectory, the interface is behaving geometrically. If behavior changes discontinuously, the SIR may function more like a discrete task identifier. If the controller ignores the SIR entirely, the downstream network has found a bypass through other inputs. For deployment, expose confidence or multiple spatial hypotheses when possible. Manipulation scenes are often ambiguous: two similar objects may satisfy the instruction, depth may be uncertain, or several grasp poses may be valid. A single deterministic latent encourages premature commitment. Sampling or ranking multiple spatial proposals allows the controller or a safety layer to reject unreachable and collision-prone options. Finally, benchmark parameter efficiency honestly. Report the total model size, trainable parameters, optimizer memory, wall-clock training cost, and closed-loop inference latency separately. A small trainable delta is valuable, but practitioners need to know whether the full backbone fits on their robot compute and whether it meets the control frequency required by the task. The Main Limitations to Watch. The first limitation is semantic inflation of the word spatial. A latent can correlate with geometric targets without becoming a transferable metric representation. If the training camera never moves, image position may be enough to solve the benchmark. Viewpoint randomization, calibration changes, and equivariance tests are essential. Second, the bottleneck may omit information. Spatial geometry is necessary for manipulation, but object compliance, friction, mass distribution, and contact state can be equally important. A rigid-object benchmark may overstate how far a visual spatial representation will transfer to cloth, cables, articulated objects, or tight insertion. Third, a learned intermediate can become an information bottleneck in the negative sense. If it is too small or too strongly constrained, it may discard subtle visual cues needed for recovery. The controller may need access to both the compact SIR and selected low-level visual features. Fourth, parameter-efficient adaptation preserves both the strengths and weaknesses of the backbone. If the VLM is poor at transparent objects, unusual viewpoints, or fine-grained state distinctions, freezing it protects those deficiencies. Spatial modules cannot manufacture evidence that the upstream encoder never preserved. Fifth, long-horizon behavior is not solved by geometry alone. The policy still needs temporal state, phase estimation, subgoal transitions, and recovery. A perfect grasp representation is of little help if the policy repeatedly closes an already closed gripper or cannot recognize that an object slipped during transport. Finally, representation improvements must survive the full robotics stack. Calibration drift, controller latency, actuator error, depth noise, and safety constraints can dominate a modest gain measured in simulation. Real-world evaluation should vary cameras and object arrangements rather than repeatedly resetting one carefully tuned workcell. The Takeaway. SIRModel’s most valuable contribution is a reminder that scaling the semantic model is not the only way to improve a VLA policy. Often, the decisive design choice is the interface between semantic understanding and physical execution. Visual tokens contain spatial clues, but clues are not a control representation. Textual plans express intent, but intent is not metric geometry. Fully explicit 3D pipelines provide metric structure, but they can be expensive to annotate, calibrate, and integrate with open-vocabulary reasoning. A learned spatial intermediate offers a compromise: preserve the pretrained VLM’s semantic competence, create a task-conditioned geometric bottleneck, and adapt the system through a small trainable path. That is especially appealing for teams with limited robot data or compute, provided the underlying VLM already captures the relevant visual evidence. The research question going forward is not simply whether intermediate representations help. It is which properties the intermediate must possess: metric calibration, equivariance, uncertainty, temporal prediction, embodiment invariance, or interpretability. SIRModel makes the case that learned spatial structure deserves to be one of those properties. For roboticists, the practical message is straightforward. When a policy understands the instruction but still behaves as though the world were made of pixels, do not immediately reach for a larger backbone. First ask whether the model has a well-designed place to represent the geometry of interaction.