Latest / Embodied AI 101 / SLA: Does 3D Pretraining Improve pi0.5 Robot Manipulation?
Transcript
- SLA: Does 3D Pretraining Improve pi0.5 Robot Manipulation?. SLA: Scaling Spatial-Language-Action Model via Continual Pre-training • Fuhao Li, Wenxuan Song et al. • HKUST (Guangzhou), OLA Dimensions, BAAI, Tsinghua University • arXiv • 2026. Cold Open. Picture a robot reaching for a piece of fruit. A person moves it just before the gripper arrives. The robot must update its reach, secure the fruit, pass it to its other arm, and finish placing it in a bowl. The instruction has not changed. The geometry has. For a manipulation policy, that interruption asks a revealing question: is the next action grounded in the current scene, or in the trajectory that usually works?. SLA approaches that question through training. The HKUST Guangzhou-led team starts with Physical Intelligence’s π0.5 and adds supervision from a frozen spatial perception model. The deployed policy keeps the geometric teacher outside its inference path. That creates an interesting engineering proposition. Instead of making the robot run another perception system whenever it moves, teach its existing visual representations to carry more useful spatial information. Where should that learning happen: during adaptation to one task, or earlier, across a broad collection of robot experience?. The Idea. The starting point is an existing VLA. In π0.5, a vision-language backbone processes observations and instructions, while a flow-matching action expert produces continuous action chunks. SLA retains that machinery and adds a representation-learning objective alongside action prediction. Its implementation captures intermediate visual features from the language backbone and supervises them through an auxiliary projector. Think of this as asking the policy to satisfy two requirements simultaneously. It must predict the demonstrated movement. It must also organize its visual features so they resemble features from a geometry-specialized teacher. The supervision targets internal representations rather than requiring the policy to generate a reconstruction as an intermediate control output. The released teacher integration extracts features with its reconstruction heads disabled. That teacher is OmniVGGT. Its important property is flexibility: it can work from RGB images alone or incorporate available depth and camera calibration. Its training exposes it to different combinations of geometric inputs, allowing it to use richer annotations without requiring every example to contain them. This suits a robot-data mixture assembled from different collection systems. The alignment is spatially organized. Teacher features are resized to correspond to the policy’s visual-token grid, with positional information retained. The objective is therefore more specific than matching a single global scene descriptor: it supervises features associated with corresponding image regions. SLA also learns which transformer layers should contribute. A projector maps visual embeddings into the teacher’s feature space, and content-dependent gates combine projected features across layers. This gives the alignment objective access to several representation depths rather than committing to one fixed layer. These gates belong to the training branch, not a new action-selection mechanism. Another ingredient is state dropout. On selected training samples, the robot-state segment is omitted from the tokenized input while the instruction remains. The released configuration makes this a probabilistic training choice. The intended effect is greater dependence on visual observations: proprioception should help control the robot, rather than simply identify which demonstrated movement comes next. The training schedule gives these components their broader purpose. SLA applies alignment during continual pretraining and again during downstream adaptation. Its corpus draws from six datasets, including DROID, RoboCOIN, AgiBot World, InternData-A1, MolmoAct, and RoboCasa365. A unified processing pipeline reconciles their action conventions and geometric annotations. The direct predecessor is Spatial Forcing, which already aligned intermediate VLA representations with a frozen 3D model during fine-tuning. That work established the teacher–student mechanism and evaluated it with OpenVLA and π0. SLA advances the training strategy: develop spatially supervised representations across heterogeneous robot experience, then reinforce them during specialization. That distinction is the central idea. A task adaptation stage starts from a representation already shaped by geometric supervision. The auxiliary machinery teaches the policy; it does not become another subsystem the deployed controller must consult. The Result. The headline concerns moving-fruit manipulation: success rises from 42% for π0.5 to 76% for SLA. On the bimanual Piper platform, each method receives fifty single-object evaluation rollouts. Success requires the complete grasp, handover, and placement sequence. Policies are post-trained on one hundred base-scene demonstrations, without additional moving-target demonstrations. That protocol asks whether a learned skill survives an intervention. The robot already knows the requested sequence. What changes is the relationship between its gripper and the object while execution is underway. Completing the sequence requires the policy to respond to fresh visual evidence and then continue toward the original goal. Spatial Forcing finishes between π0.5 and SLA on this condition. That gives the comparison a useful progression: the starting policy, the earlier spatial-alignment approach, and the expanded training recipe. The practical question becomes what happens when spatial supervision moves earlier in the learning process. Simulation supplies a broader stress test. RoboTwin 2.0 evaluates bimanual manipulation with structured variation in clutter, lighting, backgrounds, tabletop height, and language. Its Clean2Random track adapts policies using clean demonstrations and tests them under randomized conditions. The evaluation therefore asks whether familiar manipulation skills remain usable when their surroundings change. SLA’s useful ablation independently switches alignment on or off during continual pretraining and post-training. Across the tested data scales, applying it during both stages gives the strongest RoboTwin Clean2Random results. That experiment addresses a more interesting question than whether additional robot demonstrations help. It asks whether additional experience becomes more useful when the representation is supervised differently. The observed ordering supports treating spatial learning as part of the foundation-building process and preserving it during specialization. The benchmark selection also matters. LIBERO-Plus systematically perturbs camera viewpoints, robot initial states, object layouts, instructions, and visual conditions. Its design separates robustness across these changes instead of collapsing everything into success under one familiar configuration. These evaluation designs are complementary. A moving target asks for responsiveness during execution. Randomized environments ask for robustness across episodes. Together, they provide a useful way to investigate whether a representation supports action beyond the exact visual conditions of the demonstrations. The Biggest Weakness. The real-world headline does not isolate the contribution of 3D supervision. The hardware table lacks a control matching SLA’s continual pretraining, state dropout, and action processing while disabling only geometric alignment. The paper’s RoboTwin ablation also shows a substantial performance loss when state dropout is removed from the same continually pretrained starting point. That makes causal attribution the strongest objection. A moving-fruit task is exactly where encouraging visual dependence could help. A policy that pays more attention to its current images may become more responsive even before we establish which geometric information its features contain. State dropout is therefore a plausible contributor to the improvement, not evidence that the spatial teacher is irrelevant. The simulation alignment ablation is meaningful evidence against the explanation that the gains come merely from additional data. The remaining question is whether the same separation holds for the physical result attracting attention. Winning with a complete training package establishes which system performs better. It does not assign the improvement to individual ingredients. The decisive hardware experiment would keep the starting checkpoint, pretraining mixture, training budget, action representation, and state-dropout rule fixed. Train one policy with geometric alignment and one without it. Adapt both on identical demonstrations, then evaluate them under the same moving-target protocol. Matching downstream demonstration counts alone does not match the models’ learning histories. For practitioners, this distinction determines what to adopt. Should you invest in a spatial teacher, or does a substantial share of the benefit come from strengthening visual dependence while learning from broader experience? Those possibilities motivate different changes to an existing training pipeline. The missing comparison is valuable because it would turn a successful combined recipe into a more portable engineering principle: a measured contribution from geometric supervision under otherwise matched physical-task training. Verdict. If you maintain a π0.5-based manipulation stack, SLA deserves attention. Its engineering appeal is the decision to improve representations already used by the controller. The intervention happens where the policy learns what information to preserve, rather than through another decision-making module placed between perception and action. There is a concrete implementation to inspect. The official repository documents loading an SLA pretrained checkpoint, computing normalization statistics, running alignment training, and serving the resulting policy through an OpenPI-based workflow. It also points to benchmark checkpoints and real-world demonstration data. That gives a lab a starting point for experiments beyond reproducing the appearance of a demo. For foundation-model teams, the useful principle is that training supervision and deployment inputs can differ. Physical Intelligence’s π0.5 work uses heterogeneous robot experience and semantic supervision to develop a reusable policy. OmniVGGT supplies a complementary mechanism for using heterogeneous geometric annotations. Together, these approaches suggest treating every available annotation as a potential learning signal, rather than requiring it to become a permanent sensor dependency. That is the practical direction I would pursue. A robot does not need to carry every model that helped train it. Spatial teachers can belong to training infrastructure, alongside demonstration curation and augmentation. Their value is in the behavior they leave behind. For representation researchers, the interesting question is when useful spatial structure should enter the policy. A shared pretrained representation offers a place to learn something once and reuse it across later adaptations. This shifts attention from designing another input pathway to designing better supervision and a better training schedule. For manipulation engineers, it suggests a focused research bet: inspect what the visual tokens are being asked to encode, not just how much data the action head receives. Takeaway: make geometry a training objective, then let the existing policy carry it into manipulation. Outro. Return to that piece of fruit. The robot has learned a skill; now the world changes halfway through using it. Its next action must respond to the scene in front of it. SLA puts representation learning at the center of that moment. Watch whether training-only spatial teachers become a reusable ingredient across other policy backbones and robot embodiments. The appealing destination is straightforward: richer physical understanding in the weights, with a familiar policy at the controls. Better manipulation can begin with changing what the model learns to see. This is Embodied AI 101.