Latest / Embodied AI 101 / BridgeVLA++: Keep the Geometry, Give the Robot a Memory

Transcript

BridgeVLA++: Keep the Geometry, Give the Robot a Memory · Embodied AI 101

• BridgeVLA++: Keep the Geometry, Give the Robot a Memory. BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation • Peiyan Li, Yuze Zhu et al. • Institute of Automation, CAS / UCAS, with collaborators from ByteDance Seed, FiveAges, and Nanjing University • arXiv preprint • 2026. Two different reasons a robot can lose its place. Imagine a robot approaching a receptacle with an object in its gripper. The target was clearly visible a moment ago. Now the robot’s own hand blocks the camera. The policy knows what it wants to do, but its current observation no longer contains the geometry needed to finish accurately. Now imagine a different failure. A robot must repeat an operation several times before moving on. After each repetition, the scene looks almost identical. Localization is not the problem. The policy needs to know which stage of the task it has reached. These are different information deficits. One concerns previously visible geometry; the other concerns interaction history. My preferred way to read BridgeVLA++ is as an attempt to address both without abandoning a spatially structured action interface. First, a source clarification. The supplied social-media link was not directly accessible during research, and Léo Kharon is not among the paper’s listed authors. BridgeVLA++ is by Peiyan Li, Yuze Zhu, and colleagues. Its preprint was submitted on August 5, 2026, extending the NeurIPS 2025 BridgeVLA work. The headline 95.4% success with three demonstrations per task belongs to the original, memory-free BridgeVLA, rather than being the defining new result of the memory extension. That distinction gives us the right structure for this episode. BridgeVLA asks how to connect a pretrained vision-language model to precise spatial actions. BridgeVLA++ asks how to retain the information that a current-frame policy inevitably loses. The important idea is not that a sufficiently large VLM can somehow infer everything from ordinary pictures. It is that geometry, output representation, and memory should be designed together. “2D views only” describes the neural interface, not the sensing. The first misconception to remove is that this is an RGB-only manipulation system. BridgeVLA reconstructs a colored point cloud from calibrated RGB-D observations, then renders that cloud into three orthographic images: top, front, and right. The VLM receives those rendered images and the language instruction. Depth and calibration remain essential upstream, even though the backbone itself operates on images. This is a useful architectural compromise. The robot does not ask a pretrained image model to consume an unordered point set. It uses an explicit geometric transformation to express the scene in an image-like representation. Consider what that transformation buys you. In an ordinary perspective image, image scale depends on depth. Under orthographic projection, position in the rendering has a simpler relationship to position in the workspace. Multiple views expose complementary coordinate relationships. A helpful mental model is an engineering drawing. A top view tells you where something lies across the table; a side view helps resolve height. Each drawing is two-dimensional, but the collection is a representation of a three-dimensional object or scene. The direct predecessor here is RVT-2, not a generic chatbot controlling a robot. RVT-2 develops a data-efficient, precision-oriented manipulation system around rendered views, with architectural and system improvements over RVT. BridgeVLA’s contribution should be understood against that geometric-policy lineage, rather than as the invention of learning manipulation through virtual camera views. There is an important limitation to the engineering-drawing analogy, though. These renderings are made from what the physical cameras observed. Rotating a partial point cloud does not reveal the true back surface of an unseen object. It only changes how existing measurements are displayed. That observation suggests a practical distinction I would preserve throughout deployment: view normalization is not scene completion. Rendering can organize available evidence; it cannot repair missing depth measurements by itself. It also suggests where to debug first. If a target shifts when you change cameras, inspect calibration and coordinate conventions before changing the transformer. If a thin object disappears in a projection, inspect point density and rendering settings before collecting another hundred demonstrations. In this architecture, preprocessing is part of the policy’s spatial reasoning machinery—not a disposable input adapter. Making PaliGemma predict locations instead of sentences. The backbone is PaliGemma, whose original architecture combines a SigLIP vision encoder with a Gemma language model. It was developed as a transferable vision-language model, not as a robot controller with an intrinsic Cartesian action space. BridgeVLA changes how the backbone’s output is used. Rather than reading a generated action sentence, the implementation extracts the final image-token representations and restores their patch-grid organization. For the 224-pixel backbone, each view contributes a 16-by-16 grid of 2,048-dimensional tokens. Those spatial features feed a heatmap decoder. That is more consequential than attaching another prediction head to a pooled embedding. Pooling asks the network to compress the scene and then reconstruct where an action belongs. Preserving the patch grid retains an explicit connection between a feature and the region of the rendered workspace it describes. The action interface can therefore remain spatial all the way through decoding. But an off-the-shelf VLM has not necessarily learned to express language-conditioned localization through that interface. BridgeVLA adds a grounding pretraining stage using the 120,000-example object-detection portion of RoboPoint. Images and object descriptions become heatmap-learning examples before robot fine-tuning begins. The targets place probability around annotated object centers. In the default implementation, these are Gaussian-derived spatial targets trained with a soft-label cross-entropy objective. The released code also contains alternative loss experiments, but those should not be confused with the paper’s default heatmap recipe. A useful interpretation is that pretraining teaches the network a particular answer format: given language and an image, distribute probability over relevant locations. Robot fine-tuning subsequently changes the meaning of “relevant location” from an object center to a demonstrated end-effector waypoint. Those are not identical problems. A grasp approach point may float above an object. A placement waypoint may lie inside empty space. Nevertheless, both require language-conditioned spatial selection, making the transfer interface much more consistent than switching from visual question answering directly to an unrelated vector-regression head. The decoder uses learned convex upsampling. Instead of applying fixed interpolation to a coarse feature grid, it learns how neighboring predictions should contribute to the higher-resolution output. The implementation inherits this component from the RVT code lineage. I would interpret this as spending model capacity on where the answer should be, rather than asking the language-generation interface to encode a coordinate convention efficiently. It also reframes the few-demonstration claim. The robot supervision is scarce; the model’s prior experience is not. Foundation-model pretraining and additional grounding supervision have already supplied a substantial visual-language prior. “Three demonstrations” is a statement about downstream robot-label efficiency, not learning manipulation from three trajectories and nothing else. From heatmaps to an executable action. The translation decoder does not simply take three unrelated heatmap maxima and hope they intersect. It evaluates candidate locations in the three-dimensional workspace. Each candidate is projected into every view, and the corresponding heatmap values are combined. The highest-scoring candidate becomes the predicted translation. This makes the known projection geometry part of action decoding. The distinction matters. A bright pixel in one view corresponds to a set of possible three-dimensional positions, not a complete position. Agreement across views supplies the additional constraints. This representation also offers useful diagnostics. If the views disagree, you can inspect whether the error originates in language grounding, occlusion, or geometry. A single incorrect Cartesian vector does not expose those intermediate disagreements as directly. There is still a representational tradeoff. My concern would be ambiguous, multimodal scenes: several heatmap peaks could support multiple plausible three-dimensional candidates. Structured decoding reduces the search problem, but it does not automatically guarantee that the selected combination represents a valid grasp or the intended object. That deserves explicit testing in repeated-object and cluttered scenes. BridgeVLA then applies coarse-to-fine refinement. A first pass locates an approximate target over the workspace. The point cloud is cropped and rescaled around that waypoint, and a second pass refines the translation. The backbone is shared across these passes. For intuition, take a hypothetical 224-pixel rendering spanning 1.12 meters. Each pixel spans five millimeters. If the same resolution instead covers a 28-centimeter crop, the spatial interval falls to 1.25 millimeters. Those are illustrative numbers, not a reported accuracy guarantee. They explain why zooming can help without enlarging the VLM. Translation is only part of the action. Global image features and features near the predicted waypoint feed an MLP for orientation, gripper state, and, where applicable, a motion-planner collision flag. Heatmaps do not independently recover the entire six-degree-of-freedom pose. The extended formulation uses a continuous six-number representation for rotation. Here, “six-number rotation representation” must not be confused with “six-degree-of-freedom action”: rotation still has three physical degrees of freedom. The representation is converted into an orthonormal rotation matrix, following the rotation-continuity work of Zhou and colleagues. Finally, this is a keyframe policy. Its output specifies a target configuration; a motion planner or lower-level controller handles execution between decisions. That division of labor is fundamental when comparing it with models that generate dense action sequences. Memory is attached to the decisions that need it. BridgeVLA++ preserves this action interface. Its memory operates on the visual-token representations before the existing heads decode actions. The implementation groups memory into coarse-stage temporal processing and fine-stage spatial-reference processing. That placement gives the architecture a useful division of responsibility. The coarse stage decides which region or subtask matters next. The fine stage resolves the location precisely enough to act. Memory is therefore not treated as a single undifferentiated archive. Temporal memory: retaining task progress. Temporal context combines an initial scene reference with recent observations and, when enabled, selected milestone observations. The memory bank is episodic: it accumulates encoded visual information during a rollout, rather than serving as a permanent database across unrelated tasks. The initial reference answers a comparative question: what has changed since the episode began? Recent observations answer a local question: what transition just occurred? Milestones answer a longer-horizon question: which meaningful stages have already been completed?. These roles are complementary. A before-and-after comparison may reveal that a drawer is closed, but not how many times an operation inside it was attempted. Recent history may identify the latest transition while losing an early task-relevant observation. The effective recent-history window is two keyframes in the non-RMBench simulation configurations. An implementation detail worth noticing is that the RLBench configuration still allocates twelve slots for checkpoint compatibility, while only the two recency slots are populated; the others are masked. Allocated capacity is not the same thing as effective memory. RMBench additionally uses selected subgoal frames. Selection is supervised: a classifier predicts whether the current observation marks a subtask boundary worth retaining. Its training labels come from the demonstrations’ language segments, and its probability gates admission to memory at inference. That is a meaningful qualification to the word “adaptive.” The system is not discovering useful episodic events without annotation. The released RMBench preparation pipeline explicitly carries language and subtask metadata into memory supervision. For adaptation, I would ask where equivalent labels will come from. If your demonstrations already contain reliable phase annotations, this is an attractive interface. If they do not, milestone supervision becomes another engineering task. I would also distinguish remembering an attempted operation from remembering its successful completion. A policy trained on successful demonstrations might recognize the expected visual phase while handling failed contacts poorly. Testing interrupted operations and failed repetitions would tell us more than extending the nominal task by another few steps. Spatial memory: revisiting previously visible geometry. The spatial mechanism preserves the initial point cloud. For a later fine-stage prediction, that reference is cropped around the current coarse waypoint and rendered using the corresponding local view configuration. It supplies an aligned reference for geometry that may now be obscured. The key property is not simply access to an old image. It is access to an old geometric observation that can be re-expressed in the current crop’s coordinates. This is particularly appealing when the policy’s own action causes occlusion. The target may still be where it was, but the gripper now hides the evidence required for precise placement. However, a reference is not necessarily current truth. If someone moves the receptacle, the stored cloud becomes stale. My preferred stress test would deliberately move a target after the initial observation and measure whether the policy can reject obsolete geometry. The resulting design principle is broader than this particular model: history should carry its coordinate frame and its assumptions about persistence. Remembering information is only half the problem; knowing when that information still applies is the other half. Training and bimanual execution preserve the same structure. The training implementation first freezes the pretrained backbone while fitting the added heads and memory modules, then allows broader adaptation. SigLIP and the language-token embeddings remain frozen. This is not a claim that the entire pretrained network stays fixed throughout robot learning. I see the staged schedule as protection for a useful initialization. New modules initially have little reason to produce sensible gradients for the pretrained representation. Giving them an adaptation period before changing the backbone is a straightforward way to reduce that mismatch. Memory also makes augmentation more delicate. The RMBench configuration disables per-frame image augmentation and uses a fixed, bounds-centered workspace rather than recentering independently on each point cloud. These choices preserve the correspondence on which cached spatial features depend. The general implementation lesson is simple: an augmentation that is harmless for a stateless policy can corrupt a memory-conditioned one. If the current frame and its history are transformed inconsistently, the learner sees artificial motion and false scene changes. The bimanual extension shares scene processing while providing separate output heads and local crops for the arms. Its training entrypoint sums the translation, rotation, and gripper losses across arms and disables the collision-flag output for this benchmark. Shared weights should not be mistaken for identical actions or a single local computation. Each arm still needs a target and an appropriate fine-scale view. Nor does sharing a scene representation automatically solve coordination. I would separately test overlapping workspaces, tightly coupled contacts, and situations where one arm temporarily hides the other’s target. The architectural extension is economical, but the difficult coordination cases remain empirical questions. What the experiments establish—and what 95.4% leaves out. The headline Franka result is best read as a low-data adaptation experiment. Three demonstrations across each of thirteen tasks means 39 robot trajectories, not three trajectories total. The tasks are learned jointly, allowing shared structure across related manipulation conditions. In the August 2026 manuscript, BridgeVLA reaches 95.4% with three demonstrations per task, versus 96.9% with ten; RVT-2 reaches 90.0%. Evaluation uses ten trials per task. That is impressive robot-label efficiency. But the difference between 95.4% and 96.9% amounts to only two successful episodes across 130 trials. I would not interpret it as proof that three demonstrations are generally equivalent to ten. That would require repeated sampling of demonstration subsets, not just evaluation of one small-data training set. There is also a source-version issue: the project’s Franka table reports π0, whereas the August 5 manuscript reports π0.5. Those should not be silently combined into one comparison. The simulation results tell a broader story:. | Evaluation suite | BridgeVLA | BridgeVLA++ | |---|---:|---:| | RLBench | 90.5% | 93.7% | | COLOSSEUM | 64.0% | 65.2% | | GemBench | 50.0% | 51.1% | | RMBench | 18.9% | 96.0% | | MemoryBench | 11.3% | 99.7% |. These are the authors’ reported aggregate success rates. The pattern matters more than any individual maximum. Ordinary manipulation and aggregate generalization improve moderately. Memory-dependent performance changes dramatically. That pattern is consistent with an information-access explanation: a stronger current-frame policy cannot reliably reconstruct task history that is absent from the current observation. However, the evaluation protocols are not interchangeable. RMBench uses fifty demonstrations per task, separately trained task policies, best-performing checkpoint selection, and one hundred evaluation episodes per task. Its 96.0% is not the performance of one jointly trained generalist over all nine tasks. The real memory experiments also use a different setup. On Dobot CR5A, seven instructions receive ten demonstrations each. Basic-setting success on the three memory-dependent task families rises from 20.0% to 93.3%; under the four visual disturbances, BridgeVLA++ averages 78.3%. This is evidence that the framework can be trained successfully on another embodiment. It is not zero-shot transfer of the Franka checkpoint to a previously unseen arm. Finally, aggregate preservation does not imply universal improvement. On GemBench’s novel-articulated-object level, the reported average falls from 43.8% to 38.5% with memory, despite the small gain in the overall benchmark average. For practitioners, that is an important reminder: memory can add useful evidence while also adding failure modes. Evaluate it against the shifts your deployment actually encounters. The ablations are more informative than the leaderboard. Two base-model ablations strongly support taking the spatial interface seriously. Replacing heatmap decoding with direct position regression reduces RLBench success from 90.5% to 31.4%. Adding explicit per-pixel 3D-position features reduces it to 56.2%. My reading is not that coordinate regression is universally bad, or that VLMs should never receive geometric features. These experiments establish a narrower, more useful point: in this initialization and training regime, the way geometry is represented matters greatly. Additional information can make learning harder if its interface disrupts useful pretrained features. Conversely, a constrained output representation can improve learning even when it looks less expressive than a general decoder. The memory ablations provide an even cleaner separation. On RMBench, removing temporal memory drops performance from 96.0% to 21.3%; removing spatial memory leaves 95.4%. On RLBench, removing spatial memory reduces the full model from 93.7% to 92.0%. The temporal component carries the memory-benchmark improvement. The geometric reference has a smaller role there and a more relevant role in spatial manipulation. That is exactly why I would resist deploying “memory” as one indivisible feature. If failures are caused by state aliasing, prioritize history. If failures arise after the gripper occludes a static target, prioritize geometric references. If neither occurs, measure whether the added complexity earns its cost. The strongest transferable result is therefore a decomposition, not merely a larger model: different missing information calls for different memory mechanisms. Where this sits relative to other policy families. Three comparisons help locate the contribution. RVT-2 supplies the closest geometric-policy lineage. Its objective is already precise, data-efficient manipulation. BridgeVLA’s question is how to combine that kind of structure with a pretrained vision-language backbone—not whether geometry matters in the first place. 3D Diffuser Actor combines three-dimensional scene representations, language, and proprioception with trajectory denoising. Its action-generation problem differs from choosing a sparse waypoint and delegating the intervening motion to a planner. π0 also starts from PaliGemma, but adds a flow-matching action expert and proprioceptive inputs to generate continuous action chunks, including high-frequency control settings. Sharing the backbone does not make the comparison a controlled experiment on the output head alone. My practical conclusion is to compare complete control requirements before comparing model names. A task dominated by identifying a destination may suit keyframe prediction. A task requiring continuous contact regulation asks something different of the learned policy. For memory, SAM2Act+ is an especially relevant predecessor. It integrates a memory architecture into a visual-foundation-model manipulation system and introduces MemoryBench. BridgeVLA++ should therefore be understood as a particular integration of memory with a VLM-based, coarse-to-fine spatial interface—not as the first manipulation policy to remember observations. The fairest causal evidence remains the within-framework ablations. Cross-system rankings also reflect differences in sensing, pretraining, supervision, controllers, and optimization. What I would reproduce before adapting it. The release is substantive. The official repository places BridgeVLA++ on and preserves the original model on the branch. It provides benchmark training and evaluation entrypoints, while the Hugging Face release lists grounding weights, benchmark checkpoints, and prepared keyframe artifacts. The real-robot demonstration data are not published. That makes “open code and checkpoints” a fair description. “Every artifact needed to independently reproduce the real-robot numbers is available” would be too strong. Licensing also deserves precise language. The top-level code repository uses Apache-2.0, while the PaliGemma model has separate Gemma terms. Availability of weights does not collapse all upstream components into one license. For reproduction, I would begin with one released simulation checkpoint and the authors’ launch scripts. Their documentation explicitly warns that the RLBench and GemBench-lineage simulator stacks are not interchangeable. It also pins keyframe caches and task definitions rather than assuming arbitrary upstream versions will reproduce the results. That is unusually consequential here. A different keyframe extractor changes the policy’s decision timing, action targets, and memory sequence. It is not merely a data-loading implementation detail. Next, I would establish a memory-free baseline on my own tasks, then enable the two memory components separately. The training launcher exposes those ablation switches directly. I would log the rendered observations, per-view heatmaps, coarse waypoint, fine crop, selected history, and executed outcome together. Those records would let me distinguish several failures that otherwise look identical: incorrect grounding, inconsistent geometry, wrong task stage, stale memory, or an execution failure after a correct prediction. Finally, I would budget for inference honestly. The authors report 0.35 seconds per prediction for BridgeVLA and 0.57 seconds for BridgeVLA++ on an RTX 4090, with approximately 269.77 million additional memory parameters, measured against a 2.92-billion-parameter backbone. The parameter increase is about nine percent, but the reported prediction latency rises by roughly sixty-three percent. Parameter count is not a latency model. For a keyframe system whose motions take appreciable time, that may be acceptable. For rapid reactive control, it changes the design space. I would measure observation acquisition, rendering, inference, planning, and execution separately before deciding whether to optimize the network or reconsider the control architecture. The takeaway: preserve correspondence, remember selectively. My main takeaway from BridgeVLA++ is architectural rather than numerical. A pretrained model becomes more useful when the adaptation preserves relationships that the task depends on. Here, spatial observations lead to spatial predictions, and explicit projection geometry connects those predictions to robot coordinates. Memory follows the same principle. The history used to choose the next stage need not have the same structure as the geometric reference used to refine a placement. The results make a compelling case for investigating that combination. They do not establish RGB-only manipulation, universal three-demonstration learning, zero-shot embodiment transfer, or unrestricted long-horizon reasoning. For a robotics team, the actionable question is narrower: what information does your policy lose before its next decision, and can you restore that information in a representation aligned with the action it must predict?. BridgeVLA++ offers a concrete, inspectable answer. Its most useful contribution may be showing that a stronger embodied policy does not always require a more general action language. Sometimes it requires a better spatial interface—and a carefully chosen record of what the robot has already seen.