Latest / Embodied AI 101 / The Frozen VLM That Learns Anyway: Inside Spatial Memory Agent

Transcript

The Frozen VLM That Learns Anyway: Inside Spatial Memory Agent · Embodied AI 101

• The Frozen VLM That Learns Anyway: Inside Spatial Memory Agent. Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence • Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, and Hao Chen • Zhejiang University, Shanghai Jiao Tong University, and Shanghai Innovation Institute • arXiv preprint, under review • 2026. A frozen vision-language model usually has three ways to become better at spatial reasoning. You can change its weights through supervised fine-tuning or reinforcement learning. You can give it inference-time tools such as object detectors, depth estimators, camera-pose solvers, or 3D reconstruction systems. Or you can simply prompt it harder and hope that additional deliberation extracts geometry already latent in the model. Spatial Memory Agent, or SMA, proposes a fourth path: let the model accumulate an external notebook of verified reasoning procedures. The notebook does not store trainable parameters, and during final deployment it does not call specialist spatial tools or continue rewriting itself. Instead, the system turns earlier, verifiably scored attempts into short textual lessons—things like “compare vertical extent from base to top rather than apparent image area,” or “distinguish object motion from camera motion using fixed background anchors.” It then learns which lessons tend to transfer successfully. That basic idea is simple. The interesting part is the separation between semantic relevance and empirical reliability. A previous question can look almost identical to the current one while encouraging exactly the wrong reasoning shortcut. SMA’s Transfer Reliability Score, or TRS, is meant to distinguish a memory that merely sounds relevant from one that has repeatedly been associated with successful future answers. The paper was submitted to arXiv on August 13, 2026. As of August 20, 2026, it is listed as under review, and the project page says that code is still coming soon. So the analysis here is necessarily based on the paper, appendix, prompts, project materials, and reported experiments rather than an independent reproduction. The problem SMA is actually trying to solve. Spatial failures in current VLMs are often procedural rather than purely factual. A model recognizes the chair and the table, yet compares their 2D pixel area when asked about 3D height. It sees two views of a room, yet attributes every displacement to object motion rather than checking whether the camera moved. It answers a placement question from visual proximity without checking clearance. Or it understands the words “left” and “right” but silently switches from an image-centered frame to an agent-centered frame halfway through the reasoning. These are recurring reasoning habits. That makes them plausible candidates for external procedural memory. Most ordinary retrieval systems do not explicitly model this distinction. They retrieve an example because its text embedding is close to the current query. But similarity can be a poor proxy for usefulness. Two questions may share nearly identical wording while differing in viewpoint, object configuration, scale cue, or temporal interpretation. Reusing the previous answer is clearly unsafe, and even reusing the previous chain of thought can preserve a brittle shortcut. SMA therefore treats memory as a collection of transferable checks and warnings, not demonstrations to imitate. It positions itself between three established research lines. It inherits verbal reflection from systems such as Reflexion and ExpeL; procedural abstraction from memory systems such as MemP; and reward-sensitive retrieval from MemRL. It then specializes that recipe for visual-spatial tasks, adds a conservative reliability estimator, and freezes the entire memory state before deployment. The contrast with spatial post-training and spatial tool use is also important. SpatialEvo, for example, computes exact supervision from deterministic 3D geometry and uses it to update a spatial model. S-Agent instead lets a VLM request fresh 2D and 3D evidence from a hierarchy of specialist tools. SMA does neither during deployment. Its retrieved text can remind the model how to inspect the image, but it cannot produce missing depth, fix a bad detection, or directly calculate a coordinate transform. How SMA works. SMA divides operation into two phases: an experience-acquisition phase with verifier access, and a read-only deployment phase. The experience phase receives an environment split containing visual observations, task text, and verified targets. The deployment split contains separate problems whose answers are hidden from the runtime system. Model weights remain fixed in both phases. The only state that evolves is an external memory bank. Writing a procedural memory. Each memory card contains six pieces of information:. The source task text. A compact summary of the rollout and its success or failure mode. A transferable lesson. The number of subsequent times the memory has been used. Its accumulated downstream reward. Its current Transfer Reliability Score. Only the task, summary, and lesson are exposed to the VLM when the card is retrieved. Previous predictions and verified answers are deliberately hidden. During the first pass through the environment split, the frozen VLM answers each question. A verifier scores the parsed answer, normally with a binary or bounded reward. The same frozen model then acts as the reflection model: it receives the task, its output, the reward, and—crucially—the verified target. It must produce strict JSON containing a short summary and one transferable lesson. The prompts impose extensive anti-leakage rules. The reflector is told not to reveal the answer letter, coordinate, number, yes-or-no label, scene identifier, or sample-specific object configuration. Instead, it should diagnose a reusable reasoning habit and a trap to avoid. A typical requested form is essentially: when encountering this task shape, apply this inspection procedure, avoid this shortcut, and validate using this additional check. This ground-truth-guided reflection is an important part of the method. SMA is not asking a model to bootstrap reliable spatial expertise from its own unverified prose. It assumes access to a verifiable environment and uses correct targets to convert even failed episodes into potentially useful procedures. That is why a memory created from an incorrect source answer is not automatically worthless: the reflection model can use the target to identify the error and write a warning against repeating it. Conversely, a correct source answer does not guarantee a good memory. The model might have guessed, relied on a scene-specific cue, or written a lesson too narrow to transfer. SMA therefore refuses to initialize reliability from source correctness alone. One-pass writing, repeated calibration. The default protocol is called One-Pass Memory Writing. New cards are created only during the first traversal of the environment data. Once that pass ends, the number of cards is fixed. Later passes reshuffle the same environment problems, retrieve existing memories, answer the tasks again, and update the memories’ reliability values—but do not create more cards. This design matters because the obvious alternative, continual memory writing, generated many near-duplicate memories. After ten passes, the one-pass bank contained roughly one-tenth as many cards, had 21 percent less reported redundancy, and gave approximately twice as large a fraction of the bank enough retrieval visits to receive reliability updates. In other words, repeated experience is spent evaluating the existing procedural library rather than continuously expanding it. The exact first-pass ordering also creates a mild online effect. Early examples are solved with an empty or small memory bank, while later examples in the same pass may use cards written earlier. At the end of the first pass, the bank contains one card per environment example. Subsequent passes are devoted entirely to calibrating those cards. Transfer Reliability Score. Every new memory begins with the same neutral TRS of 0.5. The method also assigns the memory two virtual prior visits—conceptually one success and one failure. Whenever a memory is retrieved and the resulting answer receives a reward, that reward is added to the card’s evidence. After one successful transfer, for example, the score moves from 0.5 to roughly 0.67 rather than jumping to 1.0. After one failed transfer, it moves to roughly 0.33 rather than zero. As observations accumulate, the prior matters less and the score approaches the empirical average reward of the problems on which the memory was retrieved. This estimator is order-invariant: five successes followed by five failures yield the same score as the reverse ordering. It is also less volatile than directly adopting the most recent reward. The authors’ intended interpretation is not “was the source episode correct?” but “when this procedure is later selected, how often is the resulting task solved?”. There is, however, a subtle distinction between that interpretation and causal utility. When three memories are retrieved together, SMA assigns the final task reward to all three. It does not know whether one card helped, another was ignored, and the third was actively misleading. TRS is therefore an association between retrieval and success under the current retrieval policy, not a controlled estimate of each memory’s individual contribution. The paper explicitly identifies this as a credit-assignment limitation. Two-stage retrieval. Retrieval begins with task text rather than visual content. SMA embeds the current task and each stored source task using. Cards below a benchmark-specific similarity threshold are discarded. The surviving candidates are ranked using equal parts semantic similarity and TRS in the default configuration. Because these quantities have different scales, SMA normalizes each within the candidate set before combining them. The top three cards are placed in the VLM prompt. The logic is straightforward. Similarity supplies an applicability gate: a lesson about camera motion should not guide a simple object-counting question. TRS then prefers procedures that have worked reliably among the semantically plausible choices. The ablations suggest that both stages matter, especially the initial semantic filter. Read-only deployment. Once calibration is complete, the memory bank is frozen. Deployment performs the same semantic filtering and TRS-aware ranking, but no new cards are written and no scores are changed. Visit counts and cumulative rewards remain fixed even if a deployment answer could later be verified. This is a useful experimental discipline. It prevents the nominal test set from directly entering the memory state during inference. It also makes the deployed artifact easy to describe: a frozen VLM, a fixed text-memory library, an embedding index, and a deterministic retrieval policy. What was evaluated. The main table covers five benchmark slices. RoboSpatial tests free-space pointing, object configurations, and placement compatibility in indoor scenes. ERQA asks robot-relevant questions involving state, action, trajectories, pointing, and multiple views. Omni3D contains open-ended metric, depth, occlusion, containment, capacity, and counterfactual-placement questions. SAT focuses on dynamic spatial aptitude, including object motion, ego motion, perspective, and action consequences. EmbSpatial tests directional and near-versus-far relations. SITE-image and ViewSpatial appear in the appendix, bringing the extended evaluation to seven benchmarks. For all benchmarks except SAT, the authors perform a per-category 50/50 split with seed 42. The main deployment sets contain 175 RoboSpatial questions, 200 ERQA questions, 250 Omni3D questions, 300 SAT questions, and 1,820 EmbSpatial questions. SAT uses an expanded official test set for deployment and a matched sample from validation for memory acquisition. Macro averages weight each benchmark equally despite the large differences in sample count. The four frozen backbones are Qwen3.5-122B-A10B, Qwen3.6-35B-A3B, Qwen3.6-27B, and Qwen3.5-9B. In each run, the same model serves as both task solver and reflector. Decoding is deterministic, and the experiments were run on four NVIDIA H200 GPUs. Although the paper describes these as four VLMs, they all belong to the Qwen lineage, so the architectural diversity is narrower than a comparison spanning several independently developed model families. The baselines include direct no-memory inference; RAG over previous tasks and raw outputs; MemP-style reflected procedural memory retrieved only by similarity; reward-only MemRL; and a ground-truth-reflection version of MemRL added by the authors. This is a reasonably informative baseline set because it isolates raw example reuse, procedural abstraction, stronger reflection supervision, and reward-sensitive ranking. The headline results. SMA obtains the highest five-benchmark macro average for every backbone:. | Frozen backbone | No memory | Strongest non-SMA baseline | SMA | Gain over strongest baseline | |---|---:|---:|---:|---:| | Qwen3.5-122B-A10B | 65.3 | 66.2 | 68.8 | +2.6 | | Qwen3.6-35B-A3B | 61.6 | 63.8 | 66.7 | +2.9 | | Qwen3.6-27B | 63.3 | 68.1 | 69.8 | +1.7 | | Qwen3.5-9B | 60.6 | 60.7 | 63.5 | +2.8 |. Across the twenty model-benchmark combinations, SMA records eighteen outright wins, one tie, and one loss against the evaluated methods. It also improves over no-memory inference in every one of the twenty cells. The only main-table loss is ERQA with the 9B model, where MemP reaches 53.0 percent and SMA reaches 52.0 percent. The strongest result is RoboSpatial with Qwen3.6-27B: accuracy rises from 54.1 without memory to 68.5 with SMA. Omni3D rises from 41.6 to 47.6, while EmbSpatial moves more modestly from 85.7 to 87.9. This pattern is plausible. Procedural reminders have the most room to help when the base model is inconsistent but not completely perceptually incapable. Near a benchmark ceiling, the available lift is naturally smaller. The appendix is somewhat less uniform. SMA remains strong on SITE-image and ViewSpatial, but MemRL-GT beats it on ViewSpatial for the 122B and 27B backbones. SMA is best or tied on SITE-image across the four models and wins ViewSpatial with the 35B and 9B models. So the broader seven-benchmark picture is positive, but not a clean sweep. What the ablations say. On RoboSpatial with Qwen3.6-27B, removing the summary costs 3.2 percentage points, removing the transferable lesson costs 3.5, and removing the semantic filter costs 5.8. Including the raw prior model output makes performance 4.4 points worse, while replacing ground-truth-guided reflection with reward-only reflection costs 5.5 points. The Omni3D ablations tell a similar story. Removing the semantic filter costs 7.2 points, and removing the lesson costs 5.2. These results support two claims: distilled procedures are safer than replaying raw generations, and reliability ranking cannot compensate for memories that are structurally unrelated to the current problem. The reported sensitivity study peaks at an equal similarity/TRS weighting and three retrieved memories. Does the memory transfer?. A bank written by Qwen3.5-122B-A10B improves Qwen3.6-27B on all five main benchmarks. The gains range from 1.6 points on EmbSpatial to 9.4 on RoboSpatial. Cross-benchmark probes are also positive: ERQA memories improve RoboSpatial by 7.6 points, EmbSpatial memories improve it by 7.3, and Omni3D-to-EmbSpatial transfer adds 1.5 points. These are encouraging results because they suggest the cards encode procedures that are not entirely tied to the producing model. But the model transfer remains within the Qwen family, and the benchmark-transfer table reports representative pairs rather than a complete source-target matrix. The evidence therefore establishes promising portability, not universal transfer. Is TRS doing anything beyond similarity?. The paper reports that SMA’s retrieved memories have lower average semantic similarity than MemP’s—0.698 versus 0.792 on the five-benchmark macro analysis—yet yield higher accuracy, 69.8 versus 66.8. That is exactly the intended behavior: the nearest memory is not always the most useful one. Higher TRS is also associated with higher deployment accuracy. Memories originating from successful source questions average a TRS of 0.522 and 85.7 percent downstream accuracy, while those originating from failures average 0.452 and 61.4 percent. Questions whose three retrieved memories all came from successful source questions reach 93 percent accuracy, compared with 39 percent when all three came from failed sources. Those diagnostics are directionally useful, but they should not be read as causal proof. TRS is itself computed from previous task rewards, so a correlation between TRS and later success is partly expected. Easy task clusters can also produce both high scores and high accuracy, and co-retrieved memories share credit. The paper appropriately notes that benchmark difficulty and question composition may influence the pooled trend. How SMA differs from its closest relatives. MemP is the cleanest procedural-memory comparison. It distills trajectories into reusable instructions and higher-level scripts, then maintains that repository through build, retrieval, and update operations. The SMA baseline derived from MemP retrieves reflected spatial procedures using semantic similarity alone. SMA’s added claim is that procedure quality must be calibrated from subsequent transfer outcomes, rather than assumed from textual relevance. MemRL is algorithmically closer. It also separates a frozen model from evolving memory, filters candidates by semantic similarity, combines normalized similarity with a learned utility, and updates that utility from environmental rewards. SMA’s primary differences are its spatially specialized memory representation, verified-target reflection, pseudocount-style TRS estimator, one-pass memory-writing protocol, and read-only deployment stage. The core retrieval architecture is consequently best understood as a spatial adaptation and refinement of value-aware episodic memory, not a wholly new retrieval paradigm. SpatialEvo represents the opposite choice about where adaptation should live. It uses deterministic geometric environments to generate exact spatial supervision and updates the model parameters. Its knowledge becomes internal and does not require a memory prompt, but training is substantially more involved. SMA compares its 9B system with the released SpatialEvo-7B model and reports a 63.5 versus 47.1 macro average. That is an interesting scoreboard result, but not a controlled test of memory versus training: the systems use different base models, sizes, prompts, and adaptation budgets. S-Agent occupies another corner of the design space. It treats spatial reasoning as evidence acquisition, allowing a planner to invoke grounding, depth, 3D lifting, measurement, counting, and relative-position tools while maintaining scene and agent memory. SMA instead assumes that the current pixels contain enough evidence and supplies only reasoning advice. A likely long-term system would combine the two: procedural memory could tell an agent which evidence is missing and when a costly geometric tool call is warranted. The most important caveats. The first concern is checkpoint selection. The paper says that all reported results are measured on the held-out deployment split, but it also says the main SMA table reports the best checkpoint from a ten-pass run. The hyperparameter appendix lists a different selected pass for each model-benchmark pair—for example, pass three for Qwen3.6-27B on RoboSpatial, pass nine on ERQA, and pass two on Omni3D. No separate validation split for selecting those passes is described. Taken literally, this appears to mean that deployment accuracy was inspected across passes and the best deployment checkpoint was reported. If so, the deployment set is functioning partly as a model-selection set, which can inflate the final numbers. It also creates an operational problem: a deployed agent does not know which calibration pass is best without access to deployment labels. A stronger protocol would select the pass on an environment-side validation split, commit to a fixed final pass, or report the full pass trajectory and area under the learning curve. The second concern is split granularity. Non-SAT data are divided per category at the example level. The paper does not describe scene-disjoint, image-disjoint, or template-disjoint splitting. This matters because many spatial benchmarks contain multiple questions or highly templated questions associated with related scenes. Moreover, the retrieved card exposes the full source task text in addition to its abstract summary and lesson. The reflection prompts prevent answers from leaking into the generated lesson, but they do not remove potentially sample-specific information from the stored task itself. Nothing in the reported results proves that such overlap is driving the gains. Nevertheless, a more convincing transfer evaluation would group by scene, trajectory, source video, question template, and ideally object configuration before splitting. It would also compare retrieval using an abstracted task signature against retrieval using the full original question. Third, TRS has a credit-assignment and confounding problem. All retrieved memories receive the same outcome reward. A memory can gain value because it was repeatedly selected on easy questions, because a co-retrieved card did the real work, or because the base model would have answered correctly without any memory. There is no no-memory counterfactual for each visit. This does not make TRS useless; contextual bandits and recommender systems routinely learn from imperfect logged feedback. But “probability of success when retrieved” should not be conflated with “causal contribution to success.” The paper acknowledges this gap and points toward attribution-based memory methods as future work. Fourth, SMA remains dependent on a substantial verified experience budget. Parameter-update-free does not mean supervision-free or compute-free. The system needs ground-truth targets or reliable automated verifiers, one reflection call per environment example, repeated solver passes for calibration, embeddings, and deployment-time prompt expansion. The reported runs use four H200 GPUs, and the paper does not provide a detailed token, latency, or dollar-cost comparison against fine-tuning or tool-based alternatives. Finally, the evaluation gives no multi-seed split analysis, confidence intervals, or statistical significance tests. Deterministic decoding removes sampling variance from generation, but it does not remove uncertainty caused by finite benchmark samples, the single seed-42 partition, hyperparameter choice, or checkpoint selection. The currently unavailable code also prevents independent verification of parsing, scoring, memory construction, and exact baseline parity. What roboticists should take from this. For robotics, the most promising deployment setting is an offline or simulated experience factory with cheap verification. Manipulation attempts can be verified by grasp success, final object pose, collision checks, or simulator state. Navigation memories can be scored through goal completion and path validity. Calibration and viewpoint tasks can use known geometry. SMA then offers a way to compile those interactions into portable textual procedures without repeatedly fine-tuning a large backbone. I would modify the method in four ways before relying on it in a real robot stack. First, reliability should be contextual. A single global score cannot represent that a lesson works for tabletop RGB images but fails under a wrist camera, or transfers to one backbone but not another. A card should carry uncertainty and reliability conditioned on task family, sensor configuration, embodiment, scene type, and model identity. Second, credit assignment should include occasional counterfactual evaluation. On a small calibration subset, run the VLM with all memories, without each individual memory, and without memory altogether. Even sparse leave-one-out probes would help separate genuine contribution from easy-question correlation. Third, retrieval should be multimodal. Task text is useful for identifying question shape, but spatial applicability also depends on visual structure. A practical retriever could combine language intent, scene embeddings, estimated viewpoint, number of frames, visible object layout, and available geometric metadata. Fourth, the memory bank needs a lifecycle. SMA can downweight weak cards, but it does not merge duplicates, resolve contradictions, expire stale procedures, or prune under a latency budget. The paper identifies this explicitly. For a long-running robot, memory consolidation is not optional; it is part of the learning algorithm. The most compelling future architecture is probably hybrid rather than ideological. Use verified memory to capture recurring procedures. Use spatial tools when the current observation lacks sufficient evidence. Periodically distill mature, broadly reliable memories into a smaller policy or model. Keep rare, environment-specific procedures external so they can be updated or deleted safely. Final assessment. SMA’s central contribution is not that a frozen VLM suddenly acquires new geometric perception. It is that verified interaction can be compiled into an external library of spatial habits, and that the library can learn which habits deserve to be retrieved. The reported gains are broad, the ablations are coherent, and the transfer results make procedural memory look like a useful interface between experience and frozen multimodal models. At the same time, checkpoint selection, question-level splits, coarse memory credit, single-family backbones, and unavailable code make the exact magnitude of the gains less settled than the headline suggests. The durable idea is nevertheless strong: do not retrieve the most similar experience; retrieve a relevant procedure with evidence that it transfers. For embodied AI systems operating in verifiable environments, that is a practical design principle worth carrying forward.