Latest / Embodied AI 101 / Metis Gives Foundation Models a Native, Mutable Memory

Transcript

Metis Gives Foundation Models a Native, Mutable Memory · Embodied AI 101

• Metis Gives Foundation Models a Native, Mutable Memory. Metis: Memory Foundation Model • Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, et al. • MemTensor, Renmin University of China, National University of Singapore, Shanghai Jiao Tong University, and Tongji University • arXiv preprint • 2026. The Shift Metis Is Proposing. Most language-model agents are stateful only because the surrounding software is stateful. The model itself remains a fixed function. A vector database stores old conversations, a retrieval system selects a few passages, a prompt builder concatenates them with the current request, and the model processes everything as if it were seeing it for the first time. Metis asks what happens if memory is moved inside that computational loop. The paper, submitted to arXiv on July 29, 2026, introduces what the authors call a memory foundation model: a foundation model with a persistent, dynamically evolving state that participates directly in its forward pass. Instead of updating the model through backpropagation after deployment, Metis changes a separate set of fast memory tensors using ordinary forward computation. Those tensors survive across interaction steps and influence later generations through a dedicated memory-attention branch. That distinction is the paper’s central idea. Metis is not simply a Transformer with a larger context window, and it is not conventional test-time training. Its learned weights remain frozen during deployment. What changes is a session-specific latent state embedded in the Transformer architecture. The strongest interpretation of the work is therefore not that Metis has solved lifetime memory. It has not. The experiments still show substantial gaps relative to replaying the original evidence, particularly on long conversational histories. Rather, Metis demonstrates a new model interface: a foundation model can be treated as a state transition system, not merely a stateless conditional predictor. The authors have released code and Qwen3.5-based checkpoints at 4B, 9B, and 27B scales, making this more than a purely conceptual proposal. What Is a Memory Foundation Model?. The paper begins with a deliberately strict distinction between knowledge and memory. Information acquired before deployment through pretraining or post-training is knowledge stored in the model’s static parameters. Memory refers only to information encountered during an online interaction trajectory: facts about the present user, task, environment, or conversation that were unavailable before the trajectory began. Under that definition, a conventional language model has one parameter state for every request. An externally augmented agent still has fixed model parameters, but its prompt is supplemented with context produced by a memory system. A memory foundation model instead contains two kinds of parameters:. Slow, static parameters, covering the pretrained backbone and the learned machinery that manages memory. Fast, dynamic parameters, representing the memory state for the current interaction trajectory. At interaction step (t), the model generates an answer using the current input and the memory state inherited from previous steps. The same forward computation then produces the memory state for step (t+1). In plain language, the model behaves like a recurrent state machine whose state is distributed through its Transformer layers. The paper calls the state “native” for two reasons. First, it is consumed directly by the backbone rather than converted back into text and inserted into the prompt. Second, the procedures that transform it—remembering, updating, forgetting, and reading—are learned as continuous functions rather than implemented as a collection of external database rules. That framing produces three claimed advantages. Architecturally, the memory representation is optimized to interact with the model’s hidden states. From an optimization perspective, training losses on future answers can propagate through earlier memory writes, allowing the write and read procedures to be trained jointly. From an efficiency perspective, memory readout and storage can run alongside the ordinary attention path, while the amount of state remains fixed as history grows. There is an important systems qualification, however. “Native” does not mean that memory ceases to be an object that serving infrastructure must manage. Each user or agent still has distinct dynamic tensors that must be stored, isolated, checkpointed, restored, and possibly migrated between devices. Metis moves memory into the model’s computational representation, not out of the system architecture altogether. The paper’s roadmap makes the ambition clearer. It describes a progression from persistent state, to autonomous memory management, to learning from experience, persistent cognition, and eventually self-evolving capabilities. The current Metis prototype mostly demonstrates the first level and parts of the second. It does not yet turn accumulated experience into lasting new skills or modify the static policy itself. Architecture: Slow Weights, Fast State, and Memory Attention. Metis augments Transformer layers with a Metis block containing two components: a local memory block and a hyper memory block. The terminology is slightly confusing, but the underlying division is straightforward. The local block contains the fast, session-specific state. The hyper block contains the slow, trained functions that decide how to write and read that state. The local memory state. At each equipped Transformer layer, Metis maintains a dense matrix that associates memory keys with memory values. It also maintains a vector used to normalize query–key interactions during reading. Both are initialized to zero at the beginning of a trajectory and updated after each interaction step. This is closer to a fast-weight associative memory than to a list of memory slots. When the model stores information, it adds key–value associations to the matrix. When it later receives a query, the query is multiplied through that matrix to recover a value-like activation. Because every layer has its own memory representation, the historical signal can enter computation at multiple levels of the network. It is not appended as a single embedding at the input or injected only into the final layers. The hyper memory block. The hyper memory block contains the static parameters learned during mid-training. These include:. A learned importance scorer for token activations. Projections that map selected hidden states into memory keys and values. A separate projection that maps current hidden states into memory queries. These parameters remain frozen during deployment. Their purpose is to transform ordinary Transformer activations into writes and reads that the local memory state can support. The distinction is analogous to a learned memory controller, but unlike classical memory-augmented neural networks, the controller and memory path are inserted directly into the backbone’s layer computation. How Metis writes memory. After an interaction step has been processed, the hyper memory block examines the layer’s hidden states. It assigns an importance score to every token and forms a probability distribution over positions. Metis then performs an adaptive top-(\rho) selection. Rather than always storing a fixed number of tokens, it selects the smallest set of high-scoring positions whose cumulative importance exceeds a threshold, subject to a minimum selection count. During training, the discrete selection is handled with a straight-through estimator so that the importance scorer can still receive gradients. The selected activations are projected into memory keys and values. In the simple version of the update, their outer products are accumulated into the memory matrix with exponential decay. The normalization vector accumulates corresponding key statistics. Conceptually, each write says: when a future query resembles this key, return something resembling this value. The decay factor limits the influence of older writes and prevents the state from growing without bound. The final models replace this simple linear accumulator with a Gated Delta Network-based update, or GDU. A delta-rule update corrects the existing association rather than blindly adding another value, while gating controls retention and replacement. The paper’s ablations find this more effective and stable than the basic linear update, particularly as the model scale changes. This update is where the “gradient-free online adaptation” claim comes from. Once trained, Metis does not calculate a loss, run backward propagation, or maintain optimizer state. It transforms the memory matrix by applying the learned write function during a forward pass. How Metis reads memory. For memory retrieval, the current hidden states are mapped through the dedicated memory-query projection. Those queries are multiplied by the memory matrix, and the result is divided by a query-dependent normalization term computed from the accompanying normalization vector. The operation resembles normalized linear attention over a compressed collection of historical key–value pairs. There is no nearest-neighbor search, explicit memory slot lookup, or softmax over every historical token. The resulting memory activation is normalized to match the scale of ordinary attention. Metis then blends two branches:. Standard causal attention over tokens in the current interaction step. Memory attention over the persistent latent state. A mixing coefficient controls their relative contribution. The combined activation proceeds through the rest of the Transformer block. The separate memory-query projection is significant. If Metis reused the standard attention query projection, similarities learned for token-to-token attention would also determine cross-step memory access. By learning a dedicated projection, the model can develop a different geometry for historical relevance—for example, emphasizing identity or temporal updates while suppressing superficially related distractors. Memory as a compressed virtual prefix. The theoretical section interprets the memory branch as an approximation to attending over a virtual prefix containing selected activations from prior steps. Instead of preserving that growing prefix, Metis collapses its key–value statistics into a fixed-size matrix. The authors decompose the resulting approximation error into interference from irrelevant memories and structural error caused by normalizing many compressed writes together. Their argument is that the error remains small when the learned memory query has low similarity to irrelevant historical keys. This provides a theoretical motivation for both selective writing and the dedicated memory-query projection. This is also the architecture’s fundamental trade-off. A fixed-size associative matrix gives bounded storage and read cost, but unrelated associations eventually collide. Metis is betting that learned selection, key geometry, gating, and task-driven training will make the useful information survive those collisions. Training a Memory Procedure Rather Than a Database. Metis is built through memory-specific mid-training on top of pretrained foundation models. The backbone parameters are frozen; only the new memory-related parameters are optimized. Training trajectories contain a sequence of information, operation, and query steps. Earlier steps alter the memory state, while losses are generally applied only to later answers. This supervision scheme is one of the paper’s most important methodological choices. The model is not given a target memory matrix. Instead, it learns what and how to store from the downstream consequences of its writes. If a future answer requires an earlier fact, gradient descent encourages the write procedure to preserve a representation that the read procedure can recover. Primary memory data. The authors synthesize primary training data from 27 public benchmarks. After filtering, it contains 357,137 samples and approximately 406 million tokens. The data covers four operations:. Remember: store a fact and answer a later question about it. Update: replace an old value with a newer one. Forget: revoke a previously stored fact. Reflect: combine several stored facts to answer a multi-hop query. Here “reflect” means composition over memory, not open-ended metacognition or self-critique. The data varies along two additional axes. Some operations are explicit commands, while others are embedded implicitly in natural dialogue. Distractor turns are inserted to test retention across intervening noise. The remember subset dominates the token count because its distractor trajectories are much longer. The synthesis pipeline extracts structured subject–relation–value seeds from existing datasets, rewrites them into multiple interaction styles, inserts logically independent distractors, and uses model-based verification to reject inconsistent examples, leaked answers, and questions that can be solved without the intended memory. Auxiliary data for interference and pollution. The primary trajectories are supplemented with 609,443 auxiliary examples targeting two realistic failure modes. The first is interference. Multi-entity binding examples require Metis to store two similar facts without swapping their values. Selective-forgetting examples revoke one fact while requiring the other to remain intact. The second is memory pollution. Post-memory dialogues require the model to return to normal conversation after answering a memory question. Memory-irrelevant dialogues place an active memory state beside a request that should be answered without consulting it. Together, these examples discourage the degenerate strategy of always writing everything and always injecting memory into every response. Three optimization objectives. The training objective has three corresponding pieces. The reconstruction objective asks Metis to regenerate a previously stored passage. This provides an easy warm-up signal and pushes the state toward high-fidelity storage. The authors explicitly acknowledge that reconstruction conflicts with useful semantic compression: perfect copying favors specificity, whereas instruction following often requires discarding irrelevant details. The memory-operation objective trains remembering, updating, forgetting, and multi-fact reflection. Explicit and implicit operations are mixed with long-distance distractor trajectories so that the model cannot rely solely on keywords or immediate context. The regularization objective penalizes entity confusion, collateral forgetting, and leakage into unrelated dialogue. A curriculum gradually shifts sampling weight away from basic reconstruction and toward longer-range operation and regularization examples. For the Qwen3.5 variants, the paper reports 14,000 training steps for the 4B and 27B models and 8,000 steps for the 9B model, selected through early stopping. Because the backbone is frozen, the training effort is concentrated on aligning the new write, state, and read pathways with representations the foundation model already knows. Experimental Design. The main evaluation separates memory-operation tasks from memory-based question answering. Memory operations are evaluated on the external MemOps benchmark and on a held-out Metis test set. Question answering uses LoCoMo Gold, which contains relatively long conversational histories, and NextMem, a latent factual-memory benchmark. The baselines cover four operating modes. Full-context Qwen3.5 models receive all original evidence and serve as an approximate upper bound. Partial-context models use dense retrieval and append the top five observations. No-context Qwen models receive no historical evidence. State-based comparisons include Temp-LoRA, which performs gradient-based test-time adaptation, and (\delta)-Mem, which uses low-rank parametric corrections. Metis processes the ordered memory segments during an information stage, updates its state, and answers the final question without replaying those segments. Notably, the benchmark protocol prepends an explicit instruction telling Metis to commit each dialogue segment to memory. The training corpus includes implicit operations, but the central benchmark results should not be interpreted as fully unguided memory formation. Open-ended responses are scored by GPT-4.1-mini at temperature zero, using three judging runs and the median score. Exact-number questions use deterministic scoring where applicable. There is also a data-overlap caveat. The authors extract seed entities from public datasets including LoCoMo and LongMemEval when synthesizing training data, although they state that the benchmarks’ exact question-answer behavior is not included. This makes the two explicitly out-of-distribution evaluations especially important. What the Results Actually Show. A useful way to read the results is to distinguish proof of mechanism from proof of scalable, lifelong memory. The following are selected scores reported in the paper. Higher is better, but comparisons across differently sized backbones should be treated cautiously. | Method | LoCoMo Gold | NextMem | Metis test set | MemOps Gold | Overall | |---|---:|---:|---:|---:|---:| | Llama-3.1-8B, no context | 0.25 | 19.27 | 19.98 | 3.25 | 10.69 | | Metis on Llama-3.1-8B | 20.56 | 39.98 | 72.33 | 11.25 | 36.03 | | Llama-3.1-8B, full context | 62.80 | 75.75 | 73.42 | 70.48 | 70.61 | | Metis-4B on Qwen3.5 | 16.31 | 41.69 | 56.72 | 17.84 | 33.14 | | Metis-9B on Qwen3.5 | 16.81 | 43.39 | 57.92 | 19.63 | 34.44 |. The Llama transfer experiment is particularly informative. On the constructed Metis test set, the native-memory version reaches 72.33, nearly matching the full-context model’s 73.42 and far exceeding the no-context score of 19.98. That is strong evidence that the architecture can encode and recover the kinds of structured operations it was trained to perform. But the same model scores only 20.56 on LoCoMo, compared with 62.80 when the full conversation is available. On MemOps Gold, it reaches 11.25 against a full-context score of 70.48. Metis therefore recovers meaningful information from its latent state, but complex, long-horizon evidence remains much harder to compress than controlled fact trajectories. Scaling from Qwen3.5-4B to 9B produces only modest gains across the four central benchmarks. The paper reports larger improvements from the 27B model on some operation and QA settings, but the scaling trend is not smooth or universal. Larger backbones appear better able to formulate useful states once sufficient capacity is available, yet different tasks respond differently. Out-of-distribution transfer. The two OOD benchmarks tell a similarly mixed story. On ATM-Bench Gold, which requires integrating heterogeneous information from personal archives, the average score rises from 10.22 for Metis-4B to 16.49 for 9B and 18.56 for 27B. The 27B Temp-LoRA baseline scores 2.57. This suggests that the learned write-and-read procedure transfers beyond the exact synthetic templates used during training. On MemDaily Gold, however, the results are competitive rather than dominant. Metis-27B averages 59.04, slightly below Temp-LoRA-27B at 59.45. Metis-9B also performs worse than the 4B variant. Native memory generalizes, but not with consistent superiority across memory distributions. What the ablations reveal. Removing multi-fact auxiliary data reduces the paper’s four-benchmark aggregate from 33.14 to 29.68, a relative drop of 10.43 percent. Removing both multi-fact and memory-pollution data lowers it to 26.74, a 19.31 percent decline. The largest deterioration occurs on memory-operation tasks, supporting the idea that an associative latent state will learn brittle shortcuts unless explicitly trained against entity collisions, collateral forgetting, and irrelevant readout. The low-rank study is encouraging from a systems perspective. In the 4B model, a rank-4 decomposition retains about 94.2 percent of the full model’s aggregate performance, while rank 8 is essentially at parity. This suggests that much of the task-relevant state occupies a compact subspace, even though the full memory matrix is dense. General capabilities are preserved only while memory is empty. With an empty state, Metis-4B stays close to its Qwen3.5-4B backbone. It loses roughly one point or less on MMLU-Pro, GSM8K, and multilingual MMLU, while slightly improving on IFEval. After irrelevant information is written into memory, the picture changes. Metis loses 5.10 points on MMLU-Pro, 5.61 on GSM8K, 3.30 on multilingual MMLU, and 22.18 on IFEval relative to the corresponding backbone condition. This may be the paper’s most consequential negative result. A memory architecture should not merely recall relevant facts; it must know when its state should have effectively zero influence. The pollution training helps, but active memory still perturbs ordinary behavior substantially. The case studies expose a related inconsistency. In one forgetting example, the later memory state behaves correctly—the removed fact is no longer returned—but the model’s immediate response to the forgetting command repeats the old value rather than acknowledging its removal. State transition and surface-level instruction following are therefore not fully aligned. Efficiency advantages appear at sufficiently long histories. Metis’s bounded state eventually gives it a latency advantage, but not immediately. In the paper’s context-length sweep, the crossover against full-context Qwen3.5-4B first appears at 64K tokens. The crossover is relatively late because 24 of Qwen3.5-4B’s 32 layers already use linear attention. Between 64K and 128K, full-context prefill increases from 4.34 to 11.60 seconds. Decode time also rises as the remaining full-attention KV caches grow. The estimated Metis memory-read overhead remains small: around 0.091 seconds at 64K and 0.106 seconds at 128K for 32-token outputs. Metis also maintains lower end-to-end latency than Temp-LoRA and (\delta)-Mem throughout the sweep because it avoids optimizer-based adaptation or more expensive state writing. The practical conclusion is nuanced. For short histories, simply replaying context may still be faster and more accurate. Native memory becomes compelling when histories are long enough that repeated retrieval, prefilling, and KV-cache growth dominate the cost. Where Metis Fits Among Existing Memory Approaches. None of Metis’s individual architectural ingredients arrives without precedent. Associative fast weights, linear-attention state matrices, delta-rule updates, recurrent sequence models, and differentiable memory controllers all have substantial histories. What is new is their assembly into a persistent, cross-inference foundation-model state trained with semantic memory operations. Compared with RAG and agent memory databases, Metis eliminates discrete retrieval from the critical path. The model does not need to select passages and reinterpret them as prompt text on every query. The cost is reduced capacity, poor interpretability, and the loss of exact provenance. Compared with test-time training or Temp-LoRA, Metis moves adaptation into a learned forward recurrence. There is no deployment-time loss or gradient computation. But it also does not gain the flexibility of optimizing arbitrary new objectives at test time. Compared with classical memory-augmented neural networks, Metis’s state is coupled more tightly to the Transformer layers rather than exposed as an external table controlled by a separate read–write head. And compared with methods such as Memory3, MemGen, In-place TTT, (\delta)-Mem, MeMo, and MemFT, the paper emphasizes online trajectory information, complete removal of historical text at query time, and instruction-conditioned state transformation. For that reason, the novelty is best understood as a model abstraction and training formulation, not the discovery of fast weights themselves. Calling Metis the first memory foundation model is reasonable under the paper’s specific definition, but it should be read as the authors’ taxonomy rather than a claim that no earlier neural model possessed persistent or differentiable memory. Implications for Embodied AI. For robotics, the attractive property is not conversational recall by itself. It is the prospect of a learned, bounded, persistent state that can follow an agent across a long deployment without rerunning an optimizer or replaying its entire observation history. A household robot could maintain soft semantic information about object locations, user preferences, tool availability, or how a particular person tends to issue instructions. A mobile manipulator could carry latent summaries of recent failures and changes in local affordances. A fleet agent could maintain task-specific state without expanding every planning request with a growing textual log. The fixed-size state and forward-only updates are especially relevant to edge deployment, where optimizer state, unbounded KV caches, and repeated retrieval can be unacceptable. Because the write affects the next interaction rather than the current attention path, implementations can also schedule memory maintenance asynchronously or in parallel with other computations. But Metis should not be used as a replacement for explicit world state. Robot poses, safety constraints, object identities, calibration parameters, and action preconditions generally require inspectable and verifiable representations. A latent associative matrix can be useful for semantic context, but not as the sole authority for facts whose corruption could cause physical harm. The most promising near-term architecture is therefore hybrid:. Native memory for compact, frequently accessed, semantically soft state. External episodic logs for exact replay and provenance. Structured maps or databases for safety-critical state. Periodic consolidation that rebuilds or verifies the latent memory against trusted records. Such a design also addresses Metis’s capacity limit. Native memory becomes something like a learned working or intermediate-term memory, while external storage retains high-fidelity history. It is equally important not to confuse memory with policy learning. Metis can change what information conditions its responses, but its static policy remains frozen. A robot that remembers a failed grasp has not necessarily learned a better grasping strategy. Turning experience into new skills corresponds to the later levels of the authors’ roadmap, not to the capability demonstrated here. Limitations and Open Questions. The first limitation is fixed-capacity compression. Metis’s read cost does not grow with history precisely because all history is projected into a constant-size state. Long-term performance must eventually decline unless the model learns aggressive abstraction, hierarchical consolidation, or selective externalization. The large gap on LoCoMo is already evidence of this bottleneck. The second is interference. Multiple facts share a latent matrix, and the paper observes information confusion even after targeted regularization. Update and forgetting are especially difficult because they require modifying one association without damaging related ones. Third, the meaning of forgetting remains behavioral. The benchmarks test whether the model stops returning a revoked value. They do not show that every recoverable trace has been removed from the state, nor that deletion can be audited. That distinction matters for privacy, safety, and regulatory compliance. Fourth, native memory increases the attack surface. An untrusted observation that reaches the write procedure can alter future behavior long after the original text disappears. Persistent prompt injection becomes persistent state poisoning. Practical systems will need write authorization, per-source trust, anomaly detection, state snapshots, rollback, and strong isolation between users. Fifth, the active-state general-capability losses show that memory and ordinary reasoning are not yet cleanly separated. A useful memory model needs a reliable null operation: when stored information is irrelevant, the model should behave almost exactly like the original backbone. Metis is not there yet. Finally, the evaluation remains mostly textual, static, and benchmark-oriented. It does not test months-long deployment, multimodal sensor streams, nonstationary embodied environments, adversarial writes, or memory migration across model updates. The work is also a July 2026 version-one arXiv preprint rather than a peer-reviewed publication. Bottom Line. Metis’s major contribution is a change in how we define a foundation model’s state. Instead of treating history as text that must be repeatedly retrieved and prefixed, Metis places a learned associative state inside Transformer computation. A slow hyper memory network learns how to select, write, update, and read information. A fast local state changes through forward passes while the learned model weights remain frozen. The experiments establish that this mechanism is real: it can preserve structured facts without query-time context, transfer between Qwen and Llama backbones, generalize to some out-of-distribution tasks, and eventually outperform context replay in latency as histories become sufficiently long. They also establish that the hard problems remain hard. Full evidence is still dramatically more accurate on long and complex histories. Latent memories interfere. Forgetting is difficult to verify. Active memory can degrade unrelated reasoning. And fixed-size state cannot provide unlimited episodic capacity. So Metis is not yet a replacement for RAG, logs, maps, or structured memory. It is better understood as the first serious prototype of a complementary capability: a foundation model with its own learned, mutable, persistent internal state. For embodied agents, that is a consequential direction. The model is no longer only a policy conditioned on the current observation. It begins to look like a dynamical system whose internal state evolves with experience—and whose future behavior depends on what it has learned to retain.