Latest / Embodied AI 101 / MemoAct: Why Robot Memory Needs Both a Scratchpad and an Archive

Transcript

MemoAct: Why Robot Memory Needs Both a Scratchpad and an Archive · Embodied AI 101

• MemoAct: Why Robot Memory Needs Both a Scratchpad and an Archive. MemoAct: Atkinson–Shiffrin-Inspired Hierarchical Memory-Augmented Policy for Robotic Manipulation • Liufan Tan, Jiale Li, Gangshan Jing • Chongqing University • IEEE Robotics and Automation Letters • 2026. Research note: this episode is based on the substantially revised arXiv v2 posted on August 2, 2026, which is associated with the IEEE DOI. The original preprint appeared on March 19, 2026, and some older summaries report results from that earlier version. The Robot Has Seen This Before—but Does It Remember What Happened?. One of the easiest ways to expose a weakness in a modern manipulation policy is to show it the same scene twice and require two different actions. MemoAct opens with exactly this kind of failure. In its Sequential Hammer Tap task, the robot must move a hammer over a red target, tap red, tap blue, and then tap red again before stopping. At the first and second red taps, the current camera observation can be nearly identical. What differs is the hidden task state: whether the blue target has already been tapped. A policy conditioned only on the current image cannot reliably distinguish those moments. The problem is not visual recognition or trajectory generation. It is perceptual aliasing in a partially observable process: different underlying states generate essentially the same observation, yet demand different actions. The history is therefore part of the state. This issue appears throughout long-horizon manipulation. A robot may need to remember which drawers it has checked, where an object was originally placed, how many times it has rung a bell, or which member of a sequence comes next. Increasing model capacity does not automatically solve the problem if the relevant evidence is no longer present in the input. The obvious remedy is to provide an observation window. But a window introduces a fundamental trade-off. A short, high-fidelity window is good at tracking recent progress but forgets early events. A heavily compressed history spans more time but may blur together the small temporal differences needed to distinguish “just before the blue tap” from “just after the blue tap.”. MemoAct’s central claim is that these are not the same memory problem and should not be handled by the same storage mechanism. Recent history should remain relatively intact. Distant history can be compressed. Current perception should have its own representation before either form of temporal storage is considered. That leads to a three-tier system inspired by the Atkinson–Shiffrin model of human memory: sensory memory, short-term memory, and long-term memory. The biological analogy is loose, but the engineering principle is crisp: assign different fidelity, capacity, and update rules to information at different temporal scales. The Design Thesis: Separate Precision From Retention. Prior memory-aware robot policies tend to fall into several broad categories. FIFO memory banks keep the most recent features and discard the oldest. MemoAct associates this pattern with approaches such as SAM2Act and HAMLET. FIFO storage preserves recent details but gives the policy a hard temporal cutoff. MemoryVLA takes a different approach. Rather than always evicting the oldest item, it merges adjacent memory entries that appear similar. This keeps some representation of earlier history, but every merge is lossy. If two temporally distinct states have similar visual embeddings, their distinction may disappear. MTIL encodes full history with a Mamba-style state-space model, while SeedPolicy maintains an evolving fixed-size latent state. These approaches avoid an explicit sliding window, but they still compress potentially long trajectories into bounded hidden representations. MemoAct argues that this can make exact retrieval of a small but important historical cue difficult. The paper’s answer is not to choose one of these strategies. It combines them. The short-term bank stores recent sensory embeddings without additional temporal compression. The long-term bank receives compressed summaries of older short-term entries. When the long-term bank itself fills, similar adjacent summaries are merged. The result is a high-fidelity recent scratchpad backed by a lower-fidelity temporal archive. There is an important qualification to the paper’s use of the word “lossless.” The short-term bank does not store raw images. Every image has already been compressed into a single sensory embedding. “Lossless short-term memory” therefore means that MemoAct does not further merge or summarize those sensory tokens while they remain in the short-term bank. It does not mean that the visual observation is preserved without information loss. That distinction matters because the paper’s largest remaining weakness turns out to be the initial visual compression itself. Walking Through the MemoAct Architecture. MemoAct consists of three major components: a sensory distillation module, a hierarchical temporal-memory module, and a diffusion-based action decoder. At every policy invocation, the system extracts a compact representation of current perception, retrieves relevant history, fuses present and past, predicts an action chunk, and then writes the current representation into memory. Sensory memory: one token for the present. The sensory distillation module begins with visual observations from global and wrist viewpoints. Although the experimental cameras are RGB-D devices, the described policy architecture explicitly encodes RGB images; depth does not appear as a direct input to MemoAct’s visual path. Each image is processed by a frozen DINOv2 encoder, producing patch-level visual features. A Transformer decoder layer then uses one learned readout token to query those patches. Irrespective of the number of image patches—and apparently across the multi-view feature set—the output is compressed into one visual embedding. Robot proprioception is separately projected through a multilayer perceptron into the same latent dimension. The visual and proprioceptive representations are combined through an attention-style residual fusion block, yielding a single sensory-memory token for the current time step. This is computationally attractive. The temporal memory never has to store a full grid of DINO features. Every policy step contributes only one vector. But the architecture is making a strong information-bottleneck decision before temporal reasoning even begins. The token must simultaneously encode object identity, geometry, task progress, robot configuration, and any spatial detail that may become important much later. Once omitted, that information cannot be recovered by the long-term memory. There is also a small architectural subtlety worth noting. As written mathematically, both the distilled visual representation and projected proprioception are single tokens. With only one key-value item, the softmax in the cross-attention calculation has no competing locations over which to distribute attention. Much of the fusion’s practical value may therefore come from the learned projections, residual path, and normalization rather than selective retrieval over multiple state tokens. Short-term memory: the high-fidelity scratchpad. The short-term memory bank has a default capacity of six sensory embeddings. New sensory tokens enter this bank in chronological order. While an embedding remains here, it is not merged with neighboring entries. The retrieval network can examine the individual recent states and their temporal positions. This is intended to preserve distinctions such as whether a button was clicked one transition ago or whether the robot is immediately before or after a repeated-looking subtask. The temporal duration represented by six entries depends on the deployment loop. MemoAct predicts action chunks rather than single actions, but the paper does not specify enough timing detail to translate six policy entries into a fixed number of seconds. In practice, the useful short-term horizon would depend on the action-chunk size, replanning frequency, and whether entire chunks or only their first few controls are executed. Consolidation: moving old events into the archive. When the short-term bank reaches capacity, MemoAct takes the oldest three embeddings and adds learned temporal positional embeddings. A learned summary token and a two-layer causal Transformer encoder then distill those entries into one summary representation. The three original entries are removed, and the summary is transferred to the long-term memory bank. The default long-term bank has eight slots. Once those slots are full, MemoAct computes cosine similarity between every pair of adjacent long-term entries. It selects the most similar adjacent pair, replaces them with their arithmetic mean, and frees one slot for future information. Restricting the comparison to adjacent entries is sensible. It preserves broad temporal topology and avoids merging similar-looking events from entirely different parts of an episode. At the same time, the merge remains a heuristic. Visual similarity does not guarantee task-level redundancy. Two identical-looking button presses may be precisely the events that must remain distinguishable. The arithmetic mean also carries no explicit weight for how much history each entry summarizes. One token might represent three original observations, while another may already represent several recursively merged summaries. Averaging them equally does not preserve their relative temporal duration or evidence count. The overall memory capacity is bounded: six recent tokens and eight archived tokens by default. What grows is the span of history represented by those tokens, not the amount of retained information. For very long episodes, the archive becomes progressively coarser. Retrieval: use the present to ask the past. At action time, the short- and long-term entries are treated as a joint memory context. Learned positional embeddings are added, and a three-layer temporal Transformer encoder models relationships among the stored entries. The current sensory token then serves as a query to a two-layer Transformer decoder. Cross-attention retrieves the historical information judged relevant to the current scene. A learned gate combines that retrieved feature with the unmodified current sensory token, dimension by dimension, producing the final conditioning representation. The gate is important because not every movement should be dominated by history. Reaching toward an object may depend primarily on current geometry, while deciding which object to approach may depend on task progress. The gate gives the policy a learned continuum between reactive control and history-conditioned control. Only after action generation is the current sensory embedding written into memory. This ordering prevents the retrieval operation from trivially attending to another copy of its current input. Action generation: diffusion conditioned on memory. MemoAct’s action decoder follows the Diffusion Policy family. A UNet-style noise predictor starts from a noisy action trajectory and iteratively denoises it while conditioned on the fused current-and-history representation. The implementation uses a DDPM scheduler with 100 sampling steps. Training minimizes the standard noise-prediction mean-squared error. Memory is therefore not responsible for producing a symbolic plan. It supplies a conditioning token to a continuous trajectory generator. The same architecture can, in principle, represent multimodal low-level behavior while selecting among modes based on historical context. The 100-step sampler is nevertheless a practical concern. It is reasonable for a research system predicting action chunks, but it may create substantial inference latency for high-frequency reactive control. Faster diffusion schedulers, consistency models, or a non-diffusion action head would be natural deployment variants. Stateful training rather than shuffled imitation learning. A memory policy cannot be trained correctly by independently shuffling observation-action pairs. MemoAct instead streams each episode in strict chronological order. The memory is cleared at the beginning of an episode and populated as its observations are processed. Batches contain contiguous samples from one trajectory. If the target action chunk extends past the trajectory endpoint, the terminal action is repeated as padding. The authors train with a learning rate of (10^{-4}), and the memory write uses a stop-gradient version of the sensory embedding. This stateful data pipeline is more than an implementation detail. Researchers adapting MemoAct cannot simply attach the module to a conventional randomly shuffled imitation-learning loader. Training, validation, rollout, and checkpoint logic all need explicit memory resets and episode boundaries. The reset assumption also creates a deployment question: who tells the memory system that one task has ended and another has begun? Without reliable boundary detection, stale entries from the previous episode could contaminate the next decision. MemoryRTBench: Measuring Memory Rather Than Just Failure. The second major contribution is MemoryRTBench, a set of six RoboTwin-derived simulation tasks organized around three kinds of memory. Sequential memory is tested by Sequential Hammer Tap. The robot must preserve the order of subtasks despite repeated-looking observations. Spatial memory is tested by Block Place and Return, Sequential Transfer and Return, and Interleaved Transfer and Return. These tasks require recalling initial object configurations after intermediate manipulation has altered the visible scene. Episodic or counting memory is tested by Lift Bottle Twice and Click Bell Twice and Clock Once. Here the robot must remember how many times an event has occurred, even when each repetition returns the scene to a similar state. MemoryRTBench reports more than success rate. Sequential-memory error measures skipped, repeated, or misordered subtasks. Spatial-memory error measures failure to restore or act relative to the original scene. Episodic-memory error captures incorrect repetition counts. Low-level control error covers localization, grasping, or contact failures unrelated to memory. Success is determined automatically by task-specific rules, while failure categories are manually assigned from rollout videos. This separation is valuable: a policy that remembers the correct target but misses it physically should not be diagnosed as having forgotten the task. On the other hand, manual failure labeling introduces subjectivity, and the paper does not report annotator agreement. Simulation uses a Cobot Magic mobile manipulator in RoboTwin 2.0. Each simulation result is averaged over seeds zero, one, and two, with 50 trials per task and seed on MemoryRTBench and 100 trials per task and seed on RMBench. Real-world tests use a Realman RM65-B arm, a TEK gripper, global and wrist RealSense cameras, and 20 trials per task. All experiments were run on one RTX 4090. What the Experiments Show. The headline results are strong:. | Evaluation | MemoAct average success | Strongest compared baseline | Margin | |---|---:|---:|---:| | MemoryRTBench | 94.5% | MVMP, 75.4% | +19.1 points | | RMBench | 49.1% | MVMP, 28.5% | +20.6 points | | Real-world tasks | 77.5% | MVMP, 66.25% | +11.25 points |. The Markovian ACT and Diffusion Policy baselines average only 3.9% and 3.5% on MemoryRTBench, respectively. This is unusually clear evidence that the benchmark genuinely requires historical context rather than merely benefiting from temporal smoothing. MemoryRTBench task by task. MemoAct reaches 84.7% on Sequential Hammer Tap, 100% on Block Place and Return, 100% on Sequential Transfer and Return, 98.7% on Interleaved Transfer and Return, 90.7% on Lift Bottle Twice, and 92.7% on Click Bell Twice and Clock Once. More importantly, the diagnostic annotations assign MemoAct a zero-percent memory-related error rate across the sequential, spatial, and episodic categories. Its remaining failures are classified as low-level control errors. The architecture appears to resolve the intended history ambiguity even when execution still fails physically. The controlled memory baselines reveal the trade-off motivating MemoAct. The FIFO-style SAMP variant achieves 75.3% on Sequential Hammer Tap and 82.7% on Lift Bottle Twice, but drops to 41% on Sequential Transfer and Return and 48% on Interleaved Transfer and Return. Recent progress is available, but early spatial evidence can leave the window. The MemoryVLA-style MVMP variant gets 95% on Block Place and Return and 90.7% on Interleaved Transfer and Return, demonstrating strong retention. Yet it reaches only 45.3% on Sequential Hammer Tap and incurs a 52.7% sequential-memory error rate there. Compression appears to preserve broad historical evidence while damaging precise phase tracking. A fixed-window Transformer variant obtains 86% on Sequential Hammer Tap—slightly higher than MemoAct’s 84.7%—but falls to 58.7% on Block Place and Return and 54.7% on Sequential Transfer and Return. MemoAct is not the best entry in every table cell; its advantage is consistency across memory types. RMBench exposes the unresolved cases. On RMBench, MemoAct achieves 41% on Put Back Block, 53.3% on Swap T, 98% on Rearrange Blocks, and only 4% on Observe and Pick Up. The 98% Rearrange Blocks result is especially notable because MVMP reaches only 21%, consistent with MemoAct’s claim that pure similarity merging can disrupt ordered task progress. But MemoAct is narrowly beaten by MVMP on Swap T—53.3% versus 55%—and its four-percent result on Observe and Pick Up is essentially a failure. The authors attribute the latter kind of failure to visual compression. A cluttered scene may require retaining the exact identity and location of one target among several objects. Compressing all DINO patch features into a single token can preserve scene semantics while losing the spatial precision needed for later retrieval. The project page identifies this one-token sensory bottleneck as MemoAct’s main failure mode. The overall RMBench average of 49.1% should temper any claim that memory-dependent manipulation has been solved. MemoAct substantially improves the architecture-level trade-off, but broad, visually demanding memory remains difficult. Physical-robot performance and modest visual shifts. The four real-world tasks are Click Three Buttons in Order, Put Block Back, Grasp and Release Bowl, and Doll Swap Placement. MemoAct achieves 70%, 70%, 95%, and 75%, respectively. It ties MTIL on Click Three Buttons in Order and MVMP on Put Block Back, then leads the comparison on the other two tasks. Twenty trials per task provide useful physical evidence, although the five-percentage-point resolution and single hardware setup limit statistical conclusions. The generalization tests are narrow but informative. On Put Block Back, changing the relevant object from red to yellow leaves success unchanged at 70%; changing it to green reduces success to 65%. A background shift in Click Three Buttons in Order lowers success from 70% to 55%. The authors’ failure analysis attributes the background-shift drop primarily to localization rather than memory errors. This supports a useful distinction: the memory mechanism may remain functional under a visual shift even while the complete robot policy degrades because its present-time perception is unreliable. Intervening directly on memory. The project page adds a particularly persuasive set of targeted memory interventions. These are conducted at memory-critical states with 20 trials per setting. For Click Three Buttons in Order, deleting short-term memory drives success to zero. Deleting long-term memory retains 90% success, while shuffling stored entries gives 60%. The next-button decision is thus dominated by recent ordered progress. For Put Block Back, deleting short-term memory has no effect, but deleting long-term memory or replacing it with memory from another episode drives success to zero. The policy is relying on episode-specific archived information about the block’s original position. These experiments go beyond ordinary ablation. Rather than retraining a weaker architecture, they manipulate the state of a working policy at the decision point. The resulting task-specific degradation is good evidence that the two banks are not merely redundant extra parameters. It also highlights a safety issue: incorrect long-term memory can be worse than missing memory. Swapping in another episode’s archive causes confident but inappropriate behavior, making memory provenance and reset handling important deployment concerns. What the Ablations Tell Us. Replacing frozen DINOv2 features with ResNet-18 reduces average MemoryRTBench success from 94.5% to 75.7%. MemoAct’s gains are therefore not independent of its visual representation. Fortunately, the controlled SAMP, MVMP, and Transformer variants use the same broader policy setup, so the central comparison among memory mechanisms remains more isolated than comparisons against ACT or Diffusion Policy. Removing temporal positional embeddings lowers average success to 79.7%. Replacing the consolidation Transformer with simple addition also produces 79.7%, and replacing gated fusion with direct addition gives 81.4%. The episodic Click Bell Twice and Clock Once task is especially sensitive: its success falls from 92.7% to roughly 25–29% in these variants. Merely storing feature vectors is not enough. The archive must preserve temporal structure, and the retrieval output must be combined selectively with the present. Memory capacities behave as intended. Reducing short-term capacity from six to four and then two lowers the seed-zero average from 95.3% to 79.3% and 64%. Sequential Hammer Tap deteriorates sharply, while several spatial-return tasks are more stable. Reducing long-term capacity primarily harms tasks that need early spatial evidence. The default—six short-term slots, eight long-term slots, and groups of three short-term entries per consolidation—represents a compromise rather than a universally optimal biological constant. Finally, adding MemoAct’s memory module to the point-cloud-based DP3 policy raises average MemoryRTBench success from 22.1% to 60.1%. The improvements are dramatic on Sequential Hammer Tap, Block Place and Return, and Interleaved Transfer and Return, but much smaller on Sequential Transfer and Return and Lift Bottle Twice. The module is portable, but not a uniform cure for every backbone or failure mode. Why MemoAct Matters. MemoAct’s most important contribution is not that it borrows terminology from cognitive psychology. It is that it turns a vague instruction—“give the robot more history”—into an explicit allocation problem. Memory has at least three relevant axes: fidelity, temporal span, and retrieval cost. A single FIFO window fixes fidelity but caps span. A recurrent state or aggressively compressed archive fixes storage cost but sacrifices recoverability. MemoAct assigns different operating points to recent and distant history. That idea is applicable well beyond this exact diffusion policy. A VLA could maintain uncompressed recent interaction tokens while consolidating older events into a bounded archive. A world-model controller could retain exact recent latent states alongside coarse event summaries. A mobile manipulator could combine a short trajectory buffer with long-term object-location records. The work also demonstrates the value of controlled memory comparisons. SAMP and MVMP transplant representative update rules into a shared policy backbone, reducing the extent to which results can be explained by model size, vision encoders, or action heads. The differing error profiles are more informative than another leaderboard in which every baseline uses an entirely different stack. MemoryRTBench’s diagnostic labels are equally significant. Overall task success entangles recognition, control, planning, and memory. Separating sequential, spatial, episodic, and low-level failures makes it possible to ask whether a new memory architecture solved the intended problem or simply improved reaching accuracy. What MemoAct Does Not Yet Solve. The first limitation is the one-token sensory bottleneck. Long-term memory cannot preserve details that sensory distillation discarded before the write occurred. Object-centric tokens, spatial feature maps, keypoints, segmentation tracks, or compact 3D representations would offer a better substrate for spatial recall. Second, consolidation is driven by schedule and similarity rather than task relevance. Every observation is written; every three old short-term tokens are summarized; and the most similar neighboring long-term entries are merged. There is no explicit novelty detector, uncertainty estimate, event-boundary model, or learned write gate. A rare but crucial observation can be compressed simply because it resembles its neighbor. Third, MemoAct’s “long-term memory” exists only within the current rollout. The bank is cleared at each episode boundary and does not accumulate reusable experience across tasks or deployments. This is working memory at two temporal resolutions, not autobiographical or lifelong robot memory. The authors themselves position cross-episode reusable experience as future work. Fourth, the archive stores sensory and proprioceptive embeddings, not explicit actions, outcomes, or symbolic task events. In tasks where an interaction leaves no durable visual trace, the system must infer what happened from its sequence of sensory states. Storing action chunks and detected event outcomes could make counting and causal attribution more robust. Fifth, MemoAct should not be mistaken for a language-conditioned generalist policy. The described architecture maps current observations and memory into actions without a language or task-instruction input. The paper also does not clearly explain whether tasks use separate policies or joint training. The conservative interpretation is therefore a memory-aware visuomotor imitation architecture, not an open-vocabulary VLA that dynamically changes goals. Sixth, evaluation focuses on structured tasks with known episode boundaries and prescribed sequences. Important unanswered tests include length extrapolation, distractor events, goal changes midway through an episode, contradictory memories, dropped frames, false event detections, and tasks in which the relevant observation occurs much farther back than any training example. Finally, reproducibility remains incomplete at the time of this review, September 1, 2026. The project page’s Code button currently links back to the project page rather than to an implementation repository. MemoryRTBench is available as a 6.71-gigabyte Hugging Face upload, but it has no dataset card describing installation, task generation, splits, or evaluation procedures. That is particularly consequential here because several details matter for faithful reproduction: the ordering of short- and long-term entries, causal-mask orientation around the summary token, gradient detachment across memory updates, action-chunk execution frequency, and reset semantics. How I Would Adapt MemoAct. For a manipulation stack that already emits compact per-step embeddings, the core design is straightforward to reuse: retain a raw recent buffer, consolidate older entries into a separate archive, retrieve from both using the current observation, and gate the retrieved context into the action head. I would change the sensory representation first. Instead of one token for an entire multi-view scene, I would preserve a small set of object- or region-level tokens carrying identity, pose, confidence, camera source, and timestamp. The archive could then merge records per object rather than averaging complete scene embeddings. I would also make writing event-driven. A robot does not need to archive hundreds of nearly identical frames while moving through free space. Memory writes could be triggered by contact, gripper transitions, object displacement, high model surprise, or a learned change-point detector. That would extend the effective horizon without immediately increasing bank size. Long-term merging should be weighted by the amount of history represented by each token and constrained by semantic importance. A summary representing one rare interaction should not be averaged equally with a summary representing a long period of idle motion. Auxiliary objectives—reconstructing object locations, predicting completed subtasks, recovering event order, or estimating repetition count—could make the memory explicitly preserve what future control requires. For a VLA, I would store goal and instruction context alongside sensory and action events. Retrieval should depend jointly on the present scene and current linguistic goal; otherwise the same archive may surface history relevant to an old objective. And I would retain MemoAct’s intervention methodology. Deleting, shuffling, swapping, and corrupting memory at controlled decision points is one of the clearest ways to determine whether a policy truly uses its memory and how it fails when that memory becomes unreliable. Final Assessment. MemoAct makes a compelling architectural argument: recent task progress and distant historical evidence should not compete inside one undifferentiated context window. Its short-term scratchpad preserves phase information. Its compressed archive retains early evidence. Cross-attention retrieves from both, and a learned gate decides how much history should influence the current action. The experiments support that division, particularly through the contrasting FIFO and similarity-merge baselines, the memory-specific failure labels, and the direct intervention studies. The method is not yet a general robot-memory system. It remains episode-local, visually bottlenecked, dependent on fixed-capacity heuristic consolidation, and evaluated mostly on structured task-specific imitation problems. Its low RMBench performance on fine-grained spatial recall makes that boundary especially clear. But as a design pattern, MemoAct is valuable. The lesson is not merely to use more context. It is to decide which parts of history deserve precision, which can tolerate compression, and how the policy will retrieve the right kind of memory at the moment of action.