Latest / Embodied AI 101 / MIKASA-Robo-VLA: A Memory-Intensive Benchmark for Vision-Language-Action Robotics
Transcript
- MIKASA-Robo-VLA: A Memory-Intensive Benchmark for Vision-Language-Action Robotics. Memory is a key ingredient for intelligent robotics, especially when tasks unfold over time or with partial observations. To systematically study how much memory an embodied agent needs, the MIKASA-Robo team (led by Cherepanov, Kachaev, Kovalev, Panov, et al.) has released MIKASA-Robo-VLA v1.0.0: a new benchmark suite of tabletop manipulation tasks designed explicitly to test memory in Vision-Language-Action (VLA) policies. In plain terms, it is a collection of robotic tasks (many with natural-language instructions) where the robot must remember information across delays, occlusions, or multi-stage goals. This extends their earlier MIKASA-Robo benchmark (32 tasks) now into the language-conditioned VLA setting (90 tasks). It comes with environments based on the ManiSkill3 simulator, standardized observation and action formats, pre-generated trajectory data, and an evaluation protocol to make comparing algorithms straightforward. Background: Memory in Robotic RL and the MIKASA Framework. The need for memory in reinforcement learning (RL) and robotics is well-acknowledged but has lacked a rigorous benchmark. Many RL algorithms include recurrence or memory (e.g. LSTMs, transformers) to cope with partial observability and long-horizon dependencies. However, there was no unifying benchmark suite focused on memory. The MIKASA (Memory-Intensive Skills Assessment Suite for Agents) project aimed to fill this gap for RL. In particular, the original MIKASA-Robo benchmark introduced in 2025 presented 32 tabletop manipulation tasks, each carefully designed to require some form of memory (e.g. remembering a hidden object’s position or recalling a color shown earlier). These tasks were categorized by memory type (object memory, spatial memory, sequential memory, memory capacity) and difficulty (number of objects, obstacle speeds, etc.). The suite was built on the ManiSkill3 simulator, enabling efficient parallel physics on GPUs and a common interface for all tasks. In short, MIKASA-Robo demonstrated “a unified benchmark that enables systematic evaluation of memory-enhanced agents across diverse scenarios,” specifically tabletop manipulation. The tasks were presented in a Gym-like interface with vector state observations, and baseline controllers (open-loop or with LSTM) were evaluated. The video game “shell game” (cups covering a hidden ball), color/shape memory games, moving-target interception, and object reaching under occlusion were among the 32 tasks. By anchoring evaluations in these representative tasks, MIKASA-Robo set the stage for more systematic memory-RL research. Expanding to VLA: Language-Conditioned Tasks. While the original MIKASA-Robo covered memory in a purely RL context (vector or image observations, no language), the new MIKASA-Robo-VLA is explicitly for Vision-Language-Action research. In a VLA setting, the policy receives natural-language instructions and visual inputs, and outputs actions for the robot. For example, a task might come with the instruction “Touch the red cube after the green cube is hidden,” or “Pick up the shape that was shown first.” These instructions provide a context or goal that the robot must parse, while still requiring memory to achieve the goal when key information is hidden or delayed. The benchmark documentation emphasizes:. “MIKASA-Robo-VLA extends the MIKASA-Robo memory benchmark to language-conditioned Vision-Language-Action research. It provides tabletop robotic manipulation environments that require an agent to retain and use information across delayed, occluded, temporal, or multi-stage interactions.”. In practice, this means the VLA tasks all come with a field describing the goal. The agent must condition on this language and the vision input, then act so as to satisfy the instruction. Unlike static RL tasks, the language can make the tasks more varied and general. Importantly, every new task in the VLA suite ships with a textual instruction, ensuring that language understanding is part of the challenge. MIKASA-Robo-VLA comes with 90 distinct tasks, up from 32 in the original RL release. These tasks span ten different memory types (roughly double the types in RL release), and include many new multi-step scenarios. Tasks now cover the classic memory categories (object memory, spatial memory, sequence recall, memory capacity) and add others like temporal sequencing and combined attribute recall. For instance, tasks include “ShellGame” (find the hidden ball under shuffled cups), “RememberColor”, “RememberShape”, extended tasks like “FindImposterColor/Shape” (identify the one odd cube out by color/shape), “BunchOfColors” (touch a set of previously shown cubes in any order), “ChainOfColors” (touch cubes in the sequence they were shown), and novel ones like “BatteriesChecker” (manage battery states?), “BlinkCountButtonPress” (count blinking lights then press a button) and “GatherAndRecall” (collect then swallow information). Two illustrative examples:. Shell Game (Touch/Push/Pick): A small ball is placed under one of several cups. After a delay or shuffle (with the robot’s view occluded), the agent must remember which cup hides the ball and touch/push/pick up that specific cup. This tests object-centric memory under occlusion. RememberShapeAndColor (3×3 grid): A cube has a particular shape and color. The agent sees it, then other objects intervene. Later the agent is shown a grid of cubes with various shapes/colors and must pick the one matching both the original shape and color. This tests multi-attribute memory. Each task typically has multiple modes or difficulty levels (e.g. different horizon lengths, different numbers/colors of cubes) to vary the challenge. For example, “Maybe RememberColor3”, “RememberColor5”, and “RememberColor9” vary the number of color choices. “Intercept” tasks have Slow/Medium/Fast balls. “ShellGame” has Touch, Push, and Pick modes (the agent must touch, push, or pick the target cup). This yields 90 task-environment variants in total, grouped into categories by memory type. All these tasks use robotic manipulators (often a Franka arm or TriFinger) and objects on a table, in simulation. The environments are implemented via ManiSkill3 and Gymnasium (Gym) so that agents interact in a standard API. The PyPI project description notes key features:. 90 memory tasks across 10 memory types, horizons from 25 up to 2,160 steps, with multiple difficulty levels. Natural-language instructions for every task. Trajectories collected via PPO or motion planning oracles: 22,500 episodes (>6 million timesteps) released in standard formats. Calibrated dense and normalized rewards for each task, enabling both offline imitation learning and online RL. Full GPU-parallelizable simulation via ManiSkill3, and customizable env settings. Together, these changes make MIKASA-Robo-VLA a first-of-its-kind benchmark for VLA-memory. It preserves the “systematic evaluation of memory-enhanced agents” vision of MIKASA but adds language and scale. Environments and Observations. Under the hood, MIKASA-Robo-VLA environments are built on the ManiSkill3 simulator (which itself uses NVIDIA PhysX or other backends) and exposed via the Gymnasium API. Each task is registered under a name like or. The environments share a common structure:. Simulated Robot: Most tasks use a robotic arm (like Franka Panda) mounted over a white table, holding a gripper. Some tasks use a tri-finger robot (as ManiSkill often does) but the concept is similar. The agent controls the end-effector in Cartesian space (mode ) or sometimes joint space; typically tasks use PD control on the end-effector pose. Observations: The agent gets one or two RGB camera images and proprioceptive state. By default, two RGB arrays are provided: one top-down view of the table, and one view from a camera on the gripper or side angle. In practice the code concatenates them along channel dimension, giving shapes like (2×RGB images). Additionally, there is a 1D vector of “joints” containing the robot’s joint positions and velocities plus TCP (Tool Center Point) pose. All these come preprocessed as PyTorch tensors since the environments are vectorized in parallel. For training an RGB+joint policy, one typically sets and flags when making the env. There is also a “state” mode that bypasses vision and provides a full vector state (object positions, colors, etc, plus robot state, plus an oracle cue). This is essentially loophole-training where the agent trivially sees all needed info; it’s used mostly for debugging or to train an oracle agent. But for memory testing, the recommended mode is RGB+joints without oracle. In that mode, all memory-relevant information (like initial object positions or cues) is only visible in the images or proprioception when it actually occurs, so the agent must learn to remember it. Language Instruction: In addition to visual/state obs, each episode includes a language instruction specifying the task goal. The wrapper ensures this string appears parsed in the observation dict, typically under a field named or. For example, in a “Find Imposter Color” task, the instruction might be “Touch the cube whose color is different from all others.” The policy network can thus attend to this text along with the visual input (for a VLA model). The static language is crucial: it conditions the policy on the specific task (instead of hardcoding each task separately) and brings these tasks into the VLA paradigm. Actions: Actions are usually 7 or 8-dimensional vectors representing end-effector Cartesian deltas (and maybe a gripper open/close). The environments accept a torches Tensor of actions conforming to their action space, so agents output in the continuous space. The tasks often terminate when they detect success, or when a timestep limit is reached (most tasks have timeouts like 60–180 steps, though some multi-stage tasks can last hundreds). Rewards: Each task has its own reward function. The design philosophy in VLA version is to provide dense and normalized rewards: that is, even if the ultimate goal is binary success, the environment gives intermediate shaping or normalized incremental rewards so that offline imitation on the provided expert data is feasible. The wrappers calibrate these rewards so values remain in a similar range across tasks. Agents can train with sparse success rewards or the dense reward, as needed. (For example, “touch-the-object” tasks might give a small step reward for moving closer, and +1 at success.). Parallel Simulation: Behind the scenes, all 90 environments are implemented using parallelized simulation (the Gymnasium interface supports creating in one process). Thanks to ManiSkill3’s GPU acceleration, one can run dozens of envs in parallel on a GPU or CPU. In summary, the technical stack is similar to other modern open-source RL benchmarks: a Gym-like API, vectorized envs, and well-defined observation/action spaces. The novelty lies in the task suite and the memory focus. Task and Memory Types. Let’s examine representative tasks and what memory they test. The MIKASA-Robo-VLA suite covers ten broad “memory types,” but we can cluster tasks intuitively:. Object Memory: Tasks where one must remember which object had a certain property before occlusion. Shell Game is classic: a ball is hidden under one of three cups. After a few seconds (and possibly camera shuttered or cups shuffled), the robot must remember which cup covers the ball and interact with it. RememberColor and RememberShape tasks show a cube of a certain color/shape at the start, then present multiple choices later. The agent must recall and choose the correct cube among distractors. For example, in “RememberShapeAndColor” (mode ), the agent first sees a cube that is a blue cylinder. Later a 3×3 grid of cubes appears; to succeed it must find the cube that is both blue and cylindrical, rather than dropping color or shape alone. These tasks test selective recall of attribute information. Spatial Memory: Tasks requiring remembering where something is or was. Intercept and InterceptGrab have a rolling ball or cube that the sensor sees for a moment and then moves. The robot must predict where it will be (based on a learnt velocity) and intercept it at a goal later, implying it had to remember the trajectory. RotateLenient/Strict tasks present a peg at some angle; the agent hears a command like “rotate the peg by +45°” and must do so without displacing its base. The agent must remember the starting orientation to reach the target angle. TakeItBack is a two-phase task: move a cube to a target zone, then after a pause return it to where it started. Success requires remembering the initial location. Sequential/Temporal Memory: Tasks where a sequence of information arrives over time. BlinkCountButtonPress (new VLA task) might flash a light several times, requiring counting, then the robot must push a button a corresponding number of times or at certain intervals. TraceShape and TraceShapeSeq are like drawing: the agent sees an arbitrary shape traced on paper, then has to draw it from memory or follow a sequence. ChainOfColors shows a list of colored cubes one by one (like “red, green, blue”), then the agent must choose them in that same order across the table. This is unlike “BunchOfColors,” which shows all colors simultaneously and only requires touching the set (order doesn’t matter). Sequential memory tasks test working memory for order/time. Memory Capacity / Multi-item: Tasks that load the memory with many items at once. BunchOfColors (mode ) shows n cubes of distinct colors simultaneously. The agent can touch them in any order, but must recall all n colors at least once. As n grows (3, 5, 7), this demands higher capacity. Similarly, SeqOfColors shows a sequence of cubes one by one, then later the agent must select the set of cubes shown (any order) – again testing holding multiple elements in memory. These tasks probe how much information the agent can juggle. Complex Combined or Visual Tasks: Several tasks blend features. FindImposterColor (and shape variants) show an array of objects where one object has a unique color (or shape). The robot must pick the “imposter.” This involves scanning the array, detecting the odd attribute out, and remembering it while reaching. GatherAndRecall might involve moving objects from one region to another and then recalling something about them. BatteriesChecker sounds like monitoring multiple battery objects and acting on their final states (not entirely sure). TimedTransfer probably involves waiting for a timer or sequence before performing a transfer, testing temporal coordination. In summary, tasks are carefully constructed to isolate different memory skills: remembering what objects were (object memory), where they were and where they must go (spatial memory), the order of events (sequential memory), and handling many items (capacity memory), plus hybrid tasks. Each task has a crisp success condition (often binary success/failure) and partial rewards that encourage progress. Because language instructions accompany each task, the agent also has to parse instructions like “Pick up the cube that was red earlier” or “Press the switch after 3 blinks,” which themselves make the task definition part of the challenge. Dataset and Oracles. A major feature of MIKASA-Robo-VLA is the inclusion of expert trajectory datasets for all tasks, to facilitate offline learning or imitation. In practice, the authors trained strong oracle policies for each task (using PPO or even classical motion planning) and collected rollouts. They then released 22,500 episodes (~6 million timesteps) of high-quality demonstrations on Hugging Face, in two standard formats: RLDS (Reinforcement Learning Dataset Specification) and LeRobotDataset v3. This means you can immediately load the data for e.g. “ShellGameTouch-v0” or “RememberColor9-v0” and use them for offline imitation or offline RL without any conversion. For example, an episode file in RLDS format contains arrays for (90×128×128×6), (90×25 features), (90×8 dims), (90×1), flags, etc. A quick look at one episode shows two camera frames per step, the 25-dim joint state, etc. (The VLA RLDS format similarly includes the language instruction in the episode metadata or in the observation stream.) The episodes are long enough for the task horizon (some tasks timeout at 90 steps, others at 180 or more). These datasets have two main uses:. Offline RL / Imitation: Researchers can train policies in an offline manner (e.g. behavior cloning, offline model-based RL) on these trajectories. Because the data has “normalized-dense” rewards, one can also train with off-policy RL algorithms. Oracle Baselines: The trajectories can serve as oracles for evaluation. For instance, one can run a k-Nearest-Neighbor or behavior-cloning policy and see how close it gets to the oracle performance. Furthermore, the repository provides code to collect more data if needed. The oracle checkpoints (networks) are available, so one can generate unlimited episodes with different seeds or gather fresh data. This is useful for scaling experiments or investigating generalization. The focus on providing ready-to-use datasets shows the team’s intent to make the benchmark accessible and immediately useful. Evaluation Protocol. MIKASA-Robo-VLA includes a prescribed evaluation framework to ensure fair comparisons, similar to other RL benchmarks. Some key points:. Horizon Splits: The 90 tasks are partitioned into Short, Medium, and Long horizon sets. This grouping is designed so that multi-task training is tractable: an agent might train on the Short tasks simultaneously, then on Medium, then on Long, or some combination thereof. Each horizon split contains tasks of varying memory types but with similar time length. At evaluation time, one can test on one split at a time or all tasks. Seeds and Reproducibility: The evaluation protocol fixes random seeds for environment and agent resets, making runs reproducible. The repository even includes a reference script that runs a dummy or baseline policy for a quick sanity check. The canonical runs are expected to follow this protocol of seeds and splits. Metrics: Typically, success rate (percentage of episodes solved) and/or cumulative reward is reported per task. Because tasks have sparse success criteria, success is often the fairest metric. The instructions advise outputting JSON results for each task with fields like. There is support for resuming interrupted runs and for a rich terminal display. Bootstrap Baselines: The authors provide sample training scripts and baselines (e.g. PPO with MLP or LSTM backbones) for reference. In particular, they demonstrate (in the original RL version) that a memoryless MLP agent fails on these tasks unless given full state. The purpose is not to publish SOTA results but to validate the benchmark and to give researchers a starting point. Integration with VLA Harness: The benchmark is also architected to work with the new VLA Evaluation Harness by AllenAI. This framework unifies multiple VLA benchmarks; indeed, MIKASA-Robo is listed as one of the integrated benchmarks. This means researchers can run their VLA model across LIBERO, CALVIN, MIKASA, etc., with a single interface. The harness provides Docker images for each benchmark, including MIKASA-Robo, which simplifies the setup. In practice, a researcher using MIKASA-Robo-VLA would pick their agent (say a transformer-based VLA policy), wrap it in the provided API (applying the so the obs/actions are in the right format), and then run the agent on each task for a fixed number of episodes. The harness or the Python API collects results task-by-task. The evaluation guidelines also suggest comparing across the three horizon splits and possibly reporting aggregated metrics if doing a multi-task training evaluation. Significance in the VLA and Robotics Landscape. Why is MIKASA-Robo-VLA important? From a research perspective, it provides a structured testbed for a growing class of models – namely, vision-language-robot policies. Recently there has been a surge of interest in generalist robot models (e.g. RT-1, OpenVLA【21】, MemoryVLA【20】, etc.). These large pretrained models can interpret language and vision and output actions. However, evaluating them systematically is hard: existing benchmarks (like LIBERO, Maniskill, etc.) cover many tasks but none explicitly focus on memory demands or on articulated natural language instructions in the loop. MIKASA-Robo-VLA fills this gap by. Emphasizing Memory: Every task is contrived to break memoryless policies. A pure feedforward policy (or a Markov policy) will fail many tasks if not given explicit memory. For example, in Shell Game, if the camera view is cleared after shuffling, a simple policy that doesn’t remember the initial ball location cannot solve it. By isolating memory as the core challenge, MIKASA-Robo-VLA forces VLA models to use some form of internal state or memory (e.g. recurrent or transformer memory) to succeed. Including Language: Since each task comes with an instruction, models must also understand and ground language. A language-conditioned policy that “memorizes” everything might still need to map the instruction to the right goal. This combination of vision, language, and memory is at the frontier of robot learning, and MIKASA-Robo-VLA provides a benchmark for it. Covering Varied Memory Skills: The diversity of tasks (90 in total) means no one trick suffices. Some tasks require short-term working memory (e.g. holding a count for a few steps), others require long-term recall (e.g. remembering a position after 100 steps), and others require sequence encoding (chain of colors) or multi-object reminders. This breadth makes the benchmark comprehensive for “memory skills” evaluation. Bridging Online and Offline: By providing both environments for on-policy training (RL or imitation) and large offline datasets, the suite is useful to both RL researchers and those working on offline or imitation learning. One could evaluate a large VAE or transformer policy with offline fine-tuning, or train an LSTM-PPO online. Part of a Unified Evaluation: As mentioned, MIKASA-Robo is now part of the unified VLA evaluation harness and leaderboard. This means results on these memory tasks will be directly comparable to results on other benchmarks. If a new VLA model is proposed, a researcher can point to its MIKASA performance as evidence of how well it handles memory-intensive instructions. For practicing roboticists, MIKASA-Robo-VLA is a “lab bench” for prototyping and stress-testing policies. If you develop a new memory architecture (say, a transformer with an external memory bank), you can plug it into MIKASA and see which tasks it solves or fails. The actions are at simulation speed, but the tasks are physically realistic (manipulators on a table with realistic lighting and occlusions). One could even envision extending to real robots in future: the fundamental challenge (e.g. a shell game with hidden ping-pong balls) could be translated to hardware, allowing sim-to-real comparisons. Finally, benchmarks like this often spur innovation. When LIBERO appeared for semantics-generalization, or RLBench for manipulation skills, they led to a lot of follow-up research. We expect MIKASA-Robo-VLA to play a similar role for memory+VLA. Indeed, the references already include works like MemoryVLA which explicitly address memory in VLA models; having a standard suite to evaluate such models will accelerate progress. In summary, MIKASA-Robo-VLA v1.0.0 is a comprehensive, well-engineered benchmark tailored for the next generation of vision-language-robotics research. It provides the tasks, the data, and the evaluation pipeline to systematically study how embodied agents can remember while obeying natural language instructions. As the PyPI description highlights, it is “a memory-intensive robotic manipulation benchmark for Vision-Language-Action research.”. We highly recommend researchers in robotic learning, especially those working on VLA models or long-horizon tasks, to incorporate it into their evaluation regimen. References: The MIKASA-Robo-VLA documentation and code, as well as the original MIKASA-Robo paper, provide further technical details.