Latest / Embodied AI 101 / ASPIRE: Can Code Refinement Make Robot Manipulation Generalize?
Transcript
- ASPIRE: Can Code Refinement Make Robot Manipulation Generalize?. ASPIRE: Agentic /Skills Discovery for Robotics • Runyu Lu et al., including Linxi “Jim” Fan • NVIDIA, UMich, UIUC, UC Berkeley, CMU • arXiv • 2026. Cold Open. Imagine your robot closes its gripper, lifts, and comes away empty. The episode is marked failed. That label does not tell you whether perception selected the wrong object, the grasp slipped, or the approach geometry was impossible. You can ask a coding agent to try again. The interesting question comes afterward: what survives the retry?. ASPIRE turns that question into a learning system. It inspects robot execution evidence, repairs control programs, and preserves useful lessons for subsequent tasks. Experience accumulates in software and a reusable skill library. Picture another robot encountering a similar failure with a different arm. Does it inherit a useful strategy, or just a description of something that worked once?. That is the tension behind this episode. We are following a proposed route from failed execution to persistent knowledge—and asking what must happen for debugging experience to become a transferable robotic capability. The Idea. Start with the policy representation. ASPIRE’s coding agent produces programs that call perception, geometry, grasping, and motion-planning tools. A program can inspect a camera image, estimate an object’s geometry, request grasp candidates, execute a trajectory, and observe again. The language model organizes these operations; the robot’s underlying services carry them out. This gives the agent explicit control over both the sequence of actions and the observations used to choose them. Learning happens around a frozen model. What changes is the generated code and the persistent skill repository. The project explicitly frames training as skill refinement rather than gradient descent. Robot trials supply the experience; revised programs and retained instructions carry that experience forward. The resulting learned artifact is therefore partly executable behavior and partly guidance for generating future behavior. The first component is an instrumented execution engine. Each run records the generated program, structured calls and returns, robot actions, and visual evidence. Relevant images include keyframes around motion calls and segmentation overlays. These records connect what the agent requested to what the robot actually did. A failed manipulation becomes something the agent can investigate at the level of individual operations. Think about how that changes a repair. If the object mask is sensible and the motion planner rejects the target pose, changing the object’s name is an unmotivated intervention. Trying another approach direction addresses a different hypothesis. The useful role of the trace is to make those hypotheses distinguishable before the agent rewrites the whole behavior. The second component is a coordinator-managed skill library. Workers attempt tasks and report candidate lessons. The released coordinator instructions require entries to identify when a pattern applies, include working code where appropriate, and record supporting evidence. A separate verifier tests proposed lessons before promotion. Future workers receive the accumulated library, so one task’s repair can influence how another task is programmed. The third component is evolutionary search over programs. The agent proposes competing strategies, executes them, examines their failures, and generates another population from the strongest survivors. The released instructions explicitly ask candidates to test distinct hypotheses. This makes search more substantive than repeatedly adjusting the same offset: candidates can change grasp strategy, localization method, or motion structure. The object being optimized is executable robot software. There is a clear lineage here. Voyager already combined executable code, iterative feedback, and a growing skill library in Minecraft. CaP-X established an experimental framework for coding agents controlling robots through perception and control primitives, including the CaP-Agent0 baseline. ASPIRE brings these threads together around persistent, trace-grounded repair knowledge for manipulation. The distinctive move is the unit of memory. An entire successful trajectory is one kind of experience. A reusable instruction explaining when to change an approach, how to disambiguate a target, or what evidence confirms a grasp is another. ASPIRE organizes learning around the latter. That makes the stored knowledge suitable for interpretation by a new coding agent working through a different robot interface. The Result. First, separate the paper from the subsequent release. The original arXiv submission, dated June 30, 2026, used Claude Opus 4.6 in simulation and GPT-5.5 for real-robot transfer. The October 2026 update adds evaluations with newer models, including GPT-6 Astra and Claude Opus 5.5, alongside online real-robot demonstrations. The release headline is 99% success on its reported set of 80 LIBERO-Pro tasks and selected Robosuite and BEHAVIOR-1K tasks. The official repository now includes the execution framework, experiment workflows, and real-robot instructions. These are author-reported release results accompanying the original research contribution. LIBERO-Pro matters because it changes more than the random placement noise around a familiar demonstration. Its perturbations include moving objects and changing which object the instruction asks the robot to manipulate. A controller must respond to the actual scene and requested goal. That makes it a useful setting for testing a program that explicitly re-localizes targets and selects its actions from observations. For tabletop experiments, the workflow debugs programs on development configurations, selects a program, and evaluates it on disjoint configurations. That separation tests whether the repaired implementation survives changes in initial conditions. It also makes the execution artifact concrete: the thing being evaluated is a particular program, rather than an unspecified collection of successful attempts. The number most directly testing accumulated experience is the original paper’s 31% success on unseen LIBERO-Pro Long tasks, against baselines in the low single digits. Skills come from LIBERO-90; each new task receives a generated program without additional task-specific debugging or retries. Performance increases as the source-task library grows. The ablations support a useful engineering conclusion: adding trace-guided execution and repair produces the largest improvement, with evolutionary search providing further gains. Better evidence and an effective repair loop do substantial work. Physical transfer uses skills discovered with a simulated Franka to guide programming on a bimanual YAM station. Skill retrieval reduces tokens needed to reach a successful program; can lifting and drawer manipulation also improve in held-out physical trials. The transferred resource is guidance for adapting robot programs. Taken together, these experiments address two different questions: can the system refine a task implementation, and can earlier experience help with subsequent programming? The second question is the one that makes the skill library scientifically interesting. It gives persistence a measurable job. The Biggest Weakness. The biggest weakness is the distance between task-specific refinement and broad manipulation generalization. A held-out random seed is not a held-out task family. In the main tabletop workflow, ASPIRE practices the task before its final program encounters new configurations. That is a legitimate test of adaptation and within-task robustness. It does not establish that the same accumulated knowledge will handle unfamiliar physical problems with little additional development. The split protects the evaluation configurations; it does not remove prior access to the task being solved. BEHAVIOR-1K makes this distinction especially important. The released workflow covers two mobile-manipulation pickup tasks, and evaluation agents grow programs block by block using execution feedback. Its scoring instructions count whether the final program succeeded at least once. This measures adaptive instance solving, not a single reusable policy attempting the entire household benchmark. Reading the benchmark’s name as the breadth of demonstrated competence would greatly exaggerate the evidence. The same boundary applies to the expanding real-robot gallery. The repository explicitly describes the newer bottle and cup recipes as distilled from demonstration sessions, rather than a new real-robot validation study. Those examples show concrete behaviors. They do not, by themselves, establish broad continual learning across previously unseen physical task families. The decisive experiment would freeze accumulated experience before introducing new physical task families, then compare that library against an empty library under the same interaction budget. Measure how much new execution and repair each condition needs, across multiple environments and embodiments. That would test the ambitious thesis directly: earlier robotic experience should systematically reduce the work needed for genuinely new problems. The present evidence establishes useful reuse within a more bounded experimental scope. Verdict. Robotics teams with working perception and control stacks should pay attention. ASPIRE offers a concrete architecture for turning integration work into reusable knowledge. The valuable design choice is to expose enough execution evidence that a coding agent can make targeted changes, then retain the changes that survive testing. Its recorded-run format makes the program, actions, and relevant observations available together. For a practitioner, that suggests a different way to prioritize infrastructure. Logging is part of the learning interface. A return status, an image before contact, or the record of a rejected plan can become input to the next policy revision. The question to ask of a tool interface is not just whether an agent can call it, but whether the resulting evidence lets the agent understand what happened. Continual-learning researchers also get an inspectable experimental object. The coordinator’s library contains proposed rules, code snippets, and evidence that can be revised or rejected. That makes it possible to study the quality of accumulated experience separately from the coding model’s underlying capabilities. You can examine what a lesson says and test whether another agent can use it. For VLA researchers, the complementary direction is already visible in Harness VLA. That work exposes a frozen VLA as a callable contact-rich primitive, while an agent handles higher-level composition and maintains experience about how the primitives behave. Its design provides a concrete example of learned motor behavior operating inside an agentic execution framework. A productive synthesis is therefore straightforward: let learned controllers supply local sensorimotor competence, and let the programming layer organize observations, recovery, and reusable procedural knowledge. ASPIRE gives that programming layer a specific mechanism for accumulating experience. The takeaway is simple: a verified repair can be a learning artifact. Store it in a form that another agent can apply, and a successful debugging session becomes an input to future robot behavior. Outro. Return to the empty grasp. The immediate goal is to pick up the object. The longer-term opportunity is to leave behind something that makes the next attempt better informed. Watch what happens when persistent skill libraries become shared experimental artifacts: something another team can inspect, modify, and test on its own robot. That is the compelling idea in ASPIRE. Robotic experience can produce more than a success label or a saved trajectory. It can produce an actionable lesson—and the next program can begin with that lesson already in hand. This is Embodied AI 101.