Latest / Embodied AI 101 / ASPIRE: Automated Skill Discovery for Robotics
Transcript
- ASPIRE: Automated Skill Discovery for Robotics. Robots have long struggled to build versatile repertoires of skills without extensive hand-holding. Traditional development relies on either meticulous human engineering of control subsystems or on heavy reinforcement learning trials for each task. In recent years, researchers have begun to automate parts of this process: techniques like unsupervised skill discovery or hierarchical reinforcement learning can generate diverse behaviors given cleverly designed objectives. Language models and coding agents have also been enlisted to propose tasks and even write code to improve policies (for example, the Eureka system uses GPT-4 to auto-generate reward functions and trains policies using those rewards). But until now there has been no fully open-ended, end-to-end automated framework that continually generates*, refines, and remembers new sensorimotor skills in robotics without human crafting or per-task retraining. Enter ASPIRE, a new system introduced by researchers at NVIDIA (GEAR lab) and partners, which stands for Automated Skill Discovery for Robotics. ASPIRE uses an evolutionary search over a space of control programs to autonomously produce and accumulate reusable robotic skills. Unlike standard reinforcement learning, ASPIRE does not optimize a single end-to-end policy per task; instead, it evolves a library of specialized controllers, each mapping sensory input to motor commands. Over time ASPIRE’s library grows richer, enabling the robot to solve more complex, multi-step tasks by compounding these primitive skills, to transfer solutions from simulation to real hardware with minimal re-tuning, and even to run on different robot morphologies without re-learning the entire controller. In short, ASPIRE provides a continuous loop of discovery and improvement: propose candidate controllers, test them in simulation (or reality), keep the ones that work, and repeat. In this article we dive into ASPIRE’s approach, why it matters, and how it relates to preceding work. We begin by reviewing related ideas in skill learning and evolutionary robotics, then explain the unique automated evolutionary pipeline ASPIRE introduces. We highlight its capabilities – multi-task composition, sim-to-real, and cross-embodiment transfer – and contrast it to contemporary methods (like unsupervised RL or LLM-based agents). Finally, we discuss potential future impacts: robust robotic skill libraries that form the foundation for general-purpose robots. From Manual Engineering to Autonomous Skill Discovery. Roboticists have dreamed of robots that learn like animals: gradually building up a repertoire of abilities through exploration. For decades, evolutionary computation has been applied to robot control tuning. Early work in evolutionary robotics (e.g. genetic algorithms for legged walking) showed that controllers could be evolved for simple behaviors like forward locomotion. Over time, researchers moved away from purely manual programs and towards learned controllers, typically by reinforcement learning (RL). A rich field of unsupervised skill discovery emerged: systems like DIAYN/DADS or Quality-Diversity (QD) search methods aimed to let a robot autonomously discover a diverse set of behaviors without a human-specified reward. For example, Quality-Diversity algorithms have uncovered hundreds of distinct locomotion gaits for legged robots by maximizing diversity as well as performance. However, these approaches have limitations. Often they require hand-defined behavior spaces or rely on auxiliary tasks (e.g. reaching various positions in the plane). In fact, Batra et al. point out that simple locomotion coverage tasks (e.g. moving to every cell on a grid) can be accomplished by one general goal-conditioned policy plus planning. In practice, unsupervised methods often yield many behaviors that are hard to interpret or combine: as Zhao et al. note, RL-driven exploration might produce arbitrary motor fidgets rather than semantically meaningful skills (e.g. you won’t magically get a grasp skill if you only evolved “pushing” motions). Meanwhile, combining skills into complex tasks has traditionally been done by humans or by predefined hierarchies. The notion of a skill library – a set of reusable controllers (like “pick”, “place”, “grasp”, “push”) – is common in robotics, but such libraries have mostly been built by hand or with heavy learning resources. Some recent work has proposed ways to build skills unsupervised. For instance, Jannsonnie et al. automatically generate a large set of manipulation tasks and learn primitive control policies for them. Their system discovered dozens of diverse object-interaction behaviors that can be embedded as “primitive skills” and later composed hierarchically to solve new tasks – even transferring these skills to a real robot after simulation training. In the language-guided family of approaches, systems like the “Agentic Skill Discovery” framework use large language models (LLMs) to propose new tasks from scene descriptions, then run RL on each to learn policies. Likewise, a recent “Lifelong Robot Library Learning” (LRLL) system uses LLMs to suggest new tasks and a memory module to iterate skill-building, growing its own skill library over time. These approaches boot-strap skills from scratch, but they still rely on reinforcement learning loops (training neural policies) or LLM code generation. What has been missing is a gatekeeper-free, fully continuous pipeline that takes a blank slate and autonomously churns out & refines robot skills forever. ASPIRE aims to fill this gap by casting skill learning as a search over programmable controllers. Instead of training neural networks from scratch for each skill, ASPIRE evolves control programs – explicitly structured policies – through optimization. This promises several advantages: (1) Evolving programs can more easily encode discrete decision structure or reusable subroutines. (2) The system retains a history of successful programs and mutates them, so learned behaviors are not lost. (3) Skills come naturally in symbolic form, which aids planning and transfer. As we shall see, ASPIRE’s evolutionary approach yields a growing library of effective skills that can be chained for multi-step tasks, transported from sim to reality, and even applied across different robot embodiments – all without retraining a monolithic end-to-end policy. Autonomous Evolution of Control Programs. At the heart of ASPIRE lies a loop resembling classical evolutionary robotics, but scaled and orchestrated for open-ended skill accumulation. Imagine starting with a goal or task context; ASPIRE instantiates a population of candidate control programs (e.g. parameterized controllers or finite-state machines) for the robot to perform some task or interact with the environment. Each program is tested in simulation: the robot executes it under whatever initial conditions or inputs are relevant, and the outcome is scored by some performance criteria (success at the task, safety, novelty, etc.). Programs that perform well are kept in an archive, while the rest are discarded. New candidates are then generated by mutating or recombining programs in the archive – for example by randomly tweaking numerical gains, adding an action primitive, or rearranging decision branches. The new generation is again tested, scored, and added if they achieve new capabilities. Over many iterations, this procedure builds up a rich ensemble of controllers, each of which reliably accomplishes a certain sensorimotor skill. This process echoes the ideas of evolutionary robotics, which for decades has used genetic search to design controllers. But ASPIRE takes the general idea much further. Instead of e.g. evolving one walking gait for a quadruped, the system is continuously discovering many different behaviors. In effect it is performing a quality-diversity search over the space of controllers: not only does it want each program to work well for some scenario, but it also favors covering new kinds of tasks (for example, rotating an object vs. pushing vs. putting it down). The notion of “compounding multi-task” in ASPIRE’s description suggests that it will compose these discovered programs to solve longer tasks. One can imagine that once ASPIRE has, say, learned a “pick” program and a “place” program, it can sequence them to achieve pick-and-place tasks without having learned those from scratch. Importantly, ASPIRE’s skills are explicit programs rather than opaque policy networks. This makes them interpretable and modular. In practice, a control program might be a small piece of code or a parameterized motion primitive that reads sensors (vision, force, encoder angles) and outputs robot actions. Because it is structured, it can encode logic such as “if object seen, move above it, then grasp”. Evolution naturally explores discrete and continuous spaces of such programs. This contrasts with purely neural approaches, where the policy’s structure is fixed and evolving new “ideas” is harder. By searching in program-space, ASPIRE can mutate entire subroutines or conditional checks, giving it a rich capacity for discovering qualitatively new behaviors. While the inner machinery of ASPIRE’s evolution has not been fully disclosed, we can infer it borrows from recent ideas. Quality-Diversity algorithms have been popular for discovering robotics skills; for example, methods like MAP-Elites maintain an archive of diverse high-performing policies over a large task descriptor space. ASPIRE likely sets up a similar archive of “species” of programs. Unlike standard MAP-Elites which requires manually designing a feature space, ASPIRE naturally distinguishes skills by their achieved goals (which could also be learned). Moreover, it can leverage massive parallel simulation via NVIDIA’s Isaac framework or others, so thousands of candidate controllers can be evaluated quickly. In summary, ASPIRE removes the human in the loop for skill invention. Instead of a reward designer or engineer prescribing subtasks, the system itself continuously hypothesizes new tasks and skills. The only requirement is to define the basic sensing and actuation of the robots and possibly some success criteria (e.g. did the object get grasped). The evolutionary optimizer handles the rest, producing a steady stream of valid control programs that become the robot’s skill library. Learning and Composing Multi-Step Skills. One of the key promises in ASPIRE’s design is task compounding: using discovered skills as building blocks for more complex objectives. Once the system has learned a suite of basic behaviors – say, “grasp object,” “hinge open door,” “insert peg,” etc. – it can combine them sequentially or hierarchically to solve longer-horizon tasks. In classical AI terms, this is like having ready-made primitives in a planning library. Indeed, one can view ASPIRE’s toolkit as defining the nodes of a “skill graph,” where edges indicate feasible transitions between skills. The robot can then plan a path through this skill graph to achieve higher-level goals. This approach has strong grounding in recent research. For example, Zhao et al.’s Agentic Skill Discovery framework forms an initial skill library by bootstrapping tasks with an LLM; the skills learned are then used to solve more advanced tasks in a growing loop. Jannsonnie et al. explicitly demonstrate composing learned manipulation skills with hierarchical RL to tackle unseen assembly tasks. Their system shows that primitives learned on one set of generated tasks can safely transfer to new, related tasks: the robot trained in simulation could apply the combination of learned primitives to solve an assembly problem on a real robot. Similarly, ASPIRE’s evolved skills serve as modular components. A program discovered to, say, pick up a block at a certain angle can be paired with another that places it somewhere else, without retraining either. This is why ASPIRE can handle multi-task problems gracefully. Notably, ASPIRE needs not re-train an entire neural policy to extend tasks. Traditional reinforcement learning often requires retraining or fine-tuning a large network for each new task. ASPIRE instead re-uses the exact discovered programs. For instance, to go from a pick task to a pick-and-place task, ASPIRE might simply run “pick” then “place” programs in sequence rather than training a fresh policy for pick-and-place. This dramatically reduces training time for compound tasks. It also avoids “catastrophic forgetting”: the old skills remain intact and can be re-engaged. In effect, ASPIRE performs skill-level planning rather than policy learning. The planning comes for free because programs are symbolic descriptions of behavior. One can relate this to the Skill Graph concept introduced by Yu et al. in the context of assembly tasks. A Skill Graph is a semantic network where nodes are skills (each with pre- and post-conditions) and edges represent allowed transitions. ASPIRE implicitly builds these graphs by accumulating skills whose pre/post conditions are automatically determined by their success criteria. Once in place, the robot can search over paths in this graph to sequence skills. In complex tasks like assembly or manipulation, this is powerful: the robot can break down a task into sub-goals matching its library. For example, if the task is “open a lock,” ASPIRE might already have “grasp knob” and “rotate wrist” programs that, when chained, achieve the goal without extra learning. Importantly, ASPIRE’s skill compounding is autonomous. The system does not require a human to label or connect skills. Instead, part of the evolutionary search could involve uncovering transitions: for instance, if an evolved program “pick up peg” ends with the peg in gripper, ASPIRE can automatically see that the pre-condition matches a “place peg” program it has. This emergent chaining means ASPIRE can tackle new sequences of tasks that it has only implicitly seen through its component skills, expanding its capabilities without explicit training on each sequence. Bridging Simulation and Reality. A central challenge in robotics is moving from simulation to real hardware. Many learned controllers falter when transferred to a physical robot due to model mismatch or unmodeled dynamics. ASPIRE is designed to minimize this sim-to-real gap. By evolving explicit control programs rather than opaque neural nets, ASPIRE can bake in robustness during search, for example by testing each program under varied simulated conditions (noise, perturbations, parameter uncertainties). Programs that only work under ideal simulation will fail and be discarded. Those that survive are likely to exploit fundamentally correct modes of control that tolerably transfer to reality. In practice, ASPIRE likely leverages high-fidelity simulators (e.g. NVIDIA Isaac Gym, PhysX, or custom digital-twin frameworks) running on GPU clusters. These tools allow running thousands of trial rollouts in parallel, exploring the space of controllers quickly. ASPIRE can also use techniques like domain randomization: each candidate program might be tested on simulated robots with slightly different masses, friction, or sensor noise, to ensure it is robust. The result is a set of programs that not only solve tasks in sim, but also continue working on real robots. This approach is reminiscent of the Real-is-Sim framework. In that work, a high-speed digital twin runs the robot’s policy entirely in simulation, synchronized in real-time with the physical robot. The control commands are effectively executed on a simulated robot, and the real robot simply follows along. This shifts the burden of adaptation onto the simulator’s twin, rather than the policy. ASPIRE could use similar ideas: once a program has proven effective in sim, the virtual copy of the robot could drive the real robot’s joints, with the simulation adjusted continuously by real sensor feedback. In effect, ASPIRE-trained skills “live” in simulation but become executable on the physical robot without retraining. Empirically, the ASPIRE demonstration shows this works: skills discovered in sim are directly applied on hardware. For example, a program that learned to “move a block along a T-shaped track” in a rendered environment also succeeds on the real T-track with the physical robot, with success rates comparable to simulation. That direct transfer is exceptional, and it contrasts with end-to-end learning approaches that often require fine-tuning on real data. Here, the control logic itself is invariant to the sim-real divide, thanks to careful evolution. Transferring Skills Across Robots. Even more striking is ASPIRE’s claim of cross-embodiment transfer: a skill learned on one robot can be used on a different one without any retraining. How is this possible? Because ASPIRE’s skills are defined in a robot-agnostic way. A “pick object” program might specify motions relative to a sensed object frame, not specific joint angles. Given two robots with different geometries but both able to grasp with a compatible gripper, the same high-level program can be interpreted by each. Essentially, ASPIRE learns motion strategies rather than motor commands. It might evolve a program like: “extend end-effector toward object at this pose, close gripper, lift up 10cm,” which can be executed on many arms so long as their control API supports those primitives. This idea resonates with recent work on cross-embodiment learning. Yan et al. trained a single neural policy to control many different robots (arms, drones, mobile bases) for manipulation and navigation tasks, and found that the shared policy improved all test robots by ~20% compared to training separately. In another study, a quadruped policy learned from human demonstrators could be zero-shot adapted to a humanoid manipulator with no additional data. The key insight from these works is that task knowledge can transcend body shape. ASPIRE takes this to heart by building its skills at a semantic level. For example, a program “stir the liquid clockwise” is not tied to a specific hand or grip; if another robot has comparable sensing and a tool, it can execute the same stirring routine. Because the programs are explicit and hierarchical, ASPIRE can map them onto different embodiments via a calibration layer. Crucially, ASPIRE does this without the usual retraining. Once a skill program is evolved, it remains fixed. To transfer, ASPIRE simply adjusts its execution interface for the new robot (for instance, calibrating the pose offsets and gripper command). The program logic – the sequence of actions and decisions – carries over unchanged. As a result, if the system has 10 skills evolved for a 7-DOF arm, those same 10 programs can immediately be run on a different arm (even a mobile manipulator) or on a simulated agent with slightly different joints. Early experiments from ASPIRE’s team show that such transfer often works out-of-the-box, achieving similar success rates on the new robot as on the original. This is a powerful feature. Instead of needing to collect new data or tune policies when switching hardware, the robot simply “plugs in” the skills. In large-scale robotics fleets, this saves enormous effort. A skill discovered on a research robot in simulation can be pushed to production robots in factories without extra training. The generality comes not from big neural nets but from the structured, programmatic nature of the solutions. Relation to Other Techniques. ASPIRE’s approach sits at the intersection of several strands of research. It contrasts with end-to-end deep learning and RL: those methods typically tailor a neural policy to exactly one embodiment and task distribution. Even if some multi-task learning is used, one still usually fine-tunes per task. In ASPIRE, no monolithic network is tuned; the focus is on modular programs that each solve a particular function. This is more akin to classical AI planning, but here the “actions” (skills) are learned through evolution rather than hand-coded. Comparing to reinforcement-learning-based skill discovery, ASPIRE side-steps many pitfalls. RL methods like DIAYN or DADS maximize diversity of behaviors, but as noted earlier they require careful reward engineering and do not keep a persistent library of distinct skills. They also produce implicit, often neural, policies. ASPIRE instead finds explicit, high-level programs without even requiring smooth gradients. It can, in theory, discover behaviors that would be very unlikely to emerge in a random RL search, because evolution can combine entire subroutines in one mutation jump. Compared to LLM-driven agents, the difference is similar: systems like Eureka and the Agentic frameworks use powerful language models to propose or fine-tune training objectives and tasks. These can be effective but still rely on gradient-based training loops (RL or imitation) for the robots. ASPIRE does not use an LLM at all; no natural language is involved in its loop. This makes ASPIRE agnostic to issues of prompt design or LLM reasoning reliability. Instead, it leans on brute-force search powered by massive simulation. Interestingly, one could imagine future hybrids: an LLM might suggest high-level task goals to try, and ASPIRE would then run the low-level loops to discover how to accomplish them. But ASPIRE itself shows that even without any human-supplied task decomposition, autonomous evolution can be enough to fill a skill library. One can also see ASPIRE as complementary to transfer learning methods. For example, the ASPiRe (Adaptive Skill PrIor) work demonstrated using learned priors from previous tasks to speed up RL on new ones (sharing skill embeddings across tasks). ASPIRE learned controllers could serve as such priors or primitives. In fact, ASPIRE’s output is ideally suited to plug into frameworks like adaptive skill priors or meta-learning: each discovered program could be one dimension of a prior used to accelerate RL on unseen tasks. Instead of learning a dense skill embedding space, ASPIRE provides explicit discrete modules that are trivially combined. Lastly, it’s worth noting that ASPIRE is not the first to hint at programmatic control. Some prior work has used genetic programming or decision-tree policies in robotics. A notable early example is the Transferability-based Behavioral Repertoire Evolution (TBR-Evolution) for walking robots, which evolved controllers mapping like a finite state machine to achieve a variety of gaits (music to our context). Those efforts, however, were typically closed-world: they designed the skill genome manually, and often only evolved a set of gaits. ASPIRE generalizes this old idea to an open-ended regime, using modern compute to handle complex robots and tasks. It democratizes the creation of skill programs so that the machine, not the engineer, assembles the repertoire. Why ASPIRE Matters. The implications of ASPIRE are wide-ranging. First and foremost, it cuts down on human effort. Building each skill from scratch – by code or by teaching a policy – is time-consuming. If ASPIRE can automatically invent a skill at the level of human-designed ones, engineers can save that time and instead focus on higher-level goals. A factory could deploy robots that continuously self-improve: when new parts or tasks appear, the system itself can iterate to figure out how to handle them. Second, ASPIRE’s skills are inherently reusable and interpretable. Because each skill is an explicit program, end-users can audit or modify it. For instance, if a “grasp” program is discovered, a human can inspect its logic or adjust it for safety. This is in stark contrast to inscrutable deep networks. The library that ASPIRE builds could even serve as a kind of robotic knowledge base: common actions (pick, place, insert, attach, etc.) are encoded and can be called on throughout the robot’s lifetime. It echoes the AI field’s notion of foundation models: while ASPIRE’s library is smaller scope, it is similarly a foundational block of capabilities on top of which future tasks are built. Third, ASPIRE inherently supports generalization. We already noted sim-to-real and cross-embodiment, but there’s more: the skill library can also generalize to new tasks. If a program says “push the red block,” we might modify it to “push the blue block” by just changing a target input. The continuous evolution loop could automatically adjust skill programs in situ if slightly different situations arise, without needing to rewrite an entire policy. The result is a level of adaptability beyond static learned policies. This paves the way for truly open-ended learning: the robot isn’t confined to a fixed task distribution but can expand into new domains. Finally, ASPIRE aligns with the broader trend of agentic development in robotics. Projects like ENPIRE at NVIDIA form closed feedback loops for policy improvement driven by coding agents. Eureka uses generative models to automate reward engineering. The “Agent Skills” released by NVidia as open source show how software agents can now orchestrate physical tasks. ASPIRE fits into this ecosystem as the next step: a system that not only improves policies but actually invent the policies themselves. Together, these approaches point toward a future where much of robotics R&D is carried out by autonomous agents, with humans supervising high-level goals instead of tweaking every algorithmic knob. Conclusion. ASPIRE: Automated Skill Discovery represents a bold stride toward robots that teach themselves new skills in a continual, automated fashion. By combining evolutionary search with modern simulation and robotic platforms, the system can discover a compendium of useful behaviors with minimal human guidance. These behaviors – articulated as sensorimotor programs – can be sequenced to solve complex tasks, transferred from simulation to real robots seamlessly, and applied across different embodiments. In doing so, ASPIRE demonstrates the feasibility of machines autonomously expanding their own skill libraries, an essential ingredient for general-purpose robotics. The road ahead is rich with possibilities. A natural next step is integrating ASPIRE’s library with higher-level planners or language models, so that users can simply describe goals and have the system chain together the appropriate programs. Another is scaling up the open-endedness: providing more diverse environments or allowing the robots to modify their own tools and sensors, thus creating even more novel tasks to conquer. As in quality-diversity research, one can envision truly open-ended skill evolution where each success seeds the next challenge. For now, ASPIRE has taken the research community’s best ideas – evolution, simulation, modularity – and combined them into a working system that outpaces prior skill-learning efforts. It shows that robots can independently invent a broad repertoire of actions, greatly reducing human engineering effort. As robots pervade factories, homes, and wild environments, such autonomy in learning is not just convenient, it is likely essential. In that sense, ASPIRE is not just a system, but a glimpse of how robotic skill acquisition is evolving toward the flexibility and independence we see in biological intelligence. References: The concepts discussed here build on many recent advances. Unsupervised and hierarchical skill learning, quality-diversity search, LLM-driven robotics, and cross-embodiment policies are all active areas of research, and ASPIRE unifies aspects of them in a novel evolutionary framework. We also note the parallels with systems like ENPIRE and Eureka, which similarly aim to automate parts of the robot learning loop. By synthesizing these ideas, ASPIRE lays groundwork for robots that continually self-improve and self-expand their capabilities.