Latest / Embodied AI 101 / Playful Agentic Robot Learning
Transcript
- Playful Agentic Robot Learning. Imagine a robot that doesn’t wait for a human to give it instructions, but instead plays on its own to figure out new skills, and later applies those skills to solve real tasks. This is the vision of playful agentic robot learning. Rather than being purely reactive, the robot becomes an autonomous, lifelong learner: it explores its environment, proposes its own tasks, writes code to try them, and learns reusable routines from the experience. In a recent study, Zhang et al. introduce RATs (Robotics Agent Teams), a system that implements this idea by combining self-directed play with the emerging paradigm of Code-as-Policy robot control. Their agent uses large language models to write and execute robot programs, but uniquely, it first goes through a “play” phase to build up a library of code skills that improve its later performance on benchmark manipulation tasks. At a high level, the key contributions of playtime learning are:. Self-directed skill acquisition: Instead of only learning when explicitly told what to do, the robot autonomously generates novel/objective tasks to practice, focusing on tasks that stretch its abilities but are still doable. Code-as-Policy execution with feedback: When trying a task, the agent writes executable code (using an LLM) that interacts with the robot. It then carefully verifies each step of execution, learns from failures, and retries with better feedback. Skill library creation: Successful routines from these play sessions are turned into named code “skills” and saved in a library. Later, when solving actual tasks, the agent can simply reuse these learned code modules without retraining the model itself. This approach resembles how children learn by playing with toys: by experimenting in many ways, they acquire skills that are reused later in goal-directed tasks. RATs formalizes this idea for embodied LLM agents. In experiments on the LIBERO-PRO and MolmoSpaces robot benchmarks, the play-trained agent outperformed a non-playing baseline by 20+ percentage points on task success. Moreover, plugging these learned skills into other code-based agents (even with no additional model tuning) boosted their success in new simulation tasks and on a real robot by nearly 9 points. In what follows, we unpack how the system works, what the experiments show, and why this “playful” strategy may open new avenues for robot learning. Background: Code-as-Policy and Autonomous Agents. Recent advances in large language models (LLMs) have enabled a new style of robot control: the agent takes a task description (often in language) and generates code that drives the robot. This Code-as-Policy paradigm was demonstrated by Liang et al. (2022), showing that LLMs trained on code can be repurposed to write fully executable robot programs. For example, an LLM given the command “pick up the red block and place it in the basket” can output a Python-like script that calls perception and motion primitives to accomplish the task. By chaining language understanding with code generation, the robot effectively interprets high-level instructions into concrete control routines. In practice, this is done by few-shot prompting: the model is shown some example language commands paired with working code snippets, and then asked to produce code for new commands. The result is an “agentic” robot system that continually generates and executes its own policies in code form, even revising them on the fly if something goes wrong. However, prior work has mostly focused on task-driven scenarios: the LLM-agent only generates programs when given a specific task, and the skill repertoire grows only insofar as new tasks are instructed. In other words, there is no separate learning phase for skills — the agent does not autonomously develop new abilities unless explicitly prompted. The novel idea in RATs is to insert a self-play phase before any downstream tasks. In this stage, the embodied agent (with perception of its environment) deliberately proposes and practices interesting tasks, without any external instruction. This “playful” learning phase is used to acquire a bank of reusable skills in code form. Later, when actual tasks arrive, the agent can draw on this bank to assemble solutions quickly and robustly. To draw an analogy: imagine giving a robot arm a bin of blocks and telling it “just play”. If the robot were similar to a child, it might try stacking blocks, knocking them over, sorting them by color, etc. In doing so, it learns physics and motor skills that will be useful later. RATs implements a similar process programmatically. It structures the agent into multiple cooperating components (a Robotics Agent Team) that propose tasks, plan their execution, run the code, verify progress, and record successful strategies. The result is a persistent code skill library that captures reusable routines. Importantly, this can all happen before any formal “test tasks” are specified, so the agent is truly learning on its own. Before diving into the methodology, it is worth noting why code-based agents are a good fit for this. Because the agent’s “skills” are simply pieces of code, it is natural to store them and later retrieve them. When a new task arrives, the agent can compose previously learned code modules. This contrasts with, say, a deep RL policy that has to be fine-tuned or re-trained when the objective changes. In RATs, after the play phase, the underlying LLM remains frozen; no gradient updates are needed. New tasks are solved by retrieving relevant code snippets into the context of the LLM prompts. The project shows that this leads to significant performance boosts “for free” at inference time. In summary, RATs brings together the ideas of self-play (learning through exploration) and Code-as-Policy (LLM-written control code). The next sections describe exactly how the system is organized and what it learns during play. The RATs Methodology: Learning Skills Through Play. The core of the approach is a Robotics Agent Team (RATs) – a collection of specialized agents working together in a loop of proposal, planning, execution, verification, and memory update. This loop runs during a play phase where no external tasks are given. Here is the high-level workflow in each iteration of play:. Task Proposal (Find the competence frontier): A Task Proposer agent looks at the current state of the environment and the skills already in the library. It then generates a novel task goal that is not yet well-practiced but should be achievable. Crucially, it biases toward tasks on the edge of the robot’s current abilities – not too easy, not impossible. In their implementation, the proposer typically selects object-skill combinations that have been rarely attempted but not clearly unreachable. By exploring at this “competence frontier,” the agent efficiently broadens its skill set rather than repeating trivial tasks. For example, if the robot has frequently picked up small blocks but never tried picking up a heavier object, the proposer might suggest a task involving that heavier object, thereby challenging the robot just beyond its comfort zone. Planning: Once a task is proposed, a Planner agent takes over. The planner examines the proposed goal and decomposes it into a sequence of steps (a goal hierarchy). It also retrieves any relevant skills from the existing library that could help with those steps. In effect, the Planner “grounds” the task by mapping it onto known subroutines. For instance, if the proposed task is “stack blue block on top of red block”, the planner might break this down into steps like “reach for blue block”, “grasp blue block”, “move to above red block”, “place blue block”. It would then look in the skill library for any existing code modules like or that can be reused. This targeted retrieval means the agent does not have to reinvent the wheel each time; it composes new behavior out of previously learned pieces. The planner then outputs a structured plan (often in language or code form) specifying these ordered steps. Execution (Policy Writing): With a plan in hand, the Policy Writer agent now generates the actual executable code. It uses a large language model to turn the planned steps into robot-control code (for example, calling Motion and Perception APIs). This code is then executed in the simulator or real world. If a low-level subtask proves persistently difficult (for instance, the robot consistently fails to open a drawer), the system can spin up a SubAgent to focus on just that bottleneck by itself, essentially practicing that subskill in isolation. For example, the policy writer might generate a sequence of Python commands like:. The multi-agent structure means that while one agent (the Policy Writer) is converting text into code, another agent might recognize repeated failures and initiate targeted retries on specific subtasks (the SubAgent). This division of labor keeps the process efficient. Verification (Failure Diagnosis): After execution, the system immediately checks what happened. Crucially, RATs uses "step-level verifiers" that monitor both final success and intermediate progress. A Goal Verifier determines if the ultimate task goal was achieved (e.g. “Is the blue block on the red block?”). Concurrently, Per-step Verifiers inspect each planned step against sensor data (e.g. “Did the gripper reach the blue block’s position?”, “Did the gripper successfully close?”). This fine-grained checking localizes exactly where the attempt failed. If a sub-step failed (say the robot knocked the block instead of grasping it), the system generates detailed feedback. A Diagnoser agent takes this feedback and decides how to respond: it might simply retry the full plan with hints, or it might isolate the failed action and retry that sub-action specifically. For example, if the gripper keeps missing the block, the diagnoser might schedule additional “practice trials” focusing solely on aligning with the block. By converting failures into actionable insights, the agent learns much more efficiently than a black-box trial-and-error loop. Memory Update (Skill Extraction): After a run (and possibly multiple retries), any successful routines are distilled into the skill library. Specifically, when a plan finally succeeds – i.e., all verification checks pass – the exact code that achieved the task, together with relevant contextual hints, is saved as a named skill module. This module can later be called by the planner for new tasks. Even failures are useful: the system compacts the failed attempt and its feedback into a lesson that influences future proposals or retries. In simple terms, “successful behavior becomes reusable code; failures become compact lessons that shape future practice”. For example, the code that successfully sensed and picked up the blue block would be saved (perhaps as ), while a failed grasp attempt might generate an annotation like “do not approach from this angle” for next time. The key is that over many play iterations, the robot’s memory grows into a library of code skills. These skills are frozen (they won’t change unless explicitly relearned) and serve as building blocks. Each play iteration thus adds more to the repertoire. The Task Proposer then repeats the cycle: it proposes a new unpracticed yet achievable task, considering the enriched library. Over time, the distribution of tasks explored becomes broader (the agent systematically covers different object interactions rather than repeating easy ones), and the skill library grows richer. Figure 1 in the project (not shown here) illustrates this pipeline: from proposal to planning to execution to verification to memory update, and loops continuously. In this way, RATs systematically turns autonomous play into a performant skill library. It’s worth emphasizing that all of this happens before any downstream task instructions arrive. The robot is effectively “learning before it is asked”. Only after the play phase do we switch to the actual evaluation phase, where the agent is given novel tasks it has not seen. At that point, it no longer proposes tasks or updates its library – it simply uses what it has learned. The experiments show that having this library makes a big difference (see next section). One way to think about the RATs design is that it combines intrinsic motivation with powerful reasoning. The Task Proposer is like an intrinsic motivator, actively seeking new challenges. The LLM-based Code Writer is like a general problem solver, capable of generating any code given the right prompt. By looping with verification feedback, the system approximates an intelligent trial-and-error learner that improves itself. Importantly, it leverages off-the-shelf LLMs (such as the Qwen-2.5-Coder model used in their experiments) for code generation and reasoning, chaining them as needed without modifying them. The novel twist is simply giving these agentic capabilities a playground to roam freely and accumulate knowledge. Experimental Results. To evaluate the impact of this self-play skill learning, the authors ran experiments on two challenging robot learning benchmarks, and also a real robot setup:. LIBERO-PRO: This is an advanced extension of the LIBERO benchmark for “Vision-Language-Action” (VLA) models from Berkeley/Physical Intelligence**. It includes a variety of tabletop manipulation tasks with perturbations to test generalization. Tasks might involve interacting with objects in a scene according to language goals. The “PRO” variant emphasizes generalization by introducing new object categories and arrangements (as mentioned in its GitHub description). MolmoSpaces: A freshly-introduced massive robot environment suite from AllenAI (published 2026). MolmoSpaces provides over 230,000 procedurally-generated indoor scenes, containing 130k+ objects (48k of them manipulable, with a huge database of 42 million stable grasps). Importantly, this platform is simulator-agnostic (supporting MuJoCo, Isaac, ManiSkill, etc.) and covers tasks ranging from pick-and-place to navigation and multi-room chores. The authors focus on the MolmoSpaces-Bench, a curated set of 8 diverse manipulation tasks defined over these environments. (Examples might include things like moving a cup, opening a drawer, wiping a table, etc., each with randomized scene layouts.). RoboSuite: A smaller suite of robotic tasks typically in MuJoCo simulator (commonly used in learning-to-grasp/swipe tasks). This was used for cross-environment generalization tests. Real Robot: To test sim-to-real transfer, some experiments were run on an actual robot arm performing tasks like resetting blocks, opening a cabinet, etc. The experimental protocol was as follows: First, each agent (RATs or baselines) underwent the play phase for a fixed number of iterations in simulation (in LIBERO-PRO or MolmoSpaces). During play, RATs built up its library of skills. For comparison, the authors included two key baselines:. A No-Play baseline: This agent had no autonomous play phase. When tasked with downstream tasks, it simply attempted them from scratch (writing code with an LLM), but it had no extra skills except any that might come from the LLM’s prompting context. A Random-Play baseline: Like RATs, this agent also played before the evaluation phase, but its “Task Proposer” was a random sampler. It chose random object-skill tasks instead of intelligently targeting learnable yet unused tasks. Both baselines still used the same code generation and verification machinery when actually solving tasks. The key difference was whether and how they spent time “playing”. After the play stage, all agents were reset (the LLM weights are fixed) and then evaluated on a set of held-out downstream tasks. These tasks were chosen to be different but related to the play goals, so success depends on having useful skills. The results were striking. On in-domain downstream tasks (tasks defined within the same environment as training, i.e. LIBERO-PRO or MolmoSpaces), the RATs agent achieved much higher success rates than the baselines. Specifically, on LIBERO-PRO, adding the play-learned skills gave a +20.6 percentage-point improvement in task success over the baseline code-as-policy agent with no play. On MolmoSpaces, the gain was +17.0 points. These gains are over the “CaP-Agent0” baseline (i.e. code-as-policy agent with no play) and are also substantially higher than the random-play variant. In plain terms, if the baseline agent solved, say, 40% of tasks, the RATs-enhanced agent would solve around 60%. This demonstrates that the skills learned during play directly translate to better performance on new tasks in the same domain. The evaluation also highlights how effective the Task Proposer’s strategy is: because tasks were chosen strategically at the frontier, the learned skills covered diverse scenarios. Figures in the paper (and the results page) show that RATs quickly expands the variety of practiced goals (for example, trying different objects or novel placements) rather than getting stuck on repetitive easy tasks. The learned library likewise grows steadily with each successful play session. By contrast, the random-play agent tends to accumulate less useful or redundant skills, so it underperforms RATs. Another key result is cross-domain generalization. The authors tested whether the skills learned via play in one set of environments could help in different settings. They did this by taking the code skill library acquired in LIBERO-PRO or MolmoSpaces and using it with a standard code-as-policy agent to solve tasks in RoboSuite (a different simulator) and even on the physical robot. Remarkably, even with no further training or fine-tuning of the LLM, simply inserting the retrieved skill code into the prompt gave a significant boost: roughly +8.9 points on average in RoboSuite tasks, and +8.8 points in the real-world robot tasks. In other words, the play-learned skills truly transfer. For example, a grasping or placing function learned on block-stacking could help a different robot pick up objects in a new context. The gains are smaller than the in-domain case (as expected), but still large enough to matter. Qualitatively, videos provided by the authors show the robot performing real-world tasks (closing a drawer, transferring a block to a box, wiping a plate with a sponge, etc.) by invoking the learned primitives. In these demos, the robot uses exactly the same code skills it learned in simulation. The real-world success rate is impressive even though there was no task-specific fine-tuning of the LLM (just the usual sim-to-real adjustments for robot control). This suggests the skills captured fundamental manipulations (like , , ) that are robust across setups. Overall, the experiments confirm the hypothesis: self-play yields better skills, which in turn boost downstream task success. The numerical results are significant (20%+ in-domain gains, 8-9% transfer gains) and were achieved on challenging, multimodal tasks. The project page summarizes this concisely: “Play pays off downstream – skills improve downstream performance” and “RATs skills plug into different simulation and real-world settings without finetuning”. Ablation studies on the project site further show that each component matters. For instance, using the verification feedback markedly reduces repeated failures in code generation (the system identifies where the code went wrong and retries more intelligently). The analysis also indicates that different task categories tend to reuse different learned skills (e.g. “opening” tasks rely on certain perception and motion routines, while “placing” tasks call others). This compositional behavior is exactly what one would hope: the agent is not just memorizing full tasks, but assembling them from modular pieces. In summary, the empirical work demonstrates that purposeful play with code leads to concrete, measurable improvements in robot capabilities. Discussion and Implications. The RATs framework introduces several notable ideas for the future of robot learning:. Lifelong, open-ended learning: By decoupling skill acquisition from task solving, RATs embodies a form of lifelong learning. The robot can endlessly play and expand its repertoire without needing retraining for each new task. When confronted with something novel, it can fall back on its accumulated skills. This is reminiscent of how humans improve – we don't just learn on the fly when asked; we wander, play, and gradually build up knowledge that pays off later. Combining symbolic reasoning and embodiment: The system treats skills as code, which is a more symbolic, modular representation than, say, a neural network weight. This means the knowledge is explicit and composes well. Each skill has a name and code, so it can be debugged, upgraded, or even manually edited if needed. Yet the generation of these skills is driven by embodied interaction – the robot’s own sensors and actuators provide real feedback to refine the code. This hybrid approach marries classical planning (with discrete steps and checks) to modern LLM flexibility. Practical autonomy with LLMs: One might ask: how is the robot prompted to “play” at all? In practice, the agent may start with a set of primitive skill and object vocabulary, and then just repeatedly run its loop. What drives it is an LLM prompt that allows it to imagine tasks from the current goal. The results show that off-the-shelf LLMs (Qwen in this case) are capable enough to not only solve assigned tasks but also to autonomously dream up solvable challenges. This reduces the need for human-designed curricula or demonstration data. Zero-shot skill reuse: Perhaps most surprisingly, the skills learned require no re-learning. The authors emphasize that other code-as-policy agents can simply include the learned code snippets in their prompt to improve. This is virtually a plug-and-play mechanism for skill transfer. In practical terms, a developer could run RATs-on-play offline to build a skill library, then ship that with the LLM agent for various applications. No model retraining means the heavy lift is done once, and the base model’s parameters don’t grow suddenly with new tasks. Generalization: The strong performance on LIBERO-PRO and MolmoSpaces indicates that the skills were not just memorizations of those specific scenarios, but captured underlying capabilities (grasping, pushing, opening, etc.). The transfer gains to RoboSuite and real-world tasks (almost 9 points each) show that even across different domains and physics, the knowledge holds value. This hints that future robots could go through extensive simulated play in vast synthetic worlds (like MolmoSpaces) and emerge with general-purpose utilities applicable in reality. Efficiency: RATs is sample-efficient in a sense. It does not rely on large amounts of trial-and-error training, millions of gradient steps, or reinforcement learning. Instead, most of the “learning” is done by cleverly using the LLM’s reasoning at inference time. The agent may try hundreds of attempted play tasks, but each one is a complete planned execution, not just a random rollout. The guiding hand of the Task Proposer ensures that wasted effort is minimized by targeting novel objectives. A few caveats and future questions arise:. Scalability and safety: As the complexity of the environment grows, the space of possible play tasks also explodes. The competence frontier approach helps, but designing good task proposals might become harder. There is also the question of safe play: in the real world, an autonomous robot playing unsupervised could knock things over or break objects. The framework would need constraints or simulated play spaces to avoid damage. Quality of LLMs: The approach depends on the LLM’s ability to write correct code. While we saw that verification and retries catch many errors, fundamentally the LLM may hit its limits on very intricate manipulations. Future LLMs or multi-modal models that can directly incorporate vision (so it can see the scene) might enhance performance. Knowledge consolidation: Currently each skill is a code snippet. Over time the library could grow very large or have redundancies. Meta-learning strategies might further compress or generalize these skills. The paper’s example videos hint at hand-coded skill names like “localization” or “grasping helpers”; a fully automatic system might need its own organizing principle for the library. Application domains: The paper focuses on manipulation in indoor scenes. The same principle could, in theory, apply to other robotics domains: locomotion, aerial flight, even game-playing robots. Anywhere a simulator exists, a RATs-like system could play around to gain edge-case intuitions. In sum, “Playful Agentic Robot Learning” suggests a compelling new paradigm: give robots the freedom to self-teach, and use powerful language models to bootstrap that teaching into reusable code. It’s a step toward more autonomous, curious robots that build their own knowledge base. As LLMs and simulation platforms continue to improve, such closed-loop play systems could become a standard part of a robot’s learning toolkit. Conclusion. Zhang et al.’s work on RATs: Playful Agentic Robot Learning introduces a fresh direction for embodied AI. By allowing a robot agent to initiate its own exploration and skill acquisition, they show that an LLM-powered robot can significantly expand its capabilities in a self-supervised way. The key ideas — proposing learnable tasks, writing and verifying code, and saving successful routines — form a cycle that effectively converts play into proficiency. The experimental gains (20+ points on benchmarks, nearly 9 points in transfer) are large enough to demonstrate that this is more than a toy idea; it’s a practical technique for improving real robot performance without additional training of the model itself. For practitioners, this means that future robotic systems might employ an initial “play session” to bootstrap themselves. For researchers, it opens questions about how best to structure autonomous exploration with language models, and how far such methods can scale. As a narrative, it’s also a satisfying one: robots that learn by playing, much like living creatures, and who then recall those old play routines when a real job comes along. The RATs approach reminds us that not all learning needs to be reactive — some of the most valuable knowledge can come from merely “playing around” in the world. Sources: The description above is based on the abstract, project page, and related references of Playful Agentic Robot Learning (RATs) by Zhang et al. (Berkeley, Jun 2026), which introduces the RATs architecture and reports its performance improvements in LIBERO-PRO and MolmoSpaces tasks. These citations provide the technical details and empirical results referenced above.