Latest / Embodied AI 101 / RL-100: Toward Highly Reliable Real-World Robot Reinforcement Learning

Transcript

RL-100: Toward Highly Reliable Real-World Robot Reinforcement Learning · Embodied AI 101

• RL-100: Toward Highly Reliable Real-World Robot Reinforcement Learning. Overview. In the quest to build robotic systems that operate reliably and autonomously in messy real-world environments, the RL-100 project sets a new benchmark. Introduced by Kun Lei et al., RL-100 is a complete training framework for visuomotor robotic manipulation that combines human demonstration, offline reinforcement learning (RL), and online fine-tuning. On a suite of seven challenging real-world tasks, the authors report a jaw-dropping 900 successes out of 900 trials (100% success) – including up to 250 consecutive successes on a single task – all with a variety of manipulations that include pushing, bowling, pouring liquids and granular media, folding cloth, dexterous unscrewing, and a multi-stage orange-juicing routine. They also show that RL-100’s policies match or exceed human teleoperation in speed, gracefully handle perturbations and novel conditions, and can even continuously operate for hours (e.g. a public juicing demo ran seven hours without a single failure). In short, RL-100 demonstrates that carefully structured RL pipelines can reach deployment-grade reliability on real robots. In this article we delve into how RL-100 works, what it achieves, and why it matters. We examine the three-stage training pipeline – imitation learning, iterative offline RL, and online RL – built on top of diffusion-based visuomotor policies. We unpack the key ingredients like offline policy evaluation, variance clipping, and consistency-model distillation that make this system both effective and safe. Then we survey the experimental results: the perfect success rates, the efficiency comparisons to human operators, the zero-shot and few-shot generalization tests, and the extreme disturbance trials. Finally, we discuss the broader significance: what RL-100 tells us about scaling up learning on real robots, the “imitation ceiling” problem it overcomes, and the path forward toward truly deployable RL systems. From Demonstrations to Mastery: The RL-100 Training Pipeline. RL-100 is built around the idea that real robot learning must proceed in stages, using human intuition to build a solid foundation and then gradually improving autonomously. The authors design a three-stage pipeline, each with its own training regimen, but all working on the same learned policy representation. Roughly speaking, the policy starts out as an imitation of human demonstrations, then gets iteratively improved via offline RL using the data collected so far, and finally gets refined by actual online RL on the real robot to eliminate any remaining rare failures. At each stage, the policy is expressed as a diffusion-based visuomotor model – a powerful conditional model that can generate complex action sequences from visual inputs. Let’s break down each stage:. 1) Imitation Learning (IL) Pretraining: First, humans teleoperate the robot to collect demonstrations for each task. These trajectories are used to train a diffusion policy network via behavior cloning. In practice, the policy maps visual observations (RGB images or 3D point clouds) plus robot proprioception to actions. The key modeling choice is that actions are generated by a conditional diffusion model. Concretely, the diffusion model starts from random noise and iteratively denoises it into the desired action. By training this model to match the demonstrated actions, the policy learns a robust initial strategy that follows human priors. This gives a good starting point: the policy often already works reasonably well out of the box. In fact, on one task the raw IL policy achieves about 70.6% success, showing humans gave it a solid base. 2) Iterative Offline Reinforcement Learning: Once the IL policy exists, the next step is to improve it without risking catastrophic failures. RL-100 does this by running iterative offline RL on the robot. The trick is that they don’t simply do one round of behavior cloning or fine-tuning. Instead, they collect rollouts from the current policy (in simulation or on the real robot), add them to the dataset, and then run an offline RL update on the whole growing dataset. The core objective is a form of PPO/cutting-edge policy gradient applied across the diffusion denoising steps, ensuring we can assign credit properly over the multi-step action generation. Crucially, each new policy update is protected by an Offline Policy Evaluation (OPE) check, which only accepts policy changes that demonstrably improve estimated returns. In practice they use an Implicit Q-Learning (IQL) value function and enforce that updates are conservative. This yields monotonic policy improvement: as offline RL proceeds, success rates steadily climb (e.g. from 70.6% after IL to 91.1% after offline RL) without introducing wild new failure modes. The visual encoder is held fixed during offline RL to keep stability. This offline loop can repeat, expanding the dataset with newly successful rollouts and progressively raising performance. 3) Online Finetuning with On-Policy RL: After offline batches exhaust the easy gains, any stubborn failure modes are dealt with by going fully on-policy on the real robot. In this final stage, the robot runs episodes with the current policy, and gradients (again PPO-style) are computed on actual returns. They use Generalized Advantage Estimation (GAE) for more efficient learning. Exploration is carefully regulated by clamping the noise variance in the diffusion sampler (via variance clipping in DDIM steps) so that the robot doesn’t suddenly do crazy exploration that might knock things over. This stage typically requires only a few hours of training to polish the policy. Importantly, the entire training framework uses the same underlying policy architecture – a diffusion model – throughout all phases, only switching the training algorithm (IL, offline, online) and data used. On top of these three stages, RL-100 introduces a lightweight consistency model distillation step. A diffusion policy normally needs tens of denoising iterations to sample each action, which could be too slow for tight control loops. So the authors train a separate “consistency model” network that maps the same inputs to a final action in one shot, by distilling the multi-step diffusion process. In practice, after training the diffusion policy, they generate actions with full diffusion and train the consistency model (using mean-squared error) to imitate those one-step. The result is a single-step policy that matches the multi-step policy’s performance but can run an order of magnitude faster. In fact, they demonstrate control rates up to ~378Hz in simulation when using the distilled model. This step is crucial for deployment: it removes inference lag and allows the robot to react rapidly. Overall, this design is deliberately representation-agnostic and hardware-agnostic. The same pipeline can take either 2D RGB images or 3D point clouds as input; it can control different robot arms (UR5, Panda, custom setups, etc.); and it can output either single actions or short action “chunks” depending on task needs. The policy’s visual encoder is self-supervised and regularized to remain stable through all phases. In short, RL-100 is a general RL training framework that sits on top of any particular robot or perception stack. Diffusion Policies and the Inner MDP. A distinctive technical pillar of RL-100 is its use of diffusion models as visuomotor policies. A diffusion policy represents an action (or a short sequence of actions) as the endpoint of a gradual denoising process. Concretely, the model is trained (via standard denoising score-matching objectives) to reverse a forward diffusion that adds noise to true actions. At test time, generated actions result from starting with Gaussian noise and iteratively denoising. This framework was pioneered in “Diffusion Policy” work (Chi et al., 2023), which showed that diffusion models can represent rich, multimodal action distributions conditioned on vision. In RL-100, this diffusion policy is not only used for IL, but is fully integrated into the RL training. The key insight is to treat the denoising process itself as a short inner-MDP. Each sampling step is like taking a “micro-action” that gradually refines the eventual robot command. The outer environment step is the usual robot step, while the inner diffusion chain is a multi-step process whose trajectory ends in the final action. During training, the PPO-style policy gradient objective is applied across the entire denoising chain. As Kun Lei et al. describe it: “The denoising process is embedded as a sub-MDP within each environment step, and policy gradients are computed over the chain of denoising steps”. In practice this means using the same advantage estimate (from the environment reward) for each of the K diffusion steps. This yields a dense, step-wise learning signal and ensures that each intermediate refinement stage is optimized to support good final outcomes. Embedding diffusion in the MDP also allows the offline and online RL stages to be unified under a single objective. In RL-100, they essentially optimize one PPO-style objective continuously from offline into online training. During the offline phase, they use advantage estimates from the IQL critic and accept only updates that the OPE checker approves. During online RL, they switch to on-policy trajectories and GAE. Because the outer advantage is shared, the transition from batch updates to on-policy updates is seamless. The authors highlight that a single, unified gradient objective is applied across all K denoising steps. This design avoids the need to change the loss function midstream, and contributes to very stable learning. Crucially, after all training is done, the authors distill this iterative process into a one-step policy. The “consistency head” or model is trained so that its one-shot prediction matches the outcome of fully-iterated diffusion. This is known as a consistency model in diffusion literature. With this distillation, RL-100 attains fast closed-loop control: actions can be commanded at like 100+ Hz with negligible loss of quality. In fact, [22] reports inference at up to 378 Hz in simulation using the consistency model, whereas a raw 20-step diffusion rollout would dramatically slow the loop. In essence, RL-100 gets the best of both worlds: the expressive power of diffusion policies during learning, and the speed of a normal policy at runtime. Pushing the Envelope: Real-Robot Tasks and Full Success. To validate their approach, the authors assembled a diverse suite of seven real-world manipulation tasks. These include dynamic, contact-rich tasks as well as fluid and deformable scenarios. The tasks are:. Dynamic Push-T: Pushing a T-shaped object, requiring precise dynamic control. Agile Bowling: Knocking down bowling pins through skillful pushes. Pouring: Both granular (grains) and liquid (water) pouring experiments. Dynamic Unscrewing: Precisely unscrewing a cap using one dexterous hand. Dual-Arm Soft-Towel Folding: Folding a soft towel using two robot arms, which is a classic deformable-object challenge. Multi-Stage Orange Juicing: A two-step task involving picking/orienting and pressing oranges to juice. This task set spans rigid-body dynamics, deformable materials, and fluids. Each task has randomized initial conditions and subtle variations so that the policies must generalize beyond a fixed setup. The headline result is that RL-100 achieves 100% success across all evaluation episodes for each of these tasks. Across 900 total episodes (including different tasks and many seeds) RL-100 never fails. On the hardest task, dual-arm folding, it delivered 250 successes in 250 trials without a single failure. These are truly exceptional numbers. By comparison, the initial IL policy typically had gaps (recall the 70.6% on folding), and even standard imitation or off-policy RL methods usually leave some failures on the table. The iterative stages in RL-100 very reliably closed out these failures. The performance is not just reliable – it is also efficient. In timing tests, RL-100 finished tasks as fast or faster than expert human teleoperation. In one case, on Dynamic Push-T, the robot’s throughput was about 1.18× that of a skilled human on the same number of trials. The authors summarize: the learned policies “match or surpass expert teleoperators in time-to-completion”. In general, episodes ran with shorter wall-clock times than humans. This strong efficiency is partly thanks to good control (the robot doesn’t fumble or repeat steps) and partly due to the fast consistency-model inference. Where imitation or baselines might play conservatively, RL-100’s policy often executes briskly and confidently. In addition to raw success rate and speed, RL-100 demonstrates long-term robustness. The team ran extended continuous trials on the policy. Remarkably, one demo had the juicing robot serve random customers in a busy shopping mall for seven straight hours without a failure! Even within lab tests, policies ran continuously for up to two hours with no breakdowns. This multi-hour reliability suggests that RL-100 learned to handle real-world variance in sensors and actuators. It also contrasts sharply with typical RL experiments, where policies might work for a few minutes or fail under slight shifts. RL-100’s continuous operation is concrete evidence that its policy network is robust and its control signals stable. Adapting to Change: Zero-Shot and Few-Shot Generalization. Real-world deployment often requires adaptation to changes that were not in the training distribution. The authors stress-tested RL-100’s policies under a variety of out-of-distribution scenarios. The results are encouraging: strong zero-shot generalization and speedy few-shot adaptation. In zero-shot tests, they altered dynamics or environment factors without any additional learning. For example, on Dynamic Push-T they changed the friction of the floor; on Agile Bowling they rearranged the pins; on Pouring they switched from granular nuts to liquid water; and on Folding they presented a differently shaped towel. The learned policy handled these changes with high success rates: roughly 92.5% average success across all such zero-shot variations. (For instance, Bowling with an inverted pin layout was still 100% successful, and pouring granular→liquid was 90% success.) There were some failures, like an 80% success when adding distracting objects in pushing, but these illustrate the difficulty of extreme shifts. Notably, all this generalization happened with no retraining – the one trained policy could cope with quite different visuals and physics. When zero-shot performance did slip, a brief round of fine-tuning fixed it. In few-shot adaptation, they allowed up to 1–3 hours of additional training on the new variation. After that short training, success rates jumped. For example, changing the towel material in folding went to 100% with little effort, and adjusting the container geometry in pouring went from initial 60% up to 86.7% (on average) after a few hours of RL. Even an inverted bowling pin arrangement reached 100% after minor retraining. These results show that RL-100’s policies have learned a structured visuo-motor representation that can be tuned quickly. The high zero-shot accuracy suggests broad robustness in the learned policy. Then the data-driven nature of the approach allows rapid few-shot correction. For robotics applications, this kind of generalization is vital: one robot should ideally adapt to new kitchens, different bowls, or bumped objects without requiring days of re-training. Handling Disturbances: Human and Environmental Perturbations. Robotic systems in the wild must also handle unplanned physical perturbations. RL-100 policies proved impressively resilient to direct disturbances. The authors tested by applying strong forces or drags while the robot acted. For example, during an unscrewing task, they forced the screw counter-clockwise for up to 4 seconds – and the robot never lost the task. In a folding task, they pulled on the towel mid-step; the policy recovered 90% of the time on both early and late-stage pulls. Across all such tested disturbance types, RL-100 averaged 95% success under perturbation. This performance under stress is likely thanks to two factors. First, the iterative offline RL training probably exposed the policy to slightly varied rollouts, implicitly boosting robustness (especially if exploration noise was large enough to produce fringe conditions). Second, diffusion-based policies have inherent stochasticity (sampling noise) and the consistency distillation preserves that sense of “softness” in decision-making. When shoved, the policy uses continued observation feedback to replan. Importantly, the high success under forced-backwards movements or drag implies the policy has learned recovery strategies rather than relying on a fixed sequence. That is, it is not brittle: it knows how to regrasp or reorient when something unexpected happens. Embedding Figures. [What follows are not fully formed due to limitations, but the idea is to embed three images from unsplash of robot manipulation tasks. I cannot fetch them directly via the web interface to embed or cite them properly here, so I will skip or indicate placeholder. Normally:]. Inline image example: “figure: Robot arm performing manipulation in industrial setting,” as an illustrative image of a robot manipulator. Inline image example: etc. (embedding such images from unsplash or commons to illustrate tasks.). …. No actual images are embedded here; focus on text content instead. Comparing to Baselines and Ablations. RL-100’s achievements stand out when compared to more conventional baselines. The team implemented state-of-the-art imitation learning “diffusion policy” baselines for all tasks. One variant (“DP3”) used their full 3D diffusion model, and another (“DP-2D”) used only 2D inputs. In repeated tests, these IL-only policies often failed – i.e., needed dozens of trials to get 70–90% in reliability depending on task. In contrast, the classic RL-100 pipeline fixed those failures. Quantitatively, the IL baselines averaged ~70‡% success on folding, whereas RL-100 reached 100%. Moreover, RL-100’s execution was generally quicker and smoother: in execution efficiency tests, RL-100 with the consistency model was faster than the IL baselines, and markedly faster than doing step-by-step open-loop actions (the consistency model outpaced DP3 and DP-2D on all tasks). This shows the benefit of the multi-stage training and careful design elements (like variance clipping, use of 3D vision, etc.) beyond just using a fancy policy architecture. They also tested various ablations on tasks or in simulation. For example, in MuJoCo locomotion and Adroit dexterous benchmarks, RL-100’s same training flow outperformed recent diffusion/flow-based RL algorithms (such as Diffusion-PPO and ReinFlow) on both final scores and convergence speed. Internal ablations on their system indicated that each design choice helped:. Using 3D point clouds rather than just 2D images led to faster learning and higher final success on tasks with cluttered scenes. Clipping the noise variance in the diffusion sampler prevented runaway exploration or premature convergence, which stabilized learning. The consistency model distillation didn’t hurt performance but gave speed; without it the raw diffusion policy could not run at high control rates. Training the diffusion as an epsilon-noise predictor (as opposed to predicting the clean action $x_0$) was found to encourage larger exploration noise and turned out to be better for RL (an “epsilon prediction” parameterization). In short, nothing in RL-100 was easily removed. Putting imitation, offline RL, online RL, and distillation together – along with vision and stability tricks – was necessary to hit that 900/900 mark. It’s a testament to how fragile robot tasks can be and how careful one must be to ramp up RL to full reliability. Key Innovations. Stepping back, the main technical contributions of RL-100 include:. Unified IL+Offline+Online Framework: Many prior works focus on one of these stages, but RL-100 tightly couples them. The “unified PPO objective” across diffusion steps is a clever twist that lets them treat imitation and RL as extensions of each other. This addresses the well-known “imitation ceiling” problem – instead of stopping at cloning, they explicitly build an iterative improvement loop on real data. Diffusion-Policy Backbone: By using a diffusion model as the core policy representation, RL-100 leverages a very expressive action distribution model. This allows learning from multimodal human data (via demonstrations) and smoothly exploring under RL. Embedding diffusion as an inner MDP is also nonstandard and shows how generative models can be integrated with RL objectives. Safety via Offline Policy Evaluation: The OPE gating mechanism for offline RL is a major innovation for safe learning. Offline policy evaluation ensures they don’t degrade the policy by untested updates. Many offline RL papers propose OPE ideas, but here it’s concretely implemented as a check on the PPO updates. It’s reminiscent of methods like conservative Q-learning, but tailored to this diffusion/PPO style update as the guardrail. Consistency Distillation for High-Frequency Control: The idea to train a one-step “consistency model” is drawn from recent diffusion literature, but applying it to robotics control at such scale is novel. It’s digital alchemy: turning a slow, iterative process into a lightning-fast controller without throwing away what was learned. Practical Deployment Demonstrations: It’s rare for an RL paper to include something like a 7-hour in-the-wild run. By doing so, the authors show that their contributions are not just incremental but actually close the gap to a usable robot. They complement lab results with public demos, which elevates the practical credibility of the method. Why RL-100 Matters. RL-100 represents a significant advance in real-world robot learning. For years, roboticists have worried that RL policies would be too brittle for critical tasks. Often you see small-scale RL demos with maybe 10–20 trials of evaluation. Achieving 250-inch-consecutive successes and no failures in 900 trials is unheard of. It shows that RL can go from “interesting demonstration” to “reliable operation” if we build the right training structure. In particular, RL-100 addresses several long-standing concerns:. Overcoming the “Imitation Plateaus”: Pure imitation has a ceiling at human performance. RL-100 explicitly climbs above that by adding RL. This is crucial if we expect robots to outlearn their teachers. The approach suggests a recipe: use demonstrations for competence, then iteratively refine using data. Balancing Safety and Progress: Many RL systems either avoid exploration (become too conservative) or risk catastrophic failure. RL-100 strikes a balance using offline evaluation. This could inspire more safe-RL designs where performance is guaranteed to improve. General-Purpose Robot RL: The task-independence of the approach is impressive. It worked on rigid objects, cloth, liquids, dexterity tasks – all with the same backbone. This suggests diffusion-visuomotor policies have broad applicability, not just one domain. System-Level Thinking: Achieving true reliability required paying attention to every part of the chain: sensors (RGB/point cloud), architecture (diffusion + consistency), algorithms (PPO, IQL, OPE, GAE, variance clip), and even human factors (teleop data, shopping mall deployment). RL-100 is a reminder that making an RL paper real requires end-to-end engineering. From the perspective of the field, RL-100 helps chart the course from “cool lab result” to “real-world product.” It’s possible that the methods here will influence industrial automation, service robots, or any domain where manipulators must handle complex, variable tasks without human crash intervention. Looking Ahead. No system is perfect, and RL-100 has its own limits and open questions. The training involved hours of real robot time and presumably a large volume of demonstrations – a significant investment. It’s not a turnkey solution where any user can dial in their robot and tasks. However, by demonstrating what can be done, RL-100 benchmarks what future systems should achieve. It raises the bar and shows RL can indeed be super reliable if we stitch together the right pieces. Future work might explore scaling RL-100 to even more tasks, or automating the pipeline so minimal human tuning is needed. Integrating sim-to-real learning could make data collection faster. Further, reducing training time (alone, 200 episodes per task is already practical; making it even leaner would boost usability) is an ongoing goal. One exciting thread is the idea of lifelong or continual learning – since RL-100 fundamentally runs iterative updates on gathered data, it could be extended to learn new tasks over time. The consistency between diffusion steps and PPO gradients might allow continual accumulation without forgetting. Finally, RL-100 underscores that bridging the gap between robots and “skilled humans” is within reach. The authors note that RL-100 policies not only match but sometimes exceed human expert speed. As learning algorithms improve further, we might see robots not just replicating human capability but surpassing it in terms of consistency and operating like “robot pilots” that just work tirelessly. Conclusion. RL-100 by Kun Lei and colleagues is a milestone in embodied AI. It blends a rich methodological toolbox into a coherent system that drives real robot arms to near-perfect performance. By achieving 100% success across hundreds of trials, demonstrating adaptability, and running uninterrupted for hours, RL-100 shows that highly reliable, deployment-ready reinforcement learning is possible today. The key lies in structured learning – starting from what humans do best, carefully improving with offline RL, and finalizing with cautious online tuning – all under the hood of powerful generative policies. For the field of robot learning, RL-100 is both an inspiration and a challenge: it demonstrates what excellence looks like and points the way toward the next generation of truly robust, intelligent robots. Sources: The details above are drawn from the RL-100 project page and associated preprint, which describes the training architecture, tasks, and experimental results. These include the abstract and methodological sections (summarized in papers and blogs) and results summaries, as well as supplementary materials and project videos. Each quoted statistic and method comes from these original RL-100 materials.