Latest / Embodied AI 101 / From Demo to Duty: RL-100's Real-World Reinforcement Learning Flywheel

Transcript

From Demo to Duty: RL-100's Real-World Reinforcement Learning Flywheel · Embodied AI 101

• From Demo to Duty: RL-100’s Real-World Reinforcement Learning Flywheel. Performant robotic manipulation with real-world reinforcement learning • Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingxiao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu • Shanghai Qi Zhi Institute; Shanghai Jiao Tong University; University of Hong Kong; University of North Carolina at Chapel Hill; Carnegie Mellon University; Chinese Academy of Sciences; Tsinghua University • Science Robotics • 2026. The headline result in RL-100 is difficult to ignore: one thousand successful real-robot evaluation episodes out of one thousand attempts. But the phrase that matters just as much appears in the final paper’s abstract: this is complete task success under a predefined evaluation protocol. That qualification does not diminish the work. It tells us how to read it—as evidence that reinforcement-learning post-training can turn moderately successful imitation policies into unusually reliable, efficient task specialists, not as evidence that robotics has solved open-world manipulation. A brief version note is worthwhile. The first arXiv release, submitted on October 16, 2025, described seven task families, 900 successful evaluation trials, and uninterrupted runs of up to roughly two hours. The peer-reviewed article, published in Science Robotics on July 22, 2026, adds long-horizon box folding, raises the aggregate evaluation to 1000 out of 1000 episodes, and includes a roughly seven-hour shopping-mall deployment of the juicing system. This episode discusses that expanded, final version. The more durable contribution, however, is not the round number. It is the training recipe behind it: start with a multimodal diffusion policy learned from humans, extract most of the improvement through conservative offline reinforcement learning on a growing real-world dataset, use online RL only for the remaining rare failures, and distill the resulting multistep sampler into a fast one-step controller. RL-100 is best understood as an engineered policy-improvement flywheel. Why imitation learning is not enough. Modern imitation policies are very good at getting a robot into the right behavioral neighborhood. Diffusion Policy and its 3D variants can represent multiple valid action modes, smooth over demonstration noise, and generate coordinated action chunks. Given a few dozen or a few hundred competent demonstrations, they often produce visually impressive behavior. But behavior cloning optimizes resemblance, not deployment performance. A teleoperator may move slowly because the camera feed is delayed, pause to inspect uncertain contacts, or take a safe but circuitous trajectory. Rare recovery states may never appear in the demonstrations. Even if every demonstration succeeds, supervised training does not directly distinguish a fifteen-second success from a forty-second success, nor does it know that a particular hesitation creates a fragile contact three seconds later. That is what the RL-100 authors call the imitation ceiling. The policy inherits the demonstrator’s useful priors, but also the data’s coverage gaps, inefficiencies, and biases. Collecting more human data can expand coverage, and human-in-the-loop correction can patch failure regions, but both continue to consume skilled operator time. Reinforcement learning offers a complementary signal: optimize task return, penalize dithering, experience failures, and discover behaviors that are underrepresented—or entirely absent—in the demonstrations. The standard objection is that this sounds easier than it is. Real-world RL entails hardware wear, resets, safety supervision, reward labeling, low interaction throughput, and representation drift under sparse objectives. Sim-to-real training avoids some of that expense, but precise contact, deformable materials, fluid flow, friction, sensing artifacts, and hand-object dynamics are exactly where simulation error becomes consequential. RL-100’s answer is therefore not “replace demonstrations with RL.” It is “use demonstrations to constrain where RL begins, and spend expensive interaction only where it changes deployment outcomes.”. The three-stage recipe. RL-100 begins with a task-specific conditional diffusion policy trained by behavior cloning. The input combines recent visual observations—primarily point clouds in the real-world experiments—with robot proprioception. Depending on the task, the policy predicts either one action for immediate closed-loop execution or a short action chunk, typically on the order of eight to sixteen commands. The second stage is called iterative offline RL. “Offline” here describes the update, not the entire process. Within an iteration, learning occurs from a fixed dataset. Between iterations, the current policy is deployed, its new trajectories are added to the dataset, and another offline optimization round begins. This is closer to conservative batch policy iteration than to one-shot offline RL. Finally, the authors perform a comparatively brief online, on-policy refinement. The intended division of labor is clear:. Imitation learning places the policy on a physically plausible, human-informed action manifold. Iterative offline RL harvests most of the attainable gain without continuously changing the data distribution during every gradient update. Online RL removes the residual, low-frequency failures that become visible only once the policy is already quite strong. The empirical progression supports this allocation. Averaged across the eight evaluation rows, the 3D imitation baseline reaches 67.8% success. Iterative offline RL raises that to 91.8%. Online refinement produces the reported 100% observed success. In other words, the offline flywheel does most of the work, while online learning handles the expensive last mile. Turning a diffusion sampler into a PPO policy. The central algorithmic problem is that a diffusion policy does not emit an action in one ordinary stochastic-policy step. It starts with noise and progressively denoises that sample over several internal transitions. The environment sees only the final action, but policy-gradient learning needs to know how the probability of that action changed under the updated policy. RL-100 treats action generation as a two-level decision process. The outer level is the familiar robot-environment loop: observe the scene, execute an action, transition to a new state, and eventually receive task reward. Inside each outer step is a short denoising process. Its state contains the current noisy action, the diffusion time index, and the robot observation. Its “action” is the next, cleaner action sample. During RL training, the authors use stochastic DDIM transitions. Each denoising transition can then be viewed as a Gaussian subpolicy with a tractable log probability. At deployment, sampling can return to the deterministic form. This training-time stochasticity is what makes a PPO-style likelihood ratio available at every denoising step. The key credit-assignment decision is deliberately simple. Suppose the final robot action at one environment timestep turns out to have positive advantage. RL-100 applies that same environment-level advantage to every transition that helped denoise the action. The method computes a likelihood ratio between the candidate and behavior policies at each denoising step, clips that ratio as PPO normally would, and sums the resulting objectives over the complete chain. This does not identify which individual denoising transition deserves credit. All of them receive the same verdict. But it is substantially denser than attaching learning only to the final clean action, and it preserves a direct connection between the physical reward and the entire generative process. The same construction also extends to action chunks: the environment reward accumulated over the chunk becomes the decision-level reward, and the discount between chunks is adjusted to account for the skipped low-level timesteps. RL-100 then reuses this optimization primitive in both RL regimes. Offline learning estimates advantage using an Implicit Q-Learning critic: the action is favorable when its learned Q-value exceeds the learned state value. Online learning instead uses generalized advantage estimation from freshly collected trajectories. The data and advantage estimators change, but the actor continues to optimize the same clipped, per-denoising-step objective. That consistency is important. Offline-to-online systems often pretrain with one conservative regression objective and then abruptly switch to a different actor-critic update. Even if both methods work separately, the handoff can produce a policy-quality dip. RL-100 inherits its “one proximal objective across regimes” philosophy from the authors’ earlier Uni-O4 work and adapts it to a diffusion visuomotor policy. Iterative offline RL as a data flywheel. One outer iteration of RL-100’s offline stage has five conceptual steps. First, train Q and value critics on the current real-world dataset using IQL. Second, fit a transition model for offline policy evaluation. Third, perform several clipped policy-gradient updates to produce a candidate diffusion policy. Fourth, estimate whether that candidate is actually better than the current behavior policy. Finally, if it passes that test, deploy it, collect another batch of trajectories, merge those trajectories into the dataset, and begin the next round. The policy-evaluation gate is based on the approximate model-Q method introduced in Uni-O4. The learned transition model generates short candidate-policy rollouts, and the learned Q-function scores the resulting state-action sequence. An update is accepted only if its estimated score improves on the current behavior policy by an adaptive margin set to 5% of the magnitude of the current estimate. Rejected candidates do not become the new behavior policy. This gate is doing more than conventional PPO clipping. Clipping limits how aggressively one optimization step changes the denoising subpolicies. It does not prevent several individually small steps from walking the policy into a region where the offline critic is unsupported and overoptimistic. The OPE gate asks a coarser question: after those gradient steps, does the candidate still look better under a separate model-based estimate?. After collecting new policy rollouts, RL-100 does something especially important: it retrains the diffusion policy with imitation learning on the expanded dataset. The new dataset now contains both human behavior and RL-improved behavior. Supervised retraining consolidates these modes into a fresh behavior prior before the next RL round. That consolidation step is arguably the most underappreciated component of the framework. Pure policy-gradient updates can improve expected return while eroding useful multimodality or forgetting low-frequency but valid strategies. Re-running imitation learning anchors the model to the high-density support of everything that has worked so far. It also turns the policy’s own successful discoveries into ordinary supervised examples, making them easier to preserve in subsequent rounds. The perception stack receives similarly conservative treatment. In the offline pipeline, the actor, critics, and transition model share an encoder initialized during imitation learning. The paper also studies representation adaptation with reconstruction and variational-information-bottleneck regularization. In that ablation, regularized adaptation is more stable than allowing sparse RL rewards to alter the encoder without constraints, and it eventually outperforms keeping the encoder completely frozen. The practical message is not that vision should never change, but that it should not be rewritten solely by a fragile value estimate. Exploration inside the diffusion process is bounded as well. RL-100 clips the variance of stochastic DDIM transitions, avoiding both near-deterministic underexploration and large action deviations that leave the demonstrated manifold. The authors also find that predicting diffusion noise rather than directly predicting the clean action is better for RL fine-tuning. Although noise prediction has higher output variance, that variance behaves as structured exploration within the denoising chain and helps online learning avoid premature convergence. None of this makes the offline stage formally safe or monotonically improving in the real environment. Both the transition model and Q-function can be wrong. “OPE-monotonic” means monotonic according to the approximate evaluator, not guaranteed monotonic hardware return. The value of the gate is empirical conservatism: it creates another barrier between a noisy gradient and the physical robot. Distilling the policy for fast control. A ten-step diffusion sampler may be perfectly adequate for slow pick-and-place, but it becomes a systems bottleneck in reactive pushing, release timing, or disturbance recovery. RL-100 therefore trains a consistency-model head to approximate the multistep teacher in one network evaluation. The student receives an observation and a noisy action sample and predicts the teacher diffusion policy’s final denoised output. A stop-gradient operation prevents the distillation loss from dragging the teacher toward the student. The teacher can continue improving through RL while the student follows its evolving behavior. The distillation term is interleaved with both offline and online optimization rather than applied only as a disconnected compression phase. At the neural-network level, the speedup is substantial. In simulation, a compact 3.9-million-parameter consistency policy reaches a reported 378 action predictions per second, compared with roughly 30 to 35 hertz for representative multistep diffusion baselines. A larger U-Net consistency model reaches 133 hertz. On the physical systems, however, the effective controller is usually capped by the 30-hertz camera and the rest of the control stack. Consequently, the measured end-to-end wall-clock improvement over ten-step DDIM is a more modest 1.05 to 1.16 times. That distinction matters. Consistency distillation removes the policy network as the bottleneck; it does not make the complete robot ten times faster. The remaining benefit is still meaningful: lower and more predictable inference latency, more compute headroom, and the ability to close the loop at the sensor rate. The one-step policy is not universally interchangeable with its teacher. The authors decline to evaluate it on orange-removal because inverse-kinematics pose discontinuities in a narrow, slippery contact were already present in the demonstrations and became worse under the noise-sensitive one-step model. The deployed system therefore retains both forms: DDIM where sampling robustness is needed, and consistency inference where responsiveness dominates. A deliberately heterogeneous robot suite. The real-world evaluation spans UR5, xArm, and Franka manipulators; passive custom end effectors, Robotiq grippers, and a LeapHand; single- and dual-arm configurations; and action spaces ranging from two-dimensional planar deltas to coordinated arm-hand commands. An Intel RealSense L515 supplies depth observations. Point clouds are transformed into the robot frame, cropped to a task-relevant workspace, and reduced to either 512 or 1024 points with farthest-point sampling. Complex 3D demonstrations are collected with Apple Vision Pro hand tracking, while planar pushing and bowling use a joystick. Three tasks use single-action control: Dynamic Push-T, Agile Bowling, and Pouring. The other learned tasks use action chunks to smooth precision or bimanual behavior. This is a sensible acknowledgment that “high frequency” is not a universal virtue. A bowling policy must react and release quickly; a box-folding policy benefits from short, coherent motion segments that do not jitter at every sensor frame. The task suite is unusually broad for a real-world RL paper. Dynamic Push-T requires inserting a randomly placed and oriented T block into a slot with only about three millimeters of clearance on each side. Agile Bowling requires sending a small curling stone roughly 60 centimeters toward six pins and knocking down at least five. Pouring controls a Franka-LeapHand system and tests transfer from granular contents to water. Dynamic Unscrewing must roll a nut up a bolt, perceive when it has cleared the thread, transition to a pinch grasp, and place it on a plate. Soft-towel Folding requires two arms to flatten a randomized crumpled towel and execute two folds. Box Folding extends the horizon further: two arms must grasp thin paperboard edges, reorient a blank, manage spring-back and intermittent contacts, and form a self-supporting structure without allowing early errors to become irreversible. The orange-juicing system learns fruit placement and removal, while the lever-pressing middle stage follows a fixed kinesthetically recorded trajectory. Rewards are not fully autonomous. Except for Push-T, a human supervisor marks terminal success with a keyboard. The authors add a small step cost to encourage fast completion and a temporal action-smoothness penalty to suppress jitter. Push-T instead uses a shaped reward based on block-to-goal pose error, lack of progress, and action changes. This is pragmatic reward engineering aligned with the paper’s deployment metrics, but it remains an important part of the system. Reading the results beyond the headline. The imitation-only baselines reveal why post-training matters. The 2D Diffusion Policy averages 45.3% success, and DP3 averages 67.8%. On individual tasks, DP-2D achieves only 14% on bowling and 12% on box folding. DP3 raises bowling to 80%, but reaches only 48% on pouring and box folding. Iterative offline RL lifts the unweighted mean to 91.8%, after which online learning closes the observed gap on every task. The thousand successes combine two deployment policies. The multistep DDIM variant records 450 successes in 450 trials over all eight evaluation rows. The consistency variant records 550 in 550 trials over seven rows, including 250 consecutive towel-folding successes. Orange-removal is evaluated only with DDIM. This is still a powerful systems result, but it is not one policy executing one thousand identically distributed episodes. The stress tests are more informative about the shape of the learned robustness. Across six zero-shot variants, the corresponding task policies average 90% success: 100% under changed Push-T friction, 80% with Push-T distractor objects, 100% on a changed bowling surface, 90% when pouring water rather than nuts, 80% on an unseen towel shape, and 90% for strongly rotated, out-of-distribution box blanks. Three few-shot adaptations average 86.7% after one to three additional hours of training, although a substantially changed pouring container reaches only 60%. Human-applied disturbance tests average 96%, including successful recovery from counter-rotation during unscrewing and repeated pushing or tapping during box folding. Efficiency improves for two separate reasons: RL shortens the behavior, and distillation shortens inference. Towel folding drops from 390 environment steps for DP-2D to 312 for RL-100’s consistency policy. Unscrewing drops from 361 to 280 steps. Box folding falls from 1266 to 832 steps and from 65.1 seconds to 41.4 seconds. When Push-T failures and timeouts are included, average episode length declines from 822 steps with DP-2D to 322 with the final DDIM policy. The human comparisons should be read as task-specific throughput tests. In bowling, RL-100 succeeds on all 25 attempts, compared with 14 total successes by five human players. Under an equal wall-clock budget on Push-T, the robot completes 20 successes, an expert teleoperator 17, and a novice 13. These experiments support the narrower statement that RL post-training can remove teleoperation latency and optimize repetitive task execution beyond the demonstrations—not that the system possesses generally superhuman manipulation ability. The data budget is also revealing. Summing the final eight-task table in arXiv v4 gives 1,004 human demonstrations collected over about 15.5 hours, followed by 4,573 rollout episodes for iterative offline learning and 3,223 online episodes. Those autonomous rollouts consume roughly another 97 robot-hours. Expert labor is therefore a minority of the trajectory budget, but physical interaction is not trivial. The approach shifts the bottleneck from skilled teleoperation toward supervised robot operation, labeling, resetting, and hardware availability. The seven-hour mall demonstration is perhaps the most compelling qualitative result. The system transfers to a new public environment and repeatedly prepares juice for visitors without a reported failure. It shows that the learned placement and removal policies can survive a longer operational window than a conventional lab evaluation. It should still be seen as one carefully managed service deployment rather than a statistical estimate of months-long availability. How RL-100 relates to nearby approaches. SERL established that carefully implemented off-policy RL, demonstration replay, visual rewards, compliant control, and automated resets could train useful manipulation skills in roughly 25 to 50 minutes. HIL-SERL then added real-time human corrections and demonstrated near-perfect performance on dynamic, precise, and dual-arm tasks after approximately one to two and a half hours of training. Those systems emphasize rapid online learning and system infrastructure; HIL-SERL explicitly spends human attention in the states where intervention is most valuable. RL-100 makes a different resource trade. Humans provide the initial demonstrations, terminal labels, resets, and safety oversight, but most policy improvement comes from batches of autonomous rollout data. The offline gate and supervised reconsolidation reduce the number of policy changes made directly in a live on-policy loop. This makes sense for longer-horizon tasks where every online update is expensive to validate. Algorithmically, DPPO is the closest diffusion-policy reference point. DPPO showed that policy gradients can fine-tune diffusion policies effectively by treating denoising as an internal decision process, obtaining structured exploration and stable learning across continuous-control benchmarks. RL-100 builds on that family of ideas but adds the offline-to-online unification, OPE-gated batch improvement, iterative real-world data expansion, representation regularization, and one-step consistency deployment. The intellectual through-line to Uni-O4 is equally direct. Uni-O4 proposed using an on-policy-style proximal objective in both offline and online regimes, with offline evaluation deciding whether to advance the behavior policy. RL-100’s main algorithmic synthesis is to make that recipe work inside a high-dimensional visuomotor diffusion sampler and then demonstrate it across several physical systems. What the 100% result does—and does not—establish. First, observed perfection is not a zero failure probability. With no failures in 50 trials, the simple “rule of three” still places the approximate 95% upper bound on the underlying failure rate near 6%. For 250 out of 250, it is about 1.2%. Pooling all one thousand trials into a single confidence statement would be misleading because the episodes span different tasks, policies, robots, and conditions. The correct conclusion is that no failures were observed in a substantial but bounded protocol. Second, the algorithm is general, but the deployed policies are task specialists. Each task has its own data, reward definition, action space, workspace, perception crop, control mode, and often custom end effector. “Task- and embodiment-agnostic” describes the software and optimization scaffold; it does not mean a single learned policy can fold towels, bowl, pour, and juice oranges on command. The authors explicitly position multi-task and vision-language-action post-training as future work. Third, the environments remain structured. Cameras are calibrated, task workspaces are bounded, and object initialization follows specified distributions. The towel setup assumes that two useful grasp corners are visible after initialization. Orange pressing is scripted. The strongest geometric change in pouring still reduces few-shot success to 60%. The paper does not yet test the clutter, transparent objects, moving distractors, changing illumination, arbitrary camera placement, or severe partial observability expected in a general home environment. The authors themselves identify these as open evaluation directions. Fourth, real-world autonomy around the policy is incomplete. Humans label sparse rewards, reset tasks, approve training, and terminate unsafe episodes. RL-100 reduces reliance on expert demonstrations, but it does not solve unattended data collection. Autonomous resets and recovery are explicitly listed as remaining bottlenecks. The mall deployment demonstrates policy robustness, not an end-to-end autonomous retail operation with perception, inventory handling, cleaning, customer interaction, and fault recovery. Fifth, the OPE gate is a risk-control mechanism rather than a safety guarantee. A candidate can pass because the learned model and critic share the same blind spot. There are no formal constraints ensuring collision avoidance, bounded contact force, or monotonic true return. The system relies on conservative initialization, clipped updates, variance bounds, workspace limits, emergency stops, and human oversight. Finally, reproducibility remains partly a systems challenge. The public repository is broader than the paper—it now includes diffusion and flow policies, 2D and 3D observations, single actions and chunks, offline and online policy gradients, and one-step distillation. It provides simulation recipes and a downloadable smoke-test dataset. At the same time, its own documentation says that hardware-specific command lines and concrete real-robot recipes still need to be filled in for some setups. The code is a useful algorithmic base, but reproducing the thousand-trial result still requires substantial robotics integration. Lessons for practitioners. For teams adapting this work, the most transferable ideas are organizational rather than architectural. Treat imitation as initialization, not the final objective. Use demonstrations to establish safe, multimodal behavior, then optimize the quantities deployment actually cares about. Separate data acquisition from most optimization. Collect a rollout batch, freeze it, train critics, gate candidate updates, and validate before returning to hardware. This creates cleaner debugging boundaries than an uninterrupted online loop. Consolidate RL discoveries with supervised learning. Re-training on the merged human-and-policy dataset can preserve multimodality and reduce forgetting better than expecting policy gradients to maintain every useful behavior. Reserve online RL for rare failures. If offline policy iteration can move success from roughly 70% to above 90%, spend the difficult on-policy budget on the tail rather than relearning the center of the distribution. Make action horizon and inference latency task-dependent. Single-step control is appropriate for reactive dynamics; chunks are useful for smooth, precision-heavy coordination. Distillation is valuable when policy inference, rather than sensing or control, limits the loop. Evaluate throughput, recovery, and long runs—not just success. Completion time, failure-aware episode length, perturbation recovery, and uninterrupted operating time reveal policy quality that a short success-rate table misses. The deeper systems insight is that robot RL does not need to be a single heroic online-training run. It can be a managed progression through demonstration, conservative batch improvement, supervised consolidation, targeted online refinement, and deployment compression. The bottom line. RL-100 does not show that PPO alone solves real-world manipulation, nor that a thousand successes guarantee general-purpose reliability. It shows something more actionable: with a strong diffusion prior, bounded exploration, an offline policy-evaluation gate, repeated data consolidation, and a small online last mile, reinforcement learning can become a practical post-training tool for physical robots. That makes RL-100 an important marker for embodied AI. The field’s next question is no longer simply whether real-world RL can improve a visuomotor policy. It is whether the same flywheel can scale from carefully engineered task specialists to multi-task, cross-embodiment models—while automating rewards, resets, recovery, and safety supervision. RL-100 provides one of the clearest blueprints yet for attempting that transition.