Latest / Embodied AI 101 / Mulligan: Can Failure-Focused Data Improve Robot Imitation Learning?

Transcript

Mulligan: Can Failure-Focused Data Improve Robot Imitation Learning? · Embodied AI 101

• Mulligan: Can Failure-Focused Data Improve Robot Imitation Learning?. Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning • Lars Ankile et al. • Stanford University • arXiv preprint • 2026. When a manipulation policy already handles some object placements reliably, why spend the next collection session demonstrating those placements again?. Mulligan’s answer is to redirect supervision toward observed failures while continuing to explore unfamiliar starting configurations. The results make this worth trying: the complete system improves final autonomous success by 10–34 percentage points over uniform-sampling HG-DAgger across three real manipulation tasks. But that headline combines a collection strategy with value-based action selection. It does not isolate the benefit of choosing better resets. The distinction that matters most for practitioners is the denominator. This is evidence for better learning at matched supervised episode counts, not demonstrated savings at matched operator hours. The strongest final-checkpoint evidence is on marker insertion; the other tasks are less conclusive individually. My take: Mulligan offers a useful data-collection workflow, not a general solution to autonomous improvement. Its most transferable contribution may be the decision to treat object placement as part of the learning algorithm. What the Paper Did. The real-robot experiments use a DROID-style Franka Panda station. The tasks are inserting a marker into a holder, placing a square nut onto a peg, and seating a rope into two clips. These are contact-sensitive manipulation problems, rather than semantic instruction-following benchmarks. The starting configurations vary substantially. For example, the nut’s center ranges over approximately 25 by 46 centimeters, with a full rotation range; the peg also moves. Marker insertion ends with roughly one millimeter of clearance. This combines a variable approach-and-grasp problem with a precise final placement. That combination is important. If every episode began with the object already aligned in the gripper, targeted resets would address a different problem. Here, changing the initial placement can change which grasp is feasible, what transport motion is needed, and whether the robot reaches the insertion stage at all. Real-world training begins with 100 demonstrations per task and proceeds through five collection rounds: 50 supervised episodes per round for marker and nut, and 100 for cable routing. Policies remain fixed during collection and are updated between rounds. The collection strategy combines two operations. It returns to failed initial states, then fills the remaining budget with configurations that improve coverage. The implementation uses farthest-point selection and can add a task-specific difficulty weighting. Sobol sampling provides the initial coverage-oriented alternative to independent uniform draws. In plain language, the next session contains both “try this troublesome setup again” and “try a setup we have not adequately covered.” Those serve different purposes. Retrying exploits evidence already available. Coverage protects against building an increasingly detailed map of one failure region while overlooking another. The physical reset is essential. This is not merely resampling difficult transitions from a replay buffer. The operator actually places the objects into selected configurations, allowing the system to obtain new supervision there. Mulligan also supports a full demonstration from a failed starting configuration—a counterfactual demonstration—instead of relying exclusively on mid-trajectory corrections. Such demonstrations count against the collection quota rather than arriving as free additional data. The released collector explicitly implements this separate replay option. Consider a hypothetical marker grasp that leaves the gripper obstructing insertion. A corrective intervention might put the marker down and regrasp it. That teaches recovery. A fresh demonstration from the original configuration can instead teach the initial grasp that avoids the problem. These are different lessons. Recovery supervision answers, “What should happen after the robot makes this mistake?” A counterfactual demonstration answers, “What should it have done before making the mistake?”. That distinction helps explain why additional corrective data need not efficiently repair an upstream decision. If every intervention starts after the bad grasp, the dataset may become rich in regrasping behavior while remaining poor in examples of the correct first grasp. The complete recipe combines these collection choices with a critic that selects among proposed actions. Its real-world result is higher autonomous success at the matched episode budgets, not simply more episodes completed with human help. Separating the collection contribution from the critic contribution requires looking beyond that combined result. What Is Actually New?. The right historical comparison starts with HG-DAgger. Michael Kelly and colleagues developed human-gated DAgger so an expert can take control when necessary, rather than continuously providing labels while another controller drives the system. The original work addressed practical problems in interactive imitation learning, including label quality and the human’s experience of control. Mulligan changes a different decision: not only when the human intervenes, but which situation the robot encounters next. That is a useful separation. Intervention policy determines the supervision obtained along a trajectory. Initial-state selection determines which trajectories are likely to occur. Improving one does not automatically optimize the other. Nor is selective restarting itself new. Minqi Jiang and colleagues’ Prioritized Level Replay selects training levels according to estimated learning potential rather than sampling them uniformly. It demonstrated that adaptive selection can improve sample efficiency and generalization in procedurally generated environments. The interesting translation here is into supervised physical manipulation. With a competent human available, a configuration need not be at just the right difficulty for autonomous exploration. The operator can supply behavior the policy cannot yet discover. That changes the trade-off. An extremely difficult configuration may be unproductive for an autonomous learner that receives only failure. It can be productive for an imitation learner if a human can demonstrate the missing maneuver. Difficulty alone is therefore not the target; difficulty plus available, learnable supervision is. Recent deployment-learning work provides another useful comparison. Physical Intelligence’s RECAP combines demonstrations, autonomous experience, and expert interventions through advantage-conditioned VLA training. Its reported domains include laundry folding, box assembly, and espresso preparation. RECAP mainly asks how to learn from heterogeneous experience. Mulligan mainly asks which supervised experience to acquire next. Those questions are potentially complementary. But “potentially” matters. Compatibility at the algorithmic level is not evidence that a reset strategy will deliver the same gains when attached to a large generalist VLA. That would require another experiment. The novelty I would credit is therefore specific: making initial-state selection and corrective-data choice explicit, operational parts of a supervised robot-learning loop. The contribution does not depend on claiming that hard-example selection or human intervention was invented here. What the Critic Adds—and What It Cannot Add. The actor is a U-Net Diffusion Policy with per-camera ResNet18 encoders in the real world. It imitates successful human-controlled frames. The critic also uses autonomous experience, including failures; critic gradients do not update the actor. This separates two questions that are often conflated: “Which actions should the robot imitate?” and “Which experience can help evaluate actions?”. A failed trajectory can be inappropriate as an imitation target while still containing useful information about consequences. If the robot repeatedly approaches a fixture from an unfavorable direction, copying those motor commands is questionable. Learning that those commands tend to precede failure is a different proposition. The action-selection mechanism builds on IDQL. Philippe Hansen-Estruch and colleagues proposed using an expressive diffusion behavior model together with a separately learned critic, extracting improved behavior by reweighting or selecting policy samples. This avoids requiring the critic to directly optimize the diffusion model’s parameters. Mulligan’s released real-robot deployments use 16 or 32 candidate action chunks, depending on task and round. Its critic uses the actor’s frozen visual representation; the released configurations also specify chunked actions and a distributional value head. The appeal is straightforward: preserve an imitation-trained proposal distribution, then use experience to discriminate among its proposals. Perry Dong and colleagues’ earlier What Matters for Batch Online Reinforcement Learning in Robotics? already emphasized Q-functions, expressive policies, and implicit action selection as useful ingredients for improvement from batches of autonomous data. Mulligan extends this neighborhood of methods with targeted human supervision rather than introducing value-guided sampling from scratch. There is, however, a hard boundary to reranking. If the actor never proposes the necessary behavior, a perfect selector still cannot execute it. Imagine that all candidate chunks continue pushing against the wrong side of a fixture. Choosing the best of those candidates is not equivalent to proposing a retreat, reorientation, and new approach. A fresh demonstration can expand the actor’s behavioral repertoire; a critic can preferentially select behavior already represented within it. This gives a practical diagnostic. Before investing heavily in critic tuning, inspect candidate diversity at failure states. Does a plausible successful continuation appear among the samples? If not, the bottleneck may be proposal coverage rather than value estimation. The reverse problem also deserves attention. A critic might distinguish easy scenes from difficult scenes without reliably distinguishing good actions from bad actions within the same scene. Those are different capabilities. For deployment, the latter is what reranking needs. I would therefore evaluate the selector through paired robot trials, not only through whether its values correlate with recorded episode outcomes. A useful predictive model is not automatically a useful action selector. How Strong Are the Real-Robot Gains?. The 2,550 blinded evaluations are a substantial physical-robot evaluation effort. The released protocol uses matched starting configurations, shuffled execution order, and anonymous policy labels so the operator does not receive the policy identity while resetting and judging trials. These choices deserve credit. Pairing reduces the risk that one method simply receives easier placements. Interleaving reduces the risk that all trials for one method happen before a change in the scene or operator behavior. Blinding is not magic: visibly different behavior could still provide clues. But concealing identities is a meaningful improvement over openly evaluating a favored policy against a baseline. The final results are easiest to understand by separating three versions:. Marker insertion: uniform HG-DAgger reaches 42%; Mulligan’s actor without reranking reaches 64%; the full method reaches 76%. Nut placement: the corresponding results are 62%, 70%, and 72%. Cable routing: full two-clip completion is 18%, 28%, and 34%. Each final comparison uses 50 starts per arm. Those are useful improvements, but they do not all carry the same statistical weight. For the full method against the baseline, marker insertion has a 34-point gain with a reported paired interval of 16–52 points and an exact McNemar p-value of about 0.0015. Nut placement’s interval spans −8 to +26 points. Cable routing has a bootstrap interval of +2 to +30, but its exact McNemar p-value is about 0.077. I would call the marker result convincing within this experiment. I would call nut placement uncertain at the final checkpoint, and cable routing promising but not decisive. For cable, selecting whichever statistical procedure gives the more favorable answer would overstate the evidence. The critic’s final incremental gains over the Mulligan actor are 12, 2, and 6 points; none is individually significant in the reported paired tests. Pooled across rounds, the actor-level Mulligan comparison reports gains of approximately 11.7, 10.9, and 6 points. That makes the collection strategy a more compelling first intervention than the headline alone might suggest. It does not establish that the critic is unnecessary. It establishes that the evidence for its incremental benefit is less decisive than the evidence for the complete package on marker insertion. Another limitation concerns experimental independence. The real-world study has one collection campaign per strategy, not repeated independent campaigns. The actor-only Mulligan evaluation uses data from the Mulligan campaign rather than a separately collected no-critic campaign. Many evaluation episodes improve measurement of the trained policies. They do not measure how much the entire learning process varies with a different operator, collection history, or training campaign. Likewise, pooling related checkpoints answers a campaign-level question. It should not be interpreted as hundreds of independent repetitions of the learning algorithm. The counterfactual-demonstration ablation adds useful nuance: marker insertion improves from 14% to 38% in the first measured round, but the two variants are nearly tied in the next. My interpretation is that full demonstrations can accelerate early repair of missing behavior without necessarily producing a lasting gap once corrective data catches up. That supports selective use, not a rule that every intervention should trigger another complete demonstration. One final plotting caution: the real-world learning curves use one-standard-error whiskers, not 95% confidence intervals. The appropriate conclusion is narrower than “all three tasks are solved significantly better.” There is a strong marker result, positive point estimates elsewhere, and campaign-level evidence favoring focused collection. The Simulation Caveat That Changes the Interpretation. The simulated tasks are Square-Narrow and Square-Broad, two versions of square-nut insertion. The broader version randomizes both nut and peg placements. These are useful controlled variations, but not two independent families of manipulation skill. Simulation uses state observations rather than the real policies’ images, with five training replicates. Consequently, I would not treat agreement between the simulated and physical results as a demonstrated sim-to-real transfer result. It is evidence from separate implementations of the learning recipe. The simulation helps probe collection and optimization while removing part of the visual problem. There is also an important evaluation disclosure. In three simulated rounds—Square-Narrow round three and Square-Broad rounds two and three—the sampler uses a difficulty term derived from performance in cells of the fixed evaluation grid. The release explicitly exposes that input and identifies the rounds where it is enabled. This changes how I interpret those results. Keeping exact collection states separate from exact evaluation states is not the same as keeping evaluation information outside the learning loop. If evaluation performance tells the sampler which regions deserve more training, the collection process has adapted to that evaluation distribution. That can be a perfectly sensible deployment procedure. An operational diagnostic set is allowed to guide improvement. But it is a weaker test of generalization than a final set whose results never influence training or collection. The clean follow-up is straightforward: use one diagnostic distribution for selecting starts, then evaluate on a third, sealed set after all choices are fixed. Alternatively, rerun the affected rounds without evaluation-derived difficulty. This issue does not erase the physical-robot results. It reduces how much independent corroboration I assign to the affected simulation comparisons. The online-RL baselines require similar care. The released baseline documentation reports HiL-SERL at 94% on the narrow task and 2% on the broad task. It also shows strong sensitivity to demonstration source and budget: RLPD reaches 44% on the broad task when initialized with 1,000 released MimicGen demonstrations. These are informative outcomes, not a general law that online RL fails under broader resets. The original HiL-SERL paper reported strong real-world results across precision, dynamic, and dual-arm manipulation, using demonstrations and human corrections. Its tasks and system configuration differ from this simulated port. A fair reading is that Mulligan performs well in the authors’ low-budget, broad-placement setting, while the tested online-RL recipe struggles there. A less defensible reading would declare the broader class of human-in-the-loop RL obsolete. The uniform HG-DAgger comparison is the more direct test of the collection question. A baseline does not become a strawman merely because its algorithmic name is older; what matters is whether the comparison holds the relevant implementation and resource choices constant. Equal Episodes Are Not Equal Human Effort. The user-facing promise is data efficiency. The operational question is labor efficiency. They are related, but not interchangeable. A supervised episode can contain mostly passive observation, a short corrective intervention, or a full teleoperated demonstration. Resetting an awkward configuration may also take longer than placing an object in an easy one. As a thought experiment, compare fifty episodes requiring occasional two-second corrections with fifty episodes requiring frequent complete demonstrations. Matching the episode count does not match the human-control time. Conversely, difficult episodes could be worthwhile if they eliminate failures much faster. That is why I would keep several denominators separate: supervised episodes, human-controlled seconds, total station time, and accepted autonomous completions after learning. Mulligan reports that the human–robot team completes 98% of 1,009 fresh collection episodes. That is assisted collection productivity, not the autonomous policy’s success rate. The result matters: collecting useful data need not mean abandoning task completion. But it does not establish how much output one operator could support across multiple robots, or whether focused collection reduces the staffing needed for a production target. There is another deployment question hidden inside the reset choice. Can the operation actually select which configurations arrive next?. A laboratory workcell may permit arbitrary placement. An incoming production stream may not. In that setting, the practical adaptation might be to select among naturally arriving cases, schedule a separate improvement session, or route difficult cases to a supervised station. Those are deployment designs to test, not capabilities established by this paper. Finally, the hardest states are not automatically the best investment. A failure caused by absent visual information, an unsuitable tool, or an infeasible grasp may not be repaired by another demonstration. I would distinguish “the policy has not learned this” from “the current system cannot reliably perform this.”. Focused collection should prioritize correctable weaknesses, not simply maximize exposure to failure. Can Another Lab Reproduce It?. The release is materially useful. The official repository provides learners, collection and evaluation pipelines, locked starting-state manifests, configurations, and figure-building tools; datasets and checkpoints are publicly available. It distinguishes rebuilding reported figures, recomputing results, retraining models, and conducting a new collection campaign. That distinction is particularly appropriate for interactive learning. Retraining on recorded interventions is not equivalent to reproducing the collection process. A different policy changes when the human intervenes. A different human changes the intervention. Those differences then affect later training and future collection. The released marker round-five actor configuration, for example, pins dataset revisions, camera crops, seeds, and a 100,000-update training budget. It also specifies the batch size and image resolution. These details make the training recipe much more inspectable than a generic instruction to fine-tune a diffusion policy. There are still limits. The final marker sampler configuration includes task-specific promotion tiers and notes that its original outcome and stage-label input tables are not released. The locked selected starts are supplied. That means reusing the recorded design and reconstructing every historical design decision are different levels of reproducibility. It also shows why adapting the method may require more engineering than implementing the short conceptual sampler. The baseline documentation likewise states that baseline checkpoints are not released and identifies a simulator-version difference between historical RLPD runs and the current release environment. These disclosures are helpful rather than disqualifying. They identify which comparisons can be rebuilt from recorded evidence and which require a new experiment. A competent team has enough material to inspect the data lineage, reproduce substantial downstream computation, and implement a new campaign. That is stronger than a demonstration-only release. It is not an independent replication. I would still want another operator or lab to show that focused collection wins under a prospectively fixed protocol, particularly when measured in operator minutes rather than only episodes. What This Means for Practitioners. I would pilot Mulligan-style collection in a bounded manipulation workcell where teleoperation and manual resets already exist. I would not treat these results as justification for unattended deployment. The first step would be instrumentation, not another network. Record the starting configuration, policy version, episode outcome, intervention timing, and a compact failure category. Preserve the association between a failure and the initial arrangement that produced it. Without that association, “collect more failures” can become an informal operator habit rather than a measurable strategy. Then establish three collection controls: uniform sampling, coverage-oriented sampling without failure targeting, and coverage plus targeted retries. The coverage-only control matters. Otherwise, an apparent benefit from failure awareness may partly come from avoiding the accidental clusters and holes of a small uniform sample. Similarly, keep full counterfactual demonstrations identifiable rather than silently mixing them into the intervention pool. I would match both episode count and human time in separate comparisons. Those answer different questions, and there is no reason to force one denominator to stand in for the other. Second, preserve the intended deployment distribution for evaluation. Targeted training should not imply targeted reporting. If the robot will encounter a broad range of placements, the acceptance test should measure that range—including configurations that became less represented during focused collection. Keep diagnostic trials and acceptance trials conceptually separate. Diagnostic outcomes may guide the next training round. Acceptance outcomes should remain sealed until the relevant choices are fixed. The simulation caveat in this paper is a good reason to build that distinction into the tooling from the beginning. Third, decide whether the current bottleneck is missing behavior or poor selection. If no plausible successful action appears among policy samples, collect demonstrations that supply the missing approach, grasp, or recovery. If plausible alternatives appear but the robot frequently chooses the wrong one, value-based reranking becomes a more compelling experiment. Measure the critic against its own actor on paired starts. Do not infer its usefulness from the complete system beating a weaker collection baseline. Also check whether improved completion is purchased through excessive hesitation, retries, or reliance on eventual operator rescue. For scaling beyond one workcell, I would require additional evidence along three axes: independent collection campaigns, operator-time accounting, and changed deployment conditions. New object placements are not the same test as new objects, altered camera geometry, different lighting, or another embodiment. The same caution applies to VLAs. The collection decision could plausibly sit outside a VLA training recipe, but architectural compatibility is only the beginning. A useful test would hold the large policy’s update procedure fixed and vary the collection strategy prospectively. What is supported here is practical and valuable: supervised experience need not be acquired uniformly, and returning to selected failures can improve the usefulness of a limited collection budget. What remains uncertain is the full labor advantage, the consistency across independent campaigns, and the incremental payoff of the critic on every task. My recommendation is to adopt the measurement discipline first: preserve failure provenance, retain coverage, charge full demonstrations honestly, and evaluate on starts that have not guided collection. Then test whether focused resets outperform your current workflow. Before scaling the dataset, it may be worth changing which episode you collect next.