Latest / Embodied AI 101 / ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning

Transcript

ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning · Embodied AI 101

• ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning. In the latest wave of embodied AI, Vision-Language-Action (VLA) models have emerged as powerful generalist controllers for robots. For example, the newly introduced OpenVLA is a 7-billion parameter model trained on almost a million robot trajectories, setting a state-of-the-art for multi-robot manipulation in generalist tasks. These VLA models—essentially large language models fine-tuned to map camera images and task instructions to control commands—make it feasible for robots to interpret natural language goals and coordinate complex vision-driven actions. But deploying these models on real hardware, especially humanoid robots, often leads to failed attempts due to inevitable sim-to-real shifts or distribution differences. In practice, a human supervisor must often jump in and correct the robot’s actions in real time, a process known as human-in-the-loop intervention. Human interventions are crucial: they provide corrective feedback that can guide a policy to recover from mistakes. But in the world of humanoids, these interventions themselves are hard to obtain and suboptimal. Typical robot arms are relatively easy to teleoperate; humanoids, by contrast, have full-body balance, locomotion, and dexterous hands, making teleoperation extremely challenging. As the authors of “ROVE: Unlocking Human Interventions for Humanoid Manipulation” point out, even a skilled operator using VR headsets and motion-capture can produce hesitant, inefficient or even erroneous corrective motions when taking over a failing humanoid task. Naively treating these takeover demonstrations as “expert” behavior (as in classic learning-from-demonstration) would simply teach the robot all the mistakes. Instead, ROVE adopts a hybrid reinforcement learning (RL) approach that explicitly accounts for the mixed quality of intervention data. ROVE stands for Reinforcement learning framework with Optimistic Value Estimation. At a high level, ROVE does three things: (1) it establishes a realistic human-in-the-loop pipeline for collecting humanoid manipulation data, (2) it trains a state-value critic on this data using a bias toward high-return experiences (the “Optimistic Value Estimation” or OVE), and (3) it updates the policy (actor) by focusing only on those actions that the critic judges to have positive advantage. In effect, the policy is only asked to imitate the intervention data that demonstrably helped the task, while ignoring the noisy or unhelpful parts. In practice this means ROVE learns to steer the robot toward recoveries rather than blindly copying everything the human did. On challenging real-world tasks – for instance, using its whole body and hands to erase a whiteboard, or to put a loaf of bread into a toaster – ROVE yields dramatic improvements. In experiments, success rates climb from roughly 45% to 80% on the whiteboard task and from ~57% to 87% on the toaster task over just a few rollout-intervention cycles. By comparison, simple supervised fine-tuning on the intervention clips or naive imitation learning methods performed poorly. In short, ROVE’s RL-based filtering of human corrections lets the robot extract the good parts of imperfect data and significantly boost performance. Below we unpack ROVE’s approach in detail, situating it in the context of related work and explaining how each component works. The goal is to give robotics and ML researchers a clear picture of what ROVE did, why it matters, and how you might build on or use similar ideas in your own humanoid systems. Background: Human-In-The-Loop Learning for Robotics. Training robots from data often involves a mix of expert demonstrations, autonomous exploration, and human corrections. Classic imitation learning (IL) assumes the demonstration data is high-quality expert behavior. Algorithms like DAgger (Dataset Aggregation) collect human corrections when the policy deviates, then aggregate those new labels into the dataset. But DAgger and related IL schemes (DAgger 2008, interactive imitation, etc.) implicitly assume the human can provide ideal actions during corrections. In practice, especially for complex agents, that’s rarely true. Even without RL, recent works recognize the need to scale imitation to more realistic human data. For example, EgoMimic (Kareer et al.) shows how to leverage large collections of egocentric human videos to scale up robot imitation. EgoMimic provides a whole pipeline – from data collection to training – that uses VR setups to record humans doing tasks and then trains policies on that data. Their key insight is that humans can demonstrate tasks in VR, and the robot can imitate those trajectories. This is a related line of work to ROVE. In ROVE’s case, however, the human demonstrations come after a failure, and often involve initial alignment phases that are not purely task demonstrations. So ROVE’s algorithm must be careful to extract the useful signals. Another recent trend is using reinforcement learning with offline data and then fine-tuning. For example, AWAC (Advantage-Weighted Actor-Critic) showed that an actor-critic algorithm with advantage-weighted supervised updates can fine-tune robotic policies quickly if seeded with offline demos. The idea there is similar: use an offline dataset of demonstrations (possibly suboptimal) to shape the critic, then do a policy update that weights actions by their advantage. ROVE builds on this offline+online idea but tailors it for intervention data specifically, and for humanoid control. A final piece of context is the rise of vision-language-action (VLA) models. VLA models are a new paradigm that pretrained large language models to output actions for robots. Models like RNTR or PaLM-E, and especially the recent open-source OpenVLA, show that one can train a transformer to ingest images and text instructions and then output joint commands. In ROVE, the base humanoid controller (“the actor”) is one such VLA model. That means the policy is actually a giant transformer that has been pretrained on a lot of offline data (both robot demos and human videos). ROVE then tries to make this VLA policy better through post-training: essentially letting it learn from mistakes at deployment time. This setup – a pretrained VLA policy plus a loop of rollout and intervention – is where ROVE operates. The authors note that a humanoid VLA policy, even if trained on a huge dataset, will struggle at run time due to distribution shift and compounding errors over long tasks. Having a human intervene can fix those errors, but the way one learns from that intervention is critical. That’s what ROVE’s RL-based framework does. Human-in-the-Loop Data Collection Pipeline. A core contribution of ROVE is engineering a realistic collection system for mixed-quality humanoid data. The pipeline has two modes for an episode: autonomous rollout and human intervention. In autonomous rollout, the robot simply executes the current VLA policy (in a real environment, possibly finite-paced due to physics). If/when the robot appears to be heading for failure (by some criteria), the system pauses the autonomous policy. At that point, a human supervisor takes over control via teleoperation. The teleoperation interface is VR-based. The human wears a VR headset (for egocentric vision) and uses tracked controllers or motion capture mark-up to align their own body/hand pose with the robot. In other words, the system provides a “puppeteering” setup: as the words in the ROVE paper put it, the operator aligns their body and hand with the humanoid and then moves to finish the task. During this takeover, the operator goes through an adaptation stage to master control, and then a recovery stage to actually complete the task. The key point is: the adaptation stage is not optimal task behavior.* It’s basically the human getting the hang of the interface. Because of this two-stage nature, ROVE’s designers carefully label the data. They consider an episode that gets taken over at time $t_0$ to have an adaptation portion (from $t_0$ up to some $t_r$ when the human is fully in sync) and then a recovery portion (from $t_r$ until the end when the task finishes). In the data log, all of these human-driven steps get recorded: the actions the human took, the state transitions, and so on. Crucially, they also annotate whether the episode eventually succeeded or failed. If the robot completes the task (green light), that portion is a successful trajectory; if it fails (drops object, times out, etc.), that is a failure. This pipeline is important for multiple reasons. First, it acknowledges that real intervention data is noisy: operators might hesitate or move suboptimally at first. Second, it lets ROVE distinguish which part of the trajectory is truly valuable. The authors mention a “teleoperation gap”: humanoid teleoperation takes time to learn (an operator might fumble at first). By separating adaptation from recovery, ROVE avoids treating those shaky first steps as something to imitate. In practice, they use simple time-based reward: the whole episode gets reward $0$ if successful at the very end, or a negative penalty if the episode ultimately fails (including if the human never recovers). Meanwhile every step (during both adaptation and recovery) gets a small negative time penalty (constant $-1$). With discount factor $\gamma=1$, this effectively encourages finishing quickly and punishes failures. (In effect, success gives a total reward of 0, while a failure might be something like $-C_{fail} - (\text{trajectory length})$.). In summary, ROVE’s data pipeline records exactly what happens during rollouts and interventions, along with a labeling of success/failure, and an understanding of which portion is “true” corrective behavior. All this data (with visual observations, proprioceptive state, and actions) is stored in a buffer for later training. The pipeline runs iteratively: after collecting some interventions, the authors train/finetune their models, then go back to the environment with the improved policy, and collect more data as needed. We’ll discuss the training next. Optimistic Value Estimation (OVE) for Mixed-Quality Data. Once episodes of mixed-quality data are collected, ROVE needs to update its value function (critic) in a way that learns which states are truly good or bad. A big challenge is that the dataset contains a variety of trajectories: some originated from the robot’s original policy (often failing!), some from human teleoperation (with shy adaptation segments), and success vs failure. If we simply fit an ordinary value function to all these trajectories, we would end up averaging over suboptimal behavior, yielding a conservative estimate of value. For example, suppose in one state the robot sometimes completely fails but sometimes the human recovers and succeeds – an ordinary Monte Carlo target would give an average that underestimates the possibility of success. ROVE’s solution is Optimistic Value Estimation (OVE). In implementation, this means training the critic with a modified regression objective that focuses on higher-return outcomes. Concretely, they start by bootstrapping the value in a multi-step fashion. Given a state $s_t$, they run for $H$ steps and compute the partial return: $$ \hat{V}t = r_t + \gamma r{t+1} + \dots + \gamma^{H-1}r_{t+H-1} + \gamma^H V(,s_{t+H},) $$ using some existing target network $V(s)$ for the endpoint. This gives an $H$-step temporal difference (TD) target. Now instead of minimizing the squared error to this target in the usual way, they use an expectile regression loss. That means they weight positive and negative errors differently: they introduce an expectile parameter $\tau\in [0.5,1)$ (e.g. 0.7). In effect, if the TD target $\hat{V}_t$ is higher than the current value estimate $V(s_t)$, that error is up-weighted (by a factor related to $\tau$). If the target is lower, the weight is smaller. Mathematically, it’s like doing a weighted L2 where the weight is $\tau$ if $\hat V > V$ and $(1-\tau)$ if $\hat V < V$ (for instance). What does this achieve? When $\tau$ is set above 0.5, the critic learning essentially biases toward the higher-return trajectories. Poor suboptimal parts of the data (where $\hat V_t$ is low) get less weight in the loss, and states leading to success (where $\hat V_t$ is high) pull the value up. As the authors describe it, as $\tau\to1$, the critic starts to estimate something like the optimistic in-distribution maximum of returns rather than the mean. In effect this makes the critic track “best-case” progress. A further important detail: ROVE trains a state-value function $V(s)$, not a Q-function $Q(s,a)$. Why? Because the data comes from heterogeneous sources: some transitions have human actions in them, some robot actions, and even human videos have no robot actions at all. A unified $Q(s,a)$ across these mismatched action spaces would be very brittle (it would need to pretend like human arm joint angles are the same as robot actuators, etc.). By training only $V(s)$, they sidestep that issue. The value regressor uses only states (images, proprioception, etc.) to predict eventual success. During computing the advantage (next section), they still need to consider which action was taken, but they do that on the trajectory segments. Finally, ROVE also pretrains the critic on large offline data before fine-tuning. In practice, they first train $V(s)$ on the big VLA dataset (robot demonstrations and egocentric human videos) with a standard Monte-Carlo return objective. This gives the value network a rough sense of task progress. Then they fine-tune it on the mixed rollout+intervention data using the OVE loss described above. In experiments, this two-stage training produced a significantly sharper, more discriminative value function than a plain Monte Carlo average. The optimistic critic learned, for example, to sharply drop its value estimate when success was unlikely and to jump when a successful recovery was underway (see “Value Estimation” analysis in the paper). To summarize, OVE ensures that ROVE’s critic pays more attention to the good parts of the human-corrected trajectories. States leading to full recoveries get high value, whereas states sitting in the middle of a failed attempt do not drag the value down too much. This biased value signal is what allows ROVE to focus learning on the high-value behaviors. Advantage-Conditioned Policy Improvement. With a biased-but-informative value function in hand, ROVE turns to updating the policy. Recall the policy (“actor”) is a VLA model that maps the current visual/proprioceptive state and the task instruction into a sequence of future actions. ROVE treats this actor as a generative policy that can be trained by behavior cloning (supervised regression) on logged data. Normally one might just clone the intervention episodes directly, but that would inherit all the noise. Instead, ROVE does advantage-weighted cloning in a somewhat novel way. Here’s how it works. Consider again a transition segment starting at state $s_t$ and comprising $H$ consecutive robot actions (or human actions during intervention) $a_{t:t+H-1}$. Given the current critic $V_{\phi}(s)$, ROVE computes the $H$-step advantage of that action-segment: $$ A(s_t, a_{t:t+H-1}) = \sum_{i=0}^{H-1} \gamma^i r_{t+i} + \gamma^H V(s_{t+H}) - V(s_t). $$ Intuitively, $A>0$ means “this sequence of actions led to better-than-expected outcomes.” ROVE then marks each segment with a binary label $I=1$ if $A$ exceeds some threshold, and $I=0$ otherwise. In practice, they take the threshold to be something like the 70th percentile of advantages over the dataset, so only the top-performing chunks get $I=1$. The policy is then updated by conditional imitation learning. The VLA actor takes as input each state $s_t$ along with the binary indicator $I$ and is trained to reproduce the corresponding action-chunk. So it’s like behavior cloning, but the policy is explicitly told “this action was good” or “not so good.” In effect, $I$ tells the model whether it should try harder to match those actions. Over time, the actor learns to differentiate: if $I=1$, it really should mimic the human action, but if $I=0$ it can treat them as “negative” examples. At inference time, ROVE uses a trick borrowed from generative modeling called classifier-free guidance. The idea is that the policy defines two conditional distributions: one conditioned on $I=1$ and one on $I=0$. To bias the output toward beneficial actions, ROVE combines these distributions (for example, by adding the logits) in favor of the $I=1$ branch. Concretely, they might run the VLA with $I=1$ and with $I=0$, and then amplify the difference to steer it toward high-advantage actions. The end result is a policy that preferentially generates actions that the critic expects to yield better outcomes. In practice, this advantage-conditioning means the actor’s training objective is something like: $$ \max_\theta E_{(s_t,a_{t:t+H-1},I) \sim D} ;\big[,\log \pi_\theta(a_{t:t+H-1} \mid s_t, I)\big]. $$ They apply this loss only on segments where $I=1$ (or weight $I=0$ examples weakly), effectively focusing the gradient on good data. Importantly, they do not do any on-policy RL or policy gradient updates. It is still supervised learning, but targeted by advantage. This avoids problems of variance or destructive interference that pure RL could bring, especially when using a large transformer policy. (It’s reminiscent of Advantage-Weighted Regression or AWAC, but implemented via conditioning rather than a weighted loss.). Finally, the authors mention some practical regularization: they randomly drop out parts of the proprioceptive input and add noise during policy training. This prevents the model from “cheating” by latching onto minute joint angles (which might be brittle) and forces it to rely on broader cues like the visual context. These details help keep the policy robust to the noise inherent in real-world humanoid operation. To summarize, ROVE’s policy update is a goal-directed form of imitation: it asks the model to emulate the human actions only when they appear to have generated positive advantage. This is crucial – it prevents the robot from learning the hesitations or mistakes that occurred during teleoperation, and instead distills only the corrective behavior that actually fixed the task. Experiments: Real Humanoid Tasks. ROVE was tested on real, contact-rich humanoid manipulation tasks that would be very hard to solve without intervention. The two main tasks reported are Erase the Whiteboard and Put Bread in Toaster. In the whiteboard task, the robot must pick up a marker in its hand and wipe it off a wall-mounted board – a sequence of whole-body movements with heavy contact forces. In the bread task, the robot grasps a loaf of bread, then carries it and places it into a toaster – requiring precise fine-hand control and careful placement. In both cases, the robot had an initial VLA policy (pretrained on some offline data) that could attempt the task autonomously, but typically failed if left alone. Using ROVE, the team collected a few dozen intervention episodes. On each episode, the robot tried on its own, then the human intervened to recover. This gave mixed data where many rollouts failed and a few were rescued by human correction. Training ROVE’s critic and actor on this data, then deploying the improved policy, was done iteratively. The gains were dramatic. Initially, the base VLA policy had very low success: around 30–40%. After running ROVE for just a couple of rollout–intervention iterations, success rates skyrocketed to 80–90%. For example, on the whiteboard task, final success reached 80% (up from around 45% initially), and on putting bread into the toaster it rose to 87% (from ~57%). Figure 4 of the paper (see) charts these improvements clearly: each web-cycle of executing ROVE’s updated policy led to more successful completions. Crucially, ROVE was compared against several baselines:. Supervised Fine-Tuning (SFT): Simply taking the original VLA model and further training it on all the intervention clips via standard behavior cloning. HG-DAgger: A variant of DAgger that directly labels robot states with the human corrective action. Filtered BC: A baseline that tries to filter out bad portions of demonstrations and only clone the “good” parts (some heuristic). RECAP: (Another interactive RL approach, details in the paper, likely an offline-to-online RL method.). ROVE outperformed all of these. In particular, SFT and HG-DAgger performed poorly on these tasks, which underlines the central insight: the raw intervention data is too noisy for naive imitation. ROVE’s advantage-conditioning was the key to distilling the useful signal. (The authors show quantitatively that HG-DAgger, which blindly copies human actions, did worse than ROVE by a large margin.). In sum, these experiments demonstrate that ROVE’s combination of careful data collection and algorithmic filtering can nearly double success rates on difficult humanoid tasks, using only moderate human data and without needing additional reward shaping. The improvement was consistent across both tasks and across multiple iterations, indicating that ROVE’s closed-loop training is robust. Discussion and Context. ROVE offers several takeaways for the field of embodied AI and robotics:. Bridging offline policies with online correction. It exemplifies how you can take a pretrained VLA model (“foundation policy”) and post-train it with interactions. This is akin to reinforcement learning from human feedback but in a continuous-control setting. Instead of text-based preferences, the human here intervenes via actions and the RL leverages them. This approach nicely complements large pretrained policies by letting them correct their own failures in the real world. Learning from imperfect teachers. ROVE is another instance of an approach that exploits suboptimal demonstrations in a smart way. It has conceptual similarities to algorithms like AWAC or advantage-weighted regression, which reweight offline data by advantage. The key novelty is applying an expectile-style training for the critic and a thresholding strategy for the actor. This could inspire similar techniques in other domains (e.g. autonomous driving or manipulation with imperfect demos). Use of human video data. Incorporating cross-embodiment videos is an emerging theme. We’ve seen projects like EgoMimic and Vimeo’s advances that use large human video datasets to teach robots. ROVE uses them differently: it uses egocentric human videos (presumably of similar tasks) to further supervise the value function. This makes the critic more robust in “long-tailed” scenarios like near-misses and recoveries. This is reminiscent of a recent work ViVa (Video-based Value Models) that also uses video to predict outcomes. It’s a promising idea: humans naturally have recorded all sorts of video of similar tasks, and leveraging that to tell the robot what states look like success vs failure can fill in gaps that scarce robot data does not cover. Teleoperation and VR for real deployment. Practically, ROVE also shows the power of modern VR teleoperation setups (as also seen in NVIDIA’s Assistive Tele-Op and Isaac ROS projects). The combination of motion capture, inverse kinematics, and first-person video allows humans to effectively “dive into” a humanoid robot even in contact-rich tasks. ROVE’s explicit modeling of the teleoperation gap is a valuable lesson: robots should not simply copy all observed human moves, because the human may be calibrating their own control. Looking forward, ROVE suggests several avenues of research. One might explore different forms of optimistic updates. The expectile regression trick works well here, but could we integrate it with more powerful RL algorithms (like an off-policy actor-critic) or use it in model-based settings? One could also investigate richer conditioning on the policy. For example, in ROVE the label $I$ is binary; but perhaps a continuous measure of advantage could be incorporated to weight actions more finely (somewhat like AWAC’s exponential weighting). Another thought is to combine ROVE with even larger VLA backbones: as these language-vision policies grow stronger, ROVE-like post-training could keep them reliable in the half-sim, half-real world. Finally, on the systems side, ROVE underscores that systems enabling easy data collection are crucial. NASA robots, factory humanoids, or home assistants could all benefit from a loop where occasional human guidance translates to automated improvement. In the long run, this is part of a vision of lifelong learning robots that continually adapt with minimal human effort – using interventions only when needed, but learning from them efficiently. ROVE is a step in that direction for one of the hardest cases: a full-body, dexterous humanoid. In summary, ROVE shows how to turn our human intuition about corrections into a principled learning algorithm. By being optimistic about what worked and skipping what didn’t, it unlocks the value of human interventions that would otherwise be wasted. As models like VLA usher in a new era of generalist robots, frameworks like ROVE will be key to making them safe, robust, and genuinely helpful in the real world.