Latest / Embodied AI 101 / FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space
Transcript
- FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space. Modern robot control is increasingly driven by generative foundation models – deep policies that map random latent inputs to actions (often via diffusion or flow networks) conditioned on camera observations and goals. Examples include vision-language-action (VLA) transformers and “world-action” video-prediction policies. These large models (like the π0.5 VLA or NASA’s Cosmos video planner) have shown remarkable zero-shot capabilities across many tasks. But in practice they can fail when the robot encounters out-of-distribution situations not seen in training. Traditionally, fixing these failures would call for collecting lots of new data or running on-robot reinforcement learning—a slow, expensive, and potentially unsafe process. FlowDAgger offers a different path: it adapts a frozen generative policy using only a handful of human corrections, without ever tuning the base model’s weights. The key is to interact in the policy’s latent space. Whenever a human operator intervenes with a better action, FlowDAgger “inverts” that action back into the latent noise vector that would have produced it under the frozen model. Then it trains a small auxiliary “noise policy” on these inverted latents to gently steer the base model’s outputs. Because the foundation model itself stays fixed, this approach preserves its prior knowledge while rapidly acquiring the new skill. In experiments across simulated and real robot tasks, FlowDAgger outperforms both naive fine-tuning (behavior cloning) and prior latent-space RL methods, achieving robust success with only 5–20 correction episodes. Steering Generative Policies via Latent Corrections. Generative robot policies (e.g. diffusion policies or flow-matching controllers) produce actions by sampling a latent noise vector and running it through a learned dynamics model. Intuitively, if we could invert that generative process, we could map a desired action back to the latent input that yields it. FlowDAgger exploits this: when the human expert provides a corrective action a in state s, the method finds the latent noise w** such that the base policy πgp(s, w) ≈ a. In practice, πgp executes K small integration steps (like an ODE) to transform an initial noise w into the final action. FlowDAgger runs those steps in reverse, solving a sequence of implicit equations by fixed-point iteration. Concretely, it starts from the final action xK = a* and iterates backward:. For each step k from K–1 down to 0, solvexk = xk+1 – Δt·vθ(xk, tk, s)by iterating xk(m+1) = xk+1 – Δt·vθ(xk(m), tk, s) starting from xk(0) = xk+1. Here vθ(x, t, s) is the learned velocity field of the diffusion model. Because Δt is small and v is Lipschitz, just a few (e.g. 5) iterations suffice to converge. At the end we recover x0 = w*, the noise that the base policy would have needed to produce the expert action. This stepwise inversion is more reliable than a single backward Euler step. In practice, it costs K·M forward evaluations of the network (no gradients/backprop required), which is easily done online as corrections are collected. The result is a set of paired training examples (s, w) where s is the state and w is the latent noise corresponding to the expert’s action. This noise-space target w* then supervises a tiny neural “noise policy” Πw(s) that learns to predict the latent from state. In effect, the noise policy becomes a differentiable steering mechanism: at deployment it takes the current observation s and outputs a tweak to the latent that nudges the base policy’s action toward correcting previous mistakes. Importantly, the foundation model πgp remains frozen throughout. FlowDAgger never alters the weights of the pre-trained policy; it simply uses that model as a differentiable decoder from latents to actions. This preserves all the rich multi-task skills encoded in πgp, avoiding the “catastrophic forgetting” that can plague fine-tuning. In fact, weight-space fine-tuning or even low-rank updates often degrade unrelated capabilities, whereas FlowDAgger’s latent steering keeps non-overlapping skills intact. The overall loop is DAgger-like: deploy the current policy (base + latent steering), let the human intervene with actions a, invert them to w, add (s, w*) pairs to the dataset, and update Πw by regression. Unlike classic DAgger which retrains the action-output network, we retrain only the latent controller. This latent-space learning is extremely data-efficient – the authors report success with only tens of intervention episodes – because the heavy lifting remains in the powerful pre-trained model. Extending to Joint World-Action Models. A special case arises when the base policy is a world-action model (WAM) that jointly predicts future observations and actions (for example, a video diffusion model that outputs a whole trajectory). In those cases the model’s generative output is a multi-frame latent that contains both future state and action information. FlowDAgger handles this by inverting the joint process. Concretely (as shown for the Cosmos policy), one takes the model’s prediction for the future video and action frames, substitutes the expert’s corrected action into the corresponding frame of the “clean” latent, and then runs the reverse process on the modified latent. This recovers a joint noise w that will reproduce the expert’s action if the model were to decode from s. In other words, we still recover a noise vector even though the diffusion generates states and actions together. The key idea is that the expert’s correction is confined to the action slice of the output, so we only nudge that part of the latent and leave the rest on the manifold of the original roll-out. The resulting noise w is then used just as before to train the noise policy. The researchers verified that this joint inversion works, allowing FlowDAgger to adapt complex video-based planners without needing a separate action-head. Empirical Validation. The authors evaluated FlowDAgger both in simulation and on real robots, using a variety of models and tasks. In simulations they used MetaWorld benchmarks ( diverse pick-and-place and assembly tasks) with two types of pretrained policies: (1) π0.5, a state-of-the-art vision-language-action diffusion policy, and (2) Cosmos-Policy, a world-action video diffusion model. (They also report in appendices similar results for NVIDIA’s GR00T VLA model and a vanilla single-arm diffusion policy.) The training regime always gave FlowDAgger and all baselines the same overall interaction budget (50 on-policy rollouts, either with a human or RL sampling, as needed) and measured success at regular intervals. On these simulated tasks, FlowDAgger’s gains were dramatic. Figure 3 (from the paper) shows that with only a few corrective episodes, FlowDAgger quickly climbs to high success in tasks like Assembly, Box Closing, LeverPull, Hammer, etc., whereas action-space residual DAgger often oscillates and latent RL (DSRL) barely improves at all. Averaged over 12 tasks, the frozen π0.5 policy had a base success of 0.53. After adaptation, FlowDAgger reached a mean of 0.78 (a +0.25 gain), compared to only +0.18 for supervised fine-tuning (SFT), +0.15 for LoRA-DAgger, +0.11 for Residual-DAgger, and +0.02 for latent RL. FlowDAgger achieved the highest success on 8 of 12 tasks, significantly outpacing baseline methods with the same number of corrections. Notably, latent-space RL (DSRL) with the same rollout budget made almost no progress, underscoring that human corrections provide much richer guidance than sparse rewards in these challenging tasks. Equally important is that FlowDAgger vastly preserves the robot’s prior skills. In one study, the authors adapted the policy on a “Hammer” task and then re-evaluated 5 other tasks on which the base policy was already nearly perfect. FlowDAgger increased “Hammer” success from 0.40 to 0.84 while leaving those other tasks essentially unchanged (Door 1.00→0.88, Drawer 1.00→1.00, Faucet 0.96→0.96, Plate 0.88→1.00) – the total drop in held-out average was just 0.08. In contrast, weight-space updates (fine-tuning or LoRA) collapsed many of those tasks to near zero success. This confirms that steering in noise-space helps the adapted model retain its generalist abilities, whereas modifying the backbone parameters often overwrites them. The method’s generality was also tested across architectures. Applying the same action-inversion + noise-policy training to the world-action Cosmos model yielded similar benefits. On a shared set of MetaWorld tasks, FlowDAgger raised Cosmos-Policy performance by a comparable margin (e.g. DialTurn from 0.44→0.68, +0.24; BoxClose 0.36→0.64, +0.28) as it did for π0.5. (See Table 2 in the paper for full breakdown.) In short, FlowDAgger worked equally well on an action-head VLA and a joint-output video model, confirming that the latent-space adaptation concept is broadly applicable. Finally, FlowDAgger was tested on real robots (two bimanual setups using Franka FR3 arms and UR5e arms) on several challenging manipulation tasks. These included block pick-and-place, stacking glassware, and BusyBox bench tasks (pressing a button, sliding a slider, pulling a wire) on single arms, as well as the bimanual tasks Jenga stacking, toolbox packing, and plug insertion. Again, the frozen base policy often failed the task: for example, on Toolbox Packing the initial success rate was only 0.13. Using only 5–20 human-corrected episodes, FlowDAgger dramatically improved every task. In Toolbox Packing, just 10 intervention episodes lifted success from 13% to 80%. Glassware Stacking went from 0.26 to 0.76 with 5 episodes, Slider from 0.36 to 0.66 with 5, Cord Pull from 0.40 to 0.70 with 10, and so on (Table 4). In nearly every case, FlowDAgger outperformed a full fine-tuned baseline (SFT) under the same budget. The only exception was Jenga (where SFT had a slight edge against an already high base of 0.76). These real-world results show that FlowDAgger can turn a nearly unusable foundation policy into a reliable controller with only a handful of corrections, without ever changing the foundation model itself. The improvements were achieved within the limited deployment budget: for instance, Ladder tasks needed about 5–10 episodes (as summarized in the table) to reach high success. Practical Efficiency and Limitations. A key strength of FlowDAgger is its efficiency. Because we never backpropagate through the large base model and instead only train a tiny noise policy, adaptation is lightweight. The authors note that training the latent policy fits in the same ~8 GB of GPU memory that is required just to run π0.5. In other words, a single consumer-grade GPU can handle the entire adaptation process, whereas full fine-tuning or DAgger variants that update many weights typically need much more memory. Figure 4 in the paper plots success vs VRAM: FlowDAgger stands alone in the upper-left (high performance, low memory). In practice the only added computation is the per-step inversion (≈50 forward passes per intervention) and the small regression update – both easily done in real time. Of course, FlowDAgger has inherent limits. By design it can only adapt within the support of the pretrained model’s behavior. That is, if the human’s desired corrective action lies completely outside what the base policy can represent (its action manifold), inversion will find the closest representable action but cannot create entirely new capabilities. In such cases, one might still need additional data collection or partial fine-tuning. Also, as with any imitation-style method, the quality of adaptation depends on the human corrections: if the expert repeatedly provides biased or inconsistent demonstrations, the latent policy will simply reproduce those biases. Finally, inversion accuracy relies on the diffusion dynamics being well-behaved; highly multimodal or ill-conditioned models could make the latent recovery noisy. The authors discuss these limitations candidly, noting that FlowDAgger is best viewed as a lightweight steering mechanism that presumes the base model already “knows” something about the task domain. Conclusion. FlowDAgger introduces a pragmatic way to “steer” large pretrained robot policies using minimal human effort. Instead of editing a foundation model’s weights to fix errors, FlowDAgger asks: what latent input would cause this correct behavior? By mapping corrective actions into the latent space via reverse diffusion, then training a tiny controller on those latents, it yields rapid adaptation while preserving all of the model’s prior skills. Across both simulation and real hardware, with vision-language-action models and video-action models alike, this interaction-driven latent learning consistently outperforms vanilla fine-tuning or latent RL when only a few demonstrations are available. The result is a robotic adaptation loop that is safe (no risk of breaking the foundation model), fast (only tens of episodes), and lightweight (trainable on a commodity GPU). For practitioners, FlowDAgger offers a concrete path to improve “foundation” policies after deployment. If you have a pre-trained diffusion-based policy or VLA model that almost works in your environment but fails on certain cases, you can use FlowDAgger: have a human teleoperate or correct the policy for a few episodes, perform the action-inversion step for each correction, and learn a small latent controller. The base model never changes, but its behavior aligns with the expert guidance. In effect, we are treating human feedback as supervision on the latent decisions of the model, rather than on its raw outputs. As the authors conclude, this preserves model stability and reusability while still allowing for continual improvement through interaction. References: FlowDAgger – Murray et al., preprint 2026. (Also see code and demos at https://microsoft.github.io/FlowDAgger.).