Latest / Embodied AI 101 / EXPO-FT: Sample-Efficient Reinforcement Learning Fine-Tuning for Vision-Language-Action Models

Transcript

EXPO-FT: Sample-Efficient Reinforcement Learning Fine-Tuning for Vision-Language-Action Models · Embodied AI 101

• EXPO-FT: Sample-Efficient Reinforcement Learning Fine-Tuning for Vision-Language-Action Models. Modern robotics increasingly leverages vision-language-action (VLA) models – large neural policies that take as input a camera image plus a language instruction and output robot motions. These VLAs (for example, the OpenVLA model of Kim et al. 2025) can handle a wide variety of tasks out of the box because they’ve been pretrained on diverse data. In principle, a single VLA can generalize to many instructions: “Pick up the cube” vs “stack the blocks” might both be handled by conditioning on different language prompts. In practice, however, “zero-shot” VLA execution is often not perfectly reliable. The Stanford EXPO-FT paper stresses this shortcoming up front: “Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment”. In other words, if you deploy a VLA on a real robot right now, it will get a lot of tasks mostly right but occasionally fail in unpredictable ways. That level of reliability is usually not acceptable for real systems. A natural solution is to fine-tune the VLA on the target task. Instead of relying purely on the broad knowledge in the pretrained model, we collect some new data (or demonstrations) and adapt the policy to be precise on the task at hand. But here another problem arises: collecting data on a real robot is expensive and slow. How can we fine-tune efficiently so that the robot needs only a few minutes of trials instead of hours or days? Standard RL-from-scratch could solve each task eventually, but would typically require orders of magnitude more interaction than we want. Even naive fine-tuning of the large VLA network (for example with PPO or imitation learning) can be unstable or require lots of exploration. Prior work has begun to tackle this trade-off: for example, Kim et al. (2025) proposed a carefully designed “Optimized Fine-Tuning” recipe that uses action-chunking and other tricks to boost success rates up to 97.1% on benchmark tasks. That work highlights that choice of decoding scheme, action representation, and objectives matters a lot. Still, reaching 100% reliability on hard real tasks with minimal data remained elusive. This is the gap that EXPO-FT closes. EXPO-FT is a finetuning system that starts from a pretrained VLA and performs a small amount of online reinforcement learning to make the policy robust and precise. It incorporates several key ideas (action chunking, human-in-the-loop corrections, and most importantly an “on-the-fly” Q-guided policy update) that allow it to learn from on the order of tens of minutes of real robot experience. Indeed, on eight very hard manipulation tasks – things like routing a string of lights and plugging it in, striking a pool ball, flipping a raw egg on a spatula, etc. – the EXPO-FT policies achieved 30/30 successful trials on each task. In other words, after training they never failed any of the 30 evaluation trials on any of the eight tasks. And to do that, each task required on average only about 19.1 minutes of robot data. The Stanford team reports that this outperforms both vanilla RL-from-scratch and other VLA fine-tuning methods. They have even released (or pledged to release) an open-source codebase so that others can apply these ideas. Below we unpack how EXPO-FT works, why it’s effective, and what the experiments show. Vision-Language-Action Models and the Fine-Tuning Challenge. To set the stage, let’s briefly review what a VLA model is and why we’d want to fine-tune it. A VLA model is essentially a large neural network (often a transformer-based “policy transformer” or a diffusion policy) that is trained on hindsight data or demonstrations: each training example consists of an image of the robot’s workspace, a language instruction (like “pick up the red cube”), and a sequence of actions that solves the task. After pretraining on many tasks and environments, the model learns a mapping from (image, language) → actions. This knowledge lets it generalize: for a new task or scene, one hopes it can still spit out useful actions. In recent years, multiple groups have built VLAs. For instance, OpenVLA (Kim et al., 2025) showed that a transformer policy pretrained on diverse robot trajectories can handle new tasks in simulation and real robot (with fine-tuning). True, these models are powerful, but the Stanford paper reminds us that “strong generalization” does not mean perfect reliability. Even the best pretrained VLA might drop an egg too hard, miss a socket by an inch, or slightly flick the cue stick in the wrong direction. In other words, small errors still happen. When a policy has so many moving parts (large vision nets and long action chains), it’s surprisingly brittle to sensor noise, perturbations, or just new initial conditions. The result is that success rates on real tasks might be decent but not 100%. This is why fine-tuning is attractive. Instead of taking a pretrained policy “as is”, we use additional learning to specialize it to the task. One way is imitation learning: have a human teleoperate a few examples, then fit the VLA to that data. Another is reinforcement learning: let the robot try the task and optimize a reward signal. Each has pros and cons. Imitation can quickly reach high performance but suffers from covariate shift (the new scenario might differ from training). Pure RL can in principle reach higher robustness, but naively doing online RL (starting from scratch) is very costly in samples, and even starting from the pretrained VLA often remains unstable because the value gradients must propagate through a giant network. Indeed, previous work has studied how to best adapt a VLA. Kim et al. (2025) systematically evaluated many choices for fine-tuning OpenVLA. They found that certain design decisions make a big difference. Notably, action chunking turned out to be crucial: rather than having the model output one small control step at a time, they let it output chunks of actions (e.g. 5 or 10 steps) in one go. This reduces the action sequence length and makes control more efficient. Also, representing actions in a continuous vector space and using a simple L1 loss instead of a complex L2 or cross-entropy loss helped training stability. Putting these together into an “Optimized Fine-Tuning” (OFT) recipe, they dramatically improved performance – for example, OpenVLA’s success rate jumped from ~76.5% to ~97.1% on a multi-task benchmark, and action generation speed increased ~26×. These results show that using large pretrained VLAs plus clever training design can yield very strong polices. EXPO-FT carries forward that spirit but pushes even further. It’s still fine-tuning a pretrained VLA, but it uses an online reinforcement learning loop that is carefully engineered to be sample-efficient and reliable. The key innovation is borrowed from a related algorithm called Expressive Policy Optimization (EXPO) (Dong et al., 2025). EXPO was introduced not specifically for VLAs, but for any “expressive” policy class (like diffusion or transformer policies) that is hard to optimize directly. EXPO-FT takes that core idea and adapts it to the real-robot VLA scenario, adding vision inputs, chunked outputs, and a human-correction safety net. Before diving into EXPO-FT, let’s explain the original EXPO method and why it matters. Expressive Policy Optimization (EXPO): Refining Expressive Policies. Imagine you have a really powerful policy model – say, a huge transformer or a diffusion network – that can in principle encode complex behaviors. You pretrain this model on a lot of data (maybe via offline RL or imitation). Now you want to fine-tune it further using online RL and a reward function. A naive approach is to plug it into PPO or SAC: take the expressive network, compute gradients from the reward, and update it. But Dong et al. (2025) show that this can be unstable. The reason is subtle: these expressive policies often generate actions through a long chain of computation (like the unrolled steps of a diffusion model). A slight parameter change can vastly alter the output sequence, and the gradient from the reward ends up being very noisy. In short, directly doing gradient ascent over such a large network can fail to increase value. EXPO’s trick is conceptually elegant: don’t try to put the whole expressive policy on the hook for value gradients. Instead, use it as a powerful generator of proposals, and let a separate smaller policy chase the reward. Concretely, EXPO uses two policies and a Q-function:. A base policy $\pi_{\text{base}}$ (the large expressive one, e.g. the pretrained transformer). We initially train or keep $\pi_{\text{base}}$ using imitation from offline data, so that it is already good at producing reasonable behavior (for example, from demonstrations). The key is we treat this base policy as a fixed (or slowly updated) reference. An edit policy $\pi_{\text{edit}}$, which is a simple Gaussian policy with few parameters. It takes as input (state, base action) and outputs a small adjustment (delta) to the base action. A learned Q-function $Q(s,a)$ that estimates the expected future reward for state-action pairs. This is trained via standard temporal-difference learning using data gathered from the environment. The online RL loop works like this (see Figure 2 in their paper or project site): At each state $s$, we first sample an action $a_0 \sim \pi_{\text{base}}(\cdot\mid s)$. This is our “base” action (which could be a whole chunk of robot commands). Then we let $\pi_{\text{edit}}$ propose a corrected action: $a_1 = a_0 + \Delta a$, where $\Delta a \sim \pi_{\text{edit}}(\cdot\mid s, a_0)$. So now we have two candidate actions, $a_0$ and $a_1$. We feed them into the Q-function to get $Q(s,a_0)$ and $Q(s,a_1)$. The policy – effectively an on-the-fly controller – picks the one with higher Q-value, call it $a^$. We execute $a^$ on the robot (or simulator) and observe the reward and next state. Meanwhile, this experience $(s,a^*,r,s')$ goes into the replay buffer, and we use standard Q-learning updates to train $Q$. Meanwhile, how do we train $\pi_{\text{edit}}$? We can treat $\pi_{\text{edit}}$ as the policy of a small actor-critic: it wants to output the correction that maximizes $Q$. So for each pair $(s,a_0)$ we treat the process of selecting $a_1 = a_0 + \text{edit}(s,a_0)$ with high $Q$ as the objective. In practice, one can use an actor-critic update on $\pi_{\text{edit}}$ to maximize the Q-value. Crucially, $\pi_{\text{base}}$ (the big VLA) is not directly optimized by policy gradients. It is fixed or updated only via imitation (keeping its original knowledge). The only parameters being tuned by RL are those of $\pi_{\text{edit}}$ (and of the value network). What does this accomplish? This architecture effectively implements a local search in action space at each time step. The base policy proposes a reasonable action (leveraging all the prior knowledge). The edit policy then tries small perturbations of that action (within a Gaussian around it) to see if any yield higher value. If so, it moves to the better one. By using the Q-function to pick the higher-value choice, the system is guaranteed (at least greedily) to not make the action worse than what the base policy gave. Yet it still has flexibility to improve upon it. As the EXPO paper reports, this yields substantial gains in sample efficiency. In their experiments, using this “base + edit” scheme gave 2–3× speedups over naively fine-tuning the expressive policy with RL. The intuition is that instead of randomly perturbing the expressive policy’s parameters, you’re doing a much safer and smaller optimization in the action space at every step. The base policy remains a strong prior anchor, and only small Gaussian corrections are examined. You avoid catastrophic excursions that often kill vanilla RL on big networks. In short, Expressive Policy Optimization (EXPO) says: maximize reward by searching for the best action among batches of proposed actions, rather than by directly taking gradient steps in parameter space. The authors even call $\pi_{\text{edit}}$ + the Q-selection an “on-the-fly RL policy” because it’s only defining the actions per time step, not permanently changing the base policy. The result is a stable, sample-efficient fine-tuning method for policies that would otherwise be too expressive to handle directly. EXPO-FT: Applying EXPO to Vision-Language-Action Finetuning. EXPO-FT adapts the above framework to the specific case of finetuning a pretrained VLA for real-world tasks. Conceptually, it follows the EXPO recipe almost one-to-one, with a few important additions:. Base VLA policy. The base policy $\pi_{\text{base}}$ is a pretrained vision-language-action model. In the experiments, this could be something like OpenVLA or another transformer-based policy that was trained beforehand on diverse tasks. At the start of EXPO-FT, the base policy can already do the tasks moderately well (it “knows” how to do basic cable routing, grasping, orientation, etc., from pretraining). Action chunking. The VLA outputs are still high-dimensional robot commands. To speed up learning and execution, EXPO-FT uses action chunks: the base policy emits a sequence of several control steps in one shot. For example, instead of outputting a single joint velocity or end-effector delta each time, the network outputs a chunk of 5–10 such steps end-to-end. This means time accelerates in leaps of multiple steps. Action chunking has two benefits: it reduces the effective horizon of the RL problem (fewer decisions to sequence), and it lets the edit policy make more meaningful adjustments. As Kim et al. (2025) showed, such chunking (together with other design choices) can greatly improve inference efficiency and success rates. Image-only observations. The state $s$ fed into EXPO-FT comes from camera images and language. Concretely, $s$ could be a stack of recent RGB frames from a fixed camera plus a one-hot text instruction. No privileged state is used, so the method is broadly applicable. This means the Q-function $Q(s,a)$ is a vision-based critic, trained to predict rewards from raw images and candidate actions. All policy evaluations (base or edited) use just these images of the scene. Guess-and-refine with Q. Following vanilla EXPO, at each step EXPO-FT does:. Sample an action chunk $a_0$ from the VLA base policy given state $s$. Pass $(s,a_0)$ to the Gaussian edit actor, which proposes a residual $\Delta a$, forming $a_1 = a_0 + \Delta a$. (This might, for instance, rotate the planned gripper motion slightly, or adjust the force slightly.). Compute $Q(s,a_0)$ and $Q(s,a_1)$ with the critic. Pick the action with higher Q-value (say it’s $a^$). Execute $a^$ on the robot. Store $(s,a^*,r,s')$ in the replay buffer (where $r$ is the observed reward for that step or chunk) and train $Q$ on this data via TD updates. Also train the edit policy (and possibly Q) using standard off-policy RL: e.g. treat it like Soft Actor-Critic or DDPG – the edit actor’s objective is to choose $\Delta a$ that maximizes $Q(s,a_0+\Delta a)$. By doing this, the robot always takes the “better of” the raw VLA action and the post-edited action. Over time, $\pi_{\text{edit}}$ learns to make helpful tweaks to actions that lead to higher reward, and the Q-function learns to discriminate which actions are best. Crucially, the large base policy parameters stay mostly fixed (or are updated only via imitation to keep them from drifting), so the heavy part of the model never “goes crazy” under an RL update. Human-in-the-loop (DAgger). EXPO-FT further incorporates occasional expert overrides. If during data collection the robot is about to fail spectacularly, a human supervisor can step in, correct the action, and this corrected action-chunk is added to the replay buffer as well. In effect, this is like a DAgger-ish safety net: it prevents the robot from sampling dangerous states too often and provides more on-target training data. The EXPO-FT page briefly mentions this feature (“Operators can override actions in failure-prone states; these corrections enter the replay buffer to accelerate exploration”). While this is not quantitatively the main novelty (many deep RL works use such tricks), in practice it helps saturate success quickly. Taken together, these components make EXPO-FT both stable and efficient. The template is: start from something good (the VLA), and carefully nudge it toward better behavior with a small model that is easy to train. The use of on-the-fly Q-based selection exactly follows the EXPO idea, while the chunking and DAgger ensure that each real-world trial is as informative as possible. On the project website, the authors summarize: “EXPO-FT finetunes a pretrained Vision-Language-Action (VLA) policy with online reinforcement learning to a highly reliable performance with only a small amount of real-world interaction.” They list the main ingredients as (1) an edit actor for residual corrections, (2) Q-guided sampling among multiple action proposals, and (3) a human-in-the-loop override system. In short, the base VLA suggests a behavior, the edit policy refines it, and the Q-function picks the best – a framework that empirically closes the gap between zero-shot VLA and practical task success. Experimental Tasks and Setup. To validate EXPO-FT, the authors designed a set of eight challenging manipulation tasks on a real robot. All tasks involve the robot interacting with objects on a table and using visual feedback from cameras, under a natural language instruction. The tasks are:. String Light Routing (Route 1 & 2) and Plug Insertion. In this task (shown in the paper and code videos), the robot must pick up a flexible string of lights, route it along two different paths (one on each side of a partition or obstacle), and then insert the power plug into a socket to turn the lights on. This requires handling a deformable object (the string), planning a path around obstacles, and then performing a precise insertion. Pool Shot. The robot holds a cue stick and strikes a cue ball so that it lands in a designated pocket. This is a quasi-dynamic task: the cue impact yields a moving ball that must go through a hole. It demands forceful control and accuracy. Flower Insert. The robot must hold a delicate artificial flower (by its stem) and insert it into a narrow wine bottle neck without damaging it. This tests precision and contact dynamics with a fragile object. Egg Flip. The robot uses a spatula to flip a raw egg in a bowl or pan. The goal is to turn the egg over (like a pancake flip) without dropping it. This is a classic dexterity challenge often used in robot learning. Candy Scoop. The robot must scoop up small “candy” pieces (like small balls or beads) from a container. This involves fine manipulation and compliance (if the candy can move or slip). Cube Pick. A simpler task: pick up a small cube from the table. This ensures at least one straightforward pick-and-place case. Thus, the suite covers a range of object types (rigid, deformable, granular), speeds (fast egg flip vs slow insertion), and skills (grasping, scooping, striking). Each one is instructed by a natural language command (e.g. “Flip the egg over” or “Pick up the cube and place it in the bin”, etc., though the exact phrasing isn’t the focus). The vision input is from camera(s) mounted above/beside the workspace, and the actions are low-level arm motions (e.g. end-effector velocities) wrapped into tokens by the VLA. Importantly, the initial conditions are heavily randomized. The authors illustrate “initial state randomization” diagrams (with orange shaded regions) for each task, indicating that the objects start in a wide range of positions and orientations. This means the learned policy cannot just memorize one scene; it must truly interpret the image and generalize. For example, the plug might be on the left side of the workbench one time and on the right another time, or the candy pieces might be in different corners of the bowl. Such randomization is a common technique to improve robustness. In each training run, EXPO-FT collects episodes on the real robot. An episode might be one attempt of the task (e.g. the robot tries to route the string and plug it in). If it succeeds (lights up the string), it’s a success, and the episode ends (perhaps resetting for the next trial). If it fails (drops the string, misses the plug, etc.), the human can potentially intervene (per the DAgger method) to prevent damage, and then the scene is reset. The reward signal is binary (success vs fail, or a similar sparse reward given only at the end of the chunk sequence), so most of the learning signal comes from whether each attempted trial succeeded or end states in the buffer. The Q-function is trained with these sparse rewards using temporal-difference updates on rollouts of the chunked policy. All told, each task was trained independently. EXPO-FT used the pretrained VLA corresponding to that task and ran its finetuning loop. In total, the reported average “online robot time” per task was about 19.1 minutes. Given that each evaluation is 30 trials, this implies on the order of 30–60 episodes (roughly 0.5–1 minute per episode including resets) to reach perfect success. This is very low data for real robotics. Results: 30/30 Successes in Minutes of Data. The headline result is impressive: every evaluated task reached 100% success (30/30) after finetuning, with only ~19 minutes of data. To quote the paper’s abstract, “Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data”. We see exactly the 30 successes in one of the policy rollout videos on the project page, for each task. This means that, in the end, the finetuned policy never made any mistakes in the test trials for those tasks. For comparison, the authors note that this beats both “prior RL-from-scratch and VLA finetuning” methods. In concrete terms, a baseline policy trained from scratch via imitation or RL often fails at least a few trials (and typically needs much more data to learn), and even a straight fine-tuned VLA (e.g. turning on or near default policy gradients on the VLA) often has trouble converging reliably on all tasks, frequently plateauing below 100%. The fact that EXPO-FT hits perfect success on each task suggests it has overcome the remaining robustness gap. We should interpret this carefully: 30/30 on 8 tasks is very strong, but it is still a limited set of tasks. These are quite challenging setups, but in the real world you’d likely want a policy that can generalize to many tasks, not just the eight it was trained on. That said, the achievement here is to show what can be done when you specifically fine-tune for one task. It’s a proof of concept that near-perfect performance is achievable with small training cost. Also, note that “19.1 minutes” is an average; some tasks might have been easier (needing slightly less) and others slightly more. Even so, sub-20-minute training for a new skill is well within the realm of practical use in a lab or factory. Another important aspect is robustness of the final policies. The authors provide qualitative demonstrations (on video) showing the trained policies working under perturbations they hadn’t explicitly trained on. For example, the string-light router still succeeds if you nudge the string pole, or if the camera viewpoint is slightly changed, or if there’s a distracting object on the table. The pool-shot policy still pockets the ball if the cue start position is perturbed. These robustness tests suggest the policies did not just memorize a trajectory; they learned a strategy that generalizes over the randomized starts. This likely comes from both the wide initial randomization and the fact that the edit/Q loop kept pushing for high reward even in slightly off-nominal conditions. In terms of numbers, the key comparisons are: in simulation or smaller experiments, fine-tuning had achieved ~97% success, whereas here we see 100%. Meanwhile, a strong imitation-from-scratch baseline (Diffusion Policy or ACT, as in Kim et al.) might only reach, say, 80–85% even after extensive data. And a naive RL-from-scratch (no pretraining) might often fail entirely. The EXPO-FT results suggest that combining the VLA prior with smart online fine-tuning gets the best of both worlds: rapid learning and full success. Finally, the paper emphasizes open science. The authors state that they are releasing an open-source codebase for EXPO-FT. This is great news for the community: it means other researchers can take these methods (and the pretrained VLA) and try them on their own robots. It should help set a new standard for how VLA models are adapted: rather than hand-crafting policies by trial and error, one can use this standardized pipeline. Discussion and Future Directions. What lessons can we take from EXPO-FT, and where might this lead? First, on a broad level, this work is part of a trend of bridging large pretrained models and interactive learning. Autonomous robots need both – they need rich priors (the VLA) and cautious fine-tuning (the RL loop). EXPO-FT shows that when you combine them carefully, the result can be both sample-efficient and reliable. The “edit-actor plus Q” approach is a generally applicable trick: it does not depend heavily on the specifics of the tasks or robots. In principle, one could apply the same idea to navigation, flying drones, or any other domain where you have a good prior policy and a reward function. Second, the emphasis on chunking and continuous actions highlights that how the policy outputs actions matters a lot. By designing the outputs (chunks) properly, the learning process becomes easier. This principle has parallels in other fields: for example, RaCT (ReAct Transformers) in vision-language tasks use beam search over partial outputs. Here, sampling multiple action chunks and refining them is like doing a small beam search in action space each step. Third, the use of human corrections (DAgger) is a pragmatic recognition that real-world learning often needs supervision. It would be interesting to measure how much of the sample efficiency comes from these interventions versus from the autonomous Q-learning. The paper doesn’t quantify that breakdown, but future work might investigate it. In any case, this research shows that even with a handful of expert corrections, the policy can recover and quickly learn to avoid mistakes. Finally, there is the question of multi-task learning and generalization. EXPO-FT as presented focuses on one task at a time. A natural extension is: can we fine-tune one VLA to solve many tasks, using a similar data budget per task? That would approach the idea of a lifelong learning robotic system. Preliminary work on multi-task VLA (like Qwen-VLA or CogACT) is ongoing; maybe an “EXPO-FT multi-task” could adapt a universal VLA to a suite of tasks on the fly. Another direction is to incorporate more vision intelligence like active perception. The current EXPO-FT takes the image as given. One could imagine also learning where to look or how to move cameras to gather more helpful observations. Since EXPO-FT already uses a Q-function, an extension could be to include an attention or viewpoint selection action in the chunk. In summary, EXPO-FT is an exciting demonstration that we can “have our cake and eat it too” with VLAs: use their broad knowledge to get a good starting policy, and then use a bit of reinforcement learning – done the right way – to polish it to perfection. The result is a system that is both grounded in big-data pretraining and grounded in real-world interaction, achieving top-tier reliability with minimal data. For roboticists and ML researchers, it sets a new bar: it shows that tasks once considered very difficult (flipping a real egg, pocketing a real ball) can be learned with only a few minutes of practice if you combine the right tools. References: Dong et al. (2026). EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models. Kim et al. (2025). Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. These papers provide the background on VLA fine-tuning and the EXPO algorithm, respectively. (See also the EXPO-FT project page and code for visual demonstrations and implementation details.).