Latest / Embodied AI 101 / Real-Time EXPO-FT: Slow VLA, Fast Corrections, Better Control

Transcript

Real-Time EXPO-FT: Slow VLA, Fast Corrections, Better Control · Embodied AI 101

• Real-Time EXPO-FT: Slow VLA, Fast Corrections, Better Control. Reinforcement Learning for Real-Time Vision-Language-Action Policies • Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn • Stanford University • arXiv preprint • 2026. The ball moves while the model thinks. Imagine a robot balancing a ball on a plate. Its camera captures the ball drifting left. A large vision-language-action model processes the image and generates a corrective motion. By the time that motion reaches the robot, the ball has changed direction. The action might have been sensible for the image. It might even resemble an excellent demonstration. But it is now an answer to the wrong moment. Real-Time EXPO-FT addresses that timing problem by splitting action production into two stages. A large VLA generates candidate action sequences in the background. Immediately before a new sequence is executed, a smaller, reward-trained system uses a newer observation to edit those candidates and choose among them. The proposal can be expensive; the final decision must be comparatively cheap. The underlying work is the September 16, 2026 preprint by Perry Dong, Kuo-Han Hung, Dorsa Sadigh, and Chelsea Finn. One attribution clarification matters: the paper lists Stanford University, while Physical Intelligence developed the π0.5 model used as its backbone. This is a study of post-training and executing that model, not a new π0.5 foundation-model release. The interesting contribution is not simply that reinforcement learning improves imitation learning. It is that the structure of the RL algorithm gives the authors somewhere useful to put fresh feedback—after slow action generation, rather than only before it. Nor should we interpret the results as proving that imitation learning cannot support dynamic behavior. Real-time chunking already demonstrated demanding manipulation with imitation-trained policies. The more defensible claim is that reward-driven improvement and execution-time correction work well together, exceeding the tested imitation and delay-aware RL alternatives. Three clocks, not one. π0.5 brings a substantial behavioral starting point. Its original training combines heterogeneous robot data with semantic and vision-language supervision to support generalization across environments. That is valuable here because online learning need not discover useful manipulation behavior from random motor commands. But broad behavioral competence and timely feedback are different properties. For this episode, keep three clocks separate. The first is the command clock: how frequently the robot receives a control command. The second is the decision clock: how frequently the learned system selects a new action chunk using fresh feedback. The third is the learning clock: when new experience produces parameter updates. The released real-time example uses 30-Hz control, an eight-action execution horizon, and episode-based updates. Those are three different rates, not three descriptions of the same loop. Action chunking helps with the first clock. Instead of computing one command and immediately needing another expensive forward pass, a policy predicts a sequence. The robot can execute that sequence while additional computation happens elsewhere. However, ordinary chunking does not solve discontinuities between independently generated sequences, and it does not make an old observation current. These are central motivations behind the original real-time chunking work. Training-time real-time chunking addresses continuity by teaching the policy to condition on actions already committed to execution. The model learns to generate a continuation consistent with what the robot will do while inference is running. This is stronger than merely starting another thread: the statistical problem seen during training reflects the execution arrangement used at deployment. My interpretation is that there are therefore two separate questions. Can the runtime keep supplying coherent commands? And can new evidence still change the next decision?. Consider a defender moving into the path of a robot’s soccer shot. A coherent continuation of an old plan may still target a newly blocked route. Predicting the defender’s likely motion helps, but an observation acquired after that prediction contains information the predictor could not have known. That is the opportunity Real-Time EXPO-FT exploits. It does not require every part of the policy to run at the fastest rate. It requires a useful decision-making component to remain downstream of the expensive computation. What EXPO contributes before we add real time. The foundation is Expressive Policy Optimization, or EXPO. Its starting point is an optimization problem: diffusion and flow policies can represent rich action distributions, but directly optimizing their parameters against a learned critic can require propagating gradients through an expensive generative process. EXPO instead trains the expressive base policy with an imitation objective and trains a small Gaussian edit policy to improve its action proposals. The editor receives a state and a proposed action, then predicts a bounded additive correction. The deployed decision rule considers both original and edited proposals and selects the candidate with the highest estimated value. Importantly, EXPO also uses this value-seeking candidate construction when forming temporal-difference targets, rather than treating it as an execution-only heuristic. A useful way to understand the division of labor is that proposal generation handles broad behavioral structure, while the editor handles local improvement. The critic does not have to manufacture a complete grasp trajectory from nothing. It evaluates plausible alternatives. Keeping the original candidates available also means that applying an edit is optional: the selection mechanism can reject a correction it considers worse. The released implementation explicitly tracks whether selected actions came from the original or edited candidate set. That last point is an optimization convenience, not a safety guarantee. A critic can misrank candidates, and a bounded correction can still be physically consequential. Nevertheless, it gives us a more controlled experimental object than asking a randomly initialized actor to replace a capable VLA wholesale. The earlier EXPO-FT extended this recipe to real VLA fine-tuning, including action-chunk critics and editors, image observations, and human corrective interventions. Its tasks included precise insertion, routing string lights, and a pool shot. Real-Time EXPO-FT inherits that basic improvement mechanism; its distinctive question is how the mechanism should be arranged when action generation itself consumes significant physical time. This also clarifies the word “asynchronous.” Actor–learner asynchrony means collecting experience while another process trains. Inference asynchrony means generating the next action chunk while the current one executes. They are independent choices. The public real-time example invokes the synchronous training driver with episode-based updates while still overlapping VLA inference with execution. Calling the method “async RL” without this distinction can give the wrong systems picture. Follow one action chunk through the system. Start with the continuity problem. During the next VLA forward pass, some commands from the existing plan will inevitably execute. Those commands are no longer hypothetical. Training-time RTC supplies them as a clean action prefix and asks the model to generate the remaining continuation. During supervised training, the prefix is kept uncorrupted and the prediction loss applies only to the remaining positions. Now add the runtime. The authors’ starts background inference when the number of commands remaining in the current plan equals the configured delay. The robot continues executing those commands. At the next replanning boundary, the runtime retrieves the generated candidates and invokes the fast selection phase using the current observation. Its single-slot overlapping path assumes that the configured delay fits within the execution horizon. For a concrete mental example, imagine executing eight commands per chunk and budgeting five command intervals for inference. Once three commands have executed, five remain, so generation of the replacement begins. The existing tail supplies the committed prefix. The newly generated continuation will become eligible when the old queue runs out. These numbers also match the scheduling pattern in the released example, although experiment-specific settings need to be checked separately. At that boundary, the important informational handoff occurs. The VLA proposals reflect the observation available when generation began. The editor and action critic instead receive the newer observation. The editor modifies the portion of each proposal that can actually execute next; the critic evaluates original and corrected alternatives under the current situation. This is explicitly the fast phase implemented by. The real-time launch configuration requests 32 base proposals and 32 edited candidates. The underlying configuration uses a ten-member critic ensemble with two members selected for conservative value estimation, along with three-layer, width-256 heads and separate image and proprioceptive representations. Candidate generation, corrective exploration, and value-based selection are therefore distinct components—not one monolithic π0.5 forward pass. There is an important frequency qualification here. Eight commands at 30 Hz span about 267 milliseconds. If a fresh visual decision is made only at each chunk boundary, its nominal cadence is about 3.75 decisions per second. The robot still receives commands at 30 Hz, but that does not establish a 30-Hz visual editor correcting every individual command. “Fast” describes the correction computation relative to VLA generation; feedback frequency also depends on the execution horizon. This follows directly from the documented chunk scheduler. The latency experiment also deserves careful wording: three real-world tasks add 100 milliseconds to approximately 67 milliseconds of native VLA inference, while dynamic picking retains the native latency. For reproduction, I would treat that as a specified latency condition, not evidence that the complete system has been validated on every edge-computing platform with similar nominal inference time. Camera buffering, transport delay, scheduling jitter, and missed deadlines would need their own measurements. How reward changes the policy. The learning story is easier to follow if we separate experience reuse, critic learning, and base-policy learning. First, this is an off-policy approach. Experience enters replay and can support multiple gradient updates. That matters when physical interaction is expensive. The broader RLPD line of work establishes why combining online learning with previously collected data can be effective without turning every new update into another demand for fresh robot interaction. Second, the critic reasons about an executed action window, not an isolated command detached from its neighbors. The implementation accumulates rewards over that window, respects episode termination, and discounts the bootstrap value across the execution horizon. If eight commands execute before the next decision, the target must represent that temporal separation rather than silently treating the chunk as one ordinary control timestep. The delay arrangement must also appear inside the backup. To construct a candidate for the next decision, the VLA should receive the earlier observation it would actually have had when inference started. The editor and critic then use the observation at the next execution boundary. The replay code explicitly retrieves both ordinary next observations and delay-shifted observations, together with the relevant action history. My reading is that this is one of the most transferable implementation lessons in the paper. A correct-looking Bellman update can still train the wrong controller if its target grants the slow model access to information unavailable at deployment. The runtime and replay sampler are part of the algorithm’s definition. Third, the base VLA continues learning. In the real-robot pipeline, successful demonstrations and successful online episodes can supply its supervised training batches, while the critic uses broader replay. The batch processor keeps these sampling roles separate. “RL fine-tuning” therefore does not mean that every parameter receives a critic gradient, and it does not mean that the proposal model remains entirely frozen. The π0.5 training wrapper implements prefix-conditioned flow matching: committed actions remain clean, noise is applied to the remaining action positions, and the loss is masked accordingly. It also supports randomized prefix lengths for supervised initialization. This exposes the model to different amounts of already-committed motion rather than training only on the no-delay case. Finally, the editor is trained to increase estimated value while retaining exploratory diversity through entropy regularization. Together, these choices create an improvement cycle: proposals support useful exploration; reward trains selection and correction; successful behavior supplies new supervised targets. Notice that failures still matter even when the base-model batch is success-filtered. They help teach the critic which states and action windows do not lead to the objective. The hidden compute budget. Physical sample efficiency is only half the resource story. An off-policy learner may reuse a transition many times, and each critic target can require generating another action. With a large generative policy, reproducing a broad candidate search inside every backup can become expensive. This motivates the connection to FASTER, another closely related paper from the same research group. FASTER learns to predict the downstream value of candidates before their full generative computation has finished. Its simplest version filters candidates directly in initial-noise space, avoiding the cost of completely decoding every option before deciding which ones deserve attention. Real-Time EXPO-FT implements a noise-space filter for backup construction. A lightweight value predictor scores a pool of Gaussian seeds; the chosen seed is decoded by the VLA, then the resulting action can be edited and evaluated. The filtering code can also condition on delayed observations, preserving the timing arrangement used by the slow actor. The distinction I would retain is between search during robot execution and search during critic training. The public configuration specifies those budgets separately. Reducing the number of VLA decodes required by a backup does not imply that rollout-time action selection has become a single-candidate policy. This is a useful engineering pattern beyond this particular method. Before deciding that an RL algorithm is intrinsically too expensive, inspect where its generative calls occur. Some calls produce actions that the robot will execute. Others exist only to estimate a training target. They may deserve very different approximations—and very different validation tests. Read the results with the denominators intact. The four real-world tasks probe different timing failures: balancing a ball on a plate, receiving an object from another moving arm, striking a soccer ball into a goal, and grasping an object on a rotating base. These are not interchangeable versions of static pick-and-place. Training was capped at ten minutes of online interaction, with earlier stopping when a perfect evaluation was reached. There were no corrective human interventions during online learning, but resets could still involve a person. Here are the key comparisons. Task entries are successful trials out of 30; the final row expresses their average as a percentage. SFT denotes supervised fine-tuning. | Task | SFT | SFT + RTC | EXPO-FT + RTC | Real-Time EXPO-FT | |---|---:|---:|---:|---:| | Dynamic picking | 19/30 | 22/30 | 24/30 | 30/30 | | Ball balancing | 8/30 | 12/30 | 23/30 | 28/30 | | Object passing | 10/30 | 22/30 | 27/30 | 30/30 | | Soccer kicking | 13/30 | 16/30 | 26/30 | 28/30 | | Mean success | 41.7% | 60.0% | 83.3% | 96.7% |. The popular “42% to 97%” summary compares ordinary SFT against the complete method. That is a useful system-level result, but it combines several improvements. Reading across the table gives a more informative progression: latency-aware imitation already helps substantially; latency-aware RL helps further; moving correction and selection to the execution-time observation produces another improvement. For me, the most relevant comparison is therefore EXPO-FT plus RTC against Real-Time EXPO-FT: about 83% versus 97%, or roughly thirteen percentage points. It asks a narrower question than “does RL beat cloning?” Both sides already incorporate RL and a real-time execution mechanism. Simulation provides a complementary test. Ten Kinetix tasks use state-based flow policies—not π0.5—with four training seeds. Mean success is 96.2% for Real-Time EXPO-FT versus 81.7% for EXPO-FT with RTC. Kinetix is a physics-based control benchmark rather than a language-conditioned manipulation benchmark. Consequently, I would read these experiments as evidence about delayed control and policy improvement, not as evidence of new language understanding or broad semantic transfer. The benchmark’s scope is useful precisely because it separates those questions. The baselines represent different design choices. It is worth resisting the temptation to arrange every comparator on a single ladder of sophistication. RLPD represents a data-reuse strategy: learn an online policy while drawing on offline experience. It can benefit from demonstrations without inheriting a pretrained VLA’s policy representation. That distinction matters. Access to useful trajectories is not the same resource as access to a model that already organizes a large behavioral distribution. My practical interpretation is that a lightweight actor remains an important baseline whenever the task is narrow and sensing is good. A large model should justify its cost through useful prior structure, not merely through parameter count. Conversely, a short online budget can favor methods that arrive with a competent policy rather than only a dataset. Diffusion Steering via Reinforcement Learning, or DSRL, makes another tradeoff. It keeps the generative policy fixed and learns which latent noise inputs produce desirable actions. That avoids directly optimizing the large generator and can be highly effective, but improvement is mediated through the behaviors reachable by that frozen generator. For this episode’s timing question, the key distinction is causal. Latent steering chooses how generation begins. Execution-time editing changes an action after generation has finished. If new information arrives during the expensive forward pass, those two mechanisms do not have the same opportunity to use it. The earlier EXPO-FT work also makes clear that residual-style correction alone is not the complete story: its framework combines correction with candidate selection and continued base-policy training. Real-Time EXPO-FT’s contribution is best understood as arranging those capabilities around physical time. I would not summarize it as merely “a smaller controller on top of a bigger one.” The observation available to each component, and when that component can alter an executable action, are central parts of the design. What the headline does not establish. First, ten minutes of online experience is not ten minutes from an unconfigured robot to a finished skill. The released workflow includes demonstration collection, normalization assets, supervised initialization, and subsequent online learning. Its offline training program is a separate stage with its own data loading, optimization, and checkpoints. For a lab deciding whether to adopt the system, I would account for robot interaction time, demonstration effort, reset labor, training compute, and evaluation time separately. Those quantities answer different operational questions. Sample efficiency is valuable even when wall-clock training is longer, but the two should not be conflated. Second, the fast networks receive task-specific sensing assistance. Ball balancing uses a temporal image stack and detector-derived ball state; soccer exposes keeper motion variables. RLPD and DSRL critics receive matching additional state dimensions. I read this as a sensible engineered learning system, not evidence that an untouched generalist VLA autonomously discovered every aspect of motion estimation. When transferring the approach, I would preserve that distinction in the experiment design: what comes from pretraining, what comes from task-specific perception, and what comes from reward-driven improvement?. Third, thirty evaluation trials cannot establish deployment-grade reliability. Under a simple independent-trial model, a policy with a true 90% success probability still completes thirty consecutive trials successfully about 4.2% of the time. That calculation does not estimate these robots’ true reliability; it illustrates why “30 out of 30” is evidence of progress, not proof of near-zero failure probability. I would want independent repetitions of the entire learning run, not only additional rollouts from one final checkpoint. I would also want evaluation conditions that distinguish familiar initial-state randomization from changes in objects, sensing, dynamics, and delays. Finally, bounded edits and conservative critic estimates should not be confused with formal control guarantees. For deployment, I would maintain independent action constraints and fault handling. I would also test what happens when the candidate set is uniformly poor, the critic is confidently wrong, or background inference misses its expected completion time. Those are different failure modes from ordinary task failure. How I would port it to another robot. The authors’ repository is the practical starting point. It contains both EXPO-FT and Real-Time EXPO-FT, uses separate client and learner environments, and depends on modified OpenPI and DROID checkouts. The README identifies the appropriate branches and provides extension interfaces for alternative algorithms and VLA backends. My first experiment would not involve reward learning. I would instrument the existing policy runtime: image acquisition, observation receipt, inference launch, candidate availability, correction completion, and command execution. I would plot distributions, not only averages. If the dominant delay is upstream camera buffering, a faster editor may still be correcting from an old image. Next, I would verify the action contract. Which coordinate frame describes the command? Is the gripper command a position, velocity, or discrete decision? Where do normalization and clipping occur? A configured edit scale should never be interpreted as a universal physical distance without tracing the transformations. The π0.5 wrapper and replay code are useful audit points because they connect model inputs and outputs to the training representation. I would then establish a prefix-conditioned imitation baseline before adding the editor. The supplied offline RTC training program offers a concrete route for that stage. This keeps two debugging problems separate: whether asynchronous continuation works correctly, and whether RL improves the resulting controller. One reproduction detail deserves attention. The public dynamic-pick configuration describes randomized cube drops, and its example launch script uses a five-step delay. Those files are useful examples, but task names and script defaults should not be assumed to constitute an experiment-exact benchmark manifest. Read the configuration, task setup, and checkpoint paths together before comparing numbers. Once the timing baseline is sound, I would add components incrementally. Compare fresh-observation selection without edits, editing with selection, and continued base-policy training. Separately remove any detector-derived state features. These ablations would tell me whether the benefit on my task comes mainly from better proposals, better ranking, better corrections, or better state estimation. During learning, I would log more than success. Useful diagnostics include correction magnitudes, the fraction of selected candidates that were edited, critic disagreement, missed scheduling windows, and how often successful trajectories enter the base-model training set. The implementation already exposes original-versus-edited selection statistics, which are a good starting point for understanding whether the editor is helping or merely remaining active. Finally, I would keep an evaluation channel independent of the reward detector. A system that becomes better at triggering a task-specific success rule is not necessarily becoming better at the intended physical task. For adoption, reward auditing and reset design deserve the same seriousness as optimizer tuning. The takeaway: decide where fresh information can still matter. In their broader essay, Towards Universal Post-Training for Robotics, Perry Dong and Chelsea Finn argue that a useful robotics post-training recipe needs more than an optimization algorithm. It also needs repeatable practices for rewards, resets, and feedback. That is the right operational context for this work: the algorithm matters, but so does the process that makes repeated learning attempts possible. My takeaway from Real-Time EXPO-FT is not that imitation learning should be discarded, or that a sufficiently small residual network makes latency disappear. The more useful lesson is architectural: expensive proposal generation and immediate feedback do not have to live in the same computation. A pretrained model can supply structured possibilities. A smaller reward-trained mechanism can decide which possibility remains appropriate now. Continued learning can then improve the proposals themselves. For roboticists, that suggests a sharper question than “How fast does my VLA run?”. How late in the execution path can new evidence still change what the robot does—and does that decision improve with experience?. Real-Time EXPO-FT offers a concrete, testable answer to that question.