Latest / Embodied AI 101 / CAST: Making Language Matter to Robot Policies

Transcript

CAST: Making Language Matter to Robot Policies · Embodied AI 101

• CAST: Making Language Matter to Robot Policies. CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models • Catherine Glossop, William Chen, Arjun Bhorkar, Dhruv Shah, and Sergey Levine • UC Berkeley and Princeton University • IEEE Robotics and Automation Letters • 2026. The instruction was never necessary. Imagine evaluating a navigation policy in a corridor. You ask the robot to approach a doorway. It moves smoothly, avoids furniture, and reaches the right neighborhood. The rollout looks convincing. Now leave the robot in the same starting position and change the instruction. Ask it to follow the opposite wall instead. It executes almost the same trajectory. The first rollout demonstrated navigation competence. It did not necessarily demonstrate instruction following. That distinction is the central motivation for CAST. The authors argue that a vision-language-action model can learn to predict appropriate-looking behavior while largely ignoring its language input. If the training observations already identify the most likely task, language contributes little additional predictive information. Adding more vocabulary, more captions, or more environments does not automatically fix that problem. What matters is whether different instructions demand different actions from the same or similar observations. This is an uncomfortable possibility for anyone training generalist policies. We often inspect dataset diversity globally: the number of environments, objects, embodiments, instructions, or hours. But global diversity can coexist with a locally deterministic action distribution. A dataset might contain hundreds of tasks while presenting only one plausible demonstrated task in each particular visual situation. CAST changes that local distribution. Rather than collecting additional demonstrations, it synthesizes alternative instructions and the actions that would correspond to them at states already present in the dataset. The resulting policy cannot fit all those examples equally well by treating the image as a complete specification of the task. The instruction becomes useful for resolving what should happen next. A version clarification matters here. The original preprint appeared on August 19, 2025. The public revision dated June 8, 2026 adds manipulation experiments, a human evaluation of label quality, and robustness tests. This article follows that revised manuscript and the released implementation. The supplied DOI resolves to IEEE, but its full text was not accessible through the research interface; the authors’ institutional publication listing confirms the 2026 RA-L venue. From describing behavior to proposing alternatives. CAST stands for Counterfactual Augmentation with Synthetic Trajectories. It is primarily a data-generation method, not a new policy architecture or reinforcement-learning objective. The authors call the policies trained on its output CounterfactualVLAs. The distinction between hindsight labeling and counterfactual augmentation is essential. Hindsight labeling asks what instruction could describe an observed trajectory. A robot passed a cabinet and approached an entrance; a captioning system produces a plausible task description. This gives previously unlabeled behavior a language interface, but it does not change what the robot did. Counterfactual augmentation asks what the robot could reasonably have done instead. That requires changing both the instruction and its action target. The released pipeline explicitly separates hindsight labeling, filtering, counterfactual instruction generation, and counterfactual action generation. Consider a hypothetical scene containing a display stand on the left and an open passage on the right. The recorded robot takes the passage. Several paraphrases of that behavior still supervise the same action. An alternative instruction directing the robot toward the display stand is different only if it comes with a correspondingly different action target. These alternatives are not negative examples. CAST does not simply tell a model that the original action is wrong under a different instruction. It supplies a positive target for the alternative behavior. There is a useful connection to Hindsight Experience Replay, which reuses experience by changing the goal associated with an episode. The important difference is that hindsight replay grounds its relabeling in outcomes actually achieved. CAST introduces locally unexecuted actions, so it needs an additional mechanism for deciding what a plausible alternative action would be. That mechanism is a learned atomic policy. A vision-language model proposes the semantic alternative, while a simpler robot policy translates an associated primitive command into actions. This separates semantic invention from motor generation rather than asking an off-the-shelf VLM to invent continuous control values directly. Inside the CAST pipeline. Ground the simple behaviors first. For navigation, CAST begins with existing trajectories containing images and robot motion. Its released dataset builds on the GNM navigation mixture, which aggregates experience across different robots and environments. GNM’s underlying motivation is precisely that standardized action representations allow useful navigation behavior to be learned from heterogeneous embodiments. The first processing step divides trajectories into atomic motion segments. Navigation uses six categories: turns and smaller directional adjustments in either direction, forward motion, and stopping. These labels come from robot motion rather than visual interpretation. The implementation examines heading changes and displacement, using configurable thresholds and a maximum segment length. This has two advantages. It creates training labels for a relatively simple controller, and it identifies segment boundaries that can serve as decision points for counterfactual generation. The atomic policy combines an EfficientNet-based visual encoder, T5 language embeddings, and a diffusion action generator. It predicts short waypoint sequences conditioned on the observation and primitive instruction. The released configuration specifies an eight-step prediction horizon. The intuition is that a command specifying a basic directional change has a much tighter relationship to the action than a broad semantic task. Still, the observation remains important: executing a directional command in open space is not the same problem as executing it beside an obstacle. For someone adapting CAST, this atomic controller is not incidental infrastructure. It is the source of the synthetic motor supervision. If it cannot reliably realize the primitives used by the semantic proposer, the augmentation pipeline will manufacture contradictory training targets. Describe, then check, the recorded behavior. The navigation data does not arrive with the required task descriptions, so CAST first constructs them. The released hindsight-labeling code describes subsampled image pairs, orders those descriptions along the trajectory, and summarizes them into trajectory-level instructions. Its generation configuration selects Gemini 2.5 Pro. This decomposes a potentially difficult video-understanding problem into local description followed by temporal summarization. A separate filtering stage compares candidate instructions with the atomic motion sequence. The goal is to discard descriptions that conflict with what the robot actually did—for example, a description implying one turning direction when odometry indicates the other. The implementation passes both the candidate instructions and the relevant atomic labels into this filtering stage. This is an important distinction between semantic plausibility and behavioral accuracy. A description can mention the correct objects and still misdescribe the robot’s relationship to them. Motion-derived labels provide a check that does not depend entirely on the same visual-language interpretation that generated the caption. The revised manuscript reports an initially labeled navigation pool of approximately 65,000 trajectories and 320,000 labels, filtered to 16,000 trajectories with 65,000 labels, together with 86,000 counterfactual action chunks. The substantial reduction makes clear that filtering is a consequential part of the pipeline, not merely output-format cleanup. Propose a different ending. At a decision point, the counterfactual-generation code assembles the current image, the original instructions, preceding atomic behaviors, and the primitive-action vocabulary. The VLM proposes alternative instructions and corresponding atomic commands. Those commands are then passed to the learned action generator. The resulting synthetic ending is appended to a recorded trajectory prefix. The new instruction supervises a trajectory that shares real history with the original but branches into a different short-horizon action sequence. The conversion code makes that splice explicit. This construction is more limited—and more practical—than generating an entirely new robot episode. It does not require a simulator to render the scene after every hypothetical action. Nor does it require a video model capable of maintaining object permanence, camera geometry, and embodiment consistency throughout an imagined rollout. But that convenience creates the method’s most important boundary: the actions after the branch are synthetic; corresponding future camera observations are not generated. The public dataset represents missing observations using placeholder white images, and its documentation explicitly warns users to filter them appropriately. These placeholders are bookkeeping devices, not visual training evidence. Action chunking makes this partial supervision usable. A real observation can supervise a short future action sequence even when images along that sequence are unavailable. What it cannot provide is direct training on recovery or continuation from every state those hypothetical actions would visit. Why these examples should make language useful. CAST motivates its design using conditional mutual information: how much additional information does the instruction provide about the action once the observation is already known?. The paper develops a lower-bound argument using atomic commands. In plain language, a useful dataset should have substantial variation in the primitive behavior possible at a scene, while the scene and full instruction together should make the intended primitive predictable. The revised derivation treats atomic commands as a coarse description recoverable from the underlying actions and applies the data-processing inequality. A toy example helps. Suppose an identical image appears with two equally common target behaviors, one moving toward a left-hand destination and the other toward a right-hand destination. Before reading the instruction, the image does not resolve the choice. After reading an unambiguous instruction, it does. Now replace those alternatives with twenty paraphrases of the same behavior. There is more textual diversity, but no additional behavioral choice for language to resolve. This illustrates why CAST’s relevant unit of diversity is not simply the number of distinct strings. It is diversity in the relationship between observations, instructions, and actions. There are two qualifications to the theory. First, the paper presents mutual information as a conceptual proxy for instruction following, not a theorem guaranteeing downstream task success. Second, the argument depends on meaningful correspondence between instructions, atomic labels, and generated actions. The authors explicitly describe the connection between their practical generation procedure and these information quantities as heuristic. My interpretation is that the theory supplies a useful dataset-design principle rather than a complete learning guarantee. It tells us what statistical shortcut to disrupt. It does not establish that every added branch is physically feasible, semantically correct, or sufficient for long-horizon execution. The trained policies and the manipulation extension. The final navigation policy uses PaliGemma-3B, rather than a newly designed VLA backbone. PaliGemma itself was developed as a transferable vision-language model; CAST adapts it to action prediction. Navigation actions are normalized Cartesian displacements. An eight-step chunk has two coordinates per step and is represented by sixteen action tokens, with 128 bins available for coordinate values. A separate proportional-derivative controller translates the predicted displacements into robot velocities. The revised training recipe uses 40,000 updates, batch size 384, a learning rate of 0.0001, and a v4-16 TPU pod. The atomic policy and final VLA therefore play different roles. The atomic policy generates training targets using a restricted command interface. The final VLA maps the user’s richer instruction and camera observation directly to action predictions. The runtime system does not need to repeat the counterfactual-labeling pipeline. The manipulation extension is particularly valuable because it tests whether the idea depends on navigation’s convenient directional primitives. Here the authors use BridgeData, which already includes task-level language. BridgeData V2 contains roughly 60,000 trajectories collected with a low-cost manipulator across multiple environments, and was designed to support language- or image-conditioned multitask learning. For finer-grained decomposition, CAST uses components from Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control. That work trains policies to accept commands at several abstraction levels, including subtasks, motions, and visually grounded specifications. It provides a richer control interface than a single task-level sentence. CAST uses one model to turn a proposed alternative task into an atomic subtask command, and another—the pretrained Steerable Policy—to produce the corresponding action. The revised manuscript reports approximately 500,000 counterfactual instruction-action examples generated from the Bridge trajectories. Unlike navigation, this action generator produces non-chunked Cartesian delta actions. The manipulation VLA uses PaliGemma, 512 action bins, and 60,000 training steps on eight H200 GPUs. These are separate navigation and manipulation instantiations, not evidence of one jointly trained policy controlling both embodiments. Nevertheless, the extension supports the broader architectural idea: the atomic interface can change with the domain while the counterfactual augmentation principle remains intact. An evaluation designed around steerability. The navigation evaluation emphasizes last-mile instruction following, rather than long-distance exploration or map-based route completion. There are 27 instructions across an office hallway, a kitchen/common area, and an outdoor park, with five trials per instruction for the fully evaluated policies. The core comparison totals 645 navigation trials across methods. The tasks cover object reaching, spatially referential movement, and sustained movement relative to environmental structures. The success criteria deserve attention. Object navigation requires reaching within half a meter of the target. Referential tasks require the appropriate spatial behavior or destination. Continuous tasks involve maintaining the requested relationship to a structure over approximately two meters, with a one-meter proximity criterion for relevant behaviors. Contact with the target object is permitted in object-reaching trials, so the benchmark should not be interpreted as a uniform collision-free navigation test. The principal ablation is a standard VLA trained with hindsight labels but without counterfactual augmentation. The authors describe the two policies as otherwise identical. That is the comparison most directly aligned with CAST’s central claim. Manipulation uses six tasks on a WidowX setup: three relatively simple reaching or picking tasks and three placement tasks. Each policy receives ten trials per task with distractors. The appendix also reports an additional corresponding evaluation without distractors. This design asks a sharper question than whether a robot can perform a familiar manipulation in an uncluttered scene. When several objects or destinations are plausible, the observation alone becomes less informative about the requested task. It is not a perfect language-only intervention, because distractors also change perceptual difficulty. Still, comparing the distractor and non-distractor settings helps distinguish broad motor competence from the ability to select the behavior named by the instruction. What the results support—and what needs clarification. The authors’ headline navigation result is approximately 53% success, with the text reporting a 27% improvement over the hindsight-only VLA. The plotted comparison indicates a large absolute gain—roughly a doubling—not merely a 27% relative increase. Percentage points are therefore the more appropriate language for discussing the claimed improvement, although the exact aggregates require caution. For manipulation with distractors, the total row reports 35 successes in 60 trials for CounterfactualVLA and 19 in 60 for the standard VLA: 58.3% versus 31.7%, a 26.7-percentage-point difference. Without distractors, the reported totals are much closer: 35 versus 32 successes in 60 trials. That pattern is consistent with the proposed mechanism. Counterfactual supervision appears especially helpful when the scene permits several plausible behaviors and language must select among them. However, a thorough reading uncovers a reporting problem in the public revised manuscript. I checked the rendered PDF tables, not just their HTML extraction, and the detailed entries do not fully reconcile with the reported aggregates:. | Evaluation | Reported aggregate | Direct sum of the detailed rows | |---|---|---| | Navigation | Approximately 53% CAST success in the headline plot | Table II gives 68/135, or 50.4%, for CAST and 23/135, or 17.0%, for the standard VLA | | Manipulation with distractors | Table III totals give 35/60 for CAST | Its six CAST entries sum to 32/60, or 53.3%, rather than 35/60 |. These are my arithmetic checks of the published table entries. They are not corrected experimental results: the available material does not establish whether the individual entries, aggregate totals, or plotted values should be changed. The qualitative conclusion survives both readings: CAST improves aggregate performance over the matched standard VLA. But the precise effect size should not be treated as fully reconciled. Anyone using the results for a quantitative literature comparison should retain that qualification. There is also a substantive distinction between improvement and solved capability. Even the headline aggregate leaves many failures. The contribution is evidence that counterfactual data changes instruction-following behavior, not evidence of dependable open-ended autonomy. For a practitioner, I would put more weight on the direction of the controlled comparison and the distractor-dependent pattern than on a single advertised percentage. Those findings motivate replication. The inconsistent aggregates make replication and clarification particularly important. What the baseline comparisons actually compare. CAST also compares its navigation policy with a hierarchical baseline in which an off-the-shelf VLM selects atomic commands online and the same atomic policy executes them. That baseline performs substantially worse in the reported evaluation. This supports an interesting interpretation: using a VLM to generate offline supervision can be more effective than repeatedly asking it to choose a primitive during execution. Offline generation can exploit recorded trajectory context and consolidate many examples into a trained policy. It does not establish that hierarchical control is generally inferior. NaVILA, a closely related approach, uses a trained high-level VLA to generate spatially meaningful language commands and a learned visual locomotion policy to execute them. CAST’s generic online-planning baseline is not a direct reproduction of the complete NaVILA system. Differences in high-level training, interfaces, embodiment, and tasks matter. The other major baseline is CoNVOI. Its original method grounds VLM-selected image locations through a local occupancy map and uses the resulting reference path to guide motion planning. In CAST’s implementation, CoNVOI additionally receives LiDAR information, whereas CounterfactualVLA uses camera observations. Six task evaluations are missing because of infrastructure limitations. Consequently, this is neither a sensor-matched comparison nor a completely matched evaluation set. Finally, the authors train a ResNet-plus-FiLM policy on CAST data using a frozen CLIP language encoder. The larger VLM-based policy performs better overall, with a particularly pronounced advantage in continuous navigation. This suggests that improved data and strong pretrained representations are complementary, but the ablation changes capacity, pretraining, and trainability together; it does not isolate which factor causes the gain. The related LeLaN work is useful context: it also turns weakly labeled visual experience into language-conditioned navigation supervision, including action-free video. CAST’s distinguishing move is to expand the local instruction-action alternatives, rather than only extract supervision from the behavior already visible. The hardest limitations: grounding and missing futures. The revised paper evaluates label quality using 11 human judges and 440 label judgments. Under its strict correctness criterion, hindsight labels achieve approximately 60.5% accuracy, and counterfactual labels approximately 69.4%. Correctness requires agreement about the referenced entities, the motion, and the grounding of that motion in the scene. Those numbers are a useful antidote to interpreting synthetic supervision as ground truth. The main error source is grounding: a VLM may identify the objects and broadly characterize motion while getting their relationship wrong. Counterfactual endings can also imply collisions. The authors nevertheless observe improved downstream behavior when training on the combined data. My reading is that the experiment demonstrates a favorable net trade-off between additional supervision and label noise in this setting. It does not show that noise is harmless, or that more synthetic labels will monotonically improve performance. A plausible next step is to measure benefit as a function of both branch diversity and verified branch quality. The missing-future-observation issue is different. A synthetic action can move the robot away from the recorded state distribution without supplying corresponding examples of what to do afterward. Action chunking gets supervision across the initial branch, but it does not eliminate the continuation problem. The robustness results make that concern concrete. On a six-task hallway subset, success is 63% without dynamic obstacles in both morning and night conditions. When a person crosses the robot’s path, success drops to 37% in the morning and 43% at night. The authors report that the policy can avoid the obstacle but then fail to resume the requested behavior; they suggest inference speed and missing counterfactual observations as possible explanations, not established causes. For deployment, this separates three capabilities that should not be conflated: understanding the requested local behavior, safely accommodating an interruption, and recovering the task afterward. CAST most directly improves supervision for the first. Its evidence is much less complete for the latter two. A practical adoption plan. The release includes augmentation code, policy-training code, a navigation dataset containing original, atomic, and counterfactual components, and a downloadable checkpoint. The documented pipeline uses Google Cloud infrastructure for batch generation, while the training stack provides TPU workflows and GPU inference. The inference README reports approximately 18 GB of memory usage. That makes CAST a substantial starting point, but not a drop-in substitute for validating your own robot’s data and action conventions. My first step would be a language-sensitivity audit. Hold observations fixed and vary feasible instructions. Then check whether action predictions change appropriately. Include paraphrases that should preserve behavior and contrasting instructions that should change it. This distinguishes desirable semantic sensitivity from arbitrary prompt sensitivity. Second, validate the atomic teacher independently. In a new domain, the primitive vocabulary should correspond to behaviors you can actually supervise and verify. The objective is not to maximize the number of primitive names. It is to obtain a reliable interface between proposed semantic alternatives and executable actions. Third, concentrate generation on meaningful choice points. A scene with several feasible destinations or interaction targets is more informative than one where nearly every alternative instruction produces the same action. This is an adoption hypothesis motivated by CAST’s information argument, not a sampling ablation established by the paper. Fourth, audit the data loader carefully. The released dataset includes padded instruction fields and missing-observation placeholders. For a custom training stack, preserve the distinction between a valid action target and a valid observation from which to predict it. Also check coordinate conventions, embodiment normalization, and the boundary between recorded prefixes and synthetic endings. Fifth, run stronger controls than simply “more data versus less data.” I would compare counterfactual augmentation with repeated hindsight examples, extra paraphrases, and action-balanced resampling under comparable training budgets. Those controls would help separate the benefits of additional optimization targets, language diversity, changed action frequencies, and genuinely different behaviors at matched observations. Finally, test recovery explicitly. Introduce interruptions after the policy starts following an instruction, then evaluate whether it returns to the task. If synthetic branches improve initial selection but worsen continuation, a small amount of targeted recovery data may be more useful than another large batch of unverified alternatives. The attraction of CAST is reduced dependence on new demonstrations. It does not remove the need for careful supervision engineering. The broader lesson. CAST’s most useful contribution is a change in how we should think about dataset diversity. A dataset is not necessarily good for instruction following because it contains many instructions. It is good when instructions resolve genuine choices about behavior. The paper offers a practical way to create more of those choices: use a VLM to propose alternative tasks, connect them to grounded short-horizon controllers, and train on the resulting observation-instruction-action combinations. The revised experiments extend that idea beyond navigation into distractor-rich manipulation, while also exposing noisy grounding, incomplete future-state coverage, and unresolved inconsistencies in the reported aggregates. For robotics researchers, the takeaway is not that counterfactual labels replace better demonstrations, stronger controllers, or world models. It is that before scaling any of those, we should ask a simpler question:. Does our training data actually require the robot to listen?.