Latest / Embodied AI 101 / Co-training Large Behavior Models: Data Modalities and Training Strategies for Robot Manipulation
Transcript
- Co-training Large Behavior Models: Data Modalities and Training Strategies for Robot Manipulation. Large, multi-task robot policies – sometimes called “large behavior models” (LBMs) or vision-language-action (VLA) models – promise to bring a foundation-model approach to robotics: training a single policy with diverse, large-scale data so it can flexibly handle many tasks. In practice, however, collecting hundreds or thousands of hours of robot demonstrations across all the needed tasks and scenarios is prohibitively expensive. A clever alternative is co-training: supplementing target robot data with other, more abundant data sources (vision-language corpora, simulated and cross-robot demonstrations, egocentric human videos, etc.) to diversify the training signal. But which kinds of data actually help a robot policy generalize, and how should we mix them in training?. A recent study by Lin et al. (2026) undertakes a massive empirical investigation to answer these questions. They train 89 different VLA policies – each a fusion of a pretrained vision-language backbone with an action-sequence generator – using various combinations of data and training regimes. In total they leverage ~4,000 hours of robot and human manipulation data plus 50 million vision-language samples, and evaluate on 58,000 simulated rollouts and 2,835 real-world experiments. By ablation, they isolate the impact of five key co-training modalities and three training strategies. Their findings give clear guidance: co-training with rich, semantically meaningful data (vision-language examples and diverse robot demos) dramatically boosts out-of-distribution performance, whereas crude discretizations of actions do not help (and sometimes hurt). Moreover, combining all the effective modalities yields cumulative gains, enabling much stronger generalization and faster adaptation to new tasks. Below we explain the approach and results in detail, aiming to help researchers understand how to build and train robust, generalist robot controllers from heterogeneous data. Our focus is on what kinds of data and training protocols work best and why, rather than low-level math. We’ll outline the model architecture and objectives, then review each co-training modality and strategy, and finally delve into the key experimental findings with references to the study’s data. Vision-Language-Action Model Architecture. At the core of the study is a vision-language-action (VLA) policy: a system that takes visual observations and task instructions (text) as input and generates a sequence of robot actions. The policy uses a pretrained vision-language model (VLM) as a perceptual backbone to interpret images and text, and then appends an Action Transformer head (called “ActionFT”) that generates the robot commands. In this case the authors use a large-scale pretrained model (PaliGemma2-PT) as the VLM backbone, which provides rich semantic understanding of the scene and language. On top of this, the ActionFT is a diffusion-like transformer that outputs continuous action sequences. Training such a VLA model is formulated as imitation learning: using demonstrations of correct robot behavior to teach the policy to output similar actions given new inputs. In practice the policy is trained with two losses: a continuous action loss using flow matching (a diffusion/denoising-style objective for generating real-valued action trajectories) and a discrete token loss (cross-entropy) when training on tasks that include symbolic action tokens or text sequences. The flow-matching loss works roughly as follows: given a block of demonstration actions (A_t), a random timestep (\tau), and random noise (\epsilon), we form a noisy action (A^\tau_t = \tau A_t + (1-\tau)\epsilon) and train the model to predict the residual ((A_t - \epsilon)). This encourages the model to learn a smooth generative model of actions over time. In parallel, when training on language tokens or discrete actions (for example, labels extracted from other data), the model is trained with a standard classification loss. Together these objectives let the network align the language, vision, and action domains. Importantly, the VLM backbone means the policy starts with a rich visual and linguistic understanding, and the imitation learning adapts this understanding to action predictions. One byproduct is that training exclusively on robot data can actually damage the backbone’s semantic grasp of the world, so one of the goals of co-training is to preserve (or even improve) the VLM’s high-level knowledge. We will return to this point in the results. Co-training Data Modalities. The heart of this study is examining what data to co-train on. The authors consider five broad types of auxiliary data sources, each chosen for how it might enrich the robot policy’s understanding or behavior repertoire:. Standard Vision-Language (VL) Corpora: Large collections of images with captions or associated questions/answers from general-purpose datasets. Examples include spatial reasoning or object-commonsense benchmarks (the paper mentions RoboPoint and RefSpatial among others). These data pair visual scenes with text (descriptions, questions, spatial queries) but come from domains outside robot manipulation. The idea is that training on such data gives the model more general visual-language grounding and commonsense. In practice the policy sees images (sometimes rendered from robot scenes) paired with descriptive text, question-answer pairs, etc. As a result, the vision-language backbone gets continued exposure to diverse visual concepts and language interactions that go beyond the robot domain. Dense Language Annotations of Robot Trajectories: In this modality, the raw robot demonstrations are augmented with high-frequency textual descriptions of what the robot is doing. These annotations might be automatically generated by heuristics or by prompting the vision-language model itself on each frame (or short sequence) of a trajectory. For example, a two-arm Franka robot picking up a jar and pouring ingredients might get step-by-step captions like “reach for the jar,” “lift jar,” “tilt jar,” etc. These dense, frame-level instructions provide rich semantic signals that align robot actions with language descriptions. They serve as a kind of high-resolution instruction data, boosting the model’s language understanding specifically around robot states and actions. Cross-Embodiment Robot Data: This means demonstrations from other robots with different embodiments or kinematics (different arms, manipulators, etc.), or even the same tasks performed in different simulated environments. For instance, an open-embodiment dataset might include grasping or placing tasks executed by multiple robot platforms. Training on such data can teach the model general manipulation skills that transfer across embodiments. The vision-language representation can learn features like “how to pick up a cup” that hold true even if the robot arm is a bit different. In this study, co-training includes a dataset of diverse robot behaviors from the Open X-Embodiment collection (a large library of multi-robot demos). Egocentric Human Videos: Real-world videos of people manipulating objects from a first-person (egocentric) viewpoint. These are essentially human analogs of robot tasks: for example, a video of someone pouring sugar into a bowl, or putting dishes on a shelf, captured from chest or head mount. There are two ways these videos are used. First, one can extract latent action tokens from them: unsupervised codebook embeddings (e.g. using a model like FAST or VQ-VAE) that compress the visual motion into a discrete sequence. Second, one can generate natural language descriptions of the human’s actions (frame by frame) using the VLM itself. The result is either a sequence of embeddings or a sequence of manual-style annotations of what the human did. The intuition is that this human manipulation data injects semantic context (especially when annotated) and visual diversity (different objects/hands) into the training. Discrete Robot Action Tokens: Finally, the authors explore converting the robot’s continuous joint or end-effector actions into a discrete token representation (like words). They experiment with two methods: FAST (a learned codebook of action primitives) and VQ-VAE (vector-quantized autoencoder) to compress sequences of robot motions. The idea is to train the model on sequences of these tokens as if they were “words,” which in theory could let the language model handle actions symbolically. However, as we’ll see, this modality does not prove very helpful. These five modalities capture a spectrum of possibilities: from broad vision-language knowledge (the standard VL corpora) to robot-specific signals (dense annotations, cross-embodiment demos) to more exotic proxy tasks (human video, action tokens). Table 1 in the paper (not shown here) lists the concrete datasets: e.g. 50M image-language pairs from general corpora, ~4,000 hours of combined robot/human manipulation data, etc. The target robot is a bilingual dual-arm Franka platform, trained on 403 different tasks (seen and unseen) in both simulation and the real world. This massive data collection effort underscores the scale needed to probe these questions. In the actual training runs, the policy is fed these modalities in various combinations. For example, one ablation might train only on robot data plus VL corpora; another might do robot data plus human video annotations; another might mix in the discrete action tokens, etc. By comparing performance across these combinations, the study teases apart which data sources are truly beneficial. Co-training Strategies. How to schedule all this data during training is also crucial. The authors consider three main strategies for mixing the target robot data with the auxiliary data modalities:. Single-phase co-training: In this simplest approach, all data sources are shuffled together into one big dataset, and the model is trained on this combined set simultaneously. Concretely, each training batch might contain some robot trajectories (with whatever labels they have) and some examples from the auxiliary sets (vision-text pairs, annotated human frames, etc). This means the policy learns from both robot and non-robot data at the same time, right from the start of training. Two-phase (first-phase-only): Here training is done sequentially. In Phase 1, the model is trained exclusively on the auxiliary data (no robot demos). For example, it might train on lots of vision-language image-text pairs and maybe human video annotations, to beef up its semantic understanding. Then in Phase 2, the model is fine-tuned on just the target robot data (real and sim demos) to adapt to the actual control tasks. The intuition is to use the extra data to “pre-train” general skills and language grounding, then specialize on the robot actions. Two-phase (full co-training): This variant also uses two stages, but in a looser way. Phase 1 is still the same (train on auxiliary data only). In Phase 2, however, the model is trained on both the auxiliary data and the robot data together (like single-phase continuing). The idea is to keep seeing some of the extra data even while fine-tuning on the robot, rather than losing it completely. This is akin to progressive or curriculum learning: first learn from broad sources, then refine while still retaining the broader context. These three strategies capture a spectrum from fully interwoven to strictly sequential training. Each has pros/cons: single-phase might be easier for optimization but could “dilute” the robot signals; two-phase can let the model focus but risks catastrophic forgetting of the early skills. The study tests all of them. Experimental Setup. Putting it all together requires a unified experimental design. The VLA model (vision-language policy) is initialized with the large pretrained backbone and is trained by imitation learning on the chosen co-training data. Across all runs, the core robot data consists of dual-arm Franka demonstrations on 403 different tasks, collected in both simulation and real setups. These tasks include grasping, placing, long-horizon sequences, etc. In total the robot and human data amounts to about 4,000 hours of manipulation footage. Alongside this, the auxiliary vision-language corpora contribute roughly 50 million image-text samples. Each of the 89 policies evaluated corresponds to one particular combination of data modalities and training strategy. For example, one policy might be trained with single-phase on (robot data + vision-language + dense robot annotations), while another uses two-phase on (robot + cross-embodiment only), and so on. Training is run to convergence for each configuration. The evaluation covers both simulation benchmarks and real-world robot tests. In simulation, each policy is tested on 58,000 rollouts over all 403 tasks; this includes episodes on tasks it has seen in training, as well as on unseen new tasks. The tests include nominal conditions as well as distribution shifts (varying lighting, camera angle, textures, etc) to probe robustness. On real robots, 2,835 trials are run, focusing on language-instructed tasks. These real tests measure things like: following a natural language command with seen objects, generalizing to an unseen object referred by language, and adapting to long-horizon tasks (after fine-tuning on a small number of demos). In one set of fine-tuning tests, for example, policies trained on co-training data are given 200 demos on novel dexterous tasks (like packing items into a bag or pouring ingredients) and then evaluated on those tasks. In summary, the evaluation is extremely thorough: 58k sim runs and ~2800 physical trials with statistical rigor (t-tests, Bayesian analysis, etc). This backs up the study’s conclusions with high confidence. Key Results and Analysis. The headline findings can be grouped into two parts: (1) which data modalities improve generalization, and (2) how best to combine them. Below we break down the main outcomes. Effect of Individual Modalities. An ablation study looks at including or excluding each data modality while holding others and the strategy fixed. The results show a clear pattern:. Vision-Language Data & Cross-Embodiment Robot Data: Including standard vision-language examples (generic image-text pairs) significantly boosts out-of-distribution performance and instruction-following accuracy. Similarly, adding cross-embodiment robot demos (from different robots/environments) yields large gains on novel tasks. The model benefits from the extra semantic and spatial reasoning provided by these sources. In fact, the analysis shows that multi-phase training that preserves vision-language data into later stages is particularly effective: it keeps the backbone’s semantic knowledge intact and further improves unseen-task success. Notably, these modalities improve generalization without hurting performance on the tasks seen during training. In short, more visual-language grounding and more varied robot motions allow the policy to better cope with new objects and shifts. Dense Robot Annotations: Adding high-frequency text descriptions of the robot’s own demos also helps, though its benefits are more nuanced. Both simple scripted labels and richer labels generated by the VLM improve performance on unseen tasks, since they are informative about the actions being taken. The VLM-generated annotations (which mention objects and context in detail) give larger gains. In practice these annotations act like a strong “bootstrap” signal early in training: two-phase training with a first stage on the annotations gives a jump-start to learning that seems to carry over. Human Video Annotations: Using egocentric human videos with language annotations (via the VLM) proves beneficial. These annotated human demonstrations help the policy understand language instructions better and generalize to novel objects because they expose the model to everyday manipulation examples. However, the more exotic use of latent action tokens from these videos (the unsupervised codebook representation of human motions) is only helpful in a low-data regime. When robot data is scarce, these latent tokens provide extra signal and can slightly improve training. But once there is a decent amount of robot demonstrations, the latent-code approach does not add value. In summary: textually-labeled human video segments are useful co-training material; raw video action tokens less so. Discrete Action Tokens (FAST / VQ-VAE): Surprisingly, unlike prior hope, simply discretizing robot actions into token sequences had no positive effect in these experiments. Training on the FAST or VQ-VAE codes did not yield any statistically significant improvement, and in some cases even degraded performance. The authors attribute this to representational issues: compressing the rich continuous action space into a small codebook loses nuance and makes it harder to learn precise control. In effect, the action tokens turned out not to be an effective substitute for real action data or language guidance. In short, the effective modalities are those that carry semantic information: generic vision-language examples, textual annotations of robot motions, and annotated human tasks. The ineffective modality was the artificial discretization of actions. Put differently, “language-rich” co-training data help, but abstract tokens of motion do not. This highlights that the model benefits from aligning vision and language at the action level; simply quantizing actions into unknown symbols adds little. Cumulative Gains from Combined Modalities. Next, the study examines how these modalities stack up when combined. The take-away is that the gains largely compound. The authors construct a “Final Model” that is co-trained with all the beneficial modalities (vision-language corpora, dense annotations, cross-embodiment demos, and language-labeled human video). This model performs substantially better than any individual modality alone. In simulations on unseen tasks, the Final Model achieves a 72.6% success rate, whereas the baseline (trained only on robot demos) was roughly 36% lower. In the real-world language-following tests, the Final Model averaged 69.4% task completion compared to a far lower baseline (an improvement of 45 percentage points relative). These are huge leaps: adding the co-training signals more than doubles success on difficult unknown tasks. Figure 6 in the paper (not shown here) plots this clearly: each new modality (one at a time) lifts performance, and the final combination is far above the baseline. Crucially, even under distribution shifts and with novel instructions, the co-trained model generalizes far better. Adaptation and Fine-Tuning. One goal of large behavior models is fast adaptation to new skills. The authors test this by fine-tuning the pretrained policies on novel dexterous tasks with a limited number of demos. For example, after pretraining all modalities, the Final Model is given 200 demonstration examples of each of several new tasks (like “Pack Items into a String Bag” or “Pour Ingredients into Soup”) and then evaluated. The co-trained model quickly masters these tasks: it averages 90.2% success after fine-tuning, compared to only 67.8% if you start from the baseline (no co-training). That is a 22.8-point improvement over the baseline case, even though the same 200 demos were used. Moreover, starting from co-trained parameters outperforms training from scratch by 42.9 points. This demonstrates that co-training not only helps zero-shot generalization but also yields a representation that is malleable – the robot can latch onto new tasks with far fewer examples. In practical terms, this means a policy pretrained with diverse data can rapidly incorporate new behaviors, which is exactly what one wants in a foundation model. The co-training endows the policy with better “grasp” of semantics and manipulation structure, so that when new grasping or pouring examples are shown, the model generalizes them correctly. Preservation of World Understanding. An important side observation relates to the underlying vision-language backbone. The authors probe the semantic capabilities of the VLM component after training. It turns out that if you train only on robot demonstrations, the VLM’s capabilities in language and spatial reasoning degrade significantly. In other words, naively fine-tuning the model on just robot actions causes it to “forget” a lot of its vision-language knowledge. In contrast, co-training preserves these skills. The final co-trained policies (especially those that continued vision-language data late into training) actually match or exceed the original VLM on benchmarks of language comprehension and spatial reasoning. Thus, one side effect of the effective co-training strategy is that it keeps the model’s world understanding alive. This ensures that the policy can still interpret instructions and scenes correctly, rather than becoming a specialized but semantically shallow controller. Training Regime Insights. One more finding emerges when comparing the training strategies: it is beneficial to maintain exposure to the auxiliary data as long as possible. The two-phase “full” strategy (where vision-language data is seen again in the second phase) consistently outperforms the variant that drops the extra data after Phase 1. In fact, keeping the VL and annotated data in the later training stage was key to preventing any loss in the backbone’s reasoning. The single-phase approach (mixing everything from the start) also works reasonably well, but the staged approaches gave more flexibility. The study concludes that a multi-stage training schedule, where the model is first warmed up on the broad data and then allowed to refine on both broad+robot data, yields the best of both worlds. Finally, as noted in the abstract, the authors also experimented with chain-of-thought conditioning – having the model generate intermediate reasoning “thoughts” before actions – but found no significant benefit in the control context. This suggests that, at least in the tested benchmark, simply adding textual reasoning steps doesn’t replace the need for grounding in diverse real data. Practical Takeaways. What does all this mean for building generalist robot policies? The message is clear: diversification of training data dramatically improves robustness and generalization. Vision-language co-training and cross-robot demos should be centerpieces of any strategy aiming to produce a foundation-scale robot model. By contrast, exotic schemes like discrete action codebooks seem of low priority. In practice, one practical recipe emerging from this work could be:. Start with a strong vision-language backbone. Pretrain (or co-train) on large-scale image-text datasets to ensure broad semantic and spatial knowledge. Inject dense language captions on your robot trajectory data (either scripted or via a VLM) to align actions with words. Mix in additional robot demos from varied platforms or simulators to teach more invariant skills. Include human video data annotated in language for extra real-world variety. Use a two-phase training schedule: first learn from the broader datasets, then fine-tune with both broad and target robot data. Finally, fine-tune on any new tasks with a small number of demos, leveraging the pretrained model. This yields very sample-efficient learning on complex tasks. By following such a strategy – essentially embracing heterogeneous data sources – the resulting policy can reach roughly twice the effectiveness of a policy trained only on robot logs. In the simulations, the “best” co-trained model achieved over 72% success on unseen tasks (versus about 36% for baseline). On real-robot language-instruction tasks, it achieved 69% success (versus ~24% baseline). These are not just small margins; they represent transformational improvements that come purely from smarter data mixing. In closing, this study provides a data-driven blueprint for roboticists: when scaling up robot learning, supplement the scarce robot demos with diverse vision-language and multi-embodiment data to teach the model the “long tail” of visual and semantic understanding. According to Lin et al., such co-training is key to building truly generalist, dexterous robot controllers. As we enter an era of foundation models for embodied AI, knowing what extra data to use (and how to train with it) will make all the difference in practice. Sources: Lin et al., A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation (2026), which ablated 89 model variants and found that vision-language and cross-embodiment data yield large generalization gains, whereas discrete action tokens do not. (Results paraphrased from the paper and its online summary.).