Latest / Embodied AI 101 / Co-VLA: Coordination-Aware Structured Action Modeling for Dual-Arm VLA Systems
Transcript
- Co-VLA: Coordination-Aware Structured Action Modeling for Dual-Arm VLA Systems. Vision–Language–Action (VLA) models are emerging as a powerful paradigm in robotics, unifying perception, language understanding, and control in a single end-to-end model. In brief, a VLA model takes an image (or video) of the robot’s surroundings and a text instruction, encodes them into a joint latent representation, and decodes that into a sequence of low-level robot commands. Importantly, these models are typically built by fine-tuning a large pre-trained vision-language foundation model (like a vision-augmented LLM) on massive robot trajectory datasets, such that the same model can leverage both web-scale vision-language knowledge and robot action data. A recent flagship example is Google DeepMind’s “RT-2” model (2023), which co-trained a vision-language Transformer on internet-scale vision-text tasks alongside robot manipulation trajectories, showing that a single “one-model-fits-all” approach can learn to plan and execute robotics actions from high-level instructions. OpenVLA is another noteworthy example: a 7-billion-parameter VLA model trained on ~970k real and simulated robot episodes, which sets a new state of the art on a broad suite of manipulation benchmarks. These foundation-style VLA models can control multiple different robot platforms out of the box and generalize well to new objects, scenes, and instructions. However, almost all of this exciting progress has focused on single-arm control or treated a dual-arm system as just a higher-dimensional “monolithic” action space. In real-world robotics, many tasks require bimanual coordination – two arms working in concert – which poses unique challenges. Two arms double the dimensionality of the action and state spaces, making learning harder and execution more prone to errors. Moreover, tightly coupled tasks (e.g. carrying a tray, threading a needle, assembling two-handed tools) demand precise timing and force balance between the arms. Humans naturally coordinate their hands by semantic roles or shared intent, but end-to-end learned policies can struggle to capture that structure implicitly. As one recent work puts it, “as bimanual tasks become tightly coupled and execution constraints become critical, implicit coordination alone is insufficient for reliable, interpretable, and stable behavior”. In other words, simply scaling up a single-arm VLA model to output two arms’ joint commands often fails in complex two-handed tasks. The new Co-VLA framework directly addresses these challenges by introducing explicit coordination structure into VLA models. Rather than a single “monolithic” action head that jointly predicts all degrees-of-freedom for both arms, Co-VLA replaces it with a Structured Action Expert (SAE) that has a built-in, modular division of labor. Concretely, Co-VLA’s action decoder is factorized into multiple parts: a shared latent and residual latents for each arm. The shared latent vector encodes the task-level coordination intent – for example, moving both hands upward together, or keeping them aligned on either side of an object. In contrast, each arm’s residual latent captures its individual deviations – fine adjustments or asymmetric actions that one arm must make independently. During training, Co-VLA applies a coordination-aware loss that explicitly encourages this structure: the model is penalized if it places coordinated motions in separate latents, or if it forces asymmetric motions into the shared latent. This encourages the shared latent to learn the “common mode” of the task, while the residual latents pick up the differences. The result is a learned action representation that is more interpretable and enables stable two-arm synchrony. At deployment time, Co-VLA further adds a Latent-Aware Controller (LAC) to translate the learned latent structure into smooth and safe motor commands. The LAC operates at the level of joint velocities or torques and reads the shared vs residual latents to modulate execution. For example, if the shared latent indicates a highly synchronized motion, LAC will command the arms to move strictly in unison; if the latent signals allow asymmetry, LAC will relax coordination. The LAC also adjusts compliance and speed based on the latent: for delicate or tightly-coordinated tasks it can slow down and smooth movement, while for simpler tasks it can speed up execution. Crucially, LAC works within the standard robot control pipeline – it does not require special force sensors or impedance control schemes. Instead, it simply biases the controller on-the-fly using the latent information. This way, Co-VLA merges the best of learned structure and classical control for safety and efficiency. In experiments the authors show that these ideas are very effective. They evaluate Co-VLA on both simulated and real-world dual-arm benchmarks featuring tightly-coupled tasks. Compared to a “monolithic” VLA baseline (the same vision-language backbone with an unstructured action head), Co-VLA dramatically improves performance. For instance, on hard coordination tasks the success rate rises by 27% under Co-VLA. In out-of-distribution (OOD) trials – e.g. new objects, novel conditions, or hardware variations – the gap is even bigger: the baseline succeeds only 13% of the time, whereas Co-VLA succeeds 27%. And Co-VLA is not only more reliable, but also faster: task completion times dropped by up to 25% compared to the baseline. In short, injecting explicit bimanual structure yields a significant boost in both efficiency and robustness. The Co-VLA paper conveys these points with clear motivation. It begins by noting that large VLA models can produce some bimanual coordination through sheer capacity, but that this coordination is implicit and often unstable in the tightest-coupling tasks. By contrast, Co-VLA’s core idea is to “introduce explicit structural priors into VLA models” for the bimanual case. The Structured Action Expert is implemented simply by replacing the original action head. Instead of one part of the network generating all joint commands at once (which we call a monolithic head), Co-VLA uses a multi-branch decoder: one branch generates a “shared latent” vector for the two arms, and two other branches each generate a “residual latent” for each arm. These latents are combined to produce the final actions. During training, the authors employ a modular coordination-aware loss: for tasks that require the arms to mirror each other, the loss encourages sharing those motions in the shared latent; for tasks that require different actions, the loss lets those differences go into the residual latents. Thus the representation naturally disentangles common intent from individual adjustment. In the authors’ words: “The shared latent encodes task-level coordination intent, while residual latents capture execution adjustments for each arm”. The Latent-Aware Controller (LAC) is one of Co-VLA’s most interesting features. Once the VLA model is trained, Co-VLA does not rely solely on the neural policy to enforce safety or smoothness. Instead, the LAC watches the latent outputs in real time and tweaks the physical execution. For example, if the shared latent is changing rapidly, LAC might enforce stronger synchronization on the actual motors to keep the arms moving together. If one residual latent has a large offset, LAC might damp down that arm’s motion or add a safety margin. The LAC operates entirely at the joint-command level: it takes the neural outputs (desired joint velocities) and performs additional filtering or modulation. This means Co-VLA can be used on any standard robot controller without rewriting the whole stack. As the paper describes, LAC “remains compatible with standard control pipelines without requiring force or impedance control”. In effect, LAC bridges the gap between learned latent planning and classical control, dynamically adjusting the coupling of the arms during execution. Putting these pieces together, Co-VLA turns a single “black-box” VLA controller into a structured, modular one. The paper reports ablation studies showing that each component is important. For example, when the coordination-aware loss is removed and all latents are fused as before, performance drops back toward the monolithic baseline. Similarly, without the LAC, the raw policy tends to produce jerky or unsafe motions. But with all components – the Structured Action Expert and the LAC – Co-VLA learns policies that are noticeably smoother and more reliable in dual-arm tasks. These improvements follow a broader trend in robotics: adding inductive biases for coordination can greatly simplify learning. Other recent works have found similar ideas help. For instance, the BUDS (Bimanual Dexterity from Stabilization) framework proposed a hard role assignment: one arm is designated the “stabilizer” (holding objects in place) and the other the “actor” (performing the main task). With only 20 demonstrations, BUDS reached over 75% success on difficult dual-arm tasks like zipping jackets or cutting vegetables, whereas an unstructured baseline only succeeded ~47% of the time. That’s a 56% improvement by imposing a simple structure. Co-VLA follows this spirit but in a learned way; rather than pre-assigning “actor” vs “stabilizer”, it learns latent modes of coordination from data. Similarly, the recent InterACT method uses a hierarchical attention network to explicitly synchronize the two arms. It employs a multi-arm decoder with “synchronization blocks” so that each arm’s action prediction sees the other one’s intermediate output as context. On a range of simulated and real tasks, InterACT also outperforms naively trained policies. Co-VLA’s approach is different in architecture, but the goal is the same: make the model explicitly attend to inter-arm dependencies. Even more relevant is a line of work on latent coordination spaces. The CLAS model (2022) proposed a “central latent action space” shared among multiple robot agents, to coordinate multi-robot manipulation. Co-VLA’s “shared latent” is conceptually similar: it’s a bottleneck of information common to both arms. On the flip side, the CoFreeVLA model (Jan 2026) targets another dual-arm issue – collisions. It augments a VLA with a self-collision risk estimator that looks briefly ahead and vetoes likely collisions. Whereas CoFreeVLA focuses on safety (gating risky motions), Co-VLA focuses on stable coordination. In practice, these ideas could complement each other. In summary, Co-VLA is a significant step toward making big VLA policies work reliably on two-arm systems. By factoring the action generation into a shared coordination intent and per-arm details, it injects a useful prior that mirrors human strategies for bimanual tasks. The empirical gains are substantial: with Co-VLA, the robot is more likely to succeed on demanding two-handed tasks and even more likely to behave safely when things go slightly wrong in deployment. Importantly, Co-VLA demonstrates that such structure can be learned end-to-end: it doesn’t require the designers to hand-label which tasks are symmetric or how to micromanage each arm. Instead, the model discovers the coordination structure from data, guided by the loss. Practically, one could imagine applying Co-VLA by taking an existing VLA agent (say, a fine-tuned OpenVLA or RT-2 variant) and retraining its action head. You would implement the Structured Action Expert modules as described, then fine-tune on your dual-arm dataset with the coordination loss. At run time, you’d wrap the policy with the LAC module – which is relatively lightweight – before sending commands to the real robot. The authors report that this still runs at real-time speeds comparable to the base model. Looking forward, Co-VLA suggests that more such structured priors will benefit embodied AI. One can envision extensions to even more arms or to teams of robots: the same shared-vs-residual idea could apply to quad-robot synchronization, or to coordinating a robot with a human partner. It also raises questions about how to automatically discover the “right” coordination structure for a task. Co-VLA uses a simple shared vs residual split, but there might be richer factors (e.g. one latent for bilateral symmetry, another for depth vs sweep, etc.). Future work may explore hierarchical or task-conditioned coordination latents. For now, Co-VLA provides strong evidence that explicitly modeling how two hands should work together can greatly improve VLA-based manipulation. In the growing ecosystem of foundation models for robots, it’s a reminder that injecting the right structure – even in large neural networks – can pay off in real-world performance. References: Co-VLA’s methods and results are summarized from the authors’ arXiv abstract. Definitions of vision-language-action models are taken from robotics references. OpenVLA results are from its project page. Prior bimanual works include Stabilize to Act (BUDS), InterACT, and CoFreeVLA, each demonstrating benefits of explicit coordination or safety modules. These capture the context in which Co-VLA’s contributions sit.