Latest / Embodied AI 101 / One Policy, Three Bodies, Zero Messages: Inside CHORUS

Transcript

One Policy, Three Bodies, Zero Messages: Inside CHORUS · Embodied AI 101

• One Policy, Three Bodies, Zero Messages: Inside CHORUS. CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy • Ria Doshi, Tian Gao, Annie Chen, Chelsea Finn, Jeannette Bohg • Stanford University • arXiv preprint, under review • 2026. A metadata correction before we begin: although the supplied publication date is July 29, 2026, the public arXiv record lists the first submission on June 10, 2026. As of July 29, the project page describes the paper as under review. The Scene CHORUS Is Trying to Create. Picture two mobile manipulators standing on opposite sides of a laundry basket. One robot reaches for the left handle. The other approaches the right. If the first lifts too early, the basket tilts and slips. If either robot waits forever, the task stalls. Neither has access to the other robot’s joint state, planned trajectory, or camera feeds. There is no centralized controller issuing synchronized commands. Instead, each robot looks through its own cameras, watches what its teammate is doing, and runs an independent copy of the same learned policy. That is the central idea behind CHORUS. The work takes a pretrained vision-language-action model, , and adapts it into one shared policy for a heterogeneous robot team. Every team member runs the same weights, but conditions the forward pass on its own local observation and a text prompt identifying both its embodiment and assigned role. Coordination happens indirectly, through the visible behavior of the other robots and the physical state of the shared environment. The compelling part is not simply that multiple robots can lift a basket or exchange a book. Similar tasks have been demonstrated with centralized controllers, engineered force controllers, trajectory optimizers, and learned bimanual policies. The interesting claim is that a large pretrained visuomotor model already contains enough semantic and motor structure that, after a relatively small amount of multi-robot fine-tuning, it can produce reactive coordination without a dedicated communication protocol or an explicit model of its teammates. CHORUS is therefore less a new multi-agent algorithm in the classical sense than a new use of robot foundation models. It asks whether broad single-robot pretraining can substitute for some of the machinery that multi-robot learning has traditionally needed. Why Decentralized Collaboration Is Hard. The standard centralized formulation is conceptually clean. Gather every robot’s cameras and proprioception, concatenate them into a team-wide observation, and ask one policy to generate the team’s joint action. The policy sees everything relevant and can explicitly couple one robot’s motion to another’s. But the interface scales poorly. Adding a robot means adding cameras, state tokens, and action dimensions. All observations must reach the controller before it can generate the next joint action. The control frequency is effectively limited by the slowest robot, network link, or inference request. If the team contains different embodiments operating at different frequencies, the data must also be synchronized or resampled into a common clock. A decentralized controller avoids those bottlenecks. Each robot consumes a fixed-size local context and emits only its own actions. Different embodiments can run at different control rates, and a delayed robot does not necessarily block the rest of the team. The price is partial observability. A local policy may see that it has grasped its side of a basket but not know whether the teammate has secured the other handle. Even when the teammate is visible, its intentions remain ambiguous. And if a task admits several valid coordination modes—pass left or pass right, lead or yield, lift now or wait—independent policies can select individually reasonable but mutually incompatible behaviors. Heterogeneity makes this harder. The Kinova, ARX, and YAM robots used by CHORUS differ in kinematics, degrees of freedom, sensors, mobile bases, and control frequencies. A useful shared policy must distinguish which body it currently controls while still transferring relevant concepts between bodies: grasping, waiting, handing over, retreating, and recognizing whether a teammate is ready. The CHORUS Recipe. CHORUS starts with synchronously recorded multi-robot demonstrations. At time step (t), the data logger records an observation and commanded action for every robot. But the policy never receives that synchronized joint tuple. Instead, the authors split every collaborative trajectory into independent per-robot examples. A training example contains one robot’s local observation, its future action chunk, and its robot-identifying prompt. The examples from all embodiments are pooled into one dataset, and a sampler selects one robot’s tuple at a time. The model’s loss never directly sees the team’s joint observation or joint action. This is a deceptively important design choice. CHORUS discards explicit joint conditioning while preserving the statistical structure of the collaboration. If the Kinova’s camera shows the YAM beginning to lift a book, the correct Kinova action in that demonstration is to approach the emerging handover location. If the ARX has not yet grasped the basket, the Kinova’s image should imply that lifting is premature. The teammate’s behavior remains in the data, but only as part of the local visual scene. In that sense, the shared environment becomes the communication medium. “No communication” here means no exchange of messages, camera streams, planned trajectories, or teammate proprioception. It does not mean the robots are behaviorally isolated. They communicate through motion, object state, and visual cues, much as people coordinate while carrying furniture without continuously verbalizing joint angles. Role and Identity Are Supplied, Not Discovered. The text condition is more informative than a bare robot ID. The prompt names the embodiment and specifies its role. The project diagram gives an example along the lines of identifying the YAM and instructing it to lift and hold a tape measure for the Kinova. This removes an otherwise difficult ambiguity: the model does not have to infer from pixels alone which body it controls or which side of the interaction it should execute. This means CHORUS operates below the level of task allocation. It does not decide which robot should open a door, carry a basket, or receive a book. Those responsibilities are encoded in the prompts and demonstrations. An LLM-based planner could potentially assign those roles upstream, but CHORUS itself is the low-level collaborative controller. Each robot receives one prompt for the entire task rather than a sequence of dynamically generated subtask instructions. That asymmetry also helps resolve coordination modes. If two initially symmetric agents must independently decide who yields, decentralized execution can be fundamentally ambiguous without communication, shared randomness, or an observable convention. Embodiment-and-role prompts provide such a convention. Decentralization Is Designed Into the Demonstrations. The authors explicitly choose task strategies that make local control possible. During data collection, they position robots and select collaboration patterns so that the relevant teammate and workspace remain visible in at least one local camera view. For two-robot demonstrations, one person operates both robots using a phone in each hand. A second operator controls the third robot in the three-robot setup. This is not a weakness so much as a statement about where the engineering burden moves. CHORUS does not turn an arbitrary centralized demonstration into a decentralized policy. The demonstrations must exhibit a strategy whose hidden state is recoverable from local observations. If the decision to lift depends on a teammate completely outside the camera frustum, no amount of parameter sharing can reconstruct that information reliably. The practical lesson is that decentralized imitation learning begins with observability-aware demonstration design. Camera placement and operator strategy become parts of the algorithm. One Backbone for Different Bodies. CHORUS fine-tunes the approximately three-billion-parameter VLA backbone. That choice provides two useful ingredients. First, was trained using heterogeneous robot data, semantic prediction tasks, web data, and multiple embodiments. Its action expert generates continuous action chunks through flow matching. Although the original model focuses on single-robot mobile manipulation, its pretraining already encourages common representations for objects, spatial relationships, robot bodies, and action semantics. Second, its interface can accommodate embodiment-dependent observations and actions. CHORUS accepts a variable number of image tokens and pads each robot’s action into a shared 32-dimensional vector. Only the dimensions relevant to the current embodiment are used. This is less elegant than learning a universal geometric action representation, but it is simple and preserves the backbone’s existing cross-embodiment interface. This direction has a direct lineage in Ria Doshi’s earlier CrossFormer work. CrossFormer trained one transformer across 30 embodiments and 900,000 trajectories spanning manipulation, navigation, locomotion, and aviation, showing that shared weights could operate over very different sensors and action spaces. CHORUS moves the cross-embodiment question from independent skills to coupled behavior: can one shared representation model not only several bodies, but also how those bodies respond to one another?. Training the Action Model. The policy predicts chunks of future actions rather than one command at a time. During training, the ground-truth action chunk is mixed with Gaussian noise. The action expert learns the velocity field that transforms noise back toward a coherent robot trajectory. At inference, the model integrates that learned field to produce a continuous action chunk. The important intuition is that flow matching can represent several plausible smooth futures without averaging them into an obviously invalid command. It also lets the system plan over a short temporal horizon, reducing the need to query a large model at every low-level control tick. Fine-tuning is parameter-efficient. The authors use LoRA rank 16 on the vision-language backbone and rank 32 on the action expert. Training runs for 20,000 steps with a batch size of 64, a training chunk length of 50, AdamW, and a cosine learning-rate schedule. All reported policies were trained on one NVIDIA H100. The sampler also compensates for unequal data rates. In the three-robot experiment, the two YAMs collectively produce far more action samples than the slower Kinova. The authors therefore upweight Kinova tuples by a factor of two. This detail matters: a nominally shared policy can still become biased toward whichever embodiment contributes the densest stream of control labels. Decentralized Deployment Is an Information Constraint. At runtime, every robot runs an independent policy query using only its own observation and prompt. The copies share weights but not runtime state. They can be hosted on separate machines, or their requests can be batched on one accelerator without sharing observation tokens inside the model. In the reported deployment, policies were hosted on an RTX 5090. The robots need not share a control frequency. The YAM runs at 30 hertz while the Kinova runs at 15 hertz, so CHORUS executes 40-action chunks on the YAM and 20-action chunks on the Kinova. Both chunks cover approximately the same amount of wall-clock time. Each robot can then advance through its chunks independently, tolerating modest differences in network and inference latency. The basket experiment is somewhat more synchronized, using equal 30-action chunks for both robots. The paper’s scaling argument is specifically about model structure. CHORUS retains one three-billion-parameter weight set and one robot-sized context window as the team grows. Separate per-robot VLAs would require six billion parameters for two robots and nine billion for three, while a centralized VLA retains three billion parameters but expands its context from one robot’s observations to two or three robots’ observations. Total team inference is not constant, however. Three robots still require three policy evaluations, whether those queries are distributed or batched. CHORUS eliminates growth in model definition and per-query context, not the physical compute required to control additional machines. The Real-World Tasks. The two-robot evaluations cover three interaction patterns. In laundry basket lifting, an ARX and Kinova grasp opposite handles and move the basket onto a mat. Each robot must delay the lift until the other has secured its side. The authors collect 43 demonstrations and evaluate ten rollouts, four with previously unseen clothing distractors inside the basket. In mobile tape measurement, a YAM grasps and anchors a tape measure while a Kinova mobile manipulator takes the tape and extends it. The embodiments run at different control frequencies, and half of the twelve evaluation trials contain distractors. Training uses 29 demonstrations. In mobile book handover, a YAM presents a book and the Kinova receives it before the YAM releases. Training uses 45 demonstrations, with 18 evaluation rollouts and an extra distractor book in six of them. This task becomes the basis for the teammate-perturbation experiment. The three-robot task is more spatially extended. A Kinova opens a door and retreats while two YAMs lift a basket and transport it through the doorway. One YAM cannot directly see whether the Kinova has cleared the door and must infer the appropriate time to move from the other YAM’s behavior. The team also has to pause and adjust while negotiating the doorframe. Thirty-four demonstrations are available for the YAMs, but corrupted data leaves the Kinova with only 18. Because CHORUS trains on independent per-robot tuples, it can retain the extra YAM data. A centralized policy requiring matched team-wide samples would have to discard or otherwise reconstruct those unmatched segments. What the Results Say. Pretraining Is Doing Most of the Heavy Lifting. The first comparison replaces the VLA with separate decentralized diffusion policies trained from scratch. Across basket lifting, tape measurement, and book handover, CHORUS improves the mean evaluation score by 64 percentage points. The VLA without weight sharing also performs substantially better than the from-scratch policies, indicating that the pretrained backbone—not only parameter sharing—is the dominant factor in standard, in-distribution rollouts. The qualitative failures are informative. The diffusion policies confuse the tape measure with distractor objects and frequently enter a teammate-mismatch pattern: one robot begins its part of the task before the other is ready. The VLA has prior visual knowledge about objects and manipulation scenes, but it may also possess useful temporal priors from bimanual and cross-embodiment pretraining—approach, align, wait for contact, then move. This experiment supports the broad claim that foundation-model pretraining transfers to collaboration. It does not isolate whether the gain comes from semantic visual knowledge, motor pretraining, model capacity, action decoding, or some combination. The comparison deliberately bundles those advantages together. Shared Weights Matter Most Under Perturbation. On ordinary rollouts, CHORUS and a version that fine-tunes a separate policy for each robot perform similarly. The difference appears when the teammate behaves outside the demonstration distribution. For the perturbation test, the authors script the YAM to lift the book and then move unexpectedly left or right. The Kinova must visually track the displaced handover point. The separately trained Kinova policy succeeds in 9 of 20 trials. CHORUS succeeds in 17 of 20: 8 of 10 leftward perturbations and 9 of 10 rightward perturbations. The authors interpret this as evidence that training one network on both sides of every interaction creates a better internal model of teammate behavior. That is plausible, but it is worth being precise. CHORUS has no explicit teammate-prediction loss, paired latent alignment, or joint action objective. Its batches contain independent robot tuples. The shared model simply sees a richer distribution: sometimes it is the robot offering the book, and sometimes it is the robot receiving it. The reactivity gain could therefore be described as reciprocal representation learning. Understanding how to control the YAM may improve the features used to perceive the YAM from the Kinova’s camera. That is weaker than demonstrating a formal theory of mind, but it may be exactly the kind of practical inductive bias that large shared policies make cheap. Less Information Beats More—Under This Architecture. CHORUS also outperforms a centralized baseline that receives both robots’ observations and emits both actions jointly. On the handover task, CHORUS scores 0.833 while both tested centralized resampling strategies score 0.611. Across all three two-robot tasks, the decentralized model has the higher mean score. The explanation is distribution shift. was pretrained with camera slots carrying consistent meanings for one robot: front, rear, base, and wrist views. The centralized adaptation repopulates those slots with top and wrist cameras from two different robots. It also doubles the input state and requires temporal resampling. More information is available in principle, but it is presented in a format that violates the backbone’s learned interface. This is an important result for VLA engineering, but not a general theorem that decentralized controllers outperform centralized ones. A centralized architecture designed from the outset with robot-indexed tokens, permutation-aware attention, and asynchronous observations might use the extra information more effectively. CHORUS shows that preserving a foundation model’s input semantics can matter more than naively increasing observability. Three Robots, but Not Zero-Shot Team Expansion. The three-robot CHORUS policy completes nine of ten transport trials. The architecture and parameter count remain unchanged: another robot supplies another independent stream of queries to the same shared weights. However, the authors train a new CHORUS policy on three-robot demonstrations. They do not take the two-robot checkpoint, add a third robot at deployment, and obtain coordination zero-shot. The demonstrated scaling is architectural and data-structural: the recipe handles three robots without redesigning the network. Generalization to unseen team sizes remains open. Where CHORUS Sits Among Related Approaches. MIMIC-D is perhaps the clearest diffusion-policy comparison. It jointly trains distinct per-agent diffusion policies under a shared loss and executes them from local observations. Its focus is preserving multimodal coordination strategies, with experiments in navigation, simulated dual-arm transport, and a hardware basket task. Unlike CHORUS, each agent retains its own parameters, and local state observations may explicitly include the positions of other agents. Latent Theory of Mind takes a more structured route. Each robot learns an ego embedding and a consensus embedding, along with a decoder intended to infer its teammate’s latent state. A sheaf-inspired objective aligns the consensus representations. The system can operate without explicit messages or optionally communicate and align those embeddings online. CHORUS removes the explicit consensus machinery and asks whether shared VLA representations plus vision are sufficient. CLAS represents another tradition: learn a central latent action space through which otherwise separate controllers coordinate. That can improve reinforcement-learning efficiency, but it introduces a deliberately shared coordination channel and was evaluated in simulated multi-robot manipulation. CHORUS instead places the coupling in the demonstration distribution and the visually observed environment. Seen against these methods, CHORUS’s novelty is the combination of four properties: real visual input, heterogeneous mobile embodiments, one parameter-shared VLA, and no explicit runtime exchange of team state. None of those ingredients is individually new. Their combination, and the evidence that VLA pretraining makes it workable with tens rather than thousands of demonstrations, is the contribution. What the Paper Does Not Yet Establish. The first limitation is observability. The authors deliberately collect strategies in which teammates and relevant workspace regions remain visible. Tasks involving hidden teammates, long occlusions, or private information would require memory, communication, active viewpoint selection, or a belief-state estimator. The paper itself excludes interactions requiring exact instantaneous synchronization. Second, role assignment is given. The robot-identifying prompt specifies both body and responsibility, so CHORUS does not negotiate roles, recover from a missing team member by reallocating work, or decide which embodiment is best suited to a newly described subtask. Third, the phrase “one policy” should primarily be read as one set of weights shared by the robots in a trained team. The experiments do not show one universal checkpoint accepting arbitrary tasks, previously unseen embodiments, and new team compositions. Every evaluated embodiment and role appears in the collaborative fine-tuning data. Fourth, the evaluation is small. The primary tasks use ten to eighteen rollouts per policy, and the three-robot result is based on ten trials. Basket, tape, and handover scores also award half credit for partial completion, so the reported mean is closer to a normalized task-completion score than a purely binary success rate. The effect sizes are large, but confidence intervals and broader environment variation would make the conclusions more secure. Fifth, parameter sharing is not free scaling. Model storage and per-robot context remain constant, but inference throughput, camera bandwidth, collision risk, and the complexity of the physical interaction still grow with the team. Four or ten robots can also create severe visual occlusion even if the transformer’s input length per robot remains unchanged. Finally, decentralized policies have a fundamental multimodality problem. If several compatible joint strategies exist but no local observation initially distinguishes them, independently sampled action distributions can choose inconsistent modes. CHORUS reduces this ambiguity through asymmetric embodiment-and-role prompts and carefully demonstrated conventions. Testing it on symmetric coordination games would reveal how much correlation can arise through visual feedback alone. As of July 29, 2026, the project page links the paper and videos but does not provide a model implementation or training repository. Reproducing the work therefore requires access to a compatible pretrained backbone, multi-robot teleoperation infrastructure, and the authors’ data formatting and deployment details. How a Robotics Team Might Adapt the Idea. For practitioners, the reusable contribution is a relatively straightforward recipe. Start with local sufficiency. Before collecting data, verify that every robot can infer its next action from its own sensors. If not, change the camera layout, collaboration strategy, or communication assumptions. Record joint episodes but train on local slices. Preserve synchronization in the raw dataset for diagnostics, then convert each trajectory into embodiment-specific observation and action chunks. This allows one robot’s valid data to remain useful even when another robot’s stream is missing or corrupted. Make identity and role explicit. Use a structured prompt or learned embodiment token rather than forcing the network to infer its body from images. For larger systems, separate embodiment, team position, and task role into independently composable conditions. Balance by effective wall-clock experience. Sampling uniformly over raw control ticks can overrepresent high-frequency robots. Weight embodiments or episodes so that the shared representation does not collapse toward the densest action stream. Test teammate reactivity, not only nominal success. Shift handover locations, delay one robot, interrupt a grasp, inject network jitter, or replace one teammate with a scripted controller. Those evaluations are more diagnostic of collaboration than repeatedly replaying nominal initial states. Treat communication as a continuum. CHORUS demonstrates that explicit communication is not always necessary, not that it is undesirable. A practical system could default to visual decentralization while allowing low-bandwidth event messages—“grasp secured,” “path blocked,” or an uncertainty signal—when local perception is insufficient. The most promising research extension would combine CHORUS-style shared VLA control with learned optional communication. The policy could act locally when confident and request a compact teammate latent only when its observation is ambiguous. Other useful directions include role-conditioned generalization to unseen team compositions, recurrent memory for occlusion, force-aware coordination, and pretraining on large collaborative datasets rather than adapting a model whose experience is overwhelmingly single-robot. The Bottom Line. CHORUS’s strongest result is not simply the 90% three-robot success rate or the 64-point advantage over from-scratch diffusion. It is the demonstration that collaborative behavior can emerge from a surprisingly ordinary foundation-model recipe: pooled local imitation data, shared weights, an embodiment-and-role prompt, and independent inference. The paper also illustrates a broader principle for embodied AI. Foundation-model priors can change which coordination mechanisms are necessary. A smaller policy trained from scratch may need explicit teammate states, shared cameras, or carefully aligned latents. A sufficiently broad VLA may already recognize the behavioral cues that those mechanisms were intended to expose. CHORUS does not yet give us arbitrary robot collectives that self-organize around new goals. It gives us something more concrete: a credible path from cross-embodiment pretraining to visually mediated, decentralized teamwork—and a strong argument that the next generation of robot foundation datasets should contain not only many bodies, but bodies acting together.