Latest / Embodied AI 101 / PhysBrain 1.0 VLA (TwinBrainVLA): Dual-Brain Vision-Language-Action with Physics-Grounded Learning
Transcript
- PhysBrain 1.0 VLA (TwinBrainVLA): Dual-Brain Vision-Language-Action with Physics-Grounded Learning. Vision-language-action (VLA) models – large-scale multimodal systems that combine visual perception, language understanding, and robot control – have become a central focus in embodied AI. By leveraging pretrained vision-language models (VLMs), researchers have trained generalist manipulation policies (e.g. PaLM-E, RT-1 variants, OpenVLA, etc.) that can follow rich instructions to perform tasks. But a persistent tension has emerged: fine-tuning a single “brain” to do both high-level reasoning and low-level motor control often leads to catastrophic forgetting of the model’s general abilities and a drift toward “visual shortcuts.” In practice, a monolithic VLA policy can end up simply exploiting correlations in its training data (e.g. associating a particular image with an action) and neglecting the linguistic command. Against this backdrop, the Phys-Brain team (DeepWisdom/Ai) has introduced a new approach called PhysBrain-1.0 VLA, built around a dual-brain architecture and a physics-grounded training strategy called LangForce. Informally dubbed “TwinBrainVLA”, this model splits the work between two cooperating modules: a frozen generalist Brain that preserves world knowledge and semantics, and a trainable specialist Brain dedicated to embodied perception and proprioception. The right (specialist) Brain can dynamically query the left (generalist) Brain for semantic guidance, then fuse that knowledge with its own sensory inputs to produce precise continuous-control actions. Meanwhile, LangForce training explicitly encourages the model to ground its actions in the language instruction, penalizing policies that ignore the command and rewarding those that explain it in their behavior. The combination – a frozen semantic “Left Brain,” a flexible sensorimotor “Right Brain,” and a Bayesian instruction-focused objective – yields a VLA model that reportedly achieves far higher success rates on standard benchmarks than previous methods. In this article we unpack the PhysBrain-1.0 VLA innovations in detail. We first review the new data and model pre-training that produced the PhysBrain foundation model. Then we explain the TwinBrainVLA architecture and how its asymmetric mixture-of-transformers (AsyMoT) mechanism fuses the two brains. Next, we describe LangForce – a Bayesian decomposition training scheme that uses latent action queries to break vision-dominated policies and enforce instruction-following. Finally, we survey the empirical performance gains and discuss what this means for generalist robot learning. Throughout, we place PhysBrain-1.0 in the context of VLA research and highlight how its ideas could influence future embodied AI. PhysBrain and Embodied Pretraining. The seed of PhysBrain-1.0 VLA is the PhysBrain vision-language model, a “universal embodied brain” trained on large-scale human egocentric video. The motivation is that most VLMs (like LLaVA, Qwen-VL, etc.) are trained on third-person images or narrated videos, which poorly match a robot’s own first-person view. Instead, PhysBrain uses thousands of hours of first-person human videos to bootstrap physical intuition into its parameters. In particular, Phys-Brain’s team developed an Egocentric2Embodiment pipeline to automatically turn raw first-person videos into structured supervision involving spatial, temporal and physical questions. Over millions of video clips, they generated multi-level VQA tasks (e.g. hand trajectories, object affordances, force dynamics) that capture the causal structure of manipulation. This yields the E2E-3M dataset (about 3 million QA instances across household, industrial and lab domains) which they use to finetune a large VLM. The result is PhysBrain: an egocentric-trained VLM whose representations encode intuitive physics, interactions, and hand-centric observations. Crucially, PhysBrain is intended as a foundational embodied model. After this egocentric pretraining, the model shows much better performance on tasks requiring reasoning about sequences of actions, physical changes, and goal-oriented behaviors. For example, on specialized benchmarks like EgoThink (multi-step reasoning) and EgoPlan, a 8B-parameter PhysBrain model dramatically outperformed previous generalist VLMs. This confirms that large-scale human video can indeed endow the model with a richer notion of “state changes” and “physical commonsense.”. The PhysBrain model is then the starting point for robot policy learning. In effect, PhysBrain serves as a high-level perceptual “System 2” able to interpret complex instructions and visual context. The remaining question is how best to attach it to robot control. If one simply fine-tunes PhysBrain on manipulation data (the usual approach), the model tends to forget its breadth of world knowledge and become a narrow imitation policy. Phys-Brain’s insight is that the way we adapt PhysBrain must preserve its inherent embodied intelligence. This drives the architecture and training innovations described below. The Limits of Monolithic VLA and Vision Shortcuts. Before diving into the new approach, it’s worth crystallizing the two core problems it tackles. First is catastrophic forgetting / genericity loss: when you fine-tune or adapt a large VLM on a narrow robotics task, it often “unlearns” the general semantic capabilities that made it useful. In effect, features that were once universal become over-specialized to whatever dataset you have. This hurts open-vocabulary understanding – for instance, a robot might no longer recognize objects it wasn’t explicitly trained on or might ignore complex language cues. Second is instruction collapse/vision shortcut: many robot imitation datasets have deterministic visuals (e.g. a single block on a table with one bright goal marker); as a result, the model learns to ignore the textual instruction and just react to the scene. In information-theoretic terms, the mutual information between the language command and the action vanishes when the visual input alone already predicts the behavior. The model becomes a “vision-only policy” and fails badly when instructions truly matter (especially in out-of-distribution scenarios). Both problems stem from the classical VLA setup, which trains a single monolithic transformer (like a multimodal GPT) to map vision+language to actions. As the authors of TwinBrainVLA point out, this fixed-backbone approach creates a critical tension between high-level reasoning and low-level skills. If the backbone is updated to fit fine-grained motor control, its semantic understanding can collapse. If you freeze it to preserve semantics, the policy loses dexterity. In practice, researchers have seen that a robot policy fine-tuned on imitation often “memorizes” a few trajectories instead of truly planning. Visual quirks (lighting, background) or spurious correlations can mislead it, and it will “cheat” by associating the wrong features with a task. Thus, any solution must address both ends of this spectrum: it must allow learning precise motor commands while not losing the broad visual-language grounding. And it must explicitly discourage the model from bypassing instructions via naïve visual heuristics. The PhysBrain-1.0 VLA approach does so by splitting the responsibilities (two brains) and by augmenting the training loss (LangForce), as we now explain. TwinBrainVLA: The Dual-Brain Architecture. To reconcile generalization and specialization, PhysBrain-1.0 introduces TwinBrainVLA, a dual-transformer architecture inspired by one part by neuroscience (“left vs right brain”) and one part by modular design. In TwinBrainVLA there are literally two VLMs operating in parallel (see Figure 1). One is the Left Brain: a large pretrained vision-language model (e.g. a frozen 8B-parameter VLM) kept fixed during robot learning. This brain retains all the general semantic and reasoning power of its pretraining. The other is the Right Brain: a smaller, trainable model specialized for the robot’s embodiment. It takes in the same visual input plus proprioceptive sensors (joint angles, velocities, etc.) and is fine-tuned to perceive the world from the robot’s perspective. Crucially, the Right Brain does not learn language-vision grounding from scratch; instead it can query the Left Brain whenever it needs semantic context. In the authors’ words, “TwinBrainVLA synergizes a frozen ‘Left Brain’, which retains robust general visual reasoning, with a trainable ‘Right Brain’, specialized for embodied perception”. The Right Brain has its own transformer layers, but at each layer it can attend to the Left Brain’s hidden representations through a mechanism called Asymmetric Mixture-of-Transformers (AsyMoT). Practically, this means the Right Brain will inject “queries” into the frozen Left Brain: it asks the Left Brain for information about the visual scene or the instruction. The Left Brain responds with keys/values from its own transformer layers. The Right Brain then fuses that semantic signal with its own proprioceptive features. As a result, the control policy is conditioned on both worlds: the Left Brain guides it with high-level semantics (what objects are, where to look, what instructions say) while the Right Brain grounds it in robot space (where its arm is, which joints to move). The final action is generated by a separate Flow-Matching Action Expert that takes the combined conditioning and outputs continuous controls. (In practice the authors implement this expert as a diffusion-based transformer that predicts velocity flows, but the key point is it’s a learned dynamical model, not a frozen module.). By design, TwinBrainVLA avoids catastrophic forgetting. The Left Brain never changes – it remains a repository of general vision-language knowledge. The Right Brain learns the new task (and can overfit), but its interface to semantics is via the still-faithful Left Brain. Thus if the Right Brain drifts, the Left Brain still “remembers” what the instruction meant in general. And because the Right Brain is smaller, it can focus on sensorimotor details without having to relearn common-sense physics. The two brains are “asymmetric” in that the Left Brain does not learn from the Right (it has no gradients flowing back), while the Right Brain learns how to optimally use the Left Brain’s outputs. Critically, this architecture explicitly targets the vision-shortcut issue as well. In a single-network VLA, the model might learn a degenerate mapping from image+lang to action that relies almost entirely on vision. In TwinBrainVLA, by contrast, the Right Brain depends on querying the Left Brain for semantics. If the action were determined purely by the image, the Right Brain would learn to ignore the Left Brain outputs (its queries would add nothing). But because the architecture mixes them, the model is enticed to use the language-derived knowledge. In effect, the Left Brain acts like a fixed “oracle” of world understanding which the Right Brain has to consult to fully solve the task. In practice, the authors report that this twin setup dramatically improves performance. In simulated benchmarks (SimplerEnv and RoboCasa), TwinBrainVLA policies far outperformed earlier VLA baselines, despite using the same underlying data. In other words, letting the Left Brain retain general knowledge paid off: the robot could handle more varied tasks without forgetting what objects looked like or how language cues worked. At the same time, the Right Brain provided the fine motor tuning to execute precise placements and grasps, conditioned on the Left Brain’s cues. The result is a “best of both worlds” – high-level semantic understanding and low-level dexterity in one model. LangForce: Physics-Grounded Instruction Following. The dual-brain design tackles the architectural side of the problem, but the PhysBrain team also introduced a novel training algorithm to align vision and language. They call this method LangForce (short for “Language Forces”), which they describe as Bayesian decomposition via latent action queries. In lay terms, LangForce penalizes the model whenever it executes an action that could equally well be predicted without the instruction. It explicitly rewards actions that can only be justified by understanding the language input. Concretely, LangForce splits the policy into two hypothetical branches during training. One branch is a vision-only prior (p(a\mid v)), which tries to predict actions based on the visual input alone. The other branch is a vision+language posterior (\pi(a\mid v,\ell)), which has full access to the language command (\ell). During training, the model learns both: it computes what action it would do if it only saw the vision, and what action it does when it also hears the instruction. Then it imposes an objective to maximize the pointwise mutual information between the chosen action and the instruction. Intuitively, this means it pushes the model to only choose actions that make the instruction informative – “tell me how this action explains the command.”. Put differently, LangForce rewards the policy for following instructions and penalizes any “shortcut” where the vision alone suffices to pick an action. The paper frames this as avoiding “information collapse” (where commands carry zero extra info) by maximizing ( \mathrm{PMI}(a;\ell\mid v) ). Operationally, one can view it as “showing” the model two worlds: one where the instruction is obeyed and one where it is ignored, and forcing it to prefer the obeying outcomes. This is implemented with an additional loss that compares the log-probabilities of the posterior versus prior and pushes them apart. Importantly, LangForce does not require collecting new data or real robot experiments – it is purely an adjustment to the loss on existing vision-language-action datasets. In practice, adding LangForce during training caused large improvements on out-of-distribution tests. The authors note an 11.3% jump on the hardest SimplerEnv benchmarks under this regime, meaning the robot learned to truly “listen” to the human instruction instead of taking the lazy route. Lian et al. report that with LangForce, policies cease to blindly map scenes to actions; instead, they produce behaviors that satisfy the language command (for example, always picking the object that matches the description in the instruction). In sum, LangForce can be seen as a physics- or causality-grounded training discipline: it enforces the idea that the language command should cause a change in the action distribution. By decomposing vision vs vision+language, the model becomes aware of when language actually matters. When combined with TwinBrainVLA, LangForce ensures that the rich semantic signals available in the Left Brain are actually used by the policy. Put another way, the Left Brain contains general world knowledge, and LangForce forces the Right Brain to consult that knowledge instructively when deciding how to move. Putting It All Together: PhysBrain-1.0 VLA. By now we have three pieces: (1) PhysBrain – a VLM pretrained on first-person human data; (2) TwinBrainVLA – a dual-transformer fusion architecture; and (3) LangForce – a training regimen that grounds actions in language. The Phys-Brain team combines them into a unified system. In practice they start with the pretrained PhysBrain model (e.g. 4B or 8B parameters) which has already seen millions of minutes of human video. They then attach or initialize the Right Brain and the Flow-Matching expert and train these (with LangForce) on multitask manipulation data from simulators (SimplerEnv, RoboCasa scenarios, etc.). An alternative way to view this is as a dual-system agent: PhysBrain is the “System 2” reasoning module, and the Flow-Matching diffusion expert is “System 1” motor control. In traditional terminology, one could call the Left and Right Brains two “experts” or “moEs” (mixture of experts), but they are asymmetric: only the Right side is updated. The Flow-Matching diffusion generation (the action expert) always uses the fused representation as conditioning. Effectively, an instruction is processed by PhysBrain to form a joint vision-language embedding; the Right Brain also processes the same image plus proprioceptive state and queries PhysBrain for semantic cues; and the Flow policy consumes both to produce a trajectory. Remarkably, this system works with far less robotic data than earlier approaches. Because the Left Brain is already knowledgeable and the policy is bootstrapped on human-like reasoning, it needs only thousands of hours of first-person video plus modest simulation fine-tuning to reach state-of-the-art. By contrast, before PhysBrain, achieving similar performance often required tens of thousands of hours of real robot demonstrations or proprietary human teleoperation. The quantitative gains are substantial. In the authors’ experiments on SimplerEnv (a suite of everyday pick-and-place tasks), PhysBrain-1.0 VLA achieved around 67–80% success rates on average, whereas prior baselines (like the Roblox Pi0.5 model) were in the 30–57% range. In fact, press reports claim the new model hit ~80% average success (versus ~57% for the previous best) – a dramatic leap. Even on RoboCasa365 (which includes diverse long-horizon tasks), the TwinBrain/LangForce model won top marks. These successes are achieved without massively scaling up the number of robot demonstrations; instead, the intelligence comes from the architecture and training. For example, the PhysBrain paper reports that an 8B-parameter PhysBrain model fine-tuned under a similar dual-system (PhysVLA) setup got 67.4% success on SimplerEnv, essentially matching RoboBrain 2.5’s 67.6% score – but the latter required a large ensemble and more data. With the full twin-brain + LangForce treatment, the new results go even further. In short, the wide gap in performance isn’t from clever tuning or extra data – it’s from re-structuring the model and loss. Finally, we note that everything in PhysBrain-1.0 VLA is open-sourced (per the company’s announcement). The team has released the PhysBrain architecture and the Egocentric2Embodiment processing code, and they intend to publish even the raw data collection tools. This transparency is meant to help the community build on their foundation. Already one can see that PhysBrain establishes a new bar: a generalist embodied model that ‘understands’ instructions with real-world physics, rather than blindly mimicking actions. Implications and Context. For robotics: The primary takeaway is that combining large-scale common-sense knowledge with task-specific skill learning can dramatically boost performance. The dual-brain idea explicitly acknowledges that different parts of the network must do very different jobs. This reflects an old insight in robotics and AI: separate perception/understanding from control. TwinBrainVLA modernizes that idea within a deep-learning paradigm. In practice, a robot with PhysBrain-1.0 VLA can handle novel objects or complicated language while still moving its limbs smoothly – something monolithic end-to-end policies struggled with. It also means we can leverage cheap or readily-available human video data to train robots, sidestepping expensive physical data collection. For embodied AI research: PhysBrain-1.0 VLA shows that structural biases and training objectives (like LangForce) matter as much as model size or data scale. Just making a model bigger or giving it more trials in the simulator isn’t enough if it still “shortcuts” the logic. The Bayesian training perspective behind LangForce – decomposing vision vs language influences on actions – may inspire similar approaches. We might see more work on mutual-information losses or counterfactual action modeling to ensure multi-modal alignment. For general multi-modal learning: The frozen-vs-trainable brain split has parallels to ideas like “mixture of experts” and “learning without forgetting”. It suggests a future where we preserve one part of a network as a stable knowledge repository while continuously re-train another part for new tasks. In language models, this is akin to using a static frozen LM for background knowledge and a separate fine-tuned module for specialized tasks, but here it’s done in vision-language-action space. The success of AsyMoT (asymmetric mixture-of-transformers) might encourage more designs where one transformer serves as a knowledge database that programmable modules query. Challenges: Of course, this approach adds complexity. There are now more moving parts – a two-brain system – and training is trickier. The LangForce objective itself requires careful tuning and understanding of information measures. It remains to be seen how this scales to truly open-ended environments or to real-time physical deployment. The reported results are on simulated benchmarks; real hardware will introduce noise and un-modeled physics. Also, the reliance on first-person human video is powerful but not yet a panacea; some tasks (e.g. agile dynamic manipulation) are still hard to capture purely by human observations. Despite these caveats, PhysBrain-1.0 VLA represents a significant step. It demonstrates that we can indeed build a robot “brain” that has some grasp of the physical world, not just rote action copies. By augmenting architectures and losses that encourage deep grounding of language and vision, we move closer to robots that truly understand their commands and their environment. For AI at large, it’s a creative fusion of cognitive insights and multimodal learning. We anticipate that “dual-brain” and “Bayesian instruction grounding” will become recurring themes in the next wave of embodied intelligence research. In summary: PhysBrain-1.0 VLA (a.k.a. TwinBrainVLA) is a new vision-language-action model that splits the workload into a fixed generalist Transformer and a trainable specialist, and trains with a novel LangForce objective to enforce language grounding. This dual-brain, physics-aware strategy shatters previous performance ceilings on robot benchmarks, all while using relatively little real-robot data. It exemplifies a promising new paradigm for AI: combine pretrained common-sense with specialized motor skills, and train for true instruction following. The implications are broad – it points to a future where robots learn from watching people understand the world as much as they mimic their actions. References: The architecture and results described here are drawn from the PhysBrain-1.0 VLA release by Phys-Brain (DeepWisdom), including the TwinBrainVLA model description and the PhysBrain egocentric video training pipeline. The LangForce training methodology is detailed in Lian et al. (2026). These combined have set new state-of-the-art records on SimplerEnv and RoboCasa benchmarks.