Latest / Embodied AI 101 / Two Brains Under Pressure: How UnderwaterVLA Connects Language Models to AUV Control

Transcript

Two Brains Under Pressure: How UnderwaterVLA Connects Language Models to AUV Control · Embodied AI 101

• Two Brains Under Pressure: How UnderwaterVLA Connects Language Models to AUV Control. The UnderwaterVLA dual-brain vision-language-action architecture enables robust autonomous underwater navigation • Zhangyuan Wang, Yunpeng Zhu, Yuqi Yan, Xiaoyuan Tian, Xinhao Shao, Meixuan Li, Weikun Li, Guangsheng Su, Weicheng Cui, and Dixia Fan • Westlake University, Zhejiang University, and Australian National University • Scientific Reports • 2026. The paper’s real claim. UnderwaterVLA arrives with a compelling proposition: autonomous underwater vehicles should not ask one large model to perceive the scene, understand a mission, plan over minutes, react within seconds, and stabilize a hydrodynamically complex platform. Instead, the system divides intelligence into two layers. A large cloud-side “brain” handles strategic reasoning and mission decomposition. A smaller onboard “cerebellum” interprets the current scene and chooses short-horizon actions. Beneath both sits a physics-informed model predictive controller that converts symbolic decisions into bounded thruster commands. The journal article was published on July 28, 2026, following an arXiv preprint released on September 26, 2025. The authors frame it as the first vision-language-action architecture designed specifically for autonomous underwater vehicles, addressing limited communication, degraded vision, expensive data collection, and fluid disturbances within one integrated system. But the most useful way to understand UnderwaterVLA is slightly different from the paper’s headline. This is not an underwater version of an end-to-end action-token policy such as OpenVLA. It is a hierarchical autonomy stack in which pretrained vision-language models communicate through a small, structured action language, while a conventional optimizer remains responsible for physical execution. The novelty lies less in a new neural architecture than in the interfaces between semantic reasoning, reactive visual control, and marine dynamics. That distinction matters. UnderwaterVLA offers an argument about where foundation models belong inside a safety-critical robot, not merely an argument that a larger model can learn a better policy. Why going underwater changes the VLA problem. Many terrestrial VLA systems assume that the robot can run a large policy locally or maintain a reliable connection to substantial compute. They also benefit from environments where cameras provide relatively stable observations, radio communication is inexpensive, and low-level motion can be learned from large demonstration corpora. An AUV inherits almost none of those conveniences. Optical observations can be degraded by attenuation, color shift, backscatter, nonuniform illumination, suspended sediment, and biological matter. Sonar reaches farther but trades visual texture for noisier and lower-resolution geometric evidence. Meanwhile, vehicle motion is affected by added mass, nonlinear drag, currents, turbulence, and the coupling between body motion and the surrounding water. Conventional radio communication is ineffective at useful underwater distances, while acoustic communication is slow, noisy, and bandwidth constrained. These constraints create three different time scales. At the slowest time scale, the robot must interpret a mission such as inspecting infrastructure, locating a target, or approaching an object while maintaining a safe standoff distance. This is a semantic and long-horizon planning problem. At an intermediate time scale, it must decide whether the current observation means “continue,” “turn,” “slow down,” “the subgoal is complete,” or “the target has been lost.” This is a perceptual and reactive reasoning problem. At the fastest time scale, it must regulate velocities and thruster forces despite drag and inertia. This is a feedback-control problem. Trying to solve all three with one generative model would expose the vehicle to model latency, communication outages, action hallucinations, and distribution shift in the dynamics. UnderwaterVLA therefore makes a conservative systems decision: use foundation models for the first two time scales and leave the fastest loop to model-based control. This division also reframes the data problem. Instead of collecting enough underwater demonstrations to train a policy covering every target, obstacle arrangement, turbidity level, and fluid condition, the authors reuse pretrained multimodal models for visual and linguistic reasoning. They then use prompting and a physics-based controller to adapt those models to an AUV. The paper calls this a “zero-data” methodology, meaning that no task-specific underwater demonstration set is used to train the navigation policy. That is an attractive proposition, although—as we will discuss later—“zero underwater demonstrations” is a more precise description than “zero data.”. Inside the dual-brain architecture. The high-level component is the cloud brain, implemented with QVQ-MAX. The architecture diagram identifies the large QVQ-MAX-72B model, while the onboard component uses Qwen 2.5-VL-7B. The cloud brain receives a natural-language mission and visual context, reasons about the environment, and produces an ordered list of coarse subgoals. It is intended to operate intermittently, particularly during surfacing events or other communication opportunities, rather than remain continuously connected throughout the dive. Suppose the mission is to reach a visually identified target while avoiding an intervening obstacle. The cloud brain might produce a plan consisting of an initial alignment maneuver, forward progress past the obstacle, a fine approach, and a final hold-position phase. These are not waypoints expressed as exact coordinates. They are semantically meaningful steps that the local system can reinterpret as conditions change. That last point is central. The cloud brain proposes a plan, but the local brain is not required to execute every step mechanically. The onboard cerebellum receives the active subgoal and the latest visual observation. It runs a closed-loop perception-action process, selecting a direction, a discrete speed level, and task-status flags. Its output is serialized as structured JSON with fields such as , , , , and. A typical decision might indicate that an obstacle is visible on one side, that the vehicle should turn in the opposite direction at medium speed, and that the current subtask remains incomplete. The JSON schema is more than a logging convenience. It establishes a typed contract between an open-ended generative model and a deterministic controller. The local model cannot directly invent arbitrary thruster values. It must choose from a constrained vocabulary that the downstream stack understands. This is one of the strongest architectural ideas in the paper. Foundation models are good at reasoning over ambiguous visual and linguistic context, but their outputs are not naturally bounded, continuous, or safety certified. A finite action interface limits the consequences of a malformed or surprising generation. The controller knows what “forward at low speed” means, even if the language model does not understand the exact thrust allocation needed to realize it. The authors also prompt both brains to emit explicit chain-of-thought-style rationales. At the cloud level, the rationale explains why the mission has been decomposed into a particular sequence. At the local level, it records the visual evidence associated with each action and completion decision. The stated goals are interpretability, runtime monitoring, and easier failure diagnosis. Operationally, these rationales are best treated as diagnostic annotations rather than faithful explanations of the models’ internal computations. A generated rationale can still be post hoc or incomplete. Nevertheless, a structured trace connecting observation, declared evidence, selected action, and task status is useful when reconstructing why a robot turned, stopped, or continued. The action contract is the architecture. Seen from a robotics perspective, the most important component may not be either language model. It is the narrow interface between them and the controller. The cloud brain communicates through subgoals. The local model communicates through direction, speed, and termination signals. The MPC communicates through physically meaningful thrust commands. Each boundary removes detail that the next layer does not need. That separation makes the stack easier to modify. The cloud model could be replaced without retuning the hydrodynamic controller, provided it continues to produce compatible subgoals. The onboard model could be quantized or fine-tuned without changing the mission language. And the same semantic action vocabulary could drive a different AUV after replacing the vehicle model and thruster-allocation matrix. It also localizes failure. If cloud communication disappears, the vehicle can continue working on the plan it already has. If the local model chooses a poor action, that action is still bounded in speed and duration. If the hydrodynamic parameters change, the controller can adapt without asking either vision-language model to relearn fluid mechanics. The separation does not eliminate failures—a confidently wrong local decision remains dangerous—but it produces clearer places to detect and contain them. The physics bridge: from words to thrust. The MPC receives a discrete command consisting of a direction and a speed level. Translational commands include forward and backward motion, while rotational commands include left and right turns. The paper assigns target translational speeds of 0.2, 0.5, and 0.8 meters per second for low, medium, and high speed. The corresponding angular-rate targets are 0.5, 1.0, and 1.5 radians per second. Every command becomes a one-second motion primitive. During the first 0.2 seconds, the reference speed ramps up. It remains at the requested level until half a second, then decreases over the remainder of the interval. The MPC solves its control problem at 20-millisecond intervals, corresponding to a 50-hertz low-level loop, and applies active braking after the primitive until residual translational or angular motion drops below a small threshold. This action chunking provides a clean temporal boundary. The local VLM does not need to produce controls at motor frequency. It only needs to make another perceptual decision after a bounded, predictable piece of motion. The controller penalizes tracking error and control effort while accounting for hydrodynamic drag. Translation and yaw use quadratic drag models, meaning resistance grows with the velocity magnitude multiplied by velocity. The associated drag coefficients are estimated online using commanded thrust, known mass or inertia, and acceleration measurements. Added-mass effects are considered during acceleration, quadratic drag during the constant-speed phase, and natural fluid damping during deceleration. This is what makes the method “zero-data” at the action layer. The system does not train a neural controller to imitate underwater trajectories. It generates motion from a simplified physical model and updates drag estimates during execution. There is an important trade-off here. The controller is interpretable and easy to retarget, but the model described in the paper is much simpler than full marine vehicle dynamics. The demonstrated action space is largely planar, separating surge-like translation from yaw-like rotation. The formulation does not demonstrate general six-degree-of-freedom motion, depth-changing maneuvers, coupled roll and pitch effects, or explicit current-relative velocity estimation. Online drag estimation also depends on trustworthy thrust and acceleration measurements. Close to zero velocity, estimating a quadratic coefficient from velocity-dependent terms becomes numerically delicate, and the paper does not detail regularization, filtering, or observability safeguards. In open water, an unmodeled current could be absorbed into the estimated coefficient even though drag and ambient flow are physically different phenomena. Those are not fatal flaws for a laboratory proof of concept. They do, however, define the distance between this controller and one ready for long-duration ocean deployment. What the experiments actually show. The clearest qualitative experiment takes place in a laboratory tank containing three vertical cylinders. The vehicle begins from a predefined pose and is instructed to reach the rightmost—and therefore farthest—cylinder without colliding with the middle one. The cloud brain interprets the far cylinder as the target and the middle cylinder as an obstacle. It generates an initial rightward detour followed by forward progress and a precise final approach. The local Qwen model first commands a medium-speed clockwise turn, then a medium-speed forward movement, then a slower forward approach. Once visual feedback indicates that the target vicinity has been reached, it commands the vehicle to hold position and marks the mission complete. The sequence is simple, but it demonstrates the intended division of labor. The cloud brain decides that a detour is necessary. The local brain repeatedly grounds that plan in the latest image. The MPC turns each discrete command into a one-second trajectory and stops the vehicle before the next decision. The paper then reports four quantitative comparisons against results published by QUAR-VLA, a VLA system originally developed for quadruped robots:. Letter-distinction accuracy is reported as 51 successes in 60 trials, or 85%, compared with a 66% baseline. “Go to object” succeeds in four of five trials, or 80%, compared with 60%. Tunnel traversal succeeds in four of five trials, or 80%, compared with 53%. Object approach with obstacle avoidance succeeds in three of five trials, or 60%, compared with 41%. These yield the headline gains of 19 to 27 percentage points. The table also contrasts UnderwaterVLA’s zero task-specific training demonstrations with 262,000 samples attributed to the baseline. A separate experiment tests visual degradation. The authors reduce illumination and introduce diatomaceous earth with particle sizes from 5 to 200 micrometers, increasing turbidity from 0.5 to 18 nephelometric turbidity units. They report a 65% loss of image contrast at a distance of 1.5 meters. Under these conditions, the cloud brain initially proposes four stages: lateral alignment, medium-speed forward motion, fine pose adjustment, and final approach while maintaining a safe distance. After the first stages, the local brain performs small right and left corrections, judges that the vehicle has already reached an acceptable standoff distance, and terminates the mission without explicitly executing the final planned step. The comparison single-brain system behaves differently. After losing visual contact with the target, it continues its last valid forward command, overshoots the goal region, and violates the intended safety margin. The authors interpret this as evidence that hierarchical subgoal reasoning plus local completion logic degrades more gracefully than a monolithic policy under poor visibility. This adaptive mission closure is arguably the paper’s most interesting experimental behavior. The local system is not merely selecting a direction. It is deciding whether the semantic objective behind a planned step has already been satisfied. What is genuinely important here. UnderwaterVLA demonstrates a useful alternative to the dominant “more demonstrations, larger policy” trajectory in embodied AI. Rather than learning every layer of behavior jointly, it combines pretrained semantic competence with an engineered action abstraction and a physical controller. For underwater robotics, that may be a particularly sensible compromise. Demonstration collection is expensive, hardware is difficult to recover after failure, and many mission-level instructions do not require learning an entirely new feedback law. If a foundation model can recognize the relevant object and decide which side is safe, there is little reason to ask it to rediscover quadratic drag compensation from demonstrations. The dual-brain split is also communication-aware. Long-horizon reasoning can occur during occasional high-bandwidth contact, while the onboard system retains enough visual reasoning to continue autonomously. This is more realistic than assuming that every camera frame can be transmitted to a remote model. Finally, the work treats termination as an intelligent action. A large fraction of robot failures occur not because the platform cannot move toward a goal, but because it cannot reliably determine when to slow down, stop, replan, or abandon an obsolete instruction. UnderwaterVLA gives the local layer explicit authority to close a subtask based on current evidence. That authority is valuable—but it also makes evaluation of the local model’s false-positive and false-negative completion decisions essential. Where the evidence remains weak. The headline percentages should be interpreted cautiously. QUAR-VLA is a quadruped-robot system trained on the QUARD dataset, covering perception, navigation, terrain locomotion, and whole-body manipulation. Its paper reports thousands of evaluation trials and a learned QUART policy. UnderwaterVLA imports selected QUAR-VLA success rates as its baseline even though the embodiments, environments, controllers, training regimes, and simulation-versus-real conditions differ. The UnderwaterVLA paper acknowledges that the baseline numbers come from simulation while its own results come from physical trials. But that does not make the comparison more conservative in a scientifically controlled sense; it makes it less matched. The reported 19-to-27-point gains cannot be attributed cleanly to the dual-brain architecture because the two systems were not evaluated on the same robot, in the same environment, with the same observations and action space. The sample sizes are another issue. Three navigation categories use only five trials. One additional success changes the measured rate by 20 percentage points. That is approximately the same magnitude as several claimed improvements. Without confidence intervals, repeated environments, or significance analysis, the results establish feasibility rather than reliable effect size. The degraded-visibility experiment is valuable but primarily qualitative. It compares one dual-brain trajectory with a single-brain failure mode, yet the paper does not provide a large matched trial set over randomized turbidity, illumination, obstacle placement, and target distance. Nor does it fully specify the single-brain baseline’s model, prompting, temporal context, and termination policy. As a result, we cannot tell how much of the difference comes from hierarchy, prompt design, memory, or an explicitly added stop condition. The phrase “field tests” in the abstract should also be read against the experimental description. The demonstrated environments are controlled tanks, including a tank with artificially induced marine degradation. The conclusion lists multimodal fusion, extreme-depth operation, long-duration missions, and full-scale open-ocean deployment as future work. The paper therefore does not yet test waves, persistent currents, biological clutter, acoustic-link dropout, depth changes, long-term localization drift, pressure effects, or biofouling. Compute is a further missing variable. Running a 7-billion-parameter VLM onboard an AUV raises questions about accelerator hardware, quantization, latency, thermal management, battery draw, and inference frequency. The paper specifies a 50-hertz MPC loop but does not report the local model’s time to decision, worst-case latency, or how perception and control are scheduled when generation exceeds one motion cycle. Similarly, the communication argument would be stronger with measured bandwidth and outage experiments. How large is the cloud-generated plan? How often must it be refreshed? What happens when the local model determines that the entire plan is invalid rather than merely completing a subtask early? Can it request replanning over a low-rate acoustic channel, surface autonomously, or fall back to a conventional mission script?. “Zero-data” also needs qualification. UnderwaterVLA uses zero task-specific underwater demonstrations, which is meaningful. But QVQ-MAX and Qwen 2.5-VL are pretrained on substantial upstream corpora, and the overall system relies on human-designed prompts, an action ontology, a vehicle model, mass and inertia estimates, a thruster mapping, controller weights, and online measurements. The result is better characterized as zero-shot transfer plus physics-based control, not learning without data. There is also a safety asymmetry in the architecture. MPC can ensure that a requested turn is smooth and bounded, but it cannot determine that turning is semantically wrong. If the local VLM hallucinates free space or prematurely marks a task complete, the controller will execute that decision accurately. A deployable version therefore needs independent collision checking, depth and geofence constraints, command validation, stale-observation detection, and deterministic abort behavior. Finally, generated reasoning should not be confused with verification. A sentence explaining why the right side is safe does not prove that the perception is correct. For safety-critical use, rationales should be paired with inspectable evidence: detections, estimated ranges, visibility confidence, sonar returns, state-estimator uncertainty, and explicit satisfaction of geometric constraints. Where UnderwaterVLA sits among related systems. OpenVLA represents one end of the VLA design space. It is a 7-billion-parameter open model trained on roughly 970,000 real-world robot demonstrations, using vision and language to generate low-level manipulation actions. Adaptation generally means fine-tuning the policy for a new embodiment or task distribution. UnderwaterVLA takes nearly the opposite route. Its pretrained models are used as prompted semantic reasoners, while the final action is produced by a hand-designed symbolic interface and MPC. It sacrifices end-to-end differentiability in exchange for data efficiency, modularity, and more explicit physical constraints. QUAR-VLA is the closest named experimental baseline, but it is also a useful contrast. QUAR-VLA introduced the QUART family and the QUARD dataset for quadruped perception, navigation, locomotion, and manipulation. It seeks unified learned policies across many tasks; UnderwaterVLA seeks a reusable reasoning-and-control hierarchy with no underwater policy training. UAV-VLA is conceptually closer to UnderwaterVLA’s cloud brain. It combines satellite imagery, visual-language reasoning, and language instructions to generate aerial flight paths and mission actions. Both systems use foundation models mainly for spatial and semantic mission planning rather than direct high-frequency motor control. OceanGym provides another important reference point. It offers an 800-by-800-meter simulated underwater environment with varied seabed, wreck, pipeline, and energy-infrastructure scenarios. Its agents must reason over optical and sonar observations, maintain memory, and complete long-horizon tasks such as searching, inspection, and docking. The reported gap between multimodal agents and human performance suggests that underwater embodied reasoning remains far from solved. A strong next evaluation for UnderwaterVLA would place its hierarchy inside such a benchmark, compare it against monolithic agents using identical sensors and action primitives, and then validate the same policy structure on a physical AUV. How I would adapt the design. For teams building on this work, the first lesson is to preserve the separation of responsibilities. Let the mission model reason about objects, intent, ordering, and contingency. Let the local model reason about recent observations and subgoal status. Do not let either model send unrestricted actuator values. The interface should be even stricter than the paper’s JSON example. Each command should have bounded parameters, an expiry time, a maximum travel envelope, explicit preconditions, and separate predicates for success, failure, and uncertainty. Malformed output should trigger a hold or controlled retreat rather than an attempt to infer what the model probably meant. The local layer should also be wrapped by a deterministic executive. That executive would validate the VLM’s action against obstacle range, depth limits, remaining energy, localization uncertainty, and vehicle health. It should detect repeated oscillatory commands, stale visual observations, impossible completion claims, and disagreement between optical and acoustic sensors. For perception, adding sonar is the obvious next step. The local model could receive a compact geometric representation derived from forward-looking sonar alongside the camera image, rather than raw acoustic data alone. A visibility estimator could determine when optical evidence is trustworthy and when the controller should switch to a sonar-dominant mode. The dynamic layer should expand from decoupled planar primitives toward a six-degree-of-freedom model with current estimation and actuator constraints. The original one-second action chunks remain useful, but they could parameterize velocity, heading, depth, and standoff objectives rather than only forward-backward motion and yaw. A disturbance observer could distinguish ambient flow from changes in vehicle drag. Evaluation should then isolate each contribution on the same platform: dual brain versus single brain, structured subgoals versus direct commands, rationale prompting versus concise outputs, fixed versus online hydrodynamic parameters, and MPC versus a matched conventional controller. Hundreds of randomized trials would make the success-rate claims substantially more credible. Most importantly, testing should include safe failure, not just successful navigation. The vehicle should be evaluated on contradictory instructions, invisible targets, unexpected obstacles, communication loss, model timeouts, corrupted images, localization jumps, and goals that become physically unreachable. Bottom line. UnderwaterVLA does not yet demonstrate a general-purpose, language-driven AUV capable of independent ocean missions. Its experiments are small, its quantitative baseline is not well matched, and its physical validation remains in controlled tanks. But the paper contains a valuable systems insight: foundation models do not need to replace the classical autonomy stack to improve it. The cloud model contributes long-horizon semantic decomposition. The onboard model contributes context-sensitive visual decisions and task closure. The MPC contributes predictable timing, bounded motion, and hydrodynamic compensation. Their shared action contract keeps those capabilities modular. That may be the most practical route for bringing vision-language models into underwater robotics. Not one enormous policy trained to imitate every possible dive, but a layered system in which language models reason where semantics matter, physics governs where forces matter, and carefully designed interfaces keep the two from confusing their responsibilities.