Latest / Embodied AI 101 / Let the VLM Decide, Let the Planner Drive: Inside Senna

Transcript

Let the VLM Decide, Let the Planner Drive: Inside Senna · Embodied AI 101

• Let the VLM Decide, Let the Planner Drive: Inside Senna. Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving • Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang • Huazhong University of Science and Technology and Horizon Robotics • International Journal of Computer Vision, Volume 134, Article 407 • 2026. A brief note on versions. Senna first appeared as an arXiv preprint on October 29, 2024. The peer-reviewed journal version was published on August 25, 2026. That distinction matters because the final IJCV abstract reports larger cross-dataset transfer gains—35.59% lower planning error and 38.89% lower collision rate—than the public arXiv v1 figures of 27.12% and 33.33%. The journal version also explicitly reports both open-loop and closed-loop evaluation, while the detailed public tables in arXiv v1 focus primarily on open-loop experiments. In this episode, I will use the final journal record for publication-level claims and the public manuscript for the detailed architectural, ablation, and numerical analysis. The problem Senna is actually trying to solve. The broad motivation sounds familiar: end-to-end driving models are good at turning large quantities of sensor data into trajectories, while large vision-language models are good at describing scenes, recalling commonsense knowledge, and reasoning about unusual situations. Therefore, combine them. But Senna is interesting because it does not stop at that superficial argument. It asks a more specific interface-design question: what information should pass from the vision-language model to the driving policy?. That question matters because the two model families have very different inductive biases. A conventional end-to-end planner is optimized for continuous geometry. It can learn where a trajectory should bend, how rapidly it should change speed, and how that motion relates to lanes, agents, and vehicle dynamics. Its weakness is semantic extrapolation. A model trained mostly on ordinary urban traffic may recognize pixels and geometry correctly yet still respond poorly to an unusual arrangement—a truck carrying traffic cones, an unconventional construction zone, or an ambiguous interaction involving pedestrians. A large vision-language model has almost the opposite profile. It can often explain that a pedestrian near a crosswalk deserves caution, that cones loaded on a truck are not necessarily blocking the road, or that an emergency vehicle changes the meaning of an otherwise ordinary scene. Yet asking an autoregressive language model to emit precise metric waypoints is unnatural. Coordinates are token strings, minor numerical errors can become major trajectory errors, and ordinary language-model pretraining provides no reason to expect centimeter-level consistency. Senna’s authors therefore argue that using a VLM as a numerical trajectory generator wastes its strongest capabilities while exposing one of its largest weaknesses. Earlier hybrid systems such as DriveVLM tried to reduce this mismatch by asking the VLM for a sparse, low-frequency trajectory and then letting a conventional end-to-end model refine it. That is already better than expecting the language model to produce the final control stream, but it still requires the VLM to predict numbers. Senna replaces that numerical bridge with a semantic one. The resulting design is closer to a hierarchical policy. Senna-VLM chooses a high-level maneuver, while Senna-E2E executes it as a precise trajectory. In hierarchical reinforcement-learning language, the VLM selects something resembling an option and the lower-level policy realizes that option in continuous space. The central thesis is simple:. Give each model the part of driving that matches the representation it already handles well. That is the paper’s real contribution—not merely adding a language model to an autonomous-driving stack, but deciding where language should end and geometry should begin. A semantic bottleneck between reasoning and motion. Senna’s interface is a set of formatted meta-actions. Each action combines a lateral decision with a longitudinal one. The lateral vocabulary contains left, straight, and right. The longitudinal vocabulary contains accelerate, keep, decelerate, and stop. A complete decision might therefore be “accelerate and turn right,” “decelerate and continue straight,” or “stop and continue straight.” The labels are generated from the ego vehicle’s ground-truth future motion: lateral displacement determines the path category, while speed change over the prediction horizon determines the longitudinal category. Senna-VLM emits the decision in natural language. A meta-action encoder then maps the formatted action to a learnable embedding, which is supplied to Senna-E2E alongside images and the navigation command. This is essentially a one-to-one lookup from a human-readable action category to a high-dimensional vector. Senna-E2E never needs to parse a paragraph, and Senna-VLM never needs to generate a continuous path. This bottleneck has several attractive properties. First, it is inspectable. If the final trajectory is wrong, an engineer can ask whether the high-level decision was wrong or whether the low-level planner failed to execute a reasonable decision. Second, the action space is compact enough for reliable language-model classification. Third, it gives the trajectory planner a strong prior without prescribing the exact path. “Turn right and decelerate” constrains the solution while leaving the lower-level model free to determine curvature, clearance, and timing from geometric scene features. But the bottleneck also discards information. It cannot express “yield briefly, then merge behind the blue sedan,” “move left by half a lane to pass the obstruction,” or “maintain speed unless the cyclist enters the lane.” All variants of “decelerate, straight” share the same action embedding regardless of urgency, target speed, responsible agent, or uncertainty. That limitation becomes important later. Inside Senna-VLM. Senna-VLM is derived from the LLaVA family and uses Vicuna-v1.5-7B as its language backbone. The released configuration accepts six camera views and compresses each image to 128 visual tokens before sending them to the language model. The repository provides a 7-billion-parameter checkpoint along with training and evaluation scripts. The VLM contains four conceptual components: a vision encoder, a Driving Vision Adapter, the language model and text-tokenization path, and the meta-action encoder. The vision encoder independently extracts features from the surround-view images. Rather than building one large stitched panorama, the Driving Vision Adapter compresses the features from each camera into a fixed token budget compatible with the language model. Explicit view tags such as front, front-left, front-right, back-left, back-right, and back are placed beside the corresponding image tokens. A prompt can therefore refer specifically to the front-right camera, and the model retains a symbolic indication of where each visual observation originated. This is a small but important engineering decision. Generic VLMs tend to treat images as independent documents or as an unordered sequence. Driving requires an egocentric spatial convention. A pedestrian in a rear-side camera has a different planning implication from an identical pedestrian directly ahead. The view prompts do not create metric 3D geometry by themselves, but they give the language model a stable surround-view coordinate vocabulary. The token-compression ablation is especially revealing. With six views, performance rises as the visual budget increases from 32 to 64 and then 128 tokens per image. Meta-action accuracy reaches 71.21% at 128 tokens per view. Surprisingly, moving to 256 tokens causes accuracy to collapse to 56.13%, while configurations with 512 or 576 tokens per image fail entirely in the reported experiment. The paper treats 128 tokens as its operating point. My interpretation is that more visual tokens are not automatically more useful because they compete with prompts and generated text inside a finite context and substantially increase the optimization burden. A poorly compressed representation can omit a small traffic light; an insufficiently compressed representation can drown the language model in redundant patch features. Senna’s result is a practical reminder that multi-camera VLM design is partly an information-allocation problem, not simply a matter of preserving the maximum number of pixels. Building planning knowledge through question answering. The authors do not train Senna-VLM only on action labels. They construct six families of planning-oriented question-answer data that span perception, prediction, planning, and explanation. Scene description teaches the model to summarize road type, weather, time of day, road conditions, relevant traffic participants, and other scene-level factors. Traffic-signal detection asks for red, green, yellow, or none. Vulnerable-road-user identification asks the model to locate pedestrians, cyclists, and motorcyclists relevant to the ego vehicle. Motion-intention prediction asks about the likely behavior of surrounding agents. Meta-action planning provides the direct high-level driving target. Finally, planning explanation asks why the ground-truth ego maneuver is appropriate, with prompts encouraging attention to navigation, traffic participants, road conditions, and signal state. This is not purely human-annotated language data. Much of it can be produced automatically from trajectories, object annotations, maps, and traffic-light states. Scene descriptions and planning explanations are generated with a larger teacher VLM; the released conversion pipeline uses LLaVA-v1.6-34B for those language-heavy annotations. That division is sensible. Geometry-derived targets provide deterministic labels where possible, while the teacher model supplies the open-ended prose that is difficult to obtain programmatically. The advantage is scale. Instead of manually writing explanations for hundreds of thousands of frames, the team can transform an existing autonomous-driving corpus into instruction-tuning data. The risk is label quality. A teacher-generated explanation is necessarily post hoc: the teacher is told what the vehicle eventually did and asked to invent a plausible reason. The resulting explanation may correlate with the action without faithfully identifying its cause. A fluent statement such as “I slowed because of the pedestrian” is not proof that the trained policy actually relied on that pedestrian. The explanations are useful auxiliary supervision and a valuable diagnostic surface, but they should not be treated as causal certificates. The final journal figures also depict an inference-time consistency mechanism. Predictions from Senna-VLM and Senna-E2E concerning traffic lights, vulnerable road users, and motion intentions are compared. Where the branches disagree, confidence-bearing outputs can be used to form an enhanced result—for example, retaining the VLM’s motion-intention answer when the E2E branch has low confidence, but adopting the E2E branch’s better-localized cyclist estimate when its confidence is high. The public arXiv v1 does not expose the complete rule set for this journal-version extension, but the broader direction is notable: the two branches can act as mutual critics rather than forming a strictly one-way cascade. The three-stage curriculum. Senna-VLM is trained in three stages, and the ordering is central to the paper. In mixed pretraining, only the Driving Vision Adapter is trained, while the vision encoder and language model remain frozen. The data mix includes general LLaVA instruction-following examples and driving-scene descriptions. The purpose is to place driving visual features into the language model’s embedding space without immediately overwriting the language model’s general capabilities. In driving fine-tuning, the system switches from single-image examples to surround-view input. It trains on the planning-oriented QA tasks except meta-action planning. This stage teaches the model to perceive traffic signals, identify critical road users, reason about motion, and describe the scene before asking it to choose the ego maneuver. In planning fine-tuning, the model specializes on meta-action questions. In other words, the curriculum goes from visual alignment, to driving understanding, to action selection. The ablations support this separation. The best configuration is not generic pretraining followed by one mixed fine-tuning pass. Combining general and driving data during adapter pretraining, then separating driving understanding from final planning specialization, produces the strongest result—71.21% meta-action accuracy. Simpler mixtures remain closer to 60–65%. That result is plausible from an optimization perspective. If meta-action examples dominate too early, the model can learn shortcuts from navigation commands, ego speed, or dataset frequency rather than developing scene understanding. If every QA objective remains active during the final stage, the gradients for verbose description and compact action classification can interfere. Senna treats action prediction as the final specialization built on top of an already driving-aware representation. The public repository supports both full-parameter and LoRA fine-tuning. The authors report that full-parameter fine-tuning performs better, while recommending LoRA as a memory-constrained alternative for GPUs with roughly 24 GB of memory. Senna-E2E: turning the decision into a trajectory. The low-level component extends VADv2, a vectorized end-to-end driving model. It contains a perception module for dynamic objects and local map elements, a motion-prediction module for surrounding agents, and a planning module whose learned planning tokens attend to the scene representation. Senna adds the VLM-derived meta-action embedding as another interaction token. The navigation command remains a separate input. This distinction between navigation and behavior is useful. A route command such as “turn left” specifies where the vehicle ultimately needs to go. A meta-action such as “decelerate and turn left” specifies how it should respond to the current scene. Navigation alone cannot express that the vehicle should wait for a pedestrian before executing the turn. During training, Senna-E2E receives the ground-truth meta-action. During inference, it receives the action predicted by Senna-VLM. That makes training stable, but it introduces a familiar teacher-forcing mismatch. The planner learns under clean semantic commands and is then deployed under a noisy command distribution. The paper’s upper-bound experiment makes this mismatch visible. On DriveX, VADv2 with no meta-action produces L2 errors of 0.55, 1.46, and 3.10 meters at one, two, and three seconds. Supplying Senna’s predicted meta-actions improves those values to 0.49, 1.27, and 2.62 meters. Supplying ground-truth meta-actions improves them much further, to 0.37, 1.08, and 2.24 meters. The semantic interface clearly helps, but a considerable gap remains between predicted and oracle decisions. This gap identifies one of the most valuable directions for follow-up work: train the trajectory planner on realistic action noise, uncertain action distributions, or sampled VLM mistakes rather than only clean labels. What the experiments show. High-level decision making. On the proprietary DriveX validation set, generic VLMs perform poorly when evaluated without driving adaptation. Qwen-VL achieves 7.66% meta-action accuracy, VILA 6.95%, and LLaVA-1.5 21.34%. Fine-tuning these models on DriveX raises their accuracy to 56.38%, 63.03%, and 64.48%, respectively. Senna reaches 71.21%, a 6.73-point improvement over the strongest of those fine-tuned baselines. The per-class results show that lateral path decisions are easier than longitudinal speed decisions. Senna reports F1 scores of 95.60 for straight, 89.37 for left, and 90.09 for right. For speed behavior, the figures fall to 80.18 for keep, 58.83 for accelerate, 61.99 for decelerate, and 80.10 for stop. That asymmetry makes sense. Navigation and lane geometry strongly constrain lateral movement. The correct speed action can depend on subtler temporal cues: whether another vehicle is yielding, whether a pedestrian is about to enter the lane, or whether a distant green light will remain relevant. “Stop” is visually distinctive; choosing between keep, accelerate, and decelerate is often more ambiguous. Senna also leads the compared models on BLEU-4, CIDEr, and METEOR for scene descriptions. That confirms the effectiveness of its driving-oriented language tuning, although caption similarity should not be mistaken for planning reliability. A description can use the same nouns as the reference while missing the one causal relation that matters. Trajectory planning. The DriveX comparison is the cleanest test of the proposed interface. A reproduced DriveVLM-style model provides low-frequency trajectory points to VADv2. Its one-, two-, and three-second errors are 0.61, 1.44, and 3.00 meters. Averaged across the three horizons, it barely improves upon VADv2 with no extra planning information and is actually worse at the one-second horizon. Senna’s semantic meta-actions reduce the average error from roughly 1.70 to 1.46 meters, an improvement of about 14.3%. This result directly supports the paper’s main hypothesis. The improvement does not come merely from attaching a large model to the planner. It depends on choosing an interface compatible with the large model. Sparse coordinates remain a poor VLM output even when a second network refines them; a maneuver category transfers useful knowledge more reliably. On nuScenes, the paper’s matched comparison reports average L2 error of 0.22 meters and average collision rate of 0.08% for Senna. The corresponding DriveVLM figures are 0.31 meters and 0.10%, while VAD-Base records 0.37 meters and 0.14%. Senna’s one-, two-, and three-second displacement errors are 0.11, 0.21, and 0.35 meters. The 2026 journal abstract strengthens the cross-dataset result: pretraining on DriveX and then fine-tuning on nuScenes lowers planning error by 35.59% and collision rate by 38.89% relative to training without that pretraining. This is arguably more consequential than the in-domain leaderboard result. It suggests that the semantic planning representation transfers across sensor setups and operating domains rather than merely memorizing one fleet’s trajectory distribution. The final journal version also reports comprehensive open-loop and closed-loop testing. That is an important addition because open-loop displacement metrics cannot capture error accumulation or the response of other traffic participants. Still, a closed-loop benchmark is not equivalent to real-world safety validation, and the publisher preview does not expose enough of the revised table to reconstruct all route-level results from the public manuscript. What the ablations reveal. Using six surround views rather than only the front camera raises meta-action accuracy from 64.91% to 71.21%. The largest gains appear in the harder speed categories: deceleration F1 rises from 50.53 to 61.99, while acceleration rises from 51.78 to 58.83. Rear and side cameras therefore contribute not just to turn selection but to decisions about whether changing speed is safe. The auxiliary QA tasks also matter. Removing direct meta-action supervision predictably causes the largest failure, but removing scene descriptions or traffic-light questions produces substantial degradation as well. Excluding planning explanations, vulnerable-road-user identification, or motion prediction causes smaller yet consistent losses. Training with all six QA families gives the best result. Finally, Senna scales steadily with data. Meta-action accuracy rises from 57.45% with 100,000 training examples to 64.78% with 200,000, 67.09% with 400,000, and 71.21% with 800,000. The speed-related categories benefit particularly strongly. There is no visible plateau in the reported range. That is promising, but it creates a reproducibility tension. DriveX is proprietary and cannot currently be released under the company’s confidentiality policy. The authors provide preprocessing code and weights derived from DriveX, and state that researchers may request access subject to approval, but an independent group cannot freely reproduce the central scaling experiment from raw data. How Senna fits into the larger driving-model landscape. Relative to UniAD and VAD, Senna adds a semantic supervisory layer to a geometry-first end-to-end stack. UniAD uses multiple driving tasks—tracking, mapping, motion prediction, occupancy, and planning—to improve a shared representation. VAD builds an efficient vectorized representation of dynamic agents and map elements. Senna does not replace those ideas; its low-level planner inherits them and adds an explicit maneuver-level prior. Relative to DriveVLM, the key difference is the abstraction boundary. DriveVLM passes sparse geometry from the VLM to the planner. Senna passes semantics. The DriveX comparison suggests that semantics are the better-conditioned communication channel. Relative to fully unified vision-language-action models, Senna is deliberately less monolithic. A unified model may eventually learn perception, reasoning, and continuous control in one parameter space. Senna instead preserves specialized modules and a visible interface. That costs latency and prevents fully joint optimization, but it makes the system easier to debug, upgrade, and potentially certify component by component. The authors’ later Senna-2 work focuses explicitly on aligning VLM decisions with the end-to-end policy through hierarchical reinforcement learning. That follow-up is revealing: the original Senna establishes that a semantic bridge helps, while the next problem is ensuring that the “thinking” branch and the “acting” branch remain behaviorally consistent under closed-loop feedback. The most important limitations. The first limitation is latency. Senna-VLM is a 7B model processing six camera views. Even with token compression, it is not naturally a high-frequency controller. The authors acknowledge that large-model inference may violate real-time requirements and suggest smaller models—on the order of 2B parameters—plus hardware-level acceleration. A practical deployment might therefore run the VLM asynchronously at a lower supervisory rate while Senna-E2E continues planning at control frequency. The high-level action could be cached until an event trigger—an agent entering the path, a signal change, or a navigation transition—forces reevaluation. That would preserve much of Senna’s hierarchy while avoiding six-view VLM inference on every control tick. The second limitation is error propagation. A wrong meta-action is not just another noisy feature; it is a strong instruction. If the VLM says accelerate when the scene calls for braking, the low-level planner may be biased in precisely the wrong direction. Training Senna-E2E with oracle meta-actions exacerbates this issue. Calibration, abstention, action-distribution inputs, noisy-command augmentation, and safety filters would all be natural extensions. The third limitation is coarse expressivity. Twelve theoretical lateral-longitudinal combinations cannot capture the full structure of interactive driving. The authors themselves identify non-predefined, more granular language instructions as future work. A richer interface could represent the intended maneuver, target lane, relevant agent, right-of-way relation, target speed interval, urgency, and confidence—still structured enough for a planner to consume, but much more informative than a single categorical embedding. The fourth limitation concerns interpretability. Senna is more inspectable than a trajectory-only network, but inspectability is not the same as faithfulness. The natural-language explanation is trained as another generated output and is not the actual tensor passed to the planner. The planner receives the meta-action embedding. An explanation can therefore sound correct even if the action arose from a shortcut. Researchers adapting Senna should evaluate counterfactual faithfulness: does changing the supposedly causal traffic light or pedestrian actually change the action?. Finally, Senna still lacks the machinery expected of a safety-critical production system: explicit out-of-distribution detection, calibrated uncertainty over high-level actions, runtime verification, a documented fallback policy, and broad real-world closed-loop validation. Its state-of-the-art planning numbers demonstrate a useful research architecture, not a safety case. How roboticists might adapt the idea. For practitioners, Senna’s most transferable insight is the use of a typed semantic interface between a foundation model and a continuous controller. The VLM does not have to remain a runtime component. One could use it during training to annotate high-level maneuvers, critical agents, right-of-way relations, or risk factors, then distill those outputs into a smaller visual policy. That would preserve semantic supervision without carrying a 7B model onboard. Alternatively, the categorical meta-action could be replaced by a structured latent command. Instead of one lookup embedding, the interface might include separate embeddings for maneuver, target region, target speed, causal agent, and confidence. A diffusion planner or multimodal trajectory generator could then condition on those fields while retaining multiple geometrically feasible futures. The two modules should also be trained under each other’s real output distributions. Senna-E2E could receive sampled VLM errors during training, while Senna-VLM could be optimized using closed-loop planning outcomes rather than only action-label accuracy. A correct high-level label that causes unstable execution should not receive the same reward as one that produces safe, comfortable behavior. Finally, the intermediate QA tasks could become consistency constraints rather than merely language supervision. If the VLM says a light is red but the proposed trajectory accelerates through the intersection, the system should detect the contradiction. The confidence-aware cross-checking visible in the journal figures points in this direction. A mature architecture would turn those semantic predictions into runtime invariants that can reject or modify unsafe plans. Bottom line. Senna’s enduring idea is not that a large vision-language model should drive the vehicle. It is almost the opposite: do not ask the VLM to perform the part of driving for which language models are poorly suited. Let the VLM interpret the situation and select a semantic maneuver. Let a specialized end-to-end policy handle geometry, interaction, and precise trajectory generation. The experiments show that this separation outperforms both a trajectory-only end-to-end baseline and a hybrid that still asks the VLM to predict sparse coordinates. The auxiliary QA curriculum, surround-view tokenization, and DriveX pretraining make that high-level decision maker substantially stronger and transferable. At the same time, Senna exposes the next research problem. Once reasoning and execution are separated, they must be calibrated, aligned, and evaluated together under closed-loop feedback. The meta-action bridge is a strong first interface, but it is too coarse and too brittle to be the last one. For autonomous driving—and arguably for embodied AI more broadly—Senna offers a valuable architectural principle: foundation models may be most useful not as direct continuous controllers, but as semantic supervisors operating at the level where their knowledge, language priors, and commonsense reasoning are genuinely an advantage.