Latest / Embodied AI 101 / MJev-Spatial: Faster VLM Spatial Reasoning Without Generating Text
Transcript
- MJev-Spatial: Faster VLM Spatial Reasoning Without Generating Text. MJev-Spatial: System One Spatial Intelligence via Native Attention Matching • Luozheng Qin, Yuhan Zhu, Changhai Zhou et al. • MindLab • arXiv preprint • 2026. Cold Open. Picture a robot holding a cup above a cluttered table. You ask it to find somewhere left of the bowl. The next component in your stack needs a location. It does not need a polished sentence explaining the relationship between cups, bowls, and tables. Yet imagine sending that request through a conversational interface: generate an answer, finish writing the coordinates, then parse them back into something the controller can use. MJev-Spatial changes that interface. It selects spatial answers directly, with a reported 5.6-fold inference speedup over its starting model, Qwen3.5-4B. Choices, numerical estimates, and image locations all become candidate-selection problems. That raises a useful question: how much of a vision-language model’s output machinery does a spatial decision actually need?. This episode follows the answer through the architecture, the training recipe, and the experiments. The interesting move is putting pretrained multimodal knowledge behind a different interface—one organized around decisions rather than sentences. The Idea. Two research directions meet here: structured decision models and visual grounding. TypeSafe’s Jev established the direct-decision framing: provide context and specify possible outputs, then receive probabilities rather than generated strings. Jared Palmer’s open-source Kev implements candidate selection with a learned pointer that compares a decision representation against representations of the available options. MJev-Spatial brings that general pattern to spatial prediction. The input still contains language. Images, the question, and candidate descriptions enter the multimodal backbone together. Each candidate ends with a common marker, and a final decision marker follows the candidate list. After contextual processing, the candidate-marker states summarize individual alternatives; the decision-marker state has access to the complete menu. These are contextual representations, incorporating the observation and question. The distinctive architectural choice comes next. MJev-Spatial uses the final full-attention layer’s existing query projection for the decision state and its existing key projection for candidate states. It compares those vectors, averages the scaled matching scores across attention heads, and normalizes over the legal candidates. The implementation retains native normalization and positional transformations while removing unused final-block computation. No separately initialized prediction head is needed. Think of this as giving attention’s matching machinery a new job. Instead of using the comparison to aggregate token values, the system uses it to choose an answer. Because candidates carry their own descriptions, the same scorer can compare “left” against “right” in one request and numerical intervals in another. The meaning of a class comes from its content, rather than a permanently assigned output slot. The output controller supplies the structure. Multiple-choice questions finish after one decision. Numerical prediction generally selects a coarse interval and then a finer interval, which is decoded into a value. Pointing uses three successive selections from a three-by-three grid. After the first two selections, the chosen region is cropped and re-encoded. The final cell center supplies normalized image coordinates. Thus, a pointing answer involves three decisions, each with its own forward pass. The full image remains available alongside the crops, preserving global context as the model examines local detail. Multiple-point prediction extends the same predictor by exploring several region paths. The useful design principle is that output refinement and visual refinement advance together: a more precise question receives a more focused view. Training uses 12.3 million question–answer pairs, converted from RefSpatial, SAT, SenseNova-SI, VSI-590K-image, and VST-P. All retained language parameters are fine-tuned; the vision encoder and merger stay frozen. This is supervised adaptation of existing parameters to a new decision interface. Teacher forcing makes the multistage training efficient. Ground-truth choices determine the later candidate sets and crops, so the stages can be serialized and supervised in one causal language-backbone pass. Each original question receives equal total loss weight, regardless of its number of stages. Length grouping and cross-device balancing then organize examples to reduce padding and uneven workloads. The contribution is this combination: a shared, content-aware scorer; reuse of pretrained attention projections; and task-specific refinement that turns numbers and locations into manageable sequences of decisions. The Result. The headline quality result is a 65.64% benchmark-average score, the highest among the eight tested models. The comparison includes general-purpose instruction VLMs and spatial specialists, with Qwen3-VL-8B-Instruct the closest competitor on the overall average. MJev-Spatial also improves over its unmodified starting model across the complete twelve-benchmark suite. The suite spans visual question answering, multi-image spatial understanding, and language-guided pointing. Names include CV-Bench, BLINK, MMSI-Bench, RefSpatial, Where2Place, and Point-Bench. This makes the experiment about several forms of spatial prediction, rather than one particular question template. Where2Place illustrates the robotics relevance. Introduced with RoboPoint, it evaluates referring to free space through spatial language. A successful prediction lands inside an annotated target region. The target might be an empty area between objects, rather than an object category that a detector can identify directly. That distinction matters when the next operation is placement: the useful output is a geometrically grounded destination. On efficiency, MJev-Spatial is 2.5 times faster than the fastest autoregressive comparator. Measurements use one NVIDIA B200 at batch size one, including image loading, preprocessing, transfers, inference, and output decoding. For an engineering team, the compelling feature is the joint operating point. The system leads this reported quality comparison while also offering the lowest mean latency. That makes it worth examining as an alternative implementation of the spatial-perception interface. The controlled pilots provide the mechanism-level evidence. With matched training recipes on a smaller dataset, native Q/K scoring is competitive with a linear classifier and an added KEV pointer. Fresh crop encodings lead the tested visual-input alternatives on RefSpatial and Point-Bench. A coarser numerical codebook also improves length-estimation error on the VST-300 development set. Those experiments point toward a practical design strategy: treat the readout, visual evidence, and discretization as separate design choices. The scorer determines how alternatives compete. Cropping determines what detail is available. The codebook determines which distinctions the model must learn. There is a useful parallel in MolmoPoint. That work selects visual patches and refines locations through specialized grounding tokens, reporting benefits from changing how pointing is represented. Its mechanism uses generated grounding tokens and learned projections. Together, these approaches make output representation a concrete modeling question: which interface best connects a learned visual representation to the answer the application needs?. The Biggest Weakness. The central missing control is an autoregressive Qwen3.5-4B fine-tuned on the same spatial corpus under a matched recipe. The reported base-model comparator is unmodified. That leaves the cause of the accuracy gain unresolved. The proposed system changes the output interface and receives substantial spatial supervision. If performance improves, how much comes from candidate classification, and how much comes from teaching the backbone the domain?. This is the experiment that determines the strength of the paper’s central explanation. A fast, accurate specialized system is one claim. Showing that direct classification is a better way to learn spatial judgments than concise autoregressive prediction is a stronger claim. Molmo2-ER provides a useful partial comparison. Its official model card describes substantial embodied specialization using overlapping dataset families, including SAT and RefSpatial. It starts from a different language-and-vision backbone and follows a different training recipe. It therefore supplies a serious specialized competitor, without isolating the effect of MJev-Spatial’s decision interface. The decisive experiment would start two systems from the same checkpoint. Give them the same accepted examples, visual budgets, training exposure, and evaluation protocol. Train one to produce concise answers autoregressively and the other to select candidates. Then compare both quality and end-to-end latency. The interpretation would become straightforward. If the autoregressive counterpart recovered the accuracy improvement, the main advantage would be efficient execution and a convenient output contract. If candidate selection remained more accurate, the evidence for a superior spatial-learning formulation would become much stronger. The existing head comparisons answer whether native projections can support a competitive classifier. They do not answer that counterfactual. For practitioners choosing where to spend a fine-tuning budget, that is the paper’s most consequential unresolved question. Verdict. Roboticists building the interface between perception and planning should pay attention. The most attractive use is a component with an explicit answer contract: select a spatial relation, estimate a quantity, or nominate a location. Consider a placement pipeline. The language instruction identifies the desired relationship. A perception module supplies a point. Geometry and planning then determine how to reach it. RoboPoint already demonstrated this kind of decomposition by converting predicted image points into three-dimensional targets using depth, then passing target poses to a motion planner. MJev-Spatial offers a different implementation strategy for the language-conditioned spatial decision within such a stack. That is where I would investigate it first: at a boundary where the downstream consumer already knows what kind of answer it needs. Imagine repeatedly asking for the same placement relation as the camera view changes. The useful changing information is the destination. A candidate-based interface makes that destination the explicit prediction target. The release also gives practitioners something concrete to work with. The repository provides an Apache-licensed runtime, basic fine-tuning and evaluation entry points, examples, and tests. Released weights are linked through MindLab’s Hugging Face account. Requests specify their answer space, and predictions expose the selected answer, decision path, and candidate scores. For VLM researchers, the broader opportunity is architectural reuse. Pretraining produces representations and internal matching operations. MJev-Spatial turns the latter into a task-facing readout. That suggests a productive research direction: examine which pretrained computations already resemble the decision an application wants, then adapt those computations directly. My takeaway is simple: design the output interface as carefully as the representation. A spatial system’s interface determines what must be learned, what computation runs at inference, and what the next software component receives. MJev-Spatial makes that interface a central part of the model design. Outro. The next experiment to watch is a direct spatial-decision module inside a continuously updated perception-and-planning loop, where each answer changes what the agent sees next. For now, keep the cup-and-bowl example in mind. Before choosing how to make the model answer faster, decide what the rest of the system actually needs from it. A relation? A distance? A point?. That question is easy to postpone when a conversational model already accepts your prompt. MJev-Spatial puts it back at the center of the architecture. The useful unit of intelligence for a robot may be the next decision—not the next token.