Latest / Embodied AI 101 / LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding
Transcript
- LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding. Vision-language models (VLMs) are increasingly used as general-purpose foundations for robotic perception and interactive AI. In these systems, the model must interpret a natural-language query (“find the red cup”, “click the menu icon”, “locate the title in the document”) and return precise spatial output — typically one or more bounding boxes or points. Traditional approaches treat this as a generative task: the model emits a sequence of tokens that encode the coordinates of the answer. For example, earlier methods spelled out box corners as digit tokens or used discrete quantized tokens for each coordinate. But this sequential token generation has fundamental limitations. By breaking a 2D box into a stream of independent tokens, these methods ignore the fact that the four coordinates of a box are a tightly coupled unit. Furthermore, decoding the box coordinate by coordinate forces a strictly sequential process at inference time, creating a practical speed bottleneck. LocateAnything, introduced by NVIDIA researchers, addresses these challenges with a new idea: Parallel Box Decoding (PBD). Instead of predicting each box as a series of tokens, the model treats the entire box as an atomic unit. In one forward pass, it emits a full bounding box (all four coordinates) or point, along with any associated labels. This preserves the geometric coherence of each box and dramatically speeds up decoding, because boxes can be predicted in parallel rather than token by token. As the NVIDIA team explains, PBD “preserves intra-box geometric coherence and unlocks substantial parallelism”. In practice, this yields a model that is about 2.5× faster in decoding throughput than the best previous transformer-based detectors, while also improving localization accuracy. LocateAnything is built on a powerful vision-language backbone (Moon-ViT for high-resolution image encoding and Qwen2.5 for language decoding) and trained on an enormous new dataset. The authors curated LocateAnything-Data, a multi-domain corpus of 12 million images with 138 million language queries and 785 million annotated boxes. It covers tasks ranging from general object detection to UI element grounding, document layout parsing, OCR, and fine-grained pointing. With this scale and diversity, LocateAnything sets new state-of-the-art performance on a suite of grounding and detection benchmarks – all while running significantly faster than its predecessors. In the following sections we unpack the key ideas behind LocateAnything. We explain how Parallel Box Decoding works, how the model is trained with a clever dual-format setup, and how dynamic inference modes ensure both speed and accuracy. We also overview the massive dataset that makes it all possible. Finally, we summarize the impressive results: this single model outperforms specialist systems on tasks like UI grounding (ScreenSpot-Pro), document layout (DocLayNet, M6Doc), dense detection (VisDrone, Dense200), and more. In short, LocateAnything “advances the speed–accuracy frontier” for unified visual grounding, making it an exciting step toward practical embodied AI that truly “sees” what the human tells it. Background: Vision-Language Grounding as Sequence Prediction. Modern VLMs have enabled a unified approach to many perception tasks. Instead of standalone object detectors or specialized OCR models, we can now ask a multimodal model to directly answer queries like “Where is the red bike?” or “Locate the email button.” The model interprets the image and the text, and returns spatial outputs. Concretely, this is often framed as sequence generation: the model’s output is a sequence of tokens encoding the bounding box(es) that satisfy the query. For example, an early approach might output a formatted string like , meaning a box from (100,150) to (400,300) containing a bike. In practice, the coordinates are normalized, so that those integers might represent fractions of the image size (e.g. 0.100 to 0.400 horizontally). Regardless of exact format, the key is that the four corners of the box must be produced token by token. Prior works have largely encoded coordinates in two ways: textual digits or quantized tokens. In the textual approach, each digit of each coordinate becomes a token (so “1024” is output as “1”, “0”, “2”, “4”). In the quantized approach, each coordinate is mapped to a fixed vocabulary of possible positions and output as one token per coordinate. Either way, a box ends up serialized into a one-dimensional token stream. This token-by-token strategy is natural within the standard next-token-prediction paradigm: the transformer processes each new token sequentially with causal attention. But it has two drawbacks. First, serializing the 2D structure breaks the natural coupling of the box. The model must implicitly remember that some tokens belong together as coordinates of the same box. As the authors note, the standard approach “mismatches the coupled structure of box geometry”. This can lead to inconsistencies and errors (for example, mixing up which “x” and “y” go together or producing impossible boxes). Second, the sequential decoding becomes a practical speed bottleneck. Predicting an entire image’s worth of boxes one token at a time can be very slow, especially for scenes with many objects or dense layouts. As illustrated in the GPU throughput numbers, typical transformer VLMs achieve only around 1–5 boxes per second (BPS) on a top-end GPU, whereas LocateAnything can reach over 12 BPS (summarized below). In practice, robotics and embodied AI often need both high speed and high accuracy in localization. For example, an autonomous agent interacting with a smartphone UI must quickly find buttons and text, and a warehouse robot must rapidly count and find objects on shelves. Sequential box decoding simply won’t cut it in such real-time settings. This situation spurred NVIDIA’s team to rethink the generation process. Instead of streaming out each corner in sequence, what if the model could output a whole “box” in one shot? That is the essence of Parallel Box Decoding (PBD). Parallel Box Decoding: Predicting Boxes as Atomic Units. At the core of LocateAnything is the idea that each bounding box (or point) should be treated as an indivisible block during decoding. Concretely, the model’s output is structured into blocks. Each block corresponds to one object’s spatial output. A “box block” has a fixed length (for example, 6 tokens) that includes a marker, four coordinate tokens (for x1, y1, x2, y2), and an optional closing tag or padding. A “point block” might have fewer tokens. There are also “semantic” blocks to encode text labels (if needed) and special blocks indicating no object or end of sequence. The key is: tokens within a block are predicted together. During inference, the model does not generate coordinates one by one. Instead it emits each block’s tokens in parallel. That is, for each new object, it outputs all of that object’s coordinate tokens at once. This preserves the geometric coherence: the four corners are predicted jointly, so the model can enforce consistency (e.g. enforcing , etc.) within the block. As the Readme explains, PBD “treats each bounding box (or point) as an atomic unit and predicts the full coordinate set in a single forward pass”. Why does this speed things up? In a standard sequential decoder, generating a box of four coordinates requires four prediction steps. If you have 100 boxes in an image, that’s 400 steps. With PBD, you can generate each box in one step, so 100 boxes → 100 steps (plus overhead), achieving roughly 4× fewer steps. In practice this yields a roughly linear improvement in decoding throughput. In the reported results, LocateAnything (with hybrid mode) decodes at 12.7 boxes per second on an NVIDIA H100 GPU, compared to only 5.0 BPS for the previous best model (Rex-Omni) and about 1.1 BPS for a standard large language model (Qwen3-VL). In other words, it is over 2.5× faster than Rex-Omni and 10× faster than a textual decoder baseline. Of course, simply emitting boxes in parallel is not enough on its own – the model also has to know when to stop and how to structure the output. LocateAnything defines several block types:. Semantic blocks hold any text output (e.g. the class label or phrase describing the object). If a description is long, it can be split across multiple blocks. Box blocks specifically represent an entire bounding box. Each box block has exactly four coordinate tokens (in normalized form). Negative blocks explicitly signal that a queried object is not present (e.g. ). End blocks mark the end of the entire output sequence. Each block is padded to a fixed length (for implementation simplicity). If a block is shorter than the fixed size, remaining positions use a token, ensuring that all blocks have the same tensor shape. Figure 1 of the paper (recreated in the NVIDIA page) contrasts the old way and PBD: whereas textual or quantized decoding writes out coordinates sequentially, PBD “predicts each geometric unit (e.g., a bounding box) in a single forward pass”. This is akin to asking the model to spell out entire objects at once rather than letter-by-letter. As the authors note, this design “preserves intra-box geometric coherence and prevents irregular structural tokens” – in other words, it enforces that the coordinate tokens stay grouped correctly. In practical terms, the model’s vocabulary is extended to include special tokens for , , , , , etc. A sample output might look like:. This would be interpreted as a bounding box for “dog” whose top-left corner is at (0.345, 0.120) and bottom-right at (0.400, 0.300) in normalized image coordinates. By grouping them, the model can reason holistically about that box. Training for Parallel Decoding: NTP Meets MTP. Switching to parallel decoding presents a training challenge. During regular (next-token) training, the model learns to generate one token at a time so that each prefix is complete. But under PBD we want the model, at test time, to fill in the rest of a block all at once. How can we train it to do both without losing language coherence?. LocateAnything’s solution is to train with two aligned representations simultaneously. For each training example, the ground-truth output sequence is represented in two compatible ways:. Standard NTP (Next-Token Prediction) format: This is just the usual flattened token sequence (with box coordinates serialized) that a conventional generative model would see. We ensure the model retains full sequence modeling ability. Block-based MTP (Multi-Token Prediction) format: The same information is chunked into blocks as described above, with tokens for the parts of each block to be predicted. Specifically, within each block we keep only the first token (e.g. marker) and replace the remaining tokens (the coordinates) with a mask token. The model is then trained to predict all of those masked coordinates in one go. These are concatenated into one long input: first the shared context (vision tokens from Moon-ViT and the text query), then the NTP token sequence, then the blockwise sequence. So the model sees both representations side by side. Importantly, they represent the same ground truth, just in two formats. A single example might look like:. During training, a specialized attention mask is used so that each part learns appropriately. The next-token stream still trains with causal attention (it can only see previous tokens), preserving the model’s original language generation ability. The block stream is segmented into the block structure: tokens within the same block attend to each other bidirectionally (so the model can reason jointly over the missing coordinates), but blocks follow a causal order among themselves (later blocks can’t see future blocks). Neither stream can leak information into the other; they only share the original context. Intuitively, this dual training “aligns NTP with block-level MTP”. The model thus learns to do both sequential and parallel decoding. During training, the loss is the sum of the losses from the NTP and the MTP streams, forcing it to predict the tokenized sequence correctly and also the blocked sequence correctly. In effect, the model learns that the four coordinates of a box belong together, because in the block stream it must fill them all at once. If we set the block size to 1, this would reduce to the normal NTP case (no parallelism), but here blocks of size 4 (plus markers) enforce a strong structure. The result is a model that, when prompted at inference time, can switch to block-based generation while still maintaining the language understanding it learned. Flexible Inference Modes: Balancing Speed and Precision. In practice, pure parallel decoding can occasionally lead to glitches. The NVIDIA team identifies two failure modes during parallel (MTP) decoding: format irregularity and spatial ambiguity. Format irregularity happens in complex scenes with many objects of different classes. When predicting a new block, the model might hesitate about whether it should continue the same object or start a new class. This can result in malformed output, such as mixing an end tag with a coordinate token in the wrong order. Spatial ambiguity occurs in layouts like dense grids of similar objects: the model’s parallel guess might land between two actual objects, yielding a low-IoU box. To address this, LocateAnything uses a hybrid strategy. By default, it uses MTP (parallel blocks) for speed, but it monitors the quality of each predicted block. If it detects a malformed block or one with highly uncertain coordinates, it rolls back and uses ordinary autoregressive (NTP) decoding just for that block. Concretely, after generating a block in parallel, the model checks: if the top coordinate token had probability below a threshold (e.g. 0.7) and the margin among the top candidates is large, the block is considered unreliable. In that case, the model discards the block and regresses: it re-generates those coordinates token by token with NTP. Once the fix is done, it switches back to MTP for the next objects. Thus LocateAnything supports three modes of use:. Slow Mode: pure token-by-token NTP decoding (max stability, minimum speed). Fast Mode: pure MTP block decoding (max speed, less checks). Hybrid Mode: default MTP with on-demand NTP fallback (best of both worlds). In their experiments, the authors use Hybrid Mode as the default. This way, nearly all blocks are predicted in parallel (providing the bulk of the speed), but any problematic blocks are corrected via NTP. As they report, this hybrid strategy “seamlessly switches to NTP when parallel outputs become unreliable”. In practice, it gives both high throughput and robust results. For example, in ablation they show Hybrid Mode slightly improves accuracy over pure MTP, because it catches the few mistakes that pure parallel decoding would make. After each step (whether MTP or NTP), the newly committed tokens are appended to the cache for causal context, ready for the next prediction. The on-device pipeline is carefully managed so that, during an MTP step, all tokens within the current block attend to each other bidirectionally, while blocks attend causally between each other. Architecture: Moon-ViT + Qwen2.5. Under the hood, LocateAnything builds on a strong vision-language backbone. It uses Moon-ViT as the vision encoder – a ViT-style transformer that operates at native image resolution (detailed in internal references) – and Qwen2.5 as the language decoder. A lightweight MLP projector bridges the visual and language components. In effect, the image is encoded into a grid of visual tokens (preserving fine detail for accurate boxes), and then the decoder takes these features along with the text query to generate the output blocks. This architecture is pre-trained on large image-text corpora for general language and visual understanding, and then fine-tuned for the grounding task using the LocateAnything-Data (next section). By starting from a strong pre-trained VLM, they get robust perception and language alignment out of the box. The modifications are mainly in the output formulation (blocks and mixed training) rather than in the core architecture. LocateAnything-Data: Massive Multi-Task Grounding Data. A key part of LocateAnything’s success is the scale and diversity of its training data. The team built a LocateAnything-Data corpus with 12 million images, 138 million language queries, and 785 million bounding boxes. This is orders of magnitude larger than typical task-specific datasets and curated to cover many domains. The data spans six categories of tasks:. General Object Detection (66.9% of queries, 83.1% of boxes): This is the bulk of the data. It includes images with many annotated objects (from COCO, LVIS, MOT20Det, SKU110K, etc.) to give the model dense bounding box supervision. By training on this, the model learns precise coordinate alignments and can handle crowded scenes. GUI Element Grounding (16.5%): Queries about user interface elements, such as website or app screenshots. For example, “the submit button” or “the search icon.” This enables applications like UI automation or robotics interaction with screens. Referring Expression Comprehension (7.3%): Natural language phrases referring to objects or regions, e.g. “the person wearing a hat” or “the red car on the left” (similar to datasets like RefCOCO). This teaches the model to handle complex language that may involve comparisons or attributes. Text Localization / OCR (3.6%): Queries about text in images, e.g. “the largest word” or “the next line after ‘Chapter’.” This ensures the model can tightly locate text regions, supporting applications like reading menus or signs. Document and Scene Layout (3.5%): Complex structured images like forms or diagrams, with queries like “the header field” or “the caption area.” Data from document layout datasets (DocLayNet, M6Doc, etc.) are included to boost performance on document understanding. Point-based Localization (2.2%): Fine-grained tasks where the model must identify a point, not a full box – for example, “click the exit button” where a single point suffices. Each query in the dataset comes with one or more ground-truth locations (boxes or points) and a natural language prompt describing what to find. By mixing all these domains, the model becomes a true panoramic localizer. As the NVLabs README puts it, LocateAnything-Data covers “detection, GUI grounding, referring comprehension, OCR, layout, and pointing” across millions of examples. Training on this scale is crucial for the model to generalize: it sees many languages and contexts for “what to detect”, not just straight object names. In training, the authors actually use a two-stage process (detailed in the paper). First, they fine-tune the base VLM on the combined 138M-query mixture (so it learns all these tasks). Then, they emphasize the densest scenes for a second stage: they reduce the proportion of “one-object” data and upsample datasets with many small objects (like MOT20Det and SKU110K). This ensures the model’s bounding-box skills sharpen on crowded scenes. All told, this data strategy is about giving the model every reason to learn precise spatial reasoning across different scenarios. In short, both the algorithmic innovation (PBD and dual training) and the data scale contribute. The paper highlights this synergy: “the complementary benefits of Parallel Box Decoding and large-scale training data” enable the model to be both efficient and precise. Evaluation: Diverse Tasks, Strong Results. LocateAnything was evaluated on a wide array of benchmarks, spanning generic detection, dense scenes, UI grounding, documents, and referring tasks. Across the board it achieves state-of-the-art or near state-of-the-art performance, despite using a moderate model size (~3B parameters) and being a single unified model. We highlight some key results below. (All metrics below are reported as mean F1-scores at IoU thresholds; e.g. “F1@IoU=0.5,0.95,mean” in the tables. Higher is better.). General Object Detection (COCO, LVIS). On standard object detection, the model was evaluated zero-shot on COCO and LVIS (“common” and “long-tail” detection). This means the model had never been fine-tuned on these particular splits; it just applies its learned grounding capability. On LVIS (with any-class queries), LocateAnything-3B achieves F1 mean = 50.7, which is +3.8 points over the previous best (Rex-Omni-3B at 46.9). On COCO detection, it gets 54.7 mean F1, +1.8 over Rex-Omni’s 52.9. These improvements are significant given the identical scale of the backbone. In other words, by rethinking the output format, the model not only speeds up but also localizes slightly more accurately on these benchmarks. Dense Object Detection (VisDrone, DenseD200). For crowded scenes, the gains are even clearer. On the VisDrone drone imagery dataset (hundreds of small cars and people), LocateAnything achieves 39.9 mean F1, where Rex-Omni scored 35.8 (a +4.1 improvement). On the synthetic DenseD200 test, it reaches 58.7 vs Rex-Omni’s 58.3 – a smaller delta, but note that RA’s IoU=.95 F1 was much higher (18.5 vs 10.3 for Rex-Omni). Put simply, LocateAnything delivers superior boundary accuracy in tight clusters of objects. The higher precision at the strict IoU threshold indicates it splits overlapping instances more cleanly. In practical robotics terms, it means the model can count and separate many small objects more reliably. GUI and UI Grounding (ScreenSpot-Pro). One exciting target is GUI grounding: finding elements in images of user interfaces. ScreenSpot-Pro is a challenging benchmark for this. LocateAnything achieves 60.3 average accuracy on ScreenSpot-Pro, which the authors report as state-of-the-art. This beats much larger or specialized models, including a 30B multimodal model (Qwen3-VL-30B) and GUI-specific experts like Owl-32B. In plain terms, the model can look at a screenshot or photo of an app and correctly click buttons, icons, and text fields at over 60% accuracy – using only a 3B model. This is crucial for embodied agents that interact with software. Document Layout and OCR (DocLayNet, M6Doc, TotalText). For document understanding, LocateAnything also excels. On DocLayNet (layout parsing of pages) it scores 76.8 mean F1, and on M6Doc (business documents) 70.1 mean F1, both significantly above prior best. As a reference, a specialized object detector (DocLayout-YOLO) had an F1 of 81.1 on DocLayNet – higher in one metric – but that is a single-task model. The key is that LocateAnything is competitive while being general-purpose. For OCR (text detection in images, i.e. TotalText dataset), it gets 43.3 mean F1, which is the highest among VLM-based models and even slightly above a classic OCR system (PaddleOCRv5 had 25.7). In short, it reliably finds words and text blocks too. Referring Expression (RefCOCOg, HumanRef). Referring expression comprehension is another use case: given a phrase like “the woman in the red jacket on the right,” locate that person. On RefCOCOg, LocateAnything gets 77.6 mean F1, outperforming the predecessor (74.3). On HumanRef (free-form referring) it scores 78.7, comparable to the top specialized model (79.9 by Rex-Omni). These scores indicate the model can align nuanced instructions with visual regions competitively. Summary of Improvements. Overall, the model pushes the frontier on many fronts. The NVLabs summary tabulates these gains succinctly: for example, +3.8 F1 points on LVIS, +1.8 on COCO, +6.1 on DocLayNet, +14.5 on M6Doc (vs Rex-Omni), etc. (See the table above from the code repo.) And crucially, it does so while decoding 10× faster than a standard model and 2.5× faster than the previous VLM baseline. Another striking result is the throughput scaling. In an ablation, the authors varied the number of target boxes and measured inference time. Pure token-by-token models slowed heavily as boxes increased, since each extra box adds multiple decoding steps. In contrast, the PBD-based LocateAnything’s inference time grows much more gracefully (nearly flat). This underlines the advantage: for scenes with dozens or hundreds of objects, PBD is dramatically more efficient. Putting It All Together. LocateAnything represents a significant leap in vision-language grounding. By rethinking how spatial outputs are generated, the NVIDIA team achieved a rare combination: both higher accuracy and much lower latency. This makes the model particularly well-suited to real-world interactive robotics and agent tasks, where you want the rich understanding of an LLM but also the speed to act. Because the code and model are released (the paper links to a GitHub repo and HuggingFace models), practitioners can immediately try it. For instance, a robot with a camera can use the model to interpret spoken instructions like “grab the can”, with the model outputting the can’s bounding box in real-time. Or an automated script could parse and click through a multi-step GUI by instructing the model “click the Start button”, “enter text into the search field”, etc., seeing immediate speeded localization. The hybrid decoding mode means developers can even choose their operating point: in a safety-critical labeling application, one might prefer “Slow Mode” to ensure every box is maximal quality. In real-time vision for drones or cars, Hybrid/“Fast” mode offers the speed needed. And thanks to the large multi-task dataset, the model is surprisingly robust across domains — from street scenes to documents to UI screenshots. In summary, LocateAnything is not just a single benchmark trick, but a new paradigm for vision-language models that generate structured outputs. It demonstrates that carefully aligning the output format with the task structure (boxes as atomic units) can unlock tremendous gains. As the authors put it, they have built “a unified grounding mindset” – a model that truly aims to pinpoint anything you ask it to locate in an image. For robotics researchers and ML engineers, this is a powerful new tool: faster, more accurate, and broad in scope. It will be fascinating to see how this idea of block-based generation propagates. The notion that a model can handle spatial units (boxes, points, segments) as first-class outputs might extend to 3D (volumes) or other structured predictions. For now, LocateAnything stands as a proof-of-concept: large-scale training + intelligently designed decoding can produce a “locate-anything” model that lives up to its name. Sources: This summary is based on the LocateAnything paper and code release by Shihao Wang et al. (NVIDIA Research), which includes detailed results on benchmarks such as LVIS, COCO, VisDrone, ScreenSpot-Pro, DocLayNet, M6Doc, HumanRef, RefCOCOg, and others. Descriptions of PBD, training methodology, and inference modes draw on the authors’ explanations, and implementation details are available in the NVLabs documentation. The model and demos are publicly available on the NVIDIA/HuggingFace repositories.