Latest / Embodied AI 101 / SpatialPoint: Spatial-Aware Point Prediction for Embodied Localization
Transcript
- SpatialPoint: Spatial-Aware Point Prediction for Embodied Localization. Robots and embodied agents must not only recognize objects in images, they must also reason about where in 3D space to act. Consider an instruction like “grasp the mug” or “move the box onto the shelf.” Unlike a vision-language model (VLM) whose output might be a bounding box or text label, a robot needs a precise 3D coordinate to execute. The SpatialPoint paper introduces the notion of embodied localization: predicting an executable 3D point given an image and a language command. In other words, the model outputs a triple $(u,v,Z)$ in the camera’s frame, where $(u,v)$ is a pixel location and $Z$ is depth. This unifies many robotics tasks under one interface. As Zhu et al. explain, this is a problem of 3D spatial reasoning – essentially “where in the scene to act” – rather than just image understanding. Most existing VLMs focus on 2D semantics and work solely with RGB images. They can caption a scene or recognize objects, but fall short on metric tasks. For example, a model might “see” a mug but not know how far away it is. The authors note that no amount of natural language can replace real 3D geometry: “A robot tasked with ‘grasping the mug’ needs to know exactly where that mug or shelf is in 3D space, relative to its own sensors”. In practice, many robots have RGB-D sensors, yet most VLMs ignore depth. Instead they must implicitly guess geometry, leading to poor generalization across scenes. SpatialPoint closes this gap: its vision-language framework directly incorporates depth so that it can produce precise metric point targets. Touchable vs Air Points. SpatialPoint’s key insight is to distinguish two kinds of actionable targets. They call these touchable points and air points. Touchable points are locations on physical surfaces in the scene. These represent direct contact goals for manipulation. For instance, “grasp the handle of the mug” means the target point lies on the mug’s surface. Similarly, “press here” or “place the sensor on the table” refer to specific (u,v) on objects. In SpatialPoint, each touchable query becomes a 3D point anchored on the observed surface. Air points are targets in free space, not anchored to any visible surface. These include navigation goals, placement in open air, or spatial relationships. For example, “move to the space between the two boxes” or “position the bottle 30 cm in front of the robot” cannot be resolved by looking for a texture – the point is in empty space. Air points might specify a goal (“go to this spot”), a direction (“approach the shelf from 2 meters away”), or a geometric relation (“point left of the door”). Both types are needed for embodied tasks. Any command either asks the robot to touch something (touchable) or navigate/aim into open space (air). By representing every target as a 3D coordinate, SpatialPoint unifies manipulation and navigation goals. In practice, a touchable point will be specified to allow direct physical interaction, while an air point handles placement and navigation constraints such as relative positions or offsets. A 3D-Aware Vision-Language Architecture. The core of SpatialPoint is a vision-language model that regresses a 3D point. To do this, the authors augment a standard VLM with an explicit depth stream. Specifically, they start from a large transformer-based VLM (Qwen3-VL-Inst-4B is used in the paper) and add a parallel encoder for depth. The trick is to make the depth map compatible with the existing visual tokenizer. The depth input is originally a single-channel metric image (with each pixel’s distance in meters or millimeters). Instead of forcing the pre-trained vision model to learn a new format, SpatialPoint converts the depth map into a three-channel uint8 image (by normalizing depth into 0–255 range). This way, the same patch-based tokenizer can encode depth just as it would an RGB image. In effect, the model has two vision backbones side by side: one processes the RGB camera image, the other processes the depth image of the same scene. Because both use the same spatial tiling, the token from the RGB backbone at pixel (u,v) is naturally aligned with the token from the depth backbone at that pixel. These aligned visual tokens are then concatenated (along with the language tokens) into a single sequence for the transformer. In this joint sequence, the model can attend to image patches, depth patches, and the instruction together. Importantly, this design preserves the pre-trained weights of the original VLM. The authors only add new parameters (the depth encoder and a small part of the text-prediction head) but inherit the powerful knowledge of the pretrained model. At the output, SpatialPoint’s language head is modified to predict three numbers (u, v, Z). Conceptually, after reading the question and attending to the fused vision-depth tokens, the model emits the 3D coordinate in the camera’s frame. In this way, its output is directly executable by a controller: the robot can convert the pixel (u,v) into a ray and reach that distance $Z$ to touch or move to that point. For example, if the instruction is “place the cube to the right of the yellow box,” the model might output the pixel coordinates of the cube plus a depth a meter away, fulfilling the directional goal. Two-Stage Training. Integrating a new modality like depth into a pre-trained VLM can easily disturb the model’s existing abilities. To handle this, the authors employ a two-stage training strategy. Alignment Stage: First, they freeze the entire pretrained VLM (both the RGB backbone and the language head) and only train the new depth backbone. The depth encoder is given a higher learning rate so that it quickly learns to produce feature embeddings that “align” with the RGB embeddings in semantic space. In other words, at this stage the model is just learning a good representation of depth information that is compatible with the frozen vision-language model. The idea is to avoid any large gradient updates to the original weights early on, preserving the model’s ability to parse language and images. Fine-Tuning Stage: After the depth encoder is aligned, they unfreeze all components and fine-tune on the full SpatialPoint task. In this phase, the RGB encoder, depth encoder, and the output head are all trained jointly on the 3D point prediction task (using the SpatialPoint-Data described below). This lets the model refine how to combine visual and depth cues to satisfy spatial queries (for instance, learning that an “air point” instruction depends heavily on the depth tokens). By gradually bringing depth into the system, this process avoids catastrophic forgetting of language understanding and ensures the model retains the rich pre-trained knowledge while gaining 3D spatial reasoning. The SpatialPoint Dataset. A major contribution of SpatialPoint is its large-scale training data. The model needs example pairs of the form (RGB image plus depth map, language question, 3D point answer). No such dataset existed at the required scale, so the authors built SpatialPoint-Data from an existing robotics dataset (RoboAfford, which contains annotated 2D interaction points). Using an automated data engine (Figure 3 in the paper), they generated 2.6 million QA examples covering both touchable and air points. The process was as follows:. Touchable Points (∼1.9M samples): Starting from annotated 2D "click points" where humans indicated where to interact in the scene, they “lift” each point into 3D via depth. Concretely, given an RGB image and a depth map, a labeled point (u,v) has a measured depth value Z at that pixel. The engine simply takes that depth value to form the 3D coordinate (u,v,Z). This conveniently converts existing 2D affordance labels into metric 3D targets without extra manual labeling. In practice, they leveraged an off-the-shelf monocular depth estimator (since they work in simulated-like scenes) to obtain the Z values, thus avoiding any manual 3D annotation. The result is a large set of (“touchable”) point predictions that lie on actual object surfaces. Air Points (∼0.7M samples): Generating air-point examples is more involved, since these points do not lie on any visible surface. To handle this, the engine first detects objects in the image using a visual detector (specifically DINO-X, a robust pretrained object/mask detector). Each detected object’s 2D mask and bounding box are lifted into 3D using the depth map and camera intrinsics, giving a coarse 3D location and volume for each object. With objects mapped to 3D space, the engine can sample free-space target points based on spatial relationships. For example, it might take the 3D position of a cup and generate a query like “point 30 cm to the left of the cup.” This involves computing a point in 3D space a fixed offset from the cup’s center. Or it might create a query “point between the red and blue boxes,” which means finding a location that geometrically lies in the (3D) volume between the two box bounding volumes. In generating these air-point QA pairs, the system ensures the point is valid (not inside any object volume) and then projects it back to the 2D (u,v) plus its true depth Z. The language question is automatically assembled to describe the relation (e.g. “point 10 cm in front of the glass”, or “point between the yellow and green blocks”). A particularly novel ingredient is “body-length” queries. Here the instruction uses an object’s size as a unit. For instance: “Point out 5 points that are to the front-left of the yellow pencil, about 3 body lengths away.” In this case, the engine first estimates the pencil’s length in 3D (from the object bounding volume) to define “one body length,” then places the point using that metric. This forces the model to learn to infer object scales and use them in distance reasoning. The SpatialPoint-Data thus includes directional and distanceized air-point examples, ensuring the model sees a broad range of spatial language. Overall, this data engine produces a unified training set of (RGB, Depth, Question) → (u, v, Z) examples so the model learns from both surface contacts and free-space goals. Along with the synthetic QA generation, the authors also constructed a benchmark evaluation called SpatialPoint-Bench. Derived from a held-out set of RoboAfford scenes, this benchmark tests both point types under realistic conditions. It includes metrics for touchable points (2D accuracy and depth error) and for air points (direction and distance correctness). For instance, a predicted touchable point is counted correct if its pixel falls within the true labeled region; depth accuracy is measured by the mean absolute error of the predicted Z. For air points, they check two things: first whether the point lies in the correct directional sector or spatial relation (e.g. within a cone “in front” or literally between two object masks), and second whether the numeric distance constraints are satisfied (e.g. the predicted point is roughly 30 cm away as requested) while remaining free of collisions. This rigorous evaluation ensures the model must both respect the geometry and the semantics of the query when predicting a 3D coordinate. Training and Results. With the model and data in place, the SpatialPoint team trained their two-branch transformer on the 2.6M sample dataset. We’ve already noted the clever two-stage regimen to integrate depth. After training, the model was evaluated on SpatialPoint-Bench as well as on real robot tasks. The quantitative results are striking. On touchable-point queries, the model achieved about 79% accuracy – meaning nearly 4 out of 5 predicted points fell in the correct interaction region. By comparison, the vanilla vision-language baseline (the same VLM without the depth branch) barely reached ~50% accuracy and had a huge metric error. Crucially, the depth-aware model’s distance estimates were orders of magnitude better. SpatialPoint’s mean error along Z was only ~17 mm, whereas the RGB-only model’s depth guesses were off by hundreds of millimeters (on the order of 0.6–0.7 m). In practical terms, a robot using the depth-informed model could precisely reach 1–2 cm coordinates on a surface, whereas without depth it would be wildly inaccurate. On the more challenging air-point tasks, SpatialPoint also outperformed all baselines: it achieved about 0.51 “direction correctness” (the fraction of queries where the predicted point lay in the correct spatial cone or region). In contrast, a pure-RGB model scored around only 0.05–0.08 on the same metric. In other words, without explicit depth the model essentially guessed randomly in free space. The depth stream makes all the difference – it lets SpatialPoint “see” empty space as a real volume, so it can reliably place points according to directional instructions. In the authors’ words, the depth-aware model could perceive free-space goals in metric terms, whereas an RGB-only model could not go beyond blurring 2D textures. Overall, these benchmarks show that integrating depth yields substantial gains in embodied localization, both for touching surfaces and aiming into open air. Real-World Robotic Tasks. To prove the concept beyond simulations, the authors deployed SpatialPoint on real robots in three representative tasks:. Robot Arm Grasping/Placement: A language command specifies a target on a surface (for example, “pick up the red box at point X and place it on the marked spot”). The SpatialPoint model processes the camera’s RGB-D input and the phrase, and outputs a 3D point on the target surface. The robot arm then moves its gripper to that point and executes the grasp/place. In experiments, the system successfully localized the exact target point on various objects using the predicted (u,v,Z). Object-to-Location Placement: Similar to grasping, this involves placing an object at a specified destination. For instance, “put the cube on the first yellow square.” The model predicts the drop-off point in space (which lies on the floor or table). The robot’s arm then moves to that point. SpatialPoint handled this by interpreting the instruction, finding the corresponding air-point (which in this case was actually on a surface region of the floor designated by the square) and directing the arm there. Mobile Robot Navigation: Here the task is to move the robot base to a goal location specified by language, such as “go to the back left corner of the room” or “navigate to the space between the red chair and the blue table.” The input is an image of what the robot sees, along with the command. The SpatialPoint model outputs a point in free space – an actual goal coordinate in the map. Then a standard navigation controller uses that target to drive the robot. In trials, a mobile platform carrying an RGB-D camera used SpatialPoint to interpret commands and navigate to the specified point. In all these scenarios, the same underlying model and representation were used: language + vision → 3D point. For manipulation (the arm tasks), the “touchable point” modes were used; for navigation, the “air point” outputs guided the base. The real-robot tests qualitatively confirmed that SpatialPoint could act as a general-purpose localizer. Even though the training data was synthetic, the model transferred robustly to a physical workspace. This validation suggests that the model’s predictions were geometrically sound and language-grounded enough to work outside the lab. Discussion and Impact. SpatialPoint represents a step toward truly embodied language understanding. By forcing a VLM to literally output a 3D coordinate, the approach aligns high-level language reasoning with low-level execution. Instead of producing a caption or a bounding box, the robot gets the exact point it needs. This has several implications:. Explicit Geometric Bias: Modern transformers have powerful vision-language capabilities, but they were largely developed for 2D image understanding. SpatialPoint explicitly injects geometry through the depth channel. This gives the model a strong inductive bias: it no longer must “hallucinate” depth from RGB. As the authors note, this lets the model see empty space as a true 3D volume. The effect is dramatic: point predictions become metrically accurate across scenes (even unseen ones), whereas an RGB-only model’s outputs would be unreliable and scene-specific. In essence, SpatialPoint’s design encodes the physics of the world into the reasoning process. Unified Action Interface: By representing everything as $(u,v,Z)$, the model unifies diverse tasks. A single neural model can handle both manipulation and navigation queries, without separate modules for “grasp point detection” vs. “navigation planning.” The language prompt simply describes the goal, and SpatialPoint computes the 3D location. This contrasts with traditional robotics stacks where perception, planning, and language are siloed. Having a vision-language system that directly outputs actionable coordinates can simplify end-to-end learning and planning. For example, one could plug SpatialPoint into a reinforcement learning or control loop: the policy just needs to interpret or refine the point output. Data Generation Techniques: The clever synthetic data engine is noteworthy. It shows how to bootstrap spatial understanding from 2D annotations. In robotics, collecting metric labels is costly. Here they leveraged monocular depth and 3D heuristics to massively augment existing 2D affordance data. The novel “body-length” queries, in particular, ensure that the model practices inferring real-world scales. This data-augmentation approach could inspire other tasks where 3D grounding is needed but only 2D labels exist. Limitations and Future Work: It is worth noting boundaries. The dataset and examples are largely static and synthetically generated. Real-world scenes might have sensor noise, dynamic objects, or ambiguous depth, which could challenge the model. Also, air-point language can be quite subtle (e.g. “behind left of X”). The current system works in relatively clean, indoor layouts. Extending this to cluttered or outdoor environments would require more data and probably a larger model. Moreover, the reliance on a pretrained RGB backbone means it inherits any biases of that model’s training. Future work might explore training on more diverse 3D scenes or integrating other modalities (e.g. LiDAR, multi-view). Broader Impact: SpatialPoint points toward more fluent human-robot interaction. If a robot can truly understand instructions like “2 feet to the right” or “between the vase and lamp,” it brings robotics closer to natural language. The idea of predicting 3D points might also inspire improvements in augmented reality (e.g. communicating targets to AR systems) or in training virtual agents in simulators. It shows that vision-language research for robotics needs to go beyond image annotation to geometric understanding. In summary, SpatialPoint shows a practical way to "ground" language in 3D space by combining depth sensing with a transformer model. The results – nearly 80% accuracy on surface points and robust free-space pointing – demonstrate the strength of this approach. The model can then be plugged into real robots to carry out commands like grasping or navigation with minimal additional engineering. For researchers and practitioners, the main takeaways are: (1) Explicitly incorporate metric depth into vision-language models for any spatial task. (2) Use creative data synthesis to teach models about 3D relationships. (3) Treat all goal-directed commands as 3D point predictions, unifying planning modules under one interface. SpatialPoint is an exciting example of how next-generation VLMs can truly localize language in the world. References: This summary is based on the SpatialPoint paper by Zhu et al. (2026) and associated materials, which introduce the dataset and experiments described above.