Latest / Embodied AI 101 / Stop Teaching the VLA Your Camera: Robot-Centric Pointmaps as an Action-Aligned Visual Interface

Transcript

Stop Teaching the VLA Your Camera: Robot-Centric Pointmaps as an Action-Aligned Visual Interface · Embodied AI 101

• Stop Teaching the VLA Your Camera: Robot-Centric Pointmaps as an Action-Aligned Visual Interface. See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models • Byungkun Lee, Dongyoon Hwang, Dongjin Kim, Hojoon Lee, Minho Park, and Jaegul Choo • KAIST AI and Holiday Robotics • arXiv:2607.11498 • 2026. The Coordinate Transform Hiding Inside the Policy. There is a deterministic coordinate transform sitting in almost every vision-based robot stack. A camera sees an object at some pixel and depth. Calibration tells us how to lift that observation into 3D and express it relative to the robot. Kinematics tells us where the end effector is. None of this is conceptually mysterious. And yet, many vision-language-action models are asked to rediscover much of that relationship statistically. The input arrives as camera-view RGB. The output is usually an end-effector command expressed in the robot base frame. If the camera never moves, this arrangement can work surprisingly well. The policy learns one stable association between visual locations and robot motions. A mug appearing near a particular image coordinate tends to require a predictable Cartesian displacement. Move the camera, however, and the visual location changes while the correct robot-frame action does not. Aggregate data from several cameras, laboratories, or collection episodes, and the policy is no longer learning a single perception-to-action mapping. It is learning a family of mappings implicitly indexed by camera pose. That distinction—between handling a fixed frame mismatch and generalizing that mismatch across viewpoints—is the central insight of See like a Robot. Rather than giving a VLA more capacity and hoping it internalizes projective geometry, the paper proposes transforming sensed geometry into the robot’s coordinate system before policy learning. The result is a robot-centric pointmap: an image-shaped tensor whose pixels contain metric XYZ coordinates in the frame where actions are defined. The paper’s contribution is not that pointmaps are a new representation. Pointmaps already appear throughout modern 3D vision and have recently entered diffusion-based robot policies. The important contribution is the combination of three choices:. Express the geometry in the robot’s action frame. Preserve the dense image grid rather than converting it into an unordered point cloud. Inject it through the pretrained VLA’s existing visual pathway. That sounds almost trivial. It is also exactly the kind of representation-level decision that can remove a substantial learning burden from the network. Why Camera Diversity Makes an Old Problem Worse. It is useful to distinguish two kinds of viewpoint robustness. The first is test-time camera robustness. A policy is trained with one camera configuration and deployed with another. This is the familiar domain-shift problem. The second is training-time viewpoint diversity. Demonstrations inside the training set already come from many camera placements. The policy must fit all those views simultaneously, even before anyone asks it to extrapolate to a new one. See like a Robot primarily targets the second regime. This matters because modern robot datasets increasingly combine demonstrations from different episodes, operators, institutions, and hardware setups. In such datasets, camera variation is not merely an augmentation to survive at deployment. It becomes a source of conditional complexity inside the supervised learning problem itself. Consider two demonstrations that put the same cup into the same cabinet using the same robot-frame motion. In one demonstration, the cabinet may appear toward the upper-left of an external camera image. In another, it may appear near the center and at a different scale. From the action labels alone, those demonstrations are equivalent. From the RGB tensors, they can look dramatically different. An RGB policy could, in principle, infer the camera pose from visual context, recover the scene geometry, and map that geometry into the robot frame. But robotics datasets are small enough that the network often finds cheaper shortcuts. The independent “Know Your Camera” study demonstrated that policies without explicit camera information can use static backgrounds and workspace geometry as proxies for camera pose; those shortcuts fail when the robot, table, or environment is rearranged. The pointmap approach asks a more basic question: if calibration and depth are already available, why make the policy infer this transform at all?. A depth image is not sufficient by itself. Depth gives metric information along camera rays, but it remains camera-centric. Likewise, camera intrinsics and extrinsics contain the information required to recover robot-frame geometry, but merely conditioning a neural network on those values still leaves the coordinate conversion to the network. A point cloud solves the coordinate problem but creates an interface problem. It abandons the regular image grid on which the VLA’s pretrained visual stack was built. It usually requires subsampling, a specialized 3D encoder, and a separate fusion path. The robot-centric pointmap is intended to occupy the favorable middle ground: metric 3D like a point cloud, but dense and image-shaped like RGB. What Exactly Is a Robot-Centric Pointmap?. The construction is conventional geometry. For every RGB-D camera, the system first uses the depth value and camera intrinsics to back-project each image pixel into a 3D point in that camera’s coordinate frame. It then applies the calibrated camera-to-robot transform, producing the same point in the robot base frame. The output retains the original image height and width. Instead of storing red, green, and blue at each location, it stores the point’s X, Y, and Z coordinates. A pixel corresponding to the mug’s handle therefore contains the handle’s robot-frame position. If the camera moves, the handle may land at a different pixel, but its stored XYZ value remains the same. The authors go one step further and subtract the current end-effector position from every point. This creates an end-effector-centered pointmap. A target point is no longer encoded primarily as “position 0.6 meters forward and 0.2 meters left of the robot base.” It is encoded as “position 8 centimeters forward and 3 centimeters left of the gripper.”. That is particularly well matched to delta end-effector actions. The pointmap and action now share not only robot-base axis conventions but also a common translational origin. A point’s coordinates become an approximation to the displacement the gripper must close before interaction. One nuance is worth emphasizing. The representation is centered at the end effector, but it is not transformed by the end effector’s full orientation. Its axes remain aligned with the robot frame. It is therefore translation-canonicalized rather than fully gripper-frame canonicalized. A natural extension would be to rotate the geometry into the current tool frame as well, although whether that helps would depend on how orientation actions are parameterized. The pointmap does not make the complete tensor invariant to viewpoint. Occlusion changes which surfaces are visible. Perspective changes their arrangement on the pixel grid. The same point moves between pixels as the camera moves. What becomes consistent is the coordinate value assigned to a visible physical point, not the entire rasterized observation. That is still a substantial simplification. The network no longer has to decode a target’s robot-frame position from a camera-dependent combination of image coordinates, depth, intrinsics, and extrinsics. Integrating Pointmaps Without Rebuilding the VLA. The integration mechanism is deliberately minimal. Each RGB image goes through the VLA’s normal visual encoder. Its corresponding pointmap goes through a second image encoder with the same architecture, initialized from the RGB encoder’s pretrained weights. In the reported implementation, these are separate SigLIP vision towers. The resulting pointmap tokens have the same number, ordering, and dimensionality as the RGB tokens. The two streams are then added element by element. RGB patch token number 17 is combined with pointmap patch token number 17, preserving the fact that both describe the same image region. The fused sequence is passed to the rest of the VLA together with language and proprioception. No point-cloud encoder, voxel grid, or additional token sequence is introduced. There are two interesting implications here. First, the method treats geometry as a residual correction to the existing visual representation. The VLA retains the semantics, texture cues, object identity, and web-scale priors carried by RGB, while the second branch injects metric position. Second, preserving the token count avoids increasing the sequence length seen by the multimodal and action transformers. The cost is instead concentrated in the additional vision tower. So the project page’s description—“one extra encoder and one element-wise addition”—is accurate, but “simple” should not be confused with “computationally free.” A second SigLIP encoder can represent meaningful parameter, memory, and vision-front-end latency overhead, none of which the paper quantifies. Initializing the geometry encoder from RGB weights is also intriguing. XYZ channels are not natural images, so early RGB filters are not obviously ideal. But the initialization supplies a pretrained patch projection, spatial attention structure, and token hierarchy. The paper does not compare this initialization against a randomly initialized pointmap tower, leaving open how much of the gain comes from the pointmap itself versus transferable visual-encoder optimization priors. The Ablations Are the Most Important Part of the Paper. The headline benchmark improvements are useful, but the paper is most convincing when it decomposes the representation design. Should the Network Learn the Transform?. The first study compares four inputs using a controlled π-style policy initialized from PaliGemma rather than a robot-pretrained π checkpoint. RGB alone reaches 27.9% success. Adding per-pixel Plücker rays—six-dimensional representations of the rays implied by camera intrinsics and extrinsics—raises success slightly to 28.7%. Adding depth to those rays reaches 31.6%. At this point, the policy has all the information needed to calculate robot-frame XYZ coordinates. It has depth, intrinsics, and extrinsics. But it must learn the conversion. Precomputing that conversion and supplying a pointmap raises success to 34.7%. The 3.1-point difference between Plücker-plus-depth and pointmap is the cleanest evidence for the paper’s thesis: having the information is not equivalent to presenting it in the representation the action predictor needs. This is an instance of a broader robotics principle. Universal function approximation does not imply data-efficient discovery of known structure. If a reliable transformation is available analytically, applying it outside the network can reduce the hypothesis space that learning must search. Does Image Form Actually Matter?. The second study holds the underlying robot-frame points approximately constant while changing their representation and fusion. A lightweight point-cloud MLP reaches only 24.2%, below the RGB baseline. A pretrained Point Transformer v3 does much better at 32.8%, showing that a capable 3D encoder can make use of the geometry. An image-form pointmap whose tokens are concatenated with the RGB sequence reaches 30.7%. When those same pointmap tokens are instead added to their spatially corresponding RGB tokens, success rises to 34.7%. This result supports two claims at once. Dense image-form geometry is easier to connect to a pretrained 2D VLA than an unordered point set, and explicit patch correspondence is better than merely appending another modality. The experiment does not fully separate all contributing factors. The pointmap is denser, uses an image encoder initialized from pretrained weights, and permits element-wise fusion. The point-cloud versions are subsampled and require different encoders. The authors themselves acknowledge that they test only one point-cloud sampling budget and that a larger budget could narrow the gap. The right conclusion is therefore not that pointmaps dominate point clouds universally. It is that pointmaps are an unusually convenient representation when the downstream policy already has a strong image-based interface. Where Should the Origin Be?. A robot-base-centered pointmap reaches 34.7% under the fixed evaluation view and falls to 32.7% when evaluation views are randomized. Centering the same geometry on the current end effector raises fixed-view success to 36.9% and randomized-view success to 36.6%. The performance drop under camera randomization shrinks from two points to three-tenths of a point. The intuition is compelling. A grasp near the sink and an equivalent grasp near the microwave may occur at unrelated base-frame coordinates. Relative to the gripper, however, the target geometries can be nearly identical. End-effector centering clusters action-relevant points around a common origin and converts absolute workspace variation into local interaction geometry. This ablation also suggests that the method is doing more than camera normalization. It is introducing a form of translation canonicalization over the workspace. What Happens as Training Views Become More Diverse?. The authors then vary only the amount of camera jitter applied to RoboCasa’s two third-person cameras. The no-variation setting uses fixed cameras. The default setting perturbs position and orientation with standard deviations of five centimeters and three degrees. The high-variation setting doubles those values to ten centimeters and six degrees. The wrist camera remains fixed. As variation increases from none to high, RGB-only success falls from 34.5% to 24.9%, a loss of 9.6 points. RGB plus pointmap falls only from 37.6% to 35.8%, a loss of 1.8 points. Equivalently, the pointmap advantage grows from 3.1 points when cameras are fixed to 10.9 points under high viewpoint diversity. That trend is more informative than a single benchmark average. It directly connects the method’s benefit to the nuisance factor it is designed to remove. What the Full Experiments Show. The main simulation evaluation uses 24 RoboCasa atomic tasks spanning doors, drawers, coffee preparation, pick-and-place, and object-turning. Each task provides 50 human demonstrations and is evaluated over 50 episodes. Training demonstrations contain randomized third-person viewpoints, while the main table uses a fixed reference viewpoint at evaluation. The authors test both π0.5 and SmolVLA, which differ in scale and action-expert design. The main numbers are:. | Evaluation | RGB baseline | With pointmap | Gain | |---|---:|---:|---:| | RoboCasa, π0.5, fixed evaluation view | 55.3 | 62.9 | +7.6 | | RoboCasa, π0.5, randomized evaluation views | 54.2 | 60.3 | +6.1 | | RoboCasa, SmolVLA, fixed evaluation view | 37.2 | 41.4 | +4.2 | | Real robot, seen external-camera placement | 73.3 | 78.3 | +5.0 | | Real robot, unseen external-camera placement | 55.0 | 66.7 | +11.7 |. On the fixed-view RoboCasa evaluation, π0.5 with pointmaps outperforms the camera-aware KYC and OC-VLA baselines, the point-cloud-augmented GeoVLA and PointVLA baselines, and the standalone FP3 point-cloud foundation policy. The strongest alternative is KYC at 59.1%, compared with 62.9% for pointmaps. The per-task breakdown is particularly strong for π0.5: adding the pointmap improves or matches the RGB policy on 22 of 24 tasks. CoffeePressButton rises from 46% to 76%, and CoffeeServeMug rises from 48% to 64%, both tasks where metric target localization plausibly matters. The SmolVLA result is more mixed than its average suggests. Pointmaps strongly improve doors, drawers, and pick-and-place, but slightly reduce the fixed-view coffee and object-turning category averages. Under randomized evaluation views, SmolVLA’s coffee score drops from 41.3% to 26.0% even though its overall average improves. This heterogeneity suggests that token addition and training hyperparameters may not transfer equally well across VLA backbones. The Real-Robot Test. The real-world setup uses a Franka Research 3 with a fixed wrist-mounted RealSense D405 and an external D435i. The external camera is moved among three placements during data collection. For each placement, the authors collect 15 demonstrations for each of four tasks: pick-and-place, block stacking, drawer opening, and drawer closing. That produces 180 demonstrations in total. Evaluation is performed both at a camera placement seen during training and at a held-out placement. Each task receives 15 trials per camera condition. At the seen placement, RGB-only π0.5 achieves 73.3%, and pointmap-enhanced π0.5 reaches 78.3%. At the unseen placement, RGB-only drops to 55.0%, while the pointmap model reaches 66.7%. The advantage therefore more than doubles, from five points to 11.7 points. A from-scratch DP3 point-cloud policy reaches 63.3% and 48.3% in the two conditions. This is encouraging evidence that the method survives real stereo-depth noise and hand-eye calibration rather than depending entirely on simulation geometry. At the same time, the wrist camera remains unchanged throughout, so the experiment is not a complete shift of every visual stream. The task set is also small, and 15 trials per task leaves fairly wide statistical uncertainty. Three Ways to Resolve the Frame Mismatch. The broader literature now contains three particularly clean strategies for camera-robust policy learning. KYC, or “Know Your Camera,” conditions the observation on camera geometry. It encodes calibrated per-pixel rays using Plücker coordinates and gives them to ACT, Diffusion Policy, or SmolVLA. It does not require depth, which is an important practical advantage. Its extensive experiments show that explicit camera conditioning helps prevent RGB policies from exploiting background-based pose shortcuts. But without depth, the network must still determine where along each ray the relevant object lies. OC-VLA moves the action labels into the camera frame. During training, robot-frame actions are transformed into the observing camera’s coordinate system. At inference, predicted camera-frame actions are transformed back for execution. This adds essentially no neural architecture and performed well in both simulation and real Franka experiments. The conceptual challenge is multi-view control: an action can be expressed in one chosen camera frame, but not simultaneously in several camera frames. In the See like a Robot comparison, OC-VLA is anchored to a static third-person camera. Robot-centric pointmaps move the observation geometry into the action frame. Every camera can be mapped into the same robot-centric coordinate system, while each RGB-pointmap pair retains local pixel correspondence. These approaches are more complementary than mutually exclusive. KYC is attractive when only RGB and camera calibration are available. OC-VLA is nearly free architecturally and may be ideal for a stable single-camera setup. Pointmaps are strongest when reliable depth is available, multiple cameras must share one coordinate system, and preserving a pretrained 2D visual interface is valuable. A second comparison is against methods such as GeoVLA and PointVLA, which inject point-cloud features through dedicated geometric modules or action-expert blocks. PointVLA, for example, uses a two-stage schedule that first establishes the 2D policy and then introduces point-cloud features. These methods can learn richer set-based 3D abstractions, but their geometric path cannot directly inherit the entire image-form interface used during VLA pretraining. Finally, pointmaps themselves are not novel to this paper. PointMapPolicy uses structured pointmaps with an xLSTM-based imitation policy, while ReMAP-DP combines reprojected views with pixel-aligned pointmaps in a dual-stream diffusion policy. See like a Robot differentiates itself by asking which coordinate frame the pointmap should inhabit and by integrating it into pretrained VLA backbones with token-wise addition. What the Paper Proves—and What It Does Not. The strongest part of the evidence is the controlled sequence of ablations. The paper does not merely append depth and report a higher number. It compares implicit versus explicit coordinate conversion, pointmaps versus point clouds, concatenation versus aligned addition, base centering versus end-effector centering, and low versus high camera variation. Collectively, those studies support a specific causal account: the gains arise because the representation exposes dense metric geometry in a coordinate system aligned with action. There are nevertheless important boundaries. First, the method requires synchronized depth, camera intrinsics, and camera-to-robot extrinsics at both training and deployment. The real system uses RealSense stereo depth and a one-time hand-eye calibration. The paper does not perturb calibration numerically, measure sensitivity to drift, or evaluate depth failure around reflective, transparent, or thin objects. Second, the camera experiments change placement and orientation, but not the number of cameras or their fields of view. The authors explicitly leave those shifts to future work. New occlusion patterns or the disappearance of an entire input view may present a different problem from extrinsic variation. Third, the point-cloud comparison is not exhaustive, and the paper’s matched VLA baselines are reimplementations because official implementations were unavailable to the authors. Matching everything to π0.5 improves backbone fairness, but it introduces implementation-fidelity uncertainty. Fourth, the simulation results are reported from a single final checkpoint rather than across multiple random seeds with confidence intervals. Fifty episodes per task provide meaningful aggregate evidence, but optimization variance is not characterized. Fifth, architecturally minimal fusion may not be optimal fusion. A raw element-wise sum assumes that RGB and geometry token magnitudes, semantics, and usefulness remain compatible throughout training. The mixed SmolVLA category results suggest that a learned gate, residual scale, cross-attention adapter, or backbone-specific initialization could be worth investigating. As of September 8, 2026, the official repository contains the project website rather than the training implementation: its code button remains disabled in the page source. The method is straightforward enough to reproduce conceptually, but exact preprocessing, normalization, and training behavior would be easier to verify with an official release. How I Would Adapt This in a Robotics Stack. For a team already fine-tuning an image-based VLA, the core integration should be relatively contained. Start by making the frame contract explicit. Record exactly which axes and origins define translation and rotation actions. Confirm whether end-effector deltas are base-oriented, tool-oriented, or mixed. A pointmap should be constructed in a frame that makes the action representation as local and direct as possible. Then implement the geometry pipeline:. Align depth to the RGB image and back-project every valid pixel. Transform those points from each camera frame into the robot base frame. Subtract the current end-effector translation. Apply every crop, resize, and geometric augmentation identically to RGB and pointmap. Encode the pointmap with a cloned vision tower and fuse corresponding patch tokens. The practical details will matter. Coordinate units and ranges must be consistent across robots and datasets. Invalid depth needs an explicit policy rather than an accidental zero that might be interpreted as a point at the gripper. I would consider a validity or confidence channel, although that would require modifying the three-channel encoder interface. I would also test workspace-bounded normalization, clipping, and per-axis scaling while preserving the sign and relative metric structure. For wrist cameras, the camera-to-base transform should be recomputed using forward kinematics and the hand-eye calibration at every timestep. For movable external cameras, calibration must either be refreshed or estimated online. A stale extrinsic matrix can turn a supposedly canonical representation into a confidently wrong one. The most important evaluation would not be another average benchmark score. It would be a stress matrix varying:. camera translation and rotation beyond the training range;. calibration error;. depth noise and missing pixels;. camera count and field of view;. object position across the reachable workspace;. end-effector orientation;. action coordinate conventions. I would also test a full tool-frame pointmap, rotating points into the end effector’s orientation rather than merely subtracting its position. That might further canonicalize approach motions, although it could hurt tasks requiring stable awareness of gravity or base-aligned workspace structure. A hybrid representation—one base-oriented and one tool-oriented—may be more robust. A second promising direction is gated residual fusion. Initialize a learned geometry gate near zero, allowing the pretrained RGB policy to remain intact at the start of fine-tuning, then let training determine how strongly pointmaps should modify each layer or token. This could be especially useful for smaller backbones such as SmolVLA. Finally, if depth hardware is unavailable at deployment, the pointmap could still serve as a training-time teacher. A geometry-equipped policy might supervise an RGB-only student, or a monocular geometry model could predict the pointmap. That would sacrifice some metric reliability, but it would preserve the central principle: make action-aligned geometry an explicit intermediate target rather than expecting it to emerge accidentally inside the policy. The Larger Lesson. The message of See like a Robot is not simply that 3D inputs beat 2D inputs. Point clouds are 3D, yet they do not automatically win. Plücker rays plus depth contain enough information, yet they underperform the precomputed pointmap. The lesson is that geometry works best when it is expressed in the right coordinate frame and delivered through the right architectural interface. Robot-centric pointmaps remove a known transform from the learning problem. End-effector centering removes irrelevant workspace translation. The dense grid preserves compatibility with pretrained visual encoders. Token-wise addition preserves RGB-geometry correspondence. That combination produces strong gains on π0.5, smaller but positive average gains on SmolVLA, and an advantage that grows when the external camera moves beyond the training placements. The result is not calibration-free robot intelligence, and it is not complete viewpoint invariance. It is a carefully chosen inductive bias. As robot learning scales toward increasingly heterogeneous datasets, those biases may become more important rather than less. Bigger models can learn more transformations, but they should not have to spend data and capacity relearning geometry that the robot already knows. When calibration is available, the cleanest policy may be to let the robot see the scene in the same frame in which it acts.