Latest / Embodied AI 101 / Look Where You Mean: Gaze2Act Gives VLAs a Live Intent Channel
Transcript
- Look Where You Mean: Gaze2Act Gives VLAs a Live Intent Channel. Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation • Kuangji Zuo, Gen Li, Bofan Lyu, Yanshuo Lu, Boyu Ma, Shijia Han, Xinyu Zhou, Xichen Yuan, Chuhao Zhou, Jiaqi Bai, Geng Li, and Jianfei Yang • MARS Lab, Nanyang Technological University • arXiv cs.RO preprint • 2026. The command was never the whole intent. Suppose a humanoid is standing in front of three cups. You say, “Hand me the cup.” The instruction is semantically clear: this is a handover task involving a cup. But it is operationally incomplete. Which cup? Where should the robot grasp it? And if you change your mind after the arm begins moving, how do you redirect the motion without constructing another elaborate referring expression?. Those questions expose a distinction that language-conditioned robot learning sometimes blurs. Language is excellent for communicating task semantics—verbs, categories, constraints, and goals—but many manipulation decisions are deictic. They depend on an implicit “this one,” “right here,” or “now that one instead.” Humans normally resolve those variables with pointing, body orientation, shared attention, and gaze. Gaze2Act turns that observation into a VLA design principle: let language describe what kind of operation should occur, while gaze identifies the object, object part, or updated target to which the operation applies. Its headline contribution is therefore not simply adding another sensor stream. It is splitting human intent into complementary semantic and spatial channels, then making sure the spatial channel survives all the way from perception to action generation. The manuscript is currently an arXiv v1 preprint submitted on May 28, 2026. The twelve authors are affiliated with the MARS Lab at Nanyang Technological University, with Kuangji Zuo and Gen Li listed as equal contributors and Jianfei Yang as corresponding author. The core idea: language says what, gaze says which and where. A normal VLA maps language, visual observations, and robot state into an action sequence. Gaze2Act adds an operator’s first-person gaze coordinate as another condition. The system must then solve three different problems. First, it must determine what the person is looking at in the operator’s egocentric camera. Second, it must find the corresponding physical object and object part in the robot’s exocentric camera. Third, it must present that result to the policy in a form strong enough to affect a long, generative action process. The authors implement Gaze2Act on GR00T N1.5. In that backbone, a frozen Eagle-family vision-language model encodes the images and instruction, while a Diffusion Transformer-style action expert processes proprioception and noisy action chunks, attending to the vision-language tokens as it generates the final trajectory. NVIDIA describes the same broad N1.5 organization: a VLM produces multimodal embeddings, and a DiT processes state and noised actions through cross-attention. Gaze2Act adds three stages around and inside that backbone:. Cross-view gaze grounding converts the operator’s fixation into a robot-view object mask and a fine-grained point. Perception-level prompting draws those cues onto the policy’s visual input. Action-level conditioning supplies a structured gaze token directly to the DiT action expert. That last step matters. If the method merely painted a dot on an image, the VLM would still be responsible for discovering that the dot is operationally important, encoding it, and preserving it through every downstream action-generation layer. Gaze2Act instead gives the signal both a visible representation and a dedicated route into the controller. Crossing the ego–exo gap without camera calibration. The first technical problem is viewpoint correspondence. The operator’s gaze coordinate belongs to the camera mounted on the glasses; the robot acts from a different camera with a different pose, focal length, scale, and pattern of occlusion. A two-dimensional gaze point cannot simply be copied from one image to the other. Gaze2Act avoids explicit camera calibration and geometric reconstruction. Its cross-view mapping is semantic rather than projective. When target selection is triggered, the gaze point becomes a point prompt for SAM3 in the egocentric image. This produces a reference mask for the object under the operator’s gaze. SAM3 also proposes candidate object masks in the robot image. The system then extracts DINOv3 ViT-L/16 patch features from five transformer depths—layers 0, 5, 10, 15, and 20—normalizes them, and averages them into a cross-view feature map. For each candidate robot-view mask, it averages the features inside the mask and chooses the candidate most similar to the egocentric reference object. The result is a coarse robot-view object mask. To recover the finer interaction location, the system takes a small feature neighborhood around the egocentric gaze point and searches for its closest match among the dense DINO features inside the selected robot-view mask. Restricting the fine search to that mask prevents, for example, a hammer-handle feature from matching a handle-like region on a neighboring tool. The output is therefore a mask-plus-point representation: the mask says which object, while the point says where on that object. This coarse-to-fine decomposition is well matched to manipulation. Object identity and contact location have different spatial scales and failure modes. A point alone is too brittle for selecting an entire transparent bottle or bowl. A mask alone cannot distinguish a hammer’s head, neck, and handle. Gaze2Act explicitly carries both. It is also worth emphasizing what “calibration-free” means here. The system is not recovering a common metric coordinate frame. It is finding the robot-view region that looks semantically most like the gaze-selected egocentric region. That removes a demanding setup step and makes the interface portable, but it also creates an important identifiability limit that we will return to later. Prompt the perception stack, then condition the action stack. Once the mask and point are in robot-image coordinates, Gaze2Act exposes them to the VLA in two ways. At the perception level, it draws a colored contour around the selected object. Multi-target tasks can use different contour colors for different referents. The associated language can then be generic—something like “pick up the outlined object”—because the image, rather than a long referring expression, carries object identity. For part-level tasks, the system also renders a soft Gaussian heatmap around the fixation. The heatmap is clipped to the selected object mask, so a noisy point does not create guidance on unrelated background pixels. The paper uses contours alone for object-level and compositional tasks; adding an arbitrary point there could unnecessarily bias the grasp. It combines the contour and heatmap when the intended object part affects the grasp or action. Fine-grained prompting is applied primarily before contact. If the grounded point no longer falls inside the tracked object mask—something that can happen when the hand occludes the target—the heatmap is removed, while the object contour remains. This is a small but sensible control decision. A fixation that was useful for approach can become geometrically stale once contact and occlusion begin. The second route bypasses the possibility that the VLM might dilute or forget the visual prompt. Gaze2Act constructs a compact spatial token with two components. One component is a two-dimensional positional encoding of the gaze point, falling back to the mask center when the point is invalid. The other is a DINOv3 embedding of the selected object crop. These are concatenated and projected into the same hidden dimension used by the GR00T action expert. Every DiT block then receives a second, independent cross-attention path. The normal branch lets the current action representation attend to global vision-language tokens. The new branch uses the same action query but separate key and value projections to attend to the gaze token. The two outputs are added together before the block continues. Conceptually, the first branch answers, “Given the scene and instruction, what behavior makes sense?” The gaze branch adds, “Generate that behavior around this object and this location.”. The output projection of the new branch is initialized to zero. At the beginning of fine-tuning, it therefore contributes nothing, and the network behaves like the pretrained policy. Training can gradually increase the gaze branch’s influence instead of injecting random spatial features into every denoising layer on the first optimization step. The resulting model grows from 2.414 billion to 2.533 billion parameters—an increase of 119.5 million, or 4.95 percent. “Lightweight” is relative here. Five percent is a modest architectural increment to a multibillion-parameter VLA, but the complete system also runs SAM3 and multiple DINOv3 encoders. The policy modification is lightweight; the end-to-end perception stack is not necessarily a low-compute controller. Interactive, but deliberately not driven by every glance. The paper frequently describes gaze as a continuous intent signal, but the deployed interface is more controlled than that wording might suggest. The operator wears Meta Aria glasses, while the Unitree G1 observes the workspace through a RealSense D455 RGB camera. A voice keyword commits the current fixation as a target selection. SAM3 then tracks that selected object, so the user does not have to stare at it throughout execution. A new voice keyword captures a new fixation, reruns cross-view matching, and allows the target to change during the rollout. This is best understood as event-triggered gaze selection with persistent tracking, rather than raw gaze coordinates streaming into the policy at every control cycle. That distinction is a strength. Human eyes frequently inspect obstacles, watch the robot hand, or look ahead to the next subgoal. Treating every fixation as a command would produce a classic “Midas touch” problem in which looking becomes equivalent to selecting. The voice trigger serves as a commit signal. Gaze supplies the spatial information, while speech says when that information should become operational. In practical HRI terms, Gaze2Act is therefore a gaze-plus-voice interface, even if the spoken component is only a trigger rather than a semantic command. Its “continuous” property is the ability to revise intent online, not continuous fixation-based servoing. Training without eye-tracked robot demonstrations. An attractive engineering property is that gaze hardware is not required during demonstration collection. The authors annotate target masks and fixation points offline on robot videos, convert them into prompts and gaze tokens, and train the policy on those synthetic gaze conditions. Live Meta Aria measurements are introduced only at deployment. The task groups contain roughly 50 to 105 demonstrations each, with 57 demonstrations used for dynamic target switching. Each group is trained for 20,000 steps on one NVIDIA RTX PRO 6000 GPU with 96 gigabytes of memory and a batch size of 80. The action expert predicts a horizon of 50 steps and uses four Euler integration steps at inference. The Eagle language components and DINO encoders remain frozen, while the relevant visual, action, and conditioning modules are optimized. For practitioners, this means the idea could be retrofitted onto an existing robot dataset if target objects and intended contact regions can be annotated or reconstructed. One does not need synchronized human gaze for every demonstration. The corresponding concern is train–test mismatch. Training conditions come from clean offline annotations, whereas deployment conditions pass through eye-gaze estimation, egocentric segmentation, tracking, cross-view matching, and possible head motion. The paper does not report systematic corruption studies covering gaze jitter, delayed fixations, incorrect target commits, or mask-switching errors. Those augmentations would be high on the list for a production implementation. What the experiments are actually testing. The evaluation uses a Unitree G1 humanoid across seven categories. Fifteen static task variants cover ambiguous objects, unseen instances, transparent objects, compositional pick-and-place, subpart grasping, and actions conditioned on the selected object part. A separate dynamic setting brings the total to sixteen real-robot tasks. Each static condition is evaluated for 50 trials, with objects rearranged between trials. The ambiguous and unseen categories include cups, bread, and fruits. Transparent-object trials use cups and bottles. Compositional tasks require selecting both an object and its destination, such as one bread item and one bowl. Part-level trials ask the same policy to grasp a hammer by its handle, head, or neck. In another category, looking at different regions of a cup determines whether the resulting behavior is a handover or pouring action. The two reported metrics separate grounding from completion. Intent accuracy asks whether the robot reaches toward the user-intended object or part. Task success requires the full manipulation to finish. That distinction is useful: a robot can identify the correct cup and still drop it, or execute a stable grasp on the wrong cup. The comparison includes vanilla GR00T N1.5, RoboGround, and ControlVLA. RoboGround uses language-grounded masks and a grounded perceiver; ControlVLA injects object-centric masks through a ControlNet-style adaptation. For experimental control, the authors reimplement both methods on the GR00T N1.5 backbone and train every method from the same robot trajectories and schedule. The language-based systems receive specific, supposedly unambiguous descriptions, whereas Gaze2Act receives generic instructions and obtains the referent from gaze. This is a reasonably strong within-paper comparison, but it should not be confused with evaluating the untouched original RoboGround and ControlVLA systems. The study compares harmonized GR00T-based interpretations of their conditioning mechanisms. Reading the headline numbers carefully. Across the first ten, object-level task variants, Gaze2Act averages 93.0 percent intent accuracy and 89.0 percent task success. The strongest competing average is ControlVLA’s 68.0 percent intent accuracy and 46.4 percent success. Across all fifteen static tasks, Gaze2Act reaches 83.5 percent average task success, compared with 41.5 percent for ControlVLA, 39.6 percent for RoboGround, and 23.2 percent for vanilla GR00T. Part-level results are similarly revealing. Gaze2Act averages 80.4 percent intent accuracy and 72.4 percent success across hammer-part and cup-action conditions. The detector-based baselines do not expose a part-specific condition, so the paper does not report part-level intent accuracy for them; their best average task success is 31.6 percent. The gains are especially large on transparent objects and compositional tasks. Gaze2Act achieves 86 and 84 percent success on the transparent cup and bottle variants. On selecting a bread item and then the intended bowl, it reaches 96 percent intent accuracy and 94 percent task success. These conditions favor an explicit, persistent referent over a policy that must repeatedly infer the intended instance from language. The table is not literally a sweep, despite the surrounding prose. RoboGround slightly outperforms Gaze2Act in the ambiguous-cup condition: 96 versus 92 percent intent accuracy and 94 versus 92 percent success. The broader averages strongly favor Gaze2Act, but this row shows that a reliable language-derived mask can remain competitive when the object description and visual appearance are sufficiently favorable. Dynamic steering is the most interesting result and the least solved one. After motion has begun, the operator commits a new cup target. RoboGround succeeds in 4 of 30 trials, ControlVLA in 5, and Gaze2Act in 14. The dual gaze pathways clearly improve redirection, but 14 out of 30 is still below half. The experiment demonstrates a capability, not yet a robust interactive controller. The ablation explains why both pathways are needed. The policy ablation uses a compositional bread-to-bowl task and the hammer-part task. The unmodified baseline succeeds in 17 of 60 bread trials and 15 of 60 hammer trials. Visual prompting alone raises those counts to 34 and 28. Zero-initialized action conditioning alone reaches 40 and 19. The full Gaze2Act model reaches 55 and 39. The division of labor is intuitive. A visible heatmap is particularly effective when the robot must align its hand with a small object part. Direct action conditioning is especially useful when the controller must preserve a binding between multiple referents over time—for example, this bread and that bowl. Combining the two routes gives the VLM an interpretable spatial cue and gives the action expert a persistent control condition. Zero initialization is also consequential. Randomly initialized action conditioning drops the bread result from 40 to 24 successes. This supports the broader adapter-design lesson: when adding a new modality to a pretrained policy, begin with an exact or near-exact recovery of the old function and allow the new condition to acquire influence gradually. The paper’s attention visualization is consistent with this account: vanilla GR00T remains spatially diffuse, while Gaze2Act concentrates attention near the outlined targets. It is useful qualitative evidence, though a single attention visualization is not a causal explanation by itself. How Gaze2Act differs from nearby work. Gaze2Act sits at the intersection of two rapidly developing lines of research: gaze-informed robot learning and explicit spatial interfaces for VLAs. RoboGround and ControlVLA established that masks can be powerful intermediate representations. RoboGround uses masks to identify objects and placement regions, while ControlVLA progressively injects object-centric representations through zero-initialized adaptation layers. Their spatial information, however, is normally derived from language or task prompts rather than from a user’s live, revisable attention. Point-VLA and VP-VLA make the visual interface even more explicit. Point-VLA augments instructions with visible bounding boxes, while VP-VLA uses a high-level planner to decompose tasks and render crosshairs or boxes for a lower-level controller. These works support Gaze2Act’s premise that pixel-space prompts can be more precise than textual coordinates or elaborate referring expressions. Gaze2Act’s difference is that the prompt originates from the human’s gaze and is also injected directly into action generation. Other work uses gaze during learning rather than interactive deployment. GazeVLA pretrains on egocentric human data and learns gaze as an intermediate representation of intention before predicting robot action. Gaze-regularized VLA models use human gaze to shape attention during training without requiring eye-tracking hardware at inference. These methods ask what human gaze can teach a policy; Gaze2Act asks how a particular human can steer a policy now. GAMMA is the closest conceptual neighbor on the interactive side. It maps gaze from smart glasses into a robot workspace, uses VLM reasoning to infer a high-level task, and invokes modular perception, grasping, planning, and execution functions. Gaze2Act replaces that skill-oriented modular controller with a trained VLA whose low-level action generation is directly gaze-conditioned. So the novelty is not gaze alone, visual prompting alone, or zero-initialized conditioning alone. It is the combination of live gaze acquisition, marker-free cross-view matching, object-and-part grounding, dual injection into a modern action-generative VLA, and online target revision. What is convincing—and what remains unresolved. The most convincing aspect is the decomposition of intent. Gaze2Act does not ask gaze to encode an entire task, and it does not ask language to perform pixel-accurate selection. Each modality handles the information for which it is naturally suited. The event-triggered protocol is similarly pragmatic. Latching and tracking the selected object avoids requiring unnatural sustained fixation and reduces accidental commands. The dual conditioning architecture then addresses a common weakness of multimodal robot systems: an input can be present at the front of the network yet become functionally irrelevant by the time actions are generated. But the calibration-free matcher has a fundamental ambiguity. Its coarse stage selects the robot mask whose internal DINO features most resemble the egocentric reference mask. If two physical objects are truly identical and symmetrically observed, appearance alone contains no information establishing which instance is the same one. This is an inference from the architecture, not an empirical result. DINO’s receptive fields may leak some surrounding context, but no explicit geometry or contextual scene correspondence guarantees that the symmetry will be broken. The reported ambiguous scenes often use near-duplicates with patterns, colors, or appearance differences rather than perfectly interchangeable instances. The evaluation also does not isolate the grounding stack. Intent accuracy is measured through the robot’s reach, so errors from gaze estimation, egocentric segmentation, object tracking, cross-view object matching, part matching, and the policy itself are entangled. A stronger analysis would report object-ID correspondence accuracy, robot-view mask IoU, point-transfer error, tracking survival, and policy success given oracle grounding. Latency is another missing dimension. Dynamic steering requires gaze estimation, segmentation, candidate-mask generation, DINO feature extraction, matching, policy update, and interruption or replacement of the current action plan. The paper reports neither end-to-end retargeting latency nor control frequency. Those numbers matter when a moving humanoid arm is expected to respond safely to a changed intention. The claim that gaze is natural and low burden is plausible but not directly established here. There is no reported multi-user usability study, workload measurement, calibration analysis, or examination of individual gaze strategies. GAMMA’s user study is instructive in this respect: participants completed tasks more quickly and with lower reported demand under greater autonomy, yet sometimes preferred the direct-control baseline because it preserved agency when autonomous predictions were wrong. Finally, dynamic redirection introduces a safety problem as well as a control problem. A target switch should carry confidence, temporal hysteresis, collision checking, and an explicit stop or undo mechanism. A system that confidently follows an unintended fixation could be worse than one that asks for confirmation. The paper acknowledges sensitivity to gaze errors, occlusion, head motion, viewpoint changes, and exploratory gaze, but leaves multimodal fusion and stronger online adaptation to future work. Reproducibility is also pending. As of August 7, 2026, the public repository hosts the project website and states that project code, setup instructions, and the evaluation protocol are still forthcoming. How I would adapt Gaze2Act in a new robotics stack. For researchers considering this approach, the paper suggests a useful implementation blueprint, but I would modify it in several ways. Treat gaze as a committed observation, not an unconditional command. Preserve the event-triggered design, but support multiple commit mechanisms: a spoken keyword, blink, hand gesture, controller button, or dwell-time threshold. Maintain explicit states for preview, confirm, track, retarget, cancel, and stop. Use geometry when geometry is cheap. If both cameras are calibrated and depth is available, intersecting a three-dimensional gaze ray with a scene model may be faster and more identifiable than semantic matching. Use DINO-and-SAM correspondence when calibration is unavailable, as a fallback when geometry fails, or to associate the gaze ray with a persistent semantic object ID. Replace the hard argmax with an intent belief. Carry several cross-view candidates with probabilities rather than immediately collapsing to one mask. The policy condition should include confidence, target age, tracking quality, role—source object versus destination—and whether the point is currently visible. Low confidence should trigger confirmation or abstention rather than action. Keep the dual injection pattern. For a diffusion or flow-matching action expert, a zero-initialized parallel cross-attention branch is a clean solution. For an autoregressive action model, the equivalent might be persistent role-tagged gaze tokens, adapter attention at every action layer, or feature-wise modulation. The essential property is that gaze remains directly available during action generation rather than existing only as pixels in an early image encoder. Train on corrupted intent signals. Perturb fixation points, shift masks, insert tracking gaps, delay target updates, simulate head motion, and occasionally provide a fixation unrelated to the task. Include examples where the correct output is to continue tracking, ask for confirmation, or stop. Otherwise the model learns that every gaze condition is accurate and meaningful, which is not how human attention behaves. Make action chunks interruptible. A long horizon is useful for smooth control, but dynamic intent requires the current chunk to be cancellable. Measure the time from target commit to changed end-effector motion, and enforce safe transitional behavior while the new plan is being generated. The evaluation should be decomposed accordingly: gaze-estimation error, ego segmentation quality, cross-view object identity, part-transfer error, retarget latency, false-switch rate, intent compliance, task success, and safety interventions. It should include multiple operators, exact duplicate objects, reflective and transparent surfaces, severe occlusions, and trials where the operator looks at the robot hand or a hazard instead of the manipulation target. For compositional and long-horizon tasks, I would also generalize the single gaze token into a small set of role-tagged entities: object to pick, destination, contact region, obstacle, and perhaps a negative region to avoid. That turns gaze from a one-object selector into an interactive spatial workspace for the policy. The larger lesson. Gaze2Act is not mind reading. It is a thoughtful allocation of communication bandwidth. Language carries semantic structure. Gaze supplies a low-bandwidth but high-precision spatial reference. The robot’s visual and proprioceptive state grounds both in the current world. Gaze2Act’s most important architectural insight is that this spatial reference should not be consumed once and forgotten: it should be made visible to perception and remain explicitly available to the action generator. The static results show that this can dramatically improve object selection, multi-referent manipulation, and part-conditioned behavior. The dynamic result shows the direction of travel while also marking the remaining distance: online intent revision improves from a handful of successes to 14 out of 30, but reliable interactive steering is not solved. For embodied AI, that is still an important step. The future interface to a generalist robot is unlikely to be language alone. It will be language combined with gaze, gesture, correction, uncertainty, and the shared physical context. Gaze2Act offers a concrete example of how one of those channels can be integrated into a contemporary VLA—from the human eye, through cross-view grounding, and into the layers that actually decide how the robot moves.