Latest / Embodied AI 101 / HARP-VLA: Align the Eyes Before the Actions

Transcript

HARP-VLA: Align the Eyes Before the Actions · Embodied AI 101

• HARP-VLA: Align the Eyes Before the Actions. HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model • Xiang Zhu, Puzhen Yuan, Yichen Liu, Jianyu Chen • Institute for Interdisciplinary Information Sciences, Tsinghua University; Shanghai Qi Zhi Institute • arXiv preprint • 2026. The Paper’s Central Bet. Imagine two videos of the same manipulation. In one, a human hand reaches for a cup, rotates it, and places it upright. In the other, a robot gripper performs the same operation. From a task-level perspective, these are nearly equivalent demonstrations. But to a conventional vision encoder, they may occupy very different regions of feature space. The hand and gripper have different shapes. The trajectories follow different kinematics. Camera placement, motion blur, and occlusion patterns also differ. Latent action models are supposed to help here. Instead of asking a human video for nonexistent robot joint commands, they describe what changed between two visual states using learned action tokens. But HARP-VLA makes an important observation: these tokens are only as embodiment-agnostic as the visual representation from which they are inferred. If human and robot observations live on separate feature manifolds, a vector-quantized action model can quietly learn separate “human grasp” and “robot grasp” codes, even when the underlying manipulation is the same. The paper’s central claim is therefore simple but consequential: you cannot reliably align human and robot actions without also aligning the visual space in which those actions are defined. HARP addresses both problems jointly. It uses paired human-robot demonstrations as bridges, unpaired videos as scalable dynamics data, and an asymmetric encoder that adapts robot observations toward an anchored human-centric representation. That aligned encoder then supports a latent action model and, ultimately, a VLA policy. Why Latent Actions Are Not Automatically Embodiment-Agnostic. HARP belongs to a growing line of work that treats video prediction as a substitute for expensive action supervision. LAPA established a three-part template: learn a discrete action quantizer from visual transitions, pretrain a VLA to predict those latent actions, and use a smaller robot dataset to ground the latent space into executable commands. UniVLA extended this idea with task-centric latent actions, conditioning the action model on language and operating in a pretrained feature space to suppress task-irrelevant visual changes. In parallel, HR-Align focused on the perceptual side of the problem. It used semantically paired human and robot videos to adapt a pretrained visual encoder through lightweight robot-side modules and contrastive alignment. The goal was to preserve the useful human-centric semantics acquired during pretraining while making robot observations easier for the encoder to interpret. HARP’s novelty lies primarily in combining these directions. Visual alignment is not treated as a preprocessing step disconnected from action learning, and latent-action learning is not asked to absorb the entire embodiment discrepancy by itself. The two are trained together through frame-level dynamics prediction. HARP is therefore best understood not as a radically new VLA backbone, but as a representation-calibration layer between heterogeneous video data and an existing VLA architecture. The Data Recipe: Paired Bridges and Unpaired Scale. The framework has three stages. Stage 1 learns the aligned vision encoder, called HARP-VE, and the latent action model, HARP-LAM. Stage 2 uses them to create latent-action labels and pretrain a VLA. Stage 3 grounds the resulting model into continuous robot actions using action-labeled demonstrations. The listed Stage 1 and Stage 2 datasets total roughly 81.8 million processed frames. Human-only data include 8.9 million HOI4D frames and 36.4 million OpenEgo frames. Unpaired robot video comes from 8.6 million BridgeData V2 frames and 3.6 million frames collected with the authors’ dexterous platform. Paired sources include selected RH20T data, Human2Robot, and 5.7 million frames of self-collected human-to-dexterous-hand demonstrations. An additional 0.5 million real-robot frames participate in all three stages, while CALVIN’s 1.1 million frames are reserved for downstream action finetuning. The paired data are more than clips that happen to share a task label. HARP tries to establish frame-level correspondence. Object points are localized and tracked using a pipeline involving language-conditioned object identification, Grounding DINO, and TAPIR. Human wrist locations are obtained through WiLoR and its MANO reconstruction, while robot end-effector locations are projected into the image from recorded poses and camera extrinsics. Because humans and robots execute at different speeds, dynamic time warping aligns their wrist trajectories, using the robot sequence as the temporal reference and resampling the human sequence. That preprocessing is central to the method. “Paired” means that a human transition at a given manipulation phase is expected to correspond to a robot transition at approximately the same phase. Without that assumption, the cross-embodiment prediction objective described next would be poorly specified. Stage 1: Adapting Robot Vision Without Moving the Human Anchor. HARP begins with the fused DINOv2 and SigLIP vision encoder used by Prismatic-style VLMs. Human frames pass through the original frozen encoder. Robot frames pass through the same frozen backbone augmented with trainable residual adapters. A two-layer MLP is inserted after every attention and feed-forward block in both vision towers. The second layer is zero-initialized, so the adapted robot branch initially produces the same features as the original encoder and only moves away from them when training provides evidence to do so. This asymmetry is deliberate. Web-scale vision-language pretraining is predominantly human-centric, so the pretrained human representation is assumed to be the more semantically mature reference. Rather than moving human and robot features toward a newly learned midpoint—which could disturb the existing vision-language interface—HARP keeps the human branch fixed and maps robot observations toward it. The authors argue that this also preserves the compatibility between the vision encoder, projector, and language model inherited from Prismatic. There is an important architectural principle here: the robot adapter is not merely an image-style translator. It is trained jointly with a dynamics model. The representation is rewarded not just for looking close to a paired human feature, but for enabling actions inferred in one embodiment to explain state transitions in the other. A Small Discrete Vocabulary of Visual Transitions. HARP-LAM is a VQ-VAE-style inverse-and-forward dynamics model based on the UniVLA design. It receives patch-level features from a current and future frame, along with a frozen T5 encoding of the task instruction. A spatiotemporal transformer compresses the transition into four latent-action tokens. Each token is quantized using a 16-entry codebook, with 128-dimensional code vectors. The decoder then combines those codes, the current visual state, and language to predict the future frame’s patch-level representation rather than reconstructing pixels. Predicting features is useful because the model does not need to reproduce lighting, texture, or every pixel-level detail. It needs to explain the task-relevant evolution of the scene. The compact quantization bottleneck also discourages the action representation from memorizing an entire future observation, although it does not guarantee that every resulting token corresponds to a human-interpretable primitive. For unpaired videos, HARP performs ordinary self-prediction: infer an action from two frames in one video, then use the current frame and that action to predict the later frame in the same embodiment. This allows the system to exploit the full unpaired corpus as dynamics supervision. For paired videos, it performs the more interesting operation: cross-prediction. Suppose the action is inferred from a human transition. The decoder is conditioned on the robot’s current state and must predict the robot’s future representation. The reverse direction is trained as well: an action inferred from the robot transition must explain the corresponding human future when combined with the human current state. This creates a strong pressure on the latent code. A code that merely says “a fleshy hand moved left” will not help predict the robot future. A code that captures “move the active manipulator left while carrying the object” has a much better chance. Paired demonstrations act as a bilingual dictionary between embodiments, while unpaired videos provide the volume needed to learn broad visual dynamics. The Three Forces Shaping the Representation. Stage 1 combines three objectives. The first is future-feature prediction plus the usual vector-quantization codebook and commitment losses. This objective asks the latent action to retain enough information to explain the transition, whether the prediction remains within one embodiment or crosses between two. The second is an auxiliary manipulation loss. Dedicated tokens predict object point tracks and human-wrist or robot-end-effector locations, using visibility-aware Huber losses. These cues bias the latent action toward object motion and coarse manipulator motion—signals that are more likely to be shared across embodiments than appearance, background motion, or camera artifacts. The third component is the paper’s source-relative pair-discriminative, or SRPD, alignment loss. Its source-relative portion asks the adapted robot representation to become closer to its paired human representation than the original frozen robot representation was, by a specified margin. It does not demand that every pair reach the same absolute distance. Each robot clip is judged relative to its own starting point. That distinction matters. A naïve squared-distance objective can overcompress the space by forcing all matched examples toward near equality. Source-relative alignment only insists on a meaningful improvement. The pair-discriminative regularizer then prevents a different failure mode: every robot video collapsing into one generic human-manipulation region. For each positive pair, its distance must remain smaller than the average distance to nonmatching examples, in both human-to-robot and robot-to-human directions. Cross-prediction and SRPD consequently operate at complementary levels. Cross-prediction aligns local transition semantics; SRPD organizes pooled video representations so that matched demonstrations are close without erasing distinctions among different pairs. From Aligned Video to Executable Control. Once Stage 1 is complete, HARP-LAM assigns a sequence of four discrete action labels to transitions throughout the human and robot video corpus. Stage 2 initializes a Prismatic-7B/OpenVLA-OFT-style policy using HARP-VE, including the learned robot adapters. Given the current observation and instruction, the VLA is trained with cross-entropy to predict the HARP latent tokens. The strongest configuration freezes the aligned vision encoder during this pretraining stage. That preserves both its original web-scale visual knowledge and the human-robot geometry learned in Stage 1. The ablations later suggest that continued unrestricted updating partially overwrites the alignment. Stage 3 adds an OpenVLA-OFT-style continuous action head. For the reported downstream experiments, the policy receives third-person and wrist-camera images, language, and proprioception, and predicts ten-step action chunks. The latent space is therefore a pretraining language for manipulation, while the final action head supplies the embodiment-specific control interface. What the Experiments Actually Establish. Human-Robot Retrieval. The most direct alignment test uses 93 held-out human-robot video pairs. Each human video must retrieve its matching robot demonstration, and vice versa, using cosine similarity between pooled features. The unadapted encoder reaches 43.55% average Recall@1. Naïve L2 alignment raises that to 61.83%. The source-relative loss reaches 74.74%, and full SRPD reaches 78.50%, with 87.10% human-to-robot retrieval and 69.89% robot-to-human retrieval. Average mean reciprocal rank rises from 0.5909 to 0.8647. The UMAP plots and paired-distance distributions show the same qualitative trend for both visual and latent-action features. The asymmetry between the retrieval directions is worth noticing. Human queries retrieve robot clips substantially better than robot queries retrieve human clips. Given the one-sided adapter design, this may indicate that robot features have been successfully pulled toward human semantics but still do not reproduce the human space’s full internal structure. The pair-discriminative term narrows that problem, but does not eliminate it. Frozen-Encoder Policy Learning. The cleanest evidence that alignment helps control may be the RLBench experiment. The authors freeze each candidate vision encoder and hold the downstream policy architecture, action space, demonstration data, training schedule, and optimization budget fixed. Eighteen tasks are evaluated over 75 episodes each, for 1,350 episodes per method. The unadapted encoder averages 37.56% success. HR-Align reaches 39.70%, HARP with L2 alignment reaches 40.78%, and source-relative HARP reaches 43.41%. Full SRPD reaches 46.59%, an absolute improvement of just over nine points over the original encoder. Because only the visual representation changes, this is unusually direct evidence that the learned geometry contains more policy-relevant information. CALVIN and the Real Robot. On the difficult CALVIN ABC-to-D protocol, policies are trained in environments A, B, and C and evaluated in the unseen environment D. Success at Task 5 means completing all five instructions in a sampled sequence, while average length reports the mean number of completed subtasks. HARP-VLA achieves an average length of 4.481. It completes at least the first task in 99.8% of sequences and all five in 75.9%. OpenVLA-OFT reaches 3.917 average length, π0.5 reaches 3.875, and UniVLA reaches 3.800. Replacing SRPD with L2 alignment reduces HARP to 4.044, while allowing the vision encoder to update during Stage 2 gives 4.250. Both the alignment objective and preservation of the aligned encoder therefore contribute materially. The real-world platform combines an XArm7 with a Robotera XHand for 18 total degrees of freedom. It is evaluated on pick-and-place, pushing, button pressing, and cup flipping, with 60 trials per task. HARP-VLA averages 76.3% success: 76.7% on picking, 81.7% on pushing, 85.0% on pressing, and 61.7% on flipping. The strongest external baseline, π0.5, averages 69.2%, producing the paper’s reported 7.1-point gain. Why This Work Matters. HARP’s most useful contribution is a change in how we allocate responsibility across a learning system. A latent action model should not be expected to remove embodiment information that the visual encoder has already made dominant. Perception and action abstraction need to be co-designed. The method also offers a sensible division of labor between data types. Paired demonstrations provide calibration: they tell the model which human and robot transitions should mean the same thing. Unpaired videos provide scale: they teach the broader structure of manipulation dynamics. Robot action labels are postponed until the final grounding stage. Finally, the freezing ablation is revealing. HARP-VLA performs better when Stage 2 preserves the aligned encoder rather than continuing to optimize it freely. This suggests that cross-embodiment alignment is a valuable but somewhat fragile property—one that ordinary latent-action pretraining can overwrite unless it is explicitly protected. A Critical Reading. The first caveat concerns the word “limited.” The paired data may be limited relative to internet-scale human video, but the appendix lists roughly 23.8 million frames from explicitly paired datasets. Creating, filtering, tracking, and temporally aligning data at that scale is not a trivial bridge set. The method still reduces the need for action-labeled robot data, but it moves some of the burden toward paired cross-embodiment collection and preprocessing. The bridge itself can also be noisy. Wrist-based dynamic time warping aligns coarse execution phases, but wrist proximity does not always imply equivalent contact state, force, grasp stability, or object configuration. The auxiliary pipeline depends on object localization, point tracking, hand reconstruction, camera calibration, and visibility estimates. HARP explicitly masks unreliable tracks, but errors can propagate into both temporal alignment and the dynamics targets. The experiments do not fully isolate every ingredient. There is strong evidence for SRPD over L2 and for freezing over updating, but no comparably clean scaling study for the amount or diversity of paired data. Nor is there a full factorial ablation separating cross-prediction, unpaired self-prediction, object tracks, wrist cues, and direct feature alignment. Codebook utilization, action-token interpretability, and the ability to predict embodiment from latent tokens are also not reported. The downstream gains support the latent space’s usefulness, but do not prove that it has become strictly embodiment-invariant. Evaluation breadth remains limited. The representation experiment uses only 93 held-out pairs. The real-world study covers one robot platform and four tabletop tasks, without confidence intervals. The authors themselves identify additional embodiments, noisier cues, longer horizons, and bimanual manipulation as future work. Baseline control is mixed. RLBench is carefully controlled and is therefore particularly persuasive. The VLA comparisons are less uniform: some results are reproduced and others are taken from public reports, while OpenVLA and UniVLA use only a third-person camera under their original configurations and several other models receive both third-person and wrist views. The strongest comparisons are still useful, but they should not be interpreted as architecture-neutral rankings. The fixed human anchor is another open question. It is reasonable for present VLMs, but it assumes that human-centric representations are the appropriate semantic destination for every robot. A mobile manipulator, soft gripper, suction system, or highly non-anthropomorphic mechanism may have affordances that are poorly represented in human videos. One-sided adaptation could suppress robot-specific information that matters for control. Finally, reproducibility is currently incomplete at the documentation level. As of August 7, 2026, the public repository provides environment setup and inference commands for CALVIN and LIBERO, but its README does not document the full Stage 1 HARP-LAM and alignment training pipeline or provide a direct checkpoint download. Reconstructing the complete result therefore appears to require substantial work beyond following the published instructions. How HARP Differs from Neighboring Approaches. Against LAPA and UniVLA, HARP’s argument is that latent actions alone do not solve cross-embodiment learning. HARP preserves their scalable pseudo-labeling strategy but adds an explicit visual alignment mechanism and requires cross-embodiment decoding. Against HR-Align, HARP adds frame-level dynamics and unpaired video learning. HR-Align asks paired video embeddings to agree; HARP also asks an action extracted from one embodiment to explain the other embodiment’s future. Human2Robot and MimicDreamer take a more explicit translation route. Human2Robot jointly generates robot video and robot actions from tightly paired demonstrations. MimicDreamer uses video diffusion, egocentric stabilization, trajectory transfer, and constrained inverse kinematics to turn human footage into robot-like supervision. HARP avoids synthesizing pixels or robot trajectories, instead moving the alignment problem into feature and latent-action space. UniSkill learns embodiment-agnostic skill representations from unlabeled cross-embodiment video and uses human videos as prompts for robot policies, offering an alternative when precisely paired bridges are unavailable. A subsequent June 17, 2026 paper on motion-focused latent actions attacks the same underlying issue through physical masks, motion-background disentanglement, and separation of action intent from state-specific perception. Together, these works suggest that the next generation of video-trained VLAs will need explicit mechanisms preventing appearance and background dynamics from contaminating the action space. How a Robotics Lab Might Use HARP. For a lab with large human-video holdings, some robot video, and a modest action-labeled dataset, HARP suggests a practical sequence:. Choose the anchor carefully. Start with the exact vision tower used by the downstream VLM or VLA. Alignment is less valuable if the encoder is later replaced or its interface to the language model is changed. Treat paired data as a coverage problem. Bridge clips should span objects, motion phases, camera conditions, grasp families, and failure-relevant contacts—not merely task names. Validate alignment before training a large VLA. Monitor bidirectional retrieval, embodiment-classification accuracy, codebook usage, cross-prediction error, and nearest neighbors conditioned on task and manipulation phase. Protect the learned geometry. HARP’s results favor freezing during latent-action pretraining. If later unfreezing is necessary, retain an alignment regularizer or replay paired bridge data. Expand the auxiliary state for harder embodiments. Bimanual systems need separate wrist and contact cues; dexterous hands may require fingertip or grasp-state supervision; mobile systems need explicit ego-motion separation; 3D object tracks would be preferable to purely image-plane points when calibration permits. A multi-robot extension would naturally use a shared human anchor with separate residual adapters for each embodiment, potentially followed by a shared latent-action model. The essential test would be whether a latent code inferred from one robot can cross-decode the future of another—not merely whether both are individually close to human video. Bottom Line. HARP-VLA delivers an important correction to the latent-action story: action abstraction does not erase embodiment if embodiment is already baked into perception. Its solution—anchor human semantics, adapt robot vision, cross-predict paired dynamics, and use unpaired video for scale—is technically coherent and supported by strong retrieval, frozen-encoder, CALVIN, and real-world results. The remaining questions concern cost and reach: how little paired data is truly sufficient, how stable the alignment is across radically different robots, and whether the discrete codes capture reusable action structure rather than dataset-specific correspondence. But as a blueprint for turning heterogeneous human and robot video into a common VLA pretraining interface, HARP-VLA is a compelling step forward.