Latest / Embodied AI 101 / The Action Is Missing: Turning Human Video into Robot Control
Transcript
- The Action Is Missing: Turning Human Video into Robot Control. From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data • Zhiyuan Feng, Qixiu Li, Huizhi Liang, Rushuai Yang, Yichao Shen, Zhiying Du, Zhaowei Zhang, Yu Deng, Li Zhao, Hao Zhao, Zongqing Lu, Oier Mees, Marc Pollefeys, Jiaolong Yang, and Baining Guo • Tsinghua University, HKUST, Xi’an Jiaotong University, Fudan University, Microsoft Research Asia, Peking University, and Microsoft Zurich • IJCAI 2026 Survey Track and arXiv • 2026. The data bottleneck has changed. Foundation models succeeded in language and vision partly because their raw material was already abundant. Text comes with tokens. Images come with captions, metadata, and naturally occurring semantic structure. Robot learning has a less convenient substrate: the most valuable label is an executable action, and that label is inseparable from a particular body, control interface, sensor configuration, and collection setup. A robot demonstration may include synchronized camera streams, joint positions, end-effector poses, gripper commands, language instructions, and carefully calibrated timing. It is extremely useful—but expensive to collect, difficult to standardize, and often nonportable across embodiments. Even large efforts such as Open X-Embodiment, DROID, and AgiBot World remain constrained by the physical systems and collection protocols that produced them. Human video offers a radically larger and more diverse source of interaction experience, but strips away exactly the variable that behavior cloning normally needs: the robot command. This survey is about that missing action label. Its central question is not merely whether human video can improve robot learning. By now, the answer is clearly yes. The more useful question is: what must a human video be transformed into before it can train a Vision-Language-Action model?. That formulation is the paper’s main contribution. Instead of sorting work primarily by architecture, downstream task, or learning algorithm, the authors organize more than 30 methods around the intermediate representation that connects passive observation to active control. Their project page identifies four representation bridges, 17 human-video datasets, and three shared bottlenecks along the resulting pipeline. The survey’s central move: classify the bridge, not the backbone. The term VLA now covers a broad architectural family. Some models autoregressively emit discretized actions. Others use diffusion or flow-matching action heads. Some are built on general-purpose vision-language models, while others jointly learn video generation and control. Architecture-centric taxonomies therefore age quickly and can obscure the more fundamental decision: what supervision does the human video provide?. Feng and colleagues identify four answers:. A video can become a latent action, meaning a compact code that explains the change between observations. It can train a world model that predicts future visual states or features. It can be converted into explicit 2D cues, such as point tracks, contact locations, boxes, masks, or optical flow. It can be reconstructed into explicit 3D structure, such as wrist poses, fingertip trajectories, hand-model parameters, or object motion. These categories describe the information interface rather than the entire system. A method may use diffusion, autoregression, inverse dynamics, or conventional imitation learning after constructing the bridge. What matters for the taxonomy is the form in which knowledge crosses from the human domain into the robot-learning pipeline. The survey is also deliberately narrower than “all robot learning from human video.” Earlier approaches used video to pretrain visual encoders such as R3M and MVP, define rewards, discover affordances, or guide modular planning. More recent systems have incorporated human motion into reinforcement learning or mobile-manipulation pipelines. This paper concentrates on approaches in which video-derived signals are integrated into a unified generative or predictive VLA framework. Why a human video is not a robot demonstration. A conventional VLA policy receives visual observations, language, and usually some proprioceptive state. It then predicts a temporally coherent chunk of future commands: perhaps an end-effector displacement, a six-degree-of-freedom pose, a gripper command, arm-joint targets, or high-dimensional finger motion. Human video provides none of these targets directly. It shows that something changed, but not necessarily why. A mug may move because a hand pushed it, because the camera moved, because the table moved, or because another person outside the frame intervened. The image does not reveal the applied force, the exact contact state, the controller frequency, or the actor’s proprioception. Even when the visible human motion is perfectly reconstructed, it may not be feasible for a robot with different reachability, joint limits, compliance, or end-effector geometry. The representation bridge therefore makes a causal bet. Latent-action methods bet that action-relevant change can be separated from irrelevant visual change. World models bet that predicting consequences will induce useful physical representations. Two-dimensional methods bet that image-plane motion and contact are sufficient scaffolds for control. Three-dimensional methods bet that recovering metric geometry will justify the additional estimation cost. Seen this way, these are not four implementations of the same idea. They are four different answers to the question: which information in a human video should survive the transfer to a robot?. Bridge one: latent actions. Latent-action learning tries to manufacture an action variable without labeling one. A typical model receives observations from two moments in a video and compresses the transition into a small code. A decoder must use the initial observation and that code to reconstruct the later observation. If the information bottleneck is sufficiently strong—often through vector quantization—the code cannot preserve every pixel-level detail and is encouraged to represent the salient motion or interaction. LAPA established a particularly influential three-stage recipe. First, it learns a discrete action vocabulary from unlabeled videos using a VQ-VAE-style objective. Second, it pretrains a VLA model to predict those latent actions from an observation and task description. Third, it fine-tunes on a much smaller robot dataset, grounding the latent vocabulary in executable commands. LAPA reported positive transfer even when its video pretraining used human manipulation rather than robot footage. Later systems attack a central failure mode: a visual transition contains far more than the actor’s intent. IGOR uses asymmetric cropping to discourage absolute-position shortcuts. UniVLA predicts in a feature space rather than directly reconstructing noisy pixels and uses language to help separate task-relevant motion from camera motion. ViPRA introduces optical-flow consistency, while Motus makes optical flow itself the transition input. Villa-X conditions its latent space on embodiment, jointly using human and robot observations. CLAP makes the grounding problem more explicit. Instead of discovering a vocabulary solely by reconstructing human video, it first learns executable action tokens from robot trajectories. Human visual transitions are then contrastively aligned to that robot-derived vocabulary. This is an important conceptual reversal: rather than hoping an unsupervised video code later becomes controllable, CLAP constrains the human-video representation around a space already associated with physical actions. Latent actions are attractive because they can scale to RGB-only video and avoid expensive geometric annotation. They also fit naturally into token-based multimodal models. But their compactness is both feature and liability. A small codebook may represent “move left,” “rotate,” or “open,” while failing to preserve contact timing, finger coordination, or multiple valid ways to execute a high-degree-of-freedom task. The survey correctly identifies dexterity as the hard test: a representation that works for a parallel-jaw gripper may be too lossy for a multifingered hand. More fundamentally, reconstruction does not guarantee action identifiability. If background motion, viewpoint change, and manipulation are correlated in the training set, the latent may encode the easiest predictor rather than the most controllable one. Latent-action learning is therefore not annotation-free in the strong sense. It replaces manual action labels with assumptions encoded in bottlenecks, augmentations, architecture, and the small robot dataset used for grounding. Bridge two: predictive world models. World-model approaches retain more of the transition. Rather than compressing change into a minimal action token, they learn to predict future pixels or visual features from the observation history and language. The working hypothesis is that a model capable of anticipating what an interaction will do must internalize useful information about object persistence, motion, temporal ordering, and physical consequences. GR-1 provided early evidence for this route. Its transformer jointly predicts robot actions and future images, while video pretraining teaches the network temporal visual structure before robot fine-tuning. GR-2 scaled the idea substantially, reporting pretraining on 38 million video clips and more than 50 billion tokens before joint video-generation and action-prediction fine-tuning on robot trajectories. Other methods avoid full pixel generation. FLARE adds future-representation prediction to a diffusion-transformer policy, asking action-generation features to align with embeddings of future observations. This produces an implicit world-model objective with comparatively small architectural changes and allows co-training on egocentric human video without action labels. Mimic-video takes a pretrained internet-scale video model and adds a flow-matching action decoder that functions as inverse dynamics: video-space plans supply the dynamics prior, and the decoder translates those representations into low-level robot actions. Gen2Act treats generation as an explicit planning interface. A pretrained video generator imagines a human performing the requested task from the current scene, and a video-conditioned policy executes the generated plan. This cleanly separates broad web-scale knowledge from robot-specific control: the generator proposes what progress should look like, while a smaller amount of robot data teaches execution. World models are especially compelling for long-horizon behavior because they represent outcomes rather than only instantaneous motion. They can potentially encode multiple stages, object-state transitions, and semantic notions of task completion. Yet they are computationally expensive teachers. Generating or predicting large numbers of visual tokens spends capacity on textures, illumination, and other variables that may be irrelevant to control. A beautiful prediction is not necessarily a useful plan, and a plausible video may violate kinematic reachability or contact mechanics. The key unsolved step is knowledge distillation from prediction to control. A world model may know that a drawer should end up open without representing the force profile or grasp stability needed to open it. Predictive accuracy and controllability overlap, but they are not identical objectives. The survey consequently treats transfer from world-model features into action prediction as an open engineering and scientific problem. Bridge three: explicit 2D cues. Explicit 2D methods give up some end-to-end purity in exchange for interpretable supervision. Off-the-shelf trackers, segmenters, hand detectors, and vision-language models can convert videos into point trajectories, contact heatmaps, bounding boxes, optical flow, or object masks. These signals do not specify motor commands, but they provide a spatial-temporal scaffold for learning them. Any-point Trajectory Modeling, or ATM, pretrains a transformer to predict future trajectories for arbitrary image points. Those trajectories then serve as dynamic spatial priors for visuomotor policies trained with relatively little action-labeled data. ATM reported transfer across human videos and robot morphologies over a large collection of simulated and real tasks. Magma turns trajectories into visual-language prompts through its Trace-of-Mark representation, rendering the movement of hands or robot arms directly into the visual input. It pairs this with Set-of-Mark supervision for static spatial grounding, creating a common vocabulary for locating actionable entities and describing how they should move. A0 similarly decomposes manipulation into object-centric affordance reasoning—predicting a contact point and post-contact path—and a lower-level execution system. Masquerade illustrates how taxonomy depends on the final training signal. Its preprocessing estimates three-dimensional human hand poses, removes the visible human arms, and overlays a rendered robot following the recovered trajectory. But the learned auxiliary target is future 2D robot keypoints, so the survey places it in the explicit-2D family. On long-horizon bimanual kitchen tasks, Masquerade reported large gains when fine-tuning with only 50 robot demonstrations per task. Two-dimensional cues are relatively cheap, easy to inspect, and often close to what image-based policies need. They are also naturally compatible with heterogeneous camera setups because no globally calibrated metric frame is required. Their limitations appear near contact. Depth ambiguity makes visually similar trajectories physically different. Hands and tools occlude the most important interaction sites. A 2D path can pass directly through an obstacle while looking valid in projection. These pipelines also inherit errors and biases from their detectors, requiring filtering and confidence estimation at scale. Explicit 2D supervision is therefore an efficient middle ground, not a complete substitute for geometric or physical reasoning. Bridge four: explicit 3D structure. Explicit 3D methods attempt the most direct transfer. They reconstruct wrist poses, fingertip locations, hand meshes, object trajectories, or other metric quantities that can be expressed in camera or world coordinates. Such representations reduce viewpoint dependence and align naturally with common robot action spaces, including six-degree-of-freedom end-effector targets. Parametric hand models such as MANO are especially popular because they provide a compact, canonical representation across people and videos. EgoVLA trains a model to predict human wrist and hand actions from egocentric footage, then uses inverse kinematics and retargeting to execute them with a robot. A small set of robot demonstrations adapts the pretrained model to the target embodiment. H-RDT follows a related hand-centric strategy using large-scale egocentric video with paired 3D poses, a two-stage pretraining and fine-tuning procedure, modular action encoders and decoders, and a two-billion-parameter diffusion transformer. Being-H0 treats the human hand as a foundation manipulator. Its physical-instruction-tuning pipeline combines heterogeneous motion-capture, VR, and RGB video sources, then tokenizes fine-grained hand motion for VLA pretraining and robot adaptation. VITRA attacks both geometry and data preparation: it automatically turns unscripted human activity videos into atomic episodes with language, estimated 3D hand motion, and camera motion, producing a reported corpus of one million episodes and 26 million frames. VidBot shows an alternative that does not require instrumented capture. It combines structure from motion with a depth foundation model to recover temporally consistent, metric-scale 3D affordance trajectories from monocular, in-the-wild human videos. A coarse-to-fine model then predicts interaction regions and diffusion-based trajectories for zero-shot execution across robot systems. Three-dimensional representations offer the clearest interface to dexterous or bimanual manipulation, but they move the bottleneck into reconstruction. Wearable sensors, calibrated multi-view systems, RGB-D cameras, and motion capture yield relatively reliable labels but reduce collection scale and diversity. Monocular reconstruction scales further, yet errors arise from occlusion, scale ambiguity, deformable objects, inaccurate hand-object contact, and drift. Nor does 3D automatically solve embodiment transfer. A precise human trajectory can still be infeasible for a robot. The practically useful variable may be the object-relative contact and intended state change rather than the literal human wrist path. The survey leaves open whether noisy metric reconstruction consistently beats accurate image-space prediction—a question that should be answered task by task rather than assumed from representational richness. The dataset map: breadth versus grounding. The paper organizes human-video datasets along two mostly independent axes. The first is whether they provide explicit metric 3D action labels. The second is whether activities are scripted and staged or recorded in the wild. Together, these axes expose a scale–fidelity frontier that is often hidden when datasets are compared only by hours or frame counts. RGB-centric sources occupy the high-scale end. Something-Something V2 provides roughly 220,000 short, template-driven interaction clips. EPIC-KITCHENS-100 contains about 100 hours of egocentric kitchen activity, while Ego4D contributes roughly 3,600 hours across diverse daily settings. Ego-Exo4D adds synchronized first- and third-person views. HowTo100M supplies 136 million clips from instructional web video, and Egocentric-100K reaches a reported 100,000 hours of largely unlabeled factory footage. These sources offer enormous semantic and environmental diversity, but actions must be inferred from visual transitions. The explicit-3D group is smaller and generally more instrumented. The survey includes HOI4D, Assembly101, EgoDex, OakInk2, ARCTIC, TACO, HoloAssist, and HOT3D. Their supervision may include calibrated multi-view geometry, RGB-D, hand and object poses, parametric hand states, articulation, or contact information. Such datasets make direct grounding more plausible but usually cover a narrower distribution of settings and behaviors. The important practical lesson is that “web scale” and “robot readiness” are different properties. A million loosely aligned clips may be ideal for semantic pretraining or world modeling but poor for learning a contact-sensitive action decoder. A smaller calibrated dataset may ground geometry but fail to teach broad object and task generalization. Scalable systems will probably treat the two as complementary tiers: unstructured video for breadth, instrumented human data for intermediate grounding, and robot demonstrations for final embodiment calibration. Three interfaces where scaling breaks. Episodization is part of the learning problem. Robot datasets arrive as trajectories with explicit starts, ends, tasks, and synchronized actions. Web and egocentric video rarely does. A long recording may include navigation, conversation, preparation, failed attempts, camera adjustments, and several unrelated manipulations. Narration may describe an intention seconds before or after the corresponding physical event. Turning that stream into fixed-length windows is convenient but semantically crude. A window may begin after the grasp, end before the object state changes, or combine approach, manipulation, and recovery under one action token. The resulting label noise is structural, not random: it changes what the model believes an action is. The survey argues for semantic and interaction-driven episodization, with boundaries around manipulation phases and object-state changes rather than arbitrary durations. Standardized construction protocols would also make comparisons more meaningful. In this framing, the episodizer is effectively the first policy-learning component, because it determines the temporal abstraction on which every subsequent representation is trained. Heterogeneity has both an action side and an observation side. Embodiment mismatch is under-constrained. A high-degree-of-freedom human hand can realize many motions that collapse to the same binary gripper command, while some human strategies have no feasible realization for a given robot. The gap widens across different arm geometries, end-effectors, compliance properties, and controller frequencies. Literal trajectory imitation is therefore often the wrong objective. The survey’s preferred direction is embodiment-invariant supervision: interaction intent, object-relative motion, contact structure, and observable effects. Those shared priors should then be translated through morphology-conditioned modules that respect the target platform’s constraints. This argues for shared representations but embodiment-specific decoders, rather than forcing every robot and human into one supposedly universal joint-action space. Viewpoint mismatch creates a parallel problem. Internet video is often exocentric, with weak contact visibility and severe hand-object occlusion. Egocentric footage is closer to robot sensing but includes rapid head motion and frequent disappearance of the interaction site. Robot policies may instead use fixed head cameras, wrist cameras, or both. Cross-view alignment and outcome-centric representations are therefore as important as morphological retargeting. Evaluation must measure transfer efficiency. Current benchmarks such as CALVIN, LIBERO, and SIMPLER provide controlled comparisons, but they do not reproduce the diversity, temporal structure, and open-world shifts found in internet-scale human activity. A gain on one benchmark can show that a technique works, while revealing little about whether human-video pretraining actually reduces the amount of robot data needed for deployment. The survey recommends evaluating performance as a function of robot-data budget, with matched compute and downstream training. It also calls for held-out objects and scenes, explicit viewpoint and embodiment shifts, and comparisons between models initialized with and without human-video pretraining. Real-robot testing remains necessary, but simulation, digital twins, and automatically generated task suites can expand coverage. This is a crucial methodological point. The primary claim of this research area is not simply “our final policy has a higher success rate.” It is that human video changes the scaling relationship between expensive grounded experience and deployment performance. The natural result is therefore a curve, not a single number. What I would build from this survey. For a low-degree-of-freedom gripper where semantic and object generalization dominate, I would begin with latent actions or future-feature prediction. Exact human finger geometry is mostly nuisance in this setting. The important information is likely task phase, object motion, approach direction, and intended state change. For tasks requiring precise contact but not full dexterity, I would add explicit object-centric 2D supervision. Point tracks, contact heatmaps, and post-contact trajectories are comparatively cheap to generate and easy to inspect. They can also provide auxiliary objectives without forcing the policy to execute an externally produced plan exactly. For bimanual or dexterous manipulation, I would invest in explicit 3D hand-object reconstruction—but would avoid treating human joint angles as the final transferable target. Fingertip contact structure, object-relative wrist motion, grasp phase, and object-state changes are more likely to survive retargeting. The robot-specific action head should remain responsible for feasibility. Architecturally, the survey points toward a hybrid rather than a winner-take-all design: a predictive or latent module for high-level temporal structure, explicit affordance and trajectory heads for spatial grounding, and an embodiment-conditioned action decoder for execution. The four bridges are best understood as complementary bandwidths. A compact latent can represent task progression; a world model can anticipate outcomes; a 2D or 3D head can localize interaction; and robot data can ground the final control distribution. The data system should likewise preserve multiple views of each episode. Instead of committing early to one pseudo-label, store language, frame transitions, tracks, masks, depth estimates, hand and object poses, camera motion, reconstruction confidence, and episodization confidence. Different tasks and embodiments can then query different subsets of the same corpus. Finally, human-video sampling should be task-aware. More data is not automatically better when most of a corpus is irrelevant to the target manipulation distribution. Broad pretraining can use the full diversity, but downstream grounding should emphasize human clips whose objects, contact patterns, language, and motion resemble the available robot demonstrations. A critical reading of the survey. The strongest aspect of the paper is its signal-centric taxonomy. “Latent action,” “world model,” “2D cue,” and “3D structure” tell a practitioner more about the human-to-robot interface than labels such as transformer, diffusion policy, or foundation model. The taxonomy remains useful even as backbones and action heads change. The categories are not mutually exclusive, however. Latent-action world models explicitly combine two families. Masquerade estimates 3D motion but ultimately trains with 2D keypoints. Being-H0 discretizes explicit hand trajectories into motion tokens. VITRA is simultaneously an episodization pipeline, a 3D reconstruction system, and a VLA pretraining method. The taxonomy works because it identifies the dominant transferred representation, not because the underlying pipelines are cleanly separated. The survey also cannot offer a normalized empirical ranking. Its table compares video sources, scale, representation type, and evaluated end-effectors, but the methods use different architectures, robot-data budgets, tasks, embodiments, and evaluation protocols. One should not read the four branches as a leaderboard. The evidence currently supports “each route can help,” not a universal conclusion about which route is most data-efficient. A contemporaneous survey, Robot Learning from Human Videos, uses a broader hierarchy of task-, observation-, and action-oriented transfer and covers data foundations and video-generation schemes beyond this paper’s VLA-centered scope. The two taxonomies are complementary: one follows the functional role of transferred knowledge, while Feng and colleagues follow the representation inserted into the VLA pipeline. Several issues deserve more attention in future versions. Human videos hide forces, tactile events, unsuccessful contacts, and internal uncertainty. Public video also raises licensing, consent, privacy, and distribution-bias questions that become more serious at internet scale. Finally, polished instructional videos overrepresent successful behavior, while robust robot deployment requires recovery, hesitation, and failure handling. These are not minor dataset details; they determine what kinds of physical knowledge can actually be learned. Most importantly, the survey does not show that human video will replace robot demonstrations. Its evidence instead supports a more plausible division of labor. Human video can teach broad semantics, task structure, visual dynamics, affordances, and candidate motions. Robot data remains the mechanism that resolves control ambiguity and grounds those priors in a particular machine. What happened after the survey’s May 2026 submission. The survey’s first arXiv version was submitted on May 18, 2026. Work released shortly afterward has already reinforced its framing. On June 17, Do as I Do presented a pipeline for reconstructing hand-object interactions from monocular egocentric and exocentric videos and retargeting them to multifingered robotic hands. It sits squarely in the survey’s explicit-3D branch and pushes toward what its authors call robot-complete manipulation data. On August 5, VLAff introduced an object-centric combination of visual, grasp, and trajectory affordances extracted through structure from motion and hand reconstruction. Its EgoAffordance dataset contains a reported 204,000 episodes, with millions of visual, grasp, and trajectory annotations. This looks like a hybridized version of the survey’s explicit 2D and 3D bridges, organized around embodiment-agnostic interaction structure. And on August 4, SiMDex reframed human-data selection as a recommendation problem. Starting from a pool of roughly 32 million egocentric samples, it retrieved less than five percent for task-aware post-training and reported better dexterous-manipulation performance than equal-sized random sampling. That result targets a question the survey only begins to expose: once human video becomes abundant, which human video should a particular robot learn from?. As of August 7, 2026, these developments suggest that the field is moving beyond indiscriminate scale. The emerging agenda is to episodize selectively, reconstruct only the geometry that matters, represent outcomes rather than appearances, and ground shared priors through robot-specific control. Final takeaway. The most useful idea in this survey is that human video does not become robot intelligence simply by being added to a training mixture. It must first become an action-relevant representation. Latent actions offer scale through compression. World models offer predictive structure. Two-dimensional cues offer efficient and interpretable spatial grounding. Three-dimensional reconstruction offers the closest interface to physical execution. None removes the embodiment gap; each chooses where that gap will be handled. The likely path to scalable VLA learning is therefore not human video instead of robot data. It is human video for breadth, structured human capture for grounding, and robot experience for feasibility. Human demonstrations can teach a model what interactions mean and what successful progress looks like. Robot demonstrations teach it what this particular machine must do next. That is a much more realistic—and potentially much more powerful—scaling story.