Latest / Embodied AI 101 / RoboTok: Turning the Internet Into a Motion Search Engine for Dexterous Robots
Transcript
- RoboTok: Turning the Internet Into a Motion Search Engine for Dexterous Robots. RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning • Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen, Chen Wei, and Kaiyu Hang • Rice University and NVIDIA • arXiv preprint • 2026. A search engine for manipulation, not pixels. Imagine giving a search engine a six-second video of someone unscrewing a bottle cap. The useful result is not another kitchen video, another bottle of the same color, or another clip whose caption contains the word “open.” It is a collection of demonstrations in which the hands execute the same choreography: one hand stabilizes an object while the other establishes a grasp, applies a repeated rotation, and maintains the required relative pose. That is the central idea behind RoboTok. Instead of searching internet video by appearance or semantics, it searches by reconstructed three-dimensional hand motion. Its index is intended to treat two visually unrelated videos as neighbors when their underlying manipulation trajectories are similar—and to separate two visually similar clips when the hands are doing different things. RoboTok’s first arXiv version was submitted on September 2, 2026, only four days before this episode’s date of September 6. It is therefore best read as a promising, unusually timely preprint rather than a finished account of an established system. The project currently reports extensive retrieval and simulation experiments, but its real-world robot results are explicitly listed as forthcoming. The larger bet is important. As robot datasets grow, the bottleneck is shifting from merely collecting experience to deciding which experience should train which behavior. Internet video is effectively unlimited, but most of it is irrelevant to any particular robot task. Indiscriminately mixing that data into a model can dilute useful supervision. RoboTok asks whether the web can instead become a non-parametric memory: show the system one behavior, retrieve motion analogies, and train on only those. What RoboTok actually gives the robot. It is worth being precise about the output. RoboTok is not an end-to-end system that watches YouTube and directly emits robot torques. Given a human demonstration video, it returns human video clips with similar torso-relative 3D hand trajectories. Those clips can then be reconstructed, retargeted, or otherwise converted into supervision for a downstream learner. In the paper’s robotic experiments, the retrieved videos do not become conventional state-action behavior-cloning examples. Their reconstructed hand poses are retargeted into robot states and collected into a demonstration-state bank. A PPO policy receives extra reward for staying near and moving toward that bank while still optimizing the task reward. The retrieved videos therefore guide exploration and pose selection rather than directly supervising robot actions. So the most accurate description is that RoboTok automatically discovers and curates human motion supervision. It removes the need to label internet clips or teleoperate a robot for every retrieved example, but it still depends on a perception stack, a human query demonstration, robot retargeting, and a downstream learning objective. That distinction does not diminish the contribution. In fact, it makes the work more modular. RoboTok can potentially sit in front of reinforcement learning, imitation learning, VLA post-training, motion retargeting, or synthetic-data generation. Its claim is about finding the right experience—not prescribing the only way to consume it. Building the motion index. From Action100M to candidate manipulation clips. RoboTok begins with Action100M, a large automatically generated corpus derived from approximately 1.2 million internet instructional videos. Action100M contains on the order of 100 million temporally localized action segments spanning roughly 14.6 years of video. RoboTok does not train on all of those segments. Its reported retrieval corpus contains 100,000 selected clips, making “internet-scale” more a description of the intended data engine and source pool than of the demonstrated index size. Candidate segments are filtered automatically. Retained clips are four to eight seconds long, have near-static cameras, contain sufficiently visible hands, and include at most one detected left hand and one detected right hand. Overlapping segments are greedily removed, with longer clips preferred. This produces short units that are more likely to contain a coherent manipulation and are substantially easier to reconstruct than arbitrary web footage. These filters are practical, but they define the current operating envelope. Much useful human manipulation video is egocentric or handheld, and therefore includes significant camera motion. RoboTok’s current corpus deliberately avoids much of that complexity. The paper identifies moving-camera third-person and egocentric video as future work. Lifting hands into metric 3D. Each retained clip is sampled at five frames per second. WiLoR estimates articulated 3D hand keypoints, and detections are linked over time to stabilize handedness. Because WiLoR’s weak-perspective reconstruction does not by itself provide reliable metric placement in the camera frame, RoboTok uses MoGe-2 monocular geometry estimates to ground the hands at metric depth. Missing poses are filled using HaWoR. The released retrieval checkpoint operates on up to 42 timesteps of two-hand data. Each timestep contains 21 three-dimensional joints for each hand, producing a 126-dimensional frame representation. Unlike a wrist-path system, this retains finger articulation as well as global hand motion. The representation can therefore distinguish, at least in principle, a translation performed with a pinch from the same translation performed with a power grasp. This stage is fully automatic at RoboTok ingestion time, but it is not annotation-free in the broader foundation-model sense. It relies on several pretrained estimators whose own development required curated data. More importantly, errors can compound: an incorrect hand detection affects tracking, metric depth, torso estimation, dynamic-time-warping neighbors, and ultimately the retrieval labels used to train the embedding. RoboTok’s answer is not to make every reconstruction perfect. It is to use a compact, manipulation-focused signal and then demonstrate transfer on an external dataset with substantially more reliable 3D annotations. Canonicalizing around the actor. Camera-space hand trajectories cannot be compared directly. A hand moving from left to right in one camera may describe the same actor-relative motion as a hand moving toward the camera in another. Translation, camera orientation, shot scale, and demonstrator placement all contaminate the raw trajectory. RoboTok therefore predicts a static torso-centered coordinate frame and transforms each hand trajectory into that frame. The unusual part is that the torso estimator receives only wrist frames from the hand trajectories. The body itself need not be visible. The model is trained using SMPL-H human-body data, following a procedure adapted from the Rice group’s EgoInfinity system. The released checkpoint is described as a four-layer rotation-equivariant transformer with separate rotation and translation heads. This canonicalization is arguably the method’s most important component. Once the hands are expressed relative to the estimated actor, the retrieval encoder does not have to learn viewpoint invariance from pixels. Much of the nuisance variation has already been removed geometrically. But torso-relative is not the same as object-relative. Manipulation is fundamentally an interaction between hands and objects. Two torso-relative trajectories can look similar while acting on objects with different locations, geometry, constraints, or contact states. RoboTok captures the demonstrator’s kinematics well, but not the full physical meaning of those kinematics. Distilling dynamic time warping into vector search. After canonicalization, RoboTok needs a definition of “similar manipulation.” It uses dynamic time warping, or DTW, over the articulated hand trajectories. DTW compares two sequences while allowing local differences in timing. If one person performs a twist quickly and another pauses halfway through, DTW can stretch and compress portions of their timelines so that corresponding stages are compared. At each aligned pair of timesteps, RoboTok measures the Euclidean distance between the two 3D hand poses. The total alignment cost is normalized by the average clip length and negated to obtain a similarity score. This is a sensible teacher metric, but computing DTW between every new query and every indexed trajectory would defeat the purpose of a web-scale search engine. RoboTok therefore distills the local ranking induced by DTW into a 256-dimensional unit-normalized embedding. Database clips are encoded once; a query is encoded with the same network; and retrieval becomes ordinary cosine-similarity nearest-neighbor search. New clips can be added with one encoder forward pass rather than retraining the model. The released model is intentionally small. It projects each 126-dimensional trajectory frame into 256 dimensions, adds sinusoidal temporal position encodings, and uses one learned query token to cross-attend over the sequence. The checkpoint has one four-head cross-attention layer followed by a two-layer projection MLP that outputs the normalized retrieval vector. In this architecture, the query token behaves like a learned pooling operator that decides which temporal and articulated details should summarize the clip. Training batches are not sampled uniformly. In a 100,000-clip corpus, a random batch would rarely contain genuinely similar trajectories. The paper instead constructs anchor-centered groups. For each anchor, it samples positives from the anchor’s top 20 DTW neighbors and a boundary negative from just outside that relevant set. The loss has two roles: a set component pushes relevant clips above boundary and hard negatives, while a rank component attempts to preserve the ordering among relevant examples. This is a useful general pattern beyond RoboTok: use a slow, structured similarity function offline, then train a fast encoder to reproduce only the part of its ranking that matters for retrieval. The neural network is not being asked to discover manipulation similarity from scratch. It is compressing a hand-designed but expensive oracle into an indexable space. Retrieval results: excellent at its chosen notion of similarity. The main in-domain retrieval evaluation uses all 100,000 RoboTok clips as the database. Ninety thousand are used to train the encoder, while 10,000 held-out clips act as queries. For every query, the 20 nearest trajectories under DTW define the pseudo-ground-truth relevant set. RoboTok is compared with random retrieval, FlowRetrieval, HAND, and STRAP. The evaluation includes set-membership metrics such as mean average precision, early-hit metrics such as mean reciprocal rank, rank agreement, local-neighborhood alignment, and the actual DTW cost of the returned clips. On the paper’s v1 numbers, RoboTok reaches an mAP at 20 of 0.353, compared with 0.007 for the strongest baseline, STRAP. Its mean reciprocal rank is 0.858 versus 0.082, and its Kendall rank correlation is 0.487 versus 0.014. RoboTok also achieves recall at 20 of 0.996, meaning that almost every query retrieves at least one member of its true DTW neighborhood. Its retrieved clips have an average DTW cost of 1.333 meters, close to the oracle-neighbor value of 1.145 and far below the random value of 4.776. The out-of-domain test is more informative. AssemblyHands provides accurate 3D hand annotations for egocentric toy assembly and disassembly. RoboTok evaluates 831 two-hand clips, with each clip querying the remaining 830 and the five closest DTW trajectories serving as ground truth. AssemblyHands itself contains millions of annotated frames and was built through a multi-view annotation process designed to provide much better 3D poses than its source dataset. RoboTok again leads every reported retrieval metric. Its mAP at five is 0.261 versus 0.133 for STRAP, its mean reciprocal rank is 0.622 versus 0.397, and its mean DTW cost is 1.095 meters versus 1.304 for STRAP and 0.966 for the oracle. The reduced margin is expected under domain shift, but the ordering remains consistent. These are strong results, but the in-domain benchmark should be interpreted correctly. The ground truth is defined by RoboTok’s own canonicalized hand-DTW metric, and RoboTok is trained explicitly to imitate that metric. The experiment establishes that the encoder is a good fast approximation of its teacher. It does not by itself prove that DTW similarity always corresponds to task relevance. The downstream robot experiment is therefore essential. One minor versioning warning: the project page currently displays nDCG values that differ substantially from the arXiv v1 tables. The figures quoted here follow the paper itself, which should remain the primary reference until the code, site, and manuscript are synchronized. The downstream test: using retrieved videos as reward guidance. RoboTok’s robotic evaluation uses VTDexManip, an Isaac Gym benchmark with six dexterous manipulation tasks: bottle-cap turning, faucet screwing, lever sliding, table reorientation, in-hand reorientation, and bimanual hand-over. The original benchmark was designed to study visual-tactile pretraining and includes human manipulation data across ten tasks and 182 objects. For the RoboTok comparisons, PPO policies are trained from scratch. Policy observations contain proprioception and fingertip-force measurements over a rolling three-step window; the policy does not observe the human video or robot camera images. Retrieved clips affect learning only through a demonstration-guidance reward. The reward is based on a potential function defined as the negative weighted nearest-neighbor distance between the current robot-hand state and a bank of retargeted human demonstration states. One term continually rewards the policy for remaining near the demonstration manifold. A second, potential-difference term rewards transitions that move toward that manifold. There is a useful subtlety here. The standard policy-invariance argument applies to the potential-difference term. RoboTok also includes a separate standing reward proportional to the potential itself, which can alter the optimal policy. In practical terms, that is probably desirable—the entire purpose is to bias behavior toward human-like hand configurations—but only the potential-difference component has the classic shaping guarantee described in the paper. The guidance is also more set-based than trajectory-tracking-based. The robot is compared with a bank of demonstration states using nearest neighbors rather than being forced to match a particular video frame at a particular time. This avoids brittle synchronization and lets PPO find its own execution speed, but it weakens temporal constraints. A robot could potentially visit appropriate poses in an unusual order unless the task dynamics and base reward prevent it. On the original VTDexManip formulation, RoboTok-guided policies outperform the benchmark’s strongest reported pretrained baseline on five of the six tasks. Averaged across tasks, the paper reports gains of 7.45 percentage points on seen objects and 5.83 points on unseen objects. The exceptions are informative: VT-JointPretrain remains stronger on table reorientation, and RoboTok’s unseen bimanual hand-over result falls below several retrieval alternatives. The authors argue that the original benchmark is partly saturated, so they create harder versions of bottle-cap turning, faucet screwing, and lever sliding. These restore unrestricted 3D hand motion and remove much of the handcrafted dense reward shaping. Eight policies are evaluated per method, with 100 attempts per object. Here the retrieval choice matters dramatically. On seen objects, RoboTok reaches 77.3% success for bottle-cap turning, 44.8% for faucet screwing, and 79.3% for lever sliding. The next-best results are 59.5%, 6.8%, and 19.5%, respectively—gaps of 17.8, 38.0, and 59.8 percentage points. On unseen objects, RoboTok remains best at 38.1%, 10.9%, and 58.3%. It also learns faster according to the reported training curves. This is the paper’s most consequential evidence. It suggests that hand-motion retrieval is not merely reproducing a convenient geometric metric. Better-retrieved human poses provide a substantially more useful exploration prior when ordinary PPO and weak task rewards struggle. At the same time, this remains a simulation result with access to simulator state, task rewards, and a robot retargeting pipeline. As of September 6, 2026, the project page still lists real-world demonstrations and results as coming soon. How RoboTok differs from neighboring approaches. RoboTok belongs to a growing line of work that treats robot learning as retrieval plus specialization rather than indiscriminate pretraining. FlowRetrieval retrieves robot experience according to optical-flow similarity. Its key insight is that shared low-level motion can be useful even when task semantics differ. Optical flow, however, remains an image-space signal affected by viewpoint, camera motion, appearance, and occlusion. RoboTok pushes the same motion-centric intuition into canonicalized, articulated 3D human-hand space. STRAP retrieves robot sub-trajectories by temporally aligning visual foundation-model features with subsequence DTW. Its sub-trajectory granularity is valuable because different tasks often share reusable behavioral fragments. In RoboTok’s evaluations, STRAP is generally the strongest baseline, but it still searches representations built primarily for visual robot data rather than a camera-invariant model of dexterous human motion. HAND is closer in interface. A user supplies a human hand demonstration, which is used to retrieve compatible segments from robot play data before quickly fine-tuning a policy. HAND combines visual filtering with hand-path and robot-end-effector matching, and it reports real-robot adaptation in minutes. RoboTok expands the motion representation from a coarse path to full 3D hand pose and replaces the fixed robot-play database with an extensible human-video index. The trade-off is that RoboTok still needs a separate conversion stage before those human motions become robot supervision. The closest contemporary comparison may be SiMDex, released in August 2026. SiMDex uses robot demonstrations to mine a pool of roughly 32 million egocentric human samples through a recall, ranking, and reranking cascade, then uses the selected data for cross-embodiment VLA post-training. It reports that about 1.49 million selected samples improve success from 47.7% to 61.1% relative to an equal quantity of randomly selected human data. RoboTok cites SiMDex but does not compare against it experimentally. The two approaches differ in query direction, representation, and downstream learner, yet both argue that selective curation is more valuable than indiscriminate human-video scaling. RoboTok is also naturally complementary to EgoInfinity from the same Rice lab. EgoInfinity focuses on recovering metric 4D hand-object interactions and compiling them into executable robot motions. RoboTok focuses on discovering which videos should enter such an expensive reconstruction and retargeting pipeline. A practical system could use RoboTok as the search layer and EgoInfinity-like machinery as the video-to-action compiler. What the paper gets right. The strongest design choice is the explicit separation between invariance and retrieval. Rather than expecting a large video encoder to discover that camera viewpoint is irrelevant, RoboTok removes viewpoint geometrically. Rather than expecting contrastive learning to discover which hand motions matter, it provides a structured DTW teacher. The final network can consequently remain small and indexable. The second strength is that the authors test retrieval through a downstream control problem. Many retrieval papers stop at nearest-neighbor visualizations or retrieval metrics. RoboTok shows that, under difficult reward conditions, different returned video sets lead to radically different PPO outcomes. Third, the approach is genuinely label-light at ingestion. No action taxonomy is required, and semantically unrelated activities can become neighbors when they contain the same motion primitive. That is useful for long-tail manipulation, where the exact target object may be rare but the necessary grasp, twist, slide, transfer, or reorientation pattern recurs in other contexts. What remains unproven. The first open question is scale. The source corpus is enormous, but the experiment indexes 100,000 clips. That is large enough to demonstrate meaningful retrieval, yet three orders of magnitude short of Action100M’s nominal segment count. The v1 paper does not report end-to-end ingestion throughput or the cost of running hand reconstruction, metric grounding, infilling, torso estimation, and encoding over tens of millions of clips. The second question is whether the representation captures physical relevance rather than just kinematic resemblance. RoboTok lacks object pose, geometry, articulation, contact, force, affordances, and task direction. The same hand trajectory can have very different consequences on a rigid handle, compliant cloth, free object, or constrained mechanism. Conversely, morphologically different trajectories may achieve the same object-state transition. A natural next step is a hybrid metric containing torso-relative hand pose, object-relative motion, estimated contact, and object-state change. Hand kinematics would remain the fast recall signal, while interaction features rerank the smaller candidate set. Third, the retrieval target is partly self-referential. On the main corpus, pseudo-ground truth comes from the same reconstructed poses and DTW design that supervise RoboTok. AssemblyHands provides a valuable external check, but it contains only 831 evaluation clips and is concentrated on bimanual toy assembly. The v1 paper also lacks component ablations isolating metric depth, torso canonicalization, pose infilling, finger articulation, or the set-and-rank objective. Fourth, the released README notes that the held-out RoboTok split is clip-level rather than source-video-level. Different clips from one original video can therefore appear across training and evaluation. They need not be duplicates, and the pipeline removes overlapping segments, but shared actor, scene, camera, and recording conditions make this a less stringent test than a video-disjoint split. Fifth, the simulation protocol demonstrates data usefulness under one specific consumer: state-based PPO with task rewards and demonstration shaping. It does not yet establish that RoboTok retrievals improve behavioral cloning, diffusion policies, VLA post-training, world models, or closed-loop real-robot control. Nor does it test whether the learned hand similarity remains useful after retargeting to substantially nonhuman robot morphologies. Finally, reproducibility has practical constraints. The authors release an 11 MB retrieval encoder, a 9.9 MB torso estimator, and a 6.5 GB artifact of torso-relative evaluation keypoints. The source code is MIT-licensed, but parts of the dataset-construction pipeline expect a private clip-database module that is not included. The released checkpoints and Action100M-derived artifacts are governed by a noncommercial research license, while MANO and SMPL-H assets require separate registration and licensing. How I would use or extend RoboTok. For researchers interested in adapting the idea, I would treat RoboTok as a retrieval architecture rather than a fixed recipe. Choose the canonical frame for the downstream embodiment. Torso-relative coordinates make sense for anthropomorphic hands and humanoids. For a stationary arm, an object-, fixture-, or contact-relative frame may be more transferable. For bimanual tasks, preserve both global wrist motion and the changing relative transform between the hands. Design the teacher metric around controllable consequences. Plain positional DTW is an excellent starting point, but fingertips, wrists, and inter-hand geometry need not receive equal weight. Add velocities for motion direction, cyclic alignment for repeated twisting, uncertainty weights for unreliable reconstructions, and contact or object-motion terms where available. Use a retrieval cascade. Pure semantic search confuses visual context with behavior, while pure kinematic search can ignore the manipulated object. A practical engine could first filter by broad object and affordance compatibility, retrieve by hand motion, and rerank using interaction geometry. SiMDex’s staged search and RoboTok’s compact motion embedding point toward this hybrid design. Diversify the returned set. Nearest-neighbor search can return many segments from one source video or many near-identical executions. Clustering, source-video deduplication, quality thresholds, and diversity-aware selection could provide the downstream learner with multiple valid strategies rather than repeated copies of one motion. Match the retrieval representation to the learning interface. For PPO, RoboTok’s nearest-state reward is simple and effective. For imitation learning, preserve temporal alignment and infer executable actions. For VLA training, pair selected clips with automatically generated action descriptions or hand-object tokens. For a world model, use retrieved clips as counterfactual examples of how similar interactions unfold under different objects and scenes. The most compelling long-term system would close the loop. A robot encounters a task, records a failed or partial attempt, converts that attempt into a motion query, retrieves human analogies, retargets them, trains or plans, and then uses its new experience to refine the query. RoboTok provides the indexing primitive for that loop, even though the current paper evaluates only its first generation. The bottom line. RoboTok does not yet turn arbitrary internet video directly into deployable robot behavior. It demonstrates something narrower and potentially more foundational: articulated, actor-relative 3D hand motion is a powerful key for searching human experience. Its retrieval results show that a lightweight embedding can preserve an expensive spatiotemporal similarity metric. Its harder simulation experiments suggest that the selected clips are materially better exploration priors than those returned by appearance- or image-motion-based systems. And its limitations clearly expose the next research problems: object and contact awareness, moving cameras, larger indexes, stricter evaluation splits, and real-world policy learning. The broader lesson is that scaling embodied AI may require better search as much as bigger models. The internet already contains countless examples of dexterous behavior. RoboTok asks the right systems question: before teaching a robot from all of them, can we first find the few that move the right way?.