Latest / Embodied AI 101 / Scaling Robot Skills from Cheap Human Videos
Transcript
- Scaling Robot Skills from Cheap Human Videos. Robotics researchers have long dreamed of feeding the vast sea of everyday human videos into robot learning systems. Human demonstrations are plentiful, inexpensive, and cover a huge variety of manipulation behaviors – from wiping a countertop to sorting dishes – but turning those videos into robot skills is surprisingly tricky. A recent paper by a team from HKU-MMLab and ByteDance Seed tackles exactly this issue. They point out that the usual approach – extract full 6-DoF hand poses from videos and treat the human like another robot – runs into two big problems. First, modern vision models give only rough estimates of hand orientation, so the 3D rotations are extremely noisy. Second, the embodiment really doesn’t match: a human hand can contort in all sorts of ways, while a typical research robot arm just has a simple parallel gripper. The wrist rotations a human uses to, say, grab a microwave handle can be completely irrelevant or even confusing for a robot gripper. In practice, mapping those rotations naively leads to jittery, twisted, or simply wrong robot motions. Instead, the authors propose a radical but simple fix: throw away the problematic wrist rotations from the human data and use only the wrist’s 3D translation as the action signal. Intuitively, this captures the “intent” of the movement – for example, that the hand is moving toward a cup or away from a drawer – without worrying about the exact finger posture or orientation. In technical terms, they represent actions in a common space that all embodiments share. Concretely, each action timestep (a_t) is broken into three parts:. Bridging action (a_{3D\text{-wrist}}) – the relative 3D translation of the wrist over a short future horizon. Crucially, this translation is measured in the initial head-camera frame, so it is expressed in coordinates that both a human in a video (from a head-mounted camera) and the robot (with its fixed cameras) can understand. Robot end-effector action (a_{6D\text{-eef}}) – the full 6-DoF motion of the gripper (position + orientation) relative to the robot’s current wrist pose. This is what the robot actually needs to execute. Gripper action (a_{\text{gripper}}) – a binary open/close command for each parallel gripper. Their key idea is to supervise only the parts that make sense in each context. In particular, all human demonstrations – whether from lab setups or in-the-wild video – contribute only the bridging action (a_{3D\text{-wrist}}). The human videos either lack reliable orientation or simply can’t close the robot’s gripper, so we ignore those signals in the human data. Conversely, any real robot tele-operation data provides all three components (it has the true 6D gripper motion and the gripper open/close). By training a single policy on this mixed data (discussed below), the shared bridging-action stream ties them together. In effect the robot learns: whenever humans move their wrists along a certain path, the robot should move its grippers in a corresponding way. The authors summarize this action supervision succinctly in a table: “in-the-wild human” data only supervises the 3D-wrist translation (✓) with nothing for orientation or gripper, “in-lab human” data can supervise translation and gripper (✓ ✓), and robot tele-op gives all three (✓ ✓ ✓). This translation-centric action space solves the embodiment mismatch in a natural way. As the authors explain, “rotations are highly embodiment-specific and noise-prone, but the general translation of the wrist is a more robust and universal signal for manipulation”. Moving toward a handle looks the same in camera frames regardless of whether it’s a human hand or a gripper doing the moving. By contrast, slipping the last few degrees of rotation on to grasp the handle is something a human does with a wrist twist (and finger dexterity) but a robot cannot mimic directly. The new policy architecture is designed to handle exactly this hybrid action setup. It’s an end-to-end vision-language-action transformer (inspired by recent “π₀” models) that takes as input both area camera videos and a language instruction, then outputs a sequence of this multi-part action. Crucially, the action tokens are interleaved and masked so that missing components don’t confuse the model. For example, a chunk of tokens might be [walk1x, walk1y, walk1z, grasp1,, ,, ???] – where the blanks signal “no data” for the 6D orientation or gripper in that step. The transformer essentially skips attention on those missing streams and only predicts the parts that are present. In this way, the network smoothly trains on a mix of human and robot data. In their own words, “we adopt a π₀-like vision-language-action model with interleaved action sequences to handle missing action components across human and robot data sources”. Training the policy proceeds in stages, leveraging as much cheap human video as possible. First, the model is pre-trained on large-scale human motion data only, using only the wrist translation stream (a_{3D\text{-wrist}}). The paper mentions using on the order of hundreds of hours of diverse human egocentric recordings (for example the EgoDex dataset and other sourced videos) to cover many skills. This phase ignores any robot control or gripper data – it simply teaches the network about the typical patterns of human hand movements in space. Second, the model is co-trained on both human and robot data. Here they include generalized pick-and-place robot tele-op data (to teach basic grasping motions) and task-specific human demonstrations (e.g. clips of someone opening a microwave or stacking cups). During this stage the bridging stream ties the two domains: the network learns to align human wrist translations to corresponding robot 6D motions. Finally, the model is fine-tuned on a few robot examples for each new task (on the order of 5–10 real episodes per task). These few-shot robot trials let the policy adapt to the precise robot dynamics and camera viewpoints, polishing any remaining errors. By construction, this training pipeline scales cheaply. The bulk of the learning – the first two stages – relies heavily on free or low-cost human data. The robot-specific data is kept minimal. In fact, the authors demonstrate that after all this training, only about ten robot episodes per task are needed to achieve good performance in novel tasks. In one figure they show the robot executing 10-shot rollouts in 15 different tasks (with accompanying human demo clips), and the text notes “across 15 manipulation tasks, the robot reproduces behaviors learned from human demonstrations under the same language instructions in 10-shot rollouts, using only 10 task-specific robot episodes per task”. This level of efficiency – a handful of robot runs instead of hundreds or thousands – is rare in robot learning and is thanks largely to the massive human-video pre-training. The authors test their approach on 15 diverse bi-manual tasks using their custom ByteMini robot (a 22-DoF dual-arm platform from ByteDance’s lab). These tasks include things like opening and closing a microwave door, taking bowls in and out, wiping surfaces, opening and closing drawers, hanging mugs on a rack, stacking cups, inserting a straw into a cup, and unplugging a charger. Each task is paired with a natural-language instruction (e.g. “Take the toast from the toaster and put it on a plate”), so the policy must condition on vision and language. Performance is measured by both task “progress” (continuous metric) and binary success rates. The results show a clear win for the bridging representation. When trained in the same way from scratch, the 3D-wrist-only model outperforms the standard 6-DoF baseline by a large margin. In one comparison, the authors trained two policies with identical data except one used full 6D human wrist actions and one used only the 3D-wrist translations. The 6-DoF model often produced wildly incorrect wrist rotations, leading to low success; by contrast the translation-only policy gave natural, goal-directed motions. Quantitatively, the 6-DoF baseline averaged only 12.5% success (progress ~34.7%) across the tasks, whereas the bridging-action model reached 22.5% success (progress ~44.6%). In other words, simply removing the noisy orientation data more than doubled the success rate in this experiment. (A plotted rollout of one drawer-opening task shows the 6-DoF model’s arm stuck in a twisted pose while the bridged model cleanly aligns to the handle.). Beyond that ablation, adding the large-scale human pre-training yields huge gains. The authors report that a model trained only on generic robot pick-and-place achieves only ~10–15% success on the tasks. Once they co-train with even a small amount of task-specific human data, success jumps (~45%). Finally, after the stage-I pre-training on 600+ hours of human wrist motions, the policy already reaches 57% success before any robot fine-tuning. In practical terms, this means the robot solves well over half of the tasks “out of the box” simply by watching people. For instance, with bridging plus the full human pre-training, even a few real robot trials (e.g. 10 episodes) are enough to hit ~55% success, whereas without the pre-training the same setup was only ~35% successful. The learning curves in the paper show the robot’s loss for orientation and gripper collapsing much faster with the pre-trained model. Putting it all together, the pipeline dramatically outperforms more conventional approaches. Figure 4 in the paper (reproduced in their site) contrasts several conditions: a robot-only baseline, robot+human co-training, and robot+human pre-training. Both the “progress” and “success” bars rise steadily as human data is added. Crucially, the bridging-action policy beats the 6-DoF baseline in every case. The authors summarize the key takeaways: training purely on pick-and-place data yields very poor performance; incorporating human wrist translations dramatically lifts both progress and success; and pre-training on large-scale human video “yields substantial improvements, indicating the scalability and effectiveness of the bridging action”. Of course, this approach has its limits. By design the model ignores human hand rotations entirely, so it can struggle with tasks that really do require precise orientation. For example, inserting a thin straw into a cup or hooking something might need a very particular angle of contact that the robot has to discover on its own. In such cases, some nuanced orientation cues are lost. The authors note that future work could selectively re-introduce orientation information from the easy parts of the videos, or rely on robot data to learn fine angles after transfer. There is also the usual domain gap of visual context – human videos might show different backgrounds or viewpoints than the robot’s lab – but the multi-view cameras and language conditioning partly address that. The broader implication is clear: by finding the right shared representation, we can leverage the vast, cheap resource of human motion videos to scale up robot skill learning. This work demonstrates that translation-wise correspondences between humans and robots are real and exploitable. It’s reminiscent of recent ideas like using intermediate affordance maps or embeddings to link human and robot data, but here the “affordance” is simply the direction and speed of wrist motion. The architecture (a π₀-style vision-language-action model) also highlights the power of unified transformer policies that can assimilate heterogeneous data. In practice, a roboticist could take this idea and apply it to new hardware: capture lots of third-person or egocentric videos of people performing a range of tasks, extract relative wrist movement, and use it as extra training for a robot policy model. In summary, the paper “Translation as a Bridging Action” makes a compelling case that noisy 6-DoF hand poses aren’t the best way to copy human videos. By stripping away the troublesome rotation signals and focusing on shared translations, they unlock orders-of-magnitude more data for imitation learning. The result is a single policy that can learn new bi-manual skills from just a few robot-directed trials combined with thousands of hours of free human video. As the demand grows for robots learning ever-more complex daily tasks, techniques like this one – cleverly reducing the human-robot mismatch – will be critical to truly scale up embodied AI. Key takeaways: A joint HKU/ByteDance effort shows you can learn hard bi-manual tasks by having the robot ignore human wrist rotations entirely. Instead, it learns from the relative translation of the wrist (head/kamera coordinates), a signal shared by both human and robot. A transformer policy then trains on both human and robot data, using masking to handle the missing components. This bridging representation drastically outperforms baselines. For example, in a suite of opening, stacking, and wiping tasks, the translator-only policy more than doubles success compared to the standard 6DoF baseline, and its performance continues to climb as more human video is used. The end result is a robot that can rapidly pick up new skills from cheap human videos – exactly the kind of scaling needed for widespread autonomous robot learning.