Latest / Embodied AI 101 / Translation as a Bridging Action

Transcript

Translation as a Bridging Action · Embodied AI 101

• Translation as a Bridging Action. Imagine teaching a robot to manipulate objects by watching people do the task. Human demonstrations are easy to collect – people pick up cups, open drawers, or wipe surfaces all the time – and video of that is cheap and abundant. In theory, if a robot could learn from this human data, it could acquire a wide range of skills without costly robot programming or expert collecting hundreds of robot trials. In practice, however, making the jump from human motion to robot control has proved surprisingly hard. Humans have dexterous hands and mysterious finger contacts; robots often have simple two-finger grippers. The ways human wrists move and rotate often do not align with how a parallel–gripper robot needs to approach an object. In short, treating human videos as if they were robot “teleop” trajectories – for example by estimating a full 6-degree-of-freedom (6DoF) hand pose from video and feeding that to a robot – runs into problems. The hand-pose estimates are noisy, and crucially the human’s grasping configuration (with many fingers) does not map well to a robot gripper. A new idea from researchers at ByteDance Seed (Chen et al., 2026) is to sidestep these issues by throwing away the hand’s orientation entirely when learning from people. Instead, they use only the translational component of the wrist’s motion – essentially, “x, y, z” shifts of the hand position – as the action they learn. This relative wrist translation is measured in a consistent coordinate frame (the person’s head-mounted camera frame, for example), and it happens to be an action representation that both human videos and robot controls can share. In essence, they say: let’s focus on the common denominator of how humans and robots move during manipulation, and ignore the part that doesn’t transfer well (the rotation/orientation of the wrist). This simple change turns out to make learning from human videos much more effective. In their system, called “Translation as a Bridging Action,” training on large amounts of human video (with only translation actions) helps a bimanual robot learn new tasks faster and with higher success rates than trying to learn full 6DoF wrist actions. The key insight is that by bridging the human-to-robot “embodiment gap” through a shared representation, one can cheaply leverage human data. The authors combine this idea with a modern vision–language–action model (a transformer that ingests images, text, and action tokens) that can handle sequences with missing components (since human demonstrations lack orientation and sometimes gripper commands). On a set of 15 challenging bi-manual manipulation tasks, their “bridging action” approach significantly outperforms a baseline that uses the full 6DoF estimated human hand poses. They also show that adding ever more human video (pretraining on a large dataset) continues to improve the robot’s performance, demonstrating that the method truly “scales” with the data. In this article we’ll unpack their approach in detail. We’ll explain why 6DoF human actions are problematic, how translating just the wrist can serve as a “bridge,” and how the architecture and training handle the fact that different data sources supply different action components. We’ll review the experiments on the ByteMini bimanual robot and highlight the surprising quantitative gains. Finally, we’ll place this work in context of other efforts to transfer from human demonstration (like HumanEgo) and ponder the implications: can cheap videos really train robots?. The embodiment gap: why full 6DoF from people doesn’t work. First, let’s be clear about the problem. We have videos of people performing manipulation tasks (like opening a microwave, hanging a mug on a hook, stacking cups, etc.). We also have a two-armed robot (ByteMini) with simple parallel grippers. It would be wonderful if we could simply teach the robot by “watching” the human and doing the same, but the two embodiments (human and robot) are different in crucial ways:. Noisy hand poses: Modern motion-capture or pose-estimation methods can track a person’s hand position and orientation from video, but these are far from perfect. The wrist orientation especially can be jittery, inaccurate, or altogether missing if the camera view is bad. Relying on these noisy 6DoF signals to supervise a robot is liable to cause errors. Different contact geometry: A human hand has five fingers, each of which can make contact in complex ways with objects. A robot gripper typically just has two flat fingers. When a human rotates their wrist or curls fingers, it results in different contacts than a robot closing its fixed-angle gripper. In essence, the meaning of “rotate the wrist 45 degrees” is not the same for a human as for the robot. Indeed, as the paper notes, human wrist rotation can become semantically misaligned with gripper actions. Action relevance: In many tasks, how the human spins or tilts their wrist doesn’t change the essence of the action. Suppose you watch someone hang a mug by its handle. They might twist and turn their hand in a fluid way, but the key thing is that their hand translates to the right location so the hook goes through the handle loop. The robot just needs to move its gripper “there”; any extra wrist rotation beyond the robot’s default approach is not necessary and in fact might break the grasp if done poorly. Because of these issues, previous approaches that treated human demonstrations as if they were just another robotic pose stream have struggled. They tried to learn from human data by estimating a six-degree-of-freedom (6DoF) wrist trajectory (three for position, three for orientation) and asking the robot to imitate that. But the errors and the embodiment mismatch mean the robot learns weird or incorrect behaviors. Chen et al. say: drop the rotation. Instead, define the action as the relative wrist translation in the head camera frame (or more generally, the initial camera frame of the video). Concretely, this means if the human’s hand was at position (x0,y0,z0) in the camera view at time t=0, and then at (x1,y1,z1) some frames later, the action is effectively the vector (x1-x0, y1-y0, z1-z0) — how much the hand moved forward, sideways, and up/down, relative to where the human started. This 3D translation has a natural interpretation for the robot: it says “move your gripper this much in your camera’s coordinate frame.”. Because both humans (via head-mounted cameras or egocentric video) and robots (via the robot’s camera view) can express how the wrist moves in camera-centric coordinates, this shared action space is a bridge between the two embodiments. And by ignoring orientation, we remove a source of noise and misinterpretation. As the authors summarize:. “We argue that learning rotation-inclusive action signals from human data is therefore sub-optimal, and instead propose a bridging action representation: the relative wrist translation within the initial head-camera frame, an action space shared by humans and robots.”. Crucially, this means the model only needs to learn from the 3D translations of the human hand, which are more robustly estimated and are more directly meaningful to the robot. If the human demonstration included any gripper opening/closing, that can be mapped too when available (as we’ll see in the data sources). But the core is: no orientation. You might wonder if we’re sacrificing anything by throwing away orientation. In principle, if a task truly required a special hand orientation (like inserting a magnetic key with a specific angle), then yes, ignoring orientation might lose that nuance. But on a practical level, most simple household tasks are actually pretty robust to small orientation errors. A robot gripper always approaches an object with some default angle, and finer orientation adjustments can sometimes be learned or added later. The authors note that for tasks like those in their suite (microwave door, mug hanging, stacking), leaving orientation out still led to natural, successful grasps on the relevant handles and objects. The vision-language-action model. With a shared action representation defined, the next piece is the model: how do we learn policies that output these actions given observations and possibly instructions? The authors use what they call a vision-language-action (VLA) model. Conceptually, this is a transformer-based policy (similar in spirit to models like π₀ or OpenVLA) that takes in visual inputs, language, and optionally prior actions, and produces future actions. The architecture resembles a large seq2seq model. It ingests tokens of various kinds: visual embeddings from images, tokens from a text instruction (e.g. “open the microwave door”), and action tokens representing past or future motions. The key innovation is interleaving action tokens with the vision and language tokens, and using attention masking to handle cases where some action modalities are absent. In practice:. The policy sees an image (often from an overhead camera or the robot’s camera of the scene) showing the objects and the current state. It also takes a language instruction tokenized (like “nearby microwave, open its door”). Then it outputs a sequence of future actions. Here, an “action” is composed of up to three components: 1) the 3D-wrist translation vector (Δx, Δy, Δz), 2) optionally a 6D end-effector pose update (if using 6DoF, but humans won’t have that), and 3) a binary gripper command (open/close). Crucially, when training on human video data from the wild (like Egocentric videos of people), only the 3D wrist translation is available reliably. The pose orientation and gripper state are either not present or too noisy. The model handles this by masking out those outputs: the loss function only kicks in for the translation tokens. For in-lab human data (where the state is better controlled), the human might even have been holding a tool or controller so that a gripper open/close can be recorded, so the gripper channel is supervised but still no orientation. For robot teleoperation data (the human actually controlling the ByteMini arms), all three channels are available, so the model learns full actions there. In other words, the model has a flexible output head, but each training source turns on only the channels it can. The authors illustrate it in a table (excerpted here) showing which action components each data source provides:. | Data source | 3D-wrist | 6D end-effector | Gripper | |----------------------------------|:-------:|:---------------:|:-------:| | In-the-wild human (EgoDex + other) | ✓ | – | – | | In-lab human action data | ✓ | – | ✓ | | Robot tele-operation | ✓ | ✓ | ✓ |. Thus during training, an egocentric video snippet contributes gradients only on the translation outputs, whereas a teleop demo contributes to all outputs. The model is designed so that it can attend across these modalities – it’s a kind of transformer mixture-of-experts where some “heads” deal with vision, some with language, some with translation, etc., and they communicate via the attention mechanism (as in the π₀ model architecture). In fact, the authors explicitly describe their model as “a π₀-like vision-language-action model with interleaved action sequences”, meaning each modality is put into one sequence and processed with causal attention (each output element can attend to previous visual tokens, language tokens, and past action tokens). One detail is that they chunk the future actions into segments of length k=30. This means instead of predicting one step at a time, the model outputs an entire short plan (30 timesteps of actions) at once. This is likely done for efficiency and to capture short-term coherence. It’s similar to other VLA policies that output sequences of actions in an autoregressive fashion (instead of continuous RL). The key point is: they ask the model to output a whole upcoming trajectory of translations and gripper commands given the current observation and instruction. Importantly, the entire model is co-trained on both human and robot data, using the shared translation tokens to link them. The authors demonstrate that by sharing this representation space, the robot policy learns more from the human data. At training time, a batch might contain some egocentric video where we only compute loss on 3D translation, plus some robot episode where we compute loss on all channels. Over time the shared parameters form a policy that can interpret the visual scenes and instructions, and output the correct wrist motions for both kinds of data. Data collection and processing. What exactly are the datasets used to train this policy? The authors mention three sources:. EgoDex (and outsourced) egocentric videos: EgoDex is a large dataset of first-person videos of people performing manipulation tasks (from Egocentric vision research). It appears the authors use this and possibly additional crowdsourced video (“outsourced”) to get a wide variety of human demonstrations. From these videos, they extract the human wrist’s position over time (presumably via some vision-based motion capture or sensor) and use that as the 3D translation. Since these are “in-the-wild” head-camera videos, only the translation is reliable – we do not use any orientation or additional sensors on these. The assumption is that Egocentric video plus hand-tracking yields 3D wrist motion in camera coordinates (up to scale, perhaps with known camera intrinsics). In-lab human action data: This is their own collection of human demonstrations in a controlled lab setting. Likely, they either instrument people with motion trackers or have known camera setups, so they can capture the human wrist translation and whether the human is mimicking a gripper open or closed (maybe by just opening/closing their fingers as a stand-in, or wearing a glove). The text mentions that for in-lab data they have the gripper channel (✓ in table). They still do not use 6D poses for the human in the lab – perhaps the reasons remain the same (contact patterns, etc.) – but they do trust a “grasping/open” signal, maybe created by a human pressing a button or using their hand as a virtual gripper. Robot tele-operation data: Here a human operator controls the ByteMini robot, presumably via VR controllers or joystick, to perform the tasks. This yields true robot state sequences: 3D translation, orientation, and gripper for every step. This serves two purposes: it provides what we might call “ground truth” correct action sequences for the tasks, and it also teaches the model how the robot actually needs to move in the real world. The teleop data might be limited (they mention using “only 10 episodes per task” later for imitation fine-tuning), but it ensures the model sees the correct alignment of translation+orientation in actual robotic coordinates. In summary, they mix cheap human video data (abundant but partial signals) with expensive robot data (scarcer but complete) in training a single model. The innovation is to align them via the 3D-wrist translation. One can guess how they get these wrist translations. For Egocentric video, perhaps using a pretrained hand tracker or annotating keypoints in the video to reconstruct 3D (maybe leveraging a known object size or stereo). The details aren’t spelled out fully, but presumably they have some pipeline to produce a sequence of 3D hand positions from the head camera view. Importantly, this is all done offline: the learning is imitation learning / offline training, not real-time control yet. Training and scaling with human data. Once the data is prepared, the training proceeds in two phases:. Pre-training on human data (with bridging actions): They construct a large GPU batch containing many clips from EgoDex and other videos (including their in-lab human clips), plus some smaller amount of robot teleop. They optimize the transformer policy so that given an image+instruction, it outputs translations (and any other channels if available) matching those in the demonstration. Because the human data is large, this phase gives the model a broad base of “what does translation look like when I say ‘open microwave’ or ‘stack cups’.” The key is that the model architecture automatically ignores missing orientation/gripper inputs from humans, as we discussed. Fine-tuning or co-training with robot data: After or during pre-training, they refine the model using actual robot trials (tele-operation) specifically for the target tasks. They do 10 episodes per task – very few, given 15 tasks. These episodes supply the full action supervision for those tasks. They continue training the same model (or maybe fine-tune a separate head) so that it learns to map its learned knowledge into actual robot motions. The clever part is: by leveraging the human data in the first place, they can make do with far less real robot data. In their experiments, they indeed report that only 10 real-robot demonstrations per task were used during training, thanks to the power of pre-training on video. This is very much in the spirit of recent “foundation” models for robotics: gather internet-scale or large-scale data offline (here it happens to be available egocentric manipulation videos, instead of say web images or text), learn a general policy, then specialize with a bit of on-task data. The authors highlight that adding more human pretraining always helps. In their experiments they show that adding more hours of EgoDex (or “outsourced” video) in the training mix improves performance on the final tasks. This indicates scalability: the method can use cheap videos as fuel. Quote:. “Pre-training on large-scale human actions with a^{3D-wrist} yields substantial improvements, indicating the scalability and effectiveness of the bridging action.”. In other words, the more human demonstrations they feed (in that shared 3D translation space), the better the robot policy does, even though the human videos are not from the same robot platform or exact scenes. That’s a compelling break from the usual “hundreds of teleop episodes per task needed” standard. Evaluation tasks and setup. What tasks do they test on? The authors assembled a suite of 15 bi-manual manipulation tasks, all performed by the ByteMini robot in a kitchen-like environment (see their video demos). Examples of tasks include:. Opening/closing a microwave door. Taking a bowl out of the microwave, placing a bowl into the microwave. Wiping the top of the microwave (left-to-right and right-to-left). Opening/closing a drawer. Hanging a mug on a mug holder (for left and right mug, presumably same task but mirrored). Stacking one cup onto another (two versions, left and right). Inserting a straw into a cup. Taking toast from a toaster and placing it on a plate. These tasks involve basic everyday manipulations: moving objects in space, dealing with articulating appliances (microwave door, drawer), and multi-step behaviors (like wipe then transition). The robot is bilingual (two arms) so can do things that require two hands, but each task appears to involve a single manipulator at a time (though symmetry is there). The tasks come with language instructions (“Open the microwave door”, etc.), and the policy is conditioned on these instructions. The input to the robot policy presumably is the overhead or on-board camera view of the scene and the natural language instruction (pre-processed to tokens). The output is the sequence of wrist translations (+ gripper commands). The ByteMini then executes those as motor commands. For measuring performance, the authors use two metrics Progress and Success. “Progress” is a continuous measure of how far the task got (e.g., fraction of door opened), while “Success” is binary (task completed within tolerance). They report the averages across tasks. Performance is evaluated after training with only 10 real robot demonstrations per task, plus the human data. (Crucially, these 10 demos include no online learning or RL; they just continue the offline imitation training on them.) They compare several training regimes:. Only pick-and-place (PnP) data: A baseline trained on a generic dataset of pick-and-place trajectories (no fine tasks). This does poorly. Bridging action co-training: Our main method (with mixing human video + robot demos). 6DoF baseline: A similar model but where we use full 6DoF wrist actions from humans (as is standard practice), instead of only 3D translation. Possibly other ablations like "no human data", but key ones are above. The main result is illuminating:. “Human action co-training and large-scale human-only pre-training improve real-robot progress and success rates across the 15 evaluation tasks.”. In particular, looking at their detailed numbers: when they compare the two approaches of using human demonstrations, the 3D translation (bridging) version achieved about 44.6% progress and 22.5% success, whereas the 6DoF baseline got only 34.7% progress and 12.5% success. That is roughly a two-fold increase in success rate, which is quite significant. A nice qualitative illustration (in their Fig.6) shows that the 6DoF baseline robot often swings its wrists into odd orientations and misses the target (it’s “off-target” or “distorted”), while the translation-based policy drives the gripper naturally toward the goal (e.g., aligning with a handle). This confirms the intuition: by ignoring the confusing orientation data from the human, the robot just focuses on getting its gripper to the right spot, which is what matters. They also note an important point: using only generic pick-and-place data (for example, random motions on some blocks) did not produce a policy that could handle these tasks. The baseline that trained on pick-and-place only (and no specialized data) gave very poor progress and success. This means that even with a great model, you still need relevant examples. But the bridge is that instead of teleoperating the robot for those 15 tasks hundreds of times, we can instead leverage humans doing similar activities (even if not identical) to cover the skill distribution. In summary:. Training on only pick-and-place demos gave much lower performance (low progress and success). Incorporating human videos with the 3D-translation action lifted the robot’s ability significantly. Pretraining on large amounts of human data improved performance in a largely monotonic way: more data, better results. I would highlight the numbers as evidence of the improvement:. “Using only the relative 3D-translation from humans (and ignoring 6DoF) nearly doubled the success rate. The 3D-wrist model achieved ~44.6% progress and 22.5% success on the tasks, versus ~34.7% and 12.5% for the 6DoF human baseline.”. We’ll definitely cite that to back up the claims. What happens qualitatively?. Beyond the numbers, it’s informative to think about why this works better from some robot’s perspective. One can imagine the difference in learning signals:. With the 6DoF baseline, the policy sees that “the human rotated the hand by X degrees, I should do that too.” But maybe the human was reorienting fingers to manage an object in-hand, whereas the robot’s gripper might simply stay horizontal. The policy could be chasing irrelevant orientation cues. With 3D translation only, the policy learns “the goal is over there; move my end-effector toward it.” For a forward motion like opening a door, the human’s wrist might mostly move forward and outward; even if the human’s wrist turning is not captured, it turns out it wasn’t needed to get the door handle. The robot then learns just that “translation” is enough to open. In a hanging-a-mug task, the human might twist their hand naturally to align with a mug handle, but again, what matters is they moved to the handle location. The robot simply learns to move its gripper to that spatial location and then do a default “close gripper” action (which it did get to learn from the robot demos). The authors include a side-by-side video (in the supplementary demo) showing a human doing a task and the resulting robot action. We can mention that as proof of concept: “the robot actually reproduces behaviors it ‘learned’ from human demo under the same instruction, using only 10-shot robot rollouts per task”. Essentially, in rollouts of their learned policy, the robot often replicates the human’s action sequence (within reason) on the real robot. An interesting subtlety is coordinate frames. They always do “relative to the initial head-camera frame”. This means the orientation of the camera is used as a reference. Concretely, if the human’s head-camera started pointed at the scene, all motions are expressed in that initial camera coordinate system. For the robot, presumably its camera is aligned the same way in the environment. This solves the problem of translation aligning properly; basically, it ensures if the human moves hand straight “forward” in their view, the robot also moves straight forward in the scene. One could imagine alternatives: some works align human and robot in world coordinates, or require a known transform, but they keep it simple by using head-camera coordinates. It means the robot has to interpret the vector (dx,dy,dz) from its own camera as commanded. This “camera-centric translation” is easy to compute on the robot side. Comparison to other human-to-robot approaches. This idea is part of a growing trend: using egocentric human video to train robots. The ByteDance approach focuses on the action representation side of the problem. Others have tried different conversions:. HumanEgo (Yiannis Aloimonos et al., 2026): This is a contemporaneous paper (published arXiv May 2026) where the authors also use minutes of human egocentric video to train robots zero-shot on tasks. They take a different strategy: instead of copying raw poses, they “lift” each human demo into an intermediate representation of object interactions (things like “cup moving from A to B”) and then train a flow-matching policy with dense auxiliary objectives. They also achieve very high success on 4 tasks with only 30 minutes of video, beating matched-time teleop. HumanEgo does not explicitly mention “bridging translational actions” – they seem to abstract to entities. But conceptually they tackle the same embodiment gap. It would be interesting to compare results: HumanEgo reports 92.5% success across 4 tasks, whereas here bridging got 22.5% success across 15 tasks (with only 10 real demos each). The tasks and metrics differ, but it shows the same appetite for using human videos. EgoDex (Hoque et al., ICLR 2026): Actually a dataset (EgoDex) by Apple’s ML team claims to unlock dexterous manipulation from egocentric video. The ByteDance paper even cites using EgoDex. EgoDex is probably constructed with more annotations or constraints to afford learning dexterous tasks. The actual performance of EgoDex-like training vs bridging would be an interesting discussion point. The main difference is bridging action here explicitly discards orientation, whereas EgoDex’s approach might or might not (they don’t detail since EgoDex is a dataset, not a method paper in detail, but likely they provide 3D hand poses). Inverse maps or simulation: Some older works used intermediary models or simulated humans in robot space. This new method’s advantage is simplicity: it doesn’t try to learn a mapping from human pose to robot pose; it simply looks at the subset (translation) that is common. Other transduction methods: There’s literature on extracting keypoints or task descriptors from human video (e.g. detecting that the human is picking up “mug by handle” then replicating, etc.). This bridging approach is a bit simpler – it’s flat imitation of motion, not high-level symbolic understanding. But by using language instructions as well, it’s semi-structured. It’s fair to say this bridging action concept is novel (I’m not aware of prior art that explicitly argues to discard orientation when learning from humans). Most prior robotics learning papers assumed capturing full 6DoF. This paper convincingly argues that “no, let’s just use translation” – and provides evidence it outperforms the standard practice. What might this mean for the future?. The result has some clear implications. It suggests that the limiting factor in scaling robot skill learning might not be the lack of human video data, but how we choose to use it. By finding the right representation – in this case, a simpler action space – we can almost double sample efficiency (and ultimately performance) from the same videos. The scalability point is key: the bridging action claims to benefit from more human data. If we imagine having thousands of hours of egocentric kitchen videos (people cooking, cleaning, etc.), this approach could leverage them directly for learning. It could train a single versatile policy that generalizes across tasks, as long as it has the common denominator of “how humans move their hands around objects in a camera view.” If the vision-language policy is large (like Pi₀ or OpenVLA scale), it might start to understand subtleties of instructions and contexts. However, some caveats:. Orientation-limited tasks: Not all tasks are purely translational. Grasping slender objects might need wrist yaw, or inserting a plug at an angle might need roll. This method would ignore those. Perhaps one could extend it by having a two-stage approach: learn translation first, then maybe do a local refinement for orientation when needed. Or maybe just assume gripper defaults are sufficient for many tasks. Differences in camera viewpoint: The bridging relies on a consistent camera coordinate. If human demo is from a head camera, and robot camera is differently placed, they have to align those frames. The paper doesn’t deeply discuss calibration, but presumably they assume they can set up a coordinate system where the initial head-camera orientation roughly matches the robot’s 0-degree orientation. In practice, if the robot’s camera is fixed in the environment (like an overhead view) while the human’s is moving (like mounted on a person’s head), there must be some mapping. Possibly they simplify this by controlling camera positioning or focusing on relative moves (so slight misalignment doesn’t break the vector commands). Partial supervision: The mask approach is neat but means the model gets less information from humans (no orientation learning). But that appears fine, even beneficial, in these tasks. Generalization vs. specialization: This approach put heavy weight on the human behavior distribution. It's possible that if human videos contain weird or irrelevant motions, the model could learn spurious correlations. Thankfully, the robot fine-tuning presumably filters that. The design of the architecture (a shared model that sees both data types) means that the robot data can “correct” the model away from any overly strange human tendencies. And the results show it did that; the bridging model was better than just human-only. Overall, “translation as bridging action” is an elegant idea. It reminds me of other “dimensionality-dropping” tricks that often yield big improvements – like how AlphaFold decided certain angles didn’t matter early on, or how in RL sometimes you ignore some channels. Here, dropping rotational information turned out to improve learning. Conclusion. This work by Chen et al. offers a compelling new take on human-to-robot imitation. By recognizing that full 6DoF isn’t needed – or even counterproductive – when using human videos, they find a simpler shared action space that unlocks more data. Their experiments on real bimanual robots (ByteMini) across multiple tasks show clear wins for this bridging action: nearly doubling success rates versus the old way of doing 6D imitation. Perhaps most exciting is the scalability: adding more human video consistently raised performance. This points toward the grand vision of training robot policies on internet-scale human activity data, with only minimal real-robot tuning. For roboticists, the takeaway is tangible: if you have access to human demonstration videos, consider reframing the imitation problem. Instead of wrestling with hand pose noise, see if you can define an action that both the human and your robot can share cleanly. For tasks where orientation is secondary, translation may be the bridge. For others, maybe a similar idea could drop velocity or force signals. The principle is to find the common denominator of motion between embodiments. In the broader context of embodied AI, this is another step toward flexible learning across embodiments. Coupled with vision-language models, it means a household contains a mountain of data just from watching people do chores – data a robot can tap into with smart abstractions. The door of the microwave might not care about how your wrist twisted, just that you got there and pulled. The robot now knows HOW to "get there," thanks to all those human videos. If I were pitching the future, I’d say: imagine a house-cleaning robot that got its training from casual YouTube footage of people tidying up their kitchens. This work suggests that might not be as far-fetched as it sounds, as long as we ask the robot to imitate only the things humans do in common language – like “move your gripper from here to here,” ignoring the rest. The bridging action is one such “common language.” This could open the floodgates of human data for robot skills. Sources: The ideas and results above come from the preprint “Translation as a Bridging Action” by Sijin Chen et al. (ByteDance Seed), as well as related discussions on imitation from human video data. Where data and quotations are provided, they are drawn from that paper’s description of method and evaluations.