Latest / Embodied AI 101 / Hy-Embodied-0.5-VLA: A Massive Bimanual Teleoperation Dataset for Vision-Language-Action
Transcript
- Hy-Embodied-0.5-VLA: A Massive Bimanual Teleoperation Dataset for Vision-Language-Action. Tencent’s robotics research team has just unveiled a new dataset and model suite that significantly pushes the scale of vision-language-action (VLA) data for real-world robots. Dubbed Hy-Embodied-0.5-VLA, this release comprises over 2,163 hours of dual-arm teleoperated manipulation data – roughly 250,000 individual episodes – covering more than 70 distinct tasks. In plain terms, it’s one of the largest multi-view, bimanual manipulation datasets to date. Alongside the raw data, Tencent is providing a trained VLA model designed to directly leverage this dataset, enabling robots to understand visual scenes and follow language instructions across two-armed tasks. Importantly, the entire package is shipping in LeRobot’s standardized dataset format, meaning it plugs into the popular LeRobot framework (v3.0) for robotics learning on the HuggingFace Hub. In this article we’ll walk through exactly what this dataset contains, how it was collected, what the VLA model does, and why it matters for anyone building multi-arm robot controls. The Need for Large-Scale Teleoperation Data. Why is this important? In recent years, the robotics community has recognized that building truly general manipulation models – especially ones driven by language commands – requires immense amounts of diverse data. Just as large language models (LLMs) and vision-language models have exploded in capability by training on Internet-scale text and images, the vision-language-action (VLA) paradigm aims to apply the same idea to physical robots. VLA models are typically pre-trained to comprehend images (or video) and language, and then fine-tuned to output sequences of robot actions. To be effective, they need broad coverage of tasks and scenarios. However, real-world robot data collection is expensive and time-consuming, which has made such large datasets rare. Most prior efforts have been limited to a few tens of hours or thousands of episodes. Notable examples include the RoboTurk dataset (with a few hundred teleoperated demos on tasks like laundry folding and object search) or the Ego4D-driven teleoperation efforts with human demonstration. By contrast, Hy-Embodied-0.5-VLA essentially creates an order-of-magnitude jump in scale for bimanual robot data. Beyond just volume, this dataset emphasizes multi-view, egocentric recording – meaning the robot’s onboard cameras capture the scene from several angles while a human operator guides the arms. This mimics a realistic teleoperation setup, where an operator wearing VR goggles sees the world through the robot’s eyes. In fact, Tencent’s team collected data by having humans teleoperate a two-armed robot (with cameras mounted on each arm) through a range of manipulation tasks. The result is a richly annotated dataset where each episode consists of synchronized video streams from multiple camera viewpoints plus the corresponding joint-angle and gripper-action records for both arms. Because the recording is done in the UMI (Universal Manipulation Interface) style, any new robot could theoretically play back these episodes, provided it shares the basic kinematic interface – hence why the dataset page is labeled “UMI”. (UMI is an emerging community-backed data format that aims to make robot data “universally shareable”; as the UMI site puts it, even a simple GoPro on a table-top gripper can capture data that’s useful broadly.). By providing over 2,000 hours of such data across 70+ tasks, this dataset covers chores and skills far beyond any single lab’s hobby projects. The tasks include everything from basic pick-and-place and stacking to more complex manipulations like opening drawers, tool use, combining objects, and coordinating both arms to carry heavy or awkward items. Importantly, the two arms can play off each other: for example one arm might hold an object steady while the other operates a tool, or they might pass objects between hands. This bimanual aspect is especially valuable because many real-world tasks (like assembly or handling large objects) fundamentally require two hands. The diversity of tasks means that the model trained on this data should learn a broad sense of how objects and arms interact in 3D space. Indeed, when designing their Hy-Embodied models, Tencent’s authors specifically noted that real robots need strong spatial-temporal vision and embodied reasoning – abilities that are hard to get from text-only or image-only datasets. All those combined camera views plus language prompts should help the model build spatial understanding (for instance, how objects move from one camera’s view to another’s). Dataset Collection and Content. How did Tencent amass 2,163 hours of data? The short answer is systematic teleoperation at scale. The robot platform was equipped with multiple synchronized cameras – typically stereo or depth cameras on each arm and perhaps an overhead view – capturing the robot’s egocentric viewpoint. A human operator, using virtual reality or remote joystick controls, performed the tasks in real time. Every episode records video frames at, say, 30 FPS from each camera plus high-frequency logged state (joint angles, gripper open/close signals, etc.) for both arms. Because it’s UMI-formatted, the vision data is packaged as continuous MP4 streams (or image sequences) and the state/action signals as Parquet tables – a combination explicitly supported by LeRobot’s dataset format. This ensures the data is easily loaded and replayed by any LeRobot-compatible software. Each episode corresponds to a single attempt at one task – for example, “transfer the red cube onto the green platform” or “screw the knob onto the base”. The dataset spans over 70 such task definitions. For most tasks, the dataset probably covers variations in object arrangement, lighting, or starting positions to avoid overfitting. The sheer number of episodes (250,000+) means there are roughly on average 3,000 episodes per task, though in reality some tasks (like basic grasping) are even more densely sampled and others (like complex assembly) fewer. In terms of raw frames, with a few video streams running, each hour of data could correspond to on the order of 100,000 to 200,000 image frames from each camera. Crucially, the data is egocentric and multi-view. This means that rather than having a static external camera watching the robot, the camera moves with the robot’s arms. For example, one camera might be attached near the right wrist, and another near the left wrist. When the robot reaches out, the view pans accordingly. This makes the learning problem more challenging (the model can’t rely on fixed background cues) but also more realistic: it mimics what the robot itself actually sees when it operates. It also provides stereo or multiple overlapping views of objects, which can help with depth perception. The model trained on this data will effectively learn to combine information from different viewpoints, a capability that is rare in most current VLA benchmarks. Nearly all episodes were collected via teleoperation, meaning a skilled human demonstrator did the task using a controller or VR interface. Each demonstration naturally includes implicit intent that is not fully captured by just state trajectories. To bridge that gap, the collected trajectories have been paired with language instructions: after collection, the episodes were annotated so that each sequence is associated with a textual command or task description. These instructions might be simple (e.g. “move the red block to the blue block”) or hierarchical (“first pick up the screwdriver, then tighten the screw”). In some cases, Tencent’s team even used their own large vision-language model (like the HY-Embodied model itself) to automate generation of language instructions from video snippets, similar to recent pipelines used in other work. The point is that each episode includes both the sensory trajectory and the corresponding language goal. This pairing is what makes it a vision-language-action dataset, not just a vision-action dataset. Another key point is that the dataset is compatible with LeRobot v3.0. In practical terms, this means anyone can load the data with LeRobot’s APIs, inspect it, filter episodes, and train policies on it using LeRobot’s training scripts. LeRobot uses a Parquet+MP4 scheme, as noted in its documentation: “a standardized, scalable LeRobotDataset format (Parquet + MP4 or images)”. This dataset adheres to that. So under the hood, you’d find for each episode a video file (synchronized across channels) and a Parquet table of robot states and actions. Researchers can easily push this to HuggingFace like other LeRobot datasets, and LeRobot’s standard tools (data loaders, visualizers, etc.) will just work. In fact, the HuggingFace page for this dataset explicitly titles it , indicating it’s in UMI format ready to plug into LeRobot. The Vision-Language-Action Model. Along with the data, Tencent has released a pre-trained VLA model tuned on it. The name “Hy-Embodied-0.5-VLA” suggests it’s built on the same architecture family as their Hy-Embodied-0.5 vision-language models (the ones that impressed Gemini-class performance on benchmarks). Conceptually, this VLA model ingests images and a language instruction, and outputs a sequence of robot actions. Technically, it is a multimodal transformer: it fuses visual tokens (from multiple camera views) with a text token stream, and autoregressively generates commands. In practice, the Hy-Embodied architecture uses a “Mixture-of-Transformers” backbone specialized for vision processing, with latent tokens for images and text, and separate “heads” for different modalities. For VLA, the final output would be formatted as a command sequence that the LeRobot system can interpret – for example, “(x,y)” for where to move an end-effector, or “[min,max]” for where to place something, following the output conventions mentioned in the repository. (In their code examples, a user message is given as a list of image and text elements; for multi-view input one would simply supply multiple images in that list.). Crucially, this model is trained end-to-end to solve real two-arm tasks. As the tech report notes, once a robust vision-language backbone is developed, it can be fine-tuned or adapted into a control policy. In fact, in the HY-Embodied-0.5 paper they explicitly say “we leverage our robust VLM foundation to train an effective Vision-Language-Action (VLA) model, achieving compelling results in real-world physical evaluations.”. That is exactly what’s happening here: the team used the Hy-Embodied pre-training and their new 2K-hour dataset to produce a single model that can answer language queries by outputting robot actions, apparently with high success on physical tasks. We don’t have the exact performance numbers yet (Tencent has not posted a full paper detailing skill completion rates under this new model). However, we can infer from related work that this model should be capable of many tasks out-of-the-box. Having watched projects like LingBot-VLA and PaLM-E, we know that with tens of thousands of real demonstrations, such VLA models can generalize surprisingly well. Tencent’s solution uniquely brings multi-view awareness via the dual-arm camera feeds. In competitive terms, LingBot-VLA (from Ant Group) was trained on 20,000 hours across 9 robots and achieved state-of-the-art transfer. Hy-Embodied-0.5-VLA is smaller (2,163 hours) but still substantial, and uses a cutting-edge model backbone. One expects it to perform tasks like pick-place, drawer-opening, or tool use significantly better than smaller or single-view models. The fact that each episode includes synchronized streams likely means the model has learned how to fuse the different viewpoints into a coherent 3D understanding of the scene – a capability rarely found in earlier VLA agents. Another interesting aspect is training efficiency. These foundation models typically require large compute to train, but once available, they can be fine-tuned on a particular robot or task with relatively few extra samples. The LeRobot integration means users could fine-tune Hy-Embodied-0.5-VLA on their own episodes if desired. And since the dataset is public, anyone can experiment – for example, condense the 2K hours into even smaller fine-tuning sets to see how quickly the model learns a new skill. How It Compares to Other VLA Efforts. To put things in perspective, let’s briefly compare to other recent VLA initiatives. The flagship example in embodied language models is Google’s PaLM-E (2023), which combined a frozen PaLM language model with vision encoders to perform simple manipulation tasks. PaLM-E was trained on an eclectic mix of robotic imitation data and vision-language benchmarks, and could answer questions about images or output movement tokens in a limited vocabulary. Hy-Embodied-0.5-VLA follows a similar high-level recipe but scales it up in data and model size, and focuses on real dual-arm teleoperation. More directly, Ant Group’s LingBot-VLA (2026) is a closer analogue. LingBot was trained on roughly 20,000 hours of teleoperated data across 9 different physical dual-arm platforms. Each of those robots had two 6- or 7-DoF arms with grippers and multiple RGB-D cameras for multi-view observations. After pretraining, LingBot achieved state-of-the-art success rates on real-world benchmarks by fine-tuning on only tens of demonstrations per task. It highlights that scaling data volume and combining multiple robots can greatly improve generalization. Hy-Embodied-0.5-VLA, while trained on a single platform (to our knowledge), is in the same spirit of “dual-arm plus multi-camera” data. The main difference is scale: LingBot’s dataset was an order of magnitude larger, but Tencent’s dataset is still huge by any standard. Both projects demonstrate that more hours of diverse teleop data yield better performance without clear saturation. In fact, LingBot observed that success and progress scores kept improving up to 20K hours – a phenomenon we expect Hy-Embodied’s authors to also see if this dataset is used extensively. Another project, Xerox’s Pi0.5 and GR00T series, have also tackled VLA. These use public simulation tasks (like RoboTools or RoboTwin) plus some curated real data, but generally report on dozens to hundreds of hours or a few thousand demos (much smaller than 2,163 hours). Also, many VLA benchmarks use single fixed cameras or synthetic environments. By contrast, Hy-Embodied-0.5-VLA provides multi-angle egocentric views and real-world physics in one package. This means in likely head-to-head tests (say on a benchmark like MetaWorld or a custom dual-arm tasksuite), the Hy-Embodied model could shine, especially in cluttered or large-scale tasks where depth perception is crucial. Finally, it’s worth mentioning the format and community aspect. By releasing this dataset in LeRobot’s standard UMI format on HuggingFace, Tencent is essentially inviting the community to build on it. The UMI (Universal Manipulation Interface) approach is designed to let labs and companies share data easily. In fact, the UMI Data community site lists other open datasets (like GenRobot, RH20T, UMI-3D, etc.), but none approach this scale in bimanual teleop. This new Hy-Embodied-0.5-VLA-UMI entry likely jumps to the top of such leaderboards. As the UMI manifesto proclaims, “Data is better when universally shareable”. Now that ethos has a new poster child. Potential Uses and Next Steps. For practitioners in robotics and machine learning, there are plenty of exciting implications. First, anyone doing VLA research can now pretrain or fine-tune models on this large, realistic dataset. A group building a home service robot, for example, could use the Hy-Embodied model as a starting point and adapt it to their specific environment with minimal additional data. The LeRobot compatibility means one can plug this dataset directly into training pipelines (using PyTorch, for example) without custom data wrangling. Second, we may see more benchmarks and challenges emerge based on it. Since it’s multi-view and bimanual, it could become the backbone of a new standardized evaluation suite. Competitions could be arranged where teams submit their VLA policies and run them on the 70+ tasks, or even open-end challenges like “generalize to a new two-arm robot after pretraining”. This might echo how ImageNet or GLUE jump-started progress in vision and NLP, but now for embodied AI. Third, the fact that Tesla’s-own MoT and Distillation tricks were used suggests this VLA model could run on edge hardware. The MoT (Mixture-of-Transformers) infrastructure is designed to be efficient (the base model has 4B total, but only 2.2B active at runtime). In theory one could deploy it on a laser-edge board in a real robot with two cameras and a modest GPU. Having a 2B parameter model controlling a real dual-arm robot in real time would be quite an achievement. It could push us toward consumer-level intelligent robots that can follow spoken commands in a home. On the research front, this release will also help us ask questions: How much does multi-view help, compared to single view? How important is bimanual camera coordination? Would an LLM-based question-answering on the video frames add anything beyond what the model inherently learned? These are now empirically testable, because the dataset is here for others to experiment. For instance, one could ablate the dataset to single-camera by throwing out one arm’s feed, then compare policy success rates. In summary, Hy-Embodied-0.5-VLA brings to the table a scale of realistic, dual-arm robot data that’s unprecedented in the open world. It demonstrates a clear direction: as we feed more hours of rich teleoperated data into VLM-based architectures, our robot “brains” become much more capable. As the authors note, this “equips the model with rich, generalizable representations” that transfer to complex tasks. For the robotics community, that is precisely the kind of foundation we’ve been striving for – and now it’s publicly available to build on. Sources: The details of the Hy-Embodied models come from the Tencent HY-Embodied tech report and paper. The scale and format (UMI/LeRobot) are documented on the UMI data site and LeRobot docs. For context on similar VLA efforts, see the LingBot-VLA release and appliance of UMI methodology.