Latest / Embodied AI 101 / RynnWorld-Teleop: Digital Teleoperation via World Model Rendering
Transcript
- RynnWorld-Teleop: Digital Teleoperation via World Model Rendering. Imagine teaching a robot a complex bimanual task without ever moving a robot in the real world. Instead, a human puts on VR trackers or simply waves their hands, and a high-fidelity video appears on screen showing the robot performing the task from its own point of view. That is the promise of digital teleoperation, a concept crystallized in RynnWorld-Teleop – a recent action-conditioned world model from Alibaba’s DAMO Academy. RynnWorld-Teleop replaces the physical robot in the loop with a generative video model. An operator’s real-time hand poses drive this “virtual robot,” producing egocentric videos of a dexterous bimanual system working in a real scene. Crucially, those synthesized videos are fully consistent with the unseen robot’s joint actions. In effect, one records a task demonstration purely in a virtual space – yet that demonstration can be consumed by downstream training as if it were gathered on the actual machine. In this article, we unpack how RynnWorld-Teleop works, why it matters, and what it means for robot learning. We’ll see that it introduces three key ideas – depth-aware skeleton conditioning, a progressive human-to-robot training curriculum, and streaming diffusion distillation – all combined to yield “real-time, high-fidelity robot video” from just a single reference image and a stream of hand poses. We’ll examine its capabilities and limitations, how policies trained on pure synthetic data can solve real tasks, and how this approach fits into the broader trend of generative world models and simulation in robotics. The Vision of Digital Teleoperation. The core insight of digital teleoperation is to decouple day-to-day robot data collection from any particular physical robot or workspace. In conventional teleoperation, every demonstration requires an operator, a robot, and an environment – a resource-intensive process that bottlenecks scale. Instead, RynnWorld-Teleop turns the robot itself into a simulated video instrument. A human records a sequence of hand-motions (for example, by wearing trackers) and an egocentric camera image; the model uses those motions to render the video as if the robot were executing them in that scene. The actual robot never moves. Because the generated frames are tied to precise joint actions (the retargeted hand poses), the entire trajectory – visual observations plus underlying actions – becomes training data for imitation learning. In short, the operator teleoperates a model, and the model “acts out” the task in video. This breaks the physical-robot data bottleneck. If it works, robot policies can be trained on unlimited demonstrations: just vary the operator’s gestures in front of the world model. Operator effort still matters, but they can work in any space or on any robot, in effect. The model composes the environment and robot behavior from generative priors. As the authors emphasize, digital teleoperation means robot data collection is “bound only by operator imagination, not by physical infrastructure”. Prior work has dabbled in similar ideas, but RynnWorld-Teleop stitches them together uniquely. Some methods simply paste a robot over a video (“video-to-video translation” like Phantom or Masquerade), but these are passive – you can’t control future frames. Others build action-conditioned video models, but they almost always show human hands in the result. For example, systems like Hand2World generate interactive egocentric videos from human hand motions, but still with a human’s hand visible. To truly teleoperate a robot, the system has to produce robot-centric views (with the robot’s grippers, not human hands) and it has to generate frames from actions, not just translate pre-recorded clips. The RynnWorld team frames it this way: a digital teleoperation system must be (i) robot-centric (the output is a robot’s view), (ii) action-grounded (every frame links to a known action), and (iii) real-time (interactive control). Before this work, no video-generation framework ticked all three boxes. RynnWorld-Teleop is precisely the first system to hit all these marks. A person waves their bare hands, and the model “imagines” the robot doing it – in one’s own environment – at 40+ Hz. Figure 1 of the paper (not shown here) outlines the result: a single reference image anchors the scene, and the human’s 21-joint hand poses (encoded cleverly in the input) cause the model to roll out an egocentric video of the robot, as if it were executing that motion. In effect, the hand-pose stream becomes an embodiment-agnostic action label, directly retargetable to any robot. The video model then supplies the matching visuals, yielding a full state-action trajectory ready for imitation learning. The RynnWorld-Teleop Architecture. Under the hood, RynnWorld-Teleop is built on a state-of-the-art video diffusion framework (specifically, a version of the Wan image-to-video Transformer) but augmented to take in hand-skeleton inputs. The pipeline has three main pillars:. Depth-aware skeleton conditioning. The 21-joint hand pose of the operator is rendered into a 2D image (an “action map”), but with a twist: each joint is drawn with a color and radius that encode its depth (distance from the camera). In practice, the authors generate a “skeleton video” where the hand keypoints are color-coded by depth (e.g. closer parts brighter or larger). This depth cue in the 2D input gives the model explicit 3D structure. It helps the video model understand how the hand moves in 3D space, bridging the gap between the 2D projection and real world geometry. The paper emphasizes that this simple rendering trick – changing point color/radius by distance – supplies crucial spatial information that a raw point skeleton would lack. Large-scale pretrained model. They start from a powerful pretrained video diffusion model (Wan2.2 I2V-5B). This model uses a 3D Variational Autoencoder plus a Transformer-based denoiser, and was originally trained on a massive collection of videos without any hand or robot actions. In fact, RynnWorld-Teleop’s development is carefully staged. Initially, the model is further pretrained (stage 0) on a huge corpus of human egocentric videos without any action conditioning. This lets it absorb generic manipulation “priors” – how people reach, grasp, etc. Then, in stage 1, they fine-tune it on a smaller dataset of paired human-robot demonstrations, injecting the hand-skeleton control branch (a Conv3D embedding in the Transformer) and learning to map those human gestures to robot-centric video outputs. The key point is that the heavy lifting of learning hand-object interactions is first done with abundant human data; only after that does it see any robot. Teacher–student distillation to causal generation. The Transformer denoiser is a bidirectional diffusion model (it processes video with full temporal context), which yields high-quality video but is slow and non-causal. To make the system interactive, the authors distill this model into a causal, autoregressive student that can generate frame-by-frame (with a sliding caching attention). They employ a streaming distillation process (building on the Denoising Diffusion Models literature) that first “warms up” the student (so it learns basic denoising and becomes stable) and then applies a scheduled rollout-consistent training to compress the teacher’s knowledge into 4 diffusion steps of the student. The result is that the final student model can run at ~40 frames per second on a single NVIDIA H100 GPU, rendering long videos in real-time. Visually, the architecture looks like a standard latent-video diffusion model (with VAE encoder/decoder), except for two extra inputs: the first-frame image and the skeletal video. The first frame $I_{ref}$ defines the static environment and robot appearance. Internally it’s encoded and provided as the initial latent context. Concurrently, each hand skeleton frame (depth-colored) is encoded via the VAE (or via a small Conv3D head) and injected into the diffusion process via a learnable patch embedding. This “control branch” was added to the Transformer layers as zero-initialized extra attention (so as not to disturb the pretrained latents at first). Over training, its scale factor is learned so the model integrates the skeleton cues smoothly. The generative process then unfolds: at inference time, the student model takes the current latent (starting from the reference frame) and the next 4 diffusion noises (cause student is 4-step). It outputs the latent for the next frame, which is decoded to RGB. Because it’s causal and uses a key-value cache, it only attends to past frames. The authors cite that this causal model still preserves spatiotemporal coherence across segments, thanks to their careful distillation. Chunking and Re-anchoring. One practical trick they use is chunked re-anchoring of long sequences. Purely autoregressive generation for hundreds of frames can drift (objects may slowly shift or deform). To prevent that, they generate video in chunks of ~81 frames. The first chunk starts from the true initial frame $I_{ref}$. For subsequent chunks, they “re-anchor” by reintroducing a real frame from the robot’s viewpoint at that time, then continue generation. In practice, this means every 80 frames they reset the context with an actual egocentric image from the robot. This ensures that objects (even if moving under unknown physics) don’t wander away – the generated video gets periodic reality checks. (In effect, their “complete digital teleop” still relies on an initial real image and occasional real frames to stay grounded.). The Online System. It’s worth noting that the RynnWorld-Teleop framework was demonstrated end-to-end. Alibaba’s team built an actual teleoperation setup: a mobile base (TIANJI M6) with two 7-DoF arms and WUJI dexterous 20-DoF hands, and a head-mounted RealSense camera for an egocentric view. A human operator teleoperated this system in actual demonstrations during data collection. The operator wore HTC Vive trackers on chest, arms and wrists, providing 6-DOF pose inputs which were mapped (via IK) to the robot’s joints. These teleop sessions generated the paired human-robot demonstration videos needed for fine-tuning. Importantly, the hand-tracker motions from those demos became the skeleton streams for training, and the corresponding joint commands were the ground-truth robot actions (54-dimensional: 2×7-DOF arms + 2×20-DOF hands). Because of this setup, each generated video frame is perfectly synchronized to a vector of joint angles – exactly the imitation learning data format. Performance and Real-Time Generation. After training and distillation, the RynnWorld-Teleop model achieves real-time speeds. The paper reports about 40.0 frames per second at a resolution of 832×480 on one H100 GPU. At ~25 ms per frame, this exceeds typical robotic vision frame-rates and is far beyond prior diffusion-based video models (which might do only a few Hz). They break down the time: about 5% is spent encoding the skeletal input (converting hand coordinates to latent), 72% on the 4-step denoising, and 23% decoding back to pixels. The result is an interactive system: as soon as the human moves their hands, the new frame appears almost immediately on screen as if the robot clutched or placed an item. The authors distribute two versions of the model: the original “full” fine-tuned teacher (Diffusion) and the distilled “causal” student. For practical streaming use, the causal model is the choice. They even provide code to compile the student model to TorchScript or use 8-bit (FP8) quantization on NVIDIA Hopper GPUs for extra speed. Generative Simulation vs. Physics Simulation. It’s useful to contrast RynnWorld-Teleop with traditional teleoperation and simulators. A simulator like MuJoCo or PyBullet requires building 3D CAD models of every object and scene. RynnWorld, by contrast, requires just one real image of the scene to implicitly encode everything. In principle, it can handle textured, deformable, or fine objects without manual mesh creation – though with some limitations. Importantly, the model is pixel-based and learned, not rendering from a physics engine. This means it “knows” about the visual distribution of, say, a variety of dining plates from training data, and can realistically render lighting and appearance. Yet it also learns (from the paired demos) how objects move under the robot’s hands – it implicitly encodes those dynamics in its generative process. The tradeoff is that it may hallucinate physics not based on first-principles, but rather on patterns seen during training. The authors find that for many tasks (like picking and pushing), this data-driven approach yields convincing, useful simulations. To their benefit, RynnWorld-Teleop also embodies “zero 3D asset overhead”. The paper notes that without needing URDFs or meshes, one can move to new environments simply by feeding a new reference image and possibly collecting (or hallucinating) new object states. In effect, it is a “photo-in, video-out” simulator, with the action channel provided by human motion. Using RynnWorld-Teleop Data for Policies. Beyond being a neat visualization, RynnWorld-Teleop’s main goal is to generate training data for robot control. The entire demonstration process—hand poses + robot-state trajectories + images—is captured synthetically. The authors tested this in a true data-generation scenario: they took three state-of-the-art imitation/policy learning methods (a diffusion-policy, and two offline RL “π” networks) and gave them varying mixes of real teleoperation episodes and RynnWorld episodes. The question: does adding synthesized data help?. On the real robot, the team set up four challenging bimanual tasks:. Dual Picking: pick an apple with left gripper and a banana with right gripper from a plate, sequentially placing both on the table. Block Pushing: left arm pushes a large block from the left side to center, then right arm pushes it from center to the target right zone. Bimanual Lifting: both arms tightly grasp a heavy “watermelon” plush on the table and lift it together to place into a tray. Lid Placement (Precision): pick up a loose lid and carefully align it to close a box. Each task was randomized (object positions/orientations) and repeated ~35 times for success rates. The policies were identical except for their training data. The results (Table 3 in the paper) were remarkable. For all three policy types, augmenting 300 real examples with 300 Rynn-generated examples improved success rates in virtually every task. For instance, one diffusion-policy (DP) had about 57% success on the Lid task using only 300 real demos; with Rynn data added, it jumped to 65%. The biggest gains were in the high-precision Lid task, where success often rose by ~20 percentage points. It wasn’t a fluke of one algo: even the strong “π₀.₅” policy went from ~43% to ~63% success on Lid with the extra data. Most strikingly, the authors show a zero-real-data experiment. A policy (“π₀”) trained purely on 300 RynnWorld episodes (and no real demonstrations) could still solve some tasks reasonably well. With only synthetic data it achieved ~82.9% on Block Pushing and ~77.1% on Bimanual Lifting. In qualitative terms, that means the generative world model had captured enough of the true dynamics to train a controller that transferred to the real robot with no real examples! (As the authors emphasize, this highlights that the synthetic videos are not just visually plausible but “physically grounded” on the key manipulation motions.) Of course for harder tasks like Lid-placement the pure-synthetic policy fared worse (~28.6% success), but still non-trivial for zero real data. In short, RynnWorld-Teleop can create useful training labels where none existed. To further investigate, they compared feature distributions of real vs. synthetic data. Using a pretrained video network, they embedded 1000 frames each from real demonstrations and from RynnWorld-generated demos. The t-SNE plot shows a large overlap between real and synthetic trajectories. This confirms that the model’s output lives in the same feature space as reality, explaining why policies generalize. In their words, “our generative world model successfully captures the underlying distribution of robotic manipulation, including complex hand-object interactions and lighting”. On the vision-quality side, RynnWorld-Teleop also outshines simpler baselines. They quantify video fidelity using metrics like FVD (video Frechet distance) and PSNR. A naive baseline – simply fine-tuning the same video diffusion model on the robot data – yields very poor FVD (e.g., 1223). In contrast, RynnWorld-Teleop’s final model has FVD around 585 on key tasks, showing much crisper, coherent video. Ablations show how the design matters: removing the human pretraining stage causes the model to collapse (FVD 2598, with ghosting and vanishing objects). Using naive concatenation for skeleton conditioning also crashed quality (FVD >1000). In sum, the carefully crafted pipeline – depth encoding, staged training, distillation – is what makes these videos high-fidelity. On the real-time front, RynnWorld’s performance of ~40 fps is itself a record for an interactive world model. By comparison, prior action-conditioned video models ran at only a few Hz. Here the student model’s optimized 4-step diffusion and cached attention produce frames faster than the typical 30 Hz real camera rate, effectively eliminating perceivable lag. The authors note this means the video loop can keep up with a human’s hands even in fast interaction. Limitations and Caveats. The results are exciting, but several caveats are worth noting. First, as with any cutting-edge claim, we should be cautious about real-world reliability. An AI Weekly write-up on this paper highlights that the authors did not show a head-to-head baseline comparison against policies trained on purely real teleoperation data. In other words, while adding Rynn data improved existing methods, we haven’t seen Rynn-trained policies directly compared against a fully-real baseline on identical tasks. Similarly, there’s no systematic study of generalization to truly novel objects or extreme cases – the tasks all use objects seen during training, and lighting changes minimal. In practice, a generative model can only reproduce physics it has indirectly learned, so edge cases (liquids, very soft objects, untrained kinematics) could fail. In fact, the authors themselves mention that the model “occasionally struggles with complex physical phenomena such as fine-grained liquid dynamics or the manipulation of highly deformable objects”. Furthermore, RynnWorld-Teleop currently needs per-robot fine-tuning. The model was trained specifically on the TIANJI+WUJI hardware; to apply it to a different robot would likely require collecting some new paired demo data and re-training. The authors note this as a limitation: cross-embodiment generality is an open problem. One could imagine future “foundational world models” that take a robot’s kinematic description as input and generalize, but RynnWorld-Teleop is specialized to one embodiment. There is also the fact of chunk re-anchoring. Their system to prevent drift relies on feeding real camera frames periodically. In an online teleop session, they simply have the real camera available. But if one wanted to use RynnWorld as a fully offline data engine, this implies either one must record corresponding real frames (as they did) or figure out how to guess them. It means the environment is only simulated incrementally, not fully from scratch. Finally, since this is a learned model, it inherits biases of training data. If the human and robot demos cover only certain setup variations, the generator may hallucinate stereotyped behavior. The AI Weekly piece wisely cautions that “zero-shot Sim2Real is a claim that tends to shrink under independent replication”. We should be optimistic yet critical: this is a powerful proof-of-concept, but robotics engineers will want to see many more benchmarks across tasks and domains before treating the synthetic output as fully reliable. Related Work and Context. RynnWorld-Teleop arrives in a rapidly-moving field at the intersection of robotics, world models, and generative vision. It is one branch of a broad push at Alibaba DAMO into “embodied AI foundation models” (others include RynnWorld-4D for multi-modal 3D world modeling, and RynnBrain for vision-language-action models). Conceptually it echoes ideas of “digital twins” and simulator-in-the-loop training, but with learned visuals instead of physics-based graphics. There are a few related papers: for instance, Hand2World and GeneratedReality (Wang et al., Xie et al.) also use diffusion for egocentric video from human hands, but they leave the hand in view. Others like Phantom and Masquerade tried directly translating human videos to robot videos by inpainting robot arms, but they don’t support interaction or actions. RynnWorld’s leap is in combining these strands with a real-time causal model. There are also parallels to policy learning with learned simulators or video models. For example, recent work on video-conditioned policies (like Qwen-RobotWorld by OpenAI) use language and images to predict future robot states, but generally not in a closed-loop video sense. Reinforcement learning has seen some “imagine to control” ideas, but diffusion-model-based rollouts for robots are quite new. One could compare RynnWorld-Teleop qualitatively to learning on traditional simulations (MuJoCo) or on pure video prediction – it’s a third path that seems promising especially when creating photorealistic image data is important. Additionally, the idea of using human demonstrations to scale robot learning is an old one; what’s new here is turning human gestures into universally useful robot data. Previous work like imitation-from-video (HOWTransfer, H2R-UDA, etc.) tries to learn robot policies by watching humans, but they still needed sensors on the robot or alignment networks. Here, the “alignment” comes from the operator retargeting interface plus the world model. In a sense, RynnWorld-Teleop automates a big chunk of the embodiment transfer. Takeaways and Future Directions. RynnWorld-Teleop demonstrates a new paradigm: a generative approach to making robot data. Its biggest selling point is decoupling training data from hardware: if you buy this idea, any lab without a robotics fleet could still scale up by “teleoperating” a world model. Compared to classic simulators, it requires no 3D models or physics engines for each scenario; compared to collecting on real robots, it can potentially churn out more data at lower cost. The technical takeaways are equally interesting. Encoding 3D hand motion as a depth-colored skeleton is a clever hack that others will likely adopt. The progressive training (pretrain on humans, then adapt to robots) showcases how large unlabelled human video corpora can seed even specialized models. And the auto-regressive distillation is a neat example of compressing a diffusion model into a streaming form – following lines of works like R-Diffusion, but applied here to robotics. In short, it’s a strong demonstration that video diffusion models are not just for generating Hollywood clips – they can serve as interactive simulators for control. That said, the path forward will involve extending and hardening this approach. The authors themselves point out directions: richer datasets (covering fluids, cloth, deformables), more generalizable conditioning (e.g. provide a robot specification so the same model serves multiple robots), and maybe integrating this with physics or geometry constraints to avoid hallucinations. Also, integrating force/torque or tactile sensing remains unaddressed – RynnWorld only models vision and the taught actions. For practitioners, the immediate utility is clear: the code and models are open-sourced, so one could try generating teleop data in their own scenario. Even if you still fine-tune in a simulator or on a real robot eventually, generating synthetic training might speed up the early phases of policy learning or help with domain randomization. It’s already apparent from the experiments that even a mixed dataset of real+synthetic trajectories consistently outperforms real-only. In a future where generative models grow ever more powerful, RynnWorld-Teleop suggests a compelling vision: robot learning pipelines where human intuition (in the form of hand gestures) can be “played back” in endlessly varied virtual scenes. Rather than having to own a robot stack for data, labs could deploy video models and human controllers. It’s an intriguing step toward what the authors call “training general-purpose robotic agents by leveraging human intuition as a digital control signal”. Time, replication, and broader tests will tell how robust that leap really is – but this work firmly plants the flag for digital teleoperation as a viable component of the embodied AI toolbox. References: Zhao et al. (2026), RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation (Alibaba DAMO Academy). See also the AI Weekly summary of the paper and the RynnWorld-4D project for related embodied world-model work.