Latest / Embodied AI 101 / RynnUnify: Does Video Imagination Improve Robot Planning?
Transcript
- RynnUnify: Does Video Imagination Improve Robot Planning?. RynnUnify: Towards Unified Embodied Brain • Minghao Zhu, Bohan Hou, Kun Xiang et al. • DAMO Academy, Alibaba Group; Hupan Lab • arXiv • 2026. Cold Open. Imagine asking a robot to pack a backpack. The laptop goes in first, then a mouse, then the bag closes. Writing that sequence is easy. The interesting question arrives between steps: after the laptop moves, what does the workspace look like, which objects remain reachable, and what should happen next?. Now give the planner the ability to generate a short video of its next operation. Let it inspect the predicted ending, choose another operation, and continue. The video becomes part of the computation that produces the next decision. That is the organizing idea behind RynnUnify. The paper builds a shared foundation for scene understanding and video prediction, then develops extensions for generation-assisted reasoning and robot control. For Embodied AI 101, the question is direct: what changes when a robot’s planner can reason through imagined motion—and where does that imagined future enter the machinery that chooses an action?. The Idea. Start with the family tree. RynnUnify specializes NVIDIA’s Cosmos 3 Nano, which already has 16 billion parameters. Its architecture pairs an autoregressive reasoner with a diffusion-style generator. RynnUnify describes these as two eight-billion-parameter branches connected through joint attention, allowing the generator to use the reasoner’s representations. The research story is embodied specialization: how that capacity is organized and trained for multi-step behavior. During visual planning, the reasoner receives the task, observations, and recent history. It proposes the next subtask, a description of the expected scene change, and a duration. The generator turns that request into a clip. Its ending becomes the observation for the next planning step. In our backpack example, the next decision is made from the predicted post-packing scene. Specifying duration also gives the planner control over the temporal extent of each imagined operation. Two kinds of memory support that loop. Sparse reference frames retain the initial scene and selected history. A continuous prefix from the preceding clip carries recent motion. One supplies persistent information about appearance and layout; the other preserves how the interaction was unfolding. Think of it as a scene scrapbook plus the last few moments of movement. Training teaches these interfaces explicitly. The reasoner uses a tenth of the RynnBrain pretraining corpus alongside specialized supervision for planning, grounding, and reasoning with intermediate video context. Generation training covers egocentric activities, robot manipulation, and embodied reasoning videos. The training sequence develops the reasoner first, then trains the generator with the reasoner frozen. The generator also encounters conditions resembling its own rollouts. Training can corrupt conditioning images or replace preceding endpoints and history with cached generated content, while retaining the ground-truth continuation as the target. This self-forcing recipe makes continuation from model-produced scenes part of training. The engineering principle is to prepare the next model call for the kind of context it will actually receive. The agentic extension adds a separate agent initialized from the reasoner. It can answer directly or invoke frozen RynnUnify to generate video, inspect the result, and answer. Branch-Conditional Policy Optimization, or BCPO, explores both routes. Comparisons within each route improve reasoning; comparisons between routes teach generation selection. Only the agent is updated during this reinforcement-learning stage. Asking which answer is best and asking which route is useful become distinct learning problems. Robot control has its own post-training path. The World-Action Model, or WAM, uses a continuous subtask representation to condition video prediction. A flow-matching action head reads the generator’s hidden features and produces action chunks. It consumes those features directly rather than decoded video frames. The reasoner supplies intent, the generator supplies predictive features, and the action head translates them into commands. That makes future prediction an internal control representation, with an explicit downstream consumer. The Result. The headline planning gain is 9.75 points over Cosmos 3 Nano on Rynn-VP. This benchmark evaluates generated visual plans, including their consistency with the task and preceding operations. The comparison is particularly informative because Cosmos 3 Nano is the foundation RynnUnify starts from: this is an embodied-specialization result at the same headline parameter scale. The improvement is concentrated in the planning and visual-realization components of the score. For a long task, the interesting property is whether step four respects what happened in step three. In our backpack example, the predicted location of the laptop should change what the planner does next. The benchmark puts that continuity into the evaluation target. The public embodied-understanding results provide another piece of evidence. RynnUnify has the highest average among the unified multimodal models evaluated by the authors. On Rynn-GEU, which targets reasoning about unobserved states and interaction outcomes, the agentic extension also improves the overall score over direct RynnUnify answering. For execution, the authors turn to RoboCasa365. This is a simulation framework for household mobile manipulation, with varied kitchen environments, human demonstrations, and synthetically generated trajectories. Its benchmark design explicitly supports studying generalist policies and the effects of task and environment diversity. It provides a natural setting for asking whether predictive representations help a controller carry useful context between operations. RynnUnify’s WAM leads the compared systems on the authors’ selected twenty-task composite set. The physical tests use Astribot S1, where the model reaches 46.7 percent average complete-task success, ahead of the evaluated π₀.₅ and GR00T N1.7 policies. The tasks cover turning and grasping, block swapping, and a repeated bowl sequence followed by spoon pickup. Using completed tasks as the endpoint matters. An early grasp contributes to progress; the reported task-success measure asks whether the sequence reaches its endpoint. The bowl sequence is especially interesting as a task design because the controller must distinguish where it is in a repeated procedure before changing operations. Together, these experiments examine different outputs of the architecture: an anticipated visual sequence, an answer about an unobserved state, and an executed manipulation task. That separation makes the result legible. Each output has a distinct job, and each gets its own evaluation. The Biggest Weakness. The biggest weakness is causal attribution: the headline agentic comparison does not isolate the benefit of generated video. Direct RynnUnify is compared with an additional agent after reinforcement-learning post-training. BCPO explicitly improves reasoning within the direct route as well as within the generation-assisted route. The reported comparison therefore changes both the trained policy and its inference procedure. Several things move together: answer quality, routing behavior, and access to predicted visual context. A plausible explanation is that the additional training produces a better reasoner even on questions where generation contributes little. Another is that generated video supplies useful intermediate representations. The aggregate improvement measures the complete package and leaves those explanations entangled. The decisive control would be a direct-only agent given comparable post-training compute and the same task rewards. Pair that with enabling and disabling generation on the same agent, while holding the questions and answer budget fixed. Those comparisons would separate better answer learning from the additional computation supplied by video. This distinction matters for the central thesis. Sharing a foundation across understanding and generation is an architectural property. Showing that their interaction causes an improvement is an experimental claim. The stronger claim needs a comparison that isolates the interaction. For a practitioner, this changes where to invest. If additional reasoning training accounts for most of the gain, the valuable component is the trained answer policy. If generated context accounts for it, the world model becomes an essential part of the inference path. The engineering decision can change even when the benchmark ranking stays exactly the same. The headline comparison establishes a package-level improvement. A matched control would tell us how much credit belongs to imagination itself. Verdict. RynnUnify deserves attention from researchers building long-horizon world models and policies with predictive internal representations. Its useful design move is to give anticipation an explicit consumer: the next subtask decision, a reasoning answer, or an action head. That offers a concrete way to ask what a prediction is for. Place it in the existing lineage. Cosmos 3 supplies the two-branch foundation. RxBrain had already organized language and visual imagination into a joint planning sequence, with text specifying task structure and imagined states grounding that plan. These works establish a clear research direction: coordinating language-level decisions with representations of the physical states those decisions refer to. RynnUnify develops that direction through successive video rollouts and an explicit interface between subtask predictions and their visual consequences. Its planning implementation exposes the subtask, scene description, duration, and resulting clip as inspectable artifacts. That is useful experimental structure for a lab studying how decisions evolve over a task. For working roboticists, the design suggests a productive separation between intent, state evolution, and execution. Intent describes what the next operation should accomplish. State evolution represents the expected scene change. Execution produces commands for the current embodiment. A shared learned representation can connect those levels while preserving clear duties for each component. The release provides a concrete starting point. The official repository includes visual-planning inference and BCPO training and evaluation workflows, with links to base, agentic-reasoner, and RoboCasa WAM checkpoints. A research team can investigate the interfaces through code and model outputs, rather than reconstructing the entire proposal from diagrams. The most transferable idea is to train a prediction interface that expresses an intended transition and its timescale, then let downstream computation use the consequences. Build predictions around the decision that will consume them. Parameter count supplies capacity; the connections between reasoning, prediction, and action give that capacity an operational role. Outro. Return to the backpack. The interesting moment is the transition after the first placement: the workspace has changed, and the next decision needs to use that changed state. Imagining the transition gives us a way to structure the computation between an instruction and an action. Watch for controllers that learn their own rhythm of observation, imagination, and execution—deciding which futures to examine as part of deciding what to do. That is the idea to carry from RynnUnify: a predicted future can be a working representation inside a decision process. Its value lies in the decision it helps produce. This is Embodied AI 101.