Latest / Embodied AI 101 / VERA: Video-to-Action World Model Policy

Transcript

VERA: Video-to-Action World Model Policy · Embodied AI 101

• VERA: Video-to-Action World Model Policy. Roboticists have long dreamed of generalist control systems: one model that can pick, reorient, and otherwise manipulate objects across wildly different robot bodies and environments. A new 2026 paper from MIT introduces VERA (Video-to-Embodied Robot Action Model), a surprisingly effective step in that direction. The core idea is simple yet powerful: use a large video generation model as a task planner and then translate its imagined future frames into real robot actions using a learned inverse dynamics model that is specific to the robot. This decoupling of vision-based planning from low-level control allows the same planner to drive both a 7-DoF Panda arm and a 16-DoF Allegro robotic hand, for example, without retraining the video model. VERA achieves strong zero-shot performance on both simulation and real-world benchmarks – even matching or exceeding recent end-to-end “vision-language-action” baselines on several tasks. In short, VERA shows that “video models can be robot policies” in practice, so long as we carefully engineer the translation from pixels to motors. Decoupling Planning and Control. Traditional robot policies either require massive amounts of paired action data (as in behavior cloning) or training joint video-and-action models (“world-action models”) on internet-scale datasets of robots. Because real action-labeled robot data is scarce, recent work has explored using video generation as a backbone for control. For example, Liang et al. (“Video Generators are Robot Policies”) show that training a model to generate videos of robot tasks, and then decoding actions from that video, can dramatically improve sample efficiency and robustness. VERA takes this to the extreme: it does not try to predict actions at all in the video model. Instead, the authors keep an action-free video planner completely intact and learn a separate translator that maps the planner’s pixel predictions into motor torques or velocities. Why decouple? As the authors note, leaving the video model embodiment-agnostic has natural advantages. The video planner – a huge pretrained model – never sees or learns the specifics of a particular robot’s joints or action space. In principle that planner could then run any robot, from a simple finger to a dexterous hand, if we simply teach each robot its own “idler” to interpret the video. Concretely, VERA trains one Inverse Dynamics Model (IDM) per robot. This IDM only needs to learn how that robot’s actions move pixels in the image; it doesn’t need to regenerate high-level plans. Because IDM training data (random or scripted trajectories) is comparatively easy to collect in simulation or from self-play, each robot’s IDM can be trained separately without expensive action annotations on the video data. The result is a kind of modular pipeline where:. Planner: A large video diffusion model (14B parameters) generates a sequence of future frames showing how to accomplish the goal. Translator (IDM): A learned Jacobian-based inverse dynamics model for this robot takes each predicted frame increment and computes the low-level controls needed to produce the same pixel motion. Execution Loop: The robot executes a short chunk of those controls, observes the result, and re-plans from the video model. This decoupling brings several key benefits. The video model (planner) remains interchangeable – one could swap in a different video model without retraining the IDMs. Each IDM can be trained only on that robot’s interactions (no language or external data needed). And crucially, the video model retains its full visual reasoning power, since it never gets “muddy” with action-specific layers. As Lester Li et al. point out, jointly training vision and action heads can dilute the model’s generality. By contrast, VERA preserves the model’s raw ability to imagine strategy while handling control in a separate stage. In practice, VERA indeed shows that decoupled planning plus faithful action translation is a viable path to generalist control. The same video-planning component can direct a table-picking Panda arm or command multi-fingered reorientation in an Allegro hand – the difference is only in which IDM the system uses. One researcher even likened this to using the video model as a “scratchpad” of future camera views, with the IDM reading off the required joint moves. As the authors summarize, “decoupled video planning plus faithful video-to-action translation is a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control”. The Video World Model Planner. At the heart of VERA is a video generative model that serves as a long-horizon planner. Given a context video (e.g. a few past frames) and a language goal (e.g. “place the red cube on the blue block”), the model autoregressively generates future frames showing the scene evolving towards the goal. In the VERA paper, the planners are huge diffusion-transformer models – on the order of 14 billion parameters – pre-trained on massive video data. Specifically, they use the recent Wan family of video generation models (which support both 1.3B and 14B sizes) that have shown state-of-the-art capabilities in producing realistic, goal-directed video clips. After pre-training on general videos, this model is further fine-tuned on robot-centric videos (RGBD or RGB from the robot’s camera) of assorted manipulation tasks. Crucially, no actual robot actions are used in training the planner – only the raw video frames and textual prompts or frame seeds. This means the video model can leverage all the rich visual and linguistic knowledge from human or simulated videos without being constrained to any single robot morphology. At run-time, the planner is used in a receding horizon scheme. First, the robot captures the current observation (one or more frames). The video model, conditioned on these frames and the task instruction, samples a short “dream” of future frames (say M frames ahead). In practice the authors have used something like 8–20 future frames, depending on the task. These frames depict how the scene should change over time: an object being lifted, a cube being placed, a button being pressed, etc. Because this is a probabilistic model, it effectively scores many possible futures; VERA uses standard diffusion sampling to produce a coherent video snippet that best completes the instruction. Of course, a raw video alone isn’t a robot control sequence. But it does encode the visual intent of the task. The next stage of VERA extracts actions from those imagined frames. One can think of each predicted frame change as a visual hint of what the robot should do in the next moment. Before we describe the translation step, note a key design point: the video model itself is untouched by reinforcement learning or robotic data. Unlike some other recent approaches that fine-tune a vision-language-action model to predict both images and actions jointly, VERA’s video planner stays exactly as pretrained. This “action-agnostic” planner has two major upshots. First, it means one can swap in any comparable video model. For example, in principle you could use NVIDIA’s 14B Cosmos-Predict model or Meta’s or someone else’s 14B diffusion model without redesigning the rest of the system. Second, it frees the planner to use all the information in video+language data without being tied to a particular robot’s action space or kinematics. The authors note that video models have incredibly broad knowledge (both visual and textual), and past work suggests that forcing them to predict actions can actually hurt that understanding. VERA avoids this by relegating action prediction to the IDM. Jacobian Inverse Dynamics for Action Translation. This brings us to VERA’s other key innovation: the Jacobian Inverse Dynamics Model (J-IDM). The problem is this: given two consecutive frames (predicted by the video planner), how do we compute the low-level robot commands that would make the real robot produce the same visual change? Naively, one could train a neural network to map from to the joint velocities or torques. Indeed, past work call such networks “Direct IDMs” (D-IDMs) or similar, where the model learns an arbitrary regression from image differences to controls. However, these tend to be very data-hungry and brittle – especially as the robot’s degrees of freedom grow. A 16-DoF hand is a 16-dimensional control signal, and regressing that from pixels is extremely challenging. VERA’s solution is to bake in known structure from rigid-body kinematics. In classical robotics, the end-effector velocity $\dot x$ is related to joint velocities $\dot a$ via the robot’s Jacobian matrix $J(q)$ (which depends on the current pose). Here, instead of 3D end-effector coordinates, we consider 2D pixel motions. The authors design J-IDM to predict, for each pixel in the image, a local $2 \times n$ Jacobian that links small action increments $\delta a \in \mathbb{R}^n$ to 2D pixel displacements $\delta p \in \mathbb{R}^2$. In other words, J-IDM learns a function $J_\theta(p,o)$ that outputs a $2\times n$ matrix at each pixel location $p$, given the current observation $o$. This matrix approximates how each action dimension would shift that pixel. Concretely, training proceeds with ordinary self-play or scripted robot trajectories. An off-the-shelf optical flow solver first computes the actual per-pixel displacement field $v_t$ between $o_t$ and $o_{t+1}$. Then, J-IDM is trained with two objectives. (1) Forward consistency: $J_\theta(\cdot, o_t),Δa$ should match the observed optical flow $v_t$ (for all pixels). In practice they use a robust loss ($L_1$ or Charbonnier) over all detected flows. (2) Inverse reconstruction: since $J_\theta$ is usually non-square, they form its (damped) Moore–Penrose pseudoinverse $J^\dagger_{\theta,\lambda}$ and enforce $J^\dagger_{\theta,\lambda}(\cdot, o_t),v_t \approx Δa$. In other words, the model should also be able to recover the applied action from the predicted pixel flows. This dual loss ensures that the action-to-pixel relationship is well-learned. At inference time, the process is elegant. The video planner produces two successive frames $(\hat o_t, \hat o_{t+1})$. We compute their optical flow field $\hat v_t = \text{flow}(\hat o_t, \hat o_{t+1})$. Then we feed $\hat v_t$ and $\hat o_t$ into the learned J-IDM. Mathematically, we solve [ \delta a_t = \mathop{\mathrm{argmin}}{\delta a}; \sum_p\bigl|J\theta(p,\hat o_t),\delta a - \hat v_t(p)\bigr|^2 + \lambda |\delta a|^2, ] which has the closed-form solution $\delta a_t = J^\dagger_{\theta,\lambda}(\hat v_t)$ given the pseudoinverse formula. In words: the J-IDM uses the current predicted frame and pixel motions to compute the action fully consistent with that motion (with a small regularization to avoid nonsensical large actions). This Jacobian approach is the heart of VERA’s success in high-DOF robots. Because it explicitly models how joint motions produce image motions, it generalizes far better from modest data than an unstructured regressor. The MIT team reports that J-IDM is significantly more data-efficient than a direct image-to-action network, especially as degrees-of-freedom increase. For example, in a toy 2D finger environment, J-IDM required about half the data of a direct IDM to reach comparable accuracy. Crucially, J-IDM scales smoothly to the 16-DoF Allegro hand: it outperforms a direct IDM by a large margin on any realistic dataset size. From a practical standpoint, the Jacobian IDM is still just a learned neural network (a U-Net–style transformer) that ingests an image and outputs a dense map of local Jacobians. It does not require any kinematic model of the robot; it instead learns the mapping in image space. But by constraining the learning via the Jacobian structure and inverse objective, the model effectively builds in the idea that actions cause consistent pixel motions. The authors demonstrate that, once trained, the J-IDM can faithfully “translate” the planner’s dreamed frames into executable commands. In essence, it acts like a differentiable inverse simulator for the robot’s visual effects. Closed-Loop Planning and Execution. VERA runs in a loop of plan–execute–observe–replan. At each cycle, the system has some recent history of camera observations and the task prompt. It queries the video planner to sample a short sequence (say M frames) of where the world should go. Then, rather than blindly executing all M frames, VERA commits to only the first K steps of that imagined plan (for example, $K=4$). The J-IDM translates that $K$-frame chunk into a corresponding sequence of $K$ low-level controls. The robot then executes those $K$ actions on the real hardware (either simulation or the physical robot). This yields new camera observations after each sub-step; after executing the full chunk, the new observation replaces the history and the cycle repeats. Why chunked execution? The reasoning is that even a very good video planner will not produce exactly perfect predictions. By executing only a few steps and then re-planning, VERA corrects for model errors, unmodeled contacts, lighting changes, etc. Frequent re-planning keeps the policy closed-loop. The authors note that this balance – long enough horizon $M$ to get meaningful strategy, but short enough commitment $K$ for responsive control – is especially important in contact-rich tasks like turning a cube, where precise feedback is crucial. Essentially, VERA is doing model-predictive control, with the video model as the predictive world model and the J-IDM as the inverse dynamics solver. Because the video model is very expressive, it can handle abstract or visually rich goals. The system uses language prompts to specify tasks – for example, “place the yellow cube on top of the lego block,” or “push the blue button.” At runtime these prompts are fed into the video model (often via CLIP or a built-in text encoder), and the model dreams a sequence completing that instruction. On the user-facing demo site, the authors show a variety of commands like “yellow cube → plate,” “tennis ball → bowl,” and button pushing tasks. In each case, the video model imagines a plausible video of the goal, and VERA executes it. Importantly, the real robot never sees the language prompt or high-level explanation; everything is mediated through the video frames. The J-IDM makes no use of the prompt text. It simply knows: “whenever the video shows the cube moving forward, compute the torques that cause it to move similarly.” In other words, language is only used to steer the vision model’s imagination; the control stage is language-agnostic. This separation means that extending VERA to other languages or instruction modalities might only require changing the prompt encoder, without retraining any dynamics models. Experimental Results: Sim and Real. The VERA paper demonstrates the method on a striking range of tasks and settings, to showcase its generality. These include both simulated benchmarks and real-robot experiments, involving two very different robots: a 7-DoF Franka Panda arm using a wrist-mounted camera, and a 16-DoF Allegro five-finger hand holding a cube (seen by an eye-in-hand camera and a fixed external camera). Simulated Benchmarks. In simulation, they evaluated several tasks with increasing complexity and degrees of freedom. One simple setting is a 2D “robot finger” that can slide in the plane. In this toy domain, they compared their J-IDM against a direct IDM and a few baselines. As already noted, J-IDM learned faster and achieved better flow–action accuracy, especially as the finger’s action dimension grew. This was a warm-up to show that the idea scales beyond 2 or 3 dimensions. More instructively, they ran three main sim tasks:. PushT-Sim: A planar pushing task where a box must be moved to a target. The robot is a point or simplified finger, and success requires predicting straight-line videos and converting them to pushes. MimicGen Panda-Sim: A 7-DoF arm simulation (matching the real Panda), where the goal is to move objects in the scene (including lifting and stacking) in response to language prompts. Allegro-Sim: A 16-DoF dexterous manipulation task in Isaac Gym, where the Allegro hand must reorient a cube from one face to another. For example, the prompt might be “rotate the cube so the blue face points upward.”. Across these tasks, VERA proved robust. The J-IDM reliably translated the planner’s moves into controls. In "MimicGen Panda-Sim", for instance, if the video model showed the arm swinging and placing a block, VERA’s Panda-arm with its trained IDM would actually swing and place the block in simulation. Quantitatively, VERA matched or beat baselines on success rates – and importantly, it generalized to configurations not seen in training data. Real Panda Arm. Perhaps the most impressive results come from a real Franka Panda arm setup. Here a wrist camera looks down at a table with objects (cubes, buttons, ball, etc.). The authors tested zero-shot tasks: the video model had never seen these exact scenes or tasks in robot data but had been primed on generic robot videos. They then asked the Panda to perform a variety of new language-conditioned tasks. For example, in one trial the prompt was “place the yellow cube on top of the lego block,” and the starting configuration had a yellow cube, a lego block, and other distractors scattered about. VERA was able to infer the plan (via the video model dreaming the cube on the block) and the Panda executed it – all without any fine-tuning on that scenario. A suite of such language tasks was evaluated. These included reaching button presses (push the blue/orange button), placement tasks (cube to plate/bowl), and orientation tasks. A single video planner handled all six tasks by swapping out the linguistic prompt, with no additional training. Performance was strong in many cases. For simpler pick-and-place tasks (like cube to block), VERA succeeded nearly every time in the real world, even though lighting and camera angle varied from trial to trial. On more challenging tasks like pushing through occlusion or fine-grained offsets, it was slightly less reliable but still considerably capable. Remarkably, the authors compared VERA to recent state-of-the-art robot foundation models (“Vision-Language-Action” or world-action models like DreamZero and NVIDIA’s policies). On some of the harder reasoning-style prompts, VERA actually outperformed these baselines. For example, tasks that required offset placements or handling occlusions were done more often correctly by VERA than by end-to-end trained models. Qualitatively, many of VERA’s failures came from the IDM being slightly off (the cube didn’t end up exactly where the video showed), rather than the planner misunderstanding the task. In fact, in several failure cases the video model did predict the cube in the right place, but the real cube slipped or misaligned during execution. This suggests that VERA is indeed preserving the vision model’s reasoning power – any shortfall is in the control translation. Real Allegro Hand. Finally, the authors showed cross-embodiment transfer with a real 16-DoF Allegro hand. This is a tough hardware demo: the hand grasps a cube between its fingers, and with its many joints it must roll or reorient the cube to show a specified face upward. The camera setup included both a hand-mounted and an external watcher, similar to some recent dexterous manipulation papers. Without any additional planner training, they simply took the same video model used by the Panda and gave it new prompts like “rotate the cube forward twice”. Miraculously, VERA worked here too. The video model’s plan (a hand motion causing the cube to flip) plus the Allegro-specific J-IDM led the real hand through the multi-step flip. Success rates were lower than the Panda tasks (as expected given complexity), but they still passed a majority of test cases. Even more striking: this is zero-shot transfer across embodiments. The Panda and Allegro have completely different kinematics and visuals (one is a 2-finger electric gripper, the other a 5-finger springy hand). Yet the video model did not need to know the difference. Only the J-IDM had changed, which mapped the video’s finger-like motions into each robot’s joint actions. VERA thus demonstrates generalist control in the literal sense: one high-level model, several low-level motors. The exact same “dream” video could be sent to different robots, each guided to execute it as best as they can. This is in line with the authors’ slogan: “A single video planner, six prompts, one arm” (and by extension, “we can swap in another arm or hand with its own IDM”). Comparison to End-to-End Models. It’s worth contrasting VERA with the more conventional “vision-language-action” approach. In the latter paradigm, one trains a big model (or pair of models) on datasets of videos with actions, often using language or task labels. For instance, NVIDIA’s DreamZero or Octo policies are trained on simulation data to directly map camera images and language to motor torques. These have been shown to be very capable in many tasks, but they require large datasets of aligned observations and actions (or use heavy self-play with RL) and they still may struggle to generalize beyond their training distribution. Early work on “World Action Models” even includes a learned action head in the same architecture as the video world model, effectively merging planning and control into one net. VERA’s philosophy is quite different. Instead of binding the video and action predictions, it keeps them separate. The authors argue this both alleviates data bottlenecks (video models can leverage web-scale videos without actions) and preserves reasoning (the video model’s knowledge isn’t “diluted” by action-specific layers). Empirically, this seems to pay off in their tests: VERA matches or beats these state-of-art end-to-end models on many benchmarks, especially on the harder tasks involving long-horizon planning or fine reasoning. One should note, though, that VERA has its own limitations. The translation from pixel motion to actions still relies on accurate optical flow and linearization; very rapid or complex contacts could throw it off. And if the video model predicts something physically infeasible for the robot, the IDM can only do its best. For example, a visual plan that flicks the cube in midair might not translate into any legitimate finger torques. In such cases the policy could fail. The authors observed that VERA typically fails gracefully: often the cube ended up slightly mispositioned or dropped during an in-hand task, rather than catastrophically going astray. Importantly, when failures occurred, analysis showed that the video plan was usually still correct. This suggests that further improving the IDM (or adding compliance control, etc.) might help close the gap. In other words, the main bottleneck in VERA’s failures was usually the control fidelity, not the planning. Open-Source Release. A nice practical aspect is that the VERA team has released their code and model weights to the public. Both the 14B video planner (indeed, likely a Wan diffusion model checkpoint) and the training code for the Jacobian IDM are available on GitHub. This means researchers can fairly easily try VERA on new robots or tasks. One could imagine taking the video planner, collecting random data on a new robot to train an IDM in a few hours, and plugging it in. The released repository even includes instructions for training the J-IDM on simulated self-play data, so one could scale to custom robot geometries. Given that major robotics benchmarks now are showing off generalization to new environments and embodiments, VERA’s open-sourcing is timely. In future work, the community might extend VERA by swapping in language-conditioned video models that are continually updated, or by combining the video planner with model-based refinements. But the core idea – let the video model reach its full potential as a planner, and handle control separately – points to a compelling design pattern for embodied AI. Conclusion. VERA marks an important step toward generalist robot control by leveraging the recent boom in video foundation models. It shows that a 14B-parameter video world model can be turned into a zero-shot robot policy simply by adding a lightweight, Jacobian-informed inverse dynamics “translator” for each robot. The results are impressive: one planner handles language-specified tasks across two very different robot bodies, both in sim and on real hardware. This work suggests that highly capable video generative models (trained on text and video alone) can serve as powerful action-free “brains” for robots, as long as we correctly ground their visuals to motors. The takeaway is that decoupling planning from control – rather than end-to-end fusion – can yield more flexible and scalable robot policies. With code and models available to the public, VERA provides a new platform for researchers to build on and hopefully brings us closer to truly multipurpose robotic intelligence. In the words of the authors: using large video models as planners combined with faithful video-to-action translation is “a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control”.