Latest / Embodied AI 101 / UnifoLM: Predict the Interaction, Not the Whole World

Transcript

UnifoLM: Predict the Interaction, Not the Whole World · Embodied AI 101

• UnifoLM: Predict the Interaction, Not the Whole World. UnifoLM-WLA-1.0: One Model Driven, Whole-Body Coordination • Unitree • Unitree Robotics • Project-page and Hugging Face release • 2026. The release behind the headline. Unitree’s latest UnifoLM release starts from a deceptively simple question: how much of the visual future does a robot actually need to predict in order to act well?. A conventional video world model tries to generate the next frame—or several future frames—in full. It models the background, lighting, texture, camera noise, robot body, manipulated objects, and every other visible detail. That is attractive if the goal is photorealistic simulation. But for control, most of those pixels may be irrelevant. When a humanoid folds a towel, the policy primarily needs to anticipate where the hands and towel will move, not reproduce the grain of the table or the color gradient on the wall. UnifoLM-WLA-1.0 takes that observation seriously. Its world-modeling backbone learns to predict a compressed representation of the regions expected to change during an interaction. That representation is then combined with embodied visual reasoning, discretized robot actions, and a continuous action-generation expert. The complete system is described as a six-billion-parameter humanoid foundation model trained on approximately 2,500 hours of real-robot data. Unitree says one model supports 54 tabletop tasks and ten whole-body manipulation tasks while controlling two-finger grippers and several five-finger hands. Before getting into the architecture, however, we should correct a subtle but important simplification in the original announcement. UnifoLM does not directly predict optical flow instead of full frames. Optical flow is used offline to discover which image regions changed between two observations. Those dynamic-region masks are compressed by a VQ-VAE, and the vision-language model predicts the resulting discrete mask tokens. In other words, the learned world-model target is best understood as a tokenized map of future visual change—not future RGB pixels and not the optical-flow vectors themselves. There is also a naming distinction worth preserving. The interaction-centric component is UnifoLM-ER-Flow, a roughly 4.45-billion-parameter model derived from Qwen3-VL-4B. The six-billion-parameter figure refers to the complete UnifoLM-WLA-1.0 system, which adds an MMDiT action expert. As of September 12, 2026, Unitree has released the ER-1 and ER-Flow weights, while the complete WLA checkpoint and post-training code remain listed in the open-source plan without active release links. The official model card describes the work as a 2026 miscellaneous release rather than linking to a paper or peer-reviewed venue. So this is not yet a conventional paper review. It is an architectural analysis of a unusually revealing project release: enough artifacts exist to understand the central design, but many experimental details and ablations remain unpublished. The architectural through-line. UnifoLM-WLA-1.0 is easier to understand as four layers: an embodied reasoner, an interaction-centric prediction objective, a discrete representation of robot motion, and a continuous action expert. UnifoLM-ER-1: specializing a general VLM for embodied space. The foundation is UnifoLM-ER-1, a four-billion-parameter embodied reasoning model built from Qwen3-VL-4B. Unitree reports training it on more than five million samples covering point prediction, object detection, multi-image reasoning, 2D trajectory prediction, 3D object detection, and spatial question answering. General image-text examples are mixed into that training to reduce catastrophic specialization away from ordinary multimodal understanding. This matters because robot policies need a somewhat different visual representation from a typical visual assistant. A general VLM may recognize a mug and explain its purpose, yet still struggle to estimate a reachable contact point, determine whether the mug is behind the gripper, or predict where it will move under a particular push. UnifoLM-ER-1 therefore emphasizes operational geometry: points, trajectories, relative positions, multi-view correspondence, and 3D structure. The benchmark results broadly support the claim that this training produces a more spatially specialized model. On Unitree’s table, ER-1 scores 82.0 on Where2Place versus 63.0 for its Qwen3-VL-4B base, 88.9 versus 79.6 on EmbSpatial, and 73.1 versus 61.7 on RoboSpatial. But this is not a free, across-the-board upgrade. ER-1 trails the base model on MME and MMMU_VAL, for example. The evidence therefore points more strongly toward a useful redistribution of capacity toward embodied spatial reasoning than toward universal multimodal improvement. Unitree says ER-1 leads open models on seven of sixteen reported benchmarks, with some closed-model results obtained through the providers’ APIs and some benchmark subsets evaluated under Unitree’s own protocol. The released configuration confirms that ER-Flow remains structurally recognizable as a Qwen3-VL conditional generator rather than becoming a bespoke video diffusion network. It uses a 36-layer language transformer with a 2,560-dimensional hidden state and a 24-layer visual encoder. Unitree adds robot-state support and a custom vocabulary for actions and dynamic-region masks. The configuration reserves a 120-dimensional robot-state input, although the public materials do not yet explain how different embodiments are padded, masked, normalized, or mapped into that common state envelope. ER-Flow: turning future change into language-model tokens. The central idea enters with UnifoLM-ER-Flow. Take two robot-view images: the current observation and a later observation from the same trajectory. An optical-flow estimator identifies image motion between them. From that motion, the preprocessing pipeline extracts a dynamic-region mask indicating where the future scene differs in a task-relevant way. A VQ-VAE then compresses that spatial mask into a fixed-length sequence of discrete codes. The VLM receives the current image together with either a task description or a robot action. Its prediction target is the sequence of codes representing the future dynamic region. If the instruction is “place the plates on the rack,” the desired output is not a rendered video of the complete kitchen. It is a compact representation covering the robot hands, the plates, the rack contact area, and the visual changes induced by the placement. Unitree shows the same mechanism on table cleaning, towel folding, phone packaging, and plate placement. The released tokenizer makes this idea more concrete. It contains a bank of 256 segmentation tokens, delimited by segmentation start and end markers. The world target can consequently travel through an ordinary autoregressive vocabulary alongside text and action symbols. The checkpoint index reports 4,447,303,168 parameters and a stored BF16 size of roughly 9.7 gigabytes. This is a neat representational move. Rather than attach a high-dimensional video decoder to the VLM, Unitree translates future visual effects into something the language model already knows how to manipulate: a short sequence of categorical symbols. The VQ-VAE is doing more than compression here. It imposes a learned vocabulary over patterns of scene change. Some codes might implicitly correspond to compact moving objects, broad deformable regions, bimanual interactions, or particular spatial arrangements of motion. We cannot inspect those semantics from the project materials alone, but the codebook gives the transformer a much lower-entropy prediction problem than raw video synthesis. Discrete action learning: one vocabulary for several bodies. ER-Flow also brings robot actions into the token space. Unitree divides its unified action representation into three conceptual streams: end-effector pose, end-effector joints, and lower-body joints. End-effector joints cover either a simple gripper or a dexterous hand. Each stream receives its own residual vector quantizer, and the resulting tokens are synchronized by timestep before being fed to the VLM. Residual vector quantization approximates a continuous motion with a sequence of discrete code selections. An initial code captures a coarse movement, while later residual codes can account for what the earlier approximation missed. This gives the model a finite action vocabulary without requiring each token to represent an entire high-dimensional motion exactly. The public tokenizer exposes 256-entry banks for end-effector position, end-effector rotation, hand motion, and lower-body motion. It also includes explicit delimiters such as , , and. This suggests that the nominal end-effector-pose stream is internally separated into translational and rotational token families, while hand and lower-body motion use their own codebooks. Why tokenize actions if the final robot requires continuous commands? Because the discrete actions are primarily a representation-learning interface. They allow language, visual geometry, anticipated scene change, and physical motion to participate in one autoregressive modeling problem. The model can learn that a particular class of end-effector trajectory tends to produce a particular dynamic-region pattern, even when the exact continuous joint commands differ across robot configurations. This factorization is also Unitree’s main mechanism for cross-embodiment transfer. End-effector poses provide a relatively morphology-independent operational language. Hand tokens isolate the embodiment-specific grasping mechanism. Lower-body tokens handle balance and mobile manipulation separately from arm trajectories. The design does not eliminate embodiment differences, but it gives the network a better decomposition than concatenating every motor into one undifferentiated vector. The MMDiT action expert: returning to continuous control. The complete WLA model adds an MMDiT action expert to ER-Flow. The project diagram shows images, language, and robot state entering the ER-Flow backbone, while noisy action sequences pass through a multimodal diffusion-transformer-style flow decoder to produce continuous actions. A stop-gradient marker appears between the backbone and the action expert, suggesting that Unitree can train the continuous decoder against a relatively stable multimodal representation rather than allowing every action update to modify the VLM. Unitree has not yet published enough training detail to know when or how strictly that boundary is applied. This division of labor is sensible. Discrete action tokens provide a semantic and cross-embodiment training signal, but quantization errors are undesirable at the actuator interface. The continuous expert can generate smooth action chunks conditioned on the richer representation learned by ER-Flow. Unitree calls the full system a six-billion-parameter model, while the released ER-Flow checkpoint contains about 4.45 billion parameters. It is therefore reasonable to infer that the remaining capacity is largely associated with the action expert and connecting modules, but Unitree has not published an exact parameter breakdown. Why predicting change may beat predicting video. The strongest argument for interaction-centric modeling is not merely that masks are cheaper than videos. It is that full-frame prediction optimizes the wrong balance of information for many control problems. Consider a tabletop trajectory where the gripper moves a cup ten centimeters. Ninety percent of the image may remain unchanged. A pixel-level model can reduce its average reconstruction loss by becoming extremely good at copying the table, wall, and background while remaining comparatively poor around the cup boundary—the one region where prediction quality matters to the policy. Video diffusion can address some of this through scale and sophisticated objectives, but it still spends substantial capacity representing appearance. A dynamic-region target changes the learning pressure. Every predicted token refers to the support of future change. The model must identify the manipulated object, localize the robot’s expected intervention, and anticipate which parts of the observation will be affected. It cannot obtain a low loss simply by learning that most pixels stay the same. The result is closer to an effect model than a complete visual simulator. It answers, “Where will this instruction or action change the scene?” That may be sufficient to regularize a manipulation policy, choose among candidate motions, detect mismatches between intention and predicted consequence, or establish correspondences between actions from different embodiments. The representation is also more interpretable. If the policy fails to put a plate on the rack, a researcher can decode the predicted dynamic mask and ask whether the model expected change around the correct plate and target location. A latent feature predictor may contain equivalent information, but it is considerably harder to inspect. There is nevertheless an important distinction between output efficiency and total system efficiency. ER-Flow avoids generating dense RGB futures, but it still runs a multibillion-parameter vision-language model over full images. Optical flow and mask extraction add offline preprocessing. The final policy still needs a continuous action expert. Unitree has not published mask-token sequence lengths, world-loss training FLOPs, WLA latency, onboard hardware requirements, or a comparison against an otherwise identical full-frame model. So the safe claim is that the design reduces the complexity and entropy of the world-modeling target. We cannot yet quantify how much it reduces end-to-end training cost or control latency. Likewise, “maintaining high visual fidelity” is not quite the right description: the system intentionally abandons appearance fidelity in favor of spatial fidelity about expected change. The data and cross-embodiment gamble. Unitree reports approximately 2,500 hours of real-robot training data drawn from its own collections and external sources including BitRobot’s HIW-500. The latter contains whole-body teleoperation trajectories collected with Unitree G1 humanoids in real homes in Southeast Asia, using head and wrist cameras for household, bimanual, and mobile-manipulation tasks. Unitree’s public data footprint now covers tabletop dexterous manipulation, dual-arm gripper tasks, and whole-body teleoperation. Its UniBot-V1 collection includes tasks such as towel folding, plate arrangement, cable insertion, conveyor sorting, storage, and tool organization. The whole-body collection includes bed making, collecting clothes, loading a washing machine, placing drinks in a refrigerator, and cleaning a living room. At least some of the released data uses the LeRobot structure with synchronized images, state vectors, actions, timestamps, and episode indices. One representative dual-arm dataset is recorded at 30 hertz, contains 525 episodes and roughly 413,000 frames, and provides binocular 256-by-256 head-camera video with a sixteen-dimensional arm-and-gripper action vector. A whole-body washing-machine dataset contains separate end-effector, hand, and desired robot-joint fields, illustrating the heterogeneity that the unified representation has to absorb. The appeal of the architecture is that not all of these demonstrations must agree at the raw joint level. Two robots can produce similar end-effector paths and similar scene changes despite having different hands, kinematics, or low-level controllers. Dynamic-region prediction supplies a shared visual consequence, while the factored action tokens preserve the embodiment-specific details. But calling this a unified action space does not make the underlying feasibility constraints disappear. A five-finger hand and parallel gripper do not have identical contact modes. Whole-body motion changes reachability, balance, camera viewpoint, and collision geometry. A Cartesian end-effector trajectory that is simple for one configuration may be near a singularity for another. Independent quantizers may also neglect correlations between lower-body stance, arm reach, and grasp force unless the transformer learns to restore those dependencies. The 64-task demonstration is consequently an encouraging systems result, but the scientifically important question is not whether one set of weights can replay many trained behaviors. It is whether shared representations improve held-out tasks, unseen end effectors, and unfamiliar physical layouts relative to equally scaled policies without the world-modeling objective. Where UnifoLM sits in the world-action landscape. UnifoLM arrives as the boundary between VLA policies and world models is becoming increasingly fluid. Physical Intelligence’s π0 is an obvious architectural relative. π0 places a flow-matching action model on top of a pretrained VLM and trains across single-arm, dual-arm, and mobile-manipulator data. UnifoLM preserves that general template—a semantic multimodal backbone plus a continuous flow-style action expert—but adds explicit future-change and discrete-action objectives inside the backbone. The action expert does not have to be the sole location where physical regularities are learned. DreamZero takes a denser route. It builds a World Action Model on a pretrained video diffusion backbone and jointly models future video and robot actions. Its central argument is that video supplies a universal description of physical motion that can transfer from other robots and even human demonstrations. DreamZero reports running a fourteen-billion-parameter autoregressive video model at seven hertz and obtaining cross-embodiment benefits from small amounts of video-only data. Compared with that approach, UnifoLM’s mask tokens are a much more aggressive information bottleneck: less complete as simulation, but potentially better aligned with economical policy learning. The independently developed WLA-0 similarly unifies language reasoning, world prediction, and action synthesis. It predicts textual subtasks and fine-grained physical dynamics through a dedicated world expert, and can disable explicit world prediction during ordinary inference or activate it for additional test-time computation. That work reports two billion active parameters and 40-millisecond inference on an RTX 5090. Unitree’s system shares the high-level principle that world prediction should shape the action representation, but uses tokenized dynamic regions rather than subgoal images as its distinctive predictive interface. Unitree has not yet supplied corresponding policy-latency measurements. DINO-WM and V-JEPA 2 represent another comparison point. Both avoid RGB reconstruction by predicting in learned visual feature spaces. DINO-WM predicts future DINOv2 patch features and optimizes action sequences against feature-space goals. V-JEPA 2-AC post-trains an action-conditioned latent model using fewer than 62 hours of robot video and performs image-goal planning on Franka arms. These methods retain considerably more latent state than a binary or categorical change mask, but their futures are harder to visualize and audit. UnifoLM occupies an interesting middle ground: abstracted away from appearance, yet explicitly grounded in image-space regions. SelfWAM, released in August 2026, highlights perhaps the most important conceptual challenge. It argues that future prediction conditioned only on an observation and task prompt can learn generic task progression rather than the causal effect of the particular action being executed. SelfWAM therefore conditions its future branch on a clean copy of the demonstrated action and predicts both RGB frames and robot self-masks. UnifoLM can condition dynamic-region prediction on either a task or an action, but the public release does not show counterfactual tests proving that its masks change correctly when the action changes while the image and instruction remain fixed. That action sensitivity is the difference between a visual script and a genuine dynamics model. What the evidence supports—and what remains missing. Unitree currently provides three categories of evidence: embodied-reasoning benchmarks, qualitative dynamic-mask demonstrations, and real-robot videos covering 64 tasks. The reasoning results are the most quantitative. They suggest that ER-1 is substantially better than its base VLM on several spatial grounding and embodied perception tasks. They do not, however, isolate the contribution of dynamic-region prediction, because those results primarily characterize ER-1 before the ER-Flow extension. The dynamic-region examples show plausible masks for cleaning, folding, packaging, and plate placement. What is missing is a dedicated evaluation of those predictions. We do not know the mask horizon, mask resolution, optical-flow estimator, thresholding procedure, ego-motion compensation method, VQ-VAE architecture, reconstruction quality, codebook utilization, or sensitivity to action perturbations. There is no comparison with raw optical flow, RGB prediction, segmentation prediction, DINO features, or a simple frame-difference baseline. Most importantly, the project page does not provide task-by-task policy success rates. There are no confidence intervals, episode counts, baseline policies, perturbation conditions, held-out-scene definitions, or ablations removing the world-modeling loss. The videos demonstrate breadth, but they cannot establish how much ER-Flow improves closed-loop robustness. The public-release state is similarly partial. ER-1 and ER-Flow weights are available under Apache 2.0, including their configurations and tokenizers. The complete six-billion-parameter WLA model, post-training implementation, and detailed deployment instructions have not yet been released through the new repository. Unitree’s older WMA-0 project is more complete: it includes training, inference, checkpoints, and server-to-robot deployment code, and explicitly trains a video-generation world model in both simulation and decision-making modes. This makes WLA-1.0 best viewed as a well-motivated engineering hypothesis with unusually tangible artifacts, not yet as a fully validated scientific result. Failure modes that matter on real robots. The first and largest issue is camera motion. Optical flow does not distinguish object interaction from ego-motion on its own. When a stationary arm moves a cup under a fixed camera, the desired dynamic region is naturally sparse. When a humanoid walks, bends, or turns its head, nearly every image pixel can move. Unless Unitree compensates for camera pose, estimates background geometry, or subtracts global flow, the “interaction-centric” mask can become a camera-motion mask. The project materials do not document this step. That issue is especially relevant because WLA-1.0’s headline contribution is unifying tabletop and whole-body manipulation. The mask objective is easiest in the tabletop regime and hardest precisely where the complete model is most ambitious. The second limitation is that the support of motion does not encode the motion itself. A mask can say that the towel will change but not whether its left corner moves right, upward, or underneath another layer. It discards displacement direction, velocity, depth, appearance, object identity, and contact force. A rich VLM representation may infer some of those variables from context, but the world target does not supervise them directly. This may be enough for rigid pick-and-place. It is less obviously sufficient for cloth folding, cable insertion, pouring, slipping contacts, transparent objects, or tasks where success depends on millimeter-scale alignment. Two physically different futures can occupy almost identical dynamic regions. Third, optical-flow supervision inherits the failure modes of the flow estimator. Motion blur, specular highlights, transparent surfaces, textureless objects, occlusions, small motions, and deformable materials can all produce incomplete or misleading masks. Because the VQ-VAE adds a second compression stage, preprocessing errors can be discretized into apparently confident targets. Fourth, a dynamic mask can miss static but causally essential context. The obstacle a wrist must avoid may never move. A container’s rim, a drawer’s hinge, and a table edge can determine whether an action is feasible while remaining absent from the predicted change region. This does not mean UnifoLM ignores static context—the current RGB observation still enters the VLM—but it means the world-model loss does not explicitly reward preservation of every state variable required for planning. Finally, an autoregressive model may learn stereotyped task progressions. Given a picture of a towel and the instruction “fold towel,” predicting a towel-shaped dynamic region does not prove that the representation understands the consequences of a particular hand trajectory. The appropriate test is counterfactual: hold the image and instruction fixed, vary the candidate action, and determine whether the predicted change responds in a physically correct way. How a research team could use this idea now. Even before Unitree releases the complete training stack, the central method is straightforward enough to adapt. A team could begin with its existing VLA dataset and calculate optical flow between current and future observations. For fixed-camera manipulation, thresholded flow may be a reasonable starting point. For mobile or whole-body robots, the pipeline should first compensate for ego-motion using camera pose, depth, background homographies, or a learned decomposition of global and independently moving regions. Those masks can be compressed with a small VQ-VAE and added as an auxiliary prediction target to an existing VLM policy. There is no requirement to reproduce Unitree’s six-billion-parameter scale. The core experiment is whether forcing a policy backbone to predict action-induced visual change produces better features than behavior cloning alone. Action tokenization can be adopted independently. Represent arm behavior in a canonical end-effector frame, keep gripper or hand configuration in a separate channel, and isolate mobile-base or lower-body motion in another. Train separate quantizers, synchronize their outputs by timestep, and ask the transformer to model them jointly with language and future-change tokens. A continuous diffusion or flow-matching head can still produce the final action chunks. For a convincing evaluation, I would prioritize six tests:. Counterfactual action sensitivity: From one observation, condition on several candidate actions and measure whether predicted change tracks their actual consequences. Ego-motion robustness: Evaluate separately under fixed cameras, moving wrists, moving heads, and whole-body locomotion. World-target ablations: Compare no future objective, full RGB, latent feature prediction, raw masks, and VQ-compressed masks at matched policy scale. True cross-embodiment splits: Hold out an end effector or robot configuration rather than merely mixing all embodiments during training. Closed-loop perturbations: Move objects, disturb grasps, alter clutter, and test whether the world-trained policy recovers more reliably. Systems measurements: Report training FLOPs, output-token counts, action frequency, end-to-end latency, memory use, and performance per unit of compute. ER-Flow’s decoded masks could also become a debugging and planning interface. One could compare the predicted region against a desired goal region, reject actions whose predicted effect is off target, or monitor whether the model expects unsafe interaction near a human or obstacle. These uses go beyond what Unitree has demonstrated, but they exploit the main advantage of an image-grounded world representation: its output can be inspected without training another probe. The released checkpoint is also potentially useful as an embodied VLM baseline. It uses a standard Qwen3-VL model class, albeit with a robot-state field and custom token vocabulary. Researchers can inspect or fine-tune the ER-1 and ER-Flow weights today, but reproducing the full dynamic-mask and action-learning pipeline will remain awkward until Unitree publishes its promised post-training code and quantizers. The verdict. UnifoLM’s most interesting idea is not that it has six billion parameters or that it demonstrates 64 tasks. It is the claim that a useful robot world model does not have to reconstruct the world. By predicting a discrete map of interaction-induced change, Unitree tries to retain what a policy needs—objects, effects, and spatial consequences—while discarding much of what makes full video generation expensive. Discrete action tokens place those consequences in the same representational space as language and motion, and a continuous action expert converts that shared representation back into executable control. The design is technically plausible, interpretable, and well matched to cross-embodiment learning. But its most important claims remain experimentally open. We still need to know whether dynamic-region prediction is truly action-sensitive, whether it survives moving-camera whole-body control, whether it improves policy success over simpler auxiliary objectives, and whether the computational savings remain significant once the entire six-billion-parameter stack is counted. For now, UnifoLM-WLA-1.0 should not be treated as a compact video simulator. It is better understood as an interaction-effect model embedded inside a general robot policy. That narrower description may sound less grand—but it is also the reason the approach could prove genuinely useful.