Latest / Embodied AI 101 / GigaWorld-Policy-0.5: An Efficient Mixture-of-Transformers Policy for Real-Time Robot Control

Transcript

GigaWorld-Policy-0.5: An Efficient Mixture-of-Transformers Policy for Real-Time Robot Control · Embodied AI 101

• GigaWorld-Policy-0.5: An Efficient Mixture-of-Transformers Policy for Real-Time Robot Control. Modern robot learning is moving beyond purely reactive policies and toward what Alice Steinmann and colleagues call World-Action Models (WAMs): embodied AI systems that not only map observations to actions, but also explicitly model how the world will evolve under those actions. By predicting future visual observations jointly with actions, these models give richer supervision and can produce more physically plausible behaviors. In practice, however, naively doing joint video-and-action prediction is computationally expensive. Generating even a few frames of predicted video at inference time can take seconds on a GPU, far too slow for a real robot control loop. The new GigaWorld-Policy-0.5 model (ArXiv preprint 2607.13960) tackles this challenge head-on. Building on the team’s earlier GigaWorld-Policy work, it delivers a “real-time” WAM by cleverly decomposing the model architecture. The key idea is to separate the visual-prediction part of the network from the action-generation part. During training, the full model still learns to predict future video frames and actions together; but at inference time, the video branch can be turned off entirely, leaving a lightweight “action expert” to run at high speed. This decoupling, implemented as a Mixture-of-Transformers (MoT) architecture, yields dramatic speed gains: on an NVIDIA RTX 4090 GPU, GigaWorld-Policy-0.5 can produce actions in about 85 ms per decision. That’s roughly 10× faster than the original GigaWorld-Policy (and 38× faster than Motus, a previous WAM baseline), while still improving task success rates. Below we unpack how GigaWorld-Policy-0.5 achieves this, and why it matters for real robots. We’ll cover the core ideas of the architecture, the specialized training strategy, and the experimental results on real-world manipulation tasks. Along the way, we connect to prior work like Motus (which also used multiple transformer experts) and highlight the general principle: “train complex, infer simple.”. World-Action Models: Bringing World Models into Policies. To set the stage, recall that standard robot policies (including recent vision-language-action models) typically do reactive control: at each step they map the current observation (an image + state + possibly a language goal) directly to the next action. These policies can be very effective, but they lack a built-in model of future consequences. In contrast, a world-action model explicitly represents the joint distribution over future trajectories of both the robot and the environment. Practically, a WAM might take a current video frame (or RGB-D image, or some visual context) and a prompt for a goal, and then predict both a sequence of future actions and how the scene will look as time goes on under those actions. For example, it might predict “move arm to pick up this cup (action sequence), and show the camera view of the cup being grasped (future video)”. World-Action Models have been gaining attention recently as a way to combine the strengths of world modeling and foundation-model style policies. Researchers have noted that adding a predictive visual model can provide dense training signals and physically grounded constraints. A recent survey frames WAMs as an emerging paradigm that “unifies predictive state modeling with action generation, targeting a joint distribution over future states and actions”. In practice, some WAM implementations have simply taken a pretrained video diffusion model and fine-tuned it to also output action tokens alongside image tokens. For instance, the Motus system from Tsinghua (Bi et al.) uses optical-flow-based latent actions and a unified MoT architecture with dedicated experts for “understanding, action, and video generation”. But while these approaches can boost policy performance, they struggle with real-time control: running a full video diffusion at each step often takes seconds of GPU time per frame. GigaWorld-Policy (the predecessor to this work) already tackled some of these bottlenecks. Its fundamental insight was an “action-centered” formulation: during policy inference, you don’t have to explicitly render the future video. Instead, you train with video prediction as a auxiliary task to improve the learned action model, but at test time you drop the video branch. As the authors put it, their model uses a causal masking mechanism so that action tokens and future-visual tokens are “jointly modeled” during training, but at inference the visual branch is disabled. This idea – roughly speaking, “train complex, infer simple” – allows GigaWorld-Policy to run much faster than prior WAMs. In fact, the project page claims a 10× inference speedup over baselines like Motus, with only a modest drop in accuracy. Yet even 10× is not enough if open-loop inference still takes a few hundred milliseconds. Many tasks in the real world need control at ~10 Hz or faster to handle dynamic disturbances. This is the gap that GigaWorld-Policy-0.5 is designed to close: it re-architects the model so that action inference alone becomes extremely cheap. In doing so, it leverages two additional techniques: (1) a novel mixed pre-training strategy called “Action-Conditioned World Modeling” (AC-WM) and (2) an automated hyperparameter search pipeline (AutoResearch) to refine the training recipe. Combined, these allow a more efficient policy that retains the benefits of future-vision supervision while hitting less than 100 ms per step in practice. The Original GigaWorld-Policy Pipeline. Before diving into the 0.5 enhancements, let’s briefly review the earlier GigaWorld-Policy approach for context. The goal was exactly to address the WAM bottlenecks: inference overhead and entangled representations. To do this, the original team designed an action-centered world-action model that explicitly learned 2D pixel–action dynamics. Concretely, they trained a unified transformer model (the “world model” backbone) that took as input the current RGB observation (and robot state) and encoded it into tokens. The model was trained to output two things in parallel: (a) a future video sequence (as image tokens) conditioned on that observation and the future actions, and (b) the future action sequence itself, conditioned on the current observation. The loss combined video reconstruction and action prediction, giving the policy a rich training signal. The clever part was the causal masking trick: the transformer architecture was masked so that future-video tokens could attend to the predicted actions, but the action predictions did not depend on the future-video tokens. In other words, the model learned “what will I do?” (actions) while also learning “what will happen if I do that?” (video). But by construction, the action output was conditioned only on the past and not on the generated future frames. This meant that at inference time they could turn off the video branch completely – after all, the actions no longer needed feedback from the visuals. The result was a model that could be trained as a full video-action predictor, yet run as an action-only predictor, saving the cost of video synthesis. Under the hood, the original GigaWorld-Policy had a single transformer backbone that processed all modalities in a unified embedding space. Visual observations, robot sensory state, and action tokens were all projected into that space so the transformer could fuse them. This avoided the “multiple disjoint branches” problem and enforced multimodal alignment. On top of this shared transformer, however, two different decoding heads produced the actions and the video tokens. Thanks to the causal mask, the video head could peek at the future actions, but the action head could not peek at future video. To train this model, the authors also leveraged a huge, hierarchical data pipeline. First they pre-trained a general world model (called GigaWorld-0.5) on massive internet videos, to learn generic physics and appearance dynamics. Next they did an “embodied scene adaptation” phase, using thousands of hours of robot-manipulation video (collected from simulators, first-person demonstration datasets, and real robots). This taught the model the specific spatiotemporal patterns of how objects move under robotic interaction. Finally, they fine-tuned on a relatively small set of real robot trajectories where both observations and actions were available (the “aligned” dataset). Thanks to this three-stage pipeline, the model could leverage large unlabelled visual datasets first, then adapt to the embodied domain, and only at the end learn to map image sequences to actions. In initial experiments, the original GigaWorld-Policy was already much faster than joint video prediction baselines. On a high-end GPU (NVIDIA A100), it ran about 360 ms per inference step, roughly 9–10× faster than a comparable Motus model. On a real robot, its action success rate was on the order of 80–85% on tasks like grasping and assembly, compared to roughly 50–60% for older video-based policies. In short, the action-centered decoupling worked: the policy was both faster and more accurate. Nonetheless, working systems still saw room for improvement in speed. Anecdotally, even 360 ms (≈3 Hz control) may not suffice for fast-moving settings. The team therefore lightened the model further in version 0.5, reconsidering how to split the computation between vision and action. Architecture 0.5: Mixture-of-Transformers Decomposition. The headline innovation of GigaWorld-Policy-0.5 is a new Mixture-of-Transformers (MoT) architecture. Instead of one monolithic transformer handling everything, the model now has two specialized transformer “experts” that work together. One expert is dedicated to processing visual-video tokens, and the other is dedicated to processing action tokens. As the project page describes: the visual expert “specializes in processing video tokens,” while the action expert “focuses on action-token modeling.” Both experts still share information internally, but their separation dramatically reduces redundant computation at inference. How does this work? In the full transformer, you can imagine the tokens being split into three groups: (1) observation tokens encoding the current RGB input (and any robot state), (2) future-video tokens for the predicted frames, and (3) action tokens for the predicted actions. In prior GigaWorld-Policy, all these tokens would flow through the same transformer layers (with the causal mask controlling dependencies). In the new MoT design, we instead have two parallel streams. The visual expert transformer receives the observation tokens and (during training) the future-video tokens. The action expert transformer receives the observation tokens and the action tokens. Crucially, they are not entirely disjoint: there is cross-attention between them in some layers, so that the visual expert can inform the action expert during training. (The figure on the project page hints that they “connect through multi-modal self-attention,” preserving the necessary causal structure.). We can summarize the effect in simpler terms: at training time, the two experts collaborate – the visual branch helps supervise future dynamics and the action branch learns to generate actions given the current observation (still using the causal mask trick). But at inference time, we can choose to run only the action expert. The visual expert is simply turned off. Because the action expert was never allowed to look at the future-video tokens during training, it contains all information needed to predict actions on its own. Therefore, by dropping the visual branch, we avoid decoding a long sequence of video tokens altogether. This MoT structure is reminiscent of Motus, which also used multiple transformer experts. Indeed, Motus “introduces a Mixture-of-Transformers (MoT) architecture to integrate three experts (understanding, action, and video-generation)”. The GigaWorld team’s innovation is to apply a similar idea specifically to the action-centered policy scenario, but with two experts instead of three. They call it “action expert” vs “visual expert”, and they maintain the same cross-token causal masking that enforces the action-centered dependency. By experimentation, this gives a huge speedup in the action-only pass. On modern GPUs, running only the action expert halves the active model depth and attention span. As reported in the paper, the MoT-empowered policy achieves only 189 ms latency on an A100 GPU (down from 360 ms with the monolithic model) and only 110 ms on an RTX 4090 – purely for the PyTorch model. With a C++-optimized inference engine, that falls to 85 ms on the RTX 4090 (the cited 85 ms number). In practical terms, this means the policy can loop at ~12 Hz! In contrast, earlier baselines ran at a few Hz or less (Motus was >3 seconds per step on A100). Importantly, the MoT does not hurt training. During training, the visual and action experts still see each other’s signals (via their inter-attention layers). So the action expert still benefits from the “dense supervision” of future video dynamics. The team confirms that this architecture “preserves the training benefits of future visual dynamics” while making inference lighter. We’ll see in the ablations that performance remains high, meaning accuracy was not sacrificed for speed. Another point: by physically splitting the models, the designers can also tailor the size of each expert. For example, if the visual expert is very large (demanding input channels for images) and the action expert is smaller, then during inference only the smaller model runs. The GigaWorld-Policy-0.5 code and model card indicate the combined model has 30 layers and wide heads, but the inference FAQ suggests the action branch itself is relatively small. Regardless, the result is clear: even on a “realistic” local GPU (RTX 4090), inference is well under 100 ms. This addresses a major hurdle for WAM policies: real-time control. Mixed Pretraining: AC-WM Meets WAM as Training Recipes. Architectural change alone wouldn’t necessarily yield better policies; it just makes inference faster. To actually improve policy quality, the GigaWorld team adds two more ingredients: a mixed pretraining strategy and an automated training pipeline search. The first is mixed Action-Conditioned World Modeling (AC-WM) + WAM training. The name suggests the model is trained not only as a WAM (predicting video+actions together) but also as an action-conditioned world model. In practice, AC-WM likely means training the backbone to reconstruct future observations given action sequences (i.e. standard world model training), along with the usual WAM objective of predicting actions. By alternating or combining these objectives during the pretraining stages, the model “strengthens the coupling between visual dynamics and robot actions,” the authors say. In other words, the actions it learns to predict are even more tightly tied to the physical consequences those actions produce. This signal presumably improves the transferability of the action embeddings when fine-tuning on policy objectives. Unfortunately the summary doesn’t give many more details on exactly how AC-WM is implemented (we would need the full paper). But the ablations hint that purely treating it as a world model up front provides a better initialization for the later policy learning. The idea of hybrid training isn’t new in spirit – many works train world models on unlabeled sequences and then fine-tune for control – but here it’s baked into a single framework. The result is that, by the end of all training, the model has seen a rich mix of data: internet videos, robot videos, and eventually annotated “obs→action” pairs. Supporting this is their staged data pipeline (which also carries over from the original work). To recap from the project page:. Stage 1: General Video Pretraining. They train GigaWorld-0.5 on massive uncurated internet video (no robot demonstrations) to learn common physics and visual dynamics cues. Stage 2: Embodied Adaptation. Then they train on thousands of hours of embodied manipulation videos (first-person demo videos, simulation rollouts, some real-robot footage). This teaches the model the spatiotemporal patterns of real object manipulation under different embodiments. Stage 3: Action-Policy Alignment. Finally, they fine-tune the model on a smaller set of action-annotated robot demonstrations, to align the mapping from “observation (\to) action (\to) future observation.” This is where the AC-WM and WAM tasks are explicitly mixed, using the pretrained encoder from the previous stages. The crucial effect is that by the time they do policy training, the visual dynamics are already well understood at multiple levels of abstraction. This means the action decoder doesn’t have to learn physics from scratch, and can focus on the causal relationships. In ablation studies shown on the project page, they chart success rate as a function of training steps and demonstrate that the mixed pretraining leads to consistently higher final performance than a vanilla WAM-only baseline. We can treat this mixed training strategy as an implementation detail, albeit an important one. The main takeaway is that GigaWorld-Policy-0.5 doesn’t just slap a new architecture on top; it carefully manages the training process to fully leverage that architecture. Together, the MoT design and the mixed pretraining create a model that can “preserve the benefits of future visual dynamics while improving inference efficiency” as claimed. Tuning at Scale: AutoResearch Pipeline. The final enabler mentioned in the summary is an AutoResearch pipeline. This is essentially an automated hyperparameter/hyperconfiguration search system. Its purpose is to remove some of the tedious manual tuning when finding a good training recipe for such a complex model. According to the project page, AutoResearch “automates pilot runs, metric monitoring, and candidate selection” to discover an effective configuration. In plain terms, the team likely set up a meta-controller (perhaps an LLM or simpler agent) that runs short validation sweeps over key hyperparameters (e.g. learning rate, batch size, maybe network size etc.), observes which settings give better small-scale performance, and then chooses the best ones for a longer training run. The result is a more streamlined pipeline: the best configuration is found automatically rather than by hand. A figure on the page shows such a process applied to a fruit-picking task (sweeping learning rate and batch size, then training to 30K steps under the chosen config). Why mention all this? Because it signals that the reported results for GigaWorld-Policy-0.5 don’t come from just “guessing” a config in one try; instead, the authors used this AutoResearch method to systematically find good settings. In a way, it parallels how the architecture separated concerns: just as the model is decoupled into experts, the problem of training it has been decoupled into smaller search tasks. For our purposes, the main point is that the impressive performance numbers of GigaWorld-Policy-0.5 are achieved with a rigorously tuned setup, lending credibility to the results. We won’t need to detail the AutoResearch any further — the concept is straightforward — but it underscores the sophistication of the development process. Automated hyperparameter tuning is becoming more common in deep RL and robotics (e.g. tools like Tune, Ray, SigOpt, etc.), and GigaWorld’s deployment of it helps ensure they extract the most out of their architecture. Experimental Results: Real-Time Speed and Success. All the architectural and training innovations above would be of little interest if they didn’t actually pay off in end results. In fact, GigaWorld-Policy-0.5 shows both huge speed gains and continued or improved task performance. Let’s dig into the key numbers, comparing to relevant baselines. Inference Latency. First, the headline: on a consumer GPU, the action-decoding C++ engine achieves 85 ms per step. How big a deal is this? We can put it in context via the model’s own comparisons:. | Method | Latency A100 (ms) ↓ | Latency (RTX 4090, ms) ↓ | Real-Robot Success ↑ | |------------------------|---------------------|---------------------------|---------------------| | π0.5 | 225 | 110 | 0.76 | | Motus | 3231 | – | 0.80 | | FastWAM | 229 | 182 | 0.78 | | GigaWorld-Policy | 360 | 293 | 0.80 | | GigaWorld-Policy-0.5 | 189 | 110 | 0.85 | | + C++ Deployment | 140 | 85 | 0.85 |. (Table adapted from the project page.). Some takeaways:. Motus (a prior state-of-the-art WAM) was very slow: over 3 seconds per step on an A100 GPU. This is far too slow for practical use. The original GigaWorld-Policy already beat that by producing policies at ~0.36 s on A100 (or ~0.293 s on RTX 4090). But now GigaWorld-Policy-0.5 cuts that roughly in half again – to 0.189 s on A100, or 0.110 s on RTX 4090 for the PyTorch model. Compiling to C++ further cuts it to 0.085 s (85 ms). The table also includes something labeled “π0.5” (pi0.5), which seems to be a particular baseline (score 0.76) and “FastWAM” (another approach). Both of those sit around ~0.11–0.18 s on RTX 4090, so GigaWorld-Policy-0.5’s 0.110 s (PyTorch) and 0.085 s (C++) are at least on par or better. In relative terms, GigaWorld-Policy-0.5 with C++ is about 4.8× faster than the old GigaWorld-Policy on the 4090 (85 ms vs 293 ms), and about 38× faster than Motus on the A100 (85 ms vs 3231 ms). Even the PyTorch variant (110 ms) is an order of magnitude faster than the original GigaWorld policy. These gains primarily come from not decoding video: the mixture-of-transformers design effectively “prunes” most of the computation at inference. In human terms, 85 ms latency means the robot can replan actions at ~12 updates per second. This crosses a practical threshold for many closed-loop tasks. For example, if the robot’s gripper moves in a dynamic scene (the human moves an object at 1 m/s), an 85 ms delay means only ~8.5 cm of motion happens uncorrected, which is acceptable in many cases. The authors highlight that achieving this speedup is “crucial for robustness under dynamic disturbances”. Task Success. Crucially, all this speed does not come at the cost of performance. In fact, GigaWorld-Policy-0.5 matches or exceeds the success rates of previous methods across several tasks. Let’s consider a few representative results. On a suite of long-horizon manipulation tasks (like heating food or arranging tableware), the new model shows a clear advantage. In one table, we see:. Food Heating: π0.5 0.50 vs 0.80 for GigaWorld-Policy-0.5. Tableware Arrangement: π0.5 0.70 vs 0.80. (The averages were 0.60 for π0.5, 0.60 for Motus, 0.50 for FastWAM, 0.60 for the original GigaWorld model, and 0.80 for GigaWorld-Policy-0.5.) In other words, the new policy achieved about 80% success on these challenging tasks, while all comparators were 50–60%. For tasks with language instructions, a similar story emerges. On a fruit-picking task with instructions like “Pick the banana and place it into the basket,” GigaWorld-Policy-0.5 averaged 0.85 success, whereas the original GigaWorld policy averaged 0.80 and the best baseline (Motus at 0.80) was lower. The improvement is consistent: every single instruction-outcome pair is equal or better with the new model compared to the old. Likewise, in object-placement tasks (e.g. “Pick up the spoon and place it on the plate”), GigaWorld-Policy-0.5 achieved 0.89 average success versus 0.81 for the original. Perhaps the simplest summary is given on the project page: “GigaWorld-Policy achieves an average success rate close to 85%, about +30% absolute higher than other mainstream WAM models (~55%)”. The 0.5 version bump this from about 80% to 85%, a modest but meaningful jump. In short, GigaWorld-Policy-0.5 not only returns the benefits of future-video supervision (higher task success) but actually slightly improves upon the first GigaWorld-Policy. This is impressive given that the model now runs so much faster. Real-Robot Validation. What’s reassuring is that these results include tests on real robots with closed-loop control. The project page explicitly emphasizes “real-world robotic platforms” when reporting performance. One figure (not reproduced here) plotted success vs control frequency, showing GigaWorld-Policy-0.5 as dominating the curve: it’s the only method that achieves 10–12 Hz control and high success. Qualitatively, this means a robot arm was able to perform tasks like grasping, assembly, and rearrangement under GigaWorld-Policy-0.5 at real speed. Additionally, there are video demos (linked on the project site) for tasks like heating food and arranging tableware, showing the learned policy in action. These videos confirm that the robot adapts to small disturbances in the scene (for example, objects shifting slightly) thanks to the fast action loop. Discussion and Conclusions. GigaWorld-Policy-0.5 represents a significant step towards practical embodied AI with predictive models. The core lesson is simple: if you want to use world modeling in a real-time loop, you must decouple or compress it. By splitting the model into specialized experts and avoiding expensive video synthesis at runtime, the GigaWorld team has made an otherwise heavy approach run like a lightweight one. Of course, the gains come with some engineering complexity: the model now has two transformers and a more elaborate training regime. But the payoff is that, at deployment, it’s much simpler. The idea of “train complex, infer simple” resonates beyond this paper: other robotics researchers might adopt similar hybrid architectures (or even larger mixtures-of-experts) to get the best of both worlds. It’s worth placing GigaWorld-Policy-0.5 in the broader research landscape. See that it’s part of a rapidly evolving trend: scientists are discovering that purely reactive policies (even those using huge vision-language models) often lack physical reasoning, while fully generative world models are too slow. WAMs are the middle ground, and people have been exploring various trade-offs. Earlier systems like Cosmos Policy (not detailed here) or FastWAM tried simpler heuristics to speed up. But GigaWorld-Policy-0.5 is the first to hit true real-time territory while still being trained as a full video-action model. We also saw that Motus, a precursor on latent actions, also used multiple experts – GigaWorld’s approach can be seen as an evolution of that idea specifically optimized for low-latency inference. From an application standpoint, a model like GigaWorld-Policy-0.5 could be used as a drop-in policy for household or factory robots that need to act under vision-based goals. The fact that it runs locally on a single GPU (or even edge device, potentially) at 10–12 Hz is very attractive for safety-critical domains. Further, its reliance on action supervision means it can be fine-tuned on real-world data to adapt to a specific robot’s kinematics and sensors. One should note some caveats. The tasks tested are structured manipulation tasks, and it’s not clear how well vision predictions generalize to completely novel environments. Also, while 85 ms is fast, some high-speed tasks (e.g. catching objects, agile locomotion) might demand even lower latency or different sensor modalities (like event cameras). In those cases, one might have to further prune or quantize the model. Nevertheless, hitting sub-100 ms with rich visual context sets a new bar. In summary, GigaWorld-Policy-0.5 showcases a powerful design principle: offload heavy thinking to training, but streamline inference. By doing so, it achieves an impressive combination of speed and skill. The notion of splitting vision and action into separate, cooperating networks should inspire other embodied AI systems. And the empirical results — 10× speedup and a jump to ~85% success on real tasks — make a compelling case that “WAMs with wheels” are becoming viable. As future work, one might explore even more decomposed architectures (perhaps one expert per modality or task), or integrate learned modules with classical controllers for robustness. It would also be interesting to apply the same idea to other domains, like legged locomotion, where a fast model predictive controller can be vital. Finally, as GigaWorld demonstrates, scaling up data and using auto-tuning tools will only enhance these approaches. For now, GigaWorld-Policy-0.5 stands as a state-of-the-art example of efficient embodied foundation models. It shows that with careful architectural design and training, we can have our video cake and eat it too: train on rich visual futures, but deploy in milliseconds. References: The above discussion is based on the GigaWorld-Policy-0.5 technical report and project materials, which detail the model architecture, training pipeline, and evaluation results. These are further contextualized by prior work on world-action modeling. Each citation is linked to the relevant source for details.