Latest / Embodied AI 101 / Orca: The World is in Your Mind

Transcript

Orca: The World is in Your Mind · Embodied AI 101

• Orca: The World is in Your Mind. Modern AI has made enormous strides in specialized domains – large language models, video generators, and even robotic policies – but these advances still tend to live in silos. GPT-style models predict the next word; image generators predict the next pixel or video frame; task-oriented policies predict the next action. Each learns its own modality-specific patterns, but there is no shared “world understanding” across them. The Orca team from the Beijing Academy of AI proposes a radical shift: learn one world model that internalizes a latent state of the world and predicts how that state evolves. In other words, instead of separately learning “next token” in text, “next frame” in video, or “next action” in control, Orca is trained to predict the next world state. This unified objective – Next-State Prediction – ties together language, vision, and embodied action under one umbrella. The authors describe Orca as “an initial instantiation of a general world foundation model” that absorbs multimodal data and exposes its understanding through language, image, and action interfaces. The key idea is that a shared latent world state can serve as a common interface for any downstream task (text Q&A, image imagination, robot control, etc.), potentially leading to more consistent and versatile capabilities than separate models can achieve. From a high-level perspective, Orca’s motivation is clear. As the authors argue, intelligence should not be merely a next-token predictor (a pure language model) or a next-frame generator; rather, it should be defined by its ability to build internal world states and use those for diverse tasks. Language models and generative vision models can produce plausible continuations, but they often ignore the actual physics or causality of the world. For example, an LLM might happily describe an event that violates physical laws, or a video generator might break object permanence in a future frame. Similarly, a policy trained to mimic expert actions (“next action prediction”) may move limbs realistically but without modeling how the environment itself changes in response. Orca’s approach is to shift the training objective from “predict the next token/frame/action” to “predict the next state” of the world. In practice, Orca builds a latent world representation from incoming multimodal inputs (images, text descriptions of events, etc.) and trains the model to predict for a later time given the current state and possibly an instruction or event descriptor. Formally, if encodes the inputs (which could include the current image , an action, and any contextual query tokens), the model learns a state transition distribution. Here captures latent factors (unobserved dynamics) and encodes any explicit conditions or instructions. In this way, Orca’s latent world-state model generalizes the notion of prediction: it is not tied to a particular output format but to an abstract state with physics and causality built in. This latent state acts as the cognitive core for Orca. Once learned, it can be “read out” in multiple ways. A single world latent can be decoded into natural language, synthesized images, or sequences of actions. In other words, Orca’s backbone model maps raw multimodal inputs into , and then separate decoders translate into the specific outputs needed for each task. As the authors put it, “the same world latent can support a language readout for explanation and reasoning, a vision readout for prediction and imagination, and an action readout for intervention.” In effect, text, image, and action are just different “views” or interfaces on the same underlying state. This is a compelling vision: instead of having three disjoint models (an LLM, an image generator, a policy), we have one integrated model that understands what the world is doing, and can answer questions or take steps about it in any modality. In designing Orca, the team drew inspiration from how human infants learn. Infants initially absorb raw sensory experience without language – watching objects move, touching things, feeling forces – and only later do they learn to label events and reason about causes with language. Orca mimics this with two complementary training paradigms: “unconscious” and “conscious” learning. Unconscious learning is like passive observation. Orca is fed dense streams of unlabeled video so it can learn “physical regularities” and natural state transitions from raw pixels. It is trained to encode each frame (or representation thereof) into the latent state and then predict the latent of the next frame. This requires modeling motion, collisions, object permanence, and so on, all without any explicit supervision other than continuity. It is equivalent to self-supervised next-frame prediction, but done in latent space: Orca ingests a current image and a learned query token, and is trained to output the latent features for (extracted by a frozen vision encoder). By observing thousands of hours of “how the world moves”, Orca builds up a sense of basic physics and dynamics through this unconscious stage. Conscious learning then adds semantic structure. Once the model has seen raw dynamics, it is further trained with events and language. The team segments the videos into higher-level events (like “opening a fridge” or “a cup tipping over”) and attaches textual descriptions to these. Orca is trained to take the state at time , an event description , and predict the state at. In other words, it learns to incorporate a language-specified goal or causal premise into its predictions. It also learns from a large VQA dataset: given an image and a question about it, Orca must generate an answer. This VQA objective encourages the latent to capture facts about spatial relations, object properties, and physics that are relevant to answering questions. In sum, conscious learning ties the latent states to language and intentional actions – it “turns raw experience into causal patterns that can be communicated.” These two modes are trained together in pre-training: the model alternates or multitasks between unconstrained video prediction and event-conditioned prediction plus VQA generation. This hybrid training, the authors argue, mirrors how humans first learn by observation and then refine with language and tasks. The result is a world model grounded both in physical observation and in semantic reasoning. Orca’s architecture reflects this design. It uses an encoder–decoder framework. The encoder is a large transformer (for example a vision-language model similar to Qwen3.5) that takes in multimodal inputs: images or video frames, optionally action or other sensor readings, and learned “query” tokens. The encoder processes these to produce a latent world representation. Crucially, Orca does not feed raw pixels directly into a decoder – instead it trains against the latent outputs of a frozen vision model. Concretely, during pre-training the model takes a frame and a special learnable token ; the final hidden state of , after passing through two MLP layers, is trained to match the latent of the next frame obtained from a frozen Vision Transformer. Similarly, when an event description is provided, a second query token is fed in, and its hidden state is trained to predict the latent of the frame at. In short, features from hidden queries are trained to match future latent features under various conditions. This “learnable query” trick allows the model to perform state prediction in its latent space, guided by visual inputs and textual context. A separate loss is used for VQA: given an image and a text question, the encoder must also generate (via a language modeling head) the correct answer text. The combined pre-training loss is a weighted sum of these three components (observation-only transition, event-conditioned transition, and VQA answer prediction). To train such a model, Orca needs a massive, multi-modal dataset. The team constructed what they call a world-learning inventory: over 125,000 hours of real-world video, accompanied by 160 million event annotations (text labels describing state transitions), and 11.5 million VQA question-answer pairs. The video clips come from varied sources to cover diverse dynamics: first-person interactions (“ego-centric”), third-person manipulation of objects, action-free robot demonstrations, and general natural scene dynamics. Each video was segmented into temporal events – major steps and finer-grained sub-steps – and annotated with natural language captions that describe the transition (for example, “the person picks up the bottle and closes the cap”). These event-labeled clips train the “conscious” objective. The full video and annotation set thus teaches Orca a wide array of state changes (lifting, falling, mixing, etc.). The 11.5M VQA examples pair images from those videos with questions like “what will he do next?” or “where is the cup?” to teach language-conditioned reasoning. Altogether, the raw data is enormous, far beyond the scale of typical datasets. The authors note that this is only a fraction of the data they gathered – on the current model they actually use about one-tenth of their video inventory – with remaining data held for future scaling. All this is used to pre-train the Orca encoder. After pre-training, the backbone is frozen and detached from the pre-training tasks; it now contains a learned world latent space. The next step is to test whether that latent really captures useful knowledge. To do that, Orca is connected to various readout modules for downstream tasks. These readouts are modality-specific decoders that are trained after freezing the backbone. For language tasks, they simply re-use the model’s language modeling head (or fine-tune a small adapter) to generate text outputs from the latent. For vision tasks, they attach a generative image decoder. In practice Orca uses a stable-diffusion–style setup: they add a small MLP adapter and LoRA layers on top of a frozen Stable Diffusion 3.5 model, which is used to generate images from the latent. For actions, they attach an “Action Expert” – essentially a DiT-based (Diffusion in Transformer) trajectory generator – with its own adapter layers. Each readout is trained on a supervised or reinforcement learning objective using the frozen latent as input. In this way, the same core model can be evaluated on tasks as diverse as image prediction, language QA, and robot control by simply swapping its decoder head. A key question is whether this huge world-model approach pays off. The authors carefully evaluate three aspects: scaling behavior, and performance on prototypical downstream tasks in language, vision, and action. First, they check scalability: as they increase model size and data, does the world-model objective continue to improve? Indeed, Orca’s training loss keeps going down with more data and larger backbone, indicating that the representation is absorbing the information in the videos. In other words, next-state prediction is not saturating – it remains a fruitful training signal as scale grows. They also find a pleasing result that pre-training benefits carry over directly to tasks. Because the backbone is frozen, any gains in downstream performance come purely from having a better latent. They report that “as pre-training data scale up, text, image, and action readouts consistently improve”. The conclusion is that “stronger world latent from pre-training leads to stronger downstream readouts”. In practical terms, larger Orca models or more pre-training yield measurably better results on every task. This confirms that the paradigm is scalable: more data and model capacity help both world modeling and the later tasks. Next, they test language generation. To probe abstract reasoning about the world, they use several visual question-answering (VQA) benchmarks that focus on physical understanding. These include MVBench, TemporalBench, 3DSRBench, and a “SWITCH” category – sequences about objects switching places. These suites require commonsense reasoning about state transitions, 3D spatial relations, and temporal order. The baselines include specialized “world model” language systems (V-JEPA 2.1) as well as recent vision-language models like Qwen-3.5 and Google’s Gemma 4. In this testbed Orca shines. For example, a 0.8B-parameter Orca outperforms a 0.8B Qwen-3.5 on all benchmarks (overall score 40.8 vs 33.1), despite using the same amount of compute. At 4.0B parameters, Orca’s improvement is even more striking: the 4B Orca scores 51.8 overall, compared to 46.7 for a 4B Qwen-3.5 and just 40.8 for Gemma 4 (also 4B). In plain terms, Orca’s text generations about physics are markedly more accurate. The gains come especially in reasoning categories that closely match the training signal. The authors report detailed success rates by category. For instance, on state transition questions (e.g. “what will happen next in the scene?”), Orca achieves a 64.13% success rate versus 51.86% for Qwen-3.5 – a +12.3 percentage point jump. On dynamic motion questions (“what will the moving object do?”), Orca scores 65.55% vs Qwen’s 57.03% (+8.5 points). The gaps on commonsense reasoning and spatial relations are smaller but still in Orca’s favor. These improvements make sense: by training to predict how scenes evolve, Orca directly learned the patterns needed for answering “and then what?” type questions. In qualitative analysis, the authors note that Orca is much better at ensuring physical consistency: it won’t say objects teleport or walls vanish, because it has internalized that such world states don’t hold. Overall, Orca “possesses core advantages in temporal reasoning and spatial perception” and is more parameter-efficient. This suggests the world latent is a richer representation for visual language understanding than what generic image-text models have learned. For vision-based evaluation, the Orca team constructs a novel physical prediction benchmark called PRICE (“Prediction of Real-world Interactions with Constraints Evaluation”). In PRICE, the model is given a real scene image and a textual instruction (e.g. a robot arm should push the block), and must generate a future image showing the result of the interaction. This is stricter than unconstrained image generation because the prediction must obey physical constraints and the instruction. Orca is compared against models like OmniGen2, DeepMind’s Gemini 3.1, Doubao (an action-conditioned image model), and Google’s Gemma-4 as a “judge” for scoring. Here again the results favor Orca. A key table shows that the 4B-parameter Orca (with a 0.8B text backbone and 2B vision network, denoted “4+2” in their tables) achieves the highest average score on PRICE – 59.8 – outperforming all other models. By contrast, the next best is FLUX.2 with 56.1. Even Orca’s smaller 0.8+2B configuration (34.5 avg) beats Gemini (39.6 avg) and FLUX1 (40.9). In other words, for predicting how a physical scene will look later, Orca knows better “what will happen” than the alternative vision-language systems. The authors illustrate examples where baseline models violate scene constraints: for instance, an ordinary image generator might ignore the robot arm’s limits or generate floating objects. In contrast, Orca’s predictions stay grounded. Its decoders draw on the world latent, so they “remember” that the robot has limited reach, that gravity acts downward, or that objects don’t pass through each other. Visual comparisons in the paper show that Orca preserves the presence and location of key objects under an interaction where others fail. This demonstrates that learning a stateful model of the world can improve even pure visuospatial generation tasks – the model is effectively doing physics simulation in latent form, then decoding it. Finally, Orca is tested on embodied action tasks. Here the goal is to have the model generate or guide robot control sequences. The authors collect a small real-world dataset using a dual-arm wheeled robot (called GalBot) performing five tasks (e.g. “take book”, “stack bowls”, “pull tissue”, “stamp”, “scoop sugar”). They then design out-of-domain generalization tests: changing the environment or objects in ways not seen during training. The question: given a task instruction, can the model produce a successful action plan even in a new situation?. For evaluation, they use a combination of rule-based scoring (Did the goal get accomplished?) and a trajectory metric (PRM-as-a-Judge, which measures things like progress and recovery during the sequence). All methods have their base model frozen; only small “Action Expert” adapters are trainable. Orca is compared to V-JEPA 2.1 (a prior visual world model approach) and Qwen-3.5, as well as a prior state-of-the-art robotic method called π0.5. The results are striking: on environment generalization (the hardest case), Orca’s models succeed far more often than the baselines. For example, the rule-based success rate under environment OOD is 36.6% for Orca, compared to just 15.2% for V-JEPA and 12.4% for Qwen-3.5. In fact, Orca even beats π0.5 (27.6%) by a comfortable margin in this metric. Even when the objects themselves change (object OOD), Orca holds competitive with π0.5 (28.2% vs 33.1%), far above the other baselines. Looking at the overall aggregated score, Orca achieves 32.4% success across all tests – nearly double V-JEPA’s 17.0%. Beyond raw success rate, the PRM trajectory diagnostic (which tracks progress, failures, and recoveries) is consistently better for Orca. In short, the learned world latent appears to give Orca a much more robust policy for planning motions. The authors emphasize that no additional task-specific tricks were used; the lift in performance comes purely from the representation. The conclusion is that a stronger world model yields stronger action policies too – even in unpredictable, real-world setups. They note that Orca’s gains are principally in resilience and adaptability. Its predictions seem to recover from disturbances better (e.g. if a planned step fails, the latent state guides it back on track), resulting in more complete task executions than the LLM-based policy alone. Together, these results paint a compelling picture: a single Orca model, pre-trained only with its world-state objective, can feed separate readouts for words, images, and actions, and it outperforms specialized models in each domain. The text experiments showed better reasoning about physics, the vision tests showed more accurate future images, and the robotic tasks showed much higher success. All of this at the same or smaller model sizes used by the baselines. The authors compare Orca against state-of-the-art contemporaries in each setting: for language, models like V-JEPA, Emu3/Emu3.5, Qwen3.5, Gemma4, SmolVLM2, DeepSeek-VL2; for vision, systems like OmniGen2, FLUX.1/2, and GPT-5/Gemini judges; for control, methods like π0.5, V-JEPA, and Qwen policy. In nearly every case, Orca holds its own or wins. This is especially notable given that those baseline models were often fully specialized for one modality or task. Orca shows that by training every modality task through the lens of a world state, one obtains a surprisingly versatile representation. It’s worth contrasting Orca with prior work. DeepMind’s Gato (2022) pioneered the idea of one big model doing many tasks (text, images, joystick actions) by treating everything as token sequences. But Gato still learns task behaviors directly rather than forming an explicit world state that underpins them. Traditional “world models” in reinforcement learning (e.g. Dreamer, 2019) predict future latent states for control, but usually only from pixels in a simulated game and without any language. Orca can be seen as combining these threads: it’s a world model at vast scale, adding language and real-world data into the mix. In generative vision, latent predictive models (like V-JEPA) learn image features, but again without tying them to a shared state space that also serves text and action. Orca is, to our knowledge, the first attempt to truly train one foundation model – meaning a colossal pre-trained model – whose core objective is the evolution of a latent world. Of course, Orca is a first step, and the authors are frank about its limitations. Currently it only uses vision and text inputs, so important modalities like audio and touch are ignored. For example, things like the sound of boiling water, or the force needed to lift a heavy object, are outside its purview. The team notes that many state changes in the world are actually expressed through non-visual signals (sound, tactile, force) or through other scientific inputs (thermodynamics, chemical states, etc.). Future work will need to incorporate these richer streams (microphones, force sensors, etc.) to form a truly complete world latent. Another caveat is that Orca’s prediction target is based on a frozen Vision Transformer space. In other words, it learns to predict another mega-model’s features. This made training simpler, but it likely limits flexibility: the latent may be tied to the quirks of that vision model. An ideal world model might learn its own end-to-end state representation not bound to a frozen encoder. In practical terms, Orca challenges us to rethink how we build embodied AI. Instead of stitching together separately trained modules, it suggests training a general-purpose encoder on what actually happens in the world. If one wanted to extend this approach, the key ingredients seem to be: (1) amassing very large, multimodal experience data (videos, event scripts, questions, etc.); (2) a model architecture that can ingest that experience and learn a latent; and (3) downstream heads for any needed task. For roboticists, this is like saying “build a simulator in the model’s mind”: it should have notions of objects, physics, and cause-effect. Then planning and language tasks become just different ways of querying that simulator. The Orca results imply that this can really pay off: rather than each robot task requiring separate training, they all benefit from a common world embedding. One might imagine fine-tuning the Orca latent on new robotics domains with very few samples, or using its state predictions to drive conventional planners. Conversely, the inclusion of event-based and VQA supervision hints that the world model may also be better at open-ended reasoning (e.g. common-sense QA) than standard LLMs. What use is Orca for us right now? Even without full code or final weights (the paper is a preprint), the ideas are inspiring. We see a few takeaways: (a) Next-state prediction is a versatile training signal. If you have multi-step tasks in vision or control, consider training a model to predict future observations or internal states, not just immediate outputs. (b) Large-scale, multi-domain data helps. If you’re developing a robot or simulation model, try to collect broad data (perhaps game captures, internet videos, narrated events) and train in a self-supervised way on transitions. (c) Multimodal decoupling. You can freeze a backbone and just train light heads for each modality – meaning your expensive world model only has to do one big pre-training run, not re-train for every new problem. (d) Investigate unified architectures. While Orca used a vision-language transformer plus diffusion and adapter heads, the principle could apply across architectures (eg. graph networks for physics or NeRF worlds). In conclusion, Orca points toward a bold vision of embodied AI: one model that “lives” in a coherent, abstract world space. It has demonstrated that such a model can indeed generate better story descriptions, more accurate imagined images, and more robust actions than specialized models of the same size. Those results are impressive, but more importantly the work opens a new research direction. It leaves crucial challenges to handle more sensory modalities, richer dynamics, and long-horizon planning. But it already suggests that the key to consistency across language, vision, and action may lie in an internal world canvas. The Orca team’s work offers a compelling blueprint: train a foundation model on the world itself (via next-state prediction), and the world, in a sense, will be in the AI’s mind. References: The above summary is based on the Orca paper by Wang et al. (2026) and the accompanying materials, as well as publicly available project documentation. These sources describe Orca’s redefinition of the learning objective to next-state prediction, its large-scale multimodal training data, and the evaluation showing significant gains in text, image, and action tasks.