Latest / Embodied AI 101 / GigaBrain-0.7: Scaling Robot Intelligence by Separating Acting, Thinking, and Imagining
Transcript
- GigaBrain-0.7: Scaling Robot Intelligence by Separating Acting, Thinking, and Imagining. GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture • GigaBrain Team, Angen Ye et al. • GigaAI • arXiv:2608.15875 [cs.RO] • 2026. Release-date note: The supplied date of August 30, 2026 does not match the primary records. The first arXiv version was submitted on August 16, 2026. The repository records the technical report on August 17, the code, base model, and sample data on August 25, and additional benchmark workflows on September 2. The Core Idea. The headline claim behind GigaBrain-0.7 is that scaling an embodied foundation model requires more than a larger observation-to-action network. A robot operating over long horizons has at least three distinct problems to solve. It must interpret the scene and determine the current subtask. It must anticipate what the world may look like after making progress and judge whether execution is going well. And it must convert all of that context into high-frequency, embodiment-specific motor commands. GigaBrain-0.7 assigns these responsibilities to three systems:. System 2 handles visual-language understanding and task decomposition. System 3 predicts future visual states and estimates task progress. System 1 produces executable action chunks. The ordering is slightly counterintuitive because the numbering evokes the fast-versus-slow terminology of cognitive science. Operationally, however, Systems 2 and 3 provide semantic and predictive context to System 1, which remains the deployment policy. This architecture sits on top of an unusually broad training mixture: 37,256.98 hours of embodied trajectories from 16 robot types, plus roughly 272 million vision-language samples. Systems 1 and 2 are jointly pretrained on vision-language and action objectives; System 3 is trained separately as a world-value model; all three are connected during task-specific post-training; and the resulting policy can then be improved through offline and online reinforcement learning. The important point is that GigaBrain-0.7 is not just proposing another action decoder. It is proposing an embodied-learning stack spanning data alignment, semantic understanding, temporal memory, predictive modeling, multi-embodiment control, and deployment-time experience. Why a Reactive VLA Is Not Enough. Most modern vision-language-action models begin with a pretrained vision-language model and add some mechanism for generating robot actions. In autoregressive systems such as RT-2 and OpenVLA, actions are represented as tokens and decoded through the language-model interface. Cascaded architectures instead use a VLM to produce features that are consumed by a separate action decoder. The increasingly common Mixture-of-Transformers approach couples a VLM to a continuous Action Expert more deeply, often using flow matching to generate an entire action chunk in parallel. GigaBrain-0.7 belongs primarily to that third family. Its System 1 is close in spirit to the architecture popularized by the π-series: a semantic VLM stream operates alongside a continuous motor stream, allowing the robot to retain language and visual knowledge without forcing a language decoder to emit every low-level control dimension sequentially. The GigaBrain authors argue that even this is incomplete. A conventional VLA may recognize a bowl, understand “put the spoon in the bowl,” and output a plausible motion. But long-horizon manipulation introduces partially observed state, repeated visual configurations, recovery states, changing contact geometry, and delayed consequences. Two images that look nearly identical may require different actions because one was reached before a failed grasp and the other after a successful reorientation. An action may appear locally reasonable while moving the overall task backward. GigaBrain-0.7 therefore treats the policy as one component of a larger system. Semantic decomposition, temporal state estimation, future prediction, value estimation, and action generation remain coupled, but they are not forced into one undifferentiated next-action model. Inside the Three-System Architecture. System 2: Understanding and Planning. System 2 is described as a 3-billion-parameter PaliGemma2 vision-language model. It receives the task instruction and visual observations, reasons about relevant objects and their spatial configuration, and produces an immediate subtask. For a broad instruction such as hanging an item, System 2 may first identify the available hats and bags, reason about which object the instruction refers to, and then emit an actionable intermediate goal such as picking up the red hat. That subtask becomes a structured interface shared by the other systems: System 1 uses it to generate motor commands, while System 3 uses it to condition future prediction. The model is supervised with both task-level and subtask-level language. During some training examples, the current subtask is supplied directly. In others, only the high-level instruction is provided, forcing System 2 to infer the appropriate stage before System 1 predicts an action. Causal masking is used to prevent future subtask or action targets from leaking into the current prediction. This is an important distinction from simply prepending chain-of-thought text to an action policy. System 2 is meant to serve as a semantic bottleneck that tells the downstream components what phase of the task they are in. It turns a long-horizon instruction into something closer to an executable local objective. The report nevertheless leaves some practical questions unanswered. It does not clearly characterize how frequently System 2 replans, what triggers subtask transitions, how planning failures are detected, or how much of the reported gain can be attributed specifically to its generated reasoning. There is no broad System-2-only ablation comparable to the experiments performed for System 3. System 3: Prediction and Evaluation. System 3 is the most distinctive component. It is a 5-billion-parameter world-value model built from GigaWorld-1 and trained to provide two forms of prospective context. The first is a predicted future observation. Given the current observation, robot state, and System 2’s subtask, System 3 generates a short future video. Its final frame is extracted as a subgoal image. System 1 then receives both the current scene and this predicted future state, effectively learning a policy of the form: “Given where I am and what successful near-term progress should look like, produce the motion that connects the two.”. The second signal is a task-progress value. System 3 estimates how far the current state has advanced toward completing the active subtask. During post-training, progress estimates at two separated times are compared. If value has increased, the corresponding action segment receives a positive binary advantage label; otherwise it receives a non-positive label. This label is inserted into System 1’s prompt as another conditioning token. At inference time, the policy is asked to generate behavior conditioned on positive progress. This resembles advantage-conditioned imitation: the model sees both productive and unproductive segments during training, then is prompted to behave like the productive subset during deployment. It is important not to overstate what this means. System 3 is not described as a full model-predictive controller that samples many candidate action sequences, simulates each one, ranks their consequences, and executes the winner. The paper explicitly says that deployment does not perform candidate-action scoring or selection. Instead, the world model produces a subgoal and progress-related conditioning that guide a learned action generator. It is better understood as amortized predictive guidance than online search. That choice has practical advantages. It avoids putting an expensive planning loop around every action chunk. But it also means that the policy can inherit errors from the world model without an explicit mechanism for rejecting an implausible imagined future. To reduce this dependency, the authors independently drop the subgoal image in 50 percent of post-training samples and the value condition in 15 percent. The resulting policy is trained under all four combinations: neither signal, image only, value only, and both. System 1: Action and Control. System 1 is the executable VLA. Its core consists of a 3-billion-parameter PaliGemma2 backbone and a 0.5-billion-parameter Action Expert arranged as a dual-stream Mixture of Transformers. The interaction is asymmetric. The vision-language stream preserves causal self-attention over its semantic tokens. The Action Expert can attend to both its own hidden states and the VLM states, allowing motor generation to read the complete semantic context. The two streams have separate feed-forward parameters. This structure lets the Action Expert specialize in continuous control without feeding its action-specific hidden representation directly back into the VLM attention computation. It is a compromise between a fully shared network and a late-stage action head. System 1 supports both discrete and continuous action pathways. Discrete action representations are generated autoregressively through next-token prediction and act as a semantic bridge between reasoning and control during pretraining. Executable actions come from the continuous pathway: the Action Expert uses conditional flow matching to transform noise into a chunk of robot commands. Both pathways are trained from the beginning rather than introducing continuous control only during downstream specialization. The model also introduces Soft Knowledge Insulation. Continuous-action gradients flow normally through the Action Expert but are attenuated before entering the VLM. The aim is to let physical-control learning modify visual representations without overwhelming the language, recognition, grounding, and spatial-reasoning capabilities inherited from pretraining. Conceptually, this is a useful middle ground. Completely blocking action gradients may preserve semantics but prevent the vision stack from becoming sensitive to contact and control-relevant geometry. Allowing full gradients risks catastrophic specialization to the robot data distribution. Soft insulation supplies a tunable gradient valve. The current report, however, does not disclose the selected attenuation coefficient or provide a dedicated ablation over that coefficient. Memory Without Exploding the Token Count. System 1 does not operate on an isolated image. Temporal-Spatial Blocks are inserted into its visual encoder to combine the current frame with a short observation history. These blocks first aggregate visual features causally through time and then model spatial relationships within the current observation. Historical tokens are discarded after their information has been fused into the current-frame representation. Consequently, the high-level VLA receives approximately a single frame’s worth of visual tokens rather than a concatenation whose length grows with every history image. This is a pragmatic temporal-memory design. It provides clues about motion, prior contact, and task stage while bounding the expensive transformer context. Historical observations are randomly dropped during training so that the policy can operate with different history lengths and does not fail outright when only the current image is available. One Motor Core Across Many Bodies. Cross-embodiment learning is handled through a shared semantic action structure, a Robot ID, validity masks, and embodiment-specific projections. Missing joints are not represented as ordinary zeros. A validity mask tells the network whether a dimension is truly present, preventing an absent joint from being confused with a real joint whose current value happens to be zero. The central Action Expert is shared, while input and output projections adapt native state and action definitions to the common representation. Rotations use a continuous six-dimensional representation. This is not morphology-invariant control in the strong sense. An entirely new robot still needs a correctly designed projection, coordinate convention, normalization, action mask, and probably adaptation data. But it offers a scalable mechanism for sharing manipulation knowledge among known embodiment families without forcing every platform into an identical native control space. The Data Pyramid Is as Important as the Architecture. The embodied corpus contains 37,256.98 hours after cleaning. Real robots contribute 20,535.65 hours, or 55.12 percent. UMI demonstrations contribute 8,251.83 hours; egocentric human data, 2,862.36 hours; physics simulation, 1,453.92 hours; and world-model-generated data, 4,153.22 hours. Alongside this, the VLM is trained on 271,976,674 image-text or question-answering examples covering captioning, grounding, spatial relations, point prediction, affordances, and robotic task understanding. The real-robot portion contains roughly 1.81 million episodes and 2.08 billion frames across 16 robot types. The distribution is far from uniform: Maker H01 accounts for 43.82 percent of real-robot frames, Maker M01 for 20.14 percent, and Agibot-G1 for 18.30 percent. In other words, more than four-fifths of the real-robot frames come from three platforms. This provides genuine cross-embodiment diversity, but results on the heavily represented Maker family should not be interpreted as transfer from a balanced robot distribution. GigaBrain’s data work is unusually concrete. Raw trajectories are converted to LeRobot v3.0, cameras and metadata are standardized, and states and actions are mapped into the shared representation. GLM-5.1 rewrites inconsistent or multilingual instructions into canonical task descriptions, after which humans spot-check the output. A multimodal model segments long trajectories and produces atomic subtasks. Cleaning includes per-robot quantile filtering, removal of long stationary segments, and correction of inconsistent end-effector frames using robot descriptions and forward kinematics. Early training losses are also treated as a data diagnostic: samples producing persistent spikes are traced back to corrupted videos, discontinuous actions, temporal misalignment, or incorrect language. UMI trajectories are synchronized, filtered, retargeted to the Maker H01 tool-center-point frame, checked through inverse kinematics and simulation, and sampled for real-robot replay. Egocentric human actions are re-expressed relative to the current observation frame. Additional embodied VLM annotations are generated with Qwen3.6-27B and filtered using geometric, temporal, and answer-consistency checks. The broader lesson is that GigaBrain-0.7 treats alignment as a prerequisite for scale. Thirty-seven thousand hours of mutually inconsistent action spaces would not form a foundation model; they would form a large source of gradient interference. Training: “One Stage” Within a Four-Stage Lifecycle. The phrase “one-stage alignment training” needs careful interpretation. Systems 1 and 2 do jointly learn vision-language outputs, discrete actions, subtasks, and continuous actions during the same base VLA pretraining run. Unlike recipes that first adapt a VLM and only later introduce a continuous action expert, the flow-matching pathway is present from the beginning. The complete GigaBrain-0.7 system is nevertheless trained in four broader phases:. Systems 1 and 2 are jointly pretrained on the heterogeneous VLA mixture. System 3 undergoes robot-centric video training followed by value learning. The frozen System 3 conditions Systems 1 and 2 during task-specific post-training. The deployed policy is refined using offline and online experience. Thus, “one-stage” describes the core VLA objective, not the entire architecture. During base pretraining, next-token loss covers language, subtasks, discrete actions, captioning, VQA, grounding, spatial reasoning, and affordance tasks. A flow-matching loss simultaneously trains the continuous action chunks. Soft Knowledge Insulation reduces only the flow-matching gradient entering the VLM; the language-modeling gradient remains intact. System 3 begins from GigaWorld-1 and is adapted to robot manipulation video. A second training stage adds task-progress estimation from trajectory-level completion annotations. Afterward, System 3 is frozen. It generates fresh subgoal images and values during policy post-training but is not updated by the downstream policy loss. For deployment-driven improvement, the team first gathers successful, partial, failed, and human-corrected rollouts. Offline refinement uses an advantage-weighted-regression-style objective, emphasizing segments associated with higher return and progress. The policy is then redeployed, and humans intervene at difficult states. These corrections enter an online actor-critic loop whose rewards combine progress differences with terminal outcomes. This is sensible as a data-efficiency strategy: use offline failures before paying for more on-robot exploration, then request human corrections specifically where the improved policy still struggles. However, the authors defer the complete intervention protocol and online-RL formulation to a future report, so this portion is less reproducible than the supervised stack. What the Experiments Show. Architecture Selection. The authors compare PaliGemma2, Qwen3.5, and Gemma 4 backbones on clean-desk, fruit-picking, and shirt-folding tasks. Larger models are not consistently better. PaliGemma2 obtains 50, 88, and 30 percent success; Qwen3.5 gets 60, 12, and zero; and Gemma 4 gets 100 percent in a restricted clean-desk setting, 92 percent on fruit picking, and zero on folding. This is not a controlled parameter-scaling experiment—the models differ in size and image setup—but it motivates the choice of PaliGemma2 because it is the only candidate with nonzero shirt-folding success. The coupling experiment is more revealing. With PaliGemma2 fixed, the dual-stream design obtains 50 percent on clean desk, 88 percent on fruit picking, and 30 percent on shirt folding. A final-layer cross-attention expert falls to 20, 40, and zero percent, while multi-layer cross-attention reaches 20, 36, and zero. The price is latency. Reported System-1 inference time is 0.221 seconds for dual stream, compared with 0.073 seconds for final-layer cross-attention and 0.108 seconds for multi-layer cross-attention. Action chunking can amortize that cost over multiple control steps, but the report does not provide end-to-end latency for the complete stack including System 2 and world-video generation in System 3. Scaling and Temporal Context. The data-scaling curves show lower validation loss as the pretraining corpus grows. Fruit manipulation improves relatively smoothly, while chaotic clothes folding changes more sharply with scale. Adding UMI or egocentric data to 3,000 hours of robot data improves performance, and combining both human-data sources produces the strongest results, including after task-specific post-training. This supports the claim that heterogeneous physical experience can improve the pretrained prior. It does not yet constitute a formal embodied scaling law: no exponent is fitted, and the paper provides neither a sharply identified emergence threshold nor a complete compute-matched analysis. Temporal context is evaluated qualitatively. A single-frame policy falls into a repeated-action cycle when it revisits a visually similar state. The memory-equipped policy uses recent history to distinguish the new task phase and exits the loop. The example is persuasive as a failure mode, but aggregate quantitative memory ablations would be needed to estimate its general contribution. Does System 3 Help?. The strongest targeted ablation compares a base VLA with subgoal-image conditioning, value conditioning, and both. On clothes folding, every variant reaches 100 percent binary success, but the combined model raises average task score from 68.3 to 88.3 percent and reduces completion time from 107 to 75 seconds. On gift wrapping, success rises from zero for the base model to 20 percent with the subgoal, 60 percent with value, and 80 percent with both. Task score rises from 61.1 to 96.7 percent. On cube sorting, the combined model improves success from 40 to 55 percent and average task score from 55 to 87.5 percent. These results justify treating visual futures and values as complementary interfaces. The image describes the desired physical configuration in a high-bandwidth format, while the value condition distinguishes progress-producing behavior from stagnation or regression. The evidence is still limited to three tasks, and the effects are not uniformly additive, but it is the clearest experiment linking the proposed System 3 directly to behavior. Foundation Behavior and Post-Training. The paper shows out-of-the-box execution on AgileX PiPER and Maker H01, including language-conditioned rearrangement, previously unseen object-and-container combinations, and folding unseen garments from unstructured initial states. These examples test more than appearance variation because cloth geometry and contact conditions evolve continuously. However, this foundation-model section is mainly qualitative. It does not provide a comprehensive table of zero-shot success rates across all demonstrated tasks. Moreover, some objects are described as absent from the corresponding platform’s training data rather than necessarily absent from the full multi-embodiment corpus. The examples may therefore reflect a combination of semantic generalization, compositional reuse, and cross-embodiment transfer. After task-specific post-training, the quantitative results are stronger. On six language-following tasks, GigaBrain-0.7 averages 91.5 percent on PiPER, compared with 88.8 for π0.5, and 84.2 percent on Maker H01, compared with 75.2 for π0.5 and 69.6 for GigaBrain-0.1. On complex manipulation, GigaBrain-0.7 averages 84.9 percent on PiPER versus 76.6 for π0.5. The difference is much larger on H01: 74.1 versus 45.2 percent. Simulation and Embodied Understanding. On the held-out version of MiMo-Embodied evaluated before MiMo data entered training, GigaBrain-0.7 achieves an overall score of 0.4621. G0.5-base reaches 0.3916, and the other listed models are lower. After continued pretraining on MiMo data, GigaBrain reaches 0.5704, but the authors correctly label this as a training diagnostic rather than a held-out result. The publicly released checkpoint corresponds to that MiMo-enhanced stage. On RoboTwin 2.0, one policy is trained across all 50 tasks with 50 clean demonstrations per task. GigaBrain-0.7 obtains 66.8 percent in the clean Easy setting and 67.9 percent under domain randomization, for a 67.35 overall score. π0.5 scores higher on Easy at 70.7 percent but drops to 46 percent on Hard. The unusually strong Hard result is the most notable benchmark outcome and suggests that the pretraining mixture or temporal-policy structure provides considerable robustness to visual and environmental variation. On EBench, GigaBrain records a 33.30 percent success rate and a progress score of 46.1, versus 28.08 and 42 for π0.5 among the publicly available models selected for the main comparison. On RoboColiseum, it leads the listed open models in instruction following, spatial reasoning, robustness, and general manipulation, with scores of 0.8166, 0.4729, 0.6800, and 0.6092. The paper preserves leaderboard snapshots dated August 15, 2026 because those online rankings can change. Learning From Deployment. Finally, four real-robot tasks test the experience-reinforcement pipeline. Average success rises from 30 percent after supervised fine-tuning to 57.5 percent after offline RL and 100 percent after online RL with human correction. Link installation improves from 20 to 40 to 100 percent. Gift-box packing moves from 80 to 90 to 100. Cable-tie insertion goes from zero to 40 to 100, and bearing installation from 20 to 60 to 100. The stage-wise trend is encouraging, especially for precision-critical insertion and threading. But four tasks ending at 100 percent should be interpreted with caution until the full RL algorithm, intervention budget, number of trials, confidence intervals, and repeated-run variance are reported. For now, these experiments demonstrate that targeted correction can close residual task-specific gaps—not that the system has solved autonomous lifelong learning. Where GigaBrain-0.7 Sits in the Field. Most of its individual ingredients have nearby precedents. The π-series established deeply coupled VLM–Action Experts and flow-matching control; more recent versions added observation history, hierarchical context, and visual subgoals. GigaBrain-0 introduced world models as data generators and embodied reasoning supervision. GigaBrain-0.5M* introduced RAMP, where predicted futures and values condition policy learning. Advantage-conditioned learning from demonstrations, rollouts, and corrections also appears in systems such as RECAP. GigaBrain-0.7’s novelty is therefore mainly architectural and organizational. It consolidates these ideas into explicit functional systems, scales the underlying data mixture, adds a practical cross-embodiment interface, and connects pretraining to deployment-time improvement. That distinction matters. This is not a paper about one clever module producing a clean ten-point gain under controlled conditions. It is a foundation-model report in which data, architecture, training, hardware coverage, and post-training all change together. Its strongest numbers validate the resulting stack, but only partially isolate why the stack works. What Roboticists Can Take From It. The most reusable lessons do not require reproducing the entire 37,000-hour program. First, normalize semantics before scaling volume. Camera conventions, coordinate frames, end-effector definitions, action dimensions, instruction granularity, and stationary segments can dominate learning quality. GigaBrain’s cleaning pipeline may be more transferable than any single neural component. Second, preserve a dedicated path for continuous control while retaining semantic supervision. Joint next-token and flow-matching training gives the VLM and Action Expert a shared context without making low-level control depend on slow autoregressive decoding. Third, compress temporal history before the expensive multimodal backbone. GigaBrain’s temporal-spatial blocks offer one way to retain recent interaction state without multiplying the VLM token count. Fourth, treat a world model as a source of policy conditions before attempting full online planning. A predicted subgoal and a progress signal are much cheaper to integrate than exhaustive candidate search. Condition dropout is essential because the predictive model will sometimes be wrong. Finally, stage experience learning. Collect policy failures, use them offline, redeploy, and reserve human corrections for states that remain difficult after the first refinement. As of September 5, 2026, the public repository includes the 3.5B base checkpoint, RoboTwin and EBench fine-tuned checkpoints, sample data, PiPER and H01 configurations, deployment scripts, and benchmark workflows under Apache 2.0. New embodiments require matching LeRobot data, normalization statistics, camera mappings, state and action widths, a validity mask, and an embodiment ID. The listed release contains sample data rather than the complete 37,000-hour corpus. Practitioners should also follow the repository’s robot-specific inference scripts rather than the generic Hugging Face “Diffusers” example shown on the model page, which is a text-to-image template and does not represent VLA deployment. Final Assessment. GigaBrain-0.7’s central proposition is persuasive: an embodied foundation model should be designed as a learning system, not merely as a large reactive policy. Semantic planning, predictive state modeling, progress evaluation, continuous control, and deployment experience operate at different levels and benefit from explicit interfaces. The paper’s best evidence is the System 3 ablation, the difficult Maker H01 post-training results, and the robustness shown on RoboTwin’s randomized setting. Its weaker points are incomplete zero-shot quantification, limited component ablations, concentrated in-house data, missing end-to-end runtime analysis, and an online-RL formulation deferred to future work. So the right reading is not that GigaBrain-0.7 has conclusively discovered a human-like three-part robot mind. It has demonstrated that, at embodied-foundation-model scale, structured separation may be more effective than architectural monolithism. System 2 says what to do, System 3 imagines what progress looks like, and System 1 executes. The larger implication is that the next scaling frontier may be the coordination of specialized models and learning loops—not simply adding parameters to the action policy.