Latest / Embodied AI 101 / π0.7: When a Robot Foundation Model Learns How to Behave

Transcript

π0.7: When a Robot Foundation Model Learns How to Behave · Embodied AI 101

• π₀.₇: When a Robot Foundation Model Learns How to Behave. π₀.₇: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities • Physical Intelligence and 87 coauthors • Physical Intelligence • arXiv preprint, v1 April 16 and v2 April 24 • 2026. The headline is not the espresso. Physical Intelligence’s π₀.₇ can make espresso, fold laundry, replace a trash bag, peel vegetables, assemble boxes, and execute a surprisingly broad set of household behaviors without task-specific post-training. It can follow multi-step instructions in unfamiliar kitchens and bedrooms. With generated visual subgoals, it can transfer shirt folding to a pair of UR5e arms that never supplied laundry-folding data. And when a person verbally walks it through an unfamiliar appliance task, those coaching episodes can be converted into a high-level policy that later performs the task autonomously. Those demonstrations are impressive, but they are not the paper’s most important contribution. The central idea is much more general: a robot foundation model should be conditioned not only on what behavior is requested, but on the latent factors that explain how each training trajectory was produced. That means labeling demonstrations and autonomous rollouts with information such as execution quality, speed, local mistakes, control mode, semantic subtask, and desired future images. These labels turn an ambiguous mixture of successful demonstrations, awkward demonstrations, failures, recoveries, RL rollouts, human videos, and multiple embodiments into a conditional distribution that the policy can interrogate. At deployment, the same conditioning variables become controls for steering the policy toward fast, high-quality behavior. In that sense, π₀.₇ is less an architectural revolution than a proposal for a new interface between data and policy learning. The paper asks what happens when prompts become a structured data schema rather than a short task caption. Its answer is that much more of the robot-learning data flywheel becomes usable—including the messy parts we normally discard. Heterogeneous data is ambiguous data. Suppose a dataset contains ten ways to fold a shirt. Some operators work quickly; others pause to correct poor grasps. Some produce tight, aligned folds; others finish with the sleeves protruding. Autonomous policies add more variation: strong RL specialists generate efficient trajectories, weaker checkpoints fail halfway through, and intervention episodes splice autonomous behavior together with teleoperated corrections. Standard behavior cloning is asked to model all these actions under roughly the same observation and task label. From the model’s perspective, the data are contradictory. A command such as “fold the shirt” does not explain whether the target is the fast trajectory, the cautious trajectory, the recovery behavior, or the failed one. Scaling the dataset can therefore increase the conditional entropy of the action distribution instead of reducing uncertainty. This is one reason that large robot datasets have not always shown the monotonic scaling behavior familiar from language models. More robot data frequently means more embodiments, more controllers, more operators, more inconsistent strategies, and a lower average level of proficiency. Without additional context, the policy can learn an average strategy, overfit the dominant source, or simply become less decisive. π₀.₇ addresses this by exposing some of those hidden causes as conditioning variables. A low-quality trajectory is no longer merely a bad example of “fold the shirt.” It is an example of “fold the shirt, quality two out of five, with a mistake in this segment, at this episode duration.” A strong RL rollout is labeled as a fast, high-quality mode. At test time, the model is requested to produce quality five, no-mistake behavior at a relatively aggressive target duration. This framing has an important implication: failures do not necessarily poison a foundation model if the context explains that they are failures. They can teach state coverage, recovery-relevant visual features, and the consequences of poor action choices without being presented as the desired mode. π₀.₇’s training mixture includes lower-quality demonstrations, failed episodes, autonomous evaluation rollouts, human interventions, open robot datasets, egocentric human video, and auxiliary web-derived multimodal tasks. Autonomous data from the paper’s generalization evaluations are explicitly excluded to reduce direct leakage. The controlled laundry experiment makes the point unusually clearly. When the authors progressively expand training from the best 30 percent of folding data to the entire, lower-average-quality dataset, a model without metadata eventually degrades. The metadata-conditioned model improves throughout, rising from roughly 13 to 23 successful folds per hour. More data becomes better data only after the policy is told what kind of behavior each trajectory represents. From π₀ to π₀.₇. The architecture is easiest to understand as a synthesis of the Physical Intelligence model line. The original π₀ established the core flow-based VLA design: a pretrained vision-language backbone provides semantic and visual representations, while a smaller action expert generates continuous action chunks with flow matching. π₀ was trained across single-arm, bimanual, and mobile platforms, and demonstrated that a shared action-generation architecture could support both generalist pretraining and downstream adaptation. π₀.₅ added a more explicit hierarchy. Rather than asking a low-level policy to interpret only a long-horizon command, the system could condition on semantic subtasks supplied by a high-level policy. The MEM work then addressed partial observability by compressing multi-scale visual history into a fixed token budget, supporting tasks that require remembering earlier observations rather than reacting only to the current frame. π₀.₇ directly inherits both ideas. The other key predecessor is π₀.₆-star and its RECAP training method. RECAP uses offline and online reinforcement learning, demonstrations, interventions, and advantage conditioning to specialize VLAs for difficult tasks. It produced strong task-specific policies for espresso making, laundry, and box assembly, more than doubling throughput on some difficult tasks while roughly halving failure rates. π₀.₇ closes that loop. Instead of leaving those RL gains inside separate specialist checkpoints, it trains on their experience and distills the resulting behaviors back into one generalist policy. The distinction is subtle but strategically important: RECAP asks how a generalist can become an excellent specialist; π₀.₇ asks how the experience of many specialists can become part of a stronger generalist. The prompt as a control surface. π₀.₇’s context contains several independently droppable components. Together, they specify task identity, local intent, desired outcome, trajectory quality, and actuator interface. Semantic subtasks. The first addition is intermediate language. Alongside an overall instruction such as cleaning a kitchen, the policy can receive a changing subtask such as opening the refrigerator or picking up a knife. The subtask may come from a learned high-level policy, a person coaching the robot, or nowhere at all—the low-level model is trained to operate with that field omitted as well. This supports an unusually practical form of interactive teaching. A human does not have to supply continuous actions through teleoperation. They can issue semantic commands at the moments when task intent changes. The robot supplies the motor behavior from its existing repertoire, while the person supplies decomposition and timing. After several coached executions, the system trains a high-level policy to predict those subtask commands from the current observation, overall task, and history of previous instructions. The resulting autonomous system has learned from new interaction data, but it has not required new low-level action demonstrations. That is a more precise description than saying it learns from language alone: it learns from policy rollouts whose semantic structure is provided interactively through language. Generated visual subgoals. Language is good at expressing semantic intent but poor at specifying geometry. “Open the fridge” does not tell the policy which side of the handle to approach, what the wrist view should contain, or what opening angle counts as sufficient progress. π₀.₇ can therefore receive up to three multiview subgoal images depicting a plausible near-future state. A base camera conveys object and scene layout, while wrist images provide information about gripper pose, contact, and local geometry. Because the goal is represented in the same visual space as the observation, the low-level problem becomes closer to inverse dynamics: infer an action sequence that moves the current scene toward the depicted future. The goals come from a separate world model initialized from BAGEL, a 14-billion-parameter unified model for multimodal understanding, image editing, generation, and future prediction. Physical Intelligence fine-tunes it on carefully segmented robot and egocentric-human video, while mixing in image-editing and video datasets to preserve broader semantic knowledge. This design follows the conceptual path opened by SuSIE, which used a pretrained image-editing diffusion model to generate intermediate observations for a goal-conditioned controller. π₀.₇ scales that idea into a multiview, multi-robot VLA and deliberately trains the policy on both real future frames and world-model-generated goals, reducing the train-test gap that would arise if training used perfect images but deployment used synthetic ones. For real future images, one quarter of the goals are sampled from the end of the current annotated segment. The remaining three quarters are sampled uniformly from zero to four seconds into the future. Generated images are added as separate training examples. This gives the policy experience with different goal horizons and different image-fidelity levels. Episode metadata. The metadata prompt includes three main signals. Episode duration is discretized into bins of 500 control steps and serves as a proxy for speed. Overall quality is manually scored from one to five. Finally, segments can carry a mistake label for events such as missing a grasp or beginning the wrong subtask. At runtime, quality is set to five, the mistake flag to false, and the requested duration to the fifteenth percentile of the task’s training episodes. This resembles return- or advantage-conditioned policy learning, but the conditioning is broader and coarser. It includes both global attributes and local annotations, and it is used inside a multimodal behavior-cloning system rather than solely as an RL statistic. The paper also applies classifier-free guidance to metadata during action denoising. In effect, the system contrasts the desired high-quality context with an unconditioned context and pushes sampled actions more strongly toward the requested mode. That opens an interesting design space. A future version could condition on energy consumption, force limits, risk tolerance, preferred grasp family, intervention probability, or task-specific quality dimensions. The broader principle is to turn policy variation into named axes rather than asking one unconditional action distribution to absorb everything. Control mode and prompt dropout. The prompt also identifies whether actions are expressed in joint space or end-effector space. Both kinds of data can therefore coexist without pretending that their targets mean the same thing. Interestingly, end-effector control did not provide a clear cross-embodiment advantage over joint control in the paper’s baseline experiments, suggesting that action-space normalization alone does not explain the reported transfer. During training, subgoal images appear in only 25 percent of examples. When a subgoal is present, the semantic subtask is dropped 30 percent of the time. All metadata is removed in 15 percent of examples, and each metadata field has another independent 5 percent dropout probability. Visual history and the optional rear camera are also dropped with 30 percent probability. Control mode is never dropped. This dropout is not merely regularization. It teaches a family of conditional policies within one set of weights: language-only, image-goal-conditioned, metadata-steered, memoryless, memory-aware, and combinations thereof. It also makes classifier-free guidance possible and prevents deployment from depending on every auxiliary model being available. Model architecture and runtime stack. The main VLA has roughly five billion parameters: a Gemma 3 four-billion-parameter vision-language backbone, including a roughly 400-million-parameter vision encoder, plus an 860-million-parameter flow-matching action expert. The MEM-style encoder accepts up to four camera views, each with as many as six history frames sampled one second apart, and compresses every view’s temporal history to the token count of one image. Observations and subgoals are resized to 448 by 448 pixels. Observation and goal-image tokens receive bidirectional attention within their blocks, with goal tokens also able to attend to the current observation. Subsequent text is processed causally. Proprioceptive histories are projected directly into the backbone dimension rather than being discretized into textual state tokens. The action expert produces a 50-step continuous trajectory, with all action tokens mutually attending and cross-attending to the VLM representation. Training uses knowledge insulation: the VLM is supervised through discrete FAST action tokens, while gradients from the continuous action expert are prevented from flowing back into the backbone. The intention is to retain the stability and generalization of discrete cross-entropy training while still deploying a continuous flow policy. Training-time real-time chunking also simulates inference delays between zero and 12 control steps, corresponding to as much as 240 milliseconds at 50 hertz. At inference, five denoising steps generate each 50-action chunk, of which either 15 or 25 steps are executed before replanning. Subtask generation, visual-goal generation, and VLA inference run asynchronously. New subgoal images are requested when the semantic subtask changes or after four seconds, whichever comes first. The compute footprint matters. A minimal three-camera VLA configuration runs in about 38 milliseconds on one H100; memory and image goals increase worst-case VLA latency to 127 milliseconds. The BAGEL-derived world model requires four H100s with tensor parallelism, 8-bit matrix multiplication, and optimized attention to generate three subgoal images in 1.25 seconds. Calling that component “lightweight” makes sense only relative to large generative models, not relative to a practical edge-robot compute budget. What the experiments actually show. A generalist approaching specialist dexterity. The cleanest test compares the same π₀.₇ checkpoint against individual π₀.₆-star RL specialists on laundry folding, button-up-shirt folding, espresso making, and box construction. Success rates are broadly similar. π₀.₇ reaches about 1.5 times specialist throughput on the hardest laundry item and roughly 1.4 times throughput on box construction, while matching the espresso specialist closely. It also performs comparably to supervised specialists on sandwich making, turning a shirt right-side-out, navigating through a closing door, slicing zucchini, peeling produce, and replacing a trash bag. The ablations are particularly informative. Removing autonomous evaluation data or removing metadata hurts all four specialist-comparison tasks, with the largest gaps appearing in throughput. The result supports both halves of the proposal: specialist rollouts are valuable training data, but only if the model can distinguish stronger and weaker modes. The model also matches or exceeds memory-specialized π₀.₆-MEM policies on four partial-observability tasks, including finding an object hidden in a drawer and executing a particular sequence of mug swaps. This indicates that the generalist has retained MEM’s temporal capability rather than trading memory away for prompt breadth. Language following becomes physically consequential. In 14 scenarios spanning four unseen kitchens and two unseen bedrooms, each requiring three to six open-ended commands, π₀.₇ correctly follows roughly 87 percent of kitchen instructions and 73 percent of bedroom instructions. π₀.₆ achieves approximately 54 and 55 percent, while π₀.₅ remains below 30 percent in both settings. For standard tabletop commands, all models perform reasonably well. The gap widens when instructions use indirect references such as “pick up an object I would use to eat soup” or require relational reasoning. π₀.₇ alone reaches roughly the mid-30-percent range on the complex set, and generated subgoals raise it to around 63 percent. The world model is therefore not a decorative visualization component; in some tests it supplies a large share of the semantic improvement. The dataset-bias tests are even more revealing. In reverse bussing, the robot must place dishes in the trash and trash in the dish bin—the opposite of its training convention. π₀.₇ reaches around 80 percent task progress while earlier models remain much lower. In the reverse microwave-to-fridge task, the base policies are essentially unsuccessful, but π₀.₇ with generated visual goals reaches roughly 70 percent progress. That experiment tests more than object recognition. It asks whether language can override a strong visuomotor prior. A policy that recognizes every object but ignores the command is not meaningfully steerable. π₀.₇’s progress here suggests that its prompt has become an active control signal rather than a weak contextual hint. Cross-embodiment transfer that changes strategy. For simpler rearrangement tasks, π₀.₆ and π₀.₇ both transfer reasonably well. The larger separation appears as morphology and task complexity increase. In shirt bagging, where data came from a small static bimanual robot but evaluation uses one much larger UR5e arm, π₀.₇ approaches 95 percent task progress. Earlier models are around 60 to 70 percent. The resulting motions are not straightforward retargetings. On the source system, an operator uses one arm to hold a bag open while inserting a shirt with the other. The taller UR5e can instead perform a single-arm pick-and-place. For folding, source demonstrations often use a tilted approach that pins cloth against the table; π₀.₇ adopts a more vertical grasp suited to the UR5e’s mounting geometry and kinematics. On zero-shot UR5e shirt folding, generated subgoals raise task progress to 85.6 percent. Ten highly experienced teleoperators, each attempting the same task on that platform without practice, average 90.9 percent. Both the policy and the humans achieve approximately 80 percent full success. The human comparison covers 30 trials and uses identical initial configurations, time limits, and scoring criteria. Matching a ten-person study is not evidence of human-level manipulation in general. It is, however, a compelling result for a carefully defined transfer setting: both humans and policy know the task from one platform and must adapt it to an awkward industrial embodiment without task-specific experience. New tasks through composition and coaching. For short-horizon behaviors, π₀.₇ can directly execute interactions for which the authors collected no task-specific robot episodes: pressing a French-press plunger, scooping rice into a cooker, wiping office objects, and spinning articulated objects. Generated visual goals usually help, though not universally—the language-only policy performs better on the rice task in the reported evaluation. Long-horizon novelty remains too difficult for a single coarse prompt. The paper tests loading and unloading an air fryer and preparing a bagel in a toaster oven, with no action-level training episodes for those complete tasks. Earlier models make little progress even with coaching. π₀.₇ can be guided step by step to nearly complete air-fryer loading, reach around 75 percent on unloading, and—with image goals—around 80 percent on the bagel task. The authors then use five sets of coaching episodes to train high-level language policies. These autonomous policies roughly reproduce the performance achieved under live coaching. This is a credible path toward reducing teleoperation: let the pretrained VLA provide motor primitives, collect task structure through inexpensive verbal supervision, and learn only the semantic controller. Diversity is not interchangeable with volume. Finally, the authors remove either a random 20 percent of robot data or the 20 percent judged to contain the highest task diversity. Both subsets have the same size, but removing the diverse slice causes much larger drops on unseen rice-scooping, object-spinning, and wiping tasks. On spinning and wiping, performance falls to roughly single digits. The result is not a universal scaling law—the paper acknowledges that diversity is difficult to define and isolate in a dataset of this size. But together with the mixed-quality experiment, it supports a useful two-part claim: diversity supplies reusable behavioral components, while context prevents those components from collapsing into an incoherent mixture. Is this really emergence?. The term “emergent” is defensible here, provided it is interpreted carefully. The model develops behaviors that were not directly optimized as separate downstream objectives: adapting grasp strategies to a new morphology, overriding dataset biases, combining known interactions around unfamiliar appliances, and converting semantic coaching into autonomous execution. But the paper cannot prove that these tasks are absent from the model’s broader experience in every meaningful sense. It deliberately uses robot data, human video, open datasets, web-scale multimodal pretraining, and a generative world model. Even when no full action trajectory exists for an air-fryer task, closely related concepts—drawers, handles, baskets, appliance doors, food placement—may appear throughout the mixture. The authors explicitly acknowledge that determining whether a large foundation model has truly “seen” a task is practically impossible. The most productive interpretation is therefore compositional rather than miraculous. π₀.₇ does not invent manipulation from nothing. It appears better able to select, adapt, and recombine behavioral fragments because its training context preserves distinctions that simpler conditioning would erase. It is also important to distinguish the base VLA from the complete system. Some headline results use a learned high-level policy, and many of the strongest generalization numbers use the 14-billion-parameter world model. π₀.₇ with generated context is a modular robot stack, not just one five-billion-parameter network. That does not weaken the contribution, but it changes what must be reproduced and deployed. What roboticists should take away. The first lesson is to record more context before collecting more demonstrations. Duration, success, intervention points, operator corrections, grasp failures, controller type, embodiment, and strategy labels can be as valuable as another undifferentiated trajectory. Logging these signals automatically wherever possible may yield a higher return than aggressive post-hoc filtering. Second, do not discard autonomous rollouts by default. A failed rollout may cover states that human demonstrations systematically avoid. The requirement is to prevent its actions from masquerading as optimal actions. Quality conditioning, advantage labels, intervention masks, or explicit failure-stage annotations all provide potential mechanisms. Third, train for the context that will actually exist at deployment. If a policy will consume generated goals, include generated goals during training. If metadata may be missing, apply structured dropout. If latency is unavoidable, simulate it during action-chunk training. π₀.₇’s strongest engineering decisions repeatedly eliminate train-test mismatches rather than assuming auxiliary components will be perfect. Fourth, separate semantic teaching from motor teaching. If a generalist VLA already possesses the required contact skills, teaching a new workflow may only require demonstrations of when to invoke which semantic subtask. That is a dramatically cheaper supervision channel than teleoperating every action. Finally, quality metadata should not be treated as automatically causal. A “quality five” label may correlate with a particular operator, robot, camera, or environment. A model could learn those shortcuts rather than the intended behavioral distinction. Teams adapting this recipe should actively balance metadata across tasks and embodiments and test counterfactual prompts: can the same scene produce meaningfully different speed or quality modes when only the metadata changes?. Limitations and open questions. The largest scientific limitation is data opacity. The paper enumerates a broad mixture but does not provide the total hours, per-source proportions, task-frequency distribution, annotation cost, training compute, or enough detail to reconstruct the sampling pipeline. This makes it difficult to separate gains from context conditioning, architecture updates, improved labels, raw scale, and the accumulation of experience from earlier model generations. External comparison is also limited. Most baselines come from the same π lineage and often share data or infrastructure. That controls some variables, but it does not establish how π₀.₇ compares with other contemporary generalists under a shared public benchmark. The evaluation suite is impressively physical and broad, yet largely private. Reproducibility is correspondingly constrained. At the time of writing, the official openpi repository documents public base checkpoints for π₀, π₀-FAST, and π₀.₅, but not π₀.₇. Reimplementing the full recipe would additionally require large-scale mixed robot data, dense quality annotations, a fine-tuned BAGEL world model, and substantial inference hardware. The system’s zero-shot results also remain below its in-distribution performance. The paper reports success rates above 90 percent for many seen tasks, versus roughly 60 to 80 percent for unseen tasks or unseen task–robot combinations. That is strong research performance, but it is not yet the reliability profile required for unsupervised household deployment—especially for examples involving knives, peelers, heavy appliances, hot liquids, and human-proximate motion. A final open question is whether metadata steering will continue to scale. Quality, speed, and mistake labels are manageable with today’s datasets. A future model trained on millions of heterogeneous trajectories may need richer automated evaluators, uncertainty estimates, learned behavior descriptors, or language-generated critiques. Otherwise, the annotation layer intended to unlock scale could itself become the bottleneck. Final take. π₀.₇’s most consequential claim is not that one robot can operate an espresso machine or fold a shirt with unfamiliar arms. It is that robotic foundation models become more scalable when dataset ambiguity is represented explicitly in the prompt. That idea connects behavior cloning, offline RL, hierarchical control, goal-image conditioning, world models, and interactive language coaching through one shared mechanism: context. Diverse trajectories stop being anonymous samples from an unruly mixture and become examples of identifiable strategies, qualities, outcomes, and control regimes. If this result holds beyond Physical Intelligence’s private data and hardware stack, the next generation of generalist robot models may be built less by filtering datasets into a single idealized behavior and more by retaining everything—successes, failures, specialists, humans, multiple bodies—while teaching the model enough context to choose among them. That is the real promise of a steerable generalist: not merely a policy that knows many behaviors, but one that understands which behavior we are asking for.