Latest / Embodied AI 101 / The Optimizer Becomes a Policy: Inside RP-1's Learned Planner for Robot World Models

Transcript

The Optimizer Becomes a Policy: Inside RP-1's Learned Planner for Robot World Models · Embodied AI 101

• The Optimizer Becomes a Policy: Inside RP-1’s Learned Planner for Robot World Models. Reinforced Planning with Latent World Models • Armin Sommer and Jannik Schilling • Pantheon Industries • arXiv preprint • 2026. The Missing Intelligence in World-Model Control. A world model answers a forward question: if the robot executes this sequence of actions, what will happen? A planner answers the harder inverse question: which action sequence should the robot try?. The distinction matters because progress on the first question does not automatically solve the second. We can train increasingly capable latent dynamics models, give them compact visual representations, and roll them forward cheaply. But once the model has imagined a candidate future, somebody still has to decide how to revise the actions that produced it. In most contemporary systems, that somebody is a hand-designed optimizer. PlaNet and several pretrained latent world-model systems use the Cross-Entropy Method, or CEM. The TD-MPC family uses variants of Model Predictive Path Integral control, or MPPI. Differentiable models can optimize actions directly with Adam. These algorithms are general and familiar, but their search behavior is specified manually: population size, covariance update, temperature, learning rate, number of iterations, and so on. They may require thousands of world-model trajectories for every control decision. RP1 makes a simple but consequential proposal: turn the optimizer itself into a learned policy. Instead of asking CEM or Adam to improve a plan according to a fixed rule, RP1 trains a neural network to examine the current action sequence, the predicted value of its outcome, and the gradient of that value with respect to the actions. The network then emits a residual update to the entire multi-step plan. This update is applied repeatedly, with shared weights, creating an iterative optimization process whose search rule has been learned from a distribution of planning problems. Pantheon introduced the work on August 20, 2026, one day after the first arXiv submission. As of September 3, 2026, it remains a version-one preprint rather than a peer-reviewed result. Pantheon’s current headline says RP-1 matches or exceeds standard latent methods across 48 evaluation comparisons, has 2.2 times the success rate on its aggregate set of hard and long-horizon evaluations, and is up to 67 times faster under concurrent inference. Before unpacking those claims, a few details in the supplied summary need correction. The navigation environment is TwoRoom, not ThreeRoom. The primary Reacher experiments use 25-step goals and vary the angular success tolerance; they do not report a 100-step Reacher benchmark. And OGBench’s “hard” number is not a separate harder task distribution. It is the same evaluation normalized against a surprisingly high no-action success floor. These distinctions become important when interpreting the headline results. What “Fully Learned” Actually Means. The paper’s novelty claim is narrower—and more defensible—than saying RP1 is the first neural planner of any kind. The authors’ precise claim is that RP1 is the first model-based method to fully learn the iterative update rule over an entire multi-step action plan, without running a conventional optimizer inside that update and without imitating one as a training target. Earlier systems have learned components of planning, learned how to modify a sampling optimizer, learned from an expensive optimizer teacher, or used imagination to train an amortized action policy. RP1 instead learns the mapping from one candidate plan to the next candidate plan. “Fully learned” does not mean every part of the system is learned jointly or that hand-designed choices have disappeared. The world-model encoder and latent dynamics are frozen. The critic and plan refiner are trained separately for each environment and world-model backbone. The action chunking, planning horizon, network architecture, clipping range, intermediate loss, and training schedules are selected manually. The method also depends on gradients through a differentiable world model. So the right interpretation is:. RP1 fully learns the plan-improvement operator within a larger, still highly structured model-based control system. That is nevertheless an interesting boundary to move. A conventional optimizer has the same prescribed behavior on every task. CEM always resamples and refits a distribution. Adam always follows its moment-adjusted gradient rule. RP1 can, in principle, learn that large early changes are useful for one class of plan, that some gradient coordinates should be ignored, that correlated action blocks must move together, or that the update should become conservative near a good terminal state. This is learned optimization in a very literal sense. The planner is not merely outputting actions; it is outputting how the current actions should change. There is also an important trade. CEM can be attached to a new action dimension or horizon without retraining, even if its performance is mediocre. The RP1 refiner is tied to a fixed plan representation and must absorb the structure of its task distribution during offline training. Its low inference cost is therefore amortized: computation and experience are moved from runtime into the learned optimizer’s weights. The paper’s inference comparisons do not include that one-time planner-training cost. Inside One RP1 Planning Request. RP1 operates over image observations but does its planning in a frozen latent space. A current image and goal image are encoded by the world model. The planner then optimizes five action chunks, each containing five primitive actions, giving a 25-step imagined trajectory. In OGBench Cube, each primitive arm action is five-dimensional, so the complete plan contains 125 scalar action values. The inference loop is:. Start from a zero-action plan. Roll the complete plan through the frozen world model, producing an imagined terminal latent state. Score that terminal state with a goal-conditioned critic. Lower value means fewer predicted steps remain before reaching the goal. Differentiate the critic’s value with respect to every action in the plan. Feed the current plan, scalar value, and complete action gradient into the neural refiner. Add the refiner’s residual output to the plan and clip the result to a bounded action range. Repeat the same learned update eight times. The initial evaluation plus eight refined evaluations gives the advertised nine world-model rollouts per decision. The refiner is surprisingly small. It is a three-layer multilayer perceptron with two hidden layers of width 512 and ReLU activations. For OGBench, the input contains the 125-dimensional plan, its 125-dimensional gradient, and one scalar value. That produces a 251-dimensional input and a 125-dimensional residual update, for approximately 460,000 parameters. The same parameters are reused at every refinement iteration. Crucially, the refiner does not receive the raw current-state or goal latent. Task-specific information can reach it only through the critic’s scalar value and the gradient over the plan. This forces the network to act more like an optimizer than a conventional goal-conditioned policy. It cannot simply memorize a mapping from a visual configuration to an action trajectory. At training time, the planner is unrolled through all eight refinement steps. Its loss is the final plan’s predicted cost-to-go plus a weighted average of the costs produced at intermediate refinement iterations. The value and gradient used as refiner inputs are detached from the training graph, avoiding a higher-order optimization path through the gradient feature, while the final refined actions remain differentiable through the frozen world model and critic. No environment interaction is required during this planner-training stage. This also clarifies what “reinforcement learning” means here. The critic is trained with offline temporal-difference learning, while the planner behaves as the actor and is optimized against that critic by differentiating through imagined dynamics. It is not PPO over optimizer actions, nor is the refiner discovering updates through sparse trial-and-error rewards. It is closer to an offline actor-critic system combined with differentiable learned optimization. At inference, RP1 still calculates a conventional gradient through the world model. It has not eliminated gradients; it has learned what to do with them. The refiner can use the gradient as a proposal, precondition it, coordinate different time steps, attenuate unreliable directions, or make a non-gradient-like jump. But because it sees only the current plan, value, and gradient—without an explicit recurrent state or history of previous gradients—it is not an unrestricted learned optimizer. Its memory lives primarily in the evolving plan itself. The Critic: From Visual Closeness to Temporal Reachability. The second half of RP1 is arguably as important as the learned refiner. Many latent world-model planners score an imagined plan by measuring Euclidean distance between the predicted terminal embedding and the goal embedding. This is convenient because it requires no reward labels. But predictive representation learning does not guarantee that Euclidean distance will correspond to controllability, time-to-goal, or even meaningful physical similarity. The paper’s first theorem makes that problem precise. Imagine an exact world model whose latent transitions perfectly represent the environment. We can stretch, rotate, or skew its latent coordinates and make a compensating transformation in the transition model without changing any prediction. The transformed system remains an equally exact world model, but distances between states can change arbitrarily. With the right coordinate stretch, one goal can be made to appear closer than another; a different stretch reverses their ordering. Predictive accuracy alone therefore does not identify a useful Euclidean planning geometry. TwoRoom is the intuitive demonstration. The agent and goal may be visually and geometrically close on opposite sides of a wall, while the executable route requires moving away from the goal, passing through a doorway, and then returning. A latent-distance planner can prefer repeatedly pushing into the wall. A temporal critic can represent the actual detour. RP1 learns a separate goal-conditioned cost-to-go function from cached offline latents. Every transition has a cost of one, so the critic estimates how many steps are needed to reach a goal. The architecture is a metric residual network: one part behaves like a symmetric learned distance, while another permits directed distance. Directionality matters in manipulation because getting a cube from state A to state B need not cost the same as undoing that transition, especially after contact, dropping, or obstruction. Training combines hindsight goal relabeling, multi-step temporal-difference targets, cross-episode goals, and an Implicit Q-Learning-style expectile objective. Because the output is cost rather than reward, the authors use an expectile below one-half, penalizing overestimated cost more strongly. The intended effect is to estimate something closer to the shortest achievable path represented in the dataset, rather than the average behavior-policy path. This critic can also be useful without RP1. The experiments run CEM, MPPI, and Adam with both their usual latent-distance objective and RP1’s learned value objective. In TwoRoom especially, simply replacing latent distance with temporal cost-to-go repairs much of the apparent planning failure. That controlled comparison is one of the paper’s strongest design choices because it separates two questions that are often conflated:. Is the planner optimizing the wrong objective?. Or is it optimizing the right objective badly?. RP1 addresses both, but the answer varies by domain. A Learned Optimizer, Not an Amortized Policy. The authors position RP1 between online search and direct policy learning. Dreamer-style systems use imagined rollouts to train an actor, but at inference the actor directly emits actions; it does not repeatedly optimize a candidate sequence through the world model. Diffuser learns a trajectory distribution and improves trajectories through denoising rather than through repeated evaluation by a separate frozen dynamics model and critic. Thinker and imagination-based planners learn which hypothetical futures to inspect, but the resulting information conditions an amortized policy rather than revising a persistent action plan. Universal Planning Networks optimize multi-step actions but retain gradient descent as the hand-designed optimizer. Iterative Amortized Policy Optimization learns updates to the current action distribution, while later actions remain outputs of an amortized policy rather than jointly optimized variables. DMPO and Learning-to-Optimize MPC are the closest comparisons in spirit. DMPO retains an MPPI skeleton and learns corrections to its sampling update. L2O-MPC learns a sampling update by imitating a high-budget MPPI teacher. RP1 removes both dependencies: there is no mandatory MPPI structure and no optimizer-generated supervision. The frozen backbones also matter. LeWorldModel is a compact, end-to-end JEPA trained from pixels; its original test-time planner uses CEM and latent endpoint distance. PLDM comes from work on reward-free offline learning, where latent dynamics planning showed strong trajectory stitching and generalization. RP1 leaves both predictors unchanged and asks whether a better objective and optimizer can extract more control performance from them. The Experimental Contract. The experiments use TwoRoom navigation, visual Reacher, and OGBench Cube manipulation from the StableWorldModel ecosystem. Each domain provides 10,000 episodes. The first 8,000 train critics and planners, while the final 2,000 provide held-out starts and goals. Hyperparameters are selected on separate evaluation draws, and all reported world-model encoders and dynamics predictors remain frozen. Every planner optimizes the same five chunks of five primitive actions. CEM and MPPI each evaluate 300 candidates for 30 iterations, totaling 9,000 forward rollouts. Adam evaluates 3,000 plans and performs the corresponding backward passes. Offline-DMPO and L2O-MPC use 256 rollouts. RP1 uses nine. The “100-step” evaluations require another nuance: the goal is sampled 100 primitive steps ahead, but the planner still optimizes only 25 primitive actions at each request. It executes those open-loop chunks and plans again. RP1 is therefore not producing a single optimized 100-step action sequence. It is solving a longer task through repeated fixed-horizon segments. The 48-of-48 headline also deserves a constrained reading. The main tables contain 16 result cells across TwoRoom, Reacher, and the easy and normalized OGBench metrics. Comparing RP1 with the three standard latent-objective planners gives 48 pairwise comparisons, which appears to be the basis of the claim. It should not be interpreted as RP1 strictly beating every value-augmented or learned hybrid in every cell: value-scored CEM slightly exceeds RP1 in two TwoRoom settings, and Offline-DMPO reaches 100% on one long-distance TwoRoom setting where RP1 scores 94.2%. Reading the Results Benchmark by Benchmark. TwoRoom: Mostly a Victory for the Critic. On 25-step TwoRoom goals, RP1 averages 99.1% success across LeWM and PLDM. At a 100-step goal offset, it averages 95.1%. Standard latent-distance Adam averages 92.7% at 25 steps but only 33% at 100, while latent-distance CEM averages roughly 32.7% at the longer offset. Those are dramatic numbers, but the objective ablation changes the interpretation. Give CEM the learned temporal critic and its scores become 100%, 94.7%, 100%, and 89.3% across the four backbone-horizon combinations. Adam also improves sharply. The paper explicitly concludes that the dominant TwoRoom problem is how plans are scored, not necessarily how they are searched. RP1 still obtains comparable performance with three orders of magnitude fewer model rollouts, which is meaningful. But TwoRoom does not by itself show that learned refinement is substantially better than well-budgeted conventional optimization. It shows that latent Euclidean distance can be fundamentally wrong and that RP1 achieves nearly saturated performance extremely efficiently once equipped with a suitable critic. Reacher: Isolating Search and Terminal Precision. Reacher removes the wall and contact complications. Every joint configuration is mutually reachable through free space, so geometric and temporal proximity largely agree. Replacing latent distance with the learned critic barely changes the conventional baselines. This makes Reacher a cleaner test of the optimizer itself. At the loose 0.1-radian tolerance, RP1 scores 98.7% on LeWM and 97.8% on PLDM. CEM is already at 98.7% and 96.7%, so the gap is small. At the tighter 0.05-radian tolerance, RP1 reaches 88.7% and 82.0%, versus 80.3% and 80.0% for latent-distance CEM. The advantage is primarily in terminal precision rather than broad reachability. An appendix experiment expands the terminal cost from one frame to three, giving the objective access to velocity-like information. RP1 then reaches 97.1% and 91.2% at the tight tolerance, remaining ahead of the conventional planners. This suggests that part of fine control depends not just on where the imagined trajectory ends, but on how it arrives. The striking part is therefore not a huge absolute Reacher gain. It is that a 0.46-million-parameter learned refiner using nine trajectories can match or exceed CEM using 9,000 trajectories, even when both are already operating with a reasonable objective. OGBench Cube: The Strongest Evidence for Learned Search. OGBench Cube is the most convincing benchmark for RP1’s plan-update rule. The robot must grasp, move, and place a cube from pixels. Contact creates a discontinuous, multimodal planning landscape: a small change early in the sequence determines whether the fingers close around the cube, miss it, or knock it away. Smooth local optimization struggles, while a unimodal sampling distribution can average incompatible grasp modes. The benchmark also contains a major evaluation trap. Doing nothing already satisfies the placement criterion in 56% of 25-step-goal episodes and 45.3% of 100-step-goal episodes. The paper therefore reports the ordinary “easy” success rate and a normalized “hard” score measuring how much improvement the planner obtains above that no-op floor. Again, “hard” is a normalized metric, not a separately sampled challenge set. RP1’s easy and hard scores are:. LeWM, 25 steps: 89.1% easy, 75.2% normalized. LeWM, 100 steps: 82.4% easy, 67.8% normalized. PLDM, 25 steps: 82.9% easy, 61.1% normalized. PLDM, 100 steps: 77.1% easy, 58.1% normalized. RP1 is best in all eight Cube columns. Value-scored CEM is the strongest conventional alternative, reaching normalized scores of 63.6%, 57.4%, 31.8%, and 34.1%. The critic clearly helps CEM, especially at longer offsets, but RP1 adds a substantial search gain on top. The contrast between TwoRoom and Cube is informative: TwoRoom mainly diagnoses the objective; Cube reveals the value of the learned optimizer when contact makes the plan landscape difficult. Nine Rollouts, 30 Milliseconds, and What the Speed Claim Means. A count of model rollouts is not the same as wall-clock compute. CEM can send hundreds of candidate trajectories through a GPU in parallel, whereas RP1 must perform sequential refinement iterations and calculate a gradient through each imagined rollout. The paper acknowledges that a 1,000-fold rollout reduction does not yield a 1,000-fold latency reduction. The end-to-end latency benchmark is therefore more useful. On OGBench Cube with LeWM, using FP32 on one NVIDIA H200, RP1 returns a plan in 30 milliseconds. CEM takes 391 milliseconds, MPPI 420 milliseconds, and Adam 959 milliseconds. The learned hybrids take 45 milliseconds for L2O-MPC and 83 milliseconds for DMPO. With 50 independent planning requests batched on the same GPU, RP1 completes the batch in 79 milliseconds, compared with 5.31 seconds for CEM. That is the source of the 67-fold speedup. It represents simulated concurrent control loops sharing an accelerator, not a demonstrated deployment of 50 physical robot arms. And while the amortized GPU time is 1.58 milliseconds per request, each controller still waits roughly 79 milliseconds for the batch to finish. Even with those qualifications, the systems result is compelling. Learned optimization becomes especially attractive in a fleet setting, where a single accelerator must serve many control loops and the broad populations used by sampling planners compete for GPU capacity. The Planner Finds the World Model’s Lies. A stronger optimizer does not merely expose the capabilities of a world model. It also discovers its weaknesses. In OGBench failure cases, RP1 found plans where the frozen world model predicted that the cube would attach to the gripper even when the grasp had missed. From the planner’s perspective, these plans were excellent: the imagined cube moved toward the goal. In the real simulator, nothing had been grasped. The neural refiner had learned to exploit a model bug. Pantheon addresses this with a single Dyna-style iteration. The trained planner is deployed in the environment, its failure trajectories are added to the original data, the world model is fine-tuned, latent caches and the critic are rebuilt, and the planner is retrained. On the normalized Cube metric, this raises RP1 from 75.2% to 87.3% on LeWM’s 25-step setting and from 61.1% to 80.2% on PLDM. Gains remain at the 100-step offset even though the corrective data was collected only on 25-step tasks. This is more than an implementation detail. It illustrates an optimization version of adversarial pressure: as the planner gets better, it searches more aggressively for states where the model is optimistic. Action clipping keeps trajectories near the demonstrated range, but a box constraint measured in action standard deviations is only a crude proxy for epistemic support. A deployment-oriented version of RP1 will likely require uncertainty-aware planning. Ensembles, conservative value estimates, rollout-disagreement penalties, behavior priors, or explicit contact confidence could help distinguish a genuinely good imagined plan from one that merely exploits an unsupported prediction. Dyna-style data collection then becomes a continuing planner-model co-evolution loop rather than a one-time patch. The result also tempers the paper’s suggestion that planning quality is generally the binding constraint. In TwoRoom and Reacher, better scoring and search may unlock most of the frozen model’s capability. In contact-rich manipulation, improving the planner quickly shifts the bottleneck back to model fidelity. What Remains Unproven—and What I Would Try Next. The authors are unusually direct about several limitations. Hyperparameters differ between environments and sometimes between backbones. Execution is open loop over the complete 25-step plan, and closed-loop replanning could change the ranking. The planner assumes a fixed action interface and horizon, whereas hand-designed optimizers transfer across those choices without retraining. The evaluation covers two latent world-model families, three simulated domains, and no real robot experiments. The “attach it to any pretrained latent world model” claim should therefore be read as an architectural proposition, not yet an empirical demonstration of universal compatibility. In practice, RP1 needs a differentiable action-conditioned rollout model, a compatible fixed plan representation, offline trajectories for critic learning, and a separately trained refiner. The experiments do not show one RP1 network transferring between LeWM and PLDM, much less between action spaces or embodiments. There are also evaluation questions. The hybrid baselines are adaptations: Offline-DMPO replaces the original online PPO training with offline pathwise gradients, while L2O-MPC is trained from an MPPI teacher constructed for this study. Main tables report averages but not confidence intervals. Planner-training compute is omitted from the inference comparison. And as of September 3, 2026, I could not find an official RP1 implementation linked from the Pantheon article or arXiv record, although the StableWorldModel benchmark platform and LeWM implementations are publicly available. For researchers adapting the idea, I would prioritize four extensions:. Use the critic before replacing the optimizer. Training a temporal-distance critic and plugging it into an existing CEM or MPPI stack is the cheapest diagnostic. TwoRoom shows that a correct objective can matter more than a sophisticated search rule. Initialize RP1 from a policy proposal instead of zero. A VLA, diffusion policy, or behavior-cloned controller could supply a plausible plan, with RP1 acting as a goal- and dynamics-aware correction layer. This could combine learned priors with explicit counterfactual evaluation. Restore multimodal search. A small population of stochastic RP1 refiners, or RP1 updates embedded inside CEM, could preserve multiple grasp or navigation hypotheses while retaining learned update efficiency. Train variable-horizon and uncertainty-aware refiners. A tokenized plan representation, explicit iteration embedding, model-identity conditioning, and uncertainty features would move the method closer to a planner that genuinely transfers across world models and embodiments. Bottom Line. RP1 does not solve robot learning, and it does not establish that every handcrafted planner should immediately be discarded. The evidence is still simulation-only, domain-specific, fixed-horizon, and dependent on frozen differentiable models whose errors the planner can exploit. But the central idea is strong: planning need not mean repeatedly running the same generic optimizer over every new problem. Search behavior can itself be amortized, trained, and improved from experience. The paper’s most important achievement is not the phrase “fully learned planner.” It is the clean separation of three things that world-model research often bundles together: predicting futures, assigning useful values to those futures, and learning how to revise the actions that create them. RP1 demonstrates that the third component is a viable learning problem—and, on these benchmarks, a surprisingly efficient one.