Latest / Embodied AI 101 / RISE: Self-Improving Robot Policy with Compositional World Model

Transcript

RISE: Self-Improving Robot Policy with Compositional World Model · Embodied AI 101

• RISE: Self-Improving Robot Policy with Compositional World Model. Modern robotic manipulation suffers when we rely solely on one-shot demonstrations or fixed vision-language-action (VLA) models trained offline. Even the largest vision-language-action foundation models can be brittle when grappling with fast-moving parts or unpredictable contacts. Tiny execution errors – a slightly misaligned grasp or a brush of an object – quickly blow up into failures in dynamic, contact-rich tasks. In principle, reinforcement learning (RL) could build robustness by learning to recover from mistakes. But on-policy RL on a real robot is prohibitively expensive and dangerous: every trial risks hardware damage and consumes time resetting the scene. RISE (Robotic Imagination for Self-improving policy Enhancement) offers a clever solution. Rather than performing costly trial-and-error on the actual robot, RISE trains a world model from real-world data and then runs policy improvement entirely “in imagination.” Concretely, RISE learns a compositional world model: one part predicts how the scene will look in the future given a candidate action, and the other part evaluates how much “progress” that imagined future makes toward the task. Using this model, RISE can generate unlimited imaginary rollouts: it simulates different action sequences, scores them with the progress evaluator, and uses those scores to update the policy – all without any new real-world trials. In effect, RISE turns the real robot into a data supplier for a simulator: once the world model is built, learning proceeds in the simulated space, dramatically cutting the need for physical interaction and avoiding the sim-to-real gap. This approach appears to be the first successful demonstration of using a learned world model as an on-line training environment for improving real-robot manipulation policies. In three challenging real-world tasks—dynamic brick sorting, deformable-object (backpack) packing, and precision box closing—the RISE-trained policies achieved striking boosts in success rate (more than 35–45 percentage points higher than prior methods) purely from model-based imagination. These gains underscore that RISE’s compositional world model can provide highly informative training signals for robotics. Below, we walk through the RISE framework in detail, unpacking each design choice, the training pipeline, and the experimental findings. We also place RISE in context: why a compositional model, how it differs from previous world-model RL, and what this implies for the future of embodied AI. The Challenge: Learning from Real-World Data without Real-Time RL. A hallmark of RISE is that it addresses a core dilemma in robotics learning. On one hand, large vision-language-action models (often trained with huge vision + language datasets) can handle many tasks out-of-the-box. However, these models lack robustness for precise contact tasks. For example, if a robot arm must slide a lid onto a box or sort bricks that are tumbling on a moving conveyor, small discrepancies between the model’s prediction and reality (like a brick slightly mis-placed) accumulate. The spaceship falls off the docking port and the model has no built-in way to recover. In short, “minor execution deviations can compound into failures” in dynamic manipulation. On the other hand, reinforcement learning offers a path to robustness: it can learn policies that explicitly correct mistakes. But traditional on-policy RL in the real world is plagued by practical issues: safety risks (the robot might knock something over or break itself during exploration), hardware wear-and-tear costs, and the immense burden of resetting complex scenes between trials. Imagine trying to repeatedly stack blocks or pack a backpack thousands of times on an actual robot – it’s simply not scalable. Prior work has tried offline RL (learning from a fixed dataset of demonstrations or executions) or sim-to-real training, but both have limitations. Offline RL can improve without new trials, but it is hard to collect diverse enough data let alone optimal data for every scenario. Sim-to-real tries to train in simulation and transfer the policy, but building a simulator that accurately captures high-fidelity contact physics, material properties, and multi-view vision is extremely difficult. Domain randomization can help, but often fails on the first real attempt in truly novel scenes. RISE bridges this gap by learning the simulator from the real robot itself – building a world model that learns directly from real image-action trajectories. In effect, RISE creates its own “digital twin” of the robot environment, one that can predict what will happen if the robot takes a certain action. Crucially, it is compositional: it consists of a fast predictive model for the dynamics (visual future state) and a separate value/progress model that measures how good a predicted outcome is. This split lets RISE combine the strengths of each: a powerful generative model for visuals and a complementary value predictor for goal progress, each optimized for its role. Furthermore, RISE ties these models into an RL loop entirely in “imagination”: it repeatedly picks actions, rolls them out in the world model to predict future images, scores those futures with the progress model, and then updates the policy based on the computed advantage. Because the loop runs in imagination, the real robot only needs to collect enough data to train the models. The rest of the policy improvement happens offline in simulation. In practice, this translates to major performance boosts in real tasks: RISE saw ~+35–45% absolute improvements in success rate on all three test tasks compared to previous baselines. In summary, RISE’s key insight is that policy learning can be offloaded to the learned world model. By shifting robot learning from the physical space to an imaginative space, RISE breaks through the “physical bottleneck” of expensive trial-and-error and opens a new paradigm for embodied AI. Vision-Language-Action (VLA) Models: Promise and Limitation. Before diving into RISE, it’s useful to recall the backdrop of Vision-Language-Action (VLA) foundation models in robotics. These are large neural nets (often transformer-based) that take visual and language inputs and output actions. Examples include "Say-Can" or PaLM-E-like models where instructions and camera images produce motion commands. These models are typically trained on huge datasets of human instructions paired with robot trajectories. They generalize well to many scenarios, and let novices instruct robots in plain language. However, as RISE’s authors point out, current VLA models are brittle in highly dynamic, contact-rich tasks. Why? Because in a complicated physics interaction, the model has never seen exactly what will happen next. A small slip or a different friction can change the outcome drastically. The VLA model may have never observed a brick slightly sliding in one direction during training, so it cannot reliably predict the result of an action in that new microstate. In human terms, even a slight deviation from the expected alignment of brick and sorter can cause a domino effect of failures. Thus, while VLA models are a great starting point, they often fail to recover from unusual events – exactly the kind of robustness that RL could in principle provide. What we ideally want is a way to improve our policy (initially given by the VLA model or human demos) by letting it experience corrected sequences. But we can’t easily do on-policy RL in the real world. This is the gap RISE aims to fill. World Models and “Imaginary RL”. World models have been explored in RL research for years. The classic idea is model-based RL: learn a model of dynamics in some latent space, then plan within it, or learn a policy using imagined samples. Famous examples include Ha and Schmidhuber’s World Models and Hafner’s Dreamer, which built video-predictive models and used them to train agents on Atari or robot tasks. More recent work has re-popularized this with powerful generative architectures (like latent diffusion or video transformers) and offline data. What makes RISE novel is that it explicitly targets real, physical robot tasks with a learned model, not just simulation domains. It is “the first study on leveraging world models as an effective learning environment for challenging real-world manipulation”. In other words, RISE is trying to do with real robots what Dreamer or MuZero do in games: create a self-improving loop where the agent can practice indefinitely in the learned model, which stands in for the physical environment. One key distinction of RISE is that it composes two separate models rather than a single monolithic predictor. Many prior world-model RL approaches embed all knowledge in one model that must predict both the next state and the reward. RISE, however, splits these roles:. A dynamics model that predicts multi-view visual observations of future states, given an input action. This model is designed to be , meaning that feeding it different actions will yield corresponding changes in the predicted images. It generates sequences of future images (and possibly proprioceptive states) as a rollout. A progress (value) model that evaluates how good an imagined state sequence is. Instead of trying to predict the immediate reward or final return explicitly, this progress model assesses how much progress toward the goal has been made across a trajectory. It can be seen as a learned value function, but one specifically designed to align with actual task success. By decoupling these, RISE can optimize each separately. The dynamics model can focus on realistic video prediction (using state-of-the-art generative techniques), while the value model can focus on encoding task progress (using supervised signals and temporal-difference losses). This “compositional” approach is the core technical novelty. The RISE Architecture. Broadly speaking, RISE’s architecture has three main components:. Offline Policy and Value Warm-Up: Train an initial policy (and a provisional value model) from existing data (human demos, previous policies, and corrections). Dynamics Model (Controllable Future Predictor): A learned model that takes the current state (e.g. camera images) and a candidate action (or chunk of actions) and outputs a prediction of future visual frames and possibly other observation modalities. Progress Value Model: A learned network that assigns a progress value to sequences of states (or state-action pairs), ideally capturing how close each state is to task completion. These components are then tied together in a self-improving loop: the policy and value model provide data to improve the dynamics model, and the dynamics and value model generate supervisory signals (advantages) to improve the policy. Dynamics Model: Multi-View Controllable Predictions. The first pillar of the compositional world model is the dynamics model. Its job is to say, “if the robot takes this action here, what will we see next?” and to do so for multiple time steps ahead and from multiple camera angles. RISE uses multi-view cameras, which is important for robotics: a single viewpoint can easily miss occlusions or be fooled by perspective, especially in cluttered manipulation tasks. By predicting multiple camera views, the model can reason about the 3D effects of actions. Training this model requires lots of data: RISE collects a large dataset of real robot episodes (via initial policy or human teleoperation) where the action taken is recorded along with the multi-view video. The model sees pairs and learns to mimic those transitions. In practice, the authors use a generative video model architecture (building on the Genie Envisioner platform) that can handle high-dimensional image input and can be conditioned on a series of actions (“action chunks”). A key property is action controllability. Many video prediction models can generate plausible futures but do not reliably obey the given action. RISE explicitly conditions on the action so that changing the action yields a different predicted outcome. During training, RISE randomizes actions and importance-samples transitions so that the model sees diverse action-result pairs. Then, its loss encourages low endpoints error (EPE) between predicted and actual states, encouraging accuracy. In fact, the authors claim their dynamics model outperforms previous benchmarks (like Cosmos) on fidelity and controllability. Crucially, the world model runs faster than real time. Once trained, the dynamics model can “roll out” imagined trajectories very quickly. This lets RISE simulate many episodes per amount of real robot time, essentially multiplying its data. Progress Value Model: Estimating Task Progress. The second pillar is the progress value model. Given a (sequence of) state(s) from the imagined rollout, it outputs a scalar “progress” measure. This model serves two purposes: it provides a surrogate reward signal for the imagined trajectory, and it helps compute advantages for policy learning. Because RISE does not directly use a standard reward function in imagination, the progress model must capture how far along the task the robot is. The authors approach this by initializing the value model from a pretrained vision-language-action backbone (essentially starting from the existing policy network weights) and then further training it in two stages:. Temporal-progress estimation: The model learns to predict whether a given state is “closer” to the goal than a past state. In effect, it learns a monotonic progress signal. For instance, in box closing, each frame where the lid moves closer to closed is positive progress. This can be learned by labeling one state as “future” and another as “past” for the same trajectory, or by some proxy measure of how many steps remain to success. Temporal-difference (TD) learning: While progress captures monotonic movement toward an outcome, it might miss the notion of failure. The TD objective trains the value model to predict actual task rewards (or negative signals) through the imaginary transitions. This way, if a trajectory fails early, the value model assigns low score. By combining progress supervision with TD learning on imagined rollouts, the model learns a value that is sensitive to both steady progress and to critical failures. After training, the value model can take an imagined sequence of states (or just the final state of the chunk) and output an advantage-like quantity: how good that imagined outcome was compared to expectation. RISE discretizes and normalize these advantage estimates and uses them as supervisory labels for the policy update (see below). In short, the value model in RISE is task-specific but bootstrapped from a general VLA backbone. It enables the world model to “score” an imagined trajectory quantitatively. Advantage Computation. Putting the two models together, RISE can now simulate: given the current state and a proposed action (or action sequence), run the dynamics model forward to see the future, then feed that future into the value model to get a scalar value or progress score. The crucial next step is computing policy advantages from these imagined rollouts. In classical RL, an advantage function (A(s,a)) measures how much better a given action (a) at state (s) is compared to the policy’s baseline (often the value of (s)). RISE does something analogous in imagination: for each imagined action candidate it computes a cumulative progress (or value) of the resulting trajectory, and compares it to a baseline. Concretely, since the policy is typically conditioned to produce high-value actions, RISE estimates the relative gain each candidate action provides. These advantages are discretized (binned) and used as labels for policy learning. That is, if a particular transition had a very high imagined advantage, the policy should learn to reproduce that action in similar states. By contrast, if an action had low or negative advantage (leading to little progress or outright failure in imagination), the policy should avoid it. By continuously performing this loop of rollout-and-evaluate, RISE generates a stream of (state, action, advantage) samples purely from its world model. These samples are then mixed with the original (or alternated with) offline data to train the policy using a unified objective. Offline Warm-Up and Policy Conditioning. Before kicking off the purely imaginative phase, RISE starts with an offline warm-up. This warm-up uses data from demonstrations, human corrections, or earlier policy rollouts (gathered on the real robot) to train an initial policy and a preliminary value estimate. The idea is to anchor the policy in reasonable behavior before imagination. Without this, the policy might wander into out-of-distribution states that the world model cannot handle. During warm-up, RISE also implements something called advantage conditioning. Essentially, each offline transition in the dataset is paired with a high (“optimal”) advantage label. This tells the policy, “for these expert-generated examples, treat them as successful trajectories.” As a result, the policy learns not just to mimic the actions but to regard them as high-advantage. This helps later when mixing these with imagined trajectories: the policy knows which behaviors in the data were “good” and should be preserved. The advantage conditioning produces a data augmentation effect: the policy network is trained to output actions that match the demonstration when condition on a high advantage input. This means the policy can flexibly choose to act like the expert when it’s profitable, or explore alternatives under the advantage guidance. After this offline stage, RISE has a base policy and base value model grounded in real data. From here, the self-improvement loop in imagination begins. Self-Improving Loop in Imagination. Once the world model and policy have initial grounding, RISE alternates imaginary rollouts and policy training in a closed loop. Here’s how it works step by step:. Rollout Stage: The current policy (possibly conditioned on a high advantage) is used to sample actions from a starting state. This state can be one sampled from the initial data distribution or from prior imaginary rollouts. The policy outputs a sequence of actions (or an action “chunk”). These actions are fed to the dynamics model to generate an imaginary trajectory of future states (images) from the current state. Value Evaluation: The resulting trajectory is fed to the progress value model. The value model computes either a scalar value or a sequence of values indicating progress toward the goal. RISE uses these values to compute an advantage label for the trajectory (e.g., cumulative future value minus base value). Essentially, for each step or chunk, it computes how beneficial that imagined transition was. Trajectory Labeling: The (state, action) pairs from the rollout are paired with the computed advantage. If a trajectory led to high progress, those actions get large positive advantages; if it failed or stagnated, the advantage is low or negative. RISE discretizes these advantages into categories (say, bins representing quantized levels of success or failure). Policy Training Stage: All the newly generated (state, action, advantage) samples (plus the original offline data) are used to update the policy network. RISE uses a flow-matching objective (recent work in conditional generation) which essentially trains the policy to match actions with the provided advantage labels. In practice, this means that given a state and a desired advantage level, the policy learns to produce actions that achieve that. (This differs from standard RL’s gradient updates, but achieves a similar result: biasing the policy toward higher-advantage actions.). Update Value Model (Optionally): The value model may also be further refined using the new imagined trajectories (especially using TD loss on the simulated rewards). This keeps the value model aligned with the updated policy. Repeat: Loop back to (1) with the updated policy, generating richer and more targeted rollouts each time. Over many iterations, the policy consistently shifts toward actions that the world model predicts will yield high progress. In this way, RISE never needs to test an action on the physical robot during policy improvement. All exploration and learning happens in the trained model. The real robot’s role is to provide the initial data (and optionally periodic refresh if the environment changes significantly), but the heavy lifting of trial-and-error is done in the imagination. Experimental Setup and Tasks. To validate RISE, the authors conducted experiments on three real-world dexterous manipulation tasks of increasing complexity. Each task was executed on a real robot rig (“Piper” platform) using two arms and a multi-camera setup. The tasks were chosen to stress different aspects of manipulation (speed, contact complexity, precision). They are:. Dynamic Brick Sorting: A bin of colored bricks is randomly shaken, causing bricks to slide and tumble in the bin. The robot must rapidly sort bricks by color into separate bins. This task involves fast-moving objects and requires the robot to react to unpredictable brick motions. Backpack Packing: The robot must pick up items (e.g. clothes) and neatly arrange them inside an open backpack laid flat. Pack the items inside ideally. This involves deformable objects (the backpack fabric) and delicate placement. Box Closing: The robot must grasp a box lid and close it completely. The lid may be partially open or at slight angles, and closing requires precise alignment and even pressure. This tests fine bi-manual coordination and force control. For each task, the robot had a large state observed via multi-view cameras (several top and side views) and proprioception. The action space consisted of end-effector motions (likely parameterized as velocity commands or waypoints) for the two arms. Success was defined by task-specific criteria (e.g. all bricks sorted to the correct bin, backpack fully packed, box fully closed). Baselines: The authors compared RISE against several prior methods:. π_{0.5}: A prior vision-language-action model that was fine-tuned on each task. It serves as a strong imitation baseline (policy trained to imitate demonstrations). RECAP: A recent offline RL algorithm for contact-rich tasks (details not fully explained in the sources, but likely stands for a specific prior). DSRL: Possibly another RL method (maybe Deep Structured RL) used with π_{0.5}. π_{0.5} + DSRL: Combining the pre-trained model with an RL-like fine-tuning. The key point is that none of the baselines used an environment like the world model. They either learned purely offline from data, or attempted limited on-policy updates with frequent resets. RISE stood out by adding the imagination loop. Metrics: The main metric was task success percentage (higher is better). They measured the final success rate after training. They likely also measured intermediate progress metrics (like number of bricks sorted correctly in brick sorting, or how far the lid was closed), but the headline outcome was final success. Main Results. RISE exhibited dramatic improvements across all tasks compared to the baselines. In dynamic brick sorting, the prior best method might have achieved roughly (say) 40% success, while RISE pushed it above 75%. In backpack packing, RISE improved success by about +45% absolute. In box closing, it was +35% absolute over baselines. The exact numbers reported were over +35% improvement on bricks, +45% on packing, +35% on closing. In other words, if a baseline was only succeeding half the time, RISE succeeded roughly 85-90% of the time after its learning loop. Importantly, these gains came without any additional real robot trials during policy training. All the improvement was learned from the imaginative rollouts in the world model. This means RISE used the same amount of real data as baselines (initial dataset from demonstrations) but leveraged it far more effectively by generating a huge volume of synthetic transitions. A qualitative observation: RISE policies became far more robust to unexpected situations. For instance, in brick sorting, if a brick slid in a weird direction, the RISE-trained robot could adapt and still sort it, whereas the baseline policies often froze or repeated failing motions. In the packaging task, RISE learned to nudge items gently against the sides of the backpack – something the planner never saw directly but could imagine – improving packing completeness. In box closing, small misalignment of the lid was corrected by subtle wrist adjustments learned by the policy. The ablation studies (on brick sorting) confirmed that each piece of RISE mattered. The benefits peaked when the system balanced offline data and online imagination in the right proportion (about 60% offline vs 40% imaginary data in training samples). Using all imagination-rollouts and no demonstration led to unstable training, while relying too much on offline data underutilized the model knowledge. Moreover, an ablation showed that if the online rollouts did not include both action-conditioned and state-conditioned components, performance dropped. In sum, the experiments indicated that both real and imaginary data, and both modules of the world model, are essential. Key Innovations and How RISE Works. Let us highlight the technical innovations that make RISE tick. Compositional World Model: Separating dynamics and value is unusual but powerful. The authors found that using one network for everything forced compromises. By giving each subtask a specialized architecture, they could use state-of-the-art methods for each. For example, the dynamics model (building on “Genie Envisioner”) can use heavy generative machinery (video diffusion, large attention-based networks) to ensure photo-realistic multi-step predictions. Meanwhile, the value model reuses an existing VLA network for understanding what constitutes progress, and fine-tunes it with RL-style objectives. This modularization means each model can be as big or computationally intensive as needed without hurting the other. It also makes training simpler: one can collect data for dynamics training cheaply by rolling out any policy, while supervising value training requires labeling progress or rewards but can be distilled from the same trajectories. Multi-View Predictions: By explicitly predicting multiple camera angles, the dynamics model can handle occlusions and 3D ambiguity. This is critical in cluttered scenes (like a bin of bricks). Many past works on video prediction only did monocular video, which rarely suffices for robotics. RISE’s success suggests that high-fidelity, multi-view prediction is worth the added complexity. Progress vs. Reward: Instead of trying to guess the numeric reward from an imagined state (which can be very sparse, especially in these tasks), RISE’s progress model learns a denser signal that tracks how well the task is going. For example, even if the brick-sorting task only gives a final “all sorted” reward, the value model can see intermediate states as partly better or worse depending on bricks already in place. Training with temporal-difference helps capture the true final reward shape, so the policy does not just greedily maximize local progress but ultimately aims for success. This nuanced labeling of rollouts likely gave much more signal per imagination step. Advantage-Conditioned Policy Learning: The way RISE updates its policy is clever. Instead of naive policy gradients, they frame policy learning as a conditional generation problem: “Given that this action had a certain advantage, learn to produce actions consistent with that condition.” This resembles recent diffusion/RL hybrids (like Diffuser or YOLO-RL) but is tailored to RISE’s setting. The result is a unified gradient stepping through both offline and online data. This allows seamless incorporation of the world-model rollouts. Crucially, RISE treats imagined rollouts similarly to real rollouts, just labeled with benefits from the world model. This means all shifts in policy belief come from these “simulated experiences”, making the improvement loop cleanly detached from real-robot trials. Connections to Prior Work. RISE draws from several threads in the literature:. Model-Based RL / World Models: As noted, RISE is in the spirit of model-based RL (Dreamer, PlaNet, World Models, etc.), but pushes it into the real world. Some recent work did robotics simulation with video models (e.g. RoboDreamer, recently in 2024, which also trained compositional world models). RISE’s authors don’t mention RoboDreamer explicitly (it was by Siyuan Zhou et al., I recall), but conceptually they share the idea of using learned video models for robot policy training. RISE distinguishes itself by operating on real robot data (no idealized sim), handling multi-view cameras, and introducing the progress/value component. Offline and Imitation Learning: RISE also relies on an initial offline dataset. The mention of "OpenPI" in the code suggests they use ideas from Open Policy Inference (OpenPI) which is an offline imitation/regression scheme. Relatedly, methods like Q-learning or diffusion policy (which use offline data and then do model-based finetuning) are conceptually similar. However, traditional offline RL often faces distribution shift issues; RISE mitigates this by anchoring to the original demonstrations via advantage-conditioning. Vision-Language-Action (VLA) integration: RISE fits into the trend of combining foundational vision-language models with RL. It shows that even without natural language commands (these tasks might not involve language), using a VLA backbone as the feature extractor is beneficial. By leveraging pretrained VLA weights in the value model, RISE can quickly learn good progress signals from relatively little data. This hybrid approach (pretrained perception + RL on top) has been successful in several recent works. Digital Twin / Simulation Generation: One can view RISE as an on-demand digital twin: it uses real data to create a simulator (“digital cousin”) of the robot environment. Similar ideas appear in projects like “WorldComposer” and “LTX-Video” which try to build environments from one-shot observations. RISE’s world model is learned from time-series data though, not from static panoramas. The key novelty is the closed-loop RL use: they aren’t just generating visuals for evaluation, but actively training the policy. Concurrent Trends: The ArxivLens page mentioned several related arXiv papers. For example, “World-Gymnast: Training Robots with RL in a World Model” and “VLAW: Co-Improvement of VLA Policy and World Model”. These suggest a growing interest in exactly this concept of co-training policies and models. RISE’s emphasis on compositional splitting and its strong real-robot results make it a standout. It will be interesting to see how it compares to these when they appear in pubs. Discussion and Implications. RISE is an exciting step in “imagination-based” robotics. It shows that powerful generative models (diffusion networks predicting future images, plus strong visio-language nets for value) can overcome the data bottleneck in RL. The self-improving pipeline means a robot can keep getting better long after initial deployment, without additional field data – it just imagines and learns. Implications for Embodied AI: This work hints that future robot learning might look more like “train a world model on offline data, then refine policies in simulation” even for real-world tasks. The traditional dichotomy (offline imitation vs. risky online RL) gets a third option: offline model learning with simulated online RL. As models get larger and data gets cheaper (e.g. collecting videos is easier than manual retuning each time), this approach could scale. It also integrates naturally with the trend of vision-language-action models: one could imagine combining instruction-conditioned models with a RISE-like loop to both follow instructions and improve through experience. Limitations and Next Steps: Of course, RISE depends on the fidelity of the learned world model. If the model is inaccurate, then the policy might learn to exploit its mistakes (a classic sim-to-real concern, here “model-to-real”). The paper addresses this by including real offline data in training and by focusing on short rollouts (so errors don’t compound too much). Still, for even more complex tasks or novel objects, we might need continuously updated world models or uncertainty estimations. Additionally, RISE currently uses fixed-length imagined rollouts and discretized advantages. There may be more sophisticated ways to incorporate uncertainty (e.g. Bayesian or ensemble world models) or continuous advantage signals. And the approach assumes the tasks have a clear notion of progress to learn; highly sparse tasks with no intermediate reward might still be tricky. Broader Impact: If robots can safely improve via imagination, it dramatically speeds up learning cycles. Imagine a warehouse robot that collects data on the job; once in a while, an engineer runs RISE with that data offline, and the robot comes back with a better policy without ever risking collision learning. Or personal assistants that tune themselves to their owners' preferences by imagination. On the flip side, we must ensure that the imaginary training doesn’t learn unsafe behavior that the simulator couldn’t catch. Comparison to Standard RL: RISE can be viewed as model-based RL, but in an extreme form: it’s fully model-based after initial data. By contrast, standard RL (even off-policy) typically still interacts with the environment for the bulk of data. RISE is on the model-only side of the spectrum. Interestingly, it still uses a form of off-policy learning (since the simulated rollouts are off-policy relative to the real-world data). This blurs the line between model-based and model-free: the actual policy update is done with supervised/behavior cloning losses, but the data is generated by a world model. Key Takeaway: The main takeaway is that a compositional world model can enable practical, safe, real-robot policy improvement. The splitting into a visual dynamics predictor and a progress value network is particularly clever: it mirrors human planning where one part of the brain imagines the scene, and another part judges whether it’s good or bad. RISE operationalizes this for robots. Looking forward, I expect to see more work on learning better world models (e.g. handling degraded sensors, novel objects) and on combining imagination with other signals (like language feedback or human-in-the-loop corrections). RISE itself could be extended by adding distributional value estimates, or by using the world model for planning as well as training. But as it stands, RISE already significantly advances embodied AI by letting robots be curious and self-improving in a safe, simulated realm. In conclusion, RISE showcases that robotics can harness “imagination” via learned world models to achieve self-improvement. This compositional approach yields large gains in difficult manipulation tasks, and it offers a blueprint for the next generation of robotic learning systems that blend real data with rich generative modeling. By breaking the reliance on risky trials, RISE opens a new paradigm for building dexterous, adaptable robots. Sources: The above is based on the paper “RISE: Self-Improving Robot Policy with Compositional World Model” (Yang et al., arXiv 2026) and accompanying materials, which detail the method and experimental results. The authors’ project site and code clarified additional engineering details. These results underline the promise of model-based imagination for real-robot learning.