Latest / Embodied AI 101 / Replay Is Not Reality: Embodied RL When Both Policy and Physics Shift

Transcript

Replay Is Not Reality: Embodied RL When Both Policy and Physics Shift · Embodied AI 101

• Replay Is Not Reality: Embodied RL When Both Policy and Physics Shift. Embodied Learning under Policy and Dynamics Shifts • Yu Luo, Lei Lv, Fuchun Sun, Huaping Liu • Tsinghua University; Huawei Technologies’ 2012 Laboratories; Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University • National Science Review • 2026. The real robot never gives you one clean dataset. Imagine the replay buffer behind a deployed manipulation system. Some transitions came from simulation. Others came from yesterday’s controller. A subset was collected after the gripper pads wore down. Another batch used a different payload, camera exposure, control frequency, or safety filter. The most recent few thousand transitions reflect the robot as it exists right now—but they are mixed with millions of older samples gathered by different policies under different physics. This is not an unusual corner case. It is arguably the normal data regime for embodied learning. Standard off-policy reinforcement learning is designed to tolerate one part of this heterogeneity: the behavior policy that generated a transition may differ from the policy currently being optimized. But that tolerance usually rests on a hidden contract—the transition dynamics are assumed to remain fixed. Domain adaptation and non-stationary RL, meanwhile, often focus on changing dynamics while treating policy-induced distribution shift as a separate concern. Embodied Learning under Policy and Dynamics Shifts argues that these should not be viewed as independent problems. The policy determines which states and actions are visited, while the physical dynamics determine what happens next. Once both vary, the replay buffer’s joint distribution over states, actions, and successor states can differ fundamentally from the distribution relevant to the robot operating now. The paper calls for correcting that complete discrepancy through transition occupancy matching, and develops Occupancy-Matching Policy Optimization, or OMPO, around this principle. A publication note is important here. Oxford Academic published the article online on August 19, 2026, and currently labels it an accepted manuscript, meaning copyediting and final typesetting may still introduce changes. OMPO’s methodological lineage goes back to a six-author paper presented as an oral at ICML 2024. The 2026 article retains the central transition-occupancy framework but advertises an expanded formulation, a sign-free reward link, a distributional critic, multimodal encoding, and additional evaluation domains. I will therefore distinguish between claims made explicitly by the new accepted-manuscript abstract and technical details available in the open ICML version. The hidden stationarity contract in replay-based RL. In a conventional Markov decision process, the transition kernel answers a simple question: given the current state and action, what distribution of next states should we expect? Most reinforcement-learning derivations treat that kernel as fixed while the policy changes. Experience replay then looks benign. A transition generated by an old policy is still a valid sample of the same physical mechanism. The old controller may have selected a particular action more often, but conditional on the state and action, the observed successor remains evidence about the current environment. That reasoning breaks when the environment itself changes. Suppose an arm applies the same commanded torque before and after picking up a heavy tool. Or a quadruped’s nominal joint targets remain unchanged while the ground friction falls. The stored transition is no longer merely off-policy. It is off-dynamics: the relationship between action and consequence has changed. Now combine the two effects. Old policies visited different states, selected different actions, and interacted with different physical conditions. Learning indiscriminately from those transitions can distort both policy evaluation and improvement. A critic bootstraps through successor states whose frequency and meaning do not match present deployment. The actor then optimizes against that critic, generating fresh transitions that can amplify the original error. OMPO organizes this compound problem into three practical regimes:. Policy shift under stationary dynamics. The environment remains fixed, but replay contains data from many historical policies. Policy shift during domain adaptation. Most data comes from a source environment, while a small amount reflects target dynamics. Policy shift under non-stationary dynamics. Both the controller and the environment evolve throughout learning. Every regime contains policy shift. What changes is the source and timescale of the dynamics discrepancy. The ICML formulation emphasizes that one algorithm can address all three by changing how recent and historical data are routed into its buffers, rather than changing the policy-optimization rule itself. This framing is useful because “off-policy” is often treated as a property of an algorithm—SAC is off-policy; PPO is on-policy. OMPO instead asks whether the distribution used for an update corresponds to the current policy in the current world. That question remains relevant even for nominally off-policy algorithms, because their guarantees and heuristics do not automatically cover arbitrarily mixed dynamics. Why the right object is a transition occupancy. An occupancy distribution is the discounted frequency with which a policy visits different parts of an environment. State occupancy records how often states occur. State-action occupancy records how often the agent takes particular actions in particular states. OMPO goes one step further and works with transition occupancy: the frequency of complete state, action, next-state triples. That additional successor state is what allows one statistical object to expose both forms of shift. If the policy changes, the state and action portions of the distribution change. If the dynamics change, the conditional distribution of the next state changes. If both change, the entire transition triple moves. Rather than first diagnosing whether a discrepancy was “caused” by policy or physics, OMPO attempts to match the joint transition distribution relevant to the current deployment. Consider a pushing task. An old conservative policy might approach the object slowly and contact it near the center. A new policy might arrive with greater velocity and contact off-center. That is policy-induced shift. If the table surface also becomes more slippery, the same contact generates a different successor state. That is dynamics-induced shift. Looking only at actions misses the physical change; looking only at state visitation confounds why the state distribution moved. The transition triple contains both effects. The framework conceptually maintains two distributions:. A global distribution represented by the large historical replay buffer. A local distribution represented by a small buffer of fresh transitions collected by the current policy under current or locally consistent dynamics. The local buffer anchors learning to immediate physical reality. The global buffer provides scale, diversity, and sample efficiency. OMPO’s job is not to discard history, but to estimate how historical transitions should influence learning given what the robot has recently observed. This point distinguishes occupancy correction from simple recency weighting. A timestamp only says when a transition was collected. It does not say whether it remains representative. A six-month-old transition may still match current dynamics, while one collected an hour ago under a different payload may not. A learned transition discriminator can, in principle, judge similarity from the actual contents of the transition rather than its age alone. There is, however, an unavoidable support condition beneath this story. Reweighting can rehabilitate useful historical data, but it cannot invent transitions that neither the global nor local buffer contains. If current target behavior lies entirely outside historical support, the agent still needs new interaction, a model capable of extrapolation, or an exploratory policy that reaches the missing region. From transition matching to an optimization objective. The ideal objective would evaluate the current policy under the current target dynamics. Unfortunately, that is precisely the distribution we do not have enough samples from. If we did, there would be less need to exploit the heterogeneous replay buffer. OMPO therefore constructs a surrogate objective. In conceptual terms, the policy receives credit for task reward while paying a penalty when the transition occupancy implied by its behavior disagrees with the relevant data distribution. The original ICML derivation separates discrepancies associated with policy shift from a transition-density-ratio term that captures the combined effect of policy and dynamics changes. The 2024 formulation applies a logarithm to the reward inside this surrogate. Because an ordinary logarithm requires positive inputs, the paper explicitly identifies strictly positive rewards as a limitation. The 2026 accepted-manuscript abstract says the revised approach uses a sign-free logarithmic link and is compatible with arbitrary reward structures. The exact accepted-manuscript derivation is not exposed in the current HTML, but the intended improvement is clear: retain the mathematical convenience of the logarithmic link without requiring every environmental reward to be positive. Three transformations turn the conceptual objective into something trainable. First, OMPO estimates transition-density discrepancies with a classifier. The classifier receives a state, action, and successor state and tries to distinguish fresh local transitions from historical global transitions. At the optimum, its log odds provide an estimate of the density ratio between those distributions. This is the same general trick that connects binary classification to likelihood-ratio estimation in adversarial learning and domain adaptation. Second, OMPO imposes the Bellman flow constraint. Not every arbitrary weighting over transitions can be generated by a valid policy. Probability mass has to flow consistently: occupancy at a state comes from the initial-state distribution and from discounted transitions entering that state. The derivation introduces a value-like function as a Lagrange multiplier enforcing this conservation law. That multiplier becomes the algorithm’s critic. This is a deeper point than merely “adding a discriminator reward to SAC.” The critic arises from dualizing the constraints that define valid occupancy measures. Policy learning, value estimation, and distribution correction are therefore coupled by the same optimization problem. Third, the derivation uses a Fenchel conjugate to remove an expectation under an inaccessible distribution—roughly, what the current policy would do under dynamics represented only indirectly in the buffer. The resulting min-max objective can be estimated with samples from accessible replay data. This step closely connects OMPO to the DICE family of stationary-distribution-correction methods, which similarly use convex duality to learn from arbitrary behavior data without explicitly reconstructing the behavior policy. In plain language, the mathematics performs a useful conversion:. Start with an impossible request—optimize the current policy under the current world using a replay buffer collected by old policies in old worlds—and turn it into an actor, a critic, and a transition discriminator that can all be trained from sampled mini-batches. The result is a two-timescale optimization. The critic approximately solves the inner distribution-correction problem, while the actor improves the outer policy objective. The classifier continuously refreshes the estimate of how local reality differs from global history. What an OMPO update actually looks like. The practical system has three data structures in the ICML implementation: a global replay buffer, a small local replay buffer, and an initial-state buffer used by the dual objective. The current policy interacts with the environment, fresh transitions enter the local buffer, and historical transitions remain available globally. A simplified training cycle looks like this:. Collect fresh data from the current policy under the currently relevant dynamics. Train the transition discriminator using local samples as one class and global samples as the other. Convert its output into a transition-discrepancy correction. Update the critic using the corrected dual objective. Update the actor against that critic. Periodically merge local experience into global replay and reset or refresh the local window. The same machinery serves all three shift regimes. In stationary off-policy learning, local and global data share dynamics but reflect different policy generations. In domain adaptation, the global buffer can hold abundant source-domain experience while the local buffer holds scarce target-domain interaction. In a non-stationary environment, the local buffer acts as a moving snapshot of the current regime while global replay preserves the longer history. The local buffer’s size is not a minor implementation detail. A very small window tracks changes quickly but gives the discriminator a noisy estimate. A large window stabilizes classification but blurs together multiple policies or dynamics regimes, defeating its role as a representation of the present. The paper’s ablations show exactly this bias-variance trade-off: smaller buffers are fresher but less stable; larger buffers are easier to fit but can pull optimization toward stale local optima. The 2026 article reports two notable architectural additions. The first is a distributional critic. This should not be confused with the transition-distribution discriminator. The discriminator estimates whether a transition resembles local or historical data; a distributional critic models a distribution over possible returns rather than only their mean. That can be valuable when physical variation makes outcomes multimodal or heavy-tailed—for example, when nearly identical contacts either succeed cleanly or trigger a failure mode. The second is a multimodal encoder, which moves the framework toward contemporary embodied agents whose state is assembled from vision, proprioception, force signals, language-conditioned task information, or other modalities. The new abstract says this encoder, the distributional critic, and the local buffer together let the agent anchor large historical datasets to immediate physical context. It does not yet provide enough publicly parsed detail to determine exactly which modalities share parameters or how representation learning is stabilized against the adversarial discriminator. What the experiments establish. The 2026 abstract reports evaluation across MuJoCo locomotion, DeepMind Control Suite, Meta-World, and high-fidelity Panda manipulation. It claims consistent gains over specialized baselines in stationary, domain-shifting, and non-stationary settings. This is broader than the original study, particularly through the addition of DM-Control and the revised Panda evaluation. Because Oxford currently exposes the article as a PDF-only accepted manuscript, the abstract supports the scope and headline conclusion but not a detailed audit of every task, seed count, or effect size. The open ICML study provides a clearer view of the experimental logic. Its core evaluation uses Hopper, Walker2d, Ant, and Humanoid across three shift settings, producing twelve principal locomotion experiments. For domain adaptation, some tests substantially alter morphology or mechanics: the Hopper and Walker2d variants double torso and foot sizes, while the Ant and Humanoid variants double gravity and introduce headwind. Additional tests vary physical parameters from roughly half to three times their nominal values. The stationary experiments matter because they ask whether OMPO is useful even before dynamics begin changing. The results indicate that correcting the occupancy discrepancy created by successive behavior policies can improve exploration efficiency and training stability relative to standard actor-critic baselines. Removing the discriminator causes the method to behave more like ordinary SAC, suggesting that the gains are not solely due to a different critic loss or network architecture. In domain adaptation, the framework uses fresh target transitions to identify which portions of source replay remain compatible with target physics. In non-stationary environments, it is compared with methods such as CEMRL, which infers latent changes with a mixture model, and CaDM, which learns a context-aware dynamics model. OMPO’s attraction is that it does not require specifying the form of the dynamics context or fitting a complete forward model; it only needs enough local data to discriminate present transitions from the historical mixture. The manipulation results show where the correction can be practically significant. In the 2024 experiments, OMPO reaches a reported 68.5 percent success on Coffer-Push versus 15.8 percent for SAC, 84.2 percent on Hammer versus 15.4 percent, and 93.4 percent on Drawer-Open versus 64.3 percent for the stronger conventional baseline. On sparse Panda reaching, however, SAC slightly exceeds OMPO—94.5 versus 93.1 percent—so the table is not a story of universal dominance on every individual task. The larger gains occur on manipulation problems where stochastic contacts and difficult exploration make replay mismatch especially consequential. The ablations also reveal a second sensitive parameter: the weight placed on distribution correction. If it is too small, the algorithm approaches the no-discriminator variant and fails to correct meaningful shift. If it is too large, matching overwhelms environmental reward and prevents the policy from pursuing task improvement. This confirms that OMPO is balancing two objectives rather than discovering a free correction with no optimization cost. Finally, the original paper reports particularly strong performance when OMPO is combined with domain randomization. That combination is intuitively coherent. Domain randomization broadens the support of source experience, while occupancy matching uses local evidence to determine which parts of that broad support resemble the target. One provides diversity; the other provides selectivity. How OMPO differs from adjacent approaches. OMPO sits at the intersection of several mature research threads, but it is not identical to any of them. Stationary distribution correction, including the DICE family, estimates how off-policy occupancy should be reweighted when behavior data comes from an arbitrary policy mixture. OMPO borrows the duality machinery but enlarges the object being matched from stationary state-action statistics to transitions that can expose changing dynamics. In that sense, it asks DICE-style correction to operate across worlds, not just across controllers. Classifier-based off-dynamics adaptation is represented by methods such as DARC. DARC trains classifiers to distinguish source and target transitions, then modifies the source-domain reward so that source behavior looks more target-like. This is closely related to OMPO’s transition discriminator. The difference is largely one of scope and derivation: DARC is organized around source-to-target dynamics transfer, whereas OMPO embeds density-ratio estimation inside an occupancy-constrained policy-optimization framework that also covers ordinary policy shift and ongoing non-stationarity. Context-conditioned and meta-RL approaches attempt to infer a latent variable describing the current environment—mass, friction, morphology, task identity, or some learned equivalent—and condition the policy or dynamics model on that context. Such methods can adapt quickly when shifts lie on a reusable latent manifold. OMPO does not have to name or reconstruct that manifold. Its local discriminator asks the weaker question of whether transitions look current. The trade-off is that explicit context may generalize compositionally, while implicit density ratios can be more reactive but less interpretable. Robust MDP approaches optimize performance under a set of plausible transition kernels, often protecting against worst-case dynamics. That is attractive when an uncertainty set can be specified and target interaction is unavailable. OMPO instead adapts toward observed local dynamics. Robust optimization is precautionary; occupancy matching is evidential. Recent off-dynamics theory shows how uncertainty-set design can provide provable guarantees, but such methods currently address a different regime from deep multimodal continual adaptation. Non-stationary RL theory often studies dynamic regret, bounded variation budgets, restarts, or change-point detection. Those formulations clarify what is statistically possible but may presume episodic structure or discrete changes that are difficult to map directly onto high-dimensional robot learning. OMPO is a pragmatic deep-RL answer: define “now” through a local buffer and continually compare it with history. The emerging ODRL benchmark is a useful caution here. Its comparative study reports that no existing off-dynamics method has universal advantages across the variety of tested shifts. OMPO’s unification is therefore best understood as a strong organizing principle and algorithmic interface—not evidence that all forms of embodiment shift have been solved. The limitations that matter for robotics. The first limitation is density-ratio reliability. Classifier-based ratio estimation works best when local and global distributions overlap. If the classifier becomes perfect immediately, that may indicate the target regime is so far from replay support that its logits are saturated and its ratio estimates unusable. A high classification score is not necessarily good news. The second is the definition of locality. The correct buffer horizon depends on how rapidly both policy and dynamics change. A slowly wearing actuator might support a large local window. Payloads that switch every episode require aggressive refresh. If two regimes alternate, a simple recency window may represent neither one cleanly. A mixture of local experts or explicit change-point model could be necessary. Third, transition triples assume that the chosen representation is approximately Markov. In real robotics, hidden temperature, backlash, contact mode, object compliance, or delayed controller state may influence future behavior without being identifiable from a single state-action-next-state tuple. Multimodal encoders help only if the relevant latent information is observable in their inputs. Sequence encoders or recurrent context estimation may still be required. Fourth, the framework primarily addresses policy and transition-dynamics shift. A changing reward function is a different problem. If operators redefine success, safety costs change, or the task itself moves, matching physical transitions cannot determine which behavior should now be preferred. Fifth, OMPO needs fresh target interaction to populate the local buffer. That is reasonable in simulation and some online-control settings, but potentially expensive or unsafe on hardware. The algorithm can make better use of a small target stream; it does not eliminate the need to collect that stream. The optimization also has more moving parts than SAC: a discriminator, a dual critic, an actor, multiple buffers, and a correction coefficient. Shared encoders can create additional interference because representation updates that help transition classification may hurt value prediction, or vice versa. Careful update ratios, target networks, calibration, and encoder-gradient routing are likely to matter. Finally, the currently public code corresponds to the 2024 ICML implementation. Its repository identifies that earlier paper, contains only a few commits, and shows the main branch last updated on May 29, 2024. It should not be assumed to implement the 2026 sign-free link, distributional critic, multimodal encoder, or expanded benchmarks. A practical recipe for adapting the idea. OMPO is most compelling when three conditions hold: you possess a large heterogeneous replay dataset, you can collect a comparatively small stream from the current deployment, and state, action, and reward semantics remain sufficiently consistent across the data sources. For a new robotic system, I would treat transition occupancy matching as a module rather than immediately rebuilding the entire training stack. Start with a strong off-policy baseline and preserve two replay views: a large historical buffer and a local ring buffer. Train a discriminator on encoded state-action-next-state tuples. Monitor not only its loss but calibration, logit saturation, and performance as a function of replay age and known dynamics parameters. If the discriminator cannot identify an intentionally introduced dynamics change, it is unlikely to provide useful correction. Next, compare the full correction against simpler controls: uniform replay, exponential recency weighting, source-versus-target reward correction in the style of DARC, domain-randomized SAC, and a context-conditioned policy. The important question is not whether OMPO beats an unmodified baseline, but whether its transition-level correction adds value beyond cheaper ways of emphasizing recent data. Choose the local-buffer horizon in units of physical change rather than gradient steps. Estimate how many transitions are needed to identify a payload, friction, or actuator regime, then test windows around that scale. If the environment changes faster than the discriminator can be trained, the correction will always lag reality. For visual or multimodal systems, consider separate projection heads for policy learning and density-ratio estimation. A shared backbone may be useful, but allowing the adversarial classifier unrestricted control of the representation can encourage features that separate domains without preserving task-relevant geometry. Finally, evaluate adaptation as a time series. Report performance immediately after a shift, target interactions required to recover, asymptotic return, variance across shifts, and whether performance on recurring old regimes is retained. A final average return can hide precisely the transient failures that matter in continual embodied deployment. The same design could extend naturally to vision-language-action post-training. A massive heterogeneous robot dataset becomes global replay; a few site-specific rollouts become the local anchor; and the discriminator estimates which historical transitions remain relevant to the current robot, workspace, and controller. But high-dimensional action chunks, partial observability, and sparse target interaction make support estimation even harder, so the transition representation would likely need temporal context rather than isolated triples. The larger lesson. The strongest contribution of this work is not a particular discriminator architecture. It is the claim that policy shift and dynamics shift should be corrected at the level where they actually meet: the distribution of physical transitions. That perspective changes how we think about replay. A buffer is not simply a warehouse of valid experience. It is a mixture of occupancies produced by different decision rules in potentially different worlds. Its value depends on how those occupancies relate to the policy and embodiment operating now. OMPO’s local-global design offers a compelling pattern for continual embodied learning: retain the breadth of history, maintain a small empirical model of the present, and learn how to connect them rather than choosing one or the other. The 2026 revision pushes that pattern toward arbitrary rewards, multimodal observations, distributional value prediction, and broader robotic benchmarks. The remaining challenge is making the correction trustworthy when overlap is weak, observations are partially informative, changes are abrupt, and target interaction is costly. But as an organizing principle, transition occupancy matching is unusually well aligned with the reality of deployed robots. The robot’s policy changes. The robot’s body changes. The environment changes. A learning system that remembers the past must also know when the past no longer describes the present.