Latest / Embodied AI 101 / Continual Robot Policy Learning via Variational Neural Dynamics

Transcript

Continual Robot Policy Learning via Variational Neural Dynamics · Embodied AI 101

• Continual Robot Policy Learning via Variational Neural Dynamics. Imagine a flying drone buffeted by gusty winds: its dynamics keep shifting as wind strength and direction change. Or a mobile robot that suddenly picks up a heavy payload, tipping its balance. In the real world, these disturbances are the norm, not the exception. Yet most learned controllers are trained once (often in simulation) and then frozen for deployment. This “train-once, deploy-forever” paradigm treats deployment as the end of learning – even though deployment is exactly when the robot experiences the most relevant data. When conditions change, a fixed policy simply performs poorly or takes forever to adapt. As the new paper “Continual Robot Policy Learning via Variational Neural Dynamics” observes, robots rarely operate with a single fixed dynamics model. External wind, varying payloads, battery drain, contact properties, or hardware wear can all secretly alter the equations of motion. Crucially, these factors are often hidden, non-stationary, and recurring: you might adapt to one wind condition and later encounter it again, only to have to relearn from scratch. The key insight of this work is that a robot should recognize when a familiar condition has returned and roll out a previously learned adaptation, rather than relearning everything from scratch. In other words, deployment should be the beginning of lifelong learning, not the end. To this end, the authors introduce a continual learning framework that builds a condition-aware dynamics model from real experience and uses it to incrementally improve the control policy. A physics-based model provides a nominal prediction, and a learned latent-conditioned neural residual captures the unknown effects (like wind or payload). A recurrent encoder ingests recent history of states and actions to produce a small latent code representing the current hidden condition. This code conditions both the learned dynamics and the policy. During training, the system samples a variety of latents (from a simple prior distribution) and uses a differentiable simulation to update the policy under those varied learned dynamics. At deployment, the sampled latents are “swapped in” with latents inferred online from real observations, so that the policy can immediately adapt by recognizing the current condition. In experiments, this approach (called Variational Neural Dynamics, or VND) lets a quadrotor adapt to shifting wind about five times faster than previous online adaptation methods, and slashes large-disturbance errors by roughly 50–65% over state-of-the-art baselines. In what follows, we’ll unpack the ideas behind Variational Neural Dynamics and see why it works. We’ll start by framing the challenge of changing environments and past adaptation methods, then delve into how VND actually builds and uses its latent dynamics model. The heart of the system is learning a low-dimensional latent representation of the current “hidden” condition, so that the controller can effectively “know what environment it’s in.” We’ll explain how that latent is trained to be both informative (for prediction) and sampleable (for new rollouts). Then we describe the policy training via differentiable simulation, and the overall continual learning loop that alternates between collecting new real data and re-training in sim. Finally, we’ll survey the impressive results on real quadrotor experiments (with changing wind and payload) and simulated tasks, illustrating why this approach holds promise for lifelong robotic learning. The Challenge: Adapting to Hidden, Changing Dynamics. Robots in the wild encounter more variability than in the lab. A drone might be flown at different altitudes or carry unknown payloads; a legged robot may face slippery patches or extra weight; a manipulator deals with objects of unknown mass and friction. These hidden dynamics often change over time and can even cycle back to prior conditions. Consider a drone exposed to wind: it might fly through a gusty valley, adapt, then re-enter the same valley later. One-shot adaptation methods would treat that as a fresh problem, wasting effort relearning the same correction. Ideally, the robot should recognize the return of a known condition and instantly apply the previously learned fix. Historically, the robotics community has tackled dynamics mismatch in a few ways. Domain randomization (in sim-to-real transfer) trains a single policy over many randomly sampled physical parameters, so it becomes robust to any of them. But this yields a one-size-fits-all policy that often plays it safe for all conditions, rather than excelling in any one. It is poorly suited when certain conditions recur and could be exploited. Adaptive control methods (e.g. -MPC with model adaptation) adjust parameters online, but typically assume a fixed model form and lack flexibility for arbitrary unknowns. Bayesian system identification or Neural dynamics identification approaches learn an environment model from data and then adapt the policy. Recent work in differentiable simulation has even shown you can refit a neural residual model from real flight data in just a few seconds, then update the policy via backpropagation. However, such fast adaptation is still “one-shot”: when the drone encounters slightly different wind later, it has to run the whole fitting process again. There is no memory of past adaptations. In summary, most existing methods either (a) produce one static policy robust to all conditions, (b) adapt on the fly continuously but forget older conditions, or (c) learn a parametric ensemble but can’t identify which member is active at any moment. None learn a distribution of dynamics from real data and use it to guide continual improvements. The novelty of VND is precisely to treat these hidden dynamics as a latent variable, learned from experience, and then use it to both train and deploy the policy in a condition-aware fashion. Variational Neural Dynamics: Modeling Unknown Conditions. The core of the VND approach is a learned dynamics model that factors into two parts: a known analytical physics prior and a neural residual that accounts for unknown effects. Crucially, this neural residual is conditioned on a latent code that represents the current hidden disturbance or change. The latent code comes from an RNN-based encoder that looks at the recent state-action history of the robot and infers which condition the robot is in. Concretely, let’s say $\mathcal{H}{t-C:t}$ is the sequence of states and actions (from time $t-C$ to $t-1$) from the recent past. A GRU (recurrent neural network) encoder $E\phi$ reads this history and outputs a latent vector $\mathbf{z}_t$. This $\mathbf{z}_t$ is meant to capture the “active condition” (wind direction/speed, payload, etc.) inferred from the observed behavior. Now, given the current state $\mathbf{s}_t$ and action $\mathbf{a}t$, a known dynamics model $f{\mathrm{prior}}(\mathbf{s}_t,\mathbf{a}t)$ (e.g. the rigid-body equations for the quadrotor integrated with RK4) would predict a nominal next state. The learned residual model $D\psi$ then outputs a correction based on $(\mathbf{s}_t,\mathbf{a}_t,\mathbf{z}_t)$. The predicted next state is. $$ \hat{\mathbf{s}}{t+1} = f{\mathrm{prior}}(\mathbf{s}_t,\mathbf{a}t) ;+; D\psi(\mathbf{s}_t, \mathbf{a}_t, \mathbf{z}_t),. $$. For the quadrotor, the residual $D_\psi$ is structured as a multi-layer perceptron producing corrections in position, velocity, and orientation (encoded as a rotation vector). The key point is that since the residual is conditioned on $\mathbf{z}_t$, the same nominal transition can be corrected differently under different inferred conditions. For example, a forward thrust might accelerate differently if the latent says “heavy payload” versus “strong tailwind.” This structure means we do not need a completely separate dynamics model for each environment; instead, one model flexibly adjusts via the latent code. Training this latent-conditioned dynamics model is done in a self-supervised way from real flight data. When the robot collects real trajectories under various unknown disturbances, each consecutive transition $(\mathbf{s}t,\mathbf{a}t)\to \mathbf{s}{t+1}$ yields a residual target $$ \mathbf{r}t = \mathbf{s}{t+1} \ominus f{\mathrm{prior}}(\mathbf{s}_t,\mathbf{a}t),, $$ where $\ominus$ just means the appropriate difference (for rotations it’s a rotation-vector difference). We associate each data transition with the latent $\mathbf{z}t$ produced by the same time’s context window. Then we train $D\psi$ to predict the residual: $$ D\psi(\mathbf{s}_t,\mathbf{a}_t,\mathbf{z}_t) \approx \mathbf{r}t, $$ using a robust Huber loss on the prediction error. Importantly, during training we share the latent $\mathbf{z}$ across multiple steps in the same short trajectory context (between $t-C$ and $t$). This forces $E\phi$ to produce a latent that explains all those transitions together. In effect, the encoder learns to produce the same code for a short burst of data that was generated under (approximately) the same hidden condition. The dynamics model then learns to use that code to predict the whole sequence of residuals. Because of this, the latent truly captures the common underlying disturbance. If the physics residuals ($\mathbf{r}_t$) differ, the encoder can only make them fit by choosing a different $\mathbf{z}$. So the system self-supervises the discovery of latent conditions: no explicit labels like “wind direction = 30°” are given, but the network finds structure purely from the prediction error. One challenge is that we also want to simulate using these latent-conditioned dynamics when training the policy. During policy learning, we will be in sim and must sample a variety of hidden conditions to make the policy robust. Thus, ideally the latent $\mathbf{z}$ should be easy to sample from some simple prior (like a Gaussian) and fed into $D_\psi$ to produce realistic dynamics. But if the encoder saw only real trajectories, its outputs might live in some unknown region of latent space. Random draws from $N(0,I)$ could end up in “dead” zones where the residual network behaves unpredictably. To avoid this mismatch, the authors introduce a latent distribution alignment. They collect the set of latents $q_\phi(\mathbf{z})$ produced by encoding many real contexts, and use Maximum Mean Discrepancy (MMD) to encourage that aggregate distribution to match a standard normal prior. In practice, they minimize $\mathrm{MMD}(q_\phi(\mathbf{z}), \mathcal{N}(0,I))$ as part of the loss. In other words, the encoder is nudged so that its outputs fill a region similar to a Gaussian ball. Now, sampling $\mathbf{z} \sim \mathcal{N}(0,I)$ during training is likely to produce latent codes similar to ones seen before, avoiding “out-of-support” dynamics. Notice that this is different from a classical VAE setup. The objective contains no per-sample KL term or reconstruction likelihood. Instead, the latent is shaped purely by the residual prediction loss and the (aggregate) MMD term. In effect, it’s an autoencoder-like approach with a statistical alignment: we want the latents to carry information (because they improve prediction) but also to cover a nice region. The authors even found it useful to add a small auxiliary loss to encourage decoding the context: a network tries to reconstruct the normalized state-action history from the latent, via $\ell_2$ loss. This further pegs the latent space to hold enough information. Ablation studies showed each piece matters: removing the FiLM-style conditioning on $\mathbf{z}$, or skipping the MMD alignment, or even using a non-recurrent encoder, all hurt performance significantly. In short, the latent space becomes a meaningful, sampleable representation of hidden conditions. Training the Policy with Differentiable Simulation. With the conditional dynamics model in hand, the next piece is to use it to train a robust policy. The trick is to alternate between real-world data collection (to refine the dynamics model) and simulated policy optimization (to improve the controller). Initially, one starts with a “base policy” trained in a simplified simulator (say, with no disturbances). The product pipeline then loops as follows:. Collect Real Data & Infer Conditions: Run the robot with the current policy, under whatever hidden dynamics are present. Record state-action trajectories. For each short context window, use the encoder $E_\phi$ to infer its latent $\mathbf{z}$. Store these (states, actions, latents, next states) in a buffer. Update the Dynamics Model: Using the accumulated data, retrain (or fine-tune) the encoder $E_\phi$ and residual network $D_\psi$ to better predict the real transitions via the loss we described earlier (including MMD regularization and associated terms). This step incorporates the new experiences into the latent-aware dynamics model. Improve the Policy in Sim: Now freeze $E_\phi$ and $D_\psi$. We want to update the policy $\pi_\theta$ so that it works well under the distribution of latent conditions the model has learned. To do this efficiently, we run differentiable simulated rollouts: in each rollout, we sample a latent $\mathbf{z}\sim\mathcal{N}(0,I)$ and hold it fixed for the whole trajectory. Each simulated environment (out of $N_{\rm env}$ parallel ones) gets its own random $\mathbf{z}^{(j)}$. The policy receives at each timestep both the current (normalized) observation $\mathbf{o}t$ and the latent $\mathbf{z}$ as inputs, i.e. $\mathbf{a}t = \pi\theta(\mathbf{o}t, \mathbf{z})$. The next state is then computed by our learned simulator: $$ \mathbf{s}{t+1} = f{\mathrm{prior}}(\mathbf{s}_t,\mathbf{a}t) ;+; D\psi(\mathbf{s}_t,\mathbf{a}t,\mathbf{z}). $$ Because these dynamics are implemented in a differentiable engine (for example, a library like Brax), we can backpropagate the reward through the trajectory. We then update the policy parameters $\theta$ via gradient ascent (or Adam) to maximize expected cumulative reward $\mathbb{E}{\mathbf{z},\mathbf{s}_0}\sum_t \gamma^t r(\mathbf{s}_t,\mathbf{a}_t)$. This end-to-end gradient-based training (policy optimization by backprop) is in principle much faster than model-free RL, because no sampling of random action noise is needed – we directly compute $\nabla_\theta$. But most importantly, we here performed a data-driven domain randomization: rather than hand-specifying wind angles or friction coefficients, we sample from the learned latent prior. Each draw from $\mathcal{N}(0,I)$ is a plausible (according to our model) disturbance. Thus the policy learns to handle the variety of conditions it saw in data, and even some interpolation beyond them. After policy optimization, the improved policy is used in the real world, generating new experiences under possibly unseen or recurring conditions. Then the loop repeats: real data refines the latent-conditioned model, and then we train the policy further using the updated model. In effect, the policy continues to learn from experience, rather than remaining static. Figure 1 in the paper summarizes this loop: training alternates between “learn latent dynamics” (fitting $E,D$ to real data) and “improve policy” (differentiable rollouts with sampled $\mathbf{z}$). Meanwhile, at deployment, the encoder runs online: every 20 milliseconds or so (50 Hz), it ingests the latest state-action window and computes $\mathbf{z}_t$, feeding that to the policy. In practice, this means the policy is always stepping with the best guess of which condition it’s in, even without any “privileged” disturbance labels. Why Conditional Latents Help. This framework offers several complementary benefits. First, when a known disturbance recurs, the policy simply sees the same latent and behaves as it did before, so it recovers immediately. The drone example is telling: when a familiar wind direction reappeared, the VND-based policy re-adapted in about 1 second by inference alone, whereas a baseline that had to re-fit a residual needed around 5 seconds. That’s a five-fold speedup, and it means the drone never had to “forget” and relearn the previous wind condition. Second, because the controller is trained on a broad distribution of conditions (via the sampled latents), it becomes much more robust than a single-condition policy. The authors benchmarked on tasks like quadrotor landing and hovering under various disturbances, as well as manipulation tasks. In all cases their method was close to an oracle that did know the true disturbance ahead of time, and significantly better than baselines. For example, in quadrotor trajectory tracking under mixed winds, residual-adaptation and a one-condition policy had ~0.312 m error, while the VND method got it down to ~0.045 m. In fact, on large disturbances in hover and tracking, VND cut the errors by 53–65% compared to state-of-the-art online adaptation methods. The key was that sampling from the learned latent distribution exposed the policy to the right variety of scenarios during training. Third, VND continually improves across flights. The paper shows that as the drone repeatedly flies, collects data, and updates, its performance keeps going up. In one real-world test, the tracking error on a figure-eight trajectory started around 41 cm with the base policy and dropped to ~9 cm after several updates. This demonstrates that the strategy is not just a transient “apply correction then forget”; data from each flight get folded into the dynamics model and help refine the policy for future flights. In effect, the robot is learning new “skills” (how to handle these recurring conditions) without erasing the old ones. The learned latent space itself exhibits nice structure. In simulation tests with varying wind, the embedding vectors $\mathbf{z}$ for different wind directions formed clear clusters – even though the network was never told the wind angle explicitly. In other words, the encoder discovered a representation aligned with the physical variations. Different magnitudes and directions of wind led to distinct regions in the latent space. This emergent clustering suggests that the latent really captures meaningful features of the disturbance. It also explains why the policy can react appropriately in each case: the latent is acting like a “mode” indicator for the environment. Ablation studies confirmed that each component of the design is important. Dropping the FiLM-style conditioning (where $\mathbf{z}$ modulates the residual net) or skipping the MMD loss significantly worsened multi-step prediction errors and policy performance. Even replacing the RNN-based history encoder with a simple feedforward network degraded results, confirming that capturing temporal patterns in the recent trajectory is crucial for identifying the condition. Thus, the combination of (i) residual dynamics on top of physics, (ii) history-based latent inference, and (iii) a regularized, sampleable latent space is what unlocks the success – not simply bigger networks. Comparison to Related Methods. It’s helpful to contrast this approach with some familiar baselines. Domain randomization, for instance, does sample many disturbances in training, but it usually ignores the identity of any particular condition at runtime. Here, instead, the latent serves as an identified context. Another recent line of work fits a residual model online and updates the policy via differentiable sim in a few seconds. Those methods (e.g. LOTF 2025) are indeed sample-efficient, but as noted, they adapt reactively each time. VND adds a recognition loop – a memory of conditions – so it doesn’t start from zero if the environment reverts. The idea of latent conditioning has appeared elsewhere too. For example, Rapid Motor Adaptation (RMA) for legged robots used latent “extrinsics” predicted from history to adjust policies on uneven terrain. The twist in VND is that the latents are learned entirely from data without privileged labels: RMA assumed known simulator parameters during training, whereas VND simply uses flight data. Similarly, some vision-and-language models learn embeddings of tasks or terrains, but they often require explicit modes at training. VND’s unsupervised discovery of the latent space is closer in spirit to meta-learning approaches like RL$^2$ or Probabilistic Embeddings, except that VND goes one step further to use the latent in retraining the policy via a model. It also differs from pure meta-RL in that VND keeps improving a single policy rather than resetting. Finally, it’s worth noting a limitation mentioned by the authors: in their experiments they used high-quality state estimates from motion capture. In a practical scenario with noisy onboard odometry, measurement errors might confuse the encoder, which could mistake noise for real dynamics changes. In other words, some care would be needed so that the latent truly reflects physics and not sensor drift. This is a general issue for any system identification, but it’s especially important here since everything hinges on the latent inference being accurate. Future work might incorporate learning to filter noise, or jointly estimating when a change is due to sensors versus environment. Experiments: Quadrotor Flight Under Changing Wind. The researchers evaluated VND extensively on both simulated scenarios and real-world quadrotor flights. The star experiments involved a small agile drone (based on the Agilicious platform) tracking a figure-eight trajectory while subjected to controlled wind gusts. All computation (encoder, policy, learning loop) ran off-board, communicating with the drone at 50 Hz. Recovering from recurring wind. In one test, the drone would start in a calm environment, then strong wind would be turned on from some direction once it reached steady flight. They then measured how quickly the controller’s tracking error returned to its pre-wind level. If the wind direction was one that VND had already seen in prior flights, the learned policy took only about 1 second to recover. That is, as soon as the recurrent encoder recognized the familiar disturbance in the recent history, it updated its latent and the policy instantly compensated. By contrast, a state-of-the-art “Online Residual Fitting” method (LOTF 2025) had to perform a model fitting procedure from scratch, which took on the order of 5 seconds before tracking improved. In other words, VND was roughly five times faster to re-stabilize the drone after the same shift. This demonstrated the advantage of “recognition” (through the latent) versus “re-learning.”. Continuous refinement. In another experiment, they simply flew the drone multiple times on the figure-eight, with wind disturbances that varied each flight. After each flight, the collected data were used to update the dynamics model and then the policy. Over time, the controller improved – tracking error monotonically decreased. Initially the base policy had about 41 cm of root-mean-square position error. After a handful of updates (iterating over wind conditions), the error dropped to about 9 cm. This is a huge improvement, indicating that each bit of data was accumulated to make a better policy. Crucially, the improvements persisted: flights after refinement stayed good even under old wind conditions. This is true continual learning: the knowledge didn’t vanish. Adapting to unexpected hardware changes. To stress-test the method, they even cut one propeller blade off the drone, introducing a drastic change in actuation that was not in any simulation randomization. In this extreme case, the base policy (trained without this issue) had a whopping 52 cm error. After the first round of adaptation (collecting data and refining), that error dropped to 24 cm; a second iteration got it to 12 cm. So even though this scenario was far out-of-distribution, the same loop managed to quickly improve control. The latent dynamics model had to learn essentially new “engine” dynamics, but because the framework continually re-trained, it handled it. This shows VND’s ability to incorporate genuinely novel dynamics online. All these outcomes together paint a picture: the VND framework can leverage real flights to successively expand what the policy handles. When conditions are familiar, it recalls and applies fixes immediately. When conditions are new, it gradually learns them and folds them into its latent model and policy. In both cases, it greatly outperforms methods that either do no real learning (a fixed robust controller) or do only one-off adaptation. Beyond Quadrotors: Diverse Tasks and Generalization. While the real-world tests focused on a quadrotor in wind, the authors also ran extensive simulations on other tasks to highlight generality. They tried quadrotor landing under winds, vision-based hovering, trajectory following, and even two manipulation tasks (like pushing a box with unknown mass, or following an end-effector trajectory with changing payload). In all scenarios, the VND-trained policies almost matched an “oracle” policy that was trained knowing exactly the conditions during test. By contrast, simpler approaches faltered under large shifts. For instance, the box-pushing success rate of the baseline degraded substantially if the box mass varied, whereas VND remained high. In aggregate plots, the VND method had the lowest average error across tasks – it was consistently closest to the ground-truth condition-aware controller and far ahead of baselines like single-disturbance training or naive latent-adaptation. These results underscore an appealing property: learning a latent distribution from real data and sampling it during training leads to much better generalization. The policy effectively anticipates plausible variations even beyond what was explicitly experienced, because the latent prior fosters smooth interpolation. In contrast, a policy trained on only one disturbance or one random seed can be brittle. VND’s latent approach marries the strengths of data-driven robustness with the efficiency of sim-based optimization. Conclusion. Continual Robot Policy Learning via Variational Neural Dynamics presents a compelling vision for how robots can keep learning after deployment. By pairing a physics-based model with a learned latent-conditioned residual, and by structuring training around an inferred latent space, the system learns to both recognize and prepare for changing dynamics. It turns deployment data into persistent policy improvements, rather than into fleeting calibrations. The core idea – treat hidden environment factors as latent variables, align them to be sampleable, and condition both dynamics and policy on them – is broadly applicable. It could be extended to other domains (soft robotics, underwater vehicles, self-driving cars, etc.) where unmodeled disturbances appear. Of course, challenges remain. Real robots often have noisy, delayed perception, and identifying true physics changes amid sensor noise is tricky. Also, differentiable simulators are still an emerging technology and scaling them to very high-fidelity models can be computationally heavy. Yet the promise is clear: a new paradigm where mobile robots, drones, and manipulators continue to refine their skills on the job, leveraging latent awareness of the world. As the authors demonstrate, this is more than just adaptation – it is continual learning of better controllers through experience.