Latest / Embodied AI 101 / SRL: Combining SLIP Model and Reinforcement Learning for Agile Robotic Jumping

Transcript

SRL: Combining SLIP Model and Reinforcement Learning for Agile Robotic Jumping · Embodied AI 101

• SRL: Combining SLIP Model and Reinforcement Learning for Agile Robotic Jumping. Jumping is a core skill for agile robots: whether to vault obstacles in a disaster zone or quickly traverse uneven terrain, a good jumping gait greatly expands a robot’s capabilities. Traditional biomechanics offers the Spring-Loaded Inverted Pendulum (SLIP) as a simple template for running and hopping: essentially a point mass bouncing on springy legs. The SLIP model “captures the center of mass dynamics observed in running animals and has become a basic gait template in biomechanics and robotics”. A well-tuned SLIP can produce energy-efficient, animal-like accelerations and decelerations in legged systems. However, the simplicity of SLIP is also its limitation: it assumes ideal springs and point-foot contacts, so “its performance degrades on irregular terrain”. By contrast, deep reinforcement learning (RL) can in principle discover highly dynamic jumping policies adaptive to rough ground, since it learns from trial and error. In practice, pure RL is often data-hungry and unstable for such tasks. Training a blind RL agent from scratch to learn a jumping gait in simulation may take millions of trials, and transferring that policy to real hardware is fraught with risk and sample inefficiency. The idea of SRL is that SLIP and RL have complementary strengths. The SLIP controller is a cheap, physics-informed baseline that “naturally encodes biologically plausible hopping motions,” while RL provides online adaptation and feedback on deviations from the idealized model. By blending them, we can get the best of both worlds: a physically grounded motion plan and a learned correction to handle modeling errors and variability.†. For example, ETH Zurich researchers recently showed that a quadruped ANYmal can learn parkour by combining model-based control with learning (the robot “can negotiate difficult terrain” by blending classical planning with learned policies). Similarly, SRL follows this hybrid approach. As the authors put it, they “propose Spring-loaded Reinforcement Learning (SRL), which integrates SLIP-based feedforward control signals with RL-driven real-time feedback, enabling continuous optimization of robotic jumping”. In practice, SRL runs a SLIP model to generate a baseline jumping motion, then adds a neural-network policy that tweaks this baseline during execution. This hybrid loop achieves much more stable jumps with far less training time than a pure RL approach. The SLIP Model as a Jumping Primer. Before diving into SRL, let’s recall how the SLIP model works. Essentially, we treat a robot’s body as a point mass (the center of mass, or CoM) atop a massless leg spring. When the robot is in contact with the ground, the leg acts like a spring-damper: it compresses under load and then pushes off, storing and releasing energy. As Hartmut Geyer explains, the SLIP “describes gait with a point mass rebounding on springy legs,” and this simple abstraction “has become a basic gait template in biomechanics and robotics”. In other words, SLIP captures the core spring-mass oscillation of animal running or hopping: when you land, your leg spring compresses (storing energy), and then it releases to propel you upward in flight. Since the dynamics are simple, we can even derive analytical solutions or periodic orbits for SLIP, making it a handy offline planner. Because of its simplicity, many robot control strategies rely on SLIP-like patterns. For instance, Li et al. (2024) built a bipedal jumper whose mechanics are tuned to closely match an ideal SLIP model; in their words, “the SLIP model is widely utilized in robotics due to its passive stability and dynamic properties resembling human walking patterns”. By designing the robot around SLIP, they avoid model mismatch. Others use SLIP purely as a planning tool: at each jump the controller commands a desired spring force based on the current extension. Typically one can approximate the vertical dynamics by a simple spring law, e.g. $F = -k,\Delta L - c,v_m$ (spring minus damping force; see next section). An inverse kinematics or optimization then converts the desired CoM trajectory into joint trajectories for the legs. SLIP is powerful, but again, it assumes perfectly elastic legs and rigid contact. In real robots, the spring behavior is only approximate, legs have limits, and the ground might be uneven or slippery. A SLIP-only controller will often fail outside very flat, controlled environments. That’s why the SRL authors say SLIP can provide an excellent baseline but needs help in practice. We need something to “compensate for unstructured terrain and modeling errors.”. Learning to Jump: The Limitations of Pure RL. On the other hand, a pure RL solution is fully model-free: you give the robot (or its simulator) motor commands and let it learn from scratch how to jump. Recent breakthroughs have shown that RL can indeed master complex legged skills (running, leashless walking, even parkour-style leaping around obstacles). For example, an RL policy controlls ANYmal to jump over gaps or climb stairs by trial and error and visual feedback. But there are very real drawbacks. Training an RL controller for a high-speed jumping motion typically requires tens of millions of simulation steps, carefully shaped rewards, and a lot of debugging. And a learned controller with no built-in model can be brittle: small sensor noise or friction changes might break it. In the worst case, you can’t safely learn on a real hardware robot without crashing it. Because of this, the SRL authors note that “Reinforcement Learning (RL) can adapt to diverse and complex environments but often requires extensive data from unguided exploration”. In other words, RL is flexible but expensive and potentially unsafe. It also tends to find non-intuitive gaits that may not be energy efficient or natural. Pure RL is like letting a novice fumble around: eventually it might get good, but it’s slow and sample-hungry. A hybrid provides a shortcut. By giving the RL policy a good initial guess (the SLIP plan), the learning only needs to refine and repair, not invent the jump from zero. This can drastically reduce the training time. In effect, the known physics of SLIP “guides” the learning. Recent work in legged locomotion has started to use this idea: layering learning on top of model-based control. For instance, the ANYmal Parkour project by Hoeller et al. (2024) showed that ANYmal learns stunts like vaulting obstacles by combining perception (camera + ML) with a traditional controller – the robot “taught itself to do parkour” by trial and error, but it was always anchored by classic control loops. The take-away is that model knowledge can ground the learning. SRL takes this concept specifically for jumping. Their simple insight: use SLIP as the feedforward jumping controller, and let the RL policy add feedback corrections. They call this Spring-loaded Reinforcement Learning (SRL). The SLIP controller executes a nominal jump trajectory (in several phases), while a neural policy (trained with PPO) tweaks foot placement and timing on the fly. In effect, SRL decouples the long-term dynamics (handled by SLIP) from the real-time adaptation (handled by RL). The SRL Framework in Detail. How exactly does SRL work? The core is a two-stream controller: feedforward from SLIP, and feedback from RL. On the feedforward side, the robot’s jump is broken into discrete phases, each governed by a SLIP-like spring model. Concretely, SRL uses a six-state finite-state machine (FSM) for the jump cycle: initialization (preparing to jump), ground contact (flexing into the jump), acceleration (stretching the spring to push off), flight (in the air), landing preparation (readying legs for touchdown), and recovery (absorbing the landing). In each contact phase, SRL treats the center of mass dynamics as a spring–mass–damper:. CoM force = $-k_i ,\Delta L - c_i,v_m$,. where $\Delta L$ is the leg compression relative to rest length, $v_m$ is the vertical velocity of the mass, and $(k_i, c_i)$ are phase-dependent spring and damping constants. Put simply, each phase has its own “stiffness” and “damping” for the spring. This analytically computes how strongly to push on the ground. Then an analytic inverse kinematics solver or optimization converts the desired CoM force into specific motor (joint) positions and torques. This SLIP-based feedforward loop produces a natural jumping motion profile: it decides when to leap, how much to compress, and how to jump up. Now the feedback side: the RL agent observes the robot’s state (joint angles, velocities, maybe body orientation) and outputs a corrective action in each control step. In the SRL setup, the RL output $\mathbf{a}{fb}$ is added to the SLIP command $\mathbf{a}{ff}$ (modulated by a gain $k_b$) to form the final joint targets:. Control Command = $\mathbf{a}{ff} + k_b,\mathbf{a}{fb}$. In other words, the RL network says “add a bit more/herked little from the baseline.” This lets the learned part compensate for slippage, unmodeled leg compliance, uneven ground height, etc. Because the feedforward plan has already decided the jump timing and leg force envelope, the RL agent can focus on fine adjustments (like shifting a bit forward, or bracing the ankle). The authors use Proximal Policy Optimization (PPO) to train this feedback policy in simulation. Training SRL requires a reward that encourages both a successful jump and stable posture. The authors use a multi-term reward: it rewards keeping the robot alive (not falling), maintaining upright orientation, tracking a target velocity or displacement, and keeping leg motions smooth. Early in training, higher weight is placed on basic stability (not falling) and orientation, so the robot learns not to tip over. Later, the weight shifts toward precision in hitting a jump distance and synchronizing the legs. By shaping the reward over time, they ensure the learned policy first “figures out how to stand” before optimizing speed or jump height. (This staged strategy is common in legged RL, though not explicitly cited by SRL.). The net effect of this architecture is that SRL is a closed-loop controller: the SLIP FSM rolls out a planned trajectory (feedforward) and at each time step the RL policy corrects it (feedback). Intuitively, the SLIP covers the nonlinear game of jumping dynamics, while the RL cleans up the residual errors. Importantly, the SLIP part is analytical, so we bring in known physics, and the RL part is adaptive, so we still gain robustness. Experimental Results. Hu et al. evaluate SRL in simulation on both a bipedal humanoid and a quadrupedal model. They test a variety of jump tasks, including simple flat-ground leaps and multi-step stairs/hurdles. The findings are clear: SRL outperforms an RL-baseline controller that has no SLIP guidance, both in jump quality and learning efficiency. Quantitatively, SRL achieves very stable hopping: the center of mass stays on track within about 0.1 meters of error on average, and the velocity trajectories are within roughly ±3% of targets. This is after training only about 1–2 million simulation steps. In contrast, training an RL agent from scratch on the same tasks can take several times more data and often culminates in less stable gaits. In other words, embedding the SLIP prior dramatically speeds up learning. According to the authors, SRL attains its solution in a fraction of the time the RL-only agent needs. (They report on the order of 3–4× fewer training iterations for similar performance, though the exact numbers are not published in the abstract.). Breakdown by setting:. Flat Ground Jumps: In both biped and quadruped sims, SRL converges to a periodic jumping gait that meets the velocity targets (forward or upward) with high reliability. The jumps look smooth and consistent every cycle. An RL-only policy, by contrast, often displays jerky or unbalanced moves early on. SRL’s leg forces follow the SLIP spring trajectory with only minor RL tweaks, indicating it leans heavily on the model dynamics. Stair/Obstacle Jumps: The authors also test jumping up and down steps. Here the terrain is “discontinuous,” meaning the robot must adapt each jump because the ground height changes. SRL handles this by phase-dependent corrections: for example, it can extend the legs slightly more if it senses it’s still in the air longer. Performance remains robust – in simulation the robot clears multi-step barriers that defeat the pure model (which usually mis-times the landing on uneven ground) and the pure RL (which can’t easily learn arbitrary step patterns without structure). Ablation Studies: To verify the design choices, they turn off components one by one. Removing the FSM structure or the velocity-tracking reward term both lead to dramatic failures. Without the SLIP segmentation, learning no longer has the jump rhythm, and it may never reliably take off. Without a reward on centroid velocity, the jumps become sloppy or fall short. This confirms that the six-phase model and tailored reward are critical to making SRL work. Qualitatively, the resulting SRL controller beats baselines. The authors note it achieves “more stable jumps with much less training time than the baseline method”. The jump height and distance closely match the commanded targets, whereas the RL-only baseline either requires much more training or exhibits higher tracking error. The SRL-trained quad and biped also show better transfer across different velocity commands: once trained, the agent can robustly handle small changes in speed goals without retraining, again thanks to the structured feedforward. Sim-to-Real Validation. Crucially, SRL is not just a simulation trick. The authors report a sim-to-real experiment on a physical humanoid platform (referred to as “X02-lite” in their text). After training SRL in a simulated environment (using frameworks like Isaac Gym or MuJoCo), they deploy it on the real robot. To deal with reality gap, they apply extensive domain randomization during training (varying masses, contact friction, sensor noise, etc.) and use a robust state estimator on the real machine. Specifically, they mention using a Contact-aided Invariant EKF【22†】 (CI-EKF) to fuse leg odometry and inertial sensing under impact; this kind of estimator is known to improve legged state estimates when contact is intermittent. On the real robot, SRL produces impressive results. The humanoid consistently jumps to the desired height and distance; in their tests it reached about 15 cm of vertical lift and 20 cm of horizontal range on a single jump. These numbers roughly match the simulation (slightly lower due to real dynamics), showing the learned corrections generalized. Even on slight unevenness in the lab floor, the robot stayed upright and resumed the gait. By contrast, without the SLIP guidance the same RL network would likely stumble or fail outright under minor disturbances. Because of SRL’s success, the authors conclude it “underscores [the method’s] potential for real-world deployment”. Context and Comparison. SRL fits into a broader theme of model-informed RL. Many recent works blend analytical models with learning. For example, Yang et al. (ICRA 2024) tackle quadruped jumping over stairs by a hierarchical scheme: they learn a high-level landing policy in simulation, use an explicit map of the stairs (a “heightmap predictor”), and employ a model-based low-level controller for the leg actuators. In fact their abstract highlights this mix: “a learned heightmap predictor, a reinforcement-learning-based centroidal-level policy, and a model-based leg controller” allow a Unitree robot to continuously jump up multi-step staircases. That picture is very much in spirit with SRL: it is RL at one level, physics-based control at another. The difference is that SRL keeps things relatively simple and focused (SLIP + single PPO policy) and demonstrates it on jumping specifically. Another related idea is residual RL, where one learns a “residual” policy on top of a nominal controller. For instance, Levine’s group showed residual policies can refine task-space controllers for manipulation. SRL is essentially residual RL for locomotion: the SLIP controller is the nominal, and the policy is the residual. It’s encouraging to see this pattern succeeding in highly-dynamic gaits as well. In the grand scheme, SRL is closer to how animals behave. Real quadrupeds and bipeds don’t rely on raw trial-and-error learning for each jump; they have spinal reflexes and muscle springs that automatically shape the motion, with the brain providing adjustments. The SLIP model captures the “reflex” part, and RL is like the brain’s strategy. This bioinspiration is intentional: the SLIP’s passive springiness naturally yields stability. As Geyer and Saranli note, a SLIP-based gait is the “basic gait template” for agile locomotion. SRL leverages that insight – it doesn’t throw away the model, it uses it. How You Might Use SRL. For practitioners, the SRL paper offers a blueprint for incorporating model priors into learning. Here are some take-home ideas:. Feedforward + Feedback: Whenever you have a simple model of your system (like SLIP, or even a linearized model), use it to generate a baseline trajectory. Then train an RL policy to fine-tune that trajectory online. This can drastically cut down the random exploration needed. Think of the RL as a “residual policy” on top of your expert. Structured Phases: If your task has natural modes (like stance vs flight in jumping), explicitly encode those as phases or modes. SRL uses a finite-state machine to break the jump into six modes. This meshes well with state-of-the-art control architectures for walking and running, which often fix phase transitions (for instance, “lift off” and “touch down” events). Reward Shaping: Balance your objectives carefully. A multi-term reward that gradually shifts focus (stability first, then velocity, etc.) can address the notoriously chaotic nature of jumping. SRL’s staged reward emphasizes not falling over before pushing for distance or speed. Sim-to-Real Tricks: If you train in sim, use domain randomization and good estimation on the real bot. The authors used a contact-invariant filter and random varied jump targets. Even then, expect some tuning. Their success jump height (15 cm) might differ on larger robots or those with different actuators. Generalization: SRL, like any learned policy, is limited by its training. If the robot encounters terrain or payloads far outside what it saw, performance will degrade. But because the policy is essentially correcting a physical model, it can often generalize better than a raw black-box policy. For example, a SLIP baseline already handles changes in jump distance or small height differences easily, so RL only needs to adapt residuals. Looking ahead, SRL could be extended or combined with other methods. One could imagine a hierarchical variant: use SLIP+RL for a single jump, and another planner or learning layer for sequencing multiple jumps. Or integrate vision: in a cluttered environment the robot could use a camera to modify the SLIP plan (e.g. choose where to land) and then let the feedback policy ensure a safe landing. In fact, that is exactly what some parkour systems do: they use vision to pick footholds, then use learned controllers to execute the jumps. Conclusion. The Spring-loaded Reinforcement Learning framework shows that good old physics can still dance with modern learning. By using the SLIP model as a springy guide, SRL cuts the grinding of pure RL and yields robust hopping gaits. The authors demonstrate clear gains: stable jumping with tight tracking error, and much faster convergence than RL alone. They even leap beyond simulation into reality, with the robot achieving solid 15-cm hops in the lab. For roboticists interested in agile locomotion, SRL is a compelling lesson: leverage your model, and let learning cover the gaps. In practice, that means if you’re training a new jumping or running controller, consider hand-crafting a rough trajectory from first principles (like a SLIP plan) and use RL for the residual. This can dramatically save training time and improve safety. SRL is a concrete embodiment of that philosophy, and its success suggests similar hybrid designs will be fruitful for other dynamic skills. Agile robots of the future will likely need both the stability of springs and the adaptability of neural nets – SRL shows one way to fuse them into a single high-performance controller.