Latest / Embodied AI 101 / Calibrate the Robot, Randomize the World: PACE for Legged Sim-to-Real

Transcript

Calibrate the Robot, Randomize the World: PACE for Legged Sim-to-Real · Embodied AI 101

• Calibrate the Robot, Randomize the World: PACE for Legged Sim-to-Real. Towards bridging the gap: Systematic sim-to-real transfer for diverse legged robots • Filip Bjelonic, Fabian Tischhauser, and Marco Hutter • Robotic Systems Lab, ETH Zurich • The International Journal of Robotics Research, OnlineFirst • 2026. A familiar pattern in legged-robot learning goes something like this: build the best rigid-body model available, randomize everything that might be wrong, train a robust policy, deploy it, and then start patching whatever fails. Increase the motor-strength range. Add another action-delay distribution. Penalize joint accelerations. Adjust the foot-clearance reward. Retrain. Repeat. PACE—short for Precise Adaptation through Continuous Evolution—argues that this workflow often begins too far downstream. If the simulated mapping from joint target to joint motion is substantially wrong, then reward shaping and broad dynamics randomization are compensating for a modeling problem that could have been measured directly. The paper’s alternative is deliberately systematic. Suspend the robot. Move its legs through informative, contact-free trajectories. Record only synchronized joint commands and encoder positions. Fit a compact set of effective joint-dynamics parameters by replaying those trajectories in thousands of parallel simulations. Then train a locomotion policy inside the fitted simulator, without randomizing those identified dynamics. That description makes PACE sound like a system-identification paper with an RL demonstration. In practice, it is more ambitious. It combines actuator characterization, gray-box simulator calibration, a compact locomotion objective, a motor-aware energy model, hardware-safe action processing, and long-duration endurance measurements. The result is less a new RL algorithm than a proposed engineering discipline for legged sim-to-real. One important qualification should be stated up front. PACE is systematic across robots, but it is not a single morphology-conditioned controller shared by all of them. Each platform receives its own data collection, parameter fit, simulator, and policy. The reusable object is the pipeline. The central claim: much of the reality gap lives below the policy. A URDF or USD model is good at describing geometry, mass, inertia, and kinematic structure. It is usually much less faithful about what happens between a policy’s joint target and the resulting motion. A real actuator has rotor and reflected transmission inertia. Its gearbox, bearings, seals, and firmware introduce damping and friction. Commands arrive late. Current and voltage saturate. Velocity estimates are filtered. Embedded controllers may compensate for cogging, friction, or plant dynamics. Even when the policy uses joint-position targets and a conventional PD law, the resulting closed-loop system is not the ideal torque source assumed by a basic simulator. Those differences matter most as motion frequency increases. A modest phase mismatch may be tolerable during a slow posture change but destabilizing during rapid swing-leg recirculation. The policy may then discover a gait that depends on accelerations, impacts, or torque reversals that the real machine cannot reproduce at the same phase. PACE therefore treats achievable step frequency and speed as practical indicators of model quality, rather than viewing low-speed standing or walking as sufficient proof of transfer. The paper positions its model between two common extremes. On one side is broad domain randomization: use a simple model, sample uncertain parameters over wide ranges, and hope that reality lies inside the training distribution. On the other side are expressive learned actuator or residual models, which can capture complicated behavior but may require torque sensing, extensive hardware data, and a more opaque debugging process. PACE instead fits a small gray-box model. For every actuated joint, it identifies an effective inertia, viscous damping, Coulomb friction, and position bias. It adds one global command delay for the robot. A 12-joint quadruped therefore has 49 fitted quantities: four per joint plus one delay. The parameters remain recognizable enough to diagnose, yet they are allowed to absorb effects that are difficult to separate physically. That last point is essential. The identified “armature inertia” is not necessarily the rotor inertia one would measure on a bench. It may also absorb errors in CAD link inertias, firmware compensation, transmission behavior, and motor-constant calibration. PACE identifies a compact simulator that is dynamically equivalent over the relevant operating region—not necessarily a unique ground-truth decomposition of the hardware. PACE stage one: collect boring but informative data. The identification experiment is intentionally removed from locomotion. The robot’s base is fixed, its legs move freely in the air, and all ground and inter-leg contacts are excluded. This setup removes two major confounders. First, there are no unknown contact forces. Second, base motion cannot couple one limb’s excitation into another. For symmetric platforms, the authors use symmetric joint commands to cancel net base wrenches. The simulation replay uses the same fixed-base pose as the hardware experiment. Why not identify the model while walking? Because stance dynamics can hide precisely the actuator behavior the method is trying to measure. On a large quadruped, the effective inertia associated with moving the base can exceed the isolated leg and drive inertia by one or two orders of magnitude. Suspending the robot makes the smaller joint-level effects observable. The excitation signals are multi-joint chirps: smooth oscillations whose frequency rises over time. Typical sequences last tens of seconds, beginning around 0.1 hertz and extending toward the highest safe frequency allowed by the robot and suspension. Full-robot structural limits restricted ANYmal’s experiments to about 2 hertz and Tytan’s to about 8 hertz, while smaller systems could be excited more aggressively. The paper’s practical target is to cover the frequency range the policy can generate, or at minimum roughly twice the highest frequency expected in the eventual locomotion motion. PD gains are part of the experimental design. High gains push the closed-loop actuator dynamics into higher frequencies, potentially outside the safe excitation bandwidth. PACE therefore uses relatively low, known gains during identification and policy training. This pulls the characteristic dynamics into a measurable range. The “known” part matters. If inertia, damping, proportional gain, and derivative gain were optimized together, a common scaling of all four could preserve the same trajectory. The optimization would then have multiple equivalent explanations. PACE fixes the gains and fits only the simulator dynamics. For the headline comparison, the authors use roughly 20 seconds of encoder-only in-air data per robot. There is no requirement for joint-torque sensing. That considerably lowers the instrumentation burden relative to learned actuator models trained from torque-labelled trajectories. It does, however, require accurate command timing, synchronized logs, trustworthy encoder measurements, and a low-level controller whose behavior remains consistent between identification and deployment. PACE stage two: evolve a compact simulator. Once a real trajectory has been recorded, PACE instantiates 4,096 parallel copies of the suspended robot. Each copy receives a different candidate parameter vector. The recorded joint targets are replayed at the same simulation rate intended for policy training, and each candidate is scored by its time-averaged joint-position error relative to hardware. The optimizer is CMA-ES, the covariance matrix adaptation evolution strategy. This is a sensible choice for a moderate-dimensional objective containing saturation, delay, friction, and a non-differentiable simulator. It avoids requiring analytic gradients and can use GPU simulation to evaluate an entire evolutionary population in parallel. The authors report fitting each platform on one RTX 3080, with optimization typically completing in approximately 10 to 24 hours depending on trajectory duration, bounds, and simulator throughput. The loss uses joint positions rather than torque measurements or explicitly differentiated accelerations. Delay, damping, inertia, friction, and bias are inferred through how they shape the complete trajectory. This keeps the sensing requirements low, although it also creates correlations among parameters. The compact parameterization is doing substantial regularization work. The paper’s single-actuator study helps justify that choice. On a Tytan hip drive, the measured electrical current loop behaves approximately as a high-bandwidth, near-unit-gain torque source over the frequencies relevant to locomotion. The estimated control bandwidth is about 346 hertz, with little phase lag below 25 hertz. That allows the slower mechanical response to be approximated by a second-order system dominated by effective inertia and damping. With firmware compensation disabled, PACE recovers known inertia changes reasonably well: errors are reported at roughly 2% for the bare rotor assembly, 6% for the interface, and 15% for the largest added-mass cases. Turning firmware compensation back on introduces an approximately constant “virtual inertia” of (8.1\times10^{-3}) kilogram-square-metres. In other words, embedded software changes the apparent mechanics even though no physical mass was added. That result supports one of the paper’s most practical warnings: identification and deployment must use the same firmware modes, filtering, units, and low-level signal path. A simulator calibrated with friction compensation disabled should not be expected to transfer after that compensation is enabled. At full-robot scale, the fitted quantities become even more effective rather than literal. Tytan’s fitted hip-flexion inertia is approximately four times the value predicted from rotor and firmware effects alone. The authors attribute the remainder to underestimated link inertia, compensation effects, and other model discrepancies. This is not a failure of PACE; it clarifies what PACE is actually doing. It is finding a concise dynamic surrogate that reproduces observed behavior. A locomotion objective made simpler by a less-wrong simulator. After alignment, the authors train a blind locomotion policy using asymmetric PPO. “Blind” here means no exteroceptive terrain perception. The actor still receives state-estimated base linear and angular velocity, gravity direction, commanded velocity, joint positions and velocities, and its previous action. The critic sees additional privileged information, including contacts, external wrench, friction, and a terrain-height scan. The policy outputs joint-position offsets around a default posture. A PD controller converts these targets into torques. Before a target reaches the actuator, PACE applies a hardware-protection transform near joint limits. As a joint approaches a hard stop, any target that would command motion farther into the stop is smoothly moved toward the current hard-limit position. At the limit itself, the PD error toward the obstruction becomes zero, while commands moving away remain possible. This same transformation runs in simulation and on hardware. Once the identified dynamics are fixed, PACE does not randomize motor inertia, damping, friction, strength, or delay. It still randomizes the world: ground friction, pushes, terrain roughness, stairs, boxes, and slopes. This distinction is the paper’s clearest conceptual contribution. Calibrate persistent robot properties; randomize environmental and task variation. The locomotion reward has only four terms:. Base-velocity tracking. Actuator-aware energy consumption. Foot speed at touchdown. Collisions involving joint limits or non-foot body parts. The compactness is partly enabled by penalty scheduling. Energy and touchdown penalties are introduced gradually so they do not prevent initial gait discovery. Policy entropy is also annealed, retaining exploration early and precision later. So “four reward terms” does not mean tuning disappears; coefficients and schedules remain. But the objective is far easier to interpret than the ten- or twenty-term mixtures commonly used to sculpt a gait. The energy term is the most interesting component. It combines three forms of power. First is resistive electrical loss in the motor windings. This is computed from commanded joint torque, gear ratio, motor constant, and coil resistance. Because copper loss grows with current squared, holding large torque can consume substantial energy even when the joint is barely moving. Second is mechanical joint power: torque multiplied by velocity. Negative mechanical power is credited only according to a regeneration coefficient, acknowledging that braking energy is not necessarily returned perfectly to the battery. Third is gravitational potential power across the robot’s rigid bodies. This accounts for energy being stored or released through vertical motion, preventing a bouncing policy from exploiting an incomplete instantaneous power objective. The combined power is normalized as a function of commanded speed because joint velocities and losses naturally rise with base speed. This keeps energy pressure meaningful across a range of commands rather than making fast locomotion disproportionately unattractive. This remains a medium-fidelity model. It captures dominant motion-dependent copper and mechanical costs without numerically modelling every iron, magnet, switching, thermal, and inverter loss. The later battery experiments show why that distinction matters: locomotion is not the whole energy budget. The experimental ladder, from one motor to six kilometres. The primary quantitative study uses three quadrupeds with the same three-joint leg topology but substantially different scale and actuation. Tytan is a 52.3-kilogram research platform with pseudo-direct hip drives and a variable-ratio ball-screw knee. ANYmal is a 52.8-kilogram closed platform using high-ratio, series-elastic harmonic drives. Minimal is a 4.2-kilogram, largely 3D-printed quadruped with variable-ratio lead-screw transmissions. Thus the primary diversity is strong in actuator design, size, inertia distribution, and hardware transparency, though less radical in leg topology. The authors also report using PACE parameter fitting on more than ten other systems, including Aibo, NAO, ALMA, Spacehopper, LEVA, two Magnecko versions, the GR-1 humanoid, and two robotic hands. Those deployments support pipeline portability, but they should not be confused with the detailed locomotion evaluation conducted on the three principal quadrupeds. In-air validation. On Tytan, simulated and real in-air trajectories nearly overlap after fitting. More importantly, the fit generalizes to PD gains that were not used during optimization. Minimal shows similarly close reproduction for both chirps and random joint steps, whereas its URDF-only model diverges substantially. These tests indicate that PACE is learning a reusable dynamic model rather than merely memorizing one excitation trajectory. ANYmal provides the most informative comparison. The authors evaluate a URDF-only simulator, the vendor’s recurrent actuator network, and PACE. The URDF-only rollout drifts away from hardware. Both the actuator network and PACE reproduce the motion, but PACE produces the tighter reported position-velocity error portrait and avoids an approximately five-degree position bias visible in the actuator-network result. The data asymmetry is notable. PACE uses about 20 seconds of encoder-only motion. The original actuator-network pipeline used roughly four minutes of torque-instrumented data, while the vendor’s more recent LSTM was probably trained on a still larger collection. The comparison does not show that neural actuator models are obsolete. It shows that, for the tested operating region, a constrained physical parameterization can reach comparable fidelity at much lower data and sensing cost. On-ground transfer. The on-ground results reveal why these in-air differences matter. On ANYmal, a policy trained with only the URDF dynamics fails during one-metre-per-second forward walking and must be caught by the safety crane. Policies trained with either the actuator network or PACE transfer successfully and settle around 0.85 metres per second. The URDF-only policy can execute some yaw motion, illustrating that passing one low-dimensional behavior is not sufficient evidence that the model is accurate. On Tytan, the authors replay the same commanded base-velocity sequence in simulation that was used during a real 90-second teleoperated run. Base linear velocity, yaw rate, and commanded-versus-measured motor torques align closely, with the most visible residual mismatch occurring around knee liftoff. Endurance and cost of transport. The strongest systems result comes from repeated laps of a 400-metre running track. ANYmal D travels 4.12 kilometres over 82 minutes on one charge, with an average forward speed of 0.85 metres per second and average power of 556 watts. Tytan travels 6.10 kilometres over 104 minutes, averaging 0.978 metres per second and 504 watts. The paper reports a full dimensionless cost of transport of 1.27 for ANYmal D, compared with 1.86 for the earlier ANYmal C result—a reduction of approximately 32%. Tytan reaches 0.97. The authors further split cost of transport into electronics, idle drive and inverter overhead, and motion-dependent locomotion cost. For ANYmal D, the three contributions are 0.50, 0.24, and 0.53. For Tytan, they are 0.29, 0.21, and 0.47. Less than half of either robot’s energy budget is therefore attributed to locomotion itself. That decomposition changes how the efficiency result should be interpreted. Better control can reduce copper loss, impacts, unnecessary joint motion, and mechanical work. It cannot eliminate the power consumed by computers, sensors, communications, or enabled inverters. Once locomotion becomes reasonably efficient, hardware architecture becomes the dominant optimization target. Finally, the fitted simulators support behaviors near the systems’ dynamic limits. ANYmal balances on two legs and runs at close to three metres per second on hardware; the equivalent simulation reaches about four metres per second, with the real run capped by a 32-amp battery limit. The 4.2-kilogram Minimal continuously climbs 18-centimetre stairs, completing 20 steps in 46 seconds. How PACE differs from neighboring sim-to-real strategies. PACE’s closest historical reference is the actuator-network approach introduced by Hwangbo and colleagues for ANYmal. Actuator networks learn the mapping from command and state histories to realized torque. Their expressiveness is useful when drive behavior is strongly nonlinear or history-dependent, but training them typically depends on reliable torque labels and sufficiently broad hardware trajectories. PACE trades some of that expressiveness for lower data requirements, interpretability, and easier portability to robots without torque sensing. A more recent comparison is the RAI Institute’s Spot work. That system also uses CMA-ES and a gray-box actuator model, but it optimizes eight friction and torque-speed parameters by comparing distributions of policy rollouts using Wasserstein distance and maximum mean discrepancy. It requires deploying an initial policy, collecting several on-ground command sequences, optimizing the simulator, and retraining. PACE instead uses policy-independent, time-aligned, fixed-base chirps. Spot’s method directly models torque-speed limits that PACE identifies as future work, while PACE fits a richer per-joint dynamic model from a much simpler experiment. SPI-Active also performs massively parallel parameter identification, then optimizes an exploration policy to maximize the Fisher information in real trajectories. That is attractive when passive excitations do not sufficiently identify contact-rich parameters. PACE takes the safer and operationally simpler route of designed in-air chirps, but consequently identifies contact dynamics only indirectly. ASAP takes a more expressive residual route for humanoids. It first deploys a simulated policy, collects real whole-body motion, learns a delta-action model that compensates for the dynamics mismatch, inserts that residual into simulation, and fine-tunes the policy. Such a correction can cover errors outside PACE’s compact parameter family, but it is more dependent on real policy rollouts and potentially more skill-specific. PACE is therefore not the universal winner on an axis from “physics” to “learning.” Its natural niche is a stiff, electrically actuated robot whose low-level closed-loop dynamics are approximately linear over the relevant band, whose gains and command path are known, and whose primary discrepancy can be exposed through safe joint-space excitation. What the paper establishes—and what it does not. The most convincing result is not any single gait. It is the consistency of the experimental ladder: known inertia changes on one drive, unseen-gain generalization on a suspended robot, direct comparison against ActuatorNet, successful zero-shot walking, and then endurance and dynamic-limit demonstrations. Each stage tests a different way in which a seemingly good fit could have been misleading. Still, several boundaries matter. First, PACE identifies no contact parameters. The paper demonstrates that in-air joint calibration is sufficient for its blind locomotion tasks, not that contact is generally irrelevant. Robots with soft feet, flexible links, significant backlash, deformable terrain, manipulation contacts, or impact-dominated skills may need force-plate experiments, contact identification, residual physics, or online correction. Second, the parameters are only partly physical. Their interpretability is better than that of an unrestricted neural network, but an unexpectedly large inertia cannot automatically be assigned to one component. It may represent CAD error, embedded compensation, drive scaling, or actual inertia. PACE is best understood as structured model calibration rather than classical identification of uniquely recoverable physical constants. Third, eliminating dynamics randomization creates sensitivity to drift. Temperature, wear, firmware updates, filtering changes, battery voltage, and unit-to-unit variation can move the real system away from the fitted point. The authors note that one ANYmal parameter set can work across units, while also acknowledging residual differences from hardware wear and software changes. A production system would likely need periodic refitting or lightweight online adaptation. Fourth, the energy result is compelling systems evidence rather than a clean causal ablation. The headline comparison uses different ANYmal revisions, policies, prior methods, and ambient temperatures. It does not isolate how much of the 32% reduction comes from simulator fidelity, the energy reward, gait geometry, hardware revision, or operating conditions. A same-robot, repeated-battery comparison with and without each component would provide a stronger attribution. Finally, four reward terms are not the same as four decisions. The energy term requires motor resistance, torque constant, gear ratio, regeneration assumptions, normalization, and a coefficient. Penalties are scheduled, entropy is scheduled, and collision definitions remain task-specific. PACE substantially reduces reward clutter, but it does not make locomotion training tuning-free. The strongest defensible conclusion is narrower and more useful: for the tested PMSM-driven quadrupeds, a small joint-space parameterization fitted from short encoder-only experiments closes enough of the actuator gap to remove the need for broad dynamics randomization in blind locomotion. A practical recipe for adopting PACE. For a lab adapting this method, the most important work happens before PPO starts. Freeze the low-level system. Document PD gains, units, filtering, firmware compensations, saturation, command frequency, and transport delay. Identification is invalidated if these change before deployment. Design safe excitation around the eventual policy bandwidth. Use a fixed-base, contact-free rig and symmetric trajectories where possible. Begin with conservative chirp amplitudes, verify structural resonances, and increase bandwidth only as far as the hardware safely permits. Fit the smallest plausible model first. Start with effective inertia, damping, Coulomb friction, bias, and delay. Do not jointly optimize gains. Add filters, compliance, or nonlinear terms only when held-out trajectories show a systematic residual that the compact model cannot represent. Validate out of distribution before training. Replay unseen chirps, different amplitudes, different PD gains, and safe step inputs. Compare both time traces and position-velocity phase portraits. A low training loss is not sufficient if the model fails immediately outside the fitting sequence. Randomize uncertainty that remains genuinely variable. Terrain, friction, pushes, payload, sensing noise, and command distributions still belong in training. For deployment across aging hardware, my preferred extension would be a PACE nominal model plus narrow dynamics randomization around the identified point—not a return to broad, uninformed ranges. Measure the complete energy budget. If efficiency matters, distinguish motion-dependent losses from compute, sensing, and enabled-drive overhead. Otherwise, a controller may halve locomotion power while producing only a modest improvement in total endurance. The public implementation is an Apache-licensed Isaac Lab extension developed against Isaac Sim 5.0. That version requirement is substantive: earlier Isaac Sim releases may silently ignore joint viscous friction, undermining the fitted model. The repository currently provides one complete ANYmal example and warns that its API is still evolving. The accompanying open dataset is approximately 2.2 gigabytes and contains raw and processed data from actuator, single-joint, single-leg, and full-robot experiments, along with fitting and plotting scripts. That makes it possible to study the identification stage before committing hardware time to a new platform. The larger lesson. PACE’s most important idea is methodological: do not ask policy optimization to compensate for every lie in the simulator. A calibrated simulator does more than improve transfer probability. It changes what the policy is allowed to discover. It can exploit straighter legs, lower-impedance motion, regeneration, and platform-specific dynamics without needing to remain robust to an unnecessarily broad family of imaginary robots. At the same time, physically grounded rewards become more meaningful because simulated torque, velocity, and dissipation correspond more closely to hardware. The paper does not eliminate the reality gap, nor does it remove the need for robustness, residual learning, or adaptation. What it offers is a better nominal model—and a repeatable way to obtain one. For many legged-robot teams, that may be the highest-leverage place to begin: suspend the robot, run twenty seconds of carefully designed motion, and make the simulator tell fewer lies before asking reinforcement learning to solve the rest.