Latest / Embodied AI 101 / CycleRL: Teaching a Bicycle to Balance in Simulation—and Trusting It on Asphalt

Transcript

CycleRL: Teaching a Bicycle to Balance in Simulation—and Trusting It on Asphalt · Embodied AI 101

• CycleRL: Teaching a Bicycle to Balance in Simulation—and Trusting It on Asphalt. CycleRL: Sim-to-Real Deep Reinforcement Learning for Robust Autonomous Bicycle Control • Gelu Liu, Teng Wang, Zhijie Wu, Junliang Wu, Songyuan Li, Xiangwei Zhu • Sun Yat-sen University and Shenzhen Key Laboratory of Navigation and Communication Integration • IEEE Robotics and Automation Letters, 11(8), 9343–9350 • 2026. Source note: The paper first appeared on arXiv on March 16, 2026, with version 3 posted on June 25, 2026. This episode follows that latest manuscript. One terminology detail matters: the current paper reports a heading tracking error of 1.15 degrees, although earlier summaries—including the one supplied with this episode—describe it as a steering tracking error. Why Autonomous Bicycles Are Such an Unforgiving Control Problem. An autonomous bicycle looks mechanically simple: two wheels, one steering joint, and a drive motor. But from a control perspective, it is a compact concentration of difficult properties. The roll degree of freedom is not directly actuated. If the bicycle starts falling left, there is no motor that simply applies an opposing roll torque. The controller must steer left, shifting the tire contact geometry and generating lateral acceleration that brings the wheels back under the center of mass. That same steering action also changes the vehicle’s heading. Balance and navigation therefore compete for the same actuator. The system is nonlinear, underactuated, nonholonomic, strongly speed-dependent, and sensitive to contact mechanics. At low speed, steering produces limited lateral force, while the passive effects that help a moving bicycle remain upright are weak. At higher speed, very small steering inputs can create large lateral accelerations. A controller must continuously trade off roll recovery, heading accuracy, actuator delay, tire traction, and the risk of overcorrecting. Payload makes this harder. A delivery package changes the total mass and center of mass. Tire pressure changes contact behavior. Gravel, grass, ramps, and bumps alter friction and introduce impulses. An IMU may have bias and vibration noise; a steering drive has backlash, compliance, saturation, and latency. A controller derived from one nominal bicycle model can become brittle after surprisingly small physical changes. One solution is to add a reaction wheel or control-moment device that can balance the bicycle even while stationary. That makes roll directly controllable, but it adds mass, mechanical complexity, energy consumption, and payload penalties. CycleRL instead addresses the more natural—and more difficult—configuration in which balance is achieved primarily through steering. The paper describes the bicycle using roll angle, steering angle, planar position, and yaw. Its conceptual dynamics include inertia, Coriolis and gyroscopic terms, gravity and centrifugal forces, control inputs, and external disturbances. The authors also frame real-world uncertainty as a combination of continuous stochastic disturbances, such as terrain vibration, and discrete impulses, such as sudden road irregularities. But they do not solve this model online. It serves mainly to explain why an exact analytical controller would be difficult to identify and maintain. That distinction is central to CycleRL. The project is not trying to obtain a more accurate symbolic bicycle model. It is trying to train a feedback policy that remains useful across many plausible models. What CycleRL Actually Is. CycleRL is a command-conditioned, proprioceptive, low-level controller trained with Proximal Policy Optimization in NVIDIA Isaac Sim. It maps onboard observations and desired motion commands into actuator behavior intended to keep the bicycle upright while following speed and directional references. “End-to-end” needs to be interpreted carefully here. CycleRL is end-to-end from low-dimensional sensing and commands to low-level control. It is not an image-to-motor navigation policy. The final manuscript says that instantaneous IMU and encoder observations are treated as the state, even though the physical system is technically partially observable. The policy is a reactive stochastic multilayer perceptron rather than a recurrent network, history encoder, or explicit learned state estimator. The authors argue that the 50-hertz control rate reduces the need for temporal integration and use domain randomization to make the reactive policy tolerant of noisy observations. The final paper does not enumerate every observation and action channel in its main text. An earlier manuscript table reports an eight-dimensional input and a two-dimensional output, with actor and critic MLPs using hidden widths of 512, 256, 128, and 64 and ELU activations. The reported PPO configuration uses a learning rate of (10^{-4}) with cosine annealing, a discount factor of 0.99, generalized advantage estimation with lambda 0.95, and a clipping ratio of 0.2. The lack of a channel-by-channel interface specification in the final text is a reproducibility weakness, particularly because the hardware discussion later indicates that the longitudinal velocity command is mapped directly to motor speed rather than being fully mediated by policy inference. The Reward Is the Real Controller Specification. CycleRL’s algorithmic core is standard PPO. The more consequential design choice is the reward. At every control step, the policy receives a weighted combination of five signals:. A constant survival reward for remaining operational. An exponentially decaying velocity-tracking reward. An exponentially decaying heading-tracking reward. A penalty on action magnitude. A penalty on the change in action from the previous step. The weights are one for survival, three for velocity, five for heading, one for action magnitude, and two for action rate. The larger tracking weights discourage the trivial solution of merely surviving without following commands. The action penalty discourages unnecessarily aggressive actuation, while the rate penalty suppresses high-frequency oscillation and makes the result more compatible with real actuator bandwidth. This is a sensible decomposition of the task. Steering-only balance has a strong temptation to chatter: repeatedly applying sharp corrections may work in an ideal simulator while exciting mechanical compliance, saturating a motor, or destabilizing the real bicycle. The action-rate term makes smoothness part of the optimization objective rather than something added as an afterthought. Episodes last up to 64 seconds, or 3,200 control steps at 50 hertz. A safety termination ends an episode when the absolute roll angle exceeds 45 degrees. That gives PPO a clear failure event while avoiding the cost of simulating a bicycle that has already entered an unrealistic fallen state. The policy is optimized with PPO’s clipped objective and generalized advantage estimates. This is a pragmatic choice rather than an algorithmic contribution: PPO is stable enough for continuous control and straightforward to scale across thousands of parallel simulated bicycles. CycleRL’s contribution lies in the environment, objective, randomization distribution, hardware architecture, and deployment evidence—not in a new reinforcement-learning update rule. Massive Parallel Training. The Isaac Sim setup runs 16,384 environments in parallel on an RTX 4090, reportedly exceeding 700,000 simulated steps per second. Training curves across ten seeds stabilize after roughly 1,000 epochs, corresponding in the current manuscript to about 700 million simulation steps. The reward curves show what the authors call an implicit curriculum. Early in training, the policy discovers aggressive steering actions that prevent immediate falls. Once survival is largely solved, the action penalties move back toward zero as PPO finds smoother ways to achieve similar balance and tracking performance. In other words, the policy first learns not to crash and then learns not to look frantic while avoiding the crash. For deployment, the system is divided into input processing, policy inference, and actuator-control layers. An asynchronous multiprocess architecture carries data through a 50-hertz loop. This is another important piece of the work: a neural controller is useful only if sensing, inference, command transport, and motor control preserve the timing assumptions under which it was trained. The Sim-to-Real Strategy. CycleRL uses zero-shot transfer: the trained policy is deployed without fine-tuning on the physical bicycle. The primary mechanism for making that possible is a broad domain-randomization distribution. Total simulated mass is sampled from 15 to 45 kilograms, and center-of-mass height from 0.50 to 0.80 meters. Tire friction varies from 0.5 to 1.2. Actuator force gain ranges from 90 to 110 percent of nominal. Observation noise is Gaussian, with its scale randomly selected between one and 20 percent of each signal’s maximum magnitude. These ranges are intentionally much broader than a single carefully identified bicycle. The initial conditions are randomized as well. Episodes begin with forward velocities between 1 and 2.5 meters per second, roll angles between minus and plus 10 degrees, steering-motor positions between minus and plus 20 degrees, and different initial hub-motor velocities. This teaches recovery from imperfect launches and unstable handoff states rather than assuming that every episode starts exactly upright. Finally, the task itself changes during an episode. Target speeds are sampled from 1 to 5 meters per second, while target directional commands vary over a roughly plus-or-minus-10-degree range. Commands are resampled every three to five seconds. The policy must therefore learn a conditional feedback law, not one fixed equilibrium behavior. These randomizations address different failure modes:. Dynamics randomization reduces dependence on one mass, friction coefficient, center of mass, or actuator model. Initial-state randomization expands the policy’s recovery basin. Command randomization prevents memorization of straight, constant-speed riding. Sensor and terrain variation expose the policy to noisy and discontinuous interactions. That decomposition is one of the more transferable ideas in the paper. Many sim-to-real projects randomize physical parameters but leave initial states and task commands nearly fixed. CycleRL treats all three as parts of the transfer problem. It is also important not to equate “model-free” with “physics-free.” Considerable modeling work remains. Isaac Sim includes contact, actuator behavior, variable friction, and IMU properties, and the randomization bounds must still be chosen intelligently. The authors explicitly acknowledge that avoiding analytical system identification shifts engineering effort into simulator construction, reward shaping, and hyperparameter selection. Experimental Results: Strong, but Read the Metrics Carefully. The paper evaluates balance, command tracking, response, and robustness. Its headline metric is Balance Success Rate, defined as the percentage of episodes that keep absolute roll below 0.5 radians—about 28.6 degrees. Balance Recovery Time measures recovery from disturbances beyond 0.3 radians, or roughly 17.2 degrees. Critical Angle Tolerance is the largest initial roll disturbance from which success remains above 95 percent. The 0.5-radian threshold is fairly permissive. Balance Success Rate should therefore be read primarily as a fall-avoidance metric, not as a measurement of how tightly the bicycle remains upright. Heading error, recovery time, and the deployment videos provide complementary evidence that the controller is doing more than staying barely inside that boundary. Simulation Against Classical Baselines. The authors evaluate 10,000 episodes and compare CycleRL with a tuned PID controller, an LQR controller based on a linearized model, and the Modified Linear Control Law, or MLCL, from prior autonomous-bicycle work. | Metric | PID | LQR | MLCL | CycleRL | |---|---:|---:|---:|---:| | Balance success rate | 76.50% | 93.24% | 96.43% | 99.90% | | Recovery time | 1.41 s | 1.47 s | 1.21 s | 1.05 s | | Maximum balance duration | 15.0 s | 148.9 s | 603.2 s | >1,800 s | | Critical angle tolerance | 22.24° | 21.40° | 24.65° | 27.79° | | Heading tracking error | 4.04° | 1.82° | 2.87° | 1.15° | | Minimum sustaining speed | 1.84 m/s | 1.71 m/s | 1.50 m/s | 1.05 m/s |. CycleRL is better on every reported common metric. Its advantage is especially pronounced in maximum duration and minimum sustainable speed, both of which matter for practical steering-balanced bicycles. The paper also reports a velocity-tracking error of 0.18 meters per second, but the classical baselines are primarily evaluated as lateral controllers at constant velocity, so there is no directly comparable baseline value. There are additional protocol details worth noting. CycleRL’s target speed is randomized between 2 and 3 meters per second during this comparison, whereas the baselines operate at a fixed 2 meters per second. The evaluation disables physical-parameter randomization while retaining initial-state and command variation. The comparison is therefore informative but not a perfectly identical controller-design problem: CycleRL is a command-conditioned policy, while several baseline outputs and metrics are defined mainly around lateral stabilization. Robustness Beyond the Nominal Plane. At speeds between 1 and 3 meters per second, CycleRL maintains a 99.49 percent simulated balance success rate. Between 3 and 5 meters per second, that rate falls to 86.21 percent. With 20 percent-amplitude sensor noise, it achieves 96.38 percent, and with a 10 percent observation-dropout rate, 90.84 percent. Simulated terrain results are 88.47 percent on rough pavement, 85.87 percent on gravel, 92.05 percent on slopes, and 76.60 percent over gentle steps. These are simulation numbers, not controlled physical success rates, but they show graceful rather than catastrophic degradation. The high-speed decline is particularly revealing. A bicycle may be passively more stable while moving quickly in a straight line, but CycleRL is being asked to execute randomly changing directional commands. Lateral acceleration grows roughly with the square of velocity, making sharp steering corrections much more consequential. The authors also identify actuator phase lag, the fixed 50-hertz loop, and limitations in PhysX’s high-speed contact and collision dynamics as likely contributors. That admission is important. Domain randomization can create robustness within a family of simulated dynamics, but it cannot fully compensate for a simulator that becomes structurally inaccurate in a particular regime. What the Ablations Tell Us. The reward ablations show that Balance Success Rate is close to saturation and cannot distinguish all policy qualities. Removing the survival reward actually raises it from 99.90 to 99.99 percent, although heading error worsens slightly. Removing the action penalty produces the same nominal success rate but degrades heading and velocity tracking. Removing the action-rate penalty leaves success near 99.9 percent while roughly doubling heading error and tripling velocity error. The task-tracking terms are more obviously essential. Without the velocity reward, Balance Success Rate falls to 96.71 percent and velocity error rises from 0.18 to 0.59 meters per second. Without heading tracking, heading error jumps from 1.15 to 6.30 degrees. The survival term mainly helps learning; the tracking and smoothness terms define the quality of the eventual controller. The domain-randomization ablation presents a classic robustness trade-off. With no randomization, ideal-simulation Balance Success Rate is marginally higher at 99.99 percent, but heading error rises to 5.16 degrees and velocity error to 0.49 meters per second. Dynamics-only, initial-state-only, and terrain-only randomization all underperform the complete mixture. Full randomization sacrifices a tiny amount of nominal balance performance to gain much stronger generalization and tracking. However, these randomization ablations are evaluated in simulation. The paper does not deploy every ablated policy on hardware, so the claim that the full mixture is necessary for real transfer remains supported indirectly rather than through a controlled physical ablation. The Physical Bicycle. The hardware is a full-size, custom-built platform with a 1.1-meter wheelbase, a mass of 25 kilograms, and a center-of-gravity height of 0.65 meters. Those nominal values sit near the centers of the training distributions. Computation runs on a Jetson Orin NX with 16 gigabytes of memory and Ubuntu 20.04. The bicycle uses a WheelTec N100 IMU, a Unitree GO-M8010-6 steering servo connected through a synchronous pulley, a Bafang hub motor, and a 48-volt, 9.6-amp-hour battery. A front-mounted Intel RealSense D435i is added for the lane-tracking demonstration. On hardware, Balance Success Rate is 95 percent rather than the simulated 99.90 percent. Recovery time increases from 1.05 to 1.29 seconds, critical angle tolerance falls from 27.79 to 20.12 degrees, heading error rises from 1.15 to 2.04 degrees, and minimum sustainable speed rises from 1.05 to 1.33 meters per second. The system nevertheless balances continuously for more than 1,800 seconds—over 30 minutes—in both simulation and reality. The real velocity error is reported as only 0.01 meters per second, better than simulation. That number should not be interpreted as an unexpectedly superior learned longitudinal controller. The authors explain that the velocity command is directly mapped to motor speed, bypassing policy inference for that channel. CycleRL’s most consequential learned role on the physical system is lateral stabilization and steering coordination. The reported “transfer ratio” ranges from 0.56 to one across metrics. This is a useful compact summary, but it is not a standard normalized measure and behaves differently for higher-is-better and lower-is-better quantities. The heading ratio, for example, is obtained by comparing 1.15 degrees in simulation with 2.04 degrees in reality, while the unusually good real velocity error is effectively treated as perfect transfer. The raw measurements are more informative than the aggregate ratio. The PID, LQR, and MLCL baselines are not quantitatively reported on hardware because they failed to directly stabilize the physical platform when transferred without hardware retuning. That is relevant to CycleRL’s zero-shot objective. It does not prove that classical controllers cannot operate this bicycle: it proves that these simulation-tuned instances were more sensitive to the particular reality gap. Real demonstrations cover asphalt, gravel, rugged grass, ramps, payload changes, velocity changes, abnormal tire conditions, and lateral pushes. In a human-in-the-loop mode, an operator supplies speed and direction references while CycleRL handles low-level stabilization. For lane tracking, camera images are processed to detect the lane, and a tuned PID outer loop converts lane-center deviation into a steering reference. CycleRL remains the inner stabilizing controller; visual perception and route-level guidance are not learned end to end. The authors also report transferring the identical policy to a scooter platform without retraining. That is a promising sign that the randomization distribution captured useful cross-platform structure, although no detailed scooter metrics are provided. How CycleRL Relates to Earlier Bicycle Controllers. The strongest comparison is not simply “deep RL versus old-fashioned PID.” Modern model-based bicycle control is more capable than that framing suggests. Yeh, Lin, and Chen’s robust self-driving bicycle uses a hierarchical linear-parameter-varying design. It models speed-dependent dynamics, computes robust state-feedback gains with linear matrix inequalities, fuses GPS, wheel-speed, and IMU measurements, and reports campus trajectory tracking with RMS error below 18 centimeters. This approach requires explicit modeling and estimation, but it provides a structured controller and quantitative route-level tracking that CycleRL does not yet match. Xiong, Yu, and Liu’s Modified Linear Control Law is especially relevant because it explicitly addresses gyroscope drift—the same category of unmodeled IMU behavior that CycleRL cites as the main cause of heading degradation after transfer. Their work uses a time-varying intercept term to compensate for measured drift and demonstrates stable straight and circular riding. CycleRL beats an MLCL implementation in simulation, but the two approaches suggest a natural hybrid: a learned robust inner loop combined with explicit online bias estimation. Learning-based bicycle control also predates CycleRL. Guo and colleagues developed a synchronous actor–critic controller around a nonaffine bicycle model and feedback-linearization law, with Lyapunov analysis and prototype experiments. That method is learning-assisted but still deeply model-based: the actor structure and stability argument derive from analytical control. CycleRL’s differentiator is the combination of model-free PPO, massive parallel simulation, broad domain randomization, and direct deployment on a full-size platform across varied conditions. Weyrer and colleagues demonstrated pure RL for simultaneous path following and stabilization of a virtual Whipple bicycle between 2 and 7 meters per second, using curriculum learning and explainability tools to relate the policy back to known bicycle dynamics. That work is stronger on arbitrary simulated path following; CycleRL is stronger on physical transfer and environmental robustness. At the more aggressive end of the spectrum, the August 2026 Bicycle Acrobatics with Reinforcement Learning project uses multiple RL formulations and a policy orchestrator to execute jumps, wheelies, flips, and other maneuvers on a custom bicycle robot. Together, these papers suggest that bicycle robots are becoming a serious learning-control benchmark rather than a narrow control-theory curiosity. Why CycleRL Matters. CycleRL is significant less because it invents a new RL algorithm and more because it closes a difficult systems loop. It shows that a relatively ordinary MLP trained with ordinary PPO can become a useful physical controller when the task formulation, simulation throughput, randomization distribution, reward regularization, and onboard timing architecture are engineered together. In an era dominated by large multimodal policies, this is a reminder that low-dimensional proprioceptive feedback remains the right abstraction for many high-bandwidth control problems. It also offers a compelling interpretation of domain randomization as a form of data-driven robust control. Instead of proving stability for one uncertain model family, the policy is optimized over millions of transitions sampled from a deliberately broad family of masses, centers of mass, friction values, sensor conditions, commands, and disturbances. The resulting controller is slightly conservative in ideal simulation but considerably less attached to any one nominal plant. Perhaps most importantly, CycleRL demonstrates a useful modular pattern. A learned inner loop can provide a robust stabilizing skill while a human, planner, vision system, or classical outer-loop controller supplies high-level references. That separation is likely more practical for near-term robotic products than asking one neural network to perform perception, planning, safety reasoning, and millisecond-scale stabilization simultaneously. What the Paper Does Not Yet Establish. First, CycleRL is not a complete autonomous-cycling stack. It does not solve localization, route planning, obstacle avoidance, traffic interaction, or semantic decision-making. The lane demonstration uses a separate vision pipeline and PID reference generator. The contribution is best described as robust low-level bicycle control. Second, the controller has no formal stability or safety guarantee. The randomization results provide empirical evidence, but there is no certified region of attraction, control-barrier layer, runtime uncertainty monitor, or fallback controller. For public-space deployment, one would still want explicit roll, steering, speed, and actuator safety envelopes. Third, steering-only balance leaves important operating regimes unresolved. The real bicycle requires approximately 1.33 meters per second to maintain the paper’s success criterion. It therefore needs a strategy for launching, stopping, waiting at intersections, and handling faults below sustainable speed. A deployable platform might require a kickstand mechanism, deployable legs, a reaction device, or a separate low-speed controller. High-speed maneuvering is also not solved. Success falls to 86.21 percent in the 3-to-5-meter-per-second simulated range, and the authors themselves point to control bandwidth, actuator lag, and simulator deficiencies. Increasing policy capacity alone is unlikely to fix a mismatch rooted in contact modeling and loop timing. Fourth, the physical comparisons are incomplete. Classical baselines failed under strict zero-shot transfer, but no hardware-retuned classical controller is reported. Similarly, the domain-randomization ablations remain simulation-side. Physical deployment of selected ablated policies—or systematic variation of payload, IMU bias, tire pressure, and friction with repeated trials—would make the causal transfer story much stronger. Finally, reproducibility is currently limited. As of August 5, 2026, the public GitHub repository contains the project website and demonstration assets, but not the Isaac Sim environment, training code, robot description, learned weights, or deployment stack. Those artifacts would be especially valuable because much of CycleRL’s contribution lies in engineering details not completely specified in the eight-page paper. Lessons for Roboticists Building Similar Systems. If I were adapting CycleRL to another underactuated platform, I would carry forward five ideas. Train a command-conditioned skill, not a single behavior. Randomizing commands forces the policy to become a reusable feedback controller rather than a memorized equilibrium. Randomize starts and tasks as well as physics. Parameter randomization alone does not teach recovery from bad handoffs or abrupt reference changes. Put actuator feasibility directly into the reward. Penalizing action rate can matter more in hardware than squeezing out the final fraction of nominal simulated success. Keep the learned loop modular. Let planners and perception systems generate references while the learned policy handles the high-frequency dynamics it was trained to control. Evaluate transfer dimension by dimension. Report roll distributions, recovery basins, actuator saturation, latency, sensor bias, terrain, payload, and failure cases—not just one overall success percentage. I would also add explicit online bias estimation, safety filtering, and a low-speed transition mechanism. Those additions would not undermine the model-free controller; they would turn it into a stronger robotic system. Bottom Line. CycleRL is a convincing example of sim-to-real reinforcement learning applied to a system that is both dynamically unstable and unforgiving of modeling error. Its strongest result is not the 99.90 percent simulation success rate. It is the 95 percent physical balance rate, 30-minute continuous operation, recovery from substantial disturbances, and qualitative robustness across surfaces and mechanical variations—obtained without policy fine-tuning on the bicycle. At the same time, the paper’s central lesson is subtler than “RL replaces control theory.” CycleRL replaces explicit controller derivation with a different kind of control engineering: simulator design, distribution design, reward design, timing design, and empirical stress testing. Model-free does not mean engineering-free. In CycleRL, the engineering moves—and, in this case, it moves the bicycle with it.