Latest / Embodied AI 101 / Humanoid Scooter Control: Does Sim-to-Real Reinforcement Learning Work?

Transcript

Humanoid Scooter Control: Does Sim-to-Real Reinforcement Learning Work? · Embodied AI 101

• Humanoid Scooter Control: Does Sim-to-Real Reinforcement Learning Work?. Scooter-Driving Humanoid Robot: Sim-to-Real Transfer Through Deep Reinforcement Learning • Ugo Richard Roux, Saeed Saeedvand, and Jacky Baltes • National Taiwan Normal University • Sensors • 2026. Can a humanoid trained in simulation balance a real scooter while an operator tells it where to turn? This work reports that it can. But the achievement is autonomous balance and steering-command tracking, not autonomous transportation: a person still supplies commands, sets the throttle externally, and handles launching and stopping. That distinction is explicit in the published abstract. My take is that this is an interesting robotics integration result, rather than evidence for a breakthrough reinforcement-learning algorithm. The useful question is not whether a neural network can produce steering commands. It is whether the complete chain—from simulated dynamics, through the policy, through humanoid arm kinematics, to an actual handlebar—remains controllable when the model is wrong. Research note: The publisher dates this article to October 7, 2026. During research on October 8, its full-text and PDF endpoints could not be retrieved. This episode therefore distinguishes evidence from the accessible publisher abstract, the authors’ released PPO implementation, and related primary research. Paper-specific trial counts, baseline protocols, and complete ablation tables remain unverified, not necessarily unreported. What the Paper Did. The research concerns a full-sized humanoid physically operating a commercial scooter. The authors’ research program uses a THORMANG3 humanoid and a Gogoro vehicle; their earlier paper studied this combination in simulation and tested whether the real robot could execute the required steering motions. The present article reports actual moving-vehicle transfer. The released implementation uses proximal policy optimization, or PPO. Crucially, its learned action is a desired steering angle—not a vector of humanoid joint torques. An inverse-kinematics layer translates that steering target into arm configurations. The repository explains that this abstraction avoids directly simulating the closed kinematic loop created when both hands hold the handlebars. The observation interface is similarly compact. It includes roll and roll rate, heading and heading rate, forward speed, desired heading, the previous steering command, and the current steering angle. These are state variables describing the vehicle’s motion and control history, not images of the road. The released network configuration specifies three hidden layers of 512 units with ELU activations. That makes the architecture much more conventional than the physical demonstration might suggest. We should describe those dimensions as the released configuration, rather than assume they document every training run behind the publication. Conceptually, the learning problem is: given the present motion, where should the handlebar go next to preserve balance while reducing heading error? The policy does not need to discover human arm geometry through trial and error. Nor does it need to interpret a route, recognize traffic signals, or infer an operator’s destination. Those are different problems, outside this observation–action interface. The released task configuration exposes another important detail: 4,096 learner-facing environments correspond to 2,048 physically simulated environments plus mirrored counterparts. This is symmetry augmentation, not 4,096 independent physical simulations. The distinction matters when discussing computational efficiency and experience diversity. The real-world results reported in the abstract span speeds of 0.8–3.0 meters per second, varying during trials. Success is 96% for straight travel, 92% for light turns, 72% for sharp turns, 88% over a 60-millimeter bump, and 73% with a 70-kilogram passenger. Performance drops to 28% on the reported 4.5-meter roundabout and 33% on part of Taiwan’s scooter-license test. That distribution is more informative than a single aggregate success rate. It suggests that “remain upright while making some progress” and “sustain precise motion through a constrained maneuver” should be treated as distinct capabilities. It also establishes an important evaluation boundary. Heading tracking is not path tracking by itself. Imagine two scooters moving parallel to one another, with identical speeds, headings, and lean states, but different lateral positions. A controller receiving only the listed observations cannot distinguish them. If one is centered in a lane and the other is approaching its edge, some additional component must turn that positional difference into different commands. That is an interface limitation, not a criticism of choosing a compact controller. Compact interfaces are often exactly what makes a difficult learning problem tractable. The mistake would be silently upgrading the resulting skill into a broader autonomy claim. Why the Steering Interface Matters More Than the Network. The most reusable architectural decision here is the separation between deciding how to steer and deciding how the humanoid should move its arms. Consider two possible engineering projects. In the first, you ask reinforcement learning to control every relevant joint while also maintaining hand contact, respecting the handlebar geometry, and stabilizing the vehicle. In the second, you give learning a task-space steering interface and retain a conventional kinematic controller underneath. I would expect those projects to have very different debugging requirements. With the second architecture, an unsuccessful maneuver can be decomposed into more intelligible questions. Did the policy request an inappropriate steering angle? Did the arm controller fail to realize it? Was the requested movement physically achievable under the current load? Did the state estimate arrive too late?. This is a useful way to make a learned controller auditable. It does not require claiming that conventional kinematics solves the entire interaction problem. There is also a limit to what the demonstration says about humanoid intelligence. A successful steering policy does not establish that the robot can acquire the handlebars, change its grip, mount the scooter, recover a lost contact, or coordinate an entirely new tool. The appropriate unit of achievement is the particular closed-loop skill, with its surrounding mechanical and software support. My engineering interpretation is that the task-space interface moves the sim-to-real problem rather than eliminating it. For example, suppose the simulator treats a steering request as a predictable actuator response. On the physical robot, that request passes through an inverse-kinematics solver, joint controllers, structural compliance, and the hand–handlebar connection. The important mismatch is then the difference between the assumed steering response and the actual one. That suggests an acceptance test I would want before comparing learning algorithms: characterize the complete steering interface under controlled conditions. Record requested steering, joint targets, and measured handlebar motion. Look for delay, rate saturation, hysteresis, and load-dependent tracking error. The goal would not be to build a perfect digital twin. It would be to establish what the policy can reasonably demand from its actuator interface. This also changes how I would interpret a failure. If a controller repeatedly asks for steering transitions the physical mechanism cannot execute, increasing the policy network’s size is not an obvious remedy. A better action parameterization, a rate-aware controller, or a more faithful actuator envelope might matter more. Conversely, if the mechanism follows commands accurately but the scooter still departs from the intended maneuver, the investigation can move upward: reward design, observations, training distribution, or command feasibility. The broader lesson is methodological. Before debating whether PPO, a world model, or model-predictive control is the right algorithm, make sure they are solving the same physical problem through the same actuator interface. Training for the Errors That Matter. The repository describes domain randomization over sensing and actuation effects, including measurement noise and offsets, seating offsets, steering damping, and command delay. This is a deliberate attempt to prevent the policy from depending on one exact simulated plant. That principle is established rather than new. Peng and colleagues’ Sim-to-Real Transfer of Robotic Control with Dynamics Randomization demonstrated the broader strategy of training across varied simulated dynamics so that a policy could tolerate discrepancies on physical hardware. The scooter work applies that philosophy to a different coupled robot–vehicle system. For practitioners, the interesting question is which uncertainties deserve representation. I would start from the control interface, not from an indiscriminate list of simulator parameters. If physical steering is delayed, model delay. If the vehicle’s effective mass distribution changes with seating, vary the relevant distribution. If the estimator carries persistent bias, do not substitute only independent, zero-mean noise and assume the two disturbances are equivalent. A policy that survives broad randomization is useful only if the randomized family overlaps the errors encountered in deployment. “More randomization” is not a meaningful objective on its own. I would also separate variation within an episode from variation between episodes. A constant sensor bias creates a different inference problem from rapidly changing noise. Likewise, a fixed added load and a moving load should not automatically be treated as interchangeable robustness tests. One useful implementation detail is that the task computes reward from the clean simulator observation while corrupting the observation presented to the learner. The code also mirrors desired heading and steering variables, not just body orientation. Those choices make the observation transformation and training target worth inspecting together. The symmetry construction offers an inexpensive inductive bias: corresponding left and right situations should generally invite corresponding opposite steering responses. But I would not interpret mirrored samples as fresh independent encounters with new dynamics. They provide structured supervision, not twice as many distinct physical histories. There is some supporting ablation evidence elsewhere in the authors’ work. Their related diffusion-world-model article says that removing domain randomization from the PPO predecessor reduced real-world success to zero. That is a consequential author-reported result, but the original ablation protocol and replication counts could not be checked here. Even if that result is confirmed, its causal scope needs care. Removing an entire randomization package can establish that the package matters. It does not tell us whether sensor bias, actuator delay, mass variation, or their combination is the indispensable ingredient. A useful next experiment would remove those groups separately under matched training budgets, then test on independently measured hardware conditions. That would turn a successful recipe into a more transferable explanation. Finally, robust reaction should not automatically be described as online adaptation. Kumar and colleagues’ RMA: Rapid Motor Adaptation for Legged Robots explicitly introduces an adaptation mechanism using recent experience to accommodate changing conditions. That is a different claim from surviving disturbances with a fixed deployed policy. For this scooter system, I would ask what evidence distinguishes those possibilities. Does performance improve after encountering a changed load? Does history help identify persistent mismatch? Or does the policy simply have enough feedback authority to tolerate it? All three could be useful, but they are not the same scientific result. What the Success Rates Establish—and What They Do Not. The reported experiments provide physical feasibility evidence across multiple maneuver categories. They do not, by themselves, establish a deployment-level reliability estimate. The first issue is the denominator. I could not verify the trial counts behind the reported percentages. As a hypothetical illustration, 24 successes in 25 attempts and 96 successes in 100 attempts both produce a 96% headline. They provide different amounts of information about uncertainty. Nor would knowing the denominator settle everything. I would want to distinguish variation across repeated hardware trials from variation across independently trained policies. A single strong checkpoint can be consistent on repeated runs while the training procedure remains unreliable across seeds. Those are different reproducibility questions: “Does this controller work again?” and “Can this procedure reliably produce a controller that works?”. The second issue is what ends an unsuccessful trial. A fall, an operator intervention, a boundary crossing, excessive tracking error, and failure to finish within a time limit all matter, but they indicate different bottlenecks. Combining them into one binary score can hide whether the dominant problem is safety, precision, or task completion. I would want those failure categories reported separately. This is especially important for interpreting the harder maneuvers. A low completion rate does not necessarily mean the vehicle repeatedly fell. Conversely, a high completion rate need not imply comfortable margins from loss of balance. The third issue is conditioning on the start of the evaluated episode. Because launch and stopping remain manual, the success claim concerns the moving-control segment. It should not be read as a probability of completing an entire unattended journey. That distinction has a practical consequence. A controller may be perfectly useful inside a supervised research workflow while being unsuitable as the only controller in a transportation system. The missing transitions are not merely cosmetic additions to the demonstration; they change the overall operating envelope. The fourth issue is the command distribution. If speed varies during a run, I would want performance broken down by speed and maneuver phase, rather than inferred from the overall range. Passing through a difficult operating region briefly is different from sustaining control there. Likewise, live operator commands can complicate interpretation. An operator might naturally slow the requested heading change when the vehicle looks uncomfortable. That could be sensible supervision, but a benchmark should make the command stream visible. I would therefore preserve the exact command traces and replay matched versions across controllers wherever the experiment permits. Otherwise, differences attributed to the policy may partly reflect differences in what it was asked to do. The passenger result raises another useful question: what kind of load variation was tested? I would want the passenger instructions and, ideally, movement measurements. A person might remain nearly still, unintentionally move with the vehicle, or introduce disturbances. I am not assuming the passenger helped stabilize it. A matched ballast condition would clarify the interpretation. If both conditions succeed, that supports robustness to altered mass distribution. If they differ, the moving human becomes part of the dynamics worth studying rather than simply a heavier test case. The overall pattern also suggests a hypothesis worth testing: balance preservation and rapid heading correction may compete under tighter maneuvers. But that remains a hypothesis, not a diagnosis established by the abstract. To investigate it, I would align heading error, roll, steering saturation, and operator commands around each failure. Did tracking deteriorate before balance? Was steering authority exhausted? Did the command change faster than the plant could safely respond? Those traces would be more actionable than another aggregate score. What Would Count as a Fair Baseline?. The authors’ 2023 predecessor compared PPO with PID in Isaac Gym across low-speed balance and tracking scenarios. It also evaluated steering-command execution on the real robot. That paper is useful historical context, but it is not a substitute for a matched comparison on the present real-world task. I could not verify the current article’s complete baseline protocol. Consequently, I would not turn the reported transfer into a claim that reinforcement learning has beaten classical control generally. For a fair comparison, my first requirement would be a shared interface. Every controller should receive comparable sensor information, face the same delay and steering limits, and operate through the same humanoid arm controller. Otherwise, the experiment risks comparing engineering stacks rather than decision algorithms. My second requirement would be a credible tuning budget. A heavily developed learned policy should not be contrasted with a minimally tuned PID controller and presented as a verdict on learning versus control theory. Conversely, an extensively identified model-based controller should not receive privileged plant information unavailable to the learning system without that asymmetry being acknowledged. I would consider a speed-aware feedback baseline, a model-based controller with an explicit steering-response model, and the learned policy. These are proposed comparisons, not claims that the article already performed them. My third requirement would be common physical metrics: survival, heading error, path error when applicable, steering-rate demands, interventions, and completion. Return under one method’s reward function is not an appropriate universal score. Finally, the most important comparison might not involve a different algorithm at all. It could compare the same PPO implementation with and without a measured actuator model, or with different observation histories. That experiment would answer a practitioner’s question directly: should the next week of engineering effort go into policy optimization, sensing, or the simulator?. Open Code Helps—but Reproduction Still Needs an Exact Target. There is a public PPO repository containing the task, configuration files, scooter assets, export utilities, and real-robot components. Its documented stack includes Isaac Gym Preview 4, IsaacGymEnvs, and , with Python 3.8. The deployment code is described as platform-specific reference material rather than a turnkey system. That is substantially more useful than a paper-only release. It gives another team concrete objects to inspect and a starting point for rebuilding the simulator. But code availability and reproduction of published results remain different standards. I inspected the accessible implementation; I did not execute training or hardware deployment. One particularly instructive detail appears in the released reward calculation. The code computes a direct roll-angle reward but gives it a coefficient of zero. It still uses roll-rate shaping and terminates an episode when absolute roll reaches 0.15 radians. Thus, a casual description of the reward’s named components is not enough to reconstruct the effective objective. This does not mean the policy has no incentive to balance. Termination and the loss of future reward can make falling undesirable even without a positively weighted instantaneous roll-angle term. It does mean that a reproduction should identify the exact effective configuration. I would not call this a paper–code contradiction without the unavailable full text and the corresponding experimental commit. The public implementation could reflect a particular experiment, a later revision, or the intended final configuration. The appropriate response is to request provenance, not infer misconduct or invalidity. For a replication package, I would want the exact commit, checkpoint, normalization state, training configuration, and evaluation command traces bound together. A repository that continues to evolve needs a stable reference for each reported result. I would also test the boundary between training and deployment explicitly. Can the exported model reproduce reference outputs? Are angle conventions consistent? Are observations in the same units? Does the deployed pipeline include the same clipping and action transformations?. These tests are not glamorous, but they are highly diagnostic. A policy can appear to have a sim-to-real problem when the actual problem is an interface mismatch. The available YAML configuration makes architecture choices inspectable, including the actor–critic network specification and externally supplied seed setting. That helps reconstruction, but a seed field alone does not document training variability or tell us which checkpoint-selection rule produced a result. My reproducibility assessment is therefore split. A competent team has a meaningful starting point for reproducing the simulation task and controller design. Reproducing the reported physical success rates requires additional experimental correspondence that I could not verify. That distinction should guide expectations. The repository looks like a research implementation to adapt and audit—not evidence that another laboratory can reproduce the demonstration simply by loading a model onto nominally similar hardware. A Related World Model Offers Context, Not Independent Confirmation. The same authors published Reward-Free Scooter Balance Control via Diffusion World Models with Goal-Conditioned Trajectory Generation on September 11, 2026—before this Sensors article’s publication date. It treats PPO as predecessor work and replaces reward-based policy learning with goal-conditioned trajectory generation. It should not be presented as an independent replication. Its simulator comparison reports roughly half the per-step command variation but about 6% shorter survival than PPO. Hardware validation completed 14 of 19 runs; the matched PPO comparison was performed in simulation, not repeated as a hardware head-to-head. I read that as evidence of a trade-off worth investigating, not an algorithmic successor decisively solving the problem. Smoother commands could be valuable if the physical steering interface dislikes rapid changes. But smoothness should not be optimized in isolation: a controller that avoids necessary corrections can look mechanically gentle while losing task performance. Similarly, comparable average balance does not establish comparable worst-case recovery. The interesting next comparison would synchronize command changes, disturbances, and physical actuator limits, then examine where each controller departs from the recoverable region. For practitioners, this related work reinforces a useful priority. Before replacing a reactive policy with a generative controller, identify the observed failure mechanism. If the bottleneck is limited physical steering authority, changing the learning paradigm may not remove it. If the bottleneck is insufficient anticipation or poorly specified objectives, then the architectural change becomes more directly motivated. What This Means for Practitioners. I would take three things from this work: a useful control decomposition, encouraging physical feasibility evidence, and a clear warning against collapsing different notions of autonomy into one success claim. The part I would consider borrowing first is the task-space action interface. When a robot operates a human tool, it can be productive to let learning select a meaningful tool command while a conventional controller handles the robot’s geometry. That is particularly attractive when the alternative would force the policy to rediscover constraints that are already understood. It also creates places to measure and debug the system separately. However, I would make that interface a documented contract. What commands are admissible? What tracking error is normal? What delay does the policy assume? What happens when the low-level controller cannot satisfy a request?. A steering target is only a useful abstraction if the system communicates what it can actually deliver. For a team extending this research, my first phase would be a frozen simulation reproduction. Preserve the software versions, effective reward, observation transformations, and checkpoint-selection rule. Evaluate multiple independently trained policies, not merely repeated episodes from the best-looking one. The second phase would be a structured robustness study. Hold out combinations of uncertainties rather than only testing each variable separately. For example, a policy might tolerate an added load or additional delay individually, yet struggle when both appear together. I would choose those combinations from measured deployment discrepancies. Randomization should be connected to evidence about the plant, not just to convenient simulator switches. The third phase would be carefully supervised physical validation with clearly classified interventions and failures. The objective would be to discover the boundary of reliable control before expanding the task or adding human exposure—not to maximize the visual drama of the demonstration. For broader autonomy, I would define separate responsibilities for route following, command feasibility, starting, stopping, state-estimation health, and fallback behavior. The balance policy could remain an important component without being asked to stand in for all of them. This is also where evaluation should become more demanding. A transportation-oriented system needs to know when its own control assumptions no longer hold. Successful motion under supervision is not equivalent to recognizing that a requested maneuver is unsafe or impossible. On generalization, I would make narrow claims first. Robustness across maneuvers or loads on one robot–vehicle setup is useful. It does not automatically establish transfer to another scooter geometry, another humanoid, a different contact arrangement, or a new operating surface. Those extensions should be treated as new experiments, with the amount of retuning made explicit. What the accessible evidence cannot establish is equally important: algorithmic superiority over well-matched modern baselines, a precise reliability estimate with verified uncertainty, or a complete autonomous-driving capability. None of those omissions makes the physical achievement uninteresting. A learning-based controller successfully crossing a difficult hardware boundary can be valuable before it becomes a general solution. My final assessment is therefore positive but deliberately scoped. The interesting contribution is making a learned steering-and-balance skill work through a humanoid’s physical interaction with a scooter. The research becomes most useful when we study that interface, the uncertainties it tolerates, and the conditions under which it stops working—not when we treat the vehicle-shaped demonstration as proof of general autonomy. The remaining evidence gap is specific: the accessible abstract and implementation were sufficient for this scoped technical audit, but not for verifying the article’s complete trial statistics, baseline fairness, or ablation methodology. Those checks require the full paper and corresponding experimental artifacts.