Latest / Embodied AI 101 / PPO Beyond the Simulator: What a TurtleBot3 Mapless Navigation Study Really Demonstrates
Transcript
- PPO Beyond the Simulator: What a TurtleBot3 Mapless Navigation Study Really Demonstrates. PPO-Based Sim-to-Real Maples Navigation for TurtleBot3 Mobile Robots in Unknown and Dynamic Indoor Environments • Nabeel Muhamed and Khaleel Ali Khudhur • matching author records associate the researchers with Al-Kitab University and Northern Technical University–Kirkuk • Media Elektrik, Volume 23, Issue 3 • 2026. Research note. The paper title says “Maples Navigation,” but the intended term is almost certainly mapless navigation. As of August 24, 2026, the DOI redirected to the journal’s article record, but the full article page and PDF were not retrievable through the available research interface. The journal’s indexed site still listed Volume 23, Issue 2, published in April 2026, as its current issue, although its publication schedule includes April, August, and December. Consequently, this episode does not invent unavailable hyperparameters, architecture dimensions, or numerical results. It separates what is established by the paper’s title and supplied abstract from the technical interpretation needed to judge the work. The Paper in One Sentence. This work tackles goal-directed indoor navigation as a direct sensor-to-control problem: train a TurtleBot3 policy in simulation with Proximal Policy Optimization, then transfer that policy to a physical robot expected to reach targets and avoid both stationary and moving obstacles without relying on a precomputed environmental map. That framing makes this primarily a systems and deployment paper, rather than a new reinforcement-learning algorithm. PPO is standard. TurtleBot3 is standard. Range-sensor-based mapless navigation is also an established research formulation. The scientific value therefore has to come from how these pieces are assembled, how the simulation distribution is constructed, how the reality gap is handled, and how convincingly the physical experiments demonstrate operation in genuinely unknown and dynamic spaces. That distinction matters. A simulated policy completing a navigation benchmark is now relatively routine. A policy that survives noisy sensing, control latency, imperfect wheel dynamics, localization drift, changing layouts, and moving people or robots is more interesting. And a paper that precisely measures where transfer fails can be more useful than one reporting a high but poorly specified success rate. The right question is therefore not simply, “Does PPO work?” It is: What navigation problem was PPO actually trained to solve, and how much of that problem survived contact with the real robot?. What “Mapless Navigation” Means Here. Mapless navigation usually means that the control policy does not consume an occupancy grid or explicitly run a global graph search over a stored map. Instead, it receives a compact local observation—commonly a two-dimensional LiDAR scan, the distance and bearing to the goal, and sometimes the robot’s current velocity—and directly produces linear and angular velocity commands. This formulation removes an explicit chain of mapping, localization, global planning, local trajectory optimization, and feedback control. The learned policy compresses several of those functions into a reactive mapping from observations to actions. But “mapless” does not mean information-free, and it rarely means localization-free. If the policy is given the distance and angle to a goal, some external component must estimate the robot’s pose or otherwise express the target in the robot’s local frame. That signal may come from wheel odometry, motion capture, AMCL, SLAM running in the background, or a manually managed coordinate system. This hidden dependency is important when interpreting sim-to-real results. A robot may avoid obstacles without a map while still relying on a localization subsystem to know where the target is. If the real-world arena is small and odometry drift is negligible, this works well. Over longer missions, the localization error can dominate even if the local obstacle-avoidance policy is excellent. The “dynamic” part is equally subtle. A single LiDAR frame reports current geometry, not obstacle velocity. Unless the policy receives observation history, stacked scans, tracked obstacle states, or recurrent memory, it cannot explicitly distinguish a stationary obstacle from one moving toward the robot. It can still respond reactively as the scan changes, but that is different from predicting future occupancy. The paper’s task is thus best interpreted as goal-conditioned, locally reactive navigation under partial observability. Closely related research environments often encode observations as sparse laser ranges plus goal distance and heading, with the policy selecting velocity commands and receiving rewards for progress, short paths, collision avoidance, and reaching the goal. This is a useful abstraction for service robots, warehouse platforms, and indoor delivery systems. It is also narrower than unrestricted autonomous navigation. The policy does not necessarily build persistent spatial memory, reason semantically about people, recover from large localization errors, or plan around obstacles outside sensor range. Reconstructing the Methodological Pipeline. The accessible description fixes four essential elements: the platform is a TurtleBot3, the policy is trained with PPO, training occurs in simulation, and deployment targets unknown indoor environments containing dynamic obstacles. The unavailable full text prevents confirmation of the exact sensor vector, network architecture, reward coefficients, simulator, and training budget. Nevertheless, the methodological structure required to implement the stated system is clear. Perception and State Representation. The most transferable observation for this class of robot is a two-dimensional range scan. LiDAR is attractive because simulated and physical measurements share a relatively simple geometric meaning: each element is the distance to the nearest surface along a ray. A typical preprocessing pipeline clips every measurement to a fixed maximum range, replaces invalid or infinite readings, normalizes distances, and either downsamples the scan or groups adjacent beams into angular sectors. Goal distance and relative heading are then appended. Current linear and angular velocity may be added so that the policy knows its own motion state. This produces an observation that is independent of a global map and mostly independent of visual texture. Unlike an RGB policy, it does not need to transfer lighting, material appearance, shadows, exposure, or camera color response. That is one reason LiDAR-based navigation often represents a comparatively favorable sim-to-real problem. The trade-off is lost information. Sparse range readings do not identify whether an obstacle is a wall, chair, person, glass panel, or another robot. Reflective and absorptive materials can also produce invalid measurements. A policy trained entirely on clean simulated rays may learn dangerously sharp clearance margins unless sensor corruption and geometric uncertainty are represented during training. For truly dynamic behavior, scan history is particularly valuable. Two or more temporally separated scans let the network infer approximate relative motion. A recurrent layer can serve the same purpose, although recurrent PPO is more difficult to train and evaluate correctly. Without temporal context, “dynamic-obstacle avoidance” should be understood as fast reactive avoidance rather than motion forecasting. Continuous Actions for a Differential-Drive Base. TurtleBot3 is a differential-drive platform. The natural continuous action is therefore a pair consisting of forward velocity and yaw rate. The actor network produces a probability distribution over these values, actions are sampled during training, and deterministic means are generally used during evaluation. The raw network outputs must be constrained to the robot’s safe velocity envelope. A well-designed deployment layer also limits acceleration, rejects stale commands, and applies emergency stopping when the minimum range falls below a hard threshold. Those seemingly mundane details are part of the sim-to-real method. If simulation allows instantaneous changes from full left rotation to full right rotation, while the physical motors respond slowly, the transferred trajectory can oscillate or overshoot. Similarly, a mismatch between the training control frequency and real sensor update rate changes the effective dynamics seen by the policy. The most credible transfer pipeline therefore preserves not only the action range but also the timing contract: scan frequency, inference rate, action hold time, communication latency, and acceleration limits. Reward Shaping. A navigation policy needs feedback dense enough to learn before accidental goal-reaching becomes vanishingly rare. Most mapless navigation systems construct this from several terms:. positive reward for reducing goal distance;. a large terminal reward for reaching the target;. a large penalty for collision;. a small time penalty encouraging short missions;. possibly a clearance penalty near obstacles;. possibly smoothness or angular-velocity penalties. The exact balance determines behavior. A large progress reward can encourage cutting close to obstacles. A very strong collision penalty can make the robot overly cautious or prone to freezing. A heavy time penalty can produce aggressive trajectories. An angular penalty may create smooth motion but make escape from concave obstacles harder. Reward design also affects what “success” means. If episodes time out with only a small penalty, a policy can learn that stopping is safer than attempting a difficult passage. If near-collision states are not penalized, a high success rate may coexist with operationally unacceptable clearance. The paper’s contribution should therefore be judged partly by whether its reward is described as a reproducible engineering object rather than merely as a collection of empirically chosen constants. PPO Training. PPO alternates between collecting trajectories with the current policy and updating an actor-critic model from those trajectories. The critic estimates expected future return. The actor is trained using an advantage signal indicating whether an action performed better or worse than the critic predicted. PPO’s defining practical mechanism is its clipped objective. If an update would change the probability of previously sampled actions too dramatically, the objective stops rewarding that change. This is not a formal guarantee that every update is safe, but it usually makes optimization less fragile than unrestricted policy-gradient updates. For mobile navigation, PPO offers several practical advantages. It naturally supports continuous controls, can learn stochastic exploration, is widely implemented in mature RL libraries, and tends to be forgiving of moderately complex reward functions. Its main weakness is sample efficiency. PPO is on-policy: data become stale after the policy changes and generally cannot be reused as aggressively as the replay-buffer data used by SAC, TD3, or DDPG. Simulation makes this affordable, but only if the simulator can run faster than real time or execute multiple environments in parallel. A robust experiment should train multiple random seeds. Navigation learning is sensitive to environment generation, initial exploration, network initialization, and occasional reward exploitation. Reporting a single successful checkpoint says little about training reliability. Simulation Environment. A useful simulation distribution needs more than one fixed room. Start poses, goals, obstacle positions, obstacle shapes, corridor widths, and the number and behavior of moving objects should vary across episodes. Curriculum learning is often valuable. The agent can first learn goal seeking in open space, then static obstacle avoidance, narrow passages, clutter, and finally dynamic interactions. Prior sim-to-real navigation work has explicitly used incremental environmental complexity to prevent learning from collapsing when the initial task is too difficult. Dynamic obstacles also need behavior distributions. A policy trained only against objects moving at one speed along deterministic paths can memorize timing artifacts. Better training varies velocity, direction, size, stopping behavior, and interaction angle. Scenarios should include crossing, overtaking, head-on approaches, sudden emergence from occlusion, and temporary blockage of the direct path. Most importantly, evaluation environments must be held out. Randomizing training episodes does not automatically establish generalization if the same room geometry, obstacle library, or spawn rules are reused during testing. Why PPO Is a Reasonable Choice—but Not an Obvious Winner. The paper’s use of PPO is defensible, but PPO should not be treated as synonymous with state-of-the-art navigation. A revealing earlier TurtleBot3 study compared discrete double DQN with continuous DDPG and PPO. In that implementation, the discrete method reached more than 95 percent success after roughly 50 minutes, while PPO and DDPG required several hours and produced weaker policies. The trained models were transferred from a lightweight Unity environment to Gazebo and a physical TurtleBot3. That result does not prove that discrete control is always superior. It shows that algorithm labels are less decisive than observation design, action parameterization, simulator throughput, reward shaping, and hyperparameter tuning. PPO becomes attractive when smooth continuous motion is important and training samples are cheap. Discrete policies can converge rapidly but may produce visibly quantized trajectories unless the action set is carefully designed. Off-policy algorithms such as TD3 or SAC reuse data more efficiently but introduce their own stability and tuning challenges. A 2026 controlled benchmark comparing common continuous-control algorithms found PPO to offer the best overall trade-off in its particular protocol, reaching up to 97.6 percent seed-averaged simulation success. That work emphasized state normalization, LiDAR densification, reward refinements, tuned learning rates, multiple seeds, and aligned evaluation across a Python simulator, Gazebo, and TurtleBot3 hardware. The important lesson is not the headline PPO score; it is that PPO’s performance emerged from an unusually controlled protocol. Muhamed and Khudhur’s algorithmic contribution should therefore be interpreted modestly. PPO is the optimization tool. The interesting scientific content lies in the environment design and transfer procedure. Where the Sim-to-Real Contribution Actually Lives. The reality gap is not a single discrepancy. It is a collection of mismatches that affect different parts of the policy. The first is the perception gap. Real LiDAR contains noise, missing returns, quantization, range clipping, self-reflections, temporal jitter, and occasional measurements from surfaces that the simulator models poorly. Injecting Gaussian noise alone is rarely enough; dropout, bias, delays, and short runs of invalid readings are also useful. The second is the dynamics gap. Wheel friction, battery level, floor material, wheel radius, motor dead zones, center-of-mass placement, and collision response all affect motion. Randomizing these properties discourages the policy from depending on one precise simulated model. The third is the timing gap. Simulator steps are usually regular. Physical ROS systems experience asynchronous sensor messages, transport delays, process scheduling, and occasional dropped updates. A policy trained at ten hertz and deployed against a five-hertz sensor has effectively been placed in a different Markov decision process. The fourth is the environment gap. Simulated obstacles are geometrically clean. Real indoor spaces contain chair legs, cables, reflective surfaces, open doors, irregular edges, and objects smaller than the robot’s assumed collision model. Finally, there is the behavioral gap for moving obstacles. Simulated pedestrians or robots often follow smooth scripted trajectories. Humans stop, hesitate, reverse direction, acknowledge the robot, or move in groups. Success against scripted cylinders is evidence of collision avoidance, but not yet evidence of socially competent navigation. A strong sim-to-real system aligns the software interface and broadens the training distribution. The same scan-processing code should be used in simulation and on hardware. Velocity bounds should match. Domain randomization should cover plausible physical variations rather than arbitrary noise. Real-world measurements should then reveal which residual mismatches matter. This is why deploying on TurtleBot3 is more than a ceremonial final figure. It tests whether the policy learned geometric structure or simulator-specific shortcuts. Contributions and Likely Significance. The paper can be read as offering four system-level contributions. First, it applies a goal-conditioned PPO controller to mapless indoor navigation, bypassing explicit prior mapping in the policy’s control loop. That creates a compact reactive controller suitable for layouts that change often. Second, it includes dynamic obstacles rather than restricting evaluation to static clutter. This matters because many policies that perform well in fixed mazes fail when free space changes during execution. Third, it performs a simulation-to-hardware transition on TurtleBot3. Even a small indoor deployment exposes sensing and actuation effects absent from purely simulated studies. Fourth, it targets a relatively inexpensive, reproducible research platform. TurtleBot3 lowers the barrier to reproducing the general setup and makes the work relevant to laboratories without access to industrial autonomous mobile robots. None of these elements is individually novel. Mapless TurtleBot navigation predates this paper, as do PPO, domain randomization, and real-robot transfer. The potential contribution is their integration and evaluation in a particular unknown and dynamic setting. That kind of contribution should not be dismissed. Robotics advances through systems evidence as well as algorithms. A standard method that works reliably under a documented deployment protocol may be more valuable to practitioners than a highly novel method validated only in simulation. At the same time, the bar for significance is quantitative. The paper needs to show more than a successful demonstration video. We need repeated trials, held-out layouts, specified obstacle motion, failure categories, and comparison against credible baselines. Reading the Results Carefully. The paper’s supplied summary presents the work as addressing the limitations of pre-mapped geometric planning and reports a PPO-based sim-to-real solution. The accessible record was insufficient to verify the exact success rate, collision rate, path length, time-to-goal, number of physical trials, or whether real-world deployment was zero-shot or involved calibration or fine-tuning. The responsible conclusion is therefore qualitative: the study reports that a policy learned in simulation can control a physical TurtleBot3 in an indoor navigation task involving unknown layouts and changing obstacles. That is evidence of feasibility, but the strength of the evidence depends on protocol details. If the real experiment consists of a few hand-selected demonstrations in one room, it establishes that the software stack runs and that the learned behavior can transfer under favorable conditions. If it consists of dozens of repeated trials across unseen rooms, randomized starts and goals, multiple obstacle densities, and independently controlled moving obstacles, it supports a much stronger generalization claim. For context, a 2023 mapless local-planning study trained a modified DQN family in several static and dynamic Gazebo worlds and transferred the resulting policy to a TurtleBot3 Waffle Pi. It reported large relative success-rate improvements over its D3QN baseline, illustrating how dramatically results can depend on reward and exploration modifications even within value-based methods. A later federated growing reinforcement-learning system deployed policies on two TurtleBot3 robots acting as dynamic obstacles for one another. Across 20 repeated real-world trials, it reported an 85 percent navigation success rate and an average successful completion time of 11.89 seconds. The value of that result is not that 85 percent is universally good or bad; it is that the number of trials and interaction setup are stated. The accepted 2026 comparative benchmark goes further by using five training seeds, a fixed held-out simulation set, aligned simulator and hardware conditions, and an extended set of 100 controlled real-robot episodes. Its real layouts are primarily static, so a credible dynamic evaluation in the Muhamed and Khudhur paper could still add complementary evidence. These comparisons show why isolated success percentages are dangerous. A 90 percent result with fast crossing obstacles may be substantially more impressive than 98 percent in sparse static rooms. Without scenario distributions and uncertainty estimates, the numbers are not directly comparable. The Main Technical Limitations to Watch. Mapless Does Not Remove Global Reasoning. A reactive policy can navigate around visible obstacles, but it may fail when reaching the target requires initially moving away from it. U-shaped obstacles, long walls, dead ends, and temporarily blocked corridors create local minima. If the observation contains only local ranges and goal bearing, two globally different situations can look identical. No feedforward policy can always choose correctly because the required information is absent. The practical solution is often hybrid: retain a global planner or waypoint generator while using the learned PPO policy as the local controller. The learned module handles short-horizon motion and dynamic avoidance; the global layer supplies subgoals that avoid topological traps. Recent hierarchical research follows this pattern explicitly. One framework generates subgoals at a high level and uses a safety-oriented learned low-level controller, including a congestion-aware subgoal update mechanism for local-minimum regions. It was evaluated in simulated office, home, and restaurant environments and transferred to TurtleBot3 hardware. Reactive Avoidance Is Not Prediction. If the policy uses only the latest scan, success around moving obstacles comes from repeated reaction. It does not mean the policy understands trajectories or predicts collisions. This distinction affects safety margins. A reactive policy may turn only after a pedestrian has entered its immediate scan geometry. A predictive controller can anticipate that the pedestrian and robot will occupy the same region several seconds later. Adding stacked observations, recurrent memory, explicit velocity tracking, or occupancy-flow prediction would make the dynamic-environment claim more substantive. Reward Penalties Are Not Safety Guarantees. A collision penalty tells PPO that collisions are undesirable on average. It does not guarantee collision avoidance for every state, and it says nothing about environments outside the training distribution. This matters whenever a robot operates near people. A high expected return can tolerate rare catastrophic outcomes if successful episodes are sufficiently rewarding. Practical deployment should therefore add an independent safety layer: emergency braking, conservative range thresholds, velocity-dependent stopping distance, watchdog timers, and possibly a control-barrier or model-predictive shield. Safe-RL formulations can also treat collisions as explicit costs rather than merely negative rewards, but physical safeguards remain necessary. Sim-to-Real Needs Ablations. To establish why transfer succeeded, the experiment should compare at least:. clean simulation training versus sensor-noise randomization;. fixed robot dynamics versus randomized dynamics;. static-only versus dynamic-obstacle training;. feedforward observations versus scan history;. deployment with and without action smoothing;. simulation performance versus hardware performance under matched scenarios. Without those ablations, we can observe transfer but cannot identify its mechanism. Perhaps PPO generalized. Perhaps the simulator happened to match the low-speed physical platform closely. Perhaps the real scenario was simply easier. Reproducibility Remains Central. No associated code repository or project page was located in the accessible record. For this kind of work, code is especially important because most consequential choices live outside the PPO pseudocode: scan preprocessing, normalization statistics, reset logic, collision thresholds, timing, simulator plugins, and ROS topic handling. A reproducible release should include environment files, robot configuration, reward coefficients, trained checkpoints, normalization state, random seeds, and real-robot launch files. How Roboticists Could Build on This Work. The most useful way to adapt the approach is not to copy a PPO network verbatim. It is to preserve the observation-action contract from training to deployment. Start by defining one sensor-processing node used by both simulation and hardware. It should handle invalid LiDAR returns identically, impose the same clipping and normalization, and publish observations at a controlled rate. Record the complete real message stream so it can be replayed offline against the policy. Include goal distance and bearing, current commanded or measured velocity, and a short observation history. Even two or three recent downsampled scans can materially improve reasoning about obstacle motion. Use a curriculum, but do not let the final policy forget earlier skills. Mix open-space, static, narrow, cluttered, and dynamic scenarios during later training. Randomize geometry and physical parameters separately so that failures can be attributed to environmental generalization or dynamics mismatch. Randomize realistic sensor faults rather than only white noise. Insert missing sectors, delayed scans, range bias, isolated false returns, and variation in angular resolution. The training distribution should resemble logs from the actual robot. Constrain actions in physical units from the beginning. Apply the same velocity and acceleration limits in simulation and on hardware. If a smoothing filter will be used on the robot, place it inside the training loop; otherwise PPO learns a plant different from the one deployed. Evaluate on a matrix of conditions rather than one aggregate score. At minimum, report success, collision, timeout, path length, duration, minimum clearance, near-miss exposure, and control smoothness across static and dynamic scenarios. Dynamic tests should vary obstacle speed, density, direction, and interaction type. Use multiple training seeds and bootstrap confidence intervals over evaluation episodes. Separate failures into collisions, local-minimum timeouts, localization failures, perception failures, and policy oscillations. That taxonomy is more actionable than a single success rate. Finally, retain a conventional safety and recovery layer. A production-oriented architecture might combine a classical global route planner, PPO as a learned local planner, a hard emergency-stop monitor, and a recovery behavior invoked when progress stalls. This preserves the adaptivity of learned control without asking one neural policy to solve every level of navigation. Where the Work Sits in the Larger Field. The paper belongs to a line of research that treats navigation as an end-to-end or semi-end-to-end control problem rather than a fixed geometric pipeline. One branch emphasizes efficient discrete control. The earlier TurtleBot3 study by Marchesini and colleagues demonstrated that a carefully chosen discrete action set can learn quickly and transfer successfully, sometimes outperforming continuous PPO in the same setup. Another branch modifies value-based learning through better exploration and multi-step returns, as in the RND3QN navigation work. Such methods can be sample-efficient but inherit discretization choices and can be sensitive to exploration schedules. Continuous off-policy methods such as DDPG, TD3, and SAC trade additional implementation complexity for replay-based sample reuse. PPO instead prioritizes relatively stable, conceptually simple updates. The best choice depends more on the benchmark and engineering constraints than on a universal algorithm ranking. A third branch expands the task formulation. Dimension-variable navigation asks whether a learned policy can adapt to changes in robot footprint. Hierarchical systems add subgoals to escape local minima. Safe-RL approaches distinguish reward from collision cost. Federated systems exchange learned experience among robots. These extensions target limitations that a plain reactive PPO controller does not resolve. Seen against this landscape, Muhamed and Khudhur’s study is best understood as a deployment-focused baseline: a relatively simple policy tested across the simulator-hardware boundary. Its value is strongest if the real dynamic experiments are systematic, and weaker if the physical evidence is only demonstrative. Final Assessment. The central message of this paper is credible and practically important: a compact PPO policy can be trained in simulation to provide mapless local navigation behavior on a physical TurtleBot3, including reactive avoidance in changing indoor environments. But that message should not be inflated into “PPO solves autonomous navigation.” Mapless goal-conditioned control still depends on goal localization. A memoryless policy remains partially blind to obstacle motion. Reward shaping does not provide formal safety. Local policies can become trapped by global geometry. And sim-to-real success in one laboratory does not automatically imply robustness in crowded human environments. The work’s real significance is more grounded. It illustrates a useful embodied-AI pattern: reduce perception to transferable geometric signals, learn continuous local control in simulation, preserve the software and timing interface during deployment, and validate on an accessible physical platform. For practitioners, the takeaway is that the PPO update rule is only a small part of the solution. Transfer quality is dominated by observation preprocessing, timing, action constraints, environment diversity, sensor corruption, and evaluation discipline. For researchers, the next step is equally clear: combine this kind of reactive PPO policy with temporal reasoning, global subgoal generation, explicit safety constraints, realistic dynamic-agent models, and far more rigorous repeated hardware trials. If those pieces are added, a TurtleBot3 navigation demonstration becomes more than an RL proof of concept. It becomes a credible prototype for adaptive local navigation in the messy, changing indoor environments where embodied systems actually have to operate.