Latest / Embodied AI 101 / The Reward Behind the Gait: Learning Transferable Locomotion from a Few Stick-Insect Steps

Transcript

The Reward Behind the Gait: Learning Transferable Locomotion from a Few Stick-Insect Steps · Embodied AI 101

• The Reward Behind the Gait: Learning Transferable Locomotion from a Few Stick-Insect Steps. From Insect Behavior to Transferable Robot Locomotion: Inferring Embodied Locomotor Principles from Limited Data via Adversarial Inverse Reinforcement Learning • Yuchen Wang, Thirawat Chuthong, Mitsuhiro Hayashibe, Poramate Manoonpong, and Dai Owaki • Tohoku University and Vidyasirimedhi Institute of Science and Technology • Bioinspiration & Biomimetics • 2026. The seductive idea behind biomimetic locomotion is that if we record an animal carefully enough, we can make a robot move the same way. Capture the joint angles, fit a policy, replay the motion, and perhaps the machine will inherit some of the animal’s competence. But motion is not a portable object. An insect trajectory is inseparable from insect morphology: limb proportions, compliance, mass distribution, adhesive feet, sensory delays, and substrate interaction. Copying the trajectory onto another body usually transfers the appearance of behavior, not the principle that produced it. This new paper by Yuchen Wang and colleagues asks a more ambitious question. Instead of learning only what movements a stick insect made, can we infer what those movements were trying to preserve? And can that inferred objective be reused when the environment, dynamics, or robot morphology changes?. Their proposed answer is adversarial inverse reinforcement learning, or AIRL. From a short segment of flat-terrain stick-insect walking, the method learns both a continuous locomotion policy and a latent reward representation. The reported experiments then move through an unusually demanding sequence: adaptation to environmental changes, reward transfer between dynamical systems and morphologies, comparison with hand-shaped rewards, and preliminary deployment on a physical bio-inspired robot. The central contribution is therefore not simply an insect-like gait. It is the idea that biological demonstrations can be compressed into a reusable locomotor objective. From copying motion to inferring intent. A locomotion policy and a locomotion reward encode different kinds of knowledge. The policy answers a body-specific question: given this state, what actuator command should this machine issue next? Its answer depends on the robot’s joints, motor limits, inertia, control frequency, and contact model. Even a highly capable policy can be tightly entangled with the body on which it was trained. A reward answers a more abstract question: which transitions are desirable? It might favor forward progression without collapse, coordinated redistribution of support, appropriate swing timing, or smooth recovery from disturbances. A reward does not by itself specify how a particular machine should realize those preferences. A new policy must still be optimized for the new body. That distinction is the conceptual heart of the paper. If the robot changes, transferring the original policy asks the new machine to reuse another body’s muscle memory. Transferring the reward lets the new machine develop its own muscle memory while preserving the inferred objective. Wang and colleagues report both kinds of evidence. The policy learned from the biological demonstration generates flexible coordination under conditions not represented by the short flat-ground training segment. More importantly, the learned reward network can supervise policy learning in robot models with different dynamics and morphologies. Relative to training with manually shaped rewards alone, the inferred reward accelerates convergence and produces coordination that is closer to the biological patterns. This makes the work relevant beyond insect-inspired hexapods. Most cross-embodiment learning pipelines eventually encounter the same question: which part of a demonstration is tied to the demonstrator’s body, and which part expresses a transferable task principle?. Why stick-insect gait is a difficult learning target. Stick insects are useful precisely because their walking does not reduce cleanly to a small library of named gaits. A textbook description might divide hexapod locomotion into tripod, tetrapod, and wave gaits. Real insect behavior is less discrete. Coordination changes continuously with speed, sensory context, body geometry, and terrain. Individual steps often produce noncanonical support patterns that lie between familiar gait categories. Variability is not necessarily noise around an ideal template; it can be part of the control strategy. This creates a problem for conventional reward design. A roboticist can easily reward forward velocity and penalize falling, torque, or action variation. It is much harder to hand-code the probabilistic relationships that make a gait insect-like without forcing the controller into a rigid sequence. The present paper grows out of an earlier line of work by Wang, Hayashibe, and Owaki. In their 2025 paper, Encoding Flexible Gait Strategies in Stick Insects through Data-Driven Inverse Reinforcement Learning, they modeled gait selection using maximum-entropy inverse reinforcement learning. Walking velocity, direction, acceleration, and antennal motion were incorporated into the state, while canonical and noncanonical gait configurations formed a discrete action space. The recovered policy reproduced the observed probabilistic gait transitions and highlighted the role of tactile exploration in gait modulation. That earlier formulation was primarily explanatory. It asked why an insect selected one support pattern rather than another. It did not yet solve continuous motor control or demonstrate transfer to a robot with different dynamics. Viewed as a research progression, the new paper closes that gap. The problem moves from discrete gait classification to continuous locomotion, from retrospective explanation to policy synthesis, and from a reward that describes insect choices to one that can train robotic bodies. The collaboration itself reflects that shift. The Tohoku group brings work on extracting structure from biological motion, while Thirawat Chuthong and Poramate Manoonpong’s VISTEC research has emphasized adaptive neural control and insect-inspired robotic platforms. Their earlier systems investigate self-organized interlimb coordination, heterogeneous leg dynamics, and rapid adaptation to physical damage. AIRL as a machine for separating behavior from embodiment. Inverse reinforcement learning begins from a simple inversion of the usual RL problem. In standard reinforcement learning, the reward is specified and the policy must be discovered. In inverse reinforcement learning, demonstrations are given and the algorithm tries to infer a reward under which those demonstrations make sense. The difficulty is that behavior does not uniquely determine reward. A walking trajectory can be explained by many objectives. One reward might value speed, another stability, and a third a particular sequence of contacts. All could make the observed trajectory appear optimal within the small region covered by the data. Adversarial inverse reinforcement learning addresses this problem through a competition between a policy and a discriminator. The discriminator sees transitions from the biological demonstrations and transitions generated by the current policy. It learns to distinguish them. The policy, meanwhile, improves by seeking transitions that the discriminator will classify as expert-like. This is related to generative adversarial imitation learning, but AIRL adds an important structural ambition: it tries to expose a reward that is separable from the policy and, under the right assumptions, less dependent on the source dynamics. In the standard AIRL construction, the discriminator’s score is divided conceptually into two pieces. One represents the immediate desirability of the current situation or action. The other measures a change in a learned potential—roughly, whether the transition moves the system toward a more promising region. That second term accounts for progress-based shaping without requiring it to be mistaken for the underlying task objective. This matters because naive inverse learning can recover a reward that merely memorizes the source system’s transition mechanics. Such a reward may recognize how one particular simulator moves rather than what the demonstrator values. The AIRL parameterization is intended to reduce that entanglement. The training loop is therefore doing three things at once. It is matching the distribution of learner transitions to the biological data, improving a continuous control policy through interaction with a simulator, and extracting a scalar objective that can later be detached from that source policy. The result is not a literal model of an insect nervous system. AIRL does not identify neurons, reflex pathways, or central pattern generator circuitry. It offers a computational description at another level: a reward landscape under which the observed coordination is rational. That distinction is essential when the paper speaks of “embodied locomotor principles.” The inferred principle is not necessarily a rule represented explicitly inside the animal. It is an objective-level explanation of behavior—one that is useful if it predicts new behavior and transfers to new embodiments. There is also an important sample-efficiency nuance. The method uses limited biological demonstration data, but AIRL is not interaction-free. The learner still needs policy rollouts, typically in simulation, to create negative examples and improve its behavior. The paper’s data-efficiency claim is therefore about conserving expensive animal observations, not necessarily minimizing total environment steps. For biological robotics, that can still be a good trade. Recording clean, synchronized, full-body animal motion is difficult. Simulated robot interaction is comparatively cheap and repeatable. What a short flat-ground demonstration can contain. At first, learning adaptive locomotion from a short flat-terrain segment sounds implausible. The demonstration does not explicitly show rough terrain, morphology changes, or sim-to-real errors. But a compact locomotion sequence can contain more reusable structure than its environmental variety suggests. Each gait cycle repeatedly demonstrates how support shifts around the body, how swing events are interleaved, how forward progression couples to contact timing, and how transient deviations are absorbed. The learner sees local relationships among state, action, and next state many times, even if the global terrain remains constant. AIRL can then use those relationships as constraints rather than as a trajectory to replay. If the inferred reward favors a certain organization of support and progression, a closed-loop policy can seek that organization on a new surface without reproducing the exact original joint path. This helps explain why reward learning can extrapolate where behavior cloning often fails. A cloning policy is trained to output the action associated with a demonstrated state. Once a disturbance pushes it outside that state distribution, it may have no principled way to recover. A reward-based learner can search for actions that return it to a desirable region, including states never occupied by the demonstrator. The resulting adaptability is partly a property of the learned objective and partly a product of embodiment. Feet collide with the terrain, body motion changes sensory state, and the policy reacts. Coordination emerges from the loop among reward, controller, dynamics, and environment. So the work is not claiming that the short data segment explicitly contains every future response. It claims that the segment reveals enough of the underlying preference structure for new responses to be synthesized through interaction. The experimental ladder: imitation, adaptation, transfer, hardware. The evaluation described by Wang and colleagues has four increasingly strong levels:. Learn continuous locomotion from a limited biological demonstration. Test whether the resulting policy retains adaptive coordination under changed environmental conditions. transfer the learned reward to different dynamical systems and robot morphologies, then learn new policies there. Deploy the learned behavior preliminarily on a physical bio-inspired robot. The first level establishes imitation. The second asks whether the learner has acquired something more than an open-loop gait. The third is the most scientifically interesting because it separates objective transfer from controller transfer. The fourth tests whether the behavior survives at least some portion of the simulation-to-reality gap. The reward-transfer experiment should be read carefully. A neural reward cannot be copied blindly between arbitrary robots if its inputs are raw, morphology-specific joint coordinates. Transfer requires either a shared feature interface, a mapping between embodiments, or state variables with common behavioral meaning. For example, body-frame velocity, normalized foot position, contact state, phase relationships, and support distribution can remain meaningful across two hexapods even when their segment lengths or inertial parameters differ. Raw motor angles may not. The transfer result therefore implies some degree of representational alignment, not complete indifference to morphology. That is still a meaningful achievement. Most locomotion policies bundle together three elements: task preference, embodiment, and solution strategy. By reusing the reward while relearning the policy, this framework attempts to retain the first while allowing the other two to change. The comparison with manually shaped RL is equally important. Hand-shaped locomotion rewards often combine velocity tracking, uprightness, foot clearance, energy, smoothness, and contact penalties. These terms can produce competent walking, but they encode the designer’s assumptions about how walking should look. The paper reports that the AIRL-derived reward improves convergence and biological consistency relative to reward shaping alone. That suggests the inferred signal offers a useful learning curriculum: it makes biologically plausible coordination easier to discover than generic task rewards do. It does not eliminate engineering rewards entirely, but it can supplement them with information that would be difficult to specify by hand. The hardware result is appropriately described as preliminary. A physical demonstration establishes feasibility, but it is not yet evidence of broad real-world robustness. Long-duration walking, terrain diversity, actuator variation, delays, impacts, and mechanical wear remain much harder tests than a controlled deployment. How this differs from established bio-inspired control. The bio-inspired locomotion literature already contains remarkably capable insect-like controllers. The novelty here is not that AIRL suddenly introduces adaptability where none existed. It is that the source of the coordination changes from explicit design to inferred behavioral objective. Consider the adaptive neural controller developed by Manoonpong, Parlitz, and Wörgötter for the AMOS II hexapod. That system combined a central pattern generator, sensory feedback, adaptive forward models, and local elevation or searching responses. It could generate many gait patterns, adapt individual legs to terrain, search for footholds, handle obstacles, and tolerate simulated leg damage. Its strength came from a carefully structured, biologically inspired control architecture. That approach offers strong inductive bias and interpretability. The designer knows which subsystem generates rhythm, which predicts contact, and which triggers an elevation reflex. The cost is that the architecture, couplings, and response mechanisms must be selected and tuned. A more recent study by Larsen, Manoonpong, Chuthong, and colleagues trained a central-pattern-generator and radial-basis-function network using measured stick-insect foot trajectories. The biological data shaped intralimb motion, while decentralized neural interactions produced self-organized interlimb coordination. The authors also identified limitations: the controller relied on two-dimensional foot trajectories, lacked richer posture adaptation, required additional mechanisms for highly uneven terrain, and did not recover every gait family. That system can be viewed as direct transfer of a biological motion primitive into a structured controller. The new AIRL paper instead tries to transfer the criterion that makes a motion good. The learner is freer to choose another trajectory if the body or terrain demands it. Chuthong and Manoonpong’s work on insect-inspired resilient machines provides another complementary point. Their decentralized adaptive neural control exploits local feedback, neural dynamics, plasticity, and physical interaction to recover rapidly from leg loss without an expensive retraining phase. This is embodiment-driven online adaptation rather than inverse learning from demonstration. These approaches need not compete. A CPG or decentralized reflex network could define the policy class, while an AIRL-derived reward tunes its parameters or selects among its behaviors. That hybrid would combine biological structural priors with a data-derived objective. Compared with behavior cloning, AIRL sacrifices simplicity for recoverability. Cloning is attractive when demonstrations densely cover deployment states and embodiment remains fixed. AIRL is more compelling when disturbances and morphology changes are central to the problem. Compared with ordinary RL, AIRL replaces some reward engineering with demonstration inference. It does not remove design decisions; it relocates them. The researcher still chooses the state representation, action space, policy architecture, simulator, regularization, and which biological data count as expert behavior. That relocation may nevertheless be valuable. It is often easier to record an animal expressing a coordination strategy than to formalize the strategy as a differentiable objective. What the findings do—and do not—tell us about insect control. The strongest interpretation of the paper is that a compact set of behavioral observations contains a reward-like structure that is useful across conditions and bodies. A weaker but still interesting interpretation is that AIRL discovers a good locomotion prior correlated with insect behavior. Distinguishing those interpretations requires care. Inverse reinforcement learning is fundamentally non-identifiable. Many rewards can explain the same demonstration, particularly when observations cover only a narrow part of the state space. AIRL’s reward structure reduces some forms of ambiguity, but it cannot guarantee recovery of the animal’s true internal objective under partial observability, noisy measurement, unknown dynamics, or boundedly rational behavior. Biological behavior is also not necessarily optimal. An insect may be fatigued, exploring, reacting to hidden sensory input, or balancing several unobserved goals. Treating every recorded transition as expert evidence can fold those factors into the inferred reward. The chosen representation acts as a scientific hypothesis. If the learning state contains only limb kinematics and body velocity, the recovered reward cannot explain effects that depend on load receptors, antennal contact, substrate compliance, or internal neuromodulatory state. The earlier maximum-entropy IRL study made this issue visible by showing that antennal variables helped explain gait selection. Removing such variables would change what “principle” could be recovered. Biological consistency must likewise be interpreted behaviorally. Matching phase relations, contact sequences, or gait statistics does not establish that the robot and insect use homologous neural computations. A neural network reward and an insect thoracic network may produce similar observable coordination for entirely different mechanistic reasons. The environmental generalization result is promising, but it should not be conflated with learning the complete adaptive repertoire of a stick insect. Some robustness may arise from the robot’s mechanical design, low-level controllers, simulator randomization, or generic closed-loop feedback. Those are not confounds to be eliminated—they are part of embodied intelligence—but their contributions should be separated experimentally. Finally, “limited data” should be quantified in terms of independent information, not merely frame count. Locomotion recordings contain many highly correlated samples from repeated cycles. A short trajectory can yield thousands of transitions while representing relatively few distinct coordination events. Future comparisons should report duration, number of cycles, individuals, speeds, and effective behavioral diversity. None of these caveats undermine the paper’s central result. They clarify its level of explanation. The method recovers a transferable computational objective from behavior; it does not decode the insect nervous system. How roboticists could adapt the framework. For researchers applying this idea to another animal or robot, representation design is likely to matter more than the choice between two nearby policy optimizers. The demonstration should first be expressed in variables that capture the candidate invariant. For cross-morphology locomotion, useful features might include body velocity in a local frame, roll and pitch, contact state, normalized foot location, duty factor, relative phase between neighboring limbs, local load, and progress relative to body length. Normalizing distances by limb or body dimensions can prevent the reward from memorizing one animal’s scale. Raw joint angles can still be useful when source and target are closely matched, but they are a poor universal interface. If the eventual goal is transfer from a hexapod to a quadruped, the reward should probably operate over support and coordination relations rather than six hard-coded leg identities. The action space presents a parallel choice. A fully unconstrained torque policy offers expressive control but imposes a large learning problem. A structured policy could instead output desired foot locations, oscillator parameters, phase resets, or corrections to a reflex controller. AIRL would then infer the objective while biological priors restrict the solution space. A practical development sequence would look like this:. Build a reliable demonstration pipeline with synchronized kinematics and contact inference. Define body-centric, normalized features that remain meaningful on the intended target robots. Train AIRL in a source simulator, separating expensive expert data from inexpensive learner interaction. Test the reward interventionally by changing friction, mass, limb length, terrain, and sensory availability. Freeze the reward, discard the source policy, and optimize new policies for each target morphology. Add dynamics randomization, actuator models, delay, and conservative safety constraints before hardware transfer. The reward should be evaluated independently of imitation quality. A policy can reproduce demonstrations while relying on a brittle or misleading reward. One useful test is to generate candidate trajectories that differ from the demonstrations in controlled ways and ask whether reward rankings agree with biological or engineering expectations. Another is policy diversity. If several independently trained policies maximize the reward, do they converge on related support relations, or do some exploit simulation artifacts? Reward hacking is especially dangerous in locomotion because small inaccuracies in contact or collision models can create high-scoring but physically meaningless strategies. Reward ensembles could expose uncertainty. If several AIRL models trained on bootstrap subsets agree on a transition, confidence is relatively high. If their scores diverge in a new terrain, the controller can fall back to a conservative stabilizing policy or request additional demonstrations. The most compelling hybrid may combine a learned global reward with local embodied control. The reward can specify that stable progression and insect-like coordination are desirable, while reflexes handle immediate contact loss and obstacle collision. This avoids asking a single neural policy to rediscover every fast stabilizing mechanism from scratch. A graph-based formulation is particularly attractive. Each leg can be represented as a node, with phase, contact, load, and local kinematics as features. Edges can encode ipsilateral and contralateral relationships. A reward learned over local nodes and edges would be easier to transfer across limb lengths—and potentially across limb counts—than a monolithic reward tied to one state vector. The larger research opportunity. This paper points toward a version of bio-inspired robotics that is less literal and potentially more scalable. Traditional biomimetics often begins by identifying an anatomical or neural mechanism and reproducing it in hardware or control software. The AIRL approach begins with behavior and asks which objective can make that behavior reusable. It treats the animal less as a trajectory generator and more as evidence about a good solution landscape. The next step should be intervention-rich biological data. Flat walking identifies nominal coordination, but perturbations are more informative about objectives. Demonstrations involving slips, gaps, obstacles, altered sensory feedback, or temporary limb impairment reveal what the animal sacrifices and what it preserves. Multi-individual and multi-species learning could separate universal principles from morphology-specific habits. A shared reward might capture broad support-management rules, while species-specific terms encode ecological and anatomical specialization. There is also room to connect reward inference back to biology. If the learned reward predicts gait changes under conditions not used for training, it becomes a testable behavioral model. Researchers could compare its predictions with insect responses to tactile deprivation, load manipulation, or changed substrate properties. In that setting, AIRL is not merely a robot-training tool; it is a generator of biological hypotheses. The takeaway. The most important result in From Insect Behavior to Transferable Robot Locomotion is not that a robot can imitate a stick insect. We already know how to produce convincing insect-like gaits. The advance is the separation of movement from locomotor preference. A short biological demonstration supplies evidence about what coordinated walking should preserve. AIRL turns that evidence into both a source policy and a reward that can survive changes in environment, dynamics, and morphology. The policy is disposable; the inferred objective is the intended transferable asset. That claim still comes with familiar IRL limitations: reward ambiguity, representation dependence, simulator interaction, and preliminary sim-to-real validation. But it suggests a productive direction for embodied AI. Rather than asking robots to inherit an animal’s exact movements, we can ask them to infer the reason those movements work—and then rediscover the solution in their own bodies.