Latest / Embodied AI 101 / Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning for Long-Horizon Contact-Rich Manipulation

Transcript

Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning for Long-Horizon Contact-Rich Manipulation · Embodied AI 101

• Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning for Long-Horizon Contact-Rich Manipulation. Long-horizon manipulation – think of a robot assembling a complex device or tidily stowing a dozen dishes – is one of the hardest challenges in embodied AI. In these tasks, the robot must coordinate many sequential steps, handle uncertainty from occluded observations, and often make delicate physical contacts (say inserting a peg or nesting cups together). Traditional imitation learning or end-to-end policies tend to treat the entire sequence as one giant problem, and they often falter: subtle contact dynamics and partial observability cause flat policies to lose track of where they are in the task. Researchers have long conjectured that hierarchical structure is needed – you want a high-level planner for “which sub-goal am I on?” and a low-level controller for “how do I actually move my joints right now?” – but actually building and training such a system is quite intricate. Bi-HIL (Bilateral Hierarchical Imitation Learning), a recent framework from Buamanee et al. (2026), stitches together several ideas to stabilize long, contact-rich tasks via imitation. In a nutshell, Bi-HIL uses bilateral teleoperation to collect force-sensitive demonstrations, organizes the task into explicit subtasks with progress-tracking, and gives the high-level policy a short-term visual memory of completed subtasks (note: these are keyframes from the demonstration). The result is a two-level policy where the high level chooses which subtask to do and how far along it is, and the low level executes motions with awareness of the current force context. In experiments on real robots – both one-arm and two-arm setups – Bi-HIL showed dramatic gains over baselines. For example, in a 12-step “put three balls in drawers” experiment, the hierarchical Bi-HIL model scored 100% success (for both of two initial configurations), whereas a flat bilateral imitation model (the authors’ prior Bi-ACT system) only managed 60% on one side and 100% on the other. In even tougher bimanual tasks like stacking cups or inserting pegs, Bi-HIL again yielded ~80% success versus ~20–60% for the flat model. These results underscore a simple message: for very long, contact-rich tasks, a flat reactive policy just doesn’t cut it; you need structure and memory. In this article we’ll unpack Bi-HIL in detail. We’ll start by motivating the problem and reviewing why both hierarchy and force awareness matter. Then we’ll walk through the Bi-HIL pipeline: how bilateral teleoperation collects the data, how the high-level policy reasons over subtasks and progress, and how the low-level controller executes with forces in mind (it even uses a CVAE to handle action ambiguity). Along the way we’ll connect to related ideas (e.g. subgoal-based planning, keyframe memories, vision–language models) and interpret the authors’ results. Finally we’ll reflect on what this approach means for future embodied AI and how one could adapt it to different tasks or systems. The Challenge: Long-Horizon, Contact-Rich Tasks. Imagine a robot handed a set of building blocks and asked to stack them in a specific order, or given a set of pegs and holes and told to insert the right pegs into the right holes. These are contact-rich tasks: the robot must feel as much as see. Tiny misalignments require force-sensitive corrections, and each insertion or grasp is a delicate event. Now add long-horizon on top: perhaps the robot must do 10–20 such insertions or placements in a particular sequence. And the initial setup might not be obvious: maybe two blocks are swapped, or two pegs look almost identical, so the correct order changes. In this regime, two key problems arise:. Partial Observability: A fixed camera view often cannot capture all necessary information. The robot may have to rely on memory or inference about what it has already done. Its current sensory snapshot doesn’t tell the whole story of the task state. Contact Uncertainty: Contacts introduce noisy and unpredictable forces. For example, when inserting a peg, the friction or required torque might vary; the robot must adjust on-the-fly. A policy trained only on vision could easily slip or apply too much force, damaging the part or phantom-failing. Flat end-to-end policies (even with temporal models like recurrent nets) tend to be brittle under these conditions. Buamanee et al. point out that existing approaches “often rely on flat policies that struggle with long-horizon coordination”. Intuitively, a flat policy must learn to implicitly remember both where it is in the overall sequence and how to do the next action – a lot of burden. Hierarchical structures can help. A typical idea is to break the task into subtasks or phases. For example, “pick up green ball,” then “put green ball in top drawer,” then “pick up red ball,” etc. If the system can explicitly switch tasks at the right moments, it doesn’t have to remember everything all the time. But achieving this is tricky: how does the robot know when one subtask is done? How does it know what the next subtask should be? And how can it safely manage contacts throughout?. Previous hierarchical imitation learning (HIL) work has experimented with multi-level controllers, sometimes using language as intermediate commands or using generative subgoal predictors (see for example recent works like GHIL-Glue). GHIL-Glue in particular used image/video predictors to generate intermediate goal images for low-level policies, but this can be brittle (the generative model might hallucinate a physically impossible state, confusing the controller). Bi-HIL takes a different tack: rather than hallucinating new images, it reuses actual images from demonstrations as “keyframes.” And instead of learning a vague sense of progress, Bi-HIL explicitly computes a subtask-level progress rate (a number between 0 and 1) that indicates how far the current subtask has progressed. These innovations – keyframe memories and explicit progress – are the heart of Bi-HIL’s stability. Finally, Bi-HIL’s bilateral control aspect solves the contact problem. Instead of just collecting dry video of someone controlling a robot, they attach the demonstration robot in a “leader-follower” setup. A human operator moves the leader arm by hand (often with force feedback from the follower arm doing the task), so that the recorded data includes both joint motions and the torques (forces) encountered. This means the learned policy can be force-aware, leading to safer contacts. This idea – combining action chunking and bilateral sensation – was explored in prior work by the same team (e.g. Bi-ACT); Bi-HIL extends it by adding hierarchy and memory. In short, the authors motivate Bi-HIL by pointing at exactly these challenges: “long-horizon contact-rich manipulation remains challenging due to partial observability and unstable subtask transitions under contact uncertainty”. Bi-HIL’s design explicitly attacks those issues. Bilateral Teleoperation and Data Collection. Before any learning happens, Bi-HIL needs rich demonstration data: videos, joint states, and, crucially, force information from touch. To get that, the authors use a four-channel bilateral control setup (see Figure below, not shown) in both one-arm and two-arm experiments. What is that? Essentially, there are two copies of the robot: a leader that the human operates in real-time, and a follower that actually does the task. They are coupled so that they move together (action–reaction) yet the follower can push back on the leader to communicate forces. Imagine the human holding and moving the leader arm. The follower arm then mimics the motion in the environment (pulling a drawer, inserting a peg, etc.). If the follower experiences a contact force, that force is transmitted back to the leader arm (like a force-reflecting joystick). The synchronization is enforced at high rate (1000 Hz control loop), ensuring “force consistency via an action–reaction relationship”. During these teleoperation demonstrations, all the important signals are recorded: the joint angles, velocities, and torques of the follower (and leader) are saved, along with RGB camera images from multiple viewpoints. Because it's bilateral control, the recorded torques reflect real contact forces from the environment (rather than just control commands). After a demo run is recorded, the authors manually annotate the subtask boundaries and corresponding instructions (like “grasp green ball” or “insert peg 3 in hole 2”) based on the video. These annotations are what wires up the hierarchy during training, but importantly they do not affect how data is collected: the human just performs the task naturally, and later the trajectory is segmented by hand. This data pipeline has two benefits. First, the force feedback makes the resulting policy force-aware. Prior work (including their own Bi-ACT model) already showed that having torque information in training improves contact executions. Second, the segmentation and language instructions provide high-level labels: each frame of demo data is associated with “subtask X” and which phase of it. In Bi-HIL these labels train the high-level controller to recognize tasks and predict progress. In summary, Bi-HIL demos consist of multi-camera video, joint states, and torque readings, all synchronized. The bilateral scheme ensures the robot is “aware” of contact forces as it moves. Once collected, this data is the launchpad for the two-level policy learning. Hierarchical Policy Architecture: High-Level and Low-Level. Bi-HIL’s policy has two tiers: a High-Level Policy that reasons over subtasks, and a Low-Level Policy that generates actual joint actions. Each plays a distinct role, and crucially they communicate via a key idea: the high level tells the lower level which subtask we are on and how far along it is, and it also provides some visual memory cues. Let’s jump into each one. High-Level Policy. At the high level, the policy’s job is to decide what subtask to do next, and to estimate the progress within that subtask. For example, if the current subtask is “move green ball to top drawer,” the high-level controller should watch the video and decide when the green ball is grasped, when the drawer is reached, etc. Concretely, Bi-HIL’s high-level policy is a transformer encoder (a modern architecture suited to sequence understanding). It has two novel inputs:. Current Visual Observations: A short window of the latest frames from each camera (they use multiple cameras around the robot). Keyframe Memory: A set of representative frames from completed subtasks (which we’ll explain in a moment). On each high-level step (they run this at some slower rate than the low level), the policy takes ~N recent images plus, say, K keyframes. It encodes all of them via the transformer. The outputs of this high-level network are:. a subtask command (essentially which subtask index to execute next, in natural language token form),. a subtask-level progress rate $p_t$ (a number in [0,1]), and. keyframe scores for each frame in the current window. The progress rate $p_t$ is a core novelty: it represents how far along the current subtask is. It is defined as the normalized completion ratio of that subtask: 0 means just started, 1 means finished (reset back to 0 when a new subtask begins). During training, the true progress can be inferred from annotations: if a subtask takes 10 seconds in the demo, a frame 3 seconds in would have progress 0.3, etc. The network learns to predict this continuous progress value. In practice, the authors quantize it into 10 buckets for robustness, so it’s really a 10-way classification of “which decile of the task are we in”. The keyframe scores tell the policy which recent frames should become part of the memory. Intuitively, if the subtask is done or some landmark is hit, the policy can say “remember that frame”. Concretely, after the high-level policy outputs scores, the chosen frames with highest scores are added to the keyframe memory (and some old keyframes may be dropped if memory is limited, though the details aren’t fully specified in the paper). Thus, over time, the set of keyframes accumulates snapshots at important moments. Why keep keyframes at all? Keyframes serve as visual anchors of completed subtasks. In other words, they remind the policy of what has been achieved. In a partially observed setting, the current camera view may not capture that you actually inserted peg 1 and 2, for instance. But if you have stored an image of “peg 2 inserted” from earlier, the high-level policy can attend to that image and know that previous steps are done. This is reminiscent of memory-augmented policies or experience replay; indeed, the authors cite MemER, a recent method for scaling up memory in robot control. The keyframe mechanism is simpler, focusing only on picking a few representative frames rather than storing every past frame. Summarizing the high level: at each step it sees “recent camera frames + memory frames”, and it outputs “subtask ID, progress, which frames to add to memory”. The transformer can attend between current visuals and memory, giving it a short-term episodic memory. By conditioning on progress, the policy knows “I’m halfway through moving the green ball, so don’t start the next task yet.” If progress is near 1.0, the policy will transition to the next subtask. This structure stabilizes long-horizon reasoning: the policy explicitly tells itself where it is in the sequence, rather than trying to infer it implicitly from noisy visuals or repeatedly restarting tasks. Low-Level Policy. The high level says what to do; the low level does it. Specifically, the low-level policy receives the high-level outputs plus the live sensory data and children the actual motor commands. It runs much faster to produce smooth continuous control. The inputs to the low-level policy are:. The subtask command (encoded as text or a token) from the high level,. The predicted progress rate (as a small discrete embedding),. Current visual inputs (RGB images from the follower robot’s cameras),. Current robot state (joint angles, velocities, and torques of the follower). Because Bi-HIL is trying to execute force-sensitive tasks, including the follower’s torques in the input means the policy is aware of the contact forces right now. The output of the low-level policy is the next joint state of the leader robot (angle, velocity, torque) – effectively, the control command to apply (since the leader is being guided by the learned policy now, in the absence of the human). What is this low-level network? The authors implement it as a conditional variational autoencoder (CVAE) with a Transformer backbone. In plain terms: it’s a neural net that can represent a distribution over actions given the input context. This choice is sensible because even given “subtask command + visual scene,” there may be multiple equally good ways to move (especially under force uncertainty). A CVAE can capture such multimodal outputs better than a single deterministic network. Concretely, the model takes the text command (embedded via a pretrained SigLIP language-image model) and images, processes them, and generates a latent vector. The decoder then predicts the next joint targets. A neat detail is that the vision and language modalities are fused via FiLM (Feature-wise Linear Modulation). The SigLIP embedding of the subtask command is used to modulate the CNN processing the RGB (they use an EfficientNet). So language “interacts” with vision early on. Meanwhile, the progress scalar is appended as an extra feature after discretization into one-hot (possibly by embedding its 10 classes). All of these condition the CVAE. The policy then outputs a distribution (implicitly via latent sampling) to generate joint angles, velocities, and torques – u.s.e., actually outputs the motor torques to apply. A key point: this low-level controller has access to the torques of the follower robot as an input. In execution time, the follower’s torques are mirrored to the leader. By learning from bilateral data, the CVAE knows how to pay attention to force cues. For instance, if a peg got stuck and the torque got high, the next action might back off or rotate slightly – something a vision-only policy might miss. To sum up: Low-level Bi-HIL is a CVAE-controlled network that translates “subtask instruction + progress + current sensor (image, state, forces)” into “next motor action.” It thus enforces that all actions remain consistent with the high-level plan, while still being reactive to immediate visual and tactile feedback. Putting It All Together: Execution. During inference (i.e. when actually running the robot), the system loops as follows. The high-level policy operates at a slower cadence: every fixed interval (could be seconds, not specified but presumably matched to subtask scale), it ingests the latest camera frames and keyframe memory, and outputs (1) which subtask to target, (2) an updated progress value, and (3) which of the latest frames (if any) to save into memory. The keyframes typically correspond to momentary completion of some subtask (e.g. just after inserting the green ball). These are appended to the memory bank for future reference. The low-level policy runs every control step (the authors’ robots do 1000 Hz control) and gets the most recent observation plus the current high-level outputs (instruction, progress). It then outputs the next joint state to command. This two-tier handoff ensures that higher-order planning (when to change subtasks) is decoupled from fine motor details (joint velocities, etc.). The overall Bi-HIL loop may look like:. Initialize: high-level starts with progress=0, first subtask = “grasp green ball” (for example). Memory is empty. Low-level execution: the CVAE controller, guided by “grasp green ball”, starts moving the arm. The feed-forward policies make continuous corrections with force awareness. Progress update: as the arm closes on the ball, the high-level network incrementally increases its predicted progress. Perhaps at 100% progress, it sees that the ball has been grasped. Keyframe: the high-level assigns high score to that frame where the task completed, storing the image (this is our “keyframe: green ball now grasped”). Transition: high-level switches to next subtask, say “move green ball to top drawer.” Progress resets to 0. Repeat: the low-level now acts under the “move ball” instruction, zero progress, and so on. Execution ends when all subtasks have been done (progress=100% on final task). In practice the high level also uses the keyframe memory in its decision-making. Suppose in a later stage, the vision of the scene looks very similar whether the red ball or the white ball should come next, so visual confusion could arise. However, if the memory contains a keyframe from “red ball inserted” earlier, the system knows that subtask is done, so it correctly selects “move white ball” next. In essence, keyframes disambiguate the task context in a way raw images at a given time cannot. Experimental Tasks. Buamanee et al. evaluated Bi-HIL on challenging real-robot tasks, both single-arm and two-arm (bimanual). These tasks were carefully chosen to test long duration, ambiguity, and contact richness. Here are the two main categories:. Unimanual “Put Three Balls in Drawer”: This task uses a single 5-DOF robot arm with a gripper. The robot has three colored balls (green, red, white) and three drawers arranged vertically. The goal is to pick up each ball in sequence and place it into the correspondingly colored drawer (top to bottom). Two complication arise: the green and white balls are interchangeable in color, so the policy must look to see which one is on the left and right initially to know the ordering. Second, the task is quite long: on average 59.2 seconds with 12 discrete subtasks. For example, subtasks include “reach and grasp green ball” then “move above top drawer”, “open drawer” (if needed), “insert ball”, and similar steps for other balls. The high-level must infer when one ball placement is finished (progress=1) before switching to the next. This task tests Bi-HIL’s ability to handle ambiguous initial states (left/right), visually similar balls, and a long sequence. Bimanual Assembly Tasks: Using two ALPHA-α 7-DOF arms (each with gripper), the authors tested two dual-arm tasks simultaneously, each with its own leader-follower pair:. 6-Cup Downstack: Both arms cooperatively stack 6 open-top plastic cups one inside another, from two stacks of 3 down into one. In practice this is a highly structured “bucket-filling” or nested-cup task: one arm grabs the top cup of its stack, the other arm fetches a cup from its stack, they tandem-insert each into the bottom of an intermediate stack, progressively merging all cups into one pile. There are many sub-steps (at least the multi-step pick-and-place and stable insertion require coordination). It’s also fragile (cups can collide or tip), so sensors and smooth control matter. 4-Peg-in-Hole: Each arm holds a small pegboard with two holes. The robot must pick four geometric pegs (two per arm) and insert each accurately into its matching hole. The holes require precise alignment and a sense of contact - if the peg is slightly off, it needs to adjust force or orientation. The sequence is two by two: arm 1 does peg 1 into hole A, simultaneously arm 2 does peg 2 into hole B, then shift to next pegs. This requires synchronizing two arms and is contact-rich (the fit can jam if off). For both bimanual tasks, there are 4 total robots (two leader, two follower), so the system must control four simultaneous policies (two high-level outputs, two low-level actions per timestep). They placed multiple cameras around so each arm’s workspace was well observed. Importantly, all these tasks were performed on real hardware with real bilateral teleoperation demos. The training data was collected once (e.g. a human teleoperated both arms through many episodes of each task), and then the learned policies were tested on new attempts. Results: Why Bi-HIL Wins. The headline result is that Bi-HIL significantly outperformed baselines and ablations on these tasks. In all experiments the baseline was essentially a flat bilateral transformer policy (similar to the prior Bi-ACT model) that does not use hierarchical subtasks or progress. They also tested ablated versions of Bi-HIL where they turned off one component (no progress signal, no keyframes, or no force). Unimanual Task (“3 Balls in Drawer”). In the unimanual put-three-balls task, success means placing all three balls in the correct drawers. The authors tested on two initial configurations: a “Left” setup where, say, green and white balls start on left vs right, and “Right” swapped. The flat baseline (Bi-ACT) got 60% success on the Left config but did 100% on Right. The full Bi-HIL model, by contrast, achieved 100% success on both configurations. Ablations of Bi-HIL clarify what mattered:. No keyframes & no progress (flat): Only 30% success on each config! The robot often failed, e.g. mixing up balls part-way through. No progress only: 50% success. It could sometimes recover, but without knowing how far it was in a subtask, it was prone to premature switching or stalling. No keyframes only: 70% success. Progress alone helped partially, but without memory of what had happened, some confusion remained. Thus both progress and keyframe memory were crucial. The proposed full Bi-HIL (keyframes + progress) nailed 100% success. The authors visualize one trial in Figure 3 of their paper (here paraphrased): as each ball is inserted, the high-level subtask and progress output (green traces) show neat sawtooth patterns resetting at each phase, and the keyframe memory accumulates snapshots of each ball in its drawer. Without the progress signal, the high-level might switch tasks too early or late; without keyframes it might forget which steps were done. The results table drives the point home: only the full method solves the task reliably. Bimanual Tasks. Similar story for the two-arm scenarios. Performance is measured as trial success (complete all subtasks without dropping something or jamming). They compare three models: Bi-ACT without force feedback, Bi-ACT with force (their original bilateral baseline), and Bi-HIL. 6-Cup Downstack: Bi-ACT (no force) < 20% success, Bi-ACT (with force) 60% success, Bi-HIL 80% success. (They ran 5 trials; Bi-HIL succeeded in 4/5). The failures of Bi-ACT often happened in the “later merging stages” when finer alignment was needed; Bi-HIL’s hierarchical guidance kept it more stable. 4-Peg-in-Hole: Bi-ACT (no force) 0%, Bi-ACT 20%, Bi-HIL 80%. This insertion task is very unforgiving: without force sensation, it never worked; with force it succeeded some tries, but often the peg got stuck mid-insertion; with Bi-HIL it completed 4 out of 5 trials. The hierarchical decomposition clearly helped manage the multi-step assembly, and the force-aware controller improved insertion alignment. These tables from the paper summarize the numbers:. | Task | Bi-ACT (w/o force) | Bi-ACT (baseline) | Bi-HIL | |---------------------|--------------------|-------------------|--------| | 6-Cup Downstack | 20% | 60% | 80% | | 4-Peg-in-Hole | 0% | 20% | 80% |. And for the unimanual balls:. | Config / Model | Bi-ACT (baseline) | Bi-HIL (w/o KF&SPR) | Bi-HIL (w/o SPR) | Bi-HIL (w/o KF) | Bi-HIL (proposed) | |-----------------------|------------------|---------------------|------------------|-----------------|-------------------| | Left environment | 60% | 30% | 50% | 70% | 100% | | Right environment | 100% | 30% | 50% | 70% | 100% |. It’s worth noting that force feedback alone was not enough in many cases. The flat Bi-ACT model often shipped the same transformer architecture as Bi-HIL’s low level, but without any notion of subtasks or progress. Even with force inputs, a flat policy got only moderate success (60% on cups, 20% on pegs), failing when sequences got long. Bi-HIL adding hierarchy (progress and memory) gave big extra robustness. Of course, 80% is not perfect. The one failed cup-and-peg trial for Bi-HIL suggests there’s room to improve (maybe more demonstrations, or tightening the progress predictions). But the jump from near-zero to 80–100% is compelling evidence that Bi-HIL’s ideas work. Why It Matters and Future Directions. What does Bi-HIL teach us about embodied AI? A few insights emerge:. Explicit Progress Helps Learning: Representing “how far along a subtask is” as a training signal is quite effective. It’s akin to giving the model a shaped reward or a curriculum. Other work has studied related ideas (in reinforcement learning one might add an auxiliary “progress” reward). Bi-HIL shows that simply supervising a progress scalar – normalized subtask completion – can dramatically stabilize imitation. We foresee more use of learned or predicted progress metrics in future robotics (for example, using video models to guess percentage of assembly). Memory Clarifies Tasks: The keyframe module is a simple yet powerful memory. It’s reminiscent of ways humans use checkpoints (“Okay, I’ve already screwed in two bolts, next is third bolt”). In Bi-HIL it’s implemented via attention over stored images. Future work might explore richer memories (video segments, or using contrastive losses to find anchors). But the concept – weave in a few “illustrative examples from this episode” – is generalizable. In tasks with multimodal perception, keeping touch or vision snapshots from earlier can disambiguate current decisions. Bilateral Imitation Is Valuable: This work adds to a growing list of papers showing the value of bilateral teleop for contact tasks. Recording forces from demonstrations allows the robot to learn delicate insertion and push-down actions. For practitioners, it means investing in hardware for bilateral teleoperation can pay off on contact tasks. The drawback is needing a second robot (or a force-feedback device), but the success rates here show it may be worth it in industrial settings. Scalability and Language: Bi-HIL uses manual instruction phrases for subtasks (e.g. “insert green ball”). In this work they are fixed template instructions, but interestingly the architecture can in principle take any SigLIP-embedded phrase here. That opens the door to learning a library of verbalized skills, or maybe even deploying on tasks with natural language directions (if training data includes varied commands). Integration with large vision-language models is a promising extension. Comparison to Other Hierarchical Methods: Bi-HIL’s keyframe approach parallels ideas in generative subgoal methods like GHIL-Glue, but sidesteps pitfalls of imaginations. It’s more akin to memory-augmented RL (see e.g. MemER or LfD with memory), showing that even imitation learning policies benefit from long-term memory to solve partial-observation tasks. Limitations: A weakness is reliance on manual segmentation of subtasks. The data pipeline requires a human to label where one sub-task ends and another begins. For some complex tasks this may be costly. Future work might automate this (e.g. via change-point detection it RL reward shaping). Also, Bi-HIL’s memory only stores a handful of frames; very long tasks might need more. And all of this was done in lab conditions; deploying to the real world will require handling more visual variety. But the framework supports extension: e.g. replacing EfficientNet with a robust siamese embedding, or using continual learning to update keyframes. In practical terms: if you have a new assembly task that is tedious for flat IL, Bi-HIL suggests an approach. Record a few teleoperated demos with bilateral sensors. Manually chop the task into a handful of subgoals (ideally 5–15 subtasks). Use those annotations to train a high-level transformer that predicts subtask ID and progress. Train a low-level CVAE to “clone” the demonstrated moves under the right instruction and force. The result should be a more reliable execution policy. Importantly, Bi-HIL also shows that we don’t necessarily need hand-coded controllers or RL for these tasks – pure imitation (with modern neural nets) can do the job if structured correctly. For research teams working on robotics demos, this is encouraging: hierarchical supervision plus rich demo data can replace months of RL trial-and-error, at least for moderate-scale tasks. Related Work Highlights. Bi-HIL sits at crossroads of several active areas. We already mentioned Bi-ACT (Buamanee et al., 2024) which was the flat bilateral learning baseline; Bi-HIL can be viewed as an extension that adds hierarchy to Bi-ACT. Turnbull et al. (2025) and others have shown the benefits of language-vision models (like CLIP or SigLIP) in robotics; here Bi-HIL uses SigLIP to embed instructions, similar to how other works have used text-conditioned policies. Another relevant line is hierarchical imitation work like “One-Shot Hierarchical IL” (Singh et al., 2018) or “Modular Imitation” etc., which often segment tasks via demonstration examples. Bi-HIL differs in using an interpretable progress estimate rather than latent options. Finally, in RL we see echoes of Bi-HIL’s ideas: subgoal-conditioned policies, reward shaping by progress, and memory. For example, a reinforcement system could mirror this by training a reward that encourages progress on the current subgoal. Bi-HIL effectively synthesizes elements from these threads using only offline expert demos, which is neat. Conclusion. Bi-HIL introduces a powerful recipe for making long, hard robot tasks more reliable: combine teleoperated force-filled data, a vision-language hierarchical controller, and the twin concepts of keyframe memory plus progress estimation. The empirical gains – from 20% success to 80–100% – are striking. For roboticists and ML researchers, the lesson is: when tackling contact-rich assembly or manipulation, don’t forget where you are in the task and what you’ve done so far. Give your policy a sense of progress and memory. Tools like transformers and CVAEs make it possible to implement this elegantly. Looking ahead, one could imagine Bi-HIL being expanded with unsupervised skill discovery (to reduce the need for manual annotation), or combined with reinforcement learning to handle the residual failures. But already, Bi-HIL stands as a compelling example of how careful system design – harking back to classical ideas of hierarchy and teleoperation, but using modern deep learning – can solve problems that troubled flat models. As robotic tasks grow in complexity, approaches like Bi-HIL will likely be part of the toolkit: a blueprint for “how to structure learning” so that long-horizon, contact-heavy jobs become tractable. Sources: Buamanee et al. (2026); Kobayashi’s project page; evaluation tables from their experiments; and related works for context.