Latest / Embodied AI 101 / ForceBand: Learning Forceful Manipulation with sEMG
Transcript
- ForceBand: Learning Forceful Manipulation with sEMG. Forceful contact – the kind of gentle touch or firm squeeze needed to manipulate real-world objects – is critical for many robotic tasks, yet it is often missing from learned manipulation policies. When we humans pick up a bag of chips, twist the cap off a ketchup bottle, or carefully press a fragile object, our muscles modulate just the right amount of force. But common imitation learning setups ignore forces: human demos from motion-capture or video convey kinematics, but not the contact forces. The ForceBand system (He et al., 2026) tackles this gap by reading human muscle activity with a simple wristband. It translates those muscle signals into fingertip forces, and uses them to teach a robot how hard to push or squeeze. In this episode we’ll walk through the ForceBand approach – from its custom sEMG hardware to the data collection, machine learning models, and zero-shot human-to-robot policy learning – exploring how low-cost biosensing can bring “touch-like” information into robot imitation learning. The Missing Sense in Demonstrations. A fundamental obstacle in imitation learning is that videos and trajectories don’t show forces. Standard human demonstrations – whether captured by motion-capture suits, recorded on video, or provided by teleoperation – yield positions and motions, but not the contact forces applied by each finger. For many tasks this is a serious limitation. Imagine a policy learned only from observing someone squeeze toothpaste: without force information, the robot might not know how hard to squeeze to get toothpaste out rather than squashing the tube. Or consider lifting objects: without force sense, a delicate object might slip out of grasp or a fragile object might be broken. As the ForceBand authors emphasize, “[common] human demonstrations… do not capture the contact forces that are critical for forceful manipulation”. Robots do have force sensors, but usually only on their own grippers during training – not on the human demonstrator’s hand. Teleoperation setups with force-feedback exist, but they require an instrumented human interface and robotic feedback loop for every demonstration, which is bulky and hard to scale. And vision-based tricks (e.g. inferring forces from object deformation) can work in lab settings, but fail when fingers are occluded or forces are subtle. PressureVision++ (WACV 2024), for example, trains a vision model to estimate fingertip pressure from an RGB image, but it needed 51 participants and carefully curated prompts to collect data, and still relies on visible surface cues. ForceBand takes a different route: it equips the human demonstrator with a wearable muscle sensor on the forearm. By measuring the electrical activity of finger muscles (via surface electromyography, or sEMG), it directly captures the intent to apply force – essentially reading the forces “at their source” in the musculature. In this way, the system augments ordinary human demos (recorded egocentrically on video) with synchronized force estimates for each finger, all without putting anything on the fingers themselves during the demo. The result is force-enriched demonstrations that can train a robot to reproduce the same motion and force profile {–} in other words, force-aware imitation learning. Reading Muscles: The ForceBand Hardware. At the heart of ForceBand is a custom sEMG wristband designed to capture fine-grained finger forces. Surface EMG sensors detect the electrical potentials generated by muscle contractions. By placing electrodes on the forearm, ForceBand “eavesdrops” on the muscles that control the fingers. The band includes eight channels of input – seven of them positioned over specific finger flexor/extensor muscles, plus one over a wrist muscle. The electrode layout is anatomically guided: the designers placed each bipolar electrode pair where that muscle’s signal is strongest. In practice this means one channel per finger’s primary muscle group and one for wrist flexion. This smart layout significantly outperforms a more naive evenly-spaced ring of electrodes: in their tests, an anatomy-based placement gave about 18% lower force prediction error than a symmetric ring placement (0.77 vs 0.94 N MAE). Electrically, the band uses a low-noise front-end. The TI ADS1299 analog-to-digital converter provides high-quality amplifiers and digitization: the recorded noise is only 0.14 μVrms with a signal-to-noise ratio of ~119.5 dB. The electrodes are arranged in bipolar pairs to cancel out common-mode noise and motion artifacts. An IMU (accelerometer/gyroscope/magnetometer) in the band records hand motion and orientation, helping to filter out movement effects. Altogether, the design is entirely built from commodity parts, fits in a 3D-printed case, and costs around $300 for the bill of materials. The authors have open-sourced the hardware design and parts list, so researchers can build their own. In use, a person straps the band on their forearm (like a watch) and calibrates it by performing some varied hand movements (more on calibration below). Once calibrated, the band continuously streams 8-channel EMG data (sampled at 250 Hz) along with IMU readings. This “muscle data” silently holds rich information about how much each finger is trying to pull or push, even if you can’t see the finger itself in the video. A Rich Multimodal Dataset. To translate muscle signals to actual finger forces, the ForceBand team first needed ground truth. They collected a 10-hour multimodal dataset of people naturally manipulating objects. Volunteers wore the sEMG band and an additional fingertip sensor rig. The fingertip sensors were force-sensitive resistors (FSRs) mounted on finger cots, recording each finger’s contact force during interaction. The participants performed a wide variety of actions: pinching and grasping different objects, picking and placing them, opening containers, pouring, and more. Each session was recorded with an egocentric camera, capturing the first-person view of the hands and objects, synchronized with the band’s EMG/IMU data and the FSR force readings. Because the FSRs are small and worn under a slim glove, the hands look almost unencumbered on video – the authors specifically note this design “ensuring minimal visual occlusion”. The dataset spans a diverse range of objects and conditions: thin chips, soft metal tins, tubes of paste, bottles of sauce – weights from tens to hundreds of grams, sizes from 1 mm to 72 mm grasps. It includes gestures using different finger combinations (2-finger pinch, 3-finger grasp, full-hand grips) and ones “in the wild” (everyday scenes like kitchen counters). The planned public release of this data will be a valuable resource since synchronized EMG+video+force is rare. For ForceBand, this dataset is used to pre-train a model that maps muscle signals to fingertip forces, capturing the general relationship across people and objects. EMG2Force: From Muscle Signals to Fingertip Forces. How exactly do raw EMG channels turn into estimated fingertip forces? The ForceBand’s learned model, called EMG2Force, takes a short window of the band’s sensor output and predicts the force trajectory of each finger over that window. Concretely, the input is a 5-second clip of 8-channel EMG data plus the 10-D IMU data, sampled at 250 Hz (about 1250 timesteps). Since EMG is both noisy and nonstationary, the model processes it in two complementary ways. First, it treats the raw time series directly: a 1D convolutional encoder extracts temporal features from the EMG+IMU signals. Second, it computes a Short-Time Fourier Transform (STFT) spectrogram of the same window – essentially a time-frequency image – and feeds that into a deep visual encoder. In practice they use a pretrained vision transformer (DINOv3) to process the spectrogram “image”. These two branches – one that sees the signal in time and one in frequency – produce feature vectors that are then fused. The fused representation goes through a learnable transformer decoder (sequence model) that outputs a timeline of force values, one scalar force per finger at each timestep. Symbolically, if the raw input sequence is $X \in \mathbb{R}^{18 \times N}$ (18 channels by $N$ timesteps), EMG2Force first makes two encodings $h_{\text{time}} = \mathrm{Conv1D}(X)$ and $h_{\text{freq}} = \mathrm{DINO}( \mathrm{STFT}(X) )$. It concatenates these features and feeds them into a transformer that decodes to the force sequences $F_{\text{fingers}} \in \mathbb{R}^{5 \times N}$ (five fingers, matching the demonstrations in the dataset). During training on the 10-hour dataset, this network learns to predict the actual forces measured by the fingertip FSRs. In effect, EMG2Force learns that specific muscle activation patterns at the wrist correspond to squeezing with certain fingers. It leverages the IMU data to account for gross hand movements and to help normalize the signals (preventing motion artifacts from spoiling muscle readings). The combination of time-domain and frequency-domain processing allows the model to extract both the “rhythm” (temporal changes) and the “spectral signature” of muscle firings, roughly analogous to how speech-recognition models use spectrograms plus time features. By pretraining on the general dataset, EMG2Force acquires broad ability to infer finger forces from muscle activity. Calibrating to the User. One challenge of sEMG is person-to-person variability: muscle signal strengths differ between arms and electrode placements vary slightly. ForceBand handles this by a quick calibration step. After pretraining EMG2Force on the aggregate data, each new user performs about 15 minutes of “random play” with the device. In this calibration session, the user dons the sEMG band and also wears the fingertip FSR sensors again. They then randomly manipulate various objects (pick up items, squeeze something soft, pinch objects, etc.) for a short period. Since the fingertip sensors are on, the system now has fresh EMG/IMU versus ground-truth force data for that person. EMG2Force is fine-tuned (or its internal normalization adjusted) to align with the user’s actual force output. Remarkably, the authors report that only ~15 minutes of this data are needed to adapt the model to the new user’s physiology. Once calibration is done, the fingertip sensors can be removed. The user can then demonstrate any target tasks wearing just the ForceBand and recording video. The pre-trained-plus-calibrated EMG2Force model is run on these new EMG streams to produce per-finger force labels. In other words, force-labels come “free” in the demonstration: the human just shakes, squeezes, and manipulates objects naturally, and the system retroactively knows how much force each finger applied at each moment. This approach is powerful because the demonstrator never has to wear bulky force sensors on the fingertips during the actual demonstration. The hand looks and behaves completely natural in the camera’s eyes. Learning a Forceful Manipulation Policy. With force-annotated demonstrations in hand, how do we train a robot to imitate them? The ForceBand pipeline retargets the annotated human demonstrations to the robot’s embodiment. Since the experiments use a simple two-finger parallel-jaw gripper on a 6-DOF arm, each human demonstration (which has 5 finger forces) is converted to robot action sequences with an appropriate “grasp force” value. Essentially, the five human finger forces are aggregated into a single target gripper force (for this setup, the model predicts one grip force $f$ for the two-finger gripper). The robot’s end-effector pose and orientation come from the recorded camera viewpoint combined with a vision-based pose estimator of the object (so the policy knows where to move relative to the object). The policy model itself is trained with a flow-matching objective, a recent imitation learning technique akin to diffusion modelling. Concretely, a transformer network is trained to regress instantaneous “velocities” of action trajectories conditioned on a time index. During training, each demonstrated action trajectory (position p, orientation r6D, gripper aperture g, and grip force f over time) is split into small segments (a₀ → a₁). The network $v_\theta(a_t, t, o)$ is trained so that for a random interpolation index $t$ between 0 and 1, the predicted velocity vector v matches the true difference $a_1 - a_0$. Here o represents the current visual observations (camera images, spatial tokens for object relations, etc.). In formula form, they minimize $$ \mathbb{E}{t,a_0,a_1}\bigl|v\theta(a_t, t, o)-(a_1-a_0)\bigr|^2, $$ where $a_t$ is an interpolation between $a_0$ and $a_1$ along the trajectory. This conditional flow matching trains the policy to generate whole trajectories: at test time it can unroll predicted velocities to form a complete action sequence. The key is that the policy is conditioned on both the visual scene and the target action. In practice, the policy inputs include the current egocentric camera frame and object location tokens, and it outputs a 7D action at each time (3D translation, 4D rotation, plus grasp $g$ and force $f$). Through flow matching, the model learns to follow the human demonstration’s motion and replicate the force profile. The authors highlight that this approach is “zero-shot from human data” – no robot teleoperation or reinforcement learning is needed. Everything comes from the human demos (motion + labeled force). Why This Works: Forces Beyond Vision. What benefit does the sEMG band confer over a vision-only policy? The experiments paint a striking picture: muscle signals reveal force information that vision can’t see. First, consider contact detection. A vision-only model (trained on the same videos) can often tell if the hand as a whole is touching an object, but it struggles to tell which finger. In one evaluation, the precision-recall AUC for contact at the ring finger was only 0.398 with video, compared to 0.763 using sEMG signals. For the pinky finger, vision scored 0.314 AUC versus 0.590 for EMG. In short, if a finger is hidden or overlaps with others, a camera simply cannot “see” its subtle pressure, whereas the forearm muscle controlling it is still active. The authors note that “vision keeps up on coarse contact but trails on fingers it can’t read… occluded fingers still load their forearm muscles; pixels only read the surface”. Second, force magnitude estimation: it’s one thing to detect contact, another to judge how strong. Here too sEMG dominated. The study reports that the EMG2Force predictor’s mean absolute error was under 1 N across fingers, while the vision-based predictor’s error was about twice as large. In their language, “sEMG holds under 1 N error, vision roughly doubles it”. This is not surprising: estimating a continuous force from pixels alone is intrinsically ambiguous unless you see clear object deformation. Muscle activity, however, scales fairly directly with force. The outcome of these differences is clear in manipulation tasks. For example, in a pick–squeeze–place benchmark, a vision-only (or “gripper-only”) policy never produced an object-specific squeeze; it simply opened the gripper fully and gave up. By contrast, the force-aware policy generated different grip forces depending on the object (just as the human demo did). In a test with 9 objects (mass 43–650 g, grasps 1–72 mm wide), the ForceBand-trained robot policy succeeded 87% of the time, while motion-only baselines essentially failed the squeeze action. In other words, the muscle signals let the robot learn how hard to grip or push for each object, which pure motion imitation could not capture. Putting It All Together: Results. Summarizing the numbers, ForceBand leads to dramatically better force perception. On held-out tests, incorporating the band’s signals “recovers fine-grained fingertip interactions with over 50% lower force prediction error than vision-based baselines”. In absolute terms, the EMG-based model’s error is under 1 N on each finger, whereas the camera-only model’s error is roughly double. Contact detection accuracy is similarly improved – especially for occluded fingers. Ablation studies show that more EMG channels continually help: going from 1 channel to 8 channels reduced MAE from 1.89 N to 0.85 N. And the anatomically-informed 7+1 channel layout beats a naive uniform 8-channel ring by about 18% (0.77 vs 0.94 N). These perceptual gains translate to better robot performance. In the pick-squeeze-place experiments on a UR-5 arm, the force-aware policy succeeded in object-specific tasks on 87% of trials. It could replicate the human’s varying squeeze pressures: for example, humans applied about 10.2 N on toothpaste while robots did ~12.6 N, humans 17.3 N on BBQ sauce vs 18.4 N by robot, etc (see ForceBand site). The authors highlight that no robot data was used in training the policy – it was “zero-shot” on the robot, trained purely on human videos labeled by muscle cues. By contrast, a policy trained only on motions (and ignoring force) completely failed the “squeeze” part of the tasks (0% success on those subtasks). Toward Practical Force-Aware Learning. ForceBand is an exciting proof-of-concept for bringing touch-like signals into imitation learning. There are practical upsides: the entire setup is low-cost and reproducible. The band can be built for a few hundred dollars using off-the-shelf ADS1299 EMG boards and 3D-printed parts. After a brief calibration session (about 15 min), any person can record demonstrations with just a smartphone camera and the wristband. No bulky gloves or gloves with built-in sensors are needed during actual demos – the user’s hand is free. The idea of using sEMG also opens new avenues. Humans were already using their own proprioception to judge forces, but ForceBand externalizes part of that sensing. The authors envision that any “in-the-wild” video of someone manipulating objects could be force-labeled after the fact, provided the person wore the band during filming. This suggests a powerful pipeline: “point camera at a human showing a task, wear band on wrist, and immediately generate forceful robot policies”. Of course, there are limitations. Currently ForceBand predicts only normal fingertip forces (the kind of push directly toward the object). Lateral or complex contact forces (twisting, sliding friction, etc.) are not captured. It also assumes a relatively simple robot end-effector (a parallel gripper). Mapping five human fingers to two robot fingers requires simplification (they effectively sum finger forces). More dexterous robots could in principle benefit even more by retargeting finger forces individually. Also, sEMG can drift over long time, and electrodes must be placed carefully; real-world use would need quick recalibration or more robust signal processing. Finally, the approach requires wearing the band; while it is low-profile, some users might find any extra hardware intrusive compared to a vision-only demo. Yet even with these caveats, ForceBand marks a significant advance. It sidesteps the need for instrumented objects or teleoperated robots in the loop. Instead, it “brings force into human data” by piggybacking on our own biology. In domains like household robotics, assembly, or assistive devices, robots often interact with soft/fragile materials where force control matters. ForceBand’s method could allow learning from cheap, easy human videos enriched with muscle signals to master these tasks. Related Efforts. ForceBand builds on threads in both human-machine interfacing and imitation learning. sEMG has long been studied for prosthetic control or gesture recognition (e.g. using Myo armbands or high-density forearm arrays), but most prior work uses it to directly control a robot (mapping EMG to joint commands), rather than annotate demonstrations for offline policy learning. Teleoperation systems using EMG exist, but typically require the user to actively move in concert with the remote robot with continuous force feedback loops. ForceBand instead treats EMG as a passive sensor for labelling data. Vision-based force estimation is a hot topic, as touched on above. The emergent “Vision-Based Force Estimation” survey highlights many approaches: camera-watching a compliant gripper, using depth or event cameras to see deformations, or even GelSight-like tactile vision. Many systems report impressive accuracy (for instance, FeelAnyForce reports ~4% error on its GelSight sensor), but they rely on special sensors (like a camera under a gel pad) or ideal conditions. PressureVision++ showed it’s possible to estimate fingertip pressure from an RGB image by essentially instructing people on how hard to press during data collection, but this requires a cooperative human and only yields the pressure normal to the surface. In contrast, ForceBand’s use of sEMG means the sensing happens inside the human’s arm – it can capture intent even when fingers are out of sight, and it doesn’t require customizing the objects or environment. On the policy learning side, flow-matching (a cousin of diffusion policies) is a state-of-the-art imitation technique which the authors adopt. Other recent works (e.g. “Flow Matching for Multi-support Manipulation”) use a similar idea to train policies by regressing velocity fields along demonstration trajectories. By combining flow-based IL with force outputs, ForceBand effectively produces a hybrid force-motion trajectory for the robot, a capability reminiscent of classical hybrid force-position controllers, but learned end-to-end from data. Conclusion. ForceBand demonstrates that you can teach a robot to “feel” from watching a human’s muscle activity. The key insight is simple yet powerful: muscles fire in proportion to force, so recording sEMG from the forearm is like grabbing a hidden stream of force data during human demonstration. By adding this hidden force channel to standard imitation learning pipelines, the system can learn not just how to move, but how hard to move for each object. The result, as the creators show, is a zero-shot human-to-robot transfer of forceful pick-and-place skills on common hardware and a minimal budget. For roboticists, this is an intriguing new tool: a $300 wristband and some code can give one extra dimension of sensing that cameras cannot provide. It opens the door to richer human-demonstration datasets and more dextrous learned skills. As ForceBand’s code and potentially its dataset become available, we may see a wave of new research building on these low-cost EMG-augmented demonstrations. In domains where delicate touch matters – from kitchen helpers to assembly-line partners – muscle-driven learning might soon be part of the standard kit. References: Support for points above comes from the ForceBand project page and an independent review, as well as related vision-force estimation work. These sources detail the ForceBand hardware, dataset, EMG2Force model, and imitation learning experiments, showing the substantial improvement over vision-only baselines.