Latest / Embodied AI 101 / Reactive Diffusion Policy: Slow-Fast Visual-Tactile Learning for Contact-Rich Manipulation

Transcript

Reactive Diffusion Policy: Slow-Fast Visual-Tactile Learning for Contact-Rich Manipulation · Embodied AI 101

• Reactive Diffusion Policy: Slow-Fast Visual-Tactile Learning for Contact-Rich Manipulation. Contact-rich manipulation is one of the biggest frontiers in robotics. Humans effortlessly perform tasks like peeling a cucumber, wiping a curved vase, or lifting a delicate cup, seamlessly coordinating vision and touch. We can make coarse plans and then refine them on the fly as soon as we feel the slightest contact change – our hands constantly adjust force, orientation, and grip based on rich touch feedback. For robots, by contrast, this blend of long-horizon planning and millisecond-level reactivity is extremely challenging. Traditional imitation learning methods for dexterous tasks often “chunk” actions into open-loop sequences, predicting several seconds of motion at once from camera images. Such approaches can model complex trajectories, but they cannot adapt mid-execution when, say, an object slips or bumps. And most teleoperation systems lack good tactile feedback: the operator might see what the robot sees, but doesn’t feel what it feels – missing another key ingredient of human skill. Enter Reactive Diffusion Policy (RDP) – a new framework proposed by Xue et al. as an RSS 2025 Finalist – that explicitly combines high-level planning via diffusion models with fast closed-loop control through touch feedback. At its heart, RDP splits the control into two complementary layers: a slow, latent diffusion policy that plans long, complex action chunks based on vision, and a fast asymmetric “tokenizer” policy that fine-tunes each chunk in real time using tactile/force sensing. Intuitively, the slow policy says “move the peeler along this path,” while the fast policy says “adjust the peeler’s position and force as soon as you feel a change in contact”. This mimics how humans use vision to set a rough goal and touch to continuously refine execution. Coupled with this algorithmic advance is a novel data-collection system called TactAR: a low-cost teleoperation setup that shows the operator a real-time 3D tactile visualization in Augmented Reality. Using a VR headset (Meta Quest 3) and simple hardware, TactAR “renders” the contact forces and deformations at the robot’s fingers as a 3D deformation field floating at the gripper. In other words, as the robot squeezes or brushes an object, the operator sees along with the video what the object’s surface is doing – as if feeling texture and force through sight. By collecting demonstrations with this rich symbolic environment, the authors generate high-quality training data for RDP that encode both vision and touch. In extensive experiments on three representative contact-rich tasks – Peeling a cucumber, Wiping a vase, and Bimanual Lifting of a cup – RDP shows dramatic gains over prior baselines. With RDP’s slow-fast policies, success rates rise from near zero to over 80–90% compared to a vanilla diffusion policy. In perturbation tests where a human deliberately shifts the object mid-motion, the fast tactile controller sharply corrects the trajectory, whereas an open-loop plan can fail badly. A user study also showed the value of TactAR: users found the tactile AR feedback “very helpful” for teleoperation, and data collected with it led to more stable force control and better task success (e.g. normalized peel length rising from 0.72 to 0.91 with touch feedback). In this article we’ll peel back (pun intended) the layers of Reactive Diffusion Policy. We’ll explain why combining diffusion-based planning with high-frequency touch is a natural idea, how the RDP architecture is constructed and trained, and why TactAR solves a key practical hurdle in teleoperated data generation. Along the way, we’ll connect to related work on diffusion policies and tactile systems, and walk through the main experimental findings. The goal is to provide a deep, accessible explanation of how and why these ideas work, so that practitioners can adapt these concepts to their own robotics problems. The Challenge of Contact-Rich Manipulation. Manipulating objects that involve rich contact – for instance, sliding a tool against a surface while applying controlled force – is notoriously hard. Unlike free-space movement, these tasks actively involve force dynamics: if you press too hard or lose contact entirely, the task fails or damages the object. Think about peeling a cucumber or wiping a curved surface: precision in contact force is crucial, and the geometry changes as you move. Humans handle this easily by feeling the object and continually adjusting: if a cucumber shudders in your grasp, you apply more suction or change angle; if the cup starts to slip, you unconsciously pinch harder. Most robot learning approaches either shy away from contact altogether or treat it in a very open-loop way. End-to-end policies (reinforcement or imitation) trained on videos tend to predict short bursts of motion and then re-commit to open-loop chunks. For example, Diffusion Policy models (Cheng et al., 2024) have shown remarkable success in visuomotor tasks by representing an entire policy as a denoising diffusion process in the action space. These models can learn multi-modal or precise motions across diverse tasks, outperforming older methods by large margins. However, when it comes to contact-rich domains, an open-loop diffusion policy still suffers: once it outputs the next chunk of actions, it won’t update those actions until the next chunk, even if contact conditions have changed mid-chunk. In practice, this means it can’t react to unexpected force changes (like a perturbation or an object slipping) until possibly too late. This limitation of open-loop chunking has been noted by others working on diffusion-based policies for contact tasks. For instance, Wang et al. (2024) propose a Hierarchical Diffusion Policy that breaks the policy into a high-level diffusion predicting contact points for the robot to reach, and a low-level diffusion generating the actual motions toward those contacts. That approach significantly outperforms a flat diffusion policy by focusing on contact guidance. But even that assumes the contact point (chosen in advance) and does not incorporate continuous touch feedback during execution. In contrast, humans perform contact tasks by fusing a feedforward plan with continuous feedback, especially tactile signals. Tactile sensing has proven its value in robotics: from slip detection in grasping to fine force control in assembly, it is a powerful modality. However, many datasets and demonstrations lack tactile information. Standard cameras can’t see contact forces, and most teleoperation rigs (e.g. VR controllers) only show what the robot sees, not what it feels. The Teleoperation Hardware (TactAR) introduced by Xue et al. addresses this gap head-on by giving the operator real-time tactile/force cues in a natural way. This means the demonstrations used to train RDP embed not only the visual context and intended motions, but also how the object responded via tactile deformations. The system turns tactile arrays or force sensors into a visual 3D field so that human demonstrators can better adjust on the fly. We’ll describe TactAR in more detail below, but it’s worth emphasizing upfront: good data is crucial. If demonstrations already contain the rich interplay of touch and vision, then RDP can learn from them to mimic that interplay. We’ll see later how much of an effect this has on learned performance. In summary, the core insight of Reactive Diffusion Policy is: plan coarsely in time and space, but react finely along the way. The slow policy is a diffusion-based planner (leveraging the power of that framework for modeling complex multi-modal trajectories), and the fast policy is a small neural controller that continuously adjusts actions with tactile feedback. Together, they behave like a learned version of a classical nested loop controller: an outer loop provides setpoints or trajectory segments, and an inner loop quickly corrects for force errors. By training both via imitation learning, RDP internalizes how to coordinate vision and touch to solve hard tasks. In essence, RDP “inherits” the human strategy of feedforward+feedback control, which is something so far robot policy learning hasn’t explicitly captured at scale in contact manipulation. The TactAR Teleoperation System. Before diving into the algorithm of RDP, let’s look at how the team collects data in the first place. RDP is an imitation learning method: it needs expert demonstrations of successful contact-rich tasks, with aligned observations and actions. To gather this data, the authors built TactAR, an Augmented Reality teleoperation interface with tactile feedback for the operator. The hardware setup (Fig. 3 in the paper) centers around a Meta Quest 3 VR headset worn by the human operator. Inside the headset is a stereo video feed: multiple camera streams from the robot’s end-effector and environment are stitched into a 3D scene, so the user sees from the robot’s point of view. Crucially, TactAR also reads from a tactile or force sensor on the robot’s tool and renders that sensory data as a 3D deformable field attached to the tool. Concretely, say the robot holds a peeler bar that presses against a cucumber. The tactile sensor is either an optical tactile pad (like GelSight or MCTac) or a regular force-torque sensor at the wrist. TactAR converts that raw sensor data into a “3D deformation field”: for an optical tactile sensor, the displacement of markers on the gel is turned into an (x, y) offset and a pseudo-depth highlights how much the surface moved in/out. For a 3D force sensor, the measured (Fx, Fy, Fz) components are normalized and visualized as a colored displacement. This 3D field appears in the AR view as a translucent halo or vector arrow at the contact point. The result is that the human operator can actually see the force and deformation live, not just the camera image. For example, when peeling a cucumber held by an assistant, the operator will see the cucumber and peeler as expected, but also a small virtual grid or contour map around the peeler tip that bulges in proportion to the pressure. If contact is light and smooth, the field is barely visible; if contact is firm or uneven, the field bulges and wobbles. This visual augmentation effectively encodes touch for the human. The authors describe it as “rich contact information in 3D space” – including not only pressure magnitude, but also shear and torsional torque directions (for more advanced sensors), all captured as the 3D field. Implementing this required system integration with ROS2 and some calibration. The operator first aligns the AR coordinate system with the robot’s tool frame: a virtual white sphere is positioned at the tool center and matched to the AR world origin if needed. Then the 3D field from the tactile sensor is rendered at the tool frame, snapping it to the robot gripper. The system streams multiple RGB cameras and the tactile camera feed into the headset. Software ensures low latency: the tactile field is updated at high frequency (over 30–100 Hz, depending on sensor) to avoid lag. This way, the operator feels effectively instantaneous force feedback, analogous to feeling through a remote telepresence. Why is TactAR beneficial? The authors conducted a user study with 10 participants (mixed experience) doing the peeling task. Half the trials used a conventional VR teleop setup (no tactile feedback), and half used TactAR (with AR tactile). The results were striking: over 70% of users reported that the tactile AR feedback was “very helpful”, regardless of whether they were controlling or simply observ ing. More quantitatively, data from the AR sessions yielded longer, more precise peels. On average, the normalized peeling length (peel percentage compared to demonstration) went from 0.72 with plain VR to 0.91 with TactAR, and the fraction of time the force was stable rose from 0.58 to 0.87. In short, giving operators feel-through-sight made them naturally apply smoother, more correct forces. TactAR’s final demonstration capability is to support “example tasks” with high fidelity: the project page (Fig.2 and 3 caption) shows an operator aligning the peeler on a cucumber and seeing the tactile field attach to it in AR. The interface works for both optical tactile pads and joint-torque sensors, just visualizing either marker flow or force vectors. Also, it supports multi-view cameras, so the user can switch between wrist and external views. All of this runs on commodity hardware (Meta Quest headset, webcams, a local PC) plus the robot and sensors. Compared to expensive force-feedback gloves or haptic devices, TactAR’s approach lets the operator “feel” using only visuals. It’s low-cost and easy to replicate, which is important because building such data-hungry systems needs scalable demonstrations. In summary, TactAR is not the main novel algorithm but a critical part of the pipeline. It ensures that the dataset driving RDP contains synchronized visual, action, and most importantly tactile/force signals, captured as something learnable. Because RDP’s fast policy relies on subtle tactile cues, training without good touch data would be hopeless. The user study confirms that raw performance differs when data quality improves. As we’ll see next, the learning algorithm itself is built to exploit exactly this rich data. The Reactive Diffusion Policy Algorithm. With the training data from TactAR in hand, the core contribution is the Reactive Diffusion Policy (RDP) algorithm. Let’s unpack it step by step. Two-Level Policy Structure. At the top level, RDP is a hierarchical policy with two networks: a slow diffusion policy (also called the Latent Diffusion Policy, LDP) and a fast asymmetric tokenizer (AT). The terms slow and fast refer to control frequency. The diffusion policy thinks at a low rate (e.g. 1–2 Hz), planning seconds of motion at a time. The tokenizer runs at a high rate (20–30 Hz), enabling immediate reactions. Conceptually, during execution, RDP works like this (Fig. 4(b) in the paper): At the start of each chunk, the robot observes the scene (camera images, object states, etc.) and the current tactile reading. The slow policy sees a downsampled version of the history and outputs a high-level plan chunk in latent space (typically corresponding to, say, 1 second or more of motion). This diffusion-based plan is a sequence of intended actions (in compressed latent form), like “move peeler up by 2 cm, then down by 1 cm, etc.”. Once this chunk is decided, the robot executes it step by step. For each new control timestep within the chunk (e.g. 30 Hz rate), the fast policy takes the chunk and the latest tactile/force reading, and outputs the next motor command. In other words, the fast policy refines the planned trajectory on the fly. If the peeler suddenly hits the cucumber harder than expected, the fast policy might instantly command a slight retraction or rotation. Thus, one can think of the slow diffusion policy as a generative planner and the fast tokenizer as a reactive controller. They run sequentially rather than simultaneously, but importantly, the fast controller loops continuously while a chunk is being executed (see Fig. 5(c) in the paper). This creates a closed-loop control loop inside each chunk, based on tactile input. Contrast that with baseline IL pipelines: a vanilla diffusion policy (or any action-chunk GC) would start a chunk and apply all actions open-loop until the chunk finishes, then get the next observation. Some recent pipelining tricks try “semi-closed-loop,” like blending new feedback slowly across the chunk, but they still lack true instant response. RDP explicitly closes the loop: “the fast policy enables closed-loop control by using high-frequency tactile input and fine-tuning the latent action chunk … in an auto-regressive manner”. The fast policy is auto-regressive because at each step it takes the prior partial chunk and the new tactile sensor reading, and outputs just the next action to apply. Latent Diffusion for the Slow Policy. The slow policy (LDP) is built on a latent diffusion model. Diffusion policies for robot control (e.g. Diffusion Policy by Cheng et al.) treat trajectories as high-dimensional data to be generated. Instead of regressing directly to actions, they learn to gradually denoise a random noise action sequence until it matches a demonstrated one. The RDP authors adopt this paradigm but operate in a latent space of actions. Specifically, they use the fast tokenizer’s encoder (introduced below) to compress an action chunk into a low-dimensional latent sequence $Z$. The slow diffusion model then operates on these latent sequences. Why diffusion in latent space? Raw action chunks can still be high-dimensional (like 4-DOF controls over 30 steps = 120 numbers). By compressing into a meaningful latent representation first, the diffusion network is simpler and can better capture long-horizon structure. In practice, the latent dimension might be only a few units per step. So training the diffusion model is more efficient, and sampling (at inference time) is faster because the model is smaller. In training, the diffusion policy $\epsilon_\theta$ learns the score function on latents: it takes an observation $O$ (vision etc.) and a noisy latent sequence $Z^0 + \epsilon^k$ and predicts the noise $\epsilon^k$ added at step $k$. The loss is the usual denoising diffusion loss: an MSE between actual noise and predicted noise, averaged over noise levels. Crucially, the diffusion model’s input is the current sensor observation $O$ (and possibly a conditioning like the last tool pose), so it learns how to generate latent chunks conditioned on the visual/force context. At inference time, when the robot is performing the task, the slow policy generates a latent chunk by sampling from the diffusion process conditioned on the latest observation: starting from pure latent noise, it iteratively refines (using the learned denoising steps) until it yields a latent vector $Z^0$. We then feed this latent vector to the fast tokenizer’s decoder to begin execution (see below). The entire slow policy operates at, say, 1–2 Hz. For tasks like peeling or wiping that last many seconds, planning every second or half-second is reasonable. Within that time, the fast policy handles immediate corrections. An important implementation detail: this two-stage training (fast then slow) is a dividing the labor. The authors first train the fast policy (asymmetric tokenizer) entirely on supervised data, without the diffusion model involved. Once the tokenizer is good, they freeze it and train the diffusion model on the latent codes of all demonstration chunks. This decoupled training simplifies optimization and ensures that the slow model only has to learn how to produce the latents that the tokenizer expects. It’s like pre-training an action autoencoder and then making a generative model in its bottleneck space. The Fast Asymmetric Tokenizer (Closed-Loop Controller). The fast policy, called the asymmetric tokenizer, is a network that quickly maps partial plans into actual controls based on tactile feedback. Why “tokenizer”? Because it behaves somewhat like an action autoencoder (or sequence auto-encoder) that takes a latent chunk (a “token” of compressed actions) and decodes it back to full actions, but with a twist: it also takes sensory feedback to correct the actions on the fly. Architecturally, the tokenizer is simple but clever. It consists of:. Encoder: A 1D Temporal Convolutional Network (1D CNN) that compresses an entire multi-step action chunk (coming from human demo) into a shorter latent sequence. For example, 30 steps of 8-dimensional action might be compressed into 5 steps of a 12-dimensional latent. This encoder defines the latent space of actions. In training, this encoder is what the diffusion model ultimately conditions on. Decoder: A Gated Recurrent Unit (GRU) RNN that decodes the latent chunk back into actions, one step at a time. Crucially, at each decoding step, the decoder also takes the current tactile/force reading as input. This is what gives the policy its reactivity. The GRU has a hidden state that is updated each time step, so it inherently models sequential dependencies (how the next action should depend not just on the original plan but on what has happened so far). On each control tick, the partial latent chunk and the latest sensor reading are fed into the GRU, which outputs the next action command to the robot. Because the decoder outputs one action per time-step, it is already naturally auto-regressive: after it outputs an action and receives the next sensor reading, it continues outputting the next action, etc., until the latent chunk is exhausted. The authors refer to this design as “asymmetric” because the encoder compresses an entire chunk at once, while the decoder unfolds it sequentially with feedback. Training the tokenizer is done purely by supervised learning on the collected dataset. For each demonstration chunk of actions $A = (a_1,., a_T)$ and the corresponding tactile observation sequence, the encoder produces a latent $Z$. The decoder is then trained to reconstruct $A$ step-by-step: it inputs $Z$ and the tactile sequence, and outputs actions $\hat{A}$. The loss is a combination of a standard reconstruction loss (L1 difference between $\hat{A}$ and $A$) and a small Kullback-Leibler (KL) divergence loss that keeps the latent distribution close to a prior. In effect, this makes the latent space “nice” for diffusion. Once trained, this tokenizer does two things at execution time:. It provides the latent representation to the diffusion policy. Every demonstration chunk $A$ can be encoded to a latent $Z$, so the diffusion model learns on a consistent space. It actively controls the robot during inference. At run-time, the encoder is actually not used (since the robot doesn’t know the ground-truth next chunk in advance). Instead, only the decoder is used: the diffusion model outputs a hypothetical latent chunk $Z$, then the decoder takes $Z$ and rolls it out with real-time tactile input ${f_t}$ to produce the actual sequence of commands to send to the robot. In summary, the asymmetric tokenizer is acting like a learned impedance controller. It implicitly learns how to adjust an action given the tactile observations: if the hand is pushing too hard, the GRU will output a corrective action (like backing off slightly) in the next timestep. Because it’s trained on real demonstrations which include tactile data, it learns the subtle correlations between, say, a sudden change in force and the proper corrective motion. Putting it Together: Inference. Concretely, when executing a task, RDP loops as follows:. Perception: Gather the current observation $O_t$ (RGB images, robot pose, etc.) and current raw tactile reading $f_t$. Slow Planning: Every $N$ steps (e.g. every second or whatever the chunk length is), run the diffusion model: sample a latent chunk $Z$ conditioned on the recent low-frequency observations. Let’s call this chunk valid for the next 1 second. Execution Loop: For each control step (e.g. 30 Hz) within that second:. Read the new tactile/force measurement $f_{t}$. Feed the latent $Z$ and current hidden state into the decoder along with $f_t$. The decoder outputs the next action $a_t$ to send to the robot. Advance time $t \to t+1$. If the chunk’s time is up, or we reach the next “planning horizon,” go back to slow planning. Thus the architecture is unified: a single pipeline. It allows RDP to model non-Markovian behavior (since diffusion policies are inherently non-Markovian generative models) while still reacting instantly to contact. An analogy might help: imagine a driver on a twisting road. The slow policy is looking ahead on the map and saying “next, navigate through the next big curve.” The fast policy is like your reflexes: as you turn, you feel the steering wheel and road feedback, and you make tiny corrections to stay on track. If a gust of wind hits (a perturbation), your hands steer back immediately without re-planning the whole route. RDP formalizes this with network modules and learning. The result, as the authors emphasize, is both complex trajectory modeling (thanks to the diffusion chunk) and rapid reactive behavior (thanks to the high-frequency tactile loop) – combined in one learned framework. Comparison to Baselines. Before seeing the results, it’s worth summarizing how RDP differs from more naive approaches:. Vanilla Diffusion Policy (DP): A diffusion-based imitation learner that predicts whole action chunks (maybe at 12 Hz) from vision alone. This is like doing LDP without any tactile loop. It excels at getting the general trajectory, but won’t correct mid-chunk. In the authors’ experiments, this baseline often fails when humans intervene or conditions change. DP + Tactile (image/embedding): One could feed the raw tactile camera image or an embedding of it into the input of the diffusion model at each planned chunk. This gives the model some knowledge of contact at chunk boundaries. The authors test versions of these: “DP w. tactile img” and “DP w. tactile emb.” in their tables. They still produce open-loop chunks, though, so any contouring adjustments aren’t available during execution. In practice, these help a bit (sometimes DP with tactile image did slightly better on “perturb before contact” in wiping), but not consistently. Temporal Ensembling / Semi-closed loop: Other pipelines in the literature sometimes do partial feedback. The RDP paper’s Fig. 5 (which compares pipelines a/b/c/d) contrasts RDP with “temporal ensembling” schemes. In those schemes, the plan might be broken into sub-chunks and partial re-planning at slower rate to create a more stepwise plan. But that’s still fundamentally slower (and often hand-tuned) than RDP’s token refresh every timestep. Think of schemes that average multiple past plans or that allow small trajectory modifications – they can’t adjust as quickly or as precisely as a trained reflex controller. Human Control: If one operator with TactAR manually controls the robot, that’s obviously ideal in terms of reactivity and intelligence – but we need to automate. Still, the user study suggests how RDP should learn from human-like data. Actually RDP’s idea is partly to distill human expertise (captured via TactAR) into a neural policy. So RDP is unique in explicitly architecting for hierarchical imitation: one level predicts strategy (latent chunk), the other embodies reflex (tactile servoing). One might see it as merging ideas from imitation with latent action spaces (latent action diffusion) and classical force control (impedance/admittance control) all in one net. Experimental Results. Let’s turn concrete. The authors evaluated RDP on three contact-rich tasks, each with its own challenges:. Task 1 – Peeling: A single-armed robot holds a vegetable peeler. The cucumber is held in midair by a human. The robot must approach and peel the cucumber continuously. This requires millimeter precision (to peel without hitting too much flesh) and instant response to any perturbations. In tests, they included variations where after contact the human moves the cucumber sideways or the robot must maintain contact. Task 2 – Wiping: Similar setup, but now the robot holds an eraser brush, and the human holds a cylindrical vase. The robot must wipe a drawn pattern off the vase’s surface. This requires not only force control (to keep wiping contact on the curved surface) but also rotation tracking. Again, random perturbations are introduced by the human. Task 3 – Bimanual Lifting: A tougher scenario: a dual-arm robot each holds one handle of a paper cup. The robot must grip the cup from opposite sides and lift it along a path without crushing it. Two variations of the cup (soft and hard) are tested. This demands coordinated force: too much squeeze and the cup deforms; too lax grip and it slips. Moreover, the demonstration data included two different upward trajectories, so the policy must handle multi-modality (two valid ways to lift). For each task, the protocol was to collect many human demonstrations with TactAR (60 in peeling, etc.), then train policies from that offline dataset, and then test on unseen trials with and without disturbances. The performance metric was designed per task (fraction of peeled skin, fraction of wiped ink, or successful lift score). Overall Performance. Across all tasks, RDP blew away the baselines. The project page has detailed tables (see [19]) but here’s a summary in words:. Peeling: The diffusion policy without touch (DP) scored about 0.44 overall (normalized peel length) on average (with 10 trials under various perturbations). By comparison, RDP with any of the tactile sensors scored around 0.90. In particular, “RDP (GelSight)” got 0.90, “RDP (MCTac)” 0.88, and “RDP (Force)” 0.95, averaged over conditions. In plain terms, DP managed to peel less than half the cucumber on average (especially failing badly if disturbed), whereas RDP nearly finished the peel every time. The difference is dramatic. Wiping: The gap remained large. DP got about 0.57 success overall (see table [19]), whereas RDP with force sensing achieved 0.87. GelSight also did 0.77. In perturbation settings, DP often failed after just a little cleanup, while RDP still removed most of the residue. (The scoring was based on leftover ink, but essentially, RDP ended up wiping almost clean in most runs, far outperforming DP.). Bimanual Lifting: This was the hardest test. A vanilla DP cannot easily coordinate two arms with force control, and indeed both DP baselines got essentially 0 success (0.00) on lifting the cup without failing. RDP, by contrast, accomplished the lift frequently: with GelSight+MCTac sensors it got an overall score ~0.48 (for both soft and hard cup); with force torque sensors it was even better, around 0.70. That means in roughly 50–70% of trials, RDP lifted the cup successfully without crushing it. (Recall a partially crushed cup still got a 0.5 score, so it was quite robust.). These numbers translate to massive percentage improvements. The authors note overall “over 35% performance improvement across the three tasks” compared to the best visual-only baselines. In our terms, it was more like doubling or tripling success rates in some cases. In fact, seeing 0 vs 0.7 on a hard task is like going from zero to hero. The results tables [19] are convincing: for every scenario (no perturb, perturb before/after contact), RDP’s line is near the top whereas DP’s line languishes near zero. For example, with perturbations after contact in peeling, DP scored just 0.19, but RDP got 0.80 or higher. These are the moments where the reactivity matters the most, and RDP’s fast loop clearly helped. One more takeaway: RDP worked with different tactile inputs. The authors tested RDP trained with three kinds of sensors: a GelSight mini (optical gel), an MCTac (GelSight-like), and the built-in wrist force-torque. All three gave similar performance (GelSight/Force/MCTac lines all around 0.9 in peeling). This suggests the method is robust to sensor choice. The key is having some high-frequency contact measure; whether it’s a camera-based gel or direct force, the fast policy can use it. So teams with whatever hardware they have could try RDP too. Qualitative Observations. What does RDP feel like in action? The visualizations in Fig. 4 and Fig. 8 of the project show that the slow and fast policies actually differ in their trajectories. In several “inference process” plots, the authors overlaid the original (red/blue) planned trajectory from the slow model and the corrected (green) path after fast adjustments. You can see small but critical deviations: when the tool encounters unexpected resistance, the green path quickly dips or swerves relative to the planned red path, then continues smoothly. For instance, in a peeling scenario, the diffusion policy might plan to move horizontally along the cucumber at a fixed depth. The fast policy, noticing a sudden spike in shear force, might instead nudge the peeler slightly upward or rotate a bit to reduce force. These corrections are exactly what open-loop policies miss. In the bimanual lift, the planner might intend symmetric arm motions, but if one side’s force is too high, the fast controller can slightly slow that arm or close the grip differently on the fly. The big-picture coordination (go forward, upward, squeeze gently) comes from diffusion, but the fine details (stop if slipping, compensate if twitch) come from the GRU. Importantly, RDP does not choose actions exclusively based on force. It uses force as a refinement signal to correct a vision-based plan. This differs from fully force-based controllers (like impedance control) in that RDP still needs the vision to decide the rough trajectory. You wouldn’t want just raw impedance doing everything, because planning overall path requires understanding of object pose and shape. Here vision (and maybe other exteroceptive info) shapes the plan, and touch shapes the execution. Effect of TactAR Data. A reminder: RDP was trained on data collected with TactAR, which made users more effective. How big a difference is that? The authors report that learning from worse data degrades performance. In a final experiment (Fig. 8 on project), they collected the same number of peeling demos with a plain VR teleop (no touch feedback) and trained RDP on that. The results dipped markedly: the score (normalized peel) went down by over 30% when the data was low-quality. Users peeled less length and with more force spikes when they lacked tactile feedback knowledge. Because RDP’s fast network specifically learns “when you see this force pattern, do that correction,” if the demonstrations have unstable forces, the network gets confused. This underscores the synergy of TactAR plus RDP: better demonstrations make the policy better. In short: enabling the human teleoperator with tactile insight not only feels better, it tangibly improves the robot’s eventual skill. For practitioners, this means it’s worth investing in even simple tactile teleop ("feeling by sight") when collecting data for contact tasks. Broader Context and Related Work. Putting Reactive Diffusion Policy in context, we see threads from several areas of robotics and ML coming together. Diffusion Models in Control: Diffusion-based generation has taken off in areas from image synthesis to robotics. The idea that a robot policy could be a diffusion model (Cheng et al., 2024) is relatively new but gaining traction. These models view the entire action sequence as a “data point” to generate, which suits complex trajectories. RDP builds on that by running diffusion not in raw action space but in a learned latent space, which is reminiscent of prior work on latent action diffusion. Operating in latent space can be more efficient and focus on the essential degrees of freedom. Also, hierarchical uses of diffusion (like Wang et al.’s HDP) were explored around handling contact points and ambiguous goals. RDP can be seen as another hierarchy: chunk-level diffusion plus step-level reactive decoding. Imitation Learning & Hierarchies: Many imitation learning methods have long encoded hierarchical structures: high-level planners with lower-level controllers. Classic examples include options, skill chaining, or encoder-decoder policies. More recently, approaches like Alto et al. (2024) used low-frequency open-loop policies plus PD controllers. RDP falls in this tradition but learns both levels end-to-end (though trained sequentially). The concept of action chunking in IL – breaking an episode into fixed-length segments – is common, but RDP goes beyond by making each segment adaptive. It shows how to make chunking actually safe in contact tasks by coupling it with instantaneous feedback. Reactive Control & Tactile Feedback: RDP’s fast policy is essentially a learned impedance controller. In classical robotics, one might hand-design an impedance rule or a neural network PD controller to track a trajectory with force feedback. In learning, some works (e.g. TacRL, Grip etc.) have used depth or vision but few integrate real tactile signals into a learned policy at high rate. The idea of a small high-frequency net adjusting based on touch is straightforward once you phrase it, but actually implementing it with diffusion is novel. Related works in the recent literature (some from these same authors) include policies for in-hand manipulation that use tactile sensing (like DynaGuide, TACT, PolyTouch etc.). RDP stands alongside those as yet another demonstration that jointly learning vision+touch policies yields better contact manipulation. Augmented Reality in Teleop: Using AR for teleoperation is itself a growing area. Prior work (e.g. Pacchierotti et al., 2017) has looked at visualizing haptic information. TactAR’s specific contribution is the “3D deformation field” concept: turning raw sensor outputs into a geometric visualization. It’s akin to showing a false-color overlay on a VR view of the gripper. The novelty here is not new physics but a creative, low-cost engineering solution that proved effective. It’s reminiscent of some robotic teleop systems that overlay force vectors or tactile images, but doing it in AR with a headset for free operator head motion is neat. I expect others might adopt TactAR or similar trick for data collection. It even hints at where telepresence is going: augmenting remote robot arms with human-like sensing. Future Directions: The authors hint at an “ImplicitRDP” extension (project news), which suggests they are exploring end-to-end training (maybe joint loss for slow+fast). The two-stage training of RDP, while intuitive, is somewhat complex. An implicit formulation might simplify or speed up training. Another direction is integrating RDP with reinforcement learning or sim-to-real: use offline RDP initialization and then fine-tune with RL. Since RDP already produces closed-loop behavior, combining it with trial-and-error tuning could yield a robust controller that also adapts to new conditions. Additionally, the concept could generalize beyond manipulation. Anywhere you have a slow high-level plan and fast feedback – for example, legged locomotion with vision-planned steps and reflexive stabilization – a similar slow-fast architecture might work. Conclusion. The “Reactive Diffusion Policy” project tackles a core robotics problem: how to do rich manipulation with both foresight and reflex. By explicitly splitting control into a low-frequency, planning diffusion model and a high-frequency tactile servo network, it reconciles two often-opposed needs. The results speak for themselves: in tasks where a purely visual planner struggles, adding touch in a structured way massively improves performance. And crucially, it shows that this improvement comes from both having tactile info in the training data (via TactAR) and using it properly in the policy structure. For researchers and engineers, the takeaway is clear: if your robot’s tasks involve nuanced contact, you should incorporate tactile sensing into both the data and the model. Furthermore, combining advanced generative models (like diffusion) with reactive controllers may yield better learning-based controllers. The RDP code and models are public, so one can experiment with copying the pipeline to new tasks (maybe replacing the diffuser model or the robot). This work also raises interesting questions about imitation learning design. It suggests a more general recipe: “Learn a coarse strategy, then learn a reflex.” And it shows how augmented teleop can produce high-quality data for that. Future embodied AI systems will likely need more of this kind of integration – of multiple sensor modalities, of multiple time scales – to approach the flexibility of human hand-eye coordination. Reactive Diffusion Policy is a strong step in that direction.