Latest / Embodied AI 101 / TACTIC: Contact-Centric Control for Whole-Arm Manipulation
Transcript
- TACTIC: Contact-Centric Control for Whole-Arm Manipulation. In robotics, we often assume the gripper or end-effector is the only point of interaction with objects. But humans routinely use their entire arm surface during tasks – bracing with a forearm, sliding a hand along a body, etc. For tasks like assistive care or handling large bulky objects, whole-arm contact is essential. Madan et al. (2026) observe that whole-arm manipulation breaks many of the simplifying assumptions in typical learned robotics pipelines. The robot’s joint configuration now directly determines which links touch the world, and contacts can form or break unexpectedly as the arm moves. Vision alone may not see all contact (occlusions abound), and purely learned “end-to-end” rollouts may fail when faced with seldom-seen multi-contact states. To tackle these challenges, the paper “TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation” puts forth a new MPC-based controller that explicitly reasons about contacts through a hybrid model. In short, “TACTIC” fuses RGB-D vision, distributed tactile skin, and a learned dynamics model with analytical kinematic reasoning (via contact Jacobians) within a sampling-based planner. The result is a controller that can predict and regulate forces across the arm, enabling dexterous tasks like flipping and repositioning a mannequin or navigating a stowing maze in simulation and on a real robot. In this deep dive, we’ll unpack the TACTIC framework piece by piece. We’ll start by reviewing why whole-arm manipulation is both powerful and hard, then show how TACTIC’s contact-centric state representation and contact-aware planning address those difficulties. We’ll walk through their hybrid predictive model (analytical kinematics plus a learned latent dynamics), their cost design, and the clever way they bias MPC sampling using contact Jacobians. Finally, we’ll survey the extensive evaluations: simulated benchmarks against state-of-the-art baselines, ablations on TACTIC’s ingredients, and real-robot experiments with a Kinova arm and tactile skin. Throughout, we’ll highlight key insights – for instance, how adding vision- and force-based proximity masks turns out to be critical for success, and how the hybrid model consistently beats purely learned or purely analytic approaches. The Challenge of Whole-Arm, Contact-Rich Manipulation. Imagine an assistive robot helping a person get out of bed. To do this safely and effectively, the robot cannot rely on just its gripper. It may need to brace a forearm under the person’s back, press against an arm with its upper arm, or wrap around a limb – constantly making, sliding, and breaking contact. This is whole-arm manipulation: controlling a multi-joint arm so that multiple links contact the environment. It’s fundamental in human tasks like reorienting a patient, pushing a wheelchair, or carrying a large flexible object. It provides advantages: forces can be distributed over a larger surface, improving stability and comfort. But it also introduces serious complications for control. Many learned motion and planning methods make hidden assumptions that aren’t true in whole-arm tasks. In typical pick-and-place, contacts happen only at the hand and are rare or static; hence, people often ignore contact forces in the model or treat them as secondary. In whole-arm tasks, contact formation is inherent – the arm will intentionally slide contacts along the environment. This means the robot’s joint angles “tightly couple” with contact forces: moving one link changes which contacts exist and how they press. If you merely plan in configuration space (like usual trajectory planning) without awareness of contact, you can’t predict where or when contacts will occur. Worse, vision may not cover contacts hidden behind the body, and tactile sensors only tell you about contacts that already happened, so the state is partially observed. Finally, purely learning-based controllers suffer distribution shift: most training data or demonstrations will contain only a few contact modes. The space of possible multi-link contacts is huge, so a learned policy that hasn’t seen a certain combination may become physically inconsistent or unsafe when it encounters it. TACTIC’s main insight is to build contact reasoning directly into the controller. Instead of treating contacts as surprises, they predict contact spread and forces, and even bias action exploration toward “force-modulating directions”. Concretely, TACTIC is a receding-horizon (MPC) controller. That means at each time step it samples many candidate control sequences, rolls them out with a predictive model, scores each rollout with costs, and chooses the best sequence (executing the first few steps before re-planning). The novelty is in both the predictive model (a hybrid of learned latent dynamics and analytic kinematics through contact Jacobians) and the way the sampler is biased by local contact geometry. These elements make TACTIC “contact-centric” at every level: the state representation, the sampling process, and the model-based evaluation all explicitly consider where on the arm contacts are forming, and what those contacts’ normals are. As the authors summarize: TACTIC “uses a contact-centric state representation by coupling RGB-D, distributed tactile sensing, and a proximity map”. In the planning loop, they use “contact-aware action sampling” that pushes action perturbations into subspaces that change contact forces, and they score trajectories using objectives on predicted forces and proximities. The upshot is performance that “consistently outperforms other methods” in simulation, and impressive demos on a live robot rolling and lifting a mannequin. We’ll now unpack how it all works. A Contact-Centric State: Vision, Tactile, and Proximity. The first question is: what information does TACTIC keep track of at each step? The raw sensing they assume includes an RGB-D camera and a distributed tactile skin on the arm, plus proprioceptive joint measurements. Formally, they define the instantaneous raw observation $o_t = (I_t, D_t, f_t, q_t, \dot q_t)$ where:. $I_t$ is the RGB image,. $D_t$ is the depth map aligned to that image,. $f_t \in \mathbb{R}^{N_s}$ is the vector of tactile readings (forces) from $N_s$ taxels (tactile pixels) on the arm,. $q_t, \dot q_t$ are the robot’s joint angles and velocities. This is already quite rich, but it isn’t yet the “contact-centric” state used for planning. Raw tactile $f_t$ only reports pressures once contact has occurred at certain taxels, and they are sparse. Vision ($I_t,D_t$) sees the scene geometry but not directly where the arm will contact. TACTIC augments these with an explicit proximity mask, $M_t$, that encodes where the arm is near contact with objects or robot parts. Proximity mask: Using the segmented RGB-D, TACTIC first runs a vision-based segmentation (YOLOv11) to separate the robot arm from the object/environment. Each pixel in the RGB-D image is labeled as either “robot” ($\Omega_r$) or “obstacle” ($\Omega_o$). Then every 3D point on the robot’s arm (back-projected from $\Omega_r$) has its distance to the closest object point (from $\Omega_o$) computed (via a kd-tree). These distances are mapped to intensities (near→bright, far→dark) to yield a dense “distance field” image. In parallel, they incorporate tactile signals into this image: for any taxel that is currently sensing pressure above a threshold, they project that taxel’s 3D location into the camera frame and paint a small blob at that pixel. The final mask $M_t$ at each pixel is the maximum of the continuous distance field and the binary taxel indicators. Intuitively, $M_t$ highlights arm surface regions touching or about to touch objects, grounded in the visual frame. This gives a 2D image-aligned proximity map showing where contact is active or imminent. The augmented observation for the controller is then $$o^{cc}_t = (I_t, D_t, M_t, f_t, q_t, \dot q_t),,$$ which is a contact-centric representation of the current state. The RGB-D and the mask $M_t$ together provide vision information about the environment whereas $f_t$ gives direct force feedback if contact has been made. By encoding raw image, depth, and mask through a shared vision backbone (DINOv3 CNN), TACTIC obtains a latent embedding for the visual inputs. The tactile vector $f_t$ goes through a separate lightweight 1D convolutional network. Proprioception $(q_t,\dot q_t)$ goes through a small MLP (with joints sine/cosine encoded to handle wrap-around). All these modalities are concatenated into a single latent state $z_t$. The authors train this multi-modal encoder offline using their collected data, so $z_t$ contains fused information about arm pose, nearby obstacles, and contact status. This representation is crucial. In ablations, removing the tactile sensor and mask (i.e. using only $(I,D,q)$) leads to dramatically worse performance – the robot “goes blind” to where along the arm contact should occur. Even keeping vision but dropping tactile ($f_t$) and mask hurt performance. The proximity mask by itself helps (it’s like a learned feature from vision), but combining it with tactile sensors gives the best success rates and far fewer force violations. In short, TACTIC’s state captures both the “where and how strong is contact now?” (via tactile) and “where could contact happen next?” (via the vision-based mask). Contact-Aware Action Sampling. With a state representation in hand, TACTIC runs an MPC planner over a short horizon. It uses MPPI (Model-Predictive Path Integral control) to sample many candidate action sequences and pick the best according to a cost function. The novelty here is how to sample actions in a contact setting. Standard MPPI (especially variants like STORM) might sample Gaussian perturbations around the current plan, smoothing them with splines, etc. But in a multi-contact scenario, the relevant and feasible movements are constrained by existing contacts. For instance, if your forearm is already pushing on something, then in joint space there is a subspace of motions that push harder into the contact normal, and a complementary subspace that slides or rotates against the contact (affecting tangential velocity). Ideally, to regulate force, we want to sample more in those “push/pull” directions now that contact exists, and less in directions that would violate safety. TACTIC incorporates this intuition by computing contact Jacobians from the current tactile contacts. Concretely, at time $t$ they identify which taxels are in contact (force above threshold). For each contact point $k$, the robot can compute a 3×n Jacobian $J_{k,t}$ that maps joint velocities to the 3D velocity of that contact point. They then project out the normal direction: given the contact normal $n_{k,t}$ (derived from the surface geometry at that point), one row $J^n_{k,t} = n_{k,t}^\top J_{k,t}$ maps joint velocity to normal (perpendicular) velocity at contact. Stacking these for all K active contacts gives a matrix $J^n_t ∈ \mathbb{R}^{K×n}$. Intuitively, the row space of $J^n_t$ spans the joint-velocity directions that change normal motion at the current contacts. Any joint motion lying in the nullspace of $J^n_t$ would move tangentially or not affect those contacts. Next, the sampler uses this Jacobian to steer exploration. They compute a weighted contact Jacobian $fJ^n_t = \mathrm{diag}(\sqrt{\rho_{1,t}},…,\sqrt{\rho_{K,t}}),J^n_t$, where each contact receives a weight $\rho_{k,t}$ based on its current force magnitude (clipped at a saturation level). That way, heavier contacts (important ones) influence sampling more. Then they define two projection matrices: $G^{force}_t$ projects a motion into the subspace of “directions that affect contact normal velocity” (the row space of $fJ^n_t$), while $G^{null}_t$ projects onto its complement. These come from standard formulas: $$G^{force}_t = fJ^n_t^T (fJ^n_t fJ^n_t^T + εI)^{-1} fJ^n_t,\quad G^{null}_t = I - G^{force}_t,,$$ where $ε$ is a small ridge term for numerical stability. Now they take a standard MPPI perturbation sequence $\delta_t^{(j)}$ (over multiple timesteps) and apply a scaling matrix $S_t = σ_{force,eff} G^{force}t + σ{null,eff} G^{null}t$ to it, yielding a contact-conditioned perturbation $\delta^{(j),shaped}t = S_t , \delta_t^{(j)}$. Here $σ{force,eff}$ and $σ{null,eff}$ are scalars (with $σ_{force} > σ_{null}$) controlling how much to amplify directions that change the normal contact motion and how much to down-weight nullspace moves. The net effect: the stochastic samples are biased toward joint-velocity changes that will alter contact forces (rather than purely sliding without changing normal force). Geometrically, if a bump (force) is detected, this pushes predicted actions to “apply or release pressure” in that contact normal, which is exactly what you want for regulating force. Finally, they add an explicit safety projection. If any contact is already pushing harder than the safe threshold $F_{safe}$, they mark those contacts as violating. Then they solve, for each sampled action sequence, a small optimization to project the first-step perturbation onto the safety cone defined by the violation normals. Intuitively, they constrain that $\langle u_{t+i}, n_{viol} \rangle \ge 0$ (normal component non-negative) so the robot’s motion is away from these existing pushes. This ensures the sampler won’t propose actions that would drive into a contact beyond the safe limit. (The optimization is simply a minimum-norm projection subject to that linear inequality.). All together, the contact-aware sampler ensures that the random shooting in the MPC is not blind: it softly focuses on adjusting contact forces rather than going haywire. In ablations, using this Jacobian-guided sampling (C2) dramatically reduced the number of unsafe force violations and produced smoother motions compared to vanilla STORM sampling. Hybrid Predictive Model: Analytic Kinematics + Learned Contact Dynamics. Next, when evaluating a candidate action sequence, TACTIC uses a two-branch predictive model. This is where the “hybrid” part comes in. Some parts of the robot’s future state are easy to compute analytically – namely, the forward kinematics. But other parts – where exactly contacts will form, break, and what forces will ensue – are hard to write in closed form. TACTIC splits the difference:. Analytical branch ($F_k$): Given the current joint state $q_t$ and a candidate sequence of joint-velocity commands $u_{t:t+H-1}$, they roll out the kinematics forward deterministically. Using first-order integration, they compute predicted joint states $\hat q_{t+1:t+H}$, velocities $\hat{\dot q}{t+1:t+H}$, and from these derive task-space quantities via forward kinematics. In particular, this gives the end-effector trajectory $\hat e{t+1:t+H}$ (if there’s an explicit goal in end-effector space). Importantly, at each predicted state they also recompute the active contact Jacobian $J_{k,t+i}(\hat q_{t+i})$ for those contacts seen at time $t$. This analytic branch ensures “physical consistency”: things like joint limits, self-collisions, actual distances in space are enforced exactly. However, it cannot predict new contact events beyond the ones currently active – it essentially assumes existing contacts will persist along the predicted motion (since it only uses the Jacobians from the initial contacts). Learned latent dynamics ($F_l$): In parallel, TACTIC runs a learned dynamical model on the latent encoder $z_t$. Specifically, they feed a short history of latents $z_{t-L+1:t}$ (where $z_s = \phi(o^{cc}s)$) and the candidate action sequence $u{t:t+H-1}$ into a recurrent or transformer network that predicts future latents $\hat z_{t+1:t+H}$. This model was trained offline on demonstration data (see below), and it does capture patterns of contact formation and evolution. For example, if in training the arm often transitioned from one contact to another in certain motions, the latent model will internalize that and reflect it in $\hat z_{t+i}$ predictions. We can think of $\hat z_{t+i}$ as encoding things like “we have made contact here, now there’s force $X$, next likely contact on arm segment Y.”. In evaluation, these two righthand branches are run in parallel. From the analytic branch we get a rollout of joint states and contact Jacobians, and from the learned branch we get a latent trajectory. Some quantities are naturally predicted by one branch or the other. For example, we only really have an analytic way to compute future end-effector poses (from $\hat q$) and from that check collisions. Conversely, we only have the learned model that can guess future forces or proximities. To score a rollout (and thus compute its cost), TACTIC uses a combination of kinematic constraints, force costs, and goal terms. Concretely, their overall cost for a trajectory $u_{t:t+H-1}$ is $$ J(u) = \sum_{i=1}^H \bigl[ w_{kin} c_{kin}(\hat s_{t+i}) + w_{safety} c_{force}(\hat s_{t+i}, \hat z_{t+i}, f_t) + w_{goal} c_{goal}(\hat z_{t+i}, z_g) \bigr], $$ where $\hat s$ includes all analytic state elements (joint pose, end-effector, etc), and $\hat z$ is the predicted latent. $c_{kin}$ penalizes violations of kinematic feasibility: collisions, joint limits, or distance from an end-effector target (computed from $\hat s$). $c_{force}$ penalizes unsafe predicted forces. This is clever: it combines the learned contact predictions (do we think contact will happen, from $\hat z$) with an analytic estimate of what that would imply for force. Unfortunately the appendix must have details, but essentially they translate predicted contact contexts into expected forces and penalize if it exceeds the safe limit. $c_{goal}$ encourages progress toward the goal in latent space: they compare $\hat z_{t+i}$ to the goal latent $z_g = \phi(o_g)$ (the goal observation encoded), so the model knows to work toward matching the goal features. Thus, any candidate action sequence is judged by how well it respects contactsafety (via $c_{force}$ and $c_{kin}$) and makes progress toward $o_g$. Because we have both branches, $c_{force}$ can use analytic information (contact Jacobians from $s$) when computing normal velocities, but also a data-driven estimate of where contacts happen. To train the learned model $F_l$, they used ordinary supervised learning on logged trajectories (see below). The latent predictor is trained to predict the next latent and further out, using a consistency loss that includes one-step prediction and multi-step autoregressive consistency. They also train small decoder networks so that from $\hat z$ they can decode things like “will each taxel be in contact” and “approximate force magnitude”, which feed into $c_{force}$. In effect, $F_l$ is a world model trained on vision-tactile data, while $F_k$ is the uncompromising physics. The combination yields the best of both worlds: $F_k$ ensures physical plausibility, while $F_l$ captures complex, contact-dependent dynamics that would be hard to derive. In deployment, these predictions are evaluated in parallel for each of $N$ sampled sequences at each planning step. To keep computation real-time, they even use a two-tier “draft” model: first they roughly score all $N$ sequences with a cheaper drift ($F_l^{draft}$), then only refine the top-k with the full predictor. Collecting Data and Training. Before diving into results, a word on how they got the data to train those learned parts. TACTIC relies on a dataset of whole-arm manipulation trajectories. The authors built a full exoskeleton teleoperation rig to collect this data from human demonstrations. The exoskeleton is a scaled seven-DOF wearable that mirrors the Kinova Gen3 arm’s joint motions. When the human operator moves their arm in the exoskeleton, the robot’s arm moves accordingly. This mirrored whole-arm teleoperation is crucial: it lets humans move the robot in natural contact-rich ways without the uncanny aimless swinging of typical teleops. By doing this in tasks like wiping a mannequin or dragging a person, Madan et al. obtained rich demonstrations where forearms and upper arms bear on surfaces – exactly the regimes we care about. (All hardware designs are open-source: the paper is accompanied by exoskeleton CAD, tactile sensor mount designs, and assembly instructions.). From these teleoperated runs (plus some automated “random play” to cover space), they trained the latent dynamics $F_l$ with offline learning. Notably, no method here is learning online; even the baselines are trained offline. This means the model has seen a variety of contacts during training. They use techniques (like a latent consistency loss and a multi-step rollout penalty) to reduce drift, following best practices from latent world-model literature. They also train the tiny decoders for contact onset. In addition, for long-horizon tasks (e.g. fully rolling a person), they train a value function in latent space as a kind of terminal cost (via Implicit Q-Learning) to supplement the finite-horizon MPC. This helps because the pure MPC horizon might not capture the far-away goal well in extremely long tasks. With the predictive components trained, they freeze everything and run the MPC on actual tasks. Now let’s see how it performs. Simulation Experiments. The authors evaluated TACTIC in a suite of four simulated tasks, each designed to stress different aspects of whole-arm contact control:. 3D Maze Goal Reaching (Maze): A complex 3D maze of cylinders (built in RCareWorld) blocks the robot’s reach. Collision-free end-effector paths are often impossible; gliding with forearm contact is the key to pushing through. The robot’s goal is a target pose, with a force limit to respect. Bed Bathing / Wiping: A surgery-bed scenario where the robot wipes a sponge along a mannequin’s arm. End-effector must follow a trajectory, but wide forearm contact over a broad area occurs incidentally as it moves over the bed surface. Tabletop Reach (Reach): A Franka arm reaching for a block on a table with obstacles (from RoboHive). Here the goal is (mobility plus obstacle-averse reach) mainly testing hybrid modeling. A pretrained transformer (V-JEPA2) was fine-tuned on this. Granular Manipulation (Granular): An xArm shovels small granular objects (from FleX sim) to form a pile. This tests contact-rich physics and was used to evaluate coupling TACTIC with a pretrained grain-world model (DINO-WM). For these simulations, data came from two sources: (i) RRT-generated start/goal pairs and (ii) random “noisy joint-vel” explores of the space, both under safe conditions. No online interaction (like RL) was performed – all models are offline-trained and then tested. They compared TACTIC against a variety of baselines and ablations:. Ablations (C1-C3 tests): To isolate each contribution, they removed elements:. TACTIC-NoTacNoMask: remove tactile sensors and the proximity mask (so only raw RGB-D & proprio). (Q1: tests the value of full representation.). TACTIC-NoTac: remove tactile signals but keep the vision-based mask. (Effect of mask alone.). TACTIC-NoCaSa: remove contact-aware sampling (fall back to base STORM). TACTIC-L vs TACTIC-K vs full TACTIC: latent-only model (F_l only), kinematics-only (F_k only), or the full hybrid (F_l+F_k). Pretrained Models (B1-B2): They took powerful off-the-shelf vision-world models and plugged TACTIC ideas into them. B1 (Reach): apply V-JEPA2 (an image-based latent predictor) with only an added kinematic cost. B2 (Granular): apply DINO-WM (transformer on pixels for fluids) similarly.The point was to see if adding an analytic goal or structure to these would help (spoiler: yes, hybridized is better). Model-based RL (B3-B4): Strong latent world-model RL baselines. B3: DreamerV3, which learns a latent dynamics + actor/value. B4: TD-MPC2, an MPC-over-latent method.These methods could learn contact indirectly, but do not explicitly use contact Jacobians. Imitation (DP): A diffusion policy imitation from offline data (Diffusion Policy [42]) as a “safe imitation” baseline. They ran each method on the tasks, measuring success rate (SR), time to completion (TTC), and force violations (FV) (# of timesteps exceeding the force limit). Results driven by Tab II–V and the text:. Key findings:. Contact-centric state (C1): As expected, the full TACTIC greatly outperformed the no-tactile/no-mask variant. For instance in Maze, removing tactile and mask cut success from 87% to 63%, and unsafe forces more than doubled (84 vs 39 violations on average). Just adding the vision-based mask back (NoTac) recovered some performance. This shows that both modalities matter. The robot needs dense anticipatory contact info – either from proprioceptive touch or from the image-informed proximity. Contact-aware sampling (C2): Swapping out the STORM sampler for TACTIC’s Jacobian-biased one produced smoother, safer policies. In Maze, mean jerk (a measure of smoothness) roughly halved, and force violations plummeted (down to 39 vs 112). The contact-aware scheme found safer trajectories by actively exploring the right directions; vanilla sampling would occasionally jerk too hard against a wall, pushing forces above threshold. Hybrid modeling (C3): Purely latent predictions or purely kinematic rollouts each had flaws. The latent-only version (TACTIC-L) had decent success but huge force overshoots (lots of violations, since it lacked the grounding of physics). The kinematic-only (TACTIC-K) was safe but often failed the task entirely (it could respect contact normals but didn’t foresee contact changes). The hybrid TACTIC achieved a nice balance: high success (87%) with relatively few violations. Indeed Table IV shows hybrid TACTIC kept success ⟼ 87% with 39 force violations, whereas latent-only was 64% succ, 108 violations, and kin-only 43% succ. The analytic branch “anchors” the predictions, while the learned part adapts. Method comparisons: TACTIC beat the world-model RL baselines on multi-contact tasks. In Maze, Table V reports DreamerV3 got 76% success, TD-MPC2 65%, while TACTIC hit 87%. On the Bed-Bathing task, DreamerV3 was only 52% vs TACTIC’s 79%. TACTIC not only succeeded more often, it incurred far fewer force threshold violations – thanks to the explicit contact costs and sampling. The diffusion policy (DP imitation) did succeed sometimes, but had 0 force violations (because as an offline behavior clone it might be overly conservative or failing by stopping). Generalizing pretrained models: Even when starting from strong pretrained world models, TACTIC’s hybrid approach helped. For the Reach task with V-JEPA2, adding a simple analytic goal term allowed matching performance with 34% less fine-tuning data (and presumably fewer unsafe contacts). With the frozen granulardeco (DINO-WM), adding kin costs reduced prediction error. The lesson: injecting known kinematics into learned predictors stabilizes them, which is consistent with the paper’s theme. Overall the simulation results strongly validate TACTIC’s design choices. By integrating tactile and vision into a compact state and using contact Jacobians everywhere – in sampling and modeling – they substantially outdo both “black-box” learned approaches and naive analytic ones. Real-Robot Demonstrations. Simulations are great, but the real proof is on hardware. Madan et al. implemented TACTIC on a real Kinova Gen3 7-DoF arm. They wrapped it with a novel tactile skin of 22 pressure pads (see sidebar) and used an Intel RealSense D455 for RGB-D vision. They also built the exoskeleton mentioned earlier for data collection, but the controller itself simply runs in real time on the actual robot (no teleop once testing starts). The planned actions are joint velocities at 40 Hz, re-planning continuously. They tested three real tasks:. 3D Dynamic Maze: A physical analog of the simulation Maze. Four cylindrical obstacles move slightly (oscillating 3 cm) to add dynamism. The robot must thread its arm through without exceeding ~5 N force on any contact. Unlike simulation, precise collision avoidance is imperfect, so the robot really must feel its way through. Side Rollover: A life-size manikin (∼46 lb vinyl dummy) lies supine and must be rolled onto its side. This explicitly tests contact regulation: the robot wraps arms around, pushing against the dummy’s body, and must not exceed safe forces at any arm link. This is akin to turning a patient, a classic caregiving problem. The end goal is a target orientation of the dummy (scalar angle or bed coordinate). Limb Repositioning (LimbRepo): The robot lifts and repositions the manikin’s legs to specified joint-angle targets. To support the (heavy) legs smoothly, the robot must maintain multiple arm contacts under the dummy’s legs or body. For each task, they compared three behaviors:. Expert teleoperation: A human operator using the exoskeleton (same device as for data collection) to show best-case performance. Diffusion Policy (DP) imitation (trained on the same offline demos). TACTIC (no human input, fully autonomous). The results are summarized in Table VI of the paper. Let us translate those numbers into words:. Dynamic Maze: 5/5 successes by expert (average time ~30s, avg force violations ~97). The expert breaks the force threshold on average ~97 timesteps – surprisingly, even a human teleop often exceed the “safe” margin briefly. The diffusion policy fails most trials (2/5 success) but incurs 0 violations (it likely gives up early). TACTIC succeeds on 3/5 (so not perfect yet), taking longer (~71s) than expert, but with only ~70 violations. That is, TACTIC was safer: the expert hit the force limit more often than TACTIC did. In this very constrained space, TACTIC still managed to navigate three times even though all methods struggled. Side Rollover: Expert got 20/20 (100%) success, average ISR ~45 s, but again 141 force violations on average (so quite aggressive pushing). Diffusion policy did 3/20, very few, with lots of overshoot forces. TACTIC managed 12/20 success (60%), about 81s on average, with 104 violations. Crucially, TACTIC again incurred fewer force violations than the human (104 vs 141). In other words, TACTIC was more conservative with force even while accomplishing the roll more often than DP. This supports the claim that TACTIC regulates force actively – sometimes at the expense of speed/success. Limb Repositioning: Expert 20/20 (30s, 50 viol.), DP 5/20, TACTIC 14/20 (74s, 38 viol.). Again the robot ran more slowly (74s vs 30s) but with only 38 force violations compared to expert’s 50 (and DP’s 61). In all cases, TACTIC clearly outperforms the imitation baseline (which mostly fails to complete tasks). It doesn’t yet reach the expert’s reliability, but it’s impressive that with no online fine-tuning TACTIC can even achieve 60-70% success on these complex, risky tasks. Equally important is the force regulation: TACTIC violated force limits less often than a human moving through the same task (likely because it is continually optimizing for safety constraints). The authors note that the DP baseline, having 0 force violations in one case, probably just aborted or moved very cautiously whenever contact was sensed. One should note that real-world hardware introduces noise, model mismatch, and delays, so MPC is more challenged than in sim. The fact that TACTIC could run in real-time (with the two-tier model trick) and handle sensor noise is noteworthy. It points to a practical takeaway: for contact-rich assistive robots, a hybrid model-MPC with tactile feedback can achieve safer behavior than naive end-to-end methods. Discussion and Impact. TACTIC represents a thoughtful integration of several ideas:. Multi-modal state fusion: It brings visual, proprioceptive, and tactile data together in a unified latent. The proximity mask is an especially neat idea: linking vision and kinematics to form a “where-is-the-arm-close-to-obstacles” image. This overcomes the occlusion of pure vision and the sparsity of pure touch. Contact-aware planning: Using contact Jacobians to bias sampling is a timely twist on MPC. It’s not fully novel (the idea of projecting action perturbations by contact Jacobians has appeared in control literature), but combining that with data-driven predictions is new to whole-arm planning. Hybrid modeling: Mixing an analytical branch and a learned branch addresses distribution shift head-on. It acknowledges that we do have partial knowledge of the physics (forward kinematics), and that should guide predictions. At the same time, it leverages learning to fill the complex gaps (contact changes). The clean ablations confirm the value of this design. Hardware contributions: Although the paper’s core is algorithmic, the project page makes clear that the team also built and open-sourced an exoskeleton teleop and a tactile skin for Kinova arms. This is valuable for the community, as data for whole-arm tasks are scarce. Readers interested in experimenting can find CAD, bill-of-materials, and software links on their project site. Limitations: TACTIC is by nature closed-loop and computationally intensive. It relies on relatively short horizons (0.5-1 second) and frequent re-planning, so actions beyond the horizon are indirectly handled by a learned value function (for long tasks). The successes, while strong, still needed several re-plans per task. TACTIC also requires a pretty detailed analytic model (Jacobian, camera calibration, etc), and a good segmentation of robot vs. objects. In cluttered, real scenes, segmentation errors might degrade performance. However, those trade-offs are part of its strength: it is explicitly built for contact. By biasing exploration and scoring with contact metrics, it avoids many pitfalls of black-box learning. Future work could extend TACTIC to partially unknown objects (e.g. moving beyond known props), improve tactile skin coverage, or integrate proprioceptive force/torque sensing at joints. Another direction is to combine with high-level planners: right now tasks like “roll the manikin” required specifying subgoal angles. A complete system could incorporate human intent (language or gesture) to set those goals. In summary, the TACTIC framework is a significant step toward safe, versatile whole-arm control. It reminds us that for really contact-rich tasks, we cannot treat contact as a mere disturbance – we must put it at the center of perception, prediction, and control. In doing so, this work bridges modern learning (visual world models) with classical control (kinematics, Jacobians) in a way that practitioners in assistive robotics or manipulation can immediately leverage. Sources: The above discussion is based on Madan et al. “TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation” (2026), with additional context from their project page and supplementary materials. The real-robot evaluation results are summarized from Table VI of the paper. (All quotes and data here are from the paper’s text and tables, reformatted for clarity.).