Latest / Embodied AI 101 / VEDA and the Moment Vision Stops Being Enough
Transcript
- VEDA and the Moment Vision Stops Being Enough. VEDA: A Multimodal Dual-Attention Framework for Contact-Aware Temporal Anomaly Detection in Industrial Humanoid Manipulation • Huiqing Nong et al. (four authors) • affiliations not available in accessible public metadata • International Journal of Humanoid Robotics, World Scientific • 2026. Research note — August 22, 2026: The supplied DOI resolves to World Scientific and its identifier corresponds to the International Journal of Humanoid Robotics. However, the publisher’s full text, complete author metadata, and abstract were not accessible during research, and no public preprint, project page, or code repository surfaced in search. This episode therefore separates the paper’s confirmed framing from technical interpretations inferred from its title. I will not attribute unverified architecture details, dataset statistics, ablations, or performance numbers to the authors. The Anomaly Hides in the Relationship. Imagine a humanoid robot inserting a keyed connector on an assembly line. From the head camera, everything looks plausible. The robot’s hand is in the expected region. The connector appears aligned. The arm trajectory differs only slightly from previous successful executions. A vision-only monitor may see nothing worth reporting. But the wrist force–torque sensor tells a different story. Axial force rises too early. The fingers begin applying more grip force to compensate. The wrist controller introduces a small lateral correction, the torso shifts to preserve balance, and the connector stops advancing. No individual measurement is necessarily extreme. The anomaly is the relationship among them: the observed force is inconsistent with the visible geometry, the corrective motion is inconsistent with the expected contact phase, and the discrepancy persists across several time steps. That is the problem suggested by VEDA’s title. The interesting unit of detection is not a bad image or a large force sample. It is a temporally unfolding disagreement among vision, robot state, and contact-related signals. Industrial manipulation makes that disagreement especially subtle because large forces, abrupt accelerations, and constrained motion can all be perfectly normal once contact begins. A detector that ignores contact will confuse normal interaction with failure. A detector that ignores time will miss the difference between a harmless force transient and a developing jam. And a detector that simply concatenates every sensor stream may be overwhelmed by differences in dimensionality, sampling rate, noise, and relevance. VEDA’s stated combination—multimodality, dual attention, explicit contact awareness, and temporal anomaly detection—is therefore a sensible decomposition of the real problem. The framework appears to ask not merely, “Does this observation look unusual?” but, “Given what the robot is doing, where it is in the interaction, and what its other sensors report, is this sequence still physically coherent?”. What Kind of Anomaly Detector Does a Humanoid Need?. Anomaly detection in a stationary industrial arm is already difficult. The same problem on a humanoid introduces several additional layers of coupling. First, a humanoid’s manipulation state extends beyond the end effector. Arm contact may cause compensatory motion in the torso, waist, legs, or floating base. A local insertion failure can therefore leave a distributed signature across joint torques, balance-controller outputs, foot loads, hand state, and camera motion. Second, the platform’s compliance is both useful and ambiguous. Compliance allows a robot to tolerate pose error and explore contact safely, but it also means that early failures may not produce a sharp threshold crossing. A controller can absorb a misalignment for several moments before the execution becomes visibly unsuccessful. Third, the same sensory value can mean different things in different phases. Five newtons of lateral force may be suspicious during free-space approach, desirable during surface following, and dangerously low when a part is supposed to be clamped. Motion near zero can indicate successful seating, a stalled insertion, deliberate force regulation, or a paused controller. The detector therefore needs a conditional definition of normality. In conceptual terms, it should estimate whether the current observation is normal given the task, recent history, robot configuration, and contact state. That is substantially different from learning one global boundary between “normal” and “abnormal.”. The likely anomaly set is also heterogeneous. Industrial manipulation failures may include misgrasps, slips, dropped objects, premature contact, unexpected collisions, excessive force, tool or workpiece mismatch, insertion jams, poor weld tracking, sensor faults, controller oscillation, and whole-body balance disturbances. These events do not share one obvious signature. Some are visible but generate little force. Others are almost entirely haptic. Some develop slowly, while others require an immediate stop. A useful detector must consequently do three things at once:. Detect that execution has departed from its normal regime. Localize when the departure began. Provide enough structure for the supervisory controller to select an appropriate response. The third requirement is often underemphasized. A scalar anomaly score can stop a robot, but it cannot tell the system whether to retry a grasp, back out of an insertion, reduce force, switch sensing modes, or trigger a human inspection. Even if VEDA is primarily a detector rather than a diagnostic model, its modality- and time-resolved attention may provide useful signals for this supervisory layer. Reading the “Dual Attention” in VEDA. The accessible title does not reveal exactly what the authors mean by dual attention. In multimodal temporal models, the term commonly describes one of three arrangements:. attention within each modality followed by attention across modalities;. modality attention followed by temporal attention;. spatial attention over perceptual features paired with temporal attention over the execution sequence. These are related but not equivalent. The most natural architecture for VEDA’s stated objective would combine elements of all three: modality-specific temporal encoding, cross-modal interaction, and a contact-conditioned mechanism for emphasizing the relevant moments. What follows is the architecture I would expect from that problem formulation, not a claim about the inaccessible implementation. Preserve Each Modality Before Fusing It. The first principle is that raw concatenation is rarely an adequate multimodal model. Images, joint states, motor currents, wrench measurements, and tactile arrays have different geometries and noise characteristics. An image encoder should preserve spatial structure. Proprioception should retain joint and kinematic relationships. A six-axis wrench can be represented compactly, while dense tactile sensing may need a spatial encoder analogous to a small vision backbone. The streams also differ semantically. Vision provides relatively global information about object identity, pose, scene context, and free-space motion. Proprioception describes what the robot commanded or physically executed. Force and touch reveal the local interaction at the contact interface. That division is not absolute, but it is a useful design prior. Work on differentiable filtering for manipulation similarly emphasizes that vision, proprioception, and force are complementary while having very different uncertainty profiles. Rather than flattening those signals immediately, differentiable filters retain an explicit estimation structure so that modalities can contribute according to their learned reliability. A VEDA-like implementation should therefore begin with separate encoders that produce modality tokens in a compatible latent dimension. Crucially, “compatible” should not mean “indistinguishable.” The network benefits from knowing which tokens came from the wrist wrench, which came from the head camera, and which represent the torso or hand state. It should also model missing and degraded sensors during training. Real deployments encounter dropped frames, occluded cameras, saturated tactile cells, force bias drift, and delayed robot-state packets. A model that only sees perfectly synchronized, complete sensor bundles can turn a mundane communications problem into a stream of false alarms. One Attention Mechanism for Temporal Evidence. The first attention operation should answer: which parts of the recent history matter?. Anomalies are rarely defined by one sample. A force spike can be harmless if it decays immediately. A modest force can be dangerous if it rises steadily while insertion depth remains unchanged. A gripper opening may be correct at release time and catastrophic during transfer. Temporal attention allows the representation at the current time to consult earlier observations. For online deployment, this attention must be causal: the detector cannot use future frames that will not exist at inference time. A bidirectional model may produce attractive offline numbers while understating real detection latency. The temporal context length is itself a deployment parameter. A short window responds quickly but may miss gradual failure development. A long window captures richer context but increases memory, computation, and the risk that the model will focus on task identity rather than the local onset of failure. The best design is often multiscale. A local branch can model high-frequency force transitions around first contact, while a longer branch represents slower task progress. Dual-attention time-series models such as DTAAD demonstrate the broader appeal of combining temporal convolution with attention, while DCdetector uses dual attention and contrastive representation learning to separate anomalous multivariate sequences without relying solely on reconstruction. For robotic manipulation, however, temporal modeling has to remain phase-sensitive. The detector should not average away a brief but safety-critical collision simply because the rest of the window resembles a normal trajectory. A Second Attention Mechanism for Cross-Modal Consistency. The second attention operation should ask: which modalities explain or contradict one another right now?. Suppose the wrist experiences a lateral force increase. If vision shows the hand intentionally making surface contact and the commanded motion is tangential, that force may be expected. If vision indicates free space and joint velocity suddenly collapses, the same measurement suggests an unexpected collision. Cross-modal attention can model these conditional relationships more expressively than static concatenation. A visual token can query force and proprioceptive history; a tactile token can query visual geometry; whole-body state can contextualize apparent camera motion caused by balance correction. Recent visuotactile work offers evidence for this structured approach. A 2026 Cross-Modal Transformer for insertion uses self- and cross-attention to combine wrist-camera and tactile observations, augmented by a bilateral force-balance prior. On the TacSL benchmark, the structured model substantially outperformed naïve and gated fusion alternatives. The important lesson is not the exact success rate, but that principled cross-modal interaction and physically meaningful regularization can outperform simply providing more sensor channels. This is where contact awareness can enter VEDA most directly. Contact state can be represented as a token, a gating variable, an attention bias, or a latent state inferred jointly from the sensor streams. In each case, it changes which cross-modal relationships should be treated as normal. Attention also offers a limited form of inspectability. Engineers can examine whether a detection was associated primarily with wrist force, hand vision, torso compensation, or an earlier point in the trajectory. These maps are useful diagnostic artifacts, although they should not be confused with causal explanations. High attention weight means the model used a feature strongly; it does not prove that feature caused the physical failure. Contact Is an Operating Mode, Not Just Another Measurement. The phrase “contact-aware” is arguably the most consequential part of VEDA’s title. A weak implementation might append a binary contact flag to the fused feature vector. A stronger implementation treats contact as a changing operating mode with its own dynamics. For a representative assembly task, useful phases might include free-space approach, first touch, preload, constrained motion, seating, verification, and release. Sliding, rolling, impact, and static load are also distinct contact regimes. The expected relationship between force, motion, and visual progress changes at every transition. First contact is particularly important. The detector must determine whether contact occurred at the expected time and location, whether the resulting force direction is compatible with the intended geometry, and whether the controller’s response moves the system toward or away from success. This suggests a hierarchical model. A lower layer estimates contact or task phase. A higher layer evaluates whether the multimodal sequence is plausible under that phase. The phase representation should preferably be soft rather than a brittle one-hot label because contact transitions are uncertain and sensors can disagree about their exact onset. Contact awareness also changes how anomalies should be labeled. A collision is not simply “large force.” It is force inconsistent with the expected contact graph. A jam is not simply “low velocity.” It is persistent low progress under rising load in a phase where motion should continue. A slip is not simply tactile change. It is a tactile transition accompanied by object motion that cannot be explained by the intended manipulation. This relational view aligns with current contact-rich learning research. ContactWorld’s study of vision–tactile world models reports that successful planning depends not only on adding tactile data but on preserving spatial structure, temporal continuity, and compatibility between modalities. Tactile force-field representations were most useful when paired with a perceptual representation that expressed geometry at a comparable level of structure. For anomaly detection, the implication is clear: sensor count is less important than representational alignment. A thousand tactile values do not automatically help a model whose visual pathway emits one coarse scene vector. Fusion works when the representations describe corresponding physical phenomena. What Should Produce the Anomaly Score?. There are several viable training paradigms, and the paper’s inaccessible materials prevent us from identifying which VEDA adopts. A fully supervised model can classify normal and anomalous windows using labeled failures. This is straightforward and often effective, but industrial failure data are sparse and strongly biased toward anticipated fault categories. The result may be a good known-fault classifier rather than a true anomaly detector. A normal-only reconstruction model learns to reproduce successful multimodal sequences and scores large reconstruction errors as anomalies. The LSTM-based variational autoencoder proposed for robot-assisted feeding is an early example of this approach: it models multimodal execution trajectories so that departures from learned normal behavior become detectable. Reconstruction has an important weakness, however. Powerful models can sometimes reconstruct anomalous inputs surprisingly well. Raw reconstruction error can also be dominated by a difficult but unimportant modality—lighting variation in images, for example—while missing a small force discrepancy that matters physically. Prediction is often more useful for contact-rich manipulation. Instead of reconstructing the present, the model predicts the next visual, tactile, force, or latent state. The anomaly score then reflects a violation of learned dynamics. Imagine2touch demonstrates the broader idea by predicting tactile readings from local depth observations. For anomaly detection, the residual between predicted and observed touch could reveal contact inconsistent with visible geometry. The 2026 Touch Dreaming work takes prediction further in a humanoid policy. Its shared Transformer predicts future hand-joint forces and future tactile latents alongside action chunks, encouraging the representation to encode the physical consequences of manipulation. Although this is a control architecture rather than an anomaly detector, its predictive touch objective is highly relevant: the same latent residual could become an execution-monitoring signal. A robust VEDA-style detector might therefore combine three scores:. a discriminative probability of known failure;. a predictive residual measuring violated dynamics;. a cross-modal consistency score measuring whether sensors still tell the same physical story. The final score should be calibrated by task and contact phase. A single global threshold is unlikely to work equally well during free-space motion, impact, constrained insertion, and release. The Evaluation That Would Convince Me. For this class of work, aggregate accuracy is not enough. Frame-level AUROC can look excellent even when a detector produces many short false alarms or recognizes a failure only after the task is unrecoverable. Industrial evaluation should report event precision and recall, false alarms per operating hour, detection delay relative to annotated onset, and performance at thresholds that correspond to realistic intervention costs. Precision–recall curves are especially important when failures are rare. Results should also be separated by fault type and contact phase. A model may excel at visually obvious dropped objects while failing to detect overforce, slip, or early insertion jams. The experimental split matters just as much as the metric. Randomly splitting windows from the same execution can leak nearly identical adjacent sequences into training and testing. More persuasive protocols hold out complete runs, objects, payloads, robot speeds, environmental conditions, or fault families. For a humanoid system, I would want at least four generalization tests:. unseen workpiece poses or geometries;. unseen execution speeds and force-control gains;. changes in camera appearance or sensor calibration;. anomalies not represented during training. A leave-one-fault-family-out experiment would be particularly revealing. If a model trained on slips, collisions, and overforce can still flag an unseen jam, it has learned a meaningful boundary around normal execution rather than memorizing failure templates. Ablation studies should remove each modality, the contact representation, the temporal branch, and the cross-modal branch independently. Static concatenation is the essential fusion baseline. Other useful stress tests include shuffling the contact labels, introducing small timestamp offsets, dropping one modality, and replacing measured contact with a noisy estimate. Timestamp perturbation deserves special emphasis. A detector may appear to learn physical consistency while actually relying on perfectly aligned sensor events. In deployment, camera, tactile, and controller messages can arrive with different and variable delays. Graceful degradation under modest temporal misalignment would be strong evidence that the model learned a real interaction pattern. There is also a useful contrast with conventional robotic fault detection. A 2024 IEEE Access study compressed homogeneous force or torque signals into subsystem-level norms and trained random forests on established robotics datasets, reporting greater than 99% classification metrics and a 95.85% reduction in stored data. That is an attractive engineering baseline, but the authors explicitly identify heterogeneous signals such as vision as an open limitation. VEDA’s claimed value should therefore appear not merely as a small accuracy gain, but as better coverage of failures whose evidence is distributed across perception and contact. Where VEDA Fits in the Contact-Rich Learning Landscape. VEDA sits at the intersection of two research traditions that have often evolved separately. The first is robot introspection and execution monitoring. These systems examine force, torque, proprioception, audio, vision, or controller residuals to identify failure. Many early approaches use hand-engineered features, recurrent autoencoders, or classifiers trained for a specific task. The second is multimodal policy learning, where touch and force are used to improve the action policy itself. Recent work increasingly treats tactile information as a central representation rather than an auxiliary sensor. Touch Dreaming belongs to the second tradition: it predicts future touch to improve humanoid action generation. The visuotactile Cross-Modal Transformer uses structured attention and a physical force prior to improve insertion. ContactWorld studies which representations support long-horizon planning. These systems ask how touch can produce better actions. VEDA instead appears to ask how the same information can determine whether the current action sequence remains safe and plausible. That distinction matters architecturally. A task policy is optimized for expected return or imitation accuracy. An anomaly detector is optimized for calibrated uncertainty, onset sensitivity, and rejection of states outside the training distribution. A representation excellent for action prediction is not automatically a reliable safety monitor because the policy and monitor may share blind spots. The safest arrangement is often a partially independent monitoring model. It can consume policy internals, but it should also retain direct access to raw or lightly processed sensory evidence. Otherwise, an error in the policy’s representation may simultaneously corrupt both action selection and anomaly detection. HapticVLA offers another provocative direction. It learns tactile-aware behavior with touch during training and then distills that information into a student that operates without tactile hardware at inference. An analogous detector could use instrumented training cells to learn a privileged contact representation, then deploy a reduced model on robots with cheaper sensing. The trade-off would need careful validation: eliminating tactile hardware also removes a direct channel for detecting unexpected physical events. The broader opportunity is to connect anomaly detection with multimodal world modeling. Instead of classifying a window independently, a monitor could predict the distribution of physically plausible next states under the commanded action. An anomaly would then be a low-probability transition, and diagnosis could be phrased as identifying which modality or contact relation violated the prediction. From Anomaly Score to Industrial Safety Layer. A deployable system needs more than a neural network checkpoint. The model must run causally with bounded latency. Sensor preprocessing, synchronization, inference, thresholding, and communication with the robot controller all consume time. The relevant requirement is end-to-end reaction time, not isolated GPU throughput. Thresholding should include hysteresis and persistence. Triggering a stop on every single high score will produce nuisance interruptions. Waiting for a long sequence of high scores may be unsafe. Different anomaly classes may need different policies: an unexpected impact can trigger immediately, while a possible slip may require confirmation over several samples. The response should also depend on context. Useful reactions include slowing the motion, reducing force, freezing the current pose, retracting along the approach direction, opening the hand, transferring control to a recovery policy, or requesting human assistance. “Emergency stop” is necessary but not sufficient as a universal strategy, especially when abruptly removing support could drop a payload or destabilize the humanoid. Calibration drift is another major concern. Camera exposure changes, force sensors acquire bias, tactile materials wear, tools are replaced, and joint friction evolves. Monitoring the detector’s input statistics should be treated as part of the product. Otherwise, the anomaly monitor itself becomes an unmonitored learned component. A practical system should log the sensor window, contact-state estimate, modality contributions, model version, threshold, and chosen recovery action for every event. Those records become the most valuable source of future training examples. They also allow engineers to distinguish physical anomalies from synchronization bugs, sensor degradation, and annotation mistakes. How I Would Build on the VEDA Idea. For a research team adapting this concept, I would start with the anomaly ontology rather than the network. Define what counts as an event, when its onset occurs, and which response is expected. Separate task failure from safety violation and sensor fault. A part that is not fully inserted at the end of a cycle is a task failure; excessive force halfway through the cycle is an execution anomaly; a frozen wrench stream is a sensing anomaly. These categories may overlap, but they should not be silently collapsed. Next, annotate contact phases for successful executions. Manual labels are acceptable for an initial study, although later models can infer them. Contact-phase labels usually provide more reusable supervision than fault labels because every normal execution contains them. The model should then use modality-specific encoders, causal multiscale temporal attention, cross-modal attention, and an explicit contact token. I would train it jointly to predict near-future latent states, classify available known faults, and estimate contact phase. Modality dropout and timestamp jitter should be present from the beginning rather than added as post hoc robustness patches. The anomaly head should expose several components instead of only one opaque scalar: predictive error, cross-modal inconsistency, known-fault probability, and uncertainty. A supervisory layer can combine those signals differently for different tasks. Finally, evaluation should proceed in stages. Begin with offline event detection, then replay recorded streams in real time, then run shadow-mode monitoring without controller authority, and only afterward allow the detector to initiate conservative recovery actions. The last stage should measure not just detection quality but the rate at which intervention prevents damage without unnecessarily aborting successful tasks. The Bottom Line. The central idea behind VEDA is timely and technically well motivated: contact-rich humanoid manipulation cannot be monitored reliably by treating vision, force, touch, and robot state as independent snapshots. Normality is temporal. It is multimodal. Most importantly, it is conditional on the physical interaction regime. The promise of dual attention is that one mechanism can identify the consequential moments while another evaluates whether the modalities remain physically consistent. Contact awareness then prevents the model from treating every impact or force increase as a failure and allows it to recognize subtler anomalies such as premature touch, persistent load without progress, or visually plausible but mechanically impossible motion. The key questions for the full paper are therefore concrete. How is contact represented? What exactly are the two attention operations? Is inference causal? How are modalities synchronized? Does the evaluation measure event-level latency and false alarms? Do the improvements survive unseen objects, fault types, and sensor degradation? And does the system contribute something beyond a high-capacity classifier on a closed set of labeled failures?. If VEDA answers those questions convincingly, its contribution is larger than another anomaly-detection architecture. It points toward an execution monitor that understands manipulation as a changing physical conversation among perception, control, and contact—and notices when those participants stop agreeing.