Latest / Embodied AI 101 / Reading Trust in Motion: A Multimodal Perception Stack for Industrial HRC

Transcript

Reading Trust in Motion: A Multimodal Perception Stack for Industrial HRC · Embodied AI 101

• Reading Trust in Motion: A Multimodal Perception Stack for Industrial HRC. A Multi-Modal Model for Trust Evaluation in Industrial Human-Robot Collaboration • Giulio Campagna, Dimitrios Chrysostomou, Matthias Rehm • Smart Production Lab and Human–Robot Interaction Lab, Aalborg University • Intelligent Service Robotics • 2026. Trust as a state the robot might perceive. Trust in human–robot collaboration is usually measured after the interesting part is over. The participant completes a handover or assembly sequence, steps away from the robot, and fills out a questionnaire. That can tell us whether the robot was perceived as dependable, safe, or predictable in aggregate. It cannot tell the robot what is happening to trust while its arm is moving toward the worker’s body. For an adaptive robot, that distinction is fundamental. A post-task score is useful for system evaluation, but an online estimate could become part of the control loop. If the operator begins to hesitate, withdraw, or monitor the end effector unusually closely, the robot might slow down, increase separation, explain what it is about to do, or transfer authority back to the human. Conversely, if the interaction is stable and the operator appears appropriately confident, the robot might avoid unnecessary confirmations and preserve task fluency. This is the problem addressed by Giulio Campagna, Dimitrios Chrysostomou, and Matthias Rehm. Their model tries to classify high- and low-trust interaction states from two externally observable modalities: the operator’s face and the operator’s body motion. The motivation comes partly from the authors’ earlier systematic review of 100 HRI trust studies, which found that questionnaires remain dominant even though trust is dynamic, behaviorally expressed, and potentially accessible through machine learning and sensing. In industrial studies, robot-related factors—performance, reliability, failures, proximity, and behavior—were especially prominent. The paper’s headline numbers are strong: 88.40% classification accuracy and an area under the ROC curve of 0.97 for a beaker-handing task, followed by 85.30% accuracy and an AUC of 0.94 for a pouring task. More important than the raw scores, however, is what sits underneath them: a carefully controlled trust manipulation, participant-disjoint evaluation, frame-level multimodal fusion, and several methodological choices that sharply define what the system can—and cannot yet—claim to perceive. What the paper actually sets out to classify. The paper frames trust as a binary variable: high or low. That is a deliberate engineering choice. A robot controller may not need to respond differently to every small shift on a seven-point questionnaire. It may instead need a robust event signal indicating that the interaction has crossed a threshold requiring adaptation. The authors therefore pose two practical questions. First, can facial expressions and kinematics jointly distinguish the two trust conditions? Second, does combining them improve performance over models using only one modality? Their proposed solution is a feature-level fusion architecture: one network processes cropped facial images, another processes 92 body-motion features, and their learned representations are concatenated before binary classification. The authors call this late fusion, although in standard multimodal terminology it is closer to late feature fusion than to decision-level fusion, because the modality-specific embeddings are joined before the final prediction rather than combined as independent classifier outputs. The distinction between estimating trust and recognizing a trust-related condition will become important later. For now, the central hypothesis is straightforward: distrust is not expressed through one channel alone. A worker may lean away without showing a dramatic facial reaction, or may remain physically constrained by the task while displaying concern through a frown, brow movement, or rapid head turn. Combining modalities should make the estimate less dependent on any single behavioral response. The chemical-handling experiment. The experiment models a hazardous-material workflow. A Universal Robots UR10-CB3 manipulator equipped with an OnRobot RG6 gripper first hands a beaker to a participant. While the participant holds that container, the robot retrieves a second beaker and pours its contents into the first. The substances were harmless: baking powder and water, producing a carbon-dioxide reaction. Participants nevertheless wore lab coats, gloves, and protective glasses, and the containers were described as hazardous until the experiment had concluded. A research assistant remained ready to stop the robot with an emergency button. The purpose was not to expose anyone to real chemical risk, but to create credible uncertainty and vulnerability—the conditions under which trust becomes behaviorally consequential. The robot operated in two modes. In the high-performance condition, it followed smooth and predictable trajectories, maintained comfortable separation, positioned the beaker accurately, and poured without unexpected motion. In the low-performance condition, it moved erratically, came uncomfortably close to the operator, and created an apparent risk of spilling liquid onto the participant’s hand. These modes were inherited from the authors’ earlier work on proximity, risk, and trust. Questionnaire results using the Trust Perception Scale-HRI had validated the manipulation, and the current dataset again showed significantly higher trust scores under the high-performance condition. The resulting labels were assigned according to robot condition: samples from the high-performance trials were labeled high trust, while samples from the low-performance trials were labeled low trust. Twenty Aalborg University students participated: ten women and ten men, with a mean age of 29.1 years and limited previous robotics experience. Each completed four trials—two in each performance condition—with randomized ordering. This yielded 80 trials in total, each containing a Handing phase and a Pouring phase. The two phases were subsequently modeled as separate classification tasks. That separation is sensible because the behavioral semantics differ. During handover, the most obvious low-trust response is spatial: the robot approaches too closely and the person retreats. During pouring, the participant must continue holding the receiving beaker, so withdrawal may be constrained. Facial affect, head motion, upper-body tension, and hesitation can therefore contribute more information. Two sensors, two behavioral views. Body motion was recorded at 60 hertz using an Xsens MVN Awinda system with 17 wearable inertial trackers. After participant-specific measurements and an N-pose calibration, the system reconstructed 23 body segments. For each segment, the pipeline extracted linear velocity, angular velocity, linear acceleration, and angular acceleration. Each of those quantities began as a three-dimensional vector. The authors reduced every vector to its magnitude, producing four scalar values for each of the 23 segments, or 92 features per frame. These features were standardized with z-score normalization. The representation is compact and captures qualities such as movement intensity, abruptness, stillness, hesitation, and fluidity. It also discards direction. A retreat from the robot and an equally rapid movement toward it can have similar magnitudes. For this experiment, the network may recover useful distinctions through correlations across body segments, but a deployment-oriented representation would benefit from robot-centric directional features: distance to the end effector, torso orientation, signed radial velocity, hand–robot relative motion, and changes in protective workspace margins. Facial video came from an Azure Kinect camera positioned approximately one meter from the participant under controlled lighting. Faces were detected and aligned with the pretrained Multi-Task Cascaded Convolutional Network, cropped, resized to 200 by 200 pixels, and standardized. Online augmentation included flips and rotations of up to 40 degrees. The camera captured color at 30 hertz; its output was resampled to 60 hertz to align with the motion stream. Resampling makes the tensors align, but it does not create new facial information between camera frames. A practical implementation would need to state whether frames were duplicated or interpolated and quantify sensor timestamp error. That becomes especially important if the goal moves from classifying an entire condition to detecting a trust transition with subsecond latency. After synchronization and task segmentation, the Handing dataset contained 19,702 paired samples, while Pouring contained 15,835. These are frame-level observations rather than independent human interactions. The effective experimental sample remains 20 people and 80 trials—a distinction that matters when interpreting statistical certainty. Inside the fusion model. The face branch is a relatively modest CNN trained for the task rather than a large pretrained vision backbone. It uses three convolutional blocks with 32, 64, and 128 filters, all with three-by-three kernels and ReLU activations. The first two blocks use max pooling, while the third feeds into global average pooling. A 256-unit dense layer produces the facial representation, followed by 50% dropout. The authors compare three encoders for body motion. The dense-network variant transforms the 92 kinematic values through layers of 64, 32, and 16 ReLU units. The LSTM variant uses layers of 64 and 32 units followed by a 16-unit dense representation. The GRU variant uses recurrent layers of 128 and 64 units followed by a 32-unit dense layer. The selected face and motion embeddings are concatenated, passed through a 64-unit ReLU layer and another 50% dropout layer, and finally mapped to a high-trust probability through one sigmoid output. Training uses Adam with a learning rate of 0.001, weighted binary cross-entropy, batches of 32, and ten epochs. Random search with five-fold cross-validation selected the architecture and hyperparameters. Twelve participants formed the training set, four the validation set, and four the test set, keeping identities disjoint across the top-level split. That participant-based partition is an essential positive choice: a frame-random split would allow nearly identical images and motion patterns from the same people and trials to appear on both sides of the evaluation. There is, however, an unusual detail. The LSTM and GRU receive only a single timestep per sample. They do not observe a motion history. The authors explain that labels remain constant throughout each trial, so each prediction is generated from one synchronized face-and-motion observation. Under that formulation, the recurrent units are being used mainly as gated nonlinear feature transforms. They cannot exploit long-term dependencies, because no temporal sequence is supplied. Consequently, the LSTM’s victory should not be read as evidence that trust requires LSTM-style memory. It shows that this particular gated parameterization optimized better than the tested dense and GRU branches. A genuine temporal study would feed windows spanning several seconds, evaluate transition-detection latency, and compare the LSTM against temporal convolutions, state-space models, transformers, or even a compact hidden Markov model operating over frame-level embeddings. What the results show. Among the three multimodal configurations, CNN plus LSTM performs best on both tasks. For Handing, it reaches 88.40% accuracy, 0.97 AUC, 0.89 precision, 0.88 recall, and 0.88 F1. CNN plus DNN reaches 82.10% accuracy, while CNN plus GRU reaches 84.20%. For Pouring, CNN plus LSTM reaches 85.30% accuracy, 0.94 AUC, 0.88 precision, 0.85 recall, and 0.86 F1. The corresponding dense and GRU combinations remain near 82%. The LSTM configuration therefore provides a clear improvement within the family of fusion models evaluated in this paper. The comparison with the authors’ earlier unimodal systems is more nuanced. Their 2024 facial-expression model reached 78.61% on Handing and 73.35% on Pouring, so fusion improves substantially over facial imagery alone. Their earlier body-motion study, however, reported 96.30% accuracy for Handing using a GRU and 84.76% for Pouring using a one-dimensional CNN. The new multimodal model therefore does not beat the body-only Handing model in raw accuracy, although it reports a slightly higher AUC, 0.97 rather than 0.96. On Pouring, fusion produces a small accuracy improvement but more meaningful gains in AUC, recall, and F1. This means the strongest defensible result is not that multimodality universally dominates body motion. Rather, facial information consistently strengthens a weak facial baseline, gives the best overall Pouring results, and may provide complementary evidence when overt body withdrawal is unavailable or ambiguous. Manual behavioral annotation supports that interpretation. Positive facial expressions and engaged postures occurred more often in high-trust conditions, while frowns, frustration, backward leaning, and withdrawal were associated with low trust. All four broad behavioral categories showed statistically significant associations. Low-trust cues were particularly salient: people turned away, shifted their heads abruptly, leaned backward, paused, and made slower or more deliberate movements. High-trust trials were characterized by more open orientation, fluid action, forward engagement, and positive facial affect. The central caveat: trust labels are performance-condition labels. The paper is admirably explicit about its largest limitation. Labels are not moment-to-moment measurements of an individual’s internal trust. They are proxies derived from the robot’s performance condition. Every frame in a high-performance trial receives a high-trust label, even if the participant is briefly uncertain. Every frame in a low-performance trial receives a low-trust label, even if the participant remains confident or has already habituated to the behavior. The questionnaires establish that the manipulation changes average reported trust; they do not prove that every frame corresponds to the assigned latent state. This changes the interpretation of the reported accuracy. The model reliably recognizes human behavioral patterns elicited by two substantially different robot modes. Those patterns are plausibly associated with trust, and the annotation analysis reinforces that connection. But the classifier may also be detecting discomfort, surprise, perceived collision risk, task difficulty, negative affect, or direct biomechanical responses to proximity. Imagine the robot moves close enough that the participant leans backward. That movement is informative for safe interaction regardless of its psychological label. But it does not, by itself, identify whether the person distrusts the robot’s competence, fears physical contact, dislikes the trajectory, or is simply creating workspace for the beaker. A production system may still benefit from the signal, yet scientifically it is safer to call it a trust-related behavioral state than a direct trust measurement. A stronger follow-up would independently manipulate reliability, physical proximity, motion predictability, and task risk. Momentary self-report could be collected at sparse interruption points, while less disruptive labels could come from intervention decisions, willingness to delegate, trajectory preferences, gaze allocation, or changes in supervisory behavior. A hierarchical latent-state model could then distinguish stable individual propensity, trial-level trust, transient surprise, workload, and immediate safety response. Evaluation details that matter for roboticists. The participant-disjoint split is good practice, but the test set contains only four people. With that sample, results may depend strongly on which participants happen to be held out. Repeated group cross-validation, leave-one-participant-out evaluation, or bootstrap confidence intervals over participants would provide a more reliable estimate of generalization. The thousands of frames should not be treated as thousands of independent experimental observations. Adjacent frames from the same trial are highly correlated, and a model can achieve excellent frame-level performance by recognizing persistent posture, camera, or trial characteristics. Reporting trial-level predictions, participant-level balanced accuracy, and confidence intervals clustered by participant would clarify how often the system correctly assesses a genuinely new operator rather than a new frame. The unimodal comparison also is not fully controlled. The earlier body-motion study used a 70/30 participant split—14 people for training and six for testing—whereas the multimodal paper uses 12/4/4 training, validation, and testing. The architectures and tuning procedures also differ. The earlier 96.30% Handing result and the current 88.40% result therefore should not be interpreted as a direct ablation. A clean fusion study would retrain face-only, motion-only, and fused systems on exactly the same participant folds and preprocessing pipeline. There are also minor reproducibility inconsistencies. The camera stream is described as both 1920 by 1080 and 1980 by 1020 pixels. More visibly, the conclusion reports a Pouring AUC of 0.84, while the abstract, result table, ROC figure, and discussion all report 0.94; the latter is almost certainly the intended value. No public code repository is listed, although the authors state that code snippets, pseudocode, and implementation details are available on request. None of these issues erases the proof of concept. They define the next level of evidence required before treating the output as a calibrated online trust sensor. Where this sits in the trust-estimation landscape. Trust-aware robotics spans at least three layers: perception, belief estimation, and action selection. Classical dynamic approaches such as OPTIMo and Bayesian trust models treat trust as an evolving hidden state inferred from robot performance, failures, human interventions, and task outcomes. Trust-aware POMDP work goes further by letting the robot reason about how its actions alter that hidden state and team performance. These models provide temporal structure but frequently depend on assumptions or interaction variables that are difficult to observe continuously. The present paper occupies the perception layer: it asks whether raw human behavior can supply a useful observation model for such a state estimator. Physiological approaches offer another path. Systems have combined EEG, ECG, EMG, ocular activity, electrodermal activity, and related signals. The paper compares its Handing result with a physiological multimodal model reporting 87.5% accuracy, arguing that cameras and kinematic behavior may be easier to integrate than dense biometric instrumentation. New resources such as MultiPhysio-HRC now provide EEG, ECG, EDA, respiration, EMG, voice, and facial action units from industrial collaboration scenarios, although its baseline labels concern stress and cognitive load rather than trust itself. The authors’ own 2025 IEEE Robotics and Automation Letters paper offers a complementary direction. It uses preference-based optimization to generate candidate robot trajectories, obtains explicit human comparisons, and trains models on human- and robot-related behavioral indicators. A voting classifier reached 84.07% accuracy and 0.90 AUC while the accompanying explainability analysis examined which interaction features shaped predictions. Preference labels are more directly grounded in user choice, although selecting a trajectory is still not identical to measuring trust. Together, these works suggest a promising architecture: preference learning or validated interaction outcomes generate stronger supervision; multimodal behavior supplies continuous observations; a temporal belief model filters those observations; and a constrained controller adapts robot behavior. From laboratory classifier to deployable trust-aware robot. The first deployment change should be to preserve context. Human behavior alone is ambiguous. The estimator should also receive robot velocity, acceleration, distance to the operator, end-effector path uncertainty, task phase, recent failures, and safety-controller events. Leaning backward means something different when the robot is entering the shared workspace than when the operator is reaching for a tool. The second change should be temporal modeling in the genuine sense. Rather than classifying independent frames, the model could process two- to five-second windows and produce a filtered trust belief with uncertainty. Person-specific baselines would help distinguish an operator who is naturally still or reserved from one whose behavior has suddenly changed. Missing-modality training would let the system continue operating when a face is occluded or skeletal tracking fails. The third change should be deployment-friendly sensing. Seventeen Xsens trackers provide high-quality validation data, but they are not a frictionless production solution. The paper itself proposes replacing them with camera-based pose estimation and using multiple viewpoints to handle occlusion. A sensible transfer strategy would use the Xsens reconstruction as privileged supervision for a markerless skeletal model, then distill the trust estimator into RGB, depth, or privacy-preserving pose features. Most importantly, the objective must be appropriate trust, not maximum trust. If a robot is behaving unreliably, low trust is the correct response. The controller should fix the underlying behavior, reveal uncertainty, or reduce autonomy—not manipulate the worker into appearing more confident. High inferred trust must never justify weakening collision avoidance, force limits, or certified separation monitoring. A recent review of explainable and adaptive HRC describes trust calibration as a closed-loop system linking multimodal sensing, trust inference, decision-making, communication, and control. In a safe implementation, the millisecond-level safety loop remains independent and authoritative. Trust adaptation operates at a slower interaction level, where it can adjust speed, separation, confirmations, explanations, or task allocation within certified bounds. Evaluation should consequently move beyond classifier accuracy. Useful deployment metrics include transition-detection delay, false intervention rate, probability calibration, performance under occlusion, cross-task transfer, and the effect of adaptation on workload, safety margins, throughput, and reliance behavior. The hardest experiment is closed-loop: once the robot responds to inferred trust, it changes the behavior from which trust is inferred. The estimator and controller create a coupled dynamical system, with the possibility of oscillation or self-fulfilling predictions. Finally, workplace privacy cannot be an afterthought. Facial video and individualized behavioral profiles can become employee-surveillance infrastructure if governance is weak. A responsible design would process data locally, retain derived features rather than raw imagery where possible, expose uncertainty, permit opt-out and review, and prohibit the estimator’s use for disciplinary or productivity scoring. A cautious operator should not be penalized for displaying exactly the vigilance that a hazardous workplace requires. The bottom line. This paper does not yet deliver a universal, moment-to-moment trust meter. What it does demonstrate is still valuable: behavioral reactions to substantially different robot performance conditions are decodable across unseen participants, and body motion and facial information are complementary—especially when the task constrains overt physical withdrawal. The most important result is therefore not simply 88.40% accuracy. It is the emergence of a plausible perception module for trust-aware control, together with a clear view of the remaining gap. Crossing that gap will require temporally resolved labels, controlled ablations, participant-level uncertainty, robot and task context, deployment-friendly sensing, and closed-loop evaluation. For embodied AI researchers, the paper is best read as a strong sensor-fusion proof of concept and an invitation to connect human-state perception with belief estimation and bounded adaptive control. Trust is latent, contextual, and causal. Reading it from motion is possible—but only if we remain precise about what the model has actually learned.