Latest / Embodied AI 101 / HEAR: The Sound Your Robot Missed Between Actions
Transcript
- HEAR: The Sound Your Robot Missed Between Actions. Towards the Vision-Sound-Language-Action paradigm: The HEAR framework for sound-centric manipulation • Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu, and Hesheng Wang • Shanghai Jiao Tong University and University of Cambridge • The International Journal of Robotics Research • 2026. A competent robot with a missing observation. Imagine a robot waiting beside a microwave. It can identify the appliance, locate its handle, and generate a reliable opening trajectory. The instruction is straightforward: open the door after the completion sound. Now put the beep halfway through an action chunk. The robot is executing commands computed from an earlier observation. By the time it asks its policy what to do next, the beep has finished. Depending on the audio interface, that evidence may have disappeared from the observation entirely—or survived as a tiny fragment inside several seconds of background sound. The geometry is still available. The event that authorizes the next action is much less accessible. That is the problem HEAR makes concrete. Its proposed Vision-Sound-Language-Action, or VSLA, setting treats streaming sound as an observation that must remain useful across delayed decisions, rather than merely another input attached to a visual snapshot. Vision, audio, language, and proprioception all condition control, but they do not arrive with identical persistence properties. For this episode, we are following the September 18, 2026 arXiv revision alongside the journal publication, first published online on September 16, 2026. That distinction matters: the revised real-robot results include a calibrated experimental setup, rather than simply repeating the earlier preprint’s headline number. The framework’s four components divide the work into remembering acoustic evidence, interpreting multimodal context, learning acoustic progression, and generating action chunks. Their names give us the acronym: Historizer, Envisioner, Advancer, and Realizer. The accompanying contributions are OpenX-Sound for pretraining and HEAR-Bench for evaluation. The interesting question is not whether sound contains useful information. It is whether the complete robot system preserves that information until a decision can use it. Three clocks, and one blind interval. To understand the systems issue, separate three clocks: sensor acquisition, policy inference, and command execution. Action chunking lets a policy predict a sequence rather than one command at a time. Executing that sequence without repeatedly invoking the expensive model can support coherent motion. But observations arriving during execution cannot automatically revise commands already committed to the robot. HEAR calls this the Blind Execution Interval, or BEI. Consider a deliberately simplified example. A robot queries its policy at time zero and executes a two-second chunk. A beep occurs between 0.4 and 0.5 seconds. At the next query, the audio interface supplies only the most recent quarter-second of sound. Nothing in that input says the beep happened. Making the model better at recognizing beeps cannot solve this particular failure. The relevant samples never reach it. A buffer could fix the missing-data problem; an event latch could preserve a recognized trigger; a learned recurrent state could retain richer acoustic context. These are different implementations of the same requirement: evidence acquisition must not stop just because action selection is busy. The deployed timing is important. The authors report roughly 100-millisecond audio updates, 22 milliseconds for a Historizer update, and 199 milliseconds for a policy forward pass. Executing sixteen joint-position targets takes approximately 2.1 seconds, producing an effective gap around 2.3 seconds. That execution interval includes command-and-settle waits; it is not 2.3 seconds of neural inference. There is also a crucial qualification in the revised paper: its four-second audio window already exceeds the measured gap. Many cues therefore remain in the waveform but become poorly represented or temporally ambiguous, rather than literally disappearing. This separates two problems that are easy to conflate. One is coverage: were the samples retained? The other is usable state: can the policy identify what happened, when it happened, and whether it remains relevant?. It also separates memory from reaction speed. Preserving a cue does not make an executing chunk interruptible. For a task with a tight response deadline, I would independently measure event-to-action latency, not infer responsiveness from successful event retention. A system can remember perfectly and still react too late. Four components, four different responsibilities. Historizer: preserve the event, not just a longer recording. The Historizer is a streaming stateful transformer that maintains a compact causal audio memory. Instead of requiring the main model to repeatedly reconsider an ever-growing recording, it updates a persistent representation from incoming audio packets. That representation accompanies the next policy decision. The reported implementation uses sixteen memory tokens and four transformer layers, with width 256 and four attention heads. Conceptually, I find it useful to distinguish a recording from a task state. A recording answers, “What samples arrived?” A useful control state should support questions such as, “Has the required cue occurred since the previous action?” Those are not equivalent representations. For an alarm task, a single remembered event might be enough. For a process-monitoring task, the policy may need the direction of acoustic change. For a dialogue task, it may need to distinguish an earlier ring from a later end-of-call cue. This suggests a practical evaluation beyond aggregate success: probe how long different evidence remains recoverable from memory, and whether the model can distinguish an unconsumed event from an already handled one. Sixteen tokens specify representational capacity, not a guaranteed retention duration. Envisioner: make the acoustic state relevant to the task. The Envisioner combines visual observations, language, proprioception, the current audio window, and Historizer memory. Its hierarchy separates a high-level multimodal representation from a smaller control-oriented pathway, with cached context reused rather than recomputed independently for every downstream operation. The released implementation builds around Qwen3-Omni and a Qwen3 action-expert pathway. A structured stage-text prediction also supervises the high-level representation, making distinctions such as waiting versus reacting explicit during learning. Why care about that extra supervision? Imagine two observations with nearly identical images and joint positions. In one, the robot has not yet heard a completion cue. In the other, it has heard the cue and should begin moving. An action learner needs some representation of that difference; geometry alone cannot provide it. I would interpret stage supervision as an attempt to make the relevant hidden state easier to learn—not as evidence that verbalizing a stage is itself necessary for control. Its value should be judged against alternatives such as latent phase labels, recurrent state estimation, or explicit event-status supervision. Advancer: predict sound to shape the representation. The Advancer introduces a future-audio prediction objective. It predicts near-future discrete acoustic codes from the multimodal latent, encouraging that latent to describe how the acoustic situation is evolving. Crucially, this is a training-time branch: it is removed for deployment. HEAR does not need to synthesize future sound before selecting each action. The implementation includes an audio-code predictor and codec interfaces, including Mimi, rather than relying only on waveform reconstruction. This is a narrower use of “world model” than an online planner that rolls out candidate actions and chooses among imagined outcomes. Here, prediction supplies an auxiliary learning signal. Its intended contribution is a temporally informative representation. There is an important conceptual limit. A genuinely exogenous, randomized beep cannot become exactly predictable merely because the training objective asks for its future tokens. Prediction can learn process dynamics, event likelihoods, and recurring acoustic structure. It must not be mistaken for permission to act before an unpredictable condition actually occurs. For adaptation, I would therefore inspect whether prediction helps continuous monitoring and phase discrimination separately from transient-trigger recognition. Those capabilities need not improve together. Realizer: generate actions without treating hearing as passive. The Realizer uses conditional flow matching to generate action chunks. In plain language, flow matching learns how to transform a simple noise sample into a sample from the desired output distribution. For action learning, training examples pair noise with demonstrated chunks; the model learns directions along an interpolation between them. At inference, repeated updates transform an initial noisy chunk into a control sequence conditioned on the observation. The reported HEAR configuration uses eight Euler integration steps. The authors connect action quality to acoustic observability: unnecessarily abrupt motion can introduce mechanical noise that interferes with listening. The Realizer is therefore presented as part of the sensing solution, not merely the final actuator interface. That coupling is worth taking seriously, while keeping the mathematical claim precise. A continuous generative flow does not, by itself, guarantee bounded physical jerk or quiet execution. Those properties depend on the generated trajectories, controller, hardware, and training distribution. For a robot that must listen while manipulating, I would evaluate action generation using both mechanical and perceptual criteria: tracking error, acceleration, acoustic interference, and the reliability of the next observation. The best trajectory is not necessarily the one that reaches the next pose fastest. Training sound awareness without a large native-audio corpus. OpenX-Sound augments selected Open X-Embodiment trajectories with generated audio while retaining their original visual observations, instructions, proprioception, and actions. Its intended role is robotics-specific audio–visual–action alignment, rather than teaching an already multimodally pretrained backbone everything about sound from scratch. The supplement reports approximately 120,000 episodes spanning 100 skills. The released data preserve the existing RLDS/TFDS trajectory structure and add an episode-level audio field containing the generated waveform. That is a useful interface choice: an existing trajectory loader can retain its usual observation and action paths while an audio-aware loader additionally slices the synchronized recording. But generated sound is not recovered physical measurement. The authors acknowledge that video-to-audio synthesis can omit contacts, shift event timing, or introduce acoustically plausible events without the correct physical source. They position the dataset as pretraining support, not a replacement for task-specific real recordings. A simple thought experiment explains why that limitation is fundamental. Suppose two opaque bottles look identical, but only one contains loose objects. A silent video of both bottles sitting still cannot reveal which should rattle. A generator may create a convincing sound, but plausibility does not establish correspondence to the hidden state. That does not make synthetic pretraining useless. It changes what we should expect it to teach: useful cross-modal structure, manipulation-associated acoustic patterns, and temporal alignment—not necessarily the true hidden physics of every recorded episode. HEAR learns through offline imitation; demonstration generation uses rule-based controllers and motion planners with privileged task and cue information. Training spans 200,000 pretraining steps and 60,000 fine-tuning steps per task, with batch size 256 on six RTX 4090 GPUs. For a reproduction, I would audit synchronization before optimizing model capacity. The future waveform is a legitimate training target, but it must remain a target. Current observations, preprocessing windows, and recurrent updates must never accidentally include samples beyond their causal boundary. An apparently excellent predictor can otherwise become an equally excellent detector of dataset leakage. A benchmark where reaching the goal can be failure. HEAR-Bench makes the acoustic precondition part of success. Completing the geometric goal before the relevant sound occurs is a failure. Cue timings and initial visual states are randomized, and the reported simulation evaluation uses 100 trials per task. The task suite covers four kinds of evidence. Alarm Clock and Microwave test event-triggered behavior. Check Yes and Interrupt test speech, intonation, and timing. Pour Water and Boil Water test evolving process sounds. Check Materials tests interaction-generated evidence about otherwise ambiguous physical properties. This is a useful shift in experimental design. Consider a robot that always presses a visible alarm button immediately. A geometric metric could reward it. A sound-gated metric exposes the fact that it never waited for the event authorizing that action. Similarly, a task can require active sensing rather than passive classification. If the evidence becomes available only after an interaction, the policy must choose an action that makes the hidden state observable, then use the resulting observation. That is a more interesting control problem than classifying an audio clip independently of behavior. The simulation should nevertheless be understood at the right level. In the released Microwave task, a prerecorded is played after a randomized wait. This is event-conditioned audio playback, not a full physical model of appliance acoustics or room propagation. My reading is that such a benchmark can isolate timing and evidence-use failures very effectively. It cannot, by itself, establish robust acoustic perception across arbitrary rooms, microphones, materials, or sound-generation mechanisms. I would extend the protocol with paired counterfactuals: keep the visual trajectory similar while changing whether a cue occurs, move the cue relative to decision boundaries, and include episodes where the correct behavior is to keep waiting. Those tests distinguish genuine dependence on acoustic evidence from a policy that has learned a convenient waiting schedule. What the results actually support. On the seven-task simulation suite, HEAR reports 81% average success. The strongest waveform-as-image baseline, π0.5-Waveform, reaches 61%; π0.5-ASR reaches 35%. Play It by Ear and ManiWAV reach 28% and 26%, respectively, under this evaluation, while vision-only π0.5 reaches 14%. Those numbers establish an advantage over the evaluated systems on this sound-gated benchmark. They should not be generalized into a claim that all previous audio policies are weak, or that visual policies are broadly inadequate. The benchmark deliberately makes sound decision-critical. The most informative architectural comparison is internal. Removing the Historizer reduces success to 57%. Replacing it with GRU memory reaches 67%, and exponential-average/pooling memory reaches 62%. Replacing the flow-matching action head with regression gives 70%. Removing the Advancer gives 73%. The memory results make a stronger case than the headline comparison alone. They include simpler stateful alternatives, rather than comparing only a sophisticated complete system with a memoryless baseline. Still, I would avoid interpreting the gap as proof that this particular transformer is the only appropriate memory mechanism. A useful next comparison would match capacity, acoustic features, and timing coverage while varying the state representation. For discrete-trigger tasks, an event detector with a persistent latch would also be an instructive engineering baseline. For continuous processes, that baseline would likely need a richer state estimator. The pretraining controls are similarly valuable: synchronized audio produces 81%, versus 75% with a one-second shift, 73% with shuffled audio, and 72% with vision-only pretraining. That pattern supports the interpretation that correspondence matters beyond merely seeing additional trajectories. It does not tell us which acoustic details transfer, however. I would want downstream breakdowns by event duration, spectral complexity, and dependence on action-generated sound. One diagnostic also deserves caution. The supplement measures the fraction of steps with less than two millimeters of end-effector movement. Low movement is not inherently a control defect: during a listening phase, it may be exactly correct. I would therefore supplement that diagnostic with phase-conditioned motion statistics, explicit response latency, and acceleration or jerk measurements. The question is not simply whether the robot moves, but whether it remains still and begins moving at the appropriate times. The real robot exposes the microphone as part of the system. The Franka results are more revealing than a single aggregate suggests. HEAR reports 51% on Moka Coffee, 43% on Answer Phone, 88% on Shake Bottle, and 96% on Real Alarm Clock: approximately 70% overall, versus 45% for π0.5-Waveform. The real tasks involve weak process transitions, multi-stage dialogue, self-generated acoustic evidence, and waiting for an external trigger. The project demonstrations use coffee sputtering, call-state changes, bottle rattling, and alarm onset to expose these different requirements. The calibration story is essential. The earlier uniform setup yielded 54%. Moving the microphone closer and calibrating gain raised Moka Coffee from 18% to 51%; additional dialogue demonstrations raised Answer Phone from 15% to 43%. The revised comparison applies the changed settings to every baseline. So 54% and 70% are not interchangeable estimates of an unchanged deployment. The revised result includes improvements to acoustic capture and task-specific data. For an engineering team, that is not an embarrassing footnote. It is a reminder to define the system boundary correctly. If a camera cannot resolve a contact, we do not expect a policy architecture to manufacture the missing pixels. The corresponding discipline for audio is to inspect microphone placement, gain, clipping, reverberation, and interference before attributing every error to the model. The perturbation results reinforce this point. Stronger speech interference reduces the real-world average to 50%; a sixty-centimeter lateral microphone shift reduces it to 52%. I would read these results as encouraging but conditional. The system can exploit useful acoustic evidence under the tested conditions, yet performance remains sensitive to how that evidence is captured. Strong alarm performance should not obscure the substantially harder coffee and phone tasks. Nor should repeated rollouts be confused with broad generalization. For deployment planning, I would ask separately about variation across training seeds, rooms, microphones, objects, users, and recording sessions. A hundred trials within a task answers a different question from a hundred independently varied deployments. Where HEAR fits—and what it did not invent. Audio-driven manipulation did not begin with HEAR. Play It by Ear, introduced in 2022, explicitly addressed audio-visual imitation learning under occlusion. Its central premise already challenged the assumption that cameras provide all the task-relevant observations. That lineage matters when interpreting HEAR’s novelty: listening during manipulation is an established research direction, not a newly discovered capability. ManiWAV approached the problem through data collection and policy learning from in-the-wild human demonstrations. Its ear-in-hand collection interface captured synchronized visual and acoustic feedback. The complementary lesson is that the sensing apparatus and demonstration interface can be as important as the downstream policy architecture. If useful sound is absent from the collected data, a more capable multimodal backbone cannot retrospectively supply the missing measurements. Audio-VLA, released as a 2025 preprint, explicitly incorporates contact audio for contact events and dynamic-process feedback. It also augments simulation environments with collision-based audio. This is important context because previous audio-aware VLA work should not be characterized collectively as speech transcription or pre-execution prompting. Some work already targets physical feedback during interaction. There is also a direct neighbor to the Advancer: Fan Zhang and Michael Gienger’s Learning Robot Manipulation from Audio World Models, which investigates future-audio lookahead for manipulation. HEAR’s use of prediction is therefore best understood as part of a developing family of approaches, rather than a standalone claim that acoustic prediction has just become relevant to control. My assessment is that HEAR’s strongest contribution is the way it brings several requirements together around delayed, chunked execution. It asks not only whether a policy can interpret sound, but whether the evidence survives the execution schedule, whether the internal state tracks progression, and whether the evaluation rejects premature behavior. That is a narrower claim than “robots can now hear,” but a more useful one for designing systems. How I would adapt it in a research stack. The public release provides separate policy and benchmark codebases. The policy implementation builds on openpi and includes a Qwen3-based PyTorch training path, dataset conversion utilities, checkpoint handling, and policy serving. Its installation documentation also specifies repository-provided Transformers patches. I would treat those patches and model assets as part of the reproducibility configuration, not incidental setup details. HEAR-Bench, meanwhile, extends RoboTwin 2.0. Its documentation separates simulation collection from policy training and evaluation, with task audio assets included in the repository. The simulator and policy environments have their own dependency workflows. Before training the complete architecture, I would make three smaller experiments work. First, establish a timestamped replay test. Feed a known audio stream through the exact deployment preprocessing path, place short events at different offsets relative to policy queries, and verify that the next decision receives evidence of each event. Include delayed packets, long chunks, and episode resets. This is an interface test, not a model benchmark. Second, establish the causal data boundary. The released dataset stores full-episode audio, so the loader—not the storage format—must enforce which samples are visible at each decision. The dataset card also notes continuing uploads; reproducibility requires pinning and verifying the actual subsets used, rather than relying only on the dataset name. Third, verify the metric end to end. The released Microwave task’s local checks the door-opening angle. That function alone does not implement the published sound-causal criterion; any additional timing gate must be located and tested in the complete evaluation path. This is an audit requirement, not a claim that no such gate exists elsewhere. The benchmark’s base environment also makes recent-audio extraction configurable and exports episode audio alongside trajectory data. I would inspect those settings explicitly rather than assume that every collection or evaluation example uses the manuscript’s nominal window. Only then would I compare policy variants. Start with a native audio encoder and continuous buffering. Add simple persistent state. Add richer learned memory. Finally, test predictive supervision and action-generation changes. Otherwise, several improvements arrive simultaneously and it becomes difficult to identify which one solved the actual failure in your application. I would also log failures in operational terms: unavailable evidence, misinterpreted evidence, incorrect task phase, late response, and failed physical execution. Those categories lead to different remedies. A missed packet, a poorly positioned microphone, and a slipped grasp should not all become “the multimodal policy failed.”. Memory and faster control should be complementary. One closely related systems direction is Real-Time Execution of Action Chunking Flow Policies, by Kevin Black, Manuel Galliker, and Sergey Levine. Its real-time chunking method overlaps generation of the next chunk with execution of the current one, preserving committed actions while generating a compatible continuation. It addresses latency and continuity through execution scheduling rather than auditory memory. That suggests a natural combination—not a result demonstrated by HEAR itself. Persistent acoustic state could preserve what happened between decisions, while asynchronous generation or shorter committed horizons could reduce how long the robot must wait before responding. I would evaluate such a combination on two axes independently: evidence retention and reaction delay. Improving one does not guarantee improvement in the other. A faster policy can still ignore a brief event; a better memory can still feed an overly slow controller. The lasting lesson is a design question that applies beyond microphones:. If something important happens while the policy is busy, where does that information go?. A satisfactory answer needs more than a sensor connector. It needs a causal record, a representation of relevance and age, and an execution mechanism that can eventually consume the evidence. HEAR is useful because it turns that question into an architecture, a training setup, and a timing-sensitive evaluation problem. The next step is not simply adding more modalities. It is making sure the robot’s decision process does not forget the world while its previous decision is still running.