Latest / Embodied AI 101 / PEAfowl: Helping Two Arms Agree on What They See

Transcript

PEAfowl: Helping Two Arms Agree on What They See · Embodied AI 101

• PEAfowl: Helping Two Arms Agree on What They See. PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulation • Qingyu Fan et al. (11 authors) • Institute of Automation, Chinese Academy of Sciences; Nanjing University • IEEE Robotics and Automation Letters • 2026. The problem is not simply controlling two arms. Imagine a robot sorting objects across a crowded tabletop. Its left wrist camera sees a cup clearly, but the right wrist camera sees mostly the gripper. The overhead camera observes the arrangement, while a frontal camera reveals a handle hidden from above. The instruction identifies one particular object, perhaps by color, position, or its relationship to another object. The action model faces two different questions: Where is the relevant physical structure, and which part of the scene matters for this instruction? More cameras provide more observations, but they do not automatically resolve either question. That distinction motivates PEAfowl. The authors target two weaknesses in common multi-view VLA designs: concatenating camera features without explicit geometric correspondence, and treating language primarily as global conditioning rather than using it to retrieve relevant visual evidence. Their proposed remedy combines geometry-guided cross-view fusion, instruction-driven visual readout, and training-only depth supervision from a pretrained teacher. The emphasis is on improving what reaches the controller, not inventing a new action-generation paradigm. This episode follows the author-accepted manuscript and extended appendix uploaded on August 24, 2026, rather than relying on the earlier January preprint. That revision includes implementation, deployment, robustness, and real-world evaluation details that materially affect how the results should be interpreted. For a roboticist, the interesting hypothesis is straightforward: perhaps some apparent failures of coordinated action generation are failures of representation. A controller cannot reliably coordinate around an object if different views provide incompatible evidence about its location. Likewise, a competent motion generator can execute the wrong behavior if the perceptual representation does not distinguish the instructed target from nearby alternatives. Consider the distinction between stacking colored blocks and arranging those same blocks by color. The images may contain almost identical objects. The instruction changes the desired relations, not necessarily the object inventory. A useful perception system should therefore preserve both physical correspondence and task-dependent relevance. My reading of PEAfowl is that these should be treated as separate design problems. Geometry should organize observations according to the physical world. Language should determine which observations deserve attention. The controller should receive both, rather than being asked to reconstruct both relationships from a large undifferentiated token sequence. Start with what PEAfowl does not replace. PEAfowl builds on SEM, or Spatial Enhanced Manipulation. SEM already introduces explicit spatial structure: it lifts visual information into a common three-dimensional coordinate system and represents the robot through joint-centric states. Consequently, the relevant comparison is not simply “geometry versus no geometry.” It is whether a more deliberate perception interface improves an already spatially informed policy. The released PEAfowl implementation retains a SEM-style robot-state encoder. Rather than compressing the entire robot configuration into one undifferentiated vector, the encoder processes structured joint representations and uses joint relationships in attention. This gives the policy a way to represent the robot as an articulated mechanism, rather than merely as a list of actuator values. That distinction matters particularly for bimanual systems. Two configurations that place an end effector near the same object may leave the rest of the arm in different locations. Those differences affect reachability, occlusion, and the space available to the other arm. An embodiment-aware representation does not solve those problems automatically, but it provides a more useful basis for learning them. Action generation remains diffusion-based. During training, the implementation corrupts demonstrated joint trajectories and learns to reconstruct the clean robot-state sequence. Its decoder conditions on visual, language, and robot-state features, and includes forward-kinematics-based consistency machinery. At inference, it iteratively refines an initially noisy trajectory rather than choosing each arm’s next command independently. This sits within the broader action-diffusion approach established by Diffusion Policy: generate a temporally extended action sequence, execute a portion, then replan from new observations. The attraction is not just multimodality; a trajectory-level representation can express relationships among actions across time that would be awkward to recover from independently predicted control steps. PEAfowl’s configuration uses one observation timestep and predicts 64 future steps. The inference processor can truncate that prediction to a shorter valid prefix; the deployment configuration exposes 32 valid steps. These are distinct quantities: observation history, prediction horizon, and execution prefix should not be conflated. The central architectural change comes before this decoder. One path constructs geometry-enhanced RGB-D features. Another extracts language-conditioned evidence from frozen CLIP features. They remain separate perceptual pathways whose outputs condition the same controller; this is not a single monolithic “3D CLIP” encoder. Geometry: make camera patches communicate in physical space. The geometry pathway begins with separate RGB and depth encoders. The released configuration uses a GroundingDINO-initialized Swin backbone for RGB and a narrow ResNet-34 depth backbone, followed by multi-scale feature projections. Depth is discretized into 128 bins spanning 0.01 to 1.2 meters. This is a workspace-specific geometric representation, not an unrestricted scene model. It helps to think about the geometry before discussing the neural modules. An image patch identifies a direction through the camera, but not a unique point in space. Depth selects a location along that direction. Camera calibration then allows that location to be expressed in a shared coordinate frame. This is the basic reason calibrated RGB-D observations can relate features from cameras whose image coordinates look completely different. The depth-distribution formulation avoids selecting just one depth bin at the outset. As in SEM’s spatial enhancer, candidate positions along the camera ray are embedded and combined using predicted depth weights. The resulting geometric information accompanies the visual feature, rather than replacing its semantic content. PEAfowl adds local attention between co-located RGB and depth tokens specifically in the depth-prediction branch. Its main RGB stream is preserved before cross-view aggregation. Predicted depth probabilities produce expected three-dimensional anchors; those anchors retrieve nearby tokens from other cameras. Distance-weighted RGB evidence is added through a learned residual gate, then fused with depth and geometric features. The configuration caps retrieval at 16 neighbors and sets the distance-softmax temperature to 0.08 meters. That temperature controls how strongly proximity differentiates the selected neighbors; it should not be mistaken for a hard correspondence radius. The intuition is easiest to see with a hypothetical handle. Suppose a frontal view distinguishes the handle from the cup body, while a wrist view gives a better local observation of the intended grasp region. If the corresponding patches are placed near one another in the shared frame, the wrist-view feature can receive evidence from the frontal view without requiring the action decoder to rediscover the relationship from appearance alone. This changes the burden placed on learning. The network still has to learn useful features and actions, but it no longer has to infer every cross-camera relationship from demonstrations. Calibration supplies part of the structure. There is also an important limitation. Retrieval uses expected anchors, not a full probabilistic correspondence calculation. A patch containing foreground and background surfaces can have an average depth between them. Moreover, the released implementation computes dense pairwise distances before selecting neighbors; sparse aggregation does not imply sparse search. I would therefore treat this module as a practical geometric inductive bias, not as a guarantee of correct correspondence. Its value lies in making useful relationships easier to learn. Its failure modes still deserve direct measurement: wrong geometry can make the model combine evidence that should have remained separate. Language: let the instruction retrieve evidence. PEAfowl’s language pathway belongs to the line of work exemplified by OTTER. OTTER argues that pretrained vision-language alignment should be used explicitly to select task-relevant visual features, rather than feeding independently extracted image and text representations into a downstream policy and hoping their alignment survives adaptation. Keeping the pretrained encoders frozen is part of that strategy. This is a useful distinction between possessing language features and actually using language to inspect an image. A model can receive an instruction while still devoting most of its representational capacity to irrelevant visual structure. PEAfowl uses frozen CLIP ViT-L/14 image and text encoders. Its adapter extracts patch features from the final vision attention layer, projects and normalizes those features, and retains token-level text representations rather than relying only on one pooled sentence vector. The attention-layer choice connects to ClearCLIP, which investigates why representations effective for global image-language matching can be less suitable for dense localization. The general lesson is that “use CLIP features” is underspecified: the feature extraction point matters when downstream computation needs local visual evidence. PEAfowl then initializes latent queries with text tokens. For each view, those latents repeatedly cross-attend to image patches and undergo latent processing. The implementation uses Perceiver-style blocks with gated residual updates, allowing the instruction representation to accumulate visual evidence over successive passes. The Perceiver connection is architectural rather than rhetorical. Perceiver separates a potentially large input from a smaller latent representation that reads it through attention. That creates a controllable intermediate representation instead of requiring every downstream layer to process the entire original input. Imagine an instruction referring to a small object beside a larger one. A one-pass similarity operation can highlight patches associated with individual words. Iterative readout provides an opportunity for the latent representation to incorporate what it has already found before querying again. That is a plausible mechanism for handling relational instructions, although it should not be interpreted as an explicit symbolic reasoning trace. The resulting latents are summarized by learned attention pooling. In the released adapter, pooled view-conditioned outputs are accompanied by a separate instruction summary. The controller therefore receives both visual evidence selected through language and a representation of the instruction itself. This separation suggests a useful diagnostic for other VLA systems: when an instruction changes, what changes upstream of action decoding? If the visual representation remains almost identical, the action head must perform most of the grounding work. A text-driven readout creates an earlier location where that change can occur—and where researchers can inspect whether it actually occurs. Frozen encoders do not make the whole system frozen. The readout modules remain trainable, and the geometry pathway is separate. The design attempts to preserve pretrained semantic resources while learning how the policy should access them. Depth distillation: improve supervision without running a teacher online. The third contribution concerns the gap between useful geometric representations and imperfect depth measurements. PEAfowl uses the Camera Depth Model from Manipulation as in Simulation. That work develops RGB-D models that refine raw camera depth into more accurate metric depth, using training data generated through a camera-noise modeling pipeline. Its premise is that depth sensing quality can be a major bottleneck even when the downstream manipulation policy is otherwise capable. PEAfowl uses this resource differently from an online depth-correction frontend. Its released preprocessing pipeline runs the teacher offline and writes refined depth into a separate database alongside the original RGB-D demonstrations. The raw observations remain available; teacher outputs provide training supervision. This distinction is central. The policy still receives raw depth at deployment. The teacher is not secretly required to clean every test frame, and the refinement pipeline is not part of each policy call. The real-robot configuration separates the depth used to construct supervision from the ordinary depth input consumed by the model. At the target-construction level, refined pixel depths are interpolated between neighboring bins and pooled into patch-level distributions. The depth loss uses soft-label binary cross-entropy weighted by valid-depth coverage, with insufficiently valid regions excluded. One reason I find this design attractive is that the teacher supervises a representation the controller already needs. It is not an unrelated auxiliary prediction task added in the hope that regularization will help. The depth head determines how visual features enter the geometric computation. The distinction also matters for deployment engineering. Offline supervision can be expensive without consuming the robot’s online latency budget. Conversely, removing the teacher at inference does not make the student immune to sensor failures. It only changes where the teacher’s information enters the system. There is an additional scientific caution. A pooled depth distribution can encode multiple surfaces inside a patch as well as uncertainty about a surface. Those are not the same thing. I would not interpret a broad distribution as a calibrated confidence estimate without evaluating that interpretation separately. The practical takeaway is more general than PEAfowl: when a high-quality perception model is too costly or inconvenient to deploy, ask whether it can supervise the policy’s internal representation rather than becoming a mandatory runtime dependency. Simulation: substantial gains within a defined task suite. PEAfowl evaluates nine RoboTwin 2.0 tasks with 50 demonstrations per task and 100 evaluation rollouts per task and setting. Clean and domain-randomized models are trained and evaluated in their corresponding settings. Thus, the randomized result is not simply a clean-trained policy tested under unexpected corruption. RoboTwin 2.0 itself is broader: its benchmark contains 50 bimanual tasks, and its domain randomization spans clutter, lighting, background, tabletop height, and language. PEAfowl’s experiments sample a particular subset of that space rather than establishing performance across the entire benchmark. Average success rates are:. | Policy | Clean | Domain-randomized | |---|---:|---:| | Diffusion Policy | 26.1% | 11.3% | | DP3 | 40.0% | 14.1% | | ACT | 34.2% | 7.8% | | π₀ | 22.0% | 22.1% | | RDT | 10.6% | 6.7% | | SEM | 51.0% | 24.1% | | PEAfowl | 69.6% | 47.1% |. Visuomotor baselines train per task; the VLAs train jointly. The headline comparison is therefore an 18.6-percentage-point advantage over SEM in Clean and a 23.0-point advantage under randomization. Those are substantial improvements, especially given the limited demonstration budget. They describe this evaluated suite and training protocol, however—not a universal ranking of robot foundation models. For interpretation, I would separate three questions. First, does the proposed system improve practical task completion in this setting? The reported comparison is compelling. Second, does the improvement isolate one architectural mechanism? Not by itself. Different policies bring different pretraining, representations, optimization behavior, and capacity. A benchmark comparison establishes system performance more directly than it establishes causality. Third, does a large relative improvement imply deployment reliability? Certainly not automatically. Relative gains and absolute reliability answer different questions. For a lab selecting its next representation experiment, the former may be enough. For an unattended robot, the latter is decisive. The task-specific versus multi-task distinction also deserves attention. A jointly trained model can share information across tasks, while a task-specific policy cannot. That is a legitimate practical comparison, but it should not be described as a controlled experiment that varies only visual geometry. A useful follow-up would preserve the controller, data mixture, optimization budget, and pretrained encoders while changing only the perception modules. PEAfowl’s ablations move in that direction, which makes them especially important. Ablations: evidence for the ingredients, not a complete causal decomposition. Under randomization, one-step similarity readout yields 36.1%; instruction summaries without visually grounded text yield 38.9%. A stripped baseline achieves 31.3%, rising to 35.8% with geometry fusion. Removing the front camera gives 35.0%; head-only gives 26.8%. The language comparison is particularly relevant to the paper’s argument. OTTER-style selection already provides a strong conceptual alternative to generic global conditioning: use semantic alignment to select evidence. PEAfowl asks whether a learned iterative readout is a better interface than a single similarity operation. I would interpret these ablations as evidence that the components matter together, rather than as permission to assign an independent percentage-point value to every module. Architectural components interact. A better depth representation may make cross-view retrieval useful; better language selection may make the controller exploit the resulting features differently. The project also presents qualitative feature and depth visualizations: cross-view features become more coherent after aggregation, and depth predictions become cleaner with distillation. These are useful checks on the intended mechanism, but they are not substitutes for controlled behavioral evaluation. Held-out two-block and two-bowl stacking reach 51% and 82% under randomization, versus SEM’s 14% and 65%. For me, these are tests of nearby skill reuse. They motivate asking whether a representation transfers across related task variants. I would reserve stronger claims about general-purpose task understanding for experiments that deliberately vary the skill family, object relationships, and required action structure—not merely the number of participating objects. That is not a criticism of narrow transfer tests. Narrow tests can be scientifically useful precisely because they isolate a manageable question. The mistake would be allowing the word “generalization” to erase what changed and what remained familiar. Real robots: promising transfer with substantial task-specific adaptation. Real-world models are simulation-initialized, then fine-tuned separately using 100 teleoperated demonstrations per task. Four tasks adapt simulation tasks; two are real-only training tasks. Trials are autonomous, with a two-minute deadline. The physical platform uses two AgileX Piper arms and four RealSense D435 cameras. Across six tasks with ten trials each, the reported totals are 41/60 successes for PEAfowl, 22/60 without depth distillation, and 7/60 for SEM. These results make depth supervision an especially interesting part of the system. There is a large behavioral difference between the full model and the version without the teacher-derived depth targets, while the teacher itself is absent from online inference. The engineering proposition is therefore different from simply buying better test-time perception through more computation. The adaptation protocol is essential to understanding the claim. This is evidence for simulation initialization followed by real-world learning, not evidence that a simulation-only policy can be installed on the hardware and immediately perform all evaluated tasks. That distinction changes how I would use the work. If my team already collects demonstrations for a calibrated dual-arm platform, this is a plausible representation and supervision strategy to test. If my requirement is zero-demonstration transfer to a new robot and new task family, these experiments do not resolve that requirement. Ten trials per task also make individual task percentages coarse: one outcome changes a task’s reported rate by ten percentage points. The aggregate comparison is informative, but I would want repeated training runs, more trials, and explicit uncertainty estimates before treating small differences as stable. More importantly, success counts alone do not reveal whether failures are recoverable. For deployment decisions, I would separately log missed grasps, wrong-object choices, inter-arm interference, calibration-related errors, and states requiring a reset. Those categories would help determine whether further work belongs in perception, control, or supervisory recovery. Robustness: additive noise is not the same as missing geometry. Separate robustness checkpoints start at 38.3% randomized success. Twenty-millimeter Gaussian depth noise gives 37.8%; severe calibration perturbation gives 25.0%; randomly removing 50% of valid depth gives 7.1%. Only PEAfowl receives these test corruptions. These are within-checkpoint sensitivity measurements, not additional entries to mix directly into the main leaderboard. The distinction among perturbations is more informative than a blanket claim of robustness. Consider three possible failures in a geometric system. Small errors can perturb otherwise useful measurements. Missing measurements remove evidence. Incorrect camera parameters can systematically place evidence in the wrong location. Those conditions should not be expected to produce interchangeable behavior. My engineering interpretation is that geometric fusion introduces an explicit dependency worth monitoring. It can help a controller relate complementary observations, but the same relationship becomes harmful when its geometric basis is wrong. A model that tolerates moderate pixelwise noise may still be vulnerable to persistent misregistration. This suggests a better testing practice than reporting a single “noisy depth” number. I would test value noise, missing regions, persistent calibration bias, and actual camera relocation separately. The last two should also be distinguished: incorrect metadata and a changed visual observation are different interventions. I would likewise avoid assuming that the best next step is simply more aggressive fusion. Sometimes the correct response to uncertain correspondence is to combine less evidence. Confidence-conditioned gates, distance rejection, or explicit “no reliable neighbor” options are reasonable hypotheses to test. Those are proposed extensions, not capabilities established by the current experiments. Deployment: the timing and configuration details matter. RTX 4090 profiling reports 325.1 milliseconds versus SEM’s 314.5, with 740 million total and 312.4 million trainable parameters. Peak memory is 3.04 GB, versus 0.95 GB. The distinction between total and trainable parameters is important. Frozen weights still occupy memory and may still require substantial computation. “Only the adapter trains” and “only the adapter runs” are very different statements. The deployment package separates a 25 Hz control loop from 1 Hz policy inference. It synchronizes camera observations and arm states, invokes the local policy, and streams the predicted joint targets. Reporting the controller frequency as the VLA’s visual reaction frequency would therefore be misleading. For a manipulation researcher, that raises a useful question: what happens between policy queries? Action chunks can produce smooth motion at a higher command rate, but they do not create new visual feedback. I would measure sensitivity to observation age and execution delay alongside conventional inference latency. The training configuration is also substantial: 400,000 updates, batch size 16, and AdamW optimization. Diffusion training uses a 1,000-step schedule; inference uses ten denoising steps. Limited demonstration counts should therefore not be confused with negligible training compute. The release is useful because it includes both a training/evaluation repository and a companion ROS 2 deployment workspace, with calibration and depth-preprocessing components. That provides more of the surrounding system than a model definition alone. Nevertheless, I would begin reproduction with configuration auditing, not a training launch. One concrete example: the public simulation configuration’s default data-path list enumerates both clean and randomized datasets, while the reported protocol describes split-specific training. Reproduction should explicitly select the intended paths rather than assume the unedited starter configuration encodes the desired experiment. I would then verify camera ordering, depth units, frame conventions, joint normalization, and the robot description. The processor exposes these dependencies directly through camera intrinsics, transforms, joint histories, and ordered image/depth inputs. A simple visual check can be more valuable than another training run: project robot geometry into every image after preprocessing. If links and observed arms disagree, the learned policy is being asked to compensate for a broken interface. Similarly, inspect teacher targets independently before attributing any improvement or failure to distillation. What this changes for bimanual manipulation research. PEAfowl’s most useful contribution is a specific answer to a recurring design question: how much structure should be imposed before action learning?. Its answer is not to replace learned control with an explicit geometric planner. Nor is it to delegate everything to a larger vision-language backbone. Instead, it introduces structure at the interfaces: organize camera evidence geometrically, read visual evidence through the instruction, and use a stronger offline perception model to supervise the representation that supports action. I would borrow that decomposition before borrowing every hyperparameter. For geometry, I would ask whether cross-view information is genuinely complementary and whether calibration is accurate enough to exploit it. For language, I would ask whether changing the instruction changes the evidence selected upstream of the controller. For supervision, I would ask whether an expensive teacher can improve a policy-relevant latent without becoming an online dependency. The next convincing experiments would broaden these questions. Can the same design tolerate camera replacement or reordered views? Does it help when objects remain occluded across multiple policy queries? Can it distinguish similar scenes requiring different bimanual relationships? Does improved geometry still matter when contact state, rather than visible pose, becomes the limiting uncertainty?. I would also prioritize parameter-matched perception ablations and more demanding transfer splits. Strong benchmark gains are a reason to investigate the mechanism, not a reason to stop investigating it. The bottom line for Embodied AI 101 is this:. Before asking a larger controller to coordinate two arms, ask whether its inputs agree about the physical scene—and whether those inputs emphasize what the instruction actually asks the robot to manipulate. PEAfowl makes that question concrete enough to implement, test, and improve.