Latest / Embodied AI 101 / When Robot Eyes Converge: Active Stereo for Humanoid Depth
Transcript
- When Robot Eyes Converge: Active Stereo for Humanoid Depth. Convergent binocular stereo: Depth perception for humanoid robot vision • Mingshi Chi and John K. Tsotsos • York University • Science Robotics • 2026. The eye movement that humanoid vision left behind. Most humanoid robots have two eyes, but they do not really have binocular vision in the biological sense. They have two cameras separated by a fixed baseline, normally facing straight ahead, feeding images into a conventional stereo pipeline. The arrangement borrows the appearance of a human head while omitting one of its most consequential capabilities: each eye can rotate independently so that both optical axes meet at the point being inspected. That movement is vergence. When the eyes rotate inward, they converge on a nearby target; when they rotate outward, they diverge toward a more distant one. Humans combine vergence with coordinated gaze shifts, head movement, accommodation, foveation, attention, and prior knowledge. Roboticists have replicated pieces of this oculomotor repertoire for decades, but production stereo systems have overwhelmingly standardized on rigid, parallel cameras. Mingshi Chi and John K. Tsotsos ask what happens if we take convergence seriously as part of a robot’s depth-sensing system rather than treating movable eyes as decorative or merely useful for gaze signaling. Their Science Robotics paper, published on August 26, 2026, presents convergent binocular stereo, or CBS: an algorithm and evaluation framework for computing depth directly from cameras that actively fixate different parts of a scene. The fuller research program is documented in Chi’s 2025 master’s thesis, which introduces a matcher capable of following oblique epipolar lines and a natural-image dataset containing parallel views, ground-truth depth, and multiple convergent fixations of the same scenes. The immediate contribution is a new stereo method. The more interesting contribution is conceptual. CBS treats sensor pose as a decision variable. Depth is not simply inferred from whichever images happen to arrive; the robot moves its cameras to create images that are better suited to its current perceptual objective. That places the work squarely in active perception. Instead of asking only, “What depth estimator should process these pixels?”, it asks, “How should the robot look in order to make depth easier to estimate?”. Why parallel stereo became the default. The dominance of parallel stereo is not an oversight. It is the result of a very effective engineering abstraction. With a calibrated parallel camera pair, corresponding points appear on the same image row after rectification. Stereo matching therefore becomes a one-dimensional search: for each pixel in the left image, search left or right along the corresponding row of the right image. The horizontal displacement is disparity. Given the focal length and camera baseline, disparity maps directly to metric depth. This geometry made classical block matching, dynamic programming, semi-global matching, cost volumes, and modern recurrent stereo networks tractable. Even when physical cameras are not perfectly parallel, software rectification creates a pair of virtual parallel views. The resulting interface is exceptionally convenient: feed in two rectified images, predict one scalar disparity per pixel, and convert it to depth. Modern learned stereo has inherited this structure. The models have become dramatically better at feature extraction, context aggregation, regularization, and iterative refinement, but most still assume that correspondence lies along a horizontal scanline. Surveys of machine-learning stereo methods show how deeply rectified geometry is embedded in network design, data conventions, losses, and benchmarks. Parallel cameras also simplify the rest of the robot. Their extrinsic calibration is static. Visual-inertial odometry can treat them as a rigid sensor body. Exposure can be synchronized once. Depth maps remain in a stable camera frame. There are no eye motors introducing backlash, encoder error, vibration, or time-varying transforms. This helps explain why contemporary humanoids commonly combine fixed RGB or stereo cameras with RGB-D sensors, LiDAR, or other ranging systems rather than building mechanically articulated eyes. A fixed sensor bar may look less human, but it is easier to calibrate, manufacture, maintain, and integrate into navigation and control. CBS therefore has to clear a higher bar than showing that convergence is possible. We already know that arbitrary calibrated cameras can be triangulated. The important questions are whether active convergence changes the correspondence problem in useful ways, whether those benefits survive real calibration errors, and whether they justify the additional mechanical and computational complexity. What convergence changes in the images. Imagine that a humanoid wants to inspect a cup on a table. With parallel eyes, the cup appears at different horizontal positions in the two images. The stereo matcher must search over a disparity range that includes the cup, the table, and potentially much more distant background structure. With convergent eyes, the robot rotates both cameras until their optical axes intersect near the cup. The fixation point then appears close to the optical center of both images. In an ideal symmetric system, it has approximately zero image displacement. Nearby points have small disparities relative to fixation, while objects in front of and behind it move in opposite directions. This does not magically increase the physical baseline or evade triangulation limits. At long range, a fixed-baseline stereo system still sees very small angular differences, and depth uncertainty still grows rapidly with distance. Indeed, the reported CBS results identify reduced long-range accuracy as a limitation. What convergence changes is the distribution and direction of correspondence. In the original, unrectified images, corresponding pixels are no longer generally separated by a purely horizontal displacement. Their displacement has horizontal and vertical components. The epipolar constraint still reduces the search to a line, but that line can be oblique, and its orientation can vary across the image. A conventional stereo network cannot simply interpret that pair as a rectified stereo input. One option is to rectify the images every time the eyes move. But rectification resamples both images, crops or pads their fields of view, and can introduce locally uneven resolution. It also deliberately removes vertical disparity and other pose-dependent structure. CBS instead operates in the cameras’ native image planes and searches along the epipolar geometry induced by the current eye poses. That distinction matters most when the scene itself makes a horizontal search ambiguous. Consider a shelf filled with identical boxes, a fence, window blinds, tiled walls, cables, or a tabletop with repeated stripes. If the repetitions run along the same direction as the parallel stereo scanline, many candidate matches look nearly identical. Rotating the eyes changes the orientation with which the epipolar line cuts through that pattern. A correspondence that was ambiguous along a horizontal row may become distinctive along an oblique line because the search now intersects different edges, colors, or surrounding structures. This is the central active-perception insight: changing camera orientation changes not only where an object appears, but also the ambiguity structure of the matching problem. Convergence brings several secondary advantages. A selected target is moved toward the center of both sensors, where lenses are often better behaved and where a foveated or nonuniform camera would provide its highest resolution. Near-field overlap can be directed toward the manipulation workspace instead of being determined permanently at design time. The robot can also combine visual correspondence with proprioceptive information from the eye joints: the measured vergence angle is itself a noisy range cue. But there are corresponding costs. The shared field of view changes with every fixation. Perspective distortion between the two views becomes more pronounced. Moving cameras require continuously updated extrinsics. Small errors in joint zero positions, axis alignment, synchronization, or baseline estimation can produce significant epipolar error. The robot gains a controllable visual geometry, but it must know that geometry accurately. Inside the convergent binocular stereo pipeline. The CBS method is best understood as a classical, geometry-first stereo pipeline redesigned for articulated cameras. The first input is not just an image pair. It is an image pair plus the calibrated state of the binocular head. The system needs each camera’s intrinsics and its current pose. From those poses, it can calculate the epipolar line in one image corresponding to a location in the other. In a parallel rig, this calculation degenerates into the familiar instruction to search the same row. In CBS, it generates a potentially diagonal path through the image. The public technical descriptions of the method identify three major image-processing ingredients: Gaussian pyramids, SIFT features, and Gabor filtering. Together, they support a coarse-to-fine correspondence process that can tolerate the scale, rotation, and local appearance changes caused by convergence. The Gaussian pyramid provides multiple image resolutions. At a coarse level, fine textures and small image displacements disappear, leaving larger structural relationships. Matches found there constrain the regions that must be searched at finer levels. This is especially helpful when the fixation shift or foreground-background depth range produces large displacements away from the point of convergence. SIFT features provide sparse, relatively distinctive anchors. Their local gradient-based descriptors are more tolerant of image rotation and moderate projective change than direct pixel comparison. In the CBS setting, these anchors can establish reliable correspondences despite the two cameras looking at the scene from inward-facing orientations. Gabor filters describe local image structure at selected orientations and spatial frequencies. They respond to oriented edges and texture, making them useful for distinguishing candidate matches along an epipolar line. A multiscale, orientation-sensitive representation is a natural fit for convergent images because the relevant structures may be rotated relative to the conventional horizontal stereo axis. Conceptually, the pipeline performs the following operations:. Use camera calibration and current eye pose to determine the valid epipolar search path. Build multiscale representations of both images. Establish robust feature matches that constrain the correspondence field. Compare local oriented structure along each epipolar line. Refine correspondence estimates from coarse to fine resolution. Retain both horizontal and vertical components of image displacement. Triangulate accepted correspondences using the current camera projection matrices. The result is a depth map derived without first forcing the images into a parallel virtual rig. This should not be interpreted as a claim that rectification is fundamentally wrong. Rectification remains an excellent engineering strategy, and it allows articulated cameras to reuse mature stereo software. The CBS argument is that direct operation on convergent geometry preserves information and can expose benefits that disappear when every eye pose is normalized back into the same parallel abstraction. The algorithm is also deliberately interpretable. When CBS succeeds on a repeated pattern, one can examine the epipolar direction, filter responses, and candidate matches. That makes it useful for studying why convergence helps. A large learned matcher might eventually achieve better accuracy and speed, but it could make the geometric mechanism harder to isolate. The current implementation is not yet a real-time perception module. Public reporting places its processing time at roughly 69 seconds, making it a research proof of concept rather than a component ready for a humanoid control loop. A benchmark organized around fixation. Evaluating an active camera system is surprisingly difficult. If the parallel and convergent systems view different scenes, from different baselines, with different exposure or ground truth, one cannot tell whether a performance change comes from camera geometry or incidental data differences. Chi’s thesis addresses this by creating what it describes as the first natural-image convergent stereo dataset. It contains ground-truth information for parallel views, parallel image pairs of each scene, and images obtained while fixating multiple locations across those scenes. This allows the effect of eye orientation to be examined while holding scene content as constant as possible. The “natural-image” qualifier is important. Convergent stereo datasets existed before this work. A notable 2017 Scientific Data benchmark rendered realistic virtual kitchen and office scenes using binocular poses that included vergence and cyclotorsion. It provided 5,400 fixations, exact camera poses, depth, horizontal and vertical disparity, occlusion masks, and depth edges over a peripersonal depth range of roughly half a meter to 2.2 meters. That synthetic benchmark offers exceptionally clean geometry, but real robotic images introduce the problems simulation tends to suppress: lens distortion, photometric mismatch, mechanical tolerances, calibration residuals, sensor noise, focus differences, and small timing errors. The new CBS benchmark therefore fills a different role. It asks whether the theoretical benefits of fixation remain visible on physical hardware. The reported result is not that convergence wins everywhere. CBS is competitive with established stereo methods overall and shows lower depth errors in certain challenging scenes, with repeated patterns highlighted as a particularly favorable case. It also preserves two-dimensional disparity information that can help characterize object orientation and surface structure in the original convergent views. This selective advantage is more scientifically interesting than a blanket improvement would be. It suggests that there is no universally optimal eye configuration. Parallel and convergent stereo produce different computational problems, and their relative value depends on scene structure, range, task, and calibration quality. A humanoid looking down a long corridor may prefer near-parallel eyes and a wide shared field of view. The same robot manipulating a patterned object at arm’s length may benefit from converging on the contact region. A fixed geometry forces both tasks through one sensing configuration. An articulated head can choose. What the results do—and do not—establish. The strongest conclusion is that camera morphology can be an algorithmic resource. Convergence is not merely a biomimetic flourish. It can alter epipolar orientation, center a task-relevant target in both images, and resolve correspondence ambiguities that remain difficult in a parallel configuration. The paper also challenges a common way of framing depth estimation. We often compare algorithms under a fixed sensor and treat the camera images as immutable observations. CBS demonstrates that some apparent inference problems can be reduced by taking a perceptual action. Instead of building an ever-larger network to disambiguate repeated texture, the robot can move its eyes and acquire a geometrically different observation. But the work does not show that convergent stereo should replace parallel stereo. First, convergence does not remove the long-range limits imposed by focal length, baseline, and pixel resolution. Second, its benefits depend on accurate dynamic calibration. A rigid stereo bar can be calibrated once and monitored occasionally. A binocular head must repeatedly infer each camera’s pose from joint encoders and a kinematic model, while accounting for backlash, compliance, thermal drift, and imperfect synchronization. Third, the current algorithm is far too slow for closed-loop humanoid behavior. A manipulation controller operating at tens or hundreds of hertz cannot wait about a minute for a new depth map. The components themselves—pyramids, filtering, feature extraction, and epipolar search—are parallelizable, but substantial engineering or a learned replacement would be required. Fourth, changing fixation introduces an action-selection problem. The system must decide where to look before convergence can help. If target selection is wrong, or if the task requires simultaneous awareness of several regions, a narrow fixation-centered depth strategy may be counterproductive. Finally, depth-map error is only an intermediate metric. The decisive evaluation for robotics will be whether active convergence improves grasp success, contact localization, foot placement, collision margins, pose estimation, or recovery from perceptual failure. The present work establishes the sensing mechanism and its geometric promise; it does not yet close that downstream loop. From early active heads to learned stereo. CBS revives a branch of robotics with a long history. Early active-vision researchers built elaborate binocular heads to study saccades, pursuit, vergence, vestibulo-ocular stabilization, and even torsion. Those platforms helped establish that moving the sensor can simplify perception, but their mechanical and calibration overhead limited widespread adoption. Humanoid platforms such as iCub later used vergent eyes for target localization and developmental robotics. Some systems explicitly calibrated changing eye geometry, while others learned mappings from image coordinates and joint encoders to three-dimensional object positions. A 2012 iCub study, for example, learned centimeter-scale object localization from binocular observations and robot state without requiring an explicit camera or kinematic calibration model. What distinguishes the new work is its focus on dense stereo correspondence under natural convergent imagery, together with a benchmark designed to compare fixation against parallel capture. The surrounding hardware program is also becoming more capable. The York University group’s DIJIT robotic head provides nine mechanical degrees of freedom plus optical adjustments and explicitly supports vergence, version, and cyclotorsion. It is intended as a platform for studying active observers rather than as a pair of cameras bolted rigidly to a neck. On the algorithm side, strong learned systems such as RAFT-Stereo and CREStereo represent the alternative trajectory: keep rectified geometry and invest in better features, recurrent refinement, and cross-domain generalization. CBS suggests a complementary research direction—condition the matcher on camera pose and let it operate over arbitrary epipolar geometry. General optical-flow and image-matching networks already predict two-dimensional correspondence. In principle, they can process convergent pairs and then pass their matches to calibrated triangulation. What they generally lack is explicit exploitation of a robot’s controllable binocular geometry. The next generation of CBS may therefore look less like a handcrafted stereo matcher and more like a pose-conditioned correspondence model embedded in a gaze policy. How I would use CBS in a real humanoid stack. The practical interpretation is not “replace the fixed stereo bar with moving eyes.” It is to build a hybrid visual system with parallel and convergent operating modes. During locomotion, mapping, or broad scene understanding, the robot would keep its cameras approximately parallel. That preserves a large shared field of view, stable extrinsics, compatibility with conventional stereo networks, and straightforward integration with visual-inertial state estimation. When the robot selects a manipulation target, detects low stereo confidence, or encounters repetitive structure, it would enter a fixation mode. The eyes would converge on the region of interest, capture synchronized images and encoder readings, and compute a higher-confidence local reconstruction. Once the ambiguity is resolved, the system could return to its wide-field configuration. For that architecture to work, eye state must be treated as first-class sensory data. Every image should be timestamped against motor encoders. The calibration model should include camera intrinsics, camera-to-eye transforms, joint-axis directions, encoder zero offsets, baseline, and uncertainty. An online correction stage could minimize observed epipolar residuals to compensate for slow drift. The system should also propagate calibration uncertainty into depth confidence. A beautiful dense map is dangerous if it conceals uncertainty caused by a one-degree eye-pose error. For manipulation, the output should be a depth estimate together with a covariance or confidence measure that reflects both image-matching ambiguity and kinematic uncertainty. There are three sensible implementation paths. The lowest-risk path is dynamic rectification. After every eye movement, calculate the current relative pose, rectify the pair, and run an existing stereo network. This will not capture every advantage claimed for direct CBS, but it provides a strong baseline and may already benefit from placing the target at the center of both native views. The second path is to optimize the proposed direct CBS pipeline. Epipolar searches can be batched on a GPU, Gabor responses precomputed, and coarse-to-fine candidate sets aggressively pruned. Sparse reliable matches could guide a faster local dense reconstruction rather than computing depth over the entire image. The most promising research path is a learned arbitrary-epipolar matcher. Instead of constructing a horizontal cost volume, the network would receive image features plus relative camera pose and sample candidate correspondences along the appropriate epipolar line. An attention mechanism could parameterize each line continuously, avoiding repeated full-image rectification. Training data would randomize vergence, version, torsion, baseline perturbations, lens distortion, and calibration noise. Crucially, the gaze policy and stereo estimator should eventually be trained or optimized together. A task-conditioned controller could predict which fixation will most reduce depth uncertainty. For every candidate eye movement, it could estimate:. how much of the target will remain visible to both cameras;. the expected epipolar direction through local texture;. whether the movement is likely to break a repeated-pattern ambiguity;. the sensitivity of triangulation at the expected range;. the risk of occlusion or motion blur;. and the mechanical time required to move and settle the eyes. The action would then be selected for information gain, not merely because the target is salient. Multiple fixations are another underexplored opportunity. A robot need not trust one convergent pair. It could fixate the left edge of an object, then the grasp point, then an occluded boundary, fusing all three reconstructions in the head or world frame. Because the eye movements are low inertia compared with whole-head or whole-body motion, the robot could actively vary its correspondence geometry without significantly disturbing its manipulation posture. For vision-language-action systems, CBS need not produce a globally dense scene representation. It may be most useful as a perceptual tool invoked by the policy: “inspect the handle,” “estimate clearance behind the object,” or “verify the edge before grasping.” The output could be a targeted metric-depth crop, surface orientation, and uncertainty estimate attached to an object token or manipulation affordance. That is a more compelling use than running convergent stereo continuously. Active sensing should be purposeful. The broader lesson for embodied AI. The headline is about stereo, but the deeper message is about embodiment. Many embodied-AI systems still treat perception as though a passive camera were querying the world from a fixed API. The body supplies locomotion and manipulation, while the sensor is expected to deliver a neutral stream of observations. Convergent binocular stereo breaks that separation. Moving the eyes changes the inference problem. The robot’s action determines its epipolar geometry; the geometry determines correspondence ambiguity; correspondence determines depth; and depth informs the next action. That loop is much closer to biological perception than simply mounting two cameras in a human-shaped face. CBS is not yet a replacement for rigid stereo, RGB-D, or LiDAR. Its current runtime, calibration burden, and long-range limitations make that clear. Its significance is that it turns vergence from an anthropomorphic curiosity into a measurable computational tool. In difficult near-field scenes—especially those containing repeated structure—the best depth algorithm may begin with a motor command. For humanoid robotics, the design implication is straightforward: if we are going to give robots eyes, we should consider letting them decide how to look.