Latest / Embodied AI 101 / Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation

Transcript

Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation · Embodied AI 101

• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation. Modern robotic manipulation often relies on visual observations from multiple cameras to decide precise actions. A common approach is a view transformer, which takes a set of images (views) of the scene and predicts the robot’s next move. These models have shown impressive results but have two key weaknesses. First, they typically treat each camera view independently, merely fusing static features, which can miss the underlying 3D structure of the scene. Second, they don’t adapt well to sudden changes or motion: if an object moves while the robot is reaching toward it, a standard view-transformer policy can completely miss the change. In effect, most existing policies act like a person navigating with several still photos: they know the layout of the room if nothing moves, but they can’t react if something suddenly shifts. In “Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation” (Zhang et al., ICLR 2026), the authors take inspiration from the human visual cortex to address these issues. In human vision, two main pathways handle “what” (object recognition and spatial layout) and “where/how” (motion and action). Cortical Policy mimics this by introducing a dual-stream architecture. The static-view (ventral-like) stream focuses on building a robust 3D understanding of the scene, while the dynamic-view (dorsal-like) stream focuses on motion and attention cues, allowing the policy to react to changes. The system then fuses both streams to decide actions. Empirically, this approach significantly outperforms prior single-stream models on standard benchmarks (like RLBench) and especially on tasks where objects move or environments change (as tested by the new COLOSSEUM benchmark). In one real-robot test, a task where the target block moved in mid-reach went from 0% success under a static policy to 80% success with Cortical Policy. In the following sections we unpack this contribution. We’ll explain the architecture and training of the static and dynamic streams, how they are combined into a single policy, and the losses used to train them. We’ll then review the experiments on simulated and real robots, highlighting how Cortical Policy improves 3D spatial reasoning and real-time adaptability. Throughout, we compare to prior approaches like the NVIDIA RVT-2 system (which raised RLBench success from 65% to 82%) and discuss why Cortical Policy’s cortex-inspired design makes a difference. Background: View Transformers and the Static-Only Limit. To set the stage, recall that a view transformer for manipulation takes as input several camera images of a tabletop scene (often along with an instruction in language) and outputs a predicted robot motion. The images might come from different angles around the workspace. A transformer can fuse features across these views, theoretically allowing the policy to reason in 3D. Indeed, recent models like NVIDIA’s RVT and RVT-2 have shown very strong results by carefully designing such transformers. RVT-2, for example, used multi-view attention and system optimizations to achieve state-of-the-art on the RLBench suite of tasks. However, as Zhang et al. point out, existing view-transformer policies have bottlenecks. They typically compute static features from each view (often using a CNN or ViT encoder), then combine them in a transformer that treats them symmetrically. These static encodings capture “what is at each pixel” but ignore deeper 3D consistency. For example, a single 2D image cannot distinguish between two cups side-by-side vs. one cup on top of another without multi-view reasoning. In practice, this can lead to blurry or incorrect 3D spatial reasoning: the robot might not understand exactly where an object is in 3D space relative to others. The other issue is dynamics. If an object moves or the camera moves, a purely static representation becomes stale. The robot’s policy has no built-in mechanism to quickly attend to motion. If a block shifts position exactly as the robot is reaching, a static policy might ignore that motion and reach for where the block used to be. In human vision, this kind of situation is handled by the brain’s “dorsal” pathway, which processes motion and spatial relationships on the fly. Cortical Policy explicitly builds this idea into a transformer framework. In summary, the motivation is that multi-view “view transformers” need both better 3D consistency and a way to adapt to motion. Cortical Policy addresses both by adding a second, dynamic stream alongside the usual static one. Architecture: Dual-Stream Transformer. Cortical Policy’s architecture really consists of two parallel branches (streams) that process the input video differently, inspired by the brain’s ventral (static “what”) and dorsal (dynamic “where/how”) pathways. The outputs of the two streams are then fused into a final action prediction. We describe each stream in turn:. Static-View Stream (Ventral Pathway Analogy). The static-view stream is responsible for building a consistent 3D understanding of the scene from the multiple camera views. It starts from essentially the same kind of backbone used in previous view transformers (e.g. RVT-2): each camera image is encoded into features, and a transformer fuses information across the different camera views. However, on top of that standard backbone, Cortical Policy adds a significant 3D geometric supervision. In practice, Zhang et al. incorporate a pretrained 3D “foundation model” called VGGT (Visual Geometry Grounded Transformer). VGGT (from Meta AI, CVPR’25) is a powerful transformer trained to infer 3D scene attributes (depth, camera parameters, dense point tracks, etc.) from multiple views. The authors use VGGT to provide two things: depth/camera supervision for each view, and correspondences of keypoints across views. Specifically, VGGT can take a batch of images from the environment and predict (a) the depth map at each pixel, (b) the camera intrinsics/extrinsics, and (c) a set of 3D point tracks (keypoints) that match across different images. Cortical Policy uses these VGGT outputs during training. For each training scene, VGGT tells us which pixel in view 1 corresponds to which pixel in view 2 (because they are projections of the same 3D point). The Cortical model then encourages its own features to be aligned at those keypoint locations. In other words, if VGGT says “pixel A in the left camera and pixel B in the right camera are actually the same point in 3D”, then the static stream’s representation is trained so that the features at A and B are similar. This enforces cross-view geometric consistency. Concretely, the static stream includes:. Backbone: A multi-view transformer (adapted from RVT-2) that processes all camera images jointly. VGGT Heads: Auxiliary heads that predict (for each view) the depth, confidence (for VGGT’s keypoint predictions), and camera pose, following the pretrained VGGT. (These outputs can be compared to VGGT’s predictions as targets.). Keypoint Alignment Loss: A novel loss term that enforces consistency of features at 3D-aligned points. The authors define a Cyclic Geometric Consistency Loss ($L_{cgc}$): take a point in one view, follow its corresponding keypoint through a cycle of views (via VGGT matches) back to the original view; the representation at the end should match the one at the start. They use a differentiable “SmoothAP” loss (a rank-based metric) to softly enforce that the matching point always has the highest similarity. Intuitively, this makes the model learn features that identify the same physical point in 3D as it appears from different angles. This geometric consistency means the static stream learns true 3D-structure awareness. For example, two different cameras seeing the same cup will encode that cup’s pixels with similar latent features, rather than treating them independently. This significantly reduces the “3D ambiguity” that pure 2D view transformers suffer from. Dynamic-View Stream (Dorsal Pathway Analogy). While the static stream builds a stable 3D map of the scene, the dynamic stream focuses on motion and attention. This stream is fed a wrist-mounted (egocentric) video from the robot’s arm camera, streaming as the arm moves. So unlike the static images, which may be taken before action and are fixed inputs, the dynamic stream sees continuous video as the robot arm operates. Its goal is to alert the policy to moving objects or the robot’s own hand movements — much like the brain’s dorsal pathway signals where things are going and how to act on them. Key features of the dynamic stream:. Egocentric Gaze Model (GLC): The authors leverage a pretrained model from egocentric vision called GLC (Global-Local Correlation.), which was originally trained to predict human gaze directions in first-person (egocentric) videos. The idea is that gaze estimation models learn to identify what is important in a visual scene (often people’s hands, objects they manipulate, etc.). Cortical Policy repurposes this by synthetically retraining it for the robot’s end-effector: they build a dataset of images where the “target” (gaze) location is the robot’s hand position. In other words, they create paired frames and “ground truth” gaze points corresponding to the robotic hand. They fine-tune the egocentric gaze model so that it learns to output a heatmap centered on the end-effector. Once this egocentric gaze model is trained (the GLC network, which remains frozen for Cortical Policy’s training), it processes the wrist video frames and produces:. Feature Maps: High-level CNN features from the video frame. Saliency (Heat) Maps: A soft heatmap highlighting where the hand (or other salient motion) is. Adaptive Attention: The dynamic stream outputs are integrated to “steer” the policy’s attention. By using a gaze model pre-trained to lock onto the end-effector, the network automatically highlights moving objects that are near the hand or moving in the scene. For example, if the target block is jiggled or the arm moves near an object, the corresponding heatmap will underline that location. Importantly, the dynamic stream weights are usually frozen during policy training. That is, after pre-training the gaze model with the synthetic data, the Cortical Policy network does not update those weights; it just uses the features they produce. This “position-aware pretraining” means the policy can leverage complex motion cues without needing to train a video model from scratch. As the authors note, this mimics the human dorsal pathway: fast, reflexive interpretation of motion cues without heavy learning during task play. In summary, the dynamic stream is essentially a vision-language-action model for motion: it processes a continuous egocentric video through a pretrained gaze-capture backbone, and outputs a map of salient locations. These locations tend to correspond to moving objects and the robot’s hand. Fusing the Streams and Action Head. At the end of both streams, Cortical Policy fuses their information into action predictions. Concretely:. Static Stream Output: After the static transformer has processed all camera views, it produces per-pixel (or per-token) features for each view. These are typically turned into a top-down “heatmap” over the workspace as in standard view transformers: each heatmap score corresponds to an (x, y) location in the robot’s workspace. Dynamic Stream Output: From the wrist video, we have feature maps and saliency maps. The salient heatmap is projected into the world frame (since the wrist camera moves with the arm, this requires knowing the hand pose). This projection produces another heatmap on the workspace indicating where motion or attention is focused. These two heatmaps are fused (for example, by weighted sum or concatenation) into a combined heatmap. This fusion effectively says: “Here are the likely action points from static 3D awareness, and here are the likely action points from motion cues.” The highest-scoring point in the fused heatmap corresponds to the predicted 3D translation for the robot end-effector (within one action step). In practice, the model back-projects this point into 3D space to get the (x,y,z) translation. The policy also needs to decide the orientation (rotation) of the end-effector and possibly other signals (like whether to open/close the gripper, or which primitive to use). These are predicted by simpler heads on the fused features. In Cortical Policy, a “global” and “local” feature pooling from both streams is used to regress or classify the additional action components (rotation and discrete gripper actions). For example, the network might take the fused features, pool over all views for a global context, or focus on the local neighborhood of the chosen action point, and from those predict the angle or whether to grasp. Putting it all together, at inference time Cortical Policy:. Takes multiple fixed overhead camera images (static-view) and one or more frames from the wrist camera (dynamic-view) plus an instruction (language). Processes static images through the VGGT-supervised transformer, and the video through the pretrained gaze model. Combines features to produce a fused top-down heatmap and other action features. Picks the action (translation + rotation + open/close) with the highest score. In this way, the policy jointly reasons about the 3D layout (from static views) and any ongoing motion (from the dynamic stream) before issuing a command. Training Objectives and Pretraining. Training Cortical Policy involves several pieces:. Language Conditioning: Like other recent manipulation policies, Cortical Policy likely uses language instructions to specify tasks (e.g. “put the blue block in the box”). The architecture can attend to an instruction embedding in its transformer. (This detail is standard and not changed by this work, but it’s worth noting that the policy is multi-modal: it uses vision and language.). Static-Stream Supervision: There are multiple loss terms. The main policy loss comes from imitation or reinforcement signals on task success (usually cross-entropy or regression on the correct action). But additionally, for the static stream they add:. Depth/Camera Loss: The network predicts per-view depth (and possibly camera angles) which are compared to VGGT’s predictions, giving a learning signal to encode geometry correctly. Cyclic Geometric Consistency Loss ($L_{cgc}$): As described, this loss enforces that for any cycle of views, matching 3D points maintain consistent features. Technically, the model pools features at the VGGT keypoint locations and applies a ranking loss so that the “correct match” is most similar. This boosts multi-view 3D reasoning without actually reconstructing a full 3D voxel map (which would be expensive). Dynamic-Stream Pretraining: Before joint training, the authors create a synthetic dataset for the wrist camera. They generate frames of the robot arm and label each frame with the wrist’s 3D coordinate (projected to 2D) as the “gaze target”. They then train the GLC attention model to predict this target as a heatmap. After this pretraining, the dynamic stream is essentially fixed. During policy training, they still feed the wrist video, but the GLC network is not updated (only the final “action head” learns). Policy Loss: The final action head is trained end-to-end (static and dynamic streams) to predict the correct action for each training example. In robotic imitation or offline RL on datasets like RLBench, this means minimizing the error between the policy’s predicted translation/rotation and the ground truth expert action. A key design choice is that the dynamic stream is trained separately (frozen during policy learning) whereas the static stream is trained jointly. The authors found that this “position-aware pretraining” worked better than trying to learn the dynamic features jointly from scratch. It also ensures that the dynamic stream specializes in end-effector attention. With these components, the model is trained on a dataset of multi-view, language-conditioned manipulation demonstrations (for example, episodes from the RLBench environment). The geometric consistency and gaze objectives serve as additional regularizers that guide the network to use strong priors: 3D-aware static features and motion-aware dynamic features. Experimental Evaluation. Zhang et al. test Cortical Policy in several ways to validate its contributions:. RLBench Benchmark: RLBench is a standard simulated environment containing many tabletop manipulation tasks (e.g. stacking, opening jars, etc.) given a handful of camera views. Cortical Policy was evaluated on a set of 18 language-defined tasks from RLBench. The model achieved an 81.0% average success rate on these 18 tasks, notably higher than the previous best published result (RVT-2, at 77.5%). In fact, Cortical Policy was ranked top-1 or top-2 in 14 out of the 18 tasks, indicating broad strengths. Its gains were especially pronounced on multi-object spatial tasks – for instance, stacking cups or inserting objects precisely “between” others – as shown in the results. This suggests that the static stream’s geometric cues really help in complex 3D placement tasks. State-of-the-Art Baseline (RVT-2): For context, NVIDIA’s RVT-2 is a fair baseline. In their RSS 2024 paper, Goyal et al. report raising RLBench success from 65% to 82% by optimizing the architecture. Cortical Policy essentially matches that level of performance on those tasks, but with the key difference that its improvements come from improved spatial consistency and dynamics, not just brute-force scaling of the transformer. In other words, Cortical Policy is competitive with the very strongest vision-language policies on static tasks, while also adding robustness to dynamic changes. COLOSSEUM Generalization Benchmark: COLOSSEUM (2024) is a recent benchmark specifically for testing generalization under visual perturbations​​. It contains 20 manipulation tasks like RLBench, but with 14 types of environmental changes (object colors, lighting, distractors, camera pose, etc.). Cortical Policy was tested on COLOSSEUM in its “generalization mode” – trained on clean data, tested with perturbations. Here it shined: 69.9% average success under severe changes, compared to only ~60.5% for the RVT-2 baseline. That +9.4% gap underscores how Cortical Policy’s static stream generalizes better when appearances change (thanks to the VGGT-driven consistency) and how its dynamic stream helps cope with camera/viewpoint shifts. The authors’ ablation study on COLOSSEUM confirmed that the dynamic stream was the main driver of this robustness: removing it caused a large drop in performance. Real-World Robot Tests: To verify that these gains transfer to a real robot, the authors deployed Cortical Policy on a Franka Panda arm. They selected four tasks, some requiring precise spatial placement and others involving moving targets. Two representative findings:. Spatial Precision: In tasks that required placing an object between two others (for example “put the block between the red and blue blocks”), Cortical Policy achieved roughly 30% higher success than the best single-stream baselines. Without the cross-view consistency, the baselines often misjudged the exact “between” location in 3D. Dynamic Adaptation: In a task where the target block was suddenly moved during the robot’s reach, a static-view policy failed completely (0% success). Cortical Policy succeeded 80% of the time on these “target displacement” trials. The dynamic stream (motion cues) allowed the robot to notice and re-target the block on the fly. Ablation Studies: The authors ran several experiments to probe what makes Cortical Policy work:. Dropping the dynamic-view stream (or equivalently, ignoring its outputs) caused a significant performance drop, especially on tasks involving motion or visual distractors. Disabling the cyclic geometric consistency loss ($L_{cgc}$) hurt the static stream’s accuracy on spatial tasks, validating the 3D alignment idea. Training the gaze model jointly (end-to-end) rather than pretraining it resulted in worse motion-awareness, confirming the benefit of the “position-aware pretraining” approach. Interestingly, simply making a single stream deeper or wider (to match Cortical Policy’s compute) did not recover the performance – the dual-stream design gave better results per compute. In other words, splitting the problem into two specialized streams was more efficient than a monolithic model. Overall, these experiments show that Cortical Policy not only modestly improves static-view performance, but dramatically enhances generalization and reaction to dynamics. Its dual-stream design directly addresses the two problems (3D structure and adaptability) identified at the outset. Contributions and Significance. Cortical Policy’s main contributions can be summarized as follows:. Dual-Stream Architecture: It introduces a novel policy architecture with two parallel visual streams – a static (ventral-like) stream for spatial understanding and a dynamic (dorsal-like) stream for motion adaptation. This is the first work in robotic manipulation to explicitly mimic the cortical ventral/dorsal split in a learnable policy. Cross-View Geometric Consistency: The static stream uses a pretrained 3D vision model (VGGT) to supervise the features so that they align at geometrically consistent keypoints across views. This cyclic geometric consistency enforces true 3D reasoning without heavy 3D reconstruction. It solves the “2D ambiguity” problem of view transformers, giving a firm 3D spatial grounding to the policy’s decisions. Dynamic-View (Egocentric) Adaptation: The dynamic stream uses position-aware pretraining of an egocentric gaze model (GLC) to produce real-time saliency maps in the robot’s frame. This replicates the human dorsal pathway’s function: enabling the policy to react online to moving objects and the robot’s own hand. It thus solves the “static-only” limitation by continuously feeding motion cues into the decision pipeline. Strong Empirical Results: Cortical Policy significantly outperforms prior SOTA on benchmarks. On RLBench it outperforms RVT-2 (the prior SOTA) by several percentage points on average. More impressively, it is far more robust under visual perturbations: on the COLOSSEUM generalization suite, it achieves ~70% success vs ~60% for RVT-2. In real robot tasks, it solves problems that static policies cannot. These results validate that the dual-stream design is not just biologically plausible, but practically effective. Technically, Cortical Policy also demonstrates how to leverage “foundation models” in robotics. The use of VGGT (a large multi-view 3D model) and a pretrained Egocentric Gaze model shows a new way to fuse Big Vision models with policy learning. The system doesn’t do any expensive voxel-based 3D reconstruction – instead it transfers powerful geometric cues via feature alignment, making it scalable to many views and real-time use. Conclusion. In summary, “Cortical Policy” presents a compelling vision for bringing human-like visual processing into robot manipulation. By splitting perception into a static 3D-awareness path and a dynamic motion-attention path, it overcomes key limitations of earlier view-based policies. This work not only sets a new high-water mark for performance on both simulated and real-user benchmarks, but also suggests a general blueprint: build policies that see what the scene looks like in 3D and where things are moving, then combine those insights to act. As robotics continues to embrace large-scale pretraining, the cortical approach may become a model for future systems that must juggle rich geometry and rapid reaction under changing conditions. References: (See citations in the text above.).