Latest / Embodied AI 101 / R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation

Transcript

R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation · Embodied AI 101

• R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation. Modern robots must manipulate objects in the real world, which demands a true 3D understanding of the scene: knowing where things are in space, how they relate to each other, and how they move over time. Classical deep imitation or reinforcement policies typically rely on 2D images (RGB) or occasional depth sensors, which only give limited geometric cues. Some prior works augment vision with depth sensors or point clouds, but those solutions require extra hardware (and can fail on transparent or reflective surfaces). In contrast, recent large-scale 3D vision models – so-called foundation models – have shown an impressive ability to infer full 3D structure directly from images. For example, the Visual Geometry Grounded Transformer (VGGT) can predict camera parameters, depth maps, and dense point clouds from one or more images. But VGGT and its kin are computationally heavy, often taking hundreds of milliseconds or even seconds per frame. That latency is far too slow for closed-loop robot control, where you need new actions at tens of Hertz. The new R3DP framework (Real-time 3D-aware Policy) tackles exactly this challenge: build a manipulation policy that has access to those rich 3D features, yet still runs in real time. The key idea is an asynchronous fast–slow architecture that splits the work between a heavy 3D model and a light predictor. On occasional “key” frames, R3DP invokes a powerful pre-trained 3D backbone (VGGT) to extract detailed spatial features. During the intervening frames, a lightweight Temporal Feature Prediction Network (TFPNet) steps in, using the past 3D context to predict what VGGT would say for the current view. In this way the policy only pays the heavy cost of the 3D model some of the time, but still gets a (nearly) 3D‐aware representation every frame. R3DP also introduces a principled multi-view fusion module (MVFF) that explicitly uses camera geometry (intrinsics and extrinsics) to combine multiple camera views into a single spatial embedding. Everything is trained end-to-end with a diffusion-based imitation policy, so the final controller produces actions conditioned on rich 3D features, yet remains fast enough for live robotic control. In the experiments on simulated dual-arm tasks (the RoboTwin benchmark) and on a real robot, R3DP shows dramatic gains over vanilla 2D policies or naive 3D-augmented policies. For example, R3DP reached an average success rate of about 69 % on the ten RoboTwin tasks, versus only 36 % for a standard single-view diffusion policy and 18 % for a naive multi-view version. It even beat specialized 3D baselines that use full depth inputs: for instance, R3DP improved success by 11–41 percentage points over a depth-based diffusion policy (DP3) with ground-truth depth. Crucially, R3DP runs much faster than the naive approach of running VGGT every frame: by using the asynchronous fast-slow design, it cuts the vision encoding latency roughly in half. In simulation, R3DP processed multi-view images and fused features in about 40 ms per step (with less frequent VGGT queries) versus 73 ms if we ran VGGT on every frame. In sum, R3DP is a plug-and-play, real-time framework to inject powerful 3D priors into robotic policies – giving robots more grounded spatial awareness without sacrificing control speed. Asynchronous Fast–Slow Collaboration. At the heart of R3DP is the Asynchronous Fast–Slow Collaboration (AFSC) mechanism. Imagine two “brains” running in parallel: a slow 3D vision model (VGGT) and a fast feature predictor (TFPNet). The slow brain is very capable but costly – it can extract full 3D scene features from the current set of camera images, but only occasionally (every τ frames, say). The fast brain is lightweight – it cannot do dense 3D reconstruction from scratch, but it can take the previous 3D features and the current images and quickly predict what the 3D features should be for this frame. In practice, R3DP sets up VGGT to run on key frames (say every 4th or 8th timestep) and uses TFPNet on the frames in between. Formally, let Φ_slow be the VGGT module and Φ_fast be the TFPNet. On a key frame t₀, the system applies VGGT to get a feature map Fᵀˢ₃ᴰ = Φ_slow(I_{t₀}) (where I_{t₀} denotes the multi-view RGB input). We call Fᵀˢ₃ᴰ the Slow 3D-aware Features for that frame. For each subsequent frame t = t₀+1, t₀+2, …, t₀+τ−1, we run TFPNet to predict a feature map Fᴿ₃ᴰ_t = Φ_fast(Fᴿ₃ᴰ_{t−1}, I_t). Here Fᴿ₃ᴰ_{t−1} is the previous frame’s predicted features (or the initial Fᵀˢ₃ᴰ when t=t₀+1). The module does this sequentially: each predicted feature for frame t becomes the input for predicting frame t+1. In effect, VGGT gives a high-fidelity “snapshot” on the key frame, and then TFPNet fast-forwards the 3D representation through time by interpolation and learned dynamics. At the end of the cycle (frame t₀+τ), we again run VGGT to “reset” the features. This asynchronous design lets the policy be nearly 3D-aware on every frame, while only occasionally paying the cost of the heavy 3D model. The slow branch (VGGT) enriches the feature stream with precise spatial structure, and the fast branch (TFPNet) keeps that structure alive smoothly over intermediate frames. Concretely, in their experiments the authors achieved real-time control (up to ~20–25 Hz perceptual pipeline) by running VGGT on 1-of-4 or 1-of-8 frames and using TFPNet for the rest. As a result, the perception latency dropped by roughly 30–45% relative to running VGGT every frame, while retaining the strong 3D sensing – a crucial balance for real robots. Temporal Feature Prediction Network (TFPNet). The TFPNet module is the workhorse of the fast branch. It is a lightweight network trained to “hallucinate” the rich 3D features of the large model, using history. During policy execution, TFPNet consumes the previous frame’s fused 3D features together with the current RGB images, and outputs the current frame’s predicted 3D features. By leveraging the temporal continuity of manipulation (objects and camera motions change smoothly), TFPNet can often guess a very good approximation of what the heavy VGGT would have computed if it ran on this frame. Architecturally, TFPNet first processes each view’s image with a standard 2D encoder (they use DINOv2-S, a self-supervised vision transformer) to get per-view feature tokens. These tokens are passed through a few layers (the paper uses four Alternating-Attention transformer blocks) to fuse information across the different camera views. Finally, at the top of TFPNet the model performs a cross-attention step where the fused visual tokens attend to the historical 3D feature from the previous frame. In other words, TFPNet explicitly injects temporal context via cross-frame attention. The output of TFPNet is a feature tensor (one token embedding per view) that serves as the Real-time 3D-aware Features for the current frame. How do we train TFPNet? The authors pre-train it by distillation from VGGT. They sample short sequences of frames (random gaps of 1–8 timesteps between frames) from their manipulation data. For a given sequence of RGB images, they first run VGGT on the first frame to get its true 3D features Fᵀˢ₃ᴰ. Then TFPNet is fed that initial feature plus the next frames’ images to predict subsequent features Fᴿ₃ᴰ. Each predicted feature is compared to the ground truth VGGT feature for that frame (which is obtained offline by actually running VGGT on it). Concretely, they use a cosine-similarity loss between predicted and true features. This trains TFPNet to imitate VGGT’s outputs, leveraging temporal cues to do so. After this pre-training, TFPNet can be run much faster (with far fewer layers and smaller width than VGGT) during actual deployment. In summary, TFPNet acts as the fast temporal predictor in the slow–fast loop: it propagates the slow backbone’s understanding forward in time with minimal compute. In the experiments, R3DP’s authors show that the features predicted by TFPNet closely match the VGGT outputs. For example, decoded depth maps from TFPNet’s features nearly overlay the ground-truth VGGT depth, both in simulation and on a real robot. This confirms that a compact network can capture the large model’s 3D priors well enough to deliver robust policies. Multi-View Feature Fusion (MVFF). Most manipulation setups have multiple cameras (e.g. a head camera and a wrist camera). Simply concatenating features from all views wastes the strong geometry between them. R3DP introduces a Multi-View Feature Fuser (MVFF) to do this properly: it explicitly uses the camera intrinsics and extrinsics to merge information across views into a single consistent scene representation. In detail, for each view R3DP holds two streams of features: a 2D appearance feature (from an ordinary encoder) and the 3D-aware feature from VGGT/TFPNet that corresponded to that view. MVFF first fuses each view’s 2D and 3D features together via a cross-attention: the query is the 2D-token, and the key/value are the 3D-token for that view. This produces a hybrid 2D-3D embedding per view. Next, MVFF merges the views with a custom positional encoding called PRoPE (Projective Positional Encoding). PRoPE computes the relative camera projection between every pair of cameras (using intrinsics and extrinsics) as a 4×4 projective transformation. It then applies a block-diagonal transformation to the tokens so that attention respects these projective geometry constraints. In practice, the authors implement PRoPE by a type of RoPE-style attention that encodes the relative projection matrices into the attention computation. The fusion step can be written succinctly: let Fᵀᵒ featuring all view’s 2D-3D hybrid tokens. Then MVFF does. where K_t,E_t are the intrinsics and extrinsics at time t. The output F_fused is a single multi-view embedding that encodes one coherent 3D scene. This fused feature is then passed to the policy head (a diffusion model) as the visual condition. In practice, MVFF markedly improves performance over naive fusion. In ablations on RoboTwin, simply appending VGGT features (without MVFF) gave a big gain, but adding the geometry-informed fusion raised success even further. Intuitively, MVFF allows the policy to “see through” occlusions and reconcile viewpoints; for instance, what one camera is blocked by the robot’s arm, the other camera may see. By injecting actual camera calibration into the network, MVFF yields more accurate spatial reasoning than flat concatenation. Integration with a Diffusion Policy. R3DP wraps up 3D perception and fusion in a complete policy model. The underlying action policy is a Diffusion Policy, a recently introduced imitation-learning method that models an action sequence with a denoising diffusion model. In simple terms, a diffusion policy generates a plan of future actions by starting from noise and iteratively denoising while conditioning on observations. R3DP injects the fused 3D-aware embedding into this diffusion network as the condition. In effect, the policy head sees a temporal stack of the last few frames, each now enriched with geometric 3D features, and it produces the next sequence of control commands. Training of the diffusion policy is done end-to-end (with the 3D encoders frozen). The authors use trajectories of length 8 (spaced by 8 timesteps) to cover a long horizon (64 frames effectively). They freeze both VGGT and TFPNet parameters, so only the diffusion denoiser learns from data. This way, R3DP can leverage existing pretrained 3D models as fixed “features” and only train the lightweight parts. The losses are simply the usual diffusion losses (matching expert actions), and training is efficient in modern GPUs. Once trained, the system runs by alternating VGGT and TFPNet as described, fusing multiple views, and querying the diffusion model to sample actions (or take denoised actions directly). In summary, the R3DP architecture plugs in seamlessly to any 2D imitation recipe: you replace the vision encoder with the AFSC+MVFF module, but otherwise keep the same policy head. The result is a policy that is “3D-aware” in perception, but still operates in real time, making it a candidate for running on a real robotic system. Experimental Evaluation. Benchmark and Tasks. The paper evaluates R3DP on a suite of 10 representative manipulation tasks from RoboTwin (a dual-arm simulator). These include pick-and-place, stacking, and tool-use tasks. Notably, they add a Tube Insert task – inserting a transparent object – to stress test geometric perception. Each task comes with two camera views (head and front). The policies are trained via imitation on ~100 demonstrations per task. Baselines. Several baselines are compared. The main baseline is a standard Diffusion Policy using only RGB inputs (no 3D awareness). They examine it in two modes: single-view (head camera only) and naive multi-view (concatenating head+front features). They also compare to DP3, a diffusion policy that uses ground-truth depth (point cloud) as input, and a noisier variant DP3+DA2 which uses predicted depth from a learning model. Finally, they include a recent vision-language-action (VLA) method, dubbed π0, which is essentially a large multimodal model fine-tuned (with LoRA) for each task. Thus the baselines cover 2D policy, depth-based policy, and a generalist VLM. Simulation Results. Across the board, R3DP achieves much higher success rates. As one example: on the Block Hammer Beat task, the single-view diffusion policy essentially fails (0% success), and the multi-view diffusion is also 0%. R3DP jumps to 77% success. On Blocks Stack Easy, it goes from 6% (single-view) to 69% with R3DP. Even conservative tasks like placing a shoe or bottle see improvements of ~15–50 percentage points. Overall, R3DP attains about 69.0% average success, compared to 36.1% for the best 2D policy and just 17.6% for the naive multi-view policy. In other words, adding 3D-awareness raises success by roughly 33 percentage points on average. Compared to the depth-based baseline DP3, the picture is interesting. In these simulated tasks DP3 is not terrible (it had access to perfect depth), but R3DP still surpasses it significantly. On average, R3DP beats DP3 by about 11.4% (e.g. 69.0% vs ~57.6%). Moreover, when the authors add realistic noise (making DP3 use predicted depth), DP3+DA2’s performance collapses (e.g. from 89% on Container Place down to 37%). R3DP, which uses only RGB, is robust to those issues. Even a strong VLA model (π0, with a massive pretrained backbone) is outperformed: R3DP’s 69% vs π0’s 59.9% average. The authors suggest that π0 may struggle to fit task-specific details with limited demos. Inference Speed. R3DP’s key advantage is that it keeps latency low. Table 2 in the paper shows observation encoding time (RGB to fused feature) for various setups. A naïve policy that ran VGGT on every frame took ~73.1 ms per frame; adding an extra fusion step bumped it to 78.3 ms. In contrast, R3DP with τ=4 (VGGT every 4 frames) had 50.5 ms, and with τ=8 had 40.3 ms encoding latency. In practice they report about 40–50 ms/frame total on an RTX 4090 GPU, which is on par with or even slightly faster than depth-based methods. In the real-robot tests they measured ~62 ms to encode (compared to ~57 ms for DP3’s encoding), confirming that R3DP achieves almost real-time vision while giving much higher success. Ablations. The paper carefully ablates each R3DP component. Starting from the baseline multi-view policy (10.8% average success on a subset of geometry tasks), adding the frozen VGGT features alone (+ no temporal fusion) raises success to 39.8%. This shows that the 3D priors themselves are powerful: even without special handling, VGGT’s features make the policy far more spatially aware. Next, adding the MVFF fusion module boosts performance further (to 55.3% avg). Finally, including the full fast-slow (TFPNet temporal) pipeline yields the full 62.3% (on those tasks). These ablations confirm that each piece matters: temporal consistency and cross-view geometry each add distinct improvements. One more observation: the multi-view diffusion policy (without VGGT) actually performed worse than the single-view one on average. That surprising result highlights how naive view-concatenation can confuse a network. R3DP’s MVFF, by contrast, unlocks the benefit of multiple cameras. And thanks to its asynchronous design, R3DP can further trade off speed and accuracy by adjusting τ. For example, increasing τ from 4 to 8 (halving the frequency of VGGT) cut encoding time ~20% with only a small drop in success. This flexibility allows a user to tune for their hardware: lower τ for high-speed robots, higher τ if more compute is available. Real-World Robotic Tests. To validate transfer, the authors tested R3DP on a physical bimanual robot (ArmBot-Y1) with two fixed cameras. They chose representative tasks like placing a shoe or a glass cup – famously tricky for depth sensors when the objects are reflective. They also ran single-arm tasks (Pick Peach, Stack Bowls) in a cluttered, uneven-lit setting. Each policy was fine-tuned on 50–100 demos and then evaluated. The results were consistent: R3DP again far outperformed the baselines. On average across these tasks, the best non-R3DP method (a VGGT+MVFF ablation) scored 66.7% success, while R3DP hit 71.7%. Pure RGB-D policies lagged at ~30–50%. Perhaps more impressively, R3DP’s perceptual pipeline on a real 4090 GPU encoded an image in ~62 ms, nearly matching the 57 ms of the depth-based approach. In short, adding the fast-slow 3D perception gave real robots a strong, geometrically-informed policy almost “for free” in terms of speed. Discussion. R3DP demonstrates a powerful design pattern for embodied AI: use large, pretrained 3D models for their spatial priors, but only sparingly, and fill in intelligently. The asynchronous fast–slow approach is reminiscent of multi-rate sensor fusion or dual-rate control, but applied to perception: a slow-but-rich perception module pairs with a fast predictor. One can imagine generalizing this pattern: any heavy visual model (e.g. a large language model for vision-language reasoning, or a high-end depth estimator) could be slotted into the “slow” branch of such a framework. In R3DP’s case, VGGT was a convenient choice since it produces multi-view 3D features in one shot, but one could similarly use other 3D perception systems (NeRF backbones, occupancy networks, etc.) if they can run offline. The use of a diffusion policy is somewhat orthogonal; R3DP could in principle feed its features into value-based RL or other imitation heads just as well. The authors chose diffusion likely because of its flexibility in long-horizon planning from cameras, but the core idea of spatially grounding actions would work with any policy network. In practice, one would take their favorite policy “backbone” and replace the visual encoder with the R3DP perception module (VGGT+TFPNet+MVFF), exactly as done here. For practitioners, key takeaways are: (1) You can inject 3D geometry into your policy without new sensors by using off-the-shelf 3D models on images. (2) To keep it fast, don’t run the heavy model every step – instead, train a smaller network (like TFPNet) to predict its outputs between keyframes. (3) Always use your camera calibration when merging multiple views; simple concatenation underutilizes valuable information. One caveat is that R3DP still relies on powerful GPUs and somewhat complex pipelines. While 40 ms per step is real-time, it came from a high-end 4090 GPU. Less powerful hardware might require adjusting τ or using a smaller backbone (VGGT itself can be scaled down). Also, R3DP was built around pre-collected demonstration data; integrating it into pure online RL (where exploration happens) would be interesting future work. Nevertheless, R3DP opens an exciting avenue: it brings the promise of “foundation model” spatial reasoning into closed-loop control. As more 3D vision models appear (and more efficient ones are developed), the fast–slow strategy means robots can continuously benefit from these advances. We might soon see robots that consult a complex 3D world model intermittently, while skimming over frames in between, just like a human glances at a map every few steps instead of continuously re-triangulating their location. R3DP shows that in embodied manipulation, spatial grounding and real-time control need not be mutually exclusive – they can be achieved together with a bit of clever engineering. References: Yuhao Zhang et al., “R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation,” arXiv, Mar. 2026. (See also VGGT: Wang et al., “Visual Geometry Grounded Transformer,” CVPR 2025.).