Latest / Embodied AI 101 / Efficient Hybrid SE(3)-Equivariant Visuomotor Flow Policy via Spherical Harmonics for Robot Manipulation
Transcript
- Efficient Hybrid SE(3)-Equivariant Visuomotor Flow Policy via Spherical Harmonics for Robot Manipulation. Robotic manipulation often involves 6-DoF motions (3D position plus orientation) where the scene may be arbitrarily rotated or translated. An ideal visuomotor policy should automatically respect these symmetries: if the entire workspace is rotated, the robot’s action should rotate accordingly, not requiring new data for each orientation. In recent years, equivariant deep networks – architectures whose outputs transform predictably under rotations/translations – have been applied to robot learning to exploit these symmetries. For instance, Equivariant Diffusion Policy (CoRL 2024) injects SO(2)/SE(3) symmetry into a diffusion-based behavior cloning policy, yielding dramatic gains in sample efficiency. Likewise, the Spherical Diffusion Policy (ICML 2025) embeds states and actions into a spherical Fourier basis to produce a fully SE(3)-equivariant diffusion model. These methods validate that encoding 3D rotational invariances leads to much better generalization from limited demonstrations. However, equivariant policies have faced practical hurdles. Many equivariant networks are computationally heavy and often rely on a single type of sensor input (e.g. depth or point clouds only). Also, diffusion policies require iterative denoising steps to sample each action, which can be slow – on the order of seconds per move — making real-time deployment challenging. In the new E3Flow framework (CVPR 2026), Zhang et al. tackle these bottlenecks by combining flow matching – a fast alternative to diffusion – with a novel multimodal input pipeline, all built on an SE(3)-equivariant spherical representation. The result is a 6-DoF visuomotor policy that is not only symmetry-consistent but also orders of magnitude faster at inference, while still working with both RGB images and 3D point clouds. Equivariance in Visuomotor Policies. Let’s review why symmetry matters in manipulation. Consider a robot learning to pick up a mug. If all training demos are on a table at one pose, and then at test time the mug is rotated or the camera viewpoint changes, a plain neural policy might struggle. In contrast, an equivariant policy inherently “knows” that rotating the input (scene or object) should rotate the output action. Formally, an SE(3)-equivariant function (f) would satisfy (f(g \cdot x) = g \cdot f(x)) for any 3D rigid-body transform (g) on the input. By imposing this built-in consistency, equivariant policies tend to generalize much better to new object arrangements and viewpoints with minimal extra data, often outperforming even non-equivariant approaches by large margins (e.g. +22% success in one benchmark). Several recent works exemplify this. Equivariant Diffusion Policy showed that by making the diffusion denoising network equivariant, success rates jumped ~20% above a vanilla diffusion CNN policy. The alphabet-soup-named Diffusion Policy (DP) itself (IJRR 2025) is a baseline diffusion model for actions. And the Amazon Spherical Diffusion Policy (SDP) goes further: it embeds states and actions into a basis of spherical harmonics (“spherical Fourier representations”) so that each layer can natively rotate with the scene. SDP uses spherical FiLM layers to inject visual context equivariantly and a spatiotemporal U-Net to denoise full 6-DoF trajectories. The outcome is end-to-end SE(3) equivariance and a policy that generalizes robustly to new camera poses and object configurations, consistently outperforming previous methods on dozens of tasks. Despite these successes, diffusion-based methods remain slow at inference because they sample actions iteratively (often 50–100 network passes). By contrast, normalizing flows or flow matching can generate samples in one pass (or just a few), giving a big speed advantage. In computer vision/generative models, flow matching (aka “rectified flow”) has emerged as a way to directly learn a vector field whose ODE integrates straight paths between paired samples. Not only does this approach match diffusion’s performance, it can generate samples with far fewer steps. The trick for robotic policies is to adapt such flows into the equivariant setting and ensure stability. E3Flow is the first to do exactly that: replacing the expensive diffusion rollout with an efficient rectified flow, wrapped in an SE(3)-equivariant pipeline. The E3Flow Approach. E3Flow marries three core ideas: (1) an SE(3)-equivariant representation of the robot state and action using spherical harmonics; (2) a Feature Enhancement Module that fuses image and point-cloud features without breaking equivariance; and (3) a rectified flow matching objective to train the policy for fast sampling. We’ll unpack each piece. Spherical Harmonic Representation for SE(3) Equivariance. To guarantee equivariance, E3Flow represents 3D data in a spherical basis. In practice, the system constructs a discrete spherical embedding of the point-cloud and features on the unit sphere. Mathematically, any scalar function on the sphere can be expanded in spherical harmonics (Y_l^m(\theta,\phi)) (with degree (l) and order (m)) which form an orthonormal basis. Under a 3D rotation (R), these basis functions mix within each degree (l) according to the Wigner D-matrices, so that the transform of the coefficients is exactly equivariant to the scene’s rotation. E3Flow’s sensor input includes both a 3D point cloud (from e.g. a depth camera or LiDAR) and an RGB image (e.g. eye-in-hand camera). The raw point cloud is first encoded by an equivariant network (here an EquiformerV2 graph transformer, chosen for its native SE(3)-equivariance). This yields feature vectors at each point that live in a direct sum of features of different orders (scalar, vector, etc). The network then projects these 3D features onto a spherical grid (effectively evaluating series of spherical harmonics) to form an SE(3)-equivariant “scene tensor” on the sphere. Since all operations (point convolution, projection) commute with rotation, the result is a proper SO(3) spherical representation of the input scene. This spherical embedding now drives the policy head. Because both location and orientation are in play, the policy’s output (e.g. an SE(3) trajectory) must also equivary correctly. The E3Flow network learns a velocity field (v_\theta(x,t,s,v)) in continuous time (t parameterizes flow steps) whose output is also SE(3)-equivariant. In fact, under any world rotation (\rho), the predicted velocity obeys (v_\theta(\rho \cdot x_t, t, \rho \cdot s, \rho \cdot v) = \rho \cdot v_\theta(x_t,t,s,v)). This property is by design (the network uses equivariant convolutions) and implies that the entire learned flow model satisfies (f(\rho\cdot x)=\rho\cdot f(x)) for the policy (f). Importantly, the flow-matching training loss itself remains invariant under group transforms, ensuring the learned solution is truly equivariant. Hybrid Multimodal Fusion: The Physics of FEM. One weakness of prior equivariant policies is that they often use only a single vision modality. For example, Diffusion Policy often only used a point-cloud encoder, sometimes augmented by multiple static cameras. In practice, however, a mixed approach (RGB and depth/points) can yield richer cues. The challenge is to combine an image (which is a plain 2D signal not inherently equivariant) with an equivariant point-cloud representation without breaking symmetry. E3Flow solves this with the Feature Enhancement Module (FEM). As Figure 2 in the paper illustrates, FEM injects visual semantics from the camera images into the point-cloud features only in the invariant channel. Concretely, E3Flow first runs a standard CNN (ResNet) on the image to produce an invariant feature vector $f_\text{img}$ (scalar under rotations). Separately, it runs the Equiformer on the point cloud to get equivariant features $f_\text{pc}$, which split into a scalar (type-0) part $f_\text{pc}^{(0)}$ and higher-order parts $f_\text{pc}^{(>0)}$. Then FEM uses cross-modal attention or gating to update only the scalar part: [ f_\text{pc}^{(0)} ;\leftarrow; \text{Fuse}\bigl(f_\text{pc}^{(0)},, f_\text{img}\bigr). ] The key is that this fusion is invariant: mixing in the image changes only type-0 features, which are themselves invariant to rotation. Hence the overall spherical feature (which is the projection of all components) remains equivariant. In practice, FEM uses a learned gating $\Lambda$ to balance how much image information is inserted, and a learned projection operator $\Pi$ to merge the updated scalars with the original equivariant vectors: [ f_\text{fused} = \Pi\bigl(\Lambda,A(f_\text{pc}^{(0)}, f_\text{img}), ;f_\text{pc}^{(0)}\bigr) ;||; f_\text{pc}^{(>0)}. ] By carefully isolating the image into the invariant “channel,” E3Flow enriches the representation with color/texture cues while rigorously preserving the end-to-end SE(3) symmetry. This is a simple yet powerful insight: only the zero-order (scalar) spherical harmonics get updated by the image, leaving the higher-order ones to carry all rotationally variant spatial structure. Rectified Flow for Fast Captioning. Traditional diffusion policies train a sequence of denoisers (v_\theta(x_t,t)) and sample by repeatedly applying them from noise to target. This works, but is slow at inference (sampling might take dozens of passes). By contrast, flow matching recipes directly learn a time-indexed velocity field (v_\theta(x,t)) so that solving the ODE (dx/dt = v_\theta(x,t)) transports noise to data over a fixed interval. There is no iterative noise schedule; one can simply integrate the ODE (numerically, in a few steps) to get a sample. E3Flow uses a rectified flow formulation (as in recent generative-model work) tailored for equivariant policies. The core idea is to match straight-line displacements: each ground-truth trajectory (from state to demonstrated action) is treated as an instantaneous “flow,” and the network is trained so that [ v_\theta(x_t,t,\dots) \approx (a - x_0), ] i.e. the desired instantaneous velocity to reach the action $a$ from the initial state $x_0$. In essence, the network learns an ODE that goes as directly (along straight paths) between start and goal points as possible. Importantly, this objective requires no extra learnable parameters beyond the usual flow network; it simply fits the vector field to the demo trajectories. Because the network and loss respect symmetry, the rectified flow model is fully equivariant too. Zhang et al. show that if $v_\theta$ is implemented with an equivariant architecture (as above) then for any rotation $\rho$ of inputs we have [ v_\theta(\rho\cdot x_t, t, \rho\cdot s, \rho\cdot v) ;=; \rho\cdot v_\theta(x_t,t,s,v), ] and the squared-error loss $|v_\theta - (a-x_0)|^2$ is invariant under rotations. Thus by symmetry, the learned predictor $v_\theta$ yields an action distribution that transforms correctly when the scene rotates. In short, E3Flow’s training is end-to-end SE(3)-equivariant thanks to both network design and loss invariance. An added benefit of flow matching is speed. The E3Flow policy only needs to integrate its velocity field once per action decision, which can be done in as few as 5–10 RK4 steps. In practice, the authors found that about 10 integration steps was optimal: fewer steps gave unstable training (the flow could not steer accurately in one shot), whereas 10 steps reliably matched or exceeded the performance of diffusion. Crucially, this 10-step flow is much faster than a 50–100-step diffusion rollout. In the reported experiments, SDP required ~3–4 seconds per action (DDPM sampling), whereas E3Flow took only ~0.5s on the same GPU – a roughly 7× speedup!. (Even with 10 flow steps, the policy is real-time on moderate hardware.). Experiments and Results. Zhang et al. evaluate E3Flow on a suite of eight 6-DoF manipulation tasks drawn from the MimicGen benchmark (CoRL 2023). MimicGen is a large set of robot tasks (like stacking, threading, pick-and-place, etc.) with procedurally varied initial poses. For each task, they use only 100 demonstration trajectories (a limited-data regime) and test on novel scene configurations. The baselines include both non-equivariant and equivariant policies: (1) ACT, a transformer BC baseline; (2) Diffusion Policy (DP) with 50 denoising steps; (3) DP3, a variant with dynamic patching; (4) EquiDiff (CoRL 2024) using equivariant diffusion; (5) EquiDiff (voxel) similar but with voxel input; (6) SDP (DDPM) and SDP (DDiM) two versions of the Spherical Diffusion Policy (the latter with different sampler). There is also EquiBot (CoRL 2024), a CUTER system that supports SIM(3) scaling transformations but is weaker on rigid tasks. The results (Table 1 in the paper) show that almost all equivariant methods outperform non-equivariant ones, underscoring the power of symmetry in limited-data robot learning. E3Flow achieves the highest average success of all methods. For example, on CoffeeCleanup, SDP (DDPM) got 63% success, whereas E3Flow got 70%; on StackThree, SDP got 100% and E3Flow also 100%, but on tasks like NutAssembly or Square, E3Flow had the clear edge (e.g. 94% vs SDP’s 88–92%). Averaged over the 8 tasks, E3Flow’s success rate is about 3.12 percentage points higher than the state-of-the-art SDP. Equally important, E3Flow is far faster at inference. The paper reports roughly 7× speedup in action generation compared to SDP. In concrete terms, SDP took ~3.7 seconds per policy query (with DDPM sampling) while E3Flow took ~0.51s (10-step flow). That slashes the latency barrier for using equivariant policies on a robot. Crucially, E3Flow’s gains are not just statistical flukes: the authors conduct thorough ablations. They show that each component matters. If you remove the FEM and simply concatenate image and point features, performance drops substantially (e.g. from 79% down to 71% on average). If you revert to diffusion instead of flow, performance also dips slightly (77.6% vs 79.0% in one configuration) while running much slower. And indeed, the flow-step study shows that 1-step flows or naïve mean-field flows fail (often giving 30–50% success), whereas 5 or 10 step rectified flow recovers high success (e.g. on Coffee tasks from ~30% up to ~60% – see Table 5). In sum, the specific design (EquiformerV2 backbone + FEM + multi-step rectified flow) is validated as necessary for the final performance. Finally, E3Flow is tested on real robots to ensure it’s not just a simulator trick. They pick four real 6-DoF tasks: StoringToys, BottlePlace, StackBlocksAssembly, and another. In these experiments E3Flow achieved an average success of 76% (over 20 trials per task) compared to 62% for the SDP baseline. EquiDiff and vanilla diffusion performed much worse (~30% and 16%, respectively). These results confirm that the equivariant-flow approach scales to real hardware. The full table (Table B.1) shows E3Flow exceeding SDP on all four tasks (e.g. 95% vs 85% on stacking blocks). In summary, E3Flow outperforms prior SE(3)-equivariant visuomotor policies while running far faster. The authors describe this as trading only a modest (<5%) gain in success over SDP for a large reduction in inference cost – in effect making equivariant policies feasible for real-time control. Discussion. E3Flow is primarily an engineering advance: it combines existing ideas in a clean way to solve a practical bottleneck. The novelty is modest – spherical harmonics, EquiformerV2, attention-based fusion, and rectified flows were known – but this is the first time they are fused end-to-end for robotic policy learning. The crucial insight was the FEM design: “only inject image into type-0 features” to preserve equivariance, which is conceptually simple but effective. The experiments are thorough: they compare against all relevant baselines and vary each module. The biggest caveat is the incremental nature – the paper’s own analysis notes a ~3% improvement over SDP, which is arguably a small edge given the effort. However, in high-stakes robotics even a few percent can matter, and the speed gains are practically significant for control. The main limitation is also practical: 0.5s per step is still not super fast for highly dynamic tasks, and the tasks (MimicGen) are modest in diversity. More real-world tests and dynamic scenarios would be needed to fully validate deployment claims. Nevertheless, E3Flow highlights an important trend: equivariant generative policies are here to stay, and moving from diffusion to flows is a natural next step. Prior work showed that diffusion could be made equivariant; E3Flow shows we can get the same invariances with faster sampling. This opens the door to other hybrid architectures (e.g. combining flows with transformers, or even world-model prediction in equivariant space). The fusion of images also points to a broader lesson – navigating between invariance and multi-modal cues carefully to exploit the best of both worlds. For practitioners, E3Flow provides a recipe: if you want an SE(3)-aware policy but can’t afford diffusion’s latency, try a flow-matching loss on an equivariant latent space. EquiformerV2 or SE(3)-Transformer backbones, combined with image gating into the scalar channel, seem to work. The code is released by the authors (GitHub: zql-kk/E3Flow), so one can adapt it to other setups. In short, E3Flow demonstrates that symmetry + speed + vision fusion is achievable, moving equivariant RL closer to real robotic use. Sources: The above discussion is based on the arXiv preprint of Zhang et al.’s E3Flow (v1 submitted March 2026) and related works. Additional background on equivariant diffusion policies comes from EquiDiff (CoRL’24) and Spherical Diffusion Policy (ICML’25), which E3Flow directly compares against. All cited results and figures are from these sources.