Latest / Embodied AI 101 / ManiFlow: Manipulation via Rectified Flow
Transcript
- ManiFlow: Manipulation via Rectified Flow. Imagine a single deep neural network that can command a robot — a Franka arm or even a full humanoid — to do complicated dexterous tasks just by looking at the scene and hearing a simple instruction. That is the promise of ManiFlow, a new imitation-learning policy from NVIDIA/AI2 that uses cutting-edge generative modeling to produce robot actions. At its core, ManiFlow treats the problem of generating actions as a kind of flow matching task: it learns a continuous transformation (a “flow”) that maps noise into precise, high-dimensional robot commands. Importantly, ManiFlow introduces a consistency training objective so strong that the model only needs one or two forward passes to generate a complete action sequence, whereas previous diffusion- and flow-based policies required dozens or even hundreds of steps. In practice this means ManiFlow can execute very dexterous behaviors (single-arm, dual-arm, or even full humanoid tasks) in real time, with impressive robustness to new objects and environments. This episode takes a deep dive into how ManiFlow works and why it matters for robotic manipulation. We’ll walk through the main ideas behind flow matching and consistency training, unpack the novel DiT-X transformer architecture that ties together multi-modal inputs (vision, language, proprioception) and action outputs, and survey the benchmarks and tasks on which ManiFlow is tested. We’ll see how ManiFlow outperforms prior diffusion-based and flow-based policies — nearly doubling success rates on real-world tasks — and why it represents a significant step forward toward a more general, “LLM-like” manipulation policy. Finally, we’ll discuss how ManiFlow’s methods might be adapted or extended by other researchers working on embodied AI. Flow Matching and Consistency Training. At the highest level, ManiFlow is an imitation learning policy: it learns from demonstrations how to map observed inputs (like images, language commands, and robot proprioceptive state) to desired robot actions (joint positions, velocities, etc.). What sets it apart is the training framework. Instead of using a simple supervised network, ManiFlow uses ideas from generative modeling — specifically, flow matching and diffusion — to learn a full distribution over possible actions. You can think of this as learning a conditional probability $p(\text{action} \mid \text{observation})$, where the model can sample a realistic action trajectory. Early works in generative policy learning used diffusion models: for example, Diffusion Policy (Dong et al., CoRL 2022) shows that you can add noise to expert actions and then train a denoising network to sample from the action distribution. The downside of diffusion-based policies is that sampling requires many iterative steps: you start with pure noise and repeatedly “denoise” it through the network for tens of iterations before arriving at a final action. This is computationally expensive and often too slow for real-time control. Flow matching is an alternative generative modeling approach. Instead of discrete denoising steps, flow models learn a continuous vector field or ODE that transports noise into data samples. Concretely, imagine you have an expert action $a^$ and you add noise to it, then gradually transform it into pure Gaussian noise over time. A learned velocity field $v(x, t)$ tells you, at any point $x$ on this transformation, which direction to move in to keep heading back to the clean action. Training often samples an intermediate point $x_t = a^ + \epsilon$ on the path and trains the network to predict the corresponding velocity to recover $a^*$. Over time, integrating this velocity field from $t=0$ (noise) to $t=1$ (data) will trace out a path that maps noise to actual action. Generating an action with a flow model still normally requires integration: go from noise to action by following the vector field step by step (e.g. with an ODE solver). In practice, this is often only a handful of steps — far fewer than diffusion — but can still require something like 5–20 steps to get high fidelity. ManiFlow makes two key improvements to this picture:. Consistency Flow Training: Instead of only training the model to predict local velocities at single time points, ManiFlow enforces a global “self-consistency” across the entire flow trajectory. In practice, the ManiFlow training samples multiple points along the line (or flow path) between an action and its noise. The network is trained so that any intermediate point on this trajectory predicts the same original action. In other words, no matter where you stop along the noise->action path, the model should know how to reach the final action in one shot. Additionally, ManiFlow enforces that different points originating from the same action must agree (be consistent) in their predictions. Formally, if $x(t)$ interpolates between noisy input and the true action, the model $f_\theta$ is trained so that [ f_\theta(x(t), \text{obs}) \approx a^* ] for all $t$ and $x(t)$ on the trajectory, and these predictions must “self-consistently” map back to the same $a^*$. The practical effect is powerful: once trained with this consistency objective, ManiFlow can generate a complete action sequence in one or two forward passes of the network. Empirically, ManiFlow’s authors report that their model produces high-quality actions in just 1–2 inference steps. This is a drastic speed-up over prior methods: for example, typical diffusion policies needed on the order of 10–100 denoising steps, and standard flow models might require a dozen integration steps. By contrast, ManiFlow’s consistency training closes the “train vs inference” gap so tightly that a single network evaluation yields essentially the final action. This idea of consistency is analogous to consistency models in the diffusion literature (Song & Ermon, 2023), where a model is trained to be invariant when applied multiple times. But ManiFlow adapts this to flow fields: it learns a vector field that, effectively, collapses into direct prediction. The project’s documentation puts it clearly: “During training, we learn to map any intermediate point on the flow trajectory back to its origin…and ensure the self-consistency of sampled points on the same trajectory. With consistency flow training, ManiFlow can generate high-quality action sequences in just 1-2 inference steps”. Rectified Flows: Relatedly, ManiFlow leverages ideas from rectified flows (not to be confused with rectifier neural nets). Rectified flow methods involve learning flows that are easier to train and more stable to integrate. While ManiFlow’s main novelty is the consistency objective, the underlying flow-matching framework inherits benefits from rectified flows like uniform temporal supervision. For example, ManiFlow may use auxiliary losses to guide the velocity field at multiple time points (similar in spirit to what Riad Ahmed et al. later call “rectified flow velocity regression” in Trajectory-Consistent Flow Matching (2026)). By combining such techniques, ManiFlow’s training aligns the flow so well that the model can essentially “reset the clock” and produce the target action from anywhere on the path. In summary, ManiFlow’s core learning scheme can be thought of as: train a flow model, but add consistency constraints so strong that it no longer needs gradual integration at inference. The result is a policy that is as fast to query as a traditional reactive network, but still benefits from the flexibility and expressiveness of flow/diffusion models. The DiT-X Architecture: A Multimodal Transformer. Training a powerful flow model is only one part of the design. Robot manipulation is inherently a multimodal problem: the policy must fuse visual input, language instructions, and proprioceptive state before outputting actions. ManiFlow’s authors address this by building on the recent “Diffusion Transformer” (DiT) architectures, but with important modifications. They dub their model DiT-X (Diffusion Transformer – eXtended) to highlight these changes. At a high level, DiT-X is a single transformer model that ingests a mix of data:. Visual input: Either RGB images or point cloud representations of the scene (e.g. camera images or LiDAR/Depth). Language commands: A high-level instruction or task description (e.g. “pick up the blue cup”). Proprioceptive state: The robot’s joint angles or other low-dimensional state information. Action tokens: A set of tokens representing the output (the robot’s action trajectory). The magic is in how these are woven together. ManiFlow breaks the output action sequence into a set of learnable “action tokens”, one for each relevant degree of freedom or action dimension. The DiT-X transformer layers then allow cross-attention between action tokens and observation tokens, so that every action token can attend to all parts of the input, and vice versa. But two innovations make this particularly effective:. Adaptive Cross-Attention: In a vanilla transformer, cross-attention would simply compute dot-products between query and key vectors from different modalities. ManiFlow goes further by learning scale and shift parameters for these attentions. In other words, the model can modulate how strongly each action token attends to each visual or language token by dynamically scaling the attention weights. This adaptation is learned end-to-end, enabling very fine-grained coupling. Practically, this means if the robot sees a glass and hears “grasp it”, the attention mechanism learns to link the visual tokens for the glass with the language token “grasp” and with the relevant action tokens (e.g. those controlling the arm and gripper). The result is a precise flow of information: each part of the input can directly influence the relevant parts of the output. AdaLN-Zero Conditioning: AdaLN (Adaptive LayerNorm) is a method where external data (here, the robot’s state and possibly language or positional information) is incorporated through the normalization layers of the transformer. The “Zero” variant means the adapter starts as an identity (zero bias) and learns to adjust as needed. In ManiFlow’s DiT-X, the low-dimensional proprioceptive state (like joint PoSE) is fused via AdaLN-Zero: essentially the transformer layers can shift and scale their normalized activations based on the robot’s current configuration. This is crucial for robotics, where the same observation can demand different actions if the arm starts in a different pose. AdaLN-Zero allows the network to condition on this smoothly without destroying the pretrained stability of the transformer. The combination of these mechanisms enables “fine-grained feature interactions between action tokens and multi-modal observations”. All the pieces talk to each other. The authors illustrate this with a leveraging example: “When the robot ‘sees’ a cup and hears ‘pick it up,’ DiT-X establishes a precise connection between the cup’s position in the image, the word ‘pick’, and the corresponding movements of the hand and arm”. In effect, each token in the action sequence can attend to and incorporate signals from any token in the visual/language input. This is more flexible than older designs, which often had rigid encodings or modal splits. It helps that DiT-X is built on the strong foundation of diffusion transformers, so it can handle long-range dependencies and high-dimensional outputs. But ManiFlow explicitly repurposes it for world-to-action mapping instead of image generation. The authors even describe DiT-X as the “brain” of ManiFlow that allows full dexterity: thanks to this design, the same model can control a single arm, a dual-arm system, or even a humanoid robot with dozens of degrees of freedom, simply by changing the number of action tokens and providing the appropriate observations. Crucially, DiT-X’s design is unified and general-purpose: the same architecture is used across all their tasks and robot platforms, supporting both 2D inputs (RGB) and 3D point clouds. In practical terms, one can think of DiT-X as a multi-headed network where some attention heads process vision, others process language, and yet others link those to the actions. The AdaLN-Zero and adaptive scaling allow these heads to be reconfigured for each scenario. The end result is a single policy network that, once trained, can be queried on any of the supported platforms (see below) with minimal changes, and it will output a valid, complex trajectory for the robot. Tasks and Benchmarks. ManiFlow is evaluated on an impressively wide set of robotic manipulation tasks, spanning both simulation benchmarks and real-world robots. The authors emphasize generalization, so they test ManiFlow on many variations of tasks, new objects, and even new backgrounds. Here are the main environments and setups mentioned:. MetaWorld (Multi-Task Benchmark): MetaWorld is a popular suite of 50+ diverse manipulation tasks (e.g. reaching, pick-and-place, opening doors, pressing buttons) on a 7-DoF arm, usually with language-conditioning. ManiFlow trains on these tasks as a multi-task policy. The results show ManiFlow beating prior approaches (including both diffusion and simpler flow policies) at all difficulty levels. An example comparison from the paper (Fig. in the blog) shows ManiFlow outperforming a 3D diffusion policy and a baseline flow-matching policy on 48 MetaWorld tasks. Adroit (Dexterous Manipulation): Adroit features hand manipulation tasks: high-DoF “anthropic” hand models (like 24-DoF shadow hand) manipulating objects like hammers, doors, etc. ManiFlow trains policies on these complex hand tasks, testing its ability to handle a lot of degrees of freedom and contact-rich dynamics. DexArt (Dexterous Articulated Objects): DexArt is a benchmark of manipulation tasks involving complicated articulated objects (e.g. puppet-like structures, folding, or tools). These require precise control and vision of deformable parts. ManiFlow’s framework is tested here to see if it can generalize to novel object geometries. RoboTwin (Bimanual/Dexterous Simulator): RoboTwin is a realistic Nvidia simulator with two robot arms and high-fidelity rendering, designed for bimanual tasks like handedness puzzles, pouring, handovers, etc. ManiFlow trains on a variety of RoboTwin tasks, which often require coordinating two arms. The GitHub code specifically references “RoboTwin 1.0” and “RoboTwin 2.0” environments and includes scripts to generate demonstration data for tasks like. These cover tasks of increasing difficulty and environmental variation. In all these simulated benchmarks, ManiFlow consistently outperforms previously published methods. The developers report “superior performance across 60+ simulation tasks on 4 benchmarks”, indicating that on every one of these suites, ManiFlow did better than baselines. For example, in multi-task MetaWorld experiments ManiFlow achieved higher success rates on every task compared to both diffusion-based controllers and standard flow-matching controllers. Figure comparisons (e.g. Fig. ?? in the paper) show ManiFlow’s result curves dominating others. Perhaps even more impressively, ManiFlow is validated on real robots spanning three very different platforms: a single-arm robot, a dual-arm/bimanual setup, and a full humanoid. Specifically:. A single-arm setup using a Franka Emika Panda (7-DoF, robotiq gripper) trained on grape-sorting and fruit-mess tasks, with an overhead RGB-D camera. A bimanual manipulator with dual UFACTORY xArm7 arms (14 DoF total) each outfitted with multi-DoF hands (26 DoF combined). This rig was tasked with cloth-and-cup arrangements, stuffed-object grasping, etc. (The GitHub indicates they used high-res depth cameras like a RealSense and even VR prototypes for data collection). A humanoid robot: the Unitree H1, a full-sized humanoid with 28 DoF (arm, hands, head). This the most complex platform, where ManiFlow demonstrated pouring and grasping tasks from an egocentric ZED stereo camera. On all of these real systems, ManiFlow dramatically outperformed previous best policies. The authors say ManiFlow “nearly doubles success rates on real-world tasks” across the single-arm, bimanual, and humanoid setups. Indeed, their README boast is a “98.3% improvement on 8 real-world tasks” – meaning if a previous method had, say, 30% success, ManiFlow might be around 59%. This was true even when facing novel objects or backgrounds in the real scenes. For example, a pouring task with an unseen cup or a cluttered table still succeeded far more often with ManiFlow than with, say, a flow policy distilled from demonstrations. All in all, ManiFlow’s evaluation covers an unusually broad spectrum: simple pick-and-place, cloth folding, cloth "messy" tasks, precise insertion, to full-body humanoid manipulations. Consistently, the one-step action outputs from MamniFlow were more precise and reliable than the multi-step plans of other approaches. And critically, the consistency training seems to have paid off in that the inferred actions are smooth and physically valid, as witnessed by doubling real-world success. Results and Generalization. Putting it all together, ManiFlow “checks all the boxes” for a general manipulation policy:. Speed (1–2 steps): Because of the consistency objective, inference is extremely fast. ManiFlow never needs a long iterative rollout. This makes it practical for real-time control, even on compute-limited robotics hardware. The authors emphasize this in their results: whereas prior methods might require tens of forward passes (and accompanying computation), ManiFlow’s 1–2 step outputs mean it effectively has the latency of a single neural net evaluation. High-quality actions: Despite being fast, the actions are very precise**. The flow-matching approach models the full distribution of actions seen in demos, so ManiFlow doesn’t collapse to a mean or overly conservative policy. On the contrary, success rates are higher because the model can produce the nuanced, high-dimensional trajectories needed for dexterous tasks. For instance, it can coordinate wrists and fingers to insert a key or manipulate a soft toy, which is very hard for simpler BC networks. Multimodal input handling: ManiFlow seamlessly handles RGB or 3D point cloud visuals, plus optional language or numeric commands, within one unified network. This means one can train it on mixed data (some tasks with vision + text, others with vision only) and get one network that knows how to factor everything in. This flexibility is a boon. The DiT-X design ensures that no matter how many modalities you feed, the attention mechanisms connect them appropriately to the robot’s action. Generalization: The model generalizes in two senses. First, it can handle unseen variations of trained tasks. When tested with new object shapes or backgrounds (even in simulation and real lab scenes it hadn’t seen before), ManiFlow performance degrades much more gracefully than baselines. The rough intuition is that the attention architecture allows it to latch onto the essential features of the task (the cup’s location, the word “cup”, etc.) and not get distracted by irrelevant differences (a new wallpaper pattern, a novel-looking cup). The authors explicitly highlight robustness: ManiFlow “demonstrates strong robustness and generalizability to novel objects and background changes”. They show many images where ManiFlow-trained policies successfully transfer to new arrangements that would normally confuse fixed, pixel-based policies. Second, ManiFlow exhibits strong data scaling. This is the idea seen with large models: train on more demos and it just gets better, often more gracefully than smaller models. The paper notes that as the demonstration dataset size grows, ManiFlow’s performance continues to climb, whereas some baseline methods plateau. Impressively, using around 500 expert demos ManiFlow could reach 99.7% success on a single well-tuned task – outperforming even a large pretrained vision-and-action foundation model (they call it π0) that was much bigger. In other words, ManiFlow can fully exploit big imitation datasets and doesn’t saturate quickly. How ManiFlow Compares and When to Use It. ManiFlow represents a synthesis of many recent ideas in robotics learning:. It is like diffusion policies in that it models an action distribution and can handle multi-modal or uncertain actions. But unlike vanilla diffusion, it does inference in almost no steps thanks to consistency training. It is like flow matching policies (e.g. FlowPolicy 2024) but goes further by explicitly imposing consistency and by adding the DiT-X architecture. FlowPolicy also aimed for one-step generation, but ManiFlow’s consistency objective is novel, as is its transformer structure. It shares goals with consistency distillation methods (e.g. the “Consistency Policy” of Prasad et al., RSS 2024) which distill slow diffusion into a single step. ManiFlow essentially eats its own tail: it designs a flow that is intrinsically one-step, rather than training a separate student model after the fact. For roboticists considering using ManiFlow, some thoughts:. Data collection: ManiFlow still needs a good set of expert demonstrations for each task or environment. The paper’s results depend on having on the order of hundreds of demos per task (which is comparable to other IL research). The advantage is that ManiFlow’s architecture can reuse that data more efficiently. The GitHub provides scripts for generating simulation data (in Adroit, DexArt, MetaWorld, and their own RoboTwin sims), and notes that with ~500 demos a single-task policy can get nearly perfect. On real robots, they likely used teleoperation or kinesthetic teaching to gather a smaller set for each task. Training compute: Training these powerful transformer architectures with flow objectives will require GPUs and some engineering. The code repository provides a Docker container and PyTorch setup, but one should expect a substantial training time if tackling many tasks. The authors, for their experiments, use multiple GPUs to train on big datasets (they mention comparison to a “large π0 model”, implying they know this is heavy). That said, once trained, running ManiFlow is cheap at inference time. Applicability: ManiFlow shines at fully observed manipulation tasks where you can afford to train an imitation policy. It’s not (currently) an online RL method – it assumes you have collected the behavior you want. It's also fully deterministic: given the same state and seed, ManiFlow will output the same action, which is usually fine for manipulation. Because it’s a learned model, it can encode complex trajectories, but users should be aware that learned policies can still fail in safety-critical ways if the scene is very different from training. Integration: The authors note that the “consistency flow training” idea can actually be plugged into other flow or diffusion-based policy learners. In principle, one could take an existing diffusion-policy codebase and replace its loss with ManiFlow’s consistency objective. They explicitly say ManiFlow’s consistency objective “can be seamlessly integrated into existing vision-language-action models by replacing the diffusion loss, enabling faster inference and improved performance”. So even if you don’t want to use the entire DiT-X model, the core training trick is modular. Robustness: Because ManiFlow is trained end-to-end with multi-modal attention, it may handle minor changes better than older architectures that stacked perception then action. However, it is still an imitation system; glaringly novel scenarios (e.g. very different dynamics or objects requiring new skills) will still pose challenges. The authors’ tests on novelty (new objects, new scenes) suggest significant robustness improvement, but one should always validate with safety in mind. Conclusion. ManiFlow by Yan et al. is a compelling new approach in the lineage of generative policies for robotics. Its main contributions are the consistency flow training objective and the DiT-X transformer architecture. Together, these let a single model map from rich sensory input to complex, dexterous action sequences in only a couple of neural network calls – a feat previously thought to require dozens of diffusion steps or hand-crafted integration. The empirical results back it up: ManiFlow shows state-of-the-art success on over 60 simulated manipulation tasks and major gains in the real world, nearly doubling success rates on varied hardware. Conceptually, ManiFlow is pleasing because it unifies and extends many modern ideas. It’s essentially a “flow-based diffusion” model for actions, but trained so consistently that it behaves more like a normal neural policy at run time. Its DiT-X network bridges language, vision, proprioception, and action in one attentional framework. Philosophically, the authors even position ManiFlow as a step toward a “large-model” paradigm for robots – an “LLM of manipulation” – where scaling up data and model capacity yields broad generalization. For researchers and practitioners, ManiFlow opens new doors. It suggests that flow matching and consistency training are viable paths to fast, high-quality policy generation. It comes with code and demos, meaning it can be adapted to new robots and tasks. For instance, one could extend ManiFlow to incorporate LiDAR point clouds, handle deformable (cloth or rope) dynamics, or integrate with higher-level planners. The consistency objective is quite general, so it might also accelerate future diffusion or flow policies beyond manipulation (e.g. in drone control or legged locomotion). In summary, ManiFlow demonstrates that we are quite literally making a “straight line” from generative models to real robot actions. By rectifying the flow (training it so strongly that the path collapses), we get a direct channel from multimodal perception to dexterous actuation. That is the future of embodied AI: fast, flexible, and broadly generalizable control. Well-engineered models like ManiFlow are key stepping stones on that path. Sources: Ge Yan et al., ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training, CoRL 2025 (preprint). Practical details from the authors’ project page and GitHub. Analysis referencing the paper’s abstract and code.