Latest / Embodied AI 101 / PhysisForcing: Physics-Reinforced World Models for Robotic Manipulation
Transcript
- PhysisForcing: Physics-Reinforced World Models for Robotic Manipulation. In the quest for smarter robots, one promising approach has been to use video generation models as stand-in “world simulators” for robotic manipulation. The idea is simple: give a robot a snapshot of the scene and a desired action, and have a generative model produce the ensuing video frames. This video serves as the simulated outcome of the robot’s action, which can be used for planning or policy learning. In theory, such embodied video models can capture rich visual and temporal dynamics without hand-crafted physics engines. In practice, however, these models often err badly on the most crucial thing: physical realism. Objects can warp or float, grasps can slip or glitch, and contact events can be ignored. As the PhysisForcing paper observes, generative video models – even ones fine-tuned on robot data – tend to produce physically implausible manipulations. Motion trajectories become discontinuous, objects visibly deform, and robot–object interactions rarely follow real-world laws. Such errors may still look “okay” at a glance, but they fatally undermine the robot’s trust in its own internal simulations. The new PhysisForcing framework (Zhang et al., 2026) tackles this problem head-on by injecting physics awareness into training. Rather than redesigning the diffusion model architecture or slowing down inference, they introduce auxiliary losses during training that explicitly supervise physical consistency. Remarkably, this “plug-and-play” approach yields state-of-the-art results on several robotics video benchmarks (R-Bench, PAI-Bench, EZS-Bench) and boosts closed-loop task performance – all without any extra cost at inference time. In this article, we’ll unpack what PhysisForcing does and why it works. The Physical Instability of Video World Models. First, let’s understand the problem more concretely. Modern video diffusion models (e.g., Wan2.2, Nvidia’s Cosmos3™ models) are incredibly powerful at generating plausible-looking clips from text or image prompts. But for physical tasks, plausibility isn’t enough: physical consistency means obeying constraints like no interpenetration, satisfying kinematics, preserving mass properties (e.g. rigid objects shouldn’t squash), and honoring gravity and contact. In practice, relatively minor consistency errors multiply into nonsensical fantasies. For example, past work has documented robot hands that seem to pick up objects only half the time, or objects that jitter and deform as they move. Two main culprits emerged in thorough diagnostics by the PhysisForcing authors:. Object Deformation: Generated objects often behave like jelly rather than solids. A block being pushed might stretch or warp inexplicably slightly after contact. Over time, the model’s outputs violate mass and shape conservation. Spatio-Temporal Mismatch: The relationships among entities often don’t stay coherent. A gripper might release an object into mid-air, or two objects that start together drift apart or clip through one another. In short, interacting entities lack the proper spatio-temporal correlations seen in real physics. These pathologies manifest as jerky, disconnected motion trajectories or inconsistent contact. Crucially, simply fine-tuning a generic video model on robot datasets alleviates the problem only slightly – the models still hallucinate these physics glitches (see Figure 3 in the paper’s supplement, and preview images in the project demo). PhysisForcing’s key insight is that we can directly supervise the model on the underlying physics cues that fail. Instead of relying on end-to-end perceptual metrics, we impose structure on the model’s internal representations so that, for example, the points on a manipulator and an object follow realistic, continuous paths, and that if the object is in contact it moves rigidly with the gripper. To do this effectively, PhysisForcing identifies the physics-relevant regions of each frame (robots, objects, contact points) and applies two complementary alignment losses: one at the pixel/trajectory level, and one at a higher semantic (feature) level. Before diving into those, let’s review the challenges of implementing this idea. Region-Focused Physics Supervision. A fundamental design choice in PhysisForcing is to focus supervision on the most physics-informative parts of the frame. Not every pixel in a robotic video matters. Background walls, static tables, or irrelevant scenery can distract a loss that tries to encode “physics,” which is only crucial around the robot arm and objects. To address this, PhysisForcing uses a depth-aware motion mask that localizes the robot and moving objects. Concretely, an auxiliary perception pipeline (for example, an off-the-shelf depth model or segmentation network) identifies the pixels corresponding to the manipulator, manipulated objects, and regions of contact. By masking out all other pixels, the physics losses concentrate only on where action happens. As the authors put it, this “concentrates supervision on manipulators, objects, and contacts rather than all pixels uniformly”. In effect, the model learns to give extra attention (while training) to the robot–object interactions where physical laws must hold. This region-focus also makes the additional losses tractable. Instead of aligning every feature in a large space, only the “active” parts contribute. And by using depth or motion cues, the masking can be done automatically per frame, without manual annotation. In practice the authors find that focusing on these regions leads to better outcomes: the model doesn’t worry unnecessarily about, say, keeping a wall texture consistent, and instead learns to enforce continuity in the moving parts. With the physics mask determined, PhysisForcing applies two hierarchical losses: a low-level (pixel/trajectory) alignment and a high-level (semantic/relational) alignment. Together, these cover different aspects of physical plausibility. Pixel-Level: Trajectory Alignment. At the heart of tracking physics is trajectory consistency. Every point on an object should move along a smooth, continuous path compatible with its rigid-body motion (unless it’s being dynamically deformed, which most simple objects are not). To enforce this, PhysisForcing uses an off-the-shelf point tracker called CoTracker3 (from Meta AI) to extract “reference trajectories” for points in the ground-truth training videos. CoTracker3 is a general-purpose pixel tracker that, given a starting point (e.g. a small dot painted on the object), will follow it through the frames. Here’s the clever part: the diffusion transformer (DiT) model, during training, implicitly encodes motion in its internal features. PhysisForcing taps into that by predicting point coordinates from the DiT’s token features and comparing them to the CoTracker reference. Concretely, for each selected point (on the manipulator or object), the DiT has a learned “implied coordinates” regression (or analogous mechanism) that outputs where that point is in each frame. A masked mean-squared-error loss then penalizes the difference between the DiT-predicted coordinates and the actual tracked coordinates. Informally, this loss says: “If in the ground truth a point moves steadily from A to B, then in our video model’s feature space the corresponding point should also move from A to B.” By applying this across many points, the model must learn to generate motions that follow continuous, physics-like trajectories. The mask ensures only the tracked points (on the robot and objects) count. Two immediate benefits arise. First, local motion is constrained: jitter, teleporting, or slipping actions would incur a large error in the predicted trajectory. Second, contacts are respected: if the ground truth shows the gripper and object moving together during a grasp, the DiT must encode a similar coupled motion to keep their relative positions aligned. As the paper notes, this keeps the motion “continuous and contact-compatible”. In practice, this pixel-level alignment significantly reduces object warping and separation artifacts, because any such deformation would break the point correspondences and be penalized. Semantic-Level: Relational Alignment. Physical consistency isn’t only about individual points. Often it depends on how different parts of the scene relate to one another. For example, two parts of an object should have a constant offset if it’s rigid, a grasped object should follow the gripper, and two separate objects might follow their own paths or collide, but generally their behavior is structured. To capture these broader relationships, PhysisForcing adds a second loss that aligns semantic relations in feature space. The approach is to take all the relevant token features from the DiT (again focusing on masked “physics tokens” corresponding to robot and object regions) and look at their pairwise similarities. Simultaneously, the same ground-truth video is passed through a frozen video understanding encoder—a strong pretrained model (for example, a self-supervised transformer trained on video) that is good at capturing scene semantics. This encoder also outputs tokens, and we compute the pairwise token similarity matrix on its features, but again only for the physics-relevant tokens. The loss then tries to align these two matrices of similarities. Intuitively, if the video encoder sees that token i (say, a part of the gripper) and token j (a part of the object) have a certain relationship (e.g. very high similarity because they move together), then the DiT’s tokens should reflect the same. If the video encoder’s features indicate that two objects are far apart (low similarity), or that one token stays unchanged relative to another, the model is guided to produce a corresponding feature structure. This relational alignment enforces global consistency. The paper describes it as “transferring the encoder’s relational structure” so that globally consistent interactions emerge. For example, in a pick-and-place, the generated video should keep the object coupled to the gripper during the entire lift, or maintain an object's orientation and relative contact points. If the DiT were to let them drift apart or suddenly misalign, the similarity loss would spike. In effect, the model is learning not just to get each point trajectory right, but to preserve the geometric configuration of the scene as implied by a strong vision encoder. Importantly, because the video encoder is frozen, this acts purely as a teacher. Many video encoders are trained with self-supervision (like masked autoencoding or contrastive losses) on diverse scenes, so they have an internal notion of object coherence and physical interaction. By mimicking its embeddings’ structure, the DiT learns those priors about the physical world. Putting It All Together. In summary, the PhysisForcing learning scheme proceeds as follows during training:. Generate Mask: From the input frames (and robot state), compute a motion/depth mask to highlight the robot and manipulated objects. Pixel Alignment: Track points on those objects across frames using CoTracker3. The DiT, during its normal denoising training on video, is also used to predict those points’ positions. We apply a masked MSE loss on predicted versus reference trajectories. Semantic Alignment: Feed the same frames (or frames from the ground-truth video) into a pretrained video encoder. Compute token embeddings for the masked regions. Compute a similarity matrix of token pairs. Do the same for the DiT’s tokens on those regions. Compute a loss that encourages these two similarity matrices to match. Joint training: Both alignment losses are added to the usual video diffusion training objective (e.g. denoising loss). The DiT model’s parameters are updated to minimize all losses. Crucially, after training, none of the extra machinery is needed. At test/inference time, PhysisForcing produces a standard video diffusion model. We drop the point tracker and video encoder – they only existed to shape the training. Thus the model generates videos just as fast as the base model, with zero extra inference cost. The magic is all baked into the weights. This plug-in nature makes PhysisForcing very appealing: it can be applied to improve any suitable video generation backbone. The authors tested PhysisForcing on two backbone models in particular: Wan2.2-I2V-A14B, an image-to-video diffusion model from Wan-AI, and Cosmos3-Nano, a small-scale version of NVIDIA’s Cosmos 3 world model. In both cases, applying PhysisForcing (i.e. finetuning the pretrained model with the extra losses) significantly improved the physical quality of the generated videos. Results: More Plausible Videos, Better Performance. The ultimate goal of PhysisForcing is to generate better videos. Indeed, the qualitative improvements are striking. In the paper’s figures, side-by-side comparisons show that a vanilla fine-tuned model might have the robot’s gripper slip under an object or an object drift out of alignment, whereas the PhysisForcing model keeps a firm grasp and straight trajectories. For example, in one insert, a standard model produces a cup that wobbles as it’s lifted; the PhysisForcing version holds the cup steadily. Another example shows a ball fusion: the baseline’s pink ball falls through a plate, while the physics-enhanced version lifts it cleanly and places it accurately. In essence, PhysisForcing turns “implausible to plausible” behaviors. These visual gains are backed by numbers on three recent benchmarks designed for embodied video generation:. R-Bench (RoboTwin or “RoboBench” series): Focuses on various robotic pick-and-place and manipulation tasks across different robot embodiments. Here the metric is an average score combining task success (or plausibility) and embodiment consistency. PAI-Bench (Physical AI Bench): As introduced by Zhou et al. (2025), it evaluates video models on “quality” and “domain” metrics over robotic arm, driving, and egocentric scenes. The “domain” score emphasizes physical realism. EZS-Bench (Embodied Zero-Shot): From the ABot-PhysWorld framework (March 2026), it’s a zero-shot benchmark that scrupulously rates physical fidelity (no object penetration, correct contact, consistency) and control aspects in unseen scenarios. On all three, PhysisForcing achieved new state-of-the-art scores. In a “glance” figure, the authors report that, relative to a vanilla finetuned model, PhysisForcing yields single-digit percentage gains in these benchmarks, but in a very competitive setting where any bump is hard-earned. More concretely, on R-Bench (average over tasks and embodiments), the PhysisForcing-enhanced Wan2.2 model (called PF-Wan14B) scored 62.0% vs. 57.9% for the plain finetuned baseline. That’s a +7.1% relative boost, and relative to the original unfine-tuned model it’s a +22.3% leap. On PAI-Bench and EZS-Bench, PhysisForcing models similarly topped the leaderboard in the “domain” or physical consistency metric by a few percent over the prior best (4–5% on PAI, ~3% on EZS). It’s worth interpreting these: these video benchmarks use learned metrics and sometimes human scores, so even % points are meaningful. The domains where PhysisForcing shined were precisely the physics aspects: it maintained the highest “SOTA” ranking on all three tasks. In plain terms: a PhysisForcing-trained model generates videos that qualitatively look more physically correct, and the automated benchmarks agree. Perhaps more surprising, enforcing better video fidelity also helped downstream tasks. The authors plugged the video models into an action planner and evaluated closed-loop manipulation success. In a simplified summary:. RoboTwin-2.0 Tasks: Using a “Fast-WAM” planner backbone, the image-to-video model drives a robot to complete tasks like placing cups or pressing buttons. The success rates on individual tasks mostly improved. For example, the “place empty cup” task jumped from 41.5% success up to 63.0% (+21.5% absolute). Averaged over six tasks, success went from 68.2% to 72.8%. WorldArena IDM (Action Planner): This is a more general planning evaluation across two tasks. Here the baseline Wan2.2-5B model had only 16.0% average task success, but the PhysisForcing variant (PF-Wan5B) achieved 24.0% – a 50% relative improvement. That’s a large leap in this challenging zero-shot setup. Why does adding physics losses improve planning? One interpretation is that a more accurate world model yields better predictions, so the planner picks better actions. When the generated future frames obey real-world rules, the policy can trust them. The PhysisForcing paper explicitly notes that physically aligned video models “yield stronger representations for robotic manipulation”. In other words, it’s not just about pretty videos – it’s about useful simulations. Crucially, all these gains came “for free” at inference time. The robot runs exactly as it would with the base model; the difference is entirely in the training. This means a PhysisForcing model can drop into existing pipelines without changing deployment. Key Takeaways and Implications. PhysisForcing demonstrates a powerful principle: you can “steer” a generative model’s behavior by designing clever training losses that encode prior knowledge. Here, the prior is physics; the losses are trajectory alignment and relational consistency. Several broad points stand out:. Hierarchical Supervision: Enforcing physics is tackled at multiple levels. Low-level trajectory alignment ensures smooth motion, while high-level relational alignment keeps the global scene coherent. This hierarchical view (pixel vs. semantic) seems key to covering all failure modes. Region Focus: By automatically masking to robot and object areas, the method avoids confusing the model with irrelevant parts of the scene. In practice this targeted supervision proved much more effective than generic losses on whole images. Any-Model Applicability: PhysisForcing is model-agnostic as long as one has a differentiable generator. The paper used DiT-based diffusion models, but in principle other video generators could also benefit. In fact, it’s presented as “plug-and-play”. The core requirements are only additional offline tools (CoTracker, video encoder) that are discarded after training. Connection to World Models: This work bridges generative modeling with embodied intelligence. It can be viewed as building richer world models. Where past “world model” research mostly used metric reconstructions or dynamics models, here a generative diffusion model serves as the simulator. By forcing it to respect physics, it becomes a stronger emulator of reality, useful for downstream planning or RL. Zero Overhead at Inference: A practical high point – because no extra nets are used at test time, there is no latency hit. The augmented model runs at the same speed as the vanilla diffusion model. Of course, no solution is perfect. PhysisForcing relies on having ground truth video data for training, and on auxiliary modules for that data (tracking, video encoder). It’s not a self-contained physics engine. If the training data are limited or biased, some failure modes might persist. Also, the current method primarily enforces short-term consistency – extremely long-horizon behavior might still drift over dozens of frames. However, the diagnostics suggest the main defects are largely cured. Looking forward, this approach could inspire further hybrid methods. For instance, one could combine PhysisForcing with multi-view consistency (as in the contemporary PAIWorld work), or add more explicit physics simulation dimensions. The fact that PhysisForcing improves so readily suggests that diffusion transformers have latent capacity for realism that just needed the right pressures to surface it. In summary, PhysisForcing shows that we can have our cake and eat it too: train rich video diffusion models for robotics, and still inject hard physics constraints through judicious losses. The result is videos that not only look coherent, but also behave more like the real world – a key step for truly embodied AI. The code and models are publicly available, making this a promising tool for anyone looking to build better robotic world simulators. Citations: All claims and quotes above reference the PhysisForcing paper and its accompanying materials. Key points like the design of pixel/semantic losses, benchmark improvements, and no additional inference cost are drawn from Zhang et al. (2026). The definitions of R-Bench, PAI-Bench, and EZS-Bench and their scoring come from the project website and related papers. The improvement in closed-loop planning success (WorldArena) is also documented in their results.