Latest / Embodied AI 101 / PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation
Transcript
- PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. Modern robotic manipulation is increasingly looking to world models – learned predictors of environment dynamics – to enable robots to plan and reason about the consequences of their actions. But most prior work has focused on either simple 2D video models (pixel or feature based) or on simulated tasks with limited variability. The new PointWorld framework goes further: it builds a 3D, embodiment-agnostic world model trained on massive, real and simulated data that predicts how every point in the scene will move in response to robot actions. PointWorld represents both the state of the scene and the robot’s action in a unified 3D point-cloud space of point flows (per-point displacement vectors). By doing so it can learn across different robot bodies and diverse environments, and it can make predictions directly from raw multi-view RGB-D inputs in real time. The core insight of PointWorld is that both objects and robot behaviors can be described geometrically as point flows in 3D. In this paradigm, a camera capture of a scene is turned into a partial 3D point cloud (with associated color features), and a robot motion (e.g. a sequence of joint commands) is converted into a set of 3D “action points” that will move with the robot (e.g. points on the gripper). The PointWorld network then takes this combined cloud of scene points and robot-action points and predicts, for each point in the scene, how it will move (the 3D displacement or flow) over some future horizon in response to the given action. In effect, PointWorld is a learned physics simulator: given what it sees and how the arm will move, it generates a dense forecast of the world’s dynamics. Concretely, PointWorld is “pre-trained on 500 hours of in-the-wild 3D interaction video” amounting to around 2 million short manipulation trajectories. The result is a single large model that is embodiment-agnostic – it can condition on the shape and motion of different robot arms by treating the arm’s motion as part of the input point flow – and that runs in real time (about 0.1 s per inference). The paper shows that even without task-specific training data, a single PointWorld checkpoint can drive a real Franka robot arm in diverse tasks: pushing rigid objects, manipulating deformable objects, opening articulated mechanisms, and using tools – all by planning with the world model and no further learning. Below we unpack how PointWorld works, what the authors did, and why it matters for robotics. We discuss the unified 3D representation of state and action, the neural network architecture, the unprecedented dataset and annotation pipeline that make it possible, and the experiments demonstrating the model’s capabilities and generalization. Unifying State and Action as 3D Point Flows. Traditional model-based planners often use rigid-body simulators or low-dimensional system equations; in learning-based work, “world models” have typically been trained to predict images or latent features over time. PointWorld instead treats the entire observed scene and the robot’s motion as a collection of 3D points with associated displacement vectors, called point flows. The scene state $\mathbf{s}t$ at time $t$ is encoded as a (sub-sampled) 3D point cloud reconstructed from RGB-D sensor data, and the robot action $\mathbf{a}{t:t+H-1}$ over $H$ time steps is encoded as a set of 3D points that will move with the robot – for example, points on the robot’s gripper bodies. More formally, the model implements a function $$F_\theta^H(\mathbf{s}t, \mathbf{a}{t:t+H-1}) ;\to; \mathbf{s}{t+1:t+H},$$ mapping the current state and future actions to the next $H$ states (each represented as point flows). In practice $\mathbf{s}{t+1}$ is represented not as a new point cloud, but as a flow (vector displacement) for each input point in $\mathbf{s}_t$. In other words, PointWorld predicts how each point in the scene will move in 3D when the robot executes its planned actions. This design sidesteps the need for an explicit simulator: the model never operates in a fixed object-centric representation or requires a known physical system; it just links “current 3D points + planned motion of the robot” to “future 3D displacements of those points.” Crucially, because actions are encoded as 3D point flows too, the model is embodiment-agnostic. It does not rely on joint angles or specific kinematic parameters, but instead directly uses the geometric influence of the robot’s motion. As Huang et al. explain, “by representing actions as 3D point flows instead of embodiment-specific action spaces (e.g., joint positions), this formulation directly conditions on the physical geometries of robots while seamlessly integrating learning across embodiments”. In short, a Franka gripper and a different bimanual robot arm can both be used with the same model simply by describing their future movements as sets of point displacements. This 3D point-flow representation excels at capturing the essence of contact and motion in a manipulative scenario. If the robot gripper closes on an object, the “flow” of the gripper action will suggest contact at certain scene points, and the model learns to propagate that into object motion. Representing actions as dense point flows has a tangible advantage: in experiments it “outperform[ed] global pose, joint-space, and sparse whole-body alternatives” for multi-embodiment training. The dense representation naturally encodes where (in the 3D geometry) the robot is interacting, supports partial observability (the robot need not see the whole body), and helps transfer physics knowledge across different robots. In implementation, the robot’s point flows are obtained by sampling gripper contact points (or other relevant points on the robot) and applying forward kinematics to propagate planned joint motions. In effect, the network sees a cloud of points for the scene and a cloud of points (colored or marked differently) representing where the robot will be, and it has to predict 3D scene flow fields that respond to that action. One can view this as giving the network both “what is in the world” and “what I will do next,” and asking it to imagine consequences. By training on real video of manipulators, it learns to anticipate how piles of objects move, cloth deforms, doors open, etc. To summarize the high-level idea: PointWorld is a point-cloud-based world model where both environment and action are points in a shared 3D space, and the network learns a mapping[ (\text{points}{t}, \text{robot-action-points}{t:t+H-1}) ;\to; \text{point-flows}_{t+1:t+H}. ]As the authors put it, PointWorld “unifies state and action in a shared 3D space as 3D point flows” and predicts “per-point scene evolution directly in response to a temporal sequence of robot actions”. By working directly with 3D geometry, it makes minimal assumptions about objects or tasks. Architecture: PointCloud Transformers and Pretrained Vision Features. Building a neural network to consume these 3D point clouds and produce flows requires a powerful 3D perception backbone. PointWorld adapts recent advances in point-cloud transformers and self-supervised vision models. Concretely, the input to the network is a single aggregated point cloud at time $t$ (together with all the planned robot action points stacking $H$ future steps), and the output is a 3D flow vector for each input point indicating its predicted movement over the next time step. Backpropagation trains $\theta$ to minimize the difference between predicted flows and ground-truth flows, across millions of examples. In detail, the pipeline is as follows. First, the scene points are obtained by reconstructing the environment from synchronized RGB-D captures (potentially multi-view) and any known camera calibration. Each point carries an RGB color or feature. The robot’s action points are sampled at, say, the gripper tips, and are given 3D coordinate displacements according to the robot’s kinematic model (URDF + planned joints). These robot points may also have features indicating the time step or other pose info. All these points are merged into one point cloud. Next, each point is featurized: for scene points, PointWorld uses a frozen DINOv3 vision encoder (a self-supervised ViT) to extract high-level image features and attach them to 3D points via projection or back-projection. The robot action points do not have image appearance, so instead each is endowed with a temporal embedding that conveys when that point is active in the sequence. The result is a unified cloud of 3D points, each with a feature vector that encodes either visual appearance or temporal context. This engineered point cloud is then processed by a PointTransformerV3 (PTv3) backbone, a state-of-the-art hierarchical transformer for point clouds. The transformer carries out multiple layers of multi-head self-attention mixed with local aggregation, building per-point feature representations that mix information from nearby points (both scene and robot). Because the transformer sees the whole scene geometry, it can infer contacts, collisions, and motion propagation. After several layers, a final per-point head predicts the 3D flow vector (how far and in what direction that point will move in the next step). In summary, the model uses a pretrained 2D feature extractor (DINOv3) for raw appearance, and a 3D transformer backbone (PTv3) for spatial reasoning. This “neural mapping” has high capacity: the authors report exploring models up to ~1B parameters. Notably, they find that larger PointTransformer backbones significantly improve performance as the model scales (consistent with modern scaling laws). A key advantage of this architecture is the combination of learning and prior knowledge. The frozen DINOv3 encoder brings in rich visual features learned from Internet images, which helps the model recognize objects and textures without additional training. Meanwhile, the PTv3 backbone exploits scale and attention to capture complex interactions. The result is a model that can handle open-world scenes (trading off partial views, noise, and novel objects) with finesse. Massive, In-the-Wild Data and Annotation Pipeline. No single dataset in robotics was sufficient to train such a model. The authors curated and combined both real-world and simulated data into an unprecedented 3D dynamics corpus. In total they assembled about 2 million short trajectories (~500 hours of footage) of manipulation from diverse sources. This includes:. DROID: A recently released in-the-wild robot manipulation dataset (real-world) featuring a mobile/base robot and single-arm Franka interacting with household objects. The PointWorld team used roughly 200 hours of teleoperated demonstrations from DROID, spanning many environments, objects, and tasks. BEHAVIOR-1K (simulated): A large-scale synthetic household dataset of 3D manipulation tasks, built on iGibson/AI2-THOR or similar simulators. The team harvested roughly 1100 hours of simulated grasping, pushing, and tool-use trajectories from BEHAVIOR (after filtering irrelevant data). Because simulation can provide unlimited views and ground truth, it complements the real data and fills gaps in object diversity. These datasets were reprocessed through a custom 3D annotation pipeline that the authors developed. For each video sequence with synchronized RGB-D and known robot poses, the pipeline reconstructs accurate metric 3D geometry and per-point flows. It uses NVIDIA’s FoundationStereo to refine depth and camera poses, Voxel Global Graph Tracking (VGGT) for object consistency, and Co-Tracker3 (dense point-tracking). The outcome is millimeter-accurate 3D reconstructions and dense correspondences between frames. In DROID, for example, this process produced ~42K 1-second annotated episodes (with multi-view stereo), yielding ~400K high-fidelity point-flow segments. The authors note this is likely the largest real-world 3D dynamics dataset available to date. Having this curated data is critical. The use of markerless stereo and dense tracking means the model learns from precise geometry rather than noisy sensor depth. It can see how rigid objects slide, how cloth wrinkles, and how articulated links move, all in 3D coordinates. The scale of 2M trajectories across multiple robots and environments is orders of magnitude beyond most prior robotics datasets. It lets the model learn very general “physics” priors. As one excerpt notes, this scale is “huge for 3D dynamics” and performance empirically grows log-linearly with more data, echoing trends in large-scale vision and language models. All the code for data processing and annotations is open-sourced. By building on recent 3D vision tools and releasing their pipeline, the authors aim to make it possible for others to plug in new videos and expand the training corpus further. This level of dataset engineering – integrating real teleoperation videos, simulation scenes, and state-of-the-art 3D labeling – is one of PointWorld’s biggest contributions. Training Strategies and Loss Design. With the data prepared, the model is trained to predict 3D flows over short horizons. In practice the authors train on or predict chunks of several future steps at a time, rather than just single-step. This “chunked prediction” (also known as rollout) helps reduce drift compared to purely teacher-forced one-step prediction. During training, each training example consists of an initial RGB-D scene (converted to points) and an $H$-step sequence of robot point flows, and the supervision is the sequence of resulting scene point displacements. Crucially, the training objective is weighted to focus on the active motion. In most scenes the majority of points are static (the table, floor, background). If one treats all points equally, the learning is dominated by trivial zero-motion predictions. PointWorld addresses this by computing a “mover likelihood” weight for each point (the probability that it moves significantly under the forthcoming actions). The loss then up-weights those moving points and down-weights static ones. This concentrates the training signal on the interesting physics (an object sliding or deforming), which greatly improves quality. Another challenge is noise in the labels, especially in real data. To make training robust, PointWorld has the model predict an aleatoric uncertainty (log-variance) for each point’s motion, and uses it to weight the regression loss. In effect, points with blurry or inconsistent tracks (e.g. occlusions or sensor mismatch) will lead the model to assign higher uncertainty and thus down-weight their contribution. The authors also use a Huber (smooth L1) loss instead of pure L2 to further mitigate outliers. These techniques – motion-focused loss weighting and per-point uncertainty – are important for stabilizing learning on real-world video. The reported evidence suggests these choices significantly improve prediction accuracy by “amplifying signal-to-noise on informative interactions” while preventing overfitting to artifacts. Beyond this, standard regularization (dropout, normalization) is used, and training is done on large GPU clusters. The final model is quite large (hundreds of millions of parameters or more), reflecting the complexity of the task. The authors also experimented with various backbones (different point-neural networks) and found that modern transformers (PointTransformerV3) outperformed older GraphConv or PointNet-style networks when scaled up. This mirrors general deep learning trends: model performance improved predictably with both data and parameters. In fact, they report near log-linear gains for error vs. log(data) or log(params), suggesting rich “scaling laws” similar to those seen in vision and language. Experimental Results: From Simulation to the Real World. PointWorld’s claims are backed by extensive experiments. The core tests fall into two categories: (1) prediction accuracy in held-out datasets, and (2) closed-loop manipulation tasks using the world model for planning. Prediction and Ablation Benchmarks. First, the paper evaluates how well PointWorld can forecast 3D point flows. On held-out real-world data (DROID) and simulated data (BEHAVIOR), the model’s mean-squared error per-point is much lower than baselines (like No Vision, or a model without the scaled architecture, etc). Detailed ablations show that each design choice – action-as-flows vs. joint-pose encoding, DINOv3 image features vs. raw RGB, multi-step training vs. single-step – contributes measurable gains. For example, representing the robot action as dense gripper point flows rather than as a single end-effector pose or entire-body joint vector made a big difference in both multi-embodiment accuracy and transfer. Similarly, using the pretrained DINOv3 features for each point crucially provides the model with semantic object understanding. The model also shows remarkable generalization. A model trained only on simulated interaction can still predict fairly well on real scenes, and vice versa (with some drop). Finetuning on mixed real+sim data yields the best of both worlds. One striking result is that even without any trajectory-specific training, the network handles cases it has never seen. For instance, the paper reports that a model jointly trained on single-arm Franka data and on a separate bimanual humanoid data achieves good predictions on both robots – validating the embodiment-agnostic approach. Zero-Shot Manipulation via MPC. The most eye-opening experiments are on actual robot tasks. The authors embed the pretrained PointWorld in a Model Predictive Control (MPC) loop. Given a goal (like “push the object to a target” or “open the drawer”), the MPC repeatedly samples candidate action sequences, uses PointWorld to predict their outcomes, and scores them by how close the predicted final state is to the goal. The best action is executed on the real robot, then the loop repeats with new visual feedback. Because PointWorld is so fast (≈0.1 s per forward prediction) and accurate, this planning scheme works impressively well zero-shot – meaning no additional demonstration or fine-tuning for the task. For example, a Franka arm could push a solid block to a desired location purely by imagining the movement of the block in 3D space. It could also manipulate deformable objects like applying a cloth or sponge on the table, predicting how the material drapes under the arm. In articulated manipulation, it could open cabinet doors or flip switches by predicting the motion of the jointed object. And in tool use, it could pick up (from a single view) a hooked rod from the clutter, insert the hook under a handle, and pull – all by planning with the model. These diverse skills were not explicitly taught; they emerged from the underlying geometry-based dynamics model. Quantitatively, the video demonstrations and metrics (in the paper’s supplementary) show that PointWorld-MPC succeeds at a high rate on these tasks, often outperforming naive baselines or planners that lack visual foresight. The authors emphasize that one checkpoint does it all: “a single pre-trained checkpoint enables a real-world Franka robot to perform rigid-body pushing, deformable and articulated object manipulation, and tool use, without requiring any demonstrations or post-training”. This is a remarkable zero-shot capability and suggests that the world model has captured a broad notion of daily physics. In simulated environments (BEHAVIOR), similar experiments show that PointWorld allows agents to solve unseen tasks in new homes with minimal decline. Even untrained novel backgrounds or object sets cause only a minor performance drop, closing “the majority of the gap with domain-specific specialists” and recovering it with a quick 20× fewer finetuning steps. Scaling Laws and Ablations. PointWorld is as much about engineering scale as about clever ideas. The authors conduct rigorous ablations to distill what really matters:. Backbone and Capacity: They compare different 3D architectures. The latest PointTransformerV3 consistently outperforms older models, especially when model size is large. Scaling up the transformer layers and heads yields clear error reductions. Indeed, they chart a log-linear improvement even up to ~1B parameters, mirroring how scaling benefits other modalities. Data Mix: They test using only DROID data, only BEHAVIOR, or both. The combined set always wins, highlighting that diverse data (sim+real, different objects, various tasks) is crucial. Using simulated data alone still generalizes well, but mixing in real “wild” data reduces sim-to-real gaps. Prediction Horizons: Training to predict multiple steps in one shot (chunked rollout) is better than naive teacher forcing (predict one step repeatedly). Multi-step learning requires the model to stabilize over time and was found to reduce roll-out drift. Observability: They experiment with having the model see one vs. two camera views. As expected, giving two views (when available) improves depth accuracy. But even a single view, thanks to the strong stereo depth annotation, works surprisingly well. The model’s partial observability handling (enabled by action conditioning and geometry) is robust. Temporal Embedding: The way robot points are labeled matter too. The chosen scheme (gripper points with time indices) was better than, say, including the whole arm at every timestep. Each of these design choices was validated by corresponding metrics on held-out prediction accuracy. The upshot is that the final PointWorld recipe – DINOv3 + PTv3 + mover-weighted loss + aleatoric regularization + multi-step training + massive data – is a result of deliberate empirical study. Discussion and Implications. PointWorld represents a significant step toward “generalist” robotic models. By fusing large-scale data with a spatially-rich representation, it demonstrates that a single 3D model can handle many manipulation scenarios without task-specific retraining. This is akin to how large vision-language models, once pre-trained on internet-scale data, can perform zero-shot tasks across domains. Here, the “internet data” are hours of robot videos, and the domain is 3D physics. For robotics, the implications are wide-ranging:. Plug-and-Play Planning: A pre-trained PointWorld can be plugged into any robot with known kinematics. Since actions are just point flows, one can use a different arm or even a simulated avatar, and the model still applies. This lowers the barrier to deploying advanced planning: one doesn’t need to collect new data for each new arm or object; the geometry-aware model already knows basic physics. Bridging Sim and Real: By training on both real and simulated video, PointWorld closes the reality gap. The careful annotation of DROID data essentially lets the model learn from cameras what objects move like. In practice, the authors found that a model pre-trained on simulation (BEHAVIOR) could transfer to real tasks with only minor fine-tuning, thanks to the shared 3D language. This could herald workflows where robots continue to improve from watching synthetic or web videos of humans manipulating things. Spatial Intuition: Because the model operates in 3D, it implicitly builds an understanding of contact geometry and volumetric motion. For example, in tool use, it must infer the shape and constraint of the hook and the target. This stands in contrast to pixel-based models that often struggle with occlusion or force-based inference. The fact that PointWorld successfully plans pulling a hook under a handle suggests it has learned a form of 3D “common sense.” This paves the way for robots that reason about tools and objects more flexibly. Foundation Models for Manipulation: PointWorld can be seen as a step toward the robotics analog of a foundation model. It ingests broad sensory data (RGB-D of many scenes) and self-supervised robot signals, and produces a generalist prediction engine. In the future, one can imagine combining this with other modalities (language goals, tactile feedback, etc.), but even now it embodies rich 3D priors that could accelerate learning on downstream tasks. Of course, challenges remain. The model still has limits in longer-horizon or highly deformable interactions, and it requires RGB-D input (depth precision matters). Planning speed (even at 0.1 s per step) can be a bottleneck if more reactive control is needed. The model also does not explicitly reason about materials (e.g. how soft a toy is) or detailed safety constraints. Nevertheless, PointWorld charts a new direction: leveraging big data and big models in the 3D space of real-world robotics. Its success suggests that further scaling – more video data, bigger networks, richer annotation – could yield even more capable embodied agents. It also validates the idea that state and action belong together; by learning them jointly in 3D, the model sidesteps many complexities of traditional planning. In practical terms, anyone building a robotic manipulation system can take away that pre-trained 3D world models are now viable. If you can capture a stereo RGB-D view and have a kinematic model of your arm, you could potentially use PointWorld (or its future successors) to predict and plan motions in complex scenes. This could dramatically reduce the need for hand-engineered simulators or task-specific training data. In sum, “PointWorld” is an exciting blend of computer vision, graphics, and robotics – unifying them in service of better control. It opens the door to more robust, general, and data-driven understanding of the physical world by robots, blurring the line between seeing, thinking, and acting. References: Huang et al., PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation, CVPR 2026 (highlight). The project site (point-world.github.io) provides details on the model, dataset, and code.