Latest / Embodied AI 101 / SERF: 4D Latent Mapping for Long-Horizon Mobile Manipulation
Transcript
- SERF: 4D Latent Mapping for Long-Horizon Mobile Manipulation. Long-horizon mobile manipulation – imagine a wheeled robot entering a room, scouting around, picking up several toys, navigating around furniture, and stashing them neatly on shelves – is fundamentally hard. The robot must continually know where it is, what has changed in the environment, and how close it is to completing the task. Yet a single camera snapshot and a language instruction give very limited clues. Modern vision-language-action (VLA) models can translate images and text into robot actions, but they typically lack any explicit memory beyond the current view. In practice this means the robot forgets what it saw only a few seconds ago, and has no global map of the scene. If an object was behind the robot a moment ago, the pure image-based policy can’t recall it easily. SERF – short for Spatiotemporal Environment and Robot Feature map – addresses this by giving the robot a built-in spatial memory of the workspace that persists over time. SERF embeds both the environment and the robot’s own body into a single shared map of latent features. As the robot moves and interacts, this map updates: objects that are picked up or moved leave footprints in the latent space, and the robot’s own pose is tracked. In effect, the robot builds a 4D latent map (3D space plus time) of the world. This map is then tokenized and fed into a VLA policy, giving the policy rich spatiotemporal context – global allocentric information about object locations, along with egocentric cues about the robot’s configuration and reachability. On long-horizon mobile manipulation benchmarks like BEHAVIOR-1K, a SERF-based policy far outperforms baselines that see only images: it makes more progress on tasks, takes more direct paths, and even recovers from failures like dropped objects. In this article we unpack how SERF works and why the extra “latent map” matters. We’ll walk through the map representation itself – neural-point feature maps for the environment and the robot, how they are trained and updated – and how the vision-language-action policy is conditioned on it. We’ll also highlight the experimental results on BEHAVIOR-1K that demonstrate the gains, and discuss what this means for future embodied AI. The Memory Challenge in Long-Horizon Robot Tasks. Imagine a home robot assigned a complex task like “Tidy up this room by putting all toys on the correct shelves.” The robot’s on-board cameras capture the current view: maybe it sees some toys and furniture nearby. A standalone VLA model (for example, a vision-language model steering a policy) can parse that image and the instruction to generate the next few actions – say, drive forward or grasp a red ball. However, once the robot starts moving, cameras will no longer see the hallways it just passed, nor the toys behind it. Yet those things might still matter: perhaps the red ball it needed to pick up was actually behind it initially, or maybe as it moved a teddy wandered into a corner. Without memory, the policy has to re-discover these from scratch via new sensor views, which can be slow or fail completely. Research has long acknowledged this memory gap. Some robotic systems combine separate modules for mapping, planning, and manipulation; they build explicit maps or scene graphs where each object or location can be stored and queried. Others use recurrent networks or episodic memory buffers to hang onto past observations. But each approach has trade-offs. Explicit scene graphs or symbols may capture what objects are, but often throw away rich visual details and the fine spatial relationships needed for precise control. Pure neural-memory approaches (like storing image embeddings in a buffer) can preserve info but mixed with noise and without a clear spatial structure: the policy still has to figure out where in space an old memory is relevant. SERF’s idea is to fuse these benefits: build an implicit 3D map of features, so that the robot has a spatially grounded memory that can be updated continually. Crucially, SERF’s map is not just of the static scene – it is spatiotemporal. It explicitly represents the robot itself as part of the map, so the policy always knows how the robot’s pose relates to the environment history. In other words, SERF supports two modes of reasoning simultaneously:. Allocentric spatial memory: The map holds a persistent latent representation of the entire environment (at least the relevant parts). As the robot moves, it accumulates observations of the workspace: which objects are where, even if they are out of view right now. This is like having an internal overhead map that gets updated. When planning actions like “move to the blue cubby next to the red shelf,” the policy can refer to where those locations are in the map, not just guess from the current image. Egocentric context through embedding the robot: Every point in the robot’s body (base, joints, end-effectors) is also embedded in the same map. By combining the robot’s latent representation with the environment’s, the policy can instantly infer distances and orientations between the robot and objects in memory. For example, if the robot knows “object A” is at latent coordinate X, and its own gripper is at Y in the map, it can reason that “object A is currently 1.2 meters to my right.” This avoids having to recompute such relationships by vision alone every time. By building a 4D latent feature field that includes both environment and robot, SERF nets out the need for separate modules: the policy gets a single, rich state input that inherently encodes what has changed over time. Kim et al. spell it out as: “maintaining a unified understanding of the robot’s motion and [the] evolving environment over long horizons” is exactly the point. In practice, this means the robot is less likely to re-search for objects it already picked up or forget a subgoal it was partway toward. Constructing the 4D Feature Map. At the heart of SERF is its feature map – a function that, given any 3D point in the workspace, returns a latent feature vector (of some dimension k) summarizing what’s there. Mathematically, the map $m_\tau(x;\Theta)$ at time $\tau$ maps a 3D location $x$ to a vector in the vision-feature space (hence “latent”). But it’s not a dense function like a continuous Neural Field; instead, SERF represents $m_\tau$ by a sparse set of neural points in 3D space, each carrying a learned feature. Think of it like a point cloud with features attached, plus a decoder network to interpolate between them. The map’s neural points come in two groups: environment points and robot points. Each point $p_i$ has a 3D position and a latent feature vector $f_i$. The overall map is defined so that to query it at any location $x$, we take the nearby $K$ neural points, interpolate their features, and pass the result through a learned decoder network (e.g. an MLP) to produce the final feature $m_\tau(x)$. Concretely, the interpolation might be a distance-weighted average of the $K$ nearest point features, and the decoder just remaps that averaged feature into the space of visual embeddings. This let the map represent complex geometry or appearance cues via the features of the point set. 3D Neural Points for the Environment. The environment neural points represent the static (or slowly moving) scene. Initially, when the robot explores a new area, we lift features from its camera observations into 3D. At each time step, the robot observes its RGB-D cameras and runs a pretrained vision foundation model (e.g. DINOv3) to get a feature embedding for each image patch. Using the depth and camera pose, each image patch’s center can be back-projected to a 3D coordinate in world space, pairing a 3D point with the DINO embedding. These originate-detected points are then discretized (for instance by voxelizing space) and stored as environment neural points. Every time a previously-unseen patch appears (i.e. a voxel has an empty entry), we add a new neural point at its 3D location. We also note which object instance label (from segmentation) that point belongs to. Thus the map builds up a point cloud of the environment, each labeled and associated with a DINO feature (though the feature will be learned, not fixed to DINO). Importantly, each environment point’s learned feature $f_i$ is fixed after offline training. What we train is a common decoder (and the features themselves) via a reconstruction objective: given the true DINO embedding $y$ at $x$, we minimize $(m_\tau(x;\Theta) - y)^2$ (cosine loss). In effect, the decoder and features are learned so that the map can reproduce real vision features at all sampled locations. We also add contrastive losses to organize the feature space semantically: features of the same object category or part are pulled together, and different categories pushed apart. After this offline training on many scenes, each environment point has a feature that captures its visual semantics (e.g. “wood floor”, “red ball”), and the decoder knows how to produce scene-like embeddings from interpolated features. At run time, the robot continually updates the map’s geometry as it moves and interacts. We assume each object instance moves rigidly, so we enable the map to track object motions. Specifically, at each new time step $\tau$, we take the previous and current images and track image keypoints for each movable object instance across frames. These 2D tracks are lifted to 3D using the depth, giving correspondences $(q_k^{\tau-1}, q_k^{\tau})$ between 3D patch points of that object. We then solve a rigid registration: find the best rotation $R$ and translation $t$ in $\mathrm{SE(3)}$ that aligns the previous points to the current ones (using Iterative Closest Point, initialized by Fast Global Registration). Every environment point belonging to that object’s instance label is then moved by this $(R,t)$ transform: $p_i^{(\tau)} = R , p_i^{(\tau-1)} + t$. This effectively “drags” the neural points along with the object’s new pose. (Background points that don’t belong to any moving object stay put.). Meanwhile, new observations may introduce new points if the robot sees previously occluded areas. So the environment map both grows (when new space is scanned) and warps (as objects move), but the features of the points remain the same. 3D Neural Points for the Robot. The other half of the map is the robot itself. We predefine a set of robot neural points that sample the robot’s kinematic model. Using the robot’s URDF, we generate many 3D surface points on each link of the robot (base, arm links, grippers, etc.). Each such point is stored in its local link frame (so each point $u_j$ is defined relative to link $l_j$). We attach a learnable feature $f^r_j$ to each robot point, and give them a special “robot” label so they are treated separately from environment points. At initialization time, the learned features of robot points are trained along with the environment features together (same decoder), so that robot surfaces and environment surfaces share one coherent latent feature space. The robot points’ features start as random, but are optimized so that if you query the map at a robot surface’s world position, the decoded embedding reconstructs the actual appearance or semantics of that part. In fact, during training we render the robot’s mesh from various angles and give those renderings to the same vision network; the robot points learn to predict those embeddings from their positions. This forces the map to encode robot parts in the same feature space as objects, effectively learning part-level consistency across configurations. When the robot moves during execution, updating the robot points is trivial kinematics. Given the current joint state $s_\tau$ (base pose and joint angles), each robot point’s world position is computed by forward kinematics: if $u_j$ is in link $l_j$’s local frame, then its world position is $p^r_{j}(\tau) = R_{l_j}(s_\tau),u_j + t_{l_j}(s_\tau)$, where $(R,t)$ is the link’s transform in space. Thus all robot points coherently move as the robot moves, without changing their feature vectors. The feature decoder has already been trained to “know” what a robot surface should look like, so queries at those robot points continue to yield the expected embeddings of the robot itself, no matter how it reconfigures. In summary, SERF’s map at time $\tau$ comprises the union of environment points $P^e_\tau$ and robot points $P^r_\tau$. The only quantities that change over time are the 3D positions of those points; their learned features and labels stay fixed after training. This set of moving points, along with the trained decoder network, defines a piecewise-constant 4D latent function $m_\tau(x)$: at any query location $x$, we interpolate nearby point features $f_i$ and feed through the MLP to get a latent embedding. Because the robot and environment points are in one combined space, the map inherently captures scene geometry and the robot’s pose together. In practice this gives the policy an integrated spatial memory. Conditioning the VLA Policy on the Latent Map. Once the 4D map is available as state, the question is how to use it in the vision-language-action policy. The authors’ solution is to convert parts of the map into vector tokens, and feed those into a Transformer-based policy alongside the usual visual and language embeddings. Think of these tokens as a learned summary of the relevant portion of the map, at various scales and reference frames. This way, the VLA model sees both the live camera images and a spatial memory snapshot, all as a single context for predicting actions. Filtering and Encoding Map Points. First, the raw set of neural points is filtered down to those likely relevant to the task. The demonstrations of each task come with instance labels (for example, which objects are targets according to the task specification). The map tokenizer keeps only points whose instance labels are either the robot itself or one of the task objects; it discards generic background or structural points that the task doesn’t care about. This focuses the map on, say, the table and blocks for a stacking task, or the child’s toys for the tidying task. The remaining points (robot points plus task-object points) are passed through a point-cloud encoder – specifically, a Point Transformer network – which computes per-point feature vectors. These features embed the raw map points into a representation that can then be pooled. Essentially, the map tokenizer is a lightweight neural network that turns the 3D point set into a few summary vectors. Multi-Scale Map Tokens. SERF’s designers found it important to capture both local detail and global context, so they devised a multi-head tokenization scheme. Eight attention “heads” (parallel branches) each select a different spatial subset of the points and condense them into one token. Concretely, the eight map tokens fall into five groups:. Robot-base tokens (3 heads): These tokens capture the area near the robot’s base. Each head looks at points within a certain radius (1 m, 2 m, and 4 m) centered on the robot’s base frame. This gives a coarse-to-fine view of the surroundings right around the robot. Gripper (end-effector) tokens (2 heads): Two heads focus on points around each gripper. Each performs a 0.5 m-radius ball query around the left or right end-effector. These tokens let the model reason directly about what’s near each hand – crucial for grasping decisions. Robot-body token (1 head): One head attends to all points on the robot’s body (the union of the robot points). After passing them through the point encoder, it pools them into a single token summarizing the robot’s entire configuration. Environment token (1 head): One head pools all remaining environment points (the task-related objects) into one token. This gives a holistic summary of the current environment state. Global token (1 head): Finally, a global token aggregates every point (robot + environment). This can capture scene-level interactions or low-frequency context in one vector. Each token is produced by taking the selected subset of point-transformer features and applying attention pooling (or a simple learned reduction). The result is 8 map tokens (vectors), each encoding a different slice of the map’s information at a particular scale. Notably, the scheme omits raw coordinates from these tokens; the Point Transformer uses relative positions so that the policy doesn’t depend on absolute coordinates (improving generalization). In practice, these eight tokens now represent the key spatial memory cues: local geometry near the robot base and hands, the robot’s full posture, the environment’s summary, and global context. The Vision-Language-Action Policy. These map tokens are then fed into the vision-language-action model along with everything else. The policy backbone is based on a prompting-style Transformer (similar to the PI0.5 model from prior work, though we won’t dwell on that specific name). Concretely, at each time step $\tau$ the policy has:. Current RGB observations from cameras (encoded by a vision encoder). The discretized proprioceptive state (base pose, joint angles) as tokens. The task instruction (encoded as a learned task embedding vector). Plus the SERF map tokens $e_m$. All these are concatenated to form a single context vector sequence. In notation: $$h_\tau = \text{Concat}\bigl(E(o_\tau),;e_m,;e_s,;e_\ell\bigr),$$ where $E(o_\tau)$ are the vision-encoded image tokens, $e_s$ the state tokens, $e_\ell$ the task embedding, and $e_m=[z_1,\dots,z_8]$ are the eight map tokens. This full context $h_\tau$ is then passed into the Transformer, whose “action expert” head generates the robot’s action (or a chunk of actions). Training is done by behavior cloning from demonstrations. Each demo provides sequences of $(o_\tau, s_\tau, a_\tau)$ under a language task $\ell$, and the model learns to predict the expert actions. Kim et al. actually use a conditional flow-matching objective over chunks of actions – a kind of diffusion-based multi-step prediction – but the high-level effect is that the policy learns to map the entire observation-history and map state to the next actions. The map tokens give the model explicit memory: when it evaluates the final decision, the Transformer can attend not just to the current image but also to cues like “the red block I saw two seconds ago is in that corner” because the map token encodes that latent information. Importantly, the map is treated as part of the state in the same way as proprioceptive data. This means the policy can use allocentric info (e.g. navigate to a remembered position) or egocentric (e.g. close the gripper to a known object) seamlessly. In sum, conditioning the VLA model on SERF’s 4D map provides both local, immediate context from the eye and a persistent, spatial memory from the map. Evaluation on BEHAVIOR-1K Tasks. SERF is evaluated in simulation on long-horizon mobile manipulation tasks drawn from the BEHAVIOR-1K benchmark. BEHAVIOR-1K is a recent collection of realistic household tasks (1,000 tasks in total) in physics simulators, designed to test general robotics abilities. Each task involves multiple subgoals and object interactions (for example, picking up objects scattered through a room and placing them on specific shelves). The tasks require the robot to navigate through a simulated home environment and manipulate objects – exactly the setting where memory of space and objects matters. The authors train policies by behavior cloning on demonstration data for a handful of tasks. For a fair comparison, they use the same VLA backbone and training regime for both the SERF-augmented policy and the baseline. The baseline (“image-only VLA”) sees the exact same camera images, proprioception, and language, but no map. In practice this means its input is identical except the map tokens $e_m$ are omitted. Both policies are then tested on three representative long-horizon tasks from BEHAVIOR-1K involving navigation plus multi-object transport. (The paper refers to them as Task 21, 22, 26, but the exact tasks names aren’t as important as their nature: each requires multiple pick-and-place steps through a room.) Performance is measured by task progress, a standard metric (fraction of subgoals completed). Improved Task Completion. The results show a dramatic boost from SERF. On average across the three tasks, the map-conditioned policy’s task progress was 58.7%, compared to 44.0% for the image-only baseline. That is a relative gain of about 14+ percentage points. In practical terms, the SERF agent completes many more of the required subgoals and gets significantly further in the task. To give an example of what this means: one common outcome with the image-only agent is that it would turn back and forth or wander redundantly, failing to efficiently reach some objects it had already seen. With the latent map, the SERF agent tended to go almost straight to each subgoal. Indeed, the authors note that SERF “reaches subgoals faster by following more direct trajectories”. Essentially, the map tells the policy “the blue block you picked up two minutes ago was here on the left wall of the room,” so it won’t lose time re-scanning or backtracking. Better Generalization and Robustness. Beyond absolute performance, SERF strongly improves robustness under changes. The authors tested out-of-distribution scenarios: for example, they altered the environment layout or object placements in ways not seen in training demonstrations. In one such test, task targets were moved to a region the baseline had never encountered. On these “scene-shift” tasks, performance with the image-only model plummeted (only ~28% progress), whereas SERF’s policy managed to score about 51.0%. In other words, even though the numbers are low overall (it was a much harder setting), SERF more than doubled the success rate relative to the baseline. Qualitatively, this makes sense: an image-only agent can only act on the current view, so if a bookshelf is moved or an extra obstacle appears, it may be totally thrown off. The SERF agent, on the other hand, has a stored map that it knows should match the task’s goal locations, so it can adapt its path. Similarly, when random “distractor” objects were added (e.g., extra toys not in the instructions), the baseline would often chase the wrong object. SERF’s latent map allowed it to ignore irrelevant items (they wouldn’t have matching instance labels in memory) and focus on the actual targets. Error Recovery. One striking advantage was failure recovery. Robotic pick-and-place is prone to errors like dropping an object during transit. With only image memory, a dropped object means the policy must suddenly find it all over again. The SERF approach helps because the map already contains the object’s last known location. In tests, whenever the robot accidentally dropped an item, the SERF policy’s trajectory would usually veer back to retrieve it, whereas the baseline sometimes gave up on that object entirely. This shows up in the numbers: SERF was able to complete more subgoals even after such mishaps, which contributes to its higher task progress. Trajectory Efficiency. Efficiency is another metric where SERF excels. The learned policies output an action chunk (a sequence of base and arm controls) at every step via a diffusion-based model. The image-only agent often produced longer or less-direct action sequences between waypoints, whereas the SERF agent’s latent map basically “points” it along the optimal route. For instance, when tasked with picking an object across the room, the SERF agent moved straight there. The baseline, by contrast, sometimes approached in a convoluted path, possibly because it was reacquiring vision targets on the fly instead of knowing exactly where in space the goal lies. This matches the reported observation that SERF “follows more direct trajectories”. In practice, these quicker paths translate to finishing subgoals in fewer time steps. In short, across every dimension SERF, with its 4D memory, beats the “reactive” baseline. The added ability to remember past observations and reason about the scene spatially makes the policy smarter—almost like giving it a map and compass for the navigation part of the task. Why the 4D Map Matters. Putting it all together, SERF’s gains emphasize a few key points about embodied AI. First, where you look is often not enough in complex tasks. A static VLA policy might know exactly what to do when it sees the right object right in front of it, but without a map it can’t strategize ahead. SERF bridges that gap by making long-term geographic memory a seamlessly integrated part of the policy’s input. Second, the explicit inclusion of the robot’s body in the map is valuable. When the policy tokenizes SERF, it gets both global and local context: it knows not only “where the chair is” but also “where I am relative to it.” This dual perspective (allocentric plus egocentric) means spatial relations are automatically present. Many prior VLA systems would have to infer such relations by comparing a map-plus-planning module or by trial-and-error; SERF’s design makes them explicit. Third, using a latent map (via neural points) is flexible. It captures rich visual features (thanks to training with a vision-model encoder) without committing to a particular semantic label or occupancy grid. Moreover, updating the map with object motions is straightforward by rigid transforms on points. Had the authors used a dense voxel grid, it would be much harder to track moving objects; with points, each object just carries along its points as a rigid cluster. Finally, by conditioning a powerful Transformer policy on tokenized map features, SERF leverages recent advances in large VLMs. The map tokens act like extra “memory cells” pretrained to summarize space. It’s reminiscent of how large language models use memory or retrieval, but here it’s geometric memory. The result is a robot that can plan in space using both its eyes and its “mind’s eye” (the SERF map). Implementation Notes and Limitations. A few caveats are worth mentioning. SERF relies on instance segmentation or labels for objects, as it uses those labels to group environment points and filter map tokens. In simulation this is available, and the experiments assume perfect segmentation of task objects. In the real world one would need a segmentation model (or CLIP mask, etc.) to label objects, and errors there could affect the map. Also, the system assumes each tracked object moves rigidly. This works for typical block-and-shelf tasks, but if objects deform or move in complicated ways (e.g. a folded cloth or a swinging pendulum), the simple SE(3) updates of SERF would break down. Extending the map to elastic or articulated objects would be an interesting challenge beyond current scope. On the positive side, the method’s components (neural point map, keypoint tracking, forward kinematics) are modular and could be replaced by more advanced alternatives as needed. For example, instead of a classic 3D keypoint tracker and ICP, one could use a learned flow field to update points. Likewise, any pretrained vision model could substitute for DINOv3 to generate patch features during training. The key idea – embedding space via neural points with dynamic updates – is robust to such changes. From a learning perspective, the policy is trained by supervised imitation. This means SERF isn’t learning the mapping itself online; the map parameters (features and decoder) are fixed from a prior training run. In principle one could imagine co-training the map and policy together, or finetuning the map on task data, but that was not done here. In practice, the map learned offline on general scenes sufficed to improve performance on the specific test tasks. Conclusion. SERF showcases that spatiotemporal latent mapping can substantially elevate vision-language-action models for embodied robots. By giving the policy an explicit 4D memory of the world and of itself, SERF empowers long-horizon decision-making that image-only methods cannot match. The robot never has to blindly guess “where did I see that object?” – the answer is latent-coded in its internal map. As a result, it completes more of its mission, more efficiently and robustly. For robotics researchers and practitioners, SERF suggests an exciting direction: instead of purely reactive multi-modal policies, build persistent latent maps that the policies can query. This might extend to coupling with language-based planners (so the robot can say “go to the couch I remember,” or “fetch the last red mug I saw”) or to more complex scenes. The gains on BEHAVIOR-1K indicate that spatial memory greatly broadens what we can solve. Looking forward, one could imagine integrating SERF-like maps with learned topological or semantic maps, or combining them with world-model-based planning. The idea of embedding the robot in the same feature space as the environment could even inform multi-robot scenarios, where each agent’s latent position is encoded in a shared map. At its core, SERF points toward a vision of robot foundations: a rich, persistent scene representation, fused with advanced vision-language reasoning, to enable truly long-term autonomy in dynamic environments. Sources: The above summary is based on the UC San Diego team’s recent preprint on SERF, which describes the map construction, VLA policy, and BEHAVIOR-1K experiments in detail. The key idea is to build a spatiotemporal neural-point map of the environment and robot, and condition a vision-language-action model on it. 官方 results show the SERF-based policy significantly outperforms image-only baselines on long-horizon mobile manipulation tasks. All citations refer to passages from the paper (arXiv:2606.12956).