Latest / Embodied AI 101 / LingBot-Vision: Self-Supervised ViT with Masked Boundary Modeling

Transcript

LingBot-Vision: Self-Supervised ViT with Masked Boundary Modeling · Embodied AI 101

• LingBot-Vision: Self-Supervised ViT with Masked Boundary Modeling. Robots and embodied agents see the world differently than typical vision systems. For a robot, an image isn’t just a collection of objects to recognize – it’s a structured scene of surfaces, edges, and depth relationships. To navigate and act, a robot must know where object boundaries lie, where depth discontinuities occur, and how things are spatially arranged. Yet most modern vision foundation models (like CLIP, DINO, or masked autoencoders) have been optimized for semantics or image-level invariances, often at the expense of fine-grained spatial detail. A cat is a cat to these models, but they don’t necessarily capture where the cat ends and the floor begins, or how far away it is – information that embodied AI needs. The new work LingBot-Vision (Robbyant/Ant Group, 2026) tackles this gap head-on. The authors take a Vision Transformer (ViT) and train it with a novel self-supervised objective focused on edges and geometry, which they call Masked Boundary Modeling. Instead of randomly masking patches (as in MAE) or only distilling global semantics (as in DINO), the student model is forced to predict the most informative parts of an image: the pixels on object contours and depth discontinuities. Put differently, the model learns to pay attention to edges from day one, treating them not just as outputs but as native learning signals. The result is striking: without any labels, LingBot-Vision’s frozen features achieve state-of-the-art results on dense perception tasks. A 1.1-billion-parameter LingBot-ViT outperforms or matches much larger models on monocular depth estimation and semantic segmentation, and its features are so spatially consistent that simple label propagation solves video object tracking nearly as well as 7-billion-parameter alternatives. In short, this model shows that a geometry-centric pretraining can yield “space native” vision that is ready for embodied applications. In this episode, we’ll break down how LingBot-Vision works, why it works, and why it matters for embodied AI. We’ll contrast it with prior pretraining approaches and highlight how boundaries become part of the learning signal. We’ll also touch on its performance: e.g. on the NYU-Depth v2 benchmark, LingBot-Vision’s linear probe has lower error (RMSE 0.296) than even a 7B-parameter DINOv3 model (0.309), despite using far less data and compute. Finally, we’ll explore how to use and adapt this model – from initializing your depth or segmentation system to even doing training-free label propagation in videos. Why Boundaries and Geometry Matter for Robots. Imagine a robot grabbing a cup. It doesn’t just need to know “there’s a cup” (the cat vs. cup classification). It needs to know where the cup’s edges are, how it sits on the table, which way the handle opens, how far away it is, and where the gaps or occlusions are. Those are spatial and geometric properties. In human vision, edges and contours are fundamental cues for depth, occlusion, and segmentation. For example, an occlusion edge usually coincides with a change in depth or surface orientation. A doorframe’s outline tells you where the doorway is. A sidewalk curb delineates safe walking space. The LingBot-Vision authors point out that dense spatial perception – recovering structured, metric information from pixels – is essential for physical intelligence. Yet today’s large self-supervised models tend to favor semantic invariance. That is, they learn representations robust to color/lighting changes or local distortions (which is great for classification), but the same invariances can blur out fine geometry. For instance, methods like DINO (distillation) or masked autoencoders focus on object identity or reconstructing appearances, not on explicitly enforcing that “the boundary between object A and B must align with an edge in the image.” As the paper notes, “Modern visual foundation models tend to prioritize semantic invariance, often at the expense of detailed spatial understanding”. Boundaries and shapes are typically only considered as outputs (like segmentation masks) and are not used natively during pretraining. WingBot-Vision changes this by flipping that philosophy: boundaries and shape discontinuities become the learning target itself. The insight is simple: object and surface contours are extremely information-rich. They often coincide with depth changes, occlusions, and segmentations – exactly the things a robot needs. If a model can learn to predict edges as part of its training objective, it will inherently build more geometry-aware features. The authors describe boundaries as “fundamental organizing signals for learning dense representations,” not just optional outputs. The Lens of Masked Boundary Modeling. LingBot-Vision’s key innovation is a training paradigm called Masked Boundary Modeling (MBM). At a high level, it’s a teacher–student self-distillation scheme (in the style of DINO) with a twist: the teacher automatically finds edge-related patches in the image and forces the student to mask them out. Thus the student has to use contextual cues to predict them, essentially learning edges by doing. Let’s unpack that process. In this setup, there’s a Vision Transformer acting as the teacher (an exponential moving average of the student) and another as the student. Both process images, but under different augmentations: the teacher sees a full (global) view, while the student sees a masked version. Normally in DINO, the student tries to match the teacher’s semantic token distributions (class tokens) and patch-level features. Here, the student still learns global semantics, but crucially some masked tokens get special treatment. Teacher’s role: The teacher network, as it reads the global image, has an extra boundary head that predicts a dense “boundary field.” This is not just an edge map; it’s a sub-pixel geometric representation borrowed from line-segment detection work. Concretely, for each location (sub-pixel) the teacher outputs a vector with entries like distance to nearest line and line orientation (angle). Intuitively, this encodes where edges are and which way they point. Because early in training the network hasn’t learned anything, these raw boundary fields may be noisy. But here’s the clever part: they combine this field with a set of “corner points” (sparse anchors predicted by a frozen corner detector network) to decode line segments. Essentially, they do a Hough/voting scheme: pixels whose boundary-field vectors agree vote for line segments through corners. Even if the teacher’s output is initially random, the redundancy yields plausible line candidates (as each true edge will get many weak votes adding up). Next, the teacher applies a statistical a-contrario validation to filter only the real edges. Under the null hypothesis of “no structure,” one expects boundary orientations to be uniform. If the distribution of predicted orientations along a candidate line significantly deviates from uniform, that segment is likely real (parameter-free number-of-false-alarms test). This cleans out false positives. The remaining validated line segments are rendered back into a dense boundary field (with signed distance and orientation at each pixel). Finally, from that refined teacher field the model marks which patch tokens lie on or near edges. These become the boundary tokens. In practice the teacher produces a mask of patch indices covering the detected edges (in addition to any random masking). Student’s task: The student processes the same image but with those boundary patches forced to be masked out (along with whatever random masks). In other words, the hardest, most information-rich patches are taken away from the student. The student must then reconstruct what they should be. It does this via two heads:. A semantic distillation head, just like in DINO: it tries to match the teacher’s (sharpened) class token and patch token distributions for non-boundary masked patches. This preserves the usual self-distillation learning of global and local semantics. A boundary-field head: for every patch that was masked because it lies on a detected boundary, the student must predict the teacher’s boundary field. But instead of regressing continuous values (which would collapse), they discretize each field channel (distance, two angles) into bins and treat it as classification. The teacher produces a soft label (via narrow “Gaussian” encoding over bins). The student’s boundary head thus outputs a probability distribution per patch over these bins, using a cross-entropy loss. Crucially, after identifying which tokens are boundary-related, the losses are routed by geometry. Masked patches at boundaries get the boundary loss; other masked patches get only semantic distillation. This turns the pretraining “tension” into cooperation: the model no longer has to trade off between semantics and geometry. Instead, it is forced to do both. As the paper puts it, boundaries – the least redundant regions – become “the hardest and most informative prediction targets.” In effect, MBM says: “From now on, if there’s an object edge, we’ll hide it from you. Learn to predict it.”. To stabilize this, the boundary prediction is done categorically, so it can be centered and sharpened like the semantic distillation (no collapse). In practice, each patch token is fed through a small MLP boundary head that upsamples it (e.g. 2×) and outputs per-pixel features. These are projected against learned unit-norm “bin prototypes” (like DINO’s prototype heads) so that the teacher-to-student transfer on boundary orientation/distance is simply cross-entropy. The cooling/sharpening tricks from self-distillation now also apply to the boundary predictions, and “no-structure” regions just have a uniform target. Remarkably, this pipeline works from scratch. The teacher network starts with random weights, yet valid edges emerge almost immediately. As shown in the paper, even a totally unlearned boundary field plus corners will decode visible line segments (because many pixels give noisy votes that average out). So the model bootstraps on its own outputs. During training, the teacher gradually becomes a better edge detector, and the student follows by learning to encode spatial cues. In short, the model teaches itself to see edges without any segmentation labels or external edge detectors. Scaling Up: The LingBot-Vision Models. The authors name their scaled-up model LingBot-Vision, releasing several sizes (ViT-Small through ViT-Giant) under Apache 2.0. The flagship is a ViT-G/16 — about 1.1B parameters. They pretrain purely self-supervised on a 161-million-image corpus (carefully curated, similar to DINOv2’s 142M). By contrast, DINOv3 used ~1.69 billion images. The training used large batches (3K–4K) and multi-stage resolution: for example, LingBot was trained for 300K iterations at 256px crops, then 100K at 448px, then 100K at 640px. Even with this relatively modest schedule, they emphasize it’s only ~1/3 of DINOv3’s iterations. In practice that means you can train a geometry-aware ViT with less compute than some alternative SSL routes. During pretraining, they sample several views per image (global and local crops) per usual DINO prescriptions. The teacher-flagged boundary masks are only applied to the global views (higher resolution). Local crops still use standard distillation. They also add a few bells and whistles: an iBOT-style masked-token dense loss and a “KoLeo” loss for feature uniformity, but these are auxiliary. The core is that every epoch the teacher (EMA of the student) is actively finding edges in each image and carving them out as masked tokens for the student. After training the giant model, the authors distill it to smaller ones (ViT-L/16, ViT-B/16, ViT-S/16) by standard teacher-student training (keeping the MBM losses). They release all these checkpoints on Hugging Face (e.g. etc.), along with code to load a pure backbone and extract normalized patch tokens. The entire engineering bundle is open-source. Performance: Depth, Segmentation, and Beyond. The test for any new vision backbone is how well it does on downstream tasks, especially in a zero-shot or frozen-features setting. LingBot-Vision clearly shines on tasks requiring spatial acuity. Depth Estimation. Monocular depth estimation is a litmus test for geometric understanding. The authors evaluate linear regression on frozen patch tokens for two standard datasets: NYU Depth v2 (indoor) and KITTI (driving scenes). The results are striking: LingBot-G/16 achieves NYU RMSE 0.296, which is the best ever among published SSL models. Even 7B-parameter DINOv3 only got 0.309 under the same protocol. On KITTI, LingBot’s RMSE is 2.552, making it the top performer under 2B params (only the 7B DINOv3 at 2.346 is better). To put it another way, the 1.1B LingBot cuts NYU error by ~13% relative to any other 1B model (13% lower RMSE than the previous best same-scale model). Why the big gain? Depth discontinuities align with object edges, so a model that was trained to honor boundaries can basically "read out" depth more smoothly. The paper observes that a linear depth decoder works best only if features are "smooth within a surface and change sharply across occlusions". LingBot-Vision’s features exhibit exactly that: in qualitative PCA plots, planar regions map to coherent color blobs and transitions precisely align to contours. In simpler terms, the model has already done the “segmentation” implicitly, so a linear head easily infers depth. Even the smaller distilled models show big wins. A LingBot-ViT-L/16 (300M params) yields NYU RMSE 0.310, nearly the same as 7B DINOv3’s 0.309. That means a sub-500M model matching a 7B one in depth! Thus, the geometry awareness compresses well. In summary, LingBot-Vision dominates depth estimation: it sets state-of-art RMSE on NYUv2 and is very close to it on KITTI, outperforming larger/self-teaching models. Semantic Segmentation. How about segmentation? Boundaries should help here too. The authors test a simple linear classifier on frozen tokens for ADE20K (panoptic scene parsing), Cityscapes, and Pascal VOC. Again, LingBot is excellent. On ADE20K, the LingBot-G/16 gets about 53.5% mIoU, just a hair below the 54.8% of a distilled DINOv3-H (which itself was distilled from a 7B model). On Cityscapes, LingBot 79.6% vs DINOv3’s 79.5% (basically equal), and on VOC12 (20 classes) it exceeds DINOv3’s 85.8 with 87.5. Against a same-size DINOv2 baseline, LingBot improves by roughly 4 points on each dataset (e.g. ADE20K 53.5 vs 49.5). Notably, these results were achieved without any segmentation labels or specialized finetuning – just by a linear probe on the frozen features. For context, competing SSL models at similar scale (1B) range lower. For instance, a video-based pretraining (V-JEPA 2.1) only got ~47.8 on ADE, and even CLIP-like Saiple SigLIP2 was in the 42% range. Only DINOv3 with a giant teacher or multi-headed specialized models surpassed the LingBot on segmentation. The authors emphasize that reaching ~54 mIoU on ADE20K from scratch is remarkable and comparable to specialized dense-pretraining works, but here it all comes from the single MBM objective. In effect, LingBot-Vision delivers segmentation-quality features just by learning edges. Video Object Segmentation & Token Tracking. What about temporal consistency? The team tested training-free video object segmentation. They take the mask of the first frame and propagate it to new frames by feature matching (each query patch attends to the most similar patches in the next frame). On DAVIS-2017, LingBot-Vision (frozen) achieved a J&F-Mean of 70.0%, and on YouTube-VOS 73.5%. These are essentially on par with DINOv3’s ViT-H (71.1 / 74.0). In fact, LingBot-Vision is the best among all sub-2B models on these benchmarks. So once again, the boundary-focused features serve well for correspondence in video. Digging deeper, the paper even tracks individual tokens over video. By selecting a few "boundary token" queries in frame 1, they follow them via frozen-feature cosine similarity. Impressively, the queries remain well-anchored on the same object edge through camera pans, object motion, etc. For example, when a robot arm or a cat moves, the patch features on the gripper or the cat stay closest to the original query. This shows that LingBot’s token-space is temporally stable on edges, even without any video training. It’s a qualitative demonstration that the model’s geometry representation is persistent over time. Image Classification (ImageNet). Of course, dedicating capacity to boundaries could risk losing semantic power. But in LingBot-Vision it’s only a slight trade-off. On ImageNet-1K, the frozen backbone yields ~86.3% top-1 with a linear probe, and ~83.4% for k-NN. That’s roughly on par with DINOv2’s best (87.0 / 83.7) and a few points below the top DINOv3-7B (87.9 / 85.7). In other words, LingBot retains strong scene understanding overall, with only marginal drop in classification accuracy. The authors note that the main remaining gap to pure classification models concentrates on image-level details, consistent with their design choice to allocate more model capacity to local structure. But as the dense task results show, the trade is very favorable for embodied vision. Distilled Models. Finally, the LingBot-Vision family includes smaller distilled models (ViT-L, B, S). These all preserve the teacher’s “spatial edge focus.” For instance, the 0.3B ViT-L student matches DINOv2 in ImageNet accuracy (86.4% linear) while cutting NYU depth error by 25% (NYU RMSE 0.310 vs DINOv2’s 0.411). Compared to a same-size DINOv3-L, it sacrifices only ~1 point in ImageNet but clearly leads on depth. Even the 0.3B student attains NYU RMSE 0.310 – essentially the same as 7B DINOv3’s 0.309! Table-wise, each size of LingBot outperforms any equivalent-size DINOv2 or SigLIP model on the dense tasks, while staying within ~1% on classification. Thus the “geometry-first” signal compresses efficiently into smaller models. Overall, the metrics paint a clear picture: Masking boundaries in pretraining yields large gains in depth and segmentation with only minor semantic costs. Importantly for practitioners, these results were achieved with frozen backbones and linear heads. In practice one can fine-tune or add small decoders on top of LingBot-Vision for even better performance on tasks of interest. Putting it Together: Why it Works. Why does LingBot-Vision learn such great dense features? We can highlight a few key points:. Edge-Focused Curriculum: By design, the student always sees the most informative region as a learning challenge. This “boundary-forcing” makes the model focus on regions of high spatial information. It’s akin to a curriculum that says “you don’t get easy flat background, you must learn the tricky edges.” This forces the network to encode geometry from the start. Dual Objectives in Harmony: The boundary routing trick elegantly avoids conflict. Masking edge tokens for the boundary loss means the semantic distillation need not try to predict ambiguous mixed-edge pixels (which are hard for a global target). Hence spatial detail and object-level semantics co-emerge rather than competing. As the paper notes, “boundary-aware and semantic representations co-emerge rather than compete”. Dense Redundancy: The use of dense “boundary fields” (over-parameterized line representation) is critical. It allows edges to be discovered even from noisy beginnings. The teacher effectively bootstraps better boundaries from scratch, so no external edge detector is needed. In classical terms, each real edge is supported by many pixel votes, making its detection robust even early on. Stability via Discretization: Turning boundary regression into classification bins (and using a “no boundary” uniform class) prevents collapse and enables the normal centering/sharpening of self-distillation to apply. This is a smart hack – continuous field regression in a moving-teacher loop often diverges. By discretizing with a soft one-hot encoding, the model simply uses cross-entropy and inherits stability. Curve-fitting a continuous line field on top of this is easy afterwards. Data Efficiency: It is worth noting that LingBot-Vision achieved these results with a fraction of typical SSL data. Their 161M images are only about 10% of DINOv3’s 1.69B, and they trained far fewer epochs. The boundary signals appear to concentrate learning on high-value information, which likely improves sample efficiency. The paper points out that LingBot’s training “spends less than one third of the samples DINOv3 processes”. In summary, the combination of boundary detection and masked prediction is what gives LingBot-Vision its strength. It also adds only modest overhead: a small extra head and some line-finding logic during training, but at inference you simply drop the boundary machinery and just use the ViT backbone. Thus adoption cost is low. Using and Adapting LingBot-Vision. The LingBot-Vision models are fully open-source: code, weights, and docs are available on GitHub and Hugging Face. They provide backbone-only PyTorch checkpoints (the file) which output normalized patch-token features. To use, you can follow their guide (see the and ). For example, their package lets you easily load a pretrained variant:. These are the patch embeddings (like DINO tokens). You can then e.g. train a small linear layer for depth or segmentation. The authors suggest several direct uses:. Depth Estimation / Completion: Feed the patch features into a lightweight decoder (or even a single linear regression) to predict depth. They recommend replacing the encoder in the LingBot-Depth stack with LingBot-Vision to get “LingBot-Depth 2.0” – which indeed was how they got huge gains on depth completion benchmarks. In practice, this means your old depth network just gets a better initialization. Semantic Segmentation: Similar idea – freeze the backbone and train a small classifier per pixel. Because the features align well with contours, the segmentation boundaries should snap neatly to objects. The paper shows gains even without fine-tuning the backbone. Video Object Segmentation / Tracking: If you have object masks in one frame, you can propagate them by nearest-feature matching. The authors did this without any learning: just compute top-k cosine similarity in patch space. Given LingBot’s spatial consistency, this works impressively. So in your video tasks, you could leverage the fixed features as a “label propagation engine.”. Feature Visualization: The learned embeddings should reflect objects as distinct clusters (as seen in their PCA visualizations). You can project patch features to RGB (via PCA) and expect to see coherent regions. This can be handy for diagnostics or interactive debugging (like watching whether a robot’s vision sees objects). Backbone Initialization: Of course, one can finetune. The authors point out that better pretraining compounds with more downstream data, rather than washing out. So initialize your dense-prediction network with LingBot-Vision weights and train on your data — you will likely converge faster or to a better solution on things like refinement of depth, normal estimation, or even some shape-based recognition. On-Device Use: Note that the authors provide smaller distilled models precisely for constrained settings. The ViT-L/16 (~300M) or ViT-B/16 (~100M) may be viable for on-device inference. They still give big dense improvements. It’s also worth highlighting practical investigatory steps the authors advise (from their engineering notes): run a quick ImageNet linear probe or k-NN on your setup to ensure the model loaded correctly, and test on a few domain images for sanity. Check especially those edge cases: transparent objects, clutter, motion blur – if boundaries fail there, it’s not the model’s fault (likely annotation or calibration issues). Related Work. LingBot-Vision sits at the intersection of several threads in vision SSL:. Self-Distillation (DINO line): It builds on methods like DINO, iBOT, and DINOv2/v3, which train a student to match an EMA teacher’s outputs. Those methods found that semantic layouts and some edges “emerge” in attention maps spontaneously, but they never directly use edge information. DINOv3 even had to add a “Gram anchoring” term to keep dense features from drifting. LingBot’s approach is like a selective augmentation to DINO: it gives the distillation a spatial objective. Masked Image Modeling (MAE, BEiT): Those reconstuction-style methods tend to produce sharp low-level detail but often at cost of class semantics, or they require large finetuning. MBM uses masking too, but in a targeted way (mask edges only) and adds a classification loss instead of pixel MSE. So it’s sort of a hybrid between masked modeling and distillation. Edge/Geometric Supervision: There is prior work on self-supervised geometry, but often with explicit outputs or synthetic tasks. For instance, contrastive methods like DenseCL or PixPro aimed to align patch features across views to encourage spatial coherence – they needed complicated mining or many negatives, and still didn’t explicitly focus edges. Some older works even used self-supervised depth or ego-motion, but those rely on video and geometry sensors. LingBot’s novelty is to pull boundary learning directly from the model’s own predictions. Vision-Language Models (CLIP, SigLIP): Interestingly, vision-language pretraining (like CLIP or SigLIP) tends to degrade dense performance (they optimize entire image/text alignment, which washes out detail). LingBot takes almost the opposite tack: no language, only vision, and injecting a spatial prior. Video Pretraining (V-JEPA, MaskProp): Video-based SSL can capture some spatial dynamics, but LingBot shows that even image-only pretraining with the right inductive bias can rival or exceed them for geometry. (For example, V-JEPA 2.1 on image segmentation trails LingBot by ~5% on ADE). Prior Edge-Centric Methods: Classical computer vision long ago recognized edges (Canny, HED, etc.), and some modern neural nets predict edges given labels. But those needed supervision. LingBot is like an unsupervised edge detector that actually uses edges for general representation. It’s unique in embedding a line-segment field learning into masked SSL. In short, LingBot-Vision extends the locked-at-large-scale training trends in vision but re-orients them toward geometry. Its results suggest that explicitly teaching an SSL model about boundaries can yield powerful dense features. It’s an exciting direction for future embodied-vision research: perhaps combining this with point cloud pretraining, or using it to guide attention in multi-modal agents, or even extending masked boundary modeling into 3D (with multi-view images). But as-is, LingBot-Vision is already a major new foundation model for any spatial-prediction task. Conclusion. To wrap up, LingBot-Vision introduces a compelling idea: make your vision backbone learn edges first. By identifying and deliberately masking boundary patches during self-supervised training, the model learns highly structured, geometry-aware features. Robbyant’s results show that this pays off dramatically on dense perception. Without tuning any task-specific hyperparameters beyond a linear probe, LingBot-Vision’s features outperform much larger models on depth and segmentation accuracy. For practitioners in robotics and embodied AI, LingBot-Vision is a new tool in the toolbox. The upside is twofold: you get the convenience of a pretrained backbone, plus it comes “specialized” out-of-the-box for spatial tasks. You can drop it into depth or segmentation pipelines for better performance, or even use its raw features to propagate labels in video, as the authors demonstrated. And because it’s open source, one can experiment: try finetuning it on your robot’s sensor data, or incorporate it into a larger LLM-based vision system to give it better spatial grounding. Looking ahead, this work suggests a broader theme: spatial priors might deserve more explicit roles in foundation models. Perhaps next we’ll see similar ideas for surface normals, for physics cues, or for contact affordances. In any case, LingBot-Vision already proves that emphasizing “what’s at the edges” can make a foundation model much more useful for embodied agents. References: The core ideas here come from Zelin Fu et al., "Vision Pretraining for Dense Spatial Perception" (2026), which introduces LingBot-Vision and masked boundary modeling. We have drawn on their paper’s experiments and on the Robbyant model documentation to summarize the method and results. (Those interested should see the paper for full details, data splits, and code availability.).