Latest / Embodied AI 101 / When Every Weld Looks the Same: Language-Labeled Graphs for Industrial Localization

Transcript

When Every Weld Looks the Same: Language-Labeled Graphs for Industrial Localization · Embodied AI 101

• When Every Weld Looks the Same: Language-Labeled Graphs for Industrial Localization. Natural language-labeled keypoint graphs for industrial object localization • Patrick Tirler and Justus Piater • Universität Innsbruck and Progress Software Development GmbH • Machine Vision and Applications, Volume 37, Article 145 • 2026. Industrial pose estimation becomes a very different problem when the object is seven thousand pixels wide, contains hundreds of nearly identical welds, and may bend or change composition between instances. In that setting, “find the left elbow” is not a useful abstraction. The relevant prompt may simply be “welding point,” repeated dozens of times. Appearance and language can identify the type of point, but not which point it is within the assembly. Patrick Tirler and Justus Piater address precisely this regime. Their central proposal is to describe an expected object as a typed keypoint graph: nodes are keypoints labeled in natural language, while edges are pairwise relations that are also labeled in natural language. A modified Grounding DINO detects candidate keypoints and predicts relations between them; a separate graph-matching algorithm then assigns those candidates to the specific nodes of the requested object graph. The paper was published on August 28, 2026, and releases both its implementation and eight industrial datasets. The result is not a general-purpose object discovery system, nor does it directly output metric 6D poses. It assumes that the user already knows which object graph should be present and asks the model to localize its nodes in a 2D image. Within that scope, however, it offers a compelling answer to a problem that standard category-agnostic pose estimation largely sidesteps: how to localize composite, repetitive structures whose individual parts do not have unique names or appearances. Why conventional category-agnostic pose estimation breaks in factories. The category-agnostic pose estimation, or CAPE, literature begins from a sensible premise: instead of training a new pose model for every object category, provide a support definition of the desired keypoints and use one model across many categories. POMNet introduced the MP-100 benchmark and treated CAPE primarily as feature matching between annotated support images and query images. CapeFormer improved this with a two-stage proposal-and-refinement design. PoseAnything represented the support skeleton as a graph and inserted graph convolution into a transformer decoder, allowing structural information to help break symmetries and recover occluded points. CapeX then removed the need for a support image by attaching natural-language descriptions to graph nodes. These systems are well matched to benchmarks where most keypoints have distinctive meanings: the nose of an animal, a chair leg endpoint, a wheel center, or the corner of a garment. Even when left and right are visually symmetric, the graph and broader object appearance can often resolve the ambiguity. Industrial assemblies violate that assumption. A reinforcement cage may contain many “welding points” and many visually identical bar endpoints. Assigning ordinal names such as “welding point 17” would make the language meaningless, while assigning the same semantic label to all of them leaves a text-only detector unable to distinguish their identities. A fixed, untyped skeleton is also limiting because industrial assemblies contain several kinds of connectivity: two points may lie on the same longitudinal bar, the same transverse bar, the same lattice segment, or merely the same component. This paper’s conceptual advance is to make those relations explicit and promptable. PoseAnything and CapeX use graph structure to improve a learned pose decoder, but their graph edges primarily specify a fixed skeleton. Tirler and Piater instead ask the network to recognize what kind of relation holds between each pair of detected points. Graph topology is no longer just an architectural prior. It becomes part of the semantic input and output. The object representation: language on both nodes and edges. Suppose an assembly is modeled with 20 welding points. Every node can legitimately share the label “welding point.” Their identities arise from the edge pattern around them. One pair might be joined by a “longitudinal bar segment,” another by a “transverse bar segment,” and a third by a “lattice bar segment.” The target graph says which relation should connect which nodes. The neural network first detects all image locations that look like any requested node type. It then scores every candidate pair against each requested relation description. This produces a predicted graph whose nodes carry keypoint-class probabilities and whose edges carry relation probabilities. Classical graph matching searches for an assignment from the target graph to this predicted graph that jointly agrees with both kinds of evidence. This separation is important. The network is not forced to decide that one visually generic weld is specifically node 12. It only has to answer two easier, more transferable questions:. Is this image location a welding point?. Are these two candidate points connected by the requested type of segment?. The combinatorial node identity is deferred to graph matching. The framework presently supports symmetric pairwise relations and excludes self-relations. It therefore fits statements such as “connected by the same bar,” but not directed relations such as “upstream of,” ordered chains, ternary constraints, or asymmetric attachment roles. Nevertheless, typed symmetric edges already provide much more representational freedom than a single fixed skeleton. Turning Grounding DINO into a keypoint-and-relation model. The implementation begins with Grounding DINO, an open-vocabulary detector that aligns text tokens with image regions. The authors replace bounding-box prediction with point localization and add a relation-prediction branch. The resulting system has four main neural components: text and image backbones, a multimodal feature enhancer, a keypoint decoder, and a relation decoder. Shared visual and textual features. A pretrained BERT encodes both node descriptions and relation descriptions. Its parameters remain frozen during training. A Swin Transformer Tiny image backbone extracts multiscale visual features. The node and edge text features are concatenated and passed through six Grounding DINO-style feature-enhancer layers alongside the image features. Each layer combines text self-attention, deformable image self-attention, bidirectional text–image cross-attention, and a feed-forward network. After enhancement, the features are separated again into node-language features, relation-language features, and image memory. This shared enhancement stage matters because a phrase such as “end of bar” is not mapped to a fixed visual template independently of the image. Its text representation and the image representation are mutually conditioned before decoding. The ablations later show that weakening or removing this feature enhancer produces one of the largest drops in keypoint detection. Keypoint queries instead of box queries. The model initializes 900 keypoint queries. Each query contains a learnable 256-dimensional content embedding and a positional component. Grounding DINO normally represents position as a box; here, only the two center coordinates matter. Initial query locations are selected from image positions with strong image–text correlations. Four decoder layers then refine the queries. Self-attention allows queries to coordinate so that they do not all converge on the same prominent point. Deformable image cross-attention gathers local visual evidence, text cross-attention aligns each query with the node prompts, and a feed-forward block updates both content and position. For every final query, its position becomes a candidate keypoint coordinate. Its class scores are obtained through dot-product similarity with the enhanced node-text features, followed by a sigmoid. For descriptions spanning multiple tokens, token scores are aggregated to form the label probability. Redundant queries are trained to have low confidence and can be removed by thresholding. One implementation detail is especially relevant to localization accuracy: the authors remove Grounding DINO’s denoising queries and effectively zero the width and height dimensions used by its detector head. Regression is concentrated on the point coordinates rather than an artificial box around each keypoint. Pairwise relation queries. For every unordered pair of candidate keypoint queries, the model concatenates their content embeddings and linearly projects the result into a 128-dimensional relation embedding. The code places the relation query’s visual reference position at the midpoint between the two endpoint coordinates. A two-layer relation decoder can therefore inspect image evidence near the putative connection while also attending to the relation-language features. The relation decoder deliberately omits self-attention among relation queries. Pair count already grows quadratically with keypoint count; self-attention over all pairs would produce fourth-order scaling in the number of keypoint queries. Removing it also lets training retain only relation queries that contribute to the loss. After decoding, each pair embedding is compared with the text features for prompts such as “connected to” or “part of the same component.” The output for each relation type is a symmetric matrix over candidate keypoints. In effect, the network produces a probabilistic, multi-relational graph over its detections. This design is elegant but exposes a tension that reappears in the failure cases. A relation may require global reasoning about a repetitive grid, yet each relation query is decoded independently of all other relation queries. The endpoint embeddings contain some global context from the keypoint decoder, but the relation branch itself cannot jointly enforce that predicted edges form a coherent assembly. Learning the model and solving the final assignment. Training follows the DETR family. A Hungarian matcher pairs predicted keypoints with ground truth using classification cost and coordinate-regression cost. The released configuration weights binary focal classification cost by two and point-regression cost by seven. Matched predictions receive focal classification loss and L1 coordinate loss. Unmatched queries receive a zero-target classification loss. Ground-truth relations are transferred through the Hungarian keypoint assignment: if two predictions are matched to two annotated endpoints, their pair receives the corresponding relation target. Relations involving unmatched queries are ignored. Relation classification uses focal loss, weighted by five in the public configuration. Auxiliary losses are applied after intermediate decoder stages. At inference, keypoint and relation scores are thresholded to create a predicted graph. Node compatibility with a target node comes from the relevant keypoint-label probability. Edge compatibility comes from the predicted probability of the target edge’s relation label. Finding the globally best correspondence is a quadratic assignment problem and therefore NP-hard. The paper evaluates three approximate solvers from : spectral matching, integer projected fixed point, and reweighted random walks, or RRWM. It also tests a much simpler linear-assignment baseline using expected coordinates. For that baseline, expected points are uniformly placed along an object bounding box’s longer axis and centered on its shorter axis; matching minimizes coordinate distance and ignores predicted relations. The modularity is both a strength and a weakness. Detection and relation prediction can be inspected independently, and a user can replace the matcher or inject engineering priors without retraining the vision model. On the other hand, the neural network never learns from the final graph-matching failures. Eight unusually high-resolution industrial datasets. The industrial evaluation contains eight datasets, several of which depict steel reinforcements and shuttering profiles. They are small in image count but dense in annotation:. | Dataset | Images | Average size | Keypoints and node-label types | Relations and edge-label types | |---|---:|---:|---:|---:| | 1 | 5 | 7200 × 1700 | 3,337 (2) | 5,690 (1) | | 2 | 116 | 7098 × 1638 | 23,691 (2) | 22,496 (1) | | 3 | 9 | 3550 × 1450 | 378 (1) | 756 (2) | | 4 | 113 | 7200 × 1700 | 452 (4) | — | | 5 | 32 | 7758 × 1680 | 532 (1) | 266 (1) | | 6 | 30 | 2462 × 2056 | 2,450 (2) | 3,154 (2) | | 7 | 276 | 1000 × 250 | 1,932 (6) | — | | 8 | 93 | 1000 × 250 | 279 (3) | — |. Across the table, that amounts to 674 images, 33,051 annotated keypoints, and 32,362 annotated pairwise relations. Dataset 1 is the extreme case: only five images, but more than three thousand points and a dense, grid-like graph. Because several datasets have very few images, the authors use five-fold cross-validation. They compare models trained separately per dataset, one shared model trained across all datasets, and shared models subsequently fine-tuned on each dataset. Industrial models are trained for 500 epochs with batch size two. The learning rate starts at (10^{-4}) and drops by a factor of ten at epoch 450. Fine-tuning adds 200 epochs. The shared model takes approximately 12 hours on an RTX 4090. Images are augmented with rotations up to 30 degrees plus brightness and contrast changes. During training, the short side ranges from 480 to 800 pixels and the long side is capped at 1333; testing uses the Grounding DINO scale of 800 by 1333. That last detail is worth emphasizing. The method processes substantially more pixels than 256-by-256 CAPE systems, but it does not process the original seven-thousand-pixel images at native resolution. Much of the paper’s remaining industrial precision gap follows from this downsampling. How keypoint and relation prediction perform. A keypoint is counted as correct when it exceeds a confidence threshold and lies within a normalized distance threshold of a ground-truth point. The main industrial operating point uses a distance of 0.005 times the image’s longer side. Only one detection can count for each ground-truth point; duplicates become false positives. The authors construct precision–recall curves in two ways: varying confidence while fixing distance at 0.005, and varying distance while fixing confidence at 0.3. Relation precision and recall are evaluated only for candidate pairs whose two endpoints were already detected correctly. Relation recall is therefore conditional, not an end-to-end measure: a value of one means all eligible relations were recovered, not that every relation in the image was recovered. Fine-tuning the shared model gives the best average detection and relation results, followed by the shared model without fine-tuning and then independently trained models. Performance is relatively insensitive to confidence thresholds between roughly 0.2 and 0.8. More importantly, keypoint recall remains stable down to a normalized error of 0.005, suggesting that the detector puts most recovered points within half a percent of the long image dimension. The ranking between individual and shared training supports the paper’s transfer argument. Similar phrases and visual concepts recur across datasets, so pooling the data can improve representations of welding points, bar endpoints, and segment relations. The hashed-label ablation discussed later suggests that this is not merely a consequence of seeing more images, although the experiments do not completely separate semantic transfer from ordinary multi-dataset regularization. Graph matching is helpful—but it is also the bottleneck. The graph-matching study reports PCK at the strict 0.005 threshold. Keypoint detection recall acts as an upper bound: matching cannot recover a target node if no candidate lies near it. | Method | Mean PCK@0.005 | |---|---:| | Detection-recall upper bound | 0.93 | | Linear assignment with expected coordinates | 0.70 | | Spectral matching | 0.60 | | Integer projected fixed point | 0.57 | | RRWM | 0.81 |. RRWM performs best overall, but the per-dataset results are more informative. On Dataset 2, expected coordinates are highly effective: linear assignment reaches 0.94 against a 0.96 detection upper bound, while RRWM reaches only 0.65. On Dataset 3, the situation reverses dramatically: RRWM reaches 0.76, versus 0.13 for linear assignment. Dataset 6 shows a similar pattern, with RRWM at 0.57 and the other methods between 0.09 and 0.14. For the simpler Datasets 4, 5, 7, and 8, the methods are effectively tied. This suggests two operating regimes. When an assembly’s expected geometry is stable and accurate, even a crude coordinate prior can outperform generic graph matching. When the geometry bends or varies, language-labeled relations become much more valuable. The obvious engineering conclusion is not to choose between relational reasoning and geometry, but to combine them. There is still a sizeable assignment gap. Mean candidate recall is 0.93, whereas RRWM PCK is 0.81. Dataset 6 loses 30 percentage points between detection recall and final assignment. The authors’ visual failure cases confirm that keypoints and relations can be predicted correctly while the approximate matcher converges to the wrong symmetric configuration. The evaluation handles graph symmetry by enumerating graph automorphisms and reporting the best equivalent correspondence. That is appropriate when symmetric nodes are genuinely interchangeable under the provided graph definition. It also means that downstream users who care about an identity not encoded in the graph must add a relation, orientation cue, or geometric anchor that breaks the automorphism. The comparison with PoseAnything and CapeX. The industrial comparison is unusually favorable to the older methods in one respect: PoseAnything and CapeX receive ground-truth object bounding boxes. They work top-down on 256-by-256 crops, while the new model’s default configuration processes the complete image without bounding boxes. The authors also train a cropped, 256-by-256 variant called Ours-TD, providing a cleaner architectural comparison. PoseAnything additionally receives a randomly sampled annotated support image from the same category; CapeX and the proposed method receive textual labels. The mean PCK values across Datasets 2 through 8 are striking:. | Normalized distance | PoseAnything | CapeX | Ours-TD | Full-image model | |---:|---:|---:|---:|---:| | 0.05 | 0.61 | 0.63 | 0.90 | 0.87 | | 0.01 | 0.10 | 0.19 | 0.83 | 0.83 | | 0.005 | 0.03 | 0.06 | 0.70 | 0.81 | | 0.001 | 0.00 | 0.00 | 0.23 | 0.44 |. At coarse tolerance, cropping helps: Ours-TD slightly beats the full-image model. At strict tolerances, resolution dominates. The complete-image model exceeds Ours-TD by 11 points at 0.005 and 21 points at 0.001. Crucially, the cropped variant still dramatically outperforms both CAPE baselines, indicating that the gain cannot be explained only by input resolution. The result validates the paper’s task decomposition. PoseAnything and CapeX are expected to directly emit one coordinate for every graph node. That works when node prompts or support features are individually distinguishable. It is poorly matched to a graph containing many nodes with precisely the same text label and local appearance. Detecting a set of repeated primitives first and assigning identity from relations afterward is much better suited to this industrial regime. Still, the comparison should not be read as a universal CAPE victory. The proposed system is initialized from a pretrained Grounding DINO checkpoint and uses a different output formulation. Ours-TD equalizes crop size but does not isolate the effects of open-vocabulary detector pretraining, set detection, relational supervision, and graph matching. The tiny folds also produce large uncertainty on some datasets; the authors explicitly call Dataset 3 preliminary because each fold contains only one or two test images. Accuracy comes with substantial computational cost. The full model has 108 million trainable architecture parameters plus 61 million parameters in the frozen BERT backbone, for 169 million total. That is larger than PoseAnything’s 59 million but smaller than CapeX’s 196 million. Compute is considerably higher. PoseAnything uses about 54.8 GFLOPs per object and CapeX 28.1 GFLOPs. Ours-TD uses 219.6 GFLOPs per object, while the full-image model uses 347.5 GFLOPs per image. On an RTX 5080 Laptop GPU, measured latency is approximately 111 milliseconds per object for PoseAnything, 108 milliseconds for CapeX, 348 milliseconds for Ours-TD, and 737 milliseconds per image for the full model. RRWM then adds an average of about 125 milliseconds per object, with very high variance. The per-image versus per-object distinction matters. A full-image pass can amortize its cost across multiple instances, whereas top-down systems require an object detector and one pose pass per crop. The published baseline timings use ground-truth boxes and therefore do not include a learned detector. Even so, the proposed pipeline is not yet an obvious real-time system. Quadratic relation construction is the larger scaling concern. The authors fix 900 keypoint queries, and the number of unordered candidate pairs grows with the square of that value. Dense target graphs introduce a second combinatorial problem during assignment. Dataset 1 is excluded from graph-matching experiments because its graph is already computationally impractical. Does natural language itself help?. The ablation study halves or removes each major component. Removing the feature enhancer and feeding raw image and text features downstream causes a large keypoint-detection drop. Removing the keypoint decoder and relying on initialized proposals also hurts substantially, although reducing it from four layers to two is less damaging. Replacing the relation decoder with a two-layer ReLU MLP lowers relation quality. Interestingly, relations do not collapse completely, implying that the keypoint decoder has already embedded some information about surrounding structure. The dedicated relation decoder nevertheless provides a clear additional benefit. For the language ablation, descriptive phrases are replaced with unique but meaningless identifiers such as. The model can still use those identifiers as conditioning tokens and distinguish classes it has seen during training, but performance is lower than with meaningful language. This is evidence that BERT semantics enable transfer between datasets with related node and edge concepts rather than merely serving as arbitrary class IDs. It is not yet strong evidence for unconstrained language prompting. The industrial experiments do not systematically test paraphrases, synonyms, unseen relation descriptions, or compositional prompts. CapeX performs such text-modification tests, but this paper’s semantic claim rests mainly on the hashed-label ablation and on generalization to unseen MP-100 categories. The current system is best understood as a language-backed schema, not as a conversationally programmable perception model. Generalization on MP-100. MP-100 contains roughly 20,000 images from 100 object categories, with category-disjoint training, validation, and test sets. The authors could not access the OneHand10K subset. They therefore report splits 1, 2, 3, and 5, where the missing data affects training but not the test categories. The model is trained for 50 epochs, with gradient accumulation producing an effective batch size of 32. Only labels for annotated points are included in each training prompt so that visible but unannotated keypoints are not treated as negatives. The benchmark skeleton is represented as a relation type, and RRWM performs the final assignment. Evaluation uses the conventional, comparatively loose PCK threshold of 0.2 relative to an object bounding box. | Method | Split 1 | Split 2 | Split 3 | Split 5 | Average | |---|---:|---:|---:|---:|---:| | POMNet | 84.2 | 78.3 | 78.2 | 79.2 | 80.0 | | CapeFormer | 89.5 | 84.9 | 83.6 | 85.1 | 85.8 | | PoseAnything | 91.1 | 88.2 | 86.1 | 85.8 | 87.8 | | CapeX | 92.8 | 89.5 | 85.0 | 89.6 | 89.2 | | Proposed method | 91.9 | 85.9 | 84.1 | 86.8 | 87.2 |. The method is competitive rather than state of the art. Its 87.2 average is close to PoseAnything’s 87.8 but two points behind CapeX. The authors attribute the gap to CapeX’s use of text–image correlation for an initial pose, followed by learned graph refinement. MP-100 generally supplies unique semantic node descriptions, so that initialization is reliable. The proposed method intentionally postpones graph structure until classical matching, a choice that helps with repeated industrial points but gives up some performance where every point already has a distinctive name. This is a healthy result. It shows that the industrial gains do not come from a system incapable of conventional CAPE, while also demonstrating that no single graph integration strategy dominates every regime. The real limitations. The authors are unusually direct about precision. A normalized error of 0.005 corresponds to roughly 35 original-image pixels on a seven-thousand-pixel image. That may be excellent relative to existing CAPE systems but inadequate for millimeter-level inspection, manipulation, or metrology. PCK normalized by image size is not a physical accuracy measure; deployment ultimately requires calibrated error in millimeters. Dataset 1 exposes the scaling limit. Five images cannot adequately cover a structure containing thousands of points and a dense grid of visually similar relations. Keypoint detection degrades, relation prediction degrades more severely, and graph matching becomes infeasible. Pairwise relation decoding is precisely the wrong scaling law for a graph whose size may reach the thousands. Approximate matching is another brittle stage. Symmetric graphs create many local optima, and independently predicted pair scores do not guarantee global consistency. The authors suggest expected coordinates or stronger geometric priors. In an industrial application, those priors are often available from CAD, camera calibration, nominal dimensions, or the preceding production stage, so excluding them would be unnecessarily purist. The approach also assumes that the correct target graph and object inventory are known. It does not discover an unexpected topology, decide which product variant is present, or determine that a component is missing before matching. Missing-part inspection could be built on top of unmatched graph nodes, but confidence calibration and explicit null assignments would become critical. Finally, only symmetric binary relations are modeled. Many useful industrial constraints are richer: one endpoint precedes another along a cable, a connector inserts into a socket with a directed orientation, three points should be collinear, several welds belong to one subassembly, or an edge should have a particular metric length. Natural-language labeling makes those extensions conceptually appealing, but the current decoder and matcher cannot express all of them. How roboticists could build on it. The most promising deployment architecture would be coarse-to-fine and geometry-aware. Run the language-conditioned model once over the full scene to identify keypoint types and rough topology. Then crop native-resolution patches around candidate points and refine each coordinate with a lightweight local model. This preserves the global relational context while avoiding the 35-pixel error induced by resizing an entire seven-thousand-pixel image. Relation prediction should also become sparse. CAD adjacency, image-space neighborhoods, learned edge proposals, or approximate component grouping can reduce an all-pairs problem to a small candidate set. Global consistency could then be enforced with a factor graph or matcher incorporating edge lengths, angles, orientation, and uncertainty alongside language-conditioned relation scores. For rigid components, matched 2D keypoints with known 3D coordinates could feed PnP or bundle adjustment. For deformable reinforcements, cables, and profiles, the points could parameterize a spline or non-rigid registration model. Calibrated multiview images could triangulate the matched nodes. The project repository describes multiview reconstruction and 3D localization as natural extensions, but the journal experiments themselves validate only 2D image localization. The deeper opportunity is using the graph as an interface between perception and manufacturing knowledge. A CAD model, bill of materials, or process plan already describes components and relations. Mapping those concepts to language-labeled nodes and edges could make the perception system configurable without defining hundreds of arbitrary numerical keypoint classes. The language is useful less because an operator can chat with the model and more because it provides a reusable semantic namespace across products, datasets, and downstream robot programs. The released code is built on MMDetection and exposes separate node and relation text prompts. Its dataset format extends COCO annotations with variable-length keypoint lists and named relation adjacency structures, making it practical to experiment with new graph schemas or matching algorithms. Final assessment. The paper’s strongest contribution is not simply attaching words to keypoints. CapeX already demonstrated textual node prompts. The important step is treating edge semantics as first-class visual predictions and separating repeated-part detection from combinatorial node identity. That formulation is particularly well suited to industrial assemblies where local evidence is ambiguous but topology is informative. It produces dramatic improvements over existing CAPE methods at strict localization tolerances and remains competitive on MP-100. At the same time, native-resolution refinement, sparse relations, stronger geometry, and more reliable assignment are still needed before the system meets demanding industrial accuracy and scalability requirements. For embodied-AI researchers, the broader lesson is valuable: when objects are composite, deformable, and repetitive, a flat vocabulary of visual landmarks is not enough. What a point is and how it relates to other points should be represented separately—and both can be grounded through language.