Latest / Embodied AI 101 / ArtiFixer: Few-Step Diffusion for 3D Scene Reconstruction

Transcript

ArtiFixer: Few-Step Diffusion for 3D Scene Reconstruction · Embodied AI 101

• ArtiFixer: Few-Step Diffusion for 3D Scene Reconstruction. Imagine you scan a room from just one side: the sofa and TV look great in your 3D model, but the unseen corner behind the bookshelf is a blurry mess or an empty void. Modern 3D reconstruction methods – like NeRF variants or the recent 3D Gaussian Splatting techniques – excel at interpolating the observed parts of a scene. But where data is missing, they “hallucinate” poorly or not at all. ArtiFixer, a new model from NVIDIA (SIGGRAPH 2026), tackles exactly this problem by stringing together powerful generative priors in a carefully controlled way. It takes the broken, under-constrained output of a scene reconstructor and paints in the missing or blurry regions with content that fits the scene, boosting visual fidelity by a wide margin. In fact, on standard benchmarks it beats prior art by roughly 1–3 dB PSNR, and can recover plausible images where older approaches just fail. The trick is a two-stage diffusion pipeline: a bidirectional video diffusion teacher fine-tunes to the imperfect scene, and a fast autoregressive diffusion student generates hundreds of consistent video frames “in one shot.” The result is an end-to-end system that yields high-quality, temporally stable novel views and even refines the underlying 3D geometry if desired. In this article we’ll unpack how ArtiFixer works, why its design matters, and how it compares to earlier attempts to use diffusion for 3D recon. We’ll assume you’re familiar with basic 3D reconstruction (NeRFs, Gaussian Splatting, etc.) and diffusion models (image/video diffusion, U-Nets, transformers). We’ll avoid heavy math in favor of clear intuition. By the end, you should have a solid sense of how ArtiFixer stitches together 3D rendering and generative AI to fill in “the unknown” of a scene. The Problem: Missing Corners in 3D Reconstructions. First, let’s set the scene. Recent advances like 3D Gaussian Splatting or NeRF have given us impressively realistic novel-view renders inside a well-observed volume. If you have many images of an object or room, these methods can interpolate between them and even extrapolate a bit beyond, producing sharp textures and fine geometry. Yet all such “per-scene” pipelines share a weakness: they have no imagination for what they didn’t see. Unobserved regions (empty corners, backs of objects, occluded parts) typically become blurry or flat-colored. If you try to fly the camera into those spots, the model either collapses into artifacts or holes, breaking the illusion entirely. Generative models offer a tempting remedy: why not lean on a learned prior to invent what the scene should contain? Indeed, diffusion-based image or video models can plausibly inpaint missing areas in a 2D frame, and text-to-3D systems like DreamFusion suggest scene imagination is possible. However, applying these freely leads to hallucination: the model might fill a missing corner with a dragon or random furniture, with no regard for the existing walls and objects. Worse, independent per-frame generation often breaks temporal/photo consistency. Prior works like GenFusion (CVPR 2025) tried to “distillation loop” generative fixes, but they either handle only a few frames at a time or produce blurrier consistency. ArtiFixer is built on a simple insight: combine the hard data from the existing 3D recon with the creative power of a generative model, but constrain the generation so it respects observed scene context. To do this, the NVIDIA team devised a careful pipeline. The first stage is a bidirectional video diffusion model trained with a novel “opacity mixing” scheme so it learns to fill in unobserved regions without overwriting known parts. The second stage distills that into a causal autoregressive diffusion model that can generate a long sequence of new views quickly. Let’s go through each stage in turn. Stage I – The Bidirectional Diffusion Teacher. For the teacher stage, ArtiFixer starts with a powerful off-the-shelf video diffusion model (in fact, it’s a fine-tuned 14B-parameter “Wan2.1” model, which is a large-scale video generator). Normally such a model generates video frames from random noise or from a text prompt. Instead, ArtiFixer conditions it on the renderings and metadata of the imperfect 3D reconstruction. In practice, this means the model takes as input each degraded rendering (the RGB image produced by the base 3D method, which will have holes and noise) plus an opacity map and camera information. The opacity map is essentially a per-pixel alpha (transparency) indicating where the reconstruction has confident geometry (alpha = 1) versus none (alpha = 0). The key trick is how it uses that opacity map to mix the input with noise: rather than initializing diffusion from pure noise or directly from the blurry rendering, ArtiFixer encodes the rendered image into the model’s latent space and then mixes in Gaussian noise according to the opacity map. Concretely, pixels with high opacity (well-reconstructed) stay close to the known color in the latent. Pixels with low opacity get mostly noise inserted. This “opacity mixing” forces the model to keep known regions fixed while still allowing it to freely imagine the unknown parts. It’s a bit like saying: “Here is your best guess of the scene (the rendering), but I know it’s unreliable where alpha is low, so feel free to paint over those parts with your creativity.” This balances the two goals: consistency and creative filling-in. In addition to opacity blending, the teacher model is fed fine-grained opacity information and camera signals, along with any clean reference views and even an optional text prompt. The reference views are real images of the scene (if available) that anchor the appearance. The text prompt can describe the scene generally (though that’s an extra). All this extra conditioning helps the model learn scene-specific details and geometry cues. During training it sees many different camera paths (“bidirectional” meaning it can look both forward and backward in time) over the scene. The result is a diffusion model that has learned to take a partially wrong video (the original renderings along a camera path) and produce a corrected, complete video sequence, while never contradicting the observed bits. In other words, it’s a generative “inpainting” operation in 4D (space-time) that respects what’s already known. After enough finetuning, this bidirectional model can produce high-quality, consistent novel views of the scene. In extreme cases, even completely unobserved regions get plausible content that’s globally consistent with the reference data. The price, however, is that bidirectional diffusion is slow: each new frame essentially requires a separate diffusion process (tens or hundreds of denoising steps per frame) and the model may process chunks of video together. That’s why we move to Stage II. Stage II – The Autoregressive Diffusion Student. To speed things up and scale to longer videos, ArtiFixer next distills the bidirectional teacher into a causal autoregressive student model. “Causal autoregressive” here means it generates one frame at a time (or small batches) in temporal order, each conditioned on prior frames. Practically, this can be implemented with a transformer block and key-value caching: the model saves context from past frames and streams out new ones forward. The important part is that during generation, once a frame is produced it is not revisited or averaged with future context (hence “causal”). This is unlike bidirectional diffusion which can look ahead and behind. The benefit is speed: the student can emit hundreds of frames in a single run, at interactive rates if needed. But how to make the student faithful to the teacher’s quality? ArtiFixer uses a Self-Forcing-style Distribution-Matching Distillation (DMD) strategy. In essence, this means during training the student generates videos itself (using its own previous outputs as context, the “self-forced” part) and is trained to match the teacher’s outputs distribution. Practically, one can think of it as rolling out the student frame by frame on a camera path, obtaining a sequence (s_1, s_2, \dots), and then comparing it to the teacher’s diffused video (t_1, t_2, \dots). The loss minimizes a KL-like divergence between the student’s predicted distribution and the teacher’s distribution over images. In implementation, the gradients involve the difference of “score functions” (gradients of log-probabilities), as described in one-step diffusion distillation works. The upshot: after distillation, the student has learned to generate high-quality frames in a few diffusion steps, with nearly the same output quality as the teacher. In fact, in many distillation schemes you can collapse the 100-step process to as few as 1–4 steps with only minor quality loss. The Hugging Face model card explicitly calls ArtiFixer a “few-step causal auto-regressive model”, reflecting that it might only run a handful of denoising steps per frame at inference. In summary, Stage II yields a streamlined generator: given the underlying 3D renderer’s noisy frame and metadata at time (t), the autoregressive student will produce frame (t) using a handful of diffusion iterations, then condition on that (and optionally the rendering of frame (t+1)) to produce frame (t+1), and so on. It can generate new arbitrary camera trajectories (or loop consistent orbits) very quickly. Optionally, those generated images can themselves be used to refine the 3D model: one can take the student’s output frames as pseudo-ground-truth images and optimize the geometry or radiance field to match them (the variant ArtiFixer3D described below). But even without that, the student alone already produces significantly better novel views than the vanilla 3D recon. Putting It Together. Putting both stages in one pipeline, ArtiFixer works like this at inference time. You have some reconstructed scene (say from 3D Gaussian Splatting, also known as 3DGUT) that produces an RGB and opacity map for any given camera pose. You choose a desired novel trajectory of camera poses. For each pose:. Render the scene’s current model under that pose, getting an RGB image and corresponding opacity map (plus possibly an object segmentation or depth map, but opacity suffices). Pose and blend these with small noise as input to the model (this is how it was trained). Generate a refined image via the autoregressive diffusion student. Because it is causal, it uses any reference frames or previous outputs for context if available. Output the refined image. Because the student was distilled to run in only a few diffusion steps, this whole process is much faster than naively running the bidirectional diffusion each time. The result is a video (or novel view sequence) with the same camera motion, but the images have missing parts filled in organically. Importantly, where the original opacity was 1 (solid), the new image closely matches the original color; where opacity was low (no data), the model’s generative prior supplies plausible textures that fit the context. This yields a view-consistent completion of the scene. ArtiFixer’s architecture (an enormous 16.9B parameter model) is definitely heavy-weight, so it’s intended for offline processing, not real-time on a phone. But it leverages the latest GPU-accelerated training (FlashAttention on NVIDIA Hopper/Blackwell). NVIDIA has provided code that uses PyTorch, HuggingFace Diffusers, etc., and the authors report much faster inference than CPU baselines. Results and Comparisons. How well does this actually work? The paper reports extensive experiments on standard sparse-view benchmarks. The headline is: ArtiFixer produces radically better visual results than previous fixup methods, and measurably higher PSNR/SSIM. One key benchmark is Mip-NeRF 360 (3-view split) – it’s famously challenging because each scene has only three input images. On this, the authors compare ArtiFixer3D+ (discussed below) against the raw 3DGUT output, GenFusion (CVPR 2025, another Autodesk project), and GSFixer (ICML 2026). In every scene, ArtiFixer3D+ yields sharper, more coherent renderings. Qualitatively, corners that were blank in 3DGUT are filled with realistic detail (bricks on walls, books on shelves, etc.), where GenFusion and GSFixer either leave artifacts or paint inconsistent textures. Quantitatively, ArtiFixer beats all published baselines by about 1–3 dB in PSNR – a very large gap on these images. In fact, the project page proudly notes, “Our quality exceeds, to our knowledge, all other previously published work” on that split. They saw similar gains on other datasets. On DL3DV-10K (10,000 scenes, but tested here on very sparse scans), ArtiFixer3D+ again outperformed the original 3DGUT recon and two generative baselines. For instance, GenFusion’s model was originally trained to output 16-frame clips and required iterative passes to reach hundreds of frames; those results turned out a bit blurry, especially in empty areas. GEN3C (a separate NVIDIA model for 3D-informed video synthesis) produced sharper frames, but it often “does not respect the source content” – e.g. it might render a different object shape than in the actual room. By contrast, ArtiFixer consistently “reconstructs plausible and consistent geometry even when the initial rendering is highly degraded”. In other words, if the 3D input is bad, ArtiFixer makes it good without inventing impossible things. On the Nerfbusters benchmark (another sparse-view challenge), the pattern holds: only ArtiFixer variants manage to generate any realistic content in strongly unobserved areas, whereas other methods either produce blur or flat fill-in. The paper even tries ablations like dropping the rendering condition altogether: when forced to “hallucinate” from just reference images, ArtiFixer’s topology still holds up remarkably well. The outputs remain less detailed (since it lost the exact color cue), but the high-level layout and camera motion are still correct. Essentially it can default to being a pure video diffusion generator – and indeed, thanks to its losses, it “retains a strong generative ability” so that even a purely text-based prompt can coax it into making a scene video that matches the learned style. In summary, ArtiFixer’s evaluations show it’s not just a small tweak: it can create entirely new, coherent geometry where others couldn’t. The authors report many cases of “plausible reconstructions in scenarios where existing approaches fail completely.” Quantitatively, the multi-decibel boost in PSNR/SSIM and user study votes confirmed it. ArtiFixer Variants: 2D vs 3D Integration. It’s worth clarifying the difference between the pure-image output and the 3D-refined output. ArtiFixer actually has three working modes:. ArtiFixer (direct) – simply run the autoregressive diffusion on each novel view and take those as final images. This yields very sharp, vivid frames. However, there is a small risk of slight inconsistencies between far-apart views, since it’s not guaranteed to maintain global 3D coherence beyond what’s enforced by training. ArtiFixer3D – after generating images with ArtiFixer, use those images as “ground truth” to refine the original 3D model (e.g. optimizing the Gaussian splatting or NeRF to fit). Essentially you bake the hallucinated detail back into your 3D representation. This produces a new 3D model that, when rendered, agrees with the ArtiFixer images. The advantage is stronger multi-view consistency and exact agreement on training views. The downside is some loss of sharpness or creativity — the optimization tends to spread out detail. ArtiFixer3D+ – do both: first feed reference images into ArtiFixer3D to improve the model, then finally run ArtiFixer (the diffusion network again) on top of the refined recon. This aims to get the consistent baseline of 3D integration plus the final touch of diffusion clarity. The paper shows all three produce similar overall look. The direct ArtiFixer images are “slightly sharper”, while the pure 3D version is “more consistent with the source images at the cost of some blurriness”, and 3D+ “restores sharpness while remaining highly consistent”. In practice, ArtiFixer3D and ArtiFixer3D+ tend to score best when one cares about both metric fidelity and perceptual quality. Comparison to Related Approaches. It helps to place ArtiFixer in context. Many recent works have looked at leveraging diffusion for 3D fixes:. GSFixer/GSFix3D (ICML 2026). This system fine-tunes a latent diffusion model (like Stable Diffusion) to a scene, using the mesh and point clouds. They randomly mask regions to train the model to inpaint. That model then refines novel views which are back-projected into the Gaussians. GSFixer achieves strong results and emphasizes minimal per-scene finetune. In essence it’s similar in spirit: a diffusion inpainting then 3D optimize. However, GSFixer uses image-based diffusion (so each view one at a time), while ArtiFixer uses video diffusion which better enforces consistency. Also, ArtiFixer’s opacity mixing is a distinctive twist – we haven’t seen that elsewhere. According to NVIDIA’s comparisons, ArtiFixer outperforms GSFixer in the tested scenarios (PSNR gap of about 1-3dB) on the same tasks. GenFusion (CVPR 2025). This was an earlier NVIDIA work that “closed the loop” between reconstruction and generation with diffusion videos. It took an image diffusion model and iteratively distilled it to cover longer sequences. Its main limitation was scalability: it needed to generate only 16 frames at once and then stitch them via extra distillation passes. The result could be blurry when pushed to long sequences. ArtiFixer’s approach avoids that by directly training the autoregressive student to do hundreds of frames at once. In NVIDIA’s tests, GenFusion’s outputs were consistently blurrier than ArtiFixer’s. Gen3C (SIGGRAPH 2025). This is a text-to-video model with 3D understanding (also by NVIDIA). It can generate videos that follow a camera path described in 3D terms. Its strength is photo-realism, but it ignores any specific scene input. In other words, it tends to hallucinate freely. As the ArtiFixer paper notes, Gen3C can produce very sharp visuals, but they “often do not respect the source content”. ArtiFixer, by contrast, explicitly anchors to the actual scene textures via the opacity mixing strategy. Overall, ArtiFixer’s novelty is combining those ideas: it’s as flexible as the best text/video diffusion models, but also as anchored to the scanned data as 3D splatting. By distilling to a fast AR model, it’s far more practical than per-frame diffusion inpainting. Its 1–3 dB PSNR lead over all baselines is testament to the effectiveness of these design choices. Practical Implications. For robotics and 3D vision, ArtiFixer offers a powerful new tool. In robotic mapping or autonomous driving, one often has sparse data (e.g. one side of an object or room). Being able to realistically infer the unseen side can help in planning or simulation. For example, a robot navigating a cluttered environment could use a reconstructed scene and then ArtiFixer to guess what’s around a corner, making planning safer even with limited sensor coverage. In film or AR, it can salvage footage from just a few camera shots and generate full 3D walkthroughs. In both cases, the fact that ArtiFixer is grounded in the actual measurements (through opacity conditioning) means it’s far more reliable than unconstrained generative inpainting. That said, there are caveats. The generative element means ArtiFixer might still “guess” incorrectly surface reflectance or geometry if the data is too sparse. It should not be treated as a source of ground truth geometry – it’s giving plausible completions, which are great visually but must be used cautiously for downstream measurements or tasks that require physical accuracy. Furthermore, the model is very large (16B) and requires a powerful GPU. It’s not yet a real-time solution. However, because it only uses a few diffusion steps, it’s orders of magnitude faster than naive diffusion and can run at tens of frames per second on top-of-the-line hardware. The NVIDIA team has open-sourced the code and model weights, making it accessible for experimentation. Technically, ArtiFixer also pushes the idea of “few-step diffusion” in the vision domain. By using distribution-matching distillation, it collapses the usual many-step 3D diffusion into just a handful of steps per frame. This is a trend we see in the literature (e.g. One-Shot Diffusion) to make diffusion lighter-weight. In practice, the HuggingFace model card even emphasizes “few-step” diffusion as a selling point. Conclusion. ArtiFixer represents a significant step in applying generative AI to 3D scene reconstruction. By carefully combining the strengths of bidirectional diffusion (accuracy and coherence) with an autoregressive generator (speed), and by introducing the smart “opacity mixing” strategy, the authors have built a system that can fill in missing parts of a scene without descending into hallucination. The results speak volumes – this is one of the first times generative models have demonstrably improved the quality of a 3D reconstruction benchmark in a measurable way (1–3 dB PSNR on standard tests). For researchers and practitioners, ArtiFixer shows a viable path forward for “3D vision meets foundation models”: train on more than images, incorporate camera geometry and scene context, and distill heavy models into efficient ones. It also highlights the practical importance of things like opacity maps and masked conditioning – simple ideas that turn out to be crucial for bridging the gap between raw sensor data and fantasy. While it’s still early days, we can imagine future versions: even more efficient architectures, better integration with geometry optimization, or application to other modalities (e.g. depth completion or reflectance prediction). For now, ArtiFixer is a remarkable demonstration: it fixes broken 3D scenes with just a few diffusion steps, and does so better than anything that came before. That’s an exciting development for embodied AI and 3D understanding, pointing to a future where scenes are complete and coherent even from sparse glimpses. Sources: The above descriptions and evaluations are derived from the ArtiFixer project materials and paper (de Lutio et al., SIGGRAPH 2026), as well as associated repo documentation. These sources detail the method’s pipeline design, training strategy, and benchmark results in depth.