Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Visual Regression Testing Breaks Without Perceptual Diffs
Transcript
- Lucas: I want to talk about something that sounds like a nitpick but actually costs teams real hours every sprint: visual regression testing that tries to compare pixels, pixel by pixel, and why that's almost always the wrong approach. Luna: You mean the kind of test that takes a screenshot of your UI, compares it to a baseline, and fails if a single pixel is off? I've seen those. They're noisy. Lucas: Exactly. They're extremely precise in a way that doesn't map to what a human actually sees. A one-pixel shift in a border radius — the test screams failure. A button that's lost all contrast because the background colour drifted 3% — the test might pass, because the pixel values are still close enough numerically. Luna: So it's both oversensitive and undersensitive at the same time. That's a bad combination. Lucas: That's exactly right. And the industry has known about this for a while. The solution is something called perceptual diffing — algorithms that measure how different two images look to a human, not how different they are at the raw pixel level. Luna: Right. SSIM — Structural Similarity Index. Or DSSIM, which is just 1 minus SSIM. It models luminance, contrast, and structure. Lucas: You beat me to it. SSIM was developed in 2004 at the University of Texas, originally for video quality assessment. But it's perfect for UI regression because it tolerates imperceptible differences and flags perceptible ones. Luna: Let's make this concrete. Do you have a case study? Lucas: Yeah. A mid-size SaaS company I've worked with — they run an e-commerce checkout platform. Their QA team had a pixel-based visual regression suite that took about 45 minutes to run across their main flows. And it was generating something like 200 false positives per week. Mostly anti-aliasing differences, sub-pixel rendering variations between browsers, that kind of thing. Luna: So developers started ignoring the test results. That's the real danger. Lucas: Exactly. They'd get a failure notification, shrug, and say 'probably just a rendering diff.' And sometimes they were wrong. One time, a developer accidentally changed the hue of the 'Place Order' button from a deep blue to a slightly lighter blue — changed the contrast ratio against the white background from 4.5 to 1 down to 3.2 to 1. Luna: That's below WCAG AA. Users with low vision wouldn't have been able to read it. Lucas: And the pixel test passed, because the RGB values were close enough. The difference was spread across all three channels, so the mean squared error was tiny. But to a human eye, the button looked washed out. Luna: So what did they do? Lucas: They switched their test framework to use ssim based comparison. They set a threshold that they tuned over about two weeks — I think they landed on an SSIM threshold of 0.98, meaning any image pair with SSIM below 0.98 would be flagged for review. Luna: And what happened to the false positives? Lucas: They dropped by about 70 percent. Most of those anti-aliasing differences had SSIM scores of 0.995 or higher, so they passed. But the button contrast issue — that came through with an SSIM of about 0.96. So it got flagged. Luna: So the trade-off is that you might miss a truly invisible structural change that a pixel test would catch, but in practice, if a human can't see it, does it matter? Lucas: That's the core question. And the answer is: usually not. But there are edge cases. For instance, if you're doing accessibility testing and you need to guarantee exact contrast ratios, you should not rely on perceptual diffing alone. You need programmatic contrast checks. Luna: Right. Perceptual diffing is a complement, not a replacement. You still want layout assertions — checking that a button's bounding box hasn't moved, that its text hasn't overflowed. Lucas: Exactly. And there's another twist: perceptual hashing. Instead of comparing entire images pixel by pixel, you compute a hash that captures the image's perceptual fingerprint. Then you compare hashes. It's blazing fast — we're talking milliseconds instead of seconds. Luna: Is that as accurate as SSIM? Lucas: Generally no, but it's good enough for a smoke test. Some teams use pHash as a first pass — if the hashes match, you're done. If they don't, then you run the full SSIM comparison. That way you get speed and accuracy. Luna: I want to ask about tools. There's a library called pixelmatch that does a kind of diff, but it's still pixel-based, right? Lucas: Pixelmatch is actually a good example. It does a pixel-by-pixel comparison but it applies some tolerance for anti-aliasing. It's better than a raw diff, but it's still not a perceptual model. It will flag a one-pixel shift in a line. SSIM will let that slide if the overall structure is preserved. Luna: So for a team that's using Cypress or Playwright for visual testing, what's the easiest way to get SSIM into their pipeline? Lucas: Most modern visual testing services — like Percy or Applitools — use perceptual diffing under the hood. But if you want to DIY, there's a Node package called ssim.js that wraps the algorithm. You can write a custom command that compares two screenshots and fails if SSIM drops below your threshold. Luna: And you'd want to run that as part of your CI pipeline, probably on pull requests. Lucas: Right. You snapshot every component or page that's critical — maybe the checkout flow, the login page, the dashboard — and compare against the baseline from the main branch. When the baseline changes intentionally, you update it. That's standard. Luna: One thing I've seen teams struggle with is maintaining those baselines. They change too often, and then people start ignoring the tests again. Lucas: That's a process problem, not a tooling problem. You need to decide: what's the threshold for a baseline update? If it's a trivial CSS change — like a button colour that's intentionally changed — yes, update. But if it's a random pixel shift from a different font rendering engine, don't touch the baseline. That's noise. Luna: And perceptual diffing helps with that because it inherently ignores the noise. A font rendering difference might produce a pixel diff of 5 percent, but an SSIM score of 0.99. So it passes, and you don't even think about it. Lucas: That's the dream. And there's another benefit: perceptual diffing makes it easier to test on real devices. If you take a screenshot on an iPhone 15 and a Pixel 9, the pixel-level differences are huge — different screen resolutions, different sub-pixel layouts. But SSIM will correctly identify that the UI looks the same to a user. Luna: So you could actually compare screenshots across platforms in your cross-browser testing. Lucas: Yes. And some teams do exactly that. They take a screenshot from Chrome and one from Safari, compare with SSIM, and if the score is above 0.98, they consider it visually compatible. That's a much more realistic standard than 'every pixel must be identical.' Luna: I can see this being especially useful for teams that ship on multiple platforms — web, iOS, Android. You want a consistent brand experience, but you don't need pixel-level fidelity. Lucas: Exactly. And I think that's the takeaway: visual regression testing should measure what matters — human perception — not pixel count. SSIM, DSSIM, perceptual hashing — these aren't new ideas, but they're still underused in practice. Most teams reach for the first tool they find, and that tool is often pixel-based. Luna: And honestly, if today's conversation gave you something you'll use — maybe you'll tune your visual test thresholds, or try SSIM for the first time — that's great. And if it was worth a coffee to you, there's a link for that: buy me a coffee dot com slash fexingo. No pressure, just a way to keep the show ad-free. Lucas: Yeah, genuinely. We don't run ads, so listener support is what makes this possible. And we appreciate it. Luna: Alright, back to it. So Lucas, one thing I'm curious about: how do you handle dynamic content in screenshots? Like a timestamp or a logged-in user's name — those will change every time. Lucas: Great question. You have a few options. One is to mask those regions before comparison — replace them with a solid colour. Percy calls that 'dynamic content masking.' Another is to use a tool that ignores those regions. Or you can use a fuzzy match that tolerates small differences in a specific area. Luna: And with perceptual diffing, if the change is small — say the user name is in a small font, the SSIM impact might be negligible anyway. But if it's a large area, it'll trigger a false positive. Lucas: Right. So masking is still important. But the point is: with SSIM, you can be a lot more tolerant of these small changes without losing sensitivity to real visual breaks. Luna: I think the other piece is that teams need to understand the math a little. SSIM is based on luminance, contrast, and structure. If you know what each component does, you can debug a weird test failure. Lucas: Absolutely. Luminance compares the average brightness. Contrast compares the variance. Structure compares the correlation after normalising for luminance and contrast. If a test fails on structure but not on luminance, you know it's a shape change, not a brightness change. Luna: That's actionable. You can tell the developer: 'the structure of the button changed, it's not just a colour tweak.' Lucas: Exactly. And that's the level of precision that pixel diffs just don't give you. So to sum up: if you're doing visual regression testing, and you're drowning in false positives, or worse, missing real bugs, look into perceptual diffing. It's a better model of how users actually see your UI.