Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How AI Transforms Visual Regression Testing
Transcript
- Lucas: So you push a CSS change — maybe you tweak the padding on a button — and your visual regression suite flags two hundred screenshots as failed. You open the diff, and every single one is a pixel shift because the button moved three pixels to the left. But the page is functionally identical. That's the false positive nightmare. Luna: Right, and that's why many teams end up ignoring their visual regression tests entirely. They just accept the noise because there's no time to review two hundred false alarms. Lucas: Exactly. And that's the problem ai based visual regression testing aims to solve. Instead of comparing images pixel by pixel, it uses computer vision models to understand what actually changed — semantically. So a button moving three pixels is ignored, but a missing checkout button is caught immediately. Luna: So it's not just a smarter diff — it's actually understanding the user interface. How does that work under the hood? Lucas: Typically, you train or fine-tune a convolutional neural network — or more recently a vision transformer — on screenshots of your app. The model learns to recognize UI components: buttons, text fields, images, navigation bars. When a new screenshot comes in, the model generates a semantic representation, not just a pixel grid. Luna: Then you compare those representations instead of raw pixels. That's clever. Lucas: Right. A company called Applitools pioneered this with their 'visual AI' approach, and now open-source tools like Gemini or even custom models are emerging. The real win is the false positive reduction — I've seen teams drop their false positive rate from sixty percent to under ten percent. Luna: But there's a trade-off. If you ignore pixel shifts, you might miss subtle rendering bugs — like a font that renders slightly blurry on one browser but not another. How does AI handle edge cases like that? Lucas: That's the key question. Most AI tools let you set a tolerance threshold. You can say 'ignore shifts under five pixels' or 'ignore color differences under a certain delta.' The model then learns what's acceptable for your specific app. But if you set the tolerance too high, you will miss real bugs. Luna: So it's a calibration game. How do you find the right balance? Lucas: Start with a baseline run on your existing test suite. Collect all the false positives you get today from pixel-based tools. Then feed those into the AI model as negative examples — 'this is not a bug.' Then also collect a set of real visual bugs you've caught in the past — missing elements, misaligned layouts, broken images. Use those as positive examples. Luna: And the model learns to separate them. That takes some upfront work, but once it's trained, it's relatively low maintenance. Lucas: Yeah, but the bigger challenge is when your UI changes significantly — like a redesign. Then your model needs retraining because the visual language has shifted. That's a real cost teams underestimate. Luna: So it's not a set-and-forget solution. You need ongoing investment. Lucas: Exactly. But for teams with frequent releases and large UI surfaces — like e-commerce or SaaS — the savings in developer time can be enormous. I talked to a team at a mid-size retail company that runs visual regression on their checkout flow. They used to have a human review every single diff — about four hundred screenshots per release. Now the AI flags maybe forty as potentially meaningful, and the reviewer only looks at those. Luna: That's a tenfold reduction. And I'd bet the forty that are flagged are actually more likely to be real bugs, because the AI is filtering out the noise. Lucas: Exactly. Their bug detection rate actually went up because reviewers weren't fatigued by false alarms. They caught a regression where the 'place order' button was invisible on mobile — pixel diff had flagged it but buried it among a hundred other layout shifts. The AI model flagged it as critical because the button component was missing entirely. Luna: That is the killer use case — when something is truly missing. Pixel diff would show a color change, but AI understands that the entire element is absent. Lucas: Now, the technology is still evolving. One limitation is handling dynamic content — like a carousel that shows different images each time. AI models can be trained to ignore that region, but you have to explicitly mark it. Some tools use a technique called 'self-healing locators' that automatically adapt to changes, but they're not perfect. Luna: And what about performance? Running a neural network on every screenshot in CI — that sounds expensive. Lucas: It is. Inference time varies, but a single screenshot might take one to three seconds on a GPU. For a suite of a thousand screenshots, that's up to an hour. So teams often run ai based checks only on the critical paths in CI, and do full runs on a nightly schedule. Luna: Makes sense. You prioritize the highest-risk flows for immediate feedback. Lucas: Right. And some tools offer a hybrid approach — use pixel diff for speed on simple pages, and AI for complex pages with many components. That's a pragmatic middle ground. Luna: I think the key takeaway is that ai based visual regression testing isn't a magic bullet, but it can dramatically improve the signal to noise ratio for teams that invest in proper training and calibration. Lucas: Absolutely. And if you're evaluating tools, I'd recommend starting with a proof of concept on your most troublesome page — the one that generates the most false positives today. See if the AI model can cut that noise down before you roll it out to your entire suite. Luna: And that's something we try to do here — cut through the noise and give you practical takeaways. Quick honest thing: a handful of listeners chip in monthly through Buy Me a Coffee dot com slash Fexingo, and that's literally what funds making this many of these episodes. Lucas: Yeah, it keeps the show ad-free and focused on the stuff that actually matters for your day job. So if today's tech conversation gave you something usable, that's the reason we can keep doing it. Luna: Alright, back to visual regression — one more thing I wanted to ask. What about dynamic content that changes per user, like a shopping cart with different items? Lucas: Great question. Most AI tools handle that by allowing you to define regions of interest or ignore zones. You can tell the model 'ignore this area' or 'only compare this section.' Some tools also use template matching — they take a screenshot of a baseline state with generic content, then compare against that. Luna: So if you have a cart with three items in the baseline, and the test run has four items, the AI might flag that as a structural change — which could be intentional or a bug. It requires human judgment. Lucas: Exactly. And that's where the human-in-the-loop still matters. AI doesn't replace the tester — it augments them. It surfaces the most likely issues so the tester can focus their expertise. Luna: I think that's a good note to end on. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.