Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Visual Regression Testing Catches Hidden Bugs
Transcript
- Lucas: So you've got a test suite that passes every functional check — unit tests, integration tests, end to end — and you push a new CSS change to production. Then suddenly your checkout button is invisible on iPhone 15 in landscape mode. That's the gap visual regression testing is designed to fill. Luna: Right, because no functional test checks whether a button actually appears on screen. It just checks if the button element exists in the DOM. Lucas: Exactly. And that's a surprisingly common blind spot. I was reading about a major e-commerce company — I won't name them — but they had a deployment where a single CSS property changed the z-index of the checkout button on mobile. The button was still in the DOM, still clickable in theory, but visually hidden behind a hero image. They lost an estimated four million dollars in revenue over three days before they caught it. Luna: Yikes. And a visual regression tool would have flagged that the pixel output of that page changed from the baseline. Even a simple diff would have shown the button missing. Lucas: That's the core idea. You take a screenshot of a page or component — the 'baseline image' — and then after every change, you take a new screenshot and compare the two pixel by pixel. Any difference beyond a certain threshold gets flagged for human review. Luna: But not all pixel differences are bugs. Sometimes you intentionally change the layout. So how do you manage the flood of false positives? Lucas: That's the tricky part. The industry standard is to set a per-pixel color difference threshold — say, a delta-E value. If only a few pixels change within that tolerance, you ignore them. But if a whole section shifts, you get a visual diff. Tools like Applitools and Percy also use AI to ignore anti-aliasing and font-rendering differences across operating systems. Luna: So the tool is smart enough to know that a button rendering slightly differently on a Mac versus Windows isn't necessarily a bug. Lucas: Right. And you can also define regions of interest — areas where you expect changes — and ignore the rest. For example, a date stamp in the footer will change every day. You'd mark that region as 'floating' or 'dynamic content' so it doesn't trigger a failure. Luna: That makes sense. So where does this fit in the testing pyramid? Unit tests at the bottom, then integration, then end to end... where does visual regression sit? Lucas: I'd place it as a layer alongside end to end tests, but running in parallel. You can do visual regression at the component level — like Storybook or Playwright component testing — or at the full-page level, for critical paths like checkout, login, or onboarding. The key is to run it in CI before merge, not after deployment. Luna: And the earlier you catch a visual bug, the cheaper it is to fix. Classic shift-left principle. Lucas: Listen, if today's tech conversation gave you something usable — a concept, a tool name, a cautionary tale — that's exactly why we do this show. And we keep it ad-free, which means it relies entirely on a small group of listeners who chip in monthly at buy me a coffee dot com slash fexingo. That's literally what funds making this many episodes. Luna: Yeah, and it's not a big ask. If you've ever gotten a trick from this show that saved you a debugging session, that's the kind of listener who keeps the lights on. Lucas: Exactly. So — back to implementation. One mistake teams make is running visual regression only on desktop viewports. Mobile is where most layout bugs actually surface, because of limited width, dynamic fonts, and notch cutouts. Luna: You're right. And different devices have different pixel densities. A Retina display versus a standard screen — the baseline image has to account for that. Lucas: Tools like Percy handle that by rendering in a headless browser at a fixed viewport and DPI. They also use a consistent font stack so that rendering differences don't create noise. But you still need to define viewports for the devices your users actually use — iPhone 15, Samsung Galaxy S24, iPad Air, and a couple of desktop breakpoints. Luna: How many viewports is enough? I've seen teams test on twelve devices and the pipeline takes forever. Lucas: I'd say start with three: one mobile, one tablet, one desktop. You can always add more if you start seeing bugs on a specific device. The law of diminishing returns kicks in fast — two or three viewports catch probably ninety percent of visual regressions. Luna: And what about the baseline itself? When do you update it? Because if you merge a change that intentionally shifts the layout, the old baseline is now wrong. Lucas: That's where the review workflow comes in. In a typical setup, when a developer opens a pull request, the CI runs the visual tests. If there's a diff, the tool creates a side-by-side comparison and posts it to the PR. The developer reviews it and either accepts the new screenshot as the new baseline or rejects it and fixes the bug. Luna: So the baseline gets updated automatically on approval. That's neat. Lucas: It is. But you need discipline. If people blindly approve every diff without checking, the whole system becomes useless. I've seen teams set a policy that at least two people must approve a visual baseline change — one developer and one QA engineer. Luna: That makes sense. It also forces a conversation. 'Hey, did you mean to shift the navigation bar up by four pixels?' Lucas: Exactly. And that conversation is valuable because it catches not just bugs but also design drift. Sometimes a team accumulates small CSS changes over several sprints and suddenly the UI looks nothing like the design system. Visual regression testing acts as a guardrail against that. Luna: It's almost like a visual contract with the design team. You're saying, 'This is what the UI should look like, and we'll verify it on every change.' Lucas: Right. And the tooling has gotten much better. Open-source options like Playwright have built-in screenshot comparison — you can write a one-liner to take a screenshot and compare it to a stored baseline. And paid tools like Applitools offer more sophisticated analysis, like layout-aware comparison that ignores shifts in whitespace. Luna: So you don't need a big budget to start. You could hack together something with Playwright and a GitHub Action in an afternoon. Lucas: Absolutely. And I'd argue that any team shipping a web application should have at least one visual regression test on their most critical user flow. The cost of a single pixel bug — like the missing checkout button — can be orders of magnitude higher than the cost of setting up the tool. Luna: That four-million-dollar example is seared into my brain now. I'm going to go add a visual test to our login page tomorrow. Lucas: Do it. And then let us know how it goes. We might do a follow-up episode on visual regression in single-page applications, which have their own unique challenges — like handling async content loading that changes the layout after the screenshot is taken. Luna: That sounds like a good one. For now, I think the takeaway is clear: don't trust your UI to functional tests alone. Pixels matter. Lucas: They do. And catching a pixel bug before it hits production is one of the highest-leverage things a QA process can do. Thanks for listening — we'll see you next time.