Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Visual Regression Testing Catches What Unit Tests Miss
Transcript
- Lucas: You push a button, deploy a new CSS bundle, and suddenly the checkout button is two pixels to the left. Your unit tests all pass. Your integration tests all pass. But the button is wrong. Luna: And nobody catches it until a customer tweets a screenshot. That's the classic blind spot. Lucas: Exactly. And it's a blind spot that visual regression testing is designed to fill. This is Software Testing with Fexingo, episode 108. I'm Lucas. Luna: And I'm Luna. And Lucas, before we dive deeper — I want to say something about why we can do episodes like this without interruption. Lucas: Go ahead. Luna: This show doesn't run ads. No sponsors, no pre-rolls, no mid-rolls trying to sell you a testing tool. And that's a choice we've made from day one. Lucas: It's intentional. If today's tech conversation gave you something usable, and you want to support keeping it that way, the simplest thing is buy me a coffee dot com slash fexingo. No pressure. Just an option. Luna: Yeah. And it genuinely helps us keep the focus on the content, not the commercials. Lucas: Alright. Back to the button. So visual regression testing — sometimes called screenshot testing or visual snapshot testing — is exactly what it sounds like. You take a screenshot of a page or component at a known-good state, store it as a baseline, and then every time you run your tests, you compare the new screenshot against that baseline. Luna: Pixel by pixel? Like, literally comparing every single pixel? Lucas: Some tools do pixel-perfect diffing, yes. But modern tools like Percy or Playwright's built-in visual comparison are smarter. They handle anti-aliasing differences, they can ignore certain regions — like animated elements or dynamic content — and they give you a visual diff with the changed area highlighted in pink or red. Luna: So it's not just 'pass or fail.' It's 'here's exactly what shifted.' Lucas: Right. And that's the killer feature. Because a failing unit test tells you a function returned the wrong value. A visual regression tells you, 'the hero image is now overlapping the headline by three pixels.' That's actionable in a completely different way. Luna: Let's talk about where this matters most. I'm thinking about e-commerce. If your add to cart button moves even a few pixels on mobile — that's lost revenue. Lucas: Absolutely. There's a well-known case from a few years back — one of the big fashion retailers deployed a CSS update that pushed the 'add to bag' button off-screen on iPhone 6. Their unit and integration tests all passed because the button was still in the DOM and clickable by a test script. But the user couldn't see it. They lost an estimated eight percent of mobile revenue before they caught it. Luna: Eight percent. And a visual regression test would have caught that in the first run. Lucas: First run. And that's the argument for adding visual snapshots to your critical user flows. You don't need to screenshot every page. You need to screenshot the pages where a layout change costs money. Checkout, login, search results, product detail. Luna: What about SaaS apps? They're often more complex — dashboards, data tables, charts. Lucas: SaaS is actually a harder problem because the UI is more dynamic. A data table might show different numbers every time you load it. Charts animate. So you have to be strategic. You freeze the data, or you mock the API responses so the page renders consistently. Or you use a tool that lets you specify a stable state before taking the screenshot. Luna: So you're essentially creating a deterministic version of a non-deterministic page. Lucas: Exactly. And it's worth the effort, because the alternative is a UI bug that lives in production for weeks. I've seen teams add visual regression testing specifically for their billing page — because a misaligned price or a broken 'upgrade' button is a direct hit to revenue. Luna: How do you integrate this into a typical CI pipeline? Is it a separate step, or does it run alongside unit tests? Lucas: Most teams add it as a separate job — usually after the unit and integration tests pass, because visual tests are slower and more resource-intensive. You'd run them in a headless browser, maybe using Playwright or Cypress with a visual plugin, and then upload the screenshots to a service like Percy or Chromatic for diffing. Luna: And if the diff is just anti-aliasing or a one-pixel shift that doesn't matter? Lucas: That's where human review comes in. Most tools let you approve or reject diffs. You can set a threshold — ignore differences under a certain pixel count. But the key is that every diff is visible and traceable. You're not guessing. Luna: I want to talk about false positives. Because I've heard teams say visual regression testing is too noisy — that they spend more time approving diffs than actually developing. Lucas: That's a real complaint, and it usually comes down to two things. One: they're screenshotting too many pages. You don't need to screenshot every combination of state. Two: they haven't stabilized their test environment. If your test data changes every run, or if you have animated elements, you'll get false failures constantly. Luna: So the discipline is in choosing what to snapshot. Lucas: Exactly. Start with your top five user flows. The ones that generate the most revenue or the most user interactions. Screenshot each step in the flow. That might be fifteen to twenty screenshots total. Once those are stable, add more. But don't start with a thousand screenshots. Luna: What about mobile responsiveness? You'd need separate baselines for different viewports, right? Lucas: Yes. And that's one reason visual testing can get expensive in terms of compute time. Most tools let you define multiple viewports for the same page. So you might have a baseline for desktop, tablet, and mobile for your checkout page. That's three screenshots per step. It adds up. But again, you limit it to critical flows. Luna: Are there open-source options, or are the good tools mostly commercial? Luna: So the trade-off is infrastructure vs. convenience. Lucas: Exactly. If you're a small team, the free open-source route works fine. If you're at scale, the commercial tools save you a lot of time. Luna: I want to hear about a team that got this right. Do you have a specific example? Lucas: Yeah. There's a mid-size SaaS company — I'll keep them anonymous — that had a recurring problem where their pricing page would break on Safari every other release. They had unit tests. They had end to end tests. But the pricing page had a complex flexbox layout that behaved differently in Safari. The visual regression test caught it the first time they ran it. They now have a baseline for the pricing page on Chrome, Firefox, and Safari. Luna: And they run it on every PR. Lucas: Every PR. The test takes about thirty seconds. And it's prevented at least four layout bugs from reaching production in the last quarter alone. Luna: That's a solid ROI for a thirty-second CI job. Lucas: It is. And it's not just about catching bugs. It's about confidence. When you know your UI renders correctly, you can move faster. You can refactor CSS without fear. You can update your design system and see exactly what changed. Luna: Let's talk about the future. Where is visual regression testing heading? Lucas: I think we're moving toward ai driven visual testing. Tools that can understand semantic changes — like 'the button moved' versus 'the text changed' — and automatically categorize diffs. Some services already do that. They can tell you if a diff is likely a real bug or just a rendering artifact. Luna: So the reviewer's job becomes more about judgement and less about clicking 'approve' a hundred times. Lucas: Exactly. And I think we'll also see tighter integration with component libraries. Instead of screenshotting a full page, you screenshot individual components — buttons, cards, modals — and test them in isolation. That's already possible with Storybook and Chromatic. Luna: That feels more scalable. You test the building blocks, not every possible combination of blocks. Lucas: Right. And if you test each component in its various states — default, hover, active, error — you get a lot of coverage with relatively few screenshots. Luna: So the takeaway for someone listening: where should they start? Lucas: Pick one critical user flow. Maybe it's your sign-up flow or your checkout flow. Add a visual snapshot for each step. Run it in CI. Review the baseline. Then, on the next PR, see what happens when someone changes a CSS property. That one experiment will tell you more than a hundred articles. Luna: And you'll probably catch a bug you didn't know you had. Lucas: Almost certainly. And that's the moment visual regression testing clicks. Luna: Thanks for listening to Software Testing with Fexingo. We'll be back next week. Lucas: If this episode saved you from one layout bug, share it with a teammate. And if you want to support the show, you know where to find us. Luna: Yeah. buy me a coffee dot com slash fexingo. See you next time.