Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Visual Regression Testing Catches Layout Bugs Before Users Do
Transcript
- Lucas: So you push a CSS change — just tweaking the margin on a button — and suddenly your entire checkout flow looks like it was designed by a toddler. The button is half off-screen, the text overlaps, and somehow the logo is now in the footer. Luna: I have definitely been there. And the worst part is, functional tests might still pass because the logic works — it's just visually broken. Lucas: Exactly. That's exactly the problem visual regression testing is built to solve. The idea is straightforward: you take a screenshot of your app before a change, then another after, and you compare them pixel by pixel — or using smarter diffing algorithms — to flag any unintended visual differences. Luna: And this matters way more than it sounds like. In 2023, Twitter — back when it was still called Twitter — had a famous incident where a CSS change to the checkout button margin caused the button to render at half its intended width on mobile devices. Lucas: Right. And the crazy part is, their functional test suite passed because the button was still clickable and the API calls worked fine. But visually, it looked broken — and they estimated it cost them a measurable dip in conversion for a couple of days before someone caught it. They were using a tool called Percy at the time for visual regression, but that particular flow wasn't covered. Luna: So the lesson is: you can't just test functionality. The visual layer is a first-class concern, especially in consumer-facing apps where trust and aesthetics matter. Lucas: Right. And the industry has caught on. Companies like Airbnb and Stripe have been vocal about using visual regression testing as part of their CI pipeline. Airbnb even open-sourced their own tool called 'Screenshot Test' for React Native. Luna: How do these tools actually work? I mean, comparing every pixel sounds both expensive and noisy — what if something legitimately changes, like an A/B test variant? Lucas: Great question. Early tools did a straight pixel-by-pixel comparison, and they were indeed noisy. Modern tools use 'perceptual diffing' — they blur the images slightly and compare the perceived differences, which filters out anti-aliasing artifacts and minor color variations that humans wouldn't notice. Luna: And they let you set a threshold. So you can say: ignore differences under 5 percent, but flag anything larger. Lucas: Exactly. And you can also define 'regions of interest' — maybe you only care about the checkout form, not the background image that changes every day. Tools like Percy, Chromatic, and Applitools all support that kind of configuration. Luna: One thing I've found is that deciding what to test is the hardest part. When my team first started, we tried to screenshot every page — and the noise was unbearable. Every time we updated a date or a user avatar, the test failed. Lucas: That's a classic rookie move. The smart approach is to focus on your 'golden paths' — the user flows that directly impact revenue or retention. For an e-commerce site, that's product search, product detail, cart, and checkout. For a SaaS product, it might be login, dashboard, and billing. Luna: We ended up with about 15 critical screens per app. And we took the screenshots at the component level too, not just full pages. That way, when a button changes, you know exactly which component broke. Lucas: Another best practice is to run visual tests on every pull request, not just before release. That way you catch the regression the moment someone introduces it, instead of two weeks later when you're doing a release candidate. Luna: But that also means you need to manage the baseline images effectively. If someone intentionally updates a design, you need to approve the new screenshot as the new baseline. Lucas: Most tools have a review workflow for exactly that. A developer or QA engineer reviews the diff, and if the change is intentional, they accept it. That becomes the new baseline. It's a lot like code review, but for pixels. Luna: There's also the question of cross-browser and cross-device testing. A layout that looks fine on Chrome might break on Safari or on an older iPhone. Lucas: Yeah, that's where cloud-based testing services come in. Tools like BrowserStack and Sauce Labs integrate with visual regression tools so you can run the same screenshots across hundreds of device-browser combinations. It's expensive, but for a consumer app, it's worth it. Luna: Let me bring up a real example from my own experience. Last year, we had a single CSS change that set 'overflow: hidden' on a parent container. It was meant to fix a scroll bug on desktop, but it completely clipped the mobile navigation menu. For two days, mobile users couldn't navigate to any page except the home page. Lucas: Ouch. And I'm guessing that wasn't caught by functional tests because the navigation still rendered — it was just off-screen. Luna: Exactly. The menu was in the DOM, it was accessible via keyboard, but visually it was hidden. It wasn't until a user tweeted a screenshot that we realized. That's when we started taking visual regression seriously. Lucas: And that's the thing — users don't care if your code is technically correct. They care if the app looks and feels right. A broken layout erodes trust fast. Luna: So if a team is just getting started, what's a practical first step? Do you have to buy a tool, or are there open-source options? Lucas: You can start with free tiers or open-source tools. Percy has a free tier for small projects. Chromatic, which is built for Storybook, has a generous free tier too. On the open-source side, there's 'BackstopJS' and 'Wraith' — both are solid, though they require more setup. Luna: And I'd say start with your top three critical flows. Don't try to cover everything at once. Get those running in CI, with a review process, and then expand gradually. Lucas: I'd add one more thing: make sure your test runs are deterministic. If you're testing a page that shows live data, you need to stub that data so the screenshot is the same every time. Otherwise you'll get false positives from changing numbers or timestamps. Luna: That's a big one. We use fixtures and mock API responses for our visual test pages. It takes some upfront work, but it saves hours of false-positive triage later. Lucas: Alright, let's take a quick beat here. If today's conversation gave you something useful — maybe a new testing idea or a tool to try — I want to mention something honest. This show is entirely listener-supported. No ads, no sponsors, no corporate overlords. A handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these episodes. Luna: Yeah, and it's not a pressure thing — it's just the reality. If you find value in the show and want to keep it going, that's the way. No strings attached. Lucas: Exactly. So back to visual regression. One emerging trend I'm watching is the use of AI to automatically categorize diffs — for example, flagging whether a change is a true regression or an intentional design update. Tools like Applitools already do this with their 'Visual AI' engine. Luna: That would save a lot of manual review time. I can imagine teams that do hundreds of visual diffs a day really benefiting from that. Lucas: Right. The other thing is integrating visual regression with your component library. If you use Storybook, Chromatic can automatically snapshot every component variant every time you commit. It's a powerful way to catch regressions at the atomic level. Luna: We actually do that now. Every component in our design system has a Storybook story, and Chromatic runs on every PR. It's caught so many tiny regressions — like a button padding that got accidentally removed in a refactor. Lucas: That's the ideal setup. The earlier you catch a visual bug, the cheaper it is to fix. A misaligned button caught in code review costs five minutes. That same bug caught in production costs a support ticket, a hotfix, and maybe lost revenue. Luna: And it's not just about bugs — it's about consistency. Users expect the app to look the same across pages and sessions. Visual regression testing enforces that consistency automatically. Lucas: So if you're not doing visual regression testing yet, start small. Pick one critical flow, set up a free tool, and run it on your next PR. I think you'll be surprised how much it catches. Luna: And if you already do it, maybe share your setup with a team that doesn't. It's one of those practices that pays for itself really quickly. Lucas: Alright, that's our episode on visual regression testing. Until next time.