Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Visual AI Testing Catches Layout Bugs Humans Miss
Transcript
- Lucas: You've got a test suite that passes — unit tests green, integration tests green, end to end tests green. But you ship and suddenly users are seeing a broken layout. Buttons overlapping, text cut off, images misaligned. This happens more often than most teams want to admit. Luna: Right, because traditional tests check logic, not appearance. They assert that a function returns the right value, not that the page actually looks like it should. Lucas: Exactly. And that gap is where visual AI testing comes in. Instead of writing assertions about pixel coordinates or CSS properties, you take a screenshot — a snapshot — and compare it against a baseline image. Any difference gets flagged for review. Luna: So it's like a diff tool for the UI. I've seen teams use Applitools or Percy for this. They run in CI, right? Lucas: Yeah, they integrate into your pipeline. When you push a change, the tool takes screenshots across viewports — mobile, tablet, desktop — and compares them pixel by pixel. And here's the key: it uses AI to reduce noise. Not every pixel shift is a bug. Luna: Like anti-aliasing differences or font rendering variations across operating systems. That can produce false positives if you're doing a naive pixel compare. Lucas: Exactly. Applitools calls that their 'visual AI' — it learns which changes are meaningful. A one-pixel shift caused by a different browser engine? Probably ignore. A button that moved ten pixels down? That's a layout regression. Luna: So it's smarter than a simple screenshot diff. But it still needs a human to review and approve the baseline, right? Lucas: Right. You set the initial baseline, and then every change either matches or triggers a review. And the team can either accept it — if it's intentional — or fix it. This catches things like a CSS change that breaks a flexbox layout only on iPhone SE, which your unit tests never touch. Luna: I've seen that exact scenario. A fintech dashboard I worked with had a table that looked fine on a 27-inch monitor. But on a 375-pixel-wide phone, the columns collapsed and numbers overlapped. All tests passed because the logic was sound. Visual AI caught it. Lucas: That's a perfect example. And that's actually the case I want to drill into today — a fintech app that passed every traditional test but had a visual regression on mobile that would have confused users and potentially caused errors in data entry. Luna: Before we go further — if this kind of practical testing insight is useful to you, the way the show stays ad-free is through listener support. You can find us at buy me a coffee dot com slash fexingo. It's a small way to keep these deep dives coming. Lucas: Appreciate that. So let's get into the fintech dashboard. The team was using a standard React component library, and they had comprehensive unit tests with Jest and end to end tests with Cypress. Everything passed. Luna: But then they added a new feature: a dynamic table that resized columns based on content. The developer wrote a CSS rule using 'max-width' on a cell — but on narrow screens, the table overflowed its container. Lucas: Right. The Cypress test only checked that the table rendered and that clicking a row navigated somewhere. It never checked that all the cells were visible. So they shipped it. Luna: And production monitoring didn't catch it immediately because the layout broke only on devices with a viewport width under 400 pixels. That's a small percentage of users, but for a fintech app, even a few confused users is a problem. Lucas: So they added visual AI testing. They used Percy, which integrates with their existing Cypress test suite. The test takes a screenshot of the table page at three breakpoints. And the first time they ran it, the mobile snapshot flagged a significant difference. Luna: What did the diff show? Lucas: A horizontal scroll bar appeared on the table container. The columns were too wide, so the table extended beyond the screen. A user would have to scroll left and right to see all the data — which is terrible UX and could lead to misreading financial figures. Luna: And the developer? They probably thought they'd tested it. Lucas: Exactly. The Cypress test passed because it used a 1280-pixel viewport. The developer was testing on a 13-inch laptop. The visual AI test at 375 pixels caught it immediately. They fixed the CSS, and the next snapshot passed. Luna: So the key is having those viewport-specific baselines. And I assume you can set thresholds — like 'ignore up to 5% difference' to handle intentional changes? Lucas: Yes, and that's where the AI comes in. You can set a 'match level' — a tolerance for how similar an image must be to pass. For critical pages like a checkout flow or a transaction screen, you set it strict. For less critical pages, you allow more variance. Luna: So you're balancing between catching real bugs and not blocking deploys with noise. That's a tuning exercise. Lucas: Right. And teams typically start with a strict threshold and then relax it as they learn what changes are intentional. Over time, the AI adapts. For example, if you regularly update a banner image, the tool learns that the banner region is dynamic and flags it as a known change. Luna: That's smart. But what about dynamic content? Like a user's name or a live stock ticker — those change every time. How do you snapshot that? Lucas: Good question. You either mock the data so it's deterministic, or you use region-based ignoring. Most tools let you define a 'mask' — a region of the screenshot that's excluded from comparison. So you mask the user avatar, the ticker price, the date stamp. Luna: So you're checking the layout structure, not the dynamic data itself. That makes sense. Lucas: Exactly. You're verifying that the elements are in the right place, sized correctly, not overlapping. And that's something no assertion-based test can do easily. You'd have to write dozens of assertions checking each element's bounding box. Visual AI replaces all of that with one snapshot. Luna: But it's not a silver bullet. What about animations or transitions? Those can cause false positives because the snapshot captures a mid-animation frame. Lucas: Right. You need to wait for animations to finish before taking the screenshot. Most visual testing tools have a 'fully loaded' trigger — you wait for a certain element or a timeout. Also, you can use 'hardcoded' delay, but that's fragile. Luna: So best practice is to disable animations in test environments. Or use a CSS class that sets animation duration to zero. Lucas: Yes, that's common. You have a test-specific stylesheet that disables transitions and animations. That way the snapshot is deterministic. Then in production, animations run fine. Luna: So visual AI testing is really about adding a layer on top of your existing test pyramid. It doesn't replace unit or integration tests. Lucas: No, it complements them. Unit tests verify logic. Visual tests verify UI. And the two together catch more bugs than either alone. In fact, a study from a large e-commerce company found that 30% of their production UI bugs were not caught by any functional test — only by visual regression tests. Luna: Thirty percent is huge. That's a lot of bugs that would have reached users. Lucas: Yeah. And those bugs are often the most visible ones — literally. A broken layout is immediately obvious to a user and erodes trust. So adding visual AI testing is relatively cheap compared to the reputational cost of a bad UI release. Luna: I want to talk about the tooling landscape for a second. Percy is one option. Applitools is another. There's also Chromatic for Storybook. How do teams choose? Lucas: It depends on their stack. Percy integrates tightly with Cypress and Playwright. Applitools has more advanced AI features like root cause analysis — it tells you not just that something changed, but which CSS property caused the change. Chromatic is great if you're using Storybook for component development. Luna: And pricing? These tools aren't free for high-volume use. Lucas: They're not cheap. Most charge per screenshot or per test run. For a small team, it's manageable — maybe a few hundred dollars a month. For a large enterprise running thousands of snapshots per build, it can get expensive. But compared to the cost of a production bug, it's often worth it. Luna: So the ROI is clear if you're shipping UIs frequently. But I'm curious about integration with CI. How fast are these tests? Lucas: They're fast — typically a few seconds per snapshot. But if you take screenshots at five viewports for fifty pages, that's 250 snapshots. That can add minutes to CI. Teams often run visual tests only on critical pages in the main pipeline, and run the full suite nightly. Luna: So you parallelize the snapshot capturing across multiple CI nodes. That's common. Lucas: Yes, and tools like Percy support parallel builds. You split the snapshot list across runners, upload them, and the service aggregates the results. That keeps the pipeline fast. Luna: One more thing — what about accessibility testing? Can visual AI catch contrast issues or missing alt text? Lucas: Not directly. Visual AI is about layout, not semantics. For contrast ratios, you still need axe-core or Lighthouse. But some tools like Applitools have started combining visual and accessibility checks — they can flag regions where text might be hard to read due to overlapping elements. Luna: So it's complementary to a11y testing too. I think the takeaway is that visual AI testing closes a gap that traditional testing leaves open. It's not a replacement, but a necessary addition. Lucas: Exactly. And with more teams shipping responsive designs and using component libraries, the risk of layout regressions grows. Visual AI is becoming a standard practice, not a niche tool. Luna: So if you're a team that hasn't tried it, start with one critical page — say, your sign-up flow or checkout. Add a snapshot test at mobile and desktop. See how many false positives you get. Then tune from there. Lucas: That's exactly the right approach. Start small, prove the value, then expand. It's one of those things where the first bug it catches pays for the tool. Luna: And for our listeners, if you want to dig deeper into the tooling, we'll put links in the show notes. But for now, I think we've covered why visual AI matters. Lucas: Yeah, next time you push a UI change and all tests pass, ask yourself: did I actually look at it on a phone? If not, visual AI might be your safety net.