Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Browser Compatibility Testing Exposes Silent Revenue Leaks
Transcript
- Lucas: So there's this story from about six weeks ago — a mid-sized e-commerce platform, we'll call them ShopFlow, not their real name — they pushed a new JavaScript framework into production, and almost immediately their checkout completion rate dropped by 3.2 percent. Luna: Ouch. Was it a bug in the new framework itself? Lucas: Turns out the bug wasn't in the framework — it was in the way it interacted with older versions of Safari. Specifically, Safari 15.x on macOS Monterey. The new framework used a modern CSS property that Safari 15 just silently ignored, which caused the 'Place Order' button to render off-screen on certain viewport sizes. Luna: So users literally couldn't see the button to click it? That's a huge silent revenue leak. Lucas: Exactly. And the team had run visual regression tests on Chrome, Firefox, Edge, and the latest Safari. They just didn't test backward. Safari 15 still has about 8 percent of the desktop browser market share globally, but for their demographic — slightly older, higher-income shoppers — it was closer to 15 percent. Luna: That's interesting. I had a similar case a few years ago when I was consulting for a SaaS company. We discovered that 12 percent of their users were on a browser called Pale Moon — it's a Firefox fork that's popular in certain privacy-conscious communities. Lucas: Pale Moon — that's an obscure one. What happened? Luna: Our automated test suite only covered Chrome, Firefox, Safari, and Edge. But Pale Moon doesn't support some of the newer Web APIs that Firefox does. So certain form submissions were just failing silently. We estimated it would have caused about a 1.5 percent drop in monthly active users if we hadn't caught it. Lucas: That's the thing — browser compatibility testing is often treated as a checkbox exercise: run your screenshots on the big four and call it done. But the real value is in understanding your actual user browser distribution, and then testing the long tail. Luna: Right. And the long tail can be surprisingly large. If you look at analytics data for any decent-sized web app, the top ten browsers might cover 95 percent of traffic, but that remaining 5 percent could be dozens of different browser versions and forks. Lucas: And that 5 percent might be your most engaged users. So how do you prioritise? I think there are three levels of browser compatibility testing. Level one is manual cross-browser checks — literally opening the app in different browsers and clicking around. That's slow and error-prone, but it catches the most obvious issues. Luna: Level two being automated screenshot comparison? Lucas: Right. Tools like Percy or Applitools — they take screenshots of every page or component in multiple browsers and compare them pixel by pixel. That's great for catching visual regressions, but it has a blind spot: it can't detect functional issues. A button might look fine but not actually work. Luna: So level three would be automated functional testing on real devices or emulators. Lucas: Exactly. Services like BrowserStack or Sauce Labs let you run your actual E2E tests against hundreds of real browser os combinations. The downside is cost and speed — running a full test suite against 50 browsers can take hours and cost hundreds of dollars per month. Lucas: But here's the trick: you don't need to test all 50 on every commit. You can run a quick smoke test against the top five in CI, and then schedule a full compatibility run overnight or weekly. That way you catch the big stuff fast, and the long-tail issues within a day or two. Luna: If today's tech conversation gave you something usable — maybe a new way to think about your own test coverage — these episodes stay ad-free because of listener support. You can contribute at buy me a coffee dot com slash fexingo. Lucas: Yeah, and it genuinely makes a difference. Even a small contribution helps keep the show independent and focused on the details that matter. Luna: So back to that long-tail problem — what about the decision to drop support for legacy browsers altogether? That's a conversation I see a lot in engineering teams. Lucas: Right, and it's a business decision as much as a technical one. Shopify famously dropped support for Internet Explorer 11 in 2022, but they did it after publishing data showing that only 0.2 percent of their merchants' traffic came from IE11. And they gave a six-month warning. Luna: That's a good model. But the threshold depends on your user base. If you're building a healthcare portal for patients over 65, you might need to support older browsers much longer. Lucas: Absolutely. And there's a middle ground: you can serve a 'degraded but functional' experience to older browsers. For the ShopFlow case, the fix wasn't to drop Safari 15 — it was to add a fallback CSS rule that positioned the button correctly in all browsers. Luna: Which raises another point — feature detection versus browser detection. Instead of checking the user agent string, you can check if the browser supports a particular CSS property or API, and then provide a polyfill or fallback. Lucas: That's the modern best practice. The 'can I use' website is a great resource for that — or tools like Modernizr that automate the detection. But you still need to test, because polyfills can introduce their own bugs. Luna: Another angle I've seen teams miss is testing on different operating systems within the same browser. Chrome on Windows and Chrome on macOS can render things slightly differently, especially with fonts and subpixel rendering. Lucas: Yeah, font rendering is a classic. On Windows, ClearType makes text look thicker; on macOS, it's smoother. If you have a design that's pixel-precise with tight spacing, you might get text overlap on one platform. Lucas: And then there's mobile — not just mobile Safari versus Chrome, but also Samsung Internet, which is the default browser on Samsung phones and has around 5 percent global market share. It's based on Chromium but has its own quirks. Luna: I remember a bug where Samsung Internet didn't support the 'backdrop-filter' CSS property in certain versions, so a modal overlay with a blur effect just showed a solid grey box. It looked broken. Lucas: Right, and that's the kind of thing visual regression might catch — but only if you're testing on actual Samsung devices or a reliable emulator. The Chrome DevTools device emulator doesn't replicate Samsung Internet; it only emulates Chrome on a Samsung phone. Luna: So what's your recommended approach for a team that wants to improve their browser compatibility testing without blowing their budget? Lucas: First, audit your analytics to find your real browser distribution. You might be surprised. Second, pick a threshold — say, 99 percent of your traffic — and test every browser os combination in that set. Third, run a fast smoke test on the top five in CI, and a full compatibility suite weekly. Lucas: Fourth, for the long tail below your threshold, use feature detection and graceful degradation rather than trying to test every combination. And fifth, monitor your conversion metrics after every release — if you see a drop, check browser-specific analytics immediately. Luna: That fifth point is key. The ShopFlow team only noticed the issue because they had real-time dashboards showing checkout completion by browser. Without that, the 3.2 percent drop might have been attributed to seasonality or marketing. Lucas: Exactly. And the fix cost them maybe a day of engineering time. The revenue loss over that week was probably in the tens of thousands. So the ROI of good browser compatibility testing is enormous when you do it right. Luna: I think the takeaway is: don't assume your users are on the same setup as your developers. And don't let the complexity of the browser landscape paralyse you. Start with your analytics, test the top, monitor the rest. Lucas: Well said. And if you have a story about a browser compatibility bug that cost your company money, we'd love to hear it. You can reach us through the show's page. Until next time, keep testing.