Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Crowdsourced Testing Catches Edge Cases Your QA Team Misses
Transcript
- Lucas: So here's a question: how many devices does your QA team actually test on? Luna: If I'm honest, probably two. My iPhone and a Samsung loaner. Lucas: Right. And that's pretty typical — most internal teams have maybe a handful of reference devices. But the real world runs on thousands of different hardware-software combinations, and that's where crowdsourced testing comes in. Luna: Crowdsourced testing — like paying a global network of testers to run your app on their personal devices. Not quite bug bounties, but related. Lucas: Exactly. Platforms like uTest and Bugcrowd have been around for a while, but I think they've become much more relevant in the last two years — especially as device fragmentation in Android has actually gotten worse, not better. Luna: Worse how? I thought the Pixel and Samsung dominance was streamlining things. Lucas: That's true for the US market. But globally, you've got Xiaomi, Oppo, Vivo, Tecno — especially in India and Africa — running custom ROMs that behave differently. And then there's the Chinese Android ecosystem, which is basically a separate universe. Luna: So a real-world example — what's the best crowdsourced testing case study you've seen? Lucas: There's a fintech app — I won't name it, but it's a well-known neobank — that ran a crowdsourced test on uTest back in 2025. They had 15,000 testers across 90 countries, all running the same test script on their personal devices. Luna: Fifteen thousand testers. That's like an army. Lucas: It is. And the results were eye-opening. In two days, they found 47 unique bugs that had never been caught by their internal QA or their automated regression suite. One of them was a currency-format crash that only appeared on a specific Chinese Android build — a Lenovo tablet running a localized version of Android 14. Luna: A Lenovo tablet in China — how would an internal team ever catch that? Lucas: They wouldn't. Unless they happened to have that exact device on their test bench, which they didn't. And that's the whole point — crowdsourced testing gives you access to a long tail of device configurations that no single lab can replicate. Luna: But at what cost? Fifteen thousand testers can't be cheap. Lucas: Actually, it's surprisingly affordable. Most crowdsourced platforms work on a pay per bug model or a fixed-price per test cycle. That neobank paid around $30,000 for the full two-day run. Compare that to the cost of a single production outage that affects thousands of users. Luna: Thirty thousand dollars for 47 bugs — that's about $638 per bug. If even one of those was a revenue-critical bug, you're way ahead. Lucas: Exactly. But I want to be careful here — crowdsourced testing isn't a replacement for your internal QA or your automated test suite. It's a complement. The sweet spot is using it for specific high-risk releases, major feature launches, or when you're entering a new geographic market. Luna: So when does it make sense to skip it? I'm thinking of early-stage startups with very simple apps. Lucas: Absolutely. If your app is a CRUD form with no localization, no payment processing, and you're targeting a single market — say, us only — then crowdsourced testing is probably overkill. Your automated tests plus a small beta group might be enough. Luna: What about the regulatory angle? For fintech or healthcare, do you run into privacy issues sending your app to thousands of strangers? Lucas: It's a real concern. Most platforms handle it by running testers through background checks and signing NDAs. And the apps are usually sandboxed — you give them a test build that connects to a staging environment, not production. No real user data ever touches their devices. Luna: Still, it's a leap of faith for some compliance teams. Lucas: It is. But the numbers from that neobank case study — and others like it — are making the argument harder to ignore. When you consider that the average mobile app has a bug density of about 1 bug per 1000 lines of code, and that edge-case bugs are exactly the ones that slip through traditional testing, the risk of not testing broadly becomes the bigger concern. Luna: Do you think ai powered testing tools are making crowdsourced testing less relevant? Some of those visual regression tools we talked about in episode 61 can simulate a lot of devices. Lucas: It's a great question. ai driven visual regression tools are fantastic for catching pixel-level differences — layout shifts, color mismatches, that kind of thing. But they can't test actual device behavior. Does the app crash when you rotate the screen on a specific Motorola model? Does the NFC payment flow work on a Xiaomi? Those are real-device interactions that emulators can't replicate. Luna: So AI plus crowdsourcing — that's the combo. Lucas: I think that's where the industry is heading. Use AI for broad coverage on synthetic tests, and use crowdsourcing for targeted deep dives on real devices in markets you care about. Luna: Alright, let me push back a little. For a small team, managing a crowdsourced test cycle sounds like overhead. You have to write test scripts, triage the bugs, communicate with testers... Lucas: That's fair. The platforms do provide project managers who handle a lot of that — you just tell them what you want tested and they coordinate the testers. But there is still an upfront investment. You need to define your test scenarios clearly, or you'll get a bunch of irrelevant bugs. Luna: Have you seen any interesting trends in how companies are blending crowdsourced testing with their CI/CD pipelines? Lucas: Yes, and this is the part I find most promising. Some teams are now triggering crowdsourced test runs automatically after a successful staging deployment. So your CI pipeline runs unit and integration tests, then — if those pass — it sends a build to uTest or similar for a quick exploratory session. The results come back within hours. Luna: That's essentially continuous testing with human eyes. Lucas: Exactly. And the feedback loop is tight enough that you can fix bugs before the release candidate even goes to QA. It's a form of shift-left testing that actually scales. Luna: I want to come back to something you mentioned earlier — the long tail of devices. Is there a rule of thumb for how many devices you need to cover? Lucas: There's no hard number, but a common approach is the '80/20 rule with a twist.' Cover the top 20 devices in your target market — that gets you about 80% of your user base. Then use crowdsourcing to cover the next 200 devices, which adds maybe another 10% coverage but catches those weird edge cases. Luna: And that last 10% — the really obscure devices — you just accept the risk unless you're a global app. Lucas: Right. For most apps, the cost of testing on everything isn't justified. But for a fintech or healthcare app, that edge case might be a regulatory violation or a security breach. So the risk calculus shifts. Luna: I think this is really valuable — especially for teams that are expanding internationally. Lucas: If today's conversation gave you a concrete idea about how to broaden your testing approach, that's exactly the kind of insight we try to deliver here every week. And the way we keep these episodes ad-free is through listener support — if you found it useful, you can buy me a coffee dot com slash fexingo. Luna: It's a small way to keep the content independent and focused on what matters. Lucas: So the takeaway: crowdsourced testing is not a silver bullet, but it's a powerful layer in your test strategy — especially for catching the bugs that only appear on one weird Android build in a faraway market. And if you're building for a global audience, that's exactly the kind of testing you need. Luna: Thanks, Lucas. I'll be looking into that neobank case study a bit more. Lucas: Good idea. And if you try crowdsourcing for your next release, let us know how it goes.