Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Device Testing Prevents Your App From Failing on Real Hardware
Transcript
- Lucas: Let me start with a number that should bother every QA engineer: a well-known fintech app — one with millions of users — ran its entire Android test suite on emulators. Not a single real device in the pipeline. Luna: I've heard this story before, but remind me — what happened? Lucas: They pushed an update. A week later, crash reports spiked on roughly four percent of devices. The root cause? A GPU driver quirk on certain Qualcomm chipsets — the emulator used a generic OpenGL implementation that completely masked the bug. That four percent translated to tens of thousands of frustrated users. Luna: So the emulator passed all tests, but real hardware failed. That's the classic gap. Lucas: Exactly. And the gap is bigger than most teams realise. Emulators simulate a generic hardware profile — they don't replicate thermal throttling, memory contention from other processes, or how a specific cellular modem handles network drops. Those edge cases are where real-world bugs live. Luna: I remember when Apple transitioned to their own silicon. A lot of x86 emulators ran flawlessly, but the arm native builds exposed timing bugs that only appeared on physical M1 Macs. Lucas: That's a perfect example. Apple's Rosetta 2 translation layer made x86 apps run shockingly well — but if you only tested on an Intel Mac with an emulator, you'd never catch the race condition that only triggers on actual Apple Silicon hardware. Teams that had real M1 devices in-house found those bugs pre-release. Luna: So what's the practical split? No one can buy every phone on the market. Lucas: You don't need every phone. You need a targeted device matrix based on your user analytics. If your crash data shows that eighty percent of Android crashes come from Samsung and Xiaomi mid-range models, test on those. The Pareto principle applies here — twenty percent of device models cause eighty percent of hardware-specific bugs. Luna: And cloud device farms make that accessible now. Services like AWS Device Farm or Firebase Test Lab let you spin up a real device for a few cents per minute. Lucas: Right. The cost argument against real devices has mostly evaporated. A typical cloud device farm run might cost fifty dollars for a full test cycle across twenty devices. Compare that to the revenue lost from a four percent crash rate, and the ROI is obvious. Luna: But there's still a mindset issue, isn't there? Teams treat device testing as a pre-release gate rather than a continuous process. Lucas: That's a crucial point. If you only run device tests two days before release, you find bugs you can't fix in time. You end up shipping known issues or delaying. The smarter approach is to integrate real device testing into your CI pipeline — every pull request runs a subset of critical tests on a representative device set. That way, a GPU driver bug surfaces within hours, not weeks. Luna: I read about a team that does exactly that: they have a 'smoke test' on three real devices per PR, then a full regression on fifty devices overnight. They catch hardware issues before they ever reach staging. Lucas: That's the model. And it's not just Android. iOS has its own fragmentation — different screen sizes, notch designs, Dynamic Island interactions. An emulator can approximate an iPhone 15 Pro, but it can't replicate the actual touch sensitivity or the way the notch clips a web view on a specific iOS version. Luna: So what about emulators? Should teams abandon them entirely? Lucas: No, emulators are still incredibly useful for rapid feedback during development. They're great for testing logic, UI layout, and basic interactions. The mistake is treating them as a substitute for real hardware testing. Emulators give you speed; real devices give you fidelity. You need both. Luna: That makes sense. Emulators for the inner loop, real devices for the outer loop. Lucas: Exactly. And one area where real devices are non-negotiable is performance testing. Emulators don't throttle CPU under thermal load. They don't simulate battery drain. So if your app's background refresh kills the battery on a Pixel 7, you'll never see it in an emulator. Luna: I remember a case where a social media app was draining battery excessively — turned out it was a sensor polling bug that only manifested on physical hardware. The emulator had a fixed sensor rate that masked the issue. Lucas: Sensor bugs are a whole category of their own. Gyroscope, accelerometer, compass — emulators simulate these with perfect data. Real sensors have noise, drift, and varying update rates. If your app does anything with augmented reality or step tracking, you absolutely need real devices. Luna: So what's the one change a QA team can make this week to improve their device testing? Lucas: Audit your crash reports and identify the top five device models responsible for the most crashes. Then make sure those models are in your test matrix. If they're not, add them to your cloud device farm configuration. That's a quick win that directly reduces user-facing bugs. Luna: And if you don't have crash data yet? Lucas: Then use market share data for your target region. If you're launching in India, you need to test on Xiaomi and OnePlus devices — not just Samsung and Apple. The device distribution varies wildly by geography. Luna: That's a good point. A global app without regional device testing is essentially flying blind. Lucas: And speaking of things that help you fly less blind — if today's conversation gave you something useful, this show stays ad-free entirely through listener support. You can help keep it that way at buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small gesture that goes a long way for a niche show like this. Lucas: It really does. So back to device testing — there's another layer I want to touch on: network conditions. Real devices connected to real cellular networks behave differently than emulators on Wi-Fi. Luna: Oh, absolutely. Emulators let you simulate network conditions, but it's never the same as a device switching between LTE and 5G in a moving car. Lucas: Right. And that's where you see bugs like session timeouts during handoffs or data corruption on weak signals. The fintech app I mentioned earlier also had a bug where transactions would fail silently when the network switched from Wi-Fi to cellular mid-request. Emulators didn't catch it because they don't simulate that transition realistically. Luna: So how do you test that? Drive around with a phone in a car? Lucas: That's one way, and some teams do actual field testing. But you can approximate it with network conditioning tools on real devices — throttle bandwidth, introduce packet loss, force radio state changes. The key is that you're doing it on a physical device, not a simulated environment. Luna: There's also the question of accessories. Bluetooth headphones, smartwatches, car infotainment systems — those interactions are almost impossible to emulate. Lucas: Great point. If your app plays audio through Bluetooth, you need to test latency, codec switching, and connection drop handling on real hardware. An emulator can't pair with AirPods. I've seen apps that crash when a Bluetooth call interrupts audio playback — that's a real device only bug. Luna: So the pattern is clear: emulators for speed, real devices for coverage. But what about cost management? Testing on fifty devices every night adds up. Lucas: It does, but the cost is manageable with a tiered strategy. Run critical path tests on a 'golden device' set — say, the top five devices — after every commit. Run full regression on a broader set nightly. And use cloud device farms to parallelize, so you're not paying for idle time. Luna: And some bugs only appear on specific OS versions combined with specific hardware. That combinatorial explosion is real. Lucas: That's where data-driven test selection comes in. Use your crash analytics to prioritize combinations that have actually caused issues in production. Don't test every OS version on every device — test the ones that matter based on evidence. Luna: So the takeaway is: don't trust emulators alone, build a targeted device matrix based on real user data, and integrate real device testing into your CI pipeline. Lucas: That's it. And remember that four percent crash rate from the fintech app. They fixed it by adding that specific GPU device to their test matrix, and crashes dropped to below half a percent. Real devices pay for themselves.