Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Testing Mobile Apps on Real Devices Still Matters
Transcript
- Lucas: So, Luna, you remember that fintech app we use for splitting dinner bills? The one that crashed on my phone right when I tried to pay for a group order? Luna: Oh, yeah. That was annoying. But I thought they rolled out a fix after the initial crash reports. Lucas: They did. But the whole incident got me thinking about how that bug slipped through their testing. Because they told me later they had full emulator coverage. Every screen, every flow, automated tests running every day. But it still broke on real hardware. Luna: That's the classic emulator blind spot. What was the actual cause? Lucas: It was a connectivity handoff issue. The app would crash when you moved from Wi-Fi to a weak cellular signal while a payment request was still in flight. In an emulator, your network is basically a fixed simulated pipe. You can slow it down, sure, but you can't easily reproduce the exact chaotic way a real radio drops and reacquires signal. Luna: Right. And emulators don't have real radios. They're just virtualizing the CPU and the OS, not the actual antenna hardware. Lucas: Exactly. So that got me looking into why this still happens in 2026, even after years of mobile testing best practices. And the answer is that teams are under so much pressure to keep test suites fast and cheap that they lean on emulators and simulators almost exclusively. Luna: And then they ship a release and hope for the best. But I imagine there's a whole range of issues that only show up on physical devices. Battery drain, thermal throttling, camera behavior, sensor accuracy. Lucas: All of it. And the thing is, emulators are great for catching logic bugs and basic UI regressions. Nobody's saying they're useless. But they operate in a idealized environment. The CPU is effectively infinite. There's no other app competing for memory. Screen sizes are predictable. And the OS is a clean slate, which rarely matches what your users actually have. Luna: Yeah, not to mention the hardware variability. My phone has a weird custom chipset, and your budget Android is a totally different beast. Lucas: That's the device fragmentation problem. In a real world, your app needs to work on thousands of distinct device configurations. Emulators only mimic a handful of reference devices, and even those are approximations. A real device has a specific GPU, a specific modem, a specific battery management system — and all of those interact with your app in ways you can't fully predict. Luna: So what's the practical fix? I mean, buying a hundred physical phones for every team in the company isn't realistic. Lucas: Right. That's where the device matrix idea comes in. Instead of trying to test on everything, you pick a representative subset of devices that cover the most common chipsets, screen sizes, OS versions, and radio types. Typically you look at your analytics to see what your actual users are on, and you build your matrix around that. Luna: So it's a data-driven approach. But even a small matrix of ten devices adds up in cost if you're buying them outright. Lucas: That's why cloud device farms have become so popular. Services like Firebase Test Lab, AWS Device Farm, or even open-source setups like OpenSTF let you rent time on real hardware. You pay per minute, you can run your tests in parallel, and you get access to a much wider range of devices than you could ever buy. Luna: And that's a game changer for small teams. You don't need a hardware lab in the office. You just spin up a test run in the cloud. Lucas: Yeah. But even then, you're still not testing on every device out there. The key is to use emulators for the bulk of your automated tests — the ones that run on every commit — and then have a separate, smaller suite that runs on real devices, either in the cloud or on a few in-house physical devices, before every release candidate. Luna: That hybrid approach makes sense. But I've also seen teams skip real device testing entirely because they think a modern emulator is good enough. How do you convince them otherwise? Lucas: You point to bugs like the one in that payment app. And that's actually a perfect example to bring up here, because it's a perfect illustration of why real device testing is so important. And honestly, if today's conversation gives you one useful idea about your own testing strategy, that's a win for us. We keep making these episodes thanks to listeners who decide to support the show. If that's you, you can find us at buy me a coffee dot com slash fexingo. It's a small gesture, but it genuinely helps keep the podcast running ad-free. Luna: Yeah, and it's completely optional. We just appreciate you being here. So back to that bug — what did the team at that fintech app actually change after the incident? Lucas: They built a small device matrix around their top five devices, based on their user analytics. And they added a nightly test run on a cloud device farm that exercised the core payment flow under simulated network changes, like switching from Wi-Fi to cellular. Luna: And did that catch anything else before the next release? Lucas: It actually caught a similar issue on a budget Android phone, where the app would freeze when the screen rotated during a payment. That was a pure device-specific bug that wouldn't show up in an emulator because the virtual screen rotation doesn't trigger the same hardware reconfiguration. Luna: That's a great example. So it's not just about network or battery — it's about all the hardware-specific behaviors that are impossible to simulate faithfully. Lucas: Right. And the cost of skipping that step can be huge. There's a classic study from a few years back that estimated the average cost of a critical bug in production at around five hundred thousand dollars, once you factor in fixes, lost revenue, and reputational damage. Luna: And that's on the low end for some industries. A hospital app or a banking app could be even worse. Lucas: So when you compare that to the cost of a cloud device farm subscription — which can be as low as a couple hundred dollars a month for a small team — it's a no-brainer. The ROI is just so obvious. Luna: But I'm guessing there's a learning curve. You can't just write a test on an emulator and expect it to run perfectly on a real device. There are timing issues, UI differences, and all sorts of quirks. Lucas: Absolutely. That's something a lot of teams underestimate. When you first run your test suite on real devices, you'll likely see a lot of failures that are actually test issues, not app issues. Tests that rely on exact pixel positions might break because of different screen densities. Tests that assume a certain network latency might time out on a slower connection. Luna: So you have to invest in making your tests robust enough to handle real-world variability. But that's part of the value — it forces you to write better tests. Lucas: Exactly. And once you get there, you have a much higher confidence in your release. The question is, how do you decide which devices to include in your matrix? There's no one-size-fits-all answer, but a good starting point is to focus on your most valuable users. Luna: You mean the users who generate the most revenue or engagement. So if you're a shopping app, you'd prioritize devices used by your most frequent purchasers. Lucas: Right. And you also want to include some older, low-end devices to make sure your app doesn't become a bloated resource hog. It's a balancing act between coverage and cost. But the key is to be intentional — don't just test on whatever devices happen to be sitting in the office. Luna: Let's talk about the future. With so much moving to the cloud and the rise of on-device AI, how is real device testing going to evolve? Lucas: That's a great question. One trend I'm seeing is the use of artificial intelligence to automatically pick the most impactful devices for your matrix, based on your app's features and your user base. There are also tools that can predict which devices are likely to have issues based on historical crash data. Luna: So instead of manually curating a device list, you let the data guide you. That could save a lot of time and guesswork. Lucas: Definitely. And another trend is the integration of real device testing directly into continuous integration pipelines. You can now trigger a cloud device farm run automatically whenever a pull request is merged, and get results back in minutes. Luna: That's making it a lot more practical for teams to adopt a hybrid strategy without a huge operational burden. Lucas: Yeah, and I think that's the direction we're heading — where real device testing becomes as routine as running unit tests. The tools are getting faster, cheaper, and more accessible. The main barrier is just mindset. Once you've been burned by a bug that only appears on a real device, you never want to skip that step again. Luna: It's like the old saying: trust, but verify. Emulators give you a decent approximation, but real devices give you the truth. Lucas: Well said. And with that, we'll leave you with a question to ponder: what's in your device matrix, and does it reflect reality? Thanks for listening.