Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Record-and-Replay Testing Catches Heisenbugs That Unit Tests Miss
Transcript
- Lucas: So there's this class of bug that haunts every engineering team — the heisenbug. Named after the Heisenberg uncertainty principle, because the moment you try to observe it, it disappears. You run the test suite, it fails. You add a debugger breakpoint, it passes. You log every variable, the race condition resolves itself. Luna: Right, the flaky test that only fails in CI, never locally. I've chased those for days. Lucas: Exactly. And the traditional response has been to write more unit tests, more integration tests, more end to end tests — hoping to capture the exact sequence. But the problem isn't coverage. It's reproducibility. You can't fix what you can't consistently see. Luna: So what's the alternative? Because just adding more tests hasn't solved it for most teams I've worked with. Lucas: record and replay testing. The idea is straightforward: you record a real user session — every network request, every DOM event, every timer, every random number seed — and you store it as a deterministic log. Then you replay that exact sequence against your application, and you do it hundreds of times. If a heisenbug only happens on the 47th replay, you have the exact input that triggers it. Luna: So instead of trying to write a test that simulates the bug, you capture the real interaction and use it as a test case. Lucas: Precisely. And the key insight is that you're not just capturing the inputs — you're capturing the entire execution context. Tools like Replay.io and Mozilla's rr debugger let you step backward through time. You see the state of every variable at every instruction. So when that heisenbug surfaces on replay 47, you can rewind to the exact moment the state went wrong and see the root cause. Luna: Wow. That's a fundamentally different approach from breakpoint debugging. You're not guessing where to set the breakpoint — you just rewind to the failure. Lucas: Right. And it's been transformative for teams dealing with race conditions in JavaScript, or with C++ code where memory corruption is involved. I talked to a fintech startup — they handle about 2 million transactions per month — and they adopted Replay.io for their CI pipeline. In the first quarter, they reduced production incidents by 40 percent. Not just flaky tests — actual user-facing crashes and data corruption bugs that slipped through their existing test suite. Luna: Forty percent is huge. But what's the cost? Recording every session must generate a lot of data. And replaying it in CI adds time to the pipeline. Lucas: There's definitely a trade-off. The recording itself is compressed — Replay.io claims about 5 megabytes per minute of recorded session. But if you record every user, you're talking terabytes per day. So most teams only record a fraction — maybe 1 percent of sessions, or only sessions where an error boundary was hit. And the replay itself isn't free. It takes roughly real-time or a bit slower, depending on the engine. But you don't run it on every commit. You run it on a scheduled basis, or on pre-release branches. Luna: So it's a complement to unit tests, not a replacement. You still have fast unit tests for every commit, and then you have a slower record and replay suite that catches the weird stuff. Lucas: Exactly. And the nice thing is that the barrier to entry is lower than you'd think. If you're using Chrome DevTools, there's a built-in Recorder panel — it's been there since version 97. You can record a user flow and export it as a Puppeteer or Playwright script. That's basic record and replay. The advanced tools just add the deterministic replay and time-travel debugging. Luna: Let me push back on that. If I record a flow with Chrome DevTools and replay it with Puppeteer, I'm not getting deterministic replay of timers and random numbers, am I? It's still a normal end to end test that can flake. Lucas: You're right — that's the limitation. The DevTools Recorder is a great starting point for capturing the steps, but it doesn't solve the heisenbug problem because the replay isn't deterministic. For that, you need a tool that captures and replays the full execution context. Replay.io does that by instrumenting the browser's runtime. rr does it for native code by recording system call traces. So the entry-level tool gives you the workflow, but the production-grade tools give you the reproducibility. Luna: And those production-grade tools — are they practical for a small team or a startup? Or is this something only big tech companies can afford? Lucas: rr is free and open source, developed by Mozilla — though it only works on Linux and x86. So if you're a startup with a Linux server, you can use it today. Replay.io has a free tier for open source projects and a paid tier for commercial use. It's not cheap — it's priced per developer seat — but compared to the cost of a production outage that takes two developers a week to debug, it pays for itself quickly. Luna: That makes sense. And I've heard some teams use it not just for debugging, but for compliance — they record a session that demonstrates a specific user flow, and they can replay it during an audit to prove the system behaved correctly. Lucas: That's a really smart use case. Especially in regulated industries like fintech or healthcare. Instead of writing a post-hoc report, you can literally replay the transaction and say 'here's exactly what happened.' The recording becomes an immutable artifact. Luna: So if a user reports a bug, you can ask them to enable a recording, and then you have the exact steps plus the internal state. That's a game-changer for customer support. Lucas: It is. And some tools even let users opt in to recording when they encounter an error — so you get a replay of the crash without the user having to describe what they did. One company I know uses Replay.io in their React Native app, and they went from 'we can't reproduce this bug' to 'here's the exact frame where the state went wrong' in a matter of hours. Luna: Okay, I'm sold on the concept. But I want to talk about the practical implementation. If a team wants to start using record and replay today, what's the first step? Lucas: I'd say start with the low-hanging fruit: install the Chrome DevTools Recorder and record a critical user flow — say, sign up or checkout. Export it as a Playwright script and run it in CI. That gives you the basic workflow without any new infrastructure. Then, if you start hitting flaky tests that you can't reproduce, investigate rr or Replay.io for that specific environment. Luna: And for teams that are already using Playwright or Cypress? Can they plug record and replay into those frameworks? Lucas: Replay.io has a Playwright integration — you just add a custom fixture, and it automatically records every test run. So if a test fails, you get a replay link that you can share with your team. No extra configuration. And rr works with any test framework because it operates at the system call level — you just run your test under rr record, and if it fails, you can rr replay and step through. Luna: That's seamless. The recording is basically free — you only look at it when something fails. So the overhead is minimal. Lucas: Right. And that's the key — you don't replay every recording. You only replay when you need to debug. The storage cost is real, but it's a fraction of the cost of engineering time spent chasing heisenbugs. Luna: It reminds me of how good observability tools work — you capture all the data, but you only query it when there's an anomaly. Lucas: Exactly. And by the way, if you're finding these conversations useful, this podcast is entirely listener-supported — no ads, no sponsors. It stays that way because listeners like you chip in. If today's episode gave you something concrete you can use, consider supporting us at buy me a coffee dot com slash fexingo. That's buy me a coffee dot com slash fexingo. It genuinely helps us keep going. Luna: Yeah, it's a small gesture that makes a big difference for us. And we really appreciate it. Lucas: So, back to record and replay. One area I think will grow fast is in mobile testing. Right now, most mobile-heisenbug debugging is done by adding logs and hoping. But tools like Replay.io are starting to support Android and iOS. The same principle applies — capture the full execution, replay it deterministically, and step backward. Luna: That would be huge for mobile teams. I've seen apps crash on specific device models and it takes weeks to reproduce. If you could just replay the exact session from the user's phone... Lucas: Exactly. And the tooling is getting there. Replay.io's browser support is mature, and their native mobile support is in beta. rr already works on Android via the NDK. So within the next year or two, I expect record and replay to become a standard part of the mobile testing stack. Luna: Let's talk about the bigger picture. Do you think record and replay will eventually replace traditional unit testing for certain types of bugs? Lucas: No, I don't think so. Unit tests are still the fastest way to validate individual functions and catch regressions early. But for integration-level bugs — especially those involving timing, state, or third-party interactions — record and replay is far more efficient. It's not a replacement, it's a new layer in the testing pyramid. I'd put it between integration tests and end to end tests, but with more determinism than either. Luna: So the testing pyramid becomes a testing multivector? Unit, integration, record and replay, end to end, plus chaos engineering and all the other techniques we've covered on this show. Lucas: Exactly. Each technique covers a different class of failure. record and replay specifically closes the gap for non-deterministic, state-dependent bugs that have been the bane of QA for decades. Luna: And the tooling is finally mature enough that a mid-size team can adopt it without a dedicated infrastructure team. Lucas: That's the key takeaway. A few years ago, record and replay was a research project. Today, any team using Playwright or Chrome DevTools can get started in an afternoon. The hardest part is the cultural shift — getting developers to say 'let me record the bug' instead of 'let me try to reproduce it manually.' Luna: And once they do, the time saved on debugging alone can be enormous. I've heard estimates that developers spend 30 to 40 percent of their time debugging. Cutting that in half would be a massive productivity boost. Lucas: Absolutely. And it's not just about speed — it's about confidence. When you can prove that a bug is fixed by replaying the exact failing scenario and seeing it pass, you sleep better at night. Luna: Well said. And on that note, I think we've covered the essentials. For our listeners, if you're dealing with a flaky test that only fails in CI, give record and replay a try. Start with Chrome DevTools Recorder, then graduate to rr or Replay.io if you need the heavy artillery. Lucas: And if you do, let us know how it goes. We love hearing from listeners who try something they heard on the show.