Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Race Conditions in Concurrent Code
Transcript
- Lucas: So, we've all had that moment where a test passes a hundred times, then fails once, and you can't reproduce it no matter what you do. Luna: The classic heisenbug. It disappears the moment you try to observe it. Lucas: Exactly. And a huge chunk of those heisenbugs are race conditions. Today I want to talk about how to actually test for them, because the usual approach—just run the tests and hope—is not enough. Luna: So what's the alternative? I feel like most teams I know just cross their fingers and rerun the test suite until it goes green. Lucas: Right, and that's the problem. Let's anchor this with a real story. A fintech company I consulted with had a mobile banking app. Users could transfer money, and there was a balance check before the transfer. One day, they noticed a few customers had negative balances, and they traced it to a race condition in the balance update logic. Luna: How much money are we talking? Lucas: About forty thousand dollars in unauthorized overdrafts. The bug: two threads read the same balance, both see enough funds, both proceed with the transfer, and then both write back their computed value. The last write wins, and the other transfer is lost. Luna: Ouch. So how did they catch it? Was it in production? Lucas: Actually, they caught it in production because a customer complained. But after the fact, they realized their test suite never caught it because the timing window was so small. They had unit tests for the transfer logic, but those tests ran sequentially. No two threads ever touched the same balance simultaneously. Luna: So the tests were correct in isolation, but the concurrency aspect was never exercised. Lucas: Exactly. And that's the core challenge with race conditions: they only show up under specific interleavings of thread execution. To test them, you need to force those interleavings. There are a few strategies for that, and they're not mutually exclusive. Luna: Alright, let's hear them. I'm guessing stress testing is one. Lucas: Stress testing is the blunt instrument. You just run the concurrent code with many threads and hope you hit the bad interleaving. It's probabilistic, so you might get lucky, but it's not reliable. Better than nothing, but you can't prove absence of races with stress testing. Luna: Right. So what's the sharper tool? Lucas: Deterministic scheduling. You design your tests to control the order in which threads execute. For example, you can use barriers or latches to pause threads at specific points, then release them in a specific order to force a particular interleaving. That's how you can reproduce the exact race condition from the fintech app. Luna: So you're basically choreographing the chaos. But that requires you to know where the race might be, right? What if you don't know? Lucas: That's where race detectors come in. Tools like ThreadSanitizer for C and C++ or the race detector built into Go and Rust. They instrument the code to track memory accesses and report when two threads access the same memory without synchronization. They're great at catching data races, and they work during normal test runs. Luna: So you just run your tests with the race detector enabled, and it flags any problematic accesses? Lucas: Yes, and that's a huge step up from stress testing alone. The Go race detector, for instance, is built into the toolchain. You just add the -race flag when running tests. It's been a game-changer for a lot of Go projects. Luna: But doesn't that slow down the tests? Like, a lot? Lucas: It does. I've seen test runs go from a few seconds to a few minutes. That's why it's usually not enabled in production, but you can run it in CI on every pull request. It's a trade-off—you're trading test speed for catching a class of bugs that are notoriously painful to debug. Luna: Makes sense. But what about the fintech case—would a race detector have caught it? Lucas: Actually, yes. The race detector would have flagged the unsynchronized read and write to the balance variable. But they didn't have it enabled, so the tests passed. And I've seen plenty of teams that don't even know these tools exist. So the first step is often just turning on the race detector. Luna: So we've got stress testing, deterministic scheduling, and race detectors. What else is in the toolbox? Lucas: There's also static analysis. Tools that inspect the code without running it can spot patterns that often lead to races, like sharing mutable state between goroutines or threads. They're not perfect, but they can catch obvious issues early. Luna: And there's property-based testing, which we talked about in episode 143. How does that fit in? Lucas: Great connection. Property-based testing can help with concurrency too. You define properties that should hold regardless of interleaving, like 'the final balance should equal the starting balance minus the sum of all transfers.' Then you run many random interleavings and check the property. It's a way to catch logical race conditions, not just data races. Luna: That's interesting because a data race detector might miss a logical race, where the code is technically synchronized but the order of operations leads to a wrong result. Lucas: Exactly. Take the fintech example again. They might have used a mutex to protect the balance update, which would prevent a data race. But if they didn't check the balance inside the same lock, you could still have a logical race: two threads both read the balance, then both pass the check, then both update. No data race, but still wrong. Luna: Yeah, I've seen that. The lock is on the wrong granularity, or the check then act isn't atomic. Property-based testing would catch that because the invariant would be violated. Lucas: Right. So the ideal approach combines all of these: run stress tests, enable race detectors in CI, use deterministic scheduling to reproduce specific bugs, and write property-based tests for your core invariants. Luna: It sounds like a lot, but I guess it's about layering the tools. Lucas: Exactly. And I want to emphasize one thing: none of this works if you don't have a test that exercises the concurrent code in the first place. If your tests only call the function from a single thread, you're never going to catch a race. Luna: That's a good point. So how do you write a test that actually exercises concurrency? You need to spawn multiple threads and have them all hit the same code path simultaneously. Lucas: Right, and that's where stress testing comes in. But you can also do it in a more controlled way with deterministic scheduling. Like, you can use a barrier to make sure all threads reach a certain point before any of them proceeds. That way, you're forcing the concurrency to happen at a specific spot. Luna: So if you suspect a race in, say, a cache update, you can write a test that spawns ten goroutines, each trying to update the cache, and use a barrier to make them all hit the update at the same time. Lucas: Exactly. And then you run that test with the race detector enabled, and if there's a data race, it'll flag it. And if you have a property-based test that checks the cache is always consistent, it'll catch logical races too. Luna: But even with all that, there's still the chance that a race only happens in production, under specific load, right? Lucas: True, and that's why some teams use chaos engineering, which we discussed in episode 132, to inject latency or failures in production to surface those issues. But that's more about resilience than correctness. For correctness, the testing strategies we've talked about are your best bet. Luna: Alright, let's talk about integrating these into CI. I imagine race detectors can slow things down, but there are ways to make it work. Lucas: For sure. A common pattern is to have a separate CI job that runs the test suite with the race detector enabled, but only on code that touches concurrency. Or you can run it on every pull request but with a subset of tests. The key is to make it part of the pipeline so that races are caught before they merge. Luna: And deterministic scheduling tests can be part of your regular unit tests, right? They're not that slow. Lucas: Right. They're just as fast as any other test, but they take a bit of extra design work to set up the barriers and control the thread execution. Once you have them, they're deterministic, so they run the same every time. Luna: Speaking of deterministic tests, I read about a technique where you use a single-threaded executor to simulate concurrency by interleaving tasks manually. That gives you full control. Lucas: That's a clever trick. You can write a test that schedules two tasks on a fake executor and defines the order in which they run. That way, you can test every possible interleaving systematically, even if there are dozens of steps. Luna: It's like model-based testing, which we covered in episode 139. You're modeling the possible states and transitions. Lucas: Exactly. And that's a great way to prove that a lock is held correctly or that a state machine behaves well under concurrency. So, to wrap up, the key takeaway is: don't rely on luck. Use a combination of race detectors, stress tests, deterministic scheduling, and property-based tests to systematically uncover race conditions. Luna: And don't forget to enable those race detectors in CI, even if it slows things down. The cost of a race condition in production is way higher. Lucas: Absolutely. I've seen teams spend weeks debugging a race that could have been caught in seconds by a race detector. It's one of those investments that pays for itself quickly. Luna: So next time you see a flaky test, don't just rerun it. Treat it as a clue that there might be a race hiding underneath. Lucas: Well said. And on that note, if today's conversation gave you something you can use, and you want to keep these episodes free of ads, you can support the show at buy me a coffee dot com slash fexingo. It's a small way to keep us going. Luna: Yeah, every bit helps. And now, let's get back to the nitty-gritty of concurrency testing. What's one concrete thing a team can do this week to start improving their race condition coverage? Lucas: The first step is to check if your language has a race detector built in, and if so, turn it on for your test runs. Go, Rust, Java, and C++ all have options. Just that alone can surface existing races that you didn't know about. Then, start writing a few deterministic scheduling tests for your most critical concurrency paths. Luna: Good advice. And if you're not sure where to start, look at your version control history for past bugs that were hard to reproduce. Chances are they're race conditions. Lucas: That's a smart approach. Use your own history as a guide. And with that, I think we've covered the essentials. Thanks for listening, and happy testing.