Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How State-Based Testing Catches Integration Bugs
Transcript
- Lucas: So you've got a ride-hailing app. A user requests a ride, a driver accepts, the ride starts, ends, payment processes. That sequence of states — requested, accepted, in-progress, completed, paid — is the ride's lifecycle. And most teams would write tests for each step: does the request create a ride? Does acceptance update the driver? But here's the blind spot. Luna: Let me guess — they test each state transition in isolation, but never test what happens when the system jumps to an illegal state? Lucas: Exactly. Or when two transitions happen at once. That's where state-based testing comes in. Instead of just testing inputs and outputs, you model the system as a finite state machine — explicit states and allowed transitions — then generate tests that walk through valid and invalid paths. Luna: So it's like a graph of states, and your tests traverse edges. I've seen this called 'model-based testing' sometimes too. Lucas: Right. Model-based is broader — you could model any system behavior. State-based is a specific flavor where the model is a state machine. The key insight is that many critical bugs aren't about wrong output for a given input; they're about the system being in the wrong state or taking an illegal path. Luna: Give me a concrete example. That ride-hailing app — what kind of bug does state testing catch that unit tests miss? Lucas: Imagine a rider cancels the ride right as the driver marks it as 'picked up'. In a traditional test, you'd test cancellation separately and pickup separately. But in production, those events can arrive in the same millisecond. If your database updates are non-atomic, you could end up with a ride that's both 'cancelled' and 'in-progress' — an orphan ride. Luna: Or worse, the driver gets paid and the rider gets a refund. Double-dipping. Lucas: Exactly. State-based testing forces you to define all valid states — say, 'requested', 'accepted', 'in-progress', 'completed', 'cancelled', 'paid' — and all allowed transitions. Then you write tests that enumerate every possible pair of states and try to transition. Or you use a tool to generate paths. Luna: What tools are we talking about? Because I know XState is popular for front-end state machines, but what about backend? Lucas: XState can work on the backend too — it's JavaScript/TypeScript, so Node.js services can use it directly. For other languages, there's things like scxml based frameworks, or you can build a lightweight state machine yourself. The important part isn't the tool — it's the concept. You define the state machine as code, then write tests that probe it. Luna: So how do you write those tests? Do you manually list every transition? Lucas: You can, but it gets tedious fast. With five states and ten transitions, you're looking at maybe twenty paths. But real systems have dozens of states. The better approach is to use property-based testing to generate valid sequences and then assert that certain invariants hold — like 'the ride should never be in both cancelled and paid states'. Luna: So it's property-based testing applied to state machines. That's clever. Lucas: Right. You combine the two. The property-based framework generates a sequence of actions — request, accept, cancel, start — and you feed them into the state machine. After each action, you check invariants. If a sequence produces an invalid state, you've found a bug. Luna: And that's different from contract testing, right? Because contract testing checks that two services agree on message format, not on the state of the overall system. Lucas: Completely. Contract testing is about API compatibility — does the producer send what the consumer expects? State-based testing is about the correctness of the state machine itself. They address different risks. You need both. Luna: Let's talk about integrating this into CI. If you're generating thousands of state sequences, that's going to be slow. How do you keep pipeline time reasonable? Lucas: Good question. You don't run all generated paths on every commit. You use a technique called 'state space sampling'. Run a fixed set of critical paths — say, the happy path and a few known risky transitions — on every commit. Then run a broader random exploration nightly or on a schedule. Luna: So you treat it like a fuzz test — lightweight smoke test in CI, deep exploration offline. Lucas: Exactly. And you can also compute state coverage — what percentage of possible transitions have been tested. That's a much more meaningful metric than line coverage for these kinds of bugs. Luna: Because 100% line coverage doesn't mean you've tested two events arriving simultaneously. Lucas: Right. State coverage tells you: have we ever seen the system go from 'accepted' to 'cancelled' while 'payment' is pending? If not, you have a gap. That's actionable. Luna: I want to go back to the ride-hailing example. Let's say the state machine has 10 states and 25 transitions. How many tests would you actually write manually vs generate? Lucas: I'd write maybe 10 manual test cases for the critical paths — request to paid, request to cancelled, etc. Then I'd use a property-based framework to generate thousands more. The manual ones are for sanity. The generated ones find edge cases you didn't think of. Luna: And those generated tests — do you store them in your repo, or just run them on the fly? Lucas: On the fly. You don't want to commit thousands of generated tests. Instead, you commit the state machine model and the test generator. Then the CI runs the generator and feeds tests to a test runner. If a test fails, you can shrink the sequence to the minimal failing path using the property-based framework's shrinking feature. Luna: That's the same pattern as property-based testing for regular functions — generate, fail, shrink, debug. Lucas: Exactly. The same mindset. State-based testing is really just property-based testing with a state machine as the oracle. And it's especially valuable for distributed systems where state can get inconsistent across services. Luna: Let's talk about that — distributed state. In a microservices architecture, each service has its own state. How do you model that? Lucas: That's where it gets interesting. You need to define a global state machine that represents the system's overall state, even though state is distributed. For example, the ride service might think the ride is 'in-progress', but the payment service might think it's 'not started' because payment hasn't been initiated. Luna: So the global state is a tuple of local states. And you test transitions that might cause inconsistency. Lucas: Right. You define allowed global states — say, is valid, but is invalid. Then you generate sequences of events across services and check that you never reach an invalid global state. Luna: That's powerful. But it requires a lot of upfront modeling. How do teams get started without spending weeks on it? Lucas: Start small. Pick one critical flow — like payment settlement — and model just those states. Maybe five to ten states. Write tests for that first. Once you see the value, expand. The key is not to model everything at once. Luna: So it's another tool in the toolbox. Not a replacement for unit tests or integration tests, but a layer on top. Lucas: Exactly. It catches a specific class of bugs that other tests miss. And the cost is relatively low — just a state machine model and a test generator. The return is high for systems where state consistency is critical. Luna: Speaking of high return, if today's deep dive into state-based testing gave you something usable, you can support the show and keep it ad-free at buy me a coffee dot com slash fexingo. Lucas: Yeah, listener support is what keeps these technical deep dives coming without sponsors. Every contribution helps us spend more time on topics like this. Luna: And we genuinely appreciate it. Now, let's get back to the practical side — how do you actually implement state-based testing in a CI pipeline with a tool like XState? Lucas: Sure. Say you're using Node.js and XState. You define your state machine using XState's createMachine. Then you write a test file that imports the machine and uses a property-based framework like fast-check to generate sequences of events. Luna: And you assert that after each event, the machine is in a valid state, right? Lucas: Right. You also assert that certain invariants hold. For example, 'if the ride is paid, the payment amount should be greater than zero'. XState makes it easy to inspect the current state and context. Luna: What about services that aren't Node? Say a Java backend with Spring State Machine? Lucas: Spring State Machine works similarly. You define states and transitions via a builder, then write tests that send events and check state. For property-based testing in Java, you can use jqwik or quickcheck. The pattern is the same. Luna: So the language doesn't matter — the approach is universal. Lucas: Exactly. And that's why I think state-based testing is underused. It's not tied to any particular stack. It's a design pattern for testing that applies anywhere you have state. Luna: One last thing — what about state machines that change over time? As you add features, states and transitions evolve. How do you keep the model in sync? Lucas: That's a real challenge. The model must be a living artifact. I recommend treating the state machine definition as a source of truth that lives in the same repo as the code. Whenever you add a new state or transition, you update the model first, then write tests against it. It's a form of test-driven development for state. Luna: So the model drives the tests, not the other way around. Lucas: Yes. And if you do that, you get a self-documenting specification of your system's behavior. New team members can look at the state machine and understand all possible flows. It's a win for documentation and testing. Luna: Alright, I'm convinced. I'm going to try this on our next project. Lucas: Do it. Start with one flow, model the states, write a few property-based tests. You'll be surprised what you find.