Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Pairwise Testing Cuts Test Cases Without Cutting Coverage
Transcript
- Lucas: So you've got a feature with ten input parameters, each with, say, four possible values. The naive approach says you need four to the power of ten test cases — that's over a million. No one runs a million test cases for one feature. Luna: Right, and that's where pairwise testing comes in. It's the idea that most defects are triggered by a single parameter value or an interaction between just two parameters. Lucas: Exactly. The research goes back to the 1980s — Boris Beizer and others found that in practice, something like 90 to 95 percent of bugs are caused by a single value or a two-way interaction. Three-way and higher interactions are rare. Luna: So instead of testing every combination, you just test every pair of parameter values at least once. That dramatically cuts the number of test cases. Lucas: Right. Let's make it concrete. Say you're testing a flight booking search. You've got departure city, arrival city, departure date, return date, number of passengers, cabin class, and maybe a few more. Each has multiple options. Full combinatorial would be enormous. Luna: But with pairwise, you might end up with a few dozen test cases covering all pairs. I've seen examples where it goes from thousands down to thirty or forty. Lucas: Exactly. And the key insight is that the number of test cases grows logarithmically with the number of parameters, not exponentially. So it scales beautifully. For a system with 10 parameters each with 4 values, pairwise might give you around 30 test cases. Luna: How do you actually generate those test cases? Is there a tool or a manual method? Lucas: There are several tools. The classic one is the 'AllPairs' algorithm by James Bach. There's also the 'PICT' tool from Microsoft, which is free and widely used. You define your parameters and values, and it spits out a minimal set of test cases. Luna: And you can also do it manually with an orthogonal array, but that gets tedious fast. Lucas: Yeah, I'd never recommend manual for anything beyond three or four parameters. But the concept is straightforward: you create a matrix where each column is a parameter, each row is a test case, and every pair of values appears in at least one row. Luna: So it's like a clever way to cover all two-way interactions without brute force. Lucas: Exactly. And the beauty is that it's not just for web forms. It's used in hardware testing, API testing, configuration testing — anywhere you have multiple variables. Luna: But does it ever miss bugs that would be caught by exhaustive testing? Obviously it does, since you're not testing every combination. Lucas: Right, it's a trade-off. The assumption is that higher-order interactions are rare. But if you have a system where three-way interactions are known to cause issues — say, in some medical devices or safety-critical systems — you might need three-way or even four-way combinatorial testing. Luna: So pairwise is a practical baseline, but you need to understand your domain. Lucas: Exactly. And you can combine it with other techniques. For example, you can use pairwise for the bulk of your test cases and then add boundary value analysis for specific parameters that are known to be tricky. Luna: Let's talk about a real application. I remember a team I worked with that used pairwise testing for a configuration tool that had something like 15 different settings. Lucas: That's a perfect use case. Configuration matrices explode combinatorially. I've seen teams spend weeks writing test cases for every combination of operating system, browser, and database version. Luna: And pairwise would cover that in a fraction of the time. You'd generate maybe 20 to 30 test configurations that cover all pairs, and you'd catch the vast majority of compatibility issues. Lucas: There's a famous case from the early 2000s where a telecom company used pairwise testing to test a new switch. They had hundreds of parameters. They reduced the test suite from millions of cases to a few thousand, and they still found all the critical bugs before release. Luna: That's compelling. But what about the edge cases that pairwise misses? Do you just accept that risk? Lucas: You manage it. You prioritize based on risk. For parameters where you know three-way interactions matter — maybe because of past bugs — you can add targeted tests. Pairwise isn't a silver bullet, but it's a very efficient net. Luna: So if I'm a QA engineer on a new project, how would I start implementing pairwise testing? Lucas: First, list all your parameters and their possible values. Then pick a tool like PICT, define your model, and generate the test cases. Then review them to make sure they make sense — sometimes the tool generates odd combinations that are invalid in the real world. Luna: Like a departure date after the return date for a one-way flight? That sort of thing. Lucas: Exactly. You need to add constraints. Most tools allow you to define constraints like 'if cabin class is first class, then number of passengers must be 1' or 'departure date must be before return date'. That filters out invalid combinations. Luna: And then you automate those test cases and add them to your CI pipeline. Lucas: Right. And because pairwise reduces the number of test cases so dramatically, your CI runs faster. That's a huge win for teams that struggle with long feedback loops. Luna: Speed and coverage — that's a combination that's hard to argue with. Lucas: And it's not just for new projects. You can apply pairwise to an existing test suite to see if you're over-testing or if you have gaps. It's a great audit tool. Luna: I've seen teams that have thousands of manual test cases that are essentially redundant. Pairwise could help them trim down significantly. Lucas: Yeah, and that frees up time for exploratory testing, which can catch those rare higher-order bugs that pairwise might miss. Luna: So pairwise is a foundational technique, but it's part of a larger testing strategy. Lucas: Absolutely. It's not a replacement for everything. But if you're not using it, you're probably wasting time on redundant tests or missing coverage. Luna: And let's be honest, most teams are over-testing on some dimensions and under-testing on others. Pairwise forces you to think about coverage systematically. Lucas: That's the core value. It's a structured way to design tests that gives you high confidence with low effort. Luna: I know we've talked about tools, but are there any pitfalls with pairwise testing that beginners should watch out for? Lucas: One common mistake is not including all relevant parameters. If you leave out a parameter, you could miss important interactions. Also, make sure your parameter values are meaningful — not just placeholders. Luna: And you need to update your pairwise model when the system changes. If a new parameter is added, you need to regenerate test cases. Lucas: Right. But the good news is that with a tool, regenerating is quick. You can even automate it as part of your build process. Luna: So it's a living artifact, not a one-time exercise. Lucas: Exactly. And I think that's the right way to think about it. Pairwise testing is a practice, not a project. Luna: Well, speaking of practices that save time and improve quality — if you're finding this conversation useful, you might consider supporting the show. We keep it ad-free because of listeners like you. Lucas: Yeah, it's a small gesture that goes a long way. If you'd like to buy us a coffee, you can do that at buy me a coffee dot com slash fexingo. Every bit helps us keep bringing you deep dives like this. Luna: And we genuinely appreciate it. Now, back to pairwise — one thing I wanted to mention is that pairwise isn't just for black-box testing. You can use it for white-box testing too, like testing different code paths. Lucas: That's a good point. Pairwise is a combinatorial technique that applies anywhere you have discrete inputs. So it works for unit testing, integration testing, system testing — any level. Luna: And it pairs well with other methods like equivalence partitioning and boundary value analysis. Lucas: Absolutely. Those techniques help you choose the right values for each parameter, and pairwise helps you combine them efficiently. Luna: So the takeaway: pairwise testing is a powerful way to reduce test cases without sacrificing coverage, and it scales to complex systems. Lucas: And it's one of those techniques that every software tester should have in their toolkit. If you haven't tried it, give it a shot on your next feature. You might be surprised at how much time you save. Luna: Alright, that's a wrap for today. Thanks for listening.