Pairwise Testing is very interesting and pretty cool thing in black box testing. It enables testers to find bugs effectively and efficiently by saving time and effort. It’s a technique that helps to convert large number of test cases (exhaustive testing) to minimal.
Sometimes equivalence partitioning and boundary value analysis techniques may not uncover defects in the application under test. Â Let us take an example.
Consider a vehicle which is capable of travelling on ground, on air and in water, in manual and automatic mode. Below is the maximum speed limit:
On ground maximum speed is 150 km/hr., in air 200km/hr. and in water the maximum travel speed is 100km/hr.
To test the above scenario for bugs, for exhaustive testing the number of test case required is 18 i.e. 2*3*3. It is not possible to test vehicle using all possible combinations, as it takes more effort, time and money for fuel expenses.
Table 1 depicts the total number of exhaustive test cases for testing.
EXHAUSTIVE TESTCASES |
|||
TESTCASE ID |
MODE |
MEDIUM |
SPEED |
1 | Manual | Air | 100 |
2 | Manual | Air | 150 |
3 | Manual | Air | 200 |
4 | Manual | Water | 100 |
5 | Manual | Water | 150 |
6 | Manual | Water | 200 |
7 | Manual | Ground | 100 |
8 | Manual | Ground | 150 |
9 | Manual | Ground | 200 |
10 | Automatic | Air | 100 |
11 | Automatic | Air | 150 |
12 | Automatic | Air | 200 |
13 | Automatic | Water | 100 |
14 | Automatic | Water | 150 |
15 | Automatic | Water | 200 |
16 | Automatic | Ground | 100 |
17 | Automatic | Ground | 150 |
18 | Automatic | Ground | 200 |
Table 1
In order to avoid more number of test cases and to find bugs by executing minimum number of test cases, pairwise testing approach comes in.
Pairwise testing principle states that most bugs are caused due to interactions of 2 factors. Table2 depicts test cases generated by combining 2 factors using PICT by Microsoft.
COMBINATION DEPTH is 2 |
|||
TESTCASE ID |
MODE |
MEDIUM |
SPEED |
1 | Manual | Ground | 100 |
2 | Manual | Water | 200 |
3 | Automatic | Air | 100 |
4 | Manual | Air | 150 |
5 | Automatic | Water | 100 |
6 | Automatic | Air | 200 |
7 | Automatic | Water | 150 |
8 | Automatic | Ground | 150 |
9 | Manual | Ground | 200 |
Table 2
For the above scenario, using pairwise testing, the number of test cases (effort) can be reduced to 50% in comparison to exhaustive test cases.
Chart 1 shows pictorial view of comparison between Exhaustive and Pairwise test cases count.
 Chart 1
If you go for 3 or n factors then there will be increase in number of test cases. Table 3 in which test cases have been generated using 3 factors combination.
COMBINATION DEPTH is 3 |
|||
TESTCASE ID |
MODE |
MEDIUM |
SPEED |
1 | Manual | Water | 100 |
2 | Automatic | Air | 100 |
3 | Manual | Water | 150 |
4 | Manual | Ground | 150 |
5 | Automatic | Ground | 150 |
6 | Automatic | Water | 150 |
7 | Automatic | Water | 100 |
8 | Manual | Air | 100 |
9 | Manual | Ground | 100 |
10 | Manual | Air | 200 |
11 | Automatic | Water | 200 |
12 | Automatic | Air | 150 |
13 | Automatic | Air | 200 |
14 | Manual | Air | 150 |
15 | Manual | Water | 200 |
16 | Automatic | Ground | 100 |
17 | Automatic | Ground | 200 |
18 | Manual | Ground | 200 |
Table 3
Now, next question which comes to your mind is, will Pairwise testing approach uncover all the bugs? We will see this briefly.
There are some instances where Pairwise testing might fail or it will not be able to find bugs for your scenario under execution. Following are the instances where pairwise testing might fail to identify bug.        Â
- Pairwise Testing will not identify all important and critical combinations. Choose right values yourself.
- Understand your data. Pairwise testing may not fit for all data sets.
- If 2-way is not able to identify faults for your data sets, higher order combinations like 3-way/4-way might be necessary.
Most important consideration is while implementing pairwise testing, is that there are no best practices to uncover faults.
Conclusion
Tester needs to pay more attention while designing test cases using pairwise testing approach. It is important to understand the data and the way it interacts and care should be taken while assigning priority.
Available tools for pairwise testing
- http://www.pairwise.org/tools.asp
- Download PICT here
http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi
Further Reading and References
- Pairwise Testing: A Best Practice That Isn’t – http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
- Pairwise testing with Cucumber – http://blog.josephwilk.net/ruby/pairwise-testing-with-cucumber.html
- Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios –http://msdn.microsoft.com/en-us/library/cc150619.aspx#Pairwise_topic5
NaveenKumar,
Nice article about an important topic that too few software testers know about. This test design approach tends to improve software testing efficiency (because it prevents a great deal of wasteful repetition in software tests) and improve software testing effectiveness (because it closes a large number of gaps in coverage that would otherwise exist).
You may find a couple of these links useful to:
1) A photograph-rich, text-light presentation on both pairwise / allpairs testing and more thorough combinatorial software testing: http://www.slideshare.net/JustinHunter/combinatorial-software-testdesignbeyondpairwisetesting
2) A more scholarly but still practical introduction to pairwise and more thorough related test design methods that I co-wrote with 3 PhD’s: http://hexawise.com/combinatorial-and-pairwise-testing
3) Our Hexawise testing tool (which has a starting price point of $50,000/ year for corporate licenses but which we give away for free – with no small print – to thousands of users); any companies who want to use 5 or fewer licenses don’t pay: http://www.hexawise.com/free
Thanks.
– Justin
Many thanks Justin for your information. I will check it out. Thanks!
Hello Naveen,
What is the relationship between the number of test cases and the parameters + cardinality + order of combination (strength) in Pairwise test?
I am using the Microsoft PICT to generate Pairwise tests. For 4 parameters with cardinality of 5, 4, 4, 4 it generates 24 test cases for test strength of 2 (pairwise). For test strength of 3, it generates 115 test. How do I mathematically calculate the number of tests it will generate?
Thanks