Pairwise Testing is very interesting and pretty cool thing in black box testing. It enables testers to find bugs effectively and efficiently by saving time and effort. It’s a technique that helps to convert large number of test cases (exhaustive testing) to minimal.
Sometimes equivalence partitioning and boundary value analysis techniques may not uncover defects in the application under test. Let us take an example.
Consider a vehicle which is capable of travelling on ground, on air and in water, in manual and automatic mode. Below is the maximum speed limit:
On ground maximum speed is 150 km/hr., in air 200km/hr. and in water the maximum travel speed is 100km/hr.
To test the above scenario for bugs, for exhaustive testing the number of test case required is 18 i.e. 2*3*3. It is not possible to test vehicle using all possible combinations, as it takes more effort, time and money for fuel expenses.
Table 1 depicts the total number of exhaustive test cases for testing.
In order to avoid more number of test cases and to find bugs by executing minimum number of test cases, pairwise testing approach comes in.
Pairwise testing principle states that most bugs are caused due to interactions of 2 factors. Table2 depicts test cases generated by combining 2 factors using PICT by Microsoft.
COMBINATION DEPTH is 2
For the above scenario, using pairwise testing, the number of test cases (effort) can be reduced to 50% in comparison to exhaustive test cases.
Chart 1 shows pictorial view of comparison between Exhaustive and Pairwise test cases count.
If you go for 3 or n factors then there will be increase in number of test cases. Table 3 in which test cases have been generated using 3 factors combination.
COMBINATION DEPTH is 3
Now, next question which comes to your mind is, will Pairwise testing approach uncover all the bugs? We will see this briefly.
There are some instances where Pairwise testing might fail or it will not be able to find bugs for your scenario under execution. Following are the instances where pairwise testing might fail to identify bug.
- Pairwise Testing will not identify all important and critical combinations. Choose right values yourself.
- Understand your data. Pairwise testing may not fit for all data sets.
- If 2-way is not able to identify faults for your data sets, higher order combinations like 3-way/4-way might be necessary.
Most important consideration is while implementing pairwise testing, is that there are no best practices to uncover faults.
Tester needs to pay more attention while designing test cases using pairwise testing approach. It is important to understand the data and the way it interacts and care should be taken while assigning priority.
Available tools for pairwise testing
- Download PICT here
Further Reading and References
- Pairwise Testing: A Best Practice That Isn’t – http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
- Pairwise testing with Cucumber – http://blog.josephwilk.net/ruby/pairwise-testing-with-cucumber.html
- Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios –http://msdn.microsoft.com/en-us/library/cc150619.aspx#Pairwise_topic5