I enjoyed talking about efficient and effective combination testing strategies (and highlights of a recent empirical study) at yesterday's TISQA meeting together with Lester Bostic of Blue Cross Blue Shield North Carolina, who shared his team's experiences of adopting a combinatorial testing approach. It addresses how tools like Hexawise can help software testers quickly identify the test cases they should execute to find as many defects as possible with as few tests as possible. I wanted to share it now; once I have more time, I will comment on it and highlight some of the good questions, comments, discussion points, and tester experiences that were raised by the attendees.

The presentation focused on combinatorial testing techniques, such as pairwise testing, orthogonal array-based testing methods, and more thorough combination testing strategies (capable of identifying all defects that could be captured by, say, any possible combination of three or four "things" that you've decided to test for (regardless of whether those "things" include features configurations or equivalence class of data or type of user a mix of each).

The middle of the presentation also highlights empirical evidence that shows this method of identifying test cases often has an enormous impact on how quickly software testers are able to identify defects; citing the IEEE Computer article I co-wrote last month on Combinatorial Testing, this approach - on average - led to more than twice as many defects found per tester hour.

The final section of the presentation was delivered by Lester Bostic of Blue Cross Blue Shield and addresses his lessons learned. Lester used Hexawise to reduce 1,356,136,048,589,996,428,428,909,447,344,392,489,476,985,674,792,960 possible tests (that would have been necessary to achieve comprehensive testing of the application he was testing) to only 220 tests that proved to be extremely effective at identifying defects. .

Comments and questions are welcome.