Testers who use Hexawise consistently pack significantly more coverage into fewer software tests. It might seem counterintuitive to people unfamiliar with pairwise and orthogonal array-based testing that more thorough coverage can be achieved while using fewer tests, but this is a clear and well-established fact. This article explains how Hexawise consistently achieve superior coverage even while using fewer tests.

Time savings and thoroughness improvements achieved by testers using Hexawise at one of our insurance clients recently are typical. Let’s quickly address the first two benefits before diving deeper into a detailed explanation of how testers achieved the thoroughness improvements described in the third benefit.

The time savings in the test selection and documentation phase (shown in the top box) are easy enough to understand. Testers using Hexawise save time by automating selection and documentation steps that they would otherwise have to complete manually.

Similarly, the time savings in the test execution phase (shown in the middle box) are equally straightforward. Hexawise can generate fewer test scenarios compared to what testers would create on their own. Time savings in test execution come about simply because it takes less time to execute fewer tests.

So far so good. But how exactly do testers using Hexawise consistently achieve superior coverage while using fewer software tests? By helping testers generate optimized test sets without wasteful redundancies minimized, with the maximum amount of variation between scenarios, and with systematic coverage potential defects that could be caused by interactions.

Hexawise Minimizes Wasteful Repetition The powerful test generation algorithm inside of Hexawise systematically eliminates all wasteful repetition from every test scenario. If a given combination of test conditions has already appeared together in a test, other combinations of values will be found by the test generation algorithm and used instead of the wastefully repetitive combination. Even it if means that Hexawise’s blazingly-fast optimization algorithm needs to explore thousands of combinations of candidate values to achieve this goal. With wasteful repetition eliminated, Hexawise test sets require fewer tests to achieve thorough testing.

Hexawise Maximizes Variation Between Tests. If you take a close look at any Hexawise-generated set of tests, you will notice that variation is maximized as much as scientifically possible from one test to the next. This is the beneficial flip side of the repetition-minimization coin. Useful variation from test to test is the thoroughness-improving outcome whenever wasteful repetition is eliminated. When testers start to execute tests that explore new combinations of values and new paths through applications, they find more defects.

Superior, Systematic Coverage. Interactions between different test inputs are a major source of defects. As a result, interactions between inputs are important to test thoroughly and systematically. Testers using Hexawise use a “coverage dial” in Hexawise to determine the coverage strength they would like for a given set of tests. From there, Hexawise’s test optimization algorithms systematically detect all potential interactions that are in scope to be tested for, and Hexawise ensures tests are carefully constructed to close every such potential coverage gap. Doing this kind of analysis by hand, even using tools like Excel is time-consuming, impractical, and error-prone. There are simply too many interactions for a tester to keep track of on their own. As a result, manually-selected test sets almost always fail to test for a rather large number of potentially important interactions. In contrast, the Hexawise test optimization algorithm systematically eliminates gaps in testing coverage that manually-selected test sets routinely fail to cover. Compare the coverage achieved by the project’s 37 manually-selected “business as usual” tests (above) to the more compact, efficient, and thorough set of 25 Hexawise-generated set of tests (below).

In short, when testers select scenarios by hand, the outcome is typically too many tests that took too long to select and document, contain too much wasteful redundancy, and have an unknown number of potentially-serious gaps in coverage. In contrast, when testers use Hexawise to generate optimized sets of tests, they quickly generate unusually thorough sets of highly-varied tests at the push of a button that systematically achieve user-specified thoroughness goals, And testers can communicate coverage achieved by their test sets with stakeholders more clearly than ever before.