1) Identify the objectives and scope of your testing

Let’s create some tests to help us answer these questions – Can an applicant apply for health insurance and get it approved? What happens when we “throw everything but the kitchen sink” at this health insurance application program? Does it consistently approve and deny the applicants as it is designed to? Does it consistently quote the correct policy rates? To answer these questions, we’ll need to:

• Test all basic health insurance application functionality from start to finish
• Include a wide variety of applicants (young and old, with and without a spouse, with and without children, etc.)
• Include a broad selection of policy types (standard insurance only, with and without dental coverage, with and without maternity coverage, etc.)
• Since this is a web-based application let’s also include a variety of tests for several popular browser types and Operating Systems

2) Identify what will be excluded from this round of tests

Let’s exclude the following objectives from the tests we’re putting together now. While we could certainly use Hexawise to the kinds of tests listed below, it’s useful to clearly determine “what’s in” and “what’s out” for any set of tests you’re creating.

• Exclude security testing
• Exclude complex configuration testing
• Exclude performance testing

3) Select test inputs and input them into Hexawise

This brings us to Hexawise’s “Define Inputs” screen. To put the business rules of the mortgage application process through their paces, we’ll want to create a set of tests that include a rich mixture of different combinations of the following test inputs. To do so, let’s add the following kinds of test inputs into Hexawise.

• Applicant details (such as age, gender, and location)
• Family details (e.g., whether the primary account holder has a spouse and/or children)
• Policy details (e.g., whether or not Dental coverage or Maternity coverage are selected.)

4) Whoa! How much time do we have to test?

As we add each set of test inputs, the number of possible test cases increases dramatically. The combinatorial explosion starts innocuously enough. 4 different operating systems times 4 different types of browsers is only 16 possible tests (or 15 if you account for the fact that IE can’t be launched on Mac OSX). But by the time we add the last test inputs, there are more than 4.7 million possible tests to select from. How can you select a manageable subset of tests to execute from these that will find as many defects as possible?

Cue Hexawise to snap out of data collection mode into test creation mode by clicking on the “Create Tests” button.

5) “Voila! Here are the 27 tests you should execute first, sir”

It takes Hexawise algorithms 2.1 seconds to select the most powerful 27 tests from among the 4.7 million possible tests. In short, Hexawise generates a set of tests, so that:

• Maximum coverage is achieved in as few tests as possible
• The tester’s coverage objectives are achieved (whether the tester instructs it to create a thorough set of 2-way tests or a phenomenally thorough set of 6-way tests)
• Inefficient repetition from test case to test case is scientifically minimized

6) “Pareto on steroids” – Why those particular 48 tests?

Hexawise generates a set of 2-way tests as soon as we click the “Create Tests” button. These tests will ensure that a couple important coverage objectives are met:

• Firstly, every Parameter Value will be tested in at least one of the tests.
• Secondly, every valid pair of Parameter Values is tested together in at least one test case. In other words, every Parameter Value is tested in combination with every other Parameter Value in at least one of the tests.
• These particular 27 tests are especially powerful because, as explained below, the vast majority of defects can be triggered by just one or two test inputs.
• Decreasing marginal returns – It is quite uncommon for a defect to only be triggered by a specific combination of three or more test inputs. For that reason, we won’t focus our testing on those cases unless we have extra time after executing these higher priority 2-way tests.

7) Front-loaded test coverage

The fact that you only need 27 test cases to test for every valid combination of two Parameter Values in at least one test case is pretty cool, isn’t it? More impressive still:

• Hexawise can help you answer “How much testing is enough?” and “When should we stop testing?”
• With Hexawise, you can make fact-based decisions about these important questions.
• The most important test cases (the ones that achieve the most coverage) are placed in priority order. Inevitably, a decreasing marginal return will occur over the course of a test plan.
• Here, for example, we can see if only have time to execute just12 tests, we can see we would achieve coverage of 81% of the total number of 2-way combinations that exist in our testing scope.

8) Consider the need for more thorough testing

Pairwise (or 2-way) testing is exceptionally good at focusing testing efforts on a small set of unusually powerful tests. Pairwise testing is limited though by its lack of comprehensiveness. Hexawise makes it easy to create dramatically more thorough sets of tests. It’s like having a dial to turn: simply choose the level of thoroughness you’re looking for. Of all the defects in production,

• 85% of defects in production could have been detected by 2-way tests
• 91% by 3-way tests
• 97% by 4-way tests
• 99.99 by 5-way tests
• 100% of the defects could have been detected by 6-way tests

9) Generate a set of very thorough tests

Let’s imagine that we’ve got a lot of resources at our disposal for testing. We’d be willing to execute 100 times as many tests to be as thorough as possible. As we move from 2-way tests to 3-way or 4-way tests, we’ll increase the number of tests required from dozens to hundreds, and then thousands. Naturally, as we increase the number of tests we’ll be executing, we’d still like to be as efficient and effective as possible. Hexawise helps us ensure that every test counts.

Some defects might only be triggered by a particular combination of three specific Values. An example might be a bug that only appears when the following 4 Values are tested together in the same test case:

• To create a set of tests that will test for every possible valid combination of 4 Values in at least one test case, we just need to select “4-way” from the drop down menu listed on the “Create Tests” screen.
• When we select 4-way tests, Hexawise informs us that we would require 643 tests to achieve this coverage goal.

10) Refine our tests to focus testing on high-priority areas to address our risk-based testing considerations

Until now, we’ve been treating each of the test inputs as if they are equally important. Let’s face it; common sense and lots of testing experience suggests that this is hardly ever the case. Some test inputs are more important than others. Similarly, some combinations of test inputs are more important to test comprehensively than others.

For this set of tests, let’s make the following assumptions about where our priorities are.

Relatively high priority test inputs include the following:

• Age of Primary Applicant (N/A, 0.1-17.9, 18-39.9, 40-64.9, 65-140)
• How Many Children (0, 1-3, 4-6, 7-12)
• Dental Coverage (Selected, Not Selected)

With Hexawise, we can generate a set of Mixed-Strength tests that will achieve 2-way coverage of all the Parameter Values we’ve identified AND ALSO test for every single possible 3-way combination of Values involving these 3 sets of higher priority Parameter Values. It turns out we require 121 tests (from the unchanged possible 4.7 million tests) to achieve our new, refined, coverage objective. When we select these three particular set of Parameter Values to teset more thoroughly than the rest, Hexawise automatically generates 36 tests that are necessary to accomplish 100% of our coverage goals.

Ready to talk?


Call 1-877-HEXAWISE or send us an email.