Mortgage Application Example

Mortgage Application Example

This example shows how you can use Hexawise to select a few dozen (or a few hundred) especially powerful tests from among millions of possible functional tests of a mortgage application processing program.

1) Identify the objectives and scope of your testing

Let’s create some tests to help us answer these questions – Can an applicant apply for a loan and get it approved? What happens when we “throw everything but the kitchen sink” at this mortgage application program? Does it consistently approve and deny the applicants as it is designed to? To answer these questions, we’ll need to:

  • Test all basic loan application functionality from start to finish
  • Include a wide variety of applicants (poor and rich, with good credit ratings and bad, etc.)
  • Include a broad selection of property types (primary residences and vacation homes; houses and apartments, properties in multiple-states, etc.)

2) Identify what will be excluded from this round of tests

Let’s exclude the following objectives from the tests we’re putting together now. While we could certainly use Hexawise to the kinds of tests listed below, it’s useful to clearly determine “what’s in” and “what’s out” for any set of tests you’re creating.

  • Exclude security testing
  • Exclude configuration testing
  • Exclude performance testing

3) Select test inputs and input them into Hexawise

This brings us to Hexawise’s “Define Inputs” screen. To put the business rules of the mortgage application process through their paces, we’ll want to create a set of tests that include a rich mixture of different combinations of the following test inputs. To do so, let’s add the following kinds of test inputs into Hexawise.

  • Applicant details (such as Credit ratings of applicants, Annual incomes of applicants)
  • Mortgage details (such as Repayment terms, Sizes of loans)
  • Property details (such as Residence types, Locations of properties, etc., etc.)

4) Whoa! How much time do we have to test?

As we add each set of test inputs, the number of possible test cases increases dramatically. The combinatorial explosion starts innocuously enough. 5 different applicant credit ratings times 6 different applicant income levels is only 30 possible tests. But by the time we add the last test inputs, there are more than 3.7 million possible tests to select from. How can you select a manageable subset of tests to execute from these that will find as many defects as possible?

Cue Hexawise to snap out of data collection mode into test creation mode by clicking on the “Create Tests” button.

5) “Voila! Here are the 48 tests you should execute first, sir”

It takes Hexawise algorithms 2.1 seconds to select the most powerful 48 tests from among the 3.7 million possible tests. To see how the Hexawise optimization algorithm works, (and listen in on the “conversation” that takes place between the tester and the Hexawise optimization algorithm during that 2.1 seconds), click here.

In short, Hexawise generates a set of tests, so that:

  • Maximum coverage is achieved in as few tests as possible
  • The tester’s coverage objectives are achieved (whether the tester instructs it to create a thorough set of 2-way tests or a phenomenally thorough set of 6-way tests)
  • Inefficient repetition from test case to test case is scientifically minimized

6) “Pareto on steroids” – Why those particular 48 tests?

Hexawise generates a set of 2-way tests as soon as we click the “Create Tests” button. These tests will ensure that a couple important coverage objectives are met:

  • Firstly, every Parameter Value will be tested in at least one of the tests.
  • Secondly, every valid pair of Parameter Values is tested together in at least one test case. In other words, every Parameter Value is tested in combination with every other Parameter Value in at least one of the tests.
  • These particular 48 tests are especially powerful because, as explained below, the vast majority of defects can be triggered by just one or two test inputs.
  • Decreasing marginal returns – It is quite uncommon for a defect to only be triggered by a specific combination of three or more test inputs. If, as is usually the case for business type applications, our objective is good coverage at minimum cost, we won’t focus our testing on those cases unless we have extra time after executing these higher priority 2-way tests. In safety critical domains, we would typically go for more thorough coverage.

7) Front-loaded test coverage

The fact that you only need 48 test cases to test for every valid combination of two Parameter Values in at least one test case is pretty cool, isn’t it? More impressive still:

  • Hexawise can help you answer “How much testing is enough?” and “When should we stop testing?”
  • With Hexawise, you can make fact-based decisions about these important questions.
  • The most important test cases (the ones that achieve the most coverage) are placed in priority order. Inevitably, a decreasing marginal return will occur over the course of a test plan.
  • Here, for example, we can see if only have time to execute 20 tests, we can see we would achieve coverage of 84% of the total number of 2-way combinations that exist in our testing scope.

8 ) Consider the need for more thorough testing

Pairwise (or 2-way) testing is exceptionally good at focusing testing efforts on a small set of unusually powerful tests. Pairwise testing is limited though by its lack of comprehensiveness. Hexawise makes it easy to create dramatically more thorough sets of tests. It’s like having a dial to turn: simply choose the level of thoroughness you’re looking for.

A retrospective study of defects in production, reported in IEEE Computer, found:

  • 85% of defects in production could have been detected by 2-way tests
  • 91% by 3-way tests
  • 97% by 4-way tests
  • 99.99 by 5-way tests
  • 100% of the defects could have been detected by 6-way tests

9) Generate a set of very thorough tests

Let’s imagine that we’ve got a lot of resources at our disposal for testing. We’d be willing to execute 100 times as many tests to be as thorough as possible. As we move from 2-way tests to 3-way or 4-way tests, we’ll increase the number of tests required from dozens to hundreds, and then thousands. Naturally, as we increase the number of tests we’ll be executing, we’d still like to be as efficient and effective as possible. Hexawise helps us ensure that every test counts.

Some defects might only be triggered by a particular combination of three specific Values. An example might be a bug that only appears when the following 4 Values are tested together in the same test case:

  • Location of property = California, AND
  • Credit rating of applicant = B+, AND
  • Type of property = vacation home, AND
  • Applicant type = bank employee

To create a set of tests that will test for every possible combination of 4 Values in at least one test case, we just need to select “4-way” from the drop down menu listed on the “Create Tests” screen.
When we select 4-way tests, Hexawise informs us that we would require XXXX tests to achieve this coverage goal.

10) Refine our tests to focus testing on high-priority areas to address our risk-based testing considerations

Until now, we’ve been treating each of the test inputs as if they are equally important. Let’s face it; common sense and lots of testing experience suggests that this is hardly ever the case. Some test inputs are more important than others. In addition, some combinations of test inputs are more important to test comprehensively than others.

For this set of tests, let’s make the following assumptions about where our priorities.

Relatively high priority test inputs include the following:

  • Credit Rating of Applicant (A, B, etc.)
  • Property Type (1 Family, Condo, etc.)
  • Residence Type (Primary, Investment, etc.)

With Hexawise, we can generate a set of Mixed-Strength tests that will achieve 2-way coverage of all the Parameter Values we’ve identified AND ALSO test for every single possible 3-way combination of Values involving Credit Ratings, Property Types, and Residence Types. It turns out we require 121 tests (from the unchanged 3.7 million tests) to achieve our new, refined, coverage objective.