How to Identify Variation That Should be Tested

By John Hunter · Oct 29, 2019

When testing software (or anything for that matter) it is important to determine what is important to test.  Testing resources are limited (often, very limited) and we must be certain to focus our effort where it will do the most good.  Hexawise helps to do this by creating test plans highly efficient test cases.

When test designers struggle to use Hexawise, the most common reason is identifying what test inputs should be included in the "Inputs" screen. To help address this challenge, this lesson describes several categories of variables that are often useful to include in software tests.

At the risk of stating the obvious, you'll need to use your judgment here; not all of these categories will be necessary (or even useful!) to include in all of your sets of tests.

chart of possible input variation for software

Variables for software tests can be described in these general categories.  It is important to remember that when you're entering variables into the "Inputs" screen, you should include test inputs only.  You should not include outcomes or expected results.

Environmental variations relate to different potential combinations of hardware and software that people might use, as well as the different locations of data that might be pulled from when a given transaction is executed. Imagine you're creating some end-to-end tests for a flight reservation system, like hipmunk.com (a site widely praised for its exceptional User Interface). As a web application, you would probably want to include multiple combinations of hardware and software configurations.

Hardware configuration inputs could include:

Manufacturer: Dell, Lenovo, Apple

Processor: Intel, AMD

Age: manufactured within last 2 years, manufactured 2 or more years ago

Software configuration inputs could include:

Operating System: Windows 7, Windows 8, Windows 10, Mac OS

Browser: IE10, IE11, Firefox, Chrome, Safari

Java settings: JavaScript enabled, JavaScript disabled

Cookies: cookies enabled, cookies disabled

Channel inputs could include:

Channel: typical web transaction, typical web transaction, phone call center (Note: double-weighting 'typical web transaction' will make it show up twice as often as 'phone call center')

Additional Environmental Variations involve file types and storage locations

Flight schedule information for flights is stored: Database Huey, Database Dooey, Database Louie

Frequent Flier Mileage data stored in: United database, (Frontier Airlines database; not yet transferred into post-merger database), N/A

Continue reading about this topic in our help file: How to best identify variation in the system being testing?

Related: 84% of Software Defects Found in Production Could Have Been Found Using Pairwise Testing - How bad are normal, business-as-usual software tests? - How Not to Design Pairwise Software Tests