Hexawise welcomes Kelly Ross as Vice President of Sales. Kelly will be responsible for driving Hexawise growth by working with enterprise clients to introduce Hexawise tools into their testing environments. She will also build the sales organization through strategic hires to support the company’s growth.

Commenting on the appointment, Hexawise CEO, Justin Hunter, said: “Hexawise is poised for explosive growth in the coming year and I am delighted that Kelly will be joining us to spearhead that growth. Kelly has been highly successful both as a salesperson and as a sales leader, which is the perfect combination for our company at this time. We are excited to have her join our executive team.”

Kelly brings more than 20 years of sales leadership experience to Hexawise, including driving growth in software sales organizations at SAS Institute, Vignette Corporation and Intuit Health. She has recently worked as an independent consultant, helping start-ups and small businesses establish sales processes to accelerate revenue growth in their markets.

Kelly said: “I’m excited to join a team that is driving innovation in the software testing market. Enterprise software teams are using Hexawise to accelerate testing and get to market faster. Hexawise has a growing client base of highly satisfied customers leveraging the value of the tools, including more than 9000 users at Accenture. The revenue growth opportunities are tremendous and I am fortunate to be joining the team during such a pivotal time.”

One of Kelly’s first priorities will be to hire a new Sales Development Representative to work with her on generating new leads and growing the sales pipeline for Hexawise. “We are looking for intelligent, energetic sales professionals who would like to help us achieve our goals through strategic, directed outbound efforts.”

About Hexawise Hexawise is changing the way companies test software. Software testers at more than 100 Fortune 500 firms use the Hexawise software test design tool to scientifically prioritize scenarios they should execute in order to achieve higher coverage in fewer tests. Organizations like NASA have used sophisticated test design methods for years to learn as much as possible from each software test they execute. With its easy-to-use test design tool and unparalleled test design training and support programs, Hexawise has become the worldwide leader at making these sophisticated test design methods accessible to the masses.

Benefits delivered by Hexawise are dramatic and measurable, including:

(1) Time and cost savings from faster test case design (2) Time and cost savings from faster test execution (3) Decreased costs of defect resolution by identification of more bugs early in the development life cycle.

By: Justin Hunter on Feb 22, 2016

Categories: Hexawise

Hexawise - More Coverage Fewer Tests

Testers who use Hexawise consistently pack significantly more coverage into fewer software tests. It might seem counterintuitive to people unfamiliar with pairwise and orthogonal array-based testing that more thorough coverage can be achieved while using fewer tests, but this is a clear and well-established fact. This article explains how Hexawise consistently achieve superior coverage even while using fewer tests.

Time savings and thoroughness improvements achieved by testers using Hexawise at one of our insurance clients recently are typical. Let’s quickly address the first two benefits before diving deeper into a detailed explanation of how testers achieved the thoroughness improvements described in the third benefit.

Hexawise - Benefits Findings

The time savings in the test selection and documentation phase (shown in the top box) are easy enough to understand. Testers using Hexawise save time by automating selection and documentation steps that they would otherwise have to complete manually.

Similarly, the time savings in the test execution phase (shown in the middle box) are equally straightforward. Hexawise can generate fewer test scenarios compared to what testers would create on their own. Time savings in test execution come about simply because it takes less time to execute fewer tests.

So far so good. But how exactly do testers using Hexawise consistently achieve superior coverage while using fewer software tests? By helping testers generate optimized test sets without wasteful redundancies minimized, with the maximum amount of variation between scenarios, and with systematic coverage potential defects that could be caused by interactions.

Hexawise - Specific Benefits

Hexawise Minimizes Wasteful Repetition The powerful test generation algorithm inside of Hexawise systematically eliminates all wasteful repetition from every test scenario. If a given combination of test conditions has already appeared together in a test, other combinations of values will be found by the test generation algorithm and used instead of the wastefully repetitive combination. Even it if means that Hexawise’s blazingly-fast optimization algorithm needs to explore thousands of combinations of candidate values to achieve this goal. With wasteful repetition eliminated, Hexawise test sets require fewer tests to achieve thorough testing.

Hexawise Minimizes Wasteful Repetition

Hexawise Maximizes Variation Between Tests. If you take a close look at any Hexawise-generated set of tests, you will notice that variation is maximized as much as scientifically possible from one test to the next. This is the beneficial flip side of the repetition-minimization coin. Useful variation from test to test is the thoroughness-improving outcome whenever wasteful repetition is eliminated. When testers start to execute tests that explore new combinations of values and new paths through applications, they find more defects.

Hexawise Maximizes Variation Between Tests

Superior, Systematic Coverage. Interactions between different test inputs are a major source of defects. As a result, interactions between inputs are important to test thoroughly and systematically. Testers using Hexawise use a “coverage dial” in Hexawise to determine the coverage strength they would like for a given set of tests. From there, Hexawise’s test optimization algorithms systematically detect all potential interactions that are in scope to be tested for, and Hexawise ensures tests are carefully constructed to close every such potential coverage gap. Doing this kind of analysis by hand, even using tools like Excel is time-consuming, impractical, and error-prone. There are simply too many interactions for a tester to keep track of on their own. As a result, manually-selected test sets almost always fail to test for a rather large number of potentially important interactions. In contrast, the Hexawise test optimization algorithm systematically eliminates gaps in testing coverage that manually-selected test sets routinely fail to cover. Compare the coverage achieved by the project’s 37 manually-selected “business as usual” tests (above) to the more compact, efficient, and thorough set of 25 Hexawise-generated set of tests (below).

Gaps in Coverage Hexawise Optimized Coverage

In short, when testers select scenarios by hand, the outcome is typically too many tests that took too long to select and document, contain too much wasteful redundancy, and have an unknown number of potentially-serious gaps in coverage. In contrast, when testers use Hexawise to generate optimized sets of tests, they quickly generate unusually thorough sets of highly-varied tests at the push of a button that systematically achieve user-specified thoroughness goals, And testers can communicate coverage achieved by their test sets with stakeholders more clearly than ever before.

By: Justin Hunter on Dec 15, 2015

Categories: Pairwise Software Testing, Pairwise Testing, Combinatorial Testing, Business Case

Few things make us happier at Hexawise than hearing reports from clients about how much Hexawise is helping them improve their software testing efforts.

 

Happy

 

A recent conversation with our newest international banking client is a case in point. Carrie, a senior testing manager who is leading adoption efforts at the bank, reported the following impressive benefits:

 

Project 1

Without Hexawise: estimated testing effort for the project = 8.5 man months

With Hexawise: estimated testing effort for the project = 1.5 man months

Savings = > 80%

Hexawise Case Studies 2016 01 23

 

Project 2

Without Hexawise (e.g. immediately prior release): defects found during testing = 67%; defects found in UAT = 33%

With Hexawise (most recent release): defects found during testing = 98.5%; defects found in UAT = 1.5%

Defect Removal Effectiveness Improvement = Stunning

Hexawise Case Studies 2016 01 23 2

 

A few interesting things are worth pointing out as context behind these results.

 

First, let's be clear: these are significantly higher than normal Hexawise-generated benefits. We're not suggesting that every project will see benefits this large. They won't. "Your milage may vary." The 80% reduction in testing time is not unheard of but definitely larger than most teams tend to see. Similarly, the massive improvement to defect removal efficiency is larger than typically occurs. These are more typical benefits from case studies using Hexawise's pairwise testing methods and/or combinatorial testing methods and/or Orthogonal Array OATS testing methods.

 

Second, the test designers involved in these two projects are significantly more talented and skilled than "average" software testers working at banks. The test designers' skill has a lot to do with the unusually large successes they achieved in these projects. We know about how talented they are because we worked closely with these test designers during a 4-day onsite instructor-led test design training program we led and we have a good sense of the "average" test design skills possessed by software testers because we regularly conduct software test design training sessions around the world at hundreds of companies. During the hands-on interactive test design exercises in our face-to-face training sessions with the bank's software testers, several of their test designers demonstrated exceptional analytical thinking and problem solving skills.

 

Third, as we try to do with all of our new clients, our test design experts have actively kept in touch with the bank's test designers since the initial onsite training took place. We have been answering their test design questions when they have arisen, offering to review their draft test tests, and jumping on ad hoc screen sharing sessions to explain/demonstrate how to use Hexawise test design features. This helps our clients maximize the value they obtain from using Hexawise and helps us stay closely attuned to real-world testing challenges so that we can continuously improve our tool and fine-tune our software test design training messages.

 

If you're using Hexawise and have experiences to share with us, whether good or bad, we would love to hear about them. We're here to help. As corny as it sounds, helping clients succeed is a huge part of what motivates us at Hexawise. Please contact us at: support@hexawise.com

By: Justin Hunter on Sep 10, 2015

Categories: User Experience, Pairwise Testing, Combinatorial Testing, Business Case

Michael Bolton is one of the software testing industry's deep thinkers. He has an impressive ability to logically analyze testing problems and clearly explain complex issues.

I like how Michael summarizes what people often really mean when they say "it works"*

It works really means...

There's a lot of truth in those words, isn't there? I've shared these words from Michael in test design trainings I've done recently and found that they immediately resonate with quite a few testers. It seems that anyone who has been in testing for more than a while has seen teams of testers test a feature or function a bit, declare that "it works," only to discover later that the feature/function works in some situations but does not work in other situations.

What's a tester to do? We recommend testers use two deliberate strategies: use a rich oracle and cover critical interactions.

First, use a "rich oracle" to enage your brain more actively and train your eye to better recognize potential issues. Imagine the following scenario. 3 people are in a room. The first person, a guy plucked from the street outside at random, is given a set of 10 written test scripts to execute and told to follow the test scripts, step-by-step in return for a six pack of beer. Being a fan of beer, and endowed with the dual-abilities of being able to both read instructions and follow instructions, he performs what is asked of him dutifully.

In the room are two testers who are allowed to observe the tests being executed but who are not allowed to communicate.

  • The first tester has a list of ten numbers, each with three boxes for checkmarks. He operates in a world of black and white where if the documented "Expected Result" is consistent with what they observe, they write a green check mark. If the "Expected Result" is inconsistent, they write a green "X."
  • The second tester operates differently. She goes beyond. She goes deeper. She notices subtle things along the way that look unexpected, or not quite right. She makes notes of those things. In doing so, she thinks of new test ideas that have not been executed yet. She documents those test ideas to explore further later, provided there is time.

I think you see where I'm going with this. My point is that the more curious approach adopted by the second tester is a far more valuable one to people who care about software quality. Why is this? Here too, Michael Bolton has words of wisdom to share that resonate well:

If you insist you need written requirements to find bugs

Second, testers should adopt test case design approaches in order to avoid the "under some conditions... once" risk. One of the most important benefits of using our Hexawise test case design tool is that, even with very basic pairwise test sets, every feature or function you test will automatically be tested multiple times. And under as many different conditions as possible in the time you have available.

After close to 10 years of introducing new groups of software testers to these types of test design approaches, people have a hard time believing how big efficiency and thoroughness improvements in this area are. That's why we always strongly encourage teams using our optimized test case selection approach to do apples-to-apples comparisons of coverage and defect-finding effectiveness. We work with teams to compare the coverage gaps of their existing "business as usual" test sets to how thorough they are when Hexawise is used to generate an optimized set of tests. Images of one recent coverage analysis is shown below. Data on defect-finding effectiveness and defect finding efficiency improvements resulting from optimized test case selection can also be found here.

Testing Coverage Analysis Hexawise Pairwise Combinatorial Testing OATS

 

*With thanks to Jon Bach for sharing this on his blog.

By: Justin Hunter on Sep 25, 2014

Categories: Pairwise Software Testing, Pairwise Testing, Combinatorial Testing, Software Testing

We have written before about the general question of "Should I Use One Test Plan or Multiple Plans?" This post addresses the same question with a focus on plans that have a relatively large percentage of constraints (e.g., a relatively large number of Invalid Pairs and/or Married Pairs).

Hexawise creates a Plan Scorecard that analyzes every set of tests Hexawise users generate. The Plan Scorecard exists to help you identify potential problems with plans you create and to make you aware of possible ways to improve your sets of tests. One of the notifications the Plan Scorecard provides goes like this:

Consideration: "53% of the parameter values are directly or indirectly constrained."
Explanation: Test plans with more than half of the parameter values constrained are often trying to do too much. They may be better broken into more than one test plan.

Constraints are used to prevent "impossible to test for" scenarios from being generated by the Hexawise test case generator. For more information about entering constraints into Hexawise, see these explanations of Invalid Pairs and Married Pairs. If you do not use Invalid Pairs or Married Pairs in a plan, 0% of your parameter values would be constrained.

What should you do if you get a notification that your plan is highly constrained? Simple: consider your options. Specifically, consider the pro's and con's of splitting the plan you're working on into multiple separate plans.

 

Why can heavily-constrained plans be a problem?

  • A number above 50% or so indicates that it might make sense to consider breaking your plan into 2 or more plans.
  • Why? Because with more and more constraints in a plan, keeping track of them all, making sure they're all accurate, and making sure the constraints in certain parts of your plan do not conflict with constraints in other parts of your plan in unintended ways, can start to take a lot of mental energy.
  • Furthermore, if your constraints do begin to conflict with one another, that could make it impossible for the Hexawise algorithm to identify valid values to populate in some parts of some of your tests. When that happens, instead of an actual value appearing in a test case, you will see the words "No Possible Value" appear.

 

Why are multiple simpler plans often preferable to one more complex one?

  • It is often much easier and quicker (from a modeling standpoint) to create two different plans. Creating two separate plans instead of one single plan often makes it possible to eliminate the need for the majority of the constraints in your plan.
    • For example, if you had a pizza ordering application where there were a lot of constraints around the value "meat pizza" and a lot of constraints around the value "vegetarian pizza" it could be attractive to create one plan (e.g., one set of tests) for meat pizzas and a different set of tests for veg. pizzas.
  • Simpler plans with fewer constraints tend to be easier to understand, modify, and maintain.

 

What are practical considerations when splitting a single plans into multiple ones?

  • To determine where / how to split a plan, begin by asking "what values have the most constraints associated with them?"
    • In the example above, "meat pizza" and "veggie pizza" had the most constraints; creating one plan for meat pizzas and a separate plan for veggie pizzas was the way to go. It would not have made sense to split the plan into one plan with scenarios involving transactions paid for in cash and a different plan with scenarios involving scenarios paid for with credit cards if types of payment type did not have many Invalid Pairs or Married Pairs associated with them.
    • We were recently talking to a client recently where 58% of their plan's parameter values were constrained. We helped them look at where most of the constraints were coming from. It turned out that "Timing of Loan Payment" was the main culprit. As a result, we suggested they consider three separate plans; one plan for Delinquent Payment Scenarios, one for Regular/Timely Loan Payment Scenarios, and one plan for Loan Pre-Payment Scenarios.
    • While working with another client that was dealing with a highly constrained plan, "Type of User" was the source of most of the constraints. Super-Users were allowed to perform all kinds of activities on the System Under Test. "Regular Users" were able to perform a far more limited number of actions. It made sense in that case to break the original combined plan into two separate plans; one plan for Admin User Scenarios and one plan for Regular User scenarios.
  • After determining where to split a plan, the next steps tend to be relatively straightforward:
    • If you're starting with one combined plan and want to break it into two plans, we would recommend these steps:
    • Start by creating 3 copies of the same plan:
    • Make a copy of the original combined plan so you can easily go back to an earlier known version if things start to go horribly wrong (or if you realize that the multiple plan strategy results in the creation of significantly more tests than the original single plan version)
    • Make a copy that you will modify for, e.g., "Regular User Scenarios"
    • Make a copy that you will modify for, e.g., "Admin User Scenarios"
    • Take advantage of Hexawise's Bulk Edit feature and tailor each plan as needed.
    • Delete any unnecessary Parameters, Invalid Pairs, Married Pairs, Requirements, and Expected results
    • Add high-priority scenarios as necessary

 

What disadvantage might there be to multiple plans?

  • A potential disadvantage to a multiple plan approach is that it sometimes results in more tests generated than a single test plan approach would.

By: Justin Hunter on Aug 14, 2014

Categories: Combinatorial Software Testing, Hexawise, Hexawise tips, Testing Checklists, Testing Strategies

Hexawise helps team achieve the following qualitative benefits:

Hexawise Benefits MM

Not all of the benefits above can easily be quantified. So what should you do if you are tasked with creating a business case to support adoption of Hexawise? Simple:

  • Measure what happens when you create ~50 tests with your business as usual process; compare that to how long it takes to generate a set of ~25 tests with Hexawise.
  • Measure how long it takes to execute each sets of tests.
  • Measure how many unique defects you find executing each set of tests.
  • For typical findings, see these empirical benefit measurement studies.
  • Then put together a business case summary like the one below.
    • The summary focuses primarily on the objectively measurable efficiency savings you measured.
    • The summary also breaks out valuable qualitative benefits into separate line items (to prevent those valuable - but difficult to quantify - benefits from being forgotten about).

Hexawise - Finanacial Benefit Model 2013.10.xlsx at 2.23.05 PM

By: Justin Hunter on Aug 11, 2014

Categories: Business Case, Hexawise, ROI, Combinatorial Testing, Pairwise Testing

Here at Hexawise, we aim to make the process of testing easier and more efficient. One way in which we've done this is by promoting our test design tool. But we were seeing people make test plans too complicated. So we came up with an easy way to create super powerful test plans that stay simple and effective.

What did we come up with, you ask?

'Start with a verb and a noun.'

The idea of using a verb and a noun to describe the appropriate scope for a set of tests has been used by Eduardo Miranda. As he points out, if you find yourself tempted to add testing ideas (e.g., explore the help files in depth) that do not easily fit into your chosen verb and noun (e.g., "book a flight"), that can be a useful red flag; accordingly, you might want to exclude those new test ideas that "don't fit" from the test scenarios in your current scope of tests.

This strategy is useful for two main reasons.

One - It is a great leaping off point from which to create interesting scenarios.

The tester is forced to understand and question their system under test. For some, this is a radically different idea to what their job is. We typically hear something like "You mean, I can't just 'validate the file exists?'"

Two - The 'verb and noun' strategy requires you remain specific to one common goal.

Test plans get bloated when you start incorporating disparate ideas. This is commonly seen when testing a system that would be described as 'Apply for Loan' and you start adding in ideas to 'explore help files.' While exploring the help files will be necessary at some point, it probably will not trigger results needed for successfully testing your application process.

Now, let's explore this first reason:

You start by choosing any verb and noun.

Verb and Noun 1

Then you have to create questions to understand that verb and noun.

Nwspaper questions

Then answer your questions. This is important. If you can't answer them, how could you possibly test the system?

Answer the questions

These ideas of questions and answers lend themselves quite well to be used as test steps or scenario planning. Below you can see how well they imported into Hexawise.

Add those into Hexawise

Generating tests is the only thing left to do before testing.

5 Click on Create Tests

Hopefully you've enjoyed this exploration on making simple and effective tests using the 'Verb and Noun' process.

By: Justin Hunter on Jun 4, 2014

Categories: Combinatorial Software Testing

Coining a New Term

I'm coining a new term today, "grapefruit juice bugs."

My inspiration for this term is a blog post in the New York Times that David Pogue wrote. I was fascinated by the post and it got me to thinking about a particular kind of bugs in software that are more common than most people may realize. You could say that these bugs are surprisingly common. In fact, if you wanted to be more precise, you could even say that this term applies to a specific type of "surprisingly common type of surprising bugs." Let me explain.

There's something about the chemical makeup of grapefruit juice that makes it interact with our biology and a large number of different drugs in ways which result in dangerous conditions. For example, certain drugs lose their effectiveness dramatically when interacting with grapefruit juice which can have life-threatening consequences. Other times, the interactions with grapefruit juice can dramatically increase a drug's potency. This can result in "safe doses" becoming very unsafe.

Grapefruit Is a Culprit in More Drug Reactions

The 42-year-old was barely responding when her husband brought her to the emergency room. Her heart rate was slowing, and her blood pressure was falling. Doctors had to insert a breathing tube, and then a pacemaker, to revive her.

They were mystified: The patient’s husband said she suffered from migraines and was taking a blood pressure drug called verapamil to help prevent the headaches. But blood tests showed she had an alarming amount of the drug in her system, five times the safe level.

Did she overdose? Was she trying to commit suicide? It was only after she recovered that doctors were able to piece the story together.

“The culprit was grapefruit juice,” said Dr. Unni Pillai, a nephrologist in St. Louis, Mo. ...

The previous week, she had been subsisting mainly on grapefruit juice. Then she took verapamil, one of dozens of drugs whose potency is dramatically increased if taken with grapefruit. In her case, the interaction was life-threatening.

Last month, Dr. David Bailey, a Canadian researcher who first described this interaction more than two decades ago, released an updated list of medications affected by grapefruit. There are now 85 such drugs on the market, he noted, including common cholesterol-lowering drugs, new anticancer agents, and some synthetic opiates and psychiatric drugs, as well as certain immunosuppressant medications taken by organ transplant patients, some AIDS medications, and some birth control pills and estrogen treatments. ... Under normal circumstances, the drugs are metabolized in the gastrointestinal tract, and relatively little is absorbed, because an enzyme in the gut called CYP3A4 deactivates them. But grapefruit contains natural chemicals called furanocoumarins, that inhibit the enzyme, and without it the gut absorbs much more of a drug and blood levels rise dramatically.

For example, someone taking simvastatin (brand name Zocor) who also drinks a small 200-milliliter, or 6.7 ounces, glass of grapefruit juice once a day for three days could see blood levels of the drug triple, increasing the risk for rhabdomyolysis, a breakdown of muscle that can cause kidney damage.

 

So what do interactions between grapefruit juice and drugs have to do with software testing?

Like grapefruit juice's impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another. This is true whether you're talking about "large parts" interact in System Testing or "small parts" interact in Unit Testing.

Interactions between things are a very rich source of bugs in software. As anyone who has heard the infernal phrase "works on my machine" can tell you, software features and functions often work perfectly fine in many usage scenarios, hardware and software configurations , etc. - only to fail to work in ever-so-slightly different situations.

 

The difference between plain old every-day "Dual-Mode Faults" and "Grapefruit Juice Bugs"

A dual-mode fault occurs whenever two test inputs must both be present to trigger a defect. Most software testers start encountering them quite frequently within days of starting their jobs. Some examples:

  • This "buy" button works fine. Except when the customer is a "new user." (First, action = "click on the buy button" and Second, customer = "new user")

  • Transaction prices for share purchases are calculated correctly. Except when denominated in Japanese Yen. (First, Action = "sell shares" and Second, Currency = "Japanese Yen")

Like grapefruit juice's impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another. This is true whether you're talking about "large parts" interact in System Testing or "small parts" interact in Unit Testing.

While all grapefruit juice bugs are dual-mode faults, not all dual-mode faults are Grapefruit Juice Bugs:

  • Grapefruit juice bugs have got to have a little of the element of surprise in them. When you explain them to a developer, their first reaction should be "Huh? How is that even possible?" or at least "Hmmm... That's odd. Let me investigate."

  • Anything along the lines of "This feature usually works, except in IE6, when..." is almost definitely not a grapefruit juice bug. Problematic interactions with IE6 are an incredibly common type of dual-mode fault, not a surprising one.

Whenever you hear "works on my machine" replies to your bug reports, and it takes a while for the issue to be replicated, odds are pretty good that a grapefruit juice bug might be involved.

Here's an example of an especially surprising grapefruit juice bug. This excerpt from Apple's online help files that the company posted after users of the original iPad complained about problems with Wi-Fi connectivity. Certain screen brightness settings were causing problems with the Wi-Fi signals. I'm not even to begin to guess how one would have anything to do with the other.

Auto-Scripting-Exercises-at-1.30.13-PM1

How to identify grapefruit juice bugs during your testing?

What is a tester to do when faced with more possible potential grapefruit juice bugs than he can handle using traditional methods?

If you're a software tester trying to do your best to determine whether a feature or function in your System Under Test will work "on everyone's machine," you've got a nightmare on your hands . Really nasty combinatorial explosions arise when you consider all of the possible combinations that would be required to test multiple hardware options, multiple software options, multiple usage scenarios, multiple test data inputs (and multiple combinations of the test data itself), multiple ways in which users enter data, and all of the rest of the "stuff that could vary" when people use your application. If you take the time to think expansively about the possible variations in a medium-sized applications, Quadrillions of possible tests often result.

While not eating grapefruit and not drinking grapefruit juice might be wise if you are taking drugs, there is rarely, if ever, such an easy method for eliminating the possibility of negative results due to software interactions. Refusing to support IE 6 in order to avoid the disproportionate number of grapefruit juice-like problematic interactions associated with IE6 would be as close as you could come in the world of software.

Design of Experiments-based test design methods can help testers come to grips with this challenge. Orthogonal array software testing (often referred to as OATS or simply OA testing) is a test design strategy that allows us to efficiently detect bugs created by interactions within the system. Orthogonal array software testing is based on the principles of multifactor designed experiments as first explored by Sir RA Fisher.

Design of Experiments-based test design methods are very-closely related to pairwise testing (AKA allpairs testing, all pairs testing, and pairwise-testing). Any of these test design strategies will allow a software tester to quickly generate a set of tests that includes tests for every single pair of test inputs.

This approach to test design often has multiple advantages, including faster test creation, more varied test scenarios, 100% coverage of all potential dual-mode faults (including hard-to-predict grape-fruit juice bugs), and often a smaller resulting set of tests that will be quicker to execute. Having said that, it is by no means a magical silver bullet. This approach to test design requires test designers with above average analytical abilities to identify the appropriate Parameters and Values for their system under test; this is sometimes easier said than done because it requires a new mindset from test designers.

Software testers can take solace that the challenges of software testing, while significant, are simple when compared to trying to understand the effects of drug interactions in people.

Combinatorial testing can look at bugs created by the interaction between multiple (3, 4, 5, 6...) variables. So if there was a bug that didn't get triggered just by using Chrome on Windows but it would get triggered if you also tried to replace an existing photo in your profile with a new profile photo into your profile (test idea number 3), then pairwise testing might not catch it. Pairwise test design would create a set of tests that would include at least one test for each of these pairs:

  • Chrome & Windows and

  • Chrome & replace photo and

  • Windows & replace photo, but...

A set of pairwise might not fail to test for the specific combination of all three of those test inputs in the same test. With the use of combinatorial test design approaches, you could create test plans with 100% coverage for 3 way interactions and be sure that all 3-way interactions or 4-way interactions are covered. When you create sets of 3-way tests, 4-way tests, 5-way tests, and 6-way tests though, you'll quickly discover that the number of tests required starts to balloon.

Hexawise allows you to create test plans with the coverage interactions you desire. This allows you to create sets of tests from 2-way up all the way up to phenomenally-thorough 6-way sets of tests. In fact, it even lets you generate clever sets of risk-based tests that will, say, prioritize comprehensive 4-way coverage on 4 sets of Parameter Values while ensuring only pairwise coverage of the other, lower-priority, interactions in your system under tests. Hexawise also lets you create mixed strength test plans so if you have certain factors that you are very concerned about and want to provide coverage for more possible interactions you can set the interaction levels for those at a higher level.

 

Related: Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed - How to Model and Test CRUD Functionality - 25 Great Quotes for Software Testers

By: Justin Hunter on Feb 11, 2014

Categories: Bugs, Combinatorial Testing, Design of Experiments, Multi-variate Testing, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies

Hexawise has had another great year and we owe it to you, our users. Thank you! As a result of your input and word-of-mouth recommendations, in 2013, the Hexawise test design tool:

  • Added powerful new features,

  • Became even easier to use,

  • Introduced lots of new practical instructional content, and

  • Doubled usage again.

If you haven’t visited Hexawise lately, please login now to see all the improvements we've made (register for free).

Ease-of-Use Enhancements

Instructional Guides for Hexawise Features
We’ve added illustrated step-by-step instructions describing how to use Hexawise features.

Find them at help.hexawise.com. For our advanced features, like creating risk-based test plans, auto-generating complete scripts, and using Value Expansions, we’ve gone beyond “how?” to explain “why?” you would want to use these features.

Practical Test Design Tips
Want to see tips and tricks for creating unusually powerful tests? Want to learn about common mistakes (and how to avoid them)? Want to understand whether pairwise test design really works in practice? These topics and more can now be found at training.hexawise.com.

Frog-Powered Self-Paced Learning

frogs next

Want to become a Hexawise guru? Listen to the frogs. If you complete the achievements recommended by your friendly amphibian guides, you will level up from a Novice to Practitioner to Expert to Guru.

frogs become expert

You’ll complete two kinds of achievements on your way to guru-ness. To complete some achievements you only need to use certain Hexawise features. Completing the other achievements requires learning important test design concepts and demonstrating that you understand them. The frogs, ever-ready to guide you towards test design mastery, will greet you immediately upon logging into your account.

Powerful New Features

Recently-added features that will make you a more powerful and speedy test designer, include:

Coverage of Specific High-Priority Scenarios
You can now force specific scenarios to appear in the tests you generate using the Requirements feature.

Requirements Traceability
Requirements traceability is easier to manage than ever with the Requirements feature.

Generation of Detailed Test Scripts
The Auto-Scripting feature allows you to automatically transform sets of optimized test conditions into test scripts that contain detailed instructions in complete sentences.

Auto-Population of Expected Results in Test Scripts
If you want to, you can even automatically generate rules-based Expected Results to appear as part of your test steps by using the Expected Results feature.

To find out more about these features Hexawise added in 2013, please check out these cool slides: "Powerful New Hexawise Features".

 

Public Recognition and Rapid Growth

Kind Words
As a five-year old company working as hard as we can to make Hexawise the best damn tool it can be, hearing input from you keeps us motivated and headed in the right direction. Once a week or so, we hear users say nice things about our tool. Here are some of the nice things you guys said about Hexawise this past year:

 

“Working coaching session with customer today. Huge data/config matrix was making them weep. Stunned silence when I showed them @Hexawise :)”

-Jim Holmes (@aJimHolmes)

 

“That would be @Hexawise & combinatorial testing to the rescue once again. #Thanks”

-Vernon Richards (@TesterFromLeic)

 

Freaking awesome visualisation of test data coverage. Kind courtesy of @Hexawise at Moolya!”

-Moolya Testing (@moolyatesting)

 

“Using @Hexawise combinatorial scenarios for e-commerce basket conditions. Team suitably impressed by speed and breadth of analysis. #Win”

-Simon Knight (@sjpknight)

 

“Just discovered Hexawise today, brilliant tool for creating test cases based on coverage of many variables.”

-Stephen Blower (@badbud65)

 

“This changes everything.”

-Dan Caseley (@Fishbowler)

 

Using Hexawise is one of the highest ROI software development practices in the world.

-Results, paraphrased, of independent study by industry expert Capers Jones and colleagues.

 

Rapid Growth
Throughout 2013, Hexawise continued to be piloted and adopted at new companies week after week. Hexawise is currently being used to generate tests at:

More than 100 Fortune 500 firms More than 2,000 smaller firms

Hexawise office

New offices
Having moved into our new offices in October, the Hexawise team now gets together to do all our best stuff at this swanky new location in Chapel Hill, North Carolina.

What's Next?

Constant Improvements
We keep a public running tally of all of our enhancements. As you can see, we’re making improvements at a rate of more than once a week.

Want us to Add a New Feature?
If you have an additional feature, please let us know. We listen. There’s not much that makes us happier than finding ways to improve Hexawise.

Please Tell Your Friends
Our growth has been almost purely due to word-of-mouth recommendations from users like you. So if you find Hexawise to be helpful, please help spread the word. We appreciate it more than you know.

You can even let your friends in on a bit of a testing industry secret: while company-wide Hexawise licenses start at $50,000 per year, we allow the first five users to use fully-featured Hexawise accounts at no cost!

Thank you for all of your help and support this past year. And have a great 2014!

By: Justin Hunter on Jan 28, 2014

Categories: Hexawise test case generating tool, Recommended Tool

So, we've been experimenting with a live customer support feature in our tool lately. We're rolling out the live chat support on a beta basis to see how useful our users find it.

The motivations for creating it were two-fold. Our first motivation was the Mayday button that Amazon Kindle announced recently. How cool is that, right? Live support available on demand any time you want it! Ingenius.

Screenshot-2013-12-04-18.07.32

 

Our second motivation for building a live chat support feature into our tool is that - while software test designers consistently tell us that they find Hexawise's features to be really easy to use - new users will often encounter test design questions as they start using the tool. We wanted to be available instantly to collaborate with them - and help them address questions in real time, like: "How should I be thinking about different ways of defining equivalence classes?" "Given what I'm trying to test in my system, how much detail is too much in this context?" etc. We wanted to be there to help users answer them. We're obsessive about customer service. Having the opportunity to have our expert test designers be a click away from every user of our tool every time they encounter a question. That's just too good an opportunity to pass up.

Early indications of how useful this service is to users are extremely promising. Users are telling us it is an amazingly helpful service. And, while we were worried that we might start to feel too stretched with tons of user questions to answer, we haven't felt that way at all. Interactions have been at manageable volumes. We've found them to be really positive. Many of the interactions have helped us learn about ways to improve our tool and/or how we can make certain test design concepts easier for Hexawise users to understand. Often, customers will click on a "call me" button to talk through questions live by phone rather than by typing back and forth. I'm glad this next conversation was done with keyboards.

This conversation happened about 45 minutes ago. Everyone at Hexawise headquarters is still smiling broadly. It made all of our days and stands apart from the rest. Enjoy!

(17:15:25) Visitor Hi

(17:15:40) Sean Johnson Hey

(17:15:44) Sean Johnson What's up?

(17:15:51) Visitor Hi Sean!

(17:15:57) Visitor Hey, I created 308 test cases

(17:16:02) Sean Johnson k

(17:16:02) Visitor out of a possible 18 trillion

(17:16:07) Sean Johnson nice!

(17:16:11) Visitor I consolidated all my user stories in 6 sprints total

(17:16:13) Visitor to 1 test set

(17:16:14) Visitor which is fine

(17:16:28) Visitor but I noticed from test case # 100 plus to 308

(17:16:37) Visitor most of my test cases are now 'any value'

(17:16:48) Visitor I was wondering if there's an option to force hexawise to pick a value for me

(17:16:52) Visitor but I don't there is

(17:16:56) Visitor but that would be a good enhancement

(17:17:05) Sean Johnson ha! are you spying on me?

(17:17:09) Visitor for 'any value' you can have the app just pick a random one

(17:17:10) Visitor LOL

(17:17:11) Visitor Nope

(17:17:14) Sean Johnson seriously… that's what I'm working on right now

(17:17:19) Visitor NO WAY!

(17:17:20) Sean Johnson what are the odds?

(17:17:24) Visitor O M G

(17:17:26) Sean Johnson yes way

(17:17:33) Visitor I've been meaning to provide that feedback 2 weeks ago

(17:17:39) Visitor but never took the time!

(17:17:46) Visitor I WOULD LOVE TO HAVE THAT FEATURE!

(17:17:49) Visitor Oh

(17:17:51) Visitor in the mean time

(17:17:56) Visitor my testers workaround

(17:18:00) Visitor is to print the test plan

(17:18:17) Visitor and just pick the values randomly from the value expansion list and input parameter list

(17:18:26) Visitor AWESOME Sean!

(17:18:31) Visitor Well let me know when it's available

(17:18:41) Sean Johnson that's really crazy

(17:18:41) Sean Johnson well… I guess I'm working on the right thing!

(17:18:42) Sean Johnson Will tomorrow be soon enough for you?

(17:18:42) Sean Johnson :-)

(17:18:43) Sean Johnson for now… it's going to be hidden

(17:18:43) Sean Johnson you'll add ?full=true

(17:18:43) Sean Johnson to the end of the URL

(17:18:57) Sean Johnson and that'll force Hexawise to fill in the any_values

(17:19:03) Visitor that's AWESOME!

(17:19:06) Visitor I will do the workaround

(17:19:09) Visitor OMG, you made my day!

(17:19:13) Visitor THANKS A TON!

(17:19:15) Visitor :-)

(17:19:18) Sean Johnson I'll send you note this evening or in the morning when it's live on [your company's Hexawise instance]

(17:19:23) Visitor I am so happy

(17:19:24) Visitor LOL

(17:19:28) Visitor Thanks so much

(17:19:32) Sean Johnson thanks for chatting! made my day to know I picked the right next priority.

(17:19:32) Visitor this will make my life easier

(17:19:38) Visitor Oh yeah, totally

(17:19:38) Sean Johnson excellent.

(17:19:58) Visitor I honestly think I'm not the only one who will appreciate this enhancement

(17:20:01) Visitor you guys are the best!

(17:20:03) Visitor Thanks so much

(17:20:05) Sean Johnson :-) we try

(17:20:15) Visitor You all do an amazing job

(17:20:21) Visitor this tool is the best

(17:20:21) Sean Johnson look for an email from me shortly

(17:20:26) Visitor will do

(17:20:27) Visitor thanks!

(17:20:33) Sean Johnson thanks!

We've been working hard for the past ~5 years building and continuously improving Hexawise so that it will be a tool that software test designers find to be extremely useful and - equally importantly - a tool that software test designers will find enjoyable to use. It is hard to put into words how satisfying it is to see an interaction like this one.

By: Justin Hunter on Dec 4, 2013

Categories: Uncategorized

Since creating Hexawise, I've worked with executives at companies around the world who have found themselves convinced in the value of pairwise testing. And then they need to convince their organization of the value.

They often follow the following path: first thinking "pairwise testing is a nice method in theory, but not applicable in our case" then "pairwise is nice in theory and might be applicable in our case" to "pairwise is applicable in our case" and finally "how do I convince my organization."

In this post I review my history helping convince organizations to try and then adopt pairwise, and combinatorial, software testing methods.

About 8 years ago, I was working at a large systems integration firm and was asked to help the firm differentiate its testing offerings from the testing services provided by other firms.

While I admittedly did not know much about software testing then but by happy coincidence, my father was a leading expert in the field of Design of Experiments. Design of Experiments is a field that has applicability in many areas (including agriculture, advertising, manufacturing, etc.) The purpose of Design of Experiments is to provide people with tools and approaches to help people learn as much actionable information as possible in as few tests as possible.

I Googled "Design of Experiments Software Testing." That search led me to Dr. Madhav Phadke (who, by coincidence, had been a former student of my father). More than 20 years ago now, Dr. Phadke and his colleagues at ATT Bell Labs had asked the question you're asking now. They did an experiment using pairwise test design / orthogonal array test design to identify a subset of tests for ATT's StarMail system. The results of that testing effort were extraordinarily successful and well-documented.

Shortly after doing that, while working at that systems integration firm, I began to advocate to anyone and everyone who would listen that designing approach to designing tests promised to be both (a) more thorough and (b) require (in most but not all cases) significantly fewer tests. Results from 17 straight projects confirmed that both of these statements were true. Consistently.

 

Repeatable Steps to Confirm Whether This Approach Delivers Efficiency and Thoroughness Improvement (and/or document a business case/ROI calculation)

How did we demonstrate that this test design approach led to both more thorough testing and much more efficient testing? We followed these steps:

  1. Take an existing set of 30 - 100 existing tests that had already been created, reviewed, and approved for testing (but which had not yet been executed).

  2. Using the test ideas included in those existing tests, design a set of pairwise tests (often approximately half as many tests as were in the original set). When putting your tests together, if there are particular, known, high-priority scenarios that stakeholders believe are important to test, it is important to make sure that that you "force" your pairwise test generator to include such high-priority scenarios.

  3. Have two different testers execute both sets of tests at the same time (e.g., before developers start fixing any defects that are uncovered by testers executing either set of tests)

Document the following:

  • How long did it take to execute each set of tests?

  • How many unique defects were identified by each set of tests?

  • How long did it take to create and document each set of tests?*

 

*This third measurement was usually an estimate because a significant number of teams had not tracked the amount of time it took to create the original set of tests.

The results in 17 different pairwise testing "bake-off" projects conducted at my old firm included:

  • Defects found per tester hour during test execution: when pairwise tests were used, more than twice as many defects were found per tester hour

  • Total defects found: pairwise tests as many or more defects in every single project (despite the fact that in almost every case there were significantly more tests in the each original set of tests)

  • Defects found by pairwise tests but missed by traditional tests: a lot (I forget the exact number)

  • Defects found by traditional tests but missed by pairwise tests: zero

  • Amount of time to select and document tests: much less time required when a pairwise test generator was used (As mentioned above, precise measurements were difficult to gather here)

 

More recent project benefits have included these:

bcbs1

bcbs2

Those experiences - combined with the realization that many Fortune 500 firms were starting to try to implement smarter test design methods to achieve these kinds of benefits but were struggling to find a test design tool that was user-friendly and would integrate into their other tools - led me to the decision to create Hexawise.

 

Additional Advice and Lessons Learned Based on My Experiences

Once the testing the value of pairwise software testing at a specific organization it is very common to find the proponent of taking advantage of pairwise testing advantages to find themselves saying:

I have already elaborated some test plans that would save us up to 50% effort with that method. But now my boss and other colleagues are asking me for a proof that these pairwise test cases suffice to make sure our software is running well.

In that case, my advice is three-fold:

First, appreciate how your own thinking has evolved and understand that other people will need to follow a similar journey (and that others won't have as much time to devote as you have had to experience learnings first-hand).

When I was creating Hexawise, George Box, a Design of Experiments expert with decades of experience explaining to skeptical executives how Design of Experiments could deliver transformational improvements to their organizations' efficiency and effectiveness, told me "Justin, you'll experience three phases of acceptance and acceptance will happen more gradually than you would expect it to happen. First, people will tell you 'It won't work.' Next, they'll say "It won't work here." Eventually, he said with a smile, they'll tell you 'Of course this works. I thought of it first!'

When people hear that you can double their test execution productivity with a new approach, they won't initially believe you. They'll be skeptical. Most people you're explaining this approach to will start with the thought that "it is nice in theory but not applicable to what I'm doing." It will take some time and experience for people to understand and appreciate how dramatic the benefits are.

Second, people will instinctively be dismissive of pairwise testing case study after case study after case study that show how effective this approach has been for testers in virtually all types of industries and all types and phases of testing. George Box was right when he predicted that people will often respond with 'It won't work here.' Sometimes it is hard not to smile when people take that position.

Case in point: I will be talking to a senior executive at a large capital markets firm soon about how our tool can help them transform the efficiency and effectiveness of their testing group. And I can introduce them to a client of ours that is using our test design tool extensively in every single one of their most important IT projects. Will that executive take me up on my offer? I hope so, but based on past experience, I suspect odds are good that he'll instead react with 'Yes, yes, sure, if companies were people, that company would be our company's identical twin, but still... It won't work here.'

Third, at the end of the day, the most effective approach I have found to address that understandable skepticism and to secure organizational-level buy-in and commitment is through gathering hard, indisputable evidence on multiple projects that the approach works at the company itself through a bake-off approach (e.g., following those four steps outlined above. A few words of advice though.

My proposed approach isn't for the faint of heart. If you're working at a large company with established approaches, you'll need patience and persistence.

Even after you gather evidence that this approach works in Business Unit A, and B and C, someone from Business Unit D will be unconvinced with the compelling and irrefutable evidence you have gathered and tell you 'It won't work here. Business Unit D is unique.' The same objections may likely arise with results from "Type of Testing" A, B, and C. As powerful and widely-applicable as this test design approach is, always remember (and be clear with stakeholders) that it is not a magical silver bullet.

James Bach raises several valid limitations with using this approach. In particular, this approach won't work unless you have testers who have relatively strong analytical skills driving the test design process. Since pairwise test case generating tools are dependent upon thoughtful test designers to identify appropriate test inputs to vary, this approach (like all test design approaches) is subject to a "garbage in / garbage out" risk.

Project leads will resist "duplicating effort." But unless you do an actual bake-off stakeholders won't appreciate how broken their existing process is. There's inevitably far more wasteful repetition hidden away in standard tests than people realize. When you start reporting a doubling of tester productivity on several projects, smart managers will take notice and want to get involved. At that point - hopefully - your perseverance should be rewarded.

 

Some benefits data and case studies that you might find useful:

 

If you can't change your company, consider changing companies

Lastly, remember that your new-found skills are in high demand whether or not they're valued at your current company. And know that, despite your best efforts and intentions, your efforts might not convince skeptics. Some people inevitably won't be willing to take the time to understand. If you find yourself in a situation where you want to use this test design approach (because you know that these approaches are powerful, practical, and widely-applicable) but that you don't have management buy-in, then consider whether or not it would be worth leaving your current employer to join a company that will let you use your new-found skills.

Most of our clients, for example, are actively looking for software test designers with well developed pairwise and combinatorial test design skills. And they're even willing to pay a salary premium for highly analytical test designers who are able to design sets of powerful tests. (We publicize such job openings in the LinkedIn Hexawise Guru group for testers who have achieved "Guru" level status in the self-paced computer-based-training modules in the tool).

 

Related: Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing - Systematic Approaches to Selection of Test Data - Getting Known Good Ideas Adopted

By: Justin Hunter on Nov 21, 2013

Categories: Pairwise Software Testing, Software Testing, Software Testing Efficiency, Testing Case Studies, Testing Strategies

Recently, the following defect made the news and was one of the most widely-shared articles on the New York Times web site. Here's what the article, Computer Snag Limits Insurance Penalties on Smokers said:

A computer glitch involving the new health care law may mean that some smokers won’t bear the full brunt of tobacco-user penalties that would have made their premiums much higher — at least, not for next year.

The Obama administration has quietly notified insurers that a computer system problem will limit penalties that the law says the companies may charge smokers, The Associated Press reported Tuesday. A fix will take at least a year.

 

Tip of the Iceberg

This defect was entirely avoidable and predictable. Its safe to expect that hundreds (if not thousands) of similar defects related to Obamacare IT projects will emerge in the weeks and months to come. Had testers used straightforward software test design prioritization techniques, bugs like these would have been easily found. Let me explain.

 

There's no Way to Test Everything

If the developers and/or testers were asked how could this bug could sneak past testing, they might at first say something defensive, along the lines of: "We can't test everything! Do you know how many possible combinations there are?" If you include 40 variables (demographic information, pre-existing conditions, etc.) in the scope of this software application, there would be:

41,231,686,041,600,000

possible scenarios to test. That's not a typo: 41 QUADRILLION possible combinations. As in it would take 13 million years to execute those tests if we could execute 100 tests every second. There's no way we can test all possible combinations. So bugs like these are inevitably going to sneak through testing undetected.

 

The Wrong Question

When the developers and testers of a system say there is no way they could realistically test all the possible scenarios, they're addressing the wrong challenge. "How long would it take to execute every test we can think of?" is the wrong question. It is interesting but ultimately irrelevant that it would take 13 million years to execute those tests.

 

The Right Question

A much more important question is "Given the limited time and resources we have available for testing, how can we test this system as thoroughly as possible?" Most teams of developers and software testers are extremely bad at addressing this question. And they don't realize nearly how bad they are. The Dunning Kruger effect often prevents people from understanding the extent of their incompetence; that's a different post for a different day. After documenting a few thousand tests designed to cover all of the significant business rules and requirements they can think of, testers will run out of ideas, shrug their shoulders in the face of the overwhelming number of total possible scenarios and declare their testing strategy to be sufficiently comprehensive. Whenever you're talking about hundreds or thousands of tests, that test selection strategy is a recipe for incredibly inefficient testing that both misses large numbers of easily avoidable defects and wastes time by testing certain things again and again. There's a better way.

 

The Straightforward, Effective Solution to this Common Testing Challenge: Testers Should Use Intelligent Test Prioritization Strategies

If you create a well-designed test plan using scientific prioritization approaches, you can reduce the number of individual tests to test tremendously. It comes down to testing the system as thoroughly as possible in the time that's available for testing. There are well-proven methods for doing just that.

 

There are Two Kinds of Software Bugs in the World

Bugs that don't get found by testers sneak into production for one of two main reasons, namely:

  • "We never thought about testing that" - An example that illustrates this type of defect is one James Bach told me about. Faulty calculations were being caused by an overheated server that got that way because of a blocked vent. You can't really blame a tester who doesn't think of including a test involving a scenario with a blocked vent.

  • "We tested A; it worked. We tested B; it worked too.... But we never tested A and B together." This type of bug sneaks by testers all too often. Bugs like this should not sneak past testers. They are often very quick and easy to find. And they're so common as to be highly predictable.

 

Let's revisit the high-profile bug Obamacare bug that will impact millions of people and take more than a year to fix. Here's all that would have been required to find it:

  • Include an applicant with a relatively high (pre-Medicare) age. Oh, and they smoke.

 

Was the system tested with a scenario involving an applicant who had a relatively high age? I'm assuming it must have been.

Was the system tested with a scenario involving an applicant who smoked? Again, I'm assuming it must have been.

Was the system tested with a scenario involving an applicant who had a relatively high age who also smoked? That's what triggers this important bug; apparently it wasn't found during testing (or found early enough).

 

If You Have Limited Time, Test All Pairs

Let's revisit the claim of "we can't execute all 13 million-years-worth of tests. Combinations like these are bound to sneak through, untested. How could we be expected to test all 13 million-years-worth of tests?" The second two sentences are preposterous.

  • "Combinations like these are bound to sneak through, untested." Nonsense. In a system like this, at a minimum, every pair of test inputs should be tested together. Why? The vast majority of defects in production today would be found simply by testing every possible pair of test inputs together at least once.

  • "How could we be expected to test all 13 million-years-worth of tests?" Wrong question. Start by testing all possible pairs of test inputs you've identified. Time-wise, that's easily achievable; its also a proven way to cover a system quite thoroughly in a very limited amount of time.

 

Design of Experiments is an Established Field that was Created to Solve Problems Exactly Like This; Testers are Crazy Not to Use Design of Experiments-Based Prioritization Approaches

The almost 100 year-old field of Design of Experiments is focused on finding out as much actionable information as possible in as few experiments as possible. These prioritization approaches have been very widely used with great success in many industries, including advertising, manufacturing, drug development, agriculture, and many more. While Design of Experiments test design techniques (such as pairwise testing and orthogonal array testing / OA testing) are increasingly becoming used by software testing teams but far more teams could benefit from using these smart test prioritization approaches. We've written posts about how Design of Experiments methods are highly applicable to software testing here and here, and put an "Intro to Pairwise Testing" video here. Perhaps the reason this powerful and practical test prioritization strategy remains woefully underutilized by the software testing industry at large is that there are too few real-world examples explaining "this is what inevitably happens when this approach is not used... And here's how easy it would be to avoid this from happening to you in your next project." Hopefully this post helps raise awareness.

 

Let's Imagine We've Got One Second for Testing, Not 13 Million Years; Which Tests Should We Execute?

Remember how we said it would take 13 million years to execute all of the 41 quadrillion possible tests? That calculation assumed we could execute 100 tests a second. Let's assume we only have one second to execute tests from those 13 million years worth of tests. How should we use that second? Which 100 tests should we execute if our goal is to find as many defects as possible?

If you have a Hexawise account, you can to your Hexawise account to view the test plan details and follow along in this worked example. To create a new account in a few seconds for free, go to hexawise.com/free.

By setting the 40 different parameter values intelligently, we can maximize the testing coverage achieved in a very small number of tests. In fact, in our example, you would only need to execute only 90 tests to cover every single pairwise combination.

The number of total possible combinations (or "tests") that are generated will depend on how many parameters (items/factors) and how many options (parameter values) there are for each parameter. In this case, the number of total possible combinations of parameters and values equal 41 quadrillion.

 

insurance-bug-1

This screen shot shows a portion of the test conditions that would be included the first 4 tests of the 90 tests that are needed to provide full pairwise coverage. Sometimes people are not clear about what "test every pair" means. To make this more concrete, by way of a few specific examples, pairs of values tested together in the first part of test number 1 include:

  • Plan Type = A tested together with Deductible Amount = High

  • Plan Type = tested together with Gender = Male

  • Plan Type = A tested together with Spouse = Yes

  • Gender = Male tested together with State = California

  • Spouse = Yes tested together with Yes (and over 5 years)

  • And lots of other pairs not listed here

 

insurance-bug-2

This screen shot shows a portion of the later tests. You'll notice that the values are shown in purple italics. Those values listed in purple italics are not providing new pairwise coverage. You will note in the first tests every single parameter value is providing new pairwise coverage value, toward the end few parameter value settings are providing new pairwise coverage. Once a specific pair has been tested, retesting it doesn't provide additional pairwise coverage. Sets of Hexawise tests are "front loaded for coverage." In other words, if you need to stop testing at any point before the end of the complete set of tests, you will have achieved as much coverage as possible in the limited time you have to execute your tests (whether that is 10 tests or 30 tests or 83). The pairwise coverage chart below makes this point visually; the decreasing number of newly tested pairs of values that appear in each test accounts for the diminishing marginal returns per test.

 

You Can Even Prioritize Your First "Half Second" of Tests To Cover As Much As Possible!

insurance-bug-3

This graph shows how Hexawise orders the test plan to provide the greatest coverage quickly. So if you get through 37 of the 90 tests needed for full pairwise coverage you have already tested over 90% of all the pairwise test coverage. The implication? Even if just 37 tests were covered, there would be a 90% chance that any given pair of values that you might select at random would be tested together in the same test case by that point.

 

Was Missing This Serious Defect an Understandable Oversight (Because of Quadrillions of Possible Combinations Exist) or was it Negligent (Because Only 90 Intelligently Selected Tests Would Have Detected it)?

A generous interpretation of this situation would be that it was "unwise" for testers to fail to execute the 90 tests that would have uncovered this defect.

A less generous interpretation would be that it was idiotic not to conduct this kind of testing.

The health care reform act will introduce many such changes as this. At an absolute minimum, health insurance firms should be conducting pairwise tests of their systems. Given the defect finding effectiveness of pairwise testing coverage, testing systems any less thoroughly is patently irresponsible. And for health insurance software testing it is often wiser to expand to test all triples or all quadruples given the interaction between many variables in health insurance software.

Incidentally, to get full 3 way test coverage (using the same example as above) would require 2,090 tests.

 

Related: Getting Started with a Test Plan When Faced with a Combinatorial Explosion - How Not to Design Pairwise Software Tests - Efficient and Effective Test Design

By: Justin Hunter on Sep 26, 2013

Categories: Combinatorial Software Testing, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies

As the CEO of a small but quickly-growing SaaS (Software as a Service) firm that often doubles software tester productivity, I can attest that Fortune 500 firms I'm talking to are way less "anti-SaaS" than they were just 12 and 24 months ago. Business is booming. More than 100 Fortune 500 firms currently have testers using our tool to design their software tests.

It doesn't take Nostradamus to predict that news stories talking about how "SaaS solutions are innovative and beating out 'traditional' software" will become more and more rare. Increasingly, SaaS solutions, with data stored remotely "in the cloud" by hosting providers like Amazon Web Services, are receiving mainstream acceptance.

The situation we're in now reminds me of when I helped launch Asia's first internet-based stock brokerage firm in 1996/97. It was "big news!" that generated coverage from CNN, Time, a front page article on the South China Morning Post business section, etc. Every reporter we talked to focused a lot of their attention on the potentially grave security risks of this new way of trading stocks. Today, trillions of dollars worth of online trade executions later, a Hong Kong brokerage firm offering its customers the ability to trade stocks online wouldn't be worthy of a mention in a neighborhood newspaper. It's just accepted as the way things are done.

We're quickly heading that way with SaaS solutions too.

 

Related: Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing - A Fun Presentation on a Powerful Software Test Design Approach - What Software Testers Can Learn from the Game of 20 Questions

By: Justin Hunter on May 28, 2013

Categories: User Experience

This post addresses some comments and skeptical (in a good way) questions raised by Phil Kirkham to our recent posts: The Software Testing Community Needs More Empirical Studies.

As background to my answers: The studies and dozens of proof of concept pilot projects that I’ve been directly involved with have sought to answer these 3 questions:

1) Is it actually faster to generate tests with Hexawise than creating and documenting them manually?

Consistent findings: Yes. It takes, on average, about 40% less time to create and document tests using Hexawise because using Hexawise allows testers to partially automate test selection and test documentation steps.

2) Is it possible to generate smaller sets of tests that will be as thorough or more thorough than larger sets of manually created tests and allow testers to find more defects in less test execution time?

Consistent findings: Yes. Typically more than twice as many defects per tester hour. See, e.g., the IEEE Computer article written with 3 PhDs showing an increase in defects found per tester hour of 2.4 times. A more recent set of 10 proof of concept pilot projects at an insurance firm revealed 3.0 times as many defects per tester hour. See: Does pairwise testing really work? Evidence, data, and case studies.
This is because Hexawise-generated tests (or any pairwise tests, for that matter) consistently have dramatically less wasteful repetition than manually selected tests will and because Hexawise-generated tests leave no potential dual-mode faults untested (that is, no potential pairwise defects involving test inputs that have been contemplated by the tester and included in their models).

3) Finding more defects per tester hour is certainly nice, but do Hexawise-generated tests find MORE defects?

Consistent findings: Yes. Much smaller set of Hexawise tests have consistently found more defects. On average by about 13%

 

In answer to specific questions:

Phil Kirkham: “Seems a very basic measure of a testers productivity How about the severity of the defects ?”

JH: Agreed.

In my experience in more than 5 years of helping teams conducting these proof of concept pilot projects, pairwise and Hexawise-generated tests are just plan more effective at finding defects. They find more of ALL kinds of defects. My experience has been that the types of defects being found is not skewed towards less significant types of defects or missing severe defects. A case in point: at my old firm, one of the early adopters of orthogonal array testing approaches ran pilot project after pilot project with teams of testers reporting into him. I can’t remember the exact number of pilots he had conducted, perhaps 20 pilot projects or so, before he experienced a single defect that escaped the Hexawise-generated tests that was found by the much longer set of manually selected tests. So, a short, blunt, honest answer to your excellent question, is “Believe it or not, Phil, it almost never matters. This approach will find ALL of the defects you otherwise would have found. Plus additional ones.”*

*Major caveat here that calls into question my specific answer here (to your question concerning severity) as well as all of the results from all of the studies I’ve been involved with. Testers like you and me are strong proponents of Exploratory Testing. These studies, though, treat the test inputs and test cases as “frozen.” You have the test cases in list A (created manually) and the test cases in list B (created by using Hexawise). The ideas about what can be changed from test to test (parameters) and how each of those things can be changed (values) are identical in both lists. The difference is that list A has lots of wasteful repetition and lots of gaps in coverage. List B has neither. That’s the only difference. Then one tester executes the tests from list A and another tester executes the tests from list B. But what if you have an unskilled tester following rote scripts executing one set of tests and someone like you, Rob Sabourin, Michael Bolton, James Bach, Shmuel Gershon, Ajay Balamurugadas, etc., executing the second set? Whoa! All bets are off. What would happen is that skilled Exploratory Testers would use the Hexawise-generated test ideas (which they would not want to be overly-detailed), and go “off script” to explore interesting test ideas that they cooked up in real time as they were doing their testing. So skilled Exploratory Testers would be able to find defects (presumably including serious ones) that the written test cases, regardless of whether they were manually created or created by Hexawise, would not lead them to directly. That’s an important topic for another time. I’ll be talking about Exploratory Combinatorial Testing at the Conference of the Association of Software Testing – CAST – this year in my home town of Madison, Wisconsin. Since you’re also going, perhaps we could collaborate and you could share your experiences (good or bad) with the attendees. I’ll happily give you 10 minutes of my speaking time to share your experiences if you’d like.

PK: Are you only counting functional defects ?

JH: Not explicitly. The directions I give to teams running these pilot projects is. Try to answer the 3 questions above. Report defects. We’re not after a count of “failed test cases.” We’re after a number of defects. Having said that, as you might suspect, most of the defects reported tend to be functional defects.

PK: What about all the other types of defect ?

JH: They’re not reported as often but we count them too. In situations where one tester reports a “hard to spot” bug (e.g., one that might take a more experienced tester to identify), it raises the possibility that the bug is being reported not because one set of tests is superior to the other but because one tester is better than the other. Accordingly, in an effort to keep an apples to apples comparison, we talk with the tester and try to determine with the tester’s input whether the tester would have found that same defect with the other set of tests. If the answer is yes, we’d report the defect as “found” in both sets of tests. This doesn’t happen as often as you might suspect it would.

PK: Does the project type matter ?

JH: Yes. Benefits tend to be relatively smaller when there are a disproportionately high percentage of small, discrete, one-off tests. And higher when there are more than 5 parameters that interact in meaningful ways. And easier to capture when the System Under Test does not have a lot of conditional branching logic.

PK: How about the devs they are working with and the practices they follow ?

JH: I don’t have enough empirical evidence to say definitively. It’s used successfully by thousands of testers in waterfall projects and thousands of testers in Agile projects.

PK: What about the experience of the tester, does that make a difference?

JH: Even more important than experience level of the tester are, in order, (1) analytical ability, (2) willingness to try new things, and (3) willingness to ask questions. By my estimates about 50% of the testers I come across at our clients (almost all at Fortune 2000 firms) would not be able to design excellent sets of pairwise tests from scratch. This is because above average analytical ability is required for testers to select parameters and values from their Systems Under Test in a thoughtful way. Getting back to experience level, some of our strongest users at our clients are straight of out of college. They start work, get exposed to Hexawise, “get it” and don’t look back. Interestingly, some testers who have been testing for, say, 10 years or more – while experienced – sometimes seem to be too set in their ways to embrace this rather different approach to designing tests.

PK: If they are working with top of the range developers (as some lucky testers are cough cough) then there aren’t that many functional bugs to be found and you’re looking at browser compatibility, usability, race conditions – is combinatorial testing going to find these more quickly ?

JH: Yes. Absolutely. If you’d like to collaborate to test that and help gather empirical evidence that you could share at CAST, I would be happy to work with you to do just that. If your experience contradicts what I’m saying here, you’d have the floor to tell CAST participants what your actual experience was.

PK: I read the study in the link – 97% of defects could be found by pairwise combinatorial testing ? Really ? ALL types of defects ? Really ? How can pairwise find a defect caused by a missing or ambiguous or inconsistent requirement, or a performance or security ?

JH: The statistics I quote are a lot lower than that. The pie chart I use averages out several studies done by PhD’s that have found, on average, 84% of defects could be triggered by testing for all combinations of 2 test inputs. The 97% figure is eyebrow-raising on its own (regardless of industry). Given that it was in the medical device industry in the United States (one of the most litigious area in the history of the world?), that statistic is particularly mind-boggling. What the PhDs in that study did was take a look at all of the medical devices that had been taken off of the market in the United States as a result of software defects. Then they investigated how many test inputs would be required to trigger each of those defects. They authors of that study found that an astonishing 97% of those defects could have been triggered by just 2 test inputs.

PK: Love your passion and enthusiasm and I do have a beta of Hexawise to see if it can do anything for my productivity – and I might agree that there is a lack of empirical studies, not just among the testing community but the s/w community as a whole into the effectiveness or not of how software is produced

JH: Thanks. I hope you have positive experiences with using Hexawise and I’m happy to help you if ever have any questions about using Hexawise on your projects.

By: Justin Hunter on Mar 25, 2013

Categories: Combinatorial Software Testing, Efficiency, Pairwise Software Testing, Software Testing Efficiency, Testing Strategies

Based on my experience, over dozens of pilot projects where we've gathered hard data, many software testers would literally more than double their productivity overnight on many projects if they used combinatorial test design methods intelligently (in comparison to selecting test case conditions by hand).

In this 10 project study, Combinatorial Software Testing Case Studies, we found 2.4 times more defects per tester hour on average when we compared testers who executed manually-selected test cases to testers who executed test cases created by a combinatorial testing algorithm designed to achieve as much coverage as possible in as few tests as possible.

How many participating testers thought they would see dramatic increases before they gathered the data? Almost none (even the testers told about the prior experiences of their other colleagues on similar projects). How many participating testers are glad that they took the time to use the scientific method?

  • hypothesis

  • experiment

  • evidence

  • revise world-view

Every one of them.

What stops more people from using the scientific method on their projects and gathering data to prove or disprove hypotheses like the one addressed in the study above? A pilot could take one person's time for less than 2 days. If past experience is any indication of future results (and granted, it isn't always), odds would appear pretty good that results would show that productivity would double (as measured in defects found per tester hour).

What's stopping the testing community from doing more such analysis? Perhaps more importantly, what is stopping you from gathering this kind of data on your project?

Additional empirical studies on the effectiveness of software testing strategies would greatly benefit the software testing community.

 

Related: Hexawise case studies on software testing improvement (health insurance, IT consulting and mortgage processing studies) - How Not to Design Pairwise Software Tests - 3 Strategies to Maximize Effectiveness of Your Tests

By: Justin Hunter on Mar 5, 2013

Categories: Combinatorial Testing, Efficiency, Pairwise Software Testing, Testing Case Studies, Testing Strategies, Experimenting

I strongly agree with Cem Kaner's statement (in Inefficiency and Ineffectiveness of Software Testing: A Key Problem in Software Engineering) that: sometimes tests uncover defects that don’t fit within any coverage model because they are side effects of the tests rather than explicitly planned foci of the tests."

My experience indicates that an effective way to increase the likelihood that you will trigger such defects (without explicitly looking for them) is to try to maximize the variation between each test case you execute.

A case in point: when I sat down to dinner with James Bach a year or so ago in Boston at a testing conference, he gave me a quick testing challenge (as he is fond of doing with testers he meets for the fist time to see how we think). He asked how I would test a very simple calendar entry application that allowed users to record the start and end times of diary events. Key inputs to use as test conditions for these tests included start times and end times.

I proposed a set of times to try that were designed to provide as much variety as possible from one test case to the next. As inputs into the start and end times, I used a small number of different times spread throughout the morning, afternoon, and evening as well. The strategy I used quickly identified the testing defect the puzzle was designed to uncover in a small handful of tests. What was most memorable about the experience from my perspective was not that I "succeeded" in triggering the bug but that the tests I created triggered a type of bug that was, in Kaner's words, a "side effect of the tests rather than explicitly planned foci of the tests."

The business logic in the calendar application that should have identified invalid beginning and end time combinations was coded incorrectly. Instead of using numbers in the business logic, the business logic was ordering the numbers alphabetically. I was not consciously looking to identify that kind of a flaw in the business logic, but by maximizing the variation from test case to test case, I maximized my odds of finding it.

Efficiently achieving structured variation is difficult because it is hard for a human brain to remember whether dozens of different test conditions have been tested together (or we're accidentally repeating ourselves). This is where pairwise and combinatorial test case generating tools like our Hexawise tool come in. They are designed to achieve as much variation from test case to test case as possible. One of the relatively unsung benefits of this approach is that doing so will help find bugs, like these, that you aren't even consciously looking for.

 

Related: 3 Strategies to Maximize Effectiveness of Your Tests - Getting Started with a Test Plan When Faced with a Combinatorial Explosion - Book Review of “Explore It!” Elisabeth Hendrickson’s Excellent New Book on Software Testing

By: Justin Hunter on Feb 26, 2013

Categories: Software Testing, Testing Strategies

20100302-dnpabhucxq6ucrs5hqdnhwnbku

Design of Experiments in Software Testing - Pairwise and Combinatorial - Hexawise

Justin Hunter @Hexawise:

 

Removing inefficiency is good, sure, but it is not why Design of Experiments is so friggin' powerful. Saying DoE is interesting to know about because it can help identify and remove specific inefficiencies is a bit like saying Canada is a good country to visit because you can sometimes find a good cup of coffee there. To my mind, saying DoE is primarily about removing inefficiency misses the main point.

Design of Experiments is so powerful because it allows practitioners to predictably, systematically, and consistently find out more useful, actionable information in much less time than they would otherwise take to obtain this information (if they could find it at all with their less-structured approaches).

In manufacturing circles (e.g., when engineers produce new prototypes), DoE's ability to do this is no longer questioned. This is because leaders like George Box taught people in industry how to apply DoE and they gathered conclusive evidence that DoE allowed manufacturers to learn much faster through techniques like applying factorial designs. Box and other DoE experts (Taguchi, Montgomery, my dad, etc.) dealt with skeptical manufacturing engineers for four decades by showing them the facts and using DoE on the skeptics' own projects right under their noses. The evidence that DoE allows manufacturers to learn much faster (about a wide variety of learning goals) than the other methods they used prior to 1960 is incontrovertible.

In 2010, in the gradually maturing field of software testing, Design of Experiments-based methods of test case design has not caught on much at all yet. As an industry, it's adoption of DoE-based approaches is roughly where manufacturing was in 1960. Most software testers, even very good ones, don't know anything at all about how DoE can help them. Many other software testers have heard a bit about pairwise but mistakenly think pairwise and related, structured, DoE-based, test case selection method can't help them.

Even some of the best testers in the world who have written some of the most clearly-written and well-reasoned articles about pairwise approaches do not (in my view) seem to fully-understand: (a) how powerful the benefits are, (b) how often the approach can be applied / in how many diverse kinds of testing situations they can be utilized, and/or (c) how consistently the efficiency and effectiveness benefits are be generated when they are used properly. DoE methods, including pairwise and n-wise and mixed strength automatic test condition generation (made possible by tools like our Hexawise tool and also, to a great extent by James Bach's free AllPairs tool) allow software testers to learn much faster about critically important questions like: (1) where are the bugs?, (2) what is causing the bugs to appear?, (3) am I confident I have efficiently tested for a huge range of combinations of values in the System Under Test that might trigger defects? (4) am I succeeding in avoiding redundant repetition of steps in many test cases?, (5) how many bugs would be likely to find if we were to continue to run the next 100 tests?, etc.

In summary, the reason for the existence of Design of Experiments methods (whether we're talking about their applicability to testing software as efficiently and effectively as possible, or DoE methods' applicability to a huge variety of other objectives) - and, for that matter, the reason that they have been continuously refined and improved for 40+ years - is that DoE methods consistently and predictably allow users to learn actionable results as quickly as possible.

 

Related: Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed - Pairwise and Combinatorial Software Testing in Agile Projects - Video Highlight Reel of Hexawise

By: Justin Hunter on Feb 11, 2013

Categories: Design of Experiments, Multi-variate Testing, Recommended Tool, Testing Strategies

Hexawise was down for all of our users for most of the last hour just now. I apologize for the inconvenience caused. It was entirely my fault. I wanted to give you, our users, an explanation.

When Gmail, Twitter, and Amazon's EC2 service experience outages that impact large percentages of their users, executives at those firm sometimes need a couple days to analyze exactly what wrong and produce a report explaining the failure to their users. No complex or time-consuming analysis is required here. I messed up.

Here's the honest but somewhat embarrassing truth about what happened:

  • When I founded Hexawise, 4 years ago today, I registered the domain name Hexawise.com personally.

  • Each year, we need to pay an annual fee of $10 to keep our domain name renewed.

  • Each year, I pay it.

  • Except this year.

  • This year, I took my eye off the ball when I went on vacation last week to experience the Maha Kumbh Mela (a massive Hindu pilgrimage that takes place at the Ganges river once every 12 years that has been referred to as "the world's largest gathering of humanity").

  • I missed an email from our domain registrar highlighting that payment was due as I was setting off to the Kumbh Mela.

  • Just returning from vacation today, I realized that the site was down, jumped on the phone with Sean Johnson, our CTO, who explained what had happened (and that he didn't have access to the account I use to make the annual payment). I promptly paid it and things returned to normal.

 

450px-Third Shahi Snan in Hari Ki Pauri

Kumbh Mela (image from Wikipedia commons)

 

Here's what we'll do differently in the future to ensure this won't happen again:

  • I've updated credit card info with our domain name registrar and put this payment on auto-renew.

  • I've put calendar reminders on my calendar and our CFO's for the coming years; we'll both personally proactively confirm that it has been made.

  • I've put the email address of the domain registrar into my VIP email folder to ensure I won't miss any future emails from them.

Again, I apologize for the inconvenience caused. I've let down both you (users of Hexawise), and our engineers (who have kept Hexawise up >99.9% of the time even through hundreds of updates to Hexawise that they've put into production within the last four years). They're regularly going to great lengths to ensure the site is up so that you can generate efficient and effective software tests 24/7/365 using Hexawise. You and our engineers deserve better than a CEO who dropped the ball and brought the site down today. I hope you'll excuse this lapse. I won't make this same mistake again. We will continue to strive to keep site uptime as close to 100% as we possibly can.

 

Justin Hunter

Founder and CEO of Hexawise

By: Justin Hunter on Jan 30, 2013

Categories: Hexawise test case generating tool

I am passionate about pairwise software testing techniques. I have helped dozens of teams, for example, carefully measure the benefits that can be created when teams of testers adopt pairwise and related combinatorial testing approaches to identify the test cases they will execute (as compared to manual test case identification methods). What usually happens is that tester productivity doubles. (See Combinatorial Software Testing - pdf download).

I believe these approaches will be much more widely adopted in a few years than they are now for the simple reason that they consistently deliver dramatic benefits to both the speed of software test design and the efficiency and thoroughness of software test execution. As more teams try these methods for themselves, and measures the benefits they achieve with them, broader adoption seems highly likely to me.*

I see three main barriers to broader adoption by the testing community at large:

  1. The first barrier is that testers will not make an attempt to apply this method to their testing projects so they will never find out how effective it is. The second is that ill-informed testers will try to apply the approach but do such a poor job at implementation that they do not generate benefits.

  2. The second barrier is that even testers who use the approach effectively a few times, will not realize how much more effective it is making them. A dismissive thought process guilty of this might sound something like this: "Those 11 bugs I just found? Yeah. I found them because I'm a good tester; the fact that I happened to use pairwise tests just now? That's largely irrelevant. I'm sure I would have found them regardless.")

  3. The third barrier is that testers unfamiliar with the basics of pairwise testing principles will design test cases without thinking about what they are doing, and achieve "garbage in / garbage out" results. The benefits that would have been so easily achieved in the testing project - like Lindsey Jacobellis' opportunity to win a gold medal for Snow Boarding - disappear in a groan-worthy moment of bone-headed stupidity.

 

 

This blog post addresses this third barrier. When testers sabotage their own test plans with a poor choice of inputs, they may well blame the test design strategy rather than themselves, which would be unfortunate. Here's one common problem I see (exaggerated a bit in this example to make my point).

 

Objective: create a set of tests that will check to see if the underwriting engine for a car insurance firm is calculating premium estimates correctly.

Our aspiring pairwise test designer enters stage left and identifies a set of parameters:

First Name, Last Name, Age of Primary Driver, Credit Score, Number of Cars, Number of Accidents, Number of Speeding Tickets, and Number of Additional Drivers

So far so good. We now have the initial ingredients for a thing of beauty; we have a set of parameters that could quickly result in a combinatorial explosion of possibilities and, ready to save the day, we have a test designer who has correctly identified this as an opportunity to achieve efficiency and thoroughness benefits through the application of pairwise testing methods. Our potential hero is a couple minutes away from creating a concise set of tests that will confirm not only confirm that each of the data points in the plan work as they should but that they work as they should in combination with each of the other data points in the test plan.

In other words, the plan will not only confirm that "Number of Accidents = 3" will impact premiums as it should on its own, but also that "Number of Accidents = 3" will work as it should when tested in combination with the other relevant inputs in the application, e.g.,: 3 accidents with every relevant input for "Age of Primary Driver," 3 accidents with every relevant input for "Credit Score," 3 accidents with every relevant input for "Number of Cars," 3 accidents with every relevant combination for "Number of Speeding Tickets," and 3 accidents with every relevant input for "Number of Drivers."

He's seen the Promised Land of improved efficiency and effectiveness and he's ready to enter. Unfortunately, with his next move, he demonstrates he's a doofus. Entry to Promised Land denied. Check out the values he chose to enter for each of his parameters.

 

hexawise-screenshot

Notice anything wrong here?

 

Just for fun, let's take a close up look at Lindey's disastrous Snow Boarding maneuver here.

 

... and let's break down our shame-faced test designer's bone-headed move here. Can you notice what is wrong in with his choices of values?

There are nine different parameters in the mix here. Of those, two ("First Name" and "Second Name"), are the least important to our current objective of looking for problems in the underwriting engine calculations. And yet...

He's added ten values to each of them. Oops! Whenever you are putting together a pairwise (or 2-way) test plan, the number of tests required will never be lower than the product of the number of parameter values from the two parameters that have the highest number of values. In plain English, that high-falutin' previous sentence means: when you have a plan with 7 parameters that have a maximum of 4 values each, "10 largely irrelevant values X 10 largely irrelevant values = you're a big fat idiot" because you'll create a test plan that has 100 test cases (as compared to a test plan that could have covered the System Under Test more effectively with fewer than a quarter of the tests you've just created).

 

For more information on pairwise and combinatorial testing, I would recommend the following sources:

 

If you are attempting to use pairwise and/or combinatorial testing methods and running into questions, I'd sincerely like to help. Please consider one or more of the following:

 

Thank you,

Justin Hunter

 

*The manufacturing industry followed a similar pattern of adoption to similar methods that consistently delivered dramatic efficiency and effectiveness benefits. It took decades before multi-variate Design of Experiments methods were widely adopted by manufacturers even long after the benefits were proven to be dramatic and repeatable to anyone who would look at the clear, unambiguous, objectively-measurable evidence. Today, it is impossible to find a Fortune 500 manufacturing firm that does not regularly use multi-variate Design of Experiments in their manufacturing processes. One day it will be the same for Fortune 500 firms with respect to their adoption of multi-variate Design of Experiments methods of software testing.

By: Justin Hunter on Jan 29, 2013

Categories: Combinatorial Testing, Pairwise Software Testing, Software Testing, Software Testing Efficiency, Testing Strategies

Screen-Shot-2013-01-15-at-7.28.42-PM-250x300

 

Elisabeth Hendrickson's new book, Explore It!, will begin shipping from Amazon in a week. If you're interested in software testing, I highly recommend it without reservation. It's outstanding. It is currently available for sale on Prag Prog and for pre-order on Amazon. The paper version will be published on January 22nd. Since Amazon apparently doesn't allow people to review books until they officially go on sale, I can't yet post my review on Amazon, but here, one week early, is my glowing review:

Explore It! is one of the very best software testing books ever written. It is packed with great ideas and Elisabeth Hendrickson's writing style makes it very enjoyable to read. Elisabeth Hendrickson has a well-deserved global reputation in the software testing community as someone who has the enviable ability to clearly communicate highly-practical, well-thought-out ideas. Tens of thousands of software testers who have already read her "Test Heuristics Cheat Sheet" no doubt already appreciate her uncanny ability to clearly convey an impressive number of actionable ideas with a minimal use of ink and paper. A pdf download of the cheat sheet is available here. If you're impressed by how much useful stuff Hendrickson can pack into one double-sided sheet of paper, you should see what she can do with 160 pages.

Testers at all levels of experience will benefit from this book. Like the best TED talks, Explore It! contains advanced ideas, yet those ideas are presented in way that is both interesting and accessible to a broad audience. Beginning testers will benefit from learning about the fundamentals of Exploratory Testing (an important and incredibly useful approach to software testing that is increasingly getting the respect it deserves). Experienced testers will benefit from practical insights, frameworks for thinking about challenges that bedevil all of us, and Hendrickson's unmatched ability to clearly explain important aspects of testing (including her superb explanations of test design principles).

Chapter 4 "Find Interesting Variations" in itself is worth far more than the price of the book. It is my favorite chapter in any software testing book I have ever read. A large part of the reason I have so much appreciation for this chapter is that I have personally been teaching software testers how to create interesting variations in their testing efforts for the last six years and know from experience that it can be a challenging topic to explain. I was excited to see how thoroughly Hendrickson covered this important topic because relatively few software testing books address it. I was humbled by how effortlessly Hendrickson seemed to make this complex topic easy to understand.

Buy it. You won't regret it. I'm buying multiple copies to give to developers and testers at my company as well as multiple copies to give to our clients.

  • Justin Hunter

By: Justin Hunter on Jan 15, 2013

Categories: Book Review, Exploratory Testing, Software Testing, Testing Checklists, Testing Strategies, Training

I'll be talking at QAI's 12th Annual International Software Testing Conference on Dec 6th in Bangalore, India.

Topic: Conquering the Single Largest Challenge Facing Testers Today

"There's too much to test and not enough time to test it all." According to a recent survey conducted by Robert Sabourin, this is the single largest challenge facing test managers today. And this challenge clearly won't go away any time soon. Software is becoming increasingly complex and time pressures put on testing teams are becoming ever more extreme.

To survive and thrive as testers, we need to find ways to learn more in the limited time we have. This talk addresses:

  • Proven test design methods to learn as much as possible about a System Under Test as quickly as possible

  • How these methods were originally developed and refined in other (non-IT) industries over the last 80 years

    • How the recent Apple Maps disaster could have been easily avoided by implementing these methods
  • Real world case studies: these methods sound nice on paper, but do they actually work?

  • Reasons why these methods are being used at more than 100 Fortune 500 firms today

    • What does the future hold?

 

Attendees will learn about valuable testing strategies that are being used today by more than 100 Fortune 500 firms. In particular, attendees will hear about:

  • Practical test design approaches that they can begin implementing after the conference at their firms to:

    • Reduce the amount of time spent selecting and documenting test scripts
    • Reduce the amount of tests needed for execution by creating unusually powerful tests
    • Increase the thoroughness of software test suites

 

Related: Efficient and Effective Test Design - A Fun Presentation on a Powerful Software Test Design Approach - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

By: Justin Hunter on Nov 22, 2012

Categories: Combinatorial Testing, Efficiency, Pairwise Testing, Software Testing Efficiency, Software Testing Presentations

We have thousands of users and we're growing by hundreds of new users a month. We don't have a way to accurately breakdown our users into Agile vs Waterfall users but I know from conversations with Hexawise users that many users are using Hexawise with great results on Agile projects.

Agile projects involve short sprints in which new features are added to existing functionality. These sprints are often a couple weeks long. In such situations, it is important to be able to create tests quickly. Hexawise is extremely good at accomplishing exactly that.

Let me explain by way of an example. A Hexawise user named Sandeep asked how Hexawise could be used to create tests for an insurance ratings engine. I was able to create a sample set of 37 2-way tests within 20 minutes.

Now lets imagine that a sprint creates 2 new features that we'd like to test in combination with the existing test inputs in the existing plan. Let's assume "Feature 1" is a new ability of the application to handle input coming in from three sources: (1) web, (2) agent, and (3) call center. Let's assume "Feature 2" is a new ability of the application to handle Group Discounts. The Group Discount parameter will have 3 categories of values as well: (N/A) for people who aren't able to claim them, "Private company," and "Government / Public."

Added the new features, and created another, completely different set of tests that achieve 100% 2-way coverage. Each value in the new feature is tested in at least one test with every other value in the plan. It took less than 4 minutes to create this new set of tests incorporating the new features. It is an extremely attractive benefit of using Hexawise that is excellent for both Waterfall projects (which inevitably have late-breaking requirements changes) as well as Agile projects (where late-breaking feature additions are expected to happen as part of the process).

You want to keep testing documentation light, you want to minimize the amount of time you spend in selecting and documenting tests, and it is helpful to achieve as much coverage in as little time as possible and minimize the risk that you'll accidentally forget to test things that you meant to test (which can be easier to do if you have light test cases with enough detail in them to maximize your odds of testing the important stuff without accidentally omitting tests). Hexawise helps you achieve all of thise objectives. The example shows, as clearly and concisely as I'm able to explain it, how you can use Hexawise to create a small set of powerful tests within four minutes (from 1.7 trillion possible tests) that are well suited to test new features developed in an Agile project iteration.

 

Related: What is Agile? What is not Agile? - Why isn't Software Testing Performed as Efficiently and Effecively as it could be? - Cem Kaner: Testing Checklists = Good / Testing Scripts = Bad?

By: Justin Hunter on Oct 19, 2012

Categories: Agile, Hexawise tips, Pairwise Software Testing, Software Testing

I came across Elisabeth Hendrickson's "Test Heuristics Cheat Sheet" yesterday and developed some pairwise testing (AKA 2-way combinatorial) test cases using many of the good ideas contained in it. I would highly recommend it, I'd recommend you send it (or email a link to this blog post) to everyone on your QA team.

As an indication that the Hendrikson's Test Heuristics Cheat Sheet works well to uncover defects,

  • I wanted to create a set of pairwise tests that could be broadly applicable to test thousands of different applications, I incorporated many ideas from the Test Heuristics Cheat Sheet.

  • I intend to use those inputs to test our test design tool, Hexawise.

  • The way Hexawise works is that users enter "things they want to test" into Hexawise on the first of three screens, the "Define Inputs" screen, then click on "Create Tests." Hexawise then uses a scientific approach to maximizing coverage of the combinations of all the "stuff to be tested" in the fewest possible number of tests. This scientific approach is based on the >40 years of Design of Experiments lessons and includes both pairwise / AllPairs methods as well as more thorough 3-way, 4-way, 5-way and 6-way tests (as well

  • Ironically, even before starting to execute the test conditions suggested by Hexawise, I discovered that the special characters that I had input into the "Define Inputs" screen (which I took from the Test Heuristics Cheat Sheet) triggered a previously unidentified defect in Hexawise itself.

  • The fact that it was triggered so quickly in an application that has been live for a year and used thousands of times is a strong indication that using checklists and cheat sheets can be a great way to efficiently find defects.

Why is using checklists to guide your testing often such an efficient and effective way of finding defects? Here's my top ten list:

  1. The "bad ideas" have already been weeded out.
    1. The ideas on the list have found enough defects to make the author of the checklist think there is value in testing the particular idea.
    2. If you've got a checklist or "cheat sheet" put together by someone as thoughtful and experienced as the Bachs, Bolton, and Hendrickson, you're getting a highly-condensed executive summary version of many of their valuable insights.
    3. All testers go through many, many, "I wonder what would happen if we did this or considered that?" scenarios.
    4. The checklists referenced above represent expertise culled from thousands of testing projects.

  2. Checklists are directly actionable. You can apply them in almost no time at all.
  3. They work well. See Cem Kaner's slides on the Value of Checklists (11 Mb pdf file).
  4. They can easily evolve into some of your most powerful test artifacts.
    • Start with the lists above. See if each of the ideas for tests trigger defects in your Systems Under Test.
    • Find a lot of defects from certain test ideas? Create your own checklist of ideas that worked and iterate them over time.. Consider expanding upon the checklist items and concepts that do bear fruit.
    • Don't ever find defects from certain of the test ideas? Consider dropping those items from the checklists if they don't bear fruit for you (or put tests for those ideas at the back of your lists and only include tests for them if you have extra time).

  5. Checklists include useful, defect-triggering ideas that you may not have thought of on your own.
  6. They're free.
    • No software or books to buy.
    • No courses or conferences to attend.

  7. Using checklists mitigate the risk that you will forget to test for things that you know you should be testing for (but could well forget to test for in any specific instance).
    • As humans, we're naturally forgetful as a species despite our best efforts.
    • Checklists are widely used with good results by doctors, lawyers, pilots, software testers, and people going to grocery stores to minimize the effects of these shortcomings.

  8. Software testing checklists are an efficient way to communicate actionable information.
  9. Software testing checklists are widely applicable to all kinds of software testing.
    • Checklists can be used in creating Unit Tests, Assembly Tests, Product Tests, System Tests, Functional Tests, Load Tests, Performance Tests, User Acceptance Tests, etc.
    • Checklists can be used by Exploratory Testers and "script-everything-in-advance" test-case-centric testers.
    • Checklists can be used in Agile projects as well as Waterfall projects.

  10. Software testing checklists can be easily used in pairwise and combinatorial testing.
    • Using elements from the checklists in a pairwise test will have the added benefit that not only will you test for every one of the testing ideas on the checklist (e.g., XXX) but also, you can easily test for every idea on the checklist **in combination with** every other test idea on the checklist in at least one test case.

    By: Justin Hunter on May 3, 2012

    Categories: Checklists, Software Testing, Testing Checklists, Uncategorized

jobgraph

 

This chart shows that the number of job listings posted for testers has remained essentially flat for testers over the last six years. During this same period, the number of job listings for developers has increased almost 50%. If your company's hiring has mirrored this chart's trends, and your software testers are increasingly feeling that there's not enough time in the day to test everything that's being thrown at them, your testers might have a point.

By: Justin Hunter on Feb 8, 2011

Categories: Software Testing

Combinatorial Software Test Design - Beyond Pairwise Testing

 

I put this together to explain combinatorial software test design methods in an accessible manner. I hope you enjoy it and that, if you do, that you'll consider trying to create test cases for your next testing project (whether you choose our Hexawise test case generator or some other test design tool).

 

Where I'm Coming From

As those of you know who read my posts, read my articles, and/or have attended my testing conference presentations, I am a passionate proponent of these approaches to software test design that maximize variation from test case to test case and minimize repetition. It's not much of an exaggeration to say I hardly write or talk publicly about any other software testing-related topics. My own consistent experiences and formal studies indicate that pairwise, orthogonal array-based, and combinatorial test design approaches often lead to a doubling of tester productivity (as measured in defects found per tester hour) as compared to the far more prevalent practice in the software testing industry of selecting and documenting test cases by hand. How is it possible that this approach generates such a dramatic increase in productivity? What is so different between the manually-selected test cases and the pair-wise or combinatorial testing cases? Why isn't this test design technique far more broadly adopted than it is?

 

A Common Challenge to Understanding: Complicated, Wonky Explanation

My suspicion is that a significant reason that combinatorial software testing methods are not much more widely adopted is that many of the articles describing it are simply too complex and/or too abstract for many testers to understand and apply. Such articles say things like:

A. Mathematical Model

 

A pairwise test suite is a t-way interaction test suite where t = 2. A t-way interaction test suite is a mathematical structure, called a covering array.

Definition 1 A covering array, CA(N; t, k, |v|), is an N × k array from a set, v, of values (symbols) such that every N × t subarray contains all tuples of size t (t-tuples) from the |v| values at least once [8].

The strength of a covering array is t, which defines, for example, 2-way (pairwise) or 3-way interaction test suite. The k columns of this array are called factors, where each factor has |v| values. In general, most software systems do not have the same number of values for each factor. A more general structure can be defined that allows variability of |v|.

Definition 2 A mixed level covering array, MCA (N; t, k, (|v1|,|v2|,..., |vk|)), is an N × k array on |v| values, where

| v |␣ ␣k | vi | , with the following properties: (1) Each i␣1

column i (1 ␣ i ␣ k) contains only elements from a set Si of size |vi|. (2) The rows of each N × t subarray cover all t-tuples of values from the t columns at least once.

  • "Construct Pairwise Test Suites Based on the Bak-Sneppen Model of Biological Evolution" World Academy of Science, Engineering and Technology 59 2009 - Jianjun Yuan, Changjun Jiang

 

If you're a typical software tester, even one motivated to try new methods to improve your skills, you could be forgiven for not mustering up the enthusiasm to read such articles. The relevancy, the power, and the applicability of combinatorial testing - not to mention that this test design method can often double your software testing efficiency and increase the thoroughness of your software testing - all tend to get lost in the abstract, academic, wonky explanations that are typically used to describe combinatorial testing. Unfortunately for pragmatic, action-oriented software testing practitioners, many of the readily accessible articles on pairwise testing and combinatorial testing tend to be on the wonky end of the spectrum; an exception to that general rule are the good, practitioner-oriented introductory articles available at combinatorialtesting.com.

 

A Different Approach to Explaining Combinatorial Testing and Pairwise Testing

In the photograph-rich, numbers-light, presentation embedded above, I've tried to explain what combinatorial testing is all about without the wonky-ness. The benefits from structured variation and from using combinatorial test design is, in my view, wildly under-appreciated. It has the following extremely important benefits:

  • Less repetition from test case to test case

    • In the context of discussing testing's "pesticide paradox" James Bach, I believe, used the analogy that following in someone's footsteps is a very good way to survive traversing through a mine field but a generally lousy way to find software defects efficiently.
    • Maximizing variation from test case to test case, as a general rule, is an absolutely spectacular way to find defects quickly.
    • There are thousands, if not trillions of relevant combinations to select from when identifying test cases to execute; computer algorithms will be able to solve the problem of "how can maximum variation be achieved?" far better than human brains can.
  • More coverage of combinations of test inputs

    • Most of the time, since awareness of pairwise and combinatorial testing methods remain low in the software testing community, combining all possible pairs of values in at least one test case is not even a conscious goal of testers.
    • Even if this were a goal of their test design strategy, testers would have a tremendous challenge in trying to achieve such a goal: with hundreds, thousands or tens of thousands of targeted combinations to cover, losing track of a significant number of them and/or forgetting to include them in software tests is virtually a foregone conclusion unless a test case generator is used.
    • More thorough coverage leads to more defects being found.
  • Efficiency (Testers can "turn the coverage dial" to achieve maximum efficiency with a minimal number of tests)

    • The efficiency and effectiveness benefits of pairwise testing have been demonstrated in testing projects every major industry.
    • I wanted to prominently include the message that testers using test case generators have the option to dramatically increase the testing thoroughness levels of the tests they generate because it is a topic that often gets ignored in introductions to pairwise testing case studies and introductions
  • Thoroughness - (Testers can also "turn the coverage dial" to achieve maximum thoroughness if that is their goal)

    • Too often, tester's view pairwise as a technique that focuses on a very small number of curiously strong tests; that is only part of the story.
    • This can lead to the /false/ impression that combinatorial testing methods are inappropriate where high levels of testing thoroughness are required.
    • You can create very different sets of tests that are as thorough as possible (given your understanding of what you are testing) no matter whether you have 1 hour to execute tests or one month to test.

 

Other Recommended Sources of Information on Pairwise and Combinatorial Testing:

By: Justin Hunter on Oct 7, 2010

Categories: Combinatorial Software Testing, Combinatorial Testing, Design of Experiments, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Recommended Tool, Testing Strategies, Uncategorized

I can be too verbose with some of my posts. This will be quick.

I recommend that you read this. It is the Exploratory Testing Dynamics document, a tightly condensed list of useful testing heuristics authored by three of the most thoughtful and experienced software testers alive today: James Bach, Michael Bolton, and Jon Bach. Using it will help you improve your software testing capabilities. http://www.satisfice.com/blog/wp-content/uploads/2009/10/et-dynamics22.pdf

Told you it'd be brief. Go. Now. Read.

By: Justin Hunter on Jul 13, 2010

Categories: Exploratory Testing, Software Testing

Context is Important in Usability Testing

As Adam Goucher recently pointed out, it is important to keep in mind WHY you are testing. Testing teams working on similar projects will have different priorities that will impact how much time they test, what they look for, and what test design methods they use. (Kaner and Bach provide some great specific examples that underscore this point here). In short, the context in which you're testing should impact how you test.

The same maxim holds true when you're conducting usability testing. Considering the context is important is well, both the context of the users of the application and the context of the application itself vis a vis other similar products. Important considerations can include:

  1. What problem does the application solve for the user?

  2. What does the application you're testing aspire to be as compared to competing applications?

  3. Who is the target audience of the application? What differentiating features does the application have?

  4. What is the "personality" of the application?

  5. What benefits and values do specific segments of target users prioritize?

These questions are all important when you analyze a web site with an eye on usability. I would recommend combining both a "checklist" approach (e.g., Jakob Nielsen's well-known Ten Usability Heuristics) with an approach that takes context-specific considerations (such as the 5 questions listed above) into account.

 

The Context of a User Group I'm Familiar with: the Hexawise Team

As of the end of June, 2010, our website leaves a great deal to be desired, so say the least. Hexawise.com consists mainly of a single landing page with anemic content that we threw together a year ago thinking that we'd "turn it into a real site" when we got around to it. We then proceeded to focus all of our development efforts on the Hexawise tool itself as opposed to our website (which we've let fester). Apologies if you've visited our site and wanted to know more details about what our test design tool does and how it complements test management tools. To date, we haven't provided as much information as we should have.

We've kicked off a project now to right this wrong. To do so, we're drafting up new content and organizing our thoughts about how to present it to visitors. Our needs are relatively simple. We want to create a set of simple wireframes that will allow us to quickly experiment with a few design options, gather feedback from friends and target users. For us, ease of use is key. Quickly being able to use the tool (without needing to read through a user guide) is critical. Ability to use the tool without reading through user guides is a must. We also value a tool's ability to make it easy to collaborate with one another easily.

With that as background, what follows are some quick comments on a couple wireframing tools I've recently explored in the context of our preferences and values. Wireframing is the practice of creating a skeletal visual interface for software. It is used for the the purposes of prototyping, soliciting early user/client feedback. It comes before the more time consuming phases of design. Two popular wireframe creation tools are Balsamiq and Hotgloo. Both are flash applications. Balsamiq is a desktop app. Hotgloo is a SaaS tool used over the internet.

 

Balsamiq and HotGloo

The first thing that strikes me about Balsamiq is the rich library of UX elements neatly organized and accessible by category or through a quick add search box. Everything works as it should: the drag, drop, click and type interface follows the principle of least astonishment. Fortunately, ease of use doesn't preclude speed: modifying the content and structure of UX elements is text-based versus form-based - blending in a touch of UNIX command line efficiency into otherwise graphical interface. UNIX and IRC users will feel right at home.

HotGloo is a very promising wireframing tool. They have clearly taken a page from the 37 Signals product development playbook. They have made a tool with a smaller set of features that is very intuitive to use. They've avoided the potential risk of "feature bloat" by having fewer bells and whistles. Where I think they add value: as a SaaS tool, HotGloo is exceptionally good at allowing multiple members on a team to collaborate on iterative designs. Whereas Balsamiq uses traditional files, HotGloo is accessible from anywhere. HotGloo enables multiple users to chat and view mockups in real time. Only one user can make changes at a time. Feedback is very easy to give and I found their support to be exceptionally responsive.

HotGloo is easy to learn for the first time, but my designer felt frustrated how much time he had to spend tweaking little things (like changing the names and links of a tabbed window element). The element controller pop-ups got in the way of work and he found myself frequently dragging them away. Hotgloo also takes a more minimalist approach than Basalmiq with UX elements with respect to features. Whether this is a strength or a weakness to users is a matter of personal preference. The 37 Signals camp (which I am highly sympathetic to) argues that is often preferable to have fewer, easier-to-use features since the vast majority of users will not want or need too many bells and whistles. Our designer felt that Balsamiq's feature set fit his needs better. As a "meddlesome manager" who wants to provide regular input into the content for version 2.0 of our site, feature-richness is less important to me than the collaborative ability.

 

Usability Considerations I Shared with the Hotgloo Team

20100630-ded769wwe9a5teycpej4t7jny9

20100630-d1gfj7nyjr4naaffxffus9dmw2

 

Balsamiq

20100705-n7rymxeb8yj3dfddt24itnr2ir

 

Balsamiq has a couple usability features that make it fun to use. A case in point is how you insert an image. Balsamiq gives you three choices, the third of which is really a nice touch: You can 1. Upload a file 2. Use a photo on the web or 3. Perform a flickr search right there and then without ever leaving comfort of the Balsamiq window. In my book, that kind of thoughtful workflow integration is what makes a good product great.

 

"Postscript" - Good Karma and an Open Invitation

20100705-g1cjw9ab78ji6nqcj61uch5aau 20100705-kt8qxwjes9xabycnpm97djjtch 20100705-x3gauxcqahddtum8aj75kbfjw8

 

As a post-script of sorts, after sending 5 UX suggestions (including the 2 above) to the HotGloo team last week, I received 5 outstanding UX suggestions for our Hexawise tool this week - out of the blue - from Janesh Kodikara, a new Hexawise user based in Sri Lanka. In addition, the HotGloo team provided 5 excellent UX suggestions for improving our tool as well. Taken together, they are some of the best suggestions we've had to date. If anyone reading this would be willing to share your usability suggestions with us, I can assure you, we're extremely interested in hearing your ideas.

By: Justin Hunter on Jul 5, 2010

Categories: Context-Driven Testing, Pairwise Software Testing, Uncategorized, User Experience, User Interface

A friend passed me this set of recent tweets from Wil Shipley, a Mac developer with 11,743 followers on Twitter as of today. Wil recently encountered the familiar problem of what to do when you've got more software tests to run than you can realistically execute.

 

20100623-nixnbwu9urxaufu6hjt2143j3h

I love that. Who can't relate?

Now if only there were a good, quick way to reduce the number of tests from over a billion to a smaller, much more manageable set of tests that were "Altoid-like" in their curious strength. :) I rarely use this blog for shameless plugs of our test case generating tool, but I can't help myself here. The opening is just too inviting. So here goes:

 

"Wil,

There's an app for that... See www.hexawise.com for Hexawise, a "pairwise software test case generating tool on steroids." It eats problems like the one you encountered for breakfast. Hexawise winnows bazillions of possible test cases down in the blink of an eye to small, manageable sets of test cases that are carefully constructed to maximize coverage in the smallest amount of tests, with flexibility to adjust the solutions based upon the execution time you have available. In addition to generating pairwise testing solutions, Hexawise also generates more thorough applied statistics-based "combinatorial software testing" solutions that include tests for, say, all possible 6-way combinations of test inputs.

Where your Mac cops an attitude and tells you "Bitch, I ain't even allocating 1 billion integers to hold your results" and showers you with taunting derisive sneers, head-waggling and snaps all carefully choreographed to let you know where you stand, Hexawise, in contrast, would helpfully tell you: "Only 1 billion total possibilities to select tests from? Pfft! Child's play. Want to start testing the 100 or so most powerful tests? Want to execute an extremely thorough set of 10,000 tests? Want to select a thoroughness setting in the middle? Your wish is my command, sir. You tell me approximately how many tests you want to run and the test inputs you want to include, and I'll calculate the most powerful set of tests you can execute (based on proven applied statistics-based Design of Experiments methods) before you can say "I'm Wil Shipley and I like my TED Conference swag."

More info at:
http://hexawise.tv/intro/
or
https://hexawise.com/Hexawise_Introduction.pdf
free trials at:
http://hexawise.com/signup

By: Justin Hunter on Jun 23, 2010

Categories: Combinatorial Software Testing, Combinatorial Testing, Interesting People , Pairwise Software Testing, Pairwise Testing, Recommended Tool, Software Testing

There are good reasons James Bach is so well known among the testing community and constantly invited to give keynote presentations around the globe at software testing conferences. He's passionate about testing and educating testers; he's a gifted, energetic, and entertaining speaker with a great sense of humor; and he takes joy in rattling his saber and attacking well-established institutions and schools of thought that he disagrees with. He doesn't take kindly to people who make inflated claims of benefits that would materialize "if only you'd perform testing in XYZ way or with ABC tool" given that (a) he can always seem to find exceptions to such claims, (b) he doesn't shy away from confrontation, and (c) he (rightly, in my view) thinks that such benefits statements tend to discount the importance of critical thinking skills being used by testers and other important context-specific considerations.

Leave it up to James to create a list of 13 questions that would be great to ask the next software testing tool vendor who shows up to pitch his problem-solving product. In his blog post titled "The Essence of Heuristics," he posed this exact set of questions in a slightly different context, but as a software testing tool vendor myself, they really hit home. They are:

 

  1. Do they teach you how to tell if it’s working?
  2. Do they teach you how to tell if it’s going wrong?
  3. Do they teach you heuristics for stopping?
  4. Do they teach you heuristics for knowing when to apply it?
  5. Do they compare it to alternative heuristics?
  6. Do they show you why it works?
  7. Do they help you understand when it probably works best?
  8. Do they help you know how to re-design it, if needed?
  9. Do they let you own it?
  10. Do they ask you to practice it?
  11. Do they tell stories about how it has failed?
  12. Do they listen to you when you question or challenge it?
  13. Do they praise you for questioning and challenging it?

 

[Side note: Apparently I wasn't the only one who thought of Hexawise and pairwise / combinatorial test design approaches when they saw these 13 questions. I was amused that after I drafted this post, I saw Jared Quinert's / @xflibble's tweet just now:]

20100601-br4ud66pcc7f79q1ywgbat74jw

Where do I come down on each of James' 13 questions with respect to people I talk to about our test design tool, Hexawise, and the types of benefits and the size of benefits it typically delivers? Quite simply, "Yes" to all 13. I enjoy talking about exactly the kinds of questions that James raised in his list. In fact, when I sought out James to ask him questions at a conference in Boston earlier this year, it was because I wanted his perspective on many of the points above, particularly #11: (hearing stories about how James has seen pairwise and combinatorial approaches to test design fail), and #7 (hearing his views on where it works best and where it would be difficult to apply it). I'll save my specific answers to another post, but I am serious about wanting to share my thoughts on them; time constraints are holding me back today. I gave a speech at the ASQ World Conference on Quality Improvement in St. Louis last week though that addressed many, but not all, of James' questions.

I'm not your typical software tool vendor. Basically, my natural instincts are all wrong for sales. I agree with the premise that "a fool with a tool is still a fool"; when talking to target clients and/or potential partners, I'm inclined to point out deficiencies, limitations, and various things that could go wrong; I'm more of an introvert than an extrovert, etc. Not exactly the typical characteristics of a successful salesman... Having said that, I believe that we've built a very good tool that helps enable dramatic efficiency and thoroughness benefits in many testing situations but our tool, along with the pairwise and combinatorial test design approaches that Hexawise enables both have their limitations. It is primarily by talking to software testers about their positive and negative experiences that our company is able to improve our tool, enhance our training, and provide honest, pragmatic guidance to users about where and how to use our tool (and where and how not to).

Tool vendors who defend their tools (and/or the approaches by which their tools helps users solve problems) as magical, silver bullet solutions are being both foolish and dishonest. Tool vendors who choose not to engage in serious, honest and open discussions with users about the challenges that users have when applying their tools in different situations are being short-sighted. From my own experiences, I can say that talking about the 13 topics raised by James have been invaluable.

By: Justin Hunter on Jun 1, 2010

Categories: Combinatorial Testing, Design of Experiments, Hexawise test case generating tool, Pairwise Testing, Software Testing, Software Testing Efficiency, Uncategorized

While I wouldn't describe myself as a blanket "Microsoft hater," I have developed a no-holds-barred hatred of IE6. Our Hexawise test generation tool doesn't support it (and we have no plans to unless the surprisingly high number of our target clients in financial service firms, government agencies and buggy-whip manufacturing who are requesting IE6 support make it too financially painful for us to continue to stand on principle).

As an unapologetic IE6-hater, it made me smile to watch and listen to Scott Ward sing his witty and original song "IE is being mean to me." Enjoy.

 

 

A tip of the hat to Chris McMahon, an impressively knowledgeable software tester who played the bass professionally in bands for years, who alerted me to this gem.

By: Justin Hunter on May 14, 2010

Categories: Browser Testing, Product Management