Kathleen Poulsen of Fidelity Investments gave a presentation at STAREAST 2017 sharing her experience using Hexawise to improve their software testing performance. Watch a 10 minute video with highlights of that talk:

We didn't really have what I'd call a scientific methodology to approaching the tests...

Our regression test suites were continuously expanding... We found there was a repition of tests.

We had 3 different projects that I will talk about that I feel like combinatorial or pairwise testing was the key to answering all of those problems.

Hexawise allows you to harness the power of combinatorial software testing with test plans designed to provide thourogh testing of interaction impacts on the software being testing. Hexawise provides more coverage with fewer tests.

All the teams that are using Hexawise can use that same file, they can talk to each other. [Another] thing I liked about Hexawise was the coverage chart... I go back to my business partners and say I am not running these tests. If they are important to you I add them back in with the click of a button. I love that... it was a game changer for me.

Using the Hexawise exporting options

the tests that we produced were converted into the given then when type scenarios automatically and when they are exported into excel you can use them to drive the Sellenium test automation framework. No additional work from us involved.

Using Hexawise's ability to create highly optimized test plans Fidelity was able to greatly reduce the number of tests while also greatly improving coverage.

We were able to reduce from 12,000 tests down to 600.

This type of result sounds amazing, and it is. But it is also what we find consisently from clients over and over. There are certain things people just cannot do well and designing test plans to cover incredible large numbers of interactions between test values and conditions is one of those things. Using highly optimized alogorithms to create test plans to cover these interactions in order to reliably create software customers will love is key. This also frees people to do what they do best.

Kathleen also discussed the significant improvement in communication within Fidelity that was brought about by using Hexawise.

The common language has become the test plan that comes out of Hexawise today.

Improving communiction is an area many organization see as important but finding concrete ways to achieve better communication is often difficult. We have designed Hexwise to aid the communication between stakeholders, including: software developers, software testers, product owners, help desk support staff and senior management.

The simplicity of this tool along with the way you can enter your parameters using the mind map tool, getting that coverage chart automatically out of it, having it export your data into a pretty commonly usable format - those are things that were teribly important to me. They gave me real value... I love that.

I can accomodate many differnt types of testing. We are testing at the class method level, at the services interface level, at the UI level...

Related: 84% of Software Defects Found in Production Could Have Been Found Using Pairwise Testing - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - 2 Minute Introduction to Hexawise Software Testing Solution

By: John Hunter on Jun 9, 2017

Categories: Combinatorial Software Testing, Combinatorial Testing, Efficiency, Hexawise, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Recommended Tool, Software Testing, Software Testing Presentations, Software Testing Efficiency, Testing Case Studies, User Experience

Hexawise has had another great year and we owe it to you, our users. Thank you! As a result of your input and word-of-mouth recommendations, in 2013, the Hexawise test design tool:

  • Added powerful new features,

  • Became even easier to use,

  • Introduced lots of new practical instructional content, and

  • Doubled usage again.

If you haven’t visited Hexawise lately, please login now to see all the improvements we've made (register for free).

Ease-of-Use Enhancements

Instructional Guides for Hexawise Features
We’ve added illustrated step-by-step instructions describing how to use Hexawise features.

Find them at help.hexawise.com. For our advanced features, like creating risk-based test plans, auto-generating complete scripts, and using Value Expansions, we’ve gone beyond “how?” to explain “why?” you would want to use these features.

Practical Test Design Tips
Want to see tips and tricks for creating unusually powerful tests? Want to learn about common mistakes (and how to avoid them)? Want to understand whether pairwise test design really works in practice? These topics and more can now be found at training.hexawise.com.

Frog-Powered Self-Paced Learning

frogs next

Want to become a Hexawise guru? Listen to the frogs. If you complete the achievements recommended by your friendly amphibian guides, you will level up from a Novice to Practitioner to Expert to Guru.

frogs become expert

You’ll complete two kinds of achievements on your way to guru-ness. To complete some achievements you only need to use certain Hexawise features. Completing the other achievements requires learning important test design concepts and demonstrating that you understand them. The frogs, ever-ready to guide you towards test design mastery, will greet you immediately upon logging into your account.

Powerful New Features

Recently-added features that will make you a more powerful and speedy test designer, include:

Coverage of Specific High-Priority Scenarios
You can now force specific scenarios to appear in the tests you generate using the Requirements feature.

Requirements Traceability
Requirements traceability is easier to manage than ever with the Requirements feature.

Generation of Detailed Test Scripts
The Auto-Scripting feature allows you to automatically transform sets of optimized test conditions into test scripts that contain detailed instructions in complete sentences.

Auto-Population of Expected Results in Test Scripts
If you want to, you can even automatically generate rules-based Expected Results to appear as part of your test steps by using the Expected Results feature.

To find out more about these features Hexawise added in 2013, please check out these cool slides: "Powerful New Hexawise Features".


Public Recognition and Rapid Growth

Kind Words
As a five-year old company working as hard as we can to make Hexawise the best damn tool it can be, hearing input from you keeps us motivated and headed in the right direction. Once a week or so, we hear users say nice things about our tool. Here are some of the nice things you guys said about Hexawise this past year:


“Working coaching session with customer today. Huge data/config matrix was making them weep. Stunned silence when I showed them @Hexawise :)”

-Jim Holmes (@aJimHolmes)


“That would be @Hexawise & combinatorial testing to the rescue once again. #Thanks”

-Vernon Richards (@TesterFromLeic)


Freaking awesome visualisation of test data coverage. Kind courtesy of @Hexawise at Moolya!”

-Moolya Testing (@moolyatesting)


“Using @Hexawise combinatorial scenarios for e-commerce basket conditions. Team suitably impressed by speed and breadth of analysis. #Win”

-Simon Knight (@sjpknight)


“Just discovered Hexawise today, brilliant tool for creating test cases based on coverage of many variables.”

-Stephen Blower (@badbud65)


“This changes everything.”

-Dan Caseley (@Fishbowler)


Using Hexawise is one of the highest ROI software development practices in the world.

-Results, paraphrased, of independent study by industry expert Capers Jones and colleagues.


Rapid Growth
Throughout 2013, Hexawise continued to be piloted and adopted at new companies week after week. Hexawise is currently being used to generate tests at:

More than 100 Fortune 500 firms More than 2,000 smaller firms

Hexawise office

New offices
Having moved into our new offices in October, the Hexawise team now gets together to do all our best stuff at this swanky new location in Chapel Hill, North Carolina.

What's Next?

Constant Improvements
We keep a public running tally of all of our enhancements. As you can see, we’re making improvements at a rate of more than once a week.

Want us to Add a New Feature?
If you have an additional feature, please let us know. We listen. There’s not much that makes us happier than finding ways to improve Hexawise.

Please Tell Your Friends
Our growth has been almost purely due to word-of-mouth recommendations from users like you. So if you find Hexawise to be helpful, please help spread the word. We appreciate it more than you know.

You can even let your friends in on a bit of a testing industry secret: while company-wide Hexawise licenses start at $50,000 per year, we allow the first five users to use fully-featured Hexawise accounts at no cost!

Thank you for all of your help and support this past year. And have a great 2014!

By: Justin Hunter on Jan 28, 2014

Categories: Hexawise test case generating tool, Recommended Tool


Design of Experiments in Software Testing - Pairwise and Combinatorial - Hexawise

Justin Hunter @Hexawise:


Removing inefficiency is good, sure, but it is not why Design of Experiments is so friggin' powerful. Saying DoE is interesting to know about because it can help identify and remove specific inefficiencies is a bit like saying Canada is a good country to visit because you can sometimes find a good cup of coffee there. To my mind, saying DoE is primarily about removing inefficiency misses the main point.

Design of Experiments is so powerful because it allows practitioners to predictably, systematically, and consistently find out more useful, actionable information in much less time than they would otherwise take to obtain this information (if they could find it at all with their less-structured approaches).

In manufacturing circles (e.g., when engineers produce new prototypes), DoE's ability to do this is no longer questioned. This is because leaders like George Box taught people in industry how to apply DoE and they gathered conclusive evidence that DoE allowed manufacturers to learn much faster through techniques like applying factorial designs. Box and other DoE experts (Taguchi, Montgomery, my dad, etc.) dealt with skeptical manufacturing engineers for four decades by showing them the facts and using DoE on the skeptics' own projects right under their noses. The evidence that DoE allows manufacturers to learn much faster (about a wide variety of learning goals) than the other methods they used prior to 1960 is incontrovertible.

In 2010, in the gradually maturing field of software testing, Design of Experiments-based methods of test case design has not caught on much at all yet. As an industry, it's adoption of DoE-based approaches is roughly where manufacturing was in 1960. Most software testers, even very good ones, don't know anything at all about how DoE can help them. Many other software testers have heard a bit about pairwise but mistakenly think pairwise and related, structured, DoE-based, test case selection method can't help them.

Even some of the best testers in the world who have written some of the most clearly-written and well-reasoned articles about pairwise approaches do not (in my view) seem to fully-understand: (a) how powerful the benefits are, (b) how often the approach can be applied / in how many diverse kinds of testing situations they can be utilized, and/or (c) how consistently the efficiency and effectiveness benefits are be generated when they are used properly. DoE methods, including pairwise and n-wise and mixed strength automatic test condition generation (made possible by tools like our Hexawise tool and also, to a great extent by James Bach's free AllPairs tool) allow software testers to learn much faster about critically important questions like: (1) where are the bugs?, (2) what is causing the bugs to appear?, (3) am I confident I have efficiently tested for a huge range of combinations of values in the System Under Test that might trigger defects? (4) am I succeeding in avoiding redundant repetition of steps in many test cases?, (5) how many bugs would be likely to find if we were to continue to run the next 100 tests?, etc.

In summary, the reason for the existence of Design of Experiments methods (whether we're talking about their applicability to testing software as efficiently and effectively as possible, or DoE methods' applicability to a huge variety of other objectives) - and, for that matter, the reason that they have been continuously refined and improved for 40+ years - is that DoE methods consistently and predictably allow users to learn actionable results as quickly as possible.


Related: Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed - Pairwise and Combinatorial Software Testing in Agile Projects - Video Highlight Reel of Hexawise

By: Justin Hunter on Feb 11, 2013

Categories: Design of Experiments, Multi-variate Testing, Recommended Tool, Testing Strategies

Combinatorial Software Test Design - Beyond Pairwise Testing


I put this together to explain combinatorial software test design methods in an accessible manner. I hope you enjoy it and that, if you do, that you'll consider trying to create test cases for your next testing project (whether you choose our Hexawise test case generator or some other test design tool).


Where I'm Coming From

As those of you know who read my posts, read my articles, and/or have attended my testing conference presentations, I am a passionate proponent of these approaches to software test design that maximize variation from test case to test case and minimize repetition. It's not much of an exaggeration to say I hardly write or talk publicly about any other software testing-related topics. My own consistent experiences and formal studies indicate that pairwise, orthogonal array-based, and combinatorial test design approaches often lead to a doubling of tester productivity (as measured in defects found per tester hour) as compared to the far more prevalent practice in the software testing industry of selecting and documenting test cases by hand. How is it possible that this approach generates such a dramatic increase in productivity? What is so different between the manually-selected test cases and the pair-wise or combinatorial testing cases? Why isn't this test design technique far more broadly adopted than it is?


A Common Challenge to Understanding: Complicated, Wonky Explanation

My suspicion is that a significant reason that combinatorial software testing methods are not much more widely adopted is that many of the articles describing it are simply too complex and/or too abstract for many testers to understand and apply. Such articles say things like:

A. Mathematical Model


A pairwise test suite is a t-way interaction test suite where t = 2. A t-way interaction test suite is a mathematical structure, called a covering array.

Definition 1 A covering array, CA(N; t, k, |v|), is an N × k array from a set, v, of values (symbols) such that every N × t subarray contains all tuples of size t (t-tuples) from the |v| values at least once [8].

The strength of a covering array is t, which defines, for example, 2-way (pairwise) or 3-way interaction test suite. The k columns of this array are called factors, where each factor has |v| values. In general, most software systems do not have the same number of values for each factor. A more general structure can be defined that allows variability of |v|.

Definition 2 A mixed level covering array, MCA (N; t, k, (|v1|,|v2|,..., |vk|)), is an N × k array on |v| values, where

| v |␣ ␣k | vi | , with the following properties: (1) Each i␣1

column i (1 ␣ i ␣ k) contains only elements from a set Si of size |vi|. (2) The rows of each N × t subarray cover all t-tuples of values from the t columns at least once.

  • "Construct Pairwise Test Suites Based on the Bak-Sneppen Model of Biological Evolution" World Academy of Science, Engineering and Technology 59 2009 - Jianjun Yuan, Changjun Jiang


If you're a typical software tester, even one motivated to try new methods to improve your skills, you could be forgiven for not mustering up the enthusiasm to read such articles. The relevancy, the power, and the applicability of combinatorial testing - not to mention that this test design method can often double your software testing efficiency and increase the thoroughness of your software testing - all tend to get lost in the abstract, academic, wonky explanations that are typically used to describe combinatorial testing. Unfortunately for pragmatic, action-oriented software testing practitioners, many of the readily accessible articles on pairwise testing and combinatorial testing tend to be on the wonky end of the spectrum; an exception to that general rule are the good, practitioner-oriented introductory articles available at combinatorialtesting.com.


A Different Approach to Explaining Combinatorial Testing and Pairwise Testing

In the photograph-rich, numbers-light, presentation embedded above, I've tried to explain what combinatorial testing is all about without the wonky-ness. The benefits from structured variation and from using combinatorial test design is, in my view, wildly under-appreciated. It has the following extremely important benefits:

  • Less repetition from test case to test case

    • In the context of discussing testing's "pesticide paradox" James Bach, I believe, used the analogy that following in someone's footsteps is a very good way to survive traversing through a mine field but a generally lousy way to find software defects efficiently.
    • Maximizing variation from test case to test case, as a general rule, is an absolutely spectacular way to find defects quickly.
    • There are thousands, if not trillions of relevant combinations to select from when identifying test cases to execute; computer algorithms will be able to solve the problem of "how can maximum variation be achieved?" far better than human brains can.
  • More coverage of combinations of test inputs

    • Most of the time, since awareness of pairwise and combinatorial testing methods remain low in the software testing community, combining all possible pairs of values in at least one test case is not even a conscious goal of testers.
    • Even if this were a goal of their test design strategy, testers would have a tremendous challenge in trying to achieve such a goal: with hundreds, thousands or tens of thousands of targeted combinations to cover, losing track of a significant number of them and/or forgetting to include them in software tests is virtually a foregone conclusion unless a test case generator is used.
    • More thorough coverage leads to more defects being found.
  • Efficiency (Testers can "turn the coverage dial" to achieve maximum efficiency with a minimal number of tests)

    • The efficiency and effectiveness benefits of pairwise testing have been demonstrated in testing projects every major industry.
    • I wanted to prominently include the message that testers using test case generators have the option to dramatically increase the testing thoroughness levels of the tests they generate because it is a topic that often gets ignored in introductions to pairwise testing case studies and introductions
  • Thoroughness - (Testers can also "turn the coverage dial" to achieve maximum thoroughness if that is their goal)

    • Too often, tester's view pairwise as a technique that focuses on a very small number of curiously strong tests; that is only part of the story.
    • This can lead to the /false/ impression that combinatorial testing methods are inappropriate where high levels of testing thoroughness are required.
    • You can create very different sets of tests that are as thorough as possible (given your understanding of what you are testing) no matter whether you have 1 hour to execute tests or one month to test.


Other Recommended Sources of Information on Pairwise and Combinatorial Testing:

By: Justin Hunter on Oct 7, 2010

Categories: Combinatorial Software Testing, Combinatorial Testing, Design of Experiments, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Recommended Tool, Testing Strategies, Uncategorized