Design of Experiments is about Learning ASAP and, in Software Testing, Finding Bugs ASAP

by Justin Hunter on February 10, 2013

image of twitter exchange

Design of Experiments in Software Testing - Pairwise and Combinatorial - Hexawise

Justin Hunter@Hexawise:

Removing inefficiency is good, sure, but it is not why Design of Experiments is so friggin’ powerful. Saying DoE is interesting to know about because it can help identify and remove specific inefficiencies is a bit like saying Canada is a good country to visit because you can sometimes find a good cup of coffee there. To my mind, saying DoE is primarily about removing inefficiency misses the main point.

Design of Experiments is so powerful because it allows practitioners to predictably, systematically, and consistently find out more useful, actionable information in much less time than they would otherwise take to obtain this information (if they could find it at all with their less-structured approaches).

In manufacturing circles (e.g., when engineers produce new prototypes), DoE’s ability to do this is no longer questioned. This is because leaders like George Box taught people in industry how to apply DoE and they gathered conclusive evidence that DoE allowed manufacturers to learn much faster through techniques like applying factorial designs. Box and other DoE experts (Taguchi, Montgomery, my dad, etc.) dealt with skeptical manufacturing engineers for four decades by showing them the facts and using DoE on the skeptics’ own projects right under their noses. The evidence that DoE allows manufacturers to learn much faster (about a wide variety of learning goals) than the other methods they used prior to 1960 is incontrovertible.

In 2010, in the gradually maturing field of software testing, Design of Experiments-based methods of test case design has not caught on much at all yet. As an industry, it’s adoption of DoE-based approaches is roughly where manufacturing was in 1960. Most software testers, even very good ones, don’t know anything at all about how DoE can help them. Many other software testers have heard a bit about pairwise but mistakenly think pairwise and related, structured, DoE-based, test case selection method can’t help them.

Even some of the best testers in the world who have written some of the most clearly-written and well-reasoned articles about pairwise approaches do not (in my view) seem to fully-understand: (a) how powerful the benefits are, (b) how often the approach can be applied / in how many diverse kinds of testing situations they can be utilized, and/or (c) how consistently the efficiency and effectiveness benefits are be generated when they are used properly. DoE methods, including pairwise and n-wise and mixed strength automatic test condition generation (made possible by tools like our Hexawise tool and also, to a great extent by James Bach’s free AllPairs tool) allow software testers to learn much faster about critically important questions like: (1) where are the bugs?, (2) what is causing the bugs to appear?, (3) am I confident I have efficiently tested for a huge range of combinations of values in the System Under Test that might trigger defects? (4) am I succeeding in avoiding redundant repetition of steps in many test cases?, (5) how many bugs would be likely to find if we were to continue to run the next 100 tests?, etc.

In summary, the reason for the existence of Design of Experiments methods (whether we’re talking about their applicability to testing software as efficiently and effectively as possible, or DoE methods’ applicability to a huge variety of other objectives) – and, for that matter, the reason that they have been continuously refined and improved for 40+ years – is that DoE methods consistently and predictably allow users to learn actionable results as quickly as possible.

Related: Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed – Pairwise and Combinatorial Software Testing in Agile Projects – Video Highlight Reel of Hexawise

{ 3 comments }

Hexawise went down today. I apologize. It was my fault.

by Justin Hunter on January 30, 2013

Hexawise was down for all of our users for most of the last hour just now. I apologize for the inconvenience caused. It was entirely my fault.  I wanted to give you, our users, an explanation.

When Gmail, Twitter, and Amazon’s EC2 service experience outages that impact large percentages of their users, executives at those firm sometimes need a couple days to analyze exactly what wrong and produce a report explaining the failure to their users. No complex or time-consuming analysis is required here. I messed up.

Here’s the honest but somewhat embarrassing truth about what happened:

  • When I founded Hexawise, 4 years ago today, I registered the domain name Hexawise.com personally.
  • Each year, we need to pay an annual fee of $10 to keep our domain name renewed.
  • Each year, I pay it.
  • Except this year.
  • This year, I took my eye off the ball when I went on vacation last week to experience the Maha Kumbh Mela (a massive Hindu pilgrimage that takes place at the Ganges river once every 12 years that has been referred to as “the world’s largest gathering of humanity”).
  • I missed an email from our domain registrar highlighting that payment was due as I was setting off to the Kumbh Mela.
  • Just returning from vacation today, I realized that the site was down, jumped on the phone with Sean Johnson, our CTO, who explained what had happened (and that he didn’t have access to the account I use to make the annual payment).  I promptly paid it and things returned to normal.

Kumbh Mela (image from Wikipedia commons)

Here’s what we’ll do differently in the future to ensure this won’t happen again:

  • I’ve updated credit card info with our domain name registrar and put this payment on auto-renew.
  • I’ve put calendar reminders on my calendar and our CFO’s for the coming years; we’ll both personally proactively confirm that it has been made.
  • I’ve put the email address of the domain registrar into my VIP email folder to ensure I won’t miss any future emails from them.

Again, I apologize for the inconvenience caused.  I’ve let down both you (users of Hexawise), and our engineers (who have kept Hexawise up >99.9% of the time even through hundreds of updates to Hexawise that they’ve put into production within the last four years).  They’re regularly going to great lengths to ensure the site is up so that you can generate efficient and effective software tests 24/7/365 using Hexawise.  You and our engineers deserve better than a CEO who dropped the ball and brought the site down today.  I hope you’ll excuse this lapse.  I won’t make this same mistake again.  We will continue to strive to keep site uptime as close to 100% as we possibly can.

- Justin Hunter

Founder and CEO of Hexawise

{ 1 comment }

How Not to Design Pairwise Software Tests

by Justin Hunter on January 29, 2013

I am passionate about pairwise software testing techniques. I have helped dozens of teams, for example, carefully measure the benefits that can be created when teams of testers adopt pairwise and related combinatorial testing approaches to identify the test cases they will execute (as compared to manual test case identification methods). What usually happens is that tester productivity doubles.  (See Combinatorial Software Testing – pdf download).

I believe these approaches will be much more widely adopted in a few years than they are now for the simple reason that they consistently deliver dramatic benefits to both the speed of software test design and the efficiency and thoroughness of software test execution.  As more teams try these methods for themselves, and measures the benefits they achieve with them, broader adoption seems highly likely to me.*

I see three main barriers to broader adoption by the testing community at large:

  1. The first barrier is that testers will not make an attempt to apply this method to their testing projects so they will never find out how effective it is.  The second is that ill-informed testers will try to apply the approach but do such a poor job at implementation that they do not generate benefits.
  2. The second barrier is that even testers who use the approach effectively a few times, will not realize how much more effective it is making them.  A dismissive thought process guilty of this might sound something like this: “Those 11 bugs I just found?  Yeah.  I found them because I’m a good tester; the fact that I happened to use pairwise tests just now?  That’s largely irrelevant. I’m sure I would have found them regardless.”)
  3. The third barrier is that testers unfamiliar with the basics of pairwise testing principles will design test cases without thinking about what they are doing, and achieve “garbage in / garbage out” results.  The benefits that would have been so easily achieved in the testing project – like Lindsey Jacobellis’ opportunity to win a gold medal for Snow Boarding – disappear in a groan-worthy moment of bone-headed stupidity.

This blog post addresses this third barrier.  When testers sabotage their own test plans with a poor choice of inputs, they may well blame the test design strategy rather than themselves, which would be unfortunate.  Here’s one common problem I see (exaggerated a bit in this example to make my point).

[click to continue…]

{ 2 comments }

Software Testing Carnival #2

by John Hunter on January 21, 2013

This is the second edition of our carnival that focuses on finding interesting and useful blog posts related to software testing.

  • Testing != test execution by Jeff Fry – “We often talk about testing as if it’s only test execution, yet often the most interesting, challenging, skill-intensive aspects of testing are in creating a mental model that helps us understand the problem space, designing tests to quickly and effectively answer key questions, analyzing what specifically the problem is, and communicating it effectively.”
  • Test Mercenaries by Mike Bland – “good testing practice goes a long way towards finding and killing a lot of bugs before they can grow more expensive, and possibly multiply. The bugs that manage to pass through a healthy testing regimen are usually only the really interesting ones. Few things are less worthy of our intellectual prowess than debugging a production crash to find a head-slappingly stupid bug that a straightforward unit or integration test could’ve caught.”
    • photo of a yellow chameleon moving between to green leaves

    Yellow Cameleon in South Africa by Justin Hunter

  • The Oracle Problem and the Teaching of Software Testing by Cem Kaner – “I’ve been emphasizing the oracle problem in my testing courses for about a dozen years. I see this as one of the defining problems of software testing: one of the reasons that skilled testing is a complex cognitive activity rather than a routine activity. Most of the time, I start my courses with a survey of the fundamental challenges of software testing, including an extended discussion of the oracle problem.”
  • The new V-Model by Kim Ming Leung – “We design by specifying the measurements before coding but not writing the test before coding. We write code and invite user feedback before writing automated testing for these measurements. Code quality is still guaranteed because first, measurement is the design and second, we code with testing in mind (i.e. write testable code).”

    • [click to continue…]

{ 1 comment }

Book Review of “Explore It!” Elisabeth Hendrickson’s Excellent New Book on Software Testing

by Justin Hunter on January 15, 2013

Elisabeth Hendrickson’s new book, Explore It!, will begin shipping from Amazon in a week. If you’re interested in software testing, I highly recommend it without reservation. It’s outstanding. It is currently available for sale on Prag Prog and for pre-order on Amazon. The paper version will be published on January 22nd. Since Amazon apparently doesn’t allow people to review books until they officially go on sale, I can’t yet post my review on Amazon, but here, one week early, is my glowing review:

Explore It! is one of the very best software testing books ever written. It is packed with great ideas and Elisabeth Hendrickson’s writing style makes it very enjoyable to read. Elisabeth Hendrickson has a well-deserved global reputation in the software testing community as someone who has the enviable ability to clearly communicate highly-practical, well-thought-out ideas. Tens of thousands of software testers who have already read her “Test Heuristics Cheat Sheet” no doubt already appreciate her uncanny ability to clearly convey an impressive number of actionable ideas with a minimal use of ink and paper. A pdf download of the cheat sheet is available here. If you’re impressed by how much useful stuff Hendrickson can pack into one double-sided sheet of paper, you should see what she can do with 160 pages.

Testers at all levels of experience will benefit from this book. Like the best TED talks, Explore It! contains advanced ideas, yet those ideas are presented in way that is both interesting and accessible to a broad audience. Beginning testers will benefit from learning about the fundamentals of Exploratory Testing (an important and incredibly useful approach to software testing that is increasingly getting the respect it deserves). Experienced testers will benefit from practical insights, frameworks for thinking about challenges that bedevil all of us, and Hendrickson’s unmatched ability to clearly explain important aspects of testing (including her superb explanations of test design principles).

Chapter 4 “Find Interesting Variations” in itself is worth far more than the price of the book. It is my favorite chapter in any software testing book I have ever read. A large part of the reason I have so much appreciation for this chapter is that I have personally been teaching software testers how to create interesting variations in their testing efforts for the last six years and know from experience that it can be a challenging topic to explain. I was excited to see how thoroughly Hendrickson covered this important topic because relatively few software testing books address it. I was humbled by how effortlessly Hendrickson seemed to make this complex topic easy to understand.

Buy it. You won’t regret it. I’m buying multiple copies to give to developers and testers at my company as well as multiple copies to give to our clients.

- Justin Hunter

{ 0 comments }

Getting Started with a Test Plan When Faced with a Combinatorial Explosion

by John Hunter and Justin Hunter on January 2, 2013

A combinatorial explosion is when the configuration settings and user actions and data entered etc. makes it impossible to test everything. The number of tests required to individually test every single possibility is many thousands of times greater than could realistically be tested.

When faced with taking over an existing software applications without a good test suite (or any test plan) often is daunting. And the problems of creating an unfathomable number of tests face you due to combinatorial explosion. Hexawise is a software as a service that aids in dealing with this dilemma for software testers. Software test plans are created that provide far better coverage than is seen in practice with a tiny fraction of the test required for complete combinatorial coverage (that is testing every possible combination [pairwise or 3, 4, 5... way] individually).

The Google Maps test plan provides a good example of combinatorial explosion faced by the testers (in this case, those who tested Google Maps). Take a look at the Google Maps test plan by login to your Hexawise account (creating a demo account is free and simple). The Google Maps test plan is one of 9 samples currently provided in Hexawise.

For creating your own test plan, while you are exploring the software application and testing it out to find “where the weak points are,” you will probably find it useful to vary things as much as possible, repeat your actions as little as possible. Those points are true whether you’re doing relatively informal lightly documented exploratory testing or more heavily documented test scripts. It addition, since a large percentage of defects can be triggered by the interaction of just two test inputs, it would be nice, if you had time, to test every single possible combination involving two test inputs; that’s the rationale behind allpairs, pairwise and orthogonal array-based test case prioritization methods.

To recreate a similar – very early draft – plan for yourself, I’d suggest going through the following steps to put together a relatively small number of highly informative end-to-end-ish tests:

Ask what can change as users go through the system? Think about configuration settings, user actions, data formats, data ranges, etc. even throw in more “creative” ideas like user personas. Let your creativity and common sense guide you. Enter those in as parameters.

Ask how those parameters can change? (for the parameter “Browser” enter IE7, IE8, FF, etc.) Put those in as values under each parameter (entering constraints as required)

Ask does that variation matter? When possible (when it doesn’t matter as much) use equivalence classes and be biased towards fewer values – at least for your early draft tests.

Ask what special paths thorough the system do you want to be sure to include? (Most common happy path, paths to trigger certain business rules, etc.)

Click the Create Tests button in Hexawise and you’ll instantly get a very nice draft starter set of highly varied tests. If they look like they’re relatively interesting and don’t miss hugely important things, start informally executing them and you’ll be sure to learn some more things as you do about the system’s weak points that would result in you going back to those draft tests and iterating them to make them stronger and cover more.

To get a bit more on using this approach see our case studies. Hexawise TV provides narrated videos online showing how to make your life easier as a software tester.

Related: Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values – 3 Strategies to Maximize Effectiveness of Your Tests – Pairwise and Combinatorial Software Testing in Agile Projects

{ 0 comments }

How to Model and Test CRUD Functionality

by John Hunter and Justin Hunter on December 18, 2012

Creating test plans for create, read, update and delete (CRUD) functionality is a very common requirement. There are a few different ways to model it. Let’s review the simplest solution first then move to a few optional extra ideas that you could use in addition.

Option 1: Only one C, R, U, D action tested in each of your tests

Have one Parameter called “CRUD action” – have 4 different values: Create, Read, Update, Delete

Option 2: Include two CRUD actions in (most of) your tests

Create 2 Parameters, each with the following 4 and 3 Values:

First CRUD action: Create, Read, Update, Delete
Second CRUD action: Read, Update, Delete

You may want to add an invalid pair with the first CRUD action = Delete and each of the three values in the second CRUD action (in which case you’d need to add a 4th Value called N/A and leave it as the only available option for scenarios that deleted a record in the first test.

Option 3: Up to 4 CRUD actions in each of your tests

  1. Create a new record: create, don’t create
  2. Read a record: read, don’t read
  3. Update a record: update, don’t update
  4. Delete a record: delete, don’t delete

    5. That’s the most basic modeling decision. (e.g., one CRUD action / test, two CRUD actions / test, or potentially 4 CRUD actions / test).

    What other things might be interesting to vary?

    You might want to think about “newspaper reporter questions” – e.g., (who, what, when, where, why? how? how much?)

    Who?

    Which user type (or which system) is performing the CRUD ACTION? – Are certain types of users or systems not allowed to perform certain actions? Include test inputs to make sure they can’t. Are others allowed to? Include test inputs in your model to make sure they can.

    What?

    [click to continue…]

{ 0 comments }

Justin Hunter Speaking at QAI’s Annual International Software Testing Conference In Bangalore

by Justin Hunter on November 22, 2012

I’ll be talking at QAI’s 12th Annual International Software Testing Conference on Dec 6th in Bangalore, India.

Topic: Conquering the Single Largest Challenge Facing Testers Today

“There’s too much to test and not enough time to test it all.” According to a recent survey conducted by Robert Sabourin, this is the single largest challenge facing test managers today. And this challenge clearly won’t go away any time soon. Software is becoming increasingly complex and time pressures put on testing teams are becoming ever more extreme.

To survive and thrive as testers, we need to find ways to learn more in the limited time we have. This talk addresses:

  • Proven test design methods to learn as much as possible about a System Under Test as quickly as possible
  • How these methods were originally developed and refined in other (non-IT) industries over the last 80 years
  • How the recent Apple Maps disaster could have been easily avoided by implementing these methods
  • Real world case studies: these methods sound nice on paper, but do they actually work?
  • Reasons why these methods are being used at more than 100 Fortune 500 firms today
  • What does the future hold?

Attendees will learn about valuable testing strategies that are being used today by more than 100 Fortune 500 firms. In particular, attendees will hear about:

Hexawise TV

by John Hunter on November 20, 2012

We have created a new site to highlight Hexawise videos on combinatorial, pairwise + orthogonal array software testing. We have posted videos on a variety of software testing topics including: selecting appropriate test inputs for pairwise and combinatorial software test design, how to select the proper inputs to create a pairwise test plan, using value expansions for values in the same equivalence classes.

Here is a video with an introduction to Hexawise:

Subscribe to the Hexawise TV blog. And if you haven’t subscribed to the RSS feed for the main Hexawise blog, do so now.

{ 0 comments }

Maximizing Software Tester Value by Letting Them Spend More Time Thinking

by John Hunter on November 11, 2012

testing sucks by James Whittaker

Bet that got your attention. It’s true, but let me qualify it: Running test cases over and over in the hope that bugs will manifest sucks. It’s boring, uncreative work and since half the world thinks that is all testing is about, it is no great wonder few people covet testing positions. Testing is either too tedious and repetitive or it’s downright too hard. Either way, who would want to stay in such a position?
…
For the hard parts of the testing process like deciding what to test and determining test completeness, user scenarios and so forth we have another creative and interesting task. Testers who spend time categorizing tests and developing strategy (the interesting part) are more focused on testing and thus spend less time running tests (the boring part).
…
So all the managers out there need to ask themselves what they’ve done lately to make their testers more creative. If you don’t have an answer, then testing isn’t the only thing that sucks.

One of the great benefits of Hexawise is that it takes care of figuring out the best test plan to provide the coverage is needed. The software test planer needs to use their knowledge, experience and creativity to determine what factors and parameters are critical to test. Then Hexawise generates a test plan that provides maximum coverage with the fewest possible tests. If people try to manually create test plans to address interaction between factors to be tested it is not only extremely time consuming, not very fun and it is essentially impossible to do well.

Some things are just so complex or so effectively handled with well designed software people cannot compete. Designing software test plan coverage is one of those areas.

Hexawise also lets the software tester easily tune the test coverage based on what is most important. Certain factors can be emphasized, others can be de-emphasised. Knowledge is needed to decide what factors are most important, but after that designing a test plan based on that knowledge shouldn’t take up staff time, good software can take care of that time consuming and difficult task.

Another nice feature included with Hexawise is automated detailed tester instructions are generated. And you can easily provide customized text to assure the test instructions and the expected outcomes are clear and complete.

Hexawise greatly reduces the number of test that need to be run by creating powerful test plans that provide more coverage with fewer tests. This again, frees up tester time to focus on value added activities.

Allowing testers to focus on adding value is a key aim of ours. We strive to automate what we can and allow testers to apply their knowledge, experience and creativity to helping create great software. Hexawise grew out of the work of George Box, William Hunter (the founders father) and W. Edwards Deming, they sought to use statistical tools to free people to focus on creative tasks. For example read: Managing Our Way to Economic Success, Two Untapped Resources by William G. Hunter – “Two resources, largely untapped in American organizations, are potential information and employee creativity.”

Hexawise includes sample test plans that let you see the benefits above in action. Sign up today to try it out (free trial).

Related: Cem Kaner: Testing Checklists = Good / Testing Scripts = Bad? – A Faster Way to Enter Test Inputs – the “Bulk Add” Option – Practical Ways to Respect People

{ 5 comments }

← Previous Entries

Next Entries →