Splitting Up Test Plans

by Sean Johnson on May 21, 2013

The test cases in a given test plan should be sufficiently similar and should not have wildly divergent paths depending on the value of parameters in a test case.

When you do find your test case flow diverges too much you often want to either break your test plan down into a few different test plans, so that you have a plan for each different kind of pass through the system.

Another similar approach is that you may want to decrease the scope of your test plan a bit so that you end up with test cases that are all similar in the plan.

Lastly, let’s say the flows aren’t wildly divergent, but only slightly so. As a silly example let’s say you were testing a recipe that varied based on the fruit selected.

Fruit: Apple, Grape, Orange, Banana

And then you wanted a step for how the peeling was done.

Peeling: By hand, By manual peeling tool, By automated peeler
Peeler type: Hand crank, Battery powered, AC powered

Now… our testing flow here has some divergence. Grapes and Apples don’t get peeled in this recipe, so they never enter that flow. And Bananas are always peeled by hand so they only get a part of that flow. If this was just the tip of the iceberg of the divergence, we should create a test plan for Grapes and Apples and a different one for Oranges and Bananas.

But if this is the entire extent of the divergent flow, then we want to take advantage of N/A values and married and invalid pairs.

We update our parameter values for the peeling flow to include N/A.

Peeling: By hand, By manual peeling tool, By automated peeler, N/A
Peeler type: Hand crank, Battery powered, AC powered, N/A

We marry Grape and Orange (uni-directionally) to the two N/A’s so they don’t participate in the peeling flow. We marry Banana (unidirectionally) to “By hand” and the 2nd N/A so it has a partial and circumscribed pass through the peeling flow.

Lastly we don’t allow Orange to be paired with either N/A with an invalid pair.

That’s how a slight flow variation can be accommodated. Please comment with any questions about any of these approaches to your problem.

Related: 3 Strategies to Maximize Effectiveness of Your Tests – How Not to Design Pairwise Software Tests – How to Model and Test CRUD Functionality

{ 0 comments }

But, That Won’t Work Here – Actionable Advice from Conrad Fujimoto

by John Hunter and Justin Hunter on May 2, 2013

I saw these words of advice from Conrad Fujimoto in an email and thought they were worth passing on. I’m using them with Conrad’s permission:

Over the years, I’ve taught many software testing courses. Trainees are appreciative of the ideas, insights, and techniques presented to them.They are convinced that principles and methods taught are useful and effective. Yet, often I hear the phrase “but, that won’t work here.”

Some of the reasons given for such pessimism are resource constraints, organizational politics, lack of testing focus, and little management understanding and support. The trainees knew what adjustments needed to be made but they felt powerless to affect any meaningful change. Fortunately, much can be accomplished by strategic planning and being aware of opportunities.

Some ideas:

  1. When no one is taking a leadership role in improving the process, consider assuming that role (people are often happy to see someone take charge).
  2. Seek opportunities to form relationships and work with others who share the same concerns about the existing process.
  3. Establish your authority and credentials for speaking on testing matters by recognizing and promoting the successes of your testing team.
  4. Be proactive and constantly monitor and report the progress of both development and testing against published schedules.
  5. Be ready to implement corrective actions or invoke contingency plans in the event of schedule slippages; where appropriate, suggest process changes that reduce future slippages.=
  6. Always perform test closure activities and ensure that lessons learned are recorded and reported.
  7. Get testing representation in requirement review meetings and on the change control board.
  8. Foster an attitude of continuous improvement; build on your successes.
  9. As  software testers, we have a professional obligation to do our best in assisting our organizations to build quality software. We may not necessarily have the term “manager” in our job title, but we still have the ability to be leaders. We can guide our organizations to creating better software.

Conrad Fujimoto is an expert instructor and consultant. He teaches Software Tester Certification for SQE Training.

George Box, a good friend (and a close colleague to my father), put the problem of getting new ideas adopted this way (from Management Matters by John Hunter):

1) It won’t work
2) It won’t work here
3) I thought of it first

John’s book, and blog, discuss the challenges of actually getting improvements put into action in the workplace. Getting past the resistance to new ideas, new ways of working and change is more difficult than it should be. But there are practical steps you can take to get improvements adopted, including those mentioned above.

Sadly, George Box recently passed away. You can see George in this video of him discussing the art of discovery (which is a big part of what software testers do – discovering how the software works).

Related: Growing the Application of Management Improvement Ideas in Your Organization – Outcome and In-Process Measures – Improving Software Development with Automated Tests

{ 0 comments }

Lanette Creamer Provides Suggestions on Reducing Test Case Bloat

by John Hunter and Justin Hunter on April 24, 2013

A client informed us that they had created (and used) approximately 3,500 test cases to test the search functionality of their application.  They had a strong suspicions that (a) they should be able to test the search functionality of their application with fewer tests, (b) the tests they had accidentally omitted tests of many hundreds of plausible combinations of values that would be useful to test for (but did not know how to precisely identify were those gaps were without a huge amount of work), and  that (c) many of these tests were quite inefficient in that they repeated many steps that had already been tested in other tests in the plan (even if they were not 100% duplicative of any other single test in the plan).

This client should have spoken to Lanette Creamer before they got into that situation.  Lanette is a testing expert and blogger with ideas worth paying attention to. For example, her paper, Reducing Test Case Bloat, is well worth reading as is her blog.

There are times when what you cut may not be bloat. There are some situations where the decisions are the equivalent of “Do we cut off the arm or the head?” Well, a person can live without an arm. If you are in a situation where you are so time constrained that critical areas will be untested, you can still communicate the risk, be transparent and use a strategy to test the most important areas first. It is possible to plan for and do testing for a very time constrained project.

Of course avoiding this situation is best. Improving testing processes to use the best thoughts and tools is a better option. Cutting the bloat can allow resources to be applied to those areas they are really needed. Often though, people are scared of trying new ideas and cling to old methods, even if that results in the organization having to take increased risks by failing to test critical areas sufficiently. They are just more scared of trying new ideas than of getting away with saying we need more funding if you want more testing.

Lanette’s article provides 8 specific suggestions for process improvements to reduce bloat. The first suggestion is to use combinatorial testing tools to greatly improve coverage while reducing workload. Another suggestion is to run the bloat reduction ideas by the stakeholders.

As part of your plan to reduce bloat, it can be helpful to state your assumptions about who is important and where you are placing testing priority and why. When reducing test case bloat you are taking a calculated risk. You are weighing the risk of being unable to test new features by insisting on testing every legacy case against the risk of purposefully not running some tests. When you share your starting assumptions with your stakeholders you offer them the chance to counter with their own assumptions and often you can clarify the boundaries of your testing this way to avoid gaps in testing or duplication.

See the full article for more good ideas on how to get better results for the existing testing resources available to your organization.

[click to continue…]

{ 0 comments }

Software Testing Carnival #3

by John Hunter on April 18, 2013

The Hexawise Software Testing carnival focuses on sharing interesting and useful blog posts related to software testing.

  • Testing and Checking Refined by James Bach and Michael Bolton – “a robust role for tools in testing must be embraced. As we work toward a future of skilled, powerful, and efficient testing, this requires a careful attention to both the human side and the mechanical side of the testing equation. Tools can help us in many ways far beyond the automation of checks. But in this, they necessarily play a supporting role to skilled humans; and the unskilled use of tools may have terrible consequences.”
  • Bugs Spread Disease by Elisabeth Hendrickson – “Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified.”
    • photo of Haija Sophia, Istanbul, Turkey

    Haija Sophia in Istanbul, Turkey by Justin Hunter

  • Becoming a World-Class Tester by Ilari Henrik Aegerter – “World-class testing means following the urge to get to the bottom of things, not giving up until you have experienced enough value for yourself, thinking more expansively about the role of a tester, and thinking non-traditionally about what skills are required to thrive in the role.”
  • Test Coverage Triage by Parimala Hariprasad – “My experience shows that a mind map based test design works great at this stage. Business teams will be thrilled to have a Visual Walkthrough of tests and provide inputs quickly. As a participant observer on several dozen IT projects, I have found out that testers’ personally walking them through the tests works really well.”

    • [click to continue…]

{ 0 comments }

Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing

by Justin Hunter on March 25, 2013

This post addresses some comments and skeptical (in a good way) questions raised by Phil Kirkham to our recent posts: The Software Testing Community Needs More Empirical Studies.

As background to my answers: The studies and dozens of proof of concept pilot projects that I’ve been directly involved with have sought to answer these 3 questions:

1) Is it actually faster to generate tests with Hexawise than creating and documenting them manually?

Consistent findings: Yes. It takes, on average, about 40% less time to create and document tests using Hexawise because using Hexawise allows testers to partially automate test selection and test documentation steps.

2) Is it possible to generate smaller sets of tests that will be as thorough or more thorough than larger sets of manually created tests and allow testers to find more defects in less test execution time?

Consistent findings: Yes. Typically more than twice as many defects per tester hour. See, e.g., the IEEE Computer article written with 3 PhDs showing an increase in defects found per tester hour of 2.4 times. A more recent set of 10 proof of concept pilot projects at an insurance firm revealed 3.0 times as many defects per tester hour. See: Does pairwise testing really work? Evidence, data, and case studies.
This is because Hexawise-generated tests (or any pairwise tests, for that matter) consistently have dramatically less wasteful repetition than manually selected tests will and because Hexawise-generated tests leave no potential dual-mode faults untested (that is, no potential pairwise defects involving test inputs that have been contemplated by the tester and included in their models).

3) Finding more defects per tester hour is certainly nice, but do Hexawise-generated tests find MORE defects?

Consistent findings: Yes. Much smaller set of Hexawise tests have consistently found more defects. On average by about 13%

In answer to specific questions:

Phil Kirkham: “Seems a very basic measure of a testers productivity How about the severity of the defects ?”

JH: Agreed.

In my experience in more than 5 years of helping teams conducting these proof of concept pilot projects, pairwise and Hexawise-generated tests are just plan more effective at finding defects. They find more of ALL kinds of defects. My experience has been that the types of defects being found is not skewed towards less significant types of defects or missing severe defects. A case in point: at my old firm, one of the early adopters of orthogonal array testing approaches ran pilot project after pilot project with teams of testers reporting into him. I can’t remember the exact number of pilots he had conducted, perhaps 20 pilot projects or so, before he experienced a single defect that escaped the Hexawise-generated tests that was found by the much longer set of manually selected tests. So, a short, blunt, honest answer to your excellent question, is “Believe it or not, Phil, it almost never matters. This approach will find ALL of the defects you otherwise would have found. Plus additional ones.” *

[click to continue…]

{ 0 comments }

Hexawise 2.0 Preview

by John Hunter on March 21, 2013

We are proud of the enhancements about to be delivered in version 2.0 of Hexawise (software test plan generation solution). Hexawise a software test design tool that helps teams test their software faster (by decreasing the time it takes to design and document tests and decreasing the amount of time it takes to execute tests) and more thoroughly (by scientifically packing as much coverage into each test as possible). We provide it as software as a service solution. Three of the most imported enhancements in our dramatically improved, soon-to-be-released version are:

  • Requirements (forcing certain specific combinations to appear in the test plan)
  • Adding Expected Results, which saves time in test case documentation
  • Better auto-scripting, which also saves time in test case documentation

The embedded slide presentation above, provides graphic illustration of these features.

At the “Required” screen (short for “Requirements”), you will be able to add specific combinations of test conditions/test inputs to the tests that Hexawise generates. Tracing requirements to specific test scripts can be challenging, particularly as requirements change and sets of regression tests age. You’ll find this feature helps make requirements traceability easier and less error-prone.

You can add Expected Results to the test scripts generated. This provides test stating exactly what the result should be so that someone reviewing the result of the test can verify if the expected results conditions were actually met. If Hexawise is the first test generation tool you’ve used, you might take this for granted and think that this is just how the world should be.

If you’ve used other test generation tools before finding Hexawise though, you might feel compelled to publicly declare your love of Hexawise and/or send gifts to the engineers and designers at Hexawise who created this great feature. We believe its a feature unique to Hexawise should save you huge amounts of time when you document your test scripts.

Auto-scripting was very appreciated by users, and we have enhanced this feature of Hexawise significantly in Hexawise 2.0. Our help system includes a through review of this (and many other features of Hexawise): Auto-Script test cases to quickly include detailed written tester instructions.

Related: How do I save test documentation time by automatically generating Expected Results in test scripts? – Pairwise and Combinatorial Software Testing in Agile Projects – Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values – Maximizing Software Tester Value by Letting Them Spend More Time Thinking

{ 0 comments }

Using Hexawise is One of the Highest ROI Software Development Practices in the World According to Recent Report

by John Hunter on March 18, 2013

Attempting to assess the relative benefits of more than 200 software development practices is not for the faint of heart. Context-specific considerations run the risk of confounding the conclusions at every turn. Even so, Capers Jones, a software development expert with dozens of years of experience and nearly twenty books related to software development to his credit, recently attempted the task. He’s literally devoted decades of his career to assessing such things for clients. We’re quite pleased with how using Hexawise fared in the analysis.

Scoring and Evaluating Software Methods, Practices and Results by Capers Jones (Vice President and CTO, Namcook Analytics) provides some great idea on software project management. The article is based on the Software Engineering Best Practices with some new data is taken from The Economics of Software Quality (two of the books Capers Jones has authored).

Software development, maintenance, and software management have dozens of methodologies and hundreds of tools available that are beneficial. In addition, there are quite a few methods and practices that have been shown to be harmful, based on depositions and court documents in litigation for software project failures.

In order to evaluate the effectiveness or harm of these numerous and disparate factors, a simple scoring method has been developed. The scoring method runs from +10 for maximum benefits to -10 for maximum harm.

The scoring method is based on quality and productivity improvements or losses compared to a mid-point. The mid point is traditional waterfall development carried out by projects at about level 1 on the Software Engineering Institute capability maturity model (CMMI) using low-level programming languages. Methods and practices that improve on this mid point are assigned positive scores, while methods and practices that show declines are assigned negative scores.

The data for the scoring comes from observations among about 150 Fortune 500 companies, some 50 smaller companies, and 30 government organizations. Negative scores also include data from 15 lawsuits.

The article provides guidance, based on the results achieved by many, and varied, organizations with respect to software projects.

finding and fixing bugs is overall the most expensive activity in software development. Quality leads and productivity follows. Attempts to improve productivity without improving quality first are not effective.

This is an extremely important point for business managers to understand. Those involved in software development professionally don’t find this surprising. But business people often greatly underestimate the costs of maintaining and updating software. The costs of bugs introduced by fairly minor feature requests to a system that doesn’t have good software test coverage or test plans often create far more trouble than business managers expect.

This is especially true because there is a high correlation between software applications that have poor software testing processes (including poor test coverage and poor or completely missing test plans) and those application that were designed without long term maintenance in mind. Both deficiencies result of decisions made to minimize initial development costs and time. They both show a lack of appreciation for wise software engineering practices and software application project management.

The article discusses a complicating factor for accessing the most effective software development practices: the extremely wide differences in software engineering scope. Projects range from simple applications one software developer can create in a short period of time to massive application requiring thousands of developer-years or effort.

In order to be considered a “best practice” a method or tool has to have some quantitative proof that it actually provides value in terms of quality improvement, productivity improvement, maintainability improvement, or some other tangible factors.

Looking at the situation from the other end, there are also methods, practices, and social issues have demonstrated that they are harmful and should always be avoided.
…
Although the author’s book Software Engineering Best Practices dealt with methods and practices by size and by type, it might be of interest to show the complete range of factors ranked in descending order, with the ones having the widest and most convincing proof of usefulness at the top of the list. Table 2 lists a total of 220 methodologies, practices, and social issues that have an impact on software applications and projects.

The average scores shown in table 2 are actually based on the composite average of six separate evaluations:

  1. Small applications < 1000 function points
  2. Medium applications between 1000 and 10,000 function points
  3. Large applications > 10,000 function points
  4. Information technology and web applications
  5. Commercial, systems, and embedded applications
  6. Government and military applications

The data for the scoring comes from observations among about 150 Fortune 500 companies, some 50 smaller companies, and 30 government organizations and around 13,000 total projects. Negative scores also include data from 15 lawsuits.

The scoring method does not have high precision and the placement is somewhat subjective.

The top 10 tools and practices listed in the article were:

Practice Score
Reusability (> 85% zero-defect materials) 9.65
Requirements patterns – InteGreat 9.50
Defect potentials < 3.00 per function point 9.35
Requirements modeling (T-VEC) 9.33
Defect removal efficiency > 95% 9.32
Personal Software Process (PSP) 9.25
Team Software Process (TSP) 9.18
Automated static analysis – code 9.17
Mathematical test case design (Hexawise) 9.17
Inspections (code) 9.15

We are obviously thrilled that Hexawise is listed. We have seen the value our customers have achieved using mathematical based combinatorial software test plans (see several Hexawise case studies). It is great to see that value recognized in comparison to other software development practices and judged to be of such high value to software development projects.

The article makes it clear the importance of the results is not “the precision of the rankings, which are somewhat subjective, but in the ability of the simple scoring method to show the overall sweep of many disparate topics using a single scale.”

The methodology behind the results shown in the article can be used to evaluate your organization’s software development practice and determine opportunities for improvement. But, as stated above, software projects cover a huge range of scopes. The specific software project needs will drive which practices are most critical to achieving success for a specific project. The list in the article, of what practices have provided huge value and what practices have resulted great harm, is a very helpful resource but project managers and software developers and testers need to apply their judgement to the information the article provides in order to achieve success.

A leading company will deploy methods that, when summed, total to more than 250 and average more than 5.5. Lagging organizations and lagging projects will sum to less than 100 and average below 4.0.

The use of Hexawise has been growing; that has helped increase the number of software projects using best practices (that score 9, or higher), however as the article states there is quite a need for improvement.

From data and observations on the usage patterns of software methods and practices, it is distressing to note that practices in the harmful or worst set are actually found on about 65% of U.S. Software projects as noted when doing assessments. Conversely, best practices that score 9 or higher have only been noted on about 14% of U.S. Software projects. It is no wonder that failures far outnumber successes for large software applications!

A score of 9 to 10 for a practice means that practice results 20-30% improvement in quality and productivity of software projects.

Conclusion: while your individual mileage may vary, this report provides further evidence that using Hexawise really does lead to large, measurable improvements in efficiency and effectiveness.

We are very proud of the success of Hexawise thus far; as a new year starts we see huge potential to help many organizations improve their software development efforts.

The article includes a list of references and suggested readings that is valuable. Included in that list are:

DeMarco, Tom; Controlling Software Projects, 1986, 296 pages.

Gilb, Tom and Graham, Dorothy; Software Inspections, 1994, 496 pages.

Jones, Capers; Applied Software Measurement, 3rd edition, 2008, 662 pages.

McConnell, Code Complete, (I’m linking to the 2nd edition the article references the 1st edition) 2004, 960 pages.

Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking – A Powerful Software Test Design Approach – 3 Strategies to Maximize Effectiveness of Your Tests

{ 0 comments }

Human-Computer Cooperation

by John Hunter on March 11, 2013

The video makes the case that the value to be gained from human-computer cooperation is being ignored far too often. A focus on maximizing the results based on improving the ability to cooperate is worthwhile.

What this means in practice is people taking more responsibility for using computers as tools to accomplish what is needed. This already happens a great deal but in a way which is unexamined and often therefore the current methods leave a great deal of room for improvement. We rarely focus on how to enhance the co-operation, we mainly see the software as one separate part of the process and a person’s contribution as another separate part of the process. Focusing on computers (and software) as tools used by people to accomplish objectives is helpful.

Viewing software as a tool used to achieve an aim mirrors the idea of a company viewing their products and services as solving specific problems for customers. The tool is valuable in how it is used by people – not in some abstract way (say technical specifications).

Weaknesses in how people use the product, service or software are often weaknesses in focusing on the way people will really use it versus how it is “supposed” to be used. By understanding the process that matters is one of a person and the computer together adding value, we can create more effective software applications.

People often try to design software solutions that remove the need for humans to be involved. For complex problems, though, it is often much more effective to design solutions where people take advantage of computer tools to achieve results. People should use computers to automate the things that make sense to automate, keep track of data, and make calculations, thus leaving themselves free to use their superior insight, vision, intuition, and flexibility in making judgements.

Hexawise is built to take advantage of this type of cooperation. Even though it is a “test design tool,” Hexawise doesn’t take the lead role in designing the tests. Humans do.  Humans do the things that they’re better than computers at, such as (a) thinking up clever test ideas and test inputs, and (b) identifying, from dozens of possible parameters, which are the ones that are most important to vary in order to achieve potentially interesting interactions from test to test. Computer algorithms aren’t nearly as good as humans at such tasks.  Computers, though, will run circles around any human who tries to construct a set of tests such that (a) the variation between each test is as different as possible, (b) the wasteful repetition of combinations of values that appear together in different tests is minimized, (c) gaps in coverage are minimized (by, e.g., ensuring that every single pair or every single 3-way combination of tests appears in at least one test case), and (d) all of the above objectives are achieved in the fewest possible test cases.  Computer algorithms eat those kind of challenges for breakfast.  And complete them without error.  In seconds.

Said a different way, people are better at figuring out interesting ideas to test. Once those are identified, those test conditions and other test ideas need to be combined together and put into tests.  Generating a highly efficient, maximally varied, minimally repetitive set of tests based on a given set of test inputs is something computer algorithms are more effective at than a person.

Hexawise is not intended to eliminate the need for software testing experts. Hexawise is designed to allow software testing experts to focus on what they do well and allow Hexawise to make everything else easy (creating test plans based on software experts inputs etc.). This allows software testing experts to spend their time thinking by removing time consuming tasks they have to do. Hexawise also creates test plan coverage that is simply beyond the ability of people to create no matter how much time they were given. And software testing experts provide the inputs that no matter how much time the Hexawise software could not create in any amount of time.

Hexawise is designed to optimize the ease of cooperation. We spend a great deal of time optimizing the software to make it most useful for people. The design decisions made in creating a software application are very different if the users are meant to thoughtfully interact with the application.

We see Hexawise as an extension of the software tester. We seek to optimize how a person can use Hexawise to create the most value. The measure is how much more effective the testing solutions are, not just how Hexawise performs in isolation from the user.

A great deal of our time has been spent on how to help software testing experts use Hexawise most effectively. These efforts often take the form of many many small improvements that create an experience that is more similar to cooperation between two parties with different strengths instead of a more typical user experience of forcing you to do whatever the software demands.

Related: Gamers Use Foldit to Solve Enzyme Configuration in 3 Weeks That Stumped Scientists for Over a Decade – Ten Reasons to Use Software Testing Checklists and Cheat Sheets

{ 0 comments }

The Software Testing Community Needs More Empirical Studies

by Justin Hunter on March 4, 2013

Based on my experience, over dozens of pilot projects where we’ve gathered hard data, many software testers would literally more than double their productivity overnight on many projects if they used combinatorial test design methods intelligently (in comparison to selecting test case conditions by hand).

In this 10 project study, Combinatorial Software Testing Case Studies, we found 2.4 times more defects per tester hour on average when we compared testers who executed manually-selected test cases to testers who executed test cases created by a combinatorial testing algorithm designed to achieve as much coverage as possible in as few tests as possible.

How many participating testers thought they would see dramatic increases before they gathered the data? Almost none (even the testers told about the prior experiences of their other colleagues on similar projects). How many participating testers are glad that they took the time to use the scientific method?

  • hypothesis
  • experiment
  • evidence
  • revise world-view

Every one of them.

What stops more people from using the scientific method on their projects and gathering data to prove or disprove hypotheses like the one addressed in the study above? A pilot could take one person’s time for less than 2 days. If past experience is any indication of future results (and granted, it isn’t always), odds would appear pretty good that results would show that productivity would double (as measured in defects found per tester hour).

What’s stopping the testing community from doing more such analysis? Perhaps more importantly, what is stopping you from gathering this kind of data on your project?

Additional empirical studies on the effectiveness of software testing strategies would greatly benefit the software testing community.

Related: Hexawise case studies on software testing improvement (health insurance, IT consulting and mortgage processing studies) – How Not to Design Pairwise Software Tests – 3 Strategies to Maximize Effectiveness of Your Tests

{ 6 comments }

Maximizing Variabiliy Maximizes Your Ability to Find Bugs That You’re Not Even Consciously Looking For

by Justin Hunter on February 25, 2013

I strongly agree with Cem Kaner’s statement (in Inefficiency and Ineffectiveness of Software Testing: A Key Problem in Software Engineering) that: sometimes tests uncover defects that don’t fit within any coverage model because they are side effects of the tests rather than explicitly planned foci of the tests.”

My experience indicates that an effective way to increase the likelihood that you will trigger such defects (without explicitly looking for them) is to try to maximize the variation between each test case you execute.

A case in point: when I sat down to dinner with James Bach a year or so ago in Boston at a testing conference, he gave me a quick testing challenge (as he is fond of doing with testers he meets for the fist time to see how we think).  He asked how I would test a very simple calendar entry application that allowed users to record the start and end times of diary events.  Key inputs to use as test conditions for these tests included start times and end times.

I proposed a set of times to try that were designed to provide as much variety as possible from one test case to the next.  As inputs into the start and end times, I used a small number of different times spread throughout the morning, afternoon, and evening as well.  The strategy I used quickly identified the testing defect the puzzle was designed to uncover in a small handful of tests.  What was most memorable about the experience from my perspective was not that I “succeeded” in triggering the bug but that the tests I created triggered a type of bug that was, in Kaner’s words, a “side effect of the tests rather than explicitly planned foci of the tests.”

The business logic in the calendar application that should have identified invalid beginning and end time combinations was coded incorrectly. Instead of using numbers in the business logic, the business logic was ordering the numbers alphabetically.  I was not consciously looking to identify that kind of a flaw in the business logic, but by maximizing the variation from test case to test case, I maximized my odds of finding it.

Efficiently achieving structured variation is difficult because it is hard for a human brain to remember whether dozens of different test conditions have been tested together (or we’re accidentally repeating ourselves).  This is where pairwise and combinatorial test case generating tools like our Hexawise tool come in.  They are designed to achieve as much variation from test case to test case as possible.  One of the relatively unsung benefits of this approach is that doing so will help find bugs, like these, that you aren’t even consciously looking for.

Related: 3 Strategies to Maximize Effectiveness of Your Tests – Getting Started with a Test Plan When Faced with a Combinatorial Explosion – Book Review of “Explore It!” Elisabeth Hendrickson’s Excellent New Book on Software Testing

{ 0 comments }

← Previous Entries