Hexawise

But, That Won’t Work Here – Actionable Advice from Conrad Fujimoto

John Hunter and Justin Hunter — Thu, 02 May 2013 13:44:50 +0000

I saw these words of advice from Conrad Fujimoto in an email and thought they were worth passing on. I’m using them with Conrad’s permission:

Over the years, I’ve taught many software testing courses. Trainees are appreciative of the ideas, insights, and techniques presented to them.They are convinced that principles and methods taught are useful and effective. Yet, often I hear the phrase “but, that won’t work here.”

Some of the reasons given for such pessimism are resource constraints, organizational politics, lack of testing focus, and little management understanding and support. The trainees knew what adjustments needed to be made but they felt powerless to affect any meaningful change. Fortunately, much can be accomplished by strategic planning and being aware of opportunities.

Some ideas:

When no one is taking a leadership role in improving the process, consider assuming that role (people are often happy to see someone take charge).

Seek opportunities to form relationships and work with others who share the same concerns about the existing process.

Establish your authority and credentials for speaking on testing matters by recognizing and promoting the successes of your testing team.

Be proactive and constantly monitor and report the progress of both development and testing against published schedules.

Be ready to implement corrective actions or invoke contingency plans in the event of schedule slippages; where appropriate, suggest process changes that reduce future slippages.=

Always perform test closure activities and ensure that lessons learned are recorded and reported.

Get testing representation in requirement review meetings and on the change control board.

Foster an attitude of continuous improvement; build on your successes.

As software testers, we have a professional obligation to do our best in assisting our organizations to build quality software. We may not necessarily have the term “manager” in our job title, but we still have the ability to be leaders. We can guide our organizations to creating better software.

Conrad Fujimoto is an expert instructor and consultant. He teaches Software Tester Certification for SQE Training.

George Box, a good friend (and a close colleague to my father), put the problem of getting new ideas adopted this way (from Management Matters by John Hunter):

1) It won’t work
2) It won’t work here
3) I thought of it first

John’s book, and blog, discuss the challenges of actually getting improvements put into action in the workplace. Getting past the resistance to new ideas, new ways of working and change is more difficult than it should be. But there are practical steps you can take to get improvements adopted, including those mentioned above.

Sadly, George Box recently passed away. You can see George in this video of him discussing the art of discovery (which is a big part of what software testers do – discovering how the software works).

Lanette Creamer Provides Suggestions on Reducing Test Case Bloat

John Hunter and Justin Hunter — Wed, 24 Apr 2013 06:30:39 +0000

A client informed us that they had created (and used) approximately 3,500 test cases to test the search functionality of their application. They had a strong suspicions that (a) they should be able to test the search functionality of their application with fewer tests, (b) the tests they had accidentally omitted tests of many hundreds of plausible combinations of values that would be useful to test for (but did not know how to precisely identify were those gaps were without a huge amount of work), and that (c) many of these tests were quite inefficient in that they repeated many steps that had already been tested in other tests in the plan (even if they were not 100% duplicative of any other single test in the plan).

This client should have spoken to Lanette Creamer before they got into that situation. Lanette is a testing expert and blogger with ideas worth paying attention to. For example, her paper, Reducing Test Case Bloat, is well worth reading as is her blog.

There are times when what you cut may not be bloat. There are some situations where the decisions are the equivalent of “Do we cut off the arm or the head?” Well, a person can live without an arm. If you are in a situation where you are so time constrained that critical areas will be untested, you can still communicate the risk, be transparent and use a strategy to test the most important areas first. It is possible to plan for and do testing for a very time constrained project.

Of course avoiding this situation is best. Improving testing processes to use the best thoughts and tools is a better option. Cutting the bloat can allow resources to be applied to those areas they are really needed. Often though, people are scared of trying new ideas and cling to old methods, even if that results in the organization having to take increased risks by failing to test critical areas sufficiently. They are just more scared of trying new ideas than of getting away with saying we need more funding if you want more testing.

Lanette’s article provides 8 specific suggestions for process improvements to reduce bloat. The first suggestion is to use combinatorial testing tools to greatly improve coverage while reducing workload. Another suggestion is to run the bloat reduction ideas by the stakeholders.

As part of your plan to reduce bloat, it can be helpful to state your assumptions about who is important and where you are placing testing priority and why. When reducing test case bloat you are taking a calculated risk. You are weighing the risk of being unable to test new features by insisting on testing every legacy case against the risk of purposefully not running some tests. When you share your starting assumptions with your stakeholders you offer them the chance to counter with their own assumptions and often you can clarify the boundaries of your testing this way to avoid gaps in testing or duplication.

See the full article for more good ideas on how to get better results for the existing testing resources available to your organization.

Reducing Test Case Bloat2 1 from lanettecream

Software Testing Carnival #3

John Hunter — Thu, 18 Apr 2013 06:17:10 +0000

The Hexawise Software Testing carnival focuses on sharing interesting and useful blog posts related to software testing.

Testing and Checking Refined by James Bach and Michael Bolton – “a robust role for tools in testing must be embraced. As we work toward a future of skilled, powerful, and efficient testing, this requires a careful attention to both the human side and the mechanical side of the testing equation. Tools can help us in many ways far beyond the automation of checks. But in this, they necessarily play a supporting role to skilled humans; and the unskilled use of tools may have terrible consequences.”
Bugs Spread Disease by Elisabeth Hendrickson – “Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified.”

Haija Sophia in Istanbul, Turkey by Justin Hunter

Becoming a World-Class Tester by Ilari Henrik Aegerter – “World-class testing means following the urge to get to the bottom of things, not giving up until you have experienced enough value for yourself, thinking more expansively about the role of a tester, and thinking non-traditionally about what skills are required to thrive in the role.”
Test Coverage Triage by Parimala Hariprasad – “My experience shows that a mind map based test design works great at this stage. Business teams will be thrilled to have a Visual Walkthrough of tests and provide inputs quickly. As a participant observer on several dozen IT projects, I have found out that testers’ personally walking them through the tests works really well.”

What Does it Take to Change the Software Testing Industry? Courage! by Keith Klain – “According to Mark Twain, courage is not the absence of fear – but the mastery of it. There are people working in software testing all over the globe who are questioning long standing ways of working – some for the first time. Get yourself energized and get involved.”
Explaining Exploratory Testing Relies On Good Notes.. by Rob Lambert -
“Being able to do good exploratory testing is one thing, being able to explain this testing (and the insights it brings) to the people that matter is a different set of skills. I believe many testers are ignoring and neglecting the communication side of their skills, and it’s a shame because it may be directly affecting the opportunities they have to do exploratory testing in the first place.”
Human-Computer Cooperation by John Hunter – “people are better at figuring out interesting ideas to test. Once those are identified, those test conditions and other test ideas need to be combined together and put into tests. Generating a highly efficient, maximally varied, minimally repetitive set of tests based on a given set of test inputs is something computer algorithms are more effective at than a person.”
Software testing can be sexy, too by Ole Lensmar – “What Klain, who serves on the board of the AST, and the others would like to do is not only bring those skills back home but increase the availability and accessibility of this kind of training and job opportunity. Ideally, colleges and universities would start offering majors in Software Testing so we can set young people on a path toward testing as a career.”
Yet another future of testing post (YAFOTP) by Alan Page – “I think we’ll always need people to look at big end-to-end scenarios and determine how non-functional attributes (e.g. performance, privacy, usability, reliability, etc.) contribute to the user experience. Some of this evaluation will come from manually walking through scenarios), but there will be plenty of need for programmatic measurement and analysis as well…”

Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing

Justin Hunter — Mon, 25 Mar 2013 23:41:33 +0000

This post addresses some comments and skeptical (in a good way) questions raised by Phil Kirkham to our recent posts: The Software Testing Community Needs More Empirical Studies.

As background to my answers: The studies and dozens of proof of concept pilot projects that I’ve been directly involved with have sought to answer these 3 questions:

1) Is it actually faster to generate tests with Hexawise than creating and documenting them manually?

Consistent findings: Yes. It takes, on average, about 40% less time to create and document tests using Hexawise because using Hexawise allows testers to partially automate test selection and test documentation steps.

2) Is it possible to generate smaller sets of tests that will be as thorough or more thorough than larger sets of manually created tests and allow testers to find more defects in less test execution time?

Consistent findings: Yes. Typically more than twice as many defects per tester hour. See, e.g., the IEEE Computer article written with 3 PhDs showing an increase in defects found per tester hour of 2.4 times. A more recent set of 10 proof of concept pilot projects at an insurance firm revealed 3.0 times as many defects per tester hour. See: Does pairwise testing really work? Evidence, data, and case studies.
This is because Hexawise-generated tests (or any pairwise tests, for that matter) consistently have dramatically less wasteful repetition than manually selected tests will and because Hexawise-generated tests leave no potential dual-mode faults untested (that is, no potential pairwise defects involving test inputs that have been contemplated by the tester and included in their models).

3) Finding more defects per tester hour is certainly nice, but do Hexawise-generated tests find MORE defects?

Consistent findings: Yes. Much smaller set of Hexawise tests have consistently found more defects. On average by about 13%

In answer to specific questions:

Phil Kirkham: “Seems a very basic measure of a testers productivity How about the severity of the defects ?”

JH: Agreed.

In my experience in more than 5 years of helping teams conducting these proof of concept pilot projects, pairwise and Hexawise-generated tests are just plan more effective at finding defects. They find more of ALL kinds of defects. My experience has been that the types of defects being found is not skewed towards less significant types of defects or missing severe defects. A case in point: at my old firm, one of the early adopters of orthogonal array testing approaches ran pilot project after pilot project with teams of testers reporting into him. I can’t remember the exact number of pilots he had conducted, perhaps 20 pilot projects or so, before he experienced a single defect that escaped the Hexawise-generated tests that was found by the much longer set of manually selected tests. So, a short, blunt, honest answer to your excellent question, is “Believe it or not, Phil, it almost never matters. This approach will find ALL of the defects you otherwise would have found. Plus additional ones.” *

* Major caveat here that calls into question my specific answer here (to your question concerning severity) as well as all of the results from all of the studies I’ve been involved with. Testers like you and me are strong proponents of Exploratory Testing. These studies, though, treat the test inputs and test cases as “frozen.” You have the test cases in list A (created manually) and the test cases in list B (created by using Hexawise). The ideas about what can be changed from test to test (parameters) and how each of those things can be changed (values) are identical in both lists. The difference is that list A has lots of wasteful repetition and lots of gaps in coverage. List B has neither. That’s the only difference. Then one tester executes the tests from list A and another tester executes the tests from list B. But what if you have an unskilled tester following rote scripts executing one set of tests and someone like you, Rob Sabourin, Michael Bolton, James Bach, Shmuel Gershon, Ajay Balamurugadas, etc., executing the second set? Whoa! All bets are off. What would happen is that skilled Exploratory Testers would use the Hexawise-generated test ideas (which they would not want to be overly-detailed), and go “off script” to explore interesting test ideas that they cooked up in real time as they were doing their testing. So skilled Exploratory Testers would be able to find defects (presumably including serious ones) that the written test cases, regardless of whether they were manually created or created by Hexawise, would not lead them to directly. That’s an important topic for another time. I’ll be talking about Exploratory Combinatorial Testing at the Conference of the Association of Software Testing – CAST – this year in my home town of Madison, Wisconsin. Since you’re also going, perhaps we could collaborate and you could share your experiences (good or bad) with the attendees. I’ll happily give you 10 minutes of my speaking time to share your experiences if you’d like.

PK: Are you only counting functional defects ?

JH: Not explicitly. The directions I give to teams running these pilot projects is. Try to answer the 3 questions above. Report defects. We’re not after a count of “failed test cases.” We’re after a number of defects. Having said that, as you might suspect, most of the defects reported tend to be functional defects.

PK: What about all the other types of defect ?

JH: They’re not reported as often but we count them too. In situations where one tester reports a “hard to spot” bug (e.g., one that might take a more experienced tester to identify), it raises the possibility that the bug is being reported not because one set of tests is superior to the other but because one tester is better than the other. Accordingly, in an effort to keep an apples to apples comparison, we talk with the tester and try to determine with the tester’s input whether the tester would have found that same defect with the other set of tests. If the answer is yes, we’d report the defect as “found” in both sets of tests. This doesn’t happen as often as you might suspect it would.

PK: Does the project type matter ?

JH: Yes. Benefits tend to be relatively smaller when there are a disproportionately high percentage of small, discrete, one-off tests. And higher when there are more than 5 parameters that interact in meaningful ways. And easier to capture when the System Under Test does not have a lot of conditional branching logic.

PK: How about the devs they are working with and the practices they follow ?

JH: I don’t have enough empirical evidence to say definitively. It’s used successfully by thousands of testers in waterfall projects and thousands of testers in Agile projects.

PK: What about the experience of the tester, does that make a difference?

JH: Even more important than experience level of the tester are, in order, (1) analytical ability, (2) willingness to try new things, and (3) willingness to ask questions. By my estimates about 50% of the testers I come across at our clients (almost all at Fortune 2000 firms) would not be able to design excellent sets of pairwise tests from scratch. This is because above average analytical ability is required for testers to select parameters and values from their Systems Under Test in a thoughtful way. Getting back to experience level, some of our strongest users at our clients are straight of out of college. They start work, get exposed to Hexawise, “get it” and don’t look back. Interestingly, some testers who have been testing for, say, 10 years or more – while experienced – sometimes seem to be too set in their ways to embrace this rather different approach to designing tests.

PK: If they are working with top of the range developers (as some lucky testers are cough cough) then there aren’t that many functional bugs to be found and you’re looking at browser compatibility, usability, race conditions – is combinatorial testing going to find these more quickly ?

JH: Yes. Absolutely. If you’d like to collaborate to test that and help gather empirical evidence that you could share at CAST, I would be happy to work with you to do just that. If your experience contradicts what I’m saying here, you’d have the floor to tell CAST participants what your actual experience was.

PK: I read the study in the link – 97% of defects could be found by pairwise combinatorial testing ? Really ? ALL types of defects ? Really ? How can pairwise find a defect caused by a missing or ambiguous or inconsistent requirement, or a performance or security ?

JH: The statistics I quote are a lot lower than that. The pie chart I use averages out several studies done by PhD’s that have found, on average, 84% of defects could be triggered by testing for all combinations of 2 test inputs. The 97% figure is eyebrow-raising on its own (regardless of industry). Given that it was in the medical device industry in the United States (one of the most litigious area in the history of the world?), that statistic is particularly mind-boggling. What the PhDs in that study did was take a look at all of the medical devices that had been taken off of the market in the United States as a result of software defects. Then they investigated how many test inputs would be required to trigger each of those defects. They authors of that study found that an astonishing 97% of those defects could have been triggered by just 2 test inputs.

PK: Love your passion and enthusiasm and I do have a beta of Hexawise to see if it can do anything for my productivity – and I might agree that there is a lack of empirical studies, not just among the testing community but the s/w community as a whole into the effectiveness or not of how software is produced

JH: Thanks. I hope you have positive experiences with using Hexawise and I’m happy to help you if ever have any questions about using Hexawise on your projects.

Hexawise 2.0 Preview

John Hunter — Thu, 21 Mar 2013 07:31:10 +0000

We are proud of the enhancements about to be delivered in version 2.0 of Hexawise (software test plan generation solution). Hexawise a software test design tool that helps teams test their software faster (by decreasing the time it takes to design and document tests and decreasing the amount of time it takes to execute tests) and more thoroughly (by scientifically packing as much coverage into each test as possible). We provide it as software as a service solution. Three of the most imported enhancements in our dramatically improved, soon-to-be-released version are:

Requirements (forcing certain specific combinations to appear in the test plan)
Adding Expected Results, which saves time in test case documentation
Better auto-scripting, which also saves time in test case documentation

New Hexawise Features from Justin Hunter

The embedded slide presentation above, provides graphic illustration of these features.

At the “Required” screen (short for “Requirements”), you will be able to add specific combinations of test conditions/test inputs to the tests that Hexawise generates. Tracing requirements to specific test scripts can be challenging, particularly as requirements change and sets of regression tests age. You’ll find this feature helps make requirements traceability easier and less error-prone.

You can add Expected Results to the test scripts generated. This provides test stating exactly what the result should be so that someone reviewing the result of the test can verify if the expected results conditions were actually met. If Hexawise is the first test generation tool you’ve used, you might take this for granted and think that this is just how the world should be.

If you’ve used other test generation tools before finding Hexawise though, you might feel compelled to publicly declare your love of Hexawise and/or send gifts to the engineers and designers at Hexawise who created this great feature. We believe its a feature unique to Hexawise should save you huge amounts of time when you document your test scripts.

Auto-scripting was very appreciated by users, and we have enhanced this feature of Hexawise significantly in Hexawise 2.0. Our help system includes a through review of this (and many other features of Hexawise): Auto-Script test cases to quickly include detailed written tester instructions.

Using Hexawise is One of the Highest ROI Software Development Practices in the World According to Recent Report

John Hunter — Mon, 18 Mar 2013 14:05:07 +0000

Attempting to assess the relative benefits of more than 200 software development practices is not for the faint of heart. Context-specific considerations run the risk of confounding the conclusions at every turn. Even so, Capers Jones, a software development expert with dozens of years of experience and nearly twenty books related to software development to his credit, recently attempted the task. He’s literally devoted decades of his career to assessing such things for clients. We’re quite pleased with how using Hexawise fared in the analysis.

Scoring and Evaluating Software Methods, Practices and Results by Capers Jones (Vice President and CTO, Namcook Analytics) provides some great idea on software project management. The article is based on the Software Engineering Best Practices with some new data is taken from The Economics of Software Quality (two of the books Capers Jones has authored).

Software development, maintenance, and software management have dozens of methodologies and hundreds of tools available that are beneficial. In addition, there are quite a few methods and practices that have been shown to be harmful, based on depositions and court documents in litigation for software project failures.

In order to evaluate the effectiveness or harm of these numerous and disparate factors, a simple scoring method has been developed. The scoring method runs from +10 for maximum benefits to -10 for maximum harm.

The scoring method is based on quality and productivity improvements or losses compared to a mid-point. The mid point is traditional waterfall development carried out by projects at about level 1 on the Software Engineering Institute capability maturity model (CMMI) using low-level programming languages. Methods and practices that improve on this mid point are assigned positive scores, while methods and practices that show declines are assigned negative scores.

The data for the scoring comes from observations among about 150 Fortune 500 companies, some 50 smaller companies, and 30 government organizations. Negative scores also include data from 15 lawsuits.

The article provides guidance, based on the results achieved by many, and varied, organizations with respect to software projects.

finding and fixing bugs is overall the most expensive activity in software development. Quality leads and productivity follows. Attempts to improve productivity without improving quality first are not effective.

This is an extremely important point for business managers to understand. Those involved in software development professionally don’t find this surprising. But business people often greatly underestimate the costs of maintaining and updating software. The costs of bugs introduced by fairly minor feature requests to a system that doesn’t have good software test coverage or test plans often create far more trouble than business managers expect.

This is especially true because there is a high correlation between software applications that have poor software testing processes (including poor test coverage and poor or completely missing test plans) and those application that were designed without long term maintenance in mind. Both deficiencies result of decisions made to minimize initial development costs and time. They both show a lack of appreciation for wise software engineering practices and software application project management.

The article discusses a complicating factor for accessing the most effective software development practices: the extremely wide differences in software engineering scope. Projects range from simple applications one software developer can create in a short period of time to massive application requiring thousands of developer-years or effort.

In order to be considered a “best practice” a method or tool has to have some quantitative proof that it actually provides value in terms of quality improvement, productivity improvement, maintainability improvement, or some other tangible factors.

Looking at the situation from the other end, there are also methods, practices, and social issues have demonstrated that they are harmful and should always be avoided.
…
Although the author’s book Software Engineering Best Practices dealt with methods and practices by size and by type, it might be of interest to show the complete range of factors ranked in descending order, with the ones having the widest and most convincing proof of usefulness at the top of the list. Table 2 lists a total of 220 methodologies, practices, and social issues that have an impact on software applications and projects.

The average scores shown in table 2 are actually based on the composite average of six separate evaluations:

Small applications < 1000 function points

Medium applications between 1000 and 10,000 function points

Large applications > 10,000 function points

Information technology and web applications

Commercial, systems, and embedded applications

Government and military applications

The data for the scoring comes from observations among about 150 Fortune 500 companies, some 50 smaller companies, and 30 government organizations and around 13,000 total projects. Negative scores also include data from 15 lawsuits.

The scoring method does not have high precision and the placement is somewhat subjective.

The top 10 tools and practices listed in the article were:

Practice	Score
Reusability (> 85% zero-defect materials)	9.65
Requirements patterns – InteGreat	9.50
Defect potentials < 3.00 per function point	9.35
Requirements modeling (T-VEC)	9.33
Defect removal efficiency > 95%	9.32
Personal Software Process (PSP)	9.25
Team Software Process (TSP)	9.18
Automated static analysis – code	9.17
Mathematical test case design (Hexawise)	9.17
Inspections (code)	9.15

We are obviously thrilled that Hexawise is listed. We have seen the value our customers have achieved using mathematical based combinatorial software test plans (see several Hexawise case studies). It is great to see that value recognized in comparison to other software development practices and judged to be of such high value to software development projects.

The article makes it clear the importance of the results is not “the precision of the rankings, which are somewhat subjective, but in the ability of the simple scoring method to show the overall sweep of many disparate topics using a single scale.”

The methodology behind the results shown in the article can be used to evaluate your organization’s software development practice and determine opportunities for improvement. But, as stated above, software projects cover a huge range of scopes. The specific software project needs will drive which practices are most critical to achieving success for a specific project. The list in the article, of what practices have provided huge value and what practices have resulted great harm, is a very helpful resource but project managers and software developers and testers need to apply their judgement to the information the article provides in order to achieve success.

A leading company will deploy methods that, when summed, total to more than 250 and average more than 5.5. Lagging organizations and lagging projects will sum to less than 100 and average below 4.0.

The use of Hexawise has been growing; that has helped increase the number of software projects using best practices (that score 9, or higher), however as the article states there is quite a need for improvement.

From data and observations on the usage patterns of software methods and practices, it is distressing to note that practices in the harmful or worst set are actually found on about 65% of U.S. Software projects as noted when doing assessments. Conversely, best practices that score 9 or higher have only been noted on about 14% of U.S. Software projects. It is no wonder that failures far outnumber successes for large software applications!

A score of 9 to 10 for a practice means that practice results 20-30% improvement in quality and productivity of software projects.

Conclusion: while your individual mileage may vary, this report provides further evidence that using Hexawise really does lead to large, measurable improvements in efficiency and effectiveness.

We are very proud of the success of Hexawise thus far; as a new year starts we see huge potential to help many organizations improve their software development efforts.

The article includes a list of references and suggested readings that is valuable. Included in that list are:

DeMarco, Tom; Controlling Software Projects, 1986, 296 pages.

Gilb, Tom and Graham, Dorothy; Software Inspections, 1994, 496 pages.

Jones, Capers; Applied Software Measurement, 3rd edition, 2008, 662 pages.

McConnell, Code Complete, (I’m linking to the 2nd edition the article references the 1st edition) 2004, 960 pages.

Human-Computer Cooperation

John Hunter — Tue, 12 Mar 2013 02:25:05 +0000

The video makes the case that the value to be gained from human-computer cooperation is being ignored far too often. A focus on maximizing the results based on improving the ability to cooperate is worthwhile.

What this means in practice is people taking more responsibility for using computers as tools to accomplish what is needed. This already happens a great deal but in a way which is unexamined and often therefore the current methods leave a great deal of room for improvement. We rarely focus on how to enhance the co-operation, we mainly see the software as one separate part of the process and a person’s contribution as another separate part of the process. Focusing on computers (and software) as tools used by people to accomplish objectives is helpful.

Viewing software as a tool used to achieve an aim mirrors the idea of a company viewing their products and services as solving specific problems for customers. The tool is valuable in how it is used by people – not in some abstract way (say technical specifications).

Weaknesses in how people use the product, service or software are often weaknesses in focusing on the way people will really use it versus how it is “supposed” to be used. By understanding the process that matters is one of a person and the computer together adding value, we can create more effective software applications.

People often try to design software solutions that remove the need for humans to be involved. For complex problems, though, it is often much more effective to design solutions where people take advantage of computer tools to achieve results. People should use computers to automate the things that make sense to automate, keep track of data, and make calculations, thus leaving themselves free to use their superior insight, vision, intuition, and flexibility in making judgements.

Hexawise is built to take advantage of this type of cooperation. Even though it is a “test design tool,” Hexawise doesn’t take the lead role in designing the tests. Humans do. Humans do the things that they’re better than computers at, such as (a) thinking up clever test ideas and test inputs, and (b) identifying, from dozens of possible parameters, which are the ones that are most important to vary in order to achieve potentially interesting interactions from test to test. Computer algorithms aren’t nearly as good as humans at such tasks. Computers, though, will run circles around any human who tries to construct a set of tests such that (a) the variation between each test is as different as possible, (b) the wasteful repetition of combinations of values that appear together in different tests is minimized, (c) gaps in coverage are minimized (by, e.g., ensuring that every single pair or every single 3-way combination of tests appears in at least one test case), and (d) all of the above objectives are achieved in the fewest possible test cases. Computer algorithms eat those kind of challenges for breakfast. And complete them without error. In seconds.

Said a different way, people are better at figuring out interesting ideas to test. Once those are identified, those test conditions and other test ideas need to be combined together and put into tests. Generating a highly efficient, maximally varied, minimally repetitive set of tests based on a given set of test inputs is something computer algorithms are more effective at than a person.

Hexawise is not intended to eliminate the need for software testing experts. Hexawise is designed to allow software testing experts to focus on what they do well and allow Hexawise to make everything else easy (creating test plans based on software experts inputs etc.). This allows software testing experts to spend their time thinking by removing time consuming tasks they have to do. Hexawise also creates test plan coverage that is simply beyond the ability of people to create no matter how much time they were given. And software testing experts provide the inputs that no matter how much time the Hexawise software could not create in any amount of time.

Hexawise is designed to optimize the ease of cooperation. We spend a great deal of time optimizing the software to make it most useful for people. The design decisions made in creating a software application are very different if the users are meant to thoughtfully interact with the application.

We see Hexawise as an extension of the software tester. We seek to optimize how a person can use Hexawise to create the most value. The measure is how much more effective the testing solutions are, not just how Hexawise performs in isolation from the user.

A great deal of our time has been spent on how to help software testing experts use Hexawise most effectively. These efforts often take the form of many many small improvements that create an experience that is more similar to cooperation between two parties with different strengths instead of a more typical user experience of forcing you to do whatever the software demands.

The Software Testing Community Needs More Empirical Studies

Justin Hunter — Tue, 05 Mar 2013 02:05:43 +0000

Based on my experience, over dozens of pilot projects where we’ve gathered hard data, many software testers would literally more than double their productivity overnight on many projects if they used combinatorial test design methods intelligently (in comparison to selecting test case conditions by hand).

In this 10 project study, Combinatorial Software Testing Case Studies, we found 2.4 times more defects per tester hour on average when we compared testers who executed manually-selected test cases to testers who executed test cases created by a combinatorial testing algorithm designed to achieve as much coverage as possible in as few tests as possible.

How many participating testers thought they would see dramatic increases before they gathered the data? Almost none (even the testers told about the prior experiences of their other colleagues on similar projects). How many participating testers are glad that they took the time to use the scientific method?

hypothesis
experiment
evidence
revise world-view

Every one of them.

What stops more people from using the scientific method on their projects and gathering data to prove or disprove hypotheses like the one addressed in the study above? A pilot could take one person’s time for less than 2 days. If past experience is any indication of future results (and granted, it isn’t always), odds would appear pretty good that results would show that productivity would double (as measured in defects found per tester hour).

What’s stopping the testing community from doing more such analysis? Perhaps more importantly, what is stopping you from gathering this kind of data on your project?

Additional empirical studies on the effectiveness of software testing strategies would greatly benefit the software testing community.

Maximizing Variabiliy Maximizes Your Ability to Find Bugs That You’re Not Even Consciously Looking For

Justin Hunter — Tue, 26 Feb 2013 02:38:00 +0000

I strongly agree with Cem Kaner’s statement (in Inefficiency and Ineffectiveness of Software Testing: A Key Problem in Software Engineering) that: sometimes tests uncover defects that don’t fit within any coverage model because they are side effects of the tests rather than explicitly planned foci of the tests.”

My experience indicates that an effective way to increase the likelihood that you will trigger such defects (without explicitly looking for them) is to try to maximize the variation between each test case you execute.

A case in point: when I sat down to dinner with James Bach a year or so ago in Boston at a testing conference, he gave me a quick testing challenge (as he is fond of doing with testers he meets for the fist time to see how we think). He asked how I would test a very simple calendar entry application that allowed users to record the start and end times of diary events. Key inputs to use as test conditions for these tests included start times and end times.

I proposed a set of times to try that were designed to provide as much variety as possible from one test case to the next. As inputs into the start and end times, I used a small number of different times spread throughout the morning, afternoon, and evening as well. The strategy I used quickly identified the testing defect the puzzle was designed to uncover in a small handful of tests. What was most memorable about the experience from my perspective was not that I “succeeded” in triggering the bug but that the tests I created triggered a type of bug that was, in Kaner’s words, a “side effect of the tests rather than explicitly planned foci of the tests.”

The business logic in the calendar application that should have identified invalid beginning and end time combinations was coded incorrectly. Instead of using numbers in the business logic, the business logic was ordering the numbers alphabetically. I was not consciously looking to identify that kind of a flaw in the business logic, but by maximizing the variation from test case to test case, I maximized my odds of finding it.

Efficiently achieving structured variation is difficult because it is hard for a human brain to remember whether dozens of different test conditions have been tested together (or we’re accidentally repeating ourselves). This is where pairwise and combinatorial test case generating tools like our Hexawise tool come in. They are designed to achieve as much variation from test case to test case as possible. One of the relatively unsung benefits of this approach is that doing so will help find bugs, like these, that you aren’t even consciously looking for.

Testing Lesson from Malcolm Gladwell’s Blink

John Hunter and Justin Hunter — Mon, 18 Feb 2013 15:34:49 +0000

Malcolm Gladwell’s three best-selling books are filled with fascinating examples of all sorts of things to underscore the interesting points he makes about a wide variety of topics. I’m currently readying his book “Blink,” about how people make process information and make decisions in the ‘blink of an eye.”

The example he shares about an overcrowded hospital Emergency Room’s decision making process really resonated with me. Brendan Reilly, named the chairman of Medicine at Cook’s County Hospital in Chicago in 1996, was in dire need of a way to improve the conditions of the hospital when he showed up there. The Emergency Room faced extreme amounts of resource pressure in particular; there were too many patients needing care and too few doctors and beds to serve them all promptly. One of the most resource-sapping processes was the treatment of patients who entered the ER complaining of chest pains.

“From the beginning, the question of how to deal with heart attacks was front and center.” About 30 people a day came into the ER “were worried that they were having a heart attack. And those thirty used more than their share of beds and nurses and doctors and stayed around a lot longer than other patients.”

“Reilly’s first act was to turn to the work of a cardiologist named Lee Goldman. In the 1970′s, Goldman…p. 133

He… announced that he was holding a bake-off. For the first few months, the staff would use their own judgment in evaluating chest pain, the way they always had. Then they would use Goldman’s algorithm, and the diagnosis and outcome of every patient treated under the two systems would be compared.

For two years, data were collected, and in the end, the result wasn’t even close. Goldman’s rule won hands down in two directions; it was a whopping 70 percent better than the old method at recognizing patients who weren’t actually having a heart attack. at the same time, it was safer. the whole point of chest pain prediction is to make sure that patients end up having major complications are assigned right away to the coronary and intermediate units. Left to their own devices, the doctors guessed right on the most serious patients somewhere between 75 and 89 percent of the time. The algorithm guessed right more than 95 percent of the time…. He went to the ED and changed the rules.”

I suspect that if you asked 100 cardiologists whether they would perform more accurate assessments that the 3 question Goldman algorithm would, you would find that all 100 of those cardiologists would feel confident in their ability to outperform the 3 question algorithm. After all, the algorithm didn’t allow for subject matter expertise.

How is this Relevant to Software Testing?

When many business leaders think of software testing what they want is something that catches bugs before the product is released to the customer. To some extent this can be automated. There are many previous bugs (and experience with bugs in other software) that allow us to create specific tests which will identify known bugs.

So if we have a form on the web application, we can test various scenarios and make sure the form is processed properly: submitting the form when it should or displaying the proper error message when it should do that. It is wise to have test plans to check for these bugs.

Creating simple easy to follow guidelines to address known issues is a wise action. This is true for creating a simple checklist to use in screening for heart attacks and for creating test plans that cover the known issues.

While those are wise they are by no means sufficient. Physicians bring a great deal of expertise to evaluating patients for a huge number of symptoms. And they can use their knowledge and experience to judge where they need more evidence (for example, a diagnostic x-ray or an analysis of a blood sample) to make a judgement. And to judge is software will work as users wish simply following a test plan robotically is not sufficient. The experience and expertise of a software tester allows them to probe the software applications and spot not only bugs but weaknesses that should be addressed so users of of the software have the best experience.

The appropriate methods depend upon the need. And sometimes the expert is not as effective as simply using tools or a check sheet that has been developed specifically for that purpose. It is important to design processes in your organization to find the most effective strategies given the local circumstances.

Experimenting, by using evidence based management practices, is needed to determine what is most effective. Assuming that the best solution is always relying on the experts is not accurate (as the example Gladwell used shows). A great tool to aid in determining what practices work best is the Plan-Do-Study-Act cycle made famous by Dr. W. Edwards Deming.