Some of those using Hexawise use Gherkin as their testing framework. Gherkin is based on using a given [a], when [b] --> then [c] format. The idea is this helps make communication clear and make sure business rules are understood properly. Portions of this post may be a bit confusing for new Hexawise users, links are provided for more details on various topics. But, if you don't need to create output for Gherkin and you are confused you can just skip this post.
A simple Gherkin scenario: Making an ATM withdrawal
Given a regular account
And the account was originally opened at Goliath National
And the account has a balance of $500
When using a Goliath National Bank
And making a withdrawal of $200
Then the withdrawal should be handled appropriately
Hexawise users want to be able to specify the parameters (used in given and when statements) and then import the set of Hexawise generated test cases into a Gherkin style output.
In this example we will use Hexawise sample test plan (Gherkin example), which you can access in your Hexawise account.
I'll get into how to export the Hexawise created test plans so they can be used to create Gherkin data tables below (we do this ourselves at Hexawise).
In the then field we default to an expected value of "the withdrawal should be handled appropriately." This is something that may benefit from some explanation.
If we want to provide exact details on exactly what happens on every variation of parameter values for each test script those have to be manually created. That creates a great deal of work that has very little value. And it is an expensive way to manage for the long term as each of those has to be updated every time. So in general using a "behaves as expected" default value is best and then providing extra details when worthwhile.
For some people, this way of thinking can be a bit difficult to take in at first and they have to keep reminding themselves how to best use Hexawise to improve efficiency and effectiveness.
To enter the default expected value mouse-over the final step in the auto scripts screen. When you mouse over that step you will see the "Add Expected Results" link. Click that and add your expected result text.
The expect value entered on the last step with no conditions (the when drop down box is blank) will be the default value used for the export (and therefor the one imported into Gherkin).
In those cases when providing special notes to tester are deemed worth the extra effort, Hexawise has 2 ways of doing this. In the event a special expected value exists for the particular conditions in the individual test case then the special expected value content will be exported (and therefore used for Gherkin).
Conditional expected results can be entered using the auto scripts feature.
Or we can use the requirements feature when we want to require a specific set of parameter values to be tested. If we chose 2 way coverage (the default, pairwise coverage) every pair of parameter values will be tested at least once.
But if we wanted a specific set of say 3 exact parameter values ([account type] = VIP, [withdrawal ATM] = bank-owned ATM, [withdrawal amount] = $600 then we need to include that as a requirement. Each required test script added also includes the option to include an expected result. The sample plan includes a required test case with those parameters and an expected result of "The normal limit of $400 is raised to $600 in the special case of a VIP account using a Goliath National Bank owned ATM."
So, the most effective way to use Hexawise to create a pairwise (or higher) test plan to then use to create Gherkin data tables will be to have the then case be similar to "behaves as expected." And when there is a need for special expected results details to use the auto script or requirements features to include those details. Doing so will result the expected result entered for that special case being the value used in the Gherkin table for then.
When you click auto script button the test are then generated, you can download them using the export icon.
Then select option to download as csv file.
You will download a zip file that you can then unzip to get 2 folders with various files. The file you want to use for this is the combinations.txt file in the csv directory.
The Ruby code we use to convert the commas to pipes | used for Gherkin is
tests = CSV.read("combinations.csv")
table = 
tests.each do |test|
table << "| " + test[1..-1].join(" | ") + " |\n"
Of course, you can use whatever method to convert the format you wish, this is just what we use. See this explanation for a few more details on the process.
Now you have your Gherkin file to use however you wish. And as the code is changed over time (perhaps adding parameter value options, new parameters, etc.) you can just regenerate the test plan and export it. Then convert it and the updated Gherkin test plan is available.
Related: Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - Designing a Test Plan with Dependent Parameter Values
By: John Hunter
Coining a New Term
I'm coining a new term today, "grapefruit juice bugs."
My inspiration for this term is a blog post in the New York Times that David Pogue wrote. I was fascinated by the post and it got me to thinking about a particular kind of bugs in software that are more common than most people may realize. You could say that these bugs are surprisingly common. In fact, if you wanted to be more precise, you could even say that this term applies to a specific type of "surprisingly common type of surprising bugs." Let me explain.
There's something about the chemical makeup of grapefruit juice that makes it interact with our biology and a large number of different drugs in ways which result in dangerous conditions. For example, certain drugs lose their effectiveness dramatically when interacting with grapefruit juice which can have life-threatening consequences. Other times, the interactions with grapefruit juice can dramatically increase a drug's potency. This can result in "safe doses" becoming very unsafe.
Grapefruit Is a Culprit in More Drug Reactions
The 42-year-old was barely responding when her husband brought her to the emergency room. Her heart rate was slowing, and her blood pressure was falling. Doctors had to insert a breathing tube, and then a pacemaker, to revive her.
They were mystified: The patient’s husband said she suffered from migraines and was taking a blood pressure drug called verapamil to help prevent the headaches. But blood tests showed she had an alarming amount of the drug in her system, five times the safe level.
Did she overdose? Was she trying to commit suicide? It was only after she recovered that doctors were able to piece the story together.
“The culprit was grapefruit juice,” said Dr. Unni Pillai, a nephrologist in St. Louis, Mo. ...
The previous week, she had been subsisting mainly on grapefruit juice. Then she took verapamil, one of dozens of drugs whose potency is dramatically increased if taken with grapefruit. In her case, the interaction was life-threatening.
Last month, Dr. David Bailey, a Canadian researcher who first described this interaction more than two decades ago, released an updated list of medications affected by grapefruit. There are now 85 such drugs on the market, he noted, including common cholesterol-lowering drugs, new anticancer agents, and some synthetic opiates and psychiatric drugs, as well as certain immunosuppressant medications taken by organ transplant patients, some AIDS medications, and some birth control pills and estrogen treatments.
Under normal circumstances, the drugs are metabolized in the gastrointestinal tract, and relatively little is absorbed, because an enzyme in the gut called CYP3A4 deactivates them. But grapefruit contains natural chemicals called furanocoumarins, that inhibit the enzyme, and without it the gut absorbs much more of a drug and blood levels rise dramatically.
For example, someone taking simvastatin (brand name Zocor) who also drinks a small 200-milliliter, or 6.7 ounces, glass of grapefruit juice once a day for three days could see blood levels of the drug triple, increasing the risk for rhabdomyolysis, a breakdown of muscle that can cause kidney damage.
By: Justin Hunter
Hexawise has had another great year and we owe it to you, our users. Thank you!
As a result of your input and word-of-mouth recommendations, in 2013, the Hexawise test design tool:
“Working coaching session with customer today. Huge data/config matrix was making them weep. Stunned silence when I showed them @Hexawise :)”
“That would be @Hexawise & combinatorial testing to the rescue once again. #Thanks”
“Freaking awesome visualisation of test data coverage. Kind courtesy of @Hexawise at Moolya!”
“Using @Hexawise combinatorial scenarios for e-commerce basket conditions. Team suitably impressed by speed and breadth of analysis. #Win”
“Just discovered Hexawise today, brilliant tool for creating test cases based on coverage of many variables.”
“This changes everything.”
Using Hexawise is one of the highest ROI software development practices in the world.
By: Justin Hunter
A common mistake software companies make is creating products where features built for advanced users overwhelm and confuse average users. At Hexawise we have focused on creating a great experience for average and new users while providing advanced users powerful options.
How to Avoid a Common Product Mistake Many Teams Make by Mark Suster
The single biggest mistake most product teams make is building technology for what they believe the user would want rather than what the actual end-user needs.
My philosophy is simply. You design your product for the non-technologist. The “Normal.”
Give people fewer options, fewer choices to make. Give them the bare essentials. Design products for your mom.
power users will always find the features they need in your product even if they’re hidden away. Give them a configure button somewhere. From there give them options to soup up the product and add the complexity that goes with newfound power.
The process used to hire employees is inefficient in general and even more inefficient for knowledge work. Justin Hunter, Hexawise CEO, posted the following tweet:
The labor market is highly inefficient for software testers. Many excellent testers are undervalued relative to average testers. Agree?
Hire and promote first on the basis of integrity; second, motivation; third, capacity; fourth, understanding; fifth, knowledge; and last and least, experience. Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind. Experience is easy to provide and quickly put to good use by people with all the other qualities.
By: John Hunter
The Hexawise Software Testing blog carnival focuses on sharing interesting and useful blog posts related to software testing.
By: John Hunter
Categories: Software Testing,
So, we've been experimenting with a live customer support feature in our tool lately. We're rolling out the live chat support on a beta basis to see how useful our users find it.
The motivations for creating it were two-fold. Our first motivation was the Mayday button that Amazon Kindle announced recently. How cool is that, right? Live support available on demand any time you want it! Ingenius.
Our second motivation for building a live chat support feature into our tool is that - while software test designers consistently tell us that they find Hexawise's features to be really easy to use - new users will often encounter test design questions as they start using the tool. We wanted to be available instantly to collaborate with them - and help them address questions in real time, like: "How should I be thinking about different ways of defining equivalence classes?" "Given what I'm trying to test in my system, how much detail is too much in this context?" etc. We wanted to be there to help users answer them. We're obsessive about customer service. Having the opportunity to have our expert test designers be a click away from every user of our tool every time they encounter a question. That's just too good an opportunity to pass up.
Early indications of how useful this service is to users are extremely promising. Users are telling us it is an amazingly helpful service. And, while we were worried that we might start to feel too stretched with tons of user questions to answer, we haven't felt that way at all. Interactions have been at manageable volumes. We've found them to be really positive. Many of the interactions have helped us learn about ways to improve our tool and/or how we can make certain test design concepts easier for Hexawise users to understand. Often, customers will click on a "call me" button to talk through questions live by phone rather than by typing back and forth. I'm glad this next conversation was done with keyboards.
This conversation happened about 45 minutes ago. Everyone at Hexawise headquarters is still smiling broadly. It made all of our days and stands apart from the rest. Enjoy!
(17:15:25) Visitor Hi
(17:15:40) Sean Johnson Hey
(17:15:44) Sean Johnson What's up?
(17:15:51) Visitor Hi Sean!
(17:15:57) Visitor Hey, I created 308 test cases
(17:16:02) Sean Johnson k
(17:16:02) Visitor out of a possible 18 trillion
(17:16:07) Sean Johnson nice!
(17:16:11) Visitor I consolidated all my user stories in 6 sprints total
(17:16:13) Visitor to 1 test set
(17:16:14) Visitor which is fine
(17:16:28) Visitor but I noticed from test case # 100 plus to 308
(17:16:37) Visitor most of my test cases are now 'any value'
(17:16:48) Visitor I was wondering if there's an option to force hexawise to pick a value for me
(17:16:52) Visitor but I don't there is
(17:16:56) Visitor but that would be a good enhancement
(17:17:05) Sean Johnson ha! are you spying on me?
(17:17:09) Visitor for 'any value' you can have the app just pick a random one
(17:17:10) Visitor LOL
(17:17:11) Visitor Nope
(17:17:14) Sean Johnson seriously… that's what I'm working on right now
(17:17:19) Visitor NO WAY!
(17:17:20) Sean Johnson what are the odds?
(17:17:24) Visitor O M G
(17:17:26) Sean Johnson yes way
(17:17:33) Visitor I've been meaning to provide that feedback 2 weeks ago
(17:17:39) Visitor but never took the time!
(17:17:46) Visitor I WOULD LOVE TO HAVE THAT FEATURE!
(17:17:49) Visitor Oh
(17:17:51) Visitor in the mean time
(17:17:56) Visitor my testers workaround
(17:18:00) Visitor is to print the test plan
(17:18:17) Visitor and just pick the values randomly from the value expansion list and input parameter list
(17:18:26) Visitor AWESOME Sean!
(17:18:31) Visitor Well let me know when it's available
(17:18:41) Sean Johnson that's really crazy
(17:18:41) Sean Johnson well… I guess I'm working on the right thing!
(17:18:42) Sean Johnson Will tomorrow be soon enough for you?
(17:18:42) Sean Johnson :-)
(17:18:43) Sean Johnson for now… it's going to be hidden
(17:18:43) Sean Johnson you'll add ?full=true
(17:18:43) Sean Johnson to the end of the URL
(17:18:57) Sean Johnson and that'll force Hexawise to fill in the any_values
(17:19:03) Visitor that's AWESOME!
(17:19:06) Visitor I will do the workaround
(17:19:09) Visitor OMG, you made my day!
(17:19:13) Visitor THANKS A TON!
(17:19:15) Visitor :-)
(17:19:18) Sean Johnson I'll send you note this evening or in the morning when it's live on [your company's Hexawise instance]
(17:19:23) Visitor I am so happy
(17:19:24) Visitor LOL
(17:19:28) Visitor Thanks so much
(17:19:32) Sean Johnson thanks for chatting! made my day to know I picked the right next priority.
(17:19:32) Visitor this will make my life easier
(17:19:38) Visitor Oh yeah, totally
(17:19:38) Sean Johnson excellent.
(17:19:58) Visitor I honestly think I'm not the only one who will appreciate this enhancement
(17:20:01) Visitor you guys are the best!
(17:20:03) Visitor Thanks so much
(17:20:05) Sean Johnson :-) we try
(17:20:15) Visitor You all do an amazing job
(17:20:21) Visitor this tool is the best
(17:20:21) Sean Johnson look for an email from me shortly
(17:20:26) Visitor will do
(17:20:27) Visitor thanks!
(17:20:33) Sean Johnson thanks!
By: Justin Hunter
Since creating Hexawise, I've worked with executives at companies around the world who have found themselves convinced in the value of pairwise testing. And then they need to convince their organization of the value.
They often follow the following path: first thinking "pairwise testing is a nice method in theory, but not applicable in our case" then "pairwise is nice in theory and might be applicable in our case" to "pairwise is applicable in our case" and finally "how do I convince my organization."
In this post I review my history helping convince organizations to try and then adopt pairwise, and combinatorial, software testing methods.
About 8 years ago, I was working at a large systems integration firm and was asked to help the firm differentiate its testing offerings from the testing services provided by other firms.
While I admittedly did not know much about software testing then but by happy coincidence, my father was a leading expert in the field of Design of Experiments. Design of Experiments is a field that has applicability in many areas (including agriculture, advertising, manufacturing, etc.) The purpose of Design of Experiments is to provide people with tools and approaches to help people learn as much actionable information as possible in as few tests as possible.
I Googled "Design of Experiments Software Testing." That search led me to Dr. Madhav Phadke (who, by coincidence, had been a former student of my father). More than 20 years ago now, Dr. Phadke and his colleagues at ATT Bell Labs had asked the question you're asking now. They did an experiment using pairwise test design / orthogonal array test design to identify a subset of tests for ATT's StarMail system. The results of that testing effort were extraordinarily successful and well-documented.
Shortly after doing that, while working at that systems integration firm, I began to advocate to anyone and everyone who would listen that designing approach to designing tests promised to be both (a) more thorough and (b) require (in most but not all cases) significantly fewer tests. Results from 17 straight projects confirmed that both of these statements were true. Consistently.
Repeatable Steps to Confirm Whether This Approach Delivers Efficiency and Thoroughness Improvement (and/or document a business case/ROI calculation)
How did we demonstrate that this test design approach led to both more thorough testing and much more efficient testing? We followed these steps:
I have already elaborated some test plans that would save us up to 50% effort with that method. But now my boss and other colleagues are asking me for a proof that these pairwise test cases suffice to make sure our software is running well.
By: Justin Hunter
Software testers should be test pilots. Too many people think software testing is the pre-flight checklist an airline pilot uses.
The checklists airline pilots use before each flight are critical. Checklists are extremely valuable tools that help assure steps in a process are followed. Checklists are valuable in many professions. The Checklist – If something so simple can transform intensive care, what else can it do? by Atul Gawande
Sick people are phenomenally more various than airplanes. A study of forty-one thousand trauma patients—just trauma patients—found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations for teams to attend to. That’s like having 32,261 kinds of airplane to land. Mapping out the proper steps for each is not possible, and physicians have been skeptical that a piece of paper with a bunch of little boxes would improve matters much. In 2001, though, a critical-care specialist at Johns Hopkins Hospital named Peter Pronovost decided to give it a try.
Pronovost and his colleagues monitored what happened for a year afterward. The results were so dramatic that they weren’t sure whether to believe them: the ten-day line-infection rate went from eleven per cent to zero. So they followed patients for fifteen more months. Only two line infections occurred during the entire period. They calculated that, in this one hospital, the checklist had prevented forty-three infections and eight deaths, and saved two million dollars in costs.
Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.
(A test is an instance of testing.)
Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
By: John Hunter
Hexawise allows you to adjust testing coverage to focus more thorough coverage on selected, high-priority areas. Mixed strength test plans allow you to select different levels of coverage for different parameters.
Increasing from pairwise to "trips" (3 way) coverage increases the test plan so that bugs that are the results of 3 parameters interacting can be found. That is a good thing. But the tradeoff is that it requires more tests to catch the interactions.
The mixed-strength option that Hexawise provides allow you to do is select a higher coverage level for some parameters in your test plan. That lets you control the balance between increased test thoroughness with the workload created by additional tests.
By: John Hunter