In general, all scripts (test cases) should have the same steps when using the Hexawise Auto-Scripts feature.

An important consideration in a Hexawise test plan is, "Can I test all these test cases with the same number of steps?" If the answer is no, then you should probably reconsider the scope of your test plan, as it's likely you're trying to include too much testing scope in a single plan.

Another rule of thumb to determine what should be included in the scope of one set of Hexawise tests is to think about a verb and a noun. "Apply for" could be your verb. "A loan" might be your noun. If you have a lot of permutations in which a person could apply for a loan, those scenarios could all fall within the scope of that set of tests. If, however, you started to think about testing the contents of help files, it may well be useful to include those "help file-related" tests in a different set of tests.

Each test case in a single Hexawise test plan will often test the same functionality, but with different permutations. For example, those permutations might include variations of environment (IE or Firefox, using mouse or keyboard, etc.), user (new user, normal user, VIP customer, admin), data (Florida, New York, under 18, over 18) and actions (used the dropdown, keyed it in manually, clicked the confirmation checkbox). The parenthetical examples I provided here come from testing end user software, but the same applies to other types of systems.

If you're testing the same scope (same verb & noun), but identifying all the possible variations, your Auto-Script steps will be the same for all test cases. What might change, of course, is what the tester should expect to happen. Should they see an error dialog or a confirmation dialog? Should the border be green or blue? Should the user get an "X" email or a "Y" email? But we'll save that for another discussion.

By: Sean Johnson on May 7, 2014

Categories: Combinatorial Testing, Hexawise tips

not-possible

What Does it Mean?

One of the more common support inquiries we receive is when a Hexawise generated test case includes "No possible value" for a parameter. The first time you see this, it can be a bit unclear what it means and what you can do to address it.

A "no possible value" in a test case is telling you the test case is providing coverage for a needed pair in some other parameters, and in light of that needed pair your invalid and married pairs are then leaving then no value allowed for the parameter with "no possible value". That sounds confusing, but an example is much easier to understand.

An Example

Let's say we have a test plan with just 3 parameters, each with 2 values:

Fruit: Apple, Pear Car: Toyota, Dodge Dog: Collie, Mutt

And let's further suppose we have 2 invalid pairs:

if Fruit = Apple then Car cannot = Toyota if Car = Dodge then Dog cannot = Mutt

This all seems simple, but a hidden problem lurks in this simple setup. To create 2-way coverage, Hexawise will ensure you've paired every parameter value with every other parameter value (unless a constraint says it shouldn't be paired), which in this case means that Hexawise will necessarily pair Fruit as Apple with Dog as Mutt in at least one test case, since that pairing could be the source of a bug. You probably already see the problem!

In the test case that has Fruit as Apple and Dog as Mutt we need to have a value for the Car parameter. You can't have Car as Toyota, because Apple can't be paired with Toyota, and you can't have Car as Dodge, because Mutt can't be paired with Dodge. So what value can Hexawise provide for Car in this test case? It has no value to provide, so it provides "no possible value".

That's why you can get test cases with "no possible value". Sometimes you can leave them be, sometimes you might want to introduce a "N/A" value for a parameter, and sometimes your invalid and married pairs may need a bit of adjusting. Generally, given the real context of your actual test plan, it is clear which path to take to resolve them.

Dealing With "No Possible Value"

Let's use a more realistic flight reservation example, to show how the context of your real tests indicates what you should do to resolve instances of "no possible value".

Our flight reservation example test plan has these 5 parameters:

Destination: USA, China, Australia Class: First, Business, Economy Reserve a Car: Yes, No Type of Car: Luxury, Economy Payment Method: Credit Card, Frequent Flier Miles, Upgrade Coupons

Without any constraints, we're going to get test cases that pair Reserve a Car as No with Type of Car as Luxury. This makes no sense, and even if a tester were just to ignore it, it hides the fact that the nonsensical test case may be the only test case where we attempt to pay for a Luxury car with Frequent Flier Miles. If this pairing led to a bug, we'd miss the bug if we just ignored the car portion of the generated test case, so it's much better to use constraints to eliminate having Type of Car as Luxury when Reserve a Car is No.

We have options for how to constrain this, but let's suppose we create the following 2 uni-directional married pairs:

When Type of Car = Luxury then Reserve a Car = Yes When Type of Car = Economy then Reserve a Car = Yes

We've taken both our values for Type of Car and married them to Reserve a Car as Yes. Hexawise is still duty-bound to pair Reserve a Car as No with things like Destination as USA and Payment Method as Credit Card. For these test cases there won't be a valid value for Type of Car, so it will get "no possible value". This leads us to our first option for dealing with no possible values.

Option 1: Do nothing! There really is no possible value in that case, and you're OK with it.

If we use option 1, the "no possible value" is intentional. Option 1 is the simplest, but not always our best option. Not every "no possible value" is intentional, many in fact are unintentional, and are a result of inconsistent or missing constraint logic. A "no possible value" therefore tends to be a "quality smell" in a test plan. Meaning... you're not sure something is rotten when you see it, but it sure smells like something might be rotten. So that brings us to another option for handling intentional no possible values.

Option 2: Introduce a "Not Applicable" value.

We can add a third value to Type of Car.

Type of Car: Luxury, Economy, N/A

Now, whenever Reserve a Car is No there will be just 1 value left for Hexawise for Type of Car which is N/A so all those cases of "no possible value" will become "N/A". Problem fixed? Almost, but not quite! Hexawise is now duty-bound to pair Type of Car as N/A with Reserve a Car as Yes. To avoid this new nonsense, we add an invalid pair:

When Reserve a Car = Yes then Type of Car cannot = N/A

Option 1 and option 2 work when you discover the source of the "no possible value" is intentional. But what about when you discover it's not intentional. In that case, the "quality smell" has led us to something that truly is rotten!

Let's suppose we have Class as Business only on international, not domestic flights. Easy enough. Let's add an invalid pair to reflect this business rule.

When Destination = USA then Class cannot = Business

Let's also suppose that an Upgrade Coupon can only be used for Business class, not Economy or First Class. Also easy. Let's add a uni-directional married pair for this:

When Payment Method = Upgrade Coupon then Class = Business

Have you spotted the trouble yet? Hexawise is duty-bound to pair Destination as USA with Payment Method as Upgrade Coupon, since their could be a bug caused by that pairing, and if there is, you want to be sure to find it. In the test case that has this pairing, Hexawise is going to have "no possible value" for Class!

Option 3: Correct your constraint logic.

In this case, our constraint logic is incomplete. We need an invalid pair:

When Destination = USA then Payment Method cannot = Upgrade Coupon

This relieves Hexawise of its duty to pair Destination as USA with Payment Method as Upgrade Coupon and eliminates the "no possible value".

Mysterious "No Possible Value"

For cases of an unitentional "no possible value" where you need to use option 3 to correct an inconsistency in your constraint logic, it's not always clear where the inconsistency is. In fact, it can be downright mysterious. The more constrained your plan is, the harder it can be to find the root cause.

My best advice for tracking down the cause of a "no possible value" in complicated cases is to stay patient and stick with it. Practice makes you better at it. Here are some additional tips:

  • Look at each test case with "no possible values" in isolation, don't try to analyze more than 1 at once
  • Start with the test case with the fewest number of "no possible values" and then fix the issues your analysis turns up, then regenerate your tests and repeat, until you have none left
  • You need pencil and paper! Write out the test case that you're analyzing on paper, then do the analysis from the "Define Inputs" page where it's easiest to see all the possible values for each parameter
  • Take advantage of the hover highlighting of applicable value pairs in the Define Inputs screen. Collapse every section in the left panel except the Paired Values (constraints) so you can see as many as value pairings at one time as possible
  • If you get completely stuck, reach out to a more experienced colleague or ask Hexawise for help

Doing this analysis can be tricky, so having Hexawise do the analysis for you instead is a great idea. We'd like to have a tooltip with an explanation for the no possible value. Since it can be tricky for us humans to determine why a particular no possible value case exists in complicated cases, the algorithm to do the same is also tricky to get right, but we're working toward having a tooltip to explain the cause of the "no possible value", and are going to be very excited when this feature is ready. We might accidentally create a self-aware AI that enslaves the humans in the process though. If we do, don't say we didn't warn you.

In addition to an automated explanation of where the "no possible value" came from, we also know that prevention is better than a cure, so we're working to identify these cases as soon as the constraints are entered so we can prompt you in the moment to correct the issue. We already do this for some of the simpler causes of "no possible value", you may have been prompted by Hexawise already. We're working on Hexawise being able to detect more complicated causes so we can prevent more of these cases early.

In the end, always keep in mind that every "no possible value" is simply a conflict between a parameter value pairing Hexawise is duty-bound to provide to ensure you get 100% coverage, and the constraints you've applied to the plan in the form of married pairs and invalid pairs. With that in mind, have fun "no possible value" hunting.

spock-logical

By: Sean Johnson on May 1, 2014

Categories: Combinatorial Testing, Constraints, Hexawise test case generating tool, Hexawise tips

tl;dr: When you have parameters that only have sensible values depending on certain conditions you should include a value like "N/A" or "Does not appear" for those parameters.

 

You can try this example out yourself using your Hexawise account. If you do not have an account yet you can create a demo account for free that lets you create effective test plans.

Let's take a simple, made up example from version 1 of a restaurant ordering system that has 3 parameters:

Entree: Steak, Chicken, Salmon
Salad: Caesar, House
Side: Fries, Green Beans, Carrots, Broccoli

Everything is just fine with our test plan for version 1, but then let's suppose the business decides that in version 2, people that order "Chicken" don't get a "Salad". Easy enough, we just make an invalid pair between "Chicken" and "Caesar" and "Chicken" and "House", correct? No, Hexawise won't let us. Why? Because then it has no value available for "Salad" to pair with "Chicken" as the "Entree".

But that's what we want! "Salad" will disappear from the order screen as soon as we select "Chicken". So there is no value. That's OK. We just need to add that as the value:

Entree: Steak, Chicken, Salmon
Salad: Caesar, House, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

At this point we could create the invalid pairs between "Chicken" and "Caesar" and "Chicken" and "House", and Hexawise will allow it because there is still a parameter value, "Not Available", left to pair with "Chicken" in the "Salad" parameter.

 

If we do this though, we'll find that Hexawise will force a pairing between "Steak" and "Not Available" and "Salmon" and "Not Available". Not exactly what we wanted! So we can also add an invalid pair between "Steak" and "Not Available" and "Salmon" and "Not Available".

With these four invalid pairs, we have a working test plan for version 2, but rather than the four invalid pairs, this scenario is exactly why Hexawise has bi-directional married pairs. A bi-directional married pair between "Chicken" and "Not Available" tells Hexawise that every time "Entree" is "Chicken", "Salad" must be "Not Available" and every time "Salad" is "Not Available", "Entree" must be "Chicken". So it gives us precisely what we want for this scenario by creating just one bi-directional married pair rather than four invalid pairs.

Now let's suppose version 3 of the menu system comes out, and now there is a fourth Entree, "Pork". And "Pork", being the other white meat, also does not have a salad option:

Entree: Steak, Chicken, Salmon, Pork
Salad: Caesar, House, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

When we go to connect "Entree" as "Pork" and "Salad" as "Not Available" with a bi-directional married pair, Hexawise will rightly stop us. While we can logically say that every time "Entree" is "Chicken", "Salad" is "Not Available" and every time "Entree is Pork", "Salad" is "Not Available", we can't say the reverse. It's nonsensical to say that every time "Salad" is "Not Available", "Entree" is "Chicken" and every time "Salad" is "Not Available", "Entree" is "Pork".

This is precisely why Hexawise has uni-directional married pairs. What we do in this case is create an uni-directional married pair between "Chicken" and "Not Available" which says that every time "Entree" is "Chicken", "Salad" is "Not Available", but it's not the case that every time "Salad" is "Not Available", "Entree" is "Chicken". This of course leaves us free to create a uni-directional married pair between "Pork" and "Not Available". With this design, we're back to Hexawise wanting to pair "Steak" and "Not Available" and "Salmon" and "Not Available" since our uni-directional married pairs don't prohibit that, so we need to add our invalid pairs for those two pairings.

So our final solution for version 3 looks like:

Entree: Steak, Chicken, Salmon, Pork
Salad: Caesar, House, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

Uni-directional Married Pair - Entree:Chicken → Salad:Not Available Uni-directional Married Pair - Entree:Pork → Salad:Not Available Invalid Pair - Entree:Steak ↔ Salad:Not Available Invalid Pair - Entree:Salmon ↔ Salad:Not Available

Let's suppose the specifications for version 4 now hit our desks, and they specify that those that chose the "House" "Salad" get a choice of two dressings, "Ranch" or "Italian". We can then end up with a dependent value that's dependent on another dependent value. That's ok. We've got this!

Entree: Steak, Chicken, Salmon, Pork
Salad: Caesar, House, Not Available
Dressing: Ceasar, Ranch, Italian, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

Uni-directional Married Pair - Entree:Chicken → Salad:Not Available
Uni-directional Married Pair - Entree:Pork → Salad:Not Available
Uni-directional Married Pair - Entree:Chicken → Dressing:Not Available
Uni-directional Married Pair - Entree:Pork → Dressing:Not Available
Bi-directional Married Pair - Salad:Caesar ↔ Dressing:Caesar
Bi-directional Married Pair - Salad:Not Available ↔ Dressing:Not Available
Invalid Pair - Entree:Steak ↔ Salad:Not Available
Invalid Pair - Entree:Salmon ↔ Salad:Not Available
Invalid Pair - Entree:Steak ↔ Dressing:Not Available
Invalid Pair - Entree:Salmon ↔ Dressing:Not Available

Hexawise tests can uncover any pair-wise defects in the identified parameters for version 4 of our hypothetical menu ordering system in just 20 tests out of a possible 192. We just saved ourselves from executing 172 extra tests or missing some defects!

 

Related: How do I create an "Invalid Pair" to prevent impossible to test for Values from appearing together? - How do I prevent certain combinations from appearing using the "Married Pair" feature? - Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values

By: Sean Johnson on Sep 9, 2013

Categories: Hexawise tips, Testing Strategies

The test cases in a given test plan should be sufficiently similar and should not have wildly divergent paths depending on the value of parameters in a test case.

When you do find your test case flow diverges too much you often want to either break your test plan down into a few different test plans, so that you have a plan for each different kind of pass through the system.

Another similar approach is that you may want to decrease the scope of your test plan a bit so that you end up with test cases that are all similar in the plan.

Lastly, let's say the flows aren't wildly divergent, but only slightly so. As a silly example let's say you were testing a recipe that varied based on the fruit selected.

Fruit: Apple, Grape, Orange, Banana

And then you wanted a step for how the peeling was done.

Peeling: By hand, By manual peeling tool, By automated peeler Peeler type: Hand crank, Battery powered, AC powered

Now... our testing flow here has some divergence. Grapes and Apples don't get peeled in this recipe, so they never enter that flow. And Bananas are always peeled by hand so they only get a part of that flow. If this was just the tip of the iceberg of the divergence, we should create a test plan for Grapes and Apples and a different one for Oranges and Bananas.

But if this is the entire extent of the divergent flow, then we want to take advantage of N/A values and married and invalid pairs.

We update our parameter values for the peeling flow to include N/A.

Peeling: By hand, By manual peeling tool, By automated peeler, N/A Peeler type: Hand crank, Battery powered, AC powered, N/A

We marry Grape and Orange (uni-directionally) to the two N/A's so they don't participate in the peeling flow. We marry Banana (unidirectionally) to "By hand" and the 2nd N/A so it has a partial and circumscribed pass through the peeling flow.

Lastly we don't allow Orange to be paired with either N/A with an invalid pair.

That's how a slight flow variation can be accommodated. Please comment with any questions about any of these approaches to your problem.

 

Related: 3 Strategies to Maximize Effectiveness of Your Tests - How Not to Design Pairwise Software Tests - How to Model and Test CRUD Functionality

By: Sean Johnson on May 21, 2013

Categories: Hexawise tips, Scripted Software Testing, Software Testing, Testing Strategies