We have written before about the general question of "Should I Use One Test Plan or Multiple Plans?" This post addresses the same question with a focus on plans that have a relatively large percentage of constraints (e.g., a relatively large number of Invalid Pairs and/or Married Pairs).

Hexawise creates a Plan Scorecard that analyzes every set of tests Hexawise users generate. The Plan Scorecard exists to help you identify potential problems with plans you create and to make you aware of possible ways to improve your sets of tests. One of the notifications the Plan Scorecard provides goes like this:

Consideration: "53% of the parameter values are directly or indirectly constrained."
Explanation: Test plans with more than half of the parameter values constrained are often trying to do too much. They may be better broken into more than one test plan.

Constraints are used to prevent "impossible to test for" scenarios from being generated by the Hexawise test case generator. For more information about entering constraints into Hexawise, see these explanations of Invalid Pairs and Married Pairs. If you do not use Invalid Pairs or Married Pairs in a plan, 0% of your parameter values would be constrained.

What should you do if you get a notification that your plan is highly constrained? Simple: consider your options. Specifically, consider the pro's and con's of splitting the plan you're working on into multiple separate plans.

Why can heavily-constrained plans be a problem?

• A number above 50% or so indicates that it might make sense to consider breaking your plan into 2 or more plans.
• Why? Because with more and more constraints in a plan, keeping track of them all, making sure they're all accurate, and making sure the constraints in certain parts of your plan do not conflict with constraints in other parts of your plan in unintended ways, can start to take a lot of mental energy.
• Furthermore, if your constraints do begin to conflict with one another, that could make it impossible for the Hexawise algorithm to identify valid values to populate in some parts of some of your tests. When that happens, instead of an actual value appearing in a test case, you will see the words "No Possible Value" appear.

Why are multiple simpler plans often preferable to one more complex one?

• It is often much easier and quicker (from a modeling standpoint) to create two different plans. Creating two separate plans instead of one single plan often makes it possible to eliminate the need for the majority of the constraints in your plan.
  • For example, if you had a pizza ordering application where there were a lot of constraints around the value "meat pizza" and a lot of constraints around the value "vegetarian pizza" it could be attractive to create one plan (e.g., one set of tests) for meat pizzas and a different set of tests for veg. pizzas.
• Simpler plans with fewer constraints tend to be easier to understand, modify, and maintain.

What are practical considerations when splitting a single plans into multiple ones?

• To determine where / how to split a plan, begin by asking "what values have the most constraints associated with them?"
  • In the example above, "meat pizza" and "veggie pizza" had the most constraints; creating one plan for meat pizzas and a separate plan for veggie pizzas was the way to go. It would not have made sense to split the plan into one plan with scenarios involving transactions paid for in cash and a different plan with scenarios involving scenarios paid for with credit cards if types of payment type did not have many Invalid Pairs or Married Pairs associated with them.
  • We were recently talking to a client recently where 58% of their plan's parameter values were constrained. We helped them look at where most of the constraints were coming from. It turned out that "Timing of Loan Payment" was the main culprit. As a result, we suggested they consider three separate plans; one plan for Delinquent Payment Scenarios, one for Regular/Timely Loan Payment Scenarios, and one plan for Loan Pre-Payment Scenarios.
  • While working with another client that was dealing with a highly constrained plan, "Type of User" was the source of most of the constraints. Super-Users were allowed to perform all kinds of activities on the System Under Test. "Regular Users" were able to perform a far more limited number of actions. It made sense in that case to break the original combined plan into two separate plans; one plan for Admin User Scenarios and one plan for Regular User scenarios.
• After determining where to split a plan, the next steps tend to be relatively straightforward:
  • If you're starting with one combined plan and want to break it into two plans, we would recommend these steps:
  • Start by creating 3 copies of the same plan:
  • Make a copy of the original combined plan so you can easily go back to an earlier known version if things start to go horribly wrong (or if you realize that the multiple plan strategy results in the creation of significantly more tests than the original single plan version)
  • Make a copy that you will modify for, e.g., "Regular User Scenarios"
  • Make a copy that you will modify for, e.g., "Admin User Scenarios"
  • Take advantage of Hexawise's Bulk Edit feature and tailor each plan as needed.
  • Delete any unnecessary Parameters, Invalid Pairs, Married Pairs, Requirements, and Expected results
  • Add high-priority scenarios as necessary

What disadvantage might there be to multiple plans?

• A potential disadvantage to a multiple plan approach is that it sometimes results in more tests generated than a single test plan approach would.