When you outsource testing responsibilities to a third party, there are several key points to consider beyond the three most frequently asked questions of:

• What is the cost per hour of your services?
• What evidence can you provide of your firm’s level of expertise in our industry and/or the technical skills we feel you’ll need? and
• What is your plan for increasing testing automation? In particular, it is worth developing a thorough understanding how the vendor will identify what kinds of things should be tested, how those things should be tested, and how that information will be communicated to you.

1. How do you ensure testing is performed thoroughly - beyond just “happy path”/ confirmatory tests?

• [Someone] said “I have only confirmed that it works. I haven’t tested it.”
• Beware of answers along the lines of “we have two positive tests and one negative test” for every requirement. That’s too simplistic of an approach that potentially leaves too much of your system untested.
• James Bach: "checking may be very important to do. We are asserting that for checking to be considered good, it must happen in the context of a competent testing process."

• Do you use Exploratory Testing broadly? Would you recommend we use it? Exploratory Testing is a well-established testing approach.

• Champions of exploratory testing include James Bach and Michael Bolton.
• If your vendor hasn’t heard of exploratory testing and starts to explain the disadvantages of undisciplined “ad hoc testing” it’s a warning sign that they might have rigid tunnel vision about what good testing entails. Ad hoc testing is not what exploratory testing is but those that haven't learned about exploratory testing may get the two ideas confused.

• How do you communicate the diminishing marginal returns that exist when you execute sets of test scripts?

• In Hexawise, analyze tests screen is how we help show this concept. That screen shows both pecentage of parameter value pairs tested and the coverage matrix showing the parameter values pairs tested and untested. This view can seen for any specific number of tests along the test plan (using the adjustable bar at the top of the image). This image shows 80% of the paired values have been tested after 10 test cases. Using the slider this view lets you see exactly which parameter value pairs are untested at each point. In this example after 15 tests there is 96% pairwise coverage and 19 tests covers all pairwise combinations.
• Be conscious that your vendor has a strong conflict of interest here. 80% 2-way coverage might be sufficient for your needs on a given project but testing to that level of thoroughness (instead of the complete set of tests) could represent 50% of their billable hours in which case they may go to 100% 2-way coverage (to increase billable hours). Of course, 100% 2-way coverage may be called for, these decisions have to be made with an understanding of the software being tested.
• Implications: ideally, you would like to have a testing vendor that is happy to openly share and discuss the tradeoffs that exist and there strategy to get you the best results based on your situation.

• Do your testers use a structured test design approach such as pairwise test design?

• If so, how many of your testers regularly use pairwise test design or similar methods to design tests?
• During the process of selling you their testing services, vendors have a strong incentive to highlight their efficiency-enhancing capabilities. As soon as the ink dries on the contract though they have an incentive to keep billable hours high.

• Why (and in what types of testing) do they use it?

• Pairwise and combinatorial test design approaches can be widely used in virtually all kinds of software testing.
• If your vendor indicates that they use these approaches only in narrow areas (e.g., most likely functional testing or configuration testing), that is a large red flag that they don’t understand these test design approaches very well at all.
• If your vendor does not understand how to apply these test design methods broadly (including in systems integration testing, performance testing, etc.) then you can safely assume that there will be considerable wasteful redundancy in any areas where these test design approaches are not used.

• If we gave you 50 test scripts we already have, could you generate a set of pairwise tests for us that could test the same system more thoroughly in significantly fewer tests?

• A knowledgeable vendor should be able to analyze your tests and provide a draft set for your review within 24 hours.
• If the vendor takes multiple days and they need to search far and wide for an available expert to create a set of optimized set of tests for you, that indicates that the actual number of testers capable of performing this kind of test design is probably pretty low.
• Another practical way to double-check vendor claims is to search sites like LinkedIn for keywords such as pairwise testing, orthogonal array testing, and/or Hexawise.

Making sure your software testing process is staying current with the best ideas in software testing is an important factor in creating great software solutions that your customers love. Often companies understand the need to stay current on software coding practices that are successful but fewer organizations pay attention to good practices in software testing. This often means there is a good deal to be gained by spending some time to examine and improve your software testing practices.