Carrie Puterbaugh's presentation at the Twin Cities Quality Assurance Association on using Hexawise to improve software testing at her organization (a large bank).

In one example she discusses in the video Carrie's team used Hexawise to create an optimized test suite and provided that to the software vendor to have them run it prior to delivering the software to her bank. In this example historically they vendor was finding 67% of the defects and Carrie's bank was finding 33%. Now that the vendor is using the Hexawise test suite the vendor is finding 98.5% of the defects and fixing them prior to delivering the software to Carrie's bank.

Based on these results her team was able to move staff off of testing this application and onto other testing needs of the organization. They are saving 90% of what they used to spend on QA on this project.

Another project she talked about was a high priority and high risk release that they used Hexawise on and achieved the highest quality software release they have ever had.

It was great... We were able to go to management and say "we reduced the amount of test cases we ran and we got a better quality application.

Carrie discussed how valuable Hexawise was to improving the regression testing they must do on their large number of applications. She also mentioned how much value Hexawise added by presenting software testing plan information in an easy to visualize way that greatly improved the discussions between her (as the software testing manager) and the business executives she interacts with.

Related: Large Benefits = Happy Hexawise Clients and Happy Colleagues - Kathleen Poulsen of Fidelity Investments on Using Hexawise to Improve Software Testing Results - How to Pack More Coverage Into Fewer Software Tests

By: John Hunter on Oct 3, 2017

Categories: Business Case, Customer Success, Hexawise, Pairwise Testing, ROI, Software Testing Presentations, Testing Case Studies

We are excited to announce an ongoing partnership with Datalex to improve software test efficiency and accuracy. Datalex has achieved extreme benefits in software quality assurance and speed-to-market through their use of Hexawise. Some of these benefits include:

  • Greater than 65 percent reduction in the Datalex test suite.
  • Clearer understanding of test coverage
  • Higher confidence in the thoroughness of software tests.
  • Complete and consistently formatted tests that are simple to automate

An airline company’s regression suite typically contains thousands of test cases. Hexawise is used by Datalex to optimize these test cases, leading to fewer tests as well as greater overall testing coverage. Hexawise also provides Datalex with a complete understanding of exactly what has been tested after each and every test, allowing them to make fact-based decisions about how much testing is enough on each project.

Hexawise has been fundamental in improving the way we approach our Test Design, Test Coverage and Test Execution at Datalex... My team love using Hexawise given its intuitive interface and it’s ability to provide a risk based approach to coverage which gives them more confidence during release sign-off.

Screen shot 2016 10 04 at 12.58.53 pm

Áine Sherry

Global Test Manager at Datalex

“As a senior Engineer in a highly innovative company, I find Hexawise crucial in regards to achieving excellent coverage with a fraction of the time and effort. Hexawise will also facilitate us to scale onwards and upwards as we continue to innovate and grow,“ – Dean Richardson, Software Test Engineer at Datalex.

By eliminating duplicative tests and optimizing the test coverage of each test case Hexawise provides great time savings in the test execution phase. Hexawise can generate fewer test scenarios compared to what testers would create on their own and those test cases provide more test coverage. Time savings in test execution come about simply because it takes less time to execute fewer tests.

Related: How to Pack More Coverage Into Fewer Software Tests - Large Benefits = Happy Hexawise Clients and Happy Colleagues

By: John Hunter on Nov 10, 2016

Categories: Business Case, Customer Success, Testing Case Studies

Hexawise - More Coverage Fewer Tests

Testers who use Hexawise consistently pack significantly more coverage into fewer software tests. It might seem counterintuitive to people unfamiliar with pairwise and orthogonal array-based testing that more thorough coverage can be achieved while using fewer tests, but this is a clear and well-established fact. This article explains how Hexawise consistently achieve superior coverage even while using fewer tests.

Time savings and thoroughness improvements achieved by testers using Hexawise at one of our insurance clients recently are typical. Let’s quickly address the first two benefits before diving deeper into a detailed explanation of how testers achieved the thoroughness improvements described in the third benefit.

Hexawise - Benefits Findings

The time savings in the test selection and documentation phase (shown in the top box) are easy enough to understand. Testers using Hexawise save time by automating selection and documentation steps that they would otherwise have to complete manually.

Similarly, the time savings in the test execution phase (shown in the middle box) are equally straightforward. Hexawise can generate fewer test scenarios compared to what testers would create on their own. Time savings in test execution come about simply because it takes less time to execute fewer tests.

So far so good. But how exactly do testers using Hexawise consistently achieve superior coverage while using fewer software tests? By helping testers generate optimized test sets without wasteful redundancies minimized, with the maximum amount of variation between scenarios, and with systematic coverage potential defects that could be caused by interactions.

Hexawise - Specific Benefits

Hexawise Minimizes Wasteful Repetition The powerful test generation algorithm inside of Hexawise systematically eliminates all wasteful repetition from every test scenario. If a given combination of test conditions has already appeared together in a test, other combinations of values will be found by the test generation algorithm and used instead of the wastefully repetitive combination. Even it if means that Hexawise’s blazingly-fast optimization algorithm needs to explore thousands of combinations of candidate values to achieve this goal. With wasteful repetition eliminated, Hexawise test sets require fewer tests to achieve thorough testing.

Hexawise Minimizes Wasteful Repetition

Hexawise Maximizes Variation Between Tests. If you take a close look at any Hexawise-generated set of tests, you will notice that variation is maximized as much as scientifically possible from one test to the next. This is the beneficial flip side of the repetition-minimization coin. Useful variation from test to test is the thoroughness-improving outcome whenever wasteful repetition is eliminated. When testers start to execute tests that explore new combinations of values and new paths through applications, they find more defects.

Hexawise Maximizes Variation Between Tests

Superior, Systematic Coverage. Interactions between different test inputs are a major source of defects. As a result, interactions between inputs are important to test thoroughly and systematically. Testers using Hexawise use a “coverage dial” in Hexawise to determine the coverage strength they would like for a given set of tests. From there, Hexawise’s test optimization algorithms systematically detect all potential interactions that are in scope to be tested for, and Hexawise ensures tests are carefully constructed to close every such potential coverage gap. Doing this kind of analysis by hand, even using tools like Excel is time-consuming, impractical, and error-prone. There are simply too many interactions for a tester to keep track of on their own. As a result, manually-selected test sets almost always fail to test for a rather large number of potentially important interactions. In contrast, the Hexawise test optimization algorithm systematically eliminates gaps in testing coverage that manually-selected test sets routinely fail to cover. Compare the coverage achieved by the project’s 37 manually-selected “business as usual” tests (above) to the more compact, efficient, and thorough set of 25 Hexawise-generated set of tests (below).

Gaps in Coverage Hexawise Optimized Coverage

In short, when testers select scenarios by hand, the outcome is typically too many tests that took too long to select and document, contain too much wasteful redundancy, and have an unknown number of potentially-serious gaps in coverage. In contrast, when testers use Hexawise to generate optimized sets of tests, they quickly generate unusually thorough sets of highly-varied tests at the push of a button that systematically achieve user-specified thoroughness goals, And testers can communicate coverage achieved by their test sets with stakeholders more clearly than ever before.

By: Justin Hunter on Dec 15, 2015

Categories: Pairwise Software Testing, Pairwise Testing, Combinatorial Testing, Business Case

Few things make us happier at Hexawise than hearing reports from clients about how much Hexawise is helping them improve their software testing efforts.

 

Happy

 

A recent conversation with our newest international banking client is a case in point. Carrie, a senior testing manager who is leading adoption efforts at the bank, reported the following impressive benefits:

 

Project 1

Without Hexawise: estimated testing effort for the project = 8.5 man months

With Hexawise: estimated testing effort for the project = 1.5 man months

Savings = > 80%

Hexawise Case Studies 2016 01 23

 

Project 2

Without Hexawise (e.g. immediately prior release): defects found during testing = 67%; defects found in UAT = 33%

With Hexawise (most recent release): defects found during testing = 98.5%; defects found in UAT = 1.5%

Defect Removal Effectiveness Improvement = Stunning

Hexawise Case Studies 2016 01 23 2

 

A few interesting things are worth pointing out as context behind these results.

 

First, let's be clear: these are significantly higher than normal Hexawise-generated benefits. We're not suggesting that every project will see benefits this large. They won't. "Your milage may vary." The 80% reduction in testing time is not unheard of but definitely larger than most teams tend to see. Similarly, the massive improvement to defect removal efficiency is larger than typically occurs. These are more typical benefits from case studies using Hexawise's pairwise testing methods and/or combinatorial testing methods and/or Orthogonal Array OATS testing methods.

 

Second, the test designers involved in these two projects are significantly more talented and skilled than "average" software testers working at banks. The test designers' skill has a lot to do with the unusually large successes they achieved in these projects. We know about how talented they are because we worked closely with these test designers during a 4-day onsite instructor-led test design training program we led and we have a good sense of the "average" test design skills possessed by software testers because we regularly conduct software test design training sessions around the world at hundreds of companies. During the hands-on interactive test design exercises in our face-to-face training sessions with the bank's software testers, several of their test designers demonstrated exceptional analytical thinking and problem solving skills.

 

Third, as we try to do with all of our new clients, our test design experts have actively kept in touch with the bank's test designers since the initial onsite training took place. We have been answering their test design questions when they have arisen, offering to review their draft test tests, and jumping on ad hoc screen sharing sessions to explain/demonstrate how to use Hexawise test design features. This helps our clients maximize the value they obtain from using Hexawise and helps us stay closely attuned to real-world testing challenges so that we can continuously improve our tool and fine-tune our software test design training messages.

 

If you're using Hexawise and have experiences to share with us, whether good or bad, we would love to hear about them. We're here to help. As corny as it sounds, helping clients succeed is a huge part of what motivates us at Hexawise. Please contact us at: support@hexawise.com

By: Justin Hunter on Sep 10, 2015

Categories: User Experience, Pairwise Testing, Combinatorial Testing, Business Case

Hexawise helps team achieve the following qualitative benefits:

Hexawise Benefits MM

Not all of the benefits above can easily be quantified. So what should you do if you are tasked with creating a business case to support adoption of Hexawise? Simple:

  • Measure what happens when you create ~50 tests with your business as usual process; compare that to how long it takes to generate a set of ~25 tests with Hexawise.
  • Measure how long it takes to execute each sets of tests.
  • Measure how many unique defects you find executing each set of tests.
  • For typical findings, see these empirical benefit measurement studies.
  • Then put together a business case summary like the one below.
    • The summary focuses primarily on the objectively measurable efficiency savings you measured.
    • The summary also breaks out valuable qualitative benefits into separate line items (to prevent those valuable - but difficult to quantify - benefits from being forgotten about).

Hexawise - Finanacial Benefit Model 2013.10.xlsx at 2.23.05 PM

By: Justin Hunter on Aug 11, 2014

Categories: Business Case, Hexawise, ROI, Combinatorial Testing, Pairwise Testing