Problem

This Canadian bank was plagued with high testing costs and inefficiencies across its global QA organization. Within its SDLC, depending on the complexity and size of the project, the test design phase originally occupied up to 25% of any given testing effort schedule. Test leads were left to their own devices to find “creative” ways to optimize and compress their test schedules due to delays in environment issues, code development, and market-timing.

In addition, there was a broad, significant gap in SME knowledge coupled with limited insight into the impacted code. This led to the bank’s teams forcefully adopting a “test everything” mindset - stakeholders and testers believed that sufficient test coverage required one to test every single possible path through a set of requirements. Because of this inefficient and impossible mindset, test design alone corresponded to 30% of the cost of testing for a project on average.

Challenges

Costly test design efforts across QA organization

Slow time-to-market for customer-facing applications

Too many tests in outdated regression suites

Lack of critical testing coverage leading to too many defects caught late

Solution

The bank procured Hexawise to address these critical issues. After purchasing a global, enterprise-wide Hexawise license, stakeholders identified an initial group of analytical users who were to quickly become Hexawise experts. At the same time, nearly a dozen initial projects for the Hexawise implementation were identified based on a need to quickly deliver high-ROI within the first 6 months.

After adopting the Hexawise test design approach within a few weeks, teams sought to use the tool to optimize large, existing test sets within the initial projects. Due to Hexawise’s ability to generate objectively superior tests using its systematic, combinatorial approach to test scenario generation, the teams created sets of tests that were not only smaller, but more thorough as well. The bank initially identified the need to take a more risk-based approach to designing tests. Hexawise not only implemented that approach immediately with its optimized generated tests, but also with its ability to present stakeholders and regulators with clear, precise coverage reports. Using these reports, such as the Coverage Matrix, managers realized the magnitude of Hexawise’s ability to instantly fill in all of the bank’s critical gaps in testing coverage.

Teams used Hexawise to generate fewer tests as well - they no longer had to do their best to “test everything” within their projects. Testing every possible scenario would take years, if not lifetimes, to complete - most projects had millions, billions, or quadrillions of possible scenarios that they could run (what we call “combinatorial explosions”). Instead, Hexawise packed in as much system interaction coverage as possible into each generated test. By exercising their requirements as much as mathematically possible, Hexawise-generated scenarios allowed the projects to ultimately run significantly fewer tests (to still achieve more thorough coverage) for their manual or automated test execution efforts.

The bank's testers then turned to Tricentis Tosca’s codeless test automation capabilities for execution of the optimized Hexawise tests. Hexawise built an integration for Tosca to allow users to seamlessly transition objectively-better, model-based test sets from Hexawise into Tosca’s test automation capabilities (which is now widely available to all Hexawise client companies).