Up to $800K in annualized cost avoidance is delivered at top Canadian bank

Up to $800K in annualized cost avoidance is delivered at top Canadian bank

This bank is the largest in Canada with over CDN$1.7 trillion in assets. The bank secured a global, enterprise-wide Hexawise license in 2021 after being referred to Hexawise by a very satisfied insurance client.

Problem

This Canadian bank was plagued with high testing costs and inefficiencies across its global QA organization. Within its SDLC, depending on the complexity and size of the project, the test design phase originally occupied up to 25% of any given testing effort schedule. Test leads were left to their own devices to find “creative” ways to optimize and compress their test schedules due to delays in environment issues, code development, and market-timing.

In addition, there was a broad, significant gap in SME knowledge coupled with limited insight into the impacted code. This led to the bank’s teams forcefully adopting a “test everything” mindset - stakeholders and testers believed that sufficient test coverage required one to test every single possible path through a set of requirements. Because of this inefficient and impossible mindset, test design alone corresponded to 30% of the cost of testing for a project on average.

Challenges
Costly test design efforts across QA organization
Slow time-to-market for customer-facing applications
Too many tests in outdated regression suites
Lack of critical testing coverage leading to too many defects caught late
Solution

The bank procured Hexawise to address these critical issues. After purchasing a global, enterprise-wide Hexawise license, stakeholders identified an initial group of analytical users who were to quickly become Hexawise experts. At the same time, nearly a dozen initial projects for the Hexawise implementation were identified based on a need to quickly deliver high-ROI within the first 6 months.

After adopting the Hexawise test design approach within a few weeks, teams sought to use the tool to optimize large, existing test sets within the initial projects. Due to Hexawise’s ability to generate objectively superior tests using its systematic, combinatorial approach to test scenario generation, the teams created sets of tests that were not only smaller, but more thorough as well. The bank initially identified the need to take a more risk-based approach to designing tests. Hexawise not only implemented that approach immediately with its optimized generated tests, but also with its ability to present stakeholders and regulators with clear, precise coverage reports. Using these reports, such as the Coverage Matrix, managers realized the magnitude of Hexawise’s ability to instantly fill in all of the bank’s critical gaps in testing coverage.

Teams used Hexawise to generate fewer tests as well - they no longer had to do their best to “test everything” within their projects. Testing every possible scenario would take years, if not lifetimes, to complete - most projects had millions, billions, or quadrillions of possible scenarios that they could run (what we call “combinatorial explosions”). Instead, Hexawise packed in as much system interaction coverage as possible into each generated test. By exercising their requirements as much as mathematically possible, Hexawise-generated scenarios allowed the projects to ultimately run significantly fewer tests (to still achieve more thorough coverage) for their manual or automated test execution efforts.

The bank's testers then turned to Tricentis Tosca’s codeless test automation capabilities for execution of the optimized Hexawise tests. Hexawise built an integration for Tosca to allow users to seamlessly transition objectively-better, model-based test sets from Hexawise into Tosca’s test automation capabilities (which is now widely available to all Hexawise client companies).

Outcomes
25% reduction average in test suite sizes across the bank
Start and exit test efforts early by at least one week on average
Annualized, direct cost avoidance of up to $800K within QA organization

Based on the initial implementation efforts across the dozen projects, the bank saw that Hexawise allowed their test teams to shrink their test suites on average by 25%, while maintaining the same or more test risk coverage. Because of this test design optimization from Hexawise, test teams are now able to start and exit their test efforts early by at least one week in advance.

Across the organization, Hexawise is currently used for...

  • Requirements definition
  • Functional testing
  • User acceptance testing
  • Systems integration testing
  • API testing
  • Non-functional testing
  • Regression testing
  • Agile sprints
  • Test automation efforts, and many other projects

On one of the initial projects, designing tests for a mobile bank transfer application, the team used Hexawise test optimization capabilities and clear coverage reports to realize that in comparison to their previous, manually-selected tests, they could actually achieve the same level of test coverage in 82% fewer tests with Hexawise. In addition, if the team wanted to significantly increase their test coverage by more than double, they could use the same number of tests as before:

Hexawise’s Coverage Matrix allowed them to make these risk-based decisions in real time about their execution efforts. As a result, the gaps in communication between business and technical teams drastically shrunk.

Overall, the bank’s stakeholders translated the savings that were realized within the initial projects to an annualized, direct cost avoidance of up to $800,000 across the spectrum of projects within their Testing Center of Excellence. They are now also able to have accurate conversations with banking regulators to showcase the fact that their testing processes are implementing a quality-first and risk-based approach to running thorough examinations of their applications and systems, thus eliminating any future doubts from the agencies during audits.

Additionally, Hexawise usage continues to exponentially grow at the bank - stakeholders mandate the use of Hexawise for every testing project across the organization. At the conclusion of 2021, there were more than 220 Certified Hexawise Test Design Professionals at this top Canadian bank.

Other case studies you may like
Leading Global Bank

Leading global bank radically improves defect containment
Aine Sherry
Global Test Manager
Hexawise reduces regression tests by 65% at Datalex
Global Bank

Global bank achieves enormous savings in large regression testing suite optimization effort