Global bank achieves enormous savings in large regression testing suite optimization effort
Global bank achieves enormous savings in large regression testing suite optimization effort
As one of the largest banks in the world, this Hexawise client has almost $2 trillion in assets. The bank procured a global, enterprise-wide Hexawise license in 2018. More than 1,700 resources across the globe have signed up for Hexawise since then.
Problem
The global bank had an enormous regression testing suite challenge on a massive scale across the organization. The bank had more than 500,000 regression tests that were manually executed at the time - which took too long to complete and cost the organization too much money. In addition, stakeholders realized that “automating all of the manual tests” would be extremely challenging because they were inefficient tests with poor coverage, wasteful repetition of system ideas, and ambiguous formatting. The tests would take too long to automate because they weren’t even clear enough to be turned over to automation engineers. A different strategy needed to be implemented instead of attempting to “automate everything”.
Challenges
Solution
The bank ultimately took a smarter approach to addressing these challenges - automating only “the right” scenarios, while reworking them into a non-ambiguous, perfectly standardized format. Stakeholders selected Hexawise to meet these goals due to its proven abilities to generate objectively better tests for automation efforts, and its model-based approach to doing so.
A key vendor partner worked with Hexawise to optimize the bank’s existing test suite within its highly-visible automation center of excellence. First, using the vendor partner’s machine learning capabilities, the engagement team was able to accelerate the parameter analysis process. Dedicated resources focused on identifying existing tests within the suite that would deliver the highest return on investment - ones that could be clearly parameterized and translated into parameter models that comprehensively represented system complexities. The team then used those identified parameters to generate objectively better tests with Hexawise using the solution’s systematic approach to scenario generation. Redundancies and inefficiencies were automatically removed from the existing suite by Hexawise, which not only then filled in critical gaps in coverage, but also accomplished this in significantly fewer tests.
The team approached this broad optimization effort in “waves” - strategically choosing similar sets of tests within the suite until the entire effort was complete. Once optimized tests were generated within these waves, Hexawise Automate was used to restructure the bank’s approaches to test automation within its center of excellence. With Automate, the team took a data-driven approach to designing automated tests using the tool’s Gherkin, “Given-When-Then”, Behavior-Driven Development-style scripting methods. Since Hexawise takes a one-to-many approach to automated test generation, fewer scripts would inherently need to be maintained in the future when changes were necessary. In addition, the scripts were designed to provide reusability in the future - when new requirements would be released, teams could pull from existing assets to create new scripts. This would further accelerate the time and effort savings during the test planning and documentation phases.
Outcomes
“Hexawise clearly defines parameters, constraints, and the test scripts that use the parameters. Test cases are also optimized which will in turn reduce our automation efforts.”
More than 500,000 existing tests were optimized to reduce the number of tests needed for automation while simultaneously increasing the test coverage. At the conclusion of the engagement, Hexawise reduced the number of tests needed for automation by 30%. Due to its systematic approach to test generation, the bank was able to achieve 100% coverage within its suite. The bank’s automation specialists then converted the newly-optimized tests and placed them into its proprietary automation suite using customized, skeletal code export options developed by the Hexawise team to match the needs of the bank’s Selenium-based frameworks. Due to the reductions in tests, and the fewer automation scripts needed to maintain, the automation factory was set up for success for new releases in the future.
Across the organization, Hexawise is now used for...
- Requirements definition
- Functional testing
- User acceptance testing
- Systems integration testing
- Non-functional testing
- Regression testing
- Agile sprints
- Test automation efforts, and many other projects
Overall, the bank realized a greater than 25% reduction in testing costs - there were fewer scripts to run and maintain, and the test suite now reached 100% coverage of all critical system interactions. The bank’s stakeholders had significantly higher confidence in its test automation efforts and still uses Hexawise to generate new sets of optimized tests alongside its test automation frameworks throughout the entire organization.
