Problem

The insurance company began a multi-year transformation across the organization, combining all of its operating companies under one central platform, Guidewire, and needed to thoroughly test their massive complex system while working under aggressive delivery schedules. To state the obvious, this was a daunting task. QA leadership quickly realized that their testing efforts were going to encounter enormous “combinatorial explosions” with every release. When testing complex insurance processes, from quoting and binding (with complex discount rules), to claims and account maintenance activities, it can be difficult to achieve the right balance between important conflicting objectives:

For major insurance processes, system integration testing literally involves trillions of possible scenarios. Conducting end-to-end testing that includes multiple systems and multiple processes from quoting & binding through account closure will involve more possible tests than there are atoms in the universe. The company needed a solution to enable them to confidently answer important questions. Given the non-negotiable requirement that these complex systems must be tested very thoroughly (because more than $1 billion dollars of business will be conducted using them every month),

• How much testing will be “enough” to test the systems thoroughly?
• How will our teams be able to consistently and precisely report exactly what is (and is not) being covered by their test sets?
• How can test sets be optimized so that every test reveals the absolute maximum amount of new system information?
• How can teams systematically eliminate 100% of wasteful repetition from test sets?

According to key stakeholders, the company originally faced “serious issues” with too many defects found in production at the onset of the Guidewire transformation for high-profile, client-facing applications. There were just too many integration points to try to cover using teams’ manual approaches for test creation for their complex systems. In addition, the application teams were spending too much time manually deciding which tests to execute, leading to costly inefficiencies and release delays.

Challenges

Complex Guidewire transformation

Too much time creating tests

Unclear testing coverage for stakeholders

Defect slippage into SIT and UAT

Solution

The insurance company secured an enterprise-wide license of Hexawise to realize significant cost savings during the testing phases across the major, enterprise-wide project. Executives in the IT organization realized they could no longer continue down a path which was bound to leave too many testing coverage gaps for all of the integration points during the replatforming effort. Given the complexity of their testing challenges, in addition to securing an enterprise license and Hexawise’s standard training and rollout assistance, 'the company enlisted the support of Hexawise test design experts to play an active role in working side-by-side with its own resources to design their optimized end-to-end test sets.

The Guidewire Policy Center application team used Hexawise’s optimized, instant test generation capabilities to streamline their test planning processes. Once critical test parameters were defined, Hexawise instantly generated objectively better test scenarios for the team. In addition, using Hexawise’s one-to-many scripting capabilities, the team realized significant time and effort savings in documenting and maintaining their test steps.

Hexawise was also used to ensure quality from the onset of each sprint. To achieve more thorough testing coverage, the Policy Center team used Hexawise's systematic test generation approach to cover all critical system interactions in as mathematically few tests as possible. The team realized that it was impossible to do this on their own using previous, manual methods of test case creation. Hexawise made it possible to accomplish this with the click of a button.

The new Hexawise-generated test sets were much more thorough than their previous sets of manually-selected tests and Hexawise coverage charts made the thoroughness improvements immediately obvious to key stakeholders and business analysts.

Outcomes

>50% time savings in test planning

>70% reduction in production defects

Higher quality releases for the multi-year transformation

On average, the implementation teams realized greater than 50% time savings with Hexawise during the test planning and creation phases of their efforts. For example, within Guidewire Policy Center, the team assigned to the Defensive Driver Discount application analyzed the time spent during the test planning phase, documenting time spent using previous, manual methods, and time spent using the Hexawise approach:

Although the team spent more time with Hexawise during the initial analysis phase (shown in blue), transformational time savings were achieved - overall, it took the team 65% less time to design and document better tests with Hexawise, which in turn led to a faster time to market for the end product.

This particular Policy Center team also analyzed increases in the amount of defects found earlier in the testing lifecycle with Hexawise, which ultimately was common across the majority of projects at the company:

The team compared the defect finding effectiveness of “business as usual tests” to the Intelligently Augmented tests they generated using Hexawise. In project after project where defect-finding analyses were conducted, the results were identical: tests generated by Hexawise consistently found more defects. In this particular project, 19 defects found in production went undetected by the team’s hand-selected test set. In comparison, running optimized Hexawise tests, only 4 of those defects slipped past the test set. The reason that the Hexawise tests were dramatically more effective than the hand-selected tests is that the Hexawise test set benefitted from Hexawise’s famous Intelligent Augmentation which assured that every single critical system interaction was covered. The reason that neither test set was able to detect the final 4 defects is that they were triggered by variation ideas that did not appear in either set of tests. As soon as those ideas were identified and reincorporated into the model, they were immediately addressed - and those non-obvious variation ideas were automatically included in all future test sets generated for that system.

Although these are just two examples of the value realized with Hexawise during the major transformation, these benefits quickly spread throughout the entire replatforming effort, ultimately turning an originally complex and chaotic testing project into a systematic and thorough global quality engineering effort. All operating company systems were consolidated under a centralized, Guidewire Policy Center-based platform, ensuring all points of integration are tested as thoroughly (and efficiently) as scientifically possible.

“There is no way we would have been able to handle the magnitude of complexity for this company-wide transformation without Hexawise. Thanks to Hexawise, we achieved the coverage we needed in a much more manageable number of scenarios.”