Illinois Tool Works Inc., or ITW, is a Fortune 200 company that produces engineered fasteners and components, equipment and consumable systems, and specialty products. ITW has operations in 58 countries that employ approximately 60,000 women and men. The company holds a broad portfolio of nearly 12,000 active patents.
As part of the ITW Building Component Group’s renewed commitment to producing high quality software, the team embarked on a “process overhaul” of the software development lifecycle. There was a targeted process improvement in the area of validation. It was determined that a tool that would provide test design and coverage capabilities was necessary.
Before using Hexawise, the Building Components Group at ITW had a test bank of 3,000 manual and automated tests executed nightly to ensure one of their flagship products was regression free. These tests were written over a decade ago and their maintenance was neglected even though nearly 100,000 lines of code were altered every year. Additionally, ITW was left guessing their percentage of completion if tests failed to run, leaving the team ill-equipped to understand their gaps in testing.
The team at ITW Building Components Group knew something had to change. A 10-person team overseeing 5.5 million lines of code that added, modified or deleted 20-30 thousand lines every quarter could not possibly test everything. Luckily, Scott had an ace up his sleeve. He had been exposed to combinatorial testing at a previous employer. In searching for the tool he used previously, he found Hexawise.
After a trial period and demo, Scott quickly understood that with the GUI and constraint handling Hexawise offered his team could learn and deploy this solution faster and more efficiently than he had previously thought. He could see his long term goals coming into focus: creating a team of test architects that design high-powered tests that are then executed seamlessly.
In days, the ITW Building Components Group was up and running in Hexawise. The team was poring through their requirements and parameterizing their ideas. Building out lists of values took little time at all. The team soon realized they were being much more efficient in their test design.
Based on previous projects, the time spent creating and documenting tests would average about 54 minutes per test. Conservative estimates put the time to create tests in Hexawise at 40 minutes. ITW documented some functionality in as little as 6 minutes per test.
Given the billions of combinations of test inputs possible in ITW’s system, the need for coverage was key. ITW had no coverage measurement capabilities even with the 3000 tests – leaving the thoroughness of their test cases in serious doubt.
Three weeks into using Hexawise the team had cut their regression suite from 3,000 tests to 1,457. These tests guaranteed full pairwise coverage and in some application functionality all triples.
In 2011, a new bug tracking tool was adopted. On average, the outdated stock of 3000 tests were catching 29 regressions on a monthly basis. Midway through 2012, ITW started using Hexawise and off the bat the number of regressions found increased significantly. Since implementation, the Hexawise-designed tests have been catching an average of 71 regressions per month.
The team at ITW still had some tollgates to pass: their clients.
It's the first time I've been involved with an upgrade that didn't give us major problems introducing new bugs. The (users) were generally happy with getting the update because it fixed problems they have been having.
Another team member was getting good feedback as well:
This is all great news so far the general consensus here from the designers is that we're putting out better quality stuff.
ITW has a special team that comes in quarterly to try and break as many things as possible. Typically they would leave a report documenting 130 or more defects. After implementating Hexawise, they documented 40.
They were shocked this last time for sure. - Scott
And the rave reviews keep coming in! Through their internal feedback procedures, Scott has no shortage of compliments:
The new process seems to be working.
Every revision is better!