This interview with Angie Jones is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

photo of Angie Jones
Angie Jones

Angie Jones is a Senior Software Engineer in Test at Twitter who has developed automation strategies and frameworks for countless software products. Angie shares her wealth of knowledge by speaking and teaching at software conferences all over the world and leading tech workshops for young girls through Black Girls Code.

Personal Background

Hexawise: Wow! You’re everywhere. Conference presentations around the globe at a break-neck pace. How’d that happen, exactly?

Angie: Ha ha. I’ve been doing test automation for quite some time. In early 2016, I attended a testing conference and didn’t learn much of anything new. It wasn’t a knock on the conference, but it was a wake up call for me.

I also interviewed people for automation roles quite often and found it was extremely hard to find people that were on the same level as my team. I realized that I have something to contribute and should be working to advance the industry in any small way that I can.

Diversity is also something that’s extremely important to me, so I thought being a black female on the stages of white male dominated conferences could also be my way of shaking up the game a little bit.

Actually, seeing a black woman announced as a speaker at a tech conference on Twitter was something that stopped me in my tracks. It wasn’t something I’d seen before. The conference was Write/Speak/Code, an event that empowers women to become thought leaders by writing blogs, speaking at conferences, and contributing to open source software! It was exactly what I needed. I attended and it absolutely changed my life. In less than 6 months after that conference, I’d become an international speaker and was being invited to keynote across the world. It all happened so was crazy!

Hexawise: You have helped Black Girls Code and TechGirlz work to provide girls the opportunity to learn to code. Do those efforts leave you with hope that the future is in good hands?

Angie: You know, I set out to inspire young girls, but every time I work with them I’m the one who leaves totally inspired and renewed. The girls are so smart and innovative! I give them a little push and they end up creating things that totally blow me away. The idea is to plant the seed. I wasn’t exposed to technology much as a young girl and had no idea that computer programming was even a career option. I almost missed my calling because of that. I don’t want our industry to miss out the next Tech Rock Star because she didn’t know about us.

The greatest businesses are ones that observe how their customers are misusing their products/services and adapt accordingly to make it easier for them to do what it is that they want to do, and still gain the benefit of the product/service.

Hexawise: Recently you moved from the Raleigh-Durham area (home to Hexawise, among other technology companies) to San Francisco and took a new role at Twitter. How did that come about? What is your new job?

Angie: Yes, I’m so excited about this new opportunity at Twitter! I told myself that my next role was going to be something fun and cutting edge. Twitter is just that! I love the platform, I love the innovative culture, and I love the possibilities for growth. I’m working on critical automation tasks and helping to drive automation efforts related to ads and live video.

Hexawise: Have you gained a new insight into some aspect of software testing from your work at Twitter that you can share? You haven't been at Twitter long but sometimes in a new environment people are especially alert and gain insights that others may not notice.

Angie: When I have the privilege of choosing the work I'll be doing, I lean towards companies who understand the need for both a Tester and an Automation Engineer. Twitter gets that, which played a big role in my decision to accept this position. I, along with a few other industry leaders, have been preaching for a while now that we don't necessarily need all testers writing automation, but with the complexity of today's applications, testers do need to be technical in order to do a thorough job.

At Twitter, I've now seen just how true that is. I'm working with top notch testers who aren't programmers but they understand the intricate plumbing of our systems and are capable of digging into the guts of the application to ensure all pieces are working as they should. This requires quite a bit of technical acumen yet very little coding. I'll further explore this in my keynotes this Fall at Targeting Quality (Canada), and Agile Testing Days (Germany).

Views on Software Testing

Hexawise: Your article BDD Without the Three Amigos: Maybe Talking To Yourself Isn't So Bad is really thought provoking. In it, you acknowledge that Behavior-Driven Development (BDD) “done right” includes the 3 amigos, but you go on to explore what happens in situations when BA’s and Developers aren’t wiling to “play ball”. You suggest that you have seen testers gain significant value on their own by using BDD. Testers, for example, can create Gherkin feature files with clear “Given / When / Then” instructions that enable rapid creation of executable test scripts.

Angie: Yeah, we often make these hard fast rules and try to force everyone to follow them. Whenever someone “misuses” a practice, all hell breaks loose on the internet. I’ve consulted with a lot of teams and came to realize that these teams are not naive. They realize that they are using the approach differently than it was intended to be used. However, they’ve made it work for them. The greatest businesses are ones that observe how their customers are misusing their products/services and adapt accordingly to make it easier for them to do what it is that they want to do, and still gain the benefit of the product/service. To that point, Matt Wynne, co-founder of Cucumber BDD, actually read this piece and realized that there was more he could do in the industry to push for collaboration from the other amigos.

In a fast-paced software delivery model, automation is definitely needed, but many companies make the mistake of thinking it’s a replacement for testing... Automation should be used as a tool... a technique to enhance testing efforts.

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Angie: That’s an easy one...Automation. But there’s definitely some education needed around this, which is why I write and speak on topics in this space. In a fast-paced software delivery model, automation is definitely needed, but many companies make the mistake of thinking it’s a replacement for testing. Anytime I’ve seen companies take this approach, the quality of their product has greatly suffered. Automation should be used as a tool... a technique to enhance testing efforts.

Hexawise: In your presentation at the 2016 SeleniumConf UK conference, you explore "How to Get Automation Included in Your Definition of Done." In that talk you discuss the idea that automation is useful but also that not all tests should be automated. How should a test team go about determining what software tests should be automated?

Angie: Ha ha. That’s a talk in and of itself. Actually, I’m going to give that talk at STPCon in the Fall of 2017. I get asked this question all the time, and it’s such a tricky thing to nail down. That’s because it’s highly contextual to the needs of the business. To do this topic any justice, I plan to explore several case studies in the STPCon talk and demonstrate how context plays such an essential part in answering this question correctly.

Industry Observations / Industry Trends

Hexawise: What trends do you foresee impacting the software testing community in the next 5 to 10 years?

Angie: With the explosion of the Internet of Things (IoT) and artificial intelligence, the systems that we test are going to become a lot more complex. This will require an evolution of testing. Our roles will become even more technical, and yet also require a healthy balance of humanity. The scenarios realized by these smart systems will require thorough testing and solid judgment. Software testing will look a lot different in 10 years, but will be extremely exciting!

Hexawise: Do you see organizations integrating software testers into the software development process (as compared to those instances where the first time software testers are involved is when software is delivered to be tested)? Do you believe more integration of software testers throughout the software development and maintenance process would be useful as software testing evolves?

Angie: I’ve already seen a huge improvement in integrating software testers into the development process with the embracing of agile practices. Quality is no longer solely owned by testers. Developers are testing their features before check-in, and testers are present throughout the entire process, essentially offering insight as early as the requirements phase. This has been essential as teams are adopting continuous integration and deployment processes. Time is of the essence, and the earlier we can avoid/find/correct mistakes, the better!

Hexawise: For those who are not used to involving testers early in the software development process, how would you describe the benefits to the business of involving software testers early in the process?

Angie: By involving testers early in the software development process, the team is able to identify and correct assumptions. This essentially eliminates potential bugs before production even begins. Testers bring in a breadth of knowledge about the application as a whole and how the individual components work together. Testers also serve as a customer advocate, ensuring that their goals are being considered. So, while the team is discussing a potential feature, the tester is advocating for the customer and also calling out how this can affect existing features in the system.

I sat in a design meeting recently and watched the tester poke holes in the proposed design and call out omissions that would cost us millions of dollars. The developers left that meeting with a list of requirements and considerations that they had missed. That's beyond valuable.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices?

Angie: I read a ton. I’m subscribed to several blogs, and I use Twitter to find new ones all the time. People are always creating new tools or approaches to solve interesting problems and I try to absorb as much of it as I can.

I don’t limit myself to testing blogs. I also read development blogs and tech news sites to stay up to date with trends in the software industry as a whole. This helps me think past my current role and prepare myself for the future as well.

I’ve already seen a huge improvement in integrating software testers into the development process with the embracing of agile practices. Quality is no longer owned by testers. Developers are testing their features before check-in, and testers are present throughout the entire process, essentially offering insight as early as the requirements phases.

Hexawise: You have presented at and attended many technology conferences. What advice do you have for people attending software conferences so that they can get more out of the experience?

Angie: It’s so funny how people tend to flock towards topics they are familiar with. They end up sitting there nodding in agreement with everything the speaker says, sharing their own experiences during Q&A, and leaving thinking that was such a great talk. However, did they learn anything new?

I try to attend talks that will address an unsolved problem that I have, or is in an area that I know very little about. These are the talks where I gain the most insight and can bring something of value back to my job.

Also, be sure to network! You won’t remember everything that people said during their talks, but leaving with a catalog of names and contact information of not just the speakers, but attendees as well, is gold! I often hit problems and I recall meeting someone a year ago at a conference who talked about this very problem during the cocktail hour. Because I networked with them, I feel comfortable reaching out and asking for help.

Hexawise: What blogs would you recommend should be included in a software tester's RSS feed reader?

Angie: Fortunately, Ministry of Testing has an aggregated feed that I use all the time. It consists of more than 500 testing blogs! It’s my morning newspaper.


Angie Jones is a Senior Software Engineer in Test at Twitter who has developed automation strategies and frameworks for countless software products. As a Master Inventor, she is known for her innovative and out-of-the-box thinking style which has resulted in more than 20 patented inventions in the US and China.

Angie shares her wealth of knowledge by speaking and teaching at software conferences all over the world and leading tech workshops for young girls through Black Girls Code.


Blog: Angie Jones

Twitter: @techgirl1908

Read previous Testing Smarter with... interviews: Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham - Testing Smarter with Mike Bland

Subscribe to the RSS feed for the Hexawise software testing blog.

By: John Hunter and Justin Hunter on Jun 14, 2017

Categories: Interview, Interesting People , Software Testing, Testing Smarter with...

This interview with Hans Buwalda is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Hans Buwalda

Hans has gained experience as a developer, manager, and principal consultant for companies and organizations worldwide. His approaches to testing—action-based testing and soap opera testing—have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges.

Personal Background

Hexawise: What drew you into a career in software testing?

Hans: It was "accidental". I worked as a management consultant in my home country, the Netherlands and was sent on an assignment to a major customer. They had serious problems with both test design and automation. In particular it was difficult to involve the (very busy) domain experts, and the automation was virtually impossible to keep current. Using keywords turned out to solve both these problems, and I have been pioneering that ever since with the Action Based Testing Method.

Hexawise: You were an early pioneer and key contributor to the keyword-driven test automation framework which has stood the test of time and is widely adopted in the industry. What drove you and other early keyword-driven framework pioneers to create it and advocate for its broader adoption?

Hans: I believe the two main drivers for using keywords are readability for non-technical people, and long-term maintainability of the tests. I think my core message is not as much the keywords themselves, but more the importance of test design in achieving those goals. Without a good modularized organization of tests keywords will fail, and so will for example Behavior-Driven Development (BDD). To say it even stronger: the worst automation projects I have seen were keyword projects.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Hans: I like it when people are successful. In my world that means being able to leverage automation to achieve better quality at a faster pace. Two of my colleagues recently showed me how they were able to help a company in the oil industry reduce time needed for testing dramatically (more than twenty times).

A nice detail there was that those tests were for equipment workflows, and did not involve any User Interface (UI). It is a common misunderstanding that keyword automation is for testing via the UI, but it can work just as well for non-UI testing, like Internet of Things (IoT), services, games, telecommunication, embedded software, etc..

Views on Software Testing

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Hans: I believe that with Agile, DevOps and service oriented architectures, we're well on our way. Agile allows for good cooperation between the Quality Assurance (QA), developers, product owners, domain experts and stakeholders. DevOps allows testing to be incorporated into the build and deploy pipelines. And service oriented architectures gives more ways to test than just through the UI.

scalability in automation is not as much a technical challenge as it is a test design challenge. This gives a bit of a paradox: test designers influence the automation success, but are not developers, and do not necessarily have affinity with engineering concepts like structured programming and refactoring, which in turn are key elements of good maintainable automation.

Hexawise: Can you describe a view or opinion about software testing that you have that many or most smart experienced software testers probably disagree with? What led you to this belief?

Hans: "Disagree" is a big word. But I do like to focus on one particular area that many others tend not to address that much: It is my observation over the years that scalability in automation is not as much a technical challenge as it is a test design challenge. This gives a bit of a paradox: test designers influence the automation success, but are not developers, and do not necessarily have affinity with engineering concepts like structured programming and refactoring, which in turn are key elements of good maintainable automation. There are some more elements that can influence automation success, like "testability" of the application under test: to what extend is an application friendly to testing and automation.

Hexawise: You have extensive experience with automated testing. What interesting anecdotes would you be able to share about some of your earliest experiences automating tests?

Hans: An early experience was the very first project where action words (my term for keywords) were used. It was for a major screen trading system, and before I came in a substantial amount of tests had been created with record and playback. When I say that you can probably guess the upcoming/impending disaster... The development team made a slight change in a lay-out of the main screen and virtually all tests stopped working because of it. It was in the early days and people weren't expecting such problems.

Industry Observations / Industry Trends

Hexawise: One of my (Justin’s) favorite software testing articles of all time is “Soap Opera Testing.” What experiences led you to write it and why do you think it struck a chord with so many testers and managers?

Hans: Thanks for the kind note. It was based on a major project in a large financial institution. The project included re-architecting of all systems, and getting ready for the year 2000. Proper testing was essential, but the tests also needed to be developed and fully automated in a very short amount of time.

To get this done I introduced an approach based on real life stories from the every day practice of real users and good test design (test modules, keywords) to support successful automation. I guess the method appeals to teams and managers because it is an efficient way to translate domain knowledge into, aggressive, test cases, and to get them automated quickly as well.

I believe the two main drivers for using keywords are readability for non-technical people, and long-term maintainability of the tests. I think my core message is not as much the keywords themselves, but more the importance of test design in achieving those goals.

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of expecting more from the software testing efforts in the organization?

Hans: You have to be serious about your craft. Know testing techniques, engineering principles and the domain of an application you're testing. Your attitude is important too. Be aggressive to the system under test, but cooperative as member of a team. Second, always keep an eye on the business side.

You should not be testing because you have read somewhere that testing is important. Testing costs time and money and there must be business reasons to invest in it. Not testing saves money, but also introduces risks that can cost money later on. Saving money and losing money are business considerations that well managed large companies tend to take very seriously.

You, or your QA management, must be ready to explain the value of their testing, and automation.

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEast conference in May. What could you share with us about those topics? What gave you the idea to talk about them?

Hans: I'm doing two tutorials, one on Better Test Design for Great Test Automation. The other focuses on what makes automated testing scalable. The classes are based on real-world experiences from various projects I’ve done, across many industries. With the many recent developments like cloud and DevOps, automation is constantly evolving, so it is fun to discuss about them.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Hans: Reading, attending conferences and webinars and hands-on practice still work well. Don’t despair when current developments can be overwhelming. Concentrate on what you need, and be curious.


Hans Buwalda has been working with information technology since his high school years. In his career, Hans has gained experience as a developer, manager, and principal consultant for companies and organizations worldwide. He was a pioneer of the keyword approach to testing and automation, now widely used throughout the industry. His approaches to testing—action-based testing and soap opera testing—have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges. Hans is a frequent speaker at conferences and other events and is lead author of Integrated Test Design and Automation.


Web site: Happy Tester

Twitter: @hansbuwalda

Previous Testing Smarter with... interviews: Testing Smarter with Mike Bland - Testing Smarter with Dorothy Graham - James Bach

Subscribe to the RSS feed for our blog.

By: John Hunter and Justin Hunter on Apr 17, 2017

Categories: Interview, Software Testing, Testing Smarter with...

This interview with Michael Bolton is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Michael Bolton is a consulting software tester and testing teacher who helps people solve testing problems that they didn’t realize they could solve. He is the co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure.

Michael Bolton

Personal Background

Hexawise: Do your friends and relatives understand what you do? How do you explain it to them?

Michael: [laughs] A lot of my friends and relatives don’t really care what I do! But if they’re curious, and they want an explanation, I tell them this:

"All that software you’re using every day is hard to develop. Whenever someone wants software built, development groups try to figure out what to do and how to do it. But since people never completely understand one another, there is a risk that the proposed solution won’t solve the problem, or will introduce problems of its own.

"Then, even when the ideas about how to build something are pretty clear, we’re still building it for the first time. It’s new to us, so we have to learn how to build it. As that happens, people make mistakes, and they realize new problems along the way. It’s the tester’s job to help identify problems all the way through that process.

"And then, as people are trying to build the product, testers explore it and experiment with it, looking for problems that might threaten its value. Ideally, the important problems get found and get fixed before the software comes to you.

"In other words, testers help people to try to figure out whether the product they’ve got is the product they want. I help testers and teams learn how to do that effectively, as a consultant and as a teacher. And I do that worldwide; in classes, at client sites, at conferences, and in social networks."

If friends or family want to know more, I can tell them more. But usually they’re happy with that.

Friends and family are one thing; managers and teams are another. Some testers say their managers don’t understand them. Recognize this: managers need to know about problems that threaten the on-time, successful completion of the project, where “project” means “some burst of development work”. Think about what you’re doing and the problems you’re facing in terms of how they relate to that. Talk about your work that way, and things will become a whole lot clearer.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Michael: That’s a tough call. On projects, I really like the detective work. In one specific case I investigated the roots of a nagging customer support problem, and found six or seven wildly different factors that, if addressed, could have eliminated that problem. That was satisfying. On another project, at a bank, I constructed a really elaborate oracle [Hexawise: "If you are not familiar with using oracles in software testing we strongly recommend following the link to learn more about this important concept."] that modeled consumer financial transactions through all of the different account flows. That involved a lot of programming, and using powerful features of Excel, and learning a ton about the business domain. That was fun. I found some cool bugs and weird behaviour, too.

From 2003 through 2008, I did pretty extensive study with Jerry Weinberg —several of the AYE Conferences, some consulting workshops, and the Problem Solving Leadership class. Those were really helpful for my own personal development.

But the most satisfying thing is teaching Rapid Software Testing, helping people develop their mindsets and skill sets, and reframing the ways they think and speak about testing. It’s rewarding to hear from the class participants about their renewed energy and focus—and to hear from their organizations about the value and power of testing, value they might not have seen clearly before.

Views on Software Testing

Hexawise: What testing concept(s) do you wish more software testers understood?

Michael: Oh dear. [laughs] That’s an awfully long list, and it varies from day to day.

I wish more testers were more articulate in describing their models—their models of the product and how to cover it; their models of oracles, and how they recognize problems; their models of testing itself.

I wish that more testers understood that test cases are not central to testing. To be an excellent tester means to explore, to experiment, to study, to model, to learn, to collaborate, to investigate. To discover. None of these activities, these performances, are captured in test cases. But testers keep talking about test cases as being central to the work, as though recipe books and written ingredient lists were the most important things about cooking.

I wish more testers understood that testing is not about “building confidence in the product”. That’s not our job at all. If you want confidence, go ask a developer or a marketer. It’s our job to find out what the product is and what it does, warts and all. To investigate. It’s our job to find out where confidence isn’t warranted; where there are problems in the product that threaten its value.

I wish more testers were more articulate in describing their models—their models of the product and how to cover it; their models of oracles, and how they recognize problems; their models of testing itself.

Of course, sometimes testers find those things hard to talk about because their intuitive models are vague and fuzzy. That’s one of the important tasks in the Rapid Software Testing space: to help people sharpen up their models so that they can sound like they know what they’re doing — while actually knowing what they’re doing. Then we can talk more precisely amongst ourselves about what we’re doing, and how we’re going to approach solving problems for our clients.

Some people are bothered by our focus on expressing things precisely. They complain “If we use jargon, the non-testers we’re talking to will get confused!” If you use jargon with people who don’t need to hear it, they may get confused, so don’t use jargon with them when there’s nothing at stake.

On the other hand, when non-testers hears “automated testing”, that affords the interpretation that testing can be automated. It can’t. That’s why, in Rapid Software Testing, we talk about automated checking: to alleviate confusion between things machines can do (checking) and things that you need clever humans to do (testing, which includes the process of developing and interpreting checks).

It’s okay for different testing and development communities to adopt and adapt their own ways of speaking. But amongst ourselves, within our communities and within our teams, we had damn well better learn to speak precisely, because imprecision is where lots of problems start, and where bugs live. Don’t jump on the newbies; help them learn and guide them along.

But I would say this to people who seek respect: getting straight on what things mean and how they matter is essential in real professions. [laughs] Doctors don’t say “virus, bacteria… who cares? The guy’s sick! He’s got… a… bad thing! And he needs… stuff to make him better! Pharmacist, give him… some... better-making-stuff and tell him to take it… whenever.” Fail to make appropriate distinctions, and the attempt to help the patient will fail. And then society gets antibiotic-resistant bacteria as a byproduct.

I wish more testers recognized that testing can be applied to anything—a running product, of course, but also to documents, diagrams, models, ideas. I often hear people saying “testers should do more than just testing,” and when I ask them what they mean by “just testing”, it turns out that their notion of testing is really impoverished.

Testing isn’t just operating a product and looking for bugs. Testing involves modeling and questioning and studying, making inferences, challenging beliefs. Generating, describing, refining, revising, abandoning, and recovering ideas, all the way through the project. Designing and performing experiments, not just by interacting with running code, but also by performing thought experiments on whatever we might have in front of us, from an idea to a document to a diagram to a program. Review is testing a story or some code. Critical thinking is testing an idea — helping people sharpen their understanding. Developing tools to help test is testing. [laughs] What’s with the “just testing” business? All that isn’t enough?

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Michael: Many years ago, when I first started out as a program manager, I thought it would be a good idea to write lots of things down, formally, in advance, and hand those instructions to programmers. After that, building the product and testing it would be simple.

Well, that sure sounded appealing, but it didn’t work very well most of the time. Some programmers really liked very specific instructions, and occasionally we could give them those when we had a really good idea of what we wanted.

What I was ignoring was all the work, all of the trial and error, all of the collective tacit knowledge that gets you to the point where you can make a lot of stuff explicit… but by then you don’t need most of it to be explicit, so making it explicit is a waste of time!

Excessive or premature formality is really dangerous for testing. Notice those adjectives: excessive, premature! Sometimes, in certain contexts, there are specific things that must be tested formally, in specific ways, or to check specific facts. That formality might be important because of technical risk, business risk, legal risk, risk to finances or to human health and safety.

Much of the time, though, your testing doesn’t need to be very formal. Even when you’re in a context that requires formal testing, you need plenty of excellent informal testing in order to get to excellent formal testing.

I see a lot of people investing in formality and documentation really early in the project. But I think it’s a mistake to do that before we’ve learned about the product and the problem space. That learning is a fundamentally exploratory process, one that can’t be formalized unless you want to suppress it or damage it or slow it down needlessly.

Testing isn’t just operating a product and looking for bugs. Testing involves modeling and questioning and studying, making inferences, challenging beliefs.

Hexawise: You have clearly articulated the inherent limitations of testing coverage metrics. For example, from your blog post: 100% Coverage is Possible, you state:

To claim 100% coverage is essentially the same as saying “We’ve looked for bugs everywhere!” For a skilled tester, any “100%” claim about coverage should prompt critical thinking: “How much” compared to what? 100% of what, specifically? Some model of X—which one? Whose model? How well does the “model of X” model reality? What does the model of X leave out of the universe of possible ways of thinking about X? And what non-X things should we also be considering when we’re testing?

What advice do you give to teams you work with who use quantitative coverage metrics?

Michael: [laughs] That’s a little like asking “What advice do you give to journalists who use spreadsheets?”

Start from the default premise that we don’t need them. Don’t start from the premise that we must hang a number on our coverage. Assume that we will describe our coverage. If there’s some way in which numbers will help, assume that whatever we decide to quantify will be based on some model of the software. All models focus on something and leave everything else out, and at best we’re only aware of some of that “everything else”.

After all, what might we be covering? We could decide to model our coverage in terms of how many lines of code, or branches, or conditions have been covered. We could count them. But not all lines of code (or branches, or conditions) are equally significant. And code coverage tools report on our code, but not necessarily the code in third-party libraries and frameworks, in the operating system, in the hardware platform.

Some people model coverage in terms of “requirements”. What they really mean by that is specific statements in requirements documents. Documents model ideas about the product.

So here’s one statement: “the site shall provide scheduling information for all flights on the airline’s network”. Here’s another: “when the customer is logged in, each page shall display the customer’s first and last names in the upper-right corner of the screen”. Are those statements of equivalent significance? How would we test them?

[laughs] Don’t say something silly, like you’ll fully test your product by having “one positive and one negative test case for each sub-requirement”! (The Wikipedia article on “test cases”, as of today, says exactly that.) Your testing of a product is a performance.

Sometimes a journalist will find it useful to illustrate a point with numbers, or a table, a spreadsheet. Baseball is a great example of a game where statistics help us to evaluate performances. Bill James, the guy at the centre of Moneyball, has shown how the numbers you choose can support arguments about a player’s value. But the magic of the Baseball Abstract, which was a kind of journal he wrote for many years, was that he told great stories, using stats to illustrate them. He also showed how stats don’t tell the whole story, and how they can mislead, too.

Sometimes journalists will quote figures from researchers, or use poll numbers to back a story. But the history of polls—particularly recent history—provides a great example of the ways we can be fooled by numbers.

I wrote these articles on talking about coverage a while ago, but I think they still stand up:

You’ll find useful stuff on measurement in Quality Software Management, Vol. 2: First Order Measurement (Weinberg) or the two e-books that make up that book, How to Observe Software Systems and Responding to Significant Software Events.

But if you want to quantify coverage, be skeptical. Be professionally unsure. Read "Software Engineering Metrics: What Do They Measure and How Do We Know" (Kaner and Bond). Read How Not to Be Wrong (Ellenberg). Read Proofiness (Seife). Read How to Lie with Statistics (Huff).

Industry Observations / Industry Trends

Hexawise: What software testing industry trends make you optimistic about the future? And which software testing industry trends make you concerned?

Michael: There’s been some degree of progress over the years in refining how people think about testing, but the people who are pushing for that are in a tiny minority. We’re a gaggle of hobbits facing an army with entrenched ideas that might as well have been handed down from orcs. Those ideas didn’t work 30 years ago and are crazily inefficient now. People still don’t think very critically about testing folklore and mythodology. (Yes, I said mythodology.)

Many people treat certain ideas as Grand Truths about testing. But many of the claims that people make—especially some of the testing tool vendors—are myths, or folklore, or simply invalid. People often say things that they haven’t thought about very deeply, and some of those things don’t stand up very well to critical scrutiny.

People don’t seem to notice how rickety everything is at the best of times. Have you tried to print something lately, using the new printer software that came out yesterday? Tried to buy a plane ticket? Tried to set up or top up a pay-as-you-go mobile hotspot when you’re travelling? I do all that stuff regularly, and it’s usually time-critical, and almost every time I enter a world of pain. Technology is complex, and our lives are complex, and the least little bump in the road can make the wheels fall off. Every day, as I try to use software, I feel like I’m being pecked to death by ducklings.

We want to do stuff like move money between accounts with our smart phones, securely, but the global financial system is at the mercy of institutions that are getting testing services from the lowest bidder, trying to find ways to make testing cheaper instead of more powerful and more risk focused. That life-critical medical software comes from the same larger community that brought us JavaScript and CSS. Yikes. [laughs]

I’m concerned that we’re still overly focused on testing the clockwork, the functional aspects of the product without thinking about how people will respond to it. As Harry Collins would say, machines are social prostheses. Like insulin pumps or artificial legs, they’re being plugged into human life and human purposes to do work that humans once did (or where humans with superpowers could do that work). But protheses do don’t what the real thing does, and the surrounding body has to adapt to that fact. Software doesn’t do what humans do, so humans often must adapt to it. We should all be asking: “how does the technology change us—for good and for ill?”

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEAST conference in May. What could you share with us about what you’ll be talking about? / What gave you the idea to talk about it?

Michael: I’ll be giving two tutorials. The first is about critical thinking for testers. We describe critical thinking like this: “thinking about thinking, with the intention of avoiding being fooled”. That’s central to our work as testers. Testers think critically about software to help our clients make informed decisions about whether the product they’ve got is the product they want.

Many people treat certain ideas as Grand Truths about testing. But many of the claims that people make—especially some of the testing tool vendors—are myths, or folklore, or simply invalid. People often say things that they haven’t thought about very deeply, and some of those things don’t stand up very well to critical scrutiny. That’s one of the reasons I started developing and teaching this class in 2008. I was dismayed that testers and other people in software development were accepting certain myths about testing unskeptically.

Not only that, but testers and teams allow themselves to be fooled by focusing on confirmation, rather than challenging the software. So, in the class, we talk about the ways in which words and models can fool us. In a safe environment, it’s okay—and even fun—to be fooled, and to figure out how not to be fooled quite so easily the next time.

On Tuesday, I present on one-day class called “A Rapid Introduction to Rapid Software Testing.” RST (the methodology) is focused on the mindset and the skill set of the individual tester. It’s about focusing testing on the mission, rather than on bureaucracy and paperwork. We sometimes joke that RST (the class) is a three day class in which we attempt to cover about nine days of material. [laughs] In “A Rapid Introduction to Rapid Software Testing”, I try to do the three-day class in one day. It is a rapid introduction, but we’ll be able to explore some of the central ideas.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Michael: The two that I can recommend that are explicitly about software testing are Lessons Learned in Software Testing and Perfect Software and Other Illusions about Testing.

But you asked about testing-related books, and there’s an absurd number of those. Thinking Fast and Slow is about critical thinking and how easy it is for us to get fooled. The Shape of Actions (Collins and Kusch) is about what machines can and cannot do, and the Golem series (Collins and Pinch) is about the nature of science, technology, and medicine. Code (Petzold) is about how data is represented and processed on digital computers. Everyday Scripting in Ruby is a decent, tester-focused book on creating little tools and doing work with scripts. And the list of Jerry Weinberg books is plenty long on its own: Exploring Requirements, and An Introduction to General Systems Thinking, and Errors, and Agile Impressions.

James Bach and I have not written a book on software testing together. But you could read our blogs, which would be like reading a long-ish book of essays on testing.


Michael Bolton is a consulting software tester and testing teacher who helps people solve testing problems that they didn’t realize they could solve. He is the co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure.

With twenty-five years of experience testing, developing, managing, and writing about software, Michael has led DevelopSense, a Toronto-based testing and development consultancy, for the past fifteen years. Previously, he was with Quarterdeck Corporation where he managed the company’s flagship products and directed project and testing teams both in-house and worldwide.


Blog: DevelopSense

Twitter: @michaelbolton

Related posts: Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham

By: John Hunter and Justin Hunter on Apr 3, 2017

Categories: Interview, Software Testing, Testing Smarter with...

This interview with James Bach is the first our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

James Bach, one of the most well-known and controversial leaders in the software testing community, challenges himself and others to continually develop their software testing approaches. James believes that excellent testing is a craft that requires many skills and ongoing practice and focus to develop and maintain those skills. The skills of testing include general systems analysis and critical thinking, but also social skills. In some sense, any child can test. But children and other amateurs cannot test systematically, nor can they provide professional self-assessment and reporting on the testing they do.

photo of James Bach
James Bach, Founder and CEO of Satisfice Inc

Personal Background

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

James Bach: In 2002, Microsoft complained to a federal judge that I hadn’t given it a power cord. Yes, an ordinary power cord of the kind you can pull out of the back of any standard desktop computer. Yes, to a federal judge. No, I am not making this up. Yes, I also thought it was bizarre-- bizarre but kind of satisfying.

This happened during the Microsoft Remedies Trial, wherein nine American states were suing Microsoft and the government because they wanted a tougher punishment for Microsoft after it lost its big antitrust case. The states hired me as an expert witness to find out if Microsoft was telling the truth when it claimed it was “technically infeasible” to remove IE and the Windows Media Player. I gathered a team and went to work. I soon discovered that it was possible to remove these things-- using only public information and Microsoft’s own helpful tech support people to set up my testing to prove it. (The tech support people did not realize the purpose of my questions, and cheerily gave me all that I needed to know.)

When I revealed my results, Microsoft demanded that I turn over all the materials necessary to reproduce my results. So I gave them one of my test systems. At the last moment, I pulled out the power cord, thinking that I was causing some low-level techie 30 seconds of annoyance. But the next day Microsoft was in court acting like I had withheld the Golden Power Cord of Truth.

It was satisfying because Microsoft never even attempted to proved me wrong on the facts. They used lawyer tricks to stop my truth bombs, instead. Way to go, Bill Gates.

Another satisfying moment was watching my son find a catastrophic bug in a life-critical piece of medical equipment. He didn’t ask for a spec or a test case specification. He used video-gamer techniques to confuse the system until it overrode its own safety features and melted itself inside the simulated patient. Yeah. Melted. That was a $3000 piece of equipment he ruined. I’ve rarely been prouder of myself for having such foresight as to create a son like him: he is such a good test tool.

people who don’t “embrace exploratory testing” are, to me, not even testers. They are fact checkers, maybe. I think that’s not good enough.

Hexawise: Failures can often lead to interesting lessons learned. Do you have any noteworthy failure stories that you’d be willing to share?

James: How about the time I tried to set up my corporate server. I got it all working. Then I moved it from the conference room to the server room. I couldn’t get it to come online after that. For 12 hours I worked on it, all through the night. At long last I relented to my brother’s suggestion-- that we move it back to the conference room. I had refused to do that because it didn’t make any sense. The conference room simply connected to the server room. How would adding an extraneous variable like that change anything. But, zoom, we were back online.

After a moment of “wha??” the solution flashed into my mind: we must have two feeds to the Internet instead of one. It turned out that the conference room was patched into the open net, but the other port I had been using in the server room was routed through a firewall which in turn connected to the net. Nobody told me this during the buildout of my office space. It was a completely missing possibility in my mind.

So what did I learn? I learned about the importance of de-focusing, which includes trying apparently silly things to solve problems. I’m more open to that now.

Here is another interesting failure. I recently wrote a report involving the calculation of percentages. A non-technical person (a lawyer I worked with) checked my math and found it to be wrong. In fact, every one of her calculations was wrong. But in the process of refuting her claims, I discovered a different error in one of my own numbers. So, isn’t that interesting? Even if a critique of your work is incorrect, it could still be a useful stimulant to help you find your own problems.

Hexawise: What kinds of activities do you enjoy when you’re not at work?

James: I run a business, so I feel like I’m always at work. But I guess I do take little bits of time off each day. What do I do? I daydream. I read science news. I solve math and logic puzzles. I try to walk each day. And I watch videos with my wife. We binge on English television series, mostly.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

James: Well, many of those things I now use as testing exercises for my students, so I don’t want to spoil them. But, hmm, here’s one. I was given one day to break into an invoicing system for a large pharmaceutical company. I found three ways to do it. One of the methods I used was to get one of the sales engineers to sit with me while I tested. I asked him to demo the system to me and then he hung around while I tried to break-in. The first time I broke in (using a traversal attack if you follow such things) I didn’t even know I had done it until the sales engineer said “hey you aren’t supposed to see that data.” Good thing he was there, huh? So part of testing can be charming people into helping you, and you never know what that help will bring.

Views on Software Testing

Hexawise: Some of the thought-provoking ideas you and Michael Bolton have come up with, like the important distinction between Testing vs. Checking have received a great deal of attention within the community. Other intellectual contributions to the community you have made are not as well known but are arguably equally important and insightful. One such contribution that comes to mind which really resonates with us at Hexawise is the exploratory-scripted (or formality) continuum you and your brother Jon described.

image of software testing formality continuum

Do you have one or two intellectual contributions to the community that you wish were more widely known?

James: I wish that more people understood the folly, the sheer silliness, of counting test cases and calculating pass rates.

I don’t care if you have 80 test cases or 8 million of them. That number tells me nothing about you or your testing. It tells me nothing by itself, and it tells me nothing in conjuction with other information (except in rare cases not worth talking about). It’s like telling me that you have broken your day into 27 tasks, of 1353 tasks, or whatever. Just stop. Instead of fake science smoke rings, tell me what you actually did. Here’s a simple suggestion: instead of giving me a number, give me a list: a list of test ideas, test cases, test activities, bugs, features, people… I can do something with a list. But if you give me a number I just have to say “show me the things you are counting.”

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

James: I used to think it was useful to talk about exploratory testing. But now I think it’s more helpful to say that all testing is exploratory. To say “exploratory testing” is the same as saying “testing.” Instead, I speak of how testing can be more or less formal, but it is always informal to some degree or else it ceases to be testing.

Also, in the last few years I have concluded that term “test automation” is toxic and should be avoided. It deposits a little poison in the mind whenever it is uttered. An angel loses its wings every time someone calls himself an “automated tester.” I changed my mind on both things as the result of ongoing attempts to teach students and hearing their questions and seeing where they get confused. That, and deep conversations with Michael Bolton.

Hexawise: How do/would you test very complex systems such as genetic algorithm systems and evolutionary systems? How do you test systems when we don't understand how they work? It seems kind of like medical differential diagnosis: poke, observe, learn, hypothesize, poke again. Or is there a better way?

James: I test them using social science methods. That, after all, is how scientists attempt to test their theories about social life. That means an emphasis on qualitative analysis, but bringing in statistical methods whenever applicable.

I agree that the medical world is a good example of where statistical methods and heuristic approaches are also needed. In testing complex things, some of what you need to do includes:

  • You must use time to your advantage-- observing systems over time the way primatologists observe chimps in the wild.
  • You must use Grounded Theory, beginning with immersion and observation, until patterns begin to reveal themselves.
  • You must focus on testability. To create an environment where you can control and observe more of what is there.
  • You must pay attention to clues. Many, many clues. Stop looking for simplistic “test cases” that will “prove” that the software works.
  • You must become expert at data wrangling, since these systems usually involve huge amounts of data.
  • Let other people help you.
  • Forge partnerships with users.

If it’s a training gig then my objective is to show them what testing can be, show them a path to get there, and encourage them to walk that path. A lot of that is about removing the obstacles to moving along.

Hexawise: It is clear from your writings and frequent presentations, that you feel passionately that the software testing community would greatly benefit if far more testers embraced Exploratory Testing. It’s a deeply held conviction. What particular testing practice(s) do you most wish the software testing community would embrace?

James: Testing is exploratory. So, people who don’t “embrace exploratory testing” are, to me, not even testers. They are fact checkers, maybe. I think that’s not good enough.

I wish more testers were mathematically inclined. You must see this, too, at Hexawise-- the widespread math-phobia in our field. I want to talk about Karnaugh maps and the value of de Bruijn sequences. But I have to keep that stuff out of my classes or I will freak most people out. It’s not that I am a mathematics expert. I’m just an enthusiast who wants to be held to a higher standard. But even my dalliances in Bayesian belief nets sound like high elf incantations to most testers. Mathematical disabilities, in general, make our craft prey to quackery and fraud of all kinds.

At the same time, I want to be inclusive. Mathophobes have a lot of offer and I don’t want them to think I don’t welcome them. But must they necessarily be the majority of testers? I guess I’m saying I want a cure for mathophobia, please.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

James: I wish they understood that it benefits from specialization. When software people get heart disease they don’t limit themselves to a GP, do they? If they need surgery they don’t insist on being operated on by a rotating team of generic medical people who took a three day class in “Agile medicine” do they?

I get to be a specialist tester mainly because I do it on my own time. My clients are paying me, usually, for training, not for doing testing. So I usually test in my “free time” to sharpen my skills. I recently did have a wonderful and lucrative testing-related gig (because this particular project knew that it needed the best tester and analyst it could possibly get and became convinced I was the Chosen One), but those gigs are few and far between for someone like me.

Hexawise: When individual companies hire you for consulting engagements, how would you describe what it is that you usually seek to provide to them?

James: If it’s a training gig then my objective is to show them what testing can be, show them a path to get there, and encourage them to walk that path. A lot of that is about removing the obstacles to moving along. Chief among those obstacles is lack of confidence. So I do a lot of pep talking. Another obstacle is the very primitive, mechanical way that people think about testing. I have to replace that with systems thinking.

If it’s a testing gig then my objective is usually to provide deep, exemplary testing, that is transparent to my client. I want them to feel that they see their product in a beautiful focus. The danger I am always in when I test is that I will get too deep (and therefore be too slow and expensive). But for me, deep testing is the most fun, so it’s a constant struggle to hold myself back from using my most penetrating methods and tools.

Industry Observations / Industry Trends

Hexawise: As Artificial Intelligence increases in capability (for example the strides made by Watson) do you foresee an increase in the capabilities of computer checking? I am thinking, not of an elimination of the difference between human lead testing and what can be done without people but to what extent you see the possibility for AI to do a progressively much better job of checking.

James: I foresee a collapse of critical thinking about these complex systems, followed by some sort of disaster, followed by a new realization of the risks of surrendering human judgment to a machine. I foresee that this will be an ongoing cycle. This collapse of critical thinking will lead to more shallow testing and perfunctory checking, presented as if it were deep. For an example of what I mean, see this old computer commercial:

Pay attention starting at 2:05. Oh look, the computer is assuring us that it has no errors. Everyone relax! Its “electronic brain” can be trusted!

Hexawise: Do you have any predictions about how large an impact Artificial Intelligence and Machine Learning will have on software testing in the next 5-10 years?

James: I don’t think it will have any impact on testing as such, except inasmuch as many people (not skeptical testers, but people who might otherwise hire testers) will trust black boxes when they should be challenging them.

I suppose as machine learning become more available to the masses, someone might try to train one to recognize bad output of some kind. That’s a sort of test tool. But it would only apply to well-established kinds of badness.

If you think about it, anti-spam systems are a sort of test system. Machine learning is used in spam filters. So, I guess testing is already using machine learning in that sense. But I don’t see the average tester applying machine learning methods to testing. I don’t see a developer doing that, either. It’s too involved and complicated; too narrow in application.

I hear that people at Google are going to “put coders out of business” with a system that writes code based on people just talking. You know what that’s called? A compiler. They are inventing a high level compiler. Now the people who talk will be called developers and will have to learn to talk properly, because it will emerge that normal people can’t say what they actually want.

Hexawise: Have you seen a particularly effective process where the software testing team was integrated into the feedback from a deployed software application (getting feedback from users on problems, exploring issues the software noted as possible bugs...)? What was so effective about that instance?

James: Not really. What I see is developers ignoring feedback. It’s too overwhelming. I suspect there are people who are really good at doing that. But I haven’t run across any.

One of the things that has happened with DevOps is a de-emphasis on testing and more of an emphasis on overall risk management. That’s a valid strategy, of course, but it has interesting blind spots. Whenever I hear a developer speak about wanting feedback from users I immediately think about how abusive and incompetent most users are about reporting problems. No, my developer friends, you really don’t want to read all those Internet comments on your software. You will be demoralized. But testers? We love reading that stuff. It’s our wheelhouse. We get clues and then we can reproduce the problems and make them sensible for the devs.

My brother, at eBay, with his testing mentality, loves going over the user feedback and bringing it to the teams there. But he will tell you it’s a constant struggle to get the attention of the dev teams.

attend a conference. Don’t bother to go to the talks, though. Most of the talks are full of fluff. Instead, find people and talk to them. Compare notes, make friends. Go to the testing lab.

Hexawise: Often one of the major roadblocks to software testers is their own management. Do you think this is a fair statement? Do you have suggestions for how testers can attempt to improve the situation. My background is strongly influenced by W. Edwards Deming so I have a tendency to look at the organization as a system and see room to improve the management system. It seems to me often the biggest gains are not possible if we keep departments separate (software development, software testing, marketing, customer service...). We can make improvements in software testing even if it is largely seen as separate from the organization but in doing so we miss much greater potential improvement.

James: The collapse of the test management industry is a terrible problem. It’s getting harder to find any kind of test manager out there. Do they even exist in Silicon Valley any more or have they all been hunted down by parasitic wasps who lay “scrum master” eggs in their living carcasses?

People who seem to know little about management or testing tell me that test managers are not needed. Okay, that means a whole lot of things that test managers do will not get done. This includes: providing a protected place for testers to work, free of harassment; negotiating for testability; negotiating for resources; assuring that schedules are reasonable; assuring that testing gets the respect it requires in order to attract and keep talented people; assuring that deadlines are met; explaining testing to management; assuring that testers are properly trained. When those things aren’t happening, testers tend to become more zombie-like and reactive (I’m not speaking of those fast zombies); or they become cheerleaders for the devs, instead of critical thinkers.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices?

James: I’m not convinced there are improvements in testing practices in the absence of improvements in the thinking and social systems that drive practices-- and those things don’t improve much, as I’m sure you’ve noticed. Seems to me that the current nonsense in our craft is very similar to the old nonsense. Maybe some of the buzzwords have changed, but not much else.

The landscape of testing has definitely changed. Agile and Lean have aggressively colonized a lot of the testing space. Since most testers are young people (and test management has been eviscerated) they are easy pickings for the Agile Universalists (the people who think that we don’t need testers because we can just all test whenever we feel like it).

What this means is that testing remains rather primitive wherever I go (with a few interesting exceptions, driven inevitably by a single enlightened Elrond-like or Galadriel-like manager, who always seems to disappear off into the Grey Havens within a couple of years of me meeting him or her).

How I become aware of new and interesting ideas is through my community. For instance, a student told me about Karnaugh maps the other day and now I am trying to find a use for them.

Hexawise: How would you suggest testers stay current?

James: I don’t think currency is a thing in testing, except with respect to learning about certain emerging technologies and buzzwords.

The bigger thing in testing is to push us forward, which not enough testers are trying to do. Don’t worry about currency, worry about whether you truly understand testing, and keep working on that study.

Read widely about science. Get ideas from that. And play with the ideas. For instance, I read on Hacker News about 350,000 free images being released by the Metropolitan Museum of Art. I decided to experiment with turning that into a practical resource for test data. This led to playing with data wrangling and image analysis tools.

Also, attend a conference. Don’t bother to go to the talks, though. Most of the talks are full of fluff. Instead, find people and talk to them. Compare notes, make friends. Go to the testing lab. Or host a little conference. Invite testers to a small gathering where you can share experience reports.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?



James Bach has authored two books and consulted and presented on software testing worldwide. It is difficult to put into words how unique and insightful James is. In order to get a feel, we suggest listening to his presentations yourself and reading his excellent blog.

Some Career Highlights


By: John Hunter and Justin Hunter on Mar 2, 2017

Categories: Interesting People , Software Testing, Testing Strategies, Interview, Testing Smarter with...

I read a great post by Aleksis Tulonen Brainstorming Test Ideas With Developers and wanted to share it with you.

The ideas in the post provide a very efficient way to:

  1. increase the robustness and thoroughness of testing,
  2. prioritize among different testing ideas (both from a relative importance standpoint and from a relative timing perspective)
  3. surface new testing ideas that would not otherwise be considered
  4. provide some useful risk mitigation protection against the reality that "all models are wrong; some models are useful" problem

I'd suggest also scheduling a debrief with the same testers and developers a few weeks or months after the code was deployed.

At that time take a look at photos of the pre-testing Post It notes and a list of defects that slipped past testing and ask

  1. "what extra tests would we need to have added to have found these defects?"
  2. "what specific inputs did we forget to include?",
  3. would techniques such as pairwise testing have been beneficial?
  4. what areas did we spent a lot of effort on that did not result in learning much?
  5. what lessons learned should we incorporate for next time?

The idea of delibrately examining your software development and testing practices will be familar to those using agile retrospectives. The power of continually improving the development practices used withing the organization is hard to appreciate but it is immense. The gains compound over time so the initial benefits are only a glimpse of what can be achieve by continuing to iterate and improve.

Related: Pairwise and Combinatorial Software Testing in Agile Projects - Applying Toyota Kata to Agile Retrospectives - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas

By: John Hunter and Justin Hunter on Dec 15, 2016

Categories: Agile, Software Testing, Testing Strategies

When you outsource testing responsibilities to a third party, there are several key points to consider beyond the three most frequently asked questions of:

  • What is the cost per hour of your services?
  • What evidence can you provide of your firm’s level of expertise in our industry and/or the technical skills we feel you’ll need? and
  • What is your plan for increasing testing automation? In particular, it is worth developing a thorough understanding how the vendor will identify what kinds of things should be tested, how those things should be tested, and how that information will be communicated to you.
  1. How do you ensure testing is performed thoroughly - beyond just “happy path”/ confirmatory tests?
  2. Do you use Exploratory Testing broadly? Would you recommend we use it? Exploratory Testing is a well-established testing approach.
    • Champions of exploratory testing include James Bach and Michael Bolton.
    • If your vendor hasn’t heard of exploratory testing and starts to explain the disadvantages of undisciplined “ad hoc testing” it’s a warning sign that they might have rigid tunnel vision about what good testing entails. Ad hoc testing is not what exploratory testing is but those that haven't learned about exploratory testing may get the two ideas confused.
  3. How do you communicate the diminishing marginal returns that exist when you execute sets of test scripts?
    • In Hexawise, analyze tests screen is how we help show this concept. That screen shows both pecentage of parameter value pairs tested and the coverage matrix showing the parameter values pairs tested and untested. This view can seen for any specific number of tests along the test plan (using the adjustable bar at the top of the image). This image shows 80% of the paired values have been tested after 10 test cases. Using the slider this view lets you see exactly which parameter value pairs are untested at each point. In this example after 15 tests there is 96% pairwise coverage and 19 tests covers all pairwise combinations.
    • Be conscious that your vendor has a strong conflict of interest here. 80% 2-way coverage might be sufficient for your needs on a given project but testing to that level of thoroughness (instead of the complete set of tests) could represent 50% of their billable hours in which case they may go to 100% 2-way coverage (to increase billable hours). Of course, 100% 2-way coverage may be called for, these decisions have to be made with an understanding of the software being tested.
    • Implications: ideally, you would like to have a testing vendor that is happy to openly share and discuss the tradeoffs that exist and there strategy to get you the best results based on your situation.
  4. Do your testers use a structured test design approach such as pairwise test design?
    • If so, how many of your testers regularly use pairwise test design or similar methods to design tests?
    • During the process of selling you their testing services, vendors have a strong incentive to highlight their efficiency-enhancing capabilities. As soon as the ink dries on the contract though they have an incentive to keep billable hours high.
  5. Why (and in what types of testing) do they use it?
    • Pairwise and combinatorial test design approaches can be widely used in virtually all kinds of software testing.
    • If your vendor indicates that they use these approaches only in narrow areas (e.g., most likely functional testing or configuration testing), that is a large red flag that they don’t understand these test design approaches very well at all.
    • If your vendor does not understand how to apply these test design methods broadly (including in systems integration testing, performance testing, etc.) then you can safely assume that there will be considerable wasteful redundancy in any areas where these test design approaches are not used.
  6. If we gave you 50 test scripts we already have, could you generate a set of pairwise tests for us that could test the same system more thoroughly in significantly fewer tests?
    • A knowledgeable vendor should be able to analyze your tests and provide a draft set for your review within 24 hours.
    • If the vendor takes multiple days and they need to search far and wide for an available expert to create a set of optimized set of tests for you, that indicates that the actual number of testers capable of performing this kind of test design is probably pretty low.
    • Another practical way to double-check vendor claims is to search sites like LinkedIn for keywords such as pairwise testing, orthogonal array testing, and/or Hexawise.

Making sure your software testing process is staying current with the best ideas in software testing is an important factor in creating great software solutions that your customers love. Often companies understand the need to stay current on software coding practices that are successful but fewer organizations pay attention to good practices in software testing. This often means there is a good deal to be gained by spending some time to examine and improve your software testing practices.

By: John Hunter and Justin Hunter on Mar 29, 2016

Categories: Testing Strategies, Software Testing

When thinking about how to test software these questions can help you think of ideas you might otherwise miss. These tips are useful for anyone testing software; we do integrate tips that are specfically relevant to those using the Hexawise software testing application.

  • What additional things could I add that probably won't matter (but might?)
  • What are the two parameters in the plan with the most values? And should I shrink the number of parameter values for either case?). If you have a parameter or two with many more parameter values than other parameters have that can greatly increase the number of test cases that must be run to explore every pairwise (or higher) combination. If those are critical to test, then they should be, but often a high number of parameter values is an indication that they really could be reduced. Using value expantions may well be a wise choice.
  • Do the tests cover all the high priority scenarios that stakeholders want covered? Once you generate the tests think about high priority scenarios and if they are missing add them as required test cases. It may well be worth adding tests for the most common scenarios. Often this can be done without requiring extra tests when using Hexawise. Hexawise will just consider those and then create test cases to match your criteria which often won't require an extra test. Sometimes it will add a test or a few tests to the total, in which case you can decide the benefit is worth the added cost of those tests.
  • If you're testing a process, what are the if/then points where the process diverges? Make sure the alternative processes are being tested.
  • If the plan is heavily constrained, would it make sense to split it into multiple separate plans?
  • Might it make a difference to how the system operates if you were to include additional variation ideas related to: who, what, when, where, why, how and how many? Those questions are often helpful in identifying parameters that should be tested (and occassionally in identifying parameter values to test).
  • Have you accidentally included values in your plan that everyone is confident won't impact the system? You want to test everything that is important but if you test things that are not going to impact whether the software works or not it adds cost with no benefit.
  • Have you accidentally included "script killing" negative tests in your plan? See details on this point in our training file, different types of negative tests need to be handled differently when using Hexawise
  • Have you clearly thought about the implications of the distinction between impossible to test for combinations vs combinations that will lead the system to display error messages? Impossible to test combinations can be avoided using the invalid pair feature. But situations where users can enter values that should results in error messages should be tested to validate those error messages appear.
  • Have you considered the cost/benefit trade-off of executing, say, 50% of the Hexawise generated test set vs 75% vs 100%? It is best to think about what is the most sensible business decision instead of just automatically going for some specfic percentage. Depending on the softare being tested and the impact on your business and customers different levels of testing will make sense.

Too often software testing fails to emphasis the importance of experienced software testers using their brains, insight, experience and knowledge to make judgements about what is the best course of action. Hexawise is a wonderful tool to help software testers but the tool needs a smart and thoughtful person to wield it and produce the results we all want to see.

Related: Software Testers Are Test Pilots - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - How Much Detail to Include in Test Cases?

By: John Hunter and Justin Hunter on Mar 14, 2016

Categories: Testing Strategies, Software Testing, Hexawise test case generating tool, Combinatorial Software Testing, Hexawise tips

The amount of detail used in a test plan will depend upon your particular context. When I'm asked how much detail to include by clients, I've started drawing a simple 2 X 2 matrix that looks like this:




There is simply too much variation between different teams of testers and business contexts to provide a one-size-fits-all answer to this question. There are good and valid reasons that different teams around the world use very different test plan documentation approaches when it comes to test case writing styles.


You and your colleagues should familiarize yourself with several different approaches that other teams use. Think about the pro's and con's of using formalized and detailed script structures as opposed to the pro's and con's of other "documentation-light" approaches. Once you're aware of the options available, you should consciously adopt approach that works best for your context.


These sources are directly on point and provide more examples for your consideration than I can get into here:

  1. Dr. Cem Kaner's excellent piece "What's a Good Test Case?"

  2. A presentation I gave at an STP conference called "Documenting Software Testing Instructions - A Survey of Successful Approaches"

By: John Hunter and Justin Hunter on Oct 8, 2015

Categories: Testing Strategies

We have mentioned George Box before. He was an amazing person, scientist and statistician. One of the traditions George started in Madison, Wisconsin was the Monday Night Beer Sessions.

An excerpt of Mac Berthouex’s introduction to An Accidental Statistician: The Life and Memories of George E. P. Box:

I met George Box in 1968 at the long-running hit show that he called “The Monday Night Beer Session,” an informal discussion group that met in the basement of his house. I was taking Bill Hunter’s course in nonlinear model building. Bill suggested that I should go and talk about some research we were doing. The idea of discussing a modeling problem with the renowned Professor Box was unsettling. Bill said it would be good because George liked engineers.

Bill and several of the Monday Nighters were chemical engineers, and George’s early partnership with Olaf Hougen, then Chair of Chemical Engineering at Wisconsin, was a creative force in the early days of the newly formed Statistics Department. I tightened my belt and dropped in one night, sitting in the back and wondering whether I dared take a beer (Fauerbach brand, an appropriate choice for doing statistics because no two cases were alike).

I attended a great many sessions over almost 30 years, during which hundreds of Monday Nighters got to watch George execute an exquisite interplay of questions, quick tutorials, practical suggestions, and encouragement for anyone who had a problem and wanted to use statistics. No problem was too small, and no problem was too difficult. The output from George was always helpful and friendly advice, never discouragement. Week after week we observed the cycle of discovery and iterative experimentation.

Justin, at a meet up with a few testers in Nottingham, out of a desire to do something nice for some testers in the community found himself buying beers for a few testers. The organizer asked Justin to put a couple slides together, then he posted them on Twitter and thanked us.


Justin made a similar offer to attendees at StarEast. And as luck would have it, the first guy to respond was Alan Page who was giving the keynote speech at the conference. Alan sent out tweets with showing testers getting and sharing a few beers.


And CAST2014 didn't miss a beat.

Hexawise buys the beers cast 2014 twitter

Hexawise buys the beers cast 2014 twitter conversation

Now #HexawiseBuysTheBeers has become a way to encourage comradery among software testers at conferences and another small way the legacy of George Box lives on.

By: John Hunter and Justin Hunter on Sep 1, 2014

Categories: Hexawise, Interesting People

A common mistake software companies make is creating products where features built for advanced users overwhelm and confuse average users. At Hexawise we have focused on creating a great experience for average and new users while providing advanced users powerful options.

How to Avoid a Common Product Mistake Many Teams Make by Mark Suster

The single biggest mistake most product teams make is building technology for what they believe the user would want rather than what the actual end-user needs. … My philosophy is simply. You design your product for the non-technologist. The “Normal.”

Give people fewer options, fewer choices to make. Give them the bare essentials. Design products for your mom. … power users will always find the features they need in your product even if they’re hidden away. Give them a configure button somewhere. From there give them options to soup up the product and add the complexity that goes with newfound power.

Make sure you read his full post and subscribe to his blog, the posts are clearly-written, pragmatic, and insightful.

Our experiences at Hexawise match the points the post makes. We've designed our web-based tool with Jason Fried/37 Signals-inspired "KISS" (Keep It Simple Stupid) design principles in mind. Our interesting debates about how to add (or whether to add) new features have often been based on the exact tensions you mention here. "Power users want new features" vs. "... but users love our tool precisely because it's easy-to-use and it doesn't have 'feature bloat'."


We've experimented with the suggestion you raise here (e.g., rather than say "no" to every advanced user request, we build them in hidden places for advanced users without distracting our regular users). Results have been good so far.

The Bulk add/bulk edit feature in Hexawise is an example of a powerful feature that is implemented in a way that doesn't interfere with the ease of use for those that don't take advantage of this feature.

For us, there are few things more important to our tool's success in the marketplace than our ability to get the balance right between "uncluttered simplicity that everyone wants" vs. "with the powerful capabilities that advanced users want."

There are natural tensions between the two opposing design goals. Sean Johnson (Hexawise CTO) is the strongest advocate for simplicity. John and Justin, Hexawise's CEO, love simplicity, and understand the important of simplicity for usability, but also find ourselves pushing for advanced features.

I am a strong believer in the simplicity with the option for power users to access advanced features. Turning this concept into practice isn't easy. Thankfully Sean has the mindset and skill to make sure we don't sacrifice simplicity and usability while providing power features to power users at Hexawise. I was also lucky enough to have another amazing co-worker, Elliot Laster, at a previous employer that provided this valuable role. One of the tricks to making this work is hiring a great person with the ability to make it a reality (it requires deep technical understanding, an understanding of usability principles and a deep connection to real users, all of which Sean and Elliot have to an amazing degree).

Getting the balance right is one of the most important things a software company can do. We've tried really hard to get it right. We're probably overdue for formal usability testing of basic functionality. Reading blogs like Mark's is useful for new ideas and also for the push to do what you know you should do but seem to keep putting off.


By: John Hunter and Justin Hunter on Jan 21, 2014

Categories: Hexawise test case generating tool, Software Development

Many teams are trying to generate unusually powerful and varied sets of software tests by using Design of Experiments-based methods to generate many or most of their tests. The two most popular software test design methods are orthogonal array testing and pairwise testing. This article describes how these two approaches are similar but different and suggests that in most cases, pairwise testing is preferable.

Before advancing, it may be worth pointing out that Orthogonal Array Testing is also known as OA or OATS. Similarly, pairwise testing is sometimes referred to as all pairs testing, allpairs testing, pair testing, pair-wise testing, or simply 2-way testing. The difference between these two very similar approaches of pairwise vs. orthogonal array is that orthogonal array-based solutions require the same coverage goal that pairwise solutions do (e.g., that every pair of inputs is tested at least once) plus an additional hurdle/characteristic, that there be a uniform distribution throughout the domain.

I have studied the question of how can software testing inputs be combined most efficiently and effectively pretty steadily for the last 7 years. I started by searching the web for "Design of Experiments" and "software testing" and found references to Dr. Madhav Phadke (who, by coincidence, turns out was a former student of my father).

  • I discovered that Dr. Phadke had designed RDExpert which, although it had been primarily created to help with Research & Design projects in manufacturing settings, could also be used to select small sets of powerful test sets in software testing projects, using the Orthogonal Array-based test selection criteria.

  • I used RDExpert to create test sets (and compared those test sets against sets of tests that had been selected manually by software testers)

  • I gathered results by asking one tester to execute the manually selected tests and another tester to execute the the Orthogonal Array-based tests; the OA-based tests dramatically outperformed the manually-selected ones in terms of defects found per tester hour and defexts found overall.

So, in short, I had confirmed to my satisfaction that an OA-based test data combination strategy was far more effective than manually selecting combinations for the kinds of projects I was working on, but I was curious if other techniques worked better.


After more study I have concluded that:

  • Pairwise is more efficient and effective than orthogonal arrays for software testing.

  • Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings.


And we have built Hexawise as a software tool to help software producers test their software, based on what I have learned from my experience. We take full advantage of the greatly increased efficiency and effectiveness of letting testers to determine what needs to be tested and software algorithms to quickly create comprehensive test plans that provide more coverage with dramatically fewer tests.

But we also go well beyond this to create a software as a service solution that aids the software testing team with many huge advantages such as: automatically generating Expected Results in test scripts, automated importing of data from Excel or mind maps, exporting tests into other tools, preventing impossible to test for values from appearing together, and much more.


Why is a pairwise testing strategy better than an orthogonal array strategy?

  • Pairwise testing almost always requires fewer tests than orthogonal array-based solutions (it is possible, in some situations, for them to have an equal number of tests).

  • Remember, the reason that orthogonal array-based solutions require more tests than a pairwise solution to reach the coverage goal of testing all pairs of test conditions together in at least one test is the additional hurdle/characteristic that orthogonal array testing has, e.g., that there be a uniform distribution throughout the domain.

  • The "cost" of the extra tests (AKA experiments) is worth paying in many settings outside of the software testing industry because the results are non-binary in those tests. Someone seeking a desired darkness and gloss and luminosity and luster for a particular shade of green in the processing of film, for example, would benefit from with the information obtained from the added information gathered from orthogonal arrays.

  • In software testing, however, the added costs imposed by the the extra tests are not worth it. You're generally not seeking some ideal point in a continuum; you're looking to see what two specific pieces of data will trigger a defect when they appear in the same transaction. To identify that binary approach most efficiently and effectively, what you want is a pairwise solution (with fewer tests), not a longer list of orthogonal array-based tests.


Let me also add these points.

  • First, unlike some of my other views on combinatorial test design, my opinion on this narrow subject is not based on multiple empirical studies; it is based on (a) the reasoning I laid out above, and (b) a dozen or so conversations I've had with PhD's who specialize in the intersection of "Design of Experiments" and software test design, and (c) anecdotal evidence from using both methods.

  • Secondly, to my knowledge,very few, if any, studies have gathered empirical data showing benefits of pairwise solutions vs. orthogonal array-based solutions in software testing scenarios.

  • Thirdly, I strongly suspect that if you asked Dr. Phadke, he would give you his reasons for why orthogonal array-based solutions are appropriate (and even preferable) to pairwise test case selection methods for certain kinds of software projects. I have a huge amount of respect for both him and his son.


Time doesn't allow me to get into this last point much now, but "mixed strength" tests are another even more powerful test design approach for you to be aware of as well. With mixed strength testing solutions, the test designer is able to select a default coverage strength for the entire plan (e.g., pairwise / AKA 2-way coverage) and, in the same set of tests, select certain high priority values to receive higher coverage strength (e.g., 4-way coverage strength selected for each "Credit Rating" and "Income" and "Loan Amount" and "Loan to Value Ratio" would give you a palm that achieved pairwise coverage for everything in the plan plus comprehensive coverage for every imaginable combination of values from those four high priority parameters. This approach allows you to focus on risk-based testing considerations.


Sorry if I got a bit long-winded. It's a topic I'm passionate about.

Originally posted on Stack Exchange, Additional note added after the first 3 comments were submitted:

@Hannibal, @Peter K., and @MichaelF, Thanks for your comments! If you'd like to read more about this stuff, I recommend the multiple links available through this "bundle of links" about pairwise testing and combinatorial testing. In particular, Michael Bolton's article on pairwise testing is directly relevant and very clearly written. It is one of the few introductory articles around that accurately describes the difference between orthogonal array-based solutions and pairwise solutions. If I remember correctly though, the example Michael uses is a rare exception to the rule; the OA solution has the same number of tests as an optimal pairwise solution does.

Related: The Empirical Evidence for Using Pairwise and Combinatorial Software Testing - 3 Strategies to Maximize Effectiveness of Your Tests - Hexawise TV

More than 100 Fortune 500 firms use Hexawise to design their software tests. While large companies pay six figures per year for enterprise licenses, Hexawise is available for free to schools, open source projects, other non-profits, and teams of up to 5 users from any kind of company. Sign up for your Hexawise account.

By: John Hunter and Justin Hunter on Jun 11, 2013

Categories: Combinatorial Testing, Design of Experiments, Efficiency, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies, Experimenting

I saw these words of advice from Conrad Fujimoto in an email and thought they were worth passing on. I'm using them with Conrad's permission:

Over the years, I’ve taught many software testing courses. Trainees are appreciative of the ideas, insights, and techniques presented to them.They are convinced that principles and methods taught are useful and effective. Yet, often I hear the phrase “but, that won’t work here.”

Some of the reasons given for such pessimism are resource constraints, organizational politics, lack of testing focus, and little management understanding and support. The trainees knew what adjustments needed to be made but they felt powerless to affect any meaningful change. Fortunately, much can be accomplished by strategic planning and being aware of opportunities.


Some ideas:

  1. When no one is taking a leadership role in improving the process, consider assuming that role (people are often happy to see someone take charge).

  2. Seek opportunities to form relationships and work with others who share the same concerns about the existing process.

  3. Establish your authority and credentials for speaking on testing matters by recognizing and promoting the successes of your testing team.

  4. Be proactive and constantly monitor and report the progress of both development and testing against published schedules.

  5. Be ready to implement corrective actions or invoke contingency plans in the event of schedule slippages; where appropriate, suggest process changes that reduce future slippages.

  6. Always perform test closure activities and ensure that lessons learned are recorded and reported.

  7. Get testing representation in requirement review meetings and on the change control board.

  8. Foster an attitude of continuous improvement; build on your successes.

  9. As software testers, we have a professional obligation to do our best in assisting our organizations to build quality software. We may not necessarily have the term “manager” in our job title, but we still have the ability to be leaders. We can guide our organizations to creating better software.


Conrad Fujimoto is an expert instructor and consultant. He teaches Software Tester Certification for SQE Training.

George Box, a good friend (and a close colleague to my father), put the problem of getting new ideas adopted this way (from Management Matters by John Hunter):

  1. It won’t work
  2. It won’t work here
  3. I thought of it first

John's book, and blog, discuss the challenges of actually getting improvements put into action in the workplace. Getting past the resistance to new ideas, new ways of working and change is more difficult than it should be. But there are practical steps you can take to get improvements adopted, including those mentioned above.

Sadly, George Box recently passed away. You can see George in this video of him discussing the art of discovery (which is a big part of what software testers do - discovering how the software works).


Related: Growing the Application of Management Improvement Ideas in Your Organization - Outcome and In-Process Measures - Improving Software Development with Automated Tests

By: John Hunter and Justin Hunter on May 2, 2013

Categories: Lean, Testing Strategies, Experimenting

A client informed us that they had created (and used) approximately 3,500 test cases to test the search functionality of their application. They had a strong suspicions that (a) they should be able to test the search functionality of their application with fewer tests, (b) the tests they had accidentally omitted tests of many hundreds of plausible combinations of values that would be useful to test for (but did not know how to precisely identify were those gaps were without a huge amount of work), and that (c) many of these tests were quite inefficient in that they repeated many steps that had already been tested in other tests in the plan (even if they were not 100% duplicative of any other single test in the plan).

This client should have spoken to Lanette Creamer before they got into that situation. Lanette is a testing expert and blogger with ideas worth paying attention to. For example, her paper, Reducing Test Case Bloat, is well worth reading as is her blog.

There are times when what you cut may not be bloat. There are some situations where the decisions are the equivalent of “Do we cut off the arm or the head?” Well, a person can live without an arm. If you are in a situation where you are so time constrained that critical areas will be untested, you can still communicate the risk, be transparent and use a strategy to test the most important areas first. It is possible to plan for and do testing for a very time constrained project.


Of course avoiding this situation is best. Improving testing processes to use the best thoughts and tools is a better option. Cutting the bloat can allow resources to be applied to those areas they are really needed. Often though, people are scared of trying new ideas and cling to old methods, even if that results in the organization having to take increased risks by failing to test critical areas sufficiently. They are just more scared of trying new ideas than of getting away with saying we need more funding if you want more testing.

Lanette's article provides 8 specific suggestions for process improvements to reduce bloat. The first suggestion is to use combinatorial testing tools to greatly improve coverage while reducing workload. Another suggestion is to run the bloat reduction ideas by the stakeholders.

As part of your plan to reduce bloat, it can be helpful to state your assumptions about who is important and where you are placing testing priority and why. When reducing test case bloat you are taking a calculated risk. You are weighing the risk of being unable to test new features by insisting on testing every legacy case against the risk of purposefully not running some tests. When you share your starting assumptions with your stakeholders you offer them the chance to counter with their own assumptions and often you can clarify the boundaries of your testing this way to avoid gaps in testing or duplication.


See the full article for more good ideas on how to get better results for the existing testing resources available to your organization.


Related: Design of Experiments is about Learning ASAP and, in Software Testing, Finding Bugs ASAP - Pairwise and Combinatorial Software Testing in Agile Projects - Cem Kaner: Testing Checklists = Good / Testing Scripts = Bad?

By: John Hunter and Justin Hunter on Apr 24, 2013

Categories: Combinatorial Software Testing, Efficiency, Multi-variate Testing, Software Testing, Testing Strategies

Malcolm Gladwell's three best-selling books are filled with fascinating examples of all sorts of things to underscore the interesting points he makes about a wide variety of topics. I'm currently readying his book "Blink," about how people make process information and make decisions in the 'blink of an eye."

The example he shares about an overcrowded hospital Emergency Room's decision making process really resonated with me. Brendan Reilly, named the chairman of Medicine at Cook's County Hospital in Chicago in 1996, was in dire need of a way to improve the conditions of the hospital when he showed up there. The Emergency Room faced extreme amounts of resource pressure in particular; there were too many patients needing care and too few doctors and beds to serve them all promptly. One of the most resource-sapping processes was the treatment of patients who entered the ER complaining of chest pains.

"From the beginning, the question of how to deal with heart attacks was front and center." About 30 people a day came into the ER "were worried that they were having a heart attack. And those thirty used more than their share of beds and nurses and doctors and stayed around a lot longer than other patients."

"Reilly's first act was to turn to the work of a cardiologist named Lee Goldman. In the 1970's, Goldman...p. 133

He... announced that he was holding a bake-off. For the first few months, the staff would use their own judgment in evaluating chest pain, the way they always had. Then they would use Goldman's algorithm, and the diagnosis and outcome of every patient treated under the two systems would be compared.

For two years, data were collected, and in the end, the result wasn't even close. Goldman's rule won hands down in two directions; it was a whopping 70 percent better than the old method at recognizing patients who weren't actually having a heart attack. at the same time, it was safer. the whole point of chest pain prediction is to make sure that patients end up having major complications are assigned right away to the coronary and intermediate units. Left to their own devices, the doctors guessed right on the most serious patients somewhere between 75 and 89 percent of the time. The algorithm guessed right more than 95 percent of the time.... He went to the ED and changed the rules."


I suspect that if you asked 100 cardiologists whether they would perform more accurate assessments that the 3 question Goldman algorithm would, you would find that all 100 of those cardiologists would feel confident in their ability to outperform the 3 question algorithm. After all, the algorithm didn't allow for subject matter expertise.


How is this Relevant to Software Testing?

When many business leaders think of software testing what they want is something that catches bugs before the product is released to the customer. To some extent this can be automated. There are many previous bugs (and experience with bugs in other software) that allow us to create specific tests which will identify known bugs.

So if we have a form on the web application, we can test various scenarios and make sure the form is processed properly: submitting the form when it should or displaying the proper error message when it should do that. It is wise to have test plans to check for these bugs.

Creating simple easy to follow guidelines to address known issues is a wise action. This is true for creating a simple checklist to use in screening for heart attacks and for creating test plans that cover the known issues.

While those are wise they are by no means sufficient. Physicians bring a great deal of expertise to evaluating patients for a huge number of symptoms. And they can use their knowledge and experience to judge where they need more evidence (for example, a diagnostic x-ray or an analysis of a blood sample) to make a judgement. And to judge is software will work as users wish simply following a test plan robotically is not sufficient. The experience and expertise of a software tester allows them to probe the software applications and spot not only bugs but weaknesses that should be addressed so users of of the software have the best experience.

The appropriate methods depend upon the need. And sometimes the expert is not as effective as simply using tools or a check sheet that has been developed specifically for that purpose. It is important to design processes in your organization to find the most effective strategies given the local circumstances.

Experimenting, by using evidence based management practices, is needed to determine what is most effective. Assuming that the best solution is always relying on the experts is not accurate (as the example Gladwell used shows). A great tool to aid in determining what practices work best is the Plan-Do-Study-Act cycle made famous by Dr. W. Edwards Deming.


Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking - Mistake Proofing Strategies for Software Code - Checklists in Software Development

By: John Hunter and Justin Hunter on Feb 18, 2013

Categories: Software Testing, Testing Strategies

A combinatorial explosion is when the configuration settings and user actions and data entered etc. makes it impossible to test everything. The number of tests required to individually test every single possibility is many thousands of times greater than could realistically be tested.

When faced with taking over an existing software applications without a good test suite (or any test plan) often is daunting. And the problems of creating an unfathomable number of tests face you due to combinatorial explosion. Hexawise is a software as a service that aids in dealing with this dilemma for software testers. Software test plans are created that provide far better coverage than is seen in practice with a tiny fraction of the test required for complete combinatorial coverage (that is testing every possible combination [pairwise or 3, 4, 5... way] individually).

The Google Maps test plan provides a good example of combinatorial explosion faced by the testers (in this case, those who tested Google Maps). Take a look at the Google Maps test plan by login to your Hexawise account (creating a demo account is free and simple). The Google Maps test plan is one of 9 samples currently provided in Hexawise.

For creating your own test plan, while you are exploring the software application and testing it out to find "where the weak points are," you will probably find it useful to vary things as much as possible, repeat your actions as little as possible. Those points are true whether you're doing relatively informal lightly documented exploratory testing or more heavily documented test scripts. It addition, since a large percentage of defects can be triggered by the interaction of just two test inputs, it would be nice, if you had time, to test every single possible combination involving two test inputs; that's the rationale behind allpairs, pairwise and orthogonal array-based test case prioritization methods.

To recreate a similar - very early draft - plan for yourself, I'd suggest going through the following steps to put together a relatively small number of highly informative end-to-end-ish tests:

Ask what can change as users go through the system? Think about configuration settings, user actions, data formats, data ranges, etc. even throw in more "creative" ideas like user personas. Let your creativity and common sense guide you. Enter those in as parameters.

Ask how those parameters can change? (for the parameter "Browser" enter IE7, IE8, FF, etc.) Put those in as values under each parameter (entering constraints as required)

Ask does that variation matter? When possible (when it doesn't matter as much) use equivalence classes and be biased towards fewer values - at least for your early draft tests.

Ask what special paths thorough the system do you want to be sure to include? (Most common happy path, paths to trigger certain business rules, etc.)

Click the Create Tests button in Hexawise and you'll instantly get a very nice draft starter set of highly varied tests. If they look like they're relatively interesting and don't miss hugely important things, start informally executing them and you'll be sure to learn some more things as you do about the system's weak points that would result in you going back to those draft tests and iterating them to make them stronger and cover more.

To get a bit more on using this approach see our case studies. Hexawise TV provides narrated videos online showing how to make your life easier as a software tester.


Related: Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - 3 Strategies to Maximize Effectiveness of Your Tests - Pairwise and Combinatorial Software Testing in Agile Projects

By: John Hunter and Justin Hunter on Jan 2, 2013

Categories: Combinatorial Software Testing, Exploratory Testing, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies

Creating test plans for create, read, update and delete (CRUD) functionality is a very common requirement. There are a few different ways to model it. Let's review the simplest solution first then move to a few optional extra ideas that you could use in addition.

Option 1: Only one C, R, U, D action tested in each of your tests

Have one Parameter called "CRUD action" - have 4 different values: Create, Read, Update, Delete

Option 2: Include two CRUD actions in (most of) your tests

Create 2 Parameters, each with the following 4 and 3 Values:

First CRUD action: Create, Read, Update, Delete
Second CRUD action: Read, Update, Delete

You may want to add an invalid pair with the first CRUD action = Delete and each of the three values in the second CRUD action (in which case you'd need to add a 4th Value called N/A and leave it as the only available option for scenarios that deleted a record in the first test.

Option 3: Up to 4 CRUD actions in each of your tests

  1. Create a new record: create, don't create
  2. Read a record: read, don't read
  3. Update a record: update, don't updat
  4. Delete a record: delete, don't delete

That's the most basic modeling decision. (e.g., one CRUD action / test, two CRUD actions / test, or potentially 4 CRUD actions / test).

What other things might be interesting to vary?

You might want to think about "newspaper reporter questions" - e.g., (who, what, when, where, why? how? how much?)



Which user type (or which system) is performing the CRUD ACTION? - Are certain types of users or systems not allowed to perform certain actions? Include test inputs to make sure they can't. Are others allowed to? Include test inputs in your model to make sure they can.



What kind of CRUD actions are being made? Valid ones? Invalid ones? How many invalid types of actions can you think of? What is supposed to happen when special characters are entered? What is supposed to happen when blanks are entered? What is supposed to happen when extremely long names are entered (e.g., are there concatenation rules?)



When are the updates being made? Are there any system downtimes? What happens to CRUD actions that are attempted then? What happens if the time stamp on a CRUD action is from the future? From the past? Originates in a different time zone?


When? ("version 2")

When are the CRUD actions tested in relation to one another? e.g., Update a record and then try to read it or read it and then try to update it? If you wanted to "mix up the order" of the Option 3 actions, for example, you could add a parameter called perform CRUD actions in what order? with values such as normal create then read then update then delete order, the reverse of the normal order as much as possible, start with read or update. This would not give you a "perfect solution" to cover every timing combination but it would lead to additional variability within your tests if you wanted them.



Why is a CRUD action being conducted? Do any interesting testing ideas come to mind when you think about this angle? How about bad or malicious motivations?



How are the actions being undertaken? By a keyboard? By a mouse? By a batch processing system?

Lastly, if you have an account you created on, check out sample test plan "08. Modifications to a Database" for some additional ideas.

Bottom line: when test designers create tests (whether they use Hexawise to generate their tests or whether they select and document their tests by hand), they will have a lot of discretion over what kinds ideas to include / exclude from their tests. Using Hexawise offers these advantages over selecting tests by hand:

  1. You will maximize variation in your tests
  2. You will minimize wasteful repetition in your tests
  3. You will achieve 100% coverage of the combinations you tell the tool to include in your solution (e.g., pairwise testing coverage, three-way coverage, risk based testing coverage, etc.)
  4. You will be able to select those combinations and documented tester instructions faster and with fewer errors in the documented tests.

This post is based on a user question. See the Hexawise user forum page for some additional details related to the users follow up questions.


Related: Efficient and Effective Test Design - 3 Strategies to Maximize Effectiveness of Your Tests - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

By: John Hunter and Justin Hunter on Dec 18, 2012

Categories: Pairwise Software Testing, Software Testing, Testing Strategies