So, we've been experimenting with a live customer support feature in our tool lately. We're rolling out the live chat support on a beta basis to see how useful our users find it.

The motivations for creating it were two-fold. Our first motivation was the Mayday button that Amazon Kindle announced recently. How cool is that, right? Live support available on demand any time you want it! Ingenius.

Screenshot-2013-12-04-18.07.32

 

Our second motivation for building a live chat support feature into our tool is that - while software test designers consistently tell us that they find Hexawise's features to be really easy to use - new users will often encounter test design questions as they start using the tool. We wanted to be available instantly to collaborate with them - and help them address questions in real time, like: "How should I be thinking about different ways of defining equivalence classes?" "Given what I'm trying to test in my system, how much detail is too much in this context?" etc. We wanted to be there to help users answer them. We're obsessive about customer service. Having the opportunity to have our expert test designers be a click away from every user of our tool every time they encounter a question. That's just too good an opportunity to pass up.

Early indications of how useful this service is to users are extremely promising. Users are telling us it is an amazingly helpful service. And, while we were worried that we might start to feel too stretched with tons of user questions to answer, we haven't felt that way at all. Interactions have been at manageable volumes. We've found them to be really positive. Many of the interactions have helped us learn about ways to improve our tool and/or how we can make certain test design concepts easier for Hexawise users to understand. Often, customers will click on a "call me" button to talk through questions live by phone rather than by typing back and forth. I'm glad this next conversation was done with keyboards.

This conversation happened about 45 minutes ago. Everyone at Hexawise headquarters is still smiling broadly. It made all of our days and stands apart from the rest. Enjoy!

(17:15:25) Visitor Hi

(17:15:40) Sean Johnson Hey

(17:15:44) Sean Johnson What's up?

(17:15:51) Visitor Hi Sean!

(17:15:57) Visitor Hey, I created 308 test cases

(17:16:02) Sean Johnson k

(17:16:02) Visitor out of a possible 18 trillion

(17:16:07) Sean Johnson nice!

(17:16:11) Visitor I consolidated all my user stories in 6 sprints total

(17:16:13) Visitor to 1 test set

(17:16:14) Visitor which is fine

(17:16:28) Visitor but I noticed from test case # 100 plus to 308

(17:16:37) Visitor most of my test cases are now 'any value'

(17:16:48) Visitor I was wondering if there's an option to force hexawise to pick a value for me

(17:16:52) Visitor but I don't there is

(17:16:56) Visitor but that would be a good enhancement

(17:17:05) Sean Johnson ha! are you spying on me?

(17:17:09) Visitor for 'any value' you can have the app just pick a random one

(17:17:10) Visitor LOL

(17:17:11) Visitor Nope

(17:17:14) Sean Johnson seriously… that's what I'm working on right now

(17:17:19) Visitor NO WAY!

(17:17:20) Sean Johnson what are the odds?

(17:17:24) Visitor O M G

(17:17:26) Sean Johnson yes way

(17:17:33) Visitor I've been meaning to provide that feedback 2 weeks ago

(17:17:39) Visitor but never took the time!

(17:17:46) Visitor I WOULD LOVE TO HAVE THAT FEATURE!

(17:17:49) Visitor Oh

(17:17:51) Visitor in the mean time

(17:17:56) Visitor my testers workaround

(17:18:00) Visitor is to print the test plan

(17:18:17) Visitor and just pick the values randomly from the value expansion list and input parameter list

(17:18:26) Visitor AWESOME Sean!

(17:18:31) Visitor Well let me know when it's available

(17:18:41) Sean Johnson that's really crazy

(17:18:41) Sean Johnson well… I guess I'm working on the right thing!

(17:18:42) Sean Johnson Will tomorrow be soon enough for you?

(17:18:42) Sean Johnson :-)

(17:18:43) Sean Johnson for now… it's going to be hidden

(17:18:43) Sean Johnson you'll add ?full=true

(17:18:43) Sean Johnson to the end of the URL

(17:18:57) Sean Johnson and that'll force Hexawise to fill in the any_values

(17:19:03) Visitor that's AWESOME!

(17:19:06) Visitor I will do the workaround

(17:19:09) Visitor OMG, you made my day!

(17:19:13) Visitor THANKS A TON!

(17:19:15) Visitor :-)

(17:19:18) Sean Johnson I'll send you note this evening or in the morning when it's live on [your company's Hexawise instance]

(17:19:23) Visitor I am so happy

(17:19:24) Visitor LOL

(17:19:28) Visitor Thanks so much

(17:19:32) Sean Johnson thanks for chatting! made my day to know I picked the right next priority.

(17:19:32) Visitor this will make my life easier

(17:19:38) Visitor Oh yeah, totally

(17:19:38) Sean Johnson excellent.

(17:19:58) Visitor I honestly think I'm not the only one who will appreciate this enhancement

(17:20:01) Visitor you guys are the best!

(17:20:03) Visitor Thanks so much

(17:20:05) Sean Johnson :-) we try

(17:20:15) Visitor You all do an amazing job

(17:20:21) Visitor this tool is the best

(17:20:21) Sean Johnson look for an email from me shortly

(17:20:26) Visitor will do

(17:20:27) Visitor thanks!

(17:20:33) Sean Johnson thanks!

We've been working hard for the past ~5 years building and continuously improving Hexawise so that it will be a tool that software test designers find to be extremely useful and - equally importantly - a tool that software test designers will find enjoyable to use. It is hard to put into words how satisfying it is to see an interaction like this one.

By: Justin Hunter on Dec 4, 2013

Categories: Uncategorized

I came across Elisabeth Hendrickson's "Test Heuristics Cheat Sheet" yesterday and developed some pairwise testing (AKA 2-way combinatorial) test cases using many of the good ideas contained in it. I would highly recommend it, I'd recommend you send it (or email a link to this blog post) to everyone on your QA team.

As an indication that the Hendrikson's Test Heuristics Cheat Sheet works well to uncover defects,

  • I wanted to create a set of pairwise tests that could be broadly applicable to test thousands of different applications, I incorporated many ideas from the Test Heuristics Cheat Sheet.

  • I intend to use those inputs to test our test design tool, Hexawise.

  • The way Hexawise works is that users enter "things they want to test" into Hexawise on the first of three screens, the "Define Inputs" screen, then click on "Create Tests." Hexawise then uses a scientific approach to maximizing coverage of the combinations of all the "stuff to be tested" in the fewest possible number of tests. This scientific approach is based on the >40 years of Design of Experiments lessons and includes both pairwise / AllPairs methods as well as more thorough 3-way, 4-way, 5-way and 6-way tests (as well

  • Ironically, even before starting to execute the test conditions suggested by Hexawise, I discovered that the special characters that I had input into the "Define Inputs" screen (which I took from the Test Heuristics Cheat Sheet) triggered a previously unidentified defect in Hexawise itself.

  • The fact that it was triggered so quickly in an application that has been live for a year and used thousands of times is a strong indication that using checklists and cheat sheets can be a great way to efficiently find defects.

Why is using checklists to guide your testing often such an efficient and effective way of finding defects? Here's my top ten list:

  1. The "bad ideas" have already been weeded out.
    1. The ideas on the list have found enough defects to make the author of the checklist think there is value in testing the particular idea.
    2. If you've got a checklist or "cheat sheet" put together by someone as thoughtful and experienced as the Bachs, Bolton, and Hendrickson, you're getting a highly-condensed executive summary version of many of their valuable insights.
    3. All testers go through many, many, "I wonder what would happen if we did this or considered that?" scenarios.
    4. The checklists referenced above represent expertise culled from thousands of testing projects.

  2. Checklists are directly actionable. You can apply them in almost no time at all.
  3. They work well. See Cem Kaner's slides on the Value of Checklists (11 Mb pdf file).
  4. They can easily evolve into some of your most powerful test artifacts.
    • Start with the lists above. See if each of the ideas for tests trigger defects in your Systems Under Test.
    • Find a lot of defects from certain test ideas? Create your own checklist of ideas that worked and iterate them over time.. Consider expanding upon the checklist items and concepts that do bear fruit.
    • Don't ever find defects from certain of the test ideas? Consider dropping those items from the checklists if they don't bear fruit for you (or put tests for those ideas at the back of your lists and only include tests for them if you have extra time).

  5. Checklists include useful, defect-triggering ideas that you may not have thought of on your own.
  6. They're free.
    • No software or books to buy.
    • No courses or conferences to attend.

  7. Using checklists mitigate the risk that you will forget to test for things that you know you should be testing for (but could well forget to test for in any specific instance).
    • As humans, we're naturally forgetful as a species despite our best efforts.
    • Checklists are widely used with good results by doctors, lawyers, pilots, software testers, and people going to grocery stores to minimize the effects of these shortcomings.

  8. Software testing checklists are an efficient way to communicate actionable information.
  9. Software testing checklists are widely applicable to all kinds of software testing.
    • Checklists can be used in creating Unit Tests, Assembly Tests, Product Tests, System Tests, Functional Tests, Load Tests, Performance Tests, User Acceptance Tests, etc.
    • Checklists can be used by Exploratory Testers and "script-everything-in-advance" test-case-centric testers.
    • Checklists can be used in Agile projects as well as Waterfall projects.

  10. Software testing checklists can be easily used in pairwise and combinatorial testing.
    • Using elements from the checklists in a pairwise test will have the added benefit that not only will you test for every one of the testing ideas on the checklist (e.g., XXX) but also, you can easily test for every idea on the checklist **in combination with** every other test idea on the checklist in at least one test case.

    By: Justin Hunter on May 3, 2012

    Categories: Checklists, Software Testing, Testing Checklists, Uncategorized

Request for ideas: What are good examples of good interactive online training (including exercises & quizzes) for how to use web apps?

We continue to look for ways to improve the ability of our users to add value to their organizations. We are always in the process of improving the software tools we provide.

The value proposition for using pairwise and combinatorial tools are huge. The biggest roadblocks we see in adoption are not being aware of the advantages and not being sure how to proceed once the advantages are seen.

We designed Hexawise to be very easy to use. But even so the concepts behind combinatorial testing take some people a bit of time to operationalize. To help with this we offer on site and off site personal training.

We are looking to create some online training to ease the transition into using Hexawise's combinatorial software testing tools most effectively. We would love to know what good examples of interactive online training people have found useful.

 

Related: Why isn't Software Testing Performed as Efficiently and Effecively as it could be? - In Praise of Data-Driven Management (AKA "Why You Should be Skeptical of HiPPO's")

By: John Hunter on Apr 24, 2012

Categories: Uncategorized