Software testing is more than just automated testing.  While checking that specific details work as expected in specific situations with automated testing is very useful it is far from sufficient way to test if software will delight users.

Apple's Siri (and other "smart speaker" systems) are amazingly useful.  They all have plenty of room for improvement and effective software testers could help improve them.  Here are some of my experiences using Siri.

I use a very small set of commands (under 5, such as "Hey Siri, play business news" probably amounts to 95% of my requests).  I understand voice recognition isn't easy and I can understand Siri having difficulty understanding what is asked but it really seems to me that Siri should be able to remember that I ask for the same 5 things 95% of the time.  Check my request against those first.  Siri still attempts to play music and do other things easily 33% of the time (when I have never asked it to play music).  It never can find a matching song (often looking for a song named "business news update" or something close).

When I am on a multi-state trip why can't Siri tell me the next gas stations on my route?  When I ask Siri it will say "gas station at 123 main street" in some town I have never heard of.  I have no idea where that is.  I have been able to determine it seems to just look for the closest ones (which might be 20 miles behind me).  This seems like a poor implementation. Finding the closest makes sense in some instances but while driving on a long trip, along the route makes much more sense. The same goes for restaurants.  I have tried telling Siri "along the route" and similar things but it doesn't work.

Another odd error I experienced recently, I asked Siri for the business news update and it played one from a day or 2 earlier. That is likely a difficult thing to catch with testing and could be "blamed" on the provider of the update. The user (me in this case) doesn't know if the problem is somehow with Siri or, what I expect, is related to the provider uploading the wrong content for Siri.  But as a user of Siri it is a problem.

Another problem I experienced was that I would request Siri play the business news and then Siri would tell me the business news would play and then nothing happened.  It took me a day or 2 to realize the sound was muted. Is that a Siri problem?  Siri is responding to the user (me), and that response isn't muted, but then the content I asked for is muted: that is a confusing user experience.  Since Siri communicates using sound if the phone is muted (with Siri ignoring the muting directive) then it should at least tell the user that your speaker is muted so you can't hear the business news unless it is unmuted.  It seems to me the best interaction would be to unmute the phone to play what was asked (it seems clear the user desired to hear what they requested), but if not then ask the user if they want to hear what they requested.

Again on a long trip Siri misimplemented a request (one of my few commonly asked requests) to terminate the directions. It did so in an area with a very weak signal so when I attempted to have it reestablish the trip it took quite a long time to do so.  The poor implementation is made worse because Apple Maps doesn't let you save maps for offline use.  If the trip is active then you can continue and get directions even if you have no connection; but if you lose your internet connection and the trip is closed (by Siri or you, say if you follow the directions to a restaurant - which closes you main trip) and you don't have an internet connection you are left with no usable map.  So closing an active trip is a potentially destructive act (which it seems to me Siri should be very careful to avoid without clear direction, Siri should get confirmation before doing so).

Software testing requires searching for bugs that everyone agrees are bugs.  But software testing also requires thoughtful analysis of the experience of the user and how that is at risk with the current iteration of the software (Software Testers Are Test Pilots).  Automated testing is critical as you can create checks for thousands of situations to run extremely quickly each time any changes are made to the software.  But it is not sufficient.  Many situations can only be explored and experienced by a thinking software tester that uses the software and thinks about how a user might react and how that user could be confused or disappointed by the existing user experience.

Related: Maximize Software Tester Value by Letting Them Spend More Time Thinking - 10 Questions to Ask When Designing Software Tests

Header photo by Tim Foster  Photo of person holding a phone by Yura Fresh