User Testing

User Testing, and users can be testing…

User Testing, how many and how often?…

Gone are the days when you asked your mother to check out your website. Your mother loves everything you do. But what about the critical, impatient, spoiled consumer?

MY FAVOURITE USER TESTING PROJECTS

MetLife People Finder self service profile

OUP librarian websites

User testing has been made simpler by sites like usertesting.com. There is however a science to user testing because, when using statistical analysis, you have to make sure that the data that you gather is relevant to the results for which you’re looking. Creating case studies and asking the right questions is essential to achieve good results. There is a world of difference between asking a sample user:

1: “How easy is it for someone to buy shoes on this site?”

or

2: “I want you to buy a pair of shoes on this site that are this size, this color and in this style.”

User testing doesn’t have to be complicated. Jakob Nielsen’s group have found “A usability test with 5 users will typically uncover 80% of the site-level usability problems plus about half of the page-level usability problems on those pages that users happen to visit during the test.”

It takes 39 hours to user test a website the first time you try. This time estimate includes planning the test, defining test tasks, recruiting test users, conducting a test with five users, analyzing the results, and writing the report. With experience, Web user tests can be completed in two work days.

In a recent project at the Technical University of Denmark, Rolf Molich and Christian Gram collected data from 50 teams of students who conducted usability tests of commercial websites as part of a user interface design class. The average time spent by each team was 39 hours. Furthermore, the students had sat through 15 hours of lectures on user test methodology. These numbers are an upper estimate of the time investment the first time you decide to user test your site.

With experience it is possible to conduct much more rapid user tests of a site. Good test tasks can be written in one or two hours, recruiting can be outsourced to a recruiting firm (at a cost of less than $1,000 for five users), the actual test can be done in a day, and the results can be analyzed in a few hours. If you are a member of the design team, then there is no reason to write an extensive report which nobody will read, so reporting can be done in a one-hour meeting supplemented by a summary that takes 2-3 hours to write. In total, a discount usability study takes only two work days once you know what you are doing.

Even though experts can do the work more efficiently (and usually with better results), it is encouraging that utter beginners could complete a full Web usability project in less than one week. This truly proves that “limited budget” and “lack of time” are not valid excuses for inflicting difficult sites on your users.

The students tested nine large Danish sites: seven sites from major Danish corporations as well as the University’s own site and the university library’s site.

After rating the usability problems for severity, the study found that each site had an average of

  • 11 usability catastrophes
  • 20 serious usability problems
  • 29 cosmetic problems

In this study, a “catastrophe” was defined as a usability problem that prevented the user from completing a task. A “serious” problem was one that slowed down users significantly but did allow them to complete their task. A “cosmetic” problem delayed users slightly or annoyed the users as indicated by their verbal comments.

My experience with user testing covers the spectrum from publishing scholarly works to a global trade organization with 330 different clients in 100 countries.

Useful resources

WebTrends (http://webtrends.com)

ClickTale (http://www.clicktale.com)

  • Use ClickTale to analyze the performance of your online forms, keep visitors engaged in page content, and lead them through the conversion process. Only available for viewing desktop sites.
  • Generates heat maps of the desktop site and can record customers’ sessions.
  • Video tutorials – http://www.clicktale.com/support/video-tutorials
  • Wiki – http://wiki.clicktale.com/Article/Main_Page
  • If anyone needs training on using ClickTale, please contact our rep Esther Shatz directly at esther@clicktale.com or give her a call at 1 (800) 807-2117 x300. She will set you up with a training session via WebEx.

Optimost (http://www.autonomy.com)

  • “Gives us the What”
  • Offers A/B and multivariable testing solutions for testing Websites, including landing pages, sign-in and shopping basket pages, graphics and text, Web forms, product pages, and email campaigns.
  • Tunde suggested that you have at least 3-4 different scenarios to test, but remember that the more scenarios you use, the longer it’s going to take to get a large enough sample size.
  • For projects relating to ecommerce, you need to get approval from Alex Bennett or Liz Marshall.

Opinion Lab (http://www.opinionlab.com)

  • “Gives the Why”
  • Voice of Customer (VoC) feedback
  • Is being tied to ClickTale to view a recording of the negative (or positive) experience the customer had.
  • Video about Opinion Lab (basically a commercial) – http://youtu.be/mGzAVNDXOIg