How many people to test

A debate that rumbles on in the usability community is how many participants are needed to test usability. And, as so often with user research, the answer starts with 'it depends'. For example, it depends on whether the users of a particular product or service are likely to have very similar backgrounds (and, hence, have shared prior knowledge, goals etc.) or whether they are likely to have different backgrounds and diverse needs. It also depends on the purpose of the test: is it to demonstrate to someone that, for example, a web site has usability problems and requires re-design (if the site is that bad, very few participants are likely to be needed) or is it to tease out problems in a range of different elements of a web site (in which case more participants will be needed, possibly with testing in short iterations of test, refine, re-test)? Even then, not that many participants are needed, usually fewer than twenty. This is something that people with a traditional market research background find very hard to understand.

But I hadn't realised, although can well believe from experience, that if you want to detect key usability problems, more important than the number of people tested is the number of moderators a test has. Rolf Molich has carried out a series of comparative usability evaluation studies, tracking the usability issues picked up by different teams evaluating the same web sites. Some key findings are summarised here.While there was usually overlap across teams' detection of usability issues, sometimes this was very limited, and individual teams frequently failed to report major issues. With this evidence in mind, Molich warns against expecting that usability testing, conducted in the traditional way, usually by one or sometimes two individuals, will find all problems.

I have never seen 'multiple moderator' built into usability studies, although sometimes it happens serendipitously when time scales demand more rapid testing than an individual can handle. And, if testing is carried out with design team involvement, as I prefer, there's often more than one person watching and feeding back on a test session. Not quite multiple moderator, but at least multiple perspectives. But it's something to bear in mind as a way of maximising the output from usability research.