The authenticity of user testing

Harry Brignull comments on the artificial status of user tests: noting how in web research more participants (in his experience 10-30% of participants) click to read (or skim) T&Cs than one would anticipate from one's own experience of web use (his estimate 1-3%). He describes this as an 'experimenter effect'.

I sympathise. It's hard to get people to use technology naturally in contrived research settings. But testing still can reveal things that wouldn't have been picked up otherwise e.g. in the kind of sign up process Brignull is commenting on it can show whether form design guides people effectively or trips them up (it's so easy, when familiar with the content of a form to imagine that everyone else will navigate it successfully).

In general I'm all for giving people longer than an hour or so's test session to try out a new product or service, in order to gather more considered feedback. That's time consuming, though, and obviously not as 'clean' as some researchers might like to imply testing is. If user research is given a qualitative focus (why people find things easy or difficult, or interesting or boring, to do) it's far more informative for designers.

The interesting discussion after the post reminded me of this comment by Ian Hildreth after research on thetrainline booking service, "'...web analytics tells you the ‘what’, but it doesn’t tell you the ‘why’. The qualitative stuff really helps you get under the skin of why people are doing stuff on the site.'