Of marginal significance...

Have been wondering about the term 'marginally significant' used in research papers. It's a term that wasn't used when I learned statistics a very long time ago. Research results were either significant, at a given level, or not. So I browsed the term and found it is, indeed, acceptable in some settings, for reporting 'non-critical results' (you define what's critical).

Then I found this lovely analysis by epidemiologist, Matthew Hankins, of the p values at which effects are described as marginally significant. I love the annotations.

 And some detailed discussion by Hankins here.

Also a very nice video, 'Dance of the p Values', by Geoff Cumming which starts in the same vein has Hankins' diagram.

The file drawer problem

The Economist's recent article How science goes wrong is a useful reminder of publication biases in science. It recently came up in a discussion at work, which took me back to Rosenthal's paper on 'the file drawer problem' (null results in science are seen as boring and unpublishable, so tend to get filed away, while the study or its analysis is tweaked to produce something that shows a significant effect). Rarely mentioned in research methods teaching.

Rosenthal, R. 1979. The file drawer problem and tolerance for null results. Psychological Bulletin 86/3, 638–641

Rugged mobile phones

In past projects as a user experience researcher I could guarantee that in every piece of mobile phone research, no matter what the topic, someone would ask why phones were so flimsy and ask for a robust (and dustproof and loud) phone that could bounce around on the passenger seat of a white van and survive being stepped on by a working boot. JCB have responded.

[via Chris Heathcote's Twitter stream]

Looking to the future

Fascinating research showing a correlation between a population's predisposition to make future-focused Google searches and GDP. Not causal, of course.

[via John Naughton]

Designing for developing communities

Fastco debates the ethics of user research in developing communities, to help global companies target their products. The discussion is sparked by Jan Chipchase, once a researcher at Nokia, now at Frog, who has made a career of reaching otherwise unresearched locations and whom I suspect may have been in Don Norman's mind when he wrote critically of the relevance of user research 'you get to go to exotic locations, to watch people do intimate acts, and then to come back and tell the world what you have seen.'

I understand Norman's tetchniness; I understand Chipchase's critics who question the use of his research to create artificial need for global products; the communication of some global research projects has made me wonder 'for whose benefit' in the past. But the alternative of imposing unresearched products and services on developing communities seems even less acceptable. As Chipchase points out 'the poor can least afford poor products'. Fastco describes the development of the Firefly infant phototherapy unit, developed by Design That Matters for use in Vietnam, and I'm reminded of the incubator developed by Stanford's d.school, neither of which would have been developed in their current form without research in the context in which they are to be used.

Too much information to visualise...

...so why not develop computer programmes to visualise without human input. Agrawala, Wilmot and Berthouzoz (of Berkeley Computer Science lab and Adobe Systems) have done that and describe their methods of
- analysing hand-drawn visualisations to understand the design principles behind them
- relating those principles to research in perception and cognition
- and then, using visualisation tools based on the principles, testing people's response to the visualisations they produce.
In the process they explain why people find hand drawn maps easier to use than general-purpose maps when trying to find routes (see another discussion if the principles of hand-drawn maps in an interview with Paul Stiff), and they improve on shop assembly instructions for flat-pack furniture.

I like the strong user focus of their studies:
'We find it critical to involve users early on...[their] feedback is essential for identifying problems and ensuring our design principles and the visualisations converge on effective designs.'

Contention around user-centred design

In Interactions, John Zimmerman writes under the title (maybe not his?), 'Killing off user-centred design' that 'Not every project deserves upfront user research. Let me say this again. Not every project is deserving of the time and effort of upfront user research.'

I'm not sure that this is such a dramatic assertion that it warrants the repetition. The practitioners I know don't do research unless it's warranted; they look for ways to deliver clients the insight they need in the most efficient way. For a more nuanced explication watch Jared Spool's talk, Anatomy of a Design Decision, at UserInterface15 in 2010. Spool makes the distinction between the research needed when designing for people similar to oneself (the Apple or Basecamp scenario where, as a user of the product one is designing, no up front user research is needed) or repeat designing in a domain with which one is familiar, to designing experiences for people who differ considerably from you. That's where the research is needed: to understand what gives a good experience, and where there are gaps.

In all cases, though, I'd say testing design as it develops is needed. Even designing for people like you, there may be assumptions that will be verified or challenged by other people trying out your work.

Incidentally, Spool's talk (illustrated above) has meant I can no longer take seriously any school or college brochure that shows (in his words) 'girls under trees'.

Poor memory for intentions

Research by Kaasa, Morris and Loftus shows that memory for past intentions is poor, but that asking people to recall intentions, and then asking them again at a later date introduces a (potentially bogus) consistency across the two recall sessions. (Worth noting that the recall task was linked to a relatively trivial decision (people's reasons for buying a particular piece of music) begging the question whether the results would transfer to more significant decisions, such as reasons for choosing a particular insurance policy etc.) The results are interpreted in the context of police proceedings, where consistency of recall is regarded as an indicator of accuracy, but the research should also give qualitative researchers pause for thought about the accuracy of the reflections they elicit.

[via BPS Research Digest]

The impact of recruitment on research

'Nonetheless, the use of convenience samples place limitations on the kinds of inferences drawn from research. In the end, we strongly endorse the idea that psychological science will be improved as researchers pay increased attention to the attributes of the participants in their studies.'

From a study by Witt, Donnellan and Orlando of selection effects in psychology studies, reported in the BPS Research Digest. And just as true for design research.

[via Ben Goldacre]

Don Norman and poorly researched design

Don Norman is in typical form here. Taking as his starting point, Kevin McCullagh's comments on the relevance of traditional design skills to many modern design problems, Norman is vexed by the simplistic design solutions to difficult social and environmental problems that often win design awards (awards where, to quote, 'the uninformed judge the uninformed'). He links back from these headline winners to the inadequacies of the design education that led to them. I'm not sure a direct link is fair. Often design firms propose solutions to high-profile, complex problems as PR pieces, usually without the context of working with a client who can help focus design work. On the basis that there's no such thing as bad publicity,* these PR proposals seem to work well, and provide affirmation to the design team which, yes, can then result in naivety or optimism about the problems they can tackle well.

In Norman's eyes, though, the responsibility for the over-confidence these solutions represent lies with design education that fails to teach designers how to research problems and test solutions. He thinks design schools  perpetuate a culture that cherry picks superficial research findings, and bends them to its purpose (see Harry Brignull's comments on this tradition). Norman mitigates his criticisms (although I think he's tough enough not to feel any threat) by conceding that design is a different discipline from science and engineering and brings different qualities to product development. He also acknowledges that research in human sciences has its own agenda; not, in many cases, to inform designers.

Many of the issues Norman raises are cultural problems with foundations in the psychology of individuals and teams. I recently gave a talk about working in user experience design to a university research group of artists, designers and psychologists. One of the artists asked why my role was needed since everything I had said about understanding users and testing ideas was so obvious, don't designers do it anyway? As an example I talked about the need to test design solutions and revise them in the light of feedback from people outside the design team, and asked him how he felt when a paper he had written was returned to him for revision, editing etc. Did he ever feel reviewers had misunderstood his intentions, some reluctance to revise? Then imagine those feelings transferred to a team, committed to working on a product or service together together. He began to see what I meant.

*Tangentially, note piece in the NY Times about unscrupulous business, DecorMyEyes, using adverse commentary on the web to drive hits to its site and elevate its ranking in Google searches. Anecdotally, from Google's analytics of this blog, I see that one of the most frequent searches hits my post on IDEO's speculative work on the Bloomberg screen, a project that received as much negative as positive commentary. 

[Norman article via InfoDesign; NY Times article via Tim O'Reilly]

...and statistics

Round up of seven statistical 'sins'. Should be required reading for anyone consuming media research reports.

The power of speed

TechCrunch features Google's new instant previews and is just a little dismissive of the increase in user satisfaction they bring (according to Google, satisfaction increased by 5%, although 5% of what, we don't know). I'm not surprised that the efficiency previews bring makes an impact. Coincidentally John Naughton has just posted the viral video of comedian Louis C. K. talking about the irritation of dialing numbers with zeros in them, and waiting for the dial to cycle round, on old pre-digital telephones. When the technology gets in the way of the end goal, every second counts.

Skimming research findings

Just a note of Harry Brignull's comments on how much design research (especially in web design) is reported as press releases and 'top ten tips', without the context, study design, caveats and any counter findings, typical of scientific reporting, that help qualify the research. A comment by Rob Gillham points to this excellent example of the phenomenom (on a pet hobbyhorse: the impact of making people scroll to find information).

Focus in lifelogging and blogging

I recently read Sellen and Whittaker's excellent review of Lifelogging studies. Their key point is that, so far, prototypes haven't capitalised on what we know of how human memory works, for example
- that we store partial, associatively organised memories of events, rather than total, chronological  capture- that although we want to recall things, we also want to reminisce, and reflect on our memories
- and that one of our challenges is not remembering things from the past, but remembering to do things in the future.
Some interesting details emerge (for example, Gregory Abowd's finding that lecture recordings don't significantly improve students' grades; that whereas trials show meeting recording is popular with participants, it has never caught on as a business tool).

Sellen and Whittaker make the distinction between passive 'lifelogging' and active 'blogging' in which we choose what aspects of our lives to record, one of which is, apparently, food. TechCrunch post on recent funding for food blogging service, Foodspotting, commenting 'Don’t even bother arguing about it. It’s just the way it is'.

And so it would seem: in yesterday's Guardian Martin Parr was encouraging readers to photograph their food:
"When you are away, why not record all of the food that you eat? If someone has spent a lot of time cooking a meal, or if you're going out for a treat, photograph the food. You could make a series of each breakfast, lunch and dinner that you ate. That would be fascinating."

Indeed.

How many people to test

A debate that rumbles on in the usability community is how many participants are needed to test usability. And, as so often with user research, the answer starts with 'it depends'. For example, it depends on whether the users of a particular product or service are likely to have very similar backgrounds (and, hence, have shared prior knowledge, goals etc.) or whether they are likely to have different backgrounds and diverse needs. It also depends on the purpose of the test: is it to demonstrate to someone that, for example, a web site has usability problems and requires re-design (if the site is that bad, very few participants are likely to be needed) or is it to tease out problems in a range of different elements of a web site (in which case more participants will be needed, possibly with testing in short iterations of test, refine, re-test)? Even then, not that many participants are needed, usually fewer than twenty. This is something that people with a traditional market research background find very hard to understand.

But I hadn't realised, although can well believe from experience, that if you want to detect key usability problems, more important than the number of people tested is the number of moderators a test has. Rolf Molich has carried out a series of comparative usability evaluation studies, tracking the usability issues picked up by different teams evaluating the same web sites. Some key findings are summarised here.While there was usually overlap across teams' detection of usability issues, sometimes this was very limited, and individual teams frequently failed to report major issues. With this evidence in mind, Molich warns against expecting that usability testing, conducted in the traditional way, usually by one or sometimes two individuals, will find all problems.

I have never seen 'multiple moderator' built into usability studies, although sometimes it happens serendipitously when time scales demand more rapid testing than an individual can handle. And, if testing is carried out with design team involvement, as I prefer, there's often more than one person watching and feeding back on a test session. Not quite multiple moderator, but at least multiple perspectives. But it's something to bear in mind as a way of maximising the output from usability research.

When young people use voice

Much of my own research and that of others, such as Stefana Broadbent, has mapped out the sharing of communication across different channels and, in particular, the migration of much that was previously communicated by voice to text channels, particularly among young people. In Wired last week, Clive Thompson wrote on 'the death of the phone call', commenting that his phone bills were now much 'smaller', i.e. shorter in length, as fewer calls were itemised.

So it was interesting to see, in contrast, results of a survey by Lisa Campbell Salazar, of young people involved in political activism (e.g. environment, human rights or peace campaigns). Survey participants made heavy use of voice communication. The tasks they described using their phones for were to: 'share their message globally, mobilize protests, fundraise, educate their peers and spread solidarity'.

Not too much detail of the survey itself so it's hard to unpick this. And results were combined across several countries so we don't know how different cultural traditions are contributing to choice of channel. All these caveats aside, the finding fits with my characterisation of voice as having more 'heat' than the relative coolness of asynchronous channels such as SMS or email, or text channels where one can control one's availability. And hence it's the channel people resort to when they need to persuade, negotiate in detail, and get things done.

[Thanks to Pat Kane for the Broadbent reference, to Kristina Langhein for the Thompson reference; Salazar research via Putting People First]

Dilbert of the web development world

Cute!

[From Agents of Digital via Mashable]

Are usability agencies overblown?

As usual, some very insightful comments from Harry Brignull, on the competition usability agencies face from freelancers who offer essentially the same service, without the overhead of expensive offices or testing suites. Recently had a similar conversation with a client whom I was helping select an agency to carry out research. Not to mention that agencies, running at a low level of permanent staff in order to limit their overheads, often, themselves, use freelancers to staff their projects.

Tablet as a supplement to large-screen TV viewing

Nordkapp have published findings of some user research, carried out with Sonera, examining the relationship between tablet computers (shorthand, iPads) and TVs, and more generally how how tablets fit into domestic life. Nice to have these simple studies in the public domain.

My comment on their posting picks up on some very simple ergonomic issues that might limit tablet use: their weight, relative complexity compared to some current technology we use (such as old-fashioned transistor radios, which require a simple flick of a switch to turn them on and off) and (while they're still so new and shiny) squeamishness about using them in watery environments such as bathrooms.

What MRI scans show...and what they might in the future

Very nice exposition by Neurotopia of the limits of interpretation of MRI scans (what does it mean to say that an area of the brain is active; in what way, excitatory, inhibitory, modulatory?), and of new research by Karl Deisseroth and team showing that what we're seeing is excitatory behaviour and, furthermore, tracing the paths of the firing neurons (Deisseroth's technique involves infecting neurons with a gene producing light sensitive channels, extending right along each neuron).

All in rats, so far, which seems a far better place to be than making wild interpretations of scans of human brains.

[via Not Exactly Rocket Science]