Sep 23 - Nielsen, Formal Usability Reports vs. Quick Findings (Alertbox)

Formal Usability Reports vs. Quick Findings

Summary: Formal reports are the most common way of documenting usability studies, but informal reports are faster to produce and are often a better choice.

Introduction
I recently asked 258 usability practitioners which methods they use to communicate findings from their studies:
42% produce a formal written test report with full details on the methodology
36% write a "quick findings" report
24% circulate an email that lists the study's top findings
15% disseminate a spreadsheet of the findings
14% enter usability findings into a bug-tracking database
21% conduct a meeting in which they offer a formal presentation of the findings
27% conduct an informal meeting or debriefing to discuss the findings
1% show full-length videos from the test sessions
4% show highlights videos from the test
3% create and display posters or other physical exhibits

There's no one best approach to reporting usability study findings. Most people use more than one method, depending on their corporate culture and usability lifecycle approach.
That said, the survey clearly found that formal and brief reports are the two dominant means of disseminating usability findings. Both approaches have their place.

When to Use Quick Findings Reports

You can maximize user interface quality by conducting many rounds of testing as part of an iterative design process. To move rapidly and conduct the most tests within a given time frame and budget, informal reports are the best option.

Preparing a formal slide-based presentation will simply slow you down, as will using videos or statistics. Instead, simply hold a quick debriefing immediately after the test, structured around test observers' informal notes on user behavior. Follow this meeting with a short email to the entire team (the shorter the email, the greater the probability that it will be read).
Some organizations thrive on formal presentations and slide-deck circulation. In my view, this is a poor method for documenting usability findings. Bullet points don't capture the subtleties of user behavior, and it's almost impossible to interpret a slide presentation even a few months after it was created.
Extremely brief write-ups work well for studies aimed at finding an interface's main flaws to drive design iterations. Such studies are largely throwaway; once you've created the design's next version, the study findings are rarely useful.

When to Use Formal Reports

The formal report remains the most common format, but I think it's overused and prefer more rapid reporting and more frequent testing. The formal report definitely has its place, however, as in cases like these:

Benchmark studies or other quantitative tests. Unless you document the measurement methods in detail, you can't judge the numbers. Also, one of the main reasons to measure a benchmark is to measure it again later and compare. To do so, you need to know everything about the original study.

Competitive studies. When you test a broad sample of alternative designs, the resulting lessons are usually so fundamental and interesting that they warrant a complete report, with many screenshots and in-depth analysis.

Field studies. Most organizations rarely conduct studies at customer locations; when they do, the findings deserve an archival report that can be used for years. Also, field studies usually produce important insights that are too complex to be explained in a quick write-up.

Consulting projects. When you hire an expensive consultant to evaluate your user experience, you should milk the resulting insights for a long time to come. The outside perspective is only valuable, however, if it remains inside once the consultant has gone. To ensure this, you need a report that's both insightful and comprehensive.

Conclusion
The best usability reports are learning tools that help form a shared understanding across the team. It's worth investing the effort to produce a few formal reports each year. One way to free-up resources and make some reports extra good is to scale down your ambitions for most of your everyday reports, keeping them quick and informal.

Source:
Jakob Nielsen's Alertbox, April 25, 2005:
Formal Usability Reports vs. Quick Findings
http://www.useit.com/alertbox/20050425.html
Formal Usability Report vs. Quick Findings (Jakob Nielsen's Alertbox)