Variability in User Performance
Summary: When doing website tasks, the slowest 25% of users take 2.4 times as long as the fastest 25% of users. This difference is much higher than for other types of computer use; only programming shows a greater disparity.
Introduction
Anyone who's done user testing knows that there are tremendous individual differences among users. Some people sail through user interfaces, while others get bogged down. Even if you've never performed a formal measurement study, you've probably noticed that the fastest users are much faster than the slowest ones.
To better grasp this variability, we can look at the ratio between the top and bottom quartiles of a given study's measured task times:
Q3 is the highest number in the third quartile (the slowest 25% of users).
Q1 is the highest number in the first quartile (the fastest 25% of the users). More precisely, at Q1, 25% of users are faster and 75% are slower; at Q3, 75% of users are faster and 25% are slower. So, half of the users lie between Q1 and Q3 and the other half are evenly distributed outside this interval. We divide Q3 by Q1 to compute the Q3/Q1 ratio as a measure of individual differences between users at the low and high ends of performance.
Example Study Results
The following figure shows an example of Q1 and Q3 from one of our eyetracking studies, in which seventy-six users tried to find the location of the Agere Systems corporate headquarters using the company's website. The data plotted is for the forty-eight users who identified the correct city. (While we don't use the times from the twenty-eight users who failed the task, understanding what caused their failure is obviously important as well; see separate article on how to improve the "about us" info on a website.)
Q3/Q1 shows how much better a fast user is than a slow user. That is, it compares a somewhat fast user (25% are even faster, but 75% are slower) with a somewhat slow user (25% are even slower, but 75% are faster). The ratio doesn't consider the very fastest or very slowest users, since they're likely to be outliers.
Across our recent measurements of seventy website and intranet tasks, Q3/Q1 = 2.4. In other words, slow users spend more than twice as much time as fast users on the same task.
Comparison With Other User Interfaces
Dennis Egan has compiled studies of three interaction types: text editing, information search, and programming. In my own unpublished 1994 study, I collected performance metrics for common personal computing tasks on three different systems: Macintosh System 7, NeXTStep (the foundation for Mac OS X), and Windows 3.1. Sample tasks included adding up sales figures and sending the results out by email.
The following table shows the average Q3/Q1 ratios from the traditional-use studies compared with that of Web use:
Type of Use >> Q3/Q1
Text editing >> 1.8
Personal computing >> 1.9
Information search >> 2.2
Web use >> 2.4
Programming >> 3.0
The more difficult a problem, the more individual differences we see.
When using a website, for example, a user who can hold six chunks of knowledge in short-term memory has great superiority over someone who can hold only four chunks.
The user with the better memory is less likely to repeatedly go down the wrong path and more likely to correctly assess how a given page relates to previous pages.
In contrast, a higher-capacity short-term memory doesn't help much in simple text editing tasks, assuming you have a decent word processor that doesn't require you to remember six things to move a paragraph.
As the table shows, the Web has the second-highest individual variability of the five computer-use types.
Programming has the largest individual differences and is the most difficult task category. However, programming is not the worst problem because we can legitimately select our programmers from among those with the best performance. That is, the solution is simple: don't hire bad programmers.
For websites, we don't have the luxury of selecting only the best users. We must cater to the people who visit our website, regardless of their abstract reasoning skills. People in the last quartile are customers, too.
Source:
Jakob Nielsen's Alertbox, May 15, 2006:
Variability in User Performance
http://www.useit.com/alertbox/performance_variability.html
Variability in User Performance (Jakob Nielsen's Alertbox)
No comments:
Post a Comment