Sep 19 - Nielsen, Multiple-User Simultaneous Testing (MUST) (Alert)

Multiple-User Simultaneous Testing (MUST)

Summary: Testing 5-10 users at once lets you conduct large-scale usability testing and still meet your deadlines.

Sometimes you need to test a large number of users. ...Unfortunately, this often gets you into serious trouble with project deadlines.
Alternatively, you can use multiple-user simultaneous testing, or MUST (a term I have from Dennis Wixon). As the name indicates, with MUST, you test multiple users at the same time so you get done sooner. In most MUST studies, we test 5-10 users at once...

When to Use MUST

Most usability studies should be simple and small-scale, but in some scenarios, it's useful to conduct MUST:

* For quantitative studies and benchmarking, you typically need to test at least 20 users per condition in order to get statistical significance.

* For long-duration tasks, you need to test each user for days or weeks to observe valid behaviors. Examples here include:
*** Developer tools. You can't test a system to support professional programmers by having users develop and debug a 20-line "hello world" program; users must work through an industrial-scale problem. The same is true for other high-end problem-solving applications, such as CAD.
*** E-learning. You can't test lesson 39 unless the learners have first made it through lessons 1-38. For a sufficiently advanced e-course, each test could take a week or more.

* Usability focus groups. To alleviate the problems with traditional focus groups, each participant should start with a one-on-one session testing the live user interface. Following the test sessions, participants can then congregate to discuss the experience and how it relates to their everyday needs. This method definitely requires MUST because all participants should test the interface just before the focus group meets.

* Games design. I describe this case in detail below.

How to Run Many Users Simultaneously

When testing many users, you usually need many test facilitators.

If you're one of the lucky few companies with many usability specialists, they can facilitate MUST sessions. This is expensive, but efficient: All you need to do is turn the experts loose, since they already know how to run a study.
Most companies, however, don't have enough usability pros to assign one to each test user. Happily, non-usability staff can run user test sessions, especially if a seasoned usability expert has prepared the test plan and written the tasks.

Training Facilitators

Ideally, your new facilitators would go through a full usability training workshop, but this is rarely possible in practice. Still, it's best if you spend at least a few hours training facilitators before giving them a go with real users:

First, of course, you should explain the theory and best practices of user testing, including steps such as "keep quiet and let users do the talking," which I've discussed many times before.

Second, newbie facilitators should watch an experienced usability expert run a sample session with a pilot user. Doing so
* shows newbies how to facilitate a study, and
* better concretizes the test plan and test tasks than simply discussing them or going through them on paper.

Third, conduct a role-playing exercise in which the usability expert plays the user and simulates difficult situations that facilitators can encounter, such as users who don't talk or users who ask if they can use certain features.

Preparing the Users

There's very little special preparation to do for MUST study participants. Just follow standard procedure for recruiting test users, welcome them to the session, give them consent forms and instructions, and so on.

However, the actual MUST sessions differ from traditional sessions in two key ways:
* Thinking aloud doesn't work when people are tightly packed into small cubicles, so you shouldn't include the usual instruction for users to vocalize their thoughts as they move through your design.
* You can minimize the distraction of having multiple users by telling participants that they'll likely be working on different tasks. This reduces their natural inclination to look at other users' screens and also prevents people from feeling stupid if other people finish before them and leave the room.

Simpler MUST Labs

You can run MUST studies without the fancy lab. The following photo shows a study we ran recently, where we tested 5 users in each session.

In our lab, we simulated cubicles by taping cardboard partitions to each desk. Of course, it's better to use real cubes, but our discount cubes worked swimmingly. We used slave monitors in each cubicle so facilitators could get a good view of the action on their user's screen without having to lean in. However, for most studies, it's perfectly fine to use a single monitor.

In addition to cubicles, we've used three other setups for previous MUST studies:

* Single-person offices for each user. (For an intranet study, we used participants' actual offices.) Individual offices can be in different parts of the building (or even in different buildings) so long as each user is assigned a facilitator who stays for the session's duration.

* Large-scale usability labs with a row of test labs. In such cases, we tested each user in a separate lab. This setup is particularly suited for studies in which users work on large-scale problems for days at a time, and a few facilitators move around among users. Because the facilitators can enter and leave each lab's observation room without the user's knowledge, it's possible to watch many users in a day without disrupting their concentration. Also, you can leave users who are working on a part of the task you don't care about without communicating this fact and thereby biasing their behavior.

* Offices turned into a networked lab. This case mixes the previous two setups: many single-user offices are wired to a single observation room. You can do all this over the local-area network, which can carry the streaming screendumps for a remote slave monitor as well as webcam views of the users.

Ultimately, the vast majority of usability studies should be qualitative and test 5 users.

Still, there are situations in which you need more, and that's when it's nice to have MUST in your toolkit so that you can get the study done before your deadline.

source:

Jakob Nielsen's Alertbox, October 15, 2007:
Multiple-User Simultaneous Testing (MUST)
http://www.useit.com/alertbox/multiple-user-testing.html

Multiple-User Simultaneous Testing: MUST (Jakob Nielsen's Alertbox)