Nov 4 - Rubin, Developing the Test Plan

Chapter 5
Developing the Test Plan


Introduction

The test plan is the foundation for the entire test. It addresses the how, when, where, who, why, and what of your usability test.

Following are some important reasons why it is necessary to develop a comprehensive test plan...
1. It serves as a blueprint for the test.
2. It serves as the main communication vehicle among the main developers, the test monitor, and the rest of the development team.
3. It describes or implies required resources, both internal and external.
4. It provides a focal point for the test and a milestone for the product being tested.

Suggested Format

Test plan formats vary according to the type of test and the degree of formality required in your organization.

The following are the typical sections to include...
* Purpose
* Problem Statement / Test Objectives
* User Profile
* Method (Test Design)
* Task List
* Test Environment / Equipment
* Test Monitor Role
* Evaluation Measures (Data to be collected)
* Report Contents and Presentation.

Employ an Adequate Number of Participants

For achieving statically valid results, small sample sizes lack the statistical power to identify significant difference between groups.
For a true experimental design, a minimum of 10 to 12 participants per condition must be utilized [124].
However, for the purpose of conducting a less formal usability test, recent research has shown that four to five participants will expose 80 percent of the usability deficiencies of a product, and that this 80 percent will represent most of the major problems [138].

Be Consistent

1. Use scripts.
2. Use checklists.
3. Have the same person, if possible, conduct all test sessions.

Task List

The task list is comprised of those tasks that the participants will perform during the test. The list should consist of tasks that will ordinarily be performed during the course of using the product, documentation, and so on.

Sample Performance Measures

Time to complete each task.
Number and percentage of tasks completed correctly with and without assistance.
Number and percentage of tasks completed incorrectly.
Time required to access information in the manual.
Time required to access information in online help.
Time needed to recover from error(s).
Time spent reading a specific section of a manual.
Time spent talking to a help desk.

Count of all incorrect selections (errors).
Count of errors of communication.
Count of errors of omission.
Count of incorrect menu choices.
Count of incorrect icons selected.
Count of calls to the helpdesk.

Count of user manual accesses.
Count of visits to the index.
Count of visits to the table of contents.
Count of "negative comments or mannerisms."

Sample Preference Measures

Ratings and rationale concerning:
* Usefulness of the product.
* How well product matched expectations.
* Appropriateness of product functions to user's tasks.
* Ease of use overall.
* Ease of learning overall.
* Ease of setup and installation.
* Ease of accessibility.
* Usefulness of the index, table of contents, help, graphics, and so on.
* Help desk replies to inquiries.

Preference and rationale for:
* One prototype vs another prototype.
* This product vs a competitor's product.
* This product's conceptual model vs the old model.


Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York, 1994.