20100922 - Dykstra, Domain-specific Heuristic Checklist

A comparison of heuristic evaluation and usability testing: The efficacy of a domainspecific heuristic checklist by Dykstra, Dean Julian, Ph.D., Texas A&M University, 1993 , 228 pages; AAT 9410761 My Interest: (1) Calendar Checklist = UET = domain-specific heuristics checklist. (2) Comparison methodology of the 3 usability evaluation methods. (3) Results comparison – software developers vs usability professionals. Action: To read this Dissertation in future. Motivation The need for cost-effective usability evaluation has led to the development of alternative usability methods. Reducing the number of participants in a usability test or using heuristic evaluation are the two approaches most often associated with discount methods. Research Objective In this dissertation, an alternative heuristic method was developed based on the concept of a double specialist (knowledge of usability techniques and of a software product domain). Double specialists were previously shown to provide heuristic evaluation superior to that of regular usability professionals, primarily because of their knowledge of usability problems typically found within the given domain. Similar knowledge, based on competitive usability testing, was used to devise a domain-specific heuristic checklist. Context The product domain chosen for the research was online calendars. The calendar checklist was compared to general heuristics and to streamlined usability testing with two usability test subjects. Comments: 3 evaluation types were probably: (1) calendar checklist (domain-specific heuristics checklist), (2) general heuristics, (3) streamlined usability testing. Methodology Fifteen software developers and fifteen usability professionals were each randomly assigned to one of the three evaluation methods. They were then provided with a previously unseen software calendar and were asked to find as many usability problems as possible. Results Discussion Participants using the calendar checklist found more total problems, and a significantly higher percentage of user-oriented problems and severe problems than did those using the general heuristic method, suggesting that domain-specific heuristics do provide an improved heuristic approach. Streamlined usability testing, however, was found to be the most effective discount usability method. Usability professionals and software developers did not differ in their ability to find user-oriented or severe problems. Conclusion It was concluded that when time and resources permit, usability testing is the method of choice. When user testing is not feasible, however, domain-specific heuristics may provide results approaching the effectiveness of the double specialist. Comment: I enjoy reading this Abstract; written beautifully and systematically.

Nov 4 - Rubin, Developing the Test Plan

Chapter 5
Developing the Test Plan


Introduction

The test plan is the foundation for the entire test. It addresses the how, when, where, who, why, and what of your usability test.

Following are some important reasons why it is necessary to develop a comprehensive test plan...
1. It serves as a blueprint for the test.
2. It serves as the main communication vehicle among the main developers, the test monitor, and the rest of the development team.
3. It describes or implies required resources, both internal and external.
4. It provides a focal point for the test and a milestone for the product being tested.

Suggested Format

Test plan formats vary according to the type of test and the degree of formality required in your organization.

The following are the typical sections to include...
* Purpose
* Problem Statement / Test Objectives
* User Profile
* Method (Test Design)
* Task List
* Test Environment / Equipment
* Test Monitor Role
* Evaluation Measures (Data to be collected)
* Report Contents and Presentation.

Employ an Adequate Number of Participants

For achieving statically valid results, small sample sizes lack the statistical power to identify significant difference between groups.
For a true experimental design, a minimum of 10 to 12 participants per condition must be utilized [124].
However, for the purpose of conducting a less formal usability test, recent research has shown that four to five participants will expose 80 percent of the usability deficiencies of a product, and that this 80 percent will represent most of the major problems [138].

Be Consistent

1. Use scripts.
2. Use checklists.
3. Have the same person, if possible, conduct all test sessions.

Task List

The task list is comprised of those tasks that the participants will perform during the test. The list should consist of tasks that will ordinarily be performed during the course of using the product, documentation, and so on.

Sample Performance Measures

Time to complete each task.
Number and percentage of tasks completed correctly with and without assistance.
Number and percentage of tasks completed incorrectly.
Time required to access information in the manual.
Time required to access information in online help.
Time needed to recover from error(s).
Time spent reading a specific section of a manual.
Time spent talking to a help desk.

Count of all incorrect selections (errors).
Count of errors of communication.
Count of errors of omission.
Count of incorrect menu choices.
Count of incorrect icons selected.
Count of calls to the helpdesk.

Count of user manual accesses.
Count of visits to the index.
Count of visits to the table of contents.
Count of "negative comments or mannerisms."

Sample Preference Measures

Ratings and rationale concerning:
* Usefulness of the product.
* How well product matched expectations.
* Appropriateness of product functions to user's tasks.
* Ease of use overall.
* Ease of learning overall.
* Ease of setup and installation.
* Ease of accessibility.
* Usefulness of the index, table of contents, help, graphics, and so on.
* Help desk replies to inquiries.

Preference and rationale for:
* One prototype vs another prototype.
* This product vs a competitor's product.
* This product's conceptual model vs the old model.


Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York, 1994.

Nov 4 - Rubin, Setting Up A Testing Environment

Chapter 3
Setting Up A Testing Environment


SIMPLE SINGLE-ROOM SETUP

Advantages
1. The test monitor has an excellent sense of what is going on with the participant.
2. During early phase exploratory tests, where much interaction is desired to interrogate the participant during the test, this position accentuates a sense of teamwork.
For difficult tests, where the participant has to struggle with the material, it enables you to encourage and overcome the participant's self-consciousness.

Disadvantages
1. The test monitor's behavior can affect the behavior of the participant.
2. There is very limited space for observers.

MODIFIED SINGLE-ROOM SETUP

Advantages
1. The test monitor is more free to move about, take notes, use data logging software while test is going on, and yet is still within visual proximity of the participant.
2, The participant does not have a complete sense of isolation, since the participant is still in the room with the test monitor.
3. This setup is more likely to encourage the participant to think aloud, than if left alone in the test room.

Disadvantages
1. Loss of proximity to the participant limits what the test monitor can see directly of the proceedings.
2. If the test monitor is directly behind the participant and within ten feet or so, the test monitor would make the participant feel very uneasy and overly self-conscious, since the test monitor is not within the participant's peripheral vision.
3. As with simple single-room setup, there is limited space for observers.

ELECTRONIC OBSERVATION ROOM SETUP

Advantages
1. All the advantages of the simple single-roon setup also accure to this setup.
2. Observers get to view the test as much as they like without having to worry about interferring.

Disadvantages
1. As with the simple single-room setup, the test monitor's behavior can adversely affect the test.
2. Unless you have a permanent setup, you will need to tie up two conference rooms for up to a week.

CLASSIC USABILITY LABORATORY SETUP

Advantages
1. It allows unparalleled unobstrusive data collection.
2. Being a soundproof room, the testing staff and observers can easily communicate among themselves about the proceedings and discuss possible product improvements.
3. This environment can accomodate many observers viewing the test at one time.

Disadvantages
1. Depending on the skill of the testing staff, the control room setup can create a very impersonal environment.
2. Unless you have ideal camera placements, you may not be able to see exactly what the participant is viewing or experiencing.
3. For exploratory tests, the control room setup may offer litter advantage to monitoring the test, since you will need to be next to the participant in any case.

MOBILE, OR TRAVELING, LAB

Advantages
1. This is a very cost-effective solution since no costs are incurred for a physical location.
2. Since the equipment is portable, you are free to rent space near existing customers, such as a hotel conference room or a company sales office.
3. Storing all equipment in a locakable cart decreases the probabilty of equipment becoming damaged or misplaced and makes it very easy to "set up" the test site in very little time.

Disadvantages
1. Not having a permanent space means having to settle for whatever space is available.
2. If you will be doing a lot of testing, moving around delicate equipment can play havoc on that equipment.


Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York, 1994.

Nov 4 - Rubin, Overview of Usability Testing

Chapter 2
Overview of Usability Testing


Goals of Testing

The overall goal of usability testing is to identify and rectify usability deficiencies existing in computer-based and electronic equipment and their accompanying support materials prior to release. The intent is to ensure the creation of products that:
* are easy to learn and to use.
* are satisfying to use.
* provide utility and functionality that are highly valued by the target population [49].

More specific goals or benefits of testing are:
* Creating a historical record of usabillity benchmarks for future releases.
* Minimizing the cost of service and hotline calls.
* Increasing sales and the probability of repeat sales.
* Acquiring a competitive edge since usability has become a market separator for products.
* Minimizing risks.

Limitations of Testing

Even the most rigorously conducted formal test cannot, with 100 percent certainty, ensure that a product will be usable when released.
Here are some reasons why:
* Testing is always an artificial situation.
* Test results do not prove that a product works.
* Participants are rarely fully representative of the target population.
* Testing is not always the best technique to use.

Basics of Test Methodology

Employing true experimental designs, these require that:
* A hypotheses must be formulated.
* Randomly chosen (using a very systematic method) participants must be assigned to experimental conditions.
* Tight controls must be employed.
* Control groups must be employed.
* The sample (of users) must be of sufficient size to measure statistical significance between groups [124].

Basic Elements of Usability Testing

1, Development of problem statements or test objectives rather than hypotheses.
2, Use of a representative sample of end users which may or may not be randomly chosen.
3. Representation of the actual work environment.
4. Observation of end users who either use or review a representation of the product. Controlled and sometimes extensive interrogation and probing of the participants by the test monitor.
5. Collection of quantitative and qualitative performance and preference measures.
6. Recommendation of improvements to the design of the product.

Four Types of Tests: An Overview

In this chapter I define and discuss three tests - exploratory, assessment, and validation tests - at a high level, according to the approximate point in the product life cycle at which each would be administered.
The fourth type of test, the comparison test, can be used as an integral part of any of the other three tests, and is not associated with any specific life cycle phase.


Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York, 1994.

Nov 4 - Jeffrey Rubin, The Problem of Unusable Products and Systems

Chapter 1
The Problem of Unusable Products and Systems


Five Reasons for Hard-To-Use Products and Systems

1. During product development, the emphasis and focus have been on the machine or system, not on the person who is the ultimate end user.

2. As technology has penetrated the mainstream consumer market, the target audience has changed and continues to change dramatcially. Development organizations have been slow to react to this evolution.

3. The design of usable product is a difficult, unpredictable endeavor, yet many organizations trust it as it were just "common sense."

4. Organizations employ very specialized teams and approached to product and system development, yet fail to integrate them with each other.

5. The design of the user interface and the technical implementation of the user interface are different activities, requiring very different skills. Today, the emphasis and the need are on the design aspect, while many engineers possess the mind set and skill set for technical implementation.

User-Centered Design - Definition

User-centered design (UCD) is the most recent term coined to describe an approach that has been around for decades under different names, such as human factors engineering, ergonomics, and more recently, usability engineering.

UCD represents not only the techniques, processes, methods and procedures for designing usable products and systems, but just as important, the philosophy that places the user at the center of the process.

Three Principles of A User-Centered Design
1. An early focus on users and tasks.
2, Empirical measurement of product usage.
3. Iterative design whereby a product is designed, modified, and tested repeatedly.

Product Development Life Cycle
(Figure 1.5)
1. User and Usage Needs Analysis.
2. Specification of Requirements.
3. Preliminary (High Level) Design.
4. Detailed Design.
5. Product Build.
6. Product Release.

Hewlett Packard's Human Factors During Life Cycle
(Figure 1.6)
HUMAN FACTORS ACTIVITIES
Phase 1 (Needs Analysis)
* Identify and charaterize target user population.
* Identify and analyze user tasks.
* Identify users' physical and organizational environment.
* Identify usability problems on similar or existing products.
* Identify users' product feature list.
Phase 2 (Requirements Specification)
* Identify ergonomic or market-based requirements.
* Define and develop product usability and acceptance goals.
* Define product localization plan.
* Develop usability testing schedule.
Phase 3 (Conceptual Design)
* Develop product interface requirements.
* Define mental models describing system from the users' perspectives.
* Determine allocation of functions between user and system.
* Integrate Human Factors data, principles and guidelines.
* Test conceptual models.
Phase 4 (Prototype, Development and Test)
* Provide technical support in the development of product prototypes.
* Review production prototypes for ergonomic compliance.
* Test usability of HW, SW, and Learning Products.
* Provide feedback (including design recommendations) for designers based on results of usability test.
Phase 5 (Product Evaluation)
* Review final product specifications to ensure agreed upon usability recommendations have been satisfactorily implemented into the product design.
* Conduct on-site customer evaluations to determine product usability and how the product effectively meets user needs and expectations.
* Provide Marketing with usability advantages of the product.

Usability Goals and Objectives

It is generally accepted that an operational definition of usability include one or more of the following four factors, as outlined by Booth [15].
1. Usefulness
2. Effectiveness (ease of use)
3. Learnability
4. Attitude (likability).

Usability Testing

Usability testing is roughly divided into two main approaches.
The first approach involves formal tests conducted as true experiments, in order to confirm or refute specific hypotheses.
The second approach, a less formal one, employs an iterative cycle of tests intended to expose usability defieiencies and gradually shape or mold the product in question.

Usability testing divided into four types of tests.
1. Exploratory
2. Assessment
3. Validation
4. Comparison.


Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York. 1994.