Sep 5 - Ryu, Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods (part 1)

Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods.
Young Sam Ryu.
Dissertation Submitted to the Faculty of Virginia Polytechnic Institute and State University in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy in Industrial and Systems Engineering

July 2005
Blacksburg, Virginia


Abstract
As the growth of rapid prototyping techniques shortens the development life cycle of software and electronic products, usability inquiry methods can play a more significant role during the development life cycle, diagnosing usability problems and providing metrics for making comparative decisions. A need has been realized for questionnaires tailored to the evaluation of electronic mobile products, wherein usability is dependent on both hardware and software as well as the emotional appeal and aesthetic integrity of the design.
This research followed a systematic approach to develop a new questionnaire tailored to measure the usability of electronic mobile products. The Mobile Phone Usability Questionnaire (MPUQ) developed throughout this series of studies evaluates the usability of mobile phones for the purpose of making decisions among competing variations in the end-user market, alternatives of prototypes during the development process, and evolving versions during an iterative design process. In addition, the questionnaire can serve as a tool for identifying diagnostic information to improve specific usability dimensions and related interface elements.
Employing the refined MPUQ, decision making models were developed using Analytic Hierarchy Process (AHP) and linear regression analysis. Next, a new group of representative mobile users was employed to develop a hierarchical model representing the usability dimensions incorporated in the questionnaire and to assign priorities to each node in the hierarchy. Employing the AHP and regression models, important usability dimensions and questionnaire items for mobile products were identified. Finally, a case study of comparative usability evaluations was performed to validate the MPUQ and models.
A computerized support tool was developed to perform redundancy and relevancy analyses for the selection of appropriate questionnaire items. The weighted geometric mean was used to combine multiple numbers of matrices from pairwise comparison based on decision makers’ consistency ratio values for AHP. The AHP and regression models provided important usability dimensions so that mobile device usability practitioners can simply focus on the interface elements related to the decisive usability dimensions in order to improve the usability of mobile products. The AHP model could predict the users’ decision based on a descriptive model of purchasing the best product slightly but not significantly better than other evaluation methods. Except for memorability, the MPUQ embraced the dimensions included in the other well-known usability definitions and almost all criteria covered by the existing usability questionnaires. In addition, MPUQ incorporated new criteria, such as pleasurability and specific tasks performance.

1 INTRODUCTION

1.1 Motivation

Usability has been an important criterion of decision making for end-users, consumers, product designers and software developers for their respective purposes. In addition to the effort of defining usability concepts and dimensions to be evaluated and quantified, many usability evaluation methods and measurements have been developed and proposed. However, each method has advantages and disadvantages such that some usability measurements are difficult to apply, and some others are overly dependent on the evaluators’ levels of expertise.

As one of the effective methods of evaluating usability, various usability questionnaires have been developed by the Human Computer Interaction (HCI) research community. While these questionnaires are intended for the evaluation of computer software applications running on desktop computers, the need for a usability questionnaire for electronic consumer products has increased for various reasons.

One of the reasons is that the interface of electronic consumer products is different from that of the software products.

For example, mobile products are made up of both hardware (e.g., built-in displays, keypads, cameras, and aesthetics) and software (e.g., menus, icons, web browsers, games, calendars, and organizers) components.

Importantly, the design of electronic consumer products has been crafted by industrial designers and design artists who emphasize the emotional appeal and aesthetic integrity of the design (Ulrich & Eppinger, 1995).

For these reasons, a distinct approach and questionnaire would be helpful for the evaluation of electronic consumer products, even though some usability questionnaires claim to be relevant to products other than computer software.

Current usability questionnaires also seem to measure various usability dimensions, but the dimensions are not necessarily identical across questionnaires. Thus, the exploration of the available questionnaires provides a sound background to the development of the questionnaire items for this study.

For the purposes of this study, the term electronic mobile products refers to mobile phones, smart phones, Personal Digital Assistants (PDAs), and Handheld Personal Computers (PCs), all of which support wireless connectivity and mobility in the user’s hands.

In addition to the importance and popularity of mobile devices in consumers’ life styles, mobile products introduce new usability requirements or dimensions such as mobility and portability not possible with desktop computers. Thus, electronic mobile products were chosen here as the target products among electronic consumer products to develop a subjective usability assessment method.

As one of the usability questionnaires focusing on a specific group of products, the Quebec User Evaluation of Satisfaction with assistive Technology (QUEST) (Demers, Weiss-Lambrou, & Ska, 1996) considers absolute degrees of importance on each satisfaction variable item judged by each respondent. The purpose in considering degrees of importance on each item was to extract important variables so that evaluators could focus on finding the sources of significant dissatisfaction corresponding to the identified important variables.

Since multiple usability questionnaire items and categories of the items are necessary to represent all relevant sub-dimensions of usability in a questionnaire aimed at generating composite scores, assigning relative weights of importance to them relating to a target construct can be regarded as a multi-criteria decision making (MCDM) problem.

There are several MCDM methods3, such as weighted sum model (WSM), weighted product model (WPM), and analytic hierarchy process (AHP) (Triantaphyllou, 2000).

Among those MCDM methods, AHP has been known as the most popular across various fields because of superior capability in dealing with complexity and inter-dependency among criteria, and its dissimilar criteria units using a ratio scale. Thus, there have been a few efforts to apply AHP in the decision making stage of usability evaluation (Mitta, 1993; Park & Lim, 1999; Stanney & Mollaghasemi, 1995), but those studies considered a small number of usability criteria or used AHP in an aggregational manner.

..this research developed comparative usability evaluation methods for electronic mobile products. The methods were developed based on the construction of a new usability questionnaire scale tailored to evaluate mobile products and the application of MCDM methods (i.e., AHP combined with linear regression analysis) to the questionnaire scale in order to provide composite usability scores for the comparative evaluation.

1.2. Research Objectives

The primary objective of this research is to develop a valid and reliable method for the comparison of (1) competing electronic mobile products in the end-user market, (2) evolving
versions of the same product during an iterative design process, and (3) alternatives of prototypes to be selected during the development process. The method was based primarily on subjective usability assessments using questionnaires. Thus, the output was a set of questionnaire items integrating existing usability questionnaires adapted especially for electronic mobile products...

The objectives are summarized:
􀂃 Identify usability attributes and dimensions covered and not covered by existing usability questionnaires and generate measurement items relevant for the evaluation of electronic mobile products.
􀂃 Develop a set of items for a questionnaire according to the identified usability dimensions and expert reviews.
􀂃 Refine the set of items using factor analysis and identify the underlying structure of
the usability dimensions to be usable as input for AHP application.
􀂃 Assess the reliability and validity of the usability questionnaire so that the
questionnaire is refined based on the psychometric properties.
􀂃 Develop a hierarchical structure incorporating all of the identified usability dimensions and assign relative priorities for each element of the hierarchical structure to generate a composite score of overall usability.
􀂃 Test the applicability and validity of the developed usability questionnaire model by conducting a case study of comparative usability evaluation.

1.3. Approach

The research framework was abstracted from subjective usability assessments using questionnaires and the AHP method as the major components. In accordance with these methods, the research reviewed literature to provide a theoretical framework and employs usability experts to make critical decisions through the research, as well as reflect the user’s point of view to evaluate and validate the outcome of the research.

Table 1 summarizes the research goals and approaches of the research.

In addition, Figure 1 illustrates the conceptual summary of the usability questionnaire models, consisting of two major components of the research framework (i.e., subjective usability assessment and MCDM methods).

.....

Table 1. Research goals and approach

Phase > Goal > Approach

Figure 2. Organization of the dissertation

My Comments: The Dissertation organisation is shown in a flowchart manner, showing the phases, research flow and the outputs (deliverables).

2. LITERATURE REVIEW

2.1. Subjective Usability Assessment
2.1.1. Definitions and Perspectives of Usability

Usability has been defined by many researchers in many ways.

One of the first definitions of usability was “the quality of interaction which takes place” (Bennett, 1979, p. 8).

Shackel (1991) proposed an approach to define usability by focusing on the perception of the product and regarding acceptance of the product as the highest level of the usability concept. Considering usability in the context of acceptance, Shackel provides a definition stating that “usability of a system or equipment is the capability in human functional terms to be used easily and effectively by the specified range of users, given specified training and user support, to fulfill the specified range of tasks, within the specified range of environmental scenarios”
(Shackel, 1991, p.24). .... a set of usability criteria. Those are
􀂃 Effectiveness: level of interaction in terms of speed and errors;
􀂃 Learnability: level of learning needed to accomplish a task;
􀂃 Flexibility: level of adaptation to various tasks; and
􀂃 Attitude: level of user satisfaction with the system.

Shackel also collaborated on a later definition, stating that usability derives from “the extent to which an interface affords an effective and satisfying interaction to the intended users, performing the intended tasks within the intended environment at an acceptable cost” (Sweeney, Maguire, & Shackel, 1993, p. 690).

Another well-accepted definition of usability which received attention from the Human Computer Interaction (HCI) community was offered by Nielsen (1993). He also considers factors which may influence product acceptance. Nielsen does not provide any descriptive definition of
usability; however, he provides the operational criteria to define clearly the concept of usability:
􀂃 Learnability: ability to reach a reasonable level of performance
􀂃 Memorability: ability to remember how to use a product
􀂃 Efficiency: trained users’ level of performance
􀂃 Satisfaction: subjective assessment of how pleasurable it is to use
􀂃 Errors: number of errors, ability to recover from errors, existence of serious errors

Finally, attempts to establish standards on usability have been made by the International
Organization for Standardization (ISO). ISO 9241-11 (1998) is an international standard for the
ergonomic requirements for office work with visual display terminals and defines usability as
“the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (p. 2). Additionally, ISO
9241-11 classifies the dimensions of usability to account for the definition:
􀂃 Effectiveness: the accuracy and completeness with which users achieve goals
􀂃 Efficiency: the resources expended in relation to the accuracy and completeness
􀂃 Satisfaction: the comfort and acceptability of use

ISO/IEC 9126 elaborates on three different ways to assess usability. Part 1 (ISO/IEC 9126-1, 2001) provides the definition of usability which distinguishes clearly between the interface and task performance by designating usability as “the capability of the software to be understood, learned, used and liked by the user, when used under specified conditions” (p. 9).

The definition of ISO/IEC 9126-1 presents usability as quality-in-use. With the perception of
usability as the product quality, the dimensions of usability indicated in ISO/IEC 9126-1 became 􀂃 Understandability,
􀂃 Learnability,
􀂃 Operability, and
􀂃 Attractiveness.

Part 2 (ISO/IEC 9126-2, 2003) includes external metrics using empirical research. Part 3 (ISO/IEC 9126-3, 2003) describes internal metrics which measure interface properties.

Table 2. Comparison of usability dimensions from the usability definitions

Usability Dimensions > Shackel (1991) > Nielsen (1993) > ISO 9241 and 9126 (1998; 2001)

Given the descriptive definition of usability, new usability dimensions suggested by recent studies (e.g., aesthetic appeals and emotional dimensions) were blended in as the research progressed to develop the usability questionnaire for mobile products. For example, aesthetic appeal can be considered a sub-dimension of satisfaction, which is one of the main dimensions of ISO 9241-11.