Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods.
Young Sam Ryu.
Dissertation Submitted to the Faculty of Virginia Polytechnic Institute and State University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Industrial and Systems Engineering
July 2005
Blacksburg, Virginia
2. LITERATURE REVIEW
2.1. Subjective Usability Assessment
2.1.2. Usability Measurements
Keinonen (1998) categorized different approaches to defining usability, including usability as a design process and usability as product attributes, which contribute to the establishment of design guidelines.
From the perspective of usability as a design process, usability engineering (UE) and user-centered design (UCD) have been defined and recognized as a process whereby the usability of a product is specified quantitatively (Tyldesley, 1988).
Numerous sets of usability principles and guidelines have been developed by the HCI community, including computer companies, standard organizations, and well-known researchers. Some well-known principles and guidelines they have developed include
Shneiderman’s (1986) eight golden rules of dialogue design,
Norman’s (1988) seven principles of making tasks easy,
human interface guidelines by Apple Computer (1987),
usability heuristics by Nielsen (1993),
ISO 9241-10 (1996) for dialogue principles, and
the evaluation check list by Ravden and Johnson (1989).
These references cover many major dimensions of usability including consistency, user control, appropriate presentation, error handling, memory load, task matching, flexibility, and guidance (Keinonen, 1998).
Many research frameworks have been introduced as measures of usability at an operational level according to various usability dimensions (Nielsen, 1993; Rubin, 1994).
There are three different categories of methods to obtain measurements known as the usability
inspection method, the testing method, and the inquiry method (Avouris, 2001).
First, the usability inspection method involves having usability experts examine interfaces of the products.
Nielsen and Mack (1994) published a book focusing only on this method explaining that usability inspection aims at finding usability problems, the severity of the usability problems and the overall usability of an entire system.
The major methods in this category are heuristic evaluation, heuristic estimation, cognitive walkthrough, pluralistic walkthrough, feature inspection, consistency inspection, standards inspection, formal usability inspection, and a guidelines checklist.
One advantage of the inspection method is that it can be used in the early stages of the product development cycle..
Second, usability testing methods usually measure system performance based on welldefined usability criteria.
Those criteria can be defined according to the definitions of usability, usability attributes following standards, and empirical metrics. Typically, data on a measured performance are collected based on the observation of individual users performing specific tasks with the product (e.g., completion time and number of errors).
The most widely employed usability testing methods are think-aloud protocol, co-discovery, performance measurement, and field studies, all of which are techniques available not only for usability studies but also for numerous other fields of study. To provide for the validity of this type of evaluation, the proper design of tasks and organization of the testing laboratory are essential (Preece, Rogers, Sharp, Benyon, Holland, & Carey, 1994).
Among the techniques mentioned above, performance measurement is the one that can present objective data clearly, thus ISO 9241-11 provides example metrics for the three different usability criteria (Table 3).
Table 3. Example measures of usability (ISO 9241-11, 1998)
Effectiveness :
- Percentage of goal achieved
- Percentage of tasks completed
- Accuracy of completed task
Efficiency :
- Time to complete a task
- Monetary cost of performing the task
Satisfaction :
- Rating scale for satisfaction
- Frequency of discretionary use
- Frequency of complaints
The usability inquiry method involves communication between the users and the evaluators in the evaluation, usually by means of questions and interviews.
The evaluators question users about their thoughts on the interface or prototype of the system and the users’ ability to answer questions plays a significant role in the evaluation.
Commonly used techniques are focus groups, interviews, field observations, and questionnaires.
Inquiry methods can be used to measure various usability dimensions and attributes; however, the most common usage of inquiry methods is for the measurement of user satisfaction.
Thus, inquiry methods support the user’s point of view, the fourth perspective listed by Keinonen (1998), through the measurement of user satisfaction.
2.1.3. Subjective Measurements of Usability
Subjective usability measurements focus on an individual’s personal experience with a product or system.
Several usability questionnaires were developed by the HCI community, such as
Software Usability Measurement Inventory (SUMI) (Kirakowski, 1996; Kirakowski & Corbett, 1993; Porteous, Kirakowski, & Corbett, 1993),
the Questionnaire for User Interaction Satisfaction (QUIS) (Chin, Diehl, & Norman, 1988; Harper & Norman, 1993; Shneiderman, 1986), and
the Post-Study System Usability Questionnaire (PSSUQ) (Lewis, 1995).
SUMI is the best-known usability questionnaire to measure user satisfaction and assess user-perceived software quality. SUMI is a 50-item questionnaire, each item of which is answered with “agree”, “undecided”, or “disagree”, and is available in various languages to provide an international standard. Each set of the questionnaire (50 items) takes approximately 10 minutes to complete and needs only a small number of subjects, although at least ten subjects are recommended for the results to be used effectively (Kirakowski & Corbett, 1993).
Based on the answers collected, scores are calculated and analyzed into five subscales (Kirakowski & Corbett, 1993):
Affect: degree to which the product engages the user’s emotional responses;
Control: degree to which the user sets the pace;
Efficiency: degree to which the user can achieve the goals of interaction with the product;
Learnability: degree to which the user can easily initiate operations and learn new features; and
Helpfulness: extent to which user obtains assistance from the product.
The SUMI subscales are referenced in ISO standards on usability (ISO 9241-10, 1996) and software product quality (ISO/IEC 9126-2, 2003).
The primary advantages of SUMI over other usability evaluation methods are noted as ease of application and relatively low costs to conduct for both evaluators and participants. Several researchers argue that SUMI is the best validated subjective assessment for usability (Annett, 2002; van Veenendaal, 1998) and that it has been known to be reliable and to discriminate between different kinds of software products (van Veenendaal, 1998).
However, some disadvantages exist. For example, SUMI can only be used at relatively late stages during the product development process since a running version of the product should be available. Also, SUMI is generic so that the accuracy and level of detail of the problems or successes detected through its use are limited (Keinonen, 1998; Konradt, Wandke, Balazs, & Christophersen, 2003; van Veenendaal, 1998).
QUIS was developed at the Human Computer Interaction Laboratory at the University of Maryland, College Park (Chin et al., 1988; Harper & Norman, 1993) based on the scale for “User evaluation of interactive computer systems” introduced by Shneiderman (1986). QUIS has been updated in many versions in terms of scales, items of focus, and level of reliability. The most recent publication of QUIS, version 7, incorporates ten different dimensions of usability:
Overall user reactions,
Screen factors,
Terminology and system information,
Learning factors,
System capabilities,
Technical manuals and online help,
Online tutorials,
Multimedia,
Teleconferencing, and
Software installation.
Many items in QUIS are closer to a checklist evaluation performed by an expert, although some questions measure user satisfaction.
Therefore, there is a criticism (Keinonen, 1998) that users may not evaluate those items or attributes effectively unless they have expert knowledge. That criticism can lead to the conclusion that QUIS lies between the designer domain of concrete product attributes and the user domain of subjective experience (Keinonen, 1998).
Since QUIS questionnaires refer mostly to concrete software product attributes, its use on products other than computer software in which screen displays are used is very challenging.
According to the survey of the use of objective and subjective measures in the HCI community conducted by Nielsen and Levy (1994), 25% of 405 studies measured subjective aspects and 14% measured both objective and subjective aspects.
2.1.4. Usability Questionnaires
2.1.4.2. Questionnaires and Usability Research
One of the single greatest advantages of using questionnaires in usability research is that
questionnaires can provide evaluators with feedback from the users’ point of view (Annett, 2002; Baber, 2002; Kirakowski, 2003).
Since user-centered and participatory design is one of the most important aspects in the usability engineering process (Keinonen, 1998), questionnaires can be an essential method, assuming that the respondents are validated as representative of the whole user population.
Another big advantage insisted on by Kirakowski (2003) is that the measures from a questionnaire can provide comparable measures or scores across applications, users, and various tasks being evaluated.
As indicated in Section 2.1.2 Usability Measurements, the questionnaire is a quick and cost-effective method to conduct and measure scores compared with other inquiry methods. Thus, evaluators usually gather great amounts of data using questionnaires as surveys.
Another advantage is that there are many usability aspects or dimensions for which no established objective measurements exist, and those may only be measured by subjective assessment. New usability concepts suggested for the evaluation of consumer electronic products such as attractiveness (Caplan, 1994), emotional usability (Logan, 1994), sensuality (Hofmeester, Kemp, & Blankendaal, 1996), pleasure and displeasure in product use (Jordan, 1998) seem to be quantified effectively by subjective assessment and those usability concepts are proving to be important for software products these days.
2.2. Mobile Device Usability
2.2.1. Definition of Electronic Mobile Products
Mobile devices connected by wireless technology have changed the ways in which people communicate with each other as well as interact with computers, and the change will continue with the introduction of new mobile devices and services. Existing mobile devices include mobile phones, Personal Digital Assistants (PDAs), and smart phones.
Another term for the mobile devices provided by Weiss (2002) is ”Handheld devices,”
which is defined as “extremely portable, self-contained information management and communication devices” (p. 2). These devices are also small, lightweight, and best operated while held in the user’s hand, such as PDAs, pagers, and mobile phones.
Since computers are getting smaller, such as notebook computers and palmtop computers, Weiss (2002) suggests that a computer must meet three conditions to be considered a handheld device:
It must be used in one’s hands, not on a table,
It must operate without cables, and
It must allow the addition of new applications or support Internet connectivity.
In summary, the definition of electronic mobile products that this research focuses on includes mobile phones, smart phones, PDAs, and Handheld PCs, all of which support wireless connectivity and mobility in the user’s hands.
Figure 3. Mobile and wireless device scope diagram adapted from Gorlenko and Merrick (2003)
My Comments: There are a lot of Usability dimensions and variables discussed in Section 3. I may decide to do a Part 3 Literature Review on Ryu's PhD dissertation.
References that I may want to read further in future:
Apple Computer. (1987). Human interface guidelines: The apple desktop interface. Reading, MA: Addison-Wesley.
Baber, C. (2002). Subjective evaluation of usability. Ergonomics, 45(14), 1021-1025.
Harper, P. D., & Norman, K. L. (1993). Improving user satisfaction: The questionnaire for user interaction satisfaction version 5.5. In Proceeding of The 1st Annual Mid-Atlantic Human Factors Conference, Virginia Beach, VA, 224-228.
ISO/IEC 9126-1. (2001). Software engineering- product quality - part 1: Quality model. International Organization for Standardization.
ISO/IEC 9126-2. (2003). Software engineering - product quality - part 2: External metrics. International Organization for Standardization.
ISO/IEC 9126-3. (2003). Software engineering - product quality - part 3: Internal metrics. International Organization for Standardization.
Ketola, P. (2002). Integrating usability with concurrent engineering in mobile phone development: Tampereen yliopisto. Ketola, P., & Roykkee, M. (2001). Three facets of usability in mobile handsets. In Proceeding of CHI 2001, Workshop, Mobile Communications: Understanding Users, Adoption & Design Sunday and Monday, Seattle, Washington.
Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In P. W. Jordan & B. Thomas & B. A. Weerdmeester & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 169-178). London: Taylor & Francis.
Kirakowski, J. (2003). Questionnaires in usability engineering: A list of frequently asked questions [HTML]. Retrieved 11/26, 2003, from the World Wide Web:
Kirakowski, J., & Cierlik, B. (1998). Measuring the usability of web sites. In Proceeding of Human Factors and Ergonomics Society 42nd Annual Meeting, Santa Monica, CA.
Kirakowski, J., & Corbett, M. (1993). Sumi: The software usability measurement inventory. British Journal of Educational Technology, 24(3), 210-212.
Lewis, J. R. (1995). Ibm computer usability satisfaction questionnaire: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57-78.
Lewis, J. R. (2002). Psychometric evaluation of the pssuq using data from five years of usability studies. International Journal of Human-Computer Interaction, 14(3-4), 463-488.
Lindholm, C., Keinonen, T., & Kiljander, H. (2003). Mobile usability how nokia changed the face of the mobile phone. New York, NY: McGraw-Hill.
Porteous, M., Kirakowski, J., & Corbett, M. (1993). Sumi user handbook. University College Cork: Human Factors Research Group.
Shneiderman, B. (1986). Designing the user interface: Strategies for effective human-computer
interaction. Reading, MA: Addison-Wesley.
Smith-Jackson, T. L., Williges, R. C., Kwahk, J., Capra, M., Durak, T., Nam, C. S., & Ryu, Y. S. (2001). User requirements specification for a prototype healthcare information website and an online assessment tool (ACE/HCIL-01-01): Grado Department of Industrial and Systems Engineering, Virginia Tech.
van Veenendaal, E. (1998). Questionnaire based usability testing. In Proceeding of European Software Quality Week, Brussels.
Weiss, S. (2002). Handheld usability. Hoboken, NJ: John Wiley & Sons.
Weiss, S., Kevil, D., & Martin, R. (2001). Wireless phone usability research. New York: Useable Products Company.
