Sep 7 - about Usability Evaluation Method

My literature review about Usability Evaluation Method.

------
Usability Inspection Methods after 15 Years of Research and Practice.
Tasha Hollingsed. Lockheed Martin, 2540 North Telshor Blvd., Suite C, Las Cruces, NM 88011. +1 505-525-5267 tasha.hollingsed@lmco.com
David G. Novick. Department of Computer Science, The University of Texas at El Paso, El Paso, TX 79968-0518. +1 915-747-5725 novick@utep.edu

SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.


COGNITIVE WALKTHROUGH

The cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use [53].
Like other inspection methods, a cognitive walkthrough can be performed on an interface at any time during the development process, from the original mock-ups through the final release.

The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.
During the analysis phase, the evaluators work through the four steps of humancomputer interaction developed by Lewis and Polson [25]:
1. The user sets a goal to be completed within the system.
2. The user determines the currently available actions.
3. The user selects the action that they think will take them closer to their goal.
4. The user performs the action and evaluates the feedback given by the system.

Wharton et al. [53] surveyed the history of the cognitive walkthrough prior to 1994. At the time, the two main limitations of the cognitive walkthrough were the repetitiveness of filling out the forms and the limited range of problems the process found ([53], [51], [19]). A new method of cognitive walkthrough addressed these limitations by using small groups instead of individual evaluators and rotating the form filling within the group, evaluating the simple tasks first, and by keeping a record of all problems identified during the evaluation, not just those identified during the process ([53], [41]).

Other criticisms of the cognitive walkthrough were that it does not provide guidelines about what makes an action clearly available to a user and that it is not known what types of actions are considered by a broad range of users [51]).

Since its inception, and with its refinements and extensions, the cognitive walkthrough has been shown to be an effective inspection method that can be applied not just by cognitive scientists and usability specialists but also by novice evaluators. However, the choice of task scenario can be difficult; if the scenario is not adequately described, the evaluation is not as effective.


PLURALISTIC USABILITY WALKTHROUGH

The pluralistic usability walkthrough [3] adapted the traditional usability walkthrough to incorporate representative users, product developers, members of the product team, and usability experts in the process. It is defined by five characteristics:
1. Inclusion of representative users, product developers, and human factors professionals;
2. The application’s screens are presented in the same order as they would appear to the user;
3. All participants are asked to assume the role of the user;
4. Participants write down what actions they, as users, would take for each screen before the group discusses the screens; and
5. When discussing each screen, the representative users speak first.

...benefits and limitations of the approach.
On the positive side, this approach offers feedback from users even if the interface is not fully developed, enables rapid iteration of the design cycle, and—because the users are directly involved—can result in “on-the-fly” redesign.
On the negative side, the approach must be limited to representative rather than comprehensive user paths through the interface, and users who, at a particular step, did not choose the path the group will follow must “reset” their interaction with the interface.

My comments: I think pluralistic walkthrough has another strength of using a cross-functional team to evaluate usability.

The pluralistic walkthrough appears to be in active use for assessing usability. It is included in the Usability Professionals Association draft body of knowledge [49].
Available reports indicate that pluralistic usability walkthrough is used in industry. While some human factors experts continue to conduct usability walkthroughs that do not combine stakeholder perspectives (e.g., [7], [50], [44]), it seems likely that use of the pluralistic usability walkthrough is widespread but that teams do not refer to it as such in published reports.


FORMAL USABILITY INSPECTIONS

Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance [22].

Like the pluralistic usability walkthrough, this involves stepping through the user’s task. However, because the reviewers consist of human factors experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the interface as efficiently as possible. The review process includes task performance models and heuristics, a variety of human-factors expertise, and defect detection within the framework of the software development lifecycle.
Like the cognitive walkthrough, formal usability inspections require definitions of user profiles and task scenarios. And, like the cognitive walkthrough, the reviewers use a cognitive model of task performance, which can be extended with a checklist of cognitive steps similar to those invoked by Norman [36] to bridge the “gulf of execution.”

Hewlett Packard used this method for at least two years before 1995. The inspection team included design engineers, usability engineers, customer support engineers, and customers at times. The team inspected fourteen products and found an average of 76 usability concerns per product and an average of 74 percent of those concerns were fixed per product. While no formal evaluation of the results was done, it was found that the engineers could detect several of the usability concerns, and the engineers enjoyed using the method while increasing their awareness of user needs [15].

Digital Equipment Corporation also conducted a version of formal usability inspections from 1994 to 1995 on ten products. They found an average of 66 usability problems per product and fixed an average of 52 problems per product. Finding even small usability problems proved to be an asset, especially when a number of these problems were easily fixed. As more problems were fixed, the perceived quality of the product improved as well, even if most of these fixes were small [45].

Since then, it appears that little research has been conducted on formal usability inspections. This approach now tends to be grouped into the overall inspection method class and gets overshadowed by the better known heuristic evaluation when comparisons between inspection and empirical methods have been conducted. As a method, formal usability inspection gains speed at the cost of losing the multiple stakeholder perspectives of the pluralistic walkthrough, and its cognitive model can be seen as less comprehensive than that of the cognitive walkthrough.


CONCLUSION

Both empirical usability testing and usability inspection methods appear to be in wide use, with developers choosing the most appropriate method for their purposes and their context.

For example, half of the ten intranets winning a 2005 competition used heuristic evaluation [34]. The same report indicated that empirical usability testing was used by 80 percent of the winning intranets.

The cognitive walkthrough appears to be in continued use, although reports of use are not as frequent.
The pluralistic usability walkthrough remains in the repertoire of usability experts, although usability experts continue to conduct user-only usability walkthroughs.
And formal usability inspection, although shown to be an effective approach for identifying usability problems, appears to be used less now than in the mid-1990s.

Many have claimed that usability inspection methods make for faster and more cost-efficient evaluation of the usability of an interface than empirical user testing.
But while usability inspection methods do identify a number of usability problems faster and more cost-efficiently, the best performing evaluator and method still only found 44 percent of the usability problems found in a laboratory setting [9].
While the cognitive walkthrough is useful for predicting problems on a given task and heuristic evaluation is useful for predicting problems on the interface, empirical testing provides lots of information throughout the interface and is the benchmark against which all other methods are measured [9].
Indeed, Jeffries et al. [19] noted that evaluators of usability methods may have rated problems found though empirical usability testing as, on average, more severe precisely because the problems were identified empirically rather than analytically. While inspection methods need expert evaluators to be effective, their strengths are that they can be implemented into the early stages of the development cycle and provide a forum in which changes to interface can be discussed.

The research on comparisons of usability assessment methods suggests several lessons for practitioners.
First, while “faster, cheaper” methods such as heuristic evaluation and the pluralistic usability walkthrough can be useful for rapid iteration early in the design cycle, inspection methods cannot fully substitute for the empirical user testing needed before releasing an interface or Web site to the public.
Second, empirical methods can also be used early in the development process, via “low-tech” versions of interfaces.
Third, developers often combine multiple inspection methods— heuristic evaluation and the cognitive walkthrough—in the same project so that they obtain better coverage of usability issues.
And fourth, adding multiple perspectives—along dimensions such as the range of stakeholders or kinds of usability problems—appears to improve the effectiveness of inspection methods.

It remains an open issue as to why usability professionals, in practice, rely on single-perspective methods, typically involving users, or experts, but not both. The evidence from reports of recent uses of heuristic evaluation suggests that many usability specialists are missing the benefits of the pluralistic walkthrough and perspective-based evaluation.

At a deeper level, though, a new direction for research should complement these defect-tracking and learning approaches by seeking to understand the root causes of usability problems. The ideal solution would be to know the reasons for usability problems, so that designers can minimize the effort spent on usability inspection and testing.

My Comments: For my PhD research, shall I choose only 1 UEM (usability evaluation method) or shall I use a hybrid/combination of UEM?

---------------
Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods.
Young Sam Ryu.
Dissertation Submitted to the Faculty of Virginia Polytechnic Institute and State University in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy in Industrial and Systems Engineering
July 2005
Blacksburg, Virginia


There are three different categories of methods to obtain measurements known as the usability inspection method, the testing method, and the inquiry method (Avouris, 2001).

First, the usability inspection method involves having usability experts examine interfaces of the products.
Nielsen and Mack (1994) published a book focusing only on this method explaining that usability inspection aims at finding usability problems, the severity of the usability problems and the overall usability of an entire system.
The major methods in this category are heuristic evaluation, heuristic estimation, cognitive walkthrough, pluralistic walkthrough, feature inspection, consistency inspection, standards inspection, formal usability inspection, and a guidelines checklist.
One advantage of the inspection method is that it can be used in the early stages of the product development cycle..

Second, usability testing methods usually measure system performance based on welldefined usability criteria.
Those criteria can be defined according to the definitions of usability, usability attributes following standards, and empirical metrics. Typically, data on a measured performance are collected based on the observation of individual users performing specific tasks with the product (e.g., completion time and number of errors).
The most widely employed usability testing methods are think-aloud protocol, co-discovery, performance measurement, and field studies, all of which are techniques available not only for usability studies but also for numerous other fields of study. To provide for the validity of this type of evaluation, the proper design of tasks and organization of the testing laboratory are essential (Preece, Rogers, Sharp, Benyon, Holland, & Carey, 1994).
Among the techniques mentioned above, performance measurement is the one that can present objective data clearly, thus ISO 9241-11 provides example metrics for the three different usability criteria (Table 3).

Table 3. Example measures of usability (ISO 9241-11, 1998)
Effectiveness :
- Percentage of goal achieved
- Percentage of tasks completed
- Accuracy of completed task
Efficiency :
- Time to complete a task
- Monetary cost of performing the task
Satisfaction :
- Rating scale for satisfaction
- Frequency of discretionary use
- Frequency of complaints

The usability inquiry method involves communication between the users and the evaluators in the evaluation, usually by means of questions and interviews.
The evaluators question users about their thoughts on the interface or prototype of the system and the users’ ability to answer questions plays a significant role in the evaluation.
Commonly used techniques are focus groups, interviews, field observations, and questionnaires.
Inquiry methods can be used to measure various usability dimensions and attributes; however, the most common usage of inquiry methods is for the measurement of user satisfaction.

Thus, inquiry methods support the user’s point of view, the fourth perspective listed by Keinonen (1998), through the measurement of user satisfaction.

Subjective usability measurements focus on an individual’s personal experience with a product or system.
Several usability questionnaires were developed by the HCI community, such as
Software Usability Measurement Inventory (SUMI) (Kirakowski, 1996; Kirakowski & Corbett, 1993; Porteous, Kirakowski, & Corbett, 1993),
the Questionnaire for User Interaction Satisfaction (QUIS) (Chin, Diehl, & Norman, 1988; Harper & Norman, 1993; Shneiderman, 1986), and
the Post-Study System Usability Questionnaire (PSSUQ) (Lewis, 1995).

----------------
Usability Problem Description and the Evaluator Effect in Usability Testing.
Miranda G. Capra
Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Industrial and Systems Engineering
March 13, 2006
Blacksburg, Virginia


Formative usability evaluations are an important part of the usability life cycle, identifying usability problems present in an interface that designers should fix in the next design iteration.
Figure 1.1 provides an overview of a formative usability evaluation and the usability life cycle.
The output of a formative usability evaluation is a set of usability problems (Hartson, Andre, & Williges, 2003), but there are many different usability evaluation methods (UEMs).
Empirical evaluations involve end-users. The most common empirical method is usability testing or think aloud testing, which is generally a taskbased session in a usability laboratory.
Analytical evaluations involve expert review of an interface, such as Heuristic Evaluation (Nielsen, 1994b; Nielsen & Molich, 1990).

-----------------
Supporting Novice Usability Practitioners with Usability Engineering Tools.
Jonathan Randall Howarth.
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Computer Science and Applications.
April 13, 2007
Blacksburg, Virginia


During the usability evaluation sub-process, usability practitioners primarily conduct formative usability evaluations. As described in a Usability Professional’s Association workshop report [Theofanos et al., 2005], formative usability evaluations are conducted to “guide the improvement in design of future iterations” (p. 3).
Usability practitioners conduct formative usability evaluations to understand the strengths and weaknesses of a given interaction design.
During formative evaluations, usability practitioners collect a variety of qualitative data such as verbal protocol and subjective ratings with the goal of producing UP descriptions.

Summative usability studies represent a different type of usability evaluation that is typically performed after a product is released.
Summative usability studies provide proof in the form of statistical significance that one given interaction design is better than other designs in specific ways.
Usability practitioners may collect quantitative data such as measures of time on task and error counts that they later use in metrics.

Content of Usability Evaluation Reports

In 1997, the Industry Usability Reporting Project initiated by the National Institute of Standards and Technology developed the Common Industry Format (CIF), which is currently the most well known format for usability evaluation reports. The CIF became an American National Standard for Information Technology Standard in 2001 [ANSI, 2001].
By standardizing the reporting of usability tests, the CIF hoped to encourage the consideration of usability in purchasing software products; customer organizations that were interested could evaluate different products based on their CIF reports.The CIF includes sections for describing the product, the method used to evaluate the product, and the results of the evaluation.
The CIF is intended for summative usability evaluations, but usability practitioners most frequently perform formative usability evaluations.

Theofanos [2005] and Theofanus and Quesenbery [2005] describe efforts to develop a new CIF that would provide practitioners with guidance for performing and reporting formative studies.

The usability evaluation report consolidates usability information and provides the context for understanding UP descriptions. Without this context, UP descriptions may be misunderstood or overlooked.

----------------
Usability Inspection Criteria for e-Learning Portals.
TeckChong Yeap.
MDP7515 PROJECT Dissertation submitted in partial fulfilment of the requirement for the degree of
Master of Multimedia (E-Learning Technologies)
MULTIMEDIA UNIVERSITY, MALAYSIA
October 2008


Methods of Usability Inspection

According to Nielsen and Molich (1990a), “There are basically four ways to evaluate a user interface:
· Formally by some analysis technique,
· Automatically by a computerized procedure,
· Empirically by experiments with test users, and
· Heuristically by simply looking at the inter-face and passing judgement according to ones own opinion.”


Nielsen (1995b) classified the types of usability inspection into:

· “Heuristic evaluation is the most informal method and involves having usability specialists judge whether each dialogue element follows established usability principles (the "heuristics").” (Nielsen, 1995b)

· “Heuristic estimation is a variant in which the inspectors are asked to estimate the relative usability of two (or more) designs in quantitative terms (typically expected user performance).” (Nielsen, 1995b)

· “Cognitive walkthrough uses a more explicitly detailed procedure to simulate a user's problem-solving process at each step through the dialogue, checking if the simulated user's goals and memory content can be assumed to lead to the next correct action.” (Nielsen, 1995b)

· “Pluralistic walkthrough uses group meetings where users, developers, and human factors people step through a scenario, discussing each dialogue element.” (Nielsen, 1995b) Developers, users, and human factors people get together in a meeting and go through the e-learning portal step-by-step using a task scenario. They discuss and review the usability issues.

· “Feature inspection lists sequences of features used to accomplish typical tasks, checks for long sequences, cumbersome steps, steps that would not be natural for users to try, and steps that require extensive knowledge/experience in order to assess a proposed feature set.” (Nielsen, 1995b) Sometimes only a particular function in the e-learning portal is inspected, which is called feature inspection. For instance, a feature inspection could be done on the Quizzes feature of the e-learning portal.

· “Consistency inspection has designers who represent multiple other projects inspect an interface to see whether it does things in the same way as their own designs.” (Nielsen, 1995b) Consistency inspection involves web designers from other module of the portal to inspect the user interface, to ensure the consistency of the interaction the entire e-learning portal.

· “Standards inspection has an expert on an interface standard inspect the interface for compliance.” (Nielsen, 1995b) A usability consultant or usability professional inspects the interface to make sure it complies with the international standards or other predetermined standards.

· “Formal usability inspection combines individual and group inspections in a six-step procedure with strictly defined roles to....with elements of both heuristic evaluation and a simplified form of cognitive walkthroughs.” (Nielsen, 1995b; Nielsen, 2005e)


Most Popular Usability Inspection Methods

Hollingsed and Novick (2007) stated that heuristic evaluation and cognitive walkthrough appeared to be the most actively used and researched techniques.

Heuristic Evaluation

Heuristic evaluation is also found to be an economical and relatively easy usability evaluation method.
Ssemugabi & de Villiers (2007) had concluded in their research of usability on e-learning applications that heuristic evaluation which was conducted by a competent group of experts, was an appropriate, efficient, and highly effective usability evaluation method for elearning applications.

Heuristic evaluation could also be done with a web-based tool.
Ardito et al (2006) had used Systematic Usability Inspection Tool (SUIT) to help the evaluation conduct the heuristic evaluation. SUIT is an Internet-based tool that supports the evaluators during the usability inspection of software applications.
SUIT makes it possible to reach inspectors everywhere, guiding them in their activities. 4 major processes involved with heuristic evaluation aided with SUIT are:
1. Inspection planning
2. Problem detection
3. Report collection
4. Discussion.

Cognitive Walkthrough

“Cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use. The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.” (Hollingsed & Novick, 2007)

During the analysis phase, the usability evaluators would carry out the four steps (Lewis & Polson, 1991):
1. The user sets a goal that is to be completed within the system.
2. The user determines the existing available actions.
3. The user selects the action which will take him closer to his goal.
4. The user does the action and evaluates the feedback from the system.

Pluralistic Walkthrough

Pluralistic usability walkthrough (Bias, 1994) is usability walkthrough which incorporates usability experts, product developers, members of the product team and representative users in the process.

Pluralistic usability walkthrough has five characteristics:
1. Inclusion of usability professionals, representative users and product developers.
2. The application’s screens are similar to how they would appear to the user.
3. All evaluators would assume the role of the user.
4. Evaluators write down what actions they, as users, would take for each screen before the group discusses the screens.
5. During discussion of each screen, the representative users would speak first. (Hollingsed and Novick, 2007)

Formal Usability Inspection

Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance (Kahn & Prail, 1994). The reviewers consist of usability experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the user interface. Formal usability inspections require definitions of user profiles and task scenarios. (Hollingsed & Novick, 2007)

Empirical Usability Evaluation

Empirical usability evaluation is an approach that is very different from usability inspection.

Wiberg (2003) stated examples of empirical usability evaluation were (a) think-aloud protocol, (b) use data collection, (c) clinical experiments, (d) surveys and questionnaires, (e) interviews.

Clinical experiments emphasized on statistics for measured clinical aspects such as eye gaze, heart rate, skin color and body heat.


Advantages & Limitations of Heuristic Evaluation

A major advantage is the ease of finding usability problem with the aid of the heuristics principles. “Major usability problems have a higher probability than minor problems of being found in a heuristic evaluation, but about twice as many minor problems are found in absolute numbers.” (Nielsen, 1992a)

Another major advantage is no need for numerous users and usability lab. “Heuristic evaluation is easier to set up as there is no need to bring in users or to analyze interaction protocols, and exploits expert scientific and experiential knowledge of general problems and solutions.” (Nielsen & Landauer, 1993a)

Advantages of heuristic evaluation are
· “It is cheap.
· It is intuitive and it is easy to motivate people to do it.
· It does not require advance planning.
· It can be used early in the development process.” (Nielsen & Molich, 1990a)

Other advantages of heuristic evaluation include:
· Can start using simple usability inspection after a few hours of training.
· Usability inspection methods can be used in many different stages of the system development lifecycle.
· Usability inspection can be integrated easily into many established system development practices.
· Lists of usability problems are available immediately after the usability inspection. (Nielsen, 1995a)
· Lists of interface that need to be improved could be quickly distributed for immediate action.

Limitations of heuristic evaluation are:· Identifies usability problems without providing direct suggestions of solutions.· Influenced by the current mindset of the evaluators, so-called “evaluator effect.”· Does not normally generate breakthroughs in the evaluated design. (Nielsen & Molich, 1990a)


When To Do Heuristic Evaluation

Heuristic evaluation is relatively low cost to be conducted. It could be implemented at various stages of portal or website development cycle.

Heuristic evaluation could be conducted at the stages of:
a) Storyboard,
b) Design stage,
c) Prototype of user interface,
d) “Running version” of the prototype,
e) Partial completion of the e-Learning portal or website,
f) Completion of e-Learning portal,
g) Ongoing review of an existing running e-Learning portal.


Who To Do Heuristic Evaluation

Heuristic evaluation is conducted based on a set of guidelines or heuristics. Hence, it could be carried out by virtually anyone. Usability experts are generally more effective in identifying usability problems as compared to novice evaluators. Nielsen (1995a) stated that simple usability inspection could be carried out after a few hours of training.

“Heuristic evaluation was originally developed as a usability engineering method for evaluators who had some knowledge of usability principles but were not necessarily usability experts.” (Nielsen, 1992a)

Nielsen’s recommendation is three to five evaluators. (Nielsen, 2000a; Nielsen, 2005b) Referring to Figure 2.3, three evaluators could detect about 65% of usability problems, and five evaluators could detect about 85%. Based on Pareto concept and 80/20 rule, five evaluators would be adequate to find at least 80% of the usability problems.

Number of evaluators for satisfactory identification of high proportion of usability problems is as below.
· Double specialists (usability specialists with experience in the specific type of user interface) – 2 to 3 persons (could detect 81 to 90% of usability problems).
· Usability specialists – 3 to 5 persons (could detect 74% to 87% of usability problems.
· Novice evaluators – 14 persons (could detect more than 75% of usability problems. By using 5 novice evaluators, 51% of the usability problems could be identified. (Nielsen, 1992a; Nielsen & Molich, 1990a)

---------------
Evaluating Web Page and Web Site Usability.
Christopher C. Whitehead. Columbus State University, 4225 University Avenue, Columbus, GA 31907. 1-706-565-3527 whitehead_christopher@colstate.edu

ACM SE’06, March, 10-12, 2006, Melbourne, Florida, USA


USABILITY TESTING

Battleson et al. claim that usability testing is the most effective way to asses a site's usability [1]. They go on to specify three categories of usability testing: inquiry, inspection, and formal usability testing.

Inquiry usability testing requests information from the user and includes the use of focus groups, interviews, questionnaires, and surveys.

Inspection usability testing includes heuristic evaluation and cognitive walkthroughs.
According to Dickstein and Mills, “Heuristic evaluation is a systematic inspection of a user interface to examine if the design is in compliance with recognized usability” [2]. This typically involves checking the site “against a checklist of heuristics, or design principles” [1].
Cognitive walkthroughs involve enlisting the use of individuals “to accomplish typical user tasks with a given interface” [1].

Under formal usability testing, a formal set of tasks and goals are defined and users are observed while performing these tasks. According to Battleson et al., formal usability testing involves the following: “(1) the goal is to improve the usability of the interface; (2) testers represent real users; (3) testers perform real tasks; (4) user behavior and commentary are observed and recorded; and (5) data are analyzed to recognize problems and suggest solutions” [1].

Ivory et al. also report that simulation has been used to effectively evaluate web sites by automating the implementation of usability testing [3]. Using this methodology, user behavior is mimicked in a controlled environment or through computer simulation without actual users.

Determining the Number of Users to Involve in the TestingIn performing usability testing, Battleson et al. claim that in a homogenous user testing group, as little as five users are needed to ascertain a large percentage of the most critical errors [1].
In comparison, Dickstein and Mills found that eight to twelve users was sufficient enough to determine problems with a Web site’s usability [2]. However, they also claim that by the fifth user, they were able to determine whether the tested feature was either a problem or a success.

USABILITY METRICS

Usability metrics are measures of a particular aspect of a Web page or Web site that has an impact on usability.

In performing quantitative usability testing, Ivory, et al. suggest the following Web page metrics as the most important in evaluating usability [3]:
• Word Count--Total words on a page
• Body Text %--Percentage of words that are body vs. display text (i.e., headers)
• Emphasized Body Text %--Portion of body text that is emphasized (e.g., bold, capitalized or near !'s)
• Text Positioning Count--Changes in text position from flush left
• Text Cluster Count --Text areas highlighted with color, bordered regions, rules or lists
• Link Count--Total links on a page
• Page Size--Total bytes for the page as well as elements graphics and stylesheets
• Graphic %--Percentage of page bytes that are for graphics
• Graphics Count--Total graphics on a page (not including graphics specified in scripts, applets and objects)
• Color Count--Total colors employed
• Font Count--Total fonts employed (i.e., face + size + bold + italic)

Ivory et al. found six metrics to be of most importance: text cluster count, link count, page size, graphics count, color count, and reading complexity [3].

-----------------
Evaluating Web Usability Using Small Display Devices.
Carlos J. Costa. DCTI – ISCTE, Adetti/ISCTE, Lisboa, Portugal. carlos.costa@iscte.pt
José P. Novais Silva. Portugal Telecom, Lisboa, Portugal. jose.p.silva@telecom.pt
Manuela Aparício. ITML, Lisboa, Portugal. manuela@design.itml.org

SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.


Usability testing is an usual tool used to evaluate the usability of a mobile application in a development process. In fact, mobile and handheld usability testing could be even more important than computer-based usability testing.
Webcredible [23] explain the main reasons for this:
• Increase of small display devices sales.
• Less experience of using small display devices to navigate on the web than using desktop computers.

How usability can be tested?

Lee and Grice [15] propose a usability testing that combines heuristics, questionnaires, and scenarios for developing mobile applications.
According to Andrews [1] there are three usability inspection methods: Heuristic Evaluation, Cognitive Walkthrough and Action Analysis. In Heuristic Evaluation a small set of evaluators examines interface and judges its compliance with recognized usability principles, while in Cognitive Walkthrough, exist a taskoriented walkthrough based on formal cognitive model of user behaviour (analyses learnability).
On the other and, in Action Analysis is made an quantitative analysis of actions to predict time required for tasks, based on time estimates for typical interface actions (analyses efficiency).

---------------
Conceptual Framework and Models for Identifying and Organizing Usability Impact Factors of Mobile Phones.
Dong-Han Ham1, Jeongyun Heo2, Peter Fossick3, William Wong1, Sanghyun Park2, Chiwon Song2, Mike Bradley3
1School of Computing Science, Middlesex University, The Burroughs London, NW4 4BT UK. {d.ham, w.wong}@mdx.ac.uk
2MC R&D Centre, LG Electronics, Seoul Korea. {jy_heo, sanghyun, chiwon79}@lge.com
3Product Design Engineering, Middlesex University, Bramley Road London, N14 4YZ UK. {p.fossick, m.d.bradley}@mdx.ac.uk

OZCHI 2006, November 20-24, 2006, Sydney, Australia.


Usability can be measured in various ways; however, they can be categorized in three methods: usability testing, usability inquiry, and usability inspection (Zhang, 2003).

It cannot be said that one method is the best in all situations. Hence it is necessary to choose an appropriate method, taking into consideration evaluation purposes, available time, measures to be collected and so on.

---------------
Methods for quantitative usability requirements: a case study on the development of the user interface of a mobile phone.
Timo Jokela Æ Jussi Koivumaa Æ Jani Pirkola
Petri Salminen Æ Niina Kantola

Pers Ubiquit Comput (2006) 10: 345–355DOI 10.1007/s00779-005-0050-7

T. Jokela (&) Æ N. Kantola. Oulu University, P.O. Box 3000, Oulu, Finland. E-mail: timo.jokela@oulu.fi E-mail: niina.kantola@oulu.fi
J. Koivumaa Æ J. Pirkola. Nokia, P.O. Box 50, 90571 Oulu, Finland. E-mail: jussi.koivumaa@nokia.com E-mail: jani.pirkola@nokia.com
P. Salminen. ValueFirst, Luuvantie 28, 02620 Espoo, Finland. E-mail: petri.salminen@valuefirst.fi


Methods for quantitative evaluation of usability

Whether the quantitative requirements have been met can be determined through a usability test.
Wixon et al. [14] define the term ‘‘test’’ as a broad term that encompasses any method for assessing whether goals have been achieved, like a formal laboratory test or a collection of satisfaction data through survey.
When evaluating usability, ISO 9241-11 [19] claims it is important that the context selected be representative. Evaluations can be done in the field in a real work situation or in laboratory settings in which the relevant aspects of the context of use are re-created in a representative and controlled way. A method that includes representative users performing typical, representative tasks is generally called usability testing.

Tasks that are done in usability testing provide an objective metric for the related usability attribute. Hix and Hartson [6] indicate that tasks must be very specifically worded in order to be the same for each participant. Tasks must also be specific, so that participants do not get sidetracked into irrelevant details during testing.
Wixon et al. [14] suggest that during the test, the tester should minimize the interaction with participants. Butler [33] describes his approach where ‘‘seven test users were given an introductory level problem to solve, then left alone with a copy of the user’s guide and a 3270 terminal logged onto the system.’’
User preference questionnaires provide a subjective metric for the related usability attribute such as ease of use or usefulness. Questionnaires are commonly built using Likert and semantic differential scales and are intended for use in various circumstances [32].
There are a number of questionnaires available for quantitative usability evaluation, like SUMI [22], QUIS [24] and SUS [23]. Karat [34] states that questionnaires provide an easy and inexpensive method for obtaining measurement data on a system.

Usability can be quantitatively evaluated also with theory-based approaches such as GOMS and keystroke level model, KLM [35].

--------------
20090906-2025