The distributed cognitive walkthrough: The impact of differences in cognitive theory on usability evaluation by Eden, Joel Uzi, Ph.D., Drexel University, 2008 , 158 pages; AAT My Interest: 1) Comparison method of Cognitive Walkthrough & Distributed Cognitive Walkthrough. 2) Cognitive walkthrough. 3) Distributed cognitive walkthrough. 4) First experimental study. Action: Medium-low priority to read the Dissertation in future. Motivation A primary goal of usability related fields is the creation of products andservices that conform to or leverage theoretical views of human cognition. Therefore, most usability related methodologies are directly based on theories of human cognition. Existing user-centered design (UCD) methods are based on cognitivist theoretical views, which hold that the locus of cognition is the brain. Opposing post-cognitivist views within the cognitive science field, such as distributed and extended cognition are not represented in current UCD methods. Research Goal Our research represents a first step towards understanding the impact of differences between these opposing theories of cognition on usability evaluation and design. Research Questions In examining the differential impact of opposing cognitive views (cognitivist versus post-cognitivist), three main research questions were explored using two experimental studies. Methodology – First Study The first experimental study explored the impact of walkthrough evaluation methods with a focus on an individual user versus a focus on interaction between internal and external resources, and the number of users in a scenario on the actionability, usability relevance, and reported severity of usability evaluations. Usability evaluations were conducted by 42 participants using either the Cognitive Walkthrough (representing cognitivist theoretical views) or the Distributed Cognitive Walkthrough (novel method developed as part of this research, representing the distributed and extended cognitive views) with Results Discussion – First Study Use of two-way ANOVA showed a significant increase in all three dependent measures (actionability, usability relevance, and reported severity), with use of the Distributed Cognitive Walkthrough method over the Cognitive Walkthrough method. Methodology – Second Study A second experimental study explored the impact of materiality of artifacts (paper cup versus digital interface) on the actionability, usability relevance, and reported severity of usability evaluations. Usability evaluations were conducted by 22 participants using the Distributed Cognitive Walkthrough with scenarios set in a coffee shop that focused on either a paper coffee cup or a digital order tracking system interface. Results Discussion – Second Study Use of ANOVA showed a significant increase in reported severity for scenarios with the digital interface over scenarios with the paper cup. While differences in actionability and usability relevance by materiality were non-significant, interesting aspects of the differences are discussed. Discussion Beyond the individual results stated above, our findings suggest that an increased focus on the interaction of internal and external cognitive resources and variation in materiality of these resources during evaluation can result in making useful, yet invisible resources more visible, with respect to how an activity is being accomplished. This aspect of addressing the paradoxical invisibility of useful and usable artifacts is a key contribution of this research, showing how even simple differences in evaluation and design methods can aid in making visible these complex interactions between internal and external resources for cognition. Comments: Softcopy Dissertation is scanned version; cannot copy & paste; cannot easily blog about it. |
Showing posts with label cognitive walkthrough. Show all posts
Showing posts with label cognitive walkthrough. Show all posts
Thursday, September 23, 2010
20100924 - Eden, Distributed Cognitive Walkthrough...usability evaluation
Tuesday, September 21, 2010
20100922 - Elgin, Validating the User-CHAT (UET)
Validating the User-Centered Hybrid Assessment Tool (User-CHAT): A comparative usability evaluation by Elgin, Peter D., Ph.D., Kansas State University, 2007 , 168 pages; My Interest: (1) User-CHAT = UET. (2) Hybrid/combination of user testing, heuristic evaluation and cognitive walkthrough. (3) Effectiveness score. (4) Thoroughness, Validity. Action: To read specific parts of Dissertation in future. Therefore the present study had two principal objectives: (1) to validate a hybrid usability technique that identifies important and ignores inconsequential usability problems, and (2) to provide empirical performance data for several usability protocols on a variety of contemporary comparative metrics. The User-C entered H ybrid A ssessment T ool (User-CHAT) was developed to maximize efficient diagnosis of usability issues from a behaviorally-based perspective while minimizing time and resource limitations typically associated with usability assessment environments. Several characteristics of user-testing, the heuristic evaluation, and the cognitive walkthrough were combined to create the User-CHAT. Comments: Use-CHAT is UET. UEM used is hybrid of user testing, heuristic evaluation and cognitive walkthrough. This may be good reference/benchmark for my research. Two techniques generated comparison lists of usability problems---user-testing data and various inclusion criteria for usability problems identified by the User-CHAT, heuristic evaluation, and cognitive walkthrough. Overall the results demonstrated that the User-CHAT attained higher effectiveness scores than the heuristic evaluation and cognitive walkthrough, suggesting that it helped evaluators identify many usability problems that actually impact users, i.e., higher thoroughness, while attenuating time and effort on issues that were not important, i.e., higher validity. Comments: I want to read about how he score & analysed Effectiveness, Thoroughness and Validity. Furthermore, the User-CHAT had the greatest proportion of usability problems that were rated as serious, i.e., usability issues that hinder performance and compromise safety. Comment: How many evaluators? Who were the evaluators? How was usability problems rated – serious, what else? The User-CHAT's performance suggests that it is an appropriate usability technique to implement into the product development lifecycle. Chapter 1 -- Introduction Certification Environment Constraints Certification and Usability Usability Evaluation Methods User-testing Cognitive Walkthrough Heuristic Evaluation UEM Characteristics Incorporated into the User-CHAT Development of the User-CHAT Comparative Usability Evaluation UEM Performance Metrics * Thoroughness * Validity * Effectiveness * Reliability . * Time per Usability Problem Hypotheses. Chapter 2 -- Method Participants The System & Benchmark Tasks Procedure * User-Testing * User-CHAT. * Heuristic Evaluation * Cognitive Walkthrough Severity Ratings Summary of Procedure and Data Collection Data Synthesis.. Classifying Usability Problems into Heuristics Determining Unique or Shared Usability Problems Chapter 3 -- Results Time Spent per Usability Problem Usability Problem Partitions Detection Rates Thoroughness, Validity, and Effectiveness Reliability Severity Ratings Heuristics Classifications Generating a Comparison List without User-testing
|
20100921 - Andre, Effectiveness of Usability Problem Inspector (UET)
Determining the effectiveness of the usability problem inspector: A theory-based model and tool for finding usability problems by Andre, Terence Scott, Ph.D., Virginia Polytechnic Institute and State University, 2000 , 275 pages; AAT 9974182 My interest: (1) UET = Usability Problem Inspector. (2) Model/Theoretical Framework =User Action Framework. (3) Comparison of effectiveness of Usability Problem Inspector against heuristic evaluation and cognitive walkthrough. Action: To read the Dissertation in detail in future. Extract of abstract: The need for cost-effective usability evaluation has led to the development of methodologies to support the usability practitioner in finding usability problems during formative evaluation. Three important research activities. First, the User Action Framework was developed to help organize usability concepts and issues into a knowledge base that supports usability methods and tools. Second, the reliability of the User Action Framework was evaluated to determine if usability practitioners could use the framework in a consistent manner when classifying a set of usability problems. Third, a comprehensive comparison study was conducted to determine if the Usability Problem Inspector, based on the User Action Framework, could produce results just as effective as two other inspection methods (i.e., the heuristic evaluation and the cognitive walkthrough). Comments: Three research objectives, seemingly. From the inspection method comparison study, results showed the Usability Problem Inspector to be more effective than the heuristic evaluation and consistent with effectiveness scores from the cognitive walkthrough.
|
Sunday, September 6, 2009
Sep 7 - about Usability Evaluation Method
My literature review about Usability Evaluation Method.
------
Usability Inspection Methods after 15 Years of Research and Practice.
Tasha Hollingsed. Lockheed Martin, 2540 North Telshor Blvd., Suite C, Las Cruces, NM 88011. +1 505-525-5267 tasha.hollingsed@lmco.com
David G. Novick. Department of Computer Science, The University of Texas at El Paso, El Paso, TX 79968-0518. +1 915-747-5725 novick@utep.edu
SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.
COGNITIVE WALKTHROUGH
The cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use [53].
Like other inspection methods, a cognitive walkthrough can be performed on an interface at any time during the development process, from the original mock-ups through the final release.
The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.
During the analysis phase, the evaluators work through the four steps of humancomputer interaction developed by Lewis and Polson [25]:
1. The user sets a goal to be completed within the system.
2. The user determines the currently available actions.
3. The user selects the action that they think will take them closer to their goal.
4. The user performs the action and evaluates the feedback given by the system.
Wharton et al. [53] surveyed the history of the cognitive walkthrough prior to 1994. At the time, the two main limitations of the cognitive walkthrough were the repetitiveness of filling out the forms and the limited range of problems the process found ([53], [51], [19]). A new method of cognitive walkthrough addressed these limitations by using small groups instead of individual evaluators and rotating the form filling within the group, evaluating the simple tasks first, and by keeping a record of all problems identified during the evaluation, not just those identified during the process ([53], [41]).
Other criticisms of the cognitive walkthrough were that it does not provide guidelines about what makes an action clearly available to a user and that it is not known what types of actions are considered by a broad range of users [51]).
Since its inception, and with its refinements and extensions, the cognitive walkthrough has been shown to be an effective inspection method that can be applied not just by cognitive scientists and usability specialists but also by novice evaluators. However, the choice of task scenario can be difficult; if the scenario is not adequately described, the evaluation is not as effective.
PLURALISTIC USABILITY WALKTHROUGH
The pluralistic usability walkthrough [3] adapted the traditional usability walkthrough to incorporate representative users, product developers, members of the product team, and usability experts in the process. It is defined by five characteristics:
1. Inclusion of representative users, product developers, and human factors professionals;
2. The application’s screens are presented in the same order as they would appear to the user;
3. All participants are asked to assume the role of the user;
4. Participants write down what actions they, as users, would take for each screen before the group discusses the screens; and
5. When discussing each screen, the representative users speak first.
...benefits and limitations of the approach.
On the positive side, this approach offers feedback from users even if the interface is not fully developed, enables rapid iteration of the design cycle, and—because the users are directly involved—can result in “on-the-fly” redesign.
On the negative side, the approach must be limited to representative rather than comprehensive user paths through the interface, and users who, at a particular step, did not choose the path the group will follow must “reset” their interaction with the interface.
My comments: I think pluralistic walkthrough has another strength of using a cross-functional team to evaluate usability.
The pluralistic walkthrough appears to be in active use for assessing usability. It is included in the Usability Professionals Association draft body of knowledge [49].
Available reports indicate that pluralistic usability walkthrough is used in industry. While some human factors experts continue to conduct usability walkthroughs that do not combine stakeholder perspectives (e.g., [7], [50], [44]), it seems likely that use of the pluralistic usability walkthrough is widespread but that teams do not refer to it as such in published reports.
FORMAL USABILITY INSPECTIONS
Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance [22].
Like the pluralistic usability walkthrough, this involves stepping through the user’s task. However, because the reviewers consist of human factors experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the interface as efficiently as possible. The review process includes task performance models and heuristics, a variety of human-factors expertise, and defect detection within the framework of the software development lifecycle.
Like the cognitive walkthrough, formal usability inspections require definitions of user profiles and task scenarios. And, like the cognitive walkthrough, the reviewers use a cognitive model of task performance, which can be extended with a checklist of cognitive steps similar to those invoked by Norman [36] to bridge the “gulf of execution.”
Hewlett Packard used this method for at least two years before 1995. The inspection team included design engineers, usability engineers, customer support engineers, and customers at times. The team inspected fourteen products and found an average of 76 usability concerns per product and an average of 74 percent of those concerns were fixed per product. While no formal evaluation of the results was done, it was found that the engineers could detect several of the usability concerns, and the engineers enjoyed using the method while increasing their awareness of user needs [15].
Digital Equipment Corporation also conducted a version of formal usability inspections from 1994 to 1995 on ten products. They found an average of 66 usability problems per product and fixed an average of 52 problems per product. Finding even small usability problems proved to be an asset, especially when a number of these problems were easily fixed. As more problems were fixed, the perceived quality of the product improved as well, even if most of these fixes were small [45].
Since then, it appears that little research has been conducted on formal usability inspections. This approach now tends to be grouped into the overall inspection method class and gets overshadowed by the better known heuristic evaluation when comparisons between inspection and empirical methods have been conducted. As a method, formal usability inspection gains speed at the cost of losing the multiple stakeholder perspectives of the pluralistic walkthrough, and its cognitive model can be seen as less comprehensive than that of the cognitive walkthrough.
CONCLUSION
Both empirical usability testing and usability inspection methods appear to be in wide use, with developers choosing the most appropriate method for their purposes and their context.
For example, half of the ten intranets winning a 2005 competition used heuristic evaluation [34]. The same report indicated that empirical usability testing was used by 80 percent of the winning intranets.
The cognitive walkthrough appears to be in continued use, although reports of use are not as frequent.
The pluralistic usability walkthrough remains in the repertoire of usability experts, although usability experts continue to conduct user-only usability walkthroughs.
And formal usability inspection, although shown to be an effective approach for identifying usability problems, appears to be used less now than in the mid-1990s.
Many have claimed that usability inspection methods make for faster and more cost-efficient evaluation of the usability of an interface than empirical user testing.
But while usability inspection methods do identify a number of usability problems faster and more cost-efficiently, the best performing evaluator and method still only found 44 percent of the usability problems found in a laboratory setting [9].
While the cognitive walkthrough is useful for predicting problems on a given task and heuristic evaluation is useful for predicting problems on the interface, empirical testing provides lots of information throughout the interface and is the benchmark against which all other methods are measured [9].
Indeed, Jeffries et al. [19] noted that evaluators of usability methods may have rated problems found though empirical usability testing as, on average, more severe precisely because the problems were identified empirically rather than analytically. While inspection methods need expert evaluators to be effective, their strengths are that they can be implemented into the early stages of the development cycle and provide a forum in which changes to interface can be discussed.
The research on comparisons of usability assessment methods suggests several lessons for practitioners.
First, while “faster, cheaper” methods such as heuristic evaluation and the pluralistic usability walkthrough can be useful for rapid iteration early in the design cycle, inspection methods cannot fully substitute for the empirical user testing needed before releasing an interface or Web site to the public.
Second, empirical methods can also be used early in the development process, via “low-tech” versions of interfaces.
Third, developers often combine multiple inspection methods— heuristic evaluation and the cognitive walkthrough—in the same project so that they obtain better coverage of usability issues.
And fourth, adding multiple perspectives—along dimensions such as the range of stakeholders or kinds of usability problems—appears to improve the effectiveness of inspection methods.
It remains an open issue as to why usability professionals, in practice, rely on single-perspective methods, typically involving users, or experts, but not both. The evidence from reports of recent uses of heuristic evaluation suggests that many usability specialists are missing the benefits of the pluralistic walkthrough and perspective-based evaluation.
At a deeper level, though, a new direction for research should complement these defect-tracking and learning approaches by seeking to understand the root causes of usability problems. The ideal solution would be to know the reasons for usability problems, so that designers can minimize the effort spent on usability inspection and testing.
My Comments: For my PhD research, shall I choose only 1 UEM (usability evaluation method) or shall I use a hybrid/combination of UEM?
---------------
Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods.
Young Sam Ryu.
Dissertation Submitted to the Faculty of Virginia Polytechnic Institute and State University in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy in Industrial and Systems Engineering
July 2005
Blacksburg, Virginia
There are three different categories of methods to obtain measurements known as the usability inspection method, the testing method, and the inquiry method (Avouris, 2001).
First, the usability inspection method involves having usability experts examine interfaces of the products.
Nielsen and Mack (1994) published a book focusing only on this method explaining that usability inspection aims at finding usability problems, the severity of the usability problems and the overall usability of an entire system.
The major methods in this category are heuristic evaluation, heuristic estimation, cognitive walkthrough, pluralistic walkthrough, feature inspection, consistency inspection, standards inspection, formal usability inspection, and a guidelines checklist.
One advantage of the inspection method is that it can be used in the early stages of the product development cycle..
Second, usability testing methods usually measure system performance based on welldefined usability criteria.
Those criteria can be defined according to the definitions of usability, usability attributes following standards, and empirical metrics. Typically, data on a measured performance are collected based on the observation of individual users performing specific tasks with the product (e.g., completion time and number of errors).
The most widely employed usability testing methods are think-aloud protocol, co-discovery, performance measurement, and field studies, all of which are techniques available not only for usability studies but also for numerous other fields of study. To provide for the validity of this type of evaluation, the proper design of tasks and organization of the testing laboratory are essential (Preece, Rogers, Sharp, Benyon, Holland, & Carey, 1994).
Among the techniques mentioned above, performance measurement is the one that can present objective data clearly, thus ISO 9241-11 provides example metrics for the three different usability criteria (Table 3).
Table 3. Example measures of usability (ISO 9241-11, 1998)
Effectiveness :
- Percentage of goal achieved
- Percentage of tasks completed
- Accuracy of completed task
Efficiency :
- Time to complete a task
- Monetary cost of performing the task
Satisfaction :
- Rating scale for satisfaction
- Frequency of discretionary use
- Frequency of complaints
The usability inquiry method involves communication between the users and the evaluators in the evaluation, usually by means of questions and interviews.
The evaluators question users about their thoughts on the interface or prototype of the system and the users’ ability to answer questions plays a significant role in the evaluation.
Commonly used techniques are focus groups, interviews, field observations, and questionnaires.
Inquiry methods can be used to measure various usability dimensions and attributes; however, the most common usage of inquiry methods is for the measurement of user satisfaction.
Thus, inquiry methods support the user’s point of view, the fourth perspective listed by Keinonen (1998), through the measurement of user satisfaction.
Subjective usability measurements focus on an individual’s personal experience with a product or system.
Several usability questionnaires were developed by the HCI community, such as
Software Usability Measurement Inventory (SUMI) (Kirakowski, 1996; Kirakowski & Corbett, 1993; Porteous, Kirakowski, & Corbett, 1993),
the Questionnaire for User Interaction Satisfaction (QUIS) (Chin, Diehl, & Norman, 1988; Harper & Norman, 1993; Shneiderman, 1986), and
the Post-Study System Usability Questionnaire (PSSUQ) (Lewis, 1995).
----------------
Usability Problem Description and the Evaluator Effect in Usability Testing.
Miranda G. Capra
Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Industrial and Systems Engineering
March 13, 2006
Blacksburg, Virginia
Formative usability evaluations are an important part of the usability life cycle, identifying usability problems present in an interface that designers should fix in the next design iteration.
Figure 1.1 provides an overview of a formative usability evaluation and the usability life cycle.
The output of a formative usability evaluation is a set of usability problems (Hartson, Andre, & Williges, 2003), but there are many different usability evaluation methods (UEMs).
Empirical evaluations involve end-users. The most common empirical method is usability testing or think aloud testing, which is generally a taskbased session in a usability laboratory.
Analytical evaluations involve expert review of an interface, such as Heuristic Evaluation (Nielsen, 1994b; Nielsen & Molich, 1990).
-----------------
Supporting Novice Usability Practitioners with Usability Engineering Tools.
Jonathan Randall Howarth.
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Computer Science and Applications.
April 13, 2007
Blacksburg, Virginia
During the usability evaluation sub-process, usability practitioners primarily conduct formative usability evaluations. As described in a Usability Professional’s Association workshop report [Theofanos et al., 2005], formative usability evaluations are conducted to “guide the improvement in design of future iterations” (p. 3).
Usability practitioners conduct formative usability evaluations to understand the strengths and weaknesses of a given interaction design.
During formative evaluations, usability practitioners collect a variety of qualitative data such as verbal protocol and subjective ratings with the goal of producing UP descriptions.
Summative usability studies represent a different type of usability evaluation that is typically performed after a product is released.
Summative usability studies provide proof in the form of statistical significance that one given interaction design is better than other designs in specific ways.
Usability practitioners may collect quantitative data such as measures of time on task and error counts that they later use in metrics.
Content of Usability Evaluation Reports
In 1997, the Industry Usability Reporting Project initiated by the National Institute of Standards and Technology developed the Common Industry Format (CIF), which is currently the most well known format for usability evaluation reports. The CIF became an American National Standard for Information Technology Standard in 2001 [ANSI, 2001].
By standardizing the reporting of usability tests, the CIF hoped to encourage the consideration of usability in purchasing software products; customer organizations that were interested could evaluate different products based on their CIF reports.The CIF includes sections for describing the product, the method used to evaluate the product, and the results of the evaluation.
The CIF is intended for summative usability evaluations, but usability practitioners most frequently perform formative usability evaluations.
Theofanos [2005] and Theofanus and Quesenbery [2005] describe efforts to develop a new CIF that would provide practitioners with guidance for performing and reporting formative studies.
The usability evaluation report consolidates usability information and provides the context for understanding UP descriptions. Without this context, UP descriptions may be misunderstood or overlooked.
----------------
Usability Inspection Criteria for e-Learning Portals.
TeckChong Yeap.
MDP7515 PROJECT Dissertation submitted in partial fulfilment of the requirement for the degree of
Master of Multimedia (E-Learning Technologies)
MULTIMEDIA UNIVERSITY, MALAYSIA
October 2008
Methods of Usability Inspection
According to Nielsen and Molich (1990a), “There are basically four ways to evaluate a user interface:
· Formally by some analysis technique,
· Automatically by a computerized procedure,
· Empirically by experiments with test users, and
· Heuristically by simply looking at the inter-face and passing judgement according to ones own opinion.”
Nielsen (1995b) classified the types of usability inspection into:
· “Heuristic evaluation is the most informal method and involves having usability specialists judge whether each dialogue element follows established usability principles (the "heuristics").” (Nielsen, 1995b)
· “Heuristic estimation is a variant in which the inspectors are asked to estimate the relative usability of two (or more) designs in quantitative terms (typically expected user performance).” (Nielsen, 1995b)
· “Cognitive walkthrough uses a more explicitly detailed procedure to simulate a user's problem-solving process at each step through the dialogue, checking if the simulated user's goals and memory content can be assumed to lead to the next correct action.” (Nielsen, 1995b)
· “Pluralistic walkthrough uses group meetings where users, developers, and human factors people step through a scenario, discussing each dialogue element.” (Nielsen, 1995b) Developers, users, and human factors people get together in a meeting and go through the e-learning portal step-by-step using a task scenario. They discuss and review the usability issues.
· “Feature inspection lists sequences of features used to accomplish typical tasks, checks for long sequences, cumbersome steps, steps that would not be natural for users to try, and steps that require extensive knowledge/experience in order to assess a proposed feature set.” (Nielsen, 1995b) Sometimes only a particular function in the e-learning portal is inspected, which is called feature inspection. For instance, a feature inspection could be done on the Quizzes feature of the e-learning portal.
· “Consistency inspection has designers who represent multiple other projects inspect an interface to see whether it does things in the same way as their own designs.” (Nielsen, 1995b) Consistency inspection involves web designers from other module of the portal to inspect the user interface, to ensure the consistency of the interaction the entire e-learning portal.
· “Standards inspection has an expert on an interface standard inspect the interface for compliance.” (Nielsen, 1995b) A usability consultant or usability professional inspects the interface to make sure it complies with the international standards or other predetermined standards.
· “Formal usability inspection combines individual and group inspections in a six-step procedure with strictly defined roles to....with elements of both heuristic evaluation and a simplified form of cognitive walkthroughs.” (Nielsen, 1995b; Nielsen, 2005e)
Most Popular Usability Inspection Methods
Hollingsed and Novick (2007) stated that heuristic evaluation and cognitive walkthrough appeared to be the most actively used and researched techniques.
Heuristic Evaluation
Heuristic evaluation is also found to be an economical and relatively easy usability evaluation method.
Ssemugabi & de Villiers (2007) had concluded in their research of usability on e-learning applications that heuristic evaluation which was conducted by a competent group of experts, was an appropriate, efficient, and highly effective usability evaluation method for elearning applications.
Heuristic evaluation could also be done with a web-based tool.
Ardito et al (2006) had used Systematic Usability Inspection Tool (SUIT) to help the evaluation conduct the heuristic evaluation. SUIT is an Internet-based tool that supports the evaluators during the usability inspection of software applications.
SUIT makes it possible to reach inspectors everywhere, guiding them in their activities. 4 major processes involved with heuristic evaluation aided with SUIT are:
1. Inspection planning
2. Problem detection
3. Report collection
4. Discussion.
Cognitive Walkthrough
“Cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use. The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.” (Hollingsed & Novick, 2007)
During the analysis phase, the usability evaluators would carry out the four steps (Lewis & Polson, 1991):
1. The user sets a goal that is to be completed within the system.
2. The user determines the existing available actions.
3. The user selects the action which will take him closer to his goal.
4. The user does the action and evaluates the feedback from the system.
Pluralistic Walkthrough
Pluralistic usability walkthrough (Bias, 1994) is usability walkthrough which incorporates usability experts, product developers, members of the product team and representative users in the process.
Pluralistic usability walkthrough has five characteristics:
1. Inclusion of usability professionals, representative users and product developers.
2. The application’s screens are similar to how they would appear to the user.
3. All evaluators would assume the role of the user.
4. Evaluators write down what actions they, as users, would take for each screen before the group discusses the screens.
5. During discussion of each screen, the representative users would speak first. (Hollingsed and Novick, 2007)
Formal Usability Inspection
Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance (Kahn & Prail, 1994). The reviewers consist of usability experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the user interface. Formal usability inspections require definitions of user profiles and task scenarios. (Hollingsed & Novick, 2007)
Empirical Usability Evaluation
Empirical usability evaluation is an approach that is very different from usability inspection.
Wiberg (2003) stated examples of empirical usability evaluation were (a) think-aloud protocol, (b) use data collection, (c) clinical experiments, (d) surveys and questionnaires, (e) interviews.
Clinical experiments emphasized on statistics for measured clinical aspects such as eye gaze, heart rate, skin color and body heat.
Advantages & Limitations of Heuristic Evaluation
A major advantage is the ease of finding usability problem with the aid of the heuristics principles. “Major usability problems have a higher probability than minor problems of being found in a heuristic evaluation, but about twice as many minor problems are found in absolute numbers.” (Nielsen, 1992a)
Another major advantage is no need for numerous users and usability lab. “Heuristic evaluation is easier to set up as there is no need to bring in users or to analyze interaction protocols, and exploits expert scientific and experiential knowledge of general problems and solutions.” (Nielsen & Landauer, 1993a)
Advantages of heuristic evaluation are
· “It is cheap.
· It is intuitive and it is easy to motivate people to do it.
· It does not require advance planning.
· It can be used early in the development process.” (Nielsen & Molich, 1990a)
Other advantages of heuristic evaluation include:
· Can start using simple usability inspection after a few hours of training.
· Usability inspection methods can be used in many different stages of the system development lifecycle.
· Usability inspection can be integrated easily into many established system development practices.
· Lists of usability problems are available immediately after the usability inspection. (Nielsen, 1995a)
· Lists of interface that need to be improved could be quickly distributed for immediate action.
Limitations of heuristic evaluation are:· Identifies usability problems without providing direct suggestions of solutions.· Influenced by the current mindset of the evaluators, so-called “evaluator effect.”· Does not normally generate breakthroughs in the evaluated design. (Nielsen & Molich, 1990a)
When To Do Heuristic Evaluation
Heuristic evaluation is relatively low cost to be conducted. It could be implemented at various stages of portal or website development cycle.
Heuristic evaluation could be conducted at the stages of:
a) Storyboard,
b) Design stage,
c) Prototype of user interface,
d) “Running version” of the prototype,
e) Partial completion of the e-Learning portal or website,
f) Completion of e-Learning portal,
g) Ongoing review of an existing running e-Learning portal.
Who To Do Heuristic Evaluation
Heuristic evaluation is conducted based on a set of guidelines or heuristics. Hence, it could be carried out by virtually anyone. Usability experts are generally more effective in identifying usability problems as compared to novice evaluators. Nielsen (1995a) stated that simple usability inspection could be carried out after a few hours of training.
“Heuristic evaluation was originally developed as a usability engineering method for evaluators who had some knowledge of usability principles but were not necessarily usability experts.” (Nielsen, 1992a)
Nielsen’s recommendation is three to five evaluators. (Nielsen, 2000a; Nielsen, 2005b) Referring to Figure 2.3, three evaluators could detect about 65% of usability problems, and five evaluators could detect about 85%. Based on Pareto concept and 80/20 rule, five evaluators would be adequate to find at least 80% of the usability problems.
Number of evaluators for satisfactory identification of high proportion of usability problems is as below.
· Double specialists (usability specialists with experience in the specific type of user interface) – 2 to 3 persons (could detect 81 to 90% of usability problems).
· Usability specialists – 3 to 5 persons (could detect 74% to 87% of usability problems.
· Novice evaluators – 14 persons (could detect more than 75% of usability problems. By using 5 novice evaluators, 51% of the usability problems could be identified. (Nielsen, 1992a; Nielsen & Molich, 1990a)
---------------
Evaluating Web Page and Web Site Usability.
Christopher C. Whitehead. Columbus State University, 4225 University Avenue, Columbus, GA 31907. 1-706-565-3527 whitehead_christopher@colstate.edu
ACM SE’06, March, 10-12, 2006, Melbourne, Florida, USA
USABILITY TESTING
Battleson et al. claim that usability testing is the most effective way to asses a site's usability [1]. They go on to specify three categories of usability testing: inquiry, inspection, and formal usability testing.
Inquiry usability testing requests information from the user and includes the use of focus groups, interviews, questionnaires, and surveys.
Inspection usability testing includes heuristic evaluation and cognitive walkthroughs.
According to Dickstein and Mills, “Heuristic evaluation is a systematic inspection of a user interface to examine if the design is in compliance with recognized usability” [2]. This typically involves checking the site “against a checklist of heuristics, or design principles” [1].
Cognitive walkthroughs involve enlisting the use of individuals “to accomplish typical user tasks with a given interface” [1].
Under formal usability testing, a formal set of tasks and goals are defined and users are observed while performing these tasks. According to Battleson et al., formal usability testing involves the following: “(1) the goal is to improve the usability of the interface; (2) testers represent real users; (3) testers perform real tasks; (4) user behavior and commentary are observed and recorded; and (5) data are analyzed to recognize problems and suggest solutions” [1].
Ivory et al. also report that simulation has been used to effectively evaluate web sites by automating the implementation of usability testing [3]. Using this methodology, user behavior is mimicked in a controlled environment or through computer simulation without actual users.
Determining the Number of Users to Involve in the TestingIn performing usability testing, Battleson et al. claim that in a homogenous user testing group, as little as five users are needed to ascertain a large percentage of the most critical errors [1].
In comparison, Dickstein and Mills found that eight to twelve users was sufficient enough to determine problems with a Web site’s usability [2]. However, they also claim that by the fifth user, they were able to determine whether the tested feature was either a problem or a success.
USABILITY METRICS
Usability metrics are measures of a particular aspect of a Web page or Web site that has an impact on usability.
In performing quantitative usability testing, Ivory, et al. suggest the following Web page metrics as the most important in evaluating usability [3]:
• Word Count--Total words on a page
• Body Text %--Percentage of words that are body vs. display text (i.e., headers)
• Emphasized Body Text %--Portion of body text that is emphasized (e.g., bold, capitalized or near !'s)
• Text Positioning Count--Changes in text position from flush left
• Text Cluster Count --Text areas highlighted with color, bordered regions, rules or lists
• Link Count--Total links on a page
• Page Size--Total bytes for the page as well as elements graphics and stylesheets
• Graphic %--Percentage of page bytes that are for graphics
• Graphics Count--Total graphics on a page (not including graphics specified in scripts, applets and objects)
• Color Count--Total colors employed
• Font Count--Total fonts employed (i.e., face + size + bold + italic)
Ivory et al. found six metrics to be of most importance: text cluster count, link count, page size, graphics count, color count, and reading complexity [3].
-----------------
Evaluating Web Usability Using Small Display Devices.
Carlos J. Costa. DCTI – ISCTE, Adetti/ISCTE, Lisboa, Portugal. carlos.costa@iscte.pt
José P. Novais Silva. Portugal Telecom, Lisboa, Portugal. jose.p.silva@telecom.pt
Manuela Aparício. ITML, Lisboa, Portugal. manuela@design.itml.org
SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.
Usability testing is an usual tool used to evaluate the usability of a mobile application in a development process. In fact, mobile and handheld usability testing could be even more important than computer-based usability testing.
Webcredible [23] explain the main reasons for this:
• Increase of small display devices sales.
• Less experience of using small display devices to navigate on the web than using desktop computers.
How usability can be tested?
Lee and Grice [15] propose a usability testing that combines heuristics, questionnaires, and scenarios for developing mobile applications.
According to Andrews [1] there are three usability inspection methods: Heuristic Evaluation, Cognitive Walkthrough and Action Analysis. In Heuristic Evaluation a small set of evaluators examines interface and judges its compliance with recognized usability principles, while in Cognitive Walkthrough, exist a taskoriented walkthrough based on formal cognitive model of user behaviour (analyses learnability).
On the other and, in Action Analysis is made an quantitative analysis of actions to predict time required for tasks, based on time estimates for typical interface actions (analyses efficiency).
---------------
Conceptual Framework and Models for Identifying and Organizing Usability Impact Factors of Mobile Phones.
Dong-Han Ham1, Jeongyun Heo2, Peter Fossick3, William Wong1, Sanghyun Park2, Chiwon Song2, Mike Bradley3
1School of Computing Science, Middlesex University, The Burroughs London, NW4 4BT UK. {d.ham, w.wong}@mdx.ac.uk
2MC R&D Centre, LG Electronics, Seoul Korea. {jy_heo, sanghyun, chiwon79}@lge.com
3Product Design Engineering, Middlesex University, Bramley Road London, N14 4YZ UK. {p.fossick, m.d.bradley}@mdx.ac.uk
OZCHI 2006, November 20-24, 2006, Sydney, Australia.
Usability can be measured in various ways; however, they can be categorized in three methods: usability testing, usability inquiry, and usability inspection (Zhang, 2003).
It cannot be said that one method is the best in all situations. Hence it is necessary to choose an appropriate method, taking into consideration evaluation purposes, available time, measures to be collected and so on.
---------------
Methods for quantitative usability requirements: a case study on the development of the user interface of a mobile phone.
Timo Jokela Æ Jussi Koivumaa Æ Jani Pirkola
Petri Salminen Æ Niina Kantola
Pers Ubiquit Comput (2006) 10: 345–355DOI 10.1007/s00779-005-0050-7
T. Jokela (&) Æ N. Kantola. Oulu University, P.O. Box 3000, Oulu, Finland. E-mail: timo.jokela@oulu.fi E-mail: niina.kantola@oulu.fi
J. Koivumaa Æ J. Pirkola. Nokia, P.O. Box 50, 90571 Oulu, Finland. E-mail: jussi.koivumaa@nokia.com E-mail: jani.pirkola@nokia.com
P. Salminen. ValueFirst, Luuvantie 28, 02620 Espoo, Finland. E-mail: petri.salminen@valuefirst.fi
Methods for quantitative evaluation of usability
Whether the quantitative requirements have been met can be determined through a usability test.
Wixon et al. [14] define the term ‘‘test’’ as a broad term that encompasses any method for assessing whether goals have been achieved, like a formal laboratory test or a collection of satisfaction data through survey.
When evaluating usability, ISO 9241-11 [19] claims it is important that the context selected be representative. Evaluations can be done in the field in a real work situation or in laboratory settings in which the relevant aspects of the context of use are re-created in a representative and controlled way. A method that includes representative users performing typical, representative tasks is generally called usability testing.
Tasks that are done in usability testing provide an objective metric for the related usability attribute. Hix and Hartson [6] indicate that tasks must be very specifically worded in order to be the same for each participant. Tasks must also be specific, so that participants do not get sidetracked into irrelevant details during testing.
Wixon et al. [14] suggest that during the test, the tester should minimize the interaction with participants. Butler [33] describes his approach where ‘‘seven test users were given an introductory level problem to solve, then left alone with a copy of the user’s guide and a 3270 terminal logged onto the system.’’
User preference questionnaires provide a subjective metric for the related usability attribute such as ease of use or usefulness. Questionnaires are commonly built using Likert and semantic differential scales and are intended for use in various circumstances [32].
There are a number of questionnaires available for quantitative usability evaluation, like SUMI [22], QUIS [24] and SUS [23]. Karat [34] states that questionnaires provide an easy and inexpensive method for obtaining measurement data on a system.
Usability can be quantitatively evaluated also with theory-based approaches such as GOMS and keystroke level model, KLM [35].
--------------
20090906-2025
------
Usability Inspection Methods after 15 Years of Research and Practice.
Tasha Hollingsed. Lockheed Martin, 2540 North Telshor Blvd., Suite C, Las Cruces, NM 88011. +1 505-525-5267 tasha.hollingsed@lmco.com
David G. Novick. Department of Computer Science, The University of Texas at El Paso, El Paso, TX 79968-0518. +1 915-747-5725 novick@utep.edu
SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.
COGNITIVE WALKTHROUGH
The cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use [53].
Like other inspection methods, a cognitive walkthrough can be performed on an interface at any time during the development process, from the original mock-ups through the final release.
The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.
During the analysis phase, the evaluators work through the four steps of humancomputer interaction developed by Lewis and Polson [25]:
1. The user sets a goal to be completed within the system.
2. The user determines the currently available actions.
3. The user selects the action that they think will take them closer to their goal.
4. The user performs the action and evaluates the feedback given by the system.
Wharton et al. [53] surveyed the history of the cognitive walkthrough prior to 1994. At the time, the two main limitations of the cognitive walkthrough were the repetitiveness of filling out the forms and the limited range of problems the process found ([53], [51], [19]). A new method of cognitive walkthrough addressed these limitations by using small groups instead of individual evaluators and rotating the form filling within the group, evaluating the simple tasks first, and by keeping a record of all problems identified during the evaluation, not just those identified during the process ([53], [41]).
Other criticisms of the cognitive walkthrough were that it does not provide guidelines about what makes an action clearly available to a user and that it is not known what types of actions are considered by a broad range of users [51]).
Since its inception, and with its refinements and extensions, the cognitive walkthrough has been shown to be an effective inspection method that can be applied not just by cognitive scientists and usability specialists but also by novice evaluators. However, the choice of task scenario can be difficult; if the scenario is not adequately described, the evaluation is not as effective.
PLURALISTIC USABILITY WALKTHROUGH
The pluralistic usability walkthrough [3] adapted the traditional usability walkthrough to incorporate representative users, product developers, members of the product team, and usability experts in the process. It is defined by five characteristics:
1. Inclusion of representative users, product developers, and human factors professionals;
2. The application’s screens are presented in the same order as they would appear to the user;
3. All participants are asked to assume the role of the user;
4. Participants write down what actions they, as users, would take for each screen before the group discusses the screens; and
5. When discussing each screen, the representative users speak first.
...benefits and limitations of the approach.
On the positive side, this approach offers feedback from users even if the interface is not fully developed, enables rapid iteration of the design cycle, and—because the users are directly involved—can result in “on-the-fly” redesign.
On the negative side, the approach must be limited to representative rather than comprehensive user paths through the interface, and users who, at a particular step, did not choose the path the group will follow must “reset” their interaction with the interface.
My comments: I think pluralistic walkthrough has another strength of using a cross-functional team to evaluate usability.
The pluralistic walkthrough appears to be in active use for assessing usability. It is included in the Usability Professionals Association draft body of knowledge [49].
Available reports indicate that pluralistic usability walkthrough is used in industry. While some human factors experts continue to conduct usability walkthroughs that do not combine stakeholder perspectives (e.g., [7], [50], [44]), it seems likely that use of the pluralistic usability walkthrough is widespread but that teams do not refer to it as such in published reports.
FORMAL USABILITY INSPECTIONS
Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance [22].
Like the pluralistic usability walkthrough, this involves stepping through the user’s task. However, because the reviewers consist of human factors experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the interface as efficiently as possible. The review process includes task performance models and heuristics, a variety of human-factors expertise, and defect detection within the framework of the software development lifecycle.
Like the cognitive walkthrough, formal usability inspections require definitions of user profiles and task scenarios. And, like the cognitive walkthrough, the reviewers use a cognitive model of task performance, which can be extended with a checklist of cognitive steps similar to those invoked by Norman [36] to bridge the “gulf of execution.”
Hewlett Packard used this method for at least two years before 1995. The inspection team included design engineers, usability engineers, customer support engineers, and customers at times. The team inspected fourteen products and found an average of 76 usability concerns per product and an average of 74 percent of those concerns were fixed per product. While no formal evaluation of the results was done, it was found that the engineers could detect several of the usability concerns, and the engineers enjoyed using the method while increasing their awareness of user needs [15].
Digital Equipment Corporation also conducted a version of formal usability inspections from 1994 to 1995 on ten products. They found an average of 66 usability problems per product and fixed an average of 52 problems per product. Finding even small usability problems proved to be an asset, especially when a number of these problems were easily fixed. As more problems were fixed, the perceived quality of the product improved as well, even if most of these fixes were small [45].
Since then, it appears that little research has been conducted on formal usability inspections. This approach now tends to be grouped into the overall inspection method class and gets overshadowed by the better known heuristic evaluation when comparisons between inspection and empirical methods have been conducted. As a method, formal usability inspection gains speed at the cost of losing the multiple stakeholder perspectives of the pluralistic walkthrough, and its cognitive model can be seen as less comprehensive than that of the cognitive walkthrough.
CONCLUSION
Both empirical usability testing and usability inspection methods appear to be in wide use, with developers choosing the most appropriate method for their purposes and their context.
For example, half of the ten intranets winning a 2005 competition used heuristic evaluation [34]. The same report indicated that empirical usability testing was used by 80 percent of the winning intranets.
The cognitive walkthrough appears to be in continued use, although reports of use are not as frequent.
The pluralistic usability walkthrough remains in the repertoire of usability experts, although usability experts continue to conduct user-only usability walkthroughs.
And formal usability inspection, although shown to be an effective approach for identifying usability problems, appears to be used less now than in the mid-1990s.
Many have claimed that usability inspection methods make for faster and more cost-efficient evaluation of the usability of an interface than empirical user testing.
But while usability inspection methods do identify a number of usability problems faster and more cost-efficiently, the best performing evaluator and method still only found 44 percent of the usability problems found in a laboratory setting [9].
While the cognitive walkthrough is useful for predicting problems on a given task and heuristic evaluation is useful for predicting problems on the interface, empirical testing provides lots of information throughout the interface and is the benchmark against which all other methods are measured [9].
Indeed, Jeffries et al. [19] noted that evaluators of usability methods may have rated problems found though empirical usability testing as, on average, more severe precisely because the problems were identified empirically rather than analytically. While inspection methods need expert evaluators to be effective, their strengths are that they can be implemented into the early stages of the development cycle and provide a forum in which changes to interface can be discussed.
The research on comparisons of usability assessment methods suggests several lessons for practitioners.
First, while “faster, cheaper” methods such as heuristic evaluation and the pluralistic usability walkthrough can be useful for rapid iteration early in the design cycle, inspection methods cannot fully substitute for the empirical user testing needed before releasing an interface or Web site to the public.
Second, empirical methods can also be used early in the development process, via “low-tech” versions of interfaces.
Third, developers often combine multiple inspection methods— heuristic evaluation and the cognitive walkthrough—in the same project so that they obtain better coverage of usability issues.
And fourth, adding multiple perspectives—along dimensions such as the range of stakeholders or kinds of usability problems—appears to improve the effectiveness of inspection methods.
It remains an open issue as to why usability professionals, in practice, rely on single-perspective methods, typically involving users, or experts, but not both. The evidence from reports of recent uses of heuristic evaluation suggests that many usability specialists are missing the benefits of the pluralistic walkthrough and perspective-based evaluation.
At a deeper level, though, a new direction for research should complement these defect-tracking and learning approaches by seeking to understand the root causes of usability problems. The ideal solution would be to know the reasons for usability problems, so that designers can minimize the effort spent on usability inspection and testing.
My Comments: For my PhD research, shall I choose only 1 UEM (usability evaluation method) or shall I use a hybrid/combination of UEM?
---------------
Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods.
Young Sam Ryu.
Dissertation Submitted to the Faculty of Virginia Polytechnic Institute and State University in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy in Industrial and Systems Engineering
July 2005
Blacksburg, Virginia
There are three different categories of methods to obtain measurements known as the usability inspection method, the testing method, and the inquiry method (Avouris, 2001).
First, the usability inspection method involves having usability experts examine interfaces of the products.
Nielsen and Mack (1994) published a book focusing only on this method explaining that usability inspection aims at finding usability problems, the severity of the usability problems and the overall usability of an entire system.
The major methods in this category are heuristic evaluation, heuristic estimation, cognitive walkthrough, pluralistic walkthrough, feature inspection, consistency inspection, standards inspection, formal usability inspection, and a guidelines checklist.
One advantage of the inspection method is that it can be used in the early stages of the product development cycle..
Second, usability testing methods usually measure system performance based on welldefined usability criteria.
Those criteria can be defined according to the definitions of usability, usability attributes following standards, and empirical metrics. Typically, data on a measured performance are collected based on the observation of individual users performing specific tasks with the product (e.g., completion time and number of errors).
The most widely employed usability testing methods are think-aloud protocol, co-discovery, performance measurement, and field studies, all of which are techniques available not only for usability studies but also for numerous other fields of study. To provide for the validity of this type of evaluation, the proper design of tasks and organization of the testing laboratory are essential (Preece, Rogers, Sharp, Benyon, Holland, & Carey, 1994).
Among the techniques mentioned above, performance measurement is the one that can present objective data clearly, thus ISO 9241-11 provides example metrics for the three different usability criteria (Table 3).
Table 3. Example measures of usability (ISO 9241-11, 1998)
Effectiveness :
- Percentage of goal achieved
- Percentage of tasks completed
- Accuracy of completed task
Efficiency :
- Time to complete a task
- Monetary cost of performing the task
Satisfaction :
- Rating scale for satisfaction
- Frequency of discretionary use
- Frequency of complaints
The usability inquiry method involves communication between the users and the evaluators in the evaluation, usually by means of questions and interviews.
The evaluators question users about their thoughts on the interface or prototype of the system and the users’ ability to answer questions plays a significant role in the evaluation.
Commonly used techniques are focus groups, interviews, field observations, and questionnaires.
Inquiry methods can be used to measure various usability dimensions and attributes; however, the most common usage of inquiry methods is for the measurement of user satisfaction.
Thus, inquiry methods support the user’s point of view, the fourth perspective listed by Keinonen (1998), through the measurement of user satisfaction.
Subjective usability measurements focus on an individual’s personal experience with a product or system.
Several usability questionnaires were developed by the HCI community, such as
Software Usability Measurement Inventory (SUMI) (Kirakowski, 1996; Kirakowski & Corbett, 1993; Porteous, Kirakowski, & Corbett, 1993),
the Questionnaire for User Interaction Satisfaction (QUIS) (Chin, Diehl, & Norman, 1988; Harper & Norman, 1993; Shneiderman, 1986), and
the Post-Study System Usability Questionnaire (PSSUQ) (Lewis, 1995).
----------------
Usability Problem Description and the Evaluator Effect in Usability Testing.
Miranda G. Capra
Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Industrial and Systems Engineering
March 13, 2006
Blacksburg, Virginia
Formative usability evaluations are an important part of the usability life cycle, identifying usability problems present in an interface that designers should fix in the next design iteration.
Figure 1.1 provides an overview of a formative usability evaluation and the usability life cycle.
The output of a formative usability evaluation is a set of usability problems (Hartson, Andre, & Williges, 2003), but there are many different usability evaluation methods (UEMs).
Empirical evaluations involve end-users. The most common empirical method is usability testing or think aloud testing, which is generally a taskbased session in a usability laboratory.
Analytical evaluations involve expert review of an interface, such as Heuristic Evaluation (Nielsen, 1994b; Nielsen & Molich, 1990).
-----------------
Supporting Novice Usability Practitioners with Usability Engineering Tools.
Jonathan Randall Howarth.
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Computer Science and Applications.
April 13, 2007
Blacksburg, Virginia
During the usability evaluation sub-process, usability practitioners primarily conduct formative usability evaluations. As described in a Usability Professional’s Association workshop report [Theofanos et al., 2005], formative usability evaluations are conducted to “guide the improvement in design of future iterations” (p. 3).
Usability practitioners conduct formative usability evaluations to understand the strengths and weaknesses of a given interaction design.
During formative evaluations, usability practitioners collect a variety of qualitative data such as verbal protocol and subjective ratings with the goal of producing UP descriptions.
Summative usability studies represent a different type of usability evaluation that is typically performed after a product is released.
Summative usability studies provide proof in the form of statistical significance that one given interaction design is better than other designs in specific ways.
Usability practitioners may collect quantitative data such as measures of time on task and error counts that they later use in metrics.
Content of Usability Evaluation Reports
In 1997, the Industry Usability Reporting Project initiated by the National Institute of Standards and Technology developed the Common Industry Format (CIF), which is currently the most well known format for usability evaluation reports. The CIF became an American National Standard for Information Technology Standard in 2001 [ANSI, 2001].
By standardizing the reporting of usability tests, the CIF hoped to encourage the consideration of usability in purchasing software products; customer organizations that were interested could evaluate different products based on their CIF reports.The CIF includes sections for describing the product, the method used to evaluate the product, and the results of the evaluation.
The CIF is intended for summative usability evaluations, but usability practitioners most frequently perform formative usability evaluations.
Theofanos [2005] and Theofanus and Quesenbery [2005] describe efforts to develop a new CIF that would provide practitioners with guidance for performing and reporting formative studies.
The usability evaluation report consolidates usability information and provides the context for understanding UP descriptions. Without this context, UP descriptions may be misunderstood or overlooked.
----------------
Usability Inspection Criteria for e-Learning Portals.
TeckChong Yeap.
MDP7515 PROJECT Dissertation submitted in partial fulfilment of the requirement for the degree of
Master of Multimedia (E-Learning Technologies)
MULTIMEDIA UNIVERSITY, MALAYSIA
October 2008
Methods of Usability Inspection
According to Nielsen and Molich (1990a), “There are basically four ways to evaluate a user interface:
· Formally by some analysis technique,
· Automatically by a computerized procedure,
· Empirically by experiments with test users, and
· Heuristically by simply looking at the inter-face and passing judgement according to ones own opinion.”
Nielsen (1995b) classified the types of usability inspection into:
· “Heuristic evaluation is the most informal method and involves having usability specialists judge whether each dialogue element follows established usability principles (the "heuristics").” (Nielsen, 1995b)
· “Heuristic estimation is a variant in which the inspectors are asked to estimate the relative usability of two (or more) designs in quantitative terms (typically expected user performance).” (Nielsen, 1995b)
· “Cognitive walkthrough uses a more explicitly detailed procedure to simulate a user's problem-solving process at each step through the dialogue, checking if the simulated user's goals and memory content can be assumed to lead to the next correct action.” (Nielsen, 1995b)
· “Pluralistic walkthrough uses group meetings where users, developers, and human factors people step through a scenario, discussing each dialogue element.” (Nielsen, 1995b) Developers, users, and human factors people get together in a meeting and go through the e-learning portal step-by-step using a task scenario. They discuss and review the usability issues.
· “Feature inspection lists sequences of features used to accomplish typical tasks, checks for long sequences, cumbersome steps, steps that would not be natural for users to try, and steps that require extensive knowledge/experience in order to assess a proposed feature set.” (Nielsen, 1995b) Sometimes only a particular function in the e-learning portal is inspected, which is called feature inspection. For instance, a feature inspection could be done on the Quizzes feature of the e-learning portal.
· “Consistency inspection has designers who represent multiple other projects inspect an interface to see whether it does things in the same way as their own designs.” (Nielsen, 1995b) Consistency inspection involves web designers from other module of the portal to inspect the user interface, to ensure the consistency of the interaction the entire e-learning portal.
· “Standards inspection has an expert on an interface standard inspect the interface for compliance.” (Nielsen, 1995b) A usability consultant or usability professional inspects the interface to make sure it complies with the international standards or other predetermined standards.
· “Formal usability inspection combines individual and group inspections in a six-step procedure with strictly defined roles to....with elements of both heuristic evaluation and a simplified form of cognitive walkthroughs.” (Nielsen, 1995b; Nielsen, 2005e)
Most Popular Usability Inspection Methods
Hollingsed and Novick (2007) stated that heuristic evaluation and cognitive walkthrough appeared to be the most actively used and researched techniques.
Heuristic Evaluation
Heuristic evaluation is also found to be an economical and relatively easy usability evaluation method.
Ssemugabi & de Villiers (2007) had concluded in their research of usability on e-learning applications that heuristic evaluation which was conducted by a competent group of experts, was an appropriate, efficient, and highly effective usability evaluation method for elearning applications.
Heuristic evaluation could also be done with a web-based tool.
Ardito et al (2006) had used Systematic Usability Inspection Tool (SUIT) to help the evaluation conduct the heuristic evaluation. SUIT is an Internet-based tool that supports the evaluators during the usability inspection of software applications.
SUIT makes it possible to reach inspectors everywhere, guiding them in their activities. 4 major processes involved with heuristic evaluation aided with SUIT are:
1. Inspection planning
2. Problem detection
3. Report collection
4. Discussion.
Cognitive Walkthrough
“Cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use. The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.” (Hollingsed & Novick, 2007)
During the analysis phase, the usability evaluators would carry out the four steps (Lewis & Polson, 1991):
1. The user sets a goal that is to be completed within the system.
2. The user determines the existing available actions.
3. The user selects the action which will take him closer to his goal.
4. The user does the action and evaluates the feedback from the system.
Pluralistic Walkthrough
Pluralistic usability walkthrough (Bias, 1994) is usability walkthrough which incorporates usability experts, product developers, members of the product team and representative users in the process.
Pluralistic usability walkthrough has five characteristics:
1. Inclusion of usability professionals, representative users and product developers.
2. The application’s screens are similar to how they would appear to the user.
3. All evaluators would assume the role of the user.
4. Evaluators write down what actions they, as users, would take for each screen before the group discusses the screens.
5. During discussion of each screen, the representative users would speak first. (Hollingsed and Novick, 2007)
Formal Usability Inspection
Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance (Kahn & Prail, 1994). The reviewers consist of usability experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the user interface. Formal usability inspections require definitions of user profiles and task scenarios. (Hollingsed & Novick, 2007)
Empirical Usability Evaluation
Empirical usability evaluation is an approach that is very different from usability inspection.
Wiberg (2003) stated examples of empirical usability evaluation were (a) think-aloud protocol, (b) use data collection, (c) clinical experiments, (d) surveys and questionnaires, (e) interviews.
Clinical experiments emphasized on statistics for measured clinical aspects such as eye gaze, heart rate, skin color and body heat.
Advantages & Limitations of Heuristic Evaluation
A major advantage is the ease of finding usability problem with the aid of the heuristics principles. “Major usability problems have a higher probability than minor problems of being found in a heuristic evaluation, but about twice as many minor problems are found in absolute numbers.” (Nielsen, 1992a)
Another major advantage is no need for numerous users and usability lab. “Heuristic evaluation is easier to set up as there is no need to bring in users or to analyze interaction protocols, and exploits expert scientific and experiential knowledge of general problems and solutions.” (Nielsen & Landauer, 1993a)
Advantages of heuristic evaluation are
· “It is cheap.
· It is intuitive and it is easy to motivate people to do it.
· It does not require advance planning.
· It can be used early in the development process.” (Nielsen & Molich, 1990a)
Other advantages of heuristic evaluation include:
· Can start using simple usability inspection after a few hours of training.
· Usability inspection methods can be used in many different stages of the system development lifecycle.
· Usability inspection can be integrated easily into many established system development practices.
· Lists of usability problems are available immediately after the usability inspection. (Nielsen, 1995a)
· Lists of interface that need to be improved could be quickly distributed for immediate action.
Limitations of heuristic evaluation are:· Identifies usability problems without providing direct suggestions of solutions.· Influenced by the current mindset of the evaluators, so-called “evaluator effect.”· Does not normally generate breakthroughs in the evaluated design. (Nielsen & Molich, 1990a)
When To Do Heuristic Evaluation
Heuristic evaluation is relatively low cost to be conducted. It could be implemented at various stages of portal or website development cycle.
Heuristic evaluation could be conducted at the stages of:
a) Storyboard,
b) Design stage,
c) Prototype of user interface,
d) “Running version” of the prototype,
e) Partial completion of the e-Learning portal or website,
f) Completion of e-Learning portal,
g) Ongoing review of an existing running e-Learning portal.
Who To Do Heuristic Evaluation
Heuristic evaluation is conducted based on a set of guidelines or heuristics. Hence, it could be carried out by virtually anyone. Usability experts are generally more effective in identifying usability problems as compared to novice evaluators. Nielsen (1995a) stated that simple usability inspection could be carried out after a few hours of training.
“Heuristic evaluation was originally developed as a usability engineering method for evaluators who had some knowledge of usability principles but were not necessarily usability experts.” (Nielsen, 1992a)
Nielsen’s recommendation is three to five evaluators. (Nielsen, 2000a; Nielsen, 2005b) Referring to Figure 2.3, three evaluators could detect about 65% of usability problems, and five evaluators could detect about 85%. Based on Pareto concept and 80/20 rule, five evaluators would be adequate to find at least 80% of the usability problems.
Number of evaluators for satisfactory identification of high proportion of usability problems is as below.
· Double specialists (usability specialists with experience in the specific type of user interface) – 2 to 3 persons (could detect 81 to 90% of usability problems).
· Usability specialists – 3 to 5 persons (could detect 74% to 87% of usability problems.
· Novice evaluators – 14 persons (could detect more than 75% of usability problems. By using 5 novice evaluators, 51% of the usability problems could be identified. (Nielsen, 1992a; Nielsen & Molich, 1990a)
---------------
Evaluating Web Page and Web Site Usability.
Christopher C. Whitehead. Columbus State University, 4225 University Avenue, Columbus, GA 31907. 1-706-565-3527 whitehead_christopher@colstate.edu
ACM SE’06, March, 10-12, 2006, Melbourne, Florida, USA
USABILITY TESTING
Battleson et al. claim that usability testing is the most effective way to asses a site's usability [1]. They go on to specify three categories of usability testing: inquiry, inspection, and formal usability testing.
Inquiry usability testing requests information from the user and includes the use of focus groups, interviews, questionnaires, and surveys.
Inspection usability testing includes heuristic evaluation and cognitive walkthroughs.
According to Dickstein and Mills, “Heuristic evaluation is a systematic inspection of a user interface to examine if the design is in compliance with recognized usability” [2]. This typically involves checking the site “against a checklist of heuristics, or design principles” [1].
Cognitive walkthroughs involve enlisting the use of individuals “to accomplish typical user tasks with a given interface” [1].
Under formal usability testing, a formal set of tasks and goals are defined and users are observed while performing these tasks. According to Battleson et al., formal usability testing involves the following: “(1) the goal is to improve the usability of the interface; (2) testers represent real users; (3) testers perform real tasks; (4) user behavior and commentary are observed and recorded; and (5) data are analyzed to recognize problems and suggest solutions” [1].
Ivory et al. also report that simulation has been used to effectively evaluate web sites by automating the implementation of usability testing [3]. Using this methodology, user behavior is mimicked in a controlled environment or through computer simulation without actual users.
Determining the Number of Users to Involve in the TestingIn performing usability testing, Battleson et al. claim that in a homogenous user testing group, as little as five users are needed to ascertain a large percentage of the most critical errors [1].
In comparison, Dickstein and Mills found that eight to twelve users was sufficient enough to determine problems with a Web site’s usability [2]. However, they also claim that by the fifth user, they were able to determine whether the tested feature was either a problem or a success.
USABILITY METRICS
Usability metrics are measures of a particular aspect of a Web page or Web site that has an impact on usability.
In performing quantitative usability testing, Ivory, et al. suggest the following Web page metrics as the most important in evaluating usability [3]:
• Word Count--Total words on a page
• Body Text %--Percentage of words that are body vs. display text (i.e., headers)
• Emphasized Body Text %--Portion of body text that is emphasized (e.g., bold, capitalized or near !'s)
• Text Positioning Count--Changes in text position from flush left
• Text Cluster Count --Text areas highlighted with color, bordered regions, rules or lists
• Link Count--Total links on a page
• Page Size--Total bytes for the page as well as elements graphics and stylesheets
• Graphic %--Percentage of page bytes that are for graphics
• Graphics Count--Total graphics on a page (not including graphics specified in scripts, applets and objects)
• Color Count--Total colors employed
• Font Count--Total fonts employed (i.e., face + size + bold + italic)
Ivory et al. found six metrics to be of most importance: text cluster count, link count, page size, graphics count, color count, and reading complexity [3].
-----------------
Evaluating Web Usability Using Small Display Devices.
Carlos J. Costa. DCTI – ISCTE, Adetti/ISCTE, Lisboa, Portugal. carlos.costa@iscte.pt
José P. Novais Silva. Portugal Telecom, Lisboa, Portugal. jose.p.silva@telecom.pt
Manuela Aparício. ITML, Lisboa, Portugal. manuela@design.itml.org
SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.
Usability testing is an usual tool used to evaluate the usability of a mobile application in a development process. In fact, mobile and handheld usability testing could be even more important than computer-based usability testing.
Webcredible [23] explain the main reasons for this:
• Increase of small display devices sales.
• Less experience of using small display devices to navigate on the web than using desktop computers.
How usability can be tested?
Lee and Grice [15] propose a usability testing that combines heuristics, questionnaires, and scenarios for developing mobile applications.
According to Andrews [1] there are three usability inspection methods: Heuristic Evaluation, Cognitive Walkthrough and Action Analysis. In Heuristic Evaluation a small set of evaluators examines interface and judges its compliance with recognized usability principles, while in Cognitive Walkthrough, exist a taskoriented walkthrough based on formal cognitive model of user behaviour (analyses learnability).
On the other and, in Action Analysis is made an quantitative analysis of actions to predict time required for tasks, based on time estimates for typical interface actions (analyses efficiency).
---------------
Conceptual Framework and Models for Identifying and Organizing Usability Impact Factors of Mobile Phones.
Dong-Han Ham1, Jeongyun Heo2, Peter Fossick3, William Wong1, Sanghyun Park2, Chiwon Song2, Mike Bradley3
1School of Computing Science, Middlesex University, The Burroughs London, NW4 4BT UK. {d.ham, w.wong}@mdx.ac.uk
2MC R&D Centre, LG Electronics, Seoul Korea. {jy_heo, sanghyun, chiwon79}@lge.com
3Product Design Engineering, Middlesex University, Bramley Road London, N14 4YZ UK. {p.fossick, m.d.bradley}@mdx.ac.uk
OZCHI 2006, November 20-24, 2006, Sydney, Australia.
Usability can be measured in various ways; however, they can be categorized in three methods: usability testing, usability inquiry, and usability inspection (Zhang, 2003).
It cannot be said that one method is the best in all situations. Hence it is necessary to choose an appropriate method, taking into consideration evaluation purposes, available time, measures to be collected and so on.
---------------
Methods for quantitative usability requirements: a case study on the development of the user interface of a mobile phone.
Timo Jokela Æ Jussi Koivumaa Æ Jani Pirkola
Petri Salminen Æ Niina Kantola
Pers Ubiquit Comput (2006) 10: 345–355DOI 10.1007/s00779-005-0050-7
T. Jokela (&) Æ N. Kantola. Oulu University, P.O. Box 3000, Oulu, Finland. E-mail: timo.jokela@oulu.fi E-mail: niina.kantola@oulu.fi
J. Koivumaa Æ J. Pirkola. Nokia, P.O. Box 50, 90571 Oulu, Finland. E-mail: jussi.koivumaa@nokia.com E-mail: jani.pirkola@nokia.com
P. Salminen. ValueFirst, Luuvantie 28, 02620 Espoo, Finland. E-mail: petri.salminen@valuefirst.fi
Methods for quantitative evaluation of usability
Whether the quantitative requirements have been met can be determined through a usability test.
Wixon et al. [14] define the term ‘‘test’’ as a broad term that encompasses any method for assessing whether goals have been achieved, like a formal laboratory test or a collection of satisfaction data through survey.
When evaluating usability, ISO 9241-11 [19] claims it is important that the context selected be representative. Evaluations can be done in the field in a real work situation or in laboratory settings in which the relevant aspects of the context of use are re-created in a representative and controlled way. A method that includes representative users performing typical, representative tasks is generally called usability testing.
Tasks that are done in usability testing provide an objective metric for the related usability attribute. Hix and Hartson [6] indicate that tasks must be very specifically worded in order to be the same for each participant. Tasks must also be specific, so that participants do not get sidetracked into irrelevant details during testing.
Wixon et al. [14] suggest that during the test, the tester should minimize the interaction with participants. Butler [33] describes his approach where ‘‘seven test users were given an introductory level problem to solve, then left alone with a copy of the user’s guide and a 3270 terminal logged onto the system.’’
User preference questionnaires provide a subjective metric for the related usability attribute such as ease of use or usefulness. Questionnaires are commonly built using Likert and semantic differential scales and are intended for use in various circumstances [32].
There are a number of questionnaires available for quantitative usability evaluation, like SUMI [22], QUIS [24] and SUS [23]. Karat [34] states that questionnaires provide an easy and inexpensive method for obtaining measurement data on a system.
Usability can be quantitatively evaluated also with theory-based approaches such as GOMS and keystroke level model, KLM [35].
--------------
20090906-2025
Monday, August 31, 2009
Aug 31 - Hollingsed & Novick, Usability Inspection Methods after 15 Years of Research and Practice (part 2)
Usability Inspection Methods after 15 Years of Research and Practice.
Tasha Hollingsed. Lockheed Martin, 2540 North Telshor Blvd., Suite C, Las Cruces, NM 88011. +1 505-525-5267 tasha.hollingsed@lmco.com
David G. Novick. Department of Computer Science, The University of Texas at El Paso, El Paso, TX 79968-0518. +1 915-747-5725 novick@utep.edu
SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.
COGNITIVE WALKTHROUGH
The cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use [53].
Like other inspection methods, a cognitive walkthrough can be performed on an interface at any time during the development process, from the original mock-ups through the final release.
The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.
During the analysis phase, the evaluators work through the four steps of humancomputer interaction developed by Lewis and Polson [25]:
1. The user sets a goal to be completed within the system.
2. The user determines the currently available actions.
3. The user selects the action that they think will take them closer to their goal.
4. The user performs the action and evaluates the feedback given by the system.
Wharton et al. [53] surveyed the history of the cognitive walkthrough prior to 1994. At the time, the two main limitations of the cognitive walkthrough were the repetitiveness of filling out the forms and the limited range of problems the process found ([53], [51], [19]). A new method of cognitive walkthrough addressed these limitations by using small groups instead of individual evaluators and rotating the form filling within the group, evaluating the simple tasks first, and by keeping a record of all problems identified during the evaluation, not just those identified during the process ([53], [41]).
Other criticisms of the cognitive walkthrough were that it does not provide guidelines about what makes an action clearly available to a user and that it is not known what types of actions are considered by a broad range of users [51]).
Since its inception, and with its refinements and extensions, the cognitive walkthrough has been shown to be an effective inspection method that can be applied not just by cognitive scientists and usability specialists but also by novice evaluators. However, the choice of task scenario can be difficult; if the scenario is not adequately described, the evaluation is not as effective.
PLURALISTIC USABILITY WALKTHROUGH
The pluralistic usability walkthrough [3] adapted the traditional usability walkthrough to incorporate representative users, product developers, members of the product team, and usability experts in the process. It is defined by five characteristics:
1. Inclusion of representative users, product developers, and human factors professionals;
2. The application’s screens are presented in the same order as they would appear to the user;
3. All participants are asked to assume the role of the user;
4. Participants write down what actions they, as users, would take for each screen before the group discusses the screens; and
5. When discussing each screen, the representative users speak first.
...benefits and limitations of the approach.
On the positive side, this approach offers feedback from users even if the interface is not fully developed, enables rapid iteration of the design cycle, and—because the users are directly involved—can result in “on-the-fly” redesign.
On the negative side, the approach must be limited to representative rather than comprehensive user paths through the interface, and users who, at a particular step, did not choose the path the group will follow must “reset” their interaction with the interface.
My comments: I think pluralistic walkthrough has another strength of using a cross-functional team to evaluate usability.
The pluralistic walkthrough appears to be in active use for assessing usability. It is included in the Usability Professionals Association draft body of knowledge [49].
Available reports indicate that pluralistic usability walkthrough is used in industry. While some human factors experts continue to conduct usability walkthroughs that do not combine stakeholder perspectives (e.g., [7], [50], [44]), it seems likely that use of the pluralistic usability
walkthrough is widespread but that teams do not refer to it as such in published reports.
FORMAL USABILITY INSPECTIONS
Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance [22].
Like the pluralistic usability walkthrough, this involves stepping through the user’s task. However, because the reviewers consist of human factors experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the interface as efficiently as possible. The review process includes task performance models and heuristics, a variety of human-factors expertise, and defect detection within the framework of the software development lifecycle.
Like the cognitive walkthrough, formal usability inspections require definitions of user profiles and task scenarios. And, like the cognitive walkthrough, the reviewers use a cognitive model of task performance, which can be extended with a checklist of cognitive steps similar to those invoked by Norman [36] to bridge the “gulf of execution.”
Hewlett Packard used this method for at least two years before 1995. The inspection team included design engineers, usability engineers, customer support engineers, and customers at times. The team inspected fourteen products and found an average of 76 usability concerns per product and an average of 74 percent of those concerns were fixed per product. While no formal evaluation of the results was done, it was found that the engineers could detect several of the usability concerns, and the engineers enjoyed using the method while increasing their awareness of user needs [15].
Digital Equipment Corporation also conducted a version of formal usability inspections from 1994 to 1995 on ten products. They found an average of 66 usability problems per product and fixed an average of 52 problems per product. Finding even small usability problems proved to be an asset, especially when a number of these problems were easily fixed. As more problems were fixed, the perceived quality of the product improved as well, even if most of these fixes were small [45].
Since then, it appears that little research has been conducted on formal usability inspections. This approach now tends to be grouped into the overall inspection method class and gets overshadowed by the better known heuristic evaluation when comparisons between
inspection and empirical methods have been conducted. As a method, formal usability inspection gains speed at the cost of losing the multiple stakeholder perspectives of the pluralistic walkthrough, and its cognitive model can be seen as less comprehensive than that of the cognitive walkthrough.
CONCLUSION
Both empirical usability testing and usability inspection methods appear to be in wide use, with developers choosing the most appropriate method for their purposes and their context.
For example, half of the ten intranets winning a 2005 competition used heuristic evaluation [34]. The same report indicated that empirical usability testing was used by 80 percent of the winning intranets.
The cognitive walkthrough appears to be in continued use, although reports of use are not as frequent.
The pluralistic usability walkthrough remains in the repertoire of usability experts, although
usability experts continue to conduct user-only usability walkthroughs.
And formal usability inspection, although shown to be an effective approach for identifying usability problems, appears to be used less now than in the mid-1990s.
Many have claimed that usability inspection methods make for faster and more cost-efficient evaluation of the usability of an interface than empirical user testing.
But while usability inspection methods do identify a number of usability problems faster and more cost-efficiently, the best performing evaluator and method still only found 44 percent of the usability problems found in a laboratory setting [9].
While the cognitive walkthrough is useful for predicting problems on a given task and heuristic evaluation is useful for predicting problems on the interface, empirical testing provides lots of information throughout the interface and is the benchmark against which all other methods are measured [9].
Indeed, Jeffries et al. [19] noted that evaluators of usability methods may have rated problems found though empirical usability testing as, on average, more severe precisely because the problems were identified empirically rather than analytically. While inspection methods need expert evaluators to be effective, their strengths are that they can be implemented into the early stages of the development cycle and provide a forum in which changes to interface can be discussed.
The research on comparisons of usability assessment methods suggests several lessons for practitioners.
First, while “faster, cheaper” methods such as heuristic evaluation and the pluralistic usability walkthrough can be useful for rapid iteration early in the design cycle, inspection methods cannot fully substitute for the empirical user testing needed before releasing an interface or Web site to the public.
Second, empirical methods can also be used early in the development process, via “low-tech” versions of interfaces.
Third, developers often combine multiple inspection methods— heuristic evaluation and the cognitive walkthrough—in the same project so that they obtain better coverage of usability issues.
And fourth, adding multiple perspectives—along dimensions such as the range of stakeholders or kinds of usability problems—appears to improve the effectiveness of inspection methods.
It remains an open issue as to why usability professionals, in practice, rely on single-perspective methods, typically involving users, or experts, but not both. The evidence from reports of recent
uses of heuristic evaluation suggests that many usability specialists are missing the benefits of the pluralistic walkthrough and perspective-based evaluation.
At a deeper level, though, a new direction for research should complement these defect-tracking and learning approaches by seeking to understand the root causes of usability problems. The ideal solution would be to know the reasons for usability problems, so that designers can minimize the effort spent on usability inspection and testing.
My Comments: For my PhD research, shall I choose only 1 UEM (usability evaluation method) or shall I use a hybrid/combination of UEM?
References that I may want to read further in future:
[2] Andreasen, M. S., Nielsen, H., Schrøder, S., and Stage, J. 2007. What happened to remote usability testing?: An empirical study of three methods, Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 2007), San Jose, CA, April 28-May 3, 2007, 1405-1414.
[3] Bias, R. G. 1994. The pluralistic usability walkthrough: coordinated empathies. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 63-76.
[4] Blackmon, M., Polson, P., Kitajima, M., and Lewis, C. (2002). Cognitive walkthrough for the Web, Proceedings of the Conference on Human Factors in Computing Systems (CHI 2002), Minneapolis, MN, April 20-25, 2002, 463-470.
[6] Bradford, J. (1994). Evaluating high-level design: synergistic use of inspection and usability methods for evaluating early software designs. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 235-253.
[8] Brooks, P. (1994). Adding value to usability testing. . In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 255-271.
[9] Desurvire, H. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing? In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 173-202.
[16] Gunn, C. (1995). An example of formal usability inspections in practice at Hewlett-Packard company, Proceeding of the Conference on Human Factors in Computing System (CHI 95), Denver, Colorado, May 7-11, 1995, 103-104.
[17] Hovater, J., Krot, M., Kiskis, D. L., Holland, H., & Altman, M. (2002). Usability testing of the Virtual Data Center, Workshop on Usability of Digital Libraries, Second ACM- IEEE Joint Conference on Digital Libraries, Portland, OR, July 14-18, 2002, available at http://www.uclic.ucl.ac.uk/annb/DLUsability/Hovater7.pdf, accessed May 26, 2007.
[18] Jeffries, R. (1994). Usability problem reports: helping evaluators communicate effectively with developers. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 273-294.
[19] Jeffries, R., Miller, J., Wharton, C., and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques, Proceeding of the Conference on Human Factors in Computing System (CHI 91), New Orleans, LA, April 27-May 2, 1991, 119-124.
[20] Jeffries, R., and Desurvire, H. (1992). Usability testing vs. heuristic evaluation: was there a contest? ACM SIGCHI Bulletin 24(4), 39-41.
[21] John B., and Packer, H (1995). Learning and using the cognitive walkthrough method: A case study approach, Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 95), Denver, Colorado, May 7-11, 1995, 429-436.
[22] Kahn, M., and Prail, A. (1994). Formal usability inspections, in Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 141-171.
[23] Karat, C.-M 1994. A comparison of user interface evaluation methods, in Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 203-233.
[24] Karat, C.-M., Campbell, R. and Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the Conference in Human Factors in Computing Systems (CHI 92), Monterey, CA, May 3-7, 1992, 397-404.
[25] Lewis, C., and Polson, P. (1991). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces (tutorial), Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 91), April 27-May 2, 1991, New Orleans, LA.
[26] Mack, R. and Montaniz, F. (1994). Observing, predicting, and analyzing usability problems. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 295-339.
[27] Mack, R., and Nielsen, J. (1993). Usability inspection methods: Report on a workshop held at CHI'92, Monterey
[29] Muller, M., Dayton, T., and Root, R. (1993). Comparing studies that compare usability assessment methods: an unsuccessful search for stable criteria, INTERACT '93 and CHI '93 Conference Companion on Human Factors in Computing Systems, Amsterdam, April 24-29, 1993, 185- 186.
[31] Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proceedings of the Conference on Human Factors in Computing Systems (CHI 90), Seattle, WA, April 1-5, 1990, 249-256.
[32] Nielsen, J. (1992). Finding usability problems through heuristic evaluation, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992, 373-380.
[33] Nielsen, J. and Mack, R. (eds.) (1994), Usability inspection methods, John Wiley & Sons, Inc., New York.
[34] Nielsen, J. (2005). Ten best intranets of 2005, Jakob Nielsen's Alertbox, February 28, 2005, available at http://www.useit.com/alertbox/20050228.html, accessed May
26, 2007.
[35] Nielsen, J., and Phillips, V, (1993). Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared, Proceedings of the Conference on Human Factors in Computing System (CHI 93), Amsterdam, April 24-29, 1993, 214-221.
[37] Novick, D. and Chater, M. (1999). Evaluating the design of human-machine cooperation: The cognitive walkthrough for operating procedures, Proceedings of the Conference on
Cognitive Science Approaches to Process Control (CSAPC 99), Villeneuve d'Ascq, FR, September 21-24, 1999.
[38] Novick, D. (2000). Testing documentation with “low-tech” simulation, Proceedings of IPCC/SIGDOC 2000, Cambridge, MA, September 24-27, 2000, 55-68.
[41] Rieman, J., Franzke, M., and Redmiles, D. (1995). Usability evaluation with the cognitive walkthrough, Proceedings of the Conference on Human Factors in Computing Systems (CHI 95), Denver, Colorado, May 7-11, 1995, 387-388.
[46] Spencer, R. (2000). The streamlined cognitive walkthrough method, Proceedings of the Conference on Human Factors in Computing System (CHI 2000), The Hague, The Netherlands, April 1-6, 2000, 353-359.
[47] Tang, Z., Johnson, T., Tindall, R., and Zhang, J. (2006) Applying heuristic evaluation to improve the usability of a telemedicine system, Telemedecine Journal and E-Health
12(1), 24-34.
[49] Usability Professionals Association (undated). Methods: Pluralistic usability walkthrough, available at http://www.usabilitybok.org/methods/p2049, accessed May 26, 2007.
[50] User-Centered Web Effective Business Solutions (2007). Heuristic usability evaluation, available at http://www.ucwebs.com/usability/web-site-usability.htm, accessed May 27, 2007.
[51] Wharton, C., Bradford, J., Jeffries, R., and Franzke, M. (1992). Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992, 381-388.
[52] Wharton, C. and Lewis, C. 1994. The role of psychological theory in usability inspection methods. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 341-350
[53] Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994). The Cognitive Walkthrough Method: A Practitioner’s Guide. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 1994, 105-140.
[55] Zhang, Z., Basili, V., Shneiderman, B. (1998). An empirical study of perspective-based usability inspection, Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, Santa Monica, CA, October 5-9, 1998, 1346-1350.
My Comments: This is a very rich resource of Reference Materials. GREAT!! Extremely good for me to "Literature Review" on the subject of "Usability Evaluation Method." I should get hold to Nielsen and Mack's book on "Usability Inspection Methods."
Tasha Hollingsed. Lockheed Martin, 2540 North Telshor Blvd., Suite C, Las Cruces, NM 88011. +1 505-525-5267 tasha.hollingsed@lmco.com
David G. Novick. Department of Computer Science, The University of Texas at El Paso, El Paso, TX 79968-0518. +1 915-747-5725 novick@utep.edu
SIGDOC’07, October 22–24, 2007, El Paso, Texas, USA.
COGNITIVE WALKTHROUGH
The cognitive walkthrough is a usability inspection method that evaluates the design of a user interface for its ease of exploratory learning, based on a cognitive model of learning and use [53].
Like other inspection methods, a cognitive walkthrough can be performed on an interface at any time during the development process, from the original mock-ups through the final release.
The process of the cognitive walkthrough comprises a preparatory phase and an analysis phase. During the preparatory phase, the experimenters determine the interface to be used, its likely users, the task, and the actions to be taken during the task.
During the analysis phase, the evaluators work through the four steps of humancomputer interaction developed by Lewis and Polson [25]:
1. The user sets a goal to be completed within the system.
2. The user determines the currently available actions.
3. The user selects the action that they think will take them closer to their goal.
4. The user performs the action and evaluates the feedback given by the system.
Wharton et al. [53] surveyed the history of the cognitive walkthrough prior to 1994. At the time, the two main limitations of the cognitive walkthrough were the repetitiveness of filling out the forms and the limited range of problems the process found ([53], [51], [19]). A new method of cognitive walkthrough addressed these limitations by using small groups instead of individual evaluators and rotating the form filling within the group, evaluating the simple tasks first, and by keeping a record of all problems identified during the evaluation, not just those identified during the process ([53], [41]).
Other criticisms of the cognitive walkthrough were that it does not provide guidelines about what makes an action clearly available to a user and that it is not known what types of actions are considered by a broad range of users [51]).
Since its inception, and with its refinements and extensions, the cognitive walkthrough has been shown to be an effective inspection method that can be applied not just by cognitive scientists and usability specialists but also by novice evaluators. However, the choice of task scenario can be difficult; if the scenario is not adequately described, the evaluation is not as effective.
PLURALISTIC USABILITY WALKTHROUGH
The pluralistic usability walkthrough [3] adapted the traditional usability walkthrough to incorporate representative users, product developers, members of the product team, and usability experts in the process. It is defined by five characteristics:
1. Inclusion of representative users, product developers, and human factors professionals;
2. The application’s screens are presented in the same order as they would appear to the user;
3. All participants are asked to assume the role of the user;
4. Participants write down what actions they, as users, would take for each screen before the group discusses the screens; and
5. When discussing each screen, the representative users speak first.
...benefits and limitations of the approach.
On the positive side, this approach offers feedback from users even if the interface is not fully developed, enables rapid iteration of the design cycle, and—because the users are directly involved—can result in “on-the-fly” redesign.
On the negative side, the approach must be limited to representative rather than comprehensive user paths through the interface, and users who, at a particular step, did not choose the path the group will follow must “reset” their interaction with the interface.
My comments: I think pluralistic walkthrough has another strength of using a cross-functional team to evaluate usability.
The pluralistic walkthrough appears to be in active use for assessing usability. It is included in the Usability Professionals Association draft body of knowledge [49].
Available reports indicate that pluralistic usability walkthrough is used in industry. While some human factors experts continue to conduct usability walkthroughs that do not combine stakeholder perspectives (e.g., [7], [50], [44]), it seems likely that use of the pluralistic usability
walkthrough is widespread but that teams do not refer to it as such in published reports.
FORMAL USABILITY INSPECTIONS
Formal usability inspection is a review by the interface designer and his or her peers of users’ potential task performance [22].
Like the pluralistic usability walkthrough, this involves stepping through the user’s task. However, because the reviewers consist of human factors experts, the review can be quicker, more thorough, and more technical than in the pluralistic walkthrough. The goal is to identify the maximum number of defects in the interface as efficiently as possible. The review process includes task performance models and heuristics, a variety of human-factors expertise, and defect detection within the framework of the software development lifecycle.
Like the cognitive walkthrough, formal usability inspections require definitions of user profiles and task scenarios. And, like the cognitive walkthrough, the reviewers use a cognitive model of task performance, which can be extended with a checklist of cognitive steps similar to those invoked by Norman [36] to bridge the “gulf of execution.”
Hewlett Packard used this method for at least two years before 1995. The inspection team included design engineers, usability engineers, customer support engineers, and customers at times. The team inspected fourteen products and found an average of 76 usability concerns per product and an average of 74 percent of those concerns were fixed per product. While no formal evaluation of the results was done, it was found that the engineers could detect several of the usability concerns, and the engineers enjoyed using the method while increasing their awareness of user needs [15].
Digital Equipment Corporation also conducted a version of formal usability inspections from 1994 to 1995 on ten products. They found an average of 66 usability problems per product and fixed an average of 52 problems per product. Finding even small usability problems proved to be an asset, especially when a number of these problems were easily fixed. As more problems were fixed, the perceived quality of the product improved as well, even if most of these fixes were small [45].
Since then, it appears that little research has been conducted on formal usability inspections. This approach now tends to be grouped into the overall inspection method class and gets overshadowed by the better known heuristic evaluation when comparisons between
inspection and empirical methods have been conducted. As a method, formal usability inspection gains speed at the cost of losing the multiple stakeholder perspectives of the pluralistic walkthrough, and its cognitive model can be seen as less comprehensive than that of the cognitive walkthrough.
CONCLUSION
Both empirical usability testing and usability inspection methods appear to be in wide use, with developers choosing the most appropriate method for their purposes and their context.
For example, half of the ten intranets winning a 2005 competition used heuristic evaluation [34]. The same report indicated that empirical usability testing was used by 80 percent of the winning intranets.
The cognitive walkthrough appears to be in continued use, although reports of use are not as frequent.
The pluralistic usability walkthrough remains in the repertoire of usability experts, although
usability experts continue to conduct user-only usability walkthroughs.
And formal usability inspection, although shown to be an effective approach for identifying usability problems, appears to be used less now than in the mid-1990s.
Many have claimed that usability inspection methods make for faster and more cost-efficient evaluation of the usability of an interface than empirical user testing.
But while usability inspection methods do identify a number of usability problems faster and more cost-efficiently, the best performing evaluator and method still only found 44 percent of the usability problems found in a laboratory setting [9].
While the cognitive walkthrough is useful for predicting problems on a given task and heuristic evaluation is useful for predicting problems on the interface, empirical testing provides lots of information throughout the interface and is the benchmark against which all other methods are measured [9].
Indeed, Jeffries et al. [19] noted that evaluators of usability methods may have rated problems found though empirical usability testing as, on average, more severe precisely because the problems were identified empirically rather than analytically. While inspection methods need expert evaluators to be effective, their strengths are that they can be implemented into the early stages of the development cycle and provide a forum in which changes to interface can be discussed.
The research on comparisons of usability assessment methods suggests several lessons for practitioners.
First, while “faster, cheaper” methods such as heuristic evaluation and the pluralistic usability walkthrough can be useful for rapid iteration early in the design cycle, inspection methods cannot fully substitute for the empirical user testing needed before releasing an interface or Web site to the public.
Second, empirical methods can also be used early in the development process, via “low-tech” versions of interfaces.
Third, developers often combine multiple inspection methods— heuristic evaluation and the cognitive walkthrough—in the same project so that they obtain better coverage of usability issues.
And fourth, adding multiple perspectives—along dimensions such as the range of stakeholders or kinds of usability problems—appears to improve the effectiveness of inspection methods.
It remains an open issue as to why usability professionals, in practice, rely on single-perspective methods, typically involving users, or experts, but not both. The evidence from reports of recent
uses of heuristic evaluation suggests that many usability specialists are missing the benefits of the pluralistic walkthrough and perspective-based evaluation.
At a deeper level, though, a new direction for research should complement these defect-tracking and learning approaches by seeking to understand the root causes of usability problems. The ideal solution would be to know the reasons for usability problems, so that designers can minimize the effort spent on usability inspection and testing.
My Comments: For my PhD research, shall I choose only 1 UEM (usability evaluation method) or shall I use a hybrid/combination of UEM?
References that I may want to read further in future:
[2] Andreasen, M. S., Nielsen, H., Schrøder, S., and Stage, J. 2007. What happened to remote usability testing?: An empirical study of three methods, Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 2007), San Jose, CA, April 28-May 3, 2007, 1405-1414.
[3] Bias, R. G. 1994. The pluralistic usability walkthrough: coordinated empathies. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 63-76.
[4] Blackmon, M., Polson, P., Kitajima, M., and Lewis, C. (2002). Cognitive walkthrough for the Web, Proceedings of the Conference on Human Factors in Computing Systems (CHI 2002), Minneapolis, MN, April 20-25, 2002, 463-470.
[6] Bradford, J. (1994). Evaluating high-level design: synergistic use of inspection and usability methods for evaluating early software designs. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 235-253.
[8] Brooks, P. (1994). Adding value to usability testing. . In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 255-271.
[9] Desurvire, H. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing? In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 173-202.
[16] Gunn, C. (1995). An example of formal usability inspections in practice at Hewlett-Packard company, Proceeding of the Conference on Human Factors in Computing System (CHI 95), Denver, Colorado, May 7-11, 1995, 103-104.
[17] Hovater, J., Krot, M., Kiskis, D. L., Holland, H., & Altman, M. (2002). Usability testing of the Virtual Data Center, Workshop on Usability of Digital Libraries, Second ACM- IEEE Joint Conference on Digital Libraries, Portland, OR, July 14-18, 2002, available at http://www.uclic.ucl.ac.uk/annb/DLUsability/Hovater7.pdf, accessed May 26, 2007.
[18] Jeffries, R. (1994). Usability problem reports: helping evaluators communicate effectively with developers. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 273-294.
[19] Jeffries, R., Miller, J., Wharton, C., and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques, Proceeding of the Conference on Human Factors in Computing System (CHI 91), New Orleans, LA, April 27-May 2, 1991, 119-124.
[20] Jeffries, R., and Desurvire, H. (1992). Usability testing vs. heuristic evaluation: was there a contest? ACM SIGCHI Bulletin 24(4), 39-41.
[21] John B., and Packer, H (1995). Learning and using the cognitive walkthrough method: A case study approach, Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 95), Denver, Colorado, May 7-11, 1995, 429-436.
[22] Kahn, M., and Prail, A. (1994). Formal usability inspections, in Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 141-171.
[23] Karat, C.-M 1994. A comparison of user interface evaluation methods, in Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 203-233.
[24] Karat, C.-M., Campbell, R. and Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the Conference in Human Factors in Computing Systems (CHI 92), Monterey, CA, May 3-7, 1992, 397-404.
[25] Lewis, C., and Polson, P. (1991). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces (tutorial), Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI 91), April 27-May 2, 1991, New Orleans, LA.
[26] Mack, R. and Montaniz, F. (1994). Observing, predicting, and analyzing usability problems. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 295-339.
[27] Mack, R., and Nielsen, J. (1993). Usability inspection methods: Report on a workshop held at CHI'92, Monterey
[29] Muller, M., Dayton, T., and Root, R. (1993). Comparing studies that compare usability assessment methods: an unsuccessful search for stable criteria, INTERACT '93 and CHI '93 Conference Companion on Human Factors in Computing Systems, Amsterdam, April 24-29, 1993, 185- 186.
[31] Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proceedings of the Conference on Human Factors in Computing Systems (CHI 90), Seattle, WA, April 1-5, 1990, 249-256.
[32] Nielsen, J. (1992). Finding usability problems through heuristic evaluation, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992, 373-380.
[33] Nielsen, J. and Mack, R. (eds.) (1994), Usability inspection methods, John Wiley & Sons, Inc., New York.
[34] Nielsen, J. (2005). Ten best intranets of 2005, Jakob Nielsen's Alertbox, February 28, 2005, available at http://www.useit.com/alertbox/20050228.html, accessed May
26, 2007.
[35] Nielsen, J., and Phillips, V, (1993). Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared, Proceedings of the Conference on Human Factors in Computing System (CHI 93), Amsterdam, April 24-29, 1993, 214-221.
[37] Novick, D. and Chater, M. (1999). Evaluating the design of human-machine cooperation: The cognitive walkthrough for operating procedures, Proceedings of the Conference on
Cognitive Science Approaches to Process Control (CSAPC 99), Villeneuve d'Ascq, FR, September 21-24, 1999.
[38] Novick, D. (2000). Testing documentation with “low-tech” simulation, Proceedings of IPCC/SIGDOC 2000, Cambridge, MA, September 24-27, 2000, 55-68.
[41] Rieman, J., Franzke, M., and Redmiles, D. (1995). Usability evaluation with the cognitive walkthrough, Proceedings of the Conference on Human Factors in Computing Systems (CHI 95), Denver, Colorado, May 7-11, 1995, 387-388.
[46] Spencer, R. (2000). The streamlined cognitive walkthrough method, Proceedings of the Conference on Human Factors in Computing System (CHI 2000), The Hague, The Netherlands, April 1-6, 2000, 353-359.
[47] Tang, Z., Johnson, T., Tindall, R., and Zhang, J. (2006) Applying heuristic evaluation to improve the usability of a telemedicine system, Telemedecine Journal and E-Health
12(1), 24-34.
[49] Usability Professionals Association (undated). Methods: Pluralistic usability walkthrough, available at http://www.usabilitybok.org/methods/p2049, accessed May 26, 2007.
[50] User-Centered Web Effective Business Solutions (2007). Heuristic usability evaluation, available at http://www.ucwebs.com/usability/web-site-usability.htm, accessed May 27, 2007.
[51] Wharton, C., Bradford, J., Jeffries, R., and Franzke, M. (1992). Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations, Proceedings of the Conference on Human Factors in Computing System (CHI 92), Monterey, CA, May 3-7, 1992, 381-388.
[52] Wharton, C. and Lewis, C. 1994. The role of psychological theory in usability inspection methods. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 341-350
[53] Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994). The Cognitive Walkthrough Method: A Practitioner’s Guide. In Nielsen, J. and Mack, R. (eds.), Usability inspection methods, John Wiley & Sons, Inc., New York, 1994, 105-140.
[55] Zhang, Z., Basili, V., Shneiderman, B. (1998). An empirical study of perspective-based usability inspection, Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, Santa Monica, CA, October 5-9, 1998, 1346-1350.
My Comments: This is a very rich resource of Reference Materials. GREAT!! Extremely good for me to "Literature Review" on the subject of "Usability Evaluation Method." I should get hold to Nielsen and Mack's book on "Usability Inspection Methods."
Subscribe to:
Posts (Atom)
