Advances in heuristic usability evaluation method by Ling, Chen, Ph.D., Purdue University, 2005 , 220 pages; AAT 3210740 My Interest: 1) Heuristic evaluation. 2) Conceptual model with 3 parts. 3) Thoroughness and Validity. Action: To read again the Dissertation in future. Heuristic evaluation method is one of the most used usability evaluation method in both industry and academia (Rosenbaum, 2000). Methodology Based on the characteristics of heuristic evaluation method and E-Commerce websites, a conceptual model with three parts which address the evaluator's cognitive style, heuristic evaluation process, and the heuristics' impacts respectively is proposed. The first part examines effect of evaluator's cognitive style on the evaluation results.
The second part incorporates the Taguchi quality control method into the heuristic evaluation process to find the optimal combination of factors including task type, heuristic set, and evaluation mode which is least sensitive to variations in evaluator's cognitive style.
The third part relates website usability heuristics with user's purchase intention on Ecommerce websites. Three experiments and a survey were conducted to test the proposed four hypothesis associated with testing the validity of the proposed conceptual model. Results Discussion The results of the three experiments and survey study indicated the following: (1) The optimized evaluation process which combines the Taguchi method with the traditional heuristic evolution is to have evaluators carry out evaluation in pairs with the help of domain specific heuristic set. (2) Using the developed optimized evaluation process significantly improves the evaluation effectiveness by over 17.6% and reduced the variance among the evaluation effectiveness by 3.5 times in relation to using the traditional heuristic evaluation method. (3) Field independent (FI) evaluators find problem sets with higher thoroughness and validity than field dependent (FD) evaluators. (4) FI evaluators use more analytical approaches than FD evaluators during heuristic evaluation. (5) Using the newly developed E-Commerce heuristic set results in finding larger number of real usability problems than using Nielsen's (1994d) heuristic set. (6) Users' purchase intention on E-Commerce website is affected, in descending importance by the following five factors: prevention from error, information access quality, shopping support, ease of comprehension, and hedonic quality. Comments: Double evaluator was used.
|
Showing posts with label Ling. Show all posts
Showing posts with label Ling. Show all posts
Wednesday, September 22, 2010
20100922 - Ling, Advances in Heuristic Evaluation Method
Saturday, October 31, 2009
Oct 31 - Ling, Advances in Heuristic Usability Evaluation Method (PhD dissertation)
Oct 31 – Ling's PhD dissertation
Heuristic evaluation is one of the most popularly used usability evaluation method in
both industry and academia. (Ling, 2005; Rosenbaum, 2000)
In a major survey conducted by Rosenbaum, Rohn and Hamburg (2000), heuristic evaluation was noted as the most used usability evaluation method.
Using the newly developed E-Commerce heurist set (of usability criteria) resulted in finding larger number of real usability problems than using Nielsen’s (1994d) heuristic set. (Ling 2005)
Sweeney, Maguire and Shackel (1993) used the concept of “approach to evaluation” to classify usability methods: user-based, theory-based and expert-based. Their framework reflects different data sources that form the basis of evaluation.
Mack and Nielsen (1004) stated 4 ways to evaluate user interfaces.
Automatically
Empirically
Formally
Informally.
The informal evaluation methods are also referred to as usability inspection methods.
Ling (2005) had categorized usability evaluation methods into seven.
1) Analytic theory-based methods – include: (a) cognitive task analysis; (b) goal, operator, method and selection (GOMS) models; user action notations (UAN).
GOMS to estimate the expert’s task performance time.
2) Expert evaluation methods – include formal usability inspections, cognitive walkthroughs, pluralistic walkthroughs, guideline reviews, heuristic evaluation, claim analysis.
Guideline methods apply a set of established guidelines and rules to target system; differ from hruristic evaluation because they use a larger number of guidelines than the number of heuristics used in heuristic evaluations. Hence, guideline methods require lower level of expertise for the evaluators.
Comments: May be I could combine concept of heuristic evaluation and guidelines review method to form a new method whereby the evaluators are guided by a large set of guidelines for each heuristic.
3) Observational evaluation methods – include direct observations, videos, computer logging, verbal protocols, cooperative observations, critical incident reports and ethnographic studies.
Collect data on what users do when they interact with an interface.
4) Survey evaluation methods – include questionnaires, interviews, focus group and user feedback.
Evaluators to ask users for their subjective views of the user interface of a system.
5) Experimental evaluation methods – include beta testing, think aloud method, constructive interactions, retrospective testing, coaching methods and performance measures.
Involve laboratory experiments to analyze user’s interaction with the system.
Think aloud method asks users to verbalize their thinking throughout the test.
6) Psycho-physiological measures of satisfaction or workload – collect physiological data e.g. electrical activity in brain, heart rate, blood pressure, pupil dilation, skin conductivity, level of adrenaline in blood, mental state of users in term of satisfaction and mental workload.
Predicting user’s workload and satisfaction level.
7) Automatic testing – use programs to capture automatically the critical measures of an interface, e.g. response time, HTML broken links.
Generate a list of low-level interface problems.
Ling (2005) had described thoroughly the Nielsen’s Heuristics and heuristic evaluation method.
Muller et al (1995, 1998) added four more “participatory” heuristics (in addition to Nielsen’s ten heuristics) and validated the new set, which aims to assess how well the interactive system meets the needs of its users and their work environment (Muller et al, 1995; Muller et al, 1998). Also, added users ans inspectors to expert inspectors (traditionally used in heuristic evaluation). Four additional “participatory” heuristics are:
a) Respect the user and his/her skills
b) Promote a pleasurable experience with the system
c) Support quality work
d) Protect privacy.
Ling (2005) had explained about four drawbacks of the heuristic evaluation method.
Ling (2005) explained the factors affecting the results of heuristic evaluation.
a) Individual differences
b) Expertise level
c) Task scenario
d) Observation during evaluation
e) Heuristic set
f) Individual or group evaluations.
Domain Specific Heuristic Set:
Heuristic evaluation method was developed and applied mainly for single user, productivity-oriented desktop programs, which were the major computer applications in the early 1990s.
But with computer technologies getting more integrated into everyday life and new types of HCI emerging, Nielsen’s ten heuristics may not be able to cover usability issues in new computing systems.
For example, mobile systems need to address issues of changing context of use (Vetere et al, 2003).
Because domain-specific heuristics can be developed to supplement existing heuristics (Molich & Nielsen, 1990; Nielsen, 1993; Nielsen & Mack, 1994), researchers have derived many adapted heuristic sets to address the typical requirements and problems in different kinds of application domain.
Ling (2005) discussed how other researchers (including Baker et al, 2001; Baker et al, 2002) came out with the modified/customised heuristics set.
Ling (2005) defined usability problems, severity of usabillity problems, usability problem report.
Ling (2005) focused on usability of e-commerce websites, seemingly.
Questionnaires are the most widely used method to identify an individual's attitudes and feelings toward a software system (Kirakowski & Corbett, 1990).
What have I read:
Chapter 1 Objective and Significance
Chapter 2 Background Literature
Chapter 3 Conceptual Model and Hypotheses
Chapter 9 Conclusions and Recommendations
Ling (2005) concluded:
Based on the experimental results, the following guidelines can be derived on how to perform heuristic evaluation.
* Find evaluator of field independent cognitive style
* Use domain specific heuristic set to guide evaluation
* Have evaluators conduct the evaluation in pairs to reduce variability in results.
Source:
Ling, Chen. Advances in Heuristic Usability Evaluation Method. [Dissertation, PhD] Purdue University, West Lafayette, Indiana. Dec 2005.
Heuristic evaluation is one of the most popularly used usability evaluation method in
both industry and academia. (Ling, 2005; Rosenbaum, 2000)
In a major survey conducted by Rosenbaum, Rohn and Hamburg (2000), heuristic evaluation was noted as the most used usability evaluation method.
Using the newly developed E-Commerce heurist set (of usability criteria) resulted in finding larger number of real usability problems than using Nielsen’s (1994d) heuristic set. (Ling 2005)
Sweeney, Maguire and Shackel (1993) used the concept of “approach to evaluation” to classify usability methods: user-based, theory-based and expert-based. Their framework reflects different data sources that form the basis of evaluation.
Mack and Nielsen (1004) stated 4 ways to evaluate user interfaces.
Automatically
Empirically
Formally
Informally.
The informal evaluation methods are also referred to as usability inspection methods.
Ling (2005) had categorized usability evaluation methods into seven.
1) Analytic theory-based methods – include: (a) cognitive task analysis; (b) goal, operator, method and selection (GOMS) models; user action notations (UAN).
GOMS to estimate the expert’s task performance time.
2) Expert evaluation methods – include formal usability inspections, cognitive walkthroughs, pluralistic walkthroughs, guideline reviews, heuristic evaluation, claim analysis.
Guideline methods apply a set of established guidelines and rules to target system; differ from hruristic evaluation because they use a larger number of guidelines than the number of heuristics used in heuristic evaluations. Hence, guideline methods require lower level of expertise for the evaluators.
Comments: May be I could combine concept of heuristic evaluation and guidelines review method to form a new method whereby the evaluators are guided by a large set of guidelines for each heuristic.
3) Observational evaluation methods – include direct observations, videos, computer logging, verbal protocols, cooperative observations, critical incident reports and ethnographic studies.
Collect data on what users do when they interact with an interface.
4) Survey evaluation methods – include questionnaires, interviews, focus group and user feedback.
Evaluators to ask users for their subjective views of the user interface of a system.
5) Experimental evaluation methods – include beta testing, think aloud method, constructive interactions, retrospective testing, coaching methods and performance measures.
Involve laboratory experiments to analyze user’s interaction with the system.
Think aloud method asks users to verbalize their thinking throughout the test.
6) Psycho-physiological measures of satisfaction or workload – collect physiological data e.g. electrical activity in brain, heart rate, blood pressure, pupil dilation, skin conductivity, level of adrenaline in blood, mental state of users in term of satisfaction and mental workload.
Predicting user’s workload and satisfaction level.
7) Automatic testing – use programs to capture automatically the critical measures of an interface, e.g. response time, HTML broken links.
Generate a list of low-level interface problems.
Ling (2005) had described thoroughly the Nielsen’s Heuristics and heuristic evaluation method.
Muller et al (1995, 1998) added four more “participatory” heuristics (in addition to Nielsen’s ten heuristics) and validated the new set, which aims to assess how well the interactive system meets the needs of its users and their work environment (Muller et al, 1995; Muller et al, 1998). Also, added users ans inspectors to expert inspectors (traditionally used in heuristic evaluation). Four additional “participatory” heuristics are:
a) Respect the user and his/her skills
b) Promote a pleasurable experience with the system
c) Support quality work
d) Protect privacy.
Ling (2005) had explained about four drawbacks of the heuristic evaluation method.
Ling (2005) explained the factors affecting the results of heuristic evaluation.
a) Individual differences
b) Expertise level
c) Task scenario
d) Observation during evaluation
e) Heuristic set
f) Individual or group evaluations.
Domain Specific Heuristic Set:
Heuristic evaluation method was developed and applied mainly for single user, productivity-oriented desktop programs, which were the major computer applications in the early 1990s.
But with computer technologies getting more integrated into everyday life and new types of HCI emerging, Nielsen’s ten heuristics may not be able to cover usability issues in new computing systems.
For example, mobile systems need to address issues of changing context of use (Vetere et al, 2003).
Because domain-specific heuristics can be developed to supplement existing heuristics (Molich & Nielsen, 1990; Nielsen, 1993; Nielsen & Mack, 1994), researchers have derived many adapted heuristic sets to address the typical requirements and problems in different kinds of application domain.
Ling (2005) discussed how other researchers (including Baker et al, 2001; Baker et al, 2002) came out with the modified/customised heuristics set.
Ling (2005) defined usability problems, severity of usabillity problems, usability problem report.
Ling (2005) focused on usability of e-commerce websites, seemingly.
Questionnaires are the most widely used method to identify an individual's attitudes and feelings toward a software system (Kirakowski & Corbett, 1990).
What have I read:
Chapter 1 Objective and Significance
Chapter 2 Background Literature
Chapter 3 Conceptual Model and Hypotheses
Chapter 9 Conclusions and Recommendations
Ling (2005) concluded:
Based on the experimental results, the following guidelines can be derived on how to perform heuristic evaluation.
* Find evaluator of field independent cognitive style
* Use domain specific heuristic set to guide evaluation
* Have evaluators conduct the evaluation in pairs to reduce variability in results.
Source:
Ling, Chen. Advances in Heuristic Usability Evaluation Method. [Dissertation, PhD] Purdue University, West Lafayette, Indiana. Dec 2005.
Subscribe to:
Posts (Atom)
