20100922 - Ling, Advances in Heuristic Evaluation Method

Advances in heuristic usability evaluation method by Ling, Chen, Ph.D., Purdue University, 2005 , 220 pages; AAT 3210740 My Interest: 1) Heuristic evaluation. 2) Conceptual model with 3 parts. 3) Thoroughness and Validity. Action: To read again the Dissertation in future. Heuristic evaluation method is one of the most used usability evaluation method in both industry and academia (Rosenbaum, 2000). Methodology Based on the characteristics of heuristic evaluation method and E-Commerce websites, a conceptual model with three parts which address the evaluator's cognitive style, heuristic evaluation process, and the heuristics' impacts respectively is proposed. The first part examines effect of evaluator's cognitive style on the evaluation results. The second part incorporates the Taguchi quality control method into the heuristic evaluation process to find the optimal combination of factors including task type, heuristic set, and evaluation mode which is least sensitive to variations in evaluator's cognitive style. The third part relates website usability heuristics with user's purchase intention on Ecommerce websites. Three experiments and a survey were conducted to test the proposed four hypothesis associated with testing the validity of the proposed conceptual model. Results Discussion The results of the three experiments and survey study indicated the following: (1) The optimized evaluation process which combines the Taguchi method with the traditional heuristic evolution is to have evaluators carry out evaluation in pairs with the help of domain specific heuristic set. (2) Using the developed optimized evaluation process significantly improves the evaluation effectiveness by over 17.6% and reduced the variance among the evaluation effectiveness by 3.5 times in relation to using the traditional heuristic evaluation method. (3) Field independent (FI) evaluators find problem sets with higher thoroughness and validity than field dependent (FD) evaluators. (4) FI evaluators use more analytical approaches than FD evaluators during heuristic evaluation. (5) Using the newly developed E-Commerce heuristic set results in finding larger number of real usability problems than using Nielsen's (1994d) heuristic set. (6) Users' purchase intention on E-Commerce website is affected, in descending importance by the following five factors: prevention from error, information access quality, shopping support, ease of comprehension, and hedonic quality. Comments: Double evaluator was used.

20100921 - Capra, Usability problem description...

Usability problem description and the evaluator effect in usability testing by Capra, Miranda G., Ph.D., Virginia Polytechnic Institute and State University, 2006 , 292 pages; AAT 3207958 My interest: (1) How she developed the 10 UPD? (2) 10 UPD validated? If yes, how? (3) Methodology – 44 evaluators, usability reports, 2 categories of evaluation. (4) Comparison of 2 categories of evaluators – how? (5) Comparison of single evaluator and double evaluator – how? Action: To read the Dissertation again in future. (I have read her Dissertation before.) Previous usability evaluation method (UEM) comparison studies have noted an evaluator effect on problem detection in heuristic evaluation, with evaluators differing in problems found and problem severity judgments. The goals of this research were to develop UPD (usability problem description) guidelines, explore the evaluator effect in UT, and evaluate the usefulness of the guidelines for grading UPD content. Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are (briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the reader sympathize with the user. Comments: Capra had established 10 usability criteria (usability problem descriptions). A fourth study compared usability reports collected from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator and graded the reports for following 6 of the UPD guidelines. Comments: Two categories of evaluators were used, i.e. graduate students & usability practitioners. Three judges for each evaluation => why??? There was support for existence of an evaluator effect, even when watching prerecorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The final recommendations are to use multiple evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness. Comments: Shall I use the concept of multiple evaluators for my UEM?