20101013 - Ballard, Web site usability...educational web sites

Web site usability: A case study of student perceptions of educational web sites by Ballard, Joyce Kimberly, Ph.D., University of Minnesota, 2010 , 408 pages; AAT 3408366 My Interest: 1) Nielsen's Heuristics. 2) Xerox Heuristic Evaluation Checklist. 3) Computer Use/User Self-Efficacy Scale. 4) Quality of Web-based Instruction. 5) Usability Testing – think-aloud protocol. 6) Usability Testing – eye tracking. 7) Usability Testing – time-error log. 8) Methodology of data analysis. 9) Theoretical framework. 10) Validity & reliability. Action: To read the Dissertation in future. Research Goal The purpose of this research study was to understand the construct of usability from the perspective of 74 students enrolled in six online courses offered by one online and distance learning program at a large, public university in the Midwest. Methodology Six courses, designed and developed by two different groups, professional and nonprofessional developers, were selected. The study used both quantitative and qualitative measures to record the experiences of students enrolled in the six online courses. First, the courses were evaluated using Nielsen's (1994, 2000, 2002) heuristics as operationalized by the Xerox Heuristic Evaluation Checklist (1995) as a standard measure of usability, then rank-ordered by heuristic evaluation score. Comments: I would love to see this Xerox Heuristic Evaluation Checklist. I suppose this is a UET (usability evaluation tool) for UEM = heuristic evaluation. Would love to know WHAT are the usability criteria (heuristics) used in this Checklist. Eachus and Cassidy's (2006) Computer Use Self-efficacy Scale was used as a pre-course survey to measure students' computer self-efficacy prior to beginning their online course. Comments: Would like to see the details of Computer Use/User Self-Efficacy Scale. Stewart, Hong, and Strudler's (2004) Quality of Web-based Instruction was used as a post-course survey to measure student satisfaction with their online course experience. A subset of 29 students participated in usability testing sessions in the usability lab. A think-aloud protocol provided qualitative data in the form of verbal reports, eye-tracking recordings provided data confirming the think-aloud protocol data, and a time-error log provided "time to complete tasks," and "error rate" data as students completed seven typical tasks required to successfully participate in an online course. A summary, debriefing interview with each student was conducted to record any additional student comments and any student recommendations for improving the courses. Qualitative data were examined for themes and a coding scheme was created. This coding scheme, which illustrated the issues specific to educational web sites, was compared to Nielsen's (1994, 2000, 2002) heuristics to evaluate whether Nielsen's (1994, 2000, 2002) heuristics, widely accepted as the standard for the design and development of business and commercial websites, also apply to educational web sites. Comments: I will want to review her Guidelines for educational web sites. These can be correlated with learning-related usability criteria. Design and development guidelines for educational web sites were written by the researcher based on the study findings. These guidelines were mapped to Nielsen's heuristics as operationalized by the Xerox Heuristic Evaluation Checklist (1995). Results Discussion The results of the quantitative and qualitative measures used were analyzed by course and course development type. The most significant results of this study came from the analysis of the variables according to course development type. The results of the study findings include that the course design type, professional or nonprofessional, was related to usability as measured by students' error rates, Nielsen's heuristic evaluation scores, and student satisfaction scores. The professionally-developed courses were found to be significantly higher in usability than the non-professionally-developed courses by task error rate, Nielsen's heuristic evaluation score and student satisfaction scores. The analysis of students' verbal reports resulted in three times as many positive comments for the professionally-developed courses when compared to the positive comments for the non-professionally-developed courses. The results of the quantitative and qualitative measures used were also analyzed by course. When comparisons were made between courses using courses as the unit of analysis the findings were different. The rank-order of courses was mixed between course types when compared by error rates. The Nielsen's heuristic evaluation scores as measures of usability for educational web sites were not consistent with students' judgments of course usability as measured by error rate scores. There was no relationship between the usability ranking of courses by Nielsen's heuristics and usability as judged by students' error rates. However, an analysis of students' verbal reports identified 52 common themes and confirmed the importance of Nielsen's heuristics in educational course design. Comments: She has found that Nielsen's heuristics are important and relevant for educational course design and educational websites. This is a good support for my literature review. The correlation between the self-efficacy score and error rate means was nonsignificant. The correlation between self-efficacy and error rate was small; very close to zero. There was a small positive correlation between student satisfaction and usability as measured by error rates. Conclusion Based on the analysis of the study variables according to course development type, the results of this study found that Nielsen's usability heuristics, a respected evaluation tool used primarily to measure the usability of commercial web sites, can be used to evaluate instructional web sites and used to differentiate between levels of usability in the same way usability is judged by students. Chapter Two 14 Review of the Literature 14 Theoretical Framework 15 Usability. 17 Defining Usability 18 Usability and Web-based Instruction 19 Web-based Instruction and Instructional Design 21 Instructional Design 23 Defining Instructional Design 24 The Need to Evaluate Design and Quality 26 Instructional Design Differences . 29 Usability and Aesthetics 32 Usability Guidelines for Interface Design 34 Summary 35 Chapter Three 37 Research Method 37 Introduction . 37 Research Questions and Data Collection . 38 Rationale of the Methodology 41 Xerox Heuristics Evaluation: A System Checklist 42 Rationale 42 Validity . 44 Reliability 48 Method 48 Survey One Computer User Self-efficacy Scale (CUSE) 50 Rationale 50 Validity 52 Reliability 52 Survey Two Student Evaluation of the Quality of Web-Based Instruction 52 Rationale 52 Validity . 54 Reliability 54 Institutional context . 55 Institution . 55 College 55 Online Distance Education Program . 56 Course Management Systems 56 Course Selection 57 Student Selection/Recruitment . 60 IRB Approval 61 Recruitment Procedures 61 Data Collection Procedures . 62 General Collection Procedures . 63 Volunteers for Usability Testing . 64 Follow-up Attempts 65 Quantitative Data Collection 65 Timing of Instrument Use 65 Qualitative Data Collection . 66 Timing of Usability Testing 66 Qualitative Data from the Usability Testing Sessions 67 Usability Testing 67 Usability Testing Site 67 Pre-determined Tasks 69 Think-aloud Protocols 69 Time Log, Video, Audio, & Eye-tracking Data Collection 72 Time Log Data 72 Video, Audio, and Eye-tracking Data 73 Error Log . 75 Summary Interview 77 Post-course Student Evaluation of the Quality of Web-Based Instruction 77 Limitations of the Study 77 Summary 78 Chapter Four . 81 Research Findings 81 Research Questions 83 Summary 114

20100926 - Harmons, Usability Study of Postconference Online Selfassessment...

A usability study of a postconference online selfassessment program for health care professionals by Harmons, Eric M., M.A., Texas Woman's University, 2005 , 49 pages; AAT 1428775 My Interest: 1) Usability analysis/study. 2) QUIS – Questionnaire User Interface Survey. 3) Phone interview. 4) Post-conference online assessment. 5) Blackboard Academic Suite. Action: To read the Dissertation in the future. Motivation Continuing education (CE) conferences are widely used by professionals for the enhancement of clinical proficiency. Professionals view these types of educational activities as a vital means to enhance their professional practice and maintain clinical competency. The current environment of continuing education conferences revolves around a single-phase (didactic only) approach to learning. Current literature supports the notion of a multi-phased approach to enhance the learning experience for the lifelong learner. Research Goal The purpose of this study was a pilot usability analysis of a post-conference online selfassessment, developed within a Blackboard Academic Suite(TM) (Blackboard) classroom. Methodology The Questionnaire User Interaction Survey (QUIS) and phone interviews were utilized to assess the user interaction satisfaction with completing the self-assessment program. Results Discussion The overall response of participants to the system was a mean of 7.86 (scale from 1--9) across all the categories of usability. Based on the results, the Blackboard system is viewed by healthcare professionals as an effective medium for delivering an online self-assessment. Participants felt the self-assessment provided an effective tool to reflect on the presented material and direct their learning. Comments: Softcopy Dissertation is scanned version; cannot copy and paste; difficult to blog about it.

20100925 - Zitkute, Design & Evaluation of [e-Learning course]...usability questionnaire

Design and evaluation of a Siebel Basic Navigation course by Zitkute, Lina, M.A., Concordia University (Canada), 2006 , 52 pages; AAT MR20662 My Interest: 1) Usability Questionnaire used for usability evalution. 2) GUIQ – Generic User Interface Questionnaire – Chin et al, 1988. Action: To read this Dissertation in future. Research Goal The results of a usability evaluation of an e-learning course are presented in this thesis equivalent. The main goal of the study was to evaluate the usability of the Siebel Basic Navigation e-learning course. A secondary goal was to explore the eventual impact of elearning course usability on student performance. Background The Siebel Basic Navigation course was designed for one of the major Canadian pharmaceutical companies. This course had five modules and a test at the end. The primary purpose of this course was to introduce a new Siebel application upgrade, since this company had just upgraded from a desktop-based Siebel application to a Web-based Siebel application. Major improvements and changes needed to be introduced to the users. The secondary purpose of this course was to introduce new hires, who would not have had prior experience with Siebel, to the Siebel application. Methodology The participants, seven health care and banking professionals, completed the Siebel Basic Navigation e-learning course, and filled out the usability questionnaire. The usability evaluation was based on the Generic User Interface Questionnaire developed by Chin, Diehl, & Norman, (1988). Analysis of Data The distribution of the scores were calculated and high median (the middle value of a set of data), and high range were identified. In addition to the descriptive statistics, correlations between the software usability and student test performance emerged, underlining the importance of usability evaluation of systems supporting on-line learning. Comments: Softcopy Dissertation is scanned version; not easy to blog about it; cannot copy & paste.

20100925 - Ryu, ..Usability Questionnaires for Electronic Mobile Products...

Development of usability questionnaires for electronic mobile products and decision making methods by Ryu, Young Sam, Ph.D., Virginia Polytechnic Institute and State University, 2005 , 206 pages; AAT 3177322 My Interest: 1) MPUQ – Mobile Phone Usability Questionnaire. 2) How to select usability dimensions & questionnaire items. 3) Redundancy & Relevancy analysis. 4) How to test/validate MPUQ. Action: To read the Dissertation in future. Problem Statement As the growth of rapid prototyping techniques shortens the development life cycle of software and electronic products, usability inquiry methods can play a more significant role during the development life cycle, diagnosing usability problems and providing metrics for making comparative decisions. A need has been realized for questionnaires tailored to the evaluation of electronic mobile products, wherein usability is dependent on both hardware and software as well as the emotional appeal and aesthetic integrity of the design. Research Goal This research followed a systematic approach to develop a new questionnaire tailored to measure the usability of electronic mobile products. The Mobile Phone Usability Questionnaire (MPUQ) developed throughout this series of studies evaluates the usability of mobile phones for the purpose of making decisions among competing variations in the end-user market, alternatives of prototypes during the development process, and evolving versions during an iterative design process. In addition, the questionnaire can serve as a tool for identifying diagnostic information to improve specific usability dimensions and related interface elements. Methodology Employing the refined MPUQ, decision making models were developed using Analytic Hierarchy Process ( AHP) and linear regression analysis. Next, a new group of representative mobile users was employed to develop a hierarchical model representing the usability dimensions incorporated in the questionnaire and to assign priorities to each node in the hierarchy. Employing the AHP and regression models, important usability dimensions and questionnaire items for mobile products were identified. Finally, a case study of comparative usability evaluations was performed to validate the MPUQ and models. A computerized support tool was developed to perform redundancy and relevancy analyses for the selection of appropriate questionnaire items. The weighted geometric mean was used to combine multiple numbers of matrices from pairwise comparison based on decision makers' consistency ratio values for AHP. The AHP and regression models provided important usability dimensions so that mobile device usability practitioners can simply focus on the interface elements related to the decisive usability dimensions in order to improve the usability of mobile products. The AHP model could predict the users' decision based on a descriptive model of purchasing the best product slightly but not significantly better than other evaluation methods. Mobile Phone Usability Questionnaire (MPUQ) Except for memorability, the MPUQ embraced the dimensions included in the other well-known usability definitions and almost all criteria covered by the existing usability questionnaires. In addition, MPUQ incorporated new criteria, such as pleasurability and specific tasks performance. 2.1. Subjective Usability Assessment 2.1.1. Definitions and Perspectives of Usability 2.1.2. Usability Measurements 2.1.3. Subjective Measurements of Usability 2.1.4. Usability Questionnaires 2.1.4.1. Definition of Questionnaire 2.1.4.2. Questionnaires and Usability Research 2.2. Mobile Device Usability 2.2.1. Definition of Electronic Mobile Products 2.2.2. Usability Concept for Mobile Device 2.2.3. Mobile Device Interfaces 2.2.4. Usability Testing for Mobile Device 3. PHASE I : DEVELOPMENT OF ITEMS AND CONTENT VALIDITY 3.1. Need for a New Scale 3.2. Study 1: Conceptualization and Development of Initial Items Pool 3.2.1. Conceptualization 3.2.2. Survey on Usability Dimensions and Criteria 3.2.2.1. Usability Dimensions by Early Studies 3.2.2.2. Usability Dimensions in Existing Usability Questionnaires 3.2.2.3. Usability Dimensions for Consumer Products 3.2.2.4. Items from a Usability Questionnaire for a Specific Product 3.2.3. Creation of an Items Pool 3.2.4. Choice of Format 3.3. Study 2: Subjective Usability Assessment Support Tool and Item Judgment 3.3.1. Method 3.3.1.1. Design 3.3.1.2. Equipment 3.3.1.3. Participants 3.3.1.4. Procedure 3.3.2. Result 3.3.2.1. Part 1. Redundancy Analysis 3.3.2.2. Part 2. Relevancy Analysis 3.3.3. Discussion 3.4. Outcome of Studies 1 and 2 4. PHASE II : REFINING QUESTIONNAIRE 4.1. Study 3: Questionnaire Item Analysis 4.1.1. Method 4.1.1.1. Design 4.1.1.2. Participants 4.1.1.3. Procedure 4.1.2. Results 4.1.2.1. User Information 4.1.2.2. Factor Analysis 4.1.2.3. Scale Reliability 4.1.2.4. Known-group Validity 4.1.3. Discussion 4.1.3.1. Eliminated Questionnaire Items 4.1.3.2. Normative Patterns 4.1.3.3. Limitations 4.2. Outcome of Study 3 6. PHASE IV : VALIDATION OF MODELS 6.1. Study 6: Comparative Evaluation with the Models 6.1.1. Method 6.1.1.1. Design 6.1.1.2. Equipment 6.1.1.3. Participants 6.1.1.4. Procedure 6.1.2. Results 6.1.2.1. Mean Rankings 6.1.2.2. Preference Data Format 6.1.2.3. Friedman Test for Minimalist 6.1.2.4. Friedman Test for Voice/Text Fanatics 6.1.2.5. Comparisons Among the Methods 6.1.2.6. Important Usability Dimensions 6.1.3. Discussion 6.1.3.1. Implication of Each Evaluation Method 6.1.3.2. PSSUQ and the MPUQ 6.1.3.3. Validity of MPUQ 6.1.3.4. Usability and Actual Purchase 6.1.3.5. Limitations 6.2. Outcome of Study 6

20100925 - Rawls, Performance Support and Usability...

Performance support and usability: An experimental study of electronic performance support interfaces by Rawls, Charles L., Jr., Ed.D., University of Central Florida, 2005 , 120 pages; AAT 3193500 My Interest: 1) Checklist. 2) Online Questionnaire – PUTQ. 3) Online Questionnaire – UCFUQ. Action: To read the Dissertation in the future. Research Goal This study evaluated the usability of two types of performance-support interfaces that were designed using informational and experiential approaches. The experiment sought to determine whether there is a relationship between usability and the informational and experiential approaches. Methodology Two instruments---a checklist and an online usability questionnaire-- -were used to measure the five dependent variables: efficiency, intuitiveness, errors, satisfaction, and student performance. First, a checklist was used to assess the students' performance completing their task, which was a copyright issue request letter. The checklist was designed as a performance criterion tool for the researcher, instructor, and participants to use. Second, an online usability questionnaire was constructed based on the Purdue Usability Testing Questionnaire (PUTQ) questions to measure interface efficiency, intuitiveness, errors, and satisfaction. This study tested two approaches to user interface design for the Electronic Performance Support (EPS) using two HTML interface templates and the information from an existing training module. There were two interventions consisting of two interface types: informational and experiential. Results Discussion Results of this study indicated that students at the University of Central Florida reported no differences between the two interface types. It was postulated that the informational interface would yield a higher mean score because of its implementation of HCI guidelines, conventions, and standards. However, it was concluded that the informational interface may not be a more usable interface. Users may be as inclined to use the experiential interface as the informational interface. (Abstract shortened by UMI.) CHAPTER THREE. METHOD Participants. Research Design Instruments Intervention. Data Collection Data Analysis Limitations CHAPTER FIVE. CONCLUSIONS Hypotheses Data * Efficiency * Intuitiveness * Errors * Satisfaction * Student Performance Post Hoc Data * Interface and Age * Interface and Occupation * UCFUQ Reliability Analysis Summary Recommendations for Further Research APPENDIX D. CHECKLIST APPENDIX E. UNIVERSITY OF CENTRAL FLORIDA USABILITY QUESTIONNAIRE (UCFUQ) APPENDIX F. DEMOGRAPHIC SURVEY APPENDIX G. PURDUE USABILITY TESTING QUESTIONNAIRE (ORIGINAL)