20101013 - Hollins, ...Instrument to assess Usability of courseware

The development of an instrument to assess the usability of courseware by Hollins, Louvenia D., Ed.D., Northern Illinois University, 2008 , 270 pages; AAT 3324334 My Interest: 1) UET = Instructional Design Usability Survey (IDUS). 2) Heuristic evaluation. 3) Empirical usability testing. 4) Walkthroughs. 5) Field observations. 6) Theoretical framework. 7) Three-phases approach. 8) ANOVA. 9) Validity & reliability of UET/Survey. Action: To read the Dissertation in future. Research Goal This study describes the development and initial validation of a survey designed to measure students' perceptions of the usability of instructional design of educational courseware. Motivation There is a lack of measuring instruments to conduct this kind of research. The present study attempts to take steps to provide educators and researchers with a useful tool to facilitate their studies of this aspect or courseware creation and use. Methodology A three-phase approach is taken in this study to develop the Instructional Design Usability Survey (IDUS) instrument. Expert evaluators from the field of educational technology, research and assessment were involved in Phase I to establish content and face validity. During Phase II, student focus groups were formed from the population of courseware users to establish content validity and clarity of the instrument. In Phase III, students examined the usability of the instrument, and reliability and validity tests were conducted. Comments: This dissertation ought to be read in detail. Because a UET is developed. Second reason is to benchmark how she validated the UET. Results Discussion Reliability and validity of scores from the IDUS were assessed through statistical procedures performed on the data collected from 206 student participants. Internal consistency reliability was assessed for each construct using Cronbach's alpha. The internal consistency for the six constructs ranged from .73 to .92. Confirmatory factor analysis (CFA) was conducted to establish construct validity by examining the overall fit of the models as well as the magnitude of the factor loadings for each proposed construct. Results of the study show the IDUS is reliable but does not yet show strong enough evidence of validity. 1. INTRODUCTION TO THE STUDY 1 Statement of the Problem 3 Purpose of the Study 4 Theoretical Framework 5 Research Questions 9 Definition of Terms 9 Significance of the Study 10 Organization of the Study 10 2. REVIEW OF RELATED LITERATURE 12 CAI and User ID Guidelines 12 HCI and UEM 26 Empirical Usability Testing 29 Heuristic Evaluation 29 Field Observations 30 Walkthroughs 30 Analytical Modeling Methods 31 User-Centered Design (USD) 32 Survey Development and Validity 33 Summary of the Literature Review 41 3. RESEARCH METHODOLOGY 43 Research Methodology 44 Three-Phase Approach Overview 44 Research Site and Student Target Population 44 Instrument Development 46 Phase I: Expert Evaluation 49 Expert Target Population 49 Data Collection 50 Data Analysis 52 Phase II: Student Focus Group 56 Data Collection 56 Findings of Phase II 60 Phase III: Student Pilot Test 61 Data Collection 62 Data Analysis 63 Chapter Page Missing Data 65 Summary 66 4. FINDINGS OF THE STUDY 68 Demographic Data Analysis 68 Research Question 1 73 Research Question 1A 73 Research Question IB 88 Research Question 2 97 Posttest Reading Levels ANOVA 98 Posttest Math Levels ANOVA 100 Summary 105

20101013 - Fisher, Usability Testing...online course development

Usability testing: A model for improved online course development? by Fisher, Elizabeth Anne Bandy, Ph.D., The University of Alabama, 2009 , 196 pages; AAT 3385370 My Interest: 1) Usability testing. 2) Online course development. 3) UCD – user-centred design. 4) Usability testing protocol. Action: To read the Dissertation in future. Motivation According to the literature, usability testing as part of a user-centered design framework has been the industry standard for assessing the usability of products for quite some time. Business and industry and some governmental agencies routinely implement usability testing to test their products. Consequently, the literature is rich in this area. Despite the tremendous growth in online learning, little research has been conducted regarding the implementation of usability testing in academia, especially in regard to the development of online coursework. Research Goal Using the qualitative research paradigm, this study investigated the effectiveness of implementing usability testing as part of a user-centered design framework into the development of an online course for providing a model for improved course design. Methodology To implement the qualitative research design, a single, evaluative case study approach was employed. Usability testing was applied as part of a UCD framework, which calls for iterations of testing that inform changes in course design in a cyclic fashion. Data were collected during the spring 2009 semester at a major research university in the southeast. Fourteen participants from the freshmen student population took part in the study. Each participant was observed while completing five predefined tasks and corresponding surveys, which comprised a test session. One round consisted of 3 to 5 study participants each completing a test session. Data were collected through video recordings, surveys, observer logs, and journaling through four rounds. Findings Findings indicate that usability testing as part of a UCD framework may provide a model for improved online course design. 2. LITERATURE REVIEW 12 a. Introduction 12 b. Evolution of Distance Education and Online Learning 12 c. Usability Defined 15 d. Usability Testing Defined 16 e. Evolution of Usability Testing 17 f. Usability Testing Approaches 19 g. Usability Testing Models 21 h. Considerations for Usability Testing 28 i. User-Centered Design as a Framework 34 j . Summary 42 3. METHODOLOGY 44 a. Introduction 44 b. Researcher Positionality 45 c. Course Selection 46 d. Research Design 48 e. Rationale for Research Design 48 f. Research Question 50 g. Population and Sample 50 h. Instrumentation 51 g. Data Collection Procedures 56 h. Data Analysis 61 i. Summary 62 4. PRESENTATION OF DATA 63 a. Introduction 63 b. Demographics 64 c. Setting 65 d. Usability Testing Protocol 66 e. Summary 121

20101013 - Hu, ...Quantitative Usability Requirements Spec & Usability Evaluation..

A full life-cycle methodology for structured use-centered quantitative usability requirements specification and usability evaluation of websites by Hu, Guoqiang, Ph.D., Auburn University, 2009 , 201 pages; AAT 3386203 My Interest: 1) QUEST – Hu's usability evaluation method. 2) Expert usability review. 3) User usability testing. 4) SUS – System Usability Scale. 5) Usability metrics (usability criteria) of QUEST. Action: To search for his dissertation or journal article. Want to read more. Background World Wide Web has gained its dominant status in the cyber information and services delivery world in recent years. But how to specify website usability requirements and how to evaluate and improve website usability according to its usability requirements specification are still big issues to all the stakeholders. Research Goal To help solve this problem, we propose a website usability requirements specification and usability evaluation methodology that features a structured use-centered quantitative full life-cycle method. Methodology A validation experiment has been designed and conducted to prove the validity of the proposed methodology, QUEST (Quantitative Usability Equations SeT). Its principle is to prove that QUEST has stronger website usability evaluation capability than the most typical existing usability evaluation methods. Apparently, if QUEST's website usability evaluation capability is established, then its usability metrics can be used to quantitatively specify upfront user usability requirements for websites. In the validation experiment, 7 usability experts and 20 student subjects were recruited to perform 4 tasks on 2 open source calendar websites, WebCalendar 1.0.5 and VCalendar 1.5.3.1; 4 sets of usability data had been collected, which were corresponding to the following 4 usability evaluation methods respectively: expert usability review, traditional user usability testing, SUS (System Usability Scale), and QUEST. Comments: He should also compare against other popular usability evaluation methods such as heuristic evaluation, and cognitive walkthrough. He should also compare against other more popular Usability Questionnaires, such as QUIS and SUMI. Noted that a good method has been compared, i.e. user usability testing. Generally, he has not done a FAIR comparison. Results Discussion According to the experiment results: both the expert usability review and the traditional user usability testing were inconclusive on which of the 2 target websites had better usability; although SUS rated the overall usability of WebCalendar 1.0.5 at 66.00 and VCalendar 1.5.3.1 at 61.75, it was subjective and vague on usability problems; in contrast, QUEST not only rated the overall usability of WebCalendar 1.0.5 at 56.59 and VCalendar 1.5.3.1 at 35.97, but also revealed where the usability problems were and how severe each usability problem was in a quantitative manner. Comments: After reading Hu's abstract, an idea came to me. Initially, I have thought of selecting the type/hybrid of UEM based on literature review and comparison. Benchmarking on Hu, I could also do a study to compare various UEM and select the UEM type/hybrid based on the results of study. Conclusion In conclusion, it clearly can be stated that QUEST has stronger website usability evaluation capability than all other 3 most typical existing usability evaluation methods. So, the proposed methodology has been validated by the experiment results. Note: No Preview nor Full-Text dissertation is available for download.

20101013 - Ballard, Web site usability...educational web sites

Web site usability: A case study of student perceptions of educational web sites by Ballard, Joyce Kimberly, Ph.D., University of Minnesota, 2010 , 408 pages; AAT 3408366 My Interest: 1) Nielsen's Heuristics. 2) Xerox Heuristic Evaluation Checklist. 3) Computer Use/User Self-Efficacy Scale. 4) Quality of Web-based Instruction. 5) Usability Testing – think-aloud protocol. 6) Usability Testing – eye tracking. 7) Usability Testing – time-error log. 8) Methodology of data analysis. 9) Theoretical framework. 10) Validity & reliability. Action: To read the Dissertation in future. Research Goal The purpose of this research study was to understand the construct of usability from the perspective of 74 students enrolled in six online courses offered by one online and distance learning program at a large, public university in the Midwest. Methodology Six courses, designed and developed by two different groups, professional and nonprofessional developers, were selected. The study used both quantitative and qualitative measures to record the experiences of students enrolled in the six online courses. First, the courses were evaluated using Nielsen's (1994, 2000, 2002) heuristics as operationalized by the Xerox Heuristic Evaluation Checklist (1995) as a standard measure of usability, then rank-ordered by heuristic evaluation score. Comments: I would love to see this Xerox Heuristic Evaluation Checklist. I suppose this is a UET (usability evaluation tool) for UEM = heuristic evaluation. Would love to know WHAT are the usability criteria (heuristics) used in this Checklist. Eachus and Cassidy's (2006) Computer Use Self-efficacy Scale was used as a pre-course survey to measure students' computer self-efficacy prior to beginning their online course. Comments: Would like to see the details of Computer Use/User Self-Efficacy Scale. Stewart, Hong, and Strudler's (2004) Quality of Web-based Instruction was used as a post-course survey to measure student satisfaction with their online course experience. A subset of 29 students participated in usability testing sessions in the usability lab. A think-aloud protocol provided qualitative data in the form of verbal reports, eye-tracking recordings provided data confirming the think-aloud protocol data, and a time-error log provided "time to complete tasks," and "error rate" data as students completed seven typical tasks required to successfully participate in an online course. A summary, debriefing interview with each student was conducted to record any additional student comments and any student recommendations for improving the courses. Qualitative data were examined for themes and a coding scheme was created. This coding scheme, which illustrated the issues specific to educational web sites, was compared to Nielsen's (1994, 2000, 2002) heuristics to evaluate whether Nielsen's (1994, 2000, 2002) heuristics, widely accepted as the standard for the design and development of business and commercial websites, also apply to educational web sites. Comments: I will want to review her Guidelines for educational web sites. These can be correlated with learning-related usability criteria. Design and development guidelines for educational web sites were written by the researcher based on the study findings. These guidelines were mapped to Nielsen's heuristics as operationalized by the Xerox Heuristic Evaluation Checklist (1995). Results Discussion The results of the quantitative and qualitative measures used were analyzed by course and course development type. The most significant results of this study came from the analysis of the variables according to course development type. The results of the study findings include that the course design type, professional or nonprofessional, was related to usability as measured by students' error rates, Nielsen's heuristic evaluation scores, and student satisfaction scores. The professionally-developed courses were found to be significantly higher in usability than the non-professionally-developed courses by task error rate, Nielsen's heuristic evaluation score and student satisfaction scores. The analysis of students' verbal reports resulted in three times as many positive comments for the professionally-developed courses when compared to the positive comments for the non-professionally-developed courses. The results of the quantitative and qualitative measures used were also analyzed by course. When comparisons were made between courses using courses as the unit of analysis the findings were different. The rank-order of courses was mixed between course types when compared by error rates. The Nielsen's heuristic evaluation scores as measures of usability for educational web sites were not consistent with students' judgments of course usability as measured by error rate scores. There was no relationship between the usability ranking of courses by Nielsen's heuristics and usability as judged by students' error rates. However, an analysis of students' verbal reports identified 52 common themes and confirmed the importance of Nielsen's heuristics in educational course design. Comments: She has found that Nielsen's heuristics are important and relevant for educational course design and educational websites. This is a good support for my literature review. The correlation between the self-efficacy score and error rate means was nonsignificant. The correlation between self-efficacy and error rate was small; very close to zero. There was a small positive correlation between student satisfaction and usability as measured by error rates. Conclusion Based on the analysis of the study variables according to course development type, the results of this study found that Nielsen's usability heuristics, a respected evaluation tool used primarily to measure the usability of commercial web sites, can be used to evaluate instructional web sites and used to differentiate between levels of usability in the same way usability is judged by students. Chapter Two 14 Review of the Literature 14 Theoretical Framework 15 Usability. 17 Defining Usability 18 Usability and Web-based Instruction 19 Web-based Instruction and Instructional Design 21 Instructional Design 23 Defining Instructional Design 24 The Need to Evaluate Design and Quality 26 Instructional Design Differences . 29 Usability and Aesthetics 32 Usability Guidelines for Interface Design 34 Summary 35 Chapter Three 37 Research Method 37 Introduction . 37 Research Questions and Data Collection . 38 Rationale of the Methodology 41 Xerox Heuristics Evaluation: A System Checklist 42 Rationale 42 Validity . 44 Reliability 48 Method 48 Survey One Computer User Self-efficacy Scale (CUSE) 50 Rationale 50 Validity 52 Reliability 52 Survey Two Student Evaluation of the Quality of Web-Based Instruction 52 Rationale 52 Validity . 54 Reliability 54 Institutional context . 55 Institution . 55 College 55 Online Distance Education Program . 56 Course Management Systems 56 Course Selection 57 Student Selection/Recruitment . 60 IRB Approval 61 Recruitment Procedures 61 Data Collection Procedures . 62 General Collection Procedures . 63 Volunteers for Usability Testing . 64 Follow-up Attempts 65 Quantitative Data Collection 65 Timing of Instrument Use 65 Qualitative Data Collection . 66 Timing of Usability Testing 66 Qualitative Data from the Usability Testing Sessions 67 Usability Testing 67 Usability Testing Site 67 Pre-determined Tasks 69 Think-aloud Protocols 69 Time Log, Video, Audio, & Eye-tracking Data Collection 72 Time Log Data 72 Video, Audio, and Eye-tracking Data 73 Error Log . 75 Summary Interview 77 Post-course Student Evaluation of the Quality of Web-Based Instruction 77 Limitations of the Study 77 Summary 78 Chapter Four . 81 Research Findings 81 Research Questions 83 Summary 114