Searched ProQuest dissertations for "Usability" for the last 2 years. 46 were found. 1. A comparative review of water disinfection methods appropriate for developing countries and their efficacy, cost-efficiency, and usability by Fuqua, Gary W., M.P.H., The University of Texas School of Public Health, 2010, 85 pages; AAT 1479579 2. Backtracking events as indicators of software usability problems by Akers, David Light, Ph.D., › 115 references 3. Beyond usability: An alternative usability evaluation method, PUT-Q2 by Abel, Troy Donald, Ph.D., Iowa State University, 2010, 178 pages; AAT 3418225 4. Design and usability of a system for the study of head orientation by Chen, Ji, M.S., 5. Design and implementation of a system for evaluating the mobile Web usability by Kaveti, Rajesh, M.S., University of Massachusetts Lowell, 2010, 51 pages; AAT 1478732 › 34 references 6. Developing Academic Vocabulary Independently (DAVI): A usability study by Richards, Monica Grace, M.A., Iowa State University, 2010, 87 pages; AAT 1475935 › 44 references 7. Distributed Wizard of Oz usability testing for agile teams by Hosseini Khayat, Ali, M.Sc., 8. Enhancing usability testing with functional near infrared spectroscopy by Hirshfield, Leanne M., Ph.D., › 116 references 9. Health information technology usability evaluation: methods, models, and measures by Yen, Po-Yin, Ph.D., 10. Remote usability testing - A new approach facilitated by virtual worlds by Chalil Madathil, Kapil, M.S., 11. The usability of teacher-growth scores versus CST/API/AYP math status scores in sixth and seventh grade mathematics classes by Cameron, Garry Van, Ed.D., 12. Understanding individual results in usability assessments by Commeford, Gerett, M.Sc., 13. Understanding usability and user experience of a technology enhanced role-playing system by Jadhav, Kavita Arun, M.A., › 31 references 14. Usability of the Primary Measures of Music Audiation (PMMA) with 5-year-old Korean children by Lee, Jooyoung, Ph.D., State University of New York at Buffalo, 2010, 195 pages; AAT 3407979 › 183 references 15. Web site usability: A case study of student perceptions of educational web sites by Ballard, Joyce Kimberly, Ph.D., › 261 references 16. A cross-cultural usability study on the internationalization of user interfaces based on an empirical five factor model by Chakraborty, Joyram, Ph.D., › 152 references 17. Affect and cognition as antecedents of intergroup attitudes: The role of applicability and judged usability by Leary, Scott Paul, Ph.D., › 86 references 18. A full life-cycle methodology for structured use-centered quantitative usability requirements specification and usability evaluation of websites by Hu, Guoqiang, Ph.D., 19. Evaluating the efficiency, usability, and safety of computerized order sets in Sunnybrook Health Sciences by Chan, Julie Min-Ting, M.H.Sc., › 37 references 20. Evaluating the usability of information systems in the International Space Station integration processes by Nagy, Nelda S., Ph.D., 21. Improving the user experience of the by Berkman, Eric, M.A., › 15 references 22. Investigation of wheelchair ingress/egress activities in large accessible transit vehicles and evaluation of by Sison, Sheryll, M.Eng., › 21 references 23. Mapping usability: A critical research analysis of trends in software usability research, theory, and practice by Sousa, Morgan Margaret, Ph.D., 24. Retrospective investigation of telehealth monitor usage: Usability of the Health Buddy by Ou, Bo, M.S., State University of New York at Buffalo, 2009, 55 pages; AAT 1469115 › 40 references 25. Reverse engineering of content as a task for finding usability problems: An evaluative case study using the Wikibreathe tool for online creation of asthma action plans by Wan, Flora, M.A.Sc., › 51 references 26. Usability testing: A model for improved online course development? by Fisher, Elizabeth Anne Bandy, Ph.D., The University of Alabama, 2009, 196 pages; AAT 3385370 › 92 references 27. Verifying an integrated model of usability in games by Schaffer, Noah, Ph.D., Rensselaer Polytechnic Institute, 2009, 108 pages; AAT 3371298 › 38 references
28. Web usability or accessibility: Comparisons between people with and without intellectual disabilities in viewing complex naturalistic scenes using eye-tracking technology by Bazar, Nancy Sceery, Ph.D., George Mason University, 2009, 251 pages; AAT 3364571 › 103 references 29. A multi-faceted model of the consequences of sample size choice in usability testing by Yang, Minmin, Ph.D., › 17 references 30. An investigation into the relationship between B2C usability and profitability by Whitehead, Christopher, Ph.D., › 156 references 31. Assessing the relationships among aesthetics, complexity, and usability in treemap visualizations by Yoon, James William, M.I.St., › 85 references 32. A tool for hearing aid and cochlear implant users to judge the usability of cellular telephones in field conditions by Deer, Maria Soledad, Au.D., › 23 references 33. A virtual margin for knowledge work on the Web: Design, implementation and usability testing by Forde, Carl, M.A., › 48 references 34. Beyond usability: A rubric to evaluate the emotional impact of e-commerce homepages by Abel, › 55 references 35. Clinicians' perceptions of usability of an electronic medical record over time by Seckman, Charlotte, Ph.D., › 110 references 36. Development and usability heuristic evaluation of an application in PDA for supporting physicians tasks at the point of care by Olarte Enciso, Nadia Elizabeth, M.E., University of Puerto Rico, Mayaguez (Puerto Rico), 2008, 121 pages; AAT 1451346 › 19 references 37. Enhancing the performance and usability of software transactional memory by Marathe, Virendra Jayant, Ph.D., › 177 references 38. Evaluating the usability and security of a video CAPTCHA by Kluever, Kurt Alfred, M.S., Rochester Institute of Technology, 2008, 114 pages; AAT 1457503 › 110 references 39. Improving accessibility and usability of medical equipment for patients with disabilities by Story, Martha Follette, Ph.D., › 115 references 40. Investigation of dynamic three-dimensional tangible touchscreens: Usability and feasibility by DeWitte, Anne E., M.S., Rochester Institute of Technology, 2008, 66 pages; AAT 1457498 › 68 references 41. Investigating the correlation of usability measures and user tests: A roadmap for a predictive model by Donyaee, Mohammad Khashayar, Ph.D., 42. Older adults and cellular phones: A mixed-methods usability study by Kubik, Sara J., Ph.D., › 114 references 43. The development and usability of an informational website for family caregivers by Tarabochia, Dawn S., Ph.D., The University of Utah, 2008, 170 pages; AAT 3340349 › 70 references 44. The development of an instrument to assess the usability of courseware by Hollins, Louvenia D., Ed.D., Northern Illinois University, 2008, 270 pages; AAT 3324334 › 90 references 45. The distributed cognitive walkthrough: The impact of differences in cognitive theory on usability evaluation by › 53 references 46. Web aesthetics and usability: An empirical evaluation of white space by Kripintiris, Konstantinos Efstratios, M.A., Michigan State University, 2008, 75 pages; AAT 1459215 › 73 references
|
Showing posts with label usability. Show all posts
Showing posts with label usability. Show all posts
Friday, October 1, 2010
USABILITY Dissertation Abstracts for the last 2 years
Tuesday, September 21, 2010
20100921 - Chang, Usability Study of Mouseover
A theory-based usability study of the mouseover abstract interfaces by Chang, Yaowen, Ed.D., Columbia University Teachers College, 2005 , 125 pages; AAT 3175671 My interest: (1) Relationships among Effectiveness, Efficiency, and User Satisfaction. (2) Eight outcome measures – what? (3) Comparison methodology of 2 websites. (4) Measures of Effectiveness, Efficiency, and User Satisfaction. How to analyse? Action: To read specific parts of this Dissertation in future. This study:
The MOA interfaces provided users with an abstract for the linked page in text and/or audio format when users mouse-overed the hyperlinks. Participants explored two websites, one containing astronomy material (pertaining to the phases of the moon) and the other, facts about the Institutional Review Board. After each browsing session, the participants performed assigned tasks. Eight outcome measures were derived from their responses and used to evaluate the efficiency, effectiveness, and user satisfaction perspectives of the interface usability. Contradictory to findings in existing literature, results consistently failed to find supporting evidence for the beneficial effects of context information or modality effects in design usability. Comments: Research findings could also be contrary to popular findings. Phrase "consistently failed to find supporting evidence" is very appropriate; better than saying NO. The correlations among the perspectives of usability indicated that efficiency, effectiveness, and user satisfaction are not interchangeable. Comments: We always know that. So what? What is the impact to usability? Thus, it is essential for researcher to adopt a multi-trait/multi-method approach when conducting usability evaluation. Comments: May need to use this statement in future…to quote in my Dissertation.
|
Tuesday, November 3, 2009
Nov 4 - Jeffrey Rubin, The Problem of Unusable Products and Systems
Chapter 1
The Problem of Unusable Products and Systems
Five Reasons for Hard-To-Use Products and Systems
1. During product development, the emphasis and focus have been on the machine or system, not on the person who is the ultimate end user.
2. As technology has penetrated the mainstream consumer market, the target audience has changed and continues to change dramatcially. Development organizations have been slow to react to this evolution.
3. The design of usable product is a difficult, unpredictable endeavor, yet many organizations trust it as it were just "common sense."
4. Organizations employ very specialized teams and approached to product and system development, yet fail to integrate them with each other.
5. The design of the user interface and the technical implementation of the user interface are different activities, requiring very different skills. Today, the emphasis and the need are on the design aspect, while many engineers possess the mind set and skill set for technical implementation.
User-Centered Design - Definition
User-centered design (UCD) is the most recent term coined to describe an approach that has been around for decades under different names, such as human factors engineering, ergonomics, and more recently, usability engineering.
UCD represents not only the techniques, processes, methods and procedures for designing usable products and systems, but just as important, the philosophy that places the user at the center of the process.
Three Principles of A User-Centered Design
1. An early focus on users and tasks.
2, Empirical measurement of product usage.
3. Iterative design whereby a product is designed, modified, and tested repeatedly.
Product Development Life Cycle
(Figure 1.5)
1. User and Usage Needs Analysis.
2. Specification of Requirements.
3. Preliminary (High Level) Design.
4. Detailed Design.
5. Product Build.
6. Product Release.
Hewlett Packard's Human Factors During Life Cycle
(Figure 1.6)
HUMAN FACTORS ACTIVITIES
Phase 1 (Needs Analysis)
* Identify and charaterize target user population.
* Identify and analyze user tasks.
* Identify users' physical and organizational environment.
* Identify usability problems on similar or existing products.
* Identify users' product feature list.
Phase 2 (Requirements Specification)
* Identify ergonomic or market-based requirements.
* Define and develop product usability and acceptance goals.
* Define product localization plan.
* Develop usability testing schedule.
Phase 3 (Conceptual Design)
* Develop product interface requirements.
* Define mental models describing system from the users' perspectives.
* Determine allocation of functions between user and system.
* Integrate Human Factors data, principles and guidelines.
* Test conceptual models.
Phase 4 (Prototype, Development and Test)
* Provide technical support in the development of product prototypes.
* Review production prototypes for ergonomic compliance.
* Test usability of HW, SW, and Learning Products.
* Provide feedback (including design recommendations) for designers based on results of usability test.
Phase 5 (Product Evaluation)
* Review final product specifications to ensure agreed upon usability recommendations have been satisfactorily implemented into the product design.
* Conduct on-site customer evaluations to determine product usability and how the product effectively meets user needs and expectations.
* Provide Marketing with usability advantages of the product.
Usability Goals and Objectives
It is generally accepted that an operational definition of usability include one or more of the following four factors, as outlined by Booth [15].
1. Usefulness
2. Effectiveness (ease of use)
3. Learnability
4. Attitude (likability).
Usability Testing
Usability testing is roughly divided into two main approaches.
The first approach involves formal tests conducted as true experiments, in order to confirm or refute specific hypotheses.
The second approach, a less formal one, employs an iterative cycle of tests intended to expose usability defieiencies and gradually shape or mold the product in question.
Usability testing divided into four types of tests.
1. Exploratory
2. Assessment
3. Validation
4. Comparison.
Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York. 1994.
The Problem of Unusable Products and Systems
Five Reasons for Hard-To-Use Products and Systems
1. During product development, the emphasis and focus have been on the machine or system, not on the person who is the ultimate end user.
2. As technology has penetrated the mainstream consumer market, the target audience has changed and continues to change dramatcially. Development organizations have been slow to react to this evolution.
3. The design of usable product is a difficult, unpredictable endeavor, yet many organizations trust it as it were just "common sense."
4. Organizations employ very specialized teams and approached to product and system development, yet fail to integrate them with each other.
5. The design of the user interface and the technical implementation of the user interface are different activities, requiring very different skills. Today, the emphasis and the need are on the design aspect, while many engineers possess the mind set and skill set for technical implementation.
User-Centered Design - Definition
User-centered design (UCD) is the most recent term coined to describe an approach that has been around for decades under different names, such as human factors engineering, ergonomics, and more recently, usability engineering.
UCD represents not only the techniques, processes, methods and procedures for designing usable products and systems, but just as important, the philosophy that places the user at the center of the process.
Three Principles of A User-Centered Design
1. An early focus on users and tasks.
2, Empirical measurement of product usage.
3. Iterative design whereby a product is designed, modified, and tested repeatedly.
Product Development Life Cycle
(Figure 1.5)
1. User and Usage Needs Analysis.
2. Specification of Requirements.
3. Preliminary (High Level) Design.
4. Detailed Design.
5. Product Build.
6. Product Release.
Hewlett Packard's Human Factors During Life Cycle
(Figure 1.6)
HUMAN FACTORS ACTIVITIES
Phase 1 (Needs Analysis)
* Identify and charaterize target user population.
* Identify and analyze user tasks.
* Identify users' physical and organizational environment.
* Identify usability problems on similar or existing products.
* Identify users' product feature list.
Phase 2 (Requirements Specification)
* Identify ergonomic or market-based requirements.
* Define and develop product usability and acceptance goals.
* Define product localization plan.
* Develop usability testing schedule.
Phase 3 (Conceptual Design)
* Develop product interface requirements.
* Define mental models describing system from the users' perspectives.
* Determine allocation of functions between user and system.
* Integrate Human Factors data, principles and guidelines.
* Test conceptual models.
Phase 4 (Prototype, Development and Test)
* Provide technical support in the development of product prototypes.
* Review production prototypes for ergonomic compliance.
* Test usability of HW, SW, and Learning Products.
* Provide feedback (including design recommendations) for designers based on results of usability test.
Phase 5 (Product Evaluation)
* Review final product specifications to ensure agreed upon usability recommendations have been satisfactorily implemented into the product design.
* Conduct on-site customer evaluations to determine product usability and how the product effectively meets user needs and expectations.
* Provide Marketing with usability advantages of the product.
Usability Goals and Objectives
It is generally accepted that an operational definition of usability include one or more of the following four factors, as outlined by Booth [15].
1. Usefulness
2. Effectiveness (ease of use)
3. Learnability
4. Attitude (likability).
Usability Testing
Usability testing is roughly divided into two main approaches.
The first approach involves formal tests conducted as true experiments, in order to confirm or refute specific hypotheses.
The second approach, a less formal one, employs an iterative cycle of tests intended to expose usability defieiencies and gradually shape or mold the product in question.
Usability testing divided into four types of tests.
1. Exploratory
2. Assessment
3. Validation
4. Comparison.
Source:
Jeffrey RUBIN. Handbook of Usability Testing: How To Plan, Design and Conduct Effective Tests. John Wiley & Sons, New York. 1994.
Friday, September 25, 2009
Sep 25 - Nielsen, Risks of Quantitative Studies (Alertbox)
Summary: Number fetishism leads usability studies astray by focusing on statistical analyses that are often false, biased, misleading, or overly narrow. Better to emphasize insights and qualitative research.
Risks of Quantitative Studies
There are two main types of user research: quantitative (statistics) and qualitative (insights).
The key benefit of quantitative studies is simple: they boil a complex situation down to a single number that's easy to grasp and discuss. I exploit this communicative clarity myself, for example, in reporting that using websites is 206% more difficult for users with disabilities and 122% more difficult for senior citizens than for mainstream users.
Beware Number Fetishism
When I read reports from other people's research, I usually find that their qualitative study results are more credible and trustworthy than their quantitative results. It's a dangerous mistake to believe that statistical research is somehow more scientific or credible than insight-based observational research. In fact, most statistical research is less credible than qualitative studies.
User interfaces and usability are highly contextual, and their effectiveness depends on a broad understanding of human behavior.
Fixating on numbers rather than qualitative insights has driven many usability studies astray. As the following points illustrate, quantitative approaches are inherently risky in a host of ways.
Random Results
Researchers often perform statistical analysis to determine whether numeric results are "statistically significant." By convention, they deem an outcome significant if there is less than 5% probability that it could have occurred randomly rather than signifying a true phenomenon.
This sounds reasonable, but it implies that one out of twenty "significant" results might be random if researchers rely purely on quantitative methods.
Luckily, most good researchers -- especiaally those in the user-interface field -- use more than a simple quantitative analysis. Thus, they typically have insights beyond simple statistics when they publish a paper, which drives down, but doesn't eliminate, bogus findings.
There's a reverse phenomenon as well: Sometimes a true finding is statistically insignificant because of the experiment's design. Perhaps the study didn't include enough participants to observe a major -- but rare -- finding in sufficient numbers. It would therefore be wrong to dismiss issues as irrelevant just because they don't show up in quantitative study results.
Pulling Correlations Out of a Hat
If you measure enough variables, you will inevitably discover that some seem to correlate. Run all your stats through the software and a few "significant" correlations will surely pop out. (Remember: one out of twenty analyses are "significant," even if there is no underlying true phenomenon.)
Studies that measure seven metrics will generate twenty-one possible correlations between the variables. Thus, on average, such studies will have one bogus correlation that the statistics program deems "significant," even if the issues being measured have no real connection.
In my Web Usability 2004 project, we collected metrics on fifty-three different aspects of user behavior on websites. There are thus 1,378 possible correlations that I could throw into the hopper. Even if we didn't discover anything at all in the study, about sixty-nine correlations would emerge as "statistically significant."
Overlooking Covariants
Even when a correlation represents a true phenomenon, it can be misleading if the real action concerns a third variable that is related to the two you're studying.
For example, studies show that intelligence declines by birth order. In other words, a person who was a first-born child will on average have a higher IQ than someone who was born second. Third-, fourth-, fifth-born children and so on have progressively lower average IQs. This data seems to present a clear warning to prospective parents: Don't have too many kids, or they'll come out increasingly stupid. Not so.
There's a hidden third variable at play: smarter parents tend to have fewer children. When you want to measure the average IQ of first-born children, you sample the offspring of all parents, regardless of how many kids they have. But when you measure the average IQ of fifth-born children, you're obviously sampling only the offspring of parents who have five or more kids. There will thus be a bigger percentage of low-IQ children in the latter sample, giving us the true -- but misleading -- conclusion that fifth-born children have lower average IQs than first-born children. Any given couple can have as many children as they want, and their younger children are unlikely to be significantly less intelligent than their older ones. When you measure intelligence based on a random sample from the available pool of children, however, you're ignoring the parents, who are the true cause of the observed data.
(Update added 2007: The newest research suggests that there may actually be a tiny advantage in IQ for first-born children after correcting for family size and the parents' economic and educational status. But the point remains that you have to correct for these covariants, and when you do so, the IQ difference is much less than plain averages may lead you to believe.)
As a Web example, you might observe that longer link texts are positively correlated with user success. This doesn't mean that you should write long links. Website designers are the hidden covariant here: clueless designers tend to use short text links like "more," "click here," and made-up words.
Over-Simplified Analysis
To get good statistics, you must tightly control the experimental conditions -- often so tightly that the findings don't generalize to real problems in the real world.
This is a common problem for university research, where the test subjects tend to be undergraduate students rather than mainstream users. Also, instead of testing real websites with their myriad contextual complexities, many academic studies test scaled-back designs with a small page count and simplified content.
For example, it's easy to run a study that shows breadcrumbs are useless: just give users directed tasks that require them to go in a straight line to the desired destination and stop there. Such users will (rightly) ignore any breadcrumb trail. Breadcrumbs are still recommended for many sites, of course. Not only are they lightweight, and thus unlikely to interfere with direct-movement users, but they're helpful to users who arrive deep within a site via search engines and direct links. Breadcrumbs give these users context and help users who are doing comparisons by offering direct access to higher levels of the information architecture.
Usability-in-the-large is often neglected by narrow research that doesn't consider, for example, revisitation behavior, search engine visibility, and multi-user decision-making.
Distorted Measurements
It's easy to prejudice a usability study by helping the users at the wrong time or by using the wrong tasks. In fact, you can prove virtually anything you want if you design the study accordingly. This is often a factor behind "sponsored" studies that purport to show that one vendor's products are easier to use than a competitor's products.
Even if the experimenters aren't fraudulent, it's easy to get hoodwinked by methodological weaknesses, such as directing the users' attention to specific details on the screen.
The very fact that you're asking about some design elements rather than others makes users notice them more and thus changes their behavior.
Many Web advertising studies are misleading, possibly because most such studies come from advertising agencies.
The most common distortion is the novelty effect: whenever a new advertising format is introduced, it's always accompanied by a study showing that the new type of ad generates more user clicks. Sure, that's because the new format enjoys a temporary advantage: it gathers user attention simply because it's new and users have yet to train themselves to ignore it.
The study might be genuine as far as it goes, but it says nothing about the new advertising format's long-term advantages once the novelty effect wears off.
Publication Bias
Editors follow the "man bites dog" principle to highlight new and interesting stories. While understandable, this preference for new and different findings imposes a significant bias in the results that get exposure.
Usability is a very stable field. User behavior is pretty much the same year after year. I keep finding the same results in study after study, as do many others. Every now and then, a bogus result emerges and publication bias ensures that it gets much more attention than it deserves.
Consider the question of Web page download time. Everyone knows that faster is better. Interaction design theory has documented the importance of response times since 1968, and this importance has been seen empirically in countless Web studies since 1995. E-commerce sites that speed up response times sell more. The day your server is slow, you lose traffic. (This happened to me recently: on January 14, Tog got "slashdotted"; because we share a server, my site lost 10% of its normal pageviews for a Wednesday when AskTog's increased traffic slowed useit.com down.)
If twenty people study download times, nineteen will conclude that faster is better. But again: one of every twenty statistical analyses will give the wrong result, and this one study might be widely discussed simply because it's new. The nineteen correct studies, in contrast, might easily escape mention.
Judging Bizarre Results
Bizarre results are sometimes supported by seemingly convincing numbers. You can use the issues I've raised here as a sanity check: Did the study pull correlations out of a hat? Was it biased or overly narrow? Was it promoted purely because it's different? Or was it just a fluke?
Typically, you'll discover that deviant findings should be ignored.
The broad concepts of human behavior in interactive systems are stable and easy to understand. The exceptions usually turn out to be exactly that: exceptions.
In 1989, for example, I published a paper on discount usability engineering, stating that small, fast user studies are superior to larger studies, and that testing with about five users is typically sufficient.
This was quite contrary to the prevailing wisdom at the time, which was dominated by big-budget testing. During the fifteen years since my original claim, several other researchers reached similar conclusions, and we developed a mathematical model to substantiate the theory behind my empirical observation. Today, almost everyone who does user testing has concluded that they learn most of what they'll ever learn with about five users.
But four or five studies constitute a trend, which much enhances the finding's credibility as a general phenomenon.
Quantitative Studies: Intrinsic Risks
All the reasons I've listed for quantitative studies being misleading indicate bad research; it's possible to do good quantitative research and derive valid insights from measurements. But doing so is expensive and difficult.
Quantitative studies must be done exactly right in every detail or the numbers will be deceptive. There are so many pitfalls that you're likely to land in one of them and get into trouble.
If you rely on numbers without insights, you don't have backup when things go wrong. You'll stumble down the wrong path, because that's where the numbers will lead.
Qualitative studies are less brittle and thus less likely to break under the strain of a few methodological weaknesses. Even if your study isn't perfect in every last detail, you'll still get mostly good results from a qualitative method that relies on understanding users and their observed behavior.
Yes, experts get better results than beginners from qualitative studies.
But for quantitative studies, only the best experts get any valid results at all, and only then if they're extremely careful.
Source:
Jakob Nielsen's Alertbox, March 1, 2004:
Risks of Quantitative Studies
http://www.useit.com/alertbox/20040301.html
Risks of Quantitative Studies (Jakob Nielsen's Alertbox)
Risks of Quantitative Studies
There are two main types of user research: quantitative (statistics) and qualitative (insights).
The key benefit of quantitative studies is simple: they boil a complex situation down to a single number that's easy to grasp and discuss. I exploit this communicative clarity myself, for example, in reporting that using websites is 206% more difficult for users with disabilities and 122% more difficult for senior citizens than for mainstream users.
Beware Number Fetishism
When I read reports from other people's research, I usually find that their qualitative study results are more credible and trustworthy than their quantitative results. It's a dangerous mistake to believe that statistical research is somehow more scientific or credible than insight-based observational research. In fact, most statistical research is less credible than qualitative studies.
User interfaces and usability are highly contextual, and their effectiveness depends on a broad understanding of human behavior.
Fixating on numbers rather than qualitative insights has driven many usability studies astray. As the following points illustrate, quantitative approaches are inherently risky in a host of ways.
Random Results
Researchers often perform statistical analysis to determine whether numeric results are "statistically significant." By convention, they deem an outcome significant if there is less than 5% probability that it could have occurred randomly rather than signifying a true phenomenon.
This sounds reasonable, but it implies that one out of twenty "significant" results might be random if researchers rely purely on quantitative methods.
Luckily, most good researchers -- especiaally those in the user-interface field -- use more than a simple quantitative analysis. Thus, they typically have insights beyond simple statistics when they publish a paper, which drives down, but doesn't eliminate, bogus findings.
There's a reverse phenomenon as well: Sometimes a true finding is statistically insignificant because of the experiment's design. Perhaps the study didn't include enough participants to observe a major -- but rare -- finding in sufficient numbers. It would therefore be wrong to dismiss issues as irrelevant just because they don't show up in quantitative study results.
Pulling Correlations Out of a Hat
If you measure enough variables, you will inevitably discover that some seem to correlate. Run all your stats through the software and a few "significant" correlations will surely pop out. (Remember: one out of twenty analyses are "significant," even if there is no underlying true phenomenon.)
Studies that measure seven metrics will generate twenty-one possible correlations between the variables. Thus, on average, such studies will have one bogus correlation that the statistics program deems "significant," even if the issues being measured have no real connection.
In my Web Usability 2004 project, we collected metrics on fifty-three different aspects of user behavior on websites. There are thus 1,378 possible correlations that I could throw into the hopper. Even if we didn't discover anything at all in the study, about sixty-nine correlations would emerge as "statistically significant."
Overlooking Covariants
Even when a correlation represents a true phenomenon, it can be misleading if the real action concerns a third variable that is related to the two you're studying.
For example, studies show that intelligence declines by birth order. In other words, a person who was a first-born child will on average have a higher IQ than someone who was born second. Third-, fourth-, fifth-born children and so on have progressively lower average IQs. This data seems to present a clear warning to prospective parents: Don't have too many kids, or they'll come out increasingly stupid. Not so.
There's a hidden third variable at play: smarter parents tend to have fewer children. When you want to measure the average IQ of first-born children, you sample the offspring of all parents, regardless of how many kids they have. But when you measure the average IQ of fifth-born children, you're obviously sampling only the offspring of parents who have five or more kids. There will thus be a bigger percentage of low-IQ children in the latter sample, giving us the true -- but misleading -- conclusion that fifth-born children have lower average IQs than first-born children. Any given couple can have as many children as they want, and their younger children are unlikely to be significantly less intelligent than their older ones. When you measure intelligence based on a random sample from the available pool of children, however, you're ignoring the parents, who are the true cause of the observed data.
(Update added 2007: The newest research suggests that there may actually be a tiny advantage in IQ for first-born children after correcting for family size and the parents' economic and educational status. But the point remains that you have to correct for these covariants, and when you do so, the IQ difference is much less than plain averages may lead you to believe.)
As a Web example, you might observe that longer link texts are positively correlated with user success. This doesn't mean that you should write long links. Website designers are the hidden covariant here: clueless designers tend to use short text links like "more," "click here," and made-up words.
Over-Simplified Analysis
To get good statistics, you must tightly control the experimental conditions -- often so tightly that the findings don't generalize to real problems in the real world.
This is a common problem for university research, where the test subjects tend to be undergraduate students rather than mainstream users. Also, instead of testing real websites with their myriad contextual complexities, many academic studies test scaled-back designs with a small page count and simplified content.
For example, it's easy to run a study that shows breadcrumbs are useless: just give users directed tasks that require them to go in a straight line to the desired destination and stop there. Such users will (rightly) ignore any breadcrumb trail. Breadcrumbs are still recommended for many sites, of course. Not only are they lightweight, and thus unlikely to interfere with direct-movement users, but they're helpful to users who arrive deep within a site via search engines and direct links. Breadcrumbs give these users context and help users who are doing comparisons by offering direct access to higher levels of the information architecture.
Usability-in-the-large is often neglected by narrow research that doesn't consider, for example, revisitation behavior, search engine visibility, and multi-user decision-making.
Distorted Measurements
It's easy to prejudice a usability study by helping the users at the wrong time or by using the wrong tasks. In fact, you can prove virtually anything you want if you design the study accordingly. This is often a factor behind "sponsored" studies that purport to show that one vendor's products are easier to use than a competitor's products.
Even if the experimenters aren't fraudulent, it's easy to get hoodwinked by methodological weaknesses, such as directing the users' attention to specific details on the screen.
The very fact that you're asking about some design elements rather than others makes users notice them more and thus changes their behavior.
Many Web advertising studies are misleading, possibly because most such studies come from advertising agencies.
The most common distortion is the novelty effect: whenever a new advertising format is introduced, it's always accompanied by a study showing that the new type of ad generates more user clicks. Sure, that's because the new format enjoys a temporary advantage: it gathers user attention simply because it's new and users have yet to train themselves to ignore it.
The study might be genuine as far as it goes, but it says nothing about the new advertising format's long-term advantages once the novelty effect wears off.
Publication Bias
Editors follow the "man bites dog" principle to highlight new and interesting stories. While understandable, this preference for new and different findings imposes a significant bias in the results that get exposure.
Usability is a very stable field. User behavior is pretty much the same year after year. I keep finding the same results in study after study, as do many others. Every now and then, a bogus result emerges and publication bias ensures that it gets much more attention than it deserves.
Consider the question of Web page download time. Everyone knows that faster is better. Interaction design theory has documented the importance of response times since 1968, and this importance has been seen empirically in countless Web studies since 1995. E-commerce sites that speed up response times sell more. The day your server is slow, you lose traffic. (This happened to me recently: on January 14, Tog got "slashdotted"; because we share a server, my site lost 10% of its normal pageviews for a Wednesday when AskTog's increased traffic slowed useit.com down.)
If twenty people study download times, nineteen will conclude that faster is better. But again: one of every twenty statistical analyses will give the wrong result, and this one study might be widely discussed simply because it's new. The nineteen correct studies, in contrast, might easily escape mention.
Judging Bizarre Results
Bizarre results are sometimes supported by seemingly convincing numbers. You can use the issues I've raised here as a sanity check: Did the study pull correlations out of a hat? Was it biased or overly narrow? Was it promoted purely because it's different? Or was it just a fluke?
Typically, you'll discover that deviant findings should be ignored.
The broad concepts of human behavior in interactive systems are stable and easy to understand. The exceptions usually turn out to be exactly that: exceptions.
In 1989, for example, I published a paper on discount usability engineering, stating that small, fast user studies are superior to larger studies, and that testing with about five users is typically sufficient.
This was quite contrary to the prevailing wisdom at the time, which was dominated by big-budget testing. During the fifteen years since my original claim, several other researchers reached similar conclusions, and we developed a mathematical model to substantiate the theory behind my empirical observation. Today, almost everyone who does user testing has concluded that they learn most of what they'll ever learn with about five users.
But four or five studies constitute a trend, which much enhances the finding's credibility as a general phenomenon.
Quantitative Studies: Intrinsic Risks
All the reasons I've listed for quantitative studies being misleading indicate bad research; it's possible to do good quantitative research and derive valid insights from measurements. But doing so is expensive and difficult.
Quantitative studies must be done exactly right in every detail or the numbers will be deceptive. There are so many pitfalls that you're likely to land in one of them and get into trouble.
If you rely on numbers without insights, you don't have backup when things go wrong. You'll stumble down the wrong path, because that's where the numbers will lead.
Qualitative studies are less brittle and thus less likely to break under the strain of a few methodological weaknesses. Even if your study isn't perfect in every last detail, you'll still get mostly good results from a qualitative method that relies on understanding users and their observed behavior.
Yes, experts get better results than beginners from qualitative studies.
But for quantitative studies, only the best experts get any valid results at all, and only then if they're extremely careful.
Source:
Jakob Nielsen's Alertbox, March 1, 2004:
Risks of Quantitative Studies
http://www.useit.com/alertbox/20040301.html
Risks of Quantitative Studies (Jakob Nielsen's Alertbox)
Subscribe to:
Posts (Atom)
