The strategies used by students to read educational Websites and their relation to Website usability and text design by Elshair, Hanan Mohamed, Ed.D., University of Pittsburgh, 2002 , 165 pages; AAT 3054275 My Interest: 1) Think-aloud usability testing. 2) Catogorisation into Design-related usability problems and Navigation-related problems. Action: To read the specific parts of Dissertation in future. Research Goal This study investigated the strategies used by students to read texts published on educational websites. It was intended to help inform instructional designers and teachers of appropriate ways to design and prepare instructional web-based texts. Methodology To conduct this investigation, seven graduate students in the School of Education in different majors were non-randomly selected to participate in the study. Each participant performed the required task individually in a 90-minute session. During the session, a participant was required to read the first chapter of a book about virtual instruction, published on the Net Library website. Each participant was asked to think aloud while reading the text in order to inform the researcher of the strategies he used during reading and to report any usability problems he may have encountered. The computer screen was videotaped during the task in order to capture any additional information about each participant's use of the strategies. After participants finished reading the text, they performed a retelling task in which they retold the text in writing covering all the knowledge acquired from the text. Results Discussion Analysis of data resulted on defining 19 web-related reading strategy (WRRS) classified in the following categories: approaching the text, modifying, browsing/navigating, personalized, and evaluating, and 19 text-related strategies classified in the following categories: basic reading, text-reader interaction, personalized, and evaluating. The mean score for participants in the retelling task was defined as 21 stated and detailed ideas out of 64 possible ideas mentioned in the text. Participants also reported two types of usability problems defined as design-related usability problems, and navigation-related usability problems. Recommendations Based on the results, a number of recommendations were suggested for both websites design such as user recognition techniques and supporting of annotations, and for web-based learning such as testing students for WRRS and TRRS and train them for these types of strategies upon enrolling in web-based courses that require fair amount of online reading. Comments: Softcopy Dissertation is scanned version; cannot copy and paste; cannot easily blog about it. |
Showing posts with label usability problems. Show all posts
Showing posts with label usability problems. Show all posts
Saturday, September 25, 2010
20100926 - Elshair, Strategies..to read educational websites, relation to web usability...
Friday, September 18, 2009
Sep 18 - Nielsen, Top 10 Information Architecture Mistakes (Alertbox)
Top 10 Information Architecture Mistakes
Summary: Structure and navigation must support each other and integrate with search and across subsites. Complexity, inconsistency, hidden options, and clumsy UI mechanics prevent users from finding what they need.
Introduction
Bad information architecture causes the majority of outright user failures and isn't improving at the rate of other Web usability issues. To determine why, I've identified 10 long-term sore thumbs that together cost websites billions of dollars each year.
I divided the following list of worst IA mistakes into two parts, structure and navigation. ...The invisible way the site is structured and the visible way users understand and manage that structure.
Structure Mistakes
1. No Structure
...common on news sites and catalog-based e-commerce sites, where each item (articles and products, respectively) is treated as a stand-alone unit without connections to related items. No wonder users leave those sites so quickly.
2. Search and Structure Not Integrated
We've long known that users often exhibit search-dominant behaviors.
SERP (search engine results page) usability increases when each search hit exposes its location within the site structure. External search engines like Google can't always do this because they don't know the site's structure or which navigational dimensions are most relevant to common site tasks. But you do know your site's structure and should therefore include the info on your own SERPs.
Sadly, search and navigation fail to support each other on many sites. This problem is exacerbated by another common mistake: navigation designs that don't indicate the user's current location. That is, after users click a search result, they can't determine where they are in the site — as when you're searching for pants and click on a pair, but then have no way to see more pants.
3. Missing Category Landing Pages
We recommend that sites have a series of categories that each link to their own landing page that gives users a section overview. Sometimes, sites forego the overview page and simply offer links directly to individual pages within a section. This might reduce the number of site pages, but when no page is clearly identified as a sub-topic page, users can misunderstand the site's scope and miss important details, products, and services.
Category pages also help SEO because they're the most prominent landing place when people search for a type of product, service, or information. (Breadcrumbs facilitate users' ability to easily move up the levels.)
4. Extreme Polyhierarchy
...polyhierarchy can easily become a crutch. Rather than spend time upfront to develop several intuitive and logical top-level categories, teams rush through this important process, creating numerous weak categories and listing products multiple times within them. The usability impact? Users spend too much time agonizing over top-level categories and then get confused when they see items showing up in multiple places ("are these the same thing?").
With too many classification options and too many structured dimensions, users are forced to think harder to move forward.
5. Subsites/Microsites Poorly Integrated with Main Site
It's typically best to forego independent microsites and place new information on subsites within the main site. But you still need to integrate these subsites within the overall site structure.
For example, on both microsites and subsites, we often see product-specific pages that fail to link to information about the company or organization behind the offering. Further, many sites poorly represent their subsites in the main site search — which often ignores microsites altogether.
Navigation Mistakes
6. Invisible Navigation Options
The very worst mistake might be to have no navigation, but that's so rare that I'm not going to discuss it. Still, any feature that users can't see might as well not exist; invisible navigation is thus nearly as bad as no navigation.
Uncovering navigation shouldn't be a major task: Make it permanently visible on the page. Small children like minesweeping (passing the mouse around the screen to see what's hidden), but teenagers don't like it, and adults hate it.
7. Uncontrollable Navigation Elements
Two common offenders here are overly sensitive rollovers that launch and block content, and elements that move, spin, or rotate of their own accord. Users routinely complain about these types of elements. Designers and programmers who include them in websites severely underestimate the business impact of user frustration.
8. Inconsistent Navigation
Navigation exists to help users, not to be a puzzle in its own right. Users should be able to understand it immediately, and apply that understanding throughout the site. Sadly, lots of sites change their navigation features as users move around. Options come and go, making users feel a loss of control. How do I get that menu choice back? I saw it just a few pages ago.
9. Too Many Navigation Techniques
Our full-day seminar on navigation design covers 25 different website navigation techniques. Each approach has its own usability advantages and potential downsides, leading to the seminar's focus on design trade-offs — that is, when to use what form of navigation.
One thing is clear: each navigation technique has its place on certain types of websites and intranets. But, if you use them all, you don't get the sum of each technique's benefits. You get a mess.
10. Made-Up Menu Options
...made-up navigation terms also hurt search; users can't find something if they don't know what it's called.
Old words are better. When users understand their choices, they're more likely to pick the right one. Speak plainly and speak simply. If users don't understand a menu item, they're less likely to click on it. Paradoxically, companies are particularly prone to making up fancy terms for their newest and most important offerings, thus shooting themselves in the foot with a double-barreled rifle.
source:
Jakob Nielsen's Alertbox, May 11, 2009:
Top 10 Information Architecture Mistakes
http://www.useit.com/alertbox/ia-mistakes.html
Top-10 Information Architecture (IA) Mistakes (Jakob Nielsen's Alertbox)
Summary: Structure and navigation must support each other and integrate with search and across subsites. Complexity, inconsistency, hidden options, and clumsy UI mechanics prevent users from finding what they need.
Introduction
Bad information architecture causes the majority of outright user failures and isn't improving at the rate of other Web usability issues. To determine why, I've identified 10 long-term sore thumbs that together cost websites billions of dollars each year.
I divided the following list of worst IA mistakes into two parts, structure and navigation. ...The invisible way the site is structured and the visible way users understand and manage that structure.
Structure Mistakes
1. No Structure
...common on news sites and catalog-based e-commerce sites, where each item (articles and products, respectively) is treated as a stand-alone unit without connections to related items. No wonder users leave those sites so quickly.
2. Search and Structure Not Integrated
We've long known that users often exhibit search-dominant behaviors.
SERP (search engine results page) usability increases when each search hit exposes its location within the site structure. External search engines like Google can't always do this because they don't know the site's structure or which navigational dimensions are most relevant to common site tasks. But you do know your site's structure and should therefore include the info on your own SERPs.
Sadly, search and navigation fail to support each other on many sites. This problem is exacerbated by another common mistake: navigation designs that don't indicate the user's current location. That is, after users click a search result, they can't determine where they are in the site — as when you're searching for pants and click on a pair, but then have no way to see more pants.
3. Missing Category Landing Pages
We recommend that sites have a series of categories that each link to their own landing page that gives users a section overview. Sometimes, sites forego the overview page and simply offer links directly to individual pages within a section. This might reduce the number of site pages, but when no page is clearly identified as a sub-topic page, users can misunderstand the site's scope and miss important details, products, and services.
Category pages also help SEO because they're the most prominent landing place when people search for a type of product, service, or information. (Breadcrumbs facilitate users' ability to easily move up the levels.)
4. Extreme Polyhierarchy
...polyhierarchy can easily become a crutch. Rather than spend time upfront to develop several intuitive and logical top-level categories, teams rush through this important process, creating numerous weak categories and listing products multiple times within them. The usability impact? Users spend too much time agonizing over top-level categories and then get confused when they see items showing up in multiple places ("are these the same thing?").
With too many classification options and too many structured dimensions, users are forced to think harder to move forward.
5. Subsites/Microsites Poorly Integrated with Main Site
It's typically best to forego independent microsites and place new information on subsites within the main site. But you still need to integrate these subsites within the overall site structure.
For example, on both microsites and subsites, we often see product-specific pages that fail to link to information about the company or organization behind the offering. Further, many sites poorly represent their subsites in the main site search — which often ignores microsites altogether.
Navigation Mistakes
6. Invisible Navigation Options
The very worst mistake might be to have no navigation, but that's so rare that I'm not going to discuss it. Still, any feature that users can't see might as well not exist; invisible navigation is thus nearly as bad as no navigation.
Uncovering navigation shouldn't be a major task: Make it permanently visible on the page. Small children like minesweeping (passing the mouse around the screen to see what's hidden), but teenagers don't like it, and adults hate it.
7. Uncontrollable Navigation Elements
Two common offenders here are overly sensitive rollovers that launch and block content, and elements that move, spin, or rotate of their own accord. Users routinely complain about these types of elements. Designers and programmers who include them in websites severely underestimate the business impact of user frustration.
8. Inconsistent Navigation
Navigation exists to help users, not to be a puzzle in its own right. Users should be able to understand it immediately, and apply that understanding throughout the site. Sadly, lots of sites change their navigation features as users move around. Options come and go, making users feel a loss of control. How do I get that menu choice back? I saw it just a few pages ago.
9. Too Many Navigation Techniques
Our full-day seminar on navigation design covers 25 different website navigation techniques. Each approach has its own usability advantages and potential downsides, leading to the seminar's focus on design trade-offs — that is, when to use what form of navigation.
One thing is clear: each navigation technique has its place on certain types of websites and intranets. But, if you use them all, you don't get the sum of each technique's benefits. You get a mess.
10. Made-Up Menu Options
...made-up navigation terms also hurt search; users can't find something if they don't know what it's called.
Old words are better. When users understand their choices, they're more likely to pick the right one. Speak plainly and speak simply. If users don't understand a menu item, they're less likely to click on it. Paradoxically, companies are particularly prone to making up fancy terms for their newest and most important offerings, thus shooting themselves in the foot with a double-barreled rifle.
source:
Jakob Nielsen's Alertbox, May 11, 2009:
Top 10 Information Architecture Mistakes
http://www.useit.com/alertbox/ia-mistakes.html
Top-10 Information Architecture (IA) Mistakes (Jakob Nielsen's Alertbox)
Thursday, September 17, 2009
Sep 18 - Nielsen, Top Ten Mistakes in Web Design (Alertbox)
Top Ten Mistakes in Web Design
Summary:
The ten most egregious offenses against users. Web design disasters and HTML horrors are legion, though many usability atrocities are less common than they used to be.
1. Bad Search
Search is the user's lifeline when navigation fails. Even though advanced search can sometimes help, simple search usually works best, and search should be presented as a simple box, since that's what users are looking for.
2. PDF Files for Online Reading
PDF is great for printing and for distributing manuals and other big documents that need to be printed. Reserve it for this purpose and convert any information that needs to be browsed or read on the screen into real web pages.
> Detailed discussion of why PDF is bad for online reading
3. Not Changing the Color of Visited Links
A good grasp of past navigation helps you understand your current location, since it's the culmination of your journey. Knowing your past and present locations in turn makes it easier to decide where to go next. Most important, knowing which pages they've already visited frees users from unintentionally revisiting the same pages over and over again.
When visited links don't change color, users exhibit more navigational disorientation in usability testing and unintentionally revisit the same pages repeatedly.
> Usability implications of changing link colors
> Guidelines for showing links
4. Non-Scannable Text
Write for online, not print. To draw users into the text and support scannability, use well-documented tricks:
* subheads
* bulleted lists
* highlighted keywords
* short paragraphs
* the inverted pyramid
* a simple writing style, and
* de-fluffed language devoid of marketese.
> Eyetracking of reading patterns
5. Fixed Font Size
CSS style sheets unfortunately give websites the power to disable a Web browser's "change font size" button and specify a fixed font size. About 95% of the time, this fixed size is tiny, reducing readability significantly for most people over the age of 40.
Respect the user's preferences and let them resize text as needed. Also, specify font sizes in relative terms -- not as an absolute number of pixels.
6. Page Titles With Low Search Engine Visibility
The humble page title is your main tool to attract new visitors from search listings and to help your existing users to locate the specific pages that they need.
The page title is contained within the HTML [title] tag and is almost always used as the clickable headline for listings on search engine result pages (SERP). Search engines typically show the first 66 characters or so of the title, so it's truly microcontent.
Page titles are also used as the default entry in the Favorites when users bookmark a site. For your homepage, begin the with the company name, followed by a brief description of the site. Don't start with words like "The" or "Welcome to" unless you want to be alphabetized under "T" or "W."
For other pages than the homepage, start the title with a few of the most salient information-carrying words that describe the specifics of what users will find on that page. If all your page titles start with the same words, you have severely reduced usability for your multi-windowing users.
Taglines on homepages are a related subject: they also need to be short and quickly communicate the purpose of the site.
7. Anything That Looks Like an Advertisement
Unfortunately, users also ignore legitimate design elements that look like prevalent forms of advertising. Therefore, it is best to avoid any designs that look like advertisements.
Follow these rules:
* banner blindness means that users never fixate their eyes on anything that looks like a banner ad due to shape or position on the page
* animation avoidance makes users ignore areas with blinking or flashing text or other aggressive animations
* pop-up purges mean that users close pop-up windoids before they have even fully rendered; sometimes with great viciousness (a sort of getting-back-at-GeoCities triumph).
8. Violating Design Conventions
Consistency is one of the most powerful usability principles: when things always behave the same, users don't have to worry about what will happen. Instead, they know what will happen based on earlier experience. The more users' expectations prove right, the more they will feel in control of the system and the more they will like it. And the more the system breaks users' expectations, the more they will feel insecure.
Jakob's Law of the Web User Experience states that "users spend most of their time on other websites."
This means that they form their expectations for your site based on what's commonly done on most other sites. If you deviate, your site will be harder to use and users will leave.
9. Opening New Browser Windows
Designers open new browser windows on the theory that it keeps users on their site. But even disregarding the user-hostile message implied in taking over the user's machine, the strategy is self-defeating since it disables the Back button which is the normal way users return to previous sites. Users often don't notice that a new window has opened, especially if they are using a small monitor where the windows are maximized to fill up the screen. So a user who tries to return to the origin will be confused by a grayed out Back button.
Links that don't behave as expected undermine users' understanding of their own system. A link should be a simple hypertext reference that replaces the current page with new content. Users hate unwarranted pop-up windows.
10. Not Answering Users' Questions
The ultimate failure of a website is to fail to provide the information users are looking for.
The worst example of not answering users' questions is to avoid listing the price of products and services.
No B2C ecommerce site would make this mistake, but it's rife in B2B, where most "enterprise solutions" are presented so that you can't tell whether they are suited for 100 people or 100,000 people.
Price is the most specific piece of info customers use to understand the nature of an offering, and not providing it makes people feel lost and reduces their understanding of a product line.
Even B2C sites often make the associated mistake of forgetting prices in product lists, such as category pages or search results. Knowing the price is key in both situations; it lets users differentiate among products and click through to the most relevant ones.
source:
http://www.useit.com/alertbox/9605.html
Top 10 Mistakes in Web Design (Jakob Nielsen's Alertbox)
Jakob Nielsen's Alertbox:
Top Ten Mistakes in Web Design
Summary:
The ten most egregious offenses against users. Web design disasters and HTML horrors are legion, though many usability atrocities are less common than they used to be.
1. Bad Search
Search is the user's lifeline when navigation fails. Even though advanced search can sometimes help, simple search usually works best, and search should be presented as a simple box, since that's what users are looking for.
2. PDF Files for Online Reading
PDF is great for printing and for distributing manuals and other big documents that need to be printed. Reserve it for this purpose and convert any information that needs to be browsed or read on the screen into real web pages.
> Detailed discussion of why PDF is bad for online reading
3. Not Changing the Color of Visited Links
A good grasp of past navigation helps you understand your current location, since it's the culmination of your journey. Knowing your past and present locations in turn makes it easier to decide where to go next. Most important, knowing which pages they've already visited frees users from unintentionally revisiting the same pages over and over again.
When visited links don't change color, users exhibit more navigational disorientation in usability testing and unintentionally revisit the same pages repeatedly.
> Usability implications of changing link colors
> Guidelines for showing links
4. Non-Scannable Text
Write for online, not print. To draw users into the text and support scannability, use well-documented tricks:
* subheads
* bulleted lists
* highlighted keywords
* short paragraphs
* the inverted pyramid
* a simple writing style, and
* de-fluffed language devoid of marketese.
> Eyetracking of reading patterns
5. Fixed Font Size
CSS style sheets unfortunately give websites the power to disable a Web browser's "change font size" button and specify a fixed font size. About 95% of the time, this fixed size is tiny, reducing readability significantly for most people over the age of 40.
Respect the user's preferences and let them resize text as needed. Also, specify font sizes in relative terms -- not as an absolute number of pixels.
6. Page Titles With Low Search Engine Visibility
The humble page title is your main tool to attract new visitors from search listings and to help your existing users to locate the specific pages that they need.
The page title is contained within the HTML [title] tag and is almost always used as the clickable headline for listings on search engine result pages (SERP). Search engines typically show the first 66 characters or so of the title, so it's truly microcontent.
Page titles are also used as the default entry in the Favorites when users bookmark a site. For your homepage, begin the with the company name, followed by a brief description of the site. Don't start with words like "The" or "Welcome to" unless you want to be alphabetized under "T" or "W."
For other pages than the homepage, start the title with a few of the most salient information-carrying words that describe the specifics of what users will find on that page. If all your page titles start with the same words, you have severely reduced usability for your multi-windowing users.
Taglines on homepages are a related subject: they also need to be short and quickly communicate the purpose of the site.
7. Anything That Looks Like an Advertisement
Unfortunately, users also ignore legitimate design elements that look like prevalent forms of advertising. Therefore, it is best to avoid any designs that look like advertisements.
Follow these rules:
* banner blindness means that users never fixate their eyes on anything that looks like a banner ad due to shape or position on the page
* animation avoidance makes users ignore areas with blinking or flashing text or other aggressive animations
* pop-up purges mean that users close pop-up windoids before they have even fully rendered; sometimes with great viciousness (a sort of getting-back-at-GeoCities triumph).
8. Violating Design Conventions
Consistency is one of the most powerful usability principles: when things always behave the same, users don't have to worry about what will happen. Instead, they know what will happen based on earlier experience. The more users' expectations prove right, the more they will feel in control of the system and the more they will like it. And the more the system breaks users' expectations, the more they will feel insecure.
Jakob's Law of the Web User Experience states that "users spend most of their time on other websites."
This means that they form their expectations for your site based on what's commonly done on most other sites. If you deviate, your site will be harder to use and users will leave.
9. Opening New Browser Windows
Designers open new browser windows on the theory that it keeps users on their site. But even disregarding the user-hostile message implied in taking over the user's machine, the strategy is self-defeating since it disables the Back button which is the normal way users return to previous sites. Users often don't notice that a new window has opened, especially if they are using a small monitor where the windows are maximized to fill up the screen. So a user who tries to return to the origin will be confused by a grayed out Back button.
Links that don't behave as expected undermine users' understanding of their own system. A link should be a simple hypertext reference that replaces the current page with new content. Users hate unwarranted pop-up windows.
10. Not Answering Users' Questions
The ultimate failure of a website is to fail to provide the information users are looking for.
The worst example of not answering users' questions is to avoid listing the price of products and services.
No B2C ecommerce site would make this mistake, but it's rife in B2B, where most "enterprise solutions" are presented so that you can't tell whether they are suited for 100 people or 100,000 people.
Price is the most specific piece of info customers use to understand the nature of an offering, and not providing it makes people feel lost and reduces their understanding of a product line.
Even B2C sites often make the associated mistake of forgetting prices in product lists, such as category pages or search results. Knowing the price is key in both situations; it lets users differentiate among products and click through to the most relevant ones.
source:
http://www.useit.com/alertbox/9605.html
Top 10 Mistakes in Web Design (Jakob Nielsen's Alertbox)
Jakob Nielsen's Alertbox:
Top Ten Mistakes in Web Design
Monday, September 14, 2009
Sep 14 - Mobile Usabiity - Jakob Nielsen's Alertbox
Mobile Usability
The Mobile User Experience Is Miserable
In our mobile studies, the average success rate was 59%, which is admittedly higher than success rates in the 1990s, but substantially lower than the roughly 80% success rate when testing websites on a regular PC today.
Main Mobile Problems
Mobile users face four main usability hurdles:
Small screens. For something to be mobile, it must be easy to carry and thus relatively small. Small screens mean fewer visible options at any given time, requiring users to rely on their short-term memory to build an understanding of an online information space. This makes almost all interactions harder. It's also difficult to find room for multiple windows or other interface solutions that support advanced behaviors, such as comparative product research.
Awkward input, especially for typing. It's hard to operate GUI widgets without a mouse: menus, buttons, hypertext links, and scrolling all take longer time and are more error-prone, whether they're touch-activated or manipulated with a teensy trackball. Text entry is particularly slow and littered with typos, even on devices with dedicated mini-keyboards.
Download delays. Getting the next screen takes forever — often longer than it would on dial-up, even with a supposedly faster 3G service.
Mis-designed sites. Because websites are typically optimized for desktop usability, they don't follow the guidelines necessary for usable mobile access.
Mobile Sites Beat Full Sites
When our test participants used sites that were designed specifically for mobile devices, their success rate averaged 64%, which is substantially higher than the 53% recorded for using "full" sites — that is, the same sites that desktop users see.
Improving user performance by 1/5 is reason enough to create mobileoptimized sites. Such sites were also more pleasant to use and thus received higher subjective satisfaction ratings. This fact offers an additional rationale:
When users are successful and satisfied, they're likely to come back. So, if mobile use is important to your Internet strategy, it's smart to build a dedicated mobile site.
Still, users often had trouble getting to mobile sites, even when companies offered them. The best approach is to auto-sense users' devices and autoforward mobile users to the mobile site (even if they're using a high-end phone). You should also offer clear links from the desktop site to the mobile site, as well as a link back to the full site. As for link labels, we recommend
"Mobile Site" and "Full Site," respectively.
Better Phones Perform Better
There are 3 distinct classes of mobile user experience, and they're mainly
defined by screen size:
Feature phones (regular cellphones) with a tiny screen and a numeric
keypad. These devices account for the vast majority of the market (at
least 85% in some statistics).
Smartphones, in a range of form factors, typically with a mid-sized
screen and a full A-Z keypad.
Touch-screen phones (such as the iPhone) with a nearly device-sized
screen and a true GUI driven by direct manipulation and touch gestures.
Unsurprisingly, the bigger the screen, the better the user experience when
accessing websites. Average success rates were:
Feature phones >> 38%
Smartphones >> 55%
Touch phones >> 75%
For services highly suited for mobile use — such as news or social networking — you should probably create a dedicated feature-phone site, as well as a site optimized for higher-end phones.
Most other websites might be better off concentrating their investment on a single mobile site optimized for smartphones and touch phones.
Mobile Usability Is Hard
All of our new research findings support a single conclusion: designing for mobile is hard. Technical accessibility is very far from providing an acceptable user experience. It's not enough that your site will display on a phone.
Even touch phones that offer "full-featured" browsers don't offer PC-level usability in terms of users' ability to actually get things done on a website.
When designing for mobile, there's a tension between (a) making content and navigation salient so that people do not work too hard to get there, and (b) designing for a small screen and for slow downloading speeds.
Unless websites are redesigned for the special circumstances of mobile use, the mobile Web will remain a mirage.
Source:
Mobile Usability (Jakob Nielsen's Alertbox)
Copyright © 2009 by Jakob Nielsen. ISSN 1548-5552
http://www.useit.com/alertbox/mobile-usability.html
The Mobile User Experience Is Miserable
In our mobile studies, the average success rate was 59%, which is admittedly higher than success rates in the 1990s, but substantially lower than the roughly 80% success rate when testing websites on a regular PC today.
Main Mobile Problems
Mobile users face four main usability hurdles:
Small screens. For something to be mobile, it must be easy to carry and thus relatively small. Small screens mean fewer visible options at any given time, requiring users to rely on their short-term memory to build an understanding of an online information space. This makes almost all interactions harder. It's also difficult to find room for multiple windows or other interface solutions that support advanced behaviors, such as comparative product research.
Awkward input, especially for typing. It's hard to operate GUI widgets without a mouse: menus, buttons, hypertext links, and scrolling all take longer time and are more error-prone, whether they're touch-activated or manipulated with a teensy trackball. Text entry is particularly slow and littered with typos, even on devices with dedicated mini-keyboards.
Download delays. Getting the next screen takes forever — often longer than it would on dial-up, even with a supposedly faster 3G service.
Mis-designed sites. Because websites are typically optimized for desktop usability, they don't follow the guidelines necessary for usable mobile access.
Mobile Sites Beat Full Sites
When our test participants used sites that were designed specifically for mobile devices, their success rate averaged 64%, which is substantially higher than the 53% recorded for using "full" sites — that is, the same sites that desktop users see.
Improving user performance by 1/5 is reason enough to create mobileoptimized sites. Such sites were also more pleasant to use and thus received higher subjective satisfaction ratings. This fact offers an additional rationale:
When users are successful and satisfied, they're likely to come back. So, if mobile use is important to your Internet strategy, it's smart to build a dedicated mobile site.
Still, users often had trouble getting to mobile sites, even when companies offered them. The best approach is to auto-sense users' devices and autoforward mobile users to the mobile site (even if they're using a high-end phone). You should also offer clear links from the desktop site to the mobile site, as well as a link back to the full site. As for link labels, we recommend
"Mobile Site" and "Full Site," respectively.
Better Phones Perform Better
There are 3 distinct classes of mobile user experience, and they're mainly
defined by screen size:
Feature phones (regular cellphones) with a tiny screen and a numeric
keypad. These devices account for the vast majority of the market (at
least 85% in some statistics).
Smartphones, in a range of form factors, typically with a mid-sized
screen and a full A-Z keypad.
Touch-screen phones (such as the iPhone) with a nearly device-sized
screen and a true GUI driven by direct manipulation and touch gestures.
Unsurprisingly, the bigger the screen, the better the user experience when
accessing websites. Average success rates were:
Feature phones >> 38%
Smartphones >> 55%
Touch phones >> 75%
For services highly suited for mobile use — such as news or social networking — you should probably create a dedicated feature-phone site, as well as a site optimized for higher-end phones.
Most other websites might be better off concentrating their investment on a single mobile site optimized for smartphones and touch phones.
Mobile Usability Is Hard
All of our new research findings support a single conclusion: designing for mobile is hard. Technical accessibility is very far from providing an acceptable user experience. It's not enough that your site will display on a phone.
Even touch phones that offer "full-featured" browsers don't offer PC-level usability in terms of users' ability to actually get things done on a website.
When designing for mobile, there's a tension between (a) making content and navigation salient so that people do not work too hard to get there, and (b) designing for a small screen and for slow downloading speeds.
Unless websites are redesigned for the special circumstances of mobile use, the mobile Web will remain a mirage.
Source:
Mobile Usability (Jakob Nielsen's Alertbox)
Copyright © 2009 by Jakob Nielsen. ISSN 1548-5552
http://www.useit.com/alertbox/mobile-usability.html
Sunday, September 6, 2009
Sep 7 - Yeap, Severity of Usability Problems (part 8 of UIC for eLearn)
Nielsen's Heuristic Evaluation Methodology: Severity of Usability Problems
Blogging part 8 of USABILITY INSPECTION CRITERIA FOR E-LEARNING PORTALS.
For the practice of heuristic evaluation, usability problems are categorized into major or minor problems.
Alternatively, severity rating is assigned for each usability problem after the usability problem detected by evaluators is aggregated.
Severity of usability problems could be rated using 0 to 4 rating scale.
“ 0 = I don't agree that this is a usability problem at all.
1 = Cosmetic problem only: need not be fixed unless extra time is available on project.
2 = Minor usability problem: fixing this should be given low priority.
3 = Major usability problem: important to fix, so should be given high priority.
4 = Usability catastrophe: imperative to fix this before product can be released.” (Nielsen, 2005d)
source:
Usability Inspection Criteria for e-Learning Portals.
TeckChong Yeap. MDP7515 PROJECT Dissertation. MULTIMEDIA UNIVERSITY, MALAYSIA, October 2008.
Blogging part 8 of USABILITY INSPECTION CRITERIA FOR E-LEARNING PORTALS.
For the practice of heuristic evaluation, usability problems are categorized into major or minor problems.
Alternatively, severity rating is assigned for each usability problem after the usability problem detected by evaluators is aggregated.
Severity of usability problems could be rated using 0 to 4 rating scale.
“ 0 = I don't agree that this is a usability problem at all.
1 = Cosmetic problem only: need not be fixed unless extra time is available on project.
2 = Minor usability problem: fixing this should be given low priority.
3 = Major usability problem: important to fix, so should be given high priority.
4 = Usability catastrophe: imperative to fix this before product can be released.” (Nielsen, 2005d)
source:
Usability Inspection Criteria for e-Learning Portals.
TeckChong Yeap. MDP7515 PROJECT Dissertation. MULTIMEDIA UNIVERSITY, MALAYSIA, October 2008.
Saturday, September 5, 2009
Sep 5 - Capra, Usability Problem Description and the Evaluator Effect in Usability Testing
Usability Problem Description and the Evaluator Effect in Usability Testing.
Miranda G. Capra
Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Industrial and Systems Engineering
March 13, 2006
Blacksburg, Virginia
ABSTRACT .
Previous usability evaluation method (UEM) comparison studies have noted an evaluator effect on problem detection in heuristic evaluation, with evaluators differing in problems found and problem severity judgments. There have been few studies of the evaluator effect in usability testing (UT), task-based testing with end-users. UEM comparison studies focus on counting usability problems detected, but we also need to assess the content of usability problem descriptions (UPDs) to more fully measure evaluation effectiveness. The goals of this research were to develop UPD guidelines, explore the evaluator effect in UT, and evaluate the usefulness of the guidelines for grading UPD content.
Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are (briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the reader sympathize with the user. A fourth study compared usability reports collected from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator and graded the reports for following 6 of the UPD guidelines.
There was support for existence of an evaluator effect, even when watching prerecorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). Practitioners received higher grades averaged across the 6 guidelines than students did, suggesting that the guidelines may be useful for grading reports. The grades for the guidelines were not correlated with thoroughness, suggesting that the guideline grades complement measures of problem detection.
A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The simulation also found that thoroughness of individual evaluators would have been overestimated if the study had included a small number of evaluators. The final recommendations are to use multiple evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness.
INTRODUCTION
Formative usability evaluations are an important part of the usability life cycle, identifying usability problems present in an interface that designers should fix in the next design iteration.
Figure 1.1 provides an overview of a formative usability evaluation and the usability life cycle.
The output of a formative usability evaluation is a set of usability problems (Hartson, Andre, & Williges, 2003), but there are many different usability evaluation methods (UEMs).
Empirical evaluations involve end-users. The most common empirical method is usability testing or think aloud testing, which is generally a taskbased session in a usability laboratory.
Analytical evaluations involve expert review of an interface, such as Heuristic Evaluation (Nielsen, 1994b; Nielsen & Molich, 1990).
Formative usability evaluation is not a reliable process.
Evaluators discover different sets of usability problems depending on the usability evaluation method (UEM) used or the individual evaluator that performs an analytical evaluation (Hertzum & Jacobsen, 2003).
Other factors that can affect problem detection are the number and type of users involved in usability testing (Law & Hvannberg, 2004a; Nielsen, 1994a; Spool & Schroeder, 2001; Virzi, 1990, 1992) and the number of evaluators involved in an expert review (Dumas & Sorce, 1995; Dutt, Johnson, & Johnson, 1994).
Evaluators also differ in their judgment of the severity of usability problems (Hassenzahl, 2000; Hertzum & Jacobsen, 2003).
Hertzum and Jacobsen coined the term evaluator effect to refer to differences in problem detection and severity judgments by evaluators using the same UEM (Hertzum & Jacobsen, 2003; Jacobsen, Hertzum, & John, 1998).
Most studies comparing UEM effectiveness or measuring the evaluator effect rely on measures that involve counting usability problems identified by each evaluator, such as the any-two agreement measure of reliability (Hertzum & Jacobsen, 2003) or the thoroughness and validity of problem sets (Hartson et al., 2003). These measures are useful, but they do not give a complete measure of the effectiveness of an evaluation.
Counting usability problems detected is part of the equation, but a better question to ask is how can an evaluation help to efficiently improve a product (Wixon, 2003).
Communicating the results of the test (either through a written report or verbally) is an essential part of the usability testing process (Rubin, 1994).
Andre, Hartson, Belz, and McCreary (2001) express the opinion that poor documentation and communication of usability problems identified diminish the effectiveness of a usability evaluation and can reduce the return on the effort invested in conducting the evaluation.
Dumas, Molich and Jeffries (2004) suggest that communication style and attitude in report writing can affect recipients’ acceptance of suggestions and the number of the problems recipients choose to fix.
Jeffries (1994) suggests that developers may interpret poorly described problems as false alarms, causing the developers to ignore the poorly described problems and increasing the likelihood that developers will treat future problems as opinion or false alarms.
A more complete measure of UEM output would assess not only the quantity of problem descriptions generated but also the quality.
There has been little formal research into usability problem description.
Many authors have developed structured problem reporting forms and problem classification schemes (Andre et al., 2001; Cockton, Woolrych, & Hindmarch, 2004; Hvannberg & Law, 2003; Lavery, Cockton, & Atkinson, 1997; Sutcliffe, 2000) to increase the utility of problem descriptions and thoroughness of problem diagnosis.
However, these studies have not provided formal documentation of poor problem descriptions, nor have they provided measures of problem description quality to demonstrate the effectiveness of these tools.
1.1 Problem Statement
Formative usability evaluation is not a reliable process. There is evidence of an evaluator effect on problem detection in analytical methods and a user effect in usability testing. Usability testing had been considered the gold standard of usability evaluation, but there have been few studies of whether it is also subject to an evaluator effect.
Previous studies of the evaluator effect on problem detection in usability testing have allowed different tasks and users, used a small sample size, or used student teams; we need larger studies of the evaluator effect that focus specifically on study analysis, rather than design and execution.
In addition, previous studies of usability testing have focused primarily on comparing the number of problems identified by each evaluation. Counting usability problems identified by an evaluation is a necessary but not sufficient measure of evaluation effectiveness.
We need measures of usability problem description (UPD) content to be able to document shortcomings in UPDs and more fully measure evaluation effectiveness.
1.2 Goals
This research had three goals:
(1) develop guidelines for describing usability problems,
(2) explore the evaluator effect in usability testing, and
(3) evaluate the usefulness of the guidelines for judging the content of usability problem descriptions.
This research should lead to a better understanding of usability problem descriptions and the evaluator affecting usability testing, and provide a basis for future studies to formally develop metrics of usability problem description quality.
Research Question 1: How do practitioners describe usability problems?
Research Question 2: Is there an evaluator effect in usability testing?
Research Question 3: How can we assess the content of UPDs?
1.3 Approach
The first phase of this study focused on developing guidelines for describing usability problems.
There is little discussion of UPDs in usability articles or textbooks to form the basis of guidelines. Instead, research used an exploratory approach to develop guidelines, consulting usability practitioners about important qualities of UPDs with a series of questionnaires.
The second phase focused on the evaluator effect in usability testing.
There have been 10 previous studies of the evaluator effect in usability testing. .....
The second phase also served as a preliminary assessment of use of the guidelines to assess the content of usability reports. The guidelines from Phase I were used to evaluate the usability reports collected in Phase II. Assessment measures included the extent to which practitioners followed the guidelines, if following the guidelines was an indicator of practical usability experience, and comparing opinions about the importance of the guidelines with behavior in terms of following the guidelines when writing the usability reports.
Table 1.1 summarizes the goals and approach of this research, and Table 1.2 summarizes the phases and outputs.
My Comments: Capra had used the 2 tables to systematically explain on the Problems, Goals, Approach, Chapters, Phases, Studies, RQ and Output (deliverables).
Table 1.1 Research problems, goals and approach
Table 1.2 Summary of Chapters, Phases and Studies
2.4 Discussion
Table 2.12 Ten Guidelines for Describing Usability Problems
1 Be clear and precise while avoiding wordiness and jargon.
Define terms that you use. Be concrete, not vague. Be practical, not theoretical. Use descriptions that non-HCI people will appreciate. Avoid so much detail that no one will want to read the description.
2 Describe the impact and severity of the problem,
including business effects (support costs, time loss, etc.), impact on the user's task and importance of the task. Describe how often the problem will occur, and system components that are affected or involved.
3 Support your findings with data
such as: how many users experienced the problem and how often; task attempts, time and success/failure; critical incident descriptions; and other objective data, both quantitative and qualitative. Provide traceability of the problem to observed data.
4 Describe the cause of the problem,
including context such as the interaction architecture and the user's task. Describe the main usability issue involved in the problem. Avoid guessing about the problem cause or user's thoughts.
5 Describe observed user actions,
including specific examples from the study, such as the user's navigation flow through the system, user's subjective reactions, screen shots and task success/failure. Mention whether the problem was user-reported or experimenter observed.
6 Consider politics and diplomacy
when writing your description. Avoid judging the system, criticizing decisions made by other team members, pointing fingers or assigning blame. Point out good design elements and successful user interactions. Be practical, avoiding theory and jargon.
7 Be professional and scientific in your description.
Use only facts from the study, rather than opinions or guesses. Back your findings with sources beyond the current study, such as external classification scheme, proven usability design principles, and previous research.
8 Describe a solution to the problem,
providing alternatives and tradeoffs. Be specific enough to be helpful without dictating a solution, guessing, or jumping to conclusions. Supplement with pictures, screen capture, usability design principles and/or previous research.
9 Describe your methodology and background.
Describe how you found this problem (field study, lab study, expert evaluation, etc.). Describe the limitations of your domain knowledge. Describe the user groups that were affected and the breadth of system components involved.
10 Help the reader sympathize with the user's problem
by using descriptions that are evocative and anecdotal. Make sure the description is readable and understandable. Use user-centric language rather than system-centric. Be complete while avoiding excessive detail.
2.4.2 Applications
These guidelines will be useful for usability practitioners, instructors and
researchers.
· Usability practitioners can use this list to create usability problem reporting forms, to create checklists of items to address in writing problem descriptions, or to evaluate usability reports generated in usability studies. Usability groups could evaluate their work products to ensure that they are writing effective problem descriptions in usability reports.
· Usability instructors can use them to explain what should be in a usability problem description and to grade usability evaluations conducted by students. Following as many of the guidelines as possible would be an appropriate exercise for students, where thoroughness of the report is more important than the time spent on it. Usability students need to practice writing complete descriptions so that they will learn what information they need to take note of during a usability evaluation. Training of new practitioners could include more practice and review opportunities for the guidelines rated as more difficult.
· Usability researchers can use the guidelines to assess problem descriptions generated by different usability practitioners or usability evaluation methods. Current research in UEM effectiveness focuses on counting usability problems identified in evaluations, but more effort should be focused on the quality of descriptions as well as the quantity. This was explored further in Phase II.
The most important suggestion for using the guidelines is to select the guidelines
that fit your project and organization.
My Comments: I did literature review on Miranda Capra's PhD Dissertation when I did my Master's Project Dissertation.
2.6 Conclusion
The output of Phase I was a set of 10 guidelines for describing usability problems.
The 10 guidelines presented in this section are suggestions, not rules. It would be difficult
and time-consuming to follow every single guideline for every single problem description. Different guidelines may be more or less important for different projects, clients, and organizations.
References that I may want to read further in future:
ANSI. (2001). Common Industry Format for Usability Test Reports (ANSI NCITS 354-2001). New York: author.
Bastien, J. M. C., Scapin, D. L., & Leulier, C. (1996). Looking for usability problems with the ergonomic criteria and with the ISO 9241-10 dialogue principles. In M. J. Tauber & V. Bellotti & R. Jeffries & J. D. Mackinlay & J. Nielsen (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI '96) (pp. 77-78). New York: ACM.
Chin, J. P., Diehl, V. A., & Norman, L. K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In E. Soloway & D. Frye & S. B. Sheppart (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI '88) (pp. 213-218). New York: ACM.
Connell, I. W., & Hammond, N. V. (1999). Comparing Usability Evaluation Principles with Heuristics: Problem Instances vs. Problem Types. In M. A. Sasse & C. Johnson (Eds.), Proceedings of the Human Computer Interaction - INTERACT '99 (pp. 621-629): IOS.
De Angeli, A., Matera, M., Costabile, M. F., Garzotto, F., & Paolini, P. (2000). Validating the SUE Inspection Technique. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI 2000) (pp. 143-150). New York: ACM.
Dumas, J. S., Molich, R., & Jeffries, R. (2004). Describing usability problems: are we sending the right message? interactions, 11(4), 24-29.
Hassenzahl, M. (2000). Prioritizing usability problems: data-driven and judgement-driven severity estimates. Behaviour & Information Technology, 19(1), 29-42.
Hornbæk, K., & Frøkjær, E. (2005). Comparing Usability Problems and Redesign Proposals as Input to Practical Systems Development. In Proceedings of the Conference on Human factors in computing systems (CHI 2005) (pp. 391-400). New York: ACM.
Hvannberg, E. T., & Law, L.-C. (2003). Classification of Usability Problems (CUP) Scheme. In M. Rauterberg (Ed.) Proceedings of the Human-Computer Interaction - INTERACT'03 (pp. 655-662): IOS.
ISO. (1998). Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability (ISO 9241-11:1998(E)). Geneva: author.
ISO. (1999a). Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 10: Dialogue principles (ISO 9241-10:1998(E)). Geneva: author.
ISO. (1999b). Human -centred design processes for interactive systems (ISO 134071999(E)). Geneva: author.
Jacobsen, N. E., Hertzum, M., & John, B. E. (1998). The Evaluator Effect in Usability Tests. In C.-M. Karat & A. Lund & J. Coutaz & J. Karat (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI 98) (pp. 255-256). New York: ACM.
Jeffries, R. (1994). Usability Problem Reports: Helping Evaluators Communicate Effectively with Developers. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 273-294). New York: John Wiley.
Keenan, S. L., Hartson, H. R., Kafura, D. G., & Schulman, R. S. (1999). The usability problem taxonomy: A framework for classification and analysis. Empirical Software Engineering, 4(1), 71-104.
Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16(4/5), 246-266.
Mack, R., & Montaniz, F. (1994). Observing, Predicting, and Analyzing Usability Problems. In R. L. Mack & J. Nielsen (Eds.), Usability Inspection Methods (pp. 295-340). New York: John Wiley & Sons.
Nielsen, J. (1992). Finding Usability Problems Through Heuristic Evaluation. In P. Bauersfeld & J. Bennett & G. Lynch (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI 92) (pp. 373-380). New York: ACM.
Nielsen, J., & Landauer, T. K. (1993). A Mathematical model of the finding of usability problems. In S. Ashlund & K. Mullet & A. Henderson & E. Hollnagel & T. White (Eds.), Proceedings of the INTERCHI 93: Conference on Human Factors in Computing Systems (INTERACT 93 and CHI 93) (pp. 206-213). New York: ACM.
Rourke, C. (2003). CUE-4: Lessons in Best Practice for Usability Testing and Expert Evaluation. User Vision. Retrieved December 14, 2004 from http://www.uservision.co.uk/usability_articles/usability_CUE.asp .
Skov, M. B., & Stage, J. (2005). Supporting problem identification in usability evaluations. In Proceedings of the 19th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: citizens online: considerations for today and the future (OZCHI 2005). Narrabundah, Australia: Computer-Human Interaction Special Interest Group (CHISIG) of Australia.
Theofanos, M., & Quesenbery, W. (2005). Towards the Design of Effective Formative Test Reports. Journal of Usability Studies, 1(1), 27-45.
Theofanos, M., Quesenbery, W., Snyder, C., Dayton, D., & Lewis, J. (2005). Reporting
on Formative Testing. A UPA 2005 Workshop Report. Bloomingdale, IL: UPA. Retrieved on October 3, 2005 from the UPA website http://www.upassoc.org/usability_resources/conference/2005/formative%20reporting-upa2005.pdf.
Miranda G. Capra
Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Industrial and Systems Engineering
March 13, 2006
Blacksburg, Virginia
ABSTRACT .
Previous usability evaluation method (UEM) comparison studies have noted an evaluator effect on problem detection in heuristic evaluation, with evaluators differing in problems found and problem severity judgments. There have been few studies of the evaluator effect in usability testing (UT), task-based testing with end-users. UEM comparison studies focus on counting usability problems detected, but we also need to assess the content of usability problem descriptions (UPDs) to more fully measure evaluation effectiveness. The goals of this research were to develop UPD guidelines, explore the evaluator effect in UT, and evaluate the usefulness of the guidelines for grading UPD content.
Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are (briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the reader sympathize with the user. A fourth study compared usability reports collected from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator and graded the reports for following 6 of the UPD guidelines.
There was support for existence of an evaluator effect, even when watching prerecorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). Practitioners received higher grades averaged across the 6 guidelines than students did, suggesting that the guidelines may be useful for grading reports. The grades for the guidelines were not correlated with thoroughness, suggesting that the guideline grades complement measures of problem detection.
A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The simulation also found that thoroughness of individual evaluators would have been overestimated if the study had included a small number of evaluators. The final recommendations are to use multiple evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness.
INTRODUCTION
Formative usability evaluations are an important part of the usability life cycle, identifying usability problems present in an interface that designers should fix in the next design iteration.
Figure 1.1 provides an overview of a formative usability evaluation and the usability life cycle.
The output of a formative usability evaluation is a set of usability problems (Hartson, Andre, & Williges, 2003), but there are many different usability evaluation methods (UEMs).
Empirical evaluations involve end-users. The most common empirical method is usability testing or think aloud testing, which is generally a taskbased session in a usability laboratory.
Analytical evaluations involve expert review of an interface, such as Heuristic Evaluation (Nielsen, 1994b; Nielsen & Molich, 1990).
Formative usability evaluation is not a reliable process.
Evaluators discover different sets of usability problems depending on the usability evaluation method (UEM) used or the individual evaluator that performs an analytical evaluation (Hertzum & Jacobsen, 2003).
Other factors that can affect problem detection are the number and type of users involved in usability testing (Law & Hvannberg, 2004a; Nielsen, 1994a; Spool & Schroeder, 2001; Virzi, 1990, 1992) and the number of evaluators involved in an expert review (Dumas & Sorce, 1995; Dutt, Johnson, & Johnson, 1994).
Evaluators also differ in their judgment of the severity of usability problems (Hassenzahl, 2000; Hertzum & Jacobsen, 2003).
Hertzum and Jacobsen coined the term evaluator effect to refer to differences in problem detection and severity judgments by evaluators using the same UEM (Hertzum & Jacobsen, 2003; Jacobsen, Hertzum, & John, 1998).
Most studies comparing UEM effectiveness or measuring the evaluator effect rely on measures that involve counting usability problems identified by each evaluator, such as the any-two agreement measure of reliability (Hertzum & Jacobsen, 2003) or the thoroughness and validity of problem sets (Hartson et al., 2003). These measures are useful, but they do not give a complete measure of the effectiveness of an evaluation.
Counting usability problems detected is part of the equation, but a better question to ask is how can an evaluation help to efficiently improve a product (Wixon, 2003).
Communicating the results of the test (either through a written report or verbally) is an essential part of the usability testing process (Rubin, 1994).
Andre, Hartson, Belz, and McCreary (2001) express the opinion that poor documentation and communication of usability problems identified diminish the effectiveness of a usability evaluation and can reduce the return on the effort invested in conducting the evaluation.
Dumas, Molich and Jeffries (2004) suggest that communication style and attitude in report writing can affect recipients’ acceptance of suggestions and the number of the problems recipients choose to fix.
Jeffries (1994) suggests that developers may interpret poorly described problems as false alarms, causing the developers to ignore the poorly described problems and increasing the likelihood that developers will treat future problems as opinion or false alarms.
A more complete measure of UEM output would assess not only the quantity of problem descriptions generated but also the quality.
There has been little formal research into usability problem description.
Many authors have developed structured problem reporting forms and problem classification schemes (Andre et al., 2001; Cockton, Woolrych, & Hindmarch, 2004; Hvannberg & Law, 2003; Lavery, Cockton, & Atkinson, 1997; Sutcliffe, 2000) to increase the utility of problem descriptions and thoroughness of problem diagnosis.
However, these studies have not provided formal documentation of poor problem descriptions, nor have they provided measures of problem description quality to demonstrate the effectiveness of these tools.
1.1 Problem Statement
Formative usability evaluation is not a reliable process. There is evidence of an evaluator effect on problem detection in analytical methods and a user effect in usability testing. Usability testing had been considered the gold standard of usability evaluation, but there have been few studies of whether it is also subject to an evaluator effect.
Previous studies of the evaluator effect on problem detection in usability testing have allowed different tasks and users, used a small sample size, or used student teams; we need larger studies of the evaluator effect that focus specifically on study analysis, rather than design and execution.
In addition, previous studies of usability testing have focused primarily on comparing the number of problems identified by each evaluation. Counting usability problems identified by an evaluation is a necessary but not sufficient measure of evaluation effectiveness.
We need measures of usability problem description (UPD) content to be able to document shortcomings in UPDs and more fully measure evaluation effectiveness.
1.2 Goals
This research had three goals:
(1) develop guidelines for describing usability problems,
(2) explore the evaluator effect in usability testing, and
(3) evaluate the usefulness of the guidelines for judging the content of usability problem descriptions.
This research should lead to a better understanding of usability problem descriptions and the evaluator affecting usability testing, and provide a basis for future studies to formally develop metrics of usability problem description quality.
Research Question 1: How do practitioners describe usability problems?
Research Question 2: Is there an evaluator effect in usability testing?
Research Question 3: How can we assess the content of UPDs?
1.3 Approach
The first phase of this study focused on developing guidelines for describing usability problems.
There is little discussion of UPDs in usability articles or textbooks to form the basis of guidelines. Instead, research used an exploratory approach to develop guidelines, consulting usability practitioners about important qualities of UPDs with a series of questionnaires.
The second phase focused on the evaluator effect in usability testing.
There have been 10 previous studies of the evaluator effect in usability testing. .....
The second phase also served as a preliminary assessment of use of the guidelines to assess the content of usability reports. The guidelines from Phase I were used to evaluate the usability reports collected in Phase II. Assessment measures included the extent to which practitioners followed the guidelines, if following the guidelines was an indicator of practical usability experience, and comparing opinions about the importance of the guidelines with behavior in terms of following the guidelines when writing the usability reports.
Table 1.1 summarizes the goals and approach of this research, and Table 1.2 summarizes the phases and outputs.
My Comments: Capra had used the 2 tables to systematically explain on the Problems, Goals, Approach, Chapters, Phases, Studies, RQ and Output (deliverables).
Table 1.1 Research problems, goals and approach
Table 1.2 Summary of Chapters, Phases and Studies
2.4 Discussion
Table 2.12 Ten Guidelines for Describing Usability Problems
1 Be clear and precise while avoiding wordiness and jargon.
Define terms that you use. Be concrete, not vague. Be practical, not theoretical. Use descriptions that non-HCI people will appreciate. Avoid so much detail that no one will want to read the description.
2 Describe the impact and severity of the problem,
including business effects (support costs, time loss, etc.), impact on the user's task and importance of the task. Describe how often the problem will occur, and system components that are affected or involved.
3 Support your findings with data
such as: how many users experienced the problem and how often; task attempts, time and success/failure; critical incident descriptions; and other objective data, both quantitative and qualitative. Provide traceability of the problem to observed data.
4 Describe the cause of the problem,
including context such as the interaction architecture and the user's task. Describe the main usability issue involved in the problem. Avoid guessing about the problem cause or user's thoughts.
5 Describe observed user actions,
including specific examples from the study, such as the user's navigation flow through the system, user's subjective reactions, screen shots and task success/failure. Mention whether the problem was user-reported or experimenter observed.
6 Consider politics and diplomacy
when writing your description. Avoid judging the system, criticizing decisions made by other team members, pointing fingers or assigning blame. Point out good design elements and successful user interactions. Be practical, avoiding theory and jargon.
7 Be professional and scientific in your description.
Use only facts from the study, rather than opinions or guesses. Back your findings with sources beyond the current study, such as external classification scheme, proven usability design principles, and previous research.
8 Describe a solution to the problem,
providing alternatives and tradeoffs. Be specific enough to be helpful without dictating a solution, guessing, or jumping to conclusions. Supplement with pictures, screen capture, usability design principles and/or previous research.
9 Describe your methodology and background.
Describe how you found this problem (field study, lab study, expert evaluation, etc.). Describe the limitations of your domain knowledge. Describe the user groups that were affected and the breadth of system components involved.
10 Help the reader sympathize with the user's problem
by using descriptions that are evocative and anecdotal. Make sure the description is readable and understandable. Use user-centric language rather than system-centric. Be complete while avoiding excessive detail.
2.4.2 Applications
These guidelines will be useful for usability practitioners, instructors and
researchers.
· Usability practitioners can use this list to create usability problem reporting forms, to create checklists of items to address in writing problem descriptions, or to evaluate usability reports generated in usability studies. Usability groups could evaluate their work products to ensure that they are writing effective problem descriptions in usability reports.
· Usability instructors can use them to explain what should be in a usability problem description and to grade usability evaluations conducted by students. Following as many of the guidelines as possible would be an appropriate exercise for students, where thoroughness of the report is more important than the time spent on it. Usability students need to practice writing complete descriptions so that they will learn what information they need to take note of during a usability evaluation. Training of new practitioners could include more practice and review opportunities for the guidelines rated as more difficult.
· Usability researchers can use the guidelines to assess problem descriptions generated by different usability practitioners or usability evaluation methods. Current research in UEM effectiveness focuses on counting usability problems identified in evaluations, but more effort should be focused on the quality of descriptions as well as the quantity. This was explored further in Phase II.
The most important suggestion for using the guidelines is to select the guidelines
that fit your project and organization.
My Comments: I did literature review on Miranda Capra's PhD Dissertation when I did my Master's Project Dissertation.
2.6 Conclusion
The output of Phase I was a set of 10 guidelines for describing usability problems.
The 10 guidelines presented in this section are suggestions, not rules. It would be difficult
and time-consuming to follow every single guideline for every single problem description. Different guidelines may be more or less important for different projects, clients, and organizations.
References that I may want to read further in future:
ANSI. (2001). Common Industry Format for Usability Test Reports (ANSI NCITS 354-2001). New York: author.
Bastien, J. M. C., Scapin, D. L., & Leulier, C. (1996). Looking for usability problems with the ergonomic criteria and with the ISO 9241-10 dialogue principles. In M. J. Tauber & V. Bellotti & R. Jeffries & J. D. Mackinlay & J. Nielsen (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI '96) (pp. 77-78). New York: ACM.
Chin, J. P., Diehl, V. A., & Norman, L. K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In E. Soloway & D. Frye & S. B. Sheppart (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI '88) (pp. 213-218). New York: ACM.
Connell, I. W., & Hammond, N. V. (1999). Comparing Usability Evaluation Principles with Heuristics: Problem Instances vs. Problem Types. In M. A. Sasse & C. Johnson (Eds.), Proceedings of the Human Computer Interaction - INTERACT '99 (pp. 621-629): IOS.
De Angeli, A., Matera, M., Costabile, M. F., Garzotto, F., & Paolini, P. (2000). Validating the SUE Inspection Technique. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI 2000) (pp. 143-150). New York: ACM.
Dumas, J. S., Molich, R., & Jeffries, R. (2004). Describing usability problems: are we sending the right message? interactions, 11(4), 24-29.
Hassenzahl, M. (2000). Prioritizing usability problems: data-driven and judgement-driven severity estimates. Behaviour & Information Technology, 19(1), 29-42.
Hornbæk, K., & Frøkjær, E. (2005). Comparing Usability Problems and Redesign Proposals as Input to Practical Systems Development. In Proceedings of the Conference on Human factors in computing systems (CHI 2005) (pp. 391-400). New York: ACM.
Hvannberg, E. T., & Law, L.-C. (2003). Classification of Usability Problems (CUP) Scheme. In M. Rauterberg (Ed.) Proceedings of the Human-Computer Interaction - INTERACT'03 (pp. 655-662): IOS.
ISO. (1998). Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability (ISO 9241-11:1998(E)). Geneva: author.
ISO. (1999a). Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 10: Dialogue principles (ISO 9241-10:1998(E)). Geneva: author.
ISO. (1999b). Human -centred design processes for interactive systems (ISO 134071999(E)). Geneva: author.
Jacobsen, N. E., Hertzum, M., & John, B. E. (1998). The Evaluator Effect in Usability Tests. In C.-M. Karat & A. Lund & J. Coutaz & J. Karat (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI 98) (pp. 255-256). New York: ACM.
Jeffries, R. (1994). Usability Problem Reports: Helping Evaluators Communicate Effectively with Developers. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 273-294). New York: John Wiley.
Keenan, S. L., Hartson, H. R., Kafura, D. G., & Schulman, R. S. (1999). The usability problem taxonomy: A framework for classification and analysis. Empirical Software Engineering, 4(1), 71-104.
Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16(4/5), 246-266.
Mack, R., & Montaniz, F. (1994). Observing, Predicting, and Analyzing Usability Problems. In R. L. Mack & J. Nielsen (Eds.), Usability Inspection Methods (pp. 295-340). New York: John Wiley & Sons.
Nielsen, J. (1992). Finding Usability Problems Through Heuristic Evaluation. In P. Bauersfeld & J. Bennett & G. Lynch (Eds.), Proceedings of the Conference on Human Factors in Computing Systems (CHI 92) (pp. 373-380). New York: ACM.
Nielsen, J., & Landauer, T. K. (1993). A Mathematical model of the finding of usability problems. In S. Ashlund & K. Mullet & A. Henderson & E. Hollnagel & T. White (Eds.), Proceedings of the INTERCHI 93: Conference on Human Factors in Computing Systems (INTERACT 93 and CHI 93) (pp. 206-213). New York: ACM.
Rourke, C. (2003). CUE-4: Lessons in Best Practice for Usability Testing and Expert Evaluation. User Vision. Retrieved December 14, 2004 from http://www.uservision.co.uk/usability_articles/usability_CUE.asp .
Skov, M. B., & Stage, J. (2005). Supporting problem identification in usability evaluations. In Proceedings of the 19th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: citizens online: considerations for today and the future (OZCHI 2005). Narrabundah, Australia: Computer-Human Interaction Special Interest Group (CHISIG) of Australia.
Theofanos, M., & Quesenbery, W. (2005). Towards the Design of Effective Formative Test Reports. Journal of Usability Studies, 1(1), 27-45.
Theofanos, M., Quesenbery, W., Snyder, C., Dayton, D., & Lewis, J. (2005). Reporting
on Formative Testing. A UPA 2005 Workshop Report. Bloomingdale, IL: UPA. Retrieved on October 3, 2005 from the UPA website http://www.upassoc.org/usability_resources/conference/2005/formative%20reporting-upa2005.pdf.
Sep 5 - Howarth, Supporting Novice Usability Practitioners with Usability Engineering Tools (part 2)
Supporting Novice Usability Practitioners with Usability Engineering Tools.
Jonathan Randall Howarth.
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Computer Science and Applications.
April 13, 2007
Blacksburg, Virginia
2 RELATED WORK
2.1 Difficulties Experienced by Usability Practitioners
2.1.1 Evaluator Effect
The evaluator effect is the tendency of usability practitioners with differing knowledge and experience to find different types and numbers of UPs during usability evaluation.
Work by Rowe et al. [1994] demonstrated that different usability evaluation teams studying the same interface will find different issues.
Jacobsen et al. [1998] documented and named the evaluator effect in a study in which four usability experts were given the same video tapes of four participants performing tasks in a multimedia authoring system. Each expert identified about half of the UPs, but about half of those were unique to the individual expert.
A related study by Hertzum and Jacobsen [2003] provided more evidence of the evaluator effect by reviewing 11 studies that used one of the following usability evaluation methods: cognitive walkthroughs, heuristic evaluation, or thinkingaloud study. The authors proposed that the evaluator effect occurs because usability evaluation involves interpretation and that usability evaluation methods do not provide the guidance that usability practitioners need to perform reliable evaluations.
Vermeeren et al. [2003] conducted a study that found evidence of the evaluator effect in different domains and also proposed reasons for the evaluator effect related to interpretation, such as guessing user intentions.
2.1.2 Content of Usability Problem Descriptions
UP descriptions document interaction design flaws that cause UPs for users. They are used, in the context of a usability evaluation report, to help system analysts and designers identify specific features of an interaction design to change, add, or remove in subsequent iterations.
There have been a limited number of UP description formats documented in the literature.
Jeffries developed recommendations for what to include in a UP description while performing a review of UP descriptions to determine their shortcomings [Jeffries, 1994]. While the recommendations represent an improvement over ad hoc reporting, they do not provide a definite format and focus on solutions without addressing causes.
A study by John and Packer [1995] on the learnability and applicability of the cognitive walkthrough method contained a UP description form with a unique reference number and fields for describing the UP, estimating its severity, and assessing the source of its discovery. This form, much like Jeffries’ recommendations, did not specifically address the causes within the interaction design of problems.
In a study comparing empirical testing with usability inspections, Mack and Montaniz [1994] describe a UP description structure that includes descriptions of goal-directed behavior, interface interactions, possible causes, and severity. This report structure does address the causes of UPs, but it relies heavily on interpretation...
To enable comparative studies of usability evaluation methods, a more standard way to describe UPs was needed.
Lavery et al. [1997] developed a structured UP description format that addressed the shortcomings of previous UP description formats. The method captures the problem context, cause, outcomes, and solutions.
Cockton and Lavery [1999] leverage this structured UP description format in the Structured Usability Problem Extraction (SUPEX) framework, which separates problem context, cause, and recommendations. The SUPEX framework provides a rigorous approach to extracting problems that distinguishes among multiple levels of abstraction and handles relationships among user actions to reduce under- and over-reporting of UPs.
Later work by Cockton et al. [2003] supports the use of structured UP descriptions by demonstrating that they help to improve analysts’ performance with the heuristic evaluation method.
Capra [2006] developed guidelines for the content of UP descriptions through a series of three studies with usability practitioners. These guidelines represent an important step towards structuring the content of UP descriptions. They, however, are subject to a major limitation of guidelines in that they may be difficult to apply consistently [Borges et al., 1996, Smith, 1986].
In sum, the literature does not provide a clear answer as to what to include in a UP description. As a result, UP descriptions are often ad hoc [Andre et al., 2001].
Without a specific format, usability practitioners may not be aware of what to look for during usability data collection or what to clarify with participants during empirical testing. In addition, even if necessary usability data are observed and recorded during usability data collection, the lack of a consistent report format may make it difficult for problem analysts to understand and relate the data.
2.1.3 Content of Usability Evaluation Reports
As illustrated in Figure 3, the output of the usability evaluation sub-process is a usability evaluation report. The UP descriptions discussed in Section 2.1.2 differ from usability evaluation reports; the former is used to document an individual UP while the latter is used to convey results of an entire usability evaluation.
As discussed in Section 2.1.2, the challenge associated with UP descriptions is getting the necessary data to completely specify the UP. The challenge associated with usability evaluation reports is conveying the necessary information associated with the UP descriptions to a given audience.
In 1997, the Industry Usability Reporting Project initiated by the National Institute of Standards and Technology developed the Common Industry Format (CIF), which is currently the most well known format for usability evaluation reports. The CIF became an American National Standard for Information Technology Standard in 2001 [ANSI, 2001].
By standardizing the reporting of usability tests, the CIF hoped to encourage the consideration of usability in purchasing software products; customer organizations that were interested could evaluate different products based on their CIF reports.
The CIF includes sections for describing the product, the method used to evaluate the product, and the results of the evaluation.
The CIF is intended for summative usability evaluations, but usability practitioners most frequently perform formative usability evaluations.
Theofanos [2005] and Theofanus and Quesenbery [2005] describe efforts to develop a new
CIF that would provide practitioners with guidance for performing and reporting formative studies.
The usability evaluation report consolidates usability information and provides the context for understanding UP descriptions. Without this context, UP descriptions may be misunderstood or overlooked.
2.2 Existing Usability Engineering Tools
There are a number of tools for use in UE efforts that represent a variety of focuses and development activities.
I present a survey of these tools using a categorization scheme to structure the discussion of the state of these tools. I include tools that I found through a combination of a literature and a web search.
The list of tools is not exhaustive; instead it provides examples of each basic category of tool.
2.2.1 Tools not Included in the Survey
The following basic types were excluded from the survey:
1 custom tools,
2 tools that facilitate the construction of interfaces, and
3 tools with a business or social research focus.
Custom tools are created by an organization specifically to fit the needs of a particular usability process. As a result, these tools are generally not documented and not made available for use outside of the organization.
Two basic types of tools are used to facilitate the construction of interfaces. One type is tools that help programmers write the code for graphical user interfaces. Another type of tools is used to
help with creating interfaces quickly for prototyping. Both types of tools are excluded because of the focus on the usability evaluation sub-process instead of the design sub-process.
Some tools are specifically created for UE processes, but do not address usability evaluation activities in any detail. For example, Agility helps individuals involved in the UE process collaborate with one another and plan activities and deliverables [Classic System Solutions Inc., 2005]. The tool, however, has a business focus and is not appropriate for this survey. ... One example is Observer, which is primarily intended for collecting observational data for social research [Noldus, 2005].
2.2.2 Categorization Scheme
My categorization scheme is based on a taxonomy of usability evaluation tools
developed by Ivory and Hearst [Ivory & Hearst, 2001] and the stages of the
usability evaluation sub-process shown in Figure 3.
The three levels to this scheme are as follows:
• 1 Evaluation method class – How usability evaluations are conducted using the tool
o 1.1 Analytical – Evaluations involve inspections by experts such as heuristic evaluations or cognitive walkthroughs or static analysis of an interaction design.
o 1.2 Empirical – Evaluations involve observing a participant using a tool.
• 2 Application class – What type of application can be evaluated using the tool
o 2.1 Desktop – The tool can only be used to evaluate desktop applications.
o 2.2 Web – The tool can only be used to evaluate websites.
o 2.3 Both – The tool can evaluate both desktop applications and websites.
• 3 Supported stages of the usability evaluation sub-process – Which stages of the usability evaluation sub-process are supported by the tool (Section 1.3.3)
o 3.1 Usability data collection
o 3.2 UP analysis
o 3.3 Usability evaluation reporting
2.2.3 Tools Included in the Survey
The tools included in the survey along with their evaluation method class, application class, and supported stages of the usability evaluation sub-process are shown in Table 3.
Table 3: Tools included in the survey
Tool Name > Evaluation Method Class > Application Class > Supported Stages
My Comments: This PhD Dissertation has certain degree of relevance to my PhD research. This is because Howarth's research involved developing a usabilty engineering tool; mine would involve developing a usability evaluation tool.
Later in my research, I would definitely want to read Howarth's Dissertation in detail to understand what he had done. This could be a valuable benchmark to me.
References that I may want to read further in future:
Andre, T. S., Hartson, H. R., Belz, S. M., & McCreary, F. A. (2001). The User Action Framework: A reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54(1), 107-136.
Andre, T. S., Hartson, H. R., & Williges, R. C. (2002). Determining the effectiveness of the Usability Problem Inspector: A theory-based model and tool for finding usability problems. Human Factors, 45(3), 455-482.
ANSI. (2001). Common Industry Format for usability test reports. American National Standard for Information Technology.
Bit Debris Solutions. Usability Activity Log. Retrieved December 21, 2005, from http://www.bitdebris.com/products/ulablog/
Capra, M. (2006). Usability Problem Description and the Evaluator Effect in Usability Testing. Unpublished dissertation. Blacksburg, VA: Virginia Tech.
Capra, M. G. (2001). An Exploration of End-User Critical Incident Classification. Unpublished master's thesis. Blacksburg, VA: Virginia Tech.
Classic System Solutions Inc. Agility. Retrieved December 21, 2005, from
http://www.classicsys.com/classic_site/html/team_design.html
Cockton, G., & Lavery, D. (1999). A framework for usability problem extraction. Paper presented at the International Conference on Human-Computer Interaction.
Cockton, G., Woolrych, A., Hall, L., & Hindmarch, M. (2003). Changing analysts' tunes: The surprising impact of a new instrument for usability inspection method assessment. In People & Computers XVII (pp. 145-161).
Dumas, J. S., Molich, R., & Jeffries, R. (2004). Describing usability problems: Are we sending the right message? interactions, 11(4), 24-29.
Etgen, M., & Cantor, J. (1999). What does getting WET (web event-logging tool) mean for web usability. Paper presented at the Conference on Human Factors and the Web.
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 373-410.
Hix, D., & Hartson, H. R. (1993a). Developing User Interfaces: Ensuring Usability through Product and Process. New York, USA: John Wiley & Sons, Inc.
Hix, D., & Hartson, H. R. (1993b). Formative evaluation: Ensuring usability in user interfaces. In L. Bass & P. Dewan (Eds.), User Interface Software (pp. 1-30). New York: John Wiley & Sons.
Hoegh, R. T., Nielsen, C., Overgaard, M., Pedersen, M., & Stage, J. (2006). The impact of usability reports and user test observations on developers' understanding of usability data: An exploratory study. International Journal of Human-Computer Interaction, 21(2), 173-196.
Howarth, J. (2006). Identifying immediate intention during usability evaluation. Paper presented at the ACM Southeast Conference.
ISO. (1998). 9241 - Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability. Geneva, Switzerland: International Standards Organization.
ISO. (1999). 13407 - Human-centered design processes for interactive systems. Geneva, Switzerland: International Standards Organization.
Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470-516.
Jeffries, R. (1994). Usability problem reports: Helping evaluators communicate effectively with developers. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 273-294). New York: John Wiley and Sons.
Keenan, S. L. (1996). Product usability and process improvement based on usability problem classification. Unpublished dissertation. Blacksburg, VA: Virginia Tech.
Keenan, S. L., Hartson, H. R., Kafura, D. G., & Schulman, R. S. (1999). The Usability Problem Taxonomy: A framework for classification and analysis. Empirical Software Engineering, 4(1), 71-104.
Lavery, D., & Cockton, G. (1997). Representing predicted and actual usability problems. Paper presented at the International Workshop on Representations in Interactive Software Development.
Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour and Information Technology, 16(4-5), 246-266.
Lund, A. M. (1997). Another approach to justifying the cost of usability. interactions, 4(3), 48-56.
Mack, R., & Montaniz, F. (1994). Observing, predicting, and analyzing usability problems. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 295-339). New York: John Wiley & Sons.
Mayhew, D. (1999). The Usability Engineering Lifecycle. San Francisco, CA: Morgan Kaufmann.
Nayak, N. P., Mrazek, D., & Smith, D. R. (1995). Analyzing and communicating usability data: Now that you have the data what do you do? ACM SIGCHI Bulletin, 27(1), 22-30.
Nielsen, C., Overgaard, M., Pedersen, M., & Stage, J. (2005). Feedback from usability evaluation to user interface design: Are usability reports any good? Paper presented at the INTERACT.
Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Paper presented at the Conference on Human Factors in Computing Systems.
Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: Wiley.
Scholtz, J., & Laskowski, S. (1998). Developing usability tools and techniques for designing and testing web sites. Paper presented at the Conference on Human Factors and the Web.
TechSmith. Morae: A complete usability testing solution for web sites and software. Retrieved November 30, 2005, from http://www.techsmith.com/products/morae/default.asp
Theaker, C. J., Phillips, R., Frost, T. M. E., & Love, W. R. (1989). HIMS: A tool for HCI evaluations. In A. Sutcliffe & L. Macaulay (Eds.), People and Computers V (HCI '89 Conference) (pp. 427-439).
Theofanos, M. (2005). Towards the design of effective formative test reports. Journal of Usability Studies, 1(1), 27-45.
Theofanos, M., Quesenbery, W., Snyder, C., Dayton, D., & Lewis, J. (2005). Reporting on formative testing: A UPA 2005 workshop report. Bloomingdale, IL.
Usable Net. LIFT. Retrieved December 21, 2005, from http://www.usablenet.com/
Users First. Users First - Portable professional tools to observe users. Retrieved November 30, 2005, from http://www.usersfirst.com/index.jsp?page=products
Uzilla. Uzilla: Tools for a usable web. Retrieved November 30, 2005, from http://uzilla.net/
Weiler, P. (1993). Software for the usability lab: a sampling of current tools. Paper presented at the Conference on Human Factors in Computing Systems.
Wixon, D. (2003). Evaluating usability methods: why the current literature fails the practitioner. interactions, 10(4), 28-34.
Working Web. Usability tool: What does it do? Retrieved November 30, 2005, from http://workingweb.com.au/services/UsingUsabilityTool.php
Jonathan Randall Howarth.
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Computer Science and Applications.
April 13, 2007
Blacksburg, Virginia
2 RELATED WORK
2.1 Difficulties Experienced by Usability Practitioners
2.1.1 Evaluator Effect
The evaluator effect is the tendency of usability practitioners with differing knowledge and experience to find different types and numbers of UPs during usability evaluation.
Work by Rowe et al. [1994] demonstrated that different usability evaluation teams studying the same interface will find different issues.
Jacobsen et al. [1998] documented and named the evaluator effect in a study in which four usability experts were given the same video tapes of four participants performing tasks in a multimedia authoring system. Each expert identified about half of the UPs, but about half of those were unique to the individual expert.
A related study by Hertzum and Jacobsen [2003] provided more evidence of the evaluator effect by reviewing 11 studies that used one of the following usability evaluation methods: cognitive walkthroughs, heuristic evaluation, or thinkingaloud study. The authors proposed that the evaluator effect occurs because usability evaluation involves interpretation and that usability evaluation methods do not provide the guidance that usability practitioners need to perform reliable evaluations.
Vermeeren et al. [2003] conducted a study that found evidence of the evaluator effect in different domains and also proposed reasons for the evaluator effect related to interpretation, such as guessing user intentions.
2.1.2 Content of Usability Problem Descriptions
UP descriptions document interaction design flaws that cause UPs for users. They are used, in the context of a usability evaluation report, to help system analysts and designers identify specific features of an interaction design to change, add, or remove in subsequent iterations.
There have been a limited number of UP description formats documented in the literature.
Jeffries developed recommendations for what to include in a UP description while performing a review of UP descriptions to determine their shortcomings [Jeffries, 1994]. While the recommendations represent an improvement over ad hoc reporting, they do not provide a definite format and focus on solutions without addressing causes.
A study by John and Packer [1995] on the learnability and applicability of the cognitive walkthrough method contained a UP description form with a unique reference number and fields for describing the UP, estimating its severity, and assessing the source of its discovery. This form, much like Jeffries’ recommendations, did not specifically address the causes within the interaction design of problems.
In a study comparing empirical testing with usability inspections, Mack and Montaniz [1994] describe a UP description structure that includes descriptions of goal-directed behavior, interface interactions, possible causes, and severity. This report structure does address the causes of UPs, but it relies heavily on interpretation...
To enable comparative studies of usability evaluation methods, a more standard way to describe UPs was needed.
Lavery et al. [1997] developed a structured UP description format that addressed the shortcomings of previous UP description formats. The method captures the problem context, cause, outcomes, and solutions.
Cockton and Lavery [1999] leverage this structured UP description format in the Structured Usability Problem Extraction (SUPEX) framework, which separates problem context, cause, and recommendations. The SUPEX framework provides a rigorous approach to extracting problems that distinguishes among multiple levels of abstraction and handles relationships among user actions to reduce under- and over-reporting of UPs.
Later work by Cockton et al. [2003] supports the use of structured UP descriptions by demonstrating that they help to improve analysts’ performance with the heuristic evaluation method.
Capra [2006] developed guidelines for the content of UP descriptions through a series of three studies with usability practitioners. These guidelines represent an important step towards structuring the content of UP descriptions. They, however, are subject to a major limitation of guidelines in that they may be difficult to apply consistently [Borges et al., 1996, Smith, 1986].
In sum, the literature does not provide a clear answer as to what to include in a UP description. As a result, UP descriptions are often ad hoc [Andre et al., 2001].
Without a specific format, usability practitioners may not be aware of what to look for during usability data collection or what to clarify with participants during empirical testing. In addition, even if necessary usability data are observed and recorded during usability data collection, the lack of a consistent report format may make it difficult for problem analysts to understand and relate the data.
2.1.3 Content of Usability Evaluation Reports
As illustrated in Figure 3, the output of the usability evaluation sub-process is a usability evaluation report. The UP descriptions discussed in Section 2.1.2 differ from usability evaluation reports; the former is used to document an individual UP while the latter is used to convey results of an entire usability evaluation.
As discussed in Section 2.1.2, the challenge associated with UP descriptions is getting the necessary data to completely specify the UP. The challenge associated with usability evaluation reports is conveying the necessary information associated with the UP descriptions to a given audience.
In 1997, the Industry Usability Reporting Project initiated by the National Institute of Standards and Technology developed the Common Industry Format (CIF), which is currently the most well known format for usability evaluation reports. The CIF became an American National Standard for Information Technology Standard in 2001 [ANSI, 2001].
By standardizing the reporting of usability tests, the CIF hoped to encourage the consideration of usability in purchasing software products; customer organizations that were interested could evaluate different products based on their CIF reports.
The CIF includes sections for describing the product, the method used to evaluate the product, and the results of the evaluation.
The CIF is intended for summative usability evaluations, but usability practitioners most frequently perform formative usability evaluations.
Theofanos [2005] and Theofanus and Quesenbery [2005] describe efforts to develop a new
CIF that would provide practitioners with guidance for performing and reporting formative studies.
The usability evaluation report consolidates usability information and provides the context for understanding UP descriptions. Without this context, UP descriptions may be misunderstood or overlooked.
2.2 Existing Usability Engineering Tools
There are a number of tools for use in UE efforts that represent a variety of focuses and development activities.
I present a survey of these tools using a categorization scheme to structure the discussion of the state of these tools. I include tools that I found through a combination of a literature and a web search.
The list of tools is not exhaustive; instead it provides examples of each basic category of tool.
2.2.1 Tools not Included in the Survey
The following basic types were excluded from the survey:
1 custom tools,
2 tools that facilitate the construction of interfaces, and
3 tools with a business or social research focus.
Custom tools are created by an organization specifically to fit the needs of a particular usability process. As a result, these tools are generally not documented and not made available for use outside of the organization.
Two basic types of tools are used to facilitate the construction of interfaces. One type is tools that help programmers write the code for graphical user interfaces. Another type of tools is used to
help with creating interfaces quickly for prototyping. Both types of tools are excluded because of the focus on the usability evaluation sub-process instead of the design sub-process.
Some tools are specifically created for UE processes, but do not address usability evaluation activities in any detail. For example, Agility helps individuals involved in the UE process collaborate with one another and plan activities and deliverables [Classic System Solutions Inc., 2005]. The tool, however, has a business focus and is not appropriate for this survey. ... One example is Observer, which is primarily intended for collecting observational data for social research [Noldus, 2005].
2.2.2 Categorization Scheme
My categorization scheme is based on a taxonomy of usability evaluation tools
developed by Ivory and Hearst [Ivory & Hearst, 2001] and the stages of the
usability evaluation sub-process shown in Figure 3.
The three levels to this scheme are as follows:
• 1 Evaluation method class – How usability evaluations are conducted using the tool
o 1.1 Analytical – Evaluations involve inspections by experts such as heuristic evaluations or cognitive walkthroughs or static analysis of an interaction design.
o 1.2 Empirical – Evaluations involve observing a participant using a tool.
• 2 Application class – What type of application can be evaluated using the tool
o 2.1 Desktop – The tool can only be used to evaluate desktop applications.
o 2.2 Web – The tool can only be used to evaluate websites.
o 2.3 Both – The tool can evaluate both desktop applications and websites.
• 3 Supported stages of the usability evaluation sub-process – Which stages of the usability evaluation sub-process are supported by the tool (Section 1.3.3)
o 3.1 Usability data collection
o 3.2 UP analysis
o 3.3 Usability evaluation reporting
2.2.3 Tools Included in the Survey
The tools included in the survey along with their evaluation method class, application class, and supported stages of the usability evaluation sub-process are shown in Table 3.
Table 3: Tools included in the survey
Tool Name > Evaluation Method Class > Application Class > Supported Stages
My Comments: This PhD Dissertation has certain degree of relevance to my PhD research. This is because Howarth's research involved developing a usabilty engineering tool; mine would involve developing a usability evaluation tool.
Later in my research, I would definitely want to read Howarth's Dissertation in detail to understand what he had done. This could be a valuable benchmark to me.
References that I may want to read further in future:
Andre, T. S., Hartson, H. R., Belz, S. M., & McCreary, F. A. (2001). The User Action Framework: A reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54(1), 107-136.
Andre, T. S., Hartson, H. R., & Williges, R. C. (2002). Determining the effectiveness of the Usability Problem Inspector: A theory-based model and tool for finding usability problems. Human Factors, 45(3), 455-482.
ANSI. (2001). Common Industry Format for usability test reports. American National Standard for Information Technology.
Bit Debris Solutions. Usability Activity Log. Retrieved December 21, 2005, from http://www.bitdebris.com/products/ulablog/
Capra, M. (2006). Usability Problem Description and the Evaluator Effect in Usability Testing. Unpublished dissertation. Blacksburg, VA: Virginia Tech.
Capra, M. G. (2001). An Exploration of End-User Critical Incident Classification. Unpublished master's thesis. Blacksburg, VA: Virginia Tech.
Classic System Solutions Inc. Agility. Retrieved December 21, 2005, from
http://www.classicsys.com/classic_site/html/team_design.html
Cockton, G., & Lavery, D. (1999). A framework for usability problem extraction. Paper presented at the International Conference on Human-Computer Interaction.
Cockton, G., Woolrych, A., Hall, L., & Hindmarch, M. (2003). Changing analysts' tunes: The surprising impact of a new instrument for usability inspection method assessment. In People & Computers XVII (pp. 145-161).
Dumas, J. S., Molich, R., & Jeffries, R. (2004). Describing usability problems: Are we sending the right message? interactions, 11(4), 24-29.
Etgen, M., & Cantor, J. (1999). What does getting WET (web event-logging tool) mean for web usability. Paper presented at the Conference on Human Factors and the Web.
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 373-410.
Hix, D., & Hartson, H. R. (1993a). Developing User Interfaces: Ensuring Usability through Product and Process. New York, USA: John Wiley & Sons, Inc.
Hix, D., & Hartson, H. R. (1993b). Formative evaluation: Ensuring usability in user interfaces. In L. Bass & P. Dewan (Eds.), User Interface Software (pp. 1-30). New York: John Wiley & Sons.
Hoegh, R. T., Nielsen, C., Overgaard, M., Pedersen, M., & Stage, J. (2006). The impact of usability reports and user test observations on developers' understanding of usability data: An exploratory study. International Journal of Human-Computer Interaction, 21(2), 173-196.
Howarth, J. (2006). Identifying immediate intention during usability evaluation. Paper presented at the ACM Southeast Conference.
ISO. (1998). 9241 - Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability. Geneva, Switzerland: International Standards Organization.
ISO. (1999). 13407 - Human-centered design processes for interactive systems. Geneva, Switzerland: International Standards Organization.
Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470-516.
Jeffries, R. (1994). Usability problem reports: Helping evaluators communicate effectively with developers. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 273-294). New York: John Wiley and Sons.
Keenan, S. L. (1996). Product usability and process improvement based on usability problem classification. Unpublished dissertation. Blacksburg, VA: Virginia Tech.
Keenan, S. L., Hartson, H. R., Kafura, D. G., & Schulman, R. S. (1999). The Usability Problem Taxonomy: A framework for classification and analysis. Empirical Software Engineering, 4(1), 71-104.
Lavery, D., & Cockton, G. (1997). Representing predicted and actual usability problems. Paper presented at the International Workshop on Representations in Interactive Software Development.
Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour and Information Technology, 16(4-5), 246-266.
Lund, A. M. (1997). Another approach to justifying the cost of usability. interactions, 4(3), 48-56.
Mack, R., & Montaniz, F. (1994). Observing, predicting, and analyzing usability problems. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 295-339). New York: John Wiley & Sons.
Mayhew, D. (1999). The Usability Engineering Lifecycle. San Francisco, CA: Morgan Kaufmann.
Nayak, N. P., Mrazek, D., & Smith, D. R. (1995). Analyzing and communicating usability data: Now that you have the data what do you do? ACM SIGCHI Bulletin, 27(1), 22-30.
Nielsen, C., Overgaard, M., Pedersen, M., & Stage, J. (2005). Feedback from usability evaluation to user interface design: Are usability reports any good? Paper presented at the INTERACT.
Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Paper presented at the Conference on Human Factors in Computing Systems.
Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: Wiley.
Scholtz, J., & Laskowski, S. (1998). Developing usability tools and techniques for designing and testing web sites. Paper presented at the Conference on Human Factors and the Web.
TechSmith. Morae: A complete usability testing solution for web sites and software. Retrieved November 30, 2005, from http://www.techsmith.com/products/morae/default.asp
Theaker, C. J., Phillips, R., Frost, T. M. E., & Love, W. R. (1989). HIMS: A tool for HCI evaluations. In A. Sutcliffe & L. Macaulay (Eds.), People and Computers V (HCI '89 Conference) (pp. 427-439).
Theofanos, M. (2005). Towards the design of effective formative test reports. Journal of Usability Studies, 1(1), 27-45.
Theofanos, M., Quesenbery, W., Snyder, C., Dayton, D., & Lewis, J. (2005). Reporting on formative testing: A UPA 2005 workshop report. Bloomingdale, IL.
Usable Net. LIFT. Retrieved December 21, 2005, from http://www.usablenet.com/
Users First. Users First - Portable professional tools to observe users. Retrieved November 30, 2005, from http://www.usersfirst.com/index.jsp?page=products
Uzilla. Uzilla: Tools for a usable web. Retrieved November 30, 2005, from http://uzilla.net/
Weiler, P. (1993). Software for the usability lab: a sampling of current tools. Paper presented at the Conference on Human Factors in Computing Systems.
Wixon, D. (2003). Evaluating usability methods: why the current literature fails the practitioner. interactions, 10(4), 28-34.
Working Web. Usability tool: What does it do? Retrieved November 30, 2005, from http://workingweb.com.au/services/UsingUsabilityTool.php
Friday, September 4, 2009
Sep 5 - Hoegh, Usability Problems: Do Software Developers Already Know?
Usability Problems: Do Software Developers Already Know?
Rune Thaarup Høegh. Aalborg University, Department of Computer Science, Aalborg East, DK-9220, Denmark. runethh@cs.aau.dk
OZCHI 2006, November 20-24, 2006, Sydney, Australia.
OZCHI 2006 Proceedings ISBN: 1-59593-545-2
Abstract
The result of usability evaluations is often accentuated as a distinctive input for developers to improve the usability of a software system. On the other hand developers say that many of the results from the usability evaluations are issues already known to them. This paper presents a study of usability problems as developers perceive them in their own emerging software in relation to usability problems experienced by users in a usability evaluation. The results indicate that having developers explicating their expectation on emerging software can provide a low-cost identification of problem areas, whereas a full scale usability evaluation provides specific knowledge of usability problems and their severity.
When developing interactive software systems, a key quality factor is usability of the software for its prospective users.
Usability relates to the extent a user can achieve specified goals with effectiveness, efficiency and satisfaction (ISO 1998).
Usability evaluations are applied to assess the quality of a user interaction design and establish
a basis for improving it (Rubin, 1994). This is accomplished by identifying specific parts of a system that do not properly support the users in carrying out their work. Thus usability evaluations and the related activities can help developers make better decisions, and thereby allow them to do their jobs more effectively (Radle & Young, 2001).
Usability reports are suggested as the mean for communicating results of a usability evaluation (Dumas and Redish, 1993; Rubin 1994, Molich 2000).
The amounts and severity of identified usability problems are a typically communicated measurement (Karat et al. 1992, Nielsen 1993, Preece et al. 2002).
The traditional usability report typically includes a description of evaluation method and settings, demographic data about the test subjects, a list of usability problems sorted by severity, detailed descriptions of the usability problems, and log files with transcripts of the individual usability tests.
The list of usability problems sorted by severity is probably the most essential information in the usability report.
The study examines the amount and nature of usability problems developers are aware off prior to a usability evaluation. The developers, from the company in the study, had previously claimed that many of the problems identified in usability evaluations were issues redundant to the knowledge they already had about the software from working with it on a daily basis; hence
they do did not find the identified usability problems in the report to be a useful tool for improving the software.
Assumptions about usability and developers not recognizing usability problems are two aspects of system development that can lead to software that is hard to use for the end-users.
Experiences from projects developed by the UCD approach show that developers often get surprised by seeing that users can not use the software the way it has been designed. The developers misjudge the usability of the software they have developed.
Method
Developers from two software projects were asked to write down all known usability problems in the software they developed.
Afterwards the two software systems were usability tested with users, the results from the evaluations were analyzed, and the results were compared to the usability problems described by the developers.
Procedure
The procedure used for this study involved three overall steps; Preparing the usability evaluation, interviewing the developers and usability evaluation.
The first step was the creation of task to be used in the usability test. The usability test was to be conducted as a validation test after the think aloud protocol (Rubin, 1994). The company’s human factor experts and a domain expert created a range of tasks designed to evaluate the general usability of the software. The tasks were used both to show the developers what parts of the software that would be tested, and for the usability test.
The second step was a workshop with the GUI developers of each project. The developers were introduced to the study, and then presented the tasks that would be used in the evaluation. They were then asked to individually write down the list of usability problems they knew existed in the parts of the software that would be tested. Afterwards the developers were asked to make a common list. The developers presented the usability problems one by one, and a facilitator asked the others if they had written down the same problems or variations of it. The usability problems were all written down on a whiteboard by the facilitator. This procedure went on until all their individual lists were exhausted. Finally the developers together rated the severity of the usability problems. The problems were rated according to the three categories suggest by Molich (2000).
The third step of the study was the usability evaluation. Five individual users participated in the evaluation of the software projects. All evaluations were done within one day. The evaluation was performed by the think-aloud protocol, and each evaluation session took one hour. All of the sessions were filmed by cameras and recorded using the video equipment in the laboratory.
Data Analysis
A detailed video based analysis (VBA) was conducted, and a list of usability problems for each system was generated based on the video records. The identification of usability problems included marking on the video when the problem occurred, along with a detailed description of the problem and a severity ranking. After all video was analyzed the identified usability problems were compared, and the complete list of problems was compiled. During the study the time spend on the various tasks was recorded in a diary.
The usability problems described on the three lists were compared, and similarities and differences between the lists were noted while taking both amounts of identified problems and types of problems into account. Finally the time spent on creating the three lists was compared. Additionally the results from project A and B were examined for differences and similarities.
My Comments: the 3 lists were (a) Individual Lists, (b) Common List, (c) VBA Lists.
CONCLUSION
The developers had knowledge about 38 percent of the usability problems prior to the evaluations. The developers were however mostly able to describe the usability problems in more general terms than those identified during the usability evaluation.
Two-thirds of the usability problems identified by the usability evaluations were not known on before hand. These usability problems were problems that the developers did not expect or acknowledge. If they had an idea about them on before hand, it was not so obvious to them that they put them on any of their lists. Hence the usability evaluations have the potential to provide the developers with new knowledge of their software.
The descriptions of the usability problems written by the developers were less specific than the problem descriptions produced by VBA. The resources spent on the obtaining the usability problems were however significantly fewer.
The process of writing down the usability problems and merging the lists might have influenced the developers’ knowledge of the software. Prior to the process the developers might have had implicit ideas about the usability problems in the software, but asking them to write them down forced the developers to make their assumptions explicit.
On one hand this might be considered a flaw in the study, on the other hand having developers explicating their knowledge of the software’s usability might prove a valuable and cost-effective tool to improve the usability of the software.
References that I may want to read further in the future:
Dumas, J. S. and Redish, J. C. (1993). A practical guide to usability testing. Norwood, NJ: Ablex Publishing.
Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests, New York, NY: John Wiley & Sons.
Rune Thaarup Høegh. Aalborg University, Department of Computer Science, Aalborg East, DK-9220, Denmark. runethh@cs.aau.dk
OZCHI 2006, November 20-24, 2006, Sydney, Australia.
OZCHI 2006 Proceedings ISBN: 1-59593-545-2
Abstract
The result of usability evaluations is often accentuated as a distinctive input for developers to improve the usability of a software system. On the other hand developers say that many of the results from the usability evaluations are issues already known to them. This paper presents a study of usability problems as developers perceive them in their own emerging software in relation to usability problems experienced by users in a usability evaluation. The results indicate that having developers explicating their expectation on emerging software can provide a low-cost identification of problem areas, whereas a full scale usability evaluation provides specific knowledge of usability problems and their severity.
When developing interactive software systems, a key quality factor is usability of the software for its prospective users.
Usability relates to the extent a user can achieve specified goals with effectiveness, efficiency and satisfaction (ISO 1998).
Usability evaluations are applied to assess the quality of a user interaction design and establish
a basis for improving it (Rubin, 1994). This is accomplished by identifying specific parts of a system that do not properly support the users in carrying out their work. Thus usability evaluations and the related activities can help developers make better decisions, and thereby allow them to do their jobs more effectively (Radle & Young, 2001).
Usability reports are suggested as the mean for communicating results of a usability evaluation (Dumas and Redish, 1993; Rubin 1994, Molich 2000).
The amounts and severity of identified usability problems are a typically communicated measurement (Karat et al. 1992, Nielsen 1993, Preece et al. 2002).
The traditional usability report typically includes a description of evaluation method and settings, demographic data about the test subjects, a list of usability problems sorted by severity, detailed descriptions of the usability problems, and log files with transcripts of the individual usability tests.
The list of usability problems sorted by severity is probably the most essential information in the usability report.
The study examines the amount and nature of usability problems developers are aware off prior to a usability evaluation. The developers, from the company in the study, had previously claimed that many of the problems identified in usability evaluations were issues redundant to the knowledge they already had about the software from working with it on a daily basis; hence
they do did not find the identified usability problems in the report to be a useful tool for improving the software.
Assumptions about usability and developers not recognizing usability problems are two aspects of system development that can lead to software that is hard to use for the end-users.
Experiences from projects developed by the UCD approach show that developers often get surprised by seeing that users can not use the software the way it has been designed. The developers misjudge the usability of the software they have developed.
Method
Developers from two software projects were asked to write down all known usability problems in the software they developed.
Afterwards the two software systems were usability tested with users, the results from the evaluations were analyzed, and the results were compared to the usability problems described by the developers.
Procedure
The procedure used for this study involved three overall steps; Preparing the usability evaluation, interviewing the developers and usability evaluation.
The first step was the creation of task to be used in the usability test. The usability test was to be conducted as a validation test after the think aloud protocol (Rubin, 1994). The company’s human factor experts and a domain expert created a range of tasks designed to evaluate the general usability of the software. The tasks were used both to show the developers what parts of the software that would be tested, and for the usability test.
The second step was a workshop with the GUI developers of each project. The developers were introduced to the study, and then presented the tasks that would be used in the evaluation. They were then asked to individually write down the list of usability problems they knew existed in the parts of the software that would be tested. Afterwards the developers were asked to make a common list. The developers presented the usability problems one by one, and a facilitator asked the others if they had written down the same problems or variations of it. The usability problems were all written down on a whiteboard by the facilitator. This procedure went on until all their individual lists were exhausted. Finally the developers together rated the severity of the usability problems. The problems were rated according to the three categories suggest by Molich (2000).
The third step of the study was the usability evaluation. Five individual users participated in the evaluation of the software projects. All evaluations were done within one day. The evaluation was performed by the think-aloud protocol, and each evaluation session took one hour. All of the sessions were filmed by cameras and recorded using the video equipment in the laboratory.
Data Analysis
A detailed video based analysis (VBA) was conducted, and a list of usability problems for each system was generated based on the video records. The identification of usability problems included marking on the video when the problem occurred, along with a detailed description of the problem and a severity ranking. After all video was analyzed the identified usability problems were compared, and the complete list of problems was compiled. During the study the time spend on the various tasks was recorded in a diary.
The usability problems described on the three lists were compared, and similarities and differences between the lists were noted while taking both amounts of identified problems and types of problems into account. Finally the time spent on creating the three lists was compared. Additionally the results from project A and B were examined for differences and similarities.
My Comments: the 3 lists were (a) Individual Lists, (b) Common List, (c) VBA Lists.
CONCLUSION
The developers had knowledge about 38 percent of the usability problems prior to the evaluations. The developers were however mostly able to describe the usability problems in more general terms than those identified during the usability evaluation.
Two-thirds of the usability problems identified by the usability evaluations were not known on before hand. These usability problems were problems that the developers did not expect or acknowledge. If they had an idea about them on before hand, it was not so obvious to them that they put them on any of their lists. Hence the usability evaluations have the potential to provide the developers with new knowledge of their software.
The descriptions of the usability problems written by the developers were less specific than the problem descriptions produced by VBA. The resources spent on the obtaining the usability problems were however significantly fewer.
The process of writing down the usability problems and merging the lists might have influenced the developers’ knowledge of the software. Prior to the process the developers might have had implicit ideas about the usability problems in the software, but asking them to write them down forced the developers to make their assumptions explicit.
On one hand this might be considered a flaw in the study, on the other hand having developers explicating their knowledge of the software’s usability might prove a valuable and cost-effective tool to improve the usability of the software.
References that I may want to read further in the future:
Dumas, J. S. and Redish, J. C. (1993). A practical guide to usability testing. Norwood, NJ: Ablex Publishing.
Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests, New York, NY: John Wiley & Sons.
Subscribe to:
Posts (Atom)
