Oct 10 - Lund, Measuring Usability with the USE Questionnaire

Over the years I have worked with colleagues at Ameritech (where the work began), U.S. WEST Advanced Technologies, and most recently Sapient to create a tool that has helped in dealing with some of these questions.

The tool that we developed is called the USE Questionnaire. USE stands for Usefulness, Satisfaction, and Ease of use. These are the three dimensions that emerged most strongly in the early development of the USE Questionnaire. For many applications, Usability appears to consist of Usefulness and Ease of Use, and Usefulness and Ease of Use are correlated.

Each factor in turn drives user satisfaction and frequency of use. Users appear to have a good sense of what is usable and what is not, and can apply their internal metrics across domains.

USE Questionnaire

Usefulness
It helps me be more effective.
It helps me be more productive.
It is useful.
It gives me more control over the activities in my life.
It makes the things I want to accomplish easier to get done.
It saves me time when I use it.
It meets my needs.
It does everything I would expect it to do.

Ease of Use
It is easy to use.
It is simple to use.
It is user friendly.
It requires the fewest steps possible to accomplish what I want to do with it.
It is flexible.
Using it is effortless.
I can use it without written instructions.
I don't notice any inconsistencies as I use it.
Both occasional and regular users would like it.
I can recover from mistakes quickly and easily.
I can use it successfully every time.

Ease of Learning
I learned to use it quickly.
I easily remember how to use it.
It is easy to learn to use it.
I quickly became skillful with it.

Satisfaction
I am satisfied with it.
I would recommend it to a friend.
It is fun to use.
It works the way I want it to work.
It is wonderful.
I feel I need to have it.
It is pleasant to use.

Work to refine the items and the scales continues. There is some evidence that for web sites and certain consumer products there is an additional dimension of fun or aesthetics associated with making a product compelling.

General Background

Subjective reactions to the usability of a product or application tend to be neglected in favor of performance measures, and yet it is often the case that these metrics measure the aspects of the user experience that are most closely tied to user behavior and purchase decisions.

While some tools exist for assessing software usability, they typically are proprietary (and may only be available for a fee). More importantly, they do not do a good job of assessing usability across domains.
When re-engineering began at Ameritech, it became important to be able to set benchmarks for product usability and to be able to measure progress against those benchmarks.
It also was critical to ensure resources were being used as efficiently as possible, and so tools to help select the most cost-effective methodology and the ability to prioritize design problems to be fixed by developers became important.
Finally, it became clear that we could eliminate all the design problems and still end up with a product that would fail in the marketplace.

It was with this environment as a background that a series of studies began at Ameritech. The first one was headed by Amy Schwartz, and was a collaboration of human factors, market research in our largest marketing organization, and a researcher from the University of Michigan business school.

Building on that research, I decided to develop a short questionnaire that could be used to measure the most important dimensions of usability for users, and to measure those dimensions across domains. Ideally it should work for software, hardware, services, and user support materials. It should allow meaningful comparisons of products in different domains, even though testing of the products happened at different times and perhaps under different circumstances. In the best of all worlds, the items would have a certain amount of face validity for both users and practitioners, and it would be possible to imagine the aspects of the design that might influence ratings of the items.
It would not be intended to be a diagnostic tool, but rather would treat the dimensions of usability as dependent variables.

Subsequent research would assess how various aspects of a given category of design would impact usability ratings.
The early studies at Ameritech suggested that a viable questionnaire could be created. Interestingly, the results of those early studies were consistent with studies conducted in the MIS and technology diffusion areas, which also had identified the importance of and the relationship between Usefulness, Satisfaction, and Ease of Use.

How It Developed

The first step in identifying potential items for the questionnaire was to collect a large pool of items to test. The items were collected from previous internal studies, from the literature, and from brainstorming. The list was then massaged to eliminate or reword items that could not be applied across the hardware, software, documentation, and service domains. One goal was to make the items as simply worded as possible, and as general as possible.
As rounds of testing progressed, standard psychometric techniques were used to weed out additional items that appeared to be too idiosyncratic or to improve items through ongoing tweaking of the wording. In general, the items contributing to each scale were of approximately equal weight, the Chronbach's Alphas were very high, and for the most part the items appeared to tap slightly different aspects of the dimensions being measured.

The questionnaires were constructed as seven-point Likert rating scales. Users were asked to rate agreement with the statements, raging from strongly disagree to strongly agree.
Various forms of the questionnaires were used to evaluate user attitudes towards a variety of consumer products.

Factor analyses following each study suggested that users were evaluating the products primarily using three dimensions, Usefulness, Satisfaction, and Ease of Use. Evidence of other dimensions was found, but these three served to most effectively discriminate between interfaces.
Partial correlation calculated using scales derived for these dimensions suggested that Ease of Use and Usefulness influence one another, such that improvements in Ease of Use improve ratings of Usefulness and vice versa.
While both drive Satisfaction, Usefulness is relatively less important when the systems are internal systems that users are required to use. Users are more variable in their Usefulness ratings when they have had only limited exposure to a product.
As expected from the literature, Satisfaction was strongly related to the usage (actual or predicted).
For internal systems, the items contributing to Ease of Use for other products actually could be separated into two factors, Ease of Learning and Ease of Use (which were obviously highly correlated).

Conclusion

While the questionnaire has been used successfully by many companies around the world, and as part of several dissertation projects, the development of the questionnaire is still not over. For the reasons cited, this is an excellent starting place. The norms I have developed over the years have been useful in determining when I have achieved sufficient usability to enable success in the market. To truly develop a standardized instrument, however, the items should be taken through a complete psychometric instrument development process.
A study I have been hoping to run is one that simultaneously uses the USE Questionnaire and other questionnaires like SUMI or QUIS to evaluate applications. Once a publicly available (i.e., free) standardized questionnaire is available that applies across domains, a variety of interesting lines of research are possible. The USE Questionnaire should continue to be useful as it stands, but I hope the best is yet to come.


Measuring Usability with the USE Questionnaire
By Arnold M. Lund

Source: http://www.stcsig.org/usability/newsletter/0110_measuring_with_use.html

Oct 10 - Usabilitynet, Usability Questionnaires

My comments: This is a good summary/intro of several Usability Questionnaires.

It has been observed that questionnaires are the most frequently used tools for usability evaluation. This page is a list of usability questionnaire resources, extending the information presented on the questionnaires page of Usabilitynet.

SUMI
This is a mature questionnaire whose standardisation base and manual have been regularly updated. It is good for desktop products, but has also been used to evaluate command-and-control applications. It is a commercial product which comes complete with scoring and report generation software. It is designed and sold by the Human Factors Research Group at University College Cork.

WAMMI
This is a new questionnaire, designed to evaluate the quality of use of web sites. It is backed up by an extensive standardisation database, and it is purchased on a per report basis. It is the result of a joint development project by Jurek Kirakowski and Nigel Claridge.

SUS
This is a mature questionnaire, developed by John Brooke in 1986 and not published until years later. It is very robust and has been extensively used and adapted. It is public domain and nobody has published any standardisation data about it. Of all the public domain questionnaires, this is the most strongly recommended.

QUIS
This is a questionnaire developed by Kent Norman that has been modified many times to keep it current since its first appearance. It is commercially available and is championed by Ben Shneiderman in his book Designing the User Interface. Reading, MA: Addison-Wesley Publishing Co., 1998. Despite lack of standardisation and validation data, it has many adherents.

USE (see http://www.mindspring.com/~alund/USE/IntroductionToUse.html)
This questionnaire is still in development by Arnie Lund (last updated 11/11/98). It attempts to create a three-factor model of usability that can be applied to many situations. However, no reliability or validation data are presented. Public domain use is encouraged.

CSUQ
This is a well-designed questionnaire developed by Jim Lewis and it is public domain. It has excellent psychometric reliability properties but no standardisation base.

IsoNorm (in German only)
This questionnaire is designed to test the usability quality of software following the ISO 9241 part 10 principles. It is created by a team led by Jochim Puemper. Strong reliabilities are claimed for the sub-scales, although it appears there may be a strong inter-correlation between them as well. Downloads and an on-line version are available from the above URL, as well as articles about it (all in German.)

IsoMetrics
This questionnaire is produced by Guenter Gediga and his team. It is another attempt to produce a way of measuring ISO 9241 part 10, with reference to specific software features that may give rise to low usability data. It is therefore good both for summative and formative assessments. The questionnaire is well researched and detailed statistical information is given. Downloads of English and German versions are available. There is no standardisation base for it but it is public domain.


Questionnaire resources
Source: http://www.usabilitynet.org/tools/r_questionnaire.htm

Oct 8 - USE Questionnaire - Questionnaire for User Interface Satisfaction

Questionnaire for User Interface Satisfaction (USE Questionnaire)

Please rate your agreement with these statements.
Try to respond to all the items.
For items that are not applicable, use: NA
Likert scale:
1 = strongly disagree
2
3
4
5
6
7 = strongly agree
NA


USEFULNESS

1. It helps me be more effective.
2. It helps me be more productive.
3. It is useful.
4. It gives me more control over the activities in my life.
5. It makes the things I want to accomplish easier to get done.
6. It saves me time when I use it.
7. It meets my needs.
8. It does everything I would expect it to do.


EASE OF USE

9. It is easy to use.
10. It is simple to use.
11. It is user friendly.
12. It requires the fewest steps possible to accomplish what I want to do with it.
13. It is flexible.
14. Using it is effortless.
15. I can use it without written instructions.
16. I don't notice any inconsistencies as I use it.
17. Both occasional and regular users would like it.
18. I can recover from mistakes quickly and easily.
19. I can use it successfully every time.

EASE OF LEARNING

20. I learned to use it quickly.
21. I easily remember how to use it.
22. It is easy to learn to use it.
23. I quickly became skillful with it.


SATISFACTION

24. I am satisfied with it.
25. I would recommend it to a friend.
26. It is fun to use.
27. It works the way I want it to work.
28. It is wonderful.
29. I feel I need to have it.
30. It is pleasant to use.

List the most negative aspect(s):
1 ......................................................
2 ......................................................
3 ......................................................

List the most positive aspect(s):
1 ......................................................
2 ......................................................
3 ......................................................


Questionnaire for User Interface Satisfaction
Based on: Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2. [Abstract]

Source: http://hcibib.org/perlman/question.cgi?form=USE