Oct 9 - CSUQ: Computer System Usability Questionnaire - Lewis of IBM

The Computer System Usability Questionnaire (CSUQ)

The PSSUQ research was preliminary for two reasons. First, the sample size for the factor analysis was small, consisting of data from only 48 participants. Second, the PSSUQ data came from a usability study. This setting may have influenced the correlations among the items and, therefore, the resultant factors.
The purpose of this research (Lewis, 1992a) was to use a slightly revised version of the PSSUQ, the Computer System Usability Questionnaire (CSUQ) to obtain a database of sufficient size to calculate stable factors from a mailed survey.
If the same factors emerged from this research as from the PSSUQ research, the study would demonstrate the potential usefulness of the questionnaire across different user groups and different research settings.

Item Selection and Construction
The CSUQ is identical to the PSSUQ (Lewis, 1991c), except that the wording of the items does not refer to a usability testing situation. For example, Item 3 of the PSSUQ states, "I could effectively complete the tasks and scenarios using this system," but Item 3 of the CSUQ states, "I can effectively complete my work using this system." (See the appendix for the CSUQ items.)

Psychometric Evaluation

The mail survey using the CSUQ.
The participants were 825 IBM employees who worked at nine IBM development sites: Atlanta, Austin, Bethesda, Boca Raton, Dallas, Raleigh, Rochester, San Jose, and Tucson. I used a random number generator to select the participants' names from the IBM electronic mail directory (CALLUP), and mailed them each a copy of the CSUQ with a cover letter. Responses from the returned questionnaires that arrived within 3 months of mailing made up the database for this study.

Factor analysis.
Forty-six percent (377) of the participants returned the questionnaire.
A principal factor analysis of the returned questionnaires produced the scree plot shown in Figure 3. The scree plot was similar to that found for the PSSUQ, indicating that an appropriate factor analysis should solve for three factors. Table 6 shows the varimax-rotated 3-factor solution. The selection criterion for the factor loadings was 0.5, shown in bold type in the table.
The factor analysis showed that Item 8 ("I believe I became productive quickly using this system"), which was not a part of the original PSSUQ, should be part of Factor 1. Item 15 ("The organization of information on the system screens is clear"), which loaded on two factors in the PSSUQ study, loaded on only Factor 2 in the current study. In the PSSUQ study and in the current study, Item 19 ("Overall, I am satisfied with this system") loaded on both Factors 1 and 3, and is not part of any subscale.
Otherwise, the factor structure of the CSUQ is very similar to that of the PSSUQ, so the CSUQ and PSSUQ subscales have the same names.
The three factors accounted for 98.6% of the variability in the rating data.

Reliability.
In all cases, coefficient alpha exceeded 0.89, indicating acceptable scale reliability. The estimates of coefficient alpha for the CSUQ were .93 for SYSUSE, .91 for INFOQUAL, .89 for INTERQUAL, and .95 for the OVERALL scale. The values of coefficient alpha for the CSUQ scales were within 0.03 of those for the PSSUQ scales.

Validity/Sensitivity.
After establishing scale reliability, the next step in psychometric evaluation is to determine scale validity. However, without a concurrent or predicted measurement, it is impossible to obtain a quantitative measure of validity in the traditional psychometric sense. An indirect way to assess validity is to examine scale sensitivity to variables that should systematically affect the scale. The sensitivity analyses of the PSSUQ (Lewis, 1992b) showed significant effects of user group (business professional with mouse experience, business professional without mouse experience, and secretary/clerk without mouse experience) on the OVERALL, SYSUSE, INFOQUAL, and INTERQUAL scales. The type of computer system the participant used during the study significantly affected the INFOQUAL scale.

A comprehensive listing of the influence of respondent characteristics on the CSUQ scores is outside the scope of this paper. However, the significant findings are similar to those for the PSSUQ. The type of computer that respondents used significantly affected their responses only for the INFOQUAL score (F(5,311)=2.14, p=0.06). The number of years of experience with their computer system affected respondents' scores for OVERALL (F(4,294)=3.12, p=0.02), SYSUSE (F(4,332)=2.05, p=0.09), INFOQUAL (F(4,311)=2.59, p=0.04) and INTERQUAL (F(4,322)=2.47, p=0.04). The respondents' range of experience with computer systems (number of different computer systems that they reported having used) affected scores for OVERALL (F(3,294)=2.77, p=0.04), INFOQUAL (F(3,311)=2.60, p=0.05) and INTERQUAL (F(3,322)=2.14, p=0.10).
These significant findings provide indirect support to the hypothesis that these scales are valid.

Discussion
The key results from this study are
(1) a demonstration of stable factors for the CSUQ (and, by extension, for the PSSUQ) and
(2) evidence that the questionnaire works well in non-laboratory settings.
The CSUQ scales are comparable to the PSSUQ scales, both in terms of reliability and validity (indicated by similarity in the sensitivity analyses).
These findings substantially enhance the usefulness of the CSUQ and PSSUQ to usability practitioners. Researchers who conduct usability studies (either laboratory or non-laboratory) can use this questionnaire to assess user satisfaction with system usability.


My Comments: This would be helpful if I eventually select to develop Usability Questionnaire as usability evaluation tool.

IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - PSSUQ: Post-Study System Usability Questionnaire - Lewis of IBM

The Post-Study System Usability Questionnaire (PSSUQ)

The Post-Study System Usability Questionnaire (PSSUQ) is currently a 19-item instrument for assessing user satisfaction with system usability. (See the appendix for a copy of the questionnaire items.)
Participants need more time to complete the PSSUQ than the ASQ (about 10 minutes to complete the PSSUQ), but only complete it once, at the end of a usability study. Completing the PSSUQ allows participants to provide an overall evaluation of the system they used.

After the 48 participants in the office-applications usability study (Lewis, Henry, & Mack, 1990) completed all the scenarios, they rated their system with the PSSUQ. This data allowed preliminary psychometric evaluation of the PSSUQ (Lewis, 1992b).
This earlier version of the PSSUQ (Lewis, 1992b) had only 18 items, with the items in a different order than shown in the appendix. Recently, a series of investigations using decision support systems revealed a common set of five system characteristics associated with usability by several different user groups (Doug Antonelli, personal communication, January 5, 1991). The original 18-item PSSUQ addressed four of these five system characteristics. The 19-item version of the PSSUQ contains an additional item to cover the fifth of these five system characteristics.

Item Construction
The items are 7-point graphic scales, anchored at the end points with the terms "Strongly agree" for 1, "Strongly disagree" for 7, and a "Not applicable" (N/A) point outside the scale.

Item Selection
A group of usability evaluators selected the items on the basis of their comprehensive content regarding hypothesized constituents of usability. For example, the items assess such system characteristics as ease of use, ease of learning, simplicity, effectiveness, information, and the user interface.

Psychometric Evaluation

Factor analysis.
The scree plot for an exploratory principal factors analysis of the PSSUQ data indicated that a 3-factor solution was appropriate (see Figure 2), so the overall scale defined by the full set of items contained three subscales. Table 5 shows the varimax-rotated factor pattern, revealing the structure of the subscales. Bold type in Table 5 highlights factor loadings that exceeded .5. Items that loaded highly on two factors were ambiguous regarding the appropriate subscale of which they should be a component, so they did not become a component of any subscale. (See the appendix to examine the content of these items.)
One of the most difficult tasks following this type of exploratory factor analysis is naming the factors. After considering a number of alternatives, a group of human factors engineers named the factors (and their corresponding subscales) System Usefulness (SYSUSE), Information Quality (INFOQUAL), and Interface Quality (INTERQUAL). These three factors account for 87% of the variability in the data.

Reliability.
Coefficient alpha analyses showed that the reliability of the overall summative scale (OVERALL) was .97, and ranged from .91 to .96 for the three subscales (SYSUSE=.96, INFOQUAL=.91, and INTERQUAL=.91). Therefore, the overall scale and the three subscales have excellent reliability.

Validity.
Correlation analyses support the validity of the scales. The OVERALL scale correlated highly with the sum of the ASQ ratings that participants gave after completing each scenario (r(20)=.80, p=.0001). OVERALL also correlated significantly with the percentage of successful scenario completion (r(29)=-.40, p=.026). The SYSUSE (r(36)=-.40, p=.006) and INTERQUAL (r(35)=-.29, p=.08) correlated with the percentage of successful scenario completion.

Sensitivity.
In the sensitivity ANOVAs, the overall scale and all three subscales indicated significant differences among the user groups (OVERALL: F(2,29)=4.35, p=.02; SYSUSE: F(2,36)=6.9, p=.003; INFOQUAL: F(2,33)=3.68, p=.04; INTERQUAL: F(2,33)=3.74, p=.03). INFOQUAL showed a significant system effect (F(2,33)=3.18, p=.05).

Discussion
These findings have limited generalizability because the sample size for the factor analysis was relatively small. The usual recommendation would be 90 participants for this questionnaire.
However, the factor analysis and reliability analyses suggest that it is reasonable to define three subscales from this set of items. The PSSUQ has reasonable concurrent validity when compared with successful scenario completion rates and the ASQ scores. The overall scale and the subscales are reasonably sensitive.
The evidence provided sufficient justification to use the PSSUQ to measure user satisfaction with system usability in usability studies, but also suggested that it would be prudent to collect more data in different circumstances to extend the generalizability of the findings.


My Comments: This would be helpful if I eventually select to develop Usability Questionnaire as usability evaluation tool.

IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - ASQ: After-Scenario Questionnaire - Lewis of IBM

The After-Scenario Questionnaire (ASQ)

In a scenario-based usability study, participants use a product, such as a computer application, to do a series of realistic tasks.
The After-Scenario Questionnaire (ASQ) is a three-item questionnaire that IBM usability evaluators have used to assess participant satisfaction after the completion of each scenario. (See the appendix for a copy of the questionnaire.)
The items address three important components of user satisfaction with system usability: ease of task completion, time to complete a task, and adequacy of support information (on-line help, messages, and documentation). Because the questionnaire is very short, it takes very little time for participants to complete – an important practical consideration for usability studies.
Usability professionals have used these items (or very similar items) in usability studies at IBM for many years, but a recent series of studies has provided a database of sufficient size to allow a preliminary psychometric evaluation of the ASQ.
The ASQ items are the constituent items for a summative, or Likert, scale (McIver & Carmines, 1981; Nunnally, 1978). In developing summative scales, it is important to consider item construction, item selection and psychometric evaluation.

Item Construction
The items are 7-point graphic scales, anchored at the end points with the terms "Strongly agree" for 1 and "Strongly disagree" for 7, and a Not Applicable (N/A) point outside the scale, as shown in the appendix.

Item Selection
The content of the items reflects components of usability that usability
professionals at IBM have generally considered important.

Psychometric Evaluation
The office-applications studies. Scenario-based usability studies of three office application systems (Lewis, Henry, & Mack, 1990) provided the data for a psychometric evaluation of the ASQ. Forty-eight employees of temporary help agencies participated in the studies, with 15 hired in Hawthorne, New York; 15 hired in Boca Raton, Florida; and 18 hired in Southbury, Connecticut. Each set of participants consisted of one-third clerical/secretarial work experience with no mouse experience (SECNO), one-third business professionals with no mouse experience (BPNO), and one-third business professionals with at least three months of mouse experience (BPMS). All participants had at least three months experience using some type of computer system. They had no programming training or experience, and had no (or very limited) knowledge of operating systems.
Popular word-processing applications, mail applications, calendar applications, and spreadsheet applications installed in three different operating environments comprised the three office systems (hereafter referred to as System I, System II and System III). All three environments allowed windowing, used a mouse as a pointing device, and allowed a certain amount of integration among the applications. The systems differed in details of implementation, but were generally similar. The three wordprocessing and spreadsheet applications were similar, but the mail and calendar applications differed considerably. The studies contained eight scenarios in common,
Participants began the study with a brief lab tour, read a description of the study's purpose and the day's agenda, and completed a background questionnaire. Participants using System I completed an interactive tutorial shipped with the system. Tutors provided the other participants with a brief demonstration about how to move, point and select with a mouse; how to open the icons for each product; and how to maximize and minimize windows.
After this system exploration period (usually about 1 hour), participants performed the scenarios, completing the ASQ as they finished each scenario. While the participant performed the scenario, an observer logged the participant's activities. If the participant completed the scenario without assistance and produced the correct output, then he or she completed the scenario successfully. Either after completing all scenarios or at the end of the workday (with some scenarios never attempted), participants provided an overall system rating with the Post-Study System Usability Questionnaire (PSSUQ) (Lewis, 1992b; Lewis, Henry, & Mack, 1990).
Participants usually needed a full work day (8 hours) to complete the study.
At the end of the three studies, the researchers entered the responses to the ASQ, PSSUQ, and the scenario completion data into a database.
From this database, it was possible to conduct an exploratory factor analysis, reliability analyses, validity analyses, and a sensitivity analysis.

Factor analysis.
Due to the design of this study (eight scenarios and a 3-item questionnaire), either an 8-factor or 3-factor solution would have been reasonable. An 8-factor solution could indicate grouping by scenario, and a 3-factor solution could indicate grouping by item type. Figure 1 shows the scree plot for the eigenvalues.
The scree plot for this analysis did not support a 3-factor solution, but did support an 8-factor solution. The rotated factor pattern is in Table 2. Using a selection criterion of .5 for the factor loadings (indicated with bold type), a clear relationship existed between the factors and the scenarios. The eight factors accounted for almost all (94%) of the variance in the data.

Reliability.
For the eight summative scales derived from the eight factors, all the coefficient alphas exceeded .90. Coefficient alphas this large were surprising because each scale contained only three items, and reliability is largely a function of the number of scale items (Nunnally, 1978).

Validity.
The correlation between the ASQ scores and scenario failure or success (coded as 0=failure and 1=success) was -.40 (n=48, p<.01). This result showed that participants who successfully completed a scenario tended to give lower (more favorable) ASQ ratings – evidence of concurrent validity.

Sensitivity.
Of the 48 participants, 27 completed all of the ASQ items for all of the scenarios. This reduced database was appropriate for an analysis-of-variance (ANOVA) to assess the sensitivity of the ASQ. Specifically, did the ASQ scores discriminate among the different systems, user groups, or scenarios in the three usability studies? The main effect of Scenario was highly significant (F(7,126)=8.92, p<.0001).
The Scenario by System interaction was also significant (F(14,126)=1.75, p=.05). These results suggest that the ASQ scale score is a reasonably sensitive measure.

Discussion
These findings have limited generalizability because the sample size for the factor analysis was relatively small. The usual recommendation would require 120 participants for this analysis (5 participants x 8 scenarios/participant x 3 items/scenario). On the other hand, the resulting factor structure was very clear.
The psychometric evaluation of this questionnaire showed that it is reasonable to condense the three ASQ items into a single scale through summation (or, equivalently, averaging). The available evidence indicates that the ASQ is reliable, valid, and sensitive. This condensation should allow easier interpretation and reporting of results when usability practitioners use the ASQ.

My Comments: This would be helpful if I eventually select to develop Usability Questionnaire as usability evaluation tool.

IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - Lewis, IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use

IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use


ABSTRACT
This paper describes recent research in subjective usability measurement at IBM. The focus of the research was the application of psychometric methods to the development and evaluation of questionnaires that measure user satisfaction with system usability. The primary goals of this paper are to (1) discuss the psychometric characteristics of four IBM questionnaires that measure user satisfaction with computer system usability, and (2) provide the questionnaires, with administration and scoring instructions. Usability practitioners can use these questionnaires with confidence to help them measure users' satisfaction with the usability of computer systems.


Introduction

Customers want usable products, and developers strive to produce them. It follows that an important part of modern product engineering, both hardware and software, must be the measurement of usability.
Measuring usability is particularly difficult because usability is not a unidimensional product or user characteristic, but emerges as a multidimensional characteristic in the context of users performing tasks with a product in a specific environment (Bevan, Kirakowski, & Maissel, 1991; Shackel, 1984).
However, if you are unable to measure usability, how can you judge your product against your competitors', or even your own previous versions of the product?

Subjective and Objective Evaluation

Most usability evaluations gather both subjective and objective quantitative data in the context of realistic scenarios-of-use, as well as descriptions of the problems representative participants have trying to complete the scenarios.
Subjective data are measures of participants' opinions or attitudes concerning their perception of usability.
Objective data are measures of participants' performance (such as scenario completion time and successful scenario completion rate).

Objective usability measures include, but are not limited to, scenario completion time, successful scenario completion rate, and time spent recovering from errors (Whiteside, Bennett, & Holtzblatt, 1988). Subjective usability measures are usually responses to Likert-type questionnaire items that assess user attitude concerning attributes such as system ease-of-use and interface likeability (Alty, 1992).
Most usability evaluators collect both objective and subjective data.

Research Focus

The focus of this research was the application of psychometric methods to the development and evaluation of standard questionnaires to assess subjective usability.
The goal of psychometrics is to establish the quality of psychological measures (Nunnally, 1978). Is a measure reliable in the sense that it is consistent? Given a reliable measure, is it valid (measures the intended attribute)? Finally, is the measure appropriately sensitive to experimental manipulations?
Psychometrics is a well-developed field, but usability researchers have only recently used these methods to develop and evaluate questionnaires to assess usability (Sweeney & Dillon, 1987).
In contrast to other recent computer-user satisfaction questionnaires (Chin, Diehl, & Norman, 1988; Kirakowski & Dillon, 1988; LaLomia & Sidowski, 1990) the IBM questionnaires are specifically for use in the context of scenario-based usability testing (Lewis, 1991a; Lewis, 1991b; Lewis, 1991c; Lewis, 1992b; Lewis, Henry, & Mack, 1990), although additional research has indicated that one may be useful as an instrument for field evaluation (Lewis, 1992a). Usability practitioners can use these questionnaires to enhance their current usability methods. (The four IBM questionnaires appear in the appendix.)
Before describing the psychometric properties of the IBM questionnaires, I will briefly review the relevant elements of psychometric practice. (For a comprehensive discussion of psychometrics, see Nunnally, 1978.)

----------
ASQ
PSQ
PSSUQ
CSUQ
-----------

General Discussion

Although user satisfaction with system usability is only one component of the multifaceted construct of usability (Bevan et al., 1991), it is a very important component in many situations.
It is especially important when a primary design goal is user satisfaction.

This paper has described the psychometric qualities of four questionnaires that assess user satisfaction with system usability: the ASQ, PSQ, PSSUQ and CSUQ.

The ASQ and PSQ are both after-scenario questionnaires, intended for use in a scenario-based usability testing situation. They contain essentially the same items, but the ASQ uses a 7-point scale and the PSQ uses a 5-point scale.
Using data from very different scenario-based usability studies (one a study of software office applications, the other a study of printers), their factor analyses, validity analyses, and sensitivity analyses were virtually identical. Obtaining the same results in different settings with different user groups provides strong evidence that these results are generalizable, and the questionnaires have wide applicability. Because the ASQ has substantially better reliability than the PSQ, usability practitioners should use the ASQ rather than the PSQ as their after-scenario questionnaire.

The PSSUQ and CSUQ are both overall satisfaction questionnaires. The PSSUQ items are appropriate for a usability testing situation, and the CSUQ items are appropriate for a field testing situation. Otherwise, the questionnaires are identical.
The psychometric evaluations of the PSSUQ (using data from a usability study) and the CSUQ (using data from a mail survey) were virtually identical. As with the after-scenario questionnaires, this consistency provides strong evidence of generalizability of results and wide applicability of the questionnaires.

Because these questionnaires have acceptable psychometric properties, usability practitioners can use them with confidence as standardized measurements of satisfaction for usability studies and tests (ASQ, PSSUQ) or field research (CSUQ).
(Practitioners should note that nothing prevents the addition of items to these questionnaires if a particular situation suggests the need. However, using these questionnaires as the foundation for special-purpose questionnaires ensures that practitioners can score the scales and subscales from the questionnaires, maintaining the advantages of standardized measurement.)

Standardized satisfaction measurements offer many advantages to the usability practitioner (Nunnally, 1978). Specifically, standardized measurements provide:

1 Objectivity.
A standardized measurement supports objectivity because it allows usability practitioners to independently verify the measurement statements of other practitioners.

2 Quantification.
Standardized measurements allow practitioners to report results in finer detail than they could using only personal judgment. Standardization also permits practitioners to use powerful methods of mathematics and statistics to better understand their results (Nunnally, 1978).

3 Communication.
It is easier for practitioners to communicate effectively when standardized measures are available. Inadequate efficiency and fidelity of communication in any field is an impediment to progress.

4 Economy.
Developing standardized measures requires a substantial amount of work. However, once developed, they are economical. There is rarely any need to re-evaluate standardized measures.

5 Scientific generalization.
Scientific generalization is at the heart of scientific work. Standardization is essential for assessing the generalization of results.

Conclusion
In conclusion, these questionnaires should be valuable additions to the repertoire of techniques that usability practitioners apply in the design and evaluation of computer systems.



IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - Printer-Scenario Questionnaire (PSQ) - Lewis of IBM

Printer-Scenario Questionnaire (PSQ)

Administration and Scoring.
As indicated in the body of the paper, use the ASQ rather than the PSQ.

Instructions and Items.
The questionnaire's instructions and items are:
For each of the items below, please circle the response that best describes your experience with the printer for this scenario.

1. Time to Complete Task

1 = Acceptable as is -- less time than expected
2 = Acceptable as is -- about right
3 = Needs slight improvement
4 = Needs moderate improvement
5 = Needs a lot of improvement
- = Unable to evaluate
Comments:

2. Ease of Performing Tasks

1 = Acceptable as is -- very easy
2 = Acceptable as is -- easy
3 = Needs slight improvement
4 = Needs moderate improvement
5 = Needs a lot of improvement
- = Unable to evaluate
Comments:

3. Satisfaction with Instructions/Publications

1 = Acceptable as is -- very satisfied
2 = Acceptable as is -- satisfied
3 = Needs slight improvement
4 = Needs moderate improvement
5 = Needs a lot of improvement
- = Unable to evaluate
Comments:



IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - After-Scenario Questionnaire (ASQ) - Lewis of IBM

After-Scenario Questionnaire (ASQ)

Administration and Scoring.
Give the questionnaire to a participant after he or
she has completed a scenario during a usability evaluation. Average (with the arithmetic mean) the scores from the three items to obtain the ASQ score for a participant's satisfaction with the system for a given scenario. Low scores are better than high scores due to the anchors used in the 7-point scales. If a participant does not answer an item or marks N/A, average the remaining items to obtain the ASQ score.

Instructions and Items.
The questionnaire's instructions and items are:
For each of the statements below, circle the rating of your choice.

Likert scale:
1 = strongly agree
2
3
4
5
6
7 = strongly disagree


1. Overall, I am satisfied with the ease of completing this task.

2. Overall, I am satisfied with the amount of time it took to complete this task.

3. Overall, I am satisfied with the support information (on-line help, messages,
documentation) when completing this task.



IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - Computer System Usability Questionnaire (CSUQ)

Computer System Usability Questionnaire (CSUQ)

Administration and Scoring.
Use the CSUQ rather than the PSSUQ when the
usability study is in a non-laboratory setting. Appendix Table 1 contains the rules for
calculating the CSUQ and PSSUQ scores.
_____________________________________________________________________________
Appendix Table 1. Rules for Calculating CSUQ/PSSUQ Scores
_____________________________________________________________________________
Score Name > Average the Responses to:
_____________________________________________________________________________
OVERALL > Items 1 through 19
SYSUSE > Items 1 through 8
INFOQUAL > Items 9 through 15
INTERQUAL > Items 16 through 18
_____________________________________________________________________________
Average the scores from the appropriate items to obtain the scale and subscale
scores. Low scores are better than high scores due to the anchors used in the 7-point
scales. If a participant does not answer an item or marks "N/A," then average the
remaining item scores.

Instructions and Items.
The questionnaire's instructions and items are:
This questionnaire (which starts on the following page) gives you an opportunity to express your satisfaction with the usability of your primary computer system. Your responses will help us understand what aspects of the system you are particularly concerned about and the aspects that satisfy you.
To as great a degree as possible, think about all the tasks that you have done with the system while you answer these questions.
Please read each statement and indicate how strongly you agree or disagree with the statement by circling a number on the scale. If a statement does not apply to you, circle N/A.
Whenever it is appropriate, please write comments to explain your answers.
Thank you!

Likert scale:
1 = strongly agree
2
3
4
5
6
7 = strongly disagree


1. Overall, I am satisfied with how easy it is to use this system.

2. It is simple to use this system.

3. I can effectively complete my work using this system.

4. I am able to complete my work quickly using this system.

5. I am able to efficiently complete my work using this system.

6. I feel comfortable using this system.

7. It was easy to learn to use this system.

8. I believe I became productive quickly using this system.

9. The system gives error messages that clearly tell me how to fix problems.

10. Whenever I make a mistake using the system, I recover easily and quickly.

11. The information (such as on-line help, on-screen messages and other documentation) provided with this system is clear.

12. It is easy to find the information I need.

13. The information provided with the system is easy to understand.

14. The information is effective in helping me complete my work.

15. The organization of information on the system screens is clear.

Note: The interface includes those items that you use to interact with the system. For example, some components of the interface are the keyboard, the mouse, the screens (including their use of graphics and language).

16. The interface of this system is pleasant.

17. I like using the interface of this system.

18. This system has all the functions and capabilities I expect it to have.

19. Overall, I am satisfied with this system.



IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 9 - Post-Study System Usability Questionnaire (PSSUQ) - Lewis of IBM

The Post-Study System Usability Questionnaire (PSSUQ)

Administration and Scoring.
Give the PSSUQ to participants after they have completed all the scenarios in a usability study.
You can calculate four scores from the responses to the PSSUQ items:
* the overall satisfaction score (OVERALL),
* system usefulness (SYSUSE),
* information quality (INFOQUAL) and
* interface quality (INTERQUAL).
Because research on an alternative form of the PSSUQ (the Computer System Usability Questionnaire, or CSUQ) confirmed and clarified (and slightly modified) the factor structure of the questionnaire, refer to Appendix Table 1 in the next section of this appendix for the current scoring rules of the PSSUQ.

Instructions and Items.
The questionnaire's instructions and items are:
This questionnaire, which starts on the following page, gives you an opportunity to tell us your reactions to the system you used. Your responses will help us understand what aspects of the system you are particularly concerned about and the aspects that satisfy you.
To as great a degree as possible, think about all the tasks that you have done with the system while you answer these questions.
Please read each statement and indicate how strongly you agree or disagree with the statement by circling a number on the scale. If a statement does not apply to you, circle N/A.
Please write comments to elaborate on your answers.
After you have completed this questionnaire, I'll go over your answers with you to make sure I
understand all of your responses.
Thank you!

Likert scale
1 = strongly agree
2
3
4
5
6
7 = strongly disagree


1. Overall, I am satisfied with how easy it is to use this system.

2. It was simple to use this system.

3. I could effectively complete the tasks and scenarios using this system.

4. I was able to complete the tasks and scenarios quickly using this system.

5. I was able to efficiently complete the tasks and scenarios using this system.

6. I felt comfortable using this system.

7. It was easy to learn to use this system.

8. I believe I could become productive quickly using this system.

9. The system gave error messages that clearly told me how to fix problems.

10. Whenever I made a mistake using the system, I could recover easily and quickly.

11. The information (such as on-line help, on-screen messages and other documentation)
provided with this system was clear.

12. It was easy to find the information I needed.

13. The information provided for the system was easy to understand.

14. The information was effective in helping me complete the tasks and scenarios.

15. The organization of information on the system screens was clear.

Note: The interface includes those items that you use to interact with the system. For example, some components of the interface are the keyboard, the mouse, the screens (including their use of graphics and language).

16. The interface of this system was pleasant.

17. I liked using the interface of this system.

18. This system has all the functions and capabilities I expect it to have.

19. Overall, I am satisfied with this system.

Appendix Table 1. Rules for Calculating CSUQ/PSSUQ Scores
_____________________________________________________________________________
Score Name > Average the Responses to:
_____________________________________________________________________________
OVERALL > Items 1 through 19
SYSUSE > Items 1 through 8
INFOQUAL > Items 9 through 15
INTERQUAL > Items 16 through 18
_____________________________________________________________________________


IBM Computer Usability Satisfaction Questionnaires:
Psychometric Evaluation and Instructions for Use
Technical Report 54.786
James R. Lewis
Human Factors Group
Boca Raton, FL

Source: http://drjim.0catch.com/usabqtr.pdf

Oct 8 - ASQ: After-Scenario Questionnaire

After-Scenario Questionnaire

Please rate the usability of the system for the previous scenario.
Try to respond to all the items.
For items that are not applicable, use: NA
Likert scale:
1 = strongly disagree
2
3
4
5
6
7 = strongly agree
NA


1. Overall, I am satisfied with the ease of completing the tasks in this scenario

2. Overall, I am satisfied with the amount of time it took to complete the tasks in this scenario

3. Overall, I am satisfied with the support information (online-line help, messages, documentation) when completing the tasks

My comments: ASQ questionnaire has only 3 questions/criteria.


After-Scenario Questionnaire
Based on: Lewis, J. R. (1995) IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7:1, 57-78. Abstract

Source: http://hcibib.org/perlman/question.cgi?form=ASQ