Examples of performance and satisfaction measures
B.1 Introduction
This annex provides examples of performance and satisfaction measures, appropriate for use in usability requirements created with the CISU-R. Because the CISU-R uses the definition of usability from ISO 9241 (see the definition in Clause 4.1), these examples focus on effectiveness, efficiency, and satisfaction measures.
B.2 Effectiveness
Effectiveness relates the goals of using the product to the accuracy and completeness with which these goals can be achieved. Common measures of effectiveness include task completion rate, frequency of errors, frequency of assists to the participant from the testers. It does not take account of how the goals were achieved, only the extent to which they were achieved.
B.2.1 Task completion rate
The completion rate is the percentage of participants who completely and correctly achieve each goal. If goals can be partially achieved (e.g., by incomplete or sub-optimum results) then it may also be useful to set requirements for partial goal achievement, scored on a scale of 0 to 100 % based on specified criteria related to the value of a partial result.
EXAMPLE 1 The percentage of customers who can successfully complete a transaction on a web site, or the percentage of users who can successfully record an hour long TV program with a Digital Video Recorder (DVR).
EXAMPLE 2 A spell-checking task might involve identifying and correcting 10 spelling errors. The completion rate might be calculated based on the percent of errors corrected. Another method for calculating completion rate is weighting; e.g., spelling errors in the title page of the document are judged to be twice as important as errors in the main body of text. The rationale for choosing a particular method of partial goal analysis should be stated, if such results are included in the requirements.
B.2.2 Errors
Although the requirements for effectiveness are based on error-free completion of the task, keeping track of the errors is particularly helpful in determining what aspects of the product need improvement. Errors are instances where test participants did not complete the task successfully, or had to attempt portions of the task more than once. Scoring of data should include a classification of types of errors.
B.2.3 Assists
When participants cannot proceed on a task, the test administrator sometimes gives direct procedural help in order to allow the test to proceed. This type of intervention is called an assist for the purposes of this standard.
If it is necessary to provide participants with assists, efficiency, and effectiveness metrics shall be provided for both unassisted and assisted conditions, and the number and type of assists shall be included as part of the test results. When assists are allowed or provided, the number and type of assists shall be included as part of the test results.
B.3 Efficiency
B.3.1 Measures
Efficiency relates the level of effectiveness achieved to the quantity of resources expended. Efficiency is generally assessed by the mean time taken to achieve the task. Efficiency may also relate to other resources (e.g. total cost of usage). A common measure of efficiency is time on task, which can be defined as the mean time taken to complete each task, together with the range and standard deviation of times across participants.
B.3.2 Relative user efficiency
Relative user efficiency is the mean time taken by users who successfully achieve a goal divided by the time taken by an expert.
B.3.3 Completion rate/mean time-on-task
Completion Rate divided by Mean Time-On-Task is the core measure of efficiency. It is the percentage of users who were successful (or percentage goal achievement) for every unit of time.
This formula shows that as the time on task decreases, one would expect users to be more successful. A very efficient product has a high percentage of successful users in a small amount of time. This allows customers to compare fast error-prone interfaces to slow easy interfaces.
EXAMPLE 1 An error-prone interface such as a command line interface using wildcards to delete groups of files would typically show a low completion rate divided by mean time-on-task.
EXAMPLE 2 A slower, but easier interface such as using a mouse and keyboard to drag each file to the trash would typically show a high completion rate divided by mean time-on-task.
B.4 Satisfaction
Satisfaction describes a user’s subjective response when using the product. User satisfaction may be related to whether a users wants use a product and may affect performance in some cases. Questionnaires to measure satisfaction and associated attitudes often use Likert or semantic differential scales.
A variety of instruments are available for measuring user satisfaction of interactive products, and many organizations create their own. Some examples of standard questionnaires include: ASQ [5], CUSI [6], PSSUQ [6], QUIS [3], SUMI [4], and SUS [7]).
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment