Examples of performance and satisfaction measures
B.1 Introduction
This annex provides examples of performance and satisfaction measures, appropriate for use in usability requirements created with the CISU-R. Because the CISU-R uses the definition of usability from ISO 9241 (see the definition in Clause 4.1), these examples focus on effectiveness, efficiency, and satisfaction measures.
B.2 Effectiveness
Effectiveness relates the goals of using the product to the accuracy and completeness with which these goals can be achieved. Common measures of effectiveness include task completion rate, frequency of errors, frequency of assists to the participant from the testers. It does not take account of how the goals were achieved, only the extent to which they were achieved.
B.2.1 Task completion rate
The completion rate is the percentage of participants who completely and correctly achieve each goal. If goals can be partially achieved (e.g., by incomplete or sub-optimum results) then it may also be useful to set requirements for partial goal achievement, scored on a scale of 0 to 100 % based on specified criteria related to the value of a partial result.
EXAMPLE 1 The percentage of customers who can successfully complete a transaction on a web site, or the percentage of users who can successfully record an hour long TV program with a Digital Video Recorder (DVR).
EXAMPLE 2 A spell-checking task might involve identifying and correcting 10 spelling errors. The completion rate might be calculated based on the percent of errors corrected. Another method for calculating completion rate is weighting; e.g., spelling errors in the title page of the document are judged to be twice as important as errors in the main body of text. The rationale for choosing a particular method of partial goal analysis should be stated, if such results are included in the requirements.
B.2.2 Errors
Although the requirements for effectiveness are based on error-free completion of the task, keeping track of the errors is particularly helpful in determining what aspects of the product need improvement. Errors are instances where test participants did not complete the task successfully, or had to attempt portions of the task more than once. Scoring of data should include a classification of types of errors.
B.2.3 Assists
When participants cannot proceed on a task, the test administrator sometimes gives direct procedural help in order to allow the test to proceed. This type of intervention is called an assist for the purposes of this standard.
If it is necessary to provide participants with assists, efficiency, and effectiveness metrics shall be provided for both unassisted and assisted conditions, and the number and type of assists shall be included as part of the test results. When assists are allowed or provided, the number and type of assists shall be included as part of the test results.
B.3 Efficiency
B.3.1 Measures
Efficiency relates the level of effectiveness achieved to the quantity of resources expended. Efficiency is generally assessed by the mean time taken to achieve the task. Efficiency may also relate to other resources (e.g. total cost of usage). A common measure of efficiency is time on task, which can be defined as the mean time taken to complete each task, together with the range and standard deviation of times across participants.
B.3.2 Relative user efficiency
Relative user efficiency is the mean time taken by users who successfully achieve a goal divided by the time taken by an expert.
B.3.3 Completion rate/mean time-on-task
Completion Rate divided by Mean Time-On-Task is the core measure of efficiency. It is the percentage of users who were successful (or percentage goal achievement) for every unit of time.
This formula shows that as the time on task decreases, one would expect users to be more successful. A very efficient product has a high percentage of successful users in a small amount of time. This allows customers to compare fast error-prone interfaces to slow easy interfaces.
EXAMPLE 1 An error-prone interface such as a command line interface using wildcards to delete groups of files would typically show a low completion rate divided by mean time-on-task.
EXAMPLE 2 A slower, but easier interface such as using a mouse and keyboard to drag each file to the trash would typically show a high completion rate divided by mean time-on-task.
B.4 Satisfaction
Satisfaction describes a user’s subjective response when using the product. User satisfaction may be related to whether a users wants use a product and may affect performance in some cases. Questionnaires to measure satisfaction and associated attitudes often use Likert or semantic differential scales.
A variety of instruments are available for measuring user satisfaction of interactive products, and many organizations create their own. Some examples of standard questionnaires include: ASQ [5], CUSI [6], PSSUQ [6], QUIS [3], SUMI [4], and SUS [7]).
Showing posts with label NIST. Show all posts
Showing posts with label NIST. Show all posts
Friday, September 11, 2009
Sep 11 - Usability requirements format (CISU-R part 4)
Usability Requirements Format
Introduction
This annex provides a list of elements that should be used in a usability requirements document, including document and product information elements. It may either be used as a template, or may be used as a checklist for creating a format for usability requirements created with the CISU-R.
The format of the information in this annex is consistent with the CIF (ISO/IEC 25062).
A.2 Title page
The title page contains:
1 Identification of the requirements as conforming to Common Industry Specifications for Usability - Requirements v0.88.
2 The name of the product and version that the requirements are for.
3 The date the requirements were prepared.
4 The organization name.
5 Contact details for questions and/or clarifications.
6 Names of the people who produced the requirements.
A.3 Executive summary
The Executive Summary provides a high level overview of the requirements. The intent of this section is to provide information for people who may not read the technical body of this document. This section should begin on a new page and end with a page break to facilitate its use as a stand-alone summary.
The executive summary:
a)Explains the purpose of the requirements.
b)States the level of conformance.
c)Provides a high level overview of the requirements that includes:
−Name, version, and brief description of the product.
−Summary of the type of users, tasks, any associated equipment (hardware, software, and materials), and the physical and social environments in which the product is intended to be used.
−A list of the task scenarios, with criteria for effectiveness, efficiency, and satisfaction (for level 2 and 3 conformance).
d)Explains the reason for and nature of the requirements.
A.4 Product details
A.4.1 Product description
A product description includes:
1 The formal product name and version.
2 The parts of the product for which requirements are being provided.
3 The user groups for which the product is intended.
4 Whether the product is intended to be "walk up and use", or whether any documentation, training materials or course has to be studied before the product is used.
5 The type of user work that is supported by the product.
6 Description of the environment in which the product should be used.
7 Support for any groups with special needs.
A.4.2 Product objectives
Product objectives include:
- The overall objectives for the product and specific objectives for any subset of usage.
- Any known or intended functions and components that support key objectives.
A.5 Context of use and scenarios
The context of use and scenarios include all of the information and headings specified in Clause 6.2:
a. Stakeholders
b. User groups
c. Goals and tasks
d. Technical environment (equipment)
e. Physical and social environments
f. Scenarios of use for the most important goals
A.6 Usability performance and satisfaction criteria and values
The usability performance and satisfaction criteria include the information specified in Clause 6.3, including
a. The goals that will form the basis of the requirements, with scenarios of usage, with an explanation of why these goals and scenarios were selected.
b. The relative importance of each goal.
c. A definition of the performance and satisfaction criteria appropriate for the scenario and
d. Target values or a range of anticipated values for these criteria, which may be:
−a definite requirement, or
−a provisional requirement subject to further negotiation, or
−an objective for guidance.
A.7 Requirements for testing
If a protocol for a usability test is included, the information specified in Annex C may be used.
Introduction
This annex provides a list of elements that should be used in a usability requirements document, including document and product information elements. It may either be used as a template, or may be used as a checklist for creating a format for usability requirements created with the CISU-R.
The format of the information in this annex is consistent with the CIF (ISO/IEC 25062).
A.2 Title page
The title page contains:
1 Identification of the requirements as conforming to Common Industry Specifications for Usability - Requirements v0.88.
2 The name of the product and version that the requirements are for.
3 The date the requirements were prepared.
4 The organization name.
5 Contact details for questions and/or clarifications.
6 Names of the people who produced the requirements.
A.3 Executive summary
The Executive Summary provides a high level overview of the requirements. The intent of this section is to provide information for people who may not read the technical body of this document. This section should begin on a new page and end with a page break to facilitate its use as a stand-alone summary.
The executive summary:
a)Explains the purpose of the requirements.
b)States the level of conformance.
c)Provides a high level overview of the requirements that includes:
−Name, version, and brief description of the product.
−Summary of the type of users, tasks, any associated equipment (hardware, software, and materials), and the physical and social environments in which the product is intended to be used.
−A list of the task scenarios, with criteria for effectiveness, efficiency, and satisfaction (for level 2 and 3 conformance).
d)Explains the reason for and nature of the requirements.
A.4 Product details
A.4.1 Product description
A product description includes:
1 The formal product name and version.
2 The parts of the product for which requirements are being provided.
3 The user groups for which the product is intended.
4 Whether the product is intended to be "walk up and use", or whether any documentation, training materials or course has to be studied before the product is used.
5 The type of user work that is supported by the product.
6 Description of the environment in which the product should be used.
7 Support for any groups with special needs.
A.4.2 Product objectives
Product objectives include:
- The overall objectives for the product and specific objectives for any subset of usage.
- Any known or intended functions and components that support key objectives.
A.5 Context of use and scenarios
The context of use and scenarios include all of the information and headings specified in Clause 6.2:
a. Stakeholders
b. User groups
c. Goals and tasks
d. Technical environment (equipment)
e. Physical and social environments
f. Scenarios of use for the most important goals
A.6 Usability performance and satisfaction criteria and values
The usability performance and satisfaction criteria include the information specified in Clause 6.3, including
a. The goals that will form the basis of the requirements, with scenarios of usage, with an explanation of why these goals and scenarios were selected.
b. The relative importance of each goal.
c. A definition of the performance and satisfaction criteria appropriate for the scenario and
d. Target values or a range of anticipated values for these criteria, which may be:
−a definite requirement, or
−a provisional requirement subject to further negotiation, or
−an objective for guidance.
A.7 Requirements for testing
If a protocol for a usability test is included, the information specified in Annex C may be used.
Sep 11 - Usability Requirements Specification (CISU-R part 3)
6. Usability requirements specification
6.1 General
Usability requirements based on the CISU-R have three parts: the context of use, the performance and satisfaction criteria, and the test method. All usability requirements include all three parts, but there are three levels of compliance.
The context of use: a description of the intended users, their goals, associated equipment (including hardware, software, and materials), the physical and social environment in which the product will be used, and examples of scenarios of use. (see Clause 6.1)
- A complete description of the context of use as specified in Section 6.2 shall be included in all usability requirements (Levels 1, 2, and 3). Additional detail may be added during the design and development process.
Performance and satisfaction criteria: ways in which the usability of the product can be measured, specified for the main scenarios of use. They may include target values, or may specify how these values will be identified (see Clause 6.3). These criteria can be specified to three levels of completeness and precision:
- Level 1: the requirements shall identify the criteria appropriate for successful use of the product, and the relative importance of each and identifying those that are practical to measure directly (see Clause 6.3.1).
- Level 2: the requirements shall specify target values for the criteria, or a range of acceptable values, (see Clause 6.3.2).
- Level 3: the requirements shall include specific values for measures of user performance and satisfaction to be tested, or a method of determining those values (see Clause 6.3.3).
The test method: how the product will be tested to determine whether the usability requirements have been met. The specification of the test method may include only an identification of the appropriate test methods, may describe a method and identify the test context, or may include a complete specification of a test method (see Clause 6.4). These criteria can be specified to three levels or detail:
- Level 1: the requirements shall identify the types of methods that are appropriate for the product and criteria selected (see Clause 6.4.1).
- Level 2: the requirements shall include a preliminary test method and the criteria to be evaluated (see Clause 6.4.2).
- Level 3: the requirements shall include a complete test protocol for a user performance test, as specified in Annex C (see Clause 6.4.3).
These levels are intended to ensure that usability requirements can be developed with the CISU-R for all types of projects, from the smallest or most informal to the most complex or formally specified products.
The three levels of compliance allow usability requirements to be written to an appropriate level of completeness and precision for the product.
At the same time, they encourage all projects to reach at least Level 1 compliance.
6.2 Context of use
6.2.1. General
For all levels of compliance with this standard, the context of use shall include descriptions of:
- Stakeholders
- User groups
- Goals and tasks
- Technical environment (equipment)
- Physical and social environments
- Scenarios of use for the most important goals
6.2.2. Stakeholders
Usability requirements shall include a description of all groups of people who are identified as stakeholders, including:
- A list of all groups who have a legitimate interest in the product throughout its life cycle.
- Their roles and interests in the product. EXAMPLE 1. Role: marketing product manager; interest: business needs solutions EXAMPLE 2 Role: user/administrative assistant; interest facilitation of administrative duties
- If any of the stakeholders will not be considered in identifying requirements, the rationale for excluding them.
6.2.3. User groups
Usability requirements shall include a list of all anticipated user groups.
Requirements should be provided for the user groups who use a product most frequently or who are critical for business purposes. Key characteristics and capabilities of these primary user groups shall be defined.
6.2.4. Goals
The main goals for each user group shall be listed, without reference to any specific means of achieving them. The goals should be an intended outcome of value to the user or business, for example: accurately completing a particular form, locating the most relevant information, or successfully setting up a computer.
6.2.5. Technical environment (equipment)
Any relevant aspects of the intended or anticipated computing or other technical environment shall be specified.
Examples (particularly applicable to software products) include:
1 Hardware configuration: e.g., processor speed, memory size, network, storage, input, and output devices.
2 Screen type (CRT or LCD), resolution, and color depth. If relevant, also include display (monitor) size and whether multiple monitors are required.
3 Print media size and print resolution.
4 Whether visual interface elements (such as text) can vary in size and the size(s) available.
5 Software configuration: e.g., browser version, operating system version, middleware, database. 6 Assistive technologies available to users with disabilities.
7 Documentation and support materials.
6.2.6. Physical and social environments
Any aspects of the expected physical and social environments that can influence usability shall be specified:
1 Physical environment in which the product will be used, including location and relevant physical conditions, such as temperature or lighting.
2 Organizational environment, including group work dynamics, time pressures, supervision, and support.
3 Any physical, health or safety issues.
4 Any possible financial or security risks
6.2.7. Scenarios of use
The goals and tasks that will form the basis of the requirements shall be illustrated with scenarios of use. These scenarios describe how users meet their goals. They are examples of users’ activities, motivations, and how they carry out their tasks, using the product in a specific situation.
6.2.8. Training
If users are expected to study any documentation, training materials or courses before using the product, the following information shall be specified:
1 Which user groups are expected to study the materials.
2 What previous knowledge or experience is required.
3 Which goals and tasks are included in the training materials.
6.3 Usability performance and satisfaction criteria
6.3.1. Identifying relevant usability criteria (Level 1)
Performance and satisfaction criteria identify the measures for the usability of a product. The first level in specifying performance and satisfaction criteria is to identify the criteria relevant to the context of use for the product.
The following information shall be provided for Level 1 compliance:
a) The types of performance and satisfaction criteria appropriate for successful use of the product.
−These might include performance criteria such as task completion rate or time on task, subjective “ease of use” scores, indirect measures such as number of steps in a task, or a list of features to support usability that can be evaluated with a checklist.
−Different user groups or scenarios may have different criteria.
b) The relative importance of each criteria to the success of the product.
Examples of performance and satisfaction criteria include: the unassisted completion rate (effectiveness), the mean time taken to successfully complete each goal (efficiency), satisfaction scores on a specific questionnaire, efficiency relative to a past benchmark or other product. Annex B provides additional information about selecting appropriate criteria and best practices for how they should be measured.
6.3.2. Defining target values for criteria (Level 2)
The second step in specifying performance and satisfaction criteria is to provide target values for the criteria, or indicate a method for determining those values later in the project.
Target values are the minimum acceptable value, or a range of acceptable values, for the criteria. They may be based on the value for an existing or competitor system used for the same tasks and goals, based on expert judgment or specified by stakeholders. Additional target values better than the minimum acceptable value may also be specified.
For Level 2 compliance, the usability criteria shall include target values or a range of acceptable values for these criteria. Target values may be in actual numbers, percentages, average or means, a range of values, or a scale. they may be absolute, or relative to performance benchmarks
Each target value may be given as:
a definite requirement, or
a provisional requirement subject to further negotiation, or
an objective for guidance.
For effectiveness, efficiency, and satisfaction criteria, either target values shall be given, or a note provided explaining that either:
a target value will be provided later, or
why a target value cannot be established (for example for a completely new product), or
why no target value is needed (for example the measure is of low importance to the user or organization).
6.3.3. Defining specific criterion values (Level 3)
For Level 3 compliance the requirements shall include specific values for each criterion. These may be validated by benchmark usability testing, business requirements, or other methods.
6.3.4. Including use of learning materials in the criteria
If any documentation, training materials or course will be studied before the product is used, criteria for the usability of the training materials in at least one training scenario (effectiveness, efficiency, and satisfaction when completing training goals) should be included.
EXAMPLE 1 Given 2 hours of training with the online training program, a user will be able to complete the item identification task under 1 minute and with fewer than 2 errors.
EXAMPLE 2 After reading the owner’s manual a user will be able to access and listen to a cellphone message on the first attempt.
6.4 Methods for testing
6.4.1. Identifying evaluation methods (Level 1)
The final part of a usability requirement identifies methods to evaluate whether the criteria have been met and the context in which the criteria will be measured.
The first level in specifying test methods is to identify methods that are appropriate for the product. For some products these methods may include checklists or other inspection methods, or indirect measures such as business metrics, but ideally, they include methods in which people who are representative of the user groups test the product.
For Level 1 compliance, the usability requirements shall include a list of usability methods that can be used to determine whether the usability requirements have been met.
6.4.2. Specifying a preliminary usability evaluation method (Level 2)
The second level in specifying test methods is to describe the method to be used to test whether the usability requirements have been met, and the context in which the measurements will be made. This level adds enough information to describe the basic test method to a usability professional, but is not a complete specification of the method.
For Level 2 compliance, the preliminary evaluation method shall include a description of the methods used to measure the product on all of the criteria.
If a usability test method is specified, the description shall include:
1 Goals of the test, including task scenarios to be tested and criteria for successful completion of each goal (see Annex C.2.1.a and b).
2 The user groups (identified in the context of use) to be included in the test (see Annex C.1.b).
3 The type of test facility, including the setting and type of space, and how relevant aspects of the intended environment will be simulated (see Annex C2.2).
4 Computing environment, which may be specific or anticipated (for example, specifying the "current version at the time of the test" (see C.2.3).
5 A general description of the test procedure (see C.3.1).
My Comments: For my PhD research, one of my deliverables is procedure of usability evaluation using UET. I will follow this Standard for description of usability test/evaluation method.
6.4.3. Specifying a complete usability test method (Level 3)
For level 3 compliance, a full protocol for a usability test shall be specified, providing the information required in Annex C.
The specification in Annex C is consistent with ISO/IEC 25062 (Common Industry Format (CIF) for usability test reports), which should be used for reporting the results. ISO/IEC 25062 requires reporting on the results of measurements of efficiency, effectiveness and satisfaction, or an explanation of why any of these metrics are not meaningful for this product.
6.5 Traceability
The following information should be documented to assist the development organization in maintaining and refining the requirements:
1 The source of the requirements.
2 How the usability requirements support the stakeholders’ business objectives for the product.
3 Why the goals were selected.
4 The source of each goal.
5 Who requested this requirement.
6 Whether the requirement is essential or desirable.
6.6 Document control
The following information should be included to track different versions of the requirements document:
1 Version number
2 Date of change
3 Nature of change
4 Person making the change
5 Person approving the change.
Source:
NISTIR 7432
Common Industry Specification for Usability - Requirements
June 2007
NIST
National Institute of Standards and Technology
Technology Administration, U.S. Department of Commerce
6.1 General
Usability requirements based on the CISU-R have three parts: the context of use, the performance and satisfaction criteria, and the test method. All usability requirements include all three parts, but there are three levels of compliance.
The context of use: a description of the intended users, their goals, associated equipment (including hardware, software, and materials), the physical and social environment in which the product will be used, and examples of scenarios of use. (see Clause 6.1)
- A complete description of the context of use as specified in Section 6.2 shall be included in all usability requirements (Levels 1, 2, and 3). Additional detail may be added during the design and development process.
Performance and satisfaction criteria: ways in which the usability of the product can be measured, specified for the main scenarios of use. They may include target values, or may specify how these values will be identified (see Clause 6.3). These criteria can be specified to three levels of completeness and precision:
- Level 1: the requirements shall identify the criteria appropriate for successful use of the product, and the relative importance of each and identifying those that are practical to measure directly (see Clause 6.3.1).
- Level 2: the requirements shall specify target values for the criteria, or a range of acceptable values, (see Clause 6.3.2).
- Level 3: the requirements shall include specific values for measures of user performance and satisfaction to be tested, or a method of determining those values (see Clause 6.3.3).
The test method: how the product will be tested to determine whether the usability requirements have been met. The specification of the test method may include only an identification of the appropriate test methods, may describe a method and identify the test context, or may include a complete specification of a test method (see Clause 6.4). These criteria can be specified to three levels or detail:
- Level 1: the requirements shall identify the types of methods that are appropriate for the product and criteria selected (see Clause 6.4.1).
- Level 2: the requirements shall include a preliminary test method and the criteria to be evaluated (see Clause 6.4.2).
- Level 3: the requirements shall include a complete test protocol for a user performance test, as specified in Annex C (see Clause 6.4.3).
These levels are intended to ensure that usability requirements can be developed with the CISU-R for all types of projects, from the smallest or most informal to the most complex or formally specified products.
The three levels of compliance allow usability requirements to be written to an appropriate level of completeness and precision for the product.
At the same time, they encourage all projects to reach at least Level 1 compliance.
6.2 Context of use
6.2.1. General
For all levels of compliance with this standard, the context of use shall include descriptions of:
- Stakeholders
- User groups
- Goals and tasks
- Technical environment (equipment)
- Physical and social environments
- Scenarios of use for the most important goals
6.2.2. Stakeholders
Usability requirements shall include a description of all groups of people who are identified as stakeholders, including:
- A list of all groups who have a legitimate interest in the product throughout its life cycle.
- Their roles and interests in the product. EXAMPLE 1. Role: marketing product manager; interest: business needs solutions EXAMPLE 2 Role: user/administrative assistant; interest facilitation of administrative duties
- If any of the stakeholders will not be considered in identifying requirements, the rationale for excluding them.
6.2.3. User groups
Usability requirements shall include a list of all anticipated user groups.
Requirements should be provided for the user groups who use a product most frequently or who are critical for business purposes. Key characteristics and capabilities of these primary user groups shall be defined.
6.2.4. Goals
The main goals for each user group shall be listed, without reference to any specific means of achieving them. The goals should be an intended outcome of value to the user or business, for example: accurately completing a particular form, locating the most relevant information, or successfully setting up a computer.
6.2.5. Technical environment (equipment)
Any relevant aspects of the intended or anticipated computing or other technical environment shall be specified.
Examples (particularly applicable to software products) include:
1 Hardware configuration: e.g., processor speed, memory size, network, storage, input, and output devices.
2 Screen type (CRT or LCD), resolution, and color depth. If relevant, also include display (monitor) size and whether multiple monitors are required.
3 Print media size and print resolution.
4 Whether visual interface elements (such as text) can vary in size and the size(s) available.
5 Software configuration: e.g., browser version, operating system version, middleware, database. 6 Assistive technologies available to users with disabilities.
7 Documentation and support materials.
6.2.6. Physical and social environments
Any aspects of the expected physical and social environments that can influence usability shall be specified:
1 Physical environment in which the product will be used, including location and relevant physical conditions, such as temperature or lighting.
2 Organizational environment, including group work dynamics, time pressures, supervision, and support.
3 Any physical, health or safety issues.
4 Any possible financial or security risks
6.2.7. Scenarios of use
The goals and tasks that will form the basis of the requirements shall be illustrated with scenarios of use. These scenarios describe how users meet their goals. They are examples of users’ activities, motivations, and how they carry out their tasks, using the product in a specific situation.
6.2.8. Training
If users are expected to study any documentation, training materials or courses before using the product, the following information shall be specified:
1 Which user groups are expected to study the materials.
2 What previous knowledge or experience is required.
3 Which goals and tasks are included in the training materials.
6.3 Usability performance and satisfaction criteria
6.3.1. Identifying relevant usability criteria (Level 1)
Performance and satisfaction criteria identify the measures for the usability of a product. The first level in specifying performance and satisfaction criteria is to identify the criteria relevant to the context of use for the product.
The following information shall be provided for Level 1 compliance:
a) The types of performance and satisfaction criteria appropriate for successful use of the product.
−These might include performance criteria such as task completion rate or time on task, subjective “ease of use” scores, indirect measures such as number of steps in a task, or a list of features to support usability that can be evaluated with a checklist.
−Different user groups or scenarios may have different criteria.
b) The relative importance of each criteria to the success of the product.
Examples of performance and satisfaction criteria include: the unassisted completion rate (effectiveness), the mean time taken to successfully complete each goal (efficiency), satisfaction scores on a specific questionnaire, efficiency relative to a past benchmark or other product. Annex B provides additional information about selecting appropriate criteria and best practices for how they should be measured.
6.3.2. Defining target values for criteria (Level 2)
The second step in specifying performance and satisfaction criteria is to provide target values for the criteria, or indicate a method for determining those values later in the project.
Target values are the minimum acceptable value, or a range of acceptable values, for the criteria. They may be based on the value for an existing or competitor system used for the same tasks and goals, based on expert judgment or specified by stakeholders. Additional target values better than the minimum acceptable value may also be specified.
For Level 2 compliance, the usability criteria shall include target values or a range of acceptable values for these criteria. Target values may be in actual numbers, percentages, average or means, a range of values, or a scale. they may be absolute, or relative to performance benchmarks
Each target value may be given as:
a definite requirement, or
a provisional requirement subject to further negotiation, or
an objective for guidance.
For effectiveness, efficiency, and satisfaction criteria, either target values shall be given, or a note provided explaining that either:
a target value will be provided later, or
why a target value cannot be established (for example for a completely new product), or
why no target value is needed (for example the measure is of low importance to the user or organization).
6.3.3. Defining specific criterion values (Level 3)
For Level 3 compliance the requirements shall include specific values for each criterion. These may be validated by benchmark usability testing, business requirements, or other methods.
6.3.4. Including use of learning materials in the criteria
If any documentation, training materials or course will be studied before the product is used, criteria for the usability of the training materials in at least one training scenario (effectiveness, efficiency, and satisfaction when completing training goals) should be included.
EXAMPLE 1 Given 2 hours of training with the online training program, a user will be able to complete the item identification task under 1 minute and with fewer than 2 errors.
EXAMPLE 2 After reading the owner’s manual a user will be able to access and listen to a cellphone message on the first attempt.
6.4 Methods for testing
6.4.1. Identifying evaluation methods (Level 1)
The final part of a usability requirement identifies methods to evaluate whether the criteria have been met and the context in which the criteria will be measured.
The first level in specifying test methods is to identify methods that are appropriate for the product. For some products these methods may include checklists or other inspection methods, or indirect measures such as business metrics, but ideally, they include methods in which people who are representative of the user groups test the product.
For Level 1 compliance, the usability requirements shall include a list of usability methods that can be used to determine whether the usability requirements have been met.
6.4.2. Specifying a preliminary usability evaluation method (Level 2)
The second level in specifying test methods is to describe the method to be used to test whether the usability requirements have been met, and the context in which the measurements will be made. This level adds enough information to describe the basic test method to a usability professional, but is not a complete specification of the method.
For Level 2 compliance, the preliminary evaluation method shall include a description of the methods used to measure the product on all of the criteria.
If a usability test method is specified, the description shall include:
1 Goals of the test, including task scenarios to be tested and criteria for successful completion of each goal (see Annex C.2.1.a and b).
2 The user groups (identified in the context of use) to be included in the test (see Annex C.1.b).
3 The type of test facility, including the setting and type of space, and how relevant aspects of the intended environment will be simulated (see Annex C2.2).
4 Computing environment, which may be specific or anticipated (for example, specifying the "current version at the time of the test" (see C.2.3).
5 A general description of the test procedure (see C.3.1).
My Comments: For my PhD research, one of my deliverables is procedure of usability evaluation using UET. I will follow this Standard for description of usability test/evaluation method.
6.4.3. Specifying a complete usability test method (Level 3)
For level 3 compliance, a full protocol for a usability test shall be specified, providing the information required in Annex C.
The specification in Annex C is consistent with ISO/IEC 25062 (Common Industry Format (CIF) for usability test reports), which should be used for reporting the results. ISO/IEC 25062 requires reporting on the results of measurements of efficiency, effectiveness and satisfaction, or an explanation of why any of these metrics are not meaningful for this product.
6.5 Traceability
The following information should be documented to assist the development organization in maintaining and refining the requirements:
1 The source of the requirements.
2 How the usability requirements support the stakeholders’ business objectives for the product.
3 Why the goals were selected.
4 The source of each goal.
5 Who requested this requirement.
6 Whether the requirement is essential or desirable.
6.6 Document control
The following information should be included to track different versions of the requirements document:
1 Version number
2 Date of change
3 Nature of change
4 Person making the change
5 Person approving the change.
Source:
NISTIR 7432
Common Industry Specification for Usability - Requirements
June 2007
NIST
National Institute of Standards and Technology
Technology Administration, U.S. Department of Commerce
Wednesday, September 9, 2009
Sep 9,10 - NISTIR 7432 - Common Industry Specification for Usability - Requirements (CISU-R)
NISTIR 7432
Common Industry Specification for Usability - Requirements
Information Access Division
Information Technology Laboratory
June 2007
NIST
National Institute of Standards and Technology
Technology Administration, U.S. Department of Commerce
Introduction
The Common Industry Specification for Usability - Requirements (CISU-R) helps usability professionals, product managers, and others working in product design and development to create usability requirements.
It sets standards for specifying usability requirements, which include three types of information:
Common Industry Specification for Usability - Requirements
Information Access Division
Information Technology Laboratory
June 2007
NIST
National Institute of Standards and Technology
Technology Administration, U.S. Department of Commerce
Introduction
The Common Industry Specification for Usability - Requirements (CISU-R) helps usability professionals, product managers, and others working in product design and development to create usability requirements.
It sets standards for specifying usability requirements, which include three types of information:
The context of use: the intended users, their goals and tasks, associated equipment, and the physical and social environment in which the product can be used.
Performance and satisfaction criteria: measures of usability for the product.
The test method and context of testing: the method to be used to test whether the usability requirements have been met and the context in which the measurements will be made.
Underlying the specific goals of the standard is a deeper goal of creating useful and usable products that allow users to complete their tasks efficiently, effectively, and with satisfaction.
To support this deeper goal, the CISU-R provides a structure for requirements to help stakeholders:
Document the context of use for a product, including definitions of the expected technical, physical, and social environments, user groups, goals for use of the product and scenarios of use, as defined in Clause 6.2.
Write usability requirements in sufficient detail to make an effective contribution to design and development. (A suggested outline for usability requirements is provided in Annex A.)
Relate usability requirements to stakeholder requirements (including user, customer, and business) for successful use of a product and increased productivity.
Define usability criteria that can be empirically validated. (Annex B provides examples of usability criteria.)
Define the method for testing the product against the criteria. (Annex C identifies the information to be specified for the test method.)
Create requirements that are useful throughout the product design and development process, providing input to the design process early in a project and adding more detailed information about criteria and methods as it is available.
Usability requirements created with the CISU-R can be updated throughout a product’s lifecycle. A project may choose to create complete requirements, including all information specified in the CISU-R, or may adopt a level of conformance that meets stakeholder and business goals.
The CISU-R can help integrate the creation of usability requirements into any design or development process. (Annex D contains an example of a user centered design process that incorporates the development of usability requirements under the CISU-R.)
-Requirements can meet one of three levels of compliance with the CISU-R. Each level builds on the previous one, allowing the usability requirements to be developed over time, with increasing detail and precision. This approach allows the CISU-R to be used for all types of projects, from the smallest or most informal to complex or formally specified products.
-Customers, users, and development teams can use the CISU-R as a communications tool to understand and specify requirements within an organization or as the basis for a contractual relationship between companies. The three levels of compliance provide the flexibility to match the level of detail and formality of the usability requirements to business needs.
-Requirements may also include indirect measures of usability, such as post-release satisfaction tests, or business performance requirements, such as completed transactions. These criteria are not intended for evaluation using a summative usability test, as required by the third level of conformance, but can be useful in establishing business or performance requirements.
-The CISU-R recognizes that usability requirements are just one type of product requirement. They complement functional, business, process, safety, and non-functional (quality) requirements. The information gathered in creating usability requirements can be used to help define other user requirements (for example, for features of the interface). (Annex E lists other types of user and usability requirements.)
The CISU-R focuses on the information needed to create usability requirements that are useful for design and development, rather than dictating a specific process or user centered design activities. (Annex E contains an example of a user centered design process that incorporates the CISU-R.)
-In situations where there are already processes and document formats in place, stakeholders can incorporate the CISU-R into that process by ensuring that the information specified here is included in any usability requirements.
-If there is no established format, requirements can use the information specified in Annex A as a guide. (Examples of all three levels are included in Annexes F, G, and H as a reference.)
The process of creating requirements with the CISU-R can be incorporated into many user centred design processes. For example, in ISO 13407, specifying requirements is one activity in an iterative process. The CISU-R provides details for how to complete this activity, as shown in Figure 1.
To ensure that the requirements are correct, other user centered design activities (such as competitive analysis, interviews, surveys, focus groups, field studies, task analysis, benchmark usability tests, or paper prototyping) can be used early in the development process to obtain feedback from users to iteratively refine requirements.
The CISU-R complements other user centered design standards.
-It uses the definition of usability in ISO 9241-11: the effectiveness, efficiency, and satisfaction with which the intended users can achieve their tasks in the intended context of product use.
-Usability requirements created using the CISU-R are consistent with the recommendations of ISO 13407.
-Usability tests conducted to measure whether usability requirements have been met can be reported using the Common Industry Format (CIF) (ISO/IEC 25062).
-The development of usability requirements may be supported by the use of standards such as ISO 9241 or ISO 9126.
Performance and satisfaction criteria: measures of usability for the product.
The test method and context of testing: the method to be used to test whether the usability requirements have been met and the context in which the measurements will be made.
Underlying the specific goals of the standard is a deeper goal of creating useful and usable products that allow users to complete their tasks efficiently, effectively, and with satisfaction.
To support this deeper goal, the CISU-R provides a structure for requirements to help stakeholders:
Document the context of use for a product, including definitions of the expected technical, physical, and social environments, user groups, goals for use of the product and scenarios of use, as defined in Clause 6.2.
Write usability requirements in sufficient detail to make an effective contribution to design and development. (A suggested outline for usability requirements is provided in Annex A.)
Relate usability requirements to stakeholder requirements (including user, customer, and business) for successful use of a product and increased productivity.
Define usability criteria that can be empirically validated. (Annex B provides examples of usability criteria.)
Define the method for testing the product against the criteria. (Annex C identifies the information to be specified for the test method.)
Create requirements that are useful throughout the product design and development process, providing input to the design process early in a project and adding more detailed information about criteria and methods as it is available.
Usability requirements created with the CISU-R can be updated throughout a product’s lifecycle. A project may choose to create complete requirements, including all information specified in the CISU-R, or may adopt a level of conformance that meets stakeholder and business goals.
The CISU-R can help integrate the creation of usability requirements into any design or development process. (Annex D contains an example of a user centered design process that incorporates the development of usability requirements under the CISU-R.)
-Requirements can meet one of three levels of compliance with the CISU-R. Each level builds on the previous one, allowing the usability requirements to be developed over time, with increasing detail and precision. This approach allows the CISU-R to be used for all types of projects, from the smallest or most informal to complex or formally specified products.
-Customers, users, and development teams can use the CISU-R as a communications tool to understand and specify requirements within an organization or as the basis for a contractual relationship between companies. The three levels of compliance provide the flexibility to match the level of detail and formality of the usability requirements to business needs.
-Requirements may also include indirect measures of usability, such as post-release satisfaction tests, or business performance requirements, such as completed transactions. These criteria are not intended for evaluation using a summative usability test, as required by the third level of conformance, but can be useful in establishing business or performance requirements.
-The CISU-R recognizes that usability requirements are just one type of product requirement. They complement functional, business, process, safety, and non-functional (quality) requirements. The information gathered in creating usability requirements can be used to help define other user requirements (for example, for features of the interface). (Annex E lists other types of user and usability requirements.)
The CISU-R focuses on the information needed to create usability requirements that are useful for design and development, rather than dictating a specific process or user centered design activities. (Annex E contains an example of a user centered design process that incorporates the CISU-R.)
-In situations where there are already processes and document formats in place, stakeholders can incorporate the CISU-R into that process by ensuring that the information specified here is included in any usability requirements.
-If there is no established format, requirements can use the information specified in Annex A as a guide. (Examples of all three levels are included in Annexes F, G, and H as a reference.)
The process of creating requirements with the CISU-R can be incorporated into many user centred design processes. For example, in ISO 13407, specifying requirements is one activity in an iterative process. The CISU-R provides details for how to complete this activity, as shown in Figure 1.
To ensure that the requirements are correct, other user centered design activities (such as competitive analysis, interviews, surveys, focus groups, field studies, task analysis, benchmark usability tests, or paper prototyping) can be used early in the development process to obtain feedback from users to iteratively refine requirements.
The CISU-R complements other user centered design standards.
-It uses the definition of usability in ISO 9241-11: the effectiveness, efficiency, and satisfaction with which the intended users can achieve their tasks in the intended context of product use.
-Usability requirements created using the CISU-R are consistent with the recommendations of ISO 13407.
-Usability tests conducted to measure whether usability requirements have been met can be reported using the Common Industry Format (CIF) (ISO/IEC 25062).
-The development of usability requirements may be supported by the use of standards such as ISO 9241 or ISO 9126.
Subscribe to:
Posts (Atom)
