Sep 5 - Howarth, Supporting Novice Usability Practitioners with Usability Engineering Tools (part 2)

Supporting Novice Usability Practitioners with Usability Engineering Tools.
Jonathan Randall Howarth.
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Computer Science and Applications.
April 13, 2007
Blacksburg, Virginia


2 RELATED WORK

2.1 Difficulties Experienced by Usability Practitioners

2.1.1 Evaluator Effect

The evaluator effect is the tendency of usability practitioners with differing knowledge and experience to find different types and numbers of UPs during usability evaluation.
Work by Rowe et al. [1994] demonstrated that different usability evaluation teams studying the same interface will find different issues.
Jacobsen et al. [1998] documented and named the evaluator effect in a study in which four usability experts were given the same video tapes of four participants performing tasks in a multimedia authoring system. Each expert identified about half of the UPs, but about half of those were unique to the individual expert.
A related study by Hertzum and Jacobsen [2003] provided more evidence of the evaluator effect by reviewing 11 studies that used one of the following usability evaluation methods: cognitive walkthroughs, heuristic evaluation, or thinkingaloud study. The authors proposed that the evaluator effect occurs because usability evaluation involves interpretation and that usability evaluation methods do not provide the guidance that usability practitioners need to perform reliable evaluations.
Vermeeren et al. [2003] conducted a study that found evidence of the evaluator effect in different domains and also proposed reasons for the evaluator effect related to interpretation, such as guessing user intentions.

2.1.2 Content of Usability Problem Descriptions

UP descriptions document interaction design flaws that cause UPs for users. They are used, in the context of a usability evaluation report, to help system analysts and designers identify specific features of an interaction design to change, add, or remove in subsequent iterations.
There have been a limited number of UP description formats documented in the literature.
Jeffries developed recommendations for what to include in a UP description while performing a review of UP descriptions to determine their shortcomings [Jeffries, 1994]. While the recommendations represent an improvement over ad hoc reporting, they do not provide a definite format and focus on solutions without addressing causes.
A study by John and Packer [1995] on the learnability and applicability of the cognitive walkthrough method contained a UP description form with a unique reference number and fields for describing the UP, estimating its severity, and assessing the source of its discovery. This form, much like Jeffries’ recommendations, did not specifically address the causes within the interaction design of problems.
In a study comparing empirical testing with usability inspections, Mack and Montaniz [1994] describe a UP description structure that includes descriptions of goal-directed behavior, interface interactions, possible causes, and severity. This report structure does address the causes of UPs, but it relies heavily on interpretation...

To enable comparative studies of usability evaluation methods, a more standard way to describe UPs was needed.
Lavery et al. [1997] developed a structured UP description format that addressed the shortcomings of previous UP description formats. The method captures the problem context, cause, outcomes, and solutions.
Cockton and Lavery [1999] leverage this structured UP description format in the Structured Usability Problem Extraction (SUPEX) framework, which separates problem context, cause, and recommendations. The SUPEX framework provides a rigorous approach to extracting problems that distinguishes among multiple levels of abstraction and handles relationships among user actions to reduce under- and over-reporting of UPs.
Later work by Cockton et al. [2003] supports the use of structured UP descriptions by demonstrating that they help to improve analysts’ performance with the heuristic evaluation method.

Capra [2006] developed guidelines for the content of UP descriptions through a series of three studies with usability practitioners. These guidelines represent an important step towards structuring the content of UP descriptions. They, however, are subject to a major limitation of guidelines in that they may be difficult to apply consistently [Borges et al., 1996, Smith, 1986].

In sum, the literature does not provide a clear answer as to what to include in a UP description. As a result, UP descriptions are often ad hoc [Andre et al., 2001].

Without a specific format, usability practitioners may not be aware of what to look for during usability data collection or what to clarify with participants during empirical testing. In addition, even if necessary usability data are observed and recorded during usability data collection, the lack of a consistent report format may make it difficult for problem analysts to understand and relate the data.

2.1.3 Content of Usability Evaluation Reports

As illustrated in Figure 3, the output of the usability evaluation sub-process is a usability evaluation report. The UP descriptions discussed in Section 2.1.2 differ from usability evaluation reports; the former is used to document an individual UP while the latter is used to convey results of an entire usability evaluation.

As discussed in Section 2.1.2, the challenge associated with UP descriptions is getting the necessary data to completely specify the UP. The challenge associated with usability evaluation reports is conveying the necessary information associated with the UP descriptions to a given audience.

In 1997, the Industry Usability Reporting Project initiated by the National Institute of Standards and Technology developed the Common Industry Format (CIF), which is currently the most well known format for usability evaluation reports. The CIF became an American National Standard for Information Technology Standard in 2001 [ANSI, 2001].
By standardizing the reporting of usability tests, the CIF hoped to encourage the consideration of usability in purchasing software products; customer organizations that were interested could evaluate different products based on their CIF reports.
The CIF includes sections for describing the product, the method used to evaluate the product, and the results of the evaluation.
The CIF is intended for summative usability evaluations, but usability practitioners most frequently perform formative usability evaluations.

Theofanos [2005] and Theofanus and Quesenbery [2005] describe efforts to develop a new
CIF that would provide practitioners with guidance for performing and reporting formative studies.

The usability evaluation report consolidates usability information and provides the context for understanding UP descriptions. Without this context, UP descriptions may be misunderstood or overlooked.

2.2 Existing Usability Engineering Tools

There are a number of tools for use in UE efforts that represent a variety of focuses and development activities.
I present a survey of these tools using a categorization scheme to structure the discussion of the state of these tools. I include tools that I found through a combination of a literature and a web search.
The list of tools is not exhaustive; instead it provides examples of each basic category of tool.

2.2.1 Tools not Included in the Survey

The following basic types were excluded from the survey:
1 custom tools,
2 tools that facilitate the construction of interfaces, and
3 tools with a business or social research focus.

Custom tools are created by an organization specifically to fit the needs of a particular usability process. As a result, these tools are generally not documented and not made available for use outside of the organization.

Two basic types of tools are used to facilitate the construction of interfaces. One type is tools that help programmers write the code for graphical user interfaces. Another type of tools is used to
help with creating interfaces quickly for prototyping. Both types of tools are excluded because of the focus on the usability evaluation sub-process instead of the design sub-process.

Some tools are specifically created for UE processes, but do not address usability evaluation activities in any detail. For example, Agility helps individuals involved in the UE process collaborate with one another and plan activities and deliverables [Classic System Solutions Inc., 2005]. The tool, however, has a business focus and is not appropriate for this survey. ... One example is Observer, which is primarily intended for collecting observational data for social research [Noldus, 2005].

2.2.2 Categorization Scheme

My categorization scheme is based on a taxonomy of usability evaluation tools
developed by Ivory and Hearst [Ivory & Hearst, 2001] and the stages of the
usability evaluation sub-process shown in Figure 3.

The three levels to this scheme are as follows:
• 1 Evaluation method class – How usability evaluations are conducted using the tool
o 1.1 Analytical – Evaluations involve inspections by experts such as heuristic evaluations or cognitive walkthroughs or static analysis of an interaction design.
o 1.2 Empirical – Evaluations involve observing a participant using a tool.

• 2 Application class – What type of application can be evaluated using the tool
o 2.1 Desktop – The tool can only be used to evaluate desktop applications.
o 2.2 Web – The tool can only be used to evaluate websites.
o 2.3 Both – The tool can evaluate both desktop applications and websites.

• 3 Supported stages of the usability evaluation sub-process – Which stages of the usability evaluation sub-process are supported by the tool (Section 1.3.3)
o 3.1 Usability data collection
o 3.2 UP analysis
o 3.3 Usability evaluation reporting

2.2.3 Tools Included in the Survey

The tools included in the survey along with their evaluation method class, application class, and supported stages of the usability evaluation sub-process are shown in Table 3.

Table 3: Tools included in the survey
Tool Name > Evaluation Method Class > Application Class > Supported Stages

My Comments: This PhD Dissertation has certain degree of relevance to my PhD research. This is because Howarth's research involved developing a usabilty engineering tool; mine would involve developing a usability evaluation tool.
Later in my research, I would definitely want to read Howarth's Dissertation in detail to understand what he had done. This could be a valuable benchmark to me.

References that I may want to read further in future:
Andre, T. S., Hartson, H. R., Belz, S. M., & McCreary, F. A. (2001). The User Action Framework: A reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54(1), 107-136.
Andre, T. S., Hartson, H. R., & Williges, R. C. (2002). Determining the effectiveness of the Usability Problem Inspector: A theory-based model and tool for finding usability problems. Human Factors, 45(3), 455-482.
ANSI. (2001). Common Industry Format for usability test reports. American National Standard for Information Technology.
Bit Debris Solutions. Usability Activity Log. Retrieved December 21, 2005, from http://www.bitdebris.com/products/ulablog/
Capra, M. (2006). Usability Problem Description and the Evaluator Effect in Usability Testing. Unpublished dissertation. Blacksburg, VA: Virginia Tech.
Capra, M. G. (2001). An Exploration of End-User Critical Incident Classification. Unpublished master's thesis. Blacksburg, VA: Virginia Tech.
Classic System Solutions Inc. Agility. Retrieved December 21, 2005, from
http://www.classicsys.com/classic_site/html/team_design.html
Cockton, G., & Lavery, D. (1999). A framework for usability problem extraction. Paper presented at the International Conference on Human-Computer Interaction.
Cockton, G., Woolrych, A., Hall, L., & Hindmarch, M. (2003). Changing analysts' tunes: The surprising impact of a new instrument for usability inspection method assessment. In People & Computers XVII (pp. 145-161).
Dumas, J. S., Molich, R., & Jeffries, R. (2004). Describing usability problems: Are we sending the right message? interactions, 11(4), 24-29.
Etgen, M., & Cantor, J. (1999). What does getting WET (web event-logging tool) mean for web usability. Paper presented at the Conference on Human Factors and the Web.
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 373-410.
Hix, D., & Hartson, H. R. (1993a). Developing User Interfaces: Ensuring Usability through Product and Process. New York, USA: John Wiley & Sons, Inc.
Hix, D., & Hartson, H. R. (1993b). Formative evaluation: Ensuring usability in user interfaces. In L. Bass & P. Dewan (Eds.), User Interface Software (pp. 1-30). New York: John Wiley & Sons.
Hoegh, R. T., Nielsen, C., Overgaard, M., Pedersen, M., & Stage, J. (2006). The impact of usability reports and user test observations on developers' understanding of usability data: An exploratory study. International Journal of Human-Computer Interaction, 21(2), 173-196.
Howarth, J. (2006). Identifying immediate intention during usability evaluation. Paper presented at the ACM Southeast Conference.
ISO. (1998). 9241 - Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability. Geneva, Switzerland: International Standards Organization.
ISO. (1999). 13407 - Human-centered design processes for interactive systems. Geneva, Switzerland: International Standards Organization.
Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470-516.
Jeffries, R. (1994). Usability problem reports: Helping evaluators communicate effectively with developers. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 273-294). New York: John Wiley and Sons.
Keenan, S. L. (1996). Product usability and process improvement based on usability problem classification. Unpublished dissertation. Blacksburg, VA: Virginia Tech.
Keenan, S. L., Hartson, H. R., Kafura, D. G., & Schulman, R. S. (1999). The Usability Problem Taxonomy: A framework for classification and analysis. Empirical Software Engineering, 4(1), 71-104.
Lavery, D., & Cockton, G. (1997). Representing predicted and actual usability problems. Paper presented at the International Workshop on Representations in Interactive Software Development.
Lavery, D., Cockton, G., & Atkinson, M. P. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour and Information Technology, 16(4-5), 246-266.
Lund, A. M. (1997). Another approach to justifying the cost of usability. interactions, 4(3), 48-56.
Mack, R., & Montaniz, F. (1994). Observing, predicting, and analyzing usability problems. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 295-339). New York: John Wiley & Sons.
Mayhew, D. (1999). The Usability Engineering Lifecycle. San Francisco, CA: Morgan Kaufmann.
Nayak, N. P., Mrazek, D., & Smith, D. R. (1995). Analyzing and communicating usability data: Now that you have the data what do you do? ACM SIGCHI Bulletin, 27(1), 22-30.
Nielsen, C., Overgaard, M., Pedersen, M., & Stage, J. (2005). Feedback from usability evaluation to user interface design: Are usability reports any good? Paper presented at the INTERACT.
Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Paper presented at the Conference on Human Factors in Computing Systems.
Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: Wiley.
Scholtz, J., & Laskowski, S. (1998). Developing usability tools and techniques for designing and testing web sites. Paper presented at the Conference on Human Factors and the Web.
TechSmith. Morae: A complete usability testing solution for web sites and software. Retrieved November 30, 2005, from http://www.techsmith.com/products/morae/default.asp
Theaker, C. J., Phillips, R., Frost, T. M. E., & Love, W. R. (1989). HIMS: A tool for HCI evaluations. In A. Sutcliffe & L. Macaulay (Eds.), People and Computers V (HCI '89 Conference) (pp. 427-439).
Theofanos, M. (2005). Towards the design of effective formative test reports. Journal of Usability Studies, 1(1), 27-45.
Theofanos, M., Quesenbery, W., Snyder, C., Dayton, D., & Lewis, J. (2005). Reporting on formative testing: A UPA 2005 workshop report. Bloomingdale, IL.
Usable Net. LIFT. Retrieved December 21, 2005, from http://www.usablenet.com/
Users First. Users First - Portable professional tools to observe users. Retrieved November 30, 2005, from http://www.usersfirst.com/index.jsp?page=products
Uzilla. Uzilla: Tools for a usable web. Retrieved November 30, 2005, from http://uzilla.net/
Weiler, P. (1993). Software for the usability lab: a sampling of current tools. Paper presented at the Conference on Human Factors in Computing Systems.
Wixon, D. (2003). Evaluating usability methods: why the current literature fails the practitioner. interactions, 10(4), 28-34.
Working Web. Usability tool: What does it do? Retrieved November 30, 2005, from http://workingweb.com.au/services/UsingUsabilityTool.php