Sep 12 - Au et al, Automated Usability Testing Framework

Automated Usability Testing Framework.
Fiora T. W. Au, Simon Baker, Ian Warren, Gillian Dobbie.
Department of Electrical and Computer Engineering, University of Auckland, Auckland, New Zealand, Private Bag 92019, Auckland, New Zealand. {ian-w,gill}@cs.auckland.ac.nz

Proc. 9th Australasian User Interface Conference (AUIC2008), Wollongong, Australia

Abstract
Handheld device applications with poor usability can reduce the productivity of users and incur costs for businesses, thus usability testing should play a vital role in application development. Conventional usability testing methodologies, such as formal user testing, can be expensive, time consuming and labour intensive; less resource-demanding alternatives can yield unreliable results. Automating aspects of usability testing would improve its efficiency and make it more practical to perform throughout development.
An automated usability testing tool should capture as input the properties of an application’s graphical user interface, the sequence of user actions as they use the application to achieve particular tasks, their behaviour and comments, as well as a description of these tasks.
The tool should evaluate both the static and dynamic properties of the interface, examine navigational burden and suggest modifications or templates that would improve usability. Results should be quick and easy to interpret, and be understandable by personnel other than
specialised testers.
Several existing tools that are typical of the tools available today meet some but not all of these
requirements. In this paper we describe the design of the HUIA testing framework, in which we have to meet as many of these requirements as possible.



1 Introduction

Handheld devices continue to feature in most companies’ mobile business solutions, as a means of improving their workers and thus the companies’ productivity. Handheld device applications (HDAs) with poor usability can undermine the value of such solutions, but despite this usability testing is often neglected due to its relatively high demand in time and resources.
These issues apply also to functional testing, for which many automated testing tools and frameworks, such as JUnit, were developed and are now widely used making the process
far more efficient.

1.1 Usability

Usability can be defined by IEEE (1990) as ‘the ease with which a user can operate, prepare inputs for, and interpret outputs of a system or component’, and comprises of five main attributes as outlined in Le Peuple and Scane (2003) and Nielsen, J. (1993): Learnability, Efficiency, Memorability, Errors, and Satisfaction.

The very nature of handheld devices make their programs particularly susceptible to poor usability; the smaller size, lower resolution screens, limited memory, lack of keyboard and mouse, and frequency of use among other hardware constraints and usage factors make program and user interface design particularly challenging as reported in Weiss (2005), Lee and Grice (2004) and Moe, Dwolatzky, and Olst(2004).

1.2 Usability testing

Heuristic evaluations are relatively cheap, quick and easy to carry out, but it has been claimed by Le Peuple and Scane (2003), Scholtz (2006), and Spolsky (2001) that such evaluations are likely to identify only approximately fifty percent of actual problems, with a significant number of false problems raised and actual problems missed.

When carried out correctly, user testing is likely to identify most of the major usability issues, as it involves real users attempting real tasks, and is very useful for collecting feedback on subjective aspects of usability (such as satisfaction and aesthetic appeal).
Studies by Scholtz (2006), and Spolsky (2001) have shown that six to eight test users (per type of intended users for that particular software) are usually sufficient to identify most of the major usability issues.
The type of analyses performed on the gathered data depends on the goal of the user test.
Typical areas that are examined include mistake and failure rates, types of mistakes, time taken,
amount of interactions (such as number of clicks or amount of scrolling), user behavior and user feedback as reported in Le Peuple and Scane (2003), Nielsen, J. (1993) and Spolsky (2001).
Scholtz (2006) reports that because user tests are expensive and time consuming to conduct, they are usually performed infrequently and towards the end of the development process.

Listed below are examples of the two types of quantitative measures/statistics:
Interface Properties:
• Number of fonts used, font sizes
• Average size of buttons
• Deepest level of menus
• Average loading time of graphics
Interaction Statistics (for performing specified tasks):
• Number of clicks
• Average drag distance
• Amount of scrolling
• Error rate
• Failure rate

2.2 Automating usability testing

There are many challenges and issues associated with traditional usability testing methodologies, many of which are to do with inefficiency, management complexities and high resource demands; all these contribute to the industry’s general reluctance to integrate usability testing as an essential activity on par with functional testing, despite its importance. Instead, it is
often considered as a ‘nice-to-have’ reserved for larger projects with generous budgets.

Conducting usability testing towards the end of the development process runs the risk of leaving insufficient time and resources to respond to the usability issues raised.

User testing is arguably the most effective testing methodology, but it is very inefficient to carry out. The process is labour intensive and costly, having to design the test, recruit suitable test users, set up test sessions, run the test, collect the data and then analyze the results.
Other testing methods such as heuristic evaluations and cognitive walkthroughs involving just one or several usability experts are less costly, but not to the point where it can be performed on a frequent, iterative basis.

Currently there is a distinct separation of responsibilities between developers and usability testing specialists.
These specialists may or may not have a good understanding of the application domain, which could affect the validity of their opinions and findings.
Furthermore, developers who do not play an active role in the usability evaluations end up learning of their product’s usability ‘second-hand’, having to base design decisions on their own (sometimes incorrect) interpretations of usability evaluation results.

A logical solution would be to automate as many aspects of usability testing as possible, in much the same way that aspects of functional testing has been automated as described in Patton (2006). There has been some excellent work that investigates the state of the art in automated usability evaluation, such as Ivory and Hearst (2001).
The challenge is thus to create an automated usability testing tool that is aimed for use by developers.

3 Objectives

The main objectives of an automated usability testing tool are similar in many ways to any automated software tool as outlined in Patton (2006), and may be summarized as follows:

1. Effective usability testing:
The tool should detect as many if not more real usability issues as conventional usability testing methodologies.

2. Increased speed:
The tool should perform analyses (checks, calculations, comparisons etc.) and other processes quicker than if they were performed manually.

3. Increased efficiency:
By taking over parts of the testing process, developers/testers are free to perform other tasks, and need to dedicate less time to testing.

4. Improved accuracy and precision:
Results produced will always be functionally correct; errors are limited to those produced by inappropriate input or direction by the developer/tester.

5. Reduced resource demand:
Automation should reduce time, human resources, equipment and cost requirements.

6. Increased flexibility:
The tool should perform testing on a range of usability aspects, and allow customisation of test settings. It should also facilitate usability testing for all stages of development (as well as iterative/regression testing).

7. Consistency:
The tool should provide a means of maintaining set testing standards throughout development, so that testing can be performed by different people but the same standard of usability is enforced.

8. Promote usability:
By simplifying the usability testing process, the tool should encourage all personnel related to product development (including developers, managers, testers, sales personnel and clients) to be involved in the process.
Also, the tool itself should promote good usability practices by encouraging the use of proven design paradigms.

7 Conclusions

Usability is very important for handheld device applications, because those with poor usability can lower the productivity of its users and incur costs for businesses, thus undermining the value of a mobile business solution. Automating aspects of usability testing can improve testing efficiency and better facilitate its integration with the development process.
Ideally, an automated usability testing tool should capture a range of inputs, perform analyses on different aspects of usability, present results clearly, be simple and flexible to use, and able to be used throughout development.

None of the existing tools discussed meet all the requirements described. Notably however, none of the tools are able to suggest good usability solutions, they can only perform evaluations. Of the tools discussed, the HUIA testing framework addresses most requirements, though still requires future development, which may include GUI consistency checking such as that outlined in Mahajan et al. (1997), and the inclusion of standard interface description languages
such as those described in Souchon et al. (2003).

References that I may want to read further in future:
Baker, S; Au, F; Warren, I; Dobbie, G. (2007): HUIA: A Tool for Automated Usability Testing, UoASE-2007-1, University of Auckland.
Ivory, M. and Hearst, M. (2001): The State of the Art in Automated Usability Evaluation of User Interfaces, ACM Computing Surveys, 33 (4), pp. 173-197.
Lee, K; Grice, R. (2004): Developing a New Usability Testing Method for Mobile Devices. Professional Communication Conference, pp. 115 – 127.
Morae: Usability Testing for Software and Websites. http://www.techsmith.com/morae.asp, visited 8th March, 2006.
Scholtz, J. (2006); Usability Evaluation. http://www.itl.nist.gov/iad/IApapers/2004/Usability%20Evaluation_rev1.pdf, visited 9th March.
Weiss, S. (2005): Handheld Usability: Design, Prototyping, & Usability Testing for Mobile Phones. Proceedings of the 7th Conference on Human-Computer Interaction with Mobile Devices and Services.