Evaluation Research: an overviewComments: This is preview of a paper.Library TrendsJune 22, 2006 Powell, Ronald R. ABSTRACT Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs. After the reasons for conducting evaluation research are discussed, the general principles and types are reviewed. Several evaluation methods are then presented, including input measurement, output/ performance measurement, impact/outcomes assessment, service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. Other aspects of evaluation research considered are the steps of planning and conducting an evaluation study and the measurement process, including the gathering of statistics and the use of data collection techniques. The process of data analysis and the evaluation report are also given attention. It is concluded that evaluation research should be a rigorous, systematic process that involves collecting data about organizations, processes, programs, services, and/or resources. Evaluation research should enhance knowledge and decision making and lead to practical applications. WHAT IS EVALUATION RESEARCH? Evaluation research is not easily defined. There is not even unanimity regarding its name; it is referred to as evaluation research and evaluative research. Some individuals consider evaluation research to be a specific research method; others focus on special techniques unique, more often than not, to program evaluation; and yet others view it as a research activity that employs standard research methods for evaluative purposes. Consistent with the last perspective, Childers concludes, "The differences between evaluative research and other research center on the orientation of the research and not on the methods employed" (1989, p. 251). When evaluation research is treated as a research method, it is likely to be seen as a type of applied or action research, not as basic or theoretical research. Weiss, in her standard textbook, defines evaluation as "the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy" (1998, p. 4; emphasis in original). While certainly not incorrect, this definition, at least within a library and information (LIS) context, is too narrow or limited. Wallace and Van Fleet, for example, point out that "evaluation has to do with understanding library systems" (2001, p. 1). As will be noted later in this article, evaluative methods are used for everything from evaluating library collections to reference transactions. WHY EVALUATE? But before examining the specific techniques and methods used in LIS evaluation research, let us first briefly consider the question of why evaluation is important and then identify the desirable characteristics of evaluation, the steps involved in planning an evaluation study, and the general approaches to evaluation. With regard to the initial question, Wallace and Van Fleet (2001, pp. xx-xxi) and others have noted that there are a growing number of reasons why it is important for librarians and other information professionals to evaluate their organizations' operations, resources, and services. Among those reasons are the need for organizations to 1. account for how they use their limited resources 2. explain what they do 3. enhance their visibility 4. describe their impact 5. increase efficiency 6. avoid errors 7. support planning activities 8. express concern for their public 9. support decision making 10. strengthen their political position. In addition to some of the reasons listed above, Weiss (1998, pp. 20-28) identifies several other purposes for evaluating programs and policies. They include the following: 1. Determining how clients are faring 2. Providing legitimacy for decisions 3. Fulfilling grant requirements 4. Making midcourse corrections in programs 5. Making decisions to continue or culminate programs 6. Testing new ideas 7. Choosing the best alternatives 8. Recording program history 9. Providing feedback to staff 10. Highlighting goals "Over the past decade, both academics and practitioners in the field of library and information science (LIS) have increasingly recognized the significance of assessing library services" (Shi & Levy, 2005, p. 266). In August 2004 the National Commission on Libraries and Information Science announced three strategic goals to guide its work in the immediate future. Among those three goals was the appraising and assessing of library and information services. CHARACTERISTICS AND PRINCIPLES OF EVALUATION Childers (1989, p. 250), in an article emphasizing the evaluation of programs, notes that evaluation research Wallace and Van Fleet (2001) comment that evaluation should be carefully planned, not occur by accident; have a purpose that is usually goal oriented; focus on determining the quality of a product or service; go beyond measurement; not be any larger than necessary; and reflect the situation in which it will occur. Similarly, evaluation should contribute to an organization's planning efforts; be built into existing programs; provide useful, systematically collected data; employ an outside evaluator/consultant when possible; involve the staff; not be any fancier than necessary; and target multiple audiences and purposes (Some Practical Lessons on Evaluation, 2000). In an earlier work on the evaluation of special libraries, Griffiths and King (1991, p. 3) identify some principles for good evaluation that still bear repeating: 1. Evaluation must have a purpose; it must not be an end in itself 2. Without the potential for some action, there is no need to evaluate 3. Evaluation must be more than descriptive; it must take into account relationships among operational performance, users, and organizations 4. Evaluation should be a communication tool involving staff and users 5. Evaluation should not be sporadic but be ongoing and provide a means for continual monitoring, diagnosis, and change 6. Ongoing evaluation should provide a means for continual monitoring, diagnosis and change 7. Ongoing evaluation should be dynamic in nature, reflecting new knowledge and changes in the environment As has been implied, but not explicitly stated above, evaluation often attempts to assess the effectiveness of a program or service. On a more specific level, evaluation can be used to support accreditation reviews, needs assessments, new projects, personnel reviews, conflict resolution, and professional compliance reports. Source: http://www.accessmylibrary.com/article-1G1-151440807/evaluation-research-overview.html Action: to download the entire paper at Univ. |
Showing posts with label Powell. Show all posts
Showing posts with label Powell. Show all posts
Friday, November 26, 2010
20101127 - Evaluation Research (Powell)
Friday, September 4, 2009
Sep 5 - Whitehead, Evaluating Web Page and Web Site Usability
Evaluating Web Page and Web Site Usability.
Christopher C. Whitehead. Columbus State University, 4225 University Avenue, Columbus, GA 31907. 1-706-565-3527 whitehead_christopher@colstate.edu
ACM SE’06, March, 10-12, 2006, Melbourne, Florida, USA
ABSTRACT
As the number of Web sites continue to increase, so too does the importance of Web page/Web site usability. This paper describes what constitutes Web page/Web site usability and how it can be measured.
Why is usability important?
As Nielsen states, "Usability rules the Web. Simply stated, if the customer can't find a product, then he or she will not buy it" [5].
In this sense, usability is an extremely important aspect of individual Web page and overall Web site design, particularly for the business-oriented Web site.
In discussing e-commerce Web sites, Shacklett asserts that, "Twenty-eight percent of Web site transactions result in consumer failure and frustration....Six percent of users who leave a Web site in frustration say they won't return to the site or patronize the company” [7].
If usability is left unconsidered, then a business is likely to lose customers and miss out on profit opportunities--negatively impacting the cornerstones of any successful business.
WHAT IS WEB PAGE/SITE USABILITY?
Based on the International Standards Organization (ISO) definition of usability, Powell defines Web site usability as "the extent to which a site can be used by a specified group of users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use" [6].
Note that this definition applies equally well to Web page usability. It also implies that usability is user and task dependent, as well as being related to how well the user is able to accomplish what they set out to do, how efficiently the user can do this, and how satisfied the user is during and after the process.
..Jakob Nielsen's usability guidelines for determining the usability of a Web site [6]:
• Learnability--How easy it is to learn to use?
• Rememberability--How easy it is to remember how to use?
• Efficiency of use--How much work does it require the user to do?
• Reliability in use--Does it work correctly and does it help users perform tasks correctly?
• User Satisfaction--Is the user generally satisfied as a result of using the site?
..McLaughin and Skinner break usability down into six related but distinct components [4]:
• Checkability: The system has or allows checks that ensure the correct information is going in and going out of it.
• Confidence: Users have confidence both in their capability to use the system and in the system itself.
• Control: Users have control over the operation of the system, particularly of the information fed into and out of the system.
• Ease of Use: The system is easy to use.
• Speed: The system can be used quickly.
• Understanding: The system and its outputs are understandable.
In general then, usability is potentially complex and wide ranging,
but clearly "user-centered." In evaluating usability, it may be possible to measure each of these components separately or in combination using some form of metric or measure.
USABILITY TESTING
Battleson et al. claim that usability testing is the most effective way to asses a site's usability [1]. They go on to specify three categories of usability testing: inquiry, inspection, and formal usability testing.
Inquiry usability testing requests information from the user and includes the use of focus groups, interviews, questionnaires, and surveys.
Inspection usability testing includes heuristic evaluation and cognitive walkthroughs. According to Dickstein and Mills, “Heuristic evaluation is a systematic inspection of a user interface to examine if the design is in compliance with recognized usability” [2]. This typically involves checking the site “against a checklist of heuristics, or design principles” [1]. Cognitive walkthroughs involve enlisting the use of individuals “to accomplish typical user tasks with a given interface” [1].
Under formal usability testing, a formal set of tasks and goals are defined and users are observed while performing these tasks. According to Battleson et al., formal usability testing involves the following: “(1) the goal is to improve the usability of the interface; (2) testers represent real users; (3) testers perform real tasks; (4) user behavior and commentary are observed and recorded; and (5) data are analyzed to recognize problems and suggest solutions” [1].
Ivory et al. also report that simulation has been used to effectively evaluate web sites by automating the implementation of usability testing [3]. Using this methodology, user behavior is mimicked in a controlled environment or through computer simulation without actual users.
Determining the Number of Users to Involve in the Testing
In performing usability testing, Battleson et al. claim that in a homogenous user testing group, as little as five users are needed to ascertain a large percentage of the most critical errors [1].
In comparison, Dickstein and Mills found that eight to twelve users was sufficient enough to determine problems with a Web site’s usability [2]. However, they also claim that by the fifth user, they were able to determine whether the tested feature was either a problem or a success.
USABILITY METRICS
Usability metrics are measures of a particular aspect of a Web page or Web site that has an impact on usability.
In today's environment, there are numerous proposed metrics.
In performing quantitative usability testing, Ivory, et al. suggest the following Web page metrics as the most important in evaluating usability [3]:
• Word Count--Total words on a page
• Body Text %--Percentage of words that are body vs. display text (i.e., headers)
• Emphasized Body Text %--Portion of body text that is emphasized (e.g., bold, capitalized or near !'s)
• Text Positioning Count--Changes in text position from flush left
• Text Cluster Count --Text areas highlighted with color, bordered regions, rules or lists
• Link Count--Total links on a page
• Page Size--Total bytes for the page as well as elements graphics and stylesheets
• Graphic %--Percentage of page bytes that are for graphics
• Graphics Count--Total graphics on a page (not including graphics specified in scripts, applets and objects)
• Color Count--Total colors employed
• Font Count--Total fonts employed (i.e., face + size + bold + italic)
Ivory et al. found six metrics to be of most importance: text cluster count, link count, page size, graphics count, color count, and reading complexity [3].
CONCLUSION
In summarizing the importance of Web page/Web site usability, Nielsen states, "Usability has grown dramatically in importance for web-based companies because of an inversion in the relationship between user experience and the ability to separate customers from their money" [5]. This inversion is due to the way in which the Web works in that users are able to experience a company before they commit to either making a purchase or going elsewhere. This makes the user all-important for those businesses that expect to profit from the Web. And, as a result, as Dickstein and Mills claim, “Usability testing methods, by their very nature, keep the user in the forefront” [2]. It is because of this "user-centered" approach that evaluating the usability of Web pages and Web sites is so vitally important to today's businesses.
References that I may want to read further in future:
[3] Ivory, M. Y., Sinha, R. R., & Hearst, M. A. Empirically Validated Web Page Design Metrics. ACM SIGCHI'01, 2001.
[5] Nielsen, J. Designing Web Usability: The Practice of Simplicity. Indianapolis, IN: New Riders Publishing, 2000.
[6] Powell, T. A. Web Design: The Complete Reference. Berkeley, CA: Osborne/McGraw-Hill, 2000.
[7] Shacklett, M. Gauging Web site performance. Credit Union Magazine, 67, 6, 2001, 60-62.
Christopher C. Whitehead. Columbus State University, 4225 University Avenue, Columbus, GA 31907. 1-706-565-3527 whitehead_christopher@colstate.edu
ACM SE’06, March, 10-12, 2006, Melbourne, Florida, USA
ABSTRACT
As the number of Web sites continue to increase, so too does the importance of Web page/Web site usability. This paper describes what constitutes Web page/Web site usability and how it can be measured.
Why is usability important?
As Nielsen states, "Usability rules the Web. Simply stated, if the customer can't find a product, then he or she will not buy it" [5].
In this sense, usability is an extremely important aspect of individual Web page and overall Web site design, particularly for the business-oriented Web site.
In discussing e-commerce Web sites, Shacklett asserts that, "Twenty-eight percent of Web site transactions result in consumer failure and frustration....Six percent of users who leave a Web site in frustration say they won't return to the site or patronize the company” [7].
If usability is left unconsidered, then a business is likely to lose customers and miss out on profit opportunities--negatively impacting the cornerstones of any successful business.
WHAT IS WEB PAGE/SITE USABILITY?
Based on the International Standards Organization (ISO) definition of usability, Powell defines Web site usability as "the extent to which a site can be used by a specified group of users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use" [6].
Note that this definition applies equally well to Web page usability. It also implies that usability is user and task dependent, as well as being related to how well the user is able to accomplish what they set out to do, how efficiently the user can do this, and how satisfied the user is during and after the process.
..Jakob Nielsen's usability guidelines for determining the usability of a Web site [6]:
• Learnability--How easy it is to learn to use?
• Rememberability--How easy it is to remember how to use?
• Efficiency of use--How much work does it require the user to do?
• Reliability in use--Does it work correctly and does it help users perform tasks correctly?
• User Satisfaction--Is the user generally satisfied as a result of using the site?
..McLaughin and Skinner break usability down into six related but distinct components [4]:
• Checkability: The system has or allows checks that ensure the correct information is going in and going out of it.
• Confidence: Users have confidence both in their capability to use the system and in the system itself.
• Control: Users have control over the operation of the system, particularly of the information fed into and out of the system.
• Ease of Use: The system is easy to use.
• Speed: The system can be used quickly.
• Understanding: The system and its outputs are understandable.
In general then, usability is potentially complex and wide ranging,
but clearly "user-centered." In evaluating usability, it may be possible to measure each of these components separately or in combination using some form of metric or measure.
USABILITY TESTING
Battleson et al. claim that usability testing is the most effective way to asses a site's usability [1]. They go on to specify three categories of usability testing: inquiry, inspection, and formal usability testing.
Inquiry usability testing requests information from the user and includes the use of focus groups, interviews, questionnaires, and surveys.
Inspection usability testing includes heuristic evaluation and cognitive walkthroughs. According to Dickstein and Mills, “Heuristic evaluation is a systematic inspection of a user interface to examine if the design is in compliance with recognized usability” [2]. This typically involves checking the site “against a checklist of heuristics, or design principles” [1]. Cognitive walkthroughs involve enlisting the use of individuals “to accomplish typical user tasks with a given interface” [1].
Under formal usability testing, a formal set of tasks and goals are defined and users are observed while performing these tasks. According to Battleson et al., formal usability testing involves the following: “(1) the goal is to improve the usability of the interface; (2) testers represent real users; (3) testers perform real tasks; (4) user behavior and commentary are observed and recorded; and (5) data are analyzed to recognize problems and suggest solutions” [1].
Ivory et al. also report that simulation has been used to effectively evaluate web sites by automating the implementation of usability testing [3]. Using this methodology, user behavior is mimicked in a controlled environment or through computer simulation without actual users.
Determining the Number of Users to Involve in the Testing
In performing usability testing, Battleson et al. claim that in a homogenous user testing group, as little as five users are needed to ascertain a large percentage of the most critical errors [1].
In comparison, Dickstein and Mills found that eight to twelve users was sufficient enough to determine problems with a Web site’s usability [2]. However, they also claim that by the fifth user, they were able to determine whether the tested feature was either a problem or a success.
USABILITY METRICS
Usability metrics are measures of a particular aspect of a Web page or Web site that has an impact on usability.
In today's environment, there are numerous proposed metrics.
In performing quantitative usability testing, Ivory, et al. suggest the following Web page metrics as the most important in evaluating usability [3]:
• Word Count--Total words on a page
• Body Text %--Percentage of words that are body vs. display text (i.e., headers)
• Emphasized Body Text %--Portion of body text that is emphasized (e.g., bold, capitalized or near !'s)
• Text Positioning Count--Changes in text position from flush left
• Text Cluster Count --Text areas highlighted with color, bordered regions, rules or lists
• Link Count--Total links on a page
• Page Size--Total bytes for the page as well as elements graphics and stylesheets
• Graphic %--Percentage of page bytes that are for graphics
• Graphics Count--Total graphics on a page (not including graphics specified in scripts, applets and objects)
• Color Count--Total colors employed
• Font Count--Total fonts employed (i.e., face + size + bold + italic)
Ivory et al. found six metrics to be of most importance: text cluster count, link count, page size, graphics count, color count, and reading complexity [3].
CONCLUSION
In summarizing the importance of Web page/Web site usability, Nielsen states, "Usability has grown dramatically in importance for web-based companies because of an inversion in the relationship between user experience and the ability to separate customers from their money" [5]. This inversion is due to the way in which the Web works in that users are able to experience a company before they commit to either making a purchase or going elsewhere. This makes the user all-important for those businesses that expect to profit from the Web. And, as a result, as Dickstein and Mills claim, “Usability testing methods, by their very nature, keep the user in the forefront” [2]. It is because of this "user-centered" approach that evaluating the usability of Web pages and Web sites is so vitally important to today's businesses.
References that I may want to read further in future:
[3] Ivory, M. Y., Sinha, R. R., & Hearst, M. A. Empirically Validated Web Page Design Metrics. ACM SIGCHI'01, 2001.
[5] Nielsen, J. Designing Web Usability: The Practice of Simplicity. Indianapolis, IN: New Riders Publishing, 2000.
[6] Powell, T. A. Web Design: The Complete Reference. Berkeley, CA: Osborne/McGraw-Hill, 2000.
[7] Shacklett, M. Gauging Web site performance. Credit Union Magazine, 67, 6, 2001, 60-62.
Subscribe to:
Posts (Atom)
