20101127 - Evaluative Research (Winsett, NATCO)

Evaluative Research Rebecca P. Winsett, PhD, Associate Professor, University of ennessee-Memphis, Memphis, TN; Professor, University of Southern Indiana, Evansville, IN, NATCO Research Committee Co-Chair Introduction Evaluation of the impact of a new program is an important aspect of many organizations. Finding that implementing a new way to evaluate donors or whether or not adding a return to work program is successful may be examples of program evaluation. There is a design methodology called evaluative research that can help guide you as you design studies for future publication. As stated in Weiss (Weiss, 1998), evaluation is the "systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy" (pg 4). Elements of Evaluation Two questions that you may ask about a program may include if the program has the intended effect (What are the outcomes?) or is the program being delivered as it was originally constructed (What services are being delivered?). In these two questions, you may have several ways to evaluate the impact (the outcomes) and its effectiveness (process). Outcomes refer to the results of the program. Were the services successful? Did those in a smoking cessation program stop smoking? Did those who attended an education program increase knowledge? Was attitude about organ donor awareness changed as a result of your program? Methods to measure outcomes are not too distant from those you would find in traditional research methodology. Measuring group differences or pre/post-testing are examples of ways to measure outcomes. Process refers to the internal working of the program itself. Understanding the operations of the program and how it can be improved. Designing the evaluation of the process of the program requires thoughtful planning of what you want to examine. Examples may include monitoring staff time during the program to show cost savings or collecting surveys from different levels of staff to tease out problems with work efficiency. Qualitative studies can also be included. Each type of evaluation, whether outcome or process must be systematically developed to ensure the effects of the program are truly effects rather than opinion. Source: http://www.natco1.org/research/files/EvaluativeResearch_000.pdf Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies (2nd ed.). Upper Saddle River, NJ: Prentice Hall.