20101127 - Evaluation research/method (Patton)

Evaluation Research : an interview with Dr Patton Interviewed by Lisa Waldick Comments: Dr Michael Quinn Patton is well-known for Evaluation Research methods. So, this creates my interest to read this interview. Many development programs are evaluated to determine how effective and useful they are. But how effective and useful are the evaluations themselves? Internationally renowned evaluator, Michael Quinn Patton, recently came to IDRC (International Development Research Centre) to discuss his approach for making sure evaluations are useful for decision-makers. Dr Patton is the head of an organizational development consulting business: Utilization-Focused Information and Training. Known for five influential books on evaluation, including Qualitative/Evaluation and Research Methods, he was the 1984 recipient of the Alva and Gunnar Myrdal Award from the Evaluation Research Society for "outstanding contributions to evaluation use and practice" Why is it important to evaluate development programs? Don't people on the ground just intuitively understand what is going on in programs? Our very processes of taking in information distort reality — all the evidence of social science indicates this. We have selective perception — some of us have rose-coloured glasses, some of us are gloom-and-doomers. We are not neutral; there is an emotional content to information. We need disciplined techniques to be able to stand back from that day-to-day world and really be able to see what is going on. We need approaches to help us stand back from our tendency to have biases, prejudices, and preconceptions. Can you give an example of how a preconception can influence a project? There are examples in development that are legendary. One agriculture project grew a bean that cooked faster so it would use less fuel — and therefore you would partly reduce deforestation. But there was a lot of resistance to adopting it. Part of the resistance was because one of the few times women in this culture were able to socialize with each other was when they were cooking. They didn't want a fast cooking bean. Those are the kinds of things evaluators see when they go in — they see the things that people can't see because they are too close to it. Evaluation is about standing back and being able to see things through somebody else's eyes. What distinguishes your approach to evaluation? One of the ways of you can distinguish different evaluation approaches is by what they take as their bottom line for the evaluation. For me, it is the pragmatic use of evaluation findings and the evaluation process. In other words, it is that the evaluation is designed and implemented in a way that really makes a difference to improving programs and improving decisions about programs. So the bottom line in my approach is use — that's the reason why my approach is called utilization focussed evaluation. How do you think the usefulness of evaluations can be increased? In the timing of the evaluation, for example, it means that you time the findings to match when decisions are really going to be made. A lot of evaluations take place at the end of a project. An evaluation report gets written and it's a very good piece of work. But all the decisions have already been made about the future of the project by the time the evaluation gets done. On paper, it appears to make sense to do an evaluation right at the end of the project to try to capture everything that's gone on. But it turns out not to be useful to do that. Everything that is going to be decided about the future of a program gets decided before the end of the program. Comments: Evaluation at the end of program is summative evaluation. Formative evaluation should be done so that improvements could be made during the program, and findings could be used to support decision making for the future of the program. How can evaluations inform decision-making? By knowing what the questions are that the decision-makers bring to a project. So, for example, you have to know: is there consideration being given to expanding the project from one part of the world to another — to adapt the intervention to a new ecosystem or a new group of people? Or do decision-makers already know that resources are declining and the real question is: Can we do more with less? Knowing what the decision context is lets you gather data that is relevant. A lot of evaluations get designed generically. When decision-makers get them, the response is: Well that's interesting. But it doesn't help me with what my decision is. It doesn't answer my question. What do you see as the difference between research and evaluation? There's a whole continuum of different kinds of evaluation and different kinds of research. However, on the whole, the purpose of evaluation is to produce useful information for program improvements and decision making. And the purpose of research is to produce knowledge about how the world works. Because research is driven by the agenda of knowledge production, the standards for evidence are higher, and the time lines for generating knowledge can be longer. In evaluation, there are very concrete deadlines for when decisions have to get made, for when program action has to be taken. It often means that the levels of evidence involve less certainty than they would under a research approach and that the time lines are much shorter. If you don't have the highest possible levels of evidence in the evaluation, isn't there a risk of making bad decisions? In the real world, you don't have perfect knowledge and decisions are going to get made anyway. When a program is coming to an end and a decision has to get made about it, the decision is going to get made whether or not you have perfect knowledge. If you are saying: "No, don't decide now. Wait until I have perfect knowledge", the train is going to pass. The reality is that it's better to have some information in a timely fashion than to have perfect information too late to get used. What is participatory evaluation? What are its advantages? Participatory evaluation means involving people in the evaluation — not only to make the findings more relevant and more meaningful to them through their participation, but also to build their capacity for engaging in future evaluations and to deepen their capacity for evaluative thinking. So, let's say you want to do a serious evaluation and you are trying to decide whether or not to have an external person do it or to do it internally. The external person may do a very good job of generating findings for you. But all the things that they learn about how to do evaluation, they take away with them. If you do the evaluation with counterparts in the countries where you are working, then they get the opportunity not only to generate findings and know where those findings come from, but also to learn about evaluative thinking. What is evaluative thinking? Evaluative thinking includes a willingness to do reality testing, to ask the question: how do we know what we think that we know. To use data to inform decisions — not to make data the only basis of decisions — but to bring data to bear on decisions. Evaluative thinking is not just limited to evaluation projects, it's not even just limited to formal evaluation. It's an analytical way of thinking that infuses everything that goes on. What is the hardest thing to teach about evaluation? The hardest thing I find to teach is how to go from data to recommendations. When you are doing an evaluation, you are looking at what has gone on — a history. But when you write recommendations, you are a futurist. Evaluations can help you make forecasts, but future decisions are not just a function of data. Making good, contextually grounded, politically savvy and do-able recommendations is a sophisticated skill. A great evaluator can really show the strengths and weaknesses in a program and can gather good, credible data about what is working and not working. But that doesn't mean that they know how to turn that information into recommendations. I actually prefer to involve the primary decision-makers who are going to use the evaluation in generating their own recommendations through a process of facilitation and collaboration. I encourage them to look at the data, consider the options, and then come up with their own recommendations in a context that includes their values, experience, and resources. When is it good not to evaluate a project or program? You can overdo evaluation just because people get sick of it. There are also times of crisis when you need to take action, rather than study the questions. I've seen projects go down the tubes while people were studying and evaluating when in fact they needed to take action. What do you think is the most important key to evaluation? It is being serious diligent and disciplined about asking the questions, over and over: "What are we really going to do with this? Why are we doing it? What purpose is it going to serve? How are we going to use this information?" This typically gets answered casually: "We are going to use the evaluation to improve the program" — without asking the more detailed questions: "What do we mean by improve the program? What aspects of the program are we trying to improve?" So a focus develops, driven by use. Source: http://www.evaluationwiki.org/index.php/Michael_Quinn_Patton

20101127 - Evaluation Research (Wikipedia)

Comments: Below is my reading on Evaluation Research, taken from Wikipedia on Sat, 27 Nov 2010. Evaluation research is used to determine the impact of a social intervention. A social intervention is an action taken within a social context designed to produce an intended result. Evaluation research thus analyzes the impact of a particular program on a certain social problem the program is trying to solve. Appropriate Topics Appropriate topics for evaluation research are as diverse and extensive as any other social research. Many topics exist that are in need of evaluation; however, the ones selected for evaluation have a practical significance. Practical significance refers to a topic's application in the real-world. Evaluation research has to be conducted during real life situations, meaning researchers need real life participants who are willing to be cooperative. Types of Evaluation Research Studies In the field, there are three main types of studies: 1 Needs assessment studies Particular studies directed to determine the existence and extent of problems, usually pertaining to a specific population. 2 Cost-benefit studies Studies that decide whether the results of a program justify the expense. The cost could have been financial or non-financial. 3 Monitoring studies Studies that provide a steady flow of information about a topic of interest. These studies are usually conducted over an extended period of time. In some cases, monitoring studies require incremental interventions, meaning the results may change slightly as monitoring methods alter and changes within the topic being studied are made. Common monitoring studies focus on crime rates or epidemic outbreaks. Issues of Measurement While executing evaluation research, a researcher may encounter issues that limit the overall quality and validity of their findings. With all evaluation research , researchers must be able to operationalize, observe, and recognize the presence or absence of what is under study. A common problem is the attempt to measure a variable that is simply unmeasurable. Also, researchers must be fully aware of outside factors that may influence the variables they are studying. Researchers must also note not just the presence or absence of a variable, but they must analyze the extent to which the variable is present and the amount of its involvement. New vs. existing measures When designing an evaluation research study, a researcher must decide whether to create his or her own measures or to make use of existing ones. One consideration is that existing measures may not necessarily meet a researcher's goals; however, they do allow for comparison. Types of Research Design 1 Experimental design One evaluation research design type is experimental design. An experimental design for a study whose aim is to evaluate a new medical treatment would likely adhere to the following steps: Step 1: identify the population; identify used definitions (operationalizations) and pre-test information on patients used in the treatment Step 2: divide patients into experimental and control groups; consider ethical issues: in particular, is it ethically sound to deny treatment of the control group? Step 3: determine the length of the measurement Step 4: determine the details of measurement, context, interventions, etc. Step 5: wait for the treatment to run its course, measure what's needed, and determine how the response variable was changed (if at all) 2 Quasi-experimental design Another evaluation research design type is quasi-experimental design. Quasi-experimental designs are non-rigorous inquiries that somewhat resemble controlled experiments; however, they lack key elements that are indicative of experiments, including pre- and post-testing and/or control groups. 3 Time-series design Time-series design is an additional evaluation research design type. This design type entails measurements made over a fixed time period, such as the study of traffic accident rates before and after the lowering of the speed limit in an area. 4 Multiple time-series design Multiple time-series designs utilize more than one set of data collected over a period of time, such as accident rates over time in several states or cities. This design type allows comparisons to be made. 5 Social indicators research Social indicators research is a popular avenue that combines evaluation research with the analysis of existing data. Social indicators are measurements that are reflective of the quality or nature of social life. Social indicators are frequently monitored in order to determine the nature of social change within a society. Some commonly used social indicators include crime rates, infant mortality rates, and the number of physicians per 100,000 population. 6 Nonequivalent control groups Nonequivalent control groups share similarities with the experimental group, but they are not selected based on random assignment. For example, two comparable schools might be used; one as a nonequivalent control group, one as the experimental group. Common Problems Researchers who utilize evaluation research must contend with numerous problems. They may experience difficulty ensuring subjects' full cooperation. Additionally, they may have trouble finding willing participants, as research and participation in it is a priority for few. Ethics Like other forms of social research, evaluation research must take into account ethical considerations. As mentioned earlier, sometimes evaluation research studies may require the split of subjects into experimental and control groups. When considering controversial topics, such as sex education programs, it can be the evaluation research itself that raises ethical problems. In this case, the following question might be raised: is it ethically sound to expose only the experimental group to sex education programs? Results and Interpretation Results from evaluation research are not always put into practice. One reason for this is that implications may not be presented in such a manner that is understandable to non-researchers. Additionally, results may contrast with researchers' beliefs. Researchers may have a vested interest in the results. They may also be pressured to come up with certain results, such as positive findings for a pharmaceutical company. In this case, ethical concerns regarding the researchers' objectivity must be raised. When laypersons interpret research findings, they must always be aware of the highly political aspect of research. Political agendas can be difficult to identify, and they often cause research to be biased. This involves not just party politics, but also politics that are at play within the scientific and academic communities. Comments: I concur that the research community may tend to be biased, e.g. toward their norms, beliefs and practices. I have been advised by 2 R&D key persons to redirect (is that bias?) my research in certain manner. Do I want my PhD in easier manner? If yes, it is best to follow the advice of the University Research Community. I am in dilemma. Source: http://en.wikibooks.org/wiki/Social_Research_Methods/Evaluation_Research

20101127 - Evaluation Research (Powell)

Evaluation Research: an overview Comments: This is preview of a paper. Library Trends June 22, 2006 Powell, Ronald R. ABSTRACT Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs. After the reasons for conducting evaluation research are discussed, the general principles and types are reviewed. Several evaluation methods are then presented, including input measurement, output/ performance measurement, impact/outcomes assessment, service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. Other aspects of evaluation research considered are the steps of planning and conducting an evaluation study and the measurement process, including the gathering of statistics and the use of data collection techniques. The process of data analysis and the evaluation report are also given attention. It is concluded that evaluation research should be a rigorous, systematic process that involves collecting data about organizations, processes, programs, services, and/or resources. Evaluation research should enhance knowledge and decision making and lead to practical applications. WHAT IS EVALUATION RESEARCH? Evaluation research is not easily defined. There is not even unanimity regarding its name; it is referred to as evaluation research and evaluative research. Some individuals consider evaluation research to be a specific research method; others focus on special techniques unique, more often than not, to program evaluation; and yet others view it as a research activity that employs standard research methods for evaluative purposes. Consistent with the last perspective, Childers concludes, "The differences between evaluative research and other research center on the orientation of the research and not on the methods employed" (1989, p. 251). When evaluation research is treated as a research method, it is likely to be seen as a type of applied or action research, not as basic or theoretical research. Weiss, in her standard textbook, defines evaluation as "the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy" (1998, p. 4; emphasis in original). While certainly not incorrect, this definition, at least within a library and information (LIS) context, is too narrow or limited. Wallace and Van Fleet, for example, point out that "evaluation has to do with understanding library systems" (2001, p. 1). As will be noted later in this article, evaluative methods are used for everything from evaluating library collections to reference transactions. WHY EVALUATE? But before examining the specific techniques and methods used in LIS evaluation research, let us first briefly consider the question of why evaluation is important and then identify the desirable characteristics of evaluation, the steps involved in planning an evaluation study, and the general approaches to evaluation. With regard to the initial question, Wallace and Van Fleet (2001, pp. xx-xxi) and others have noted that there are a growing number of reasons why it is important for librarians and other information professionals to evaluate their organizations' operations, resources, and services. Among those reasons are the need for organizations to 1. account for how they use their limited resources 2. explain what they do 3. enhance their visibility 4. describe their impact 5. increase efficiency 6. avoid errors 7. support planning activities 8. express concern for their public 9. support decision making 10. strengthen their political position. In addition to some of the reasons listed above, Weiss (1998, pp. 20-28) identifies several other purposes for evaluating programs and policies. They include the following: 1. Determining how clients are faring 2. Providing legitimacy for decisions 3. Fulfilling grant requirements 4. Making midcourse corrections in programs 5. Making decisions to continue or culminate programs 6. Testing new ideas 7. Choosing the best alternatives 8. Recording program history 9. Providing feedback to staff 10. Highlighting goals "Over the past decade, both academics and practitioners in the field of library and information science (LIS) have increasingly recognized the significance of assessing library services" (Shi & Levy, 2005, p. 266). In August 2004 the National Commission on Libraries and Information Science announced three strategic goals to guide its work in the immediate future. Among those three goals was the appraising and assessing of library and information services. CHARACTERISTICS AND PRINCIPLES OF EVALUATION Childers (1989, p. 250), in an article emphasizing the evaluation of programs, notes that evaluation research (1) is usually employed for decision making; (2) deals with research questions about a program; (3) takes place in the real world of the program; and (4) usually represents a compromise between pure and applied research. Wallace and Van Fleet (2001) comment that evaluation should be carefully planned, not occur by accident; have a purpose that is usually goal oriented; focus on determining the quality of a product or service; go beyond measurement; not be any larger than necessary; and reflect the situation in which it will occur. Similarly, evaluation should contribute to an organization's planning efforts; be built into existing programs; provide useful, systematically collected data; employ an outside evaluator/consultant when possible; involve the staff; not be any fancier than necessary; and target multiple audiences and purposes (Some Practical Lessons on Evaluation, 2000). In an earlier work on the evaluation of special libraries, Griffiths and King (1991, p. 3) identify some principles for good evaluation that still bear repeating: 1. Evaluation must have a purpose; it must not be an end in itself 2. Without the potential for some action, there is no need to evaluate 3. Evaluation must be more than descriptive; it must take into account relationships among operational performance, users, and organizations 4. Evaluation should be a communication tool involving staff and users 5. Evaluation should not be sporadic but be ongoing and provide a means for continual monitoring, diagnosis, and change 6. Ongoing evaluation should provide a means for continual monitoring, diagnosis and change 7. Ongoing evaluation should be dynamic in nature, reflecting new knowledge and changes in the environment As has been implied, but not explicitly stated above, evaluation often attempts to assess the effectiveness of a program or service. On a more specific level, evaluation can be used to support accreditation reviews, needs assessments, new projects, personnel reviews, conflict resolution, and professional compliance reports. Source: http://www.accessmylibrary.com/article-1G1-151440807/evaluation-research-overview.html Action: to download the entire paper at Univ.

20101127 - Evaluative Research (Winsett, NATCO)

Evaluative Research Rebecca P. Winsett, PhD, Associate Professor, University of ennessee-Memphis, Memphis, TN; Professor, University of Southern Indiana, Evansville, IN, NATCO Research Committee Co-Chair Introduction Evaluation of the impact of a new program is an important aspect of many organizations. Finding that implementing a new way to evaluate donors or whether or not adding a return to work program is successful may be examples of program evaluation. There is a design methodology called evaluative research that can help guide you as you design studies for future publication. As stated in Weiss (Weiss, 1998), evaluation is the "systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy" (pg 4). Elements of Evaluation Two questions that you may ask about a program may include if the program has the intended effect (What are the outcomes?) or is the program being delivered as it was originally constructed (What services are being delivered?). In these two questions, you may have several ways to evaluate the impact (the outcomes) and its effectiveness (process). Outcomes refer to the results of the program. Were the services successful? Did those in a smoking cessation program stop smoking? Did those who attended an education program increase knowledge? Was attitude about organ donor awareness changed as a result of your program? Methods to measure outcomes are not too distant from those you would find in traditional research methodology. Measuring group differences or pre/post-testing are examples of ways to measure outcomes. Process refers to the internal working of the program itself. Understanding the operations of the program and how it can be improved. Designing the evaluation of the process of the program requires thoughtful planning of what you want to examine. Examples may include monitoring staff time during the program to show cost savings or collecting surveys from different levels of staff to tease out problems with work efficiency. Qualitative studies can also be included. Each type of evaluation, whether outcome or process must be systematically developed to ensure the effects of the program are truly effects rather than opinion. Source: http://www.natco1.org/research/files/EvaluativeResearch_000.pdf Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

20101127 - The Planning-Evaluation Cycle (SocialResearchMethods.net)

Often, evaluation is construed as part of a larger managerial or administrative process. Sometimes this is referred to as the planning-evaluation cycle. The distinctions between planning and evaluation are not always clear; this cycle is described in many different ways with various phases claimed by both planners and evaluators. Usually, the first stage of such a cycle -- the planning phase -- is designed to elaborate a set of potential actions, programs, or technologies, and select the best for implementation. Depending on the organization and the problem being addressed, a planning process could involve any or all of these stages: the formulation of the problem, issue, or concern; the broad conceptualization of the major alternatives that might be considered; the detailing of these alternatives and their potential implications; the evaluation of the alternatives and the selection of the best one; and the implementation of the selected alternative. Although these stages are traditionally considered planning, there is a lot of evaluation work involved. Evaluators are trained in needs assessment, they use methodologies -- like the concept mapping one presented later -- that help in conceptualization and detailing, and they have the skills to help assess alternatives and make a choice of the best one. The evaluation phase also involves a sequence of stages that typically includes: the formulation of the major objectives, goals, and hypotheses of the program or technology; the conceptualization and operationalization of the major components of the evaluation -- the program, participants, setting, and measures; the design of the evaluation, detailing how these components will be coordinated; the analysis of the information, both qualitative and quantitative; and the utilization of the evaluation results. Source: http://www.socialresearchmethods.net/kb/pecycle.php

20101127 - Evaluation Research (SocialResearchMethods.net)

One specific form of social research -- evaluation research -- is of particular interest here. The Introduction to Evaluation Research presents an overview of what evaluation is and how it differs from social research generally. We also introduce several evaluation models to give you some perspective on the evaluation endeavor. Evaluation should not be considered in a vacuum. Here, we consider evaluation as embedded within a larger Planning-Evaluation Cycle. Evaluation can be a threatening activity. Many groups and organizations struggle with how to build a good evaluation capability into their everyday activities and procedures. This is essentially an organizational culture issue. Here we consider some of the issues a group or organization needs to address in order to develop an evaluation culture that works in their context. Source: http://www.socialresearchmethods.net/kb/evaluation.php Introduction to Evaluation Evaluation is a methodological area that is closely related to, but distinguishable from more traditional social research. Evaluation utilizes many of the same methodologies used in traditional social research, but because evaluation takes place within a political and organizational context, it requires group skills, management ability, political dexterity, sensitivity to multiple stakeholders and other skills that social research in general does not rely on as much. Here we introduce the idea of evaluation and some of the major terms and issues in the field. Definitions of Evaluation Probably the most frequently given definition is: Evaluation is the systematic assessment of the worth or merit of some object This definition is hardly perfect. There are many types of evaluations that do not necessarily result in an assessment of worth or merit -- descriptive studies, implementation analyses, and formative evaluations, to name a few. Better perhaps is a definition that emphasizes the information-processing and feedback functions of evaluation. For instance, one might say: Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object Both definitions agree that evaluation is a systematic endeavor and both use the deliberately ambiguous term 'object' which could refer to a program, policy, technology, person, need, activity, and so on. The latter definition emphasizes acquiring and assessing information rather than assessing worth or merit because all evaluation work involves collecting and sifting through data, making judgements about the validity of the information and of inferences we derive from it, whether or not an assessment of worth or merit results. The Goals of Evaluation The generic goal of most evaluations is to provide "useful feedback" to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies. Most often, feedback is perceived as "useful" if it aids in decision-making. But the relationship between an evaluation and its impact is not a simple one -- studies that seem critical sometimes fail to influence short-term decisions, and studies that initially seem to have no influence can have a delayed impact when more congenial conditions arise. Despite this, there is broad consensus that the major goal of evaluation should be to influence decision-making or policy formulation through the provision of empirically-driven feedback. Evaluation Strategies 'Evaluation strategies' means broad, overarching perspectives on evaluation. They encompass the most general groups or "camps" of evaluators; although, at its best, evaluation work borrows eclectically from the perspectives of all these camps. Four major groups of evaluation strategies are discussed here. Scientific-experimental models are probably the most historically dominant evaluation strategies. Taking their values and methods from the sciences -- especially the social sciences -- they prioritize on the desirability of impartiality, accuracy, objectivity and the validity of the information generated. Included under scientific-experimental models would be: the tradition of experimental and quasi-experimental designs; objectives-based research that comes from education; econometrically-oriented perspectives including cost-effectiveness and cost-benefit analysis; and the recent articulation of theory-driven evaluation. The second class of strategies are management-oriented systems models. Two of the most common of these are PERT, the Program Evaluation and Review Technique, and CPM, the Critical Path Method. Both have been widely used in business and government in this country. It would also be legitimate to include the Logical Framework or "Logframe" model developed at U.S. Agency for International Development and general systems theory and operations research approaches in this category. Two management-oriented systems models were originated by evaluators: the UTOS model where U stands for Units, T for Treatments, O for Observing Observations and S for Settings; and the CIPP model where the C stands for Context, the I for Input, the first P for Process and the second P for Product. These management-oriented systems models emphasize comprehensiveness in evaluation, placing evaluation within a larger framework of organizational activities. The third class of strategies are the qualitative/anthropological models. They emphasize the importance of observation, the need to retain the phenomenological quality of the evaluation context, and the value of subjective human interpretation in the evaluation process. Included in this category are the approaches known in evaluation as naturalistic or 'Fourth Generation' evaluation; the various qualitative schools; critical theory and art criticism approaches; and, the 'grounded theory' approach of Glaser and Strauss among others. Finally, a fourth class of strategies is termed participant-oriented models. As the term suggests, they emphasize the central importance of the evaluation participants, especially clients and users of the program or technology. Client-centered and stakeholder approaches are examples of participant-oriented models, as are consumer-oriented evaluation systems. With all of these strategies to choose from, how to decide? Debates that rage within the evaluation profession -- and they do rage -- are generally battles between these different strategists, with each claiming the superiority of their position. In reality, most good evaluators are familiar with all four categories and borrow from each as the need arises. There is no inherent incompatibility between these broad strategies -- each of them brings something valuable to the evaluation table. In fact, in recent years attention has increasingly turned to how one might integrate results from evaluations that use different strategies, carried out from different perspectives, and using different methods. Clearly, there are no simple answers here. The problems are complex and the methodologies needed will and should be varied. Types of Evaluation There are many different types of evaluations depending on the object being evaluated and the purpose of the evaluation. Perhaps the most important basic distinction in evaluation types is that between formative and summative evaluation. Formative evaluations strengthen or improve the object being evaluated -- they help form it by examining the delivery of the program or technology, the quality of its implementation, and the assessment of the organizational context, personnel, procedures, inputs, and so on. Summative evaluations, in contrast, examine the effects or outcomes of some object -- they summarize it by describing what happens subsequent to delivery of the program or technology; assessing whether the object can be said to have caused the outcome; determining the overall impact of the causal factor beyond only the immediate target outcomes; and, estimating the relative costs associated with the object. Formative evaluation includes several evaluation types: needs assessment determines who needs the program, how great the need is, and what might work to meet the need evaluability assessment determines whether an evaluation is feasible and how stakeholders can help shape its usefulness structured conceptualization helps stakeholders define the program or technology, the target population, and the possible outcomes implementation evaluation monitors the fidelity of the program or technology delivery process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures Summative evaluation can also be subdivided: outcome evaluations investigate whether the program or technology caused demonstrable effects on specifically defined target outcomes impact evaluation is broader and assesses the overall or net effects -- intended or unintended -- of the program or technology as a whole cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs and values secondary analysis reexamines existing data to address new questions or use methods not previously employed meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question Evaluation Questions and Methods Evaluators ask many different kinds of questions and use a variety of methods to address them. These are considered within the framework of formative and summative evaluation as presented above. In formative research the major questions and methodologies are: What is the definition and scope of the problem or issue, or what's the question? Formulating and conceptualizing methods might be used including brainstorming, focus groups, nominal group techniques, Delphi methods, brainwriting, stakeholder analysis, synectics, lateral thinking, input-output analysis, and concept mapping. Where is the problem and how big or serious is it? The most common method used here is "needs assessment" which can include: analysis of existing data sources, and the use of sample surveys, interviews of constituent populations, qualitative research, expert testimony, and focus groups. How should the program or technology be delivered to address the problem? Some of the methods already listed apply here, as do detailing methodologies like simulation techniques, or multivariate methods like multiattribute utility theory or exploratory causal modeling; decision-making methods; and project planning and implementation methods like flow charting, PERT/CPM, and project scheduling. How well is the program or technology delivered? Qualitative and quantitative monitoring techniques, the use of management information systems, and implementation assessment would be appropriate methodologies here. The questions and methods addressed under summative evaluation include: What type of evaluation is feasible? Evaluability assessment can be used here, as well as standard approaches for selecting an appropriate evaluation design. What was the effectiveness of the program or technology? One would choose from observational and correlational methods for demonstrating whether desired effects occurred, and quasi-experimental and experimental designs for determining whether observed effects can reasonably be attributed to the intervention and not to other sources. What is the net impact of the program? Econometric methods for assessing cost effectiveness and cost/benefits would apply here, along with qualitative methods that enable us to summarize the full range of intended and unintended impacts. Clearly, this introduction is not meant to be exhaustive. Each of these methods, and the many not mentioned, are supported by an extensive methodological research literature. This is a formidable set of tools. But the need to improve, update and adapt these methods to changing circumstances means that methodological research and development needs to have a major place in evaluation work. Source: http://www.socialresearchmethods.net/kb/intreval.php