Usability Framework (Shackel)

Human factors for informatics usability By Brian Shackel, Simon J. Richardson Chapter 2 Usability - Context, Framework, Definition, Design and Evaluation Brian Shackel http://books.google.com.my/books?hl=en&lr=&id=KSHrPgLlMJIC&oi=fnd&pg=PA21&dq=usability+framework&ots=ITWmRUSVB9&sig=5q50cAypuH-8I1LN4KPWtRRIoEs#v=onepage&q=usability%20framework&f=false

20101127 - concepts & defintions of Evaluation research (Stern)

2. Can evaluation be defined?   Stern, E. (2004).  Philosophies and types of evaluation research.  In Descy, P.; Tessaring, M. (eds), The foundations of evaluation and impact research.   There are numerous definitions and types of evaluation. There are, for example, many definitions of evaluation put forward in handbooks, evaluation guidelines and administrative procedures, by bodies that commission and use evaluation. All of these definitions draw selectively on a wider debate as to the scope and focus of evaluation. A recent book identifies 22 foundation models for 21st century programme evaluation (Stufflebeam, 2000a), although the authors suggest that a smaller subset of nine are the strongest. Rather than begin with types and models, this chapter begins with an attempt to review and bring together the main ideas and orientations that underpin evaluation thinking.   Indicating potential problems with 'definition' by a question mark in the title of this section warns the reader not to expect straightforward or consistent statements. Evaluation has grown up through different historical periods in different policy environments, with inputs from many disciplines and methodologies, from diverse value positions and rooted in hard fought debates in philosophy of science and theories of knowledge.   While there is some agreement, there is also persistent difference: evaluation is contested terrain. Most of these sources are from North America where evaluation has been established – as a discipline and practice – and debated for 30 or more years.   2.1. Assessing or explaining outcomes   Among the most frequently quoted definitions is that of Scriven who has produced an evaluation Thesaurus, his own extensive handbook of evaluation terminology: '"evaluation" refers to the process of determining the merit, worth or value of something, or the product of that process […]  The evaluation process normally involves some identification of relevant standards or merit, worth or value; some investigation of the performance of evaluands on these standards; and some integration or synthesis of the results to achieve an overall evaluation or set of associated evaluations.' (Scriven, 1991; p. 139).   This definition prepares the way for what has been called 'the logic of evaluation' (Scriven, 1991; Fournier, 1995). This logic is expressed in a sequence of four stages: (a) establishing evaluation criteria and related dimensions; (b) constructing standards of performance in relation to these criteria and dimensions; (c) measuring performance in practice; (d) reaching a conclusion about the worth of the object in question.   This logic is not without its critics (e.g. Schwandt, 1997) especially among those of a naturalistic or constructivist turn who cast doubt on the claims of evaluators to know, to judge and ultimately to control. Other stakeholders, it is argued, have a role and this changed relationship with stakeholders is discussed further below.   The most popular textbook definition of evaluation can be found in Rossi et. al.'s book Evaluation – a systematic approach: 'Program evaluation is the use of social research procedures to systematically investigate the effectiveness of social intervention programs. More specifically, evaluation researchers (evaluators) use social research methods to study, appraise, and help improve social programmes in all their important aspects, including the diagnosis of the social problems they address, their conceptualization and design, their implementation and administration, their outcomes, and their efficiency.' (Rossi et al., 1999; p. 4).   Using words such as effectiveness rather than Scriven's favoured 'merit worth or value' begins to shift the perspective of this definition towards the explanation of outcomes and impacts. This is partly because Rossi and his colleagues identify helping improve social programmes as one of the purposes of evaluation. Once there is an intention to make programmes more effective, the need to explain how they work becomes more important.   Yet, explanation is an important and intentionally absent element in Scriven's definitions of evaluation: 'By contrast with evaluation, which identifies the value of something, explanation involves answering a Why or How question about it or a call for some other type of understanding. Often, explanation involves identifying the cause of a phenomenon, rather than its effects (which is a major part of evaluation). When it is possible, without jeopardizing the main goals of an evaluation, a good evaluation design tries to uncover microexplanations (e.g. by identifying those components of the curriculum package that are producing the major part of the good or bad effects, and/or those that are having little effect). The first priority, however, is to resolve the evaluation issues (is the package any good at all, the best available? etc.). Too often the research orientation and training of evaluators leads them to do a poor job on evaluation because they became interested in explanation.' (Scriven, 1991, p. 158).   Scriven himself recognises that one pressure moving evaluation to pay greater attention to explanation is the emergence of programme theory, with its concern about how programmes operate so that they can be improved or better implemented. A parallel pressure comes from the uptake of impact assessment associated with the growth of performance management and other managerial reforms within public sector administrations.   The intellectual basis for this work was most consistently elaborated by Wholey and colleagues. They start from the position that evaluation should be concerned with the efficiency and effectiveness of the way governments deliver public services. A core concept within this approach is what is called 'evaluability assessment' (Wholey, 1981). The starting point for this assessment is a critical review of the logic of programmes and the assumptions that underpin them. This work constitutes the foundation for most of the thinking about programme theory and logical frameworks. It also prefigures a later generation of evaluation thinking rooted more in policy analysis that is concerned with the institutionalisation of evaluation within public agencies (Boyle and Lemaire, 1999), as discussed further below.   These management reforms generally link interventions with outcomes. As Rossi et al. recognise, this takes us to the heart of broader debates in the social sciences about causality: 'The problem of establishing a program's impact is identical to the problem of establishing that the program is a cause of some specified effect. Hence, establishing impact essentially amounts to establishing causality.' (Rossi et al., 1999).   The difficulties of establishing perfect, rather than good enough, impact assessments are recognised by Rossi and colleagues. This takes us into the territory of experimentation and causal inference associated with some of the most influential founders of North American evaluations such as Campbell, with his interest in experimental and quasi-experimental designs, but also his interest in later years in the explanatory potential of qualitative evaluation methods.   The debate about experimentation and causality in evaluation continues to be vigorously pursued in various guises. For example, in a recent authoritative text on experimentation and causal inference, (Shadish et al., 2002) the authors begin to take on board contemporary criticisms of experimental methods that have come from the philosophy of science and the social sciences more generally. In recent years, we have also seen a sustained realist critique on experimental methods led in Europe by Pawson and Tilley (1997). But, whatever their orientations to experimentation and causal inference, explanations remain at the heart of the concerns of an important constituency within evaluation.   2.2. Evaluation, change and values   Another important strand in evaluation thinking concerns the relationship between evaluation and action or change. One comparison is between 'summative' and 'formative' evaluation methods, terms also coined by Scriven. The former assesses or judges results and the latter seeks to influence or promote change.   Various authors have contributed to an understanding of the role of evaluation and change. For example, Cronbach (1982, 1989) rooted in policy analysis and education, sees an important if limited role for evaluation in shaping policy 'at the margins' through 'piecemeal adaptations'. The role of evaluation in Cronbach's framework is to inform policies and programmes through the generation of knowledge that feeds into the 'policy shaping community' of experts, administrators and policy-makers.   Stake (1996) on the other hand, with his notion of 'responsive evaluation', sees this as a 'service' to programme stakeholders and to participants. By working with those who are directly involved in a programme, Stake sees the evaluator as supporting their participation and possibilities for initiating change. This contrasts with Cronbach's position and even more strongly with that of Wholey (referred to earlier) given Stake's scepticism about the possibilities of change at the level of large scale national (or in the US context Federal and State) programmes and their management.   Similarly, Patton, (1997 and earlier editions) who has tended to eschew work at programme and national level, shares with Stake a commitment to working with stakeholders and (local) users. His concern is for 'intended use by intended users'.   Virtually everyone in the field recognises the political and value basis of much evaluation activity, albeit in different ways. While Stake, Cronbach and Wholey may recognise the importance of values within evaluation, the values that they recognise are variously those of stakeholders, participants and programme managers.   There is another strand within the general orientation towards evaluation and change which is decidedly normative. This category includes House, with his emphasis on evaluation for social justice and the emancipatory logic of Fetterman et al. (1996) and 'empowerment evaluation'. Within the view of Fetterman and his colleagues, evaluation itself is not undertaken by external experts but rather is a self-help activity in which – because people empower themselves – the role of any external input is to support self-help.   So, one of the main differences among those evaluators who explicitly address issues of programme and societal change is in terms of the role of evaluators, be they experts who act, facilitators and advocates, or enablers of self help.     Source:  http://www.cedefop.europa.eu/EN/Files/BgR1_Stern.pdf   Comments:  Boring. Reading thru all these concepts and definitions on Evaluation Research.

 

20101127 - Evaluation research/method (Patton)

Evaluation Research : an interview with Dr Patton Interviewed by Lisa Waldick Comments: Dr Michael Quinn Patton is well-known for Evaluation Research methods. So, this creates my interest to read this interview. Many development programs are evaluated to determine how effective and useful they are. But how effective and useful are the evaluations themselves? Internationally renowned evaluator, Michael Quinn Patton, recently came to IDRC (International Development Research Centre) to discuss his approach for making sure evaluations are useful for decision-makers. Dr Patton is the head of an organizational development consulting business: Utilization-Focused Information and Training. Known for five influential books on evaluation, including Qualitative/Evaluation and Research Methods, he was the 1984 recipient of the Alva and Gunnar Myrdal Award from the Evaluation Research Society for "outstanding contributions to evaluation use and practice" Why is it important to evaluate development programs? Don't people on the ground just intuitively understand what is going on in programs? Our very processes of taking in information distort reality — all the evidence of social science indicates this. We have selective perception — some of us have rose-coloured glasses, some of us are gloom-and-doomers. We are not neutral; there is an emotional content to information. We need disciplined techniques to be able to stand back from that day-to-day world and really be able to see what is going on. We need approaches to help us stand back from our tendency to have biases, prejudices, and preconceptions. Can you give an example of how a preconception can influence a project? There are examples in development that are legendary. One agriculture project grew a bean that cooked faster so it would use less fuel — and therefore you would partly reduce deforestation. But there was a lot of resistance to adopting it. Part of the resistance was because one of the few times women in this culture were able to socialize with each other was when they were cooking. They didn't want a fast cooking bean. Those are the kinds of things evaluators see when they go in — they see the things that people can't see because they are too close to it. Evaluation is about standing back and being able to see things through somebody else's eyes. What distinguishes your approach to evaluation? One of the ways of you can distinguish different evaluation approaches is by what they take as their bottom line for the evaluation. For me, it is the pragmatic use of evaluation findings and the evaluation process. In other words, it is that the evaluation is designed and implemented in a way that really makes a difference to improving programs and improving decisions about programs. So the bottom line in my approach is use — that's the reason why my approach is called utilization focussed evaluation. How do you think the usefulness of evaluations can be increased? In the timing of the evaluation, for example, it means that you time the findings to match when decisions are really going to be made. A lot of evaluations take place at the end of a project. An evaluation report gets written and it's a very good piece of work. But all the decisions have already been made about the future of the project by the time the evaluation gets done. On paper, it appears to make sense to do an evaluation right at the end of the project to try to capture everything that's gone on. But it turns out not to be useful to do that. Everything that is going to be decided about the future of a program gets decided before the end of the program. Comments: Evaluation at the end of program is summative evaluation. Formative evaluation should be done so that improvements could be made during the program, and findings could be used to support decision making for the future of the program. How can evaluations inform decision-making? By knowing what the questions are that the decision-makers bring to a project. So, for example, you have to know: is there consideration being given to expanding the project from one part of the world to another — to adapt the intervention to a new ecosystem or a new group of people? Or do decision-makers already know that resources are declining and the real question is: Can we do more with less? Knowing what the decision context is lets you gather data that is relevant. A lot of evaluations get designed generically. When decision-makers get them, the response is: Well that's interesting. But it doesn't help me with what my decision is. It doesn't answer my question. What do you see as the difference between research and evaluation? There's a whole continuum of different kinds of evaluation and different kinds of research. However, on the whole, the purpose of evaluation is to produce useful information for program improvements and decision making. And the purpose of research is to produce knowledge about how the world works. Because research is driven by the agenda of knowledge production, the standards for evidence are higher, and the time lines for generating knowledge can be longer. In evaluation, there are very concrete deadlines for when decisions have to get made, for when program action has to be taken. It often means that the levels of evidence involve less certainty than they would under a research approach and that the time lines are much shorter. If you don't have the highest possible levels of evidence in the evaluation, isn't there a risk of making bad decisions? In the real world, you don't have perfect knowledge and decisions are going to get made anyway. When a program is coming to an end and a decision has to get made about it, the decision is going to get made whether or not you have perfect knowledge. If you are saying: "No, don't decide now. Wait until I have perfect knowledge", the train is going to pass. The reality is that it's better to have some information in a timely fashion than to have perfect information too late to get used. What is participatory evaluation? What are its advantages? Participatory evaluation means involving people in the evaluation — not only to make the findings more relevant and more meaningful to them through their participation, but also to build their capacity for engaging in future evaluations and to deepen their capacity for evaluative thinking. So, let's say you want to do a serious evaluation and you are trying to decide whether or not to have an external person do it or to do it internally. The external person may do a very good job of generating findings for you. But all the things that they learn about how to do evaluation, they take away with them. If you do the evaluation with counterparts in the countries where you are working, then they get the opportunity not only to generate findings and know where those findings come from, but also to learn about evaluative thinking. What is evaluative thinking? Evaluative thinking includes a willingness to do reality testing, to ask the question: how do we know what we think that we know. To use data to inform decisions — not to make data the only basis of decisions — but to bring data to bear on decisions. Evaluative thinking is not just limited to evaluation projects, it's not even just limited to formal evaluation. It's an analytical way of thinking that infuses everything that goes on. What is the hardest thing to teach about evaluation? The hardest thing I find to teach is how to go from data to recommendations. When you are doing an evaluation, you are looking at what has gone on — a history. But when you write recommendations, you are a futurist. Evaluations can help you make forecasts, but future decisions are not just a function of data. Making good, contextually grounded, politically savvy and do-able recommendations is a sophisticated skill. A great evaluator can really show the strengths and weaknesses in a program and can gather good, credible data about what is working and not working. But that doesn't mean that they know how to turn that information into recommendations. I actually prefer to involve the primary decision-makers who are going to use the evaluation in generating their own recommendations through a process of facilitation and collaboration. I encourage them to look at the data, consider the options, and then come up with their own recommendations in a context that includes their values, experience, and resources. When is it good not to evaluate a project or program? You can overdo evaluation just because people get sick of it. There are also times of crisis when you need to take action, rather than study the questions. I've seen projects go down the tubes while people were studying and evaluating when in fact they needed to take action. What do you think is the most important key to evaluation? It is being serious diligent and disciplined about asking the questions, over and over: "What are we really going to do with this? Why are we doing it? What purpose is it going to serve? How are we going to use this information?" This typically gets answered casually: "We are going to use the evaluation to improve the program" — without asking the more detailed questions: "What do we mean by improve the program? What aspects of the program are we trying to improve?" So a focus develops, driven by use. Source: http://www.evaluationwiki.org/index.php/Michael_Quinn_Patton

20101127 - Evaluation Research (Wikipedia)

Comments: Below is my reading on Evaluation Research, taken from Wikipedia on Sat, 27 Nov 2010. Evaluation research is used to determine the impact of a social intervention. A social intervention is an action taken within a social context designed to produce an intended result. Evaluation research thus analyzes the impact of a particular program on a certain social problem the program is trying to solve. Appropriate Topics Appropriate topics for evaluation research are as diverse and extensive as any other social research. Many topics exist that are in need of evaluation; however, the ones selected for evaluation have a practical significance. Practical significance refers to a topic's application in the real-world. Evaluation research has to be conducted during real life situations, meaning researchers need real life participants who are willing to be cooperative. Types of Evaluation Research Studies In the field, there are three main types of studies: 1 Needs assessment studies Particular studies directed to determine the existence and extent of problems, usually pertaining to a specific population. 2 Cost-benefit studies Studies that decide whether the results of a program justify the expense. The cost could have been financial or non-financial. 3 Monitoring studies Studies that provide a steady flow of information about a topic of interest. These studies are usually conducted over an extended period of time. In some cases, monitoring studies require incremental interventions, meaning the results may change slightly as monitoring methods alter and changes within the topic being studied are made. Common monitoring studies focus on crime rates or epidemic outbreaks. Issues of Measurement While executing evaluation research, a researcher may encounter issues that limit the overall quality and validity of their findings. With all evaluation research , researchers must be able to operationalize, observe, and recognize the presence or absence of what is under study. A common problem is the attempt to measure a variable that is simply unmeasurable. Also, researchers must be fully aware of outside factors that may influence the variables they are studying. Researchers must also note not just the presence or absence of a variable, but they must analyze the extent to which the variable is present and the amount of its involvement. New vs. existing measures When designing an evaluation research study, a researcher must decide whether to create his or her own measures or to make use of existing ones. One consideration is that existing measures may not necessarily meet a researcher's goals; however, they do allow for comparison. Types of Research Design 1 Experimental design One evaluation research design type is experimental design. An experimental design for a study whose aim is to evaluate a new medical treatment would likely adhere to the following steps: Step 1: identify the population; identify used definitions (operationalizations) and pre-test information on patients used in the treatment Step 2: divide patients into experimental and control groups; consider ethical issues: in particular, is it ethically sound to deny treatment of the control group? Step 3: determine the length of the measurement Step 4: determine the details of measurement, context, interventions, etc. Step 5: wait for the treatment to run its course, measure what's needed, and determine how the response variable was changed (if at all) 2 Quasi-experimental design Another evaluation research design type is quasi-experimental design. Quasi-experimental designs are non-rigorous inquiries that somewhat resemble controlled experiments; however, they lack key elements that are indicative of experiments, including pre- and post-testing and/or control groups. 3 Time-series design Time-series design is an additional evaluation research design type. This design type entails measurements made over a fixed time period, such as the study of traffic accident rates before and after the lowering of the speed limit in an area. 4 Multiple time-series design Multiple time-series designs utilize more than one set of data collected over a period of time, such as accident rates over time in several states or cities. This design type allows comparisons to be made. 5 Social indicators research Social indicators research is a popular avenue that combines evaluation research with the analysis of existing data. Social indicators are measurements that are reflective of the quality or nature of social life. Social indicators are frequently monitored in order to determine the nature of social change within a society. Some commonly used social indicators include crime rates, infant mortality rates, and the number of physicians per 100,000 population. 6 Nonequivalent control groups Nonequivalent control groups share similarities with the experimental group, but they are not selected based on random assignment. For example, two comparable schools might be used; one as a nonequivalent control group, one as the experimental group. Common Problems Researchers who utilize evaluation research must contend with numerous problems. They may experience difficulty ensuring subjects' full cooperation. Additionally, they may have trouble finding willing participants, as research and participation in it is a priority for few. Ethics Like other forms of social research, evaluation research must take into account ethical considerations. As mentioned earlier, sometimes evaluation research studies may require the split of subjects into experimental and control groups. When considering controversial topics, such as sex education programs, it can be the evaluation research itself that raises ethical problems. In this case, the following question might be raised: is it ethically sound to expose only the experimental group to sex education programs? Results and Interpretation Results from evaluation research are not always put into practice. One reason for this is that implications may not be presented in such a manner that is understandable to non-researchers. Additionally, results may contrast with researchers' beliefs. Researchers may have a vested interest in the results. They may also be pressured to come up with certain results, such as positive findings for a pharmaceutical company. In this case, ethical concerns regarding the researchers' objectivity must be raised. When laypersons interpret research findings, they must always be aware of the highly political aspect of research. Political agendas can be difficult to identify, and they often cause research to be biased. This involves not just party politics, but also politics that are at play within the scientific and academic communities. Comments: I concur that the research community may tend to be biased, e.g. toward their norms, beliefs and practices. I have been advised by 2 R&D key persons to redirect (is that bias?) my research in certain manner. Do I want my PhD in easier manner? If yes, it is best to follow the advice of the University Research Community. I am in dilemma. Source: http://en.wikibooks.org/wiki/Social_Research_Methods/Evaluation_Research

20101127 - Evaluation Research (Powell)

Evaluation Research: an overview Comments: This is preview of a paper. Library Trends June 22, 2006 Powell, Ronald R. ABSTRACT Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs. After the reasons for conducting evaluation research are discussed, the general principles and types are reviewed. Several evaluation methods are then presented, including input measurement, output/ performance measurement, impact/outcomes assessment, service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. Other aspects of evaluation research considered are the steps of planning and conducting an evaluation study and the measurement process, including the gathering of statistics and the use of data collection techniques. The process of data analysis and the evaluation report are also given attention. It is concluded that evaluation research should be a rigorous, systematic process that involves collecting data about organizations, processes, programs, services, and/or resources. Evaluation research should enhance knowledge and decision making and lead to practical applications. WHAT IS EVALUATION RESEARCH? Evaluation research is not easily defined. There is not even unanimity regarding its name; it is referred to as evaluation research and evaluative research. Some individuals consider evaluation research to be a specific research method; others focus on special techniques unique, more often than not, to program evaluation; and yet others view it as a research activity that employs standard research methods for evaluative purposes. Consistent with the last perspective, Childers concludes, "The differences between evaluative research and other research center on the orientation of the research and not on the methods employed" (1989, p. 251). When evaluation research is treated as a research method, it is likely to be seen as a type of applied or action research, not as basic or theoretical research. Weiss, in her standard textbook, defines evaluation as "the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy" (1998, p. 4; emphasis in original). While certainly not incorrect, this definition, at least within a library and information (LIS) context, is too narrow or limited. Wallace and Van Fleet, for example, point out that "evaluation has to do with understanding library systems" (2001, p. 1). As will be noted later in this article, evaluative methods are used for everything from evaluating library collections to reference transactions. WHY EVALUATE? But before examining the specific techniques and methods used in LIS evaluation research, let us first briefly consider the question of why evaluation is important and then identify the desirable characteristics of evaluation, the steps involved in planning an evaluation study, and the general approaches to evaluation. With regard to the initial question, Wallace and Van Fleet (2001, pp. xx-xxi) and others have noted that there are a growing number of reasons why it is important for librarians and other information professionals to evaluate their organizations' operations, resources, and services. Among those reasons are the need for organizations to 1. account for how they use their limited resources 2. explain what they do 3. enhance their visibility 4. describe their impact 5. increase efficiency 6. avoid errors 7. support planning activities 8. express concern for their public 9. support decision making 10. strengthen their political position. In addition to some of the reasons listed above, Weiss (1998, pp. 20-28) identifies several other purposes for evaluating programs and policies. They include the following: 1. Determining how clients are faring 2. Providing legitimacy for decisions 3. Fulfilling grant requirements 4. Making midcourse corrections in programs 5. Making decisions to continue or culminate programs 6. Testing new ideas 7. Choosing the best alternatives 8. Recording program history 9. Providing feedback to staff 10. Highlighting goals "Over the past decade, both academics and practitioners in the field of library and information science (LIS) have increasingly recognized the significance of assessing library services" (Shi & Levy, 2005, p. 266). In August 2004 the National Commission on Libraries and Information Science announced three strategic goals to guide its work in the immediate future. Among those three goals was the appraising and assessing of library and information services. CHARACTERISTICS AND PRINCIPLES OF EVALUATION Childers (1989, p. 250), in an article emphasizing the evaluation of programs, notes that evaluation research (1) is usually employed for decision making; (2) deals with research questions about a program; (3) takes place in the real world of the program; and (4) usually represents a compromise between pure and applied research. Wallace and Van Fleet (2001) comment that evaluation should be carefully planned, not occur by accident; have a purpose that is usually goal oriented; focus on determining the quality of a product or service; go beyond measurement; not be any larger than necessary; and reflect the situation in which it will occur. Similarly, evaluation should contribute to an organization's planning efforts; be built into existing programs; provide useful, systematically collected data; employ an outside evaluator/consultant when possible; involve the staff; not be any fancier than necessary; and target multiple audiences and purposes (Some Practical Lessons on Evaluation, 2000). In an earlier work on the evaluation of special libraries, Griffiths and King (1991, p. 3) identify some principles for good evaluation that still bear repeating: 1. Evaluation must have a purpose; it must not be an end in itself 2. Without the potential for some action, there is no need to evaluate 3. Evaluation must be more than descriptive; it must take into account relationships among operational performance, users, and organizations 4. Evaluation should be a communication tool involving staff and users 5. Evaluation should not be sporadic but be ongoing and provide a means for continual monitoring, diagnosis, and change 6. Ongoing evaluation should provide a means for continual monitoring, diagnosis and change 7. Ongoing evaluation should be dynamic in nature, reflecting new knowledge and changes in the environment As has been implied, but not explicitly stated above, evaluation often attempts to assess the effectiveness of a program or service. On a more specific level, evaluation can be used to support accreditation reviews, needs assessments, new projects, personnel reviews, conflict resolution, and professional compliance reports. Source: http://www.accessmylibrary.com/article-1G1-151440807/evaluation-research-overview.html Action: to download the entire paper at Univ.

20101127 - Evaluative Research (Winsett, NATCO)

Evaluative Research Rebecca P. Winsett, PhD, Associate Professor, University of ennessee-Memphis, Memphis, TN; Professor, University of Southern Indiana, Evansville, IN, NATCO Research Committee Co-Chair Introduction Evaluation of the impact of a new program is an important aspect of many organizations. Finding that implementing a new way to evaluate donors or whether or not adding a return to work program is successful may be examples of program evaluation. There is a design methodology called evaluative research that can help guide you as you design studies for future publication. As stated in Weiss (Weiss, 1998), evaluation is the "systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy" (pg 4). Elements of Evaluation Two questions that you may ask about a program may include if the program has the intended effect (What are the outcomes?) or is the program being delivered as it was originally constructed (What services are being delivered?). In these two questions, you may have several ways to evaluate the impact (the outcomes) and its effectiveness (process). Outcomes refer to the results of the program. Were the services successful? Did those in a smoking cessation program stop smoking? Did those who attended an education program increase knowledge? Was attitude about organ donor awareness changed as a result of your program? Methods to measure outcomes are not too distant from those you would find in traditional research methodology. Measuring group differences or pre/post-testing are examples of ways to measure outcomes. Process refers to the internal working of the program itself. Understanding the operations of the program and how it can be improved. Designing the evaluation of the process of the program requires thoughtful planning of what you want to examine. Examples may include monitoring staff time during the program to show cost savings or collecting surveys from different levels of staff to tease out problems with work efficiency. Qualitative studies can also be included. Each type of evaluation, whether outcome or process must be systematically developed to ensure the effects of the program are truly effects rather than opinion. Source: http://www.natco1.org/research/files/EvaluativeResearch_000.pdf Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies (2nd ed.). Upper Saddle River, NJ: Prentice Hall.