Our Evaluation Model: Evaluating Comprehensive Community Initiatives
-
WHY EVALUATE?
-
CHALLENGES TO COMMUNITY PARTICIPATORY EVALUATION
-
A MODEL: COMMUNITY INITIATIVES AS CATALYSTS FOR CHANGES
-
PRINCIPLES, ASSUMPTIONS, AND VALUES OF COMMUNITY EVALUATION
-
A "LOGICAL MODEL" FOR COMMUNITY EVALUATION
-
RECOMMENDATIONS
Use this model to evaluate comprehensive community initiatives working to improve quality of life in the community.
Throughout the world, people and organizations come together to address issues that matter to them. For example, some community partnerships have formed to reduce substance abuse, teen pregnancy, or violence. Other community-based efforts attempt to lower risks for HIV/AIDS, cardiovascular diseases, or injuries. Alliances among community people have also focused on promoting urban economic development, access to decent housing, and quality education.
These initiatives try to improve the quality of life for everyone in a community. Often, they do this in two ways. Initiatives use universal approaches -- that is, they try to reach everyone who could possibly be affected by the concern. They also use targeted approaches, which try to affect conditions for people who are at higher risk for the problem. Through these two approaches, initiatives try to change people's behavior, such as using illegal drugs, being physically active, or caring for children. They also might go deeper and try to change the conditions, such as the availability of drugs, or opportunity for drugs or daycare, under which these behaviors occur.
Community health promotion is a process that includes many things at many levels. For example, efforts use multiple strategies, such as providing information about the problem or improving people's access to assistance. They also operate at multiple levels, including individuals, families and organizations, and through a variety of community sectors, such as schools, businesses, and religious organizations. All of this works together to make small but widespread changes in the health of the community. The goal is to promote healthy behaviors by making them easier to do and more likely to meet with positive reinforcement.
There are a lot of different models that describe how to best promote community health and development. Some of the more popular models include the Healthy Cities/Healthy Communities Model, the PRECEDE/PROCEED model, and the Planned Approach to Community Health [PATCH]. Similarly, our University of Kansas (U.S.A.) Center for Community Health and Development's model of Building Capacity for Community Change is outlined elsewhere. While how things should be done differs in each model, the basic goal of these and other community approaches is the same. They aim to increase opportunities for community members to work together to improve their quality of life.
Unfortunately, only modest information on the effectiveness of community-based initiatives exists. That's because evaluation practice hasn't fully caught up with a recent shift towards community control of programs. Although there are models for studying community health efforts, community initiatives are often evaluated using research methods borrowed from clinical trials and other researcher-controlled techniques. While these methods work very well in the fields for which they were developed, they're not necessarily a "good fit" for evaluating community work. It's like trying to put a square peg into a round hole -- with a lot of work, you might be able to do it, but it will never be as smooth as you want.
New ideas about community evaluation have their roots in several different models and traditions. These include:
- Action anthropology, which refers to the use of research to empower communities.
- Qualitative research, which highlights the value of the experience as an important part of understanding the effort. That is, it looks at other things besides statistics as important.
- Community-based participatory research (CBPR), which uses dialogue among community and scientific partners to produce knowledge and guide the actions taken by a group or community.
- Empowerment evaluation, which aims to assess the effort's worth while improving the community's desire and ability to take care of its own problems.
These and other types of research actively involve community members in designing and conducting the evaluation. They all have two primary goals: understanding what is going on, and empowering communities to take care of themselves. What is different between these methods is the various balances they strike between these two ends.
In this section, we'll look at models, methods, and applications of community evaluation in understanding and improving comprehensive community initiatives. We'll start with a look at some of the reasons why community groups should evaluate their efforts. Then, we'll describe some of the major challenges to evaluation. We'll also describe a model of community initiatives as catalysts for change. Then, we'll discuss some principles, assumptions, and values that guide community evaluation and outline a "logic model" for our KU Center for Community Health and Development's system of evaluation. We'll also make some specific recommendations to practitioners and policymakers about how these issues can be addressed. Finally, we'll end with a discussion examining some of the broad issues and opportunities in community evaluation.
WHY EVALUATE?
There are many good reasons for a community group to evaluate its efforts. When done properly, evaluation can improve efforts to promote health and development at any level -- from a small local nonprofit group to a statewide or even national effort. Evaluation offers the following advantages for groups of almost any size:
- Collecting information about how things are done and the results help us understand how community initiatives develop, offering lessons other groups can profit from.
- Providing ongoing feedback can improve community work by encouraging continuous adjustments of programs, policies, and other interventions.
- By involving community members, people who haven't had a voice may gain the opportunity to better understand and improve local efforts.
- Finally, evaluation can help hold groups accountable to the community and to the grantmakers who provide funding. It can also help hold grantmakers accountable to the communities that they serve.
CHALLENGES TO COMMUNITY PARTICIPATORY EVALUATION
Although there are a lot of advantages to evaluating community efforts, that doesn't mean it's an easy thing to do. There are some serious challenges that make it difficult to do a meaningful evaluation of community work. They are:
- The causes of a lot of community problems, such as substance abuse or violence, aren't very well understood. Without this better understanding of the causes, it's hard to decide what needs to be done and if the work has been successful.
- Some of the more important things to evaluate, such as the ability of the group to successfully accomplish its goals or the quality of life of community members, can be very difficult to measure.
- Community initiatives are very complex. They include doing a lot of things on many levels with a lot of different people. Because of this, it is daunting to describe what's been done thoroughly enough for another community to try to do the same thing.
- For some community issues, such as child abuse or domestic violence, researchers haven't yet come up with valid ways to determine if efforts are working. Because of this, it's difficult to evaluate the effectiveness of efforts.
- For many issues, it takes a long time to move the bottom line. For example, if a group is trying to reduce HIV/AIDS in the community, they won't know if they have really affected the number of people who contract HIV for years and years. Therefore, it's necessary to identify markers along the path -- measures of intermediate outcome, such as changes in the community or system, which give community members an idea of whether or not they are going in the right direction.
- It's very hard to estimate how strong a community-driven intervention is -- will it make a large impact, or just a ripple? Evaluators will need to collect precise information on what happened, who it happened to, and for how long the intervention occurred.
- Information collected on individuals can't always be generalized to come to a conclusion about the community as a whole.
- Because there aren't always suitable experimental designs or fitting comparisons (for example, it's hard to say that two towns are exactly alike), it is not always possible to say that the results were really because of the community initiative, and not because of something else that was going on.
- Because community initiatives change with time and circumstances, what they do gets modified as well. Since they are so malleable, it can be difficult to assess the generality of effects, and decide if a given program is good in general or just worked in one particular circumstance.
- People see things differently. Evaluators, especially those in the field of participatory evaluation, must guard against potential confusion resulting from conflicting ways of looking at things when interviewing different people about the same event.
- To evaluate a community initiative fully and well takes a lot of time and work. For an already overburdened organization, it may not be feasible to do all of this properly.
- As we discussed above, there are two primary goals of evaluation: understanding what is going on and empowering communities to take care of themselves. It can be very difficult to try and attain both of these goals at the same time. Often, one seems to need to give.
Despite the challenges that evaluation poses, our belief is that it is a very worthwhile pursuit. In order to minimize these challenges, the KU Center for Community Health and Development has developed a model and some principles that may provide guidance for people trying to evaluate the work done in their community.
A MODEL: COMMUNITY INITIATIVES AS CATALYSTS FOR CHANGE
Access Checklist, Examples, Tools, and Powerpoint.