Our Model for Community Change and Improvement:
Participatory Evaluation
-
WHAT IS PARTICIPATORY EVALUATION?
-
WHY WOULD (AND WHY WOULDN'T) YOU USE PARTICIPATORY EVALUATION?
-
WHEN WOULD YOU USE PARTICIPATORY EVALUATION?
-
WHO SHOULD BE INVOLVED IN PARTICIPATORY EVALUATION?
-
HOW DO YOU CONDUCT A PARTICIPATORY EVALUATION?
Experienced community builders know that involving stakeholders - the people directly connected to and affected by their projects - in their work is tremendously important. It gives them the information they need to design, and to adjust or change, what they do to best meet the needs of the community and of the particular populations that an intervention or initiative is meant to benefit. This is particularly true in relation to evaluation.
As we have previously discussed, community-based participatory research can be employed in describing the community, assessing community issues and needs, finding and choosing best practices, and/or evaluation. We consider the topic of participatory evaluation important enough to give it a section of its own, and to show how it fits into the larger participatory research picture.
It's a good idea to build stakeholder participation into a project from the beginning. One of the best ways to choose the proper direction for your work is to involve stakeholders in identifying real community needs, and the ways in which a project will have the greatest impact. One of the best ways to find out what kinds of effects your work is having on the people it's aimed at is to include those on the receiving end of information or services or advocacy on your evaluation team.
Often, you can see most clearly what's actually happening through the eyes of those directly involved in it - participants, staff, and others who are involved in taking part in and carrying out a program, initiative, or other project. Previously, we have discussed how you can involve those people in conducting research on the community and choosing issues to address and directions to go in. This section is about how you can involve them in the whole scope of the project, including its evaluation, and how that's likely to benefit the project's final outcomes.
WHAT IS PARTICIPATORY EVALUATION?
When most people think of evaluation, they think of something that happens at the end of a project - that looks at the project after it's over and decides whether it was any good or not. Evaluation actually needs to be an integral part of any project from the beginning. Participatory evaluation involves all the stakeholders in a project - those directly affected by it or by carrying it out - in contributing to the understanding of it, and in applying that understanding to the improvement of the work.
Participatory evaluation, as we shall see, isn't simply a matter of asking stakeholders to take part. Involving everyone affected changes the whole nature of a project from something done for a group of people or a community to a partnership between the beneficiaries and the project implementers. Rather than powerless people who are acted on, beneficiaries become the copilots of a project, making sure that their real needs and those of the community are recognized and addressed. Professional evaluators, project staff, project beneficiaries or participants, and other community members all become colleagues in an effort to improve the community's quality of life.
This approach to planning and evaluation isn't possible without mutual trust and respect. These have to develop over time, but that development is made more probable by starting out with an understanding of the local culture and customs - whether you're working in a developing country or in an American urban neighborhood. Respecting individuals and the knowledge and skills they have will go a long way toward promoting long-term trust and involvement.
The other necessary aspect of any participatory process is appropriate training for everyone involved. Some stakeholders may not even be aware that project research takes place; others may have no idea how to work alongside people from different backgrounds; and still others may not know what to do with evaluation results once they have them. We'll discuss all of these issues - stakeholder involvement, establishing trust, and training - as the section progresses.
The real purpose of an evaluation is not just to find out what happened, but to use the information to make the project better In order to accomplish this, evaluation should include examining at least two areas:
- Process. The process of a project includes the planning and logistical activities needed to set up and run it. Did we do a proper assessment beforehand so we would know what the real needs were? Did we use the results of the assessment to identify and respond to those needs in the design of the project? Did we set up and run the project within the timelines and other structures that we intended? Did we involve the people we intended to? Did we have or get the resources we expected? Were staff and others trained and prepared to do the work? Did we have the community support we expected? Did we record what we did accurately and on time? Did we monitor and evaluate as we intended?
- Implementation. Project implementation is the actual work of running it. Did we do what we intended? Did we serve or affect the number of people we proposed to? Did we use the methods we set out to use? Was the level of our activity what we intended (e.g., did we provide the number of hours of service we intended to)? Did we reach the population(s) we aimed at? What exactly did we provide or do? Did we make intentional or unintentional changes, and why?
- Outcomes. The project's outcomes are its results - what actually happened as a consequence of the project's existence. Did our work have the effects we hoped for? Did it have other, unforeseen effects? Were they positive or negative (or neither)? Do we know why we got the results we did? What can we change, and how, to make our work more effective?
Many who write about participatory evaluation combine the first two of these areas into process evaluation, and add a third - impact evaluation - in addition to outcome evaluation. Impact evaluation looks at the long-term results of a project, whether the project continues, or does its work and ends.
Rural development projects in the developing world, for example, often exist simply to pass on specific skills to local people, who are expected to then both practice those skills and teach them to others. Once people have learned the skills - perhaps particular cultivation techniques, or water purification - the project ends. If in five or ten years, an impact evaluation shows that the skills the project taught are not only still being practiced, but have spread, then the project's impact was both long-term and positive.
In order for these areas to be covered properly, evaluation has to start at the very beginning of the project, with assessment and planning. In a participatory evaluation, stakeholders should be involved in:
- Naming and framing the problem or goal to be addressed
- Developing a theory of practice (process, logic model) for how to achieve success
- Identifying the questions to ask about the project and the best ways to ask them - these questions will identify what the project means to do, and therefore what should be evaluated
What's the real goal, for instance, of a program to introduce healthier foods in school lunches? It could be simply to convince children to eat more fruits, vegetables, and whole grains. It could be to get them to eat less junk food. It could be to encourage weight loss in kids who are overweight or obese. It could simply be to educate them about healthy eating, and to persuade them to be more adventurous eaters. The evaluation questions you ask both reflect and determine your goals for the program. If you don't measure weight loss, for instance, then clearly that's not what you're aiming at. If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight. Is that still better than not eating the healthy foods? You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.
- Collecting information about the project
- Making sense of that information
- Deciding what to celebrate, and what to adjust or change, based on information from the evaluation
WHY WOULD (AND WHY WOULDN'T) YOU USE PARTICIPATORY EVALUATION?
Why would you use participatory evaluation? The short answer is that it's often the most effective way to find out what you need to know, both at the beginning of and throughout the course of a project. In addition, it carries benefits for both individual participants and the community that other methods don't.
Some of the major advantages of participatory evaluation:
- It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects. If stakeholders, including project beneficiaries, are involved from the beginning in determining what needs to be evaluated and why - not to mention what the focus of the project needs to be - you're much more likely to aim your work in the right direction, to correctly determine whether your project is effective or not, and to understand how to change it to make it moreso.
- It can get you information you wouldn't get otherwise. When project direction and evaluation depend, at least in part, on information from people in the community, that information will often be more forthcoming if it's asked for by someone familiar. Community people interviewing their friends and neighbors may get information that an outside person wouldn't be offered.
- It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff. Those implementing the project and those who are directly affected by it are most capable of sorting out the effective from the ineffective.
- It can tell you why something does or doesn't work. Beneficiaries are often able to explain exactly why they didn't respond to a particular technique or approach, thus giving you a better chance to adjust it properly.
- It results in a more effective project. For the reasons just described, you're much more apt to start out in the right direction, and to know when you need to change direction if you haven't. The consequence is a project that addresses the appropriate issues in the appropriate way, and accomplishes what it sets out to do.
- It empowers stakeholders. Participatory evaluation gives those who are often not consulted - line staff and beneficiaries particularly - the chance to be full partners in determining the direction and effectiveness of a project.
- It can provide a voice for those who are often not heard. Project beneficiaries are often low-income people with relatively low levels of education, who seldom have - and often don't think they have a right to - the chance to speak for themselves. By involving them from the beginning in project evaluation, you assure that their voices are heard, and they learn that they have the ability and the right to speak for themselves.
- It teaches skills that can be used in employment and other areas of life. In addition to the development of basic skills and specific research capabilities, participatory evaluation encourages critical thinking, collaboration, problem-solving, independent action, meeting deadlines...all skills valued by employers, and useful in family life, education, civic participation, and other areas.
- It bolsters self-confidence and self-esteem in those who may have little of either. This category can include not only project beneficiaries, but also others who may, because of circumstance, have been given little reason to believe in their own competence or value to society. The opportunity to engage in a meaningful and challenging activity, and to be treated as a colleague by professionals, can make a huge difference for folks who are seldom granted respect or given a chance to prove themselves.
- It demonstrates to people ways in which they can take more control of their lives. Working with professionals and others to complete a complex task with real-world consequences can show people how they can take action to influence people and events.
- It encourages stakeholder ownership of the project. If those involved feel the project is theirs, rather than something imposed on them by others, they'll work hard both in implementing it, and in conducting a thorough and informative evaluation in order to improve it.
- It can spark creativity in everyone involved. For those who've never been involved in anything similar, a participatory evaluation can be a revelation, opening doors to a whole new way of thinking and looking at the world. To those who have taken part in evaluation before, the opportunity to exchange ideas with people who may have new ways of looking at the familiar can lead to a fresh perspective on what may have seemed to be a settled issue.
- It encourages working collaboratively. For participatory evaluation to work well, it has to be viewed by everyone involved as a collaboration, where each participant brings specific tools and skills to the effort, and everyone is valued for what she can contribute. Collaboration of this sort not only leads to many of the advantages described above, but also fosters a more collaborative spirit for the future as well, leading to other successful community projects.
- It fits into a larger participatory effort. When community assessment and the planning of a project have been a collaboration among project beneficiaries, staff, and community members, it only makes sense to include evaluation in the overall plan, and to approach it in the same way as the rest of the project. In order to conduct a good evaluation, its planning should be part of the overall planning of the project. Furthermore, participatory process generally matches well with the philosophy of community-based or grass roots groups or organizations.
With all these positive aspects, participatory evaluation carries some negative ones as well. Whether its disadvantages outweigh its advantages depend on your circumstances, but whether you decide to engage in it or not, it's important to understand what kinds of drawbacks it might have.
The significant disadvantages of participatory evaluation include:
- It takes more time than conventional process. Because there are so many people with different perspectives involved, a number of whom have never taken part in planning or evaluation before, everything takes longer than if a professional evaluator or a team familiar with evaluation simply set up and conducted everything. Decision-making involves a great deal of discussion, gathering people together may be difficult, evaluators need to be trained, etc.
- It takes the establishment of trust among all participants in the process. If you're starting something new (or, all too often, even if the project is ongoing), there are likely to be issues of class distinction, cultural differences, etc., dividing groups of stakeholders. These can lead to snags and slowdowns until they're resolved, which won't happen overnight. It will take time and a good deal of conscious effort before all stakeholders feel comfortable and confident that their needs and culture are being addressed.
- You have to make sure that everyone's involved, not just "leaders" of various groups. All too often, "participatory" means the participation of an already-existing power structure. Most leaders are actually that - people who are most concerned with the best interests of the group, and whom others trust to represent them and steer them in the direction that best reflects those interests. Sometimes, however, leaders are those who push their way to the front, and try to confirm their own importance by telling others what to do.
By involving only leaders of a population or community, you run the risk of losing - or never gaining - the confidence and perspective of the rest of the population, which may dislike and distrust a leader of the second type, or may simply see themselves shut out of the process.. They may see the participatory evaluation as a function of authority, and be uninterested in taking part in it. Working to recruit "regular" people as well as, or instead of, leaders may be an important step for the credibility of the process. But it's a lot of work and may be tough to sell.
- You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills. There are really a number of potential disadvantages here. The obvious one is that of time, which we've already raised - training takes time to prepare, time to implement, and time to sink in. Another is the question of what kind of training participants will respond to. Still another concerns recruitment - will people be willing to put in the time necessary to prepare them for the process, let alone the time for the process itself?
- You have to get buy-in and commitment from participants. Given what evaluators will have to do, they need to be committed to the process, and to feel ownership of it. You have to structure both the training and the process itself to bring about this commitment.
- People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation. Poor people everywhere live on the edge, which means they're engaged in a delicate balancing act. The least tilt to one side or the other - a sick child, too many days of rain in a row - can cause a disruption that may result in an inability to participate on a given day, or at all. If you're dealing with a rural village that's dependent on agriculture, for instance, an accident of weather can derail the whole process, either temporarily or permanently.
- You may have to be creative about how you get, record, and report information. If some of the participants in an evaluation are non- or semi-literate, or if participants speak a number of different languages (English, Spanish, and Lao, for instance), a way to record information will have to be found that everyone can understand, and that can, in turn, be understood by others outside the group.
- Funders and policy makers may not understand or believe in participatory evaluation. At worst, this can lose you your funding, or the opportunity to apply for funding. At best, you'll have to spend a good deal of time and effort convincing funders and policy makers that participatory evaluation is a good idea, and obtaining their support for your effort.
Some of these disadvantages could also be seen as advantages: the training people receive blends in with their development of new skills that can be transferred to other areas of life, for instance; coming up with creative ways to express ideas benefits everyone; once funders and policy makers are persuaded of the benefits of participatory process and participatory evaluation, they may encourage others to employ it as well. Nonetheless, all of these potential negatives eat up time, which can be crucial. If it's absolutely necessary that things happen quickly (which is true not nearly as often as most of us think it is), participatory evaluation is probably not the way to go.
WHEN MIGHT YOU USE PARTICIPATORY EVALUATION?
Access Checklist, Tools, & Powerpoint.