Evaluate - Western Pacific Community Engagement Toolkit

Group of Vanuatu looking at a book. — Credit: WHO/ Y. Shimizu

Why Evaluate?

Monitoring and evaluating progress throughout community engagement efforts is essential for accountability and making adjustments along the way. This involves monitoring the process, activities, and accomplishments along the way. It also involves assessing the impact and merit of the effort, and using the information for adjustment and improvement.

Key Questions to Consider

What would success look like for the community and other stakeholders, including those most affected?
What is the framework (or logic model) by which our action will lead to changes in conditions and improvement in short- and longer-term outcomes (see Plan section)?

How will we monitor and assess participation throughout the process? How have our actions strengthened community assets and capacity to improve conditions and outcomes?

What questions do we hope to answer with the evaluation? This includes process, outcomes, and impact of the community engagement efforts.

What methods will we use to answer the evaluation questions, and to monitor and evaluate the effort?

How will we use an equity lens to see whether the effects of the effort benefit all people, including hard to reach or vulnerable groups?
How will we make sense of the data? How will community and other stakeholders, including those most affected, be involved in interpreting the data and what it means?

How will we support or create mechanisms to enable people, including those most affected and those who can address the issue, to be involved in monitoring and evaluation?
How will we use the evaluation information for celebration, improvement, accountability, sustainability, and advocacy?

Some Recommended Actions

__ Review the framework for action or logic model previously developed during the planning phase (see Plan section). It should include information about:
- - Purpose or mission (i.e. the issue or goal that the effort is addressing)
  - Objectives (i.e. measurable statements about what you plan to accomplish)
  - Context or conditions (i.e. the situation in which the effort will take place; factors that may affect outcomes)
- - - Factors may include barriers such as history of conflict, environmental factors, economic conditions, etc.
  - Inputs (i.e. the resources necessary to implement the action)
    - Resources may include time, talent, equipment, information, money, etc.
  - Strategies or interventions (i.e. what the initiative will do to effect change and improvement)
    - Interventions may include providing information and enhancing skills; enhancing services and support; modifying access, barriers and opportunities; changing the consequences; modifying policies and broader systems
  - Outputs (i.e. direct evidence of having performed the activities)
    - Outputs may include targeted actions to those most affected or number of services provided
  - Intended effects or outcomes (i.e. the changes to be made in the short, medium and long term)
    - Short-term and intermediate term (e.g. community and systems changes, changes in behavior)
    - Longer-term (e.g. improvement in population-level outcomes and reduced inequities)
1. __ Decide how the community and other stakeholders (including those most affected) will be involved in identifying and monitoring indicators of success, documenting evidence of progress, and sense making about the overall initiative and how it can be improved.
1. __ Outline an evaluation plan with evaluation questions, methods, and indicators.
  - - - Define evaluation questions
        
        What information about the effort is important to stakeholders? How well has the program met its stated objectives? How much/what kind of difference has the program or initiative made in the community as a whole? for those groups experiencing poorer outcomes?
      - Define methods
2. - - - What methods will be used to evaluate the effects of the program or initiative? Consider the level of participation you want to foster in evaluation methods. This may include surveys, interviews with key participants, goal attainment reports, archival records, observations, self-reporting, case studies and experiments, documentation system and/or monitoring and evaluation system.
3. - - - Define indicators of success
4. - - - How you will judge the success of the community engagement effort? Match the indicators to the evaluation questions. Potential indicators may be related to the community engagement process itself (e.g. building trust) or the purpose of the effort (e.g. behaviour change). This may include:
        
        Process measures
        
        Program outputs
        
        Participation rates
        
        Levels of satisfaction
        
        Changes in behavior
        
        Community or system changes
        
        Improvements in indicators of outcomes and inequities
1. __ Implement the evaluation plan. This includes:
  - - Gathering or reviewing existing data on the goal, objections, and outcomes
    - Tracking and documenting implementation of the action plan
    - Monitoring the indicators of success
    - Assessing ongoing changes in specific objectives (e.g. changes in behavior)
    - Assessing ongoing changes in specific outcomes (e.g. indicators, inequities)
    - Considering the ethical implications of the initiative
2. __ Make sense of the data and justify conclusions.
  - - Sensemaking and interpretation – Engage those responsible and those most affected in making sense of the data by asking what we are seeing, why and what it means. How will we use the information to help answer the evaluation questions?
    - Judgments – Make statements of worth or merit of the community engagement effort. How will we communicate what the findings suggest about the value added by the effort?
    - Recommendations – Identify recommendations based on the results of the evaluation.
3. __ Indicate how you intend to use the information from the evaluation, including to:
  - - Celebrate accomplishments
    - Adjust strategies and interventions (e.g. guide improvements, resource allocation)
    - Be accountable to funders, the community, and other stakeholders
    - Communicate lessons learned to stakeholders, decision-makers and wider audiences in other communities and countries
    - Advocate for sustained support of the efforts (e.g. institutionalize effective programs and policies)
    - Advocate for further practice, program and/or policy changes needed