Outcome evaluations measure the effect a programme has had on a target population by assessing the degree to which it has met desired outcomes. Outcomes are positive changes the programme intends to achieve. Examples include changes in knowledge, skills, behaviours, capacity, or health status of the target population. Outcomes are typically categorised by short-term, medium-term, and long-term goals. What these temporal categories mean depend on the programme, but short-term could refer to 1–3 years, medium-term to 2–5 years, and long-term to 5+ years.1, 2
The following stages, while presented chronologically, are not always linear. Depending on the evaluation, stages might need to be completed simultaneously or be revisited.
The first step of an outcome evaluation is to create an advisory group. This group consists of evaluators and key stakeholders who will plan for and oversee the evaluation. Advisory groups lead to better quality and useful evaluations.3, 4
Note: Advisory groups may also be called reference groups, working groups, or advisory panels.
The advisory group should clarify what is to be evaluated, develop or revisit a theory of change, discuss possible evaluation methods, and agree on the audience of the evaluation. These evaluation characteristics should be used to create a terms of reference and/or an evaluation plan.
Create an evaluation plan to outline what the evaluation is trying to do and how it will be performed. At minimum, the plan should list evaluation questions, determine what data are required, choose methods for obtaining and analysing the data, and include a schedule of dates, milestones, and responsibilities. Other information, such as a theory of change or Gantt chart, can also be included in the plan.5
Some examples of outcome evaluation questions include: 6
When selecting data collection and analysis methods, consider the following: 7
Collect and analyse the data as outlined in the evaluation plan.
Interpret the data, make a list of recommended actions, and present your findings. Results are typically presented in a final written report, but slide shows or informational videos can also be used. Visual aids, such as short videos or infographics, are great methods for communicating evaluation results in a clear and simple way. Results should be shared with all stakeholders.8
The advisory group should review the recommendations, discuss what has been learned from the evaluation, and identify next steps for applying the results. Prioritise the actions and create an action plan to ensure the results are used.8
The terms “outcome” and “impact” are sometimes used interchangeably in evaluation. However, impacts typically refer to the broad, long-term outcomes of a programme. For example, an outcome of a gambling self-exclusion programme might be to help gamblers feel more in control of their gambling behaviour, whereas an impact of that same programme might be to reduce long-term rates of harmful gambling and improve population health.9
Programme impacts can be difficult to measure and attribute to a single activity. Likewise, impact evaluations can be time consuming and difficult to perform. For these reasons, outcome evaluations are more common.
For more information on how to design and conduct outcome evaluations, visit the resources below, or contact evaluation@greo.ca to see how Greo can help.
1 Centers for Disease Control and Prevention. (n.d.). Types of Evaluation. https://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdf
2 W.K. Kellogg Foundation. (1998). Evaluation Handbook. https://www.wkkf.org/resource-directory/resources/1998/01/evaluation-handbook
3 BetterEvaluation. (2020). Advisory Group. https://www.betterevaluation.org/en/evaluation-options/advisory_group
4 Child Trends. (2008). Process evaluations: A guide for out-of-school time practitioners. https://www.childtrends.org/wp-content/uploads/2008/01/Process-Evaluation.pdf
5 BetterEvaluation. (2018). Evaluation Plan. https://www.betterevaluation.org/en/evaluation-options/evaluation_planning_templates
6 BetterEvaluation. (2016). Specify the Key Evaluation Questions (KEQs). https://www.betterevaluation.org/en/rainbow_framework/frame/specify_key_evaluation_questions
7 International Program for Development Evaluation Training. (2007). Module 8: Data Collection Methods. IPDET Handbook. https://www.measureevaluation.org/resources/training/capacity-building-resources/data-quality-portuguese/DATA_COLECTION.pdf
8 Ontario Agency for Health Protection and Promotion (Public Health Ontario). (2015). At a glance: The ten steps for conducting an evaluation. Toronto: Queen’s Printer for Ontario. https://www.publichealthontario.ca/-/media/documents/A/2015/at-a-glance-10step-evaluation.pdf
9 Public Health England. (2018). Guidance: Outcome evaluation. https://www.gov.uk/government/publications/evaluation-in-health-and-well-being-overview/outcome-evaluation