Evaluations during the COVID-19 Pandemic

Adapting evaluations during the COVID-19 pandemic

The COVID-19 pandemic has required rapid changes to the design and delivery of safer gambling programmes and policies. These changes may range from shifting service offerings online to extensive reallocation of organisational priorities, including staff time and programme funding. Despite the considerable state of flux, evaluation is still critical to informing programme and policy effectiveness and highlighting opportunities for continuous improvement.¹

Changing approaches

Performing an evaluation during a pandemic requires a different lens than is used in traditional evaluations where programmes, policies, and the contexts in which they operate are more stable. Programmes or policies will likely need to change in real-time to respond to the rapidly changing and uncertain situation. This could mean that programme plans evolve or are entirely abandoned for alternatives. In these situations, evaluations should adopt a real time feedback approach to quickly inform your team of what is working, why, for whom, and under what circumstances.²

Real time evaluations

Real time evaluations can be used during the pandemic for the purposes below. Note: these purposes are not mutually exclusive.

1. Adapting planned or ideal evaluation methodologies based on what is realistic given changes in priorities or resources.

When priorities are in flux or resources are impacted, consider reallocating resources or simplifying the evaluation methods. Methods can be simplified by focusing on purposive sampling and qualitative approaches that can provide rapid feedback, such as interviews, focus groups, or direct observations.³

Ask yourself how the pandemic has affected the programme or policy being evaluated and what changes can be made to the evaluation process to best inform the activities. Additionally, consider how the pandemic has affected ongoing evaluation efforts and what changes can be made to ensure both rapid and high-quality findings.

2. Testing and refining small changes to programmes or policies made in response to the pandemic in real time.

Real time testing should include the use of iterative feedback loops during which small changes to programmes or policies are rapidly evaluated so that lessons learned can quickly inform ongoing quality improvements. By doing so, problems will be quickly identified, alternative solutions will be considered, and small changes can be implemented and subsequently re-evaluated. Consider incorporating responsive quality improvement processes such as Plan-Do-Study-Act.⁴

Evaluation principles

Evaluating during times of rapid change requires us to adopt the following principles:^2,5

• Keep learning: Focus on how evaluation can, not can’t, be meaningfully used to gain insight, whilst focusing on facilitators rather than barriers to learning.
• Be responsive: Crises affect people differently, therefore ensuring a human rights-based, equity-focused, and gender responsive evaluation is essential.
• Emphasise process: Evaluating change rapidly or change in unusual contexts requires paying as much attention, if not more, to how or why changes occurred, not just whether they did.
• Keep it simple: Avoid complex evaluation questions or methodologies in favour of those that are streamlined and critical–shift from “nice to have” to “must have”.
• Take action: Whether evaluating rapid changes in real time or changing the approach of a planned longer-term evaluation, staying focused and committed to using the results to inform action, despite unknowns, is critical.

Additional resources

For more information on how to design or conduct evaluations in complex and emerging situations, such as during the COVID-19 pandemic, see the resources below and/or contact evaluation@greo.ca.

• Supplementary Guide: Handling Complexity in Policy Evaluation (HM Treasury’s Magenta Book 2020)
• Developmental Evaluation Primer (J.W. McConnell Family Foundation)
• Toolkit for Adapting Evaluations in the Time of COVID-19 (BetterEvaluation)
• Plan-Do-Study-Act (PDSA) worksheet (Institute for Healthcare Improvement)
• Evaluating Complex and Unfolding Interventions in Real Time (article in journal Evaluation by Tom Ling)

¹ Centers for Disease Control and Prevention Program Performance and Evaluation Office (PPEO). Introduction to Program Evaluation for Public Health Programs: A self-study guide. https://cdc.gov/eval/guide/introduction/

² United Nations Population Fund. Adapting evaluations to the COVID-19 pandemic. https://unfpa.org/admin-resource/adapting-evaluations-covid-19-pandemic

³ United Nations Office on Drug and Crime. Evaluation and COVID-19. https://unodc.org/unodc/en/evaluation/evaluation-and-covid-19.html

⁴ Riley, W., Moran, J.W., Corso, L., & Beitsch, L. M. (2010). Defining quality improvement in public health. Journal of Public Health Management and Practice, 16(1):5-7. https://researchgate.net/publication/40687297_Defining_Quality_Improvement_in_Public_Health

⁵ INTRACT (2017). Real-time evaluation. https://intrac.org/wpcms/wp-content/uploads/2017/01/Real-time-evaluation.pdf