The COVID-19 pandemic has required rapid changes to the design and delivery of safer gambling programmes and policies. These changes may range from shifting service offerings online to extensive reallocation of organisational priorities, including staff time and programme funding. Despite the considerable state of flux, evaluation is still critical to informing programme and policy effectiveness and highlighting opportunities for continuous improvement.1
Performing an evaluation during a pandemic requires a different lens than is used in traditional evaluations where programmes, policies, and the contexts in which they operate are more stable. Programmes or policies will likely need to change in real-time to respond to the rapidly changing and uncertain situation. This could mean that programme plans evolve or are entirely abandoned for alternatives. In these situations, evaluations should adopt a real time feedback approach to quickly inform your team of what is working, why, for whom, and under what circumstances.2
Real time evaluations can be used during the pandemic for the purposes below. Note: these purposes are not mutually exclusive.
1. Adapting planned or ideal evaluation methodologies based on what is realistic given changes in priorities or resources.
When priorities are in flux or resources are impacted, consider reallocating resources or simplifying the evaluation methods. Methods can be simplified by focusing on purposive sampling and qualitative approaches that can provide rapid feedback, such as interviews, focus groups, or direct observations.3
Ask yourself how the pandemic has affected the programme or policy being evaluated and what changes can be made to the evaluation process to best inform the activities. Additionally, consider how the pandemic has affected ongoing evaluation efforts and what changes can be made to ensure both rapid and high-quality findings.
2. Testing and refining small changes to programmes or policies made in response to the pandemic in real time.
Real time testing should include the use of iterative feedback loops during which small changes to programmes or policies are rapidly evaluated so that lessons learned can quickly inform ongoing quality improvements. By doing so, problems will be quickly identified, alternative solutions will be considered, and small changes can be implemented and subsequently re-evaluated. Consider incorporating responsive quality improvement processes such as Plan-Do-Study-Act.4
Evaluating during times of rapid change requires us to adopt the following principles:2,5
For more information on how to design or conduct evaluations in complex and emerging situations, such as during the COVID-19 pandemic, see the resources below and/or contact evaluation@greo.ca.
1 Centers for Disease Control and Prevention Program Performance and Evaluation Office (PPEO). Introduction to Program Evaluation for Public Health Programs: A self-study guide. https://cdc.gov/eval/guide/introduction/
2 United Nations Population Fund. Adapting evaluations to the COVID-19 pandemic. https://unfpa.org/admin-resource/adapting-evaluations-covid-19-pandemic
3 United Nations Office on Drug and Crime. Evaluation and COVID-19. https://unodc.org/unodc/en/evaluation/evaluation-and-covid-19.html
4 Riley, W., Moran, J.W., Corso, L., & Beitsch, L. M. (2010). Defining quality improvement in public health. Journal of Public Health Management and Practice, 16(1):5-7. https://researchgate.net/publication/40687297_Defining_Quality_Improvement_in_Public_Health
5 INTRACT (2017). Real-time evaluation. https://intrac.org/wpcms/wp-content/uploads/2017/01/Real-time-evaluation.pdf