As evaluators, we consider numerous ethical issues, especially when we evaluate programs that have a direct impact on public health and community wellbeing. Of course, evaluators and researchers must abide by organizational ethical standards, institutional review boards, and codes of ethics. But there are also more nuanced considerations that arise, which are inherent to the practice of evaluation and often bring up difficult decisions and tradeoffs. In this blog post, I outline several of these key issues, and suggest strategies for evaluators to consider when making these decisions.
1. What does program “success” mean?
Evaluators are often considered “objective” third parties, who are able to determine program success as external observers. However, evaluators may face challenges in their efforts to balance competing priorities in defining what program “success” means.
Different stakeholders such as clients, funders, program leaders and participants may have different concepts of how program success is defined. For example, funders may be most interested in measurable quantitative outcomes of the program like higher test scores, while program leaders or staff may be more interested in process evaluation to understand how the program was implemented, and program participants may be interested in participant experiences as a measure of success.
Each of these stakeholders have varying degrees of power, and evaluators may struggle to balance varying interests in an ethical manner. For example, evaluators may prioritize (or feel pressure to prioritize) the definition of success important to funders or clients, as they have the greatest amount of power and resources, with less attention or inclusion of other stakeholders such as program staff, participants, or affected communities. Evaluators must acknowledge that determining the metrics of “success” itself involves ethical decision-making.
Strategies to consider:
- Be transparent about how metrics for program success were chosen
- Utilize participatory evaluation methods, to ensure that those impacted by the program evaluation are meaningfully engaged and able to have a voice in determining metrics for success
- Use equitable evaluation as a framework to consider the values and biases that determine what is considered “valuable evidence”, and to move towards metrics of success that are oriented towards equity
2. Potential harm of evaluation activities
Evaluators work from a position of power, and as such, have the potential to cause harm to programs. Challenges or issues within programs unearthed through evaluation can negatively impact future funding decisions and threaten program continuation or the reputation of the organization. Due to this power dynamic, program leadership, staff, or other stakeholders may be (rightfully) distrustful of evaluators.
Evaluators may find themselves struggling with how to report program challenges and other potentially damaging findings, as evaluation decisions have the potential to cause harm to programs and participants, but evaluators also have a responsibility to report both program successes and challenges to determine efficacy.
Strategies to consider:
- Acknowledge evaluators’ position of power, and the impact of evaluators’ choices on programs and staff
- Use appreciative inquiry, which is a strengths-based rather than deficit-based evaluation approach
- Utilize evaluation findings to encourage the distribution of further resources and support to organizations to address program challenges
3. Impact of evaluator identity and values
Lastly, evaluators may struggle with the impact of their own values, morals, worldviews, and identities on the practice of program evaluation. External evaluators often present themselves as “neutral” individuals, but they always bring their own frameworks, beliefs, biases, and lived experiences to their evaluation work. Some evaluators may be hesitant to acknowledge this reality, which can contribute to the perpetuation of existing power structures and dynamics. Without an examination of their own perceptions of “truth” and “value,” evaluators may act on their own biases, and reinforce oppressive structures, frameworks, and beliefs in their evaluation choices, which have the potential to cause harm to programs, participants, and communities.
Strategies to consider:
- Use reflexivity, which involves a process of self-reflection about one’s own perspectives and beliefs. As discussed in this AEA blog post, reflexivity “requires us to ask ourselves how our role, power, and intersectional positionality influences the evaluation, and how ALL our stakeholders’ various positionality/ies and powers impact decision-making”
- Disclose your perspective as an evaluator, including your values and identity and their impact, as part of the evaluation findings
These examples represent just a few of the ethical questions that may arise during the practice of program evaluation. The challenges and strategies outlined in this blog post provide an opportunity for evaluators to reflect on their ethical responsibilities, and the impact of their decisions on programs, policies, and communities.