Voices for Health Justice: 2023 evaluation report

Type: Published Report
Date: July 2023


Voices for Health Justice: 2023 evaluation report. Ranjani Paradise, Carolyn Fisher, Amanda Robinson, Sofia Ladner, Nubia Goodwin, Benjamin Goldberg. July 2023.


Background and evaluation methods
Voices for Health Justice (Voices) is a program funded by the Robert Wood Johnson Foundation (RWJF) that supports organizations committed to health justice, racial justice, and anti-racism work. The overarching program goals are to increase access to health care, make healthcare more affordable, and increase the ability of the healthcare system to treat all people with dignity. The first grant cycle, referred to here as Voices 1.0, began in December 2020 and ran through March 2023. A second grant cycle, Voices 2.0, will begin in summer/fall 2023.

Voices 1.0 supported Community Catalyst, Community Change, and the Center on Budget and Policy Priorities, which together comprised the Voices Steering Committee (SC). This national infrastructure funded and provided technical assistance (TA) to 25 state grantees across 24 states, each of which had between zero and six subgrantees. State projects were rooted in building the power of communities facing disproportionate health inequities, including low-income communities and communities of color.

The Institute for Community Health (ICH) is the evaluation partner for the Voices program. ICH formed and engaged with an Evaluation Advisory Committee (EAC) made up of grantee and subgrantee representatives to guide the direction of the evaluation and the interpretation of findings. During an initial participatory design phase, ICH worked with the EAC and the Voices SC to develop a program theory of change and an evaluation plan. Over the course of the Voices 1.0 evaluation, we identified evaluation questions in collaboration with the EAC, Voices SC, and RWJF, and explored these questions utilizing a range of data collection approaches, including qualitative interviews, social network analysis, document review, secondary data analysis, reflections sessions with the SC, and in-depth case studies with six state teams.

In this report, we integrate all data collected over the Voices 1.0 evaluation period to answer seven evaluation questions and identify recommendations for future work. We also reflect on the Voices 1.0 evaluation process and how we plan to evolve our approach for the Voices 2.0 evaluation.

This report focuses on themes that we identified across the overall Voices 1.0 cohort as a whole. Findings from case studies of six state projects (Colorado, Georgia, Illinois, Louisiana, Massachusetts, and Washington, DC) can be found in the companion report, Implementation of Voices for Health Justice in diverse contexts: Case studies of six state projects.

How did Voices state teams understand community power building and how did this affect their approach to their work?
Though community power building has always been central to Voices, the program did not set a common definition or framework, and states had some room to interpret what community power meant to them and to design their grant programs accordingly. Voices state teams generally agreed that the goal of community power building is for the people affected by issues to have the influence and ability to bring about change that meets their needs and benefits their communities. State teams articulated that power building can be a vehicle for policy/systems change, and vice versa, and many emphasized that community power is a valued outcome in its own right. Teams described narrative change, fostering connections between people, and capacity-building as essential components to the process of building community power for health justice. There was variation across state teams in which elements of power building they prioritized in their projects, their goals, and what they accomplished over the grant period. Some were building upon a long history of work and were operating in more favorable political climates, making it easier to build power quickly or exercise built power towards specific policy advocacy efforts. Others were at earlier stages and designed their work accordingly.

Which policy, budget, and administrative outcomes did the Voices state teams contribute to?
Over the first 20 months of the Voices 1.0 grant period, state teams contributed to a number of policy, budget, and administrative wins that aligned with their objectives. Seventeen states had documented victories relevant to Voices, many of which were related to expanding Medicaid or other insurance coverage, often for undocumented people and/or prenatal or postpartum services, including doula services. Though these policy, budget, and administrative outcomes were all related to Voices goals, some of them were the result of many years of community organizing, advocacy, and legislator education, much of which predated the Voices funding. The power building, organizing, and advocacy of Voices state teams during the life of the grant is meaningful, but is just one factor within a complex set of contributions.

How did the design, structure, and implementation of Voices affect state teams’ ability to do community power building and policy advocacy?
There were five areas in which the design and implementation of Voices 1.0 had a significant impact on the state projects:

  1. The team-based grant structure: Most of the Voices state grants were multiorganizational partnerships bringing together organizations with complementary skills and expertise. State grantees and subgrantees felt that there was value in this structure and that it made their work more effective. However, some teams were more well-aligned than others, and some had challenges coming to agreement about the project focus or identifying the best role for each partner.
  2. Grant size and timescale: A theme that came up consistently throughout the evaluation process was that it was challenging for state teams to accomplish everything they wanted to accomplish within the timeline and budget parameters of the grant. Building community power to improve healthcare access, affordability, and dignity is long-term work, and state teams felt that the grant did not allow enough time for all of the steps required to reach this goal. Some teams also felt that the amount of funding was insufficient, particularly after the grant was divided among partner organizations.
  3. TA provision: Many state teams found the TA offered through Voices to be helpful for their work, particularly when TA providers helped support strategic planning, problem solving, idea development, brainstorming, and resource sharing. However, there were some challenges related to power dynamics and relationship-building between teams and TA providers, misalignments between TA providers’ expertise and teams’ needs, lack of understanding about how best to use TA, turnover among TA providers, and organizational capacity to engage with various TA offerings.
  4. Reporting structures: The Voices Steering Committee did not require any reporting from state teams after the first year of the grant. This helped reduce burden on teams, but also limited their opportunities to present their progress and highlight what they saw as their main accomplishments over the second half of the grant.
  5. Communication around success criteria and grant renewal for Voices 2.0: State teams had received feedback and guidance from their TA providers over the course of the Voices 1.0 grant about their project plans, success and challenges, and how to move forward effectively. However, because of the renewal approval timeline, teams were largely unaware for most of the 1.0 grant period that their progress in specific areas could impact their opportunity to receive continued funding. Some state teams expressed that they felt there was a lack of clarity from the TA providers about the Steering Committee’s expectations for the projects, with one particular area of confusion being whether they should prioritize grassroots power building or policy outcomes.

How did state teams develop over the course of the Voices 1.0 period?
By funding multi-organizational partnerships to implement state projects, Voices sought to foster collaborative relationships among organizations working within each state, theorizing that if successful, the team would develop into a whole that was greater than the sum of its parts, and would be more effective in making progress towards Voices goals. Social network analysis (SNA) showed that there was notable growth in the density of the state team networks from 2020 to 2023, demonstrating that organizational relationships within each state were built over the Voices 1.0 period. In addition, there was growth in the strength of the relationships between organizations on Voices state teams over the grant period, measured by the Collaboration Index on the SNA survey. Many state teams highlighted their partnership development as one of their biggest achievements under the Voices grant, describing their collaborations as “strong” and “invaluable”. Teams were able to build new relationships, deepen existing relationships, have meaningful and open conversations with partners, and build trust and respect. They viewed their relationships as a sustainable element of the work that would continue beyond the funding period, and many organizations hoped to continue working with their partner organizations in the future outside of Voices.

What factors facilitated or hindered successful collaboration within state teams?
There were several factors that impacted the development of strong collaborations within the state teams. Facilitators to effective collaboration included having shared values, goals, and understanding of issues; communicating regularly and transparently; devoting time to resolving challenges and aligning priorities; having previous experience working together; having partner organizations that could complement each other’s work; and identifying a clear role for each partner. Some teams that had not worked together previously found it difficult to get their collaborations off the ground, as they had to devote more time to partnership development work before moving forward on other project activities. In addition, there were sometimes challenges related to organizational capacity, typically around the amount of time and attention specific organizations could devote to Voices. When some organizations were not able to engage fully in the partnership or in the project work, this made it harder for teams to work together effectively.

How did state-level political context affect teams’ ability to do community power building and policy advocacy?
Many state teams experienced challenges related to their state’s political climate, and one clear theme from the evaluation was that multiple factors made power building and advocacy particularly challenging in politically conservative states. In these states, legislatures had greater opposition to the Voices advocacy goals, community power was weakened due to redistricting or gerrymandering, and teams had to change strategies in response to harmful conservative rhetoric and budgetary challenges. In addition, even when favorable legislation is passed, there can be challenges securing funding or implementing policies effectively. All these challenges ultimately impact community power building, as community members can become disenchanted by politics when they don’t see positive changes, which leads them to disengage from organizing efforts. The evaluation also suggests that political context may affect partnership-building, and that it may have been more difficult for projects in conservative states to succeed with their Voices 1.0 grants, achieve policy wins, and meet the criteria for renewal.

What other contextual factors impacted the work? How did these factors affect teams’ ability to do community power building and policy advocacy?
Voices for Health Justice unfolded during a particularly tumultuous time in history, and there were a number of contextual factors that affected the state projects. These included COVID-19 and its impacts on communities of color, the murder of George Floyd and increased national attention on structural racism, economic stimulus measures, national disasters, and a widespread pattern of high worker turnover. In some cases, these external factors forced organizations to focus on urgent community needs rather than longer-term Voices goals and/or reduced their capacity for grant-related work during certain periods of time.

Based on the findings of the Voices 1.0 evaluation, we offer the following four recommendations for the implementation of Voices 2.0 and/or future programs seeking to promote community power building for health justice:

  1. Dedicate time up front to developing a common understanding of key concepts and definitions of success
  2. Support development of specific power ecosystem elements within state teams
  3. Offer more customized TA
  4. Structure funding to align with the long-term nature of community power building and to meet the needs of projects in different political environments

Reflections on evaluation
In reflecting on the Voices 1.0 evaluation, we identified a number of successes and challenges with the methods we used and our application of the principles of equitable evaluation. We see the engagement with the EAC as a key success that strengthened our evaluation, and we plan to continue working closely with this group for the Voices 2.0 evaluation. We also plan to refine our data collection approaches, including doing more targeted sampling for interviews and scaling back our dependence on secondary data. Finally, a core goal for the Voices 2.0 evaluation is to
develop a rubric to use for evaluating community power building progress, grounded in the Barsoum and Farrow power ecosystem framework. We expect that developing and applying this rubric will help us to capture progress in a nuanced way and understand the different ways in which state power ecosystems develop through Voices for Health Justice 2.0.

View Publication