Voices for Health Justice: 2022 evaluation report

Type: Published Report
Date: July 2022

Citation

Voices for Health Justice: 2022 evaluation report. Ranjani Paradise, Carolyn Fisher, Sofia Ladner, Amanda Robinson, Nubia GoodwinBenjamin Goldberg, Sean Merritt, Kevalyn Bharadwaj. July 2022.

Abstract

Voices for Health Justice (Voices) is a program funded in part by the Robert Wood Johnson Foundation (RWJF) that provides grants and other support to organizations committed to health justice, racial justice, and anti-racism work. The overarching program goals are to increase access to health care, make healthcare more affordable, and increase the ability of the healthcare system to treat all people with dignity. With the core Voices program, RWJF is supporting Community Catalyst, Community Change, and the Center on Budget and Policy Priorities (together comprising the Voices Steering Committee), as well as the Altarum Healthcare Value Hub through an adjacent grant. This national infrastructure funds state 25 grantees across 24 states, each of which has between zero and six subgrantees. The grantee funding began in December 2020 and runs through March 2023. Voices supports projects that are rooted in building the power of communities facing disproportionate health inequities, including low-income communities and communities of color (Black, Indigenous, Hispanic, Latino/a/e/X, Arab/Arab American, Southeast Asian, Asian, Asian Pacific Islander, Native Hawaiian, Desi and/or immigrant communities). The support consists of both funding and the provision of technical assistance (TA) and connections with other programs. The program is also running a communications strategy called the National Wave which seeks to build momentum for initiatives and priorities held in common across the various projects. Finally, the program has funded additional states and organizations through strategic Rapid Response grants (see Appendix A for more details on Rapid Response grants).

The Institute for Community Health (ICH) is the evaluation partner for the Voices program. Our evaluation is guided by equitable evaluation principles and participatory approaches. Our evaluation plan identifies six domains of inquiry along with a cross-cutting focus on structural racism: 1) deep and broad community engagement, 2) power ecosystems, 3) sustained capacity growth, 4) narrative change, 5) policy, budget, and administrative outcomes, and 6) community power. To explore evaluation questions within each of the domains, we use a range of data collection approaches, including group interviews with funded state project teams, a longitudinal social network analysis (SNA) survey, reflection sessions with the Voices Steering Committee, review of secondary data, and in-depth case studies with six project teams. We are partnering with an Evaluation Advisory Committee (EAC) made up of grantee and subgrantee representatives to guide the direction of the evaluation and the interpretation of findings, and we have also engaged with the Voices Steering Committee to provide input into our work.

This report describes our findings from the time period that corresponds roughly with months 7-19 out of 28 months of grantees’ work. This is still relatively early in the timeline of our planned data collection, and the objective of this report is to review the learnings we have identified from various data sources so far and begin to draw together the major themes that unite them. This should be considered a preliminary report that can be used to facilitate ongoing productive discussions among stakeholders regarding current and future programming.

In this report, we first review updates on Voices activities that are coordinated by the Steering Committee and TA partners: TA provision, the National Wave, and the Rapid Response grants. Next, we discuss evaluation process activities, including our work with the Evaluation Advisory Committee (EAC), the reflection sessions with the Steering Committee, and the process we used to select case study states. After that, we discuss our primary data collection results to date: the group interviews with each state team and the first administration of the Social Network Analysis. Next, we discuss our findings from secondary data analysis of grantee annual reports and of states’ reported policy and administrative wins. We conclude by discussing uniting themes across our data collection methods.

Update on Voices for Health Justice activities

Technical assistance: As part of the program, grantees and sub-grantees receive individual project TA from a small team of people from the Steering Committee organizations and the TA provider Altarum Healthcare Value Hub, as well as McCabe Message Partners. Small group TA was added in response to grantee requests on initial capacity assessments. Small groups TA is available for states working on similar issues, and cohort-wide TA opportunities are available to all projects.

TA providers give assistance in several major areas based on Community Catalyst’s six capacities for effective advocacy: campaign development and execution, grassroots organizing, policy analysis and advocacy; communications, coalition and stakeholder alliance, resource development, and organizational development. We found that campaign, communications, and coalition and stakeholder alliances were the most in-demand TA areas during the early quarters and steadily less utilized during later quarters, while demand for other topics was more consistent over the timeframe.

Successful practices identified by TA providers included offering grantees a menu of TA options including specific skills and areas of expertise they could focus on, sharing resources available from the Steering Committee national organizations, and taking the role of a partner in the team rather than that of an outside expert. In addition, grantees noted that they found it most useful when TA providers were flexible and focused on problem solving, idea development, and brainstorming during meetings. Grantee teams also shared that TA brought added capacity, policy expertise, and relevant knowledge to support their work, in addition to opportunities to build coalition relationships both within and between states.

National Wave: The National Wave supports coordinated media campaigns that strategically elevate the work of state partners. National Wave topics are identified by members of the Steering Committee and TA providers, and campaigns consist of multiple media centered activities supported by the Voices communications manager. The first ‘mini’ or pilot wave occurred in March-May 2021, and focused on equitable COVID-19 vaccine distribution, where participating state partners were provided support and tools (including media materials o draft OpEds) to help build a communitycentered narrative to illustrate gaps in vaccine access and actionable policy solutions. The current wave focuses on campaigns to support Cover All Kids state legislation.

Learnings from the pilot National Wave identified the following recommendations: focus on state and local initiatives instead of national policies; build in enough time to help each state team modify and align their efforts; be highly responsive to states’ needs; and ensure the National Wave furthers existing work of the state partners. After the pilot wave, these recommendations were integrated into the approach for the current National Wave.

Rapid Response funds: The Steering Committee also has a reserve of Rapid Response funds to support strategic and timely work that advances Voices goals of increasing health care access, affordability, and dignity. The work supported by these funds is aligned with but not central to the efforts of the Voices state grantees, and we report on the progress with Rapid Response funds in Appendix A.

Evaluation process activities

Evaluation Advisory Committee learnings: Throughout the evaluation, we have been advised by the Evaluation Advisory Committee (EAC), a group of people representing a wide variety of grantee and subgrantee perspectives. The purpose of the EAC is not only to gather ideas and input from people that are closest to the work of the program, but also to create a space that promotes reflection and evaluative thinking, to build relationships and community, as well as to promote networking among the grantees. We strongly feel that the Voices evaluation is stronger for incorporating grantee and subgrantee perspectives from the beginning and throughout the evaluation process. The EAC has guided us to understand how grantee teams were conceptualizing the program and community power overall, and has advised us on identifying evaluation activities that will be effective in helping us gain understanding of grantees’ work.

Learnings about implementing an evaluation advisory committee have include the following: it is important to begin by building authentic, genuine relationships before committing to the work; to maximize engagement, information should be kept focused, relevant, and straightforward; to promote fruitful discussions, meetings should balance structure and flexibility; it is important to simultaneously recognize the value of all members’ contributions AND the unavoidable power dynamics in a mixed group; people of color might be more hesitant to demonstrate vulnerability by sharing early or emerging ideas; to foster comfort, it is important to provide multiple channels to contribute to the discussion including real-time, offline, written, and spoken channels.

Steering Committee reflection sessions: Through our partnership with the Steering Committee, we have facilitated three reflection sessions to promote reflection and evaluative thinking and to gather data and input into the evaluation approach. The first reflection session, in September 2021, focused on identifying criteria to use for selecting case study projects. The second session, in January 2022, focused on how the Steering Committee has worked to center race and anti-racism in the Voices program design, grantee selection, and ongoing implementation. The third session, in May 2022, focused on reactions to the findings from the qualitative interviews with the state project teams, with emphasis on how those findings could inform future programs. In all of these sessions, we have been able to observe Steering Committee members reflecting on past progress, absorbing lessons, and proactively deciding upon improvements. We integrate learnings from these sessions throughout the report.

Case study selection: A central part of the Voices evaluation is conducting in-depth case studies with a sample of state project teams. Case studies data will be gathered through in-depth engagements with six individual state teams. We used a systematic process to select a diverse set of projects, taking into account a number of factors, including geography, partisan lean, intra-coalition relationships, and the state teams’ willingness and capacity to participate in the case study process. The six projects we selected are Colorado, Georgia, Illinois, Massachusetts, Louisiana, and Washington, DC. Because each project is taking a different approach and is at a different point in the process, each case study will be customized and will focus on the specific lessons we can learn from that project. Compared to the overall cohort of Voices states, the case study states are more liberal (the states who declined to participate were all more conservative), have the same number of sub-grantees on average, and have policy and legislative aims that are resonant across all Voices state teams. Case studies are only just beginning, and case study findings will be available towards the end of the grantees’ funding period.

Primary data collection results

Group interviews: We conducted semi-structured qualitative group interviews with state project teams in the fall of 2021. A brief summary of what we learned from the interviews is provided in this report, and more details can be found in the full report on interview findings.

When asked about the power ecosystems within their states, interview participants reflected on their coalitions’ challenges and facilitators and many described significant benefits and value added by working in collaboration with one another, including thought partnership, increased credibility with various communities, and emotional support. Challenges in forming these beneficial collaborations included coming to agreement when organizations have different political objectives, capacities, and understandings of racial and health justice issues across different communities; and working virtually due to the COVID-19 pandemic. Interview participants agreed with the Steering Committee that a history of prior collaboration, complementary skill sets and capacities, and shared values made the work easier. Participants also suggested that good practices to support strong partnerships include transparent and regular communication and role clarity. Finally, some teams commented that the flexibility of the Voices program fostered stronger relationships.

State project teams were also asked to discuss their approaches to building community power. Teams shared their approaches to organizing and leadership development as components of powerbuilding, highlighting that they are focusing on the most marginalized populations, including lowincome communities and communities of color. Many state project teams were in early phases of their organizing work, and described their efforts around building relationships, earning trust, and learning from communities. Teams highlighted that they were hoping to build a broad base of support and bring together different communities of color to further the work. Interviewees also described their approaches to leadership development, including informal ways to identify and encourage leaders as well as more concrete leadership training programs and curricula.

When asked about the anti-racism focus of their work, various grantees and sub-grantees expressed that a racial justice or anti-racism lens is core to everything they do, although this was operationalized in different ways among the Voices organizations. Interview participants described activities such as internal trainings on racial justice topics, hiring efforts to diversify staff, looking at data disaggregated by race, centering race in communications, and re-orienting organizing work to have a racial justice focus by more intentionally following the community’s lead in identifying priorities. Some interviewees also shared that the Voices partnerships have helped them further their racial justice priorities.

A salient theme that cut across interview topics was the dynamic and long-term nature of this work, with teams highlighting multiple stages of work involved in the path to long-term change. A critical step involves building long-term relationships and trust within coalitions, as well as relationships with the communities of focus. Policy goals and associated plans identified in the grant proposal phase may need to be reoriented as the work proceeds, necessitating a responsive and iterative approach.

Social network analysis findings: The first question we explored with the data from the social network analysis (SNA) survey was whether organizations of color were substantively leading the work, using centrality within the state network as an indicator. The average centrality of organizations of color was not statistically significantly higher than that of organizations not meeting the definition of organizations of color. However, both the average centrality and the range of the variation of centrality are higher for organizations of color, suggesting that there may be a meaningful difference even in the absence of statistical significance. Trends in these numbers in future iterations of the survey will be useful in understanding how meaningful these differences are. The second question we explored was whether state networks were more closely connected after one year of work than before the Voices project started; we calculated this through examining the density of the network before the Voices project (2020) compared to at the end of the first year of the grant (2021). We found that the networks were significantly denser in 2021 than in 2020.

Secondary data analysis

Policy wins and related achievements: In our analysis of policy wins compiled by Community Catalyst, it was notable that there were fewer outcomes reported in the second 9-month period (September 2021 – July 2022) than in the first 9-month period (December 2020 – August 2021), and also that the states reporting policy wins were substantially more politically liberal than the overall set of Voices states. In addition, it is important to note that there are many factors that contribute to a policy win; specifically, although the policy outcomes compiled here are all related to Voices goals, some of them are the result of many years of community organizing, advocacy, and legislator education on related issues, much of which predated the Voices funding.

Learnings from grantee annual reports: Grantees are asked to periodically report on their activities to the Steering Committee. In these reports, grantees share their reflections on accomplishments, challenges, goals, and experience with Voices TA. In reflecting back on what they have accomplished in their first year, many grantees highlighted the successes they have had in getting their Voices projects started, as well as their thoughts on successful organizing strategies and techniques for messaging to their base. Many grantees wrote about the importance of clear and understandable communication, and recommended tying concrete realities and experience to more abstract concepts around health inequity. Some noted that outreach should tailored to community preferences about communication formats and should be accessible to people who speak different languages. Grantees also shared that direct engagement through one-on-ones is most effective for bringing people into active roles, but that other forms of outreach (e.g., texts, emails, virtual events) can be effective for sharing information and raising awareness.

Grantees were also asked about what direct impacts on decision-makers they have been able to achieve through their organizing. A significant number of programs reported that they have been successful in connecting members of their communities with lawmakers to share their stories, which they feel impacted lawmakers’ decisions.

Grantees were also asked to reflect on challenges they encountered. Almost all grantees discussed the ongoing external challenge of COVID-19 and the way it has impacted and will continue to impact their work. Grantees also emphasized that the state and national political climate is a significant external factor affecting their ability to reach goals. This includes projects in conservative political settings as well as some in more liberal states, who noted that in election years, legislators tend to be less willing to take policy action that may be controversial. Grantees also experienced challenges building coalition relationships and staffing key positions; while these challenges are experienced within project partnerships and individual organizations, they are related to broader macroeconomic and political environments that affect organizational priorities and capacities.

Finally, grantees shared how they have changed their strategies in light of their experiences and learning. Grassroots engagement remains central to most organizations’ work, and many grantees noted that they are becoming more effective in virtual and remote engagement, which is now a longer-term reality than initially expected. A number of grantees noted that they are now able to move towards broader community engagement and messaging, and several noted that they are focused more now on collecting stories and lifting up community voices, sometimes as part of a messaging strategy for pushing back against misinformation. Finally, some grantees shared that they are moving their focus to administrative advocacy; this includes grantees who had success passing policies in their first year who are working on ensuring that the policy is enacted in the best way possible, as well as some who experienced challenges building support for legislative change.

Discussion

Although we are still early in our evaluation, themes have started to emerge around several of our domains of inquiry that cross-cut the various data collection methods we are using.

Deep and broad community engagement: It was clear that many state coalitions feel that at 18 months into the grant, this work is only beginning. Although a number of obstacles influenced the pace of work, including the need to do defensive advocacy, natural disasters, and COVID-19-related barriers, it is apparent that even in the absence of emergencies the work of engaging community takes significant time, and that a 2-year funding cycle can at best lay groundwork for a longer-term change in patterns.

Power ecosystem: The results of our inquiries using the social network analysis around the leadership of lead grantees and organizations meeting the project definition of “organizations of color” were indeterminate. Through our qualitative work, we have gained additional perspective on the various roles of organizations within the state coalitions, and come to understand that leadership has more nuance than we originally understood. The role of project lead may look very different in different projects, and in some cases a subgrantee may be acting as the project convener, for example. We will explore this further through the rest of the evaluation.

Policy, budget, and administrative outcomes: It was notable that there were fewer policy wins reported in the second year than in the first, and also that the states reporting outcomes were substantially more politically liberal than the overall set of Voices states. In addition, it is important to recognize that there are many factors that contribute to a policy win, and that some of the biggest policy outcomes reported by Voices grantees are the result of many years of effort predating the grant funding. The advocacy and organizing work of Voices teams during the life of the grant is meaningful, but just one factor within a complex set of contributions.

Community power: It clearly emerged from our qualitative data that building community power is much longer-term than the duration of the Voices grants. Although the Voices grants have allowed grantees to begin this work – or in some cases to continue work that was previously funded through other mechanisms – in order to have long term impact the funding needs to be both longer-term and less project-specific. Significant funding and time need to be dedicated to participatory processes for developing project plans that include relationship-building between organizations, organizing within the communities, and collectively developing objectives with participation from both community and organization stakeholders.

Looking forward: Now that the grantees are more than 18 months into their 26-month funding period, the next phase of the evaluation will focus on understanding changes over time as well as identifying additional lessons that can inform future programs and the broader power-building field.