Evaluation of the Research to Policy Collaboration Model

Overview

This work aims to evaluate an approach for improving federal legislators' use of evidence-known as the Research-to-Policy Collaboration (RPC) - which seeks to address known barriers to policymakers' use of research, including a lack of personal contact between researchers and policymakers and limited relevance of research translation efforts to current policy priorities. The RPC involves structured processes for identifying policymakers' priorities, building researchers' capacity for nonpartisan responses to current policy priorities, and facilitating ongoing and productive researcher-policymaker interactions. This implementation of the RPC will focus on child and family policies relevant to child maltreatment. This study assesses both processes for collaboration and policymakers' use of research within a randomized controlled trial (RCT) employing a mixed methods approach-including quantitative and qualitative evaluation of impact. The proposed project will be guided by three overarching questions: 1. How does the RPC impact researchers and legislative staff? 2. How does the RPC impact legislative activity? 3. How might perceptions and experiences of collaboration through the RPC relate to different forms of evidence use among researchers and policymakers? The RPC's effectiveness will be tested through experimental design (randomization) using qualitative and quantitative assessments of researcher-policymaker interactions and impact. This includes surveying congressional staff and researchers, reviewing records of policymaker's public statements and introduced legislation, and conducting qualitative interviews around researchers' and legislative staffs' experiences with researcher-policymaker collaboration prior to and during the RPC.

Objective: The aim of this study is to examine the effectiveness of a model for supporting policymakers' use of research evidence within a randomized controlled trial (RCT) employing both qualitative and quantitative analyses of survey, interview, and observational data to understand the use of research in policy and collaboration. Legislators' research use will be assessed with multiple methods, including (1) self-report via a survey protocol, (2) legislative activity (i.e., introduced bills and official statements), and (3) qualitative interviews. Additionally, interviews and surveys will assess researchers' reported efficacy, skill and engagement in public policy, and the extent to which their work is informed by the policy process. Intervention: The Research-to-Policy Collaboration (RPC) builds upon theoretical and empirical literature that suggests the need to (a) translate research relevant to current policy priorities, and (b) facilitate productive interactions between policymakers and researchers. The RPC is a nonpartisan, manualized model implemented by an intermediary organization that cultivates relationships between researchers and legislative offices. Implementation occurs in two phases involving a series of seven interrelated activities. During the capacity-building phase, the RPC simultaneously aims to support researcher's development of honest knowledge broker skills with training and coaching, while also conducting iterative needs assessments with congressional offices regarding their current priorities and desire for research evidence. In the collaboration phase, researchers with relevant expertise are coalesced into rapid response teams that are matched with legislative offices. Through a scaffolded series of interactions, the RPC model is used to cultivate productive researcher-policymaker relationships and support responses to legislative requests. Evaluating the effectiveness of the RPC is the focus of the current study. Study Design: This study employs both qualitative and quantitative analyses of survey, interview, and observational data within the context of an RCT. Three overarching research questions guide this work and align with the study's three different components - self-report via a quantitative survey, observation of research use via coded legislative activity, and ethnography via in-depth interviews and participant observations. 1. How does the RPC impact researchers and legislative staff? (Self-Report) 1. Do researchers report improved policy competencies and motivation for conducting policy-relevant research? 2. Will legislative staff report increases in positive attitudes toward, knowledge of, intended application, and actual use of research evidence? 3. Might the impact on legislative and research participants will vary as a function of aspects of their collaborative relationship (e.g., enhanced by mutual trust)? 2. How does the RPC impact legislative activity? (Observation) a) Will congressional offices increase their use of research evidence in legislative activities (i.e., bills, official statements)? 3. How might perceptions and experiences of collaboration through the RPC relate to different forms of evidence use among researchers and policymakers? (Ethnography) 1. What are the barriers and facilitators of participation in the RPC? 2. What types of interactions between researchers and policymakers does the RPC influence? 3. How does the RPC affect participants' assumptions about each other, interactions with each other, and their intentions? 4. What types of evidence use are supported by RPC participation? Sampling Framework All study components draw from an initial sampling framework through which researchers and congressional staff who opt into participating in the RPC will be asked to participate in the study; those who agree to participate in the study will be randomized for either the full RPC intervention or a light-touch control group that receives little direct engagement. This study will oversample to combat anticipated attrition. The final sample includes 96 congressional offices (48 in RPC, 48 in control group) and 226 researchers (151 in RPC, 75 in control group). These samples were derived from an identified population of eligible intervention participants. * A population of relevant legislative offices (Senate and House) will be identified based on the number of child/family bills legislator have (co)sponsored. The random selection process will be stratified to ensure proportional partisan representation and representation of legislators who range in activity related to child welfare. Prior to randomization, the population of relevant legislators will be assigned an Activity Ranking based on the number of times a legislator has sponsored a child/family policy bill. Activity Rankings will range from 1-10 such that 10% of legislators who (co)sponsored the fewest bills will be ranked 1, and 10% of legislators who co-sponsored the most bills will be ranked 10. Recruitment and stratified sampling across a continuum of activity on child/family policy will improve bipartisan representation of the sample. Also aiding with bipartisan representation are targeted outreach efforts that intensify along party lines if participation becomes skewed. * Researchers who work on relevant child/family research will be identified via research institutions, professional networks and listservs, literature searches, and referrals from other researchers. Researchers include research-oriented practitioners and program evaluators from both academic and non-academic settings who are interested in translating their expertise with policymakers. Subsequent to opting-in to participate in a policy engagement intervention, researchers are asked to participate in the study. Study participants will be randomized to receive the RPC or the control condition. Study Component 1: Quantitative Evaluation of Self-Reported RPC Impact A structured survey will be used to assess researchers' capacity for engaging in public policy processes. A corresponding survey protocol will assess legislative staff's attitudes, knowledge, intent, and reported behavior regarding the use of research evidence. Both researchers and legislative staff will be asked about the nature of their collaborative interactions with one another. Surveys will be conducted approximately every three months. Survey constructs are described in outcome measures. Missing data will be handled with multiple imputation because this is best practice when data are not missing at random. Multilevel modelling will be used to examine individual trajectories in outcome variables. Multiple regression will examine changes between select time points; for instance, whether researchers' reported policy skills improve between baseline and subsequent to trainings. These analyses will allow us to examine whether RPC participation is associated with increases in (1) researchers' reported efficacy to engage with policymakers, (2) legislative staff's interest and willingness to use research evidence, and (3) both researchers' and policymakers' positive sentiment toward working together. Longitudinal analyses will further allow us to explore the trajectory of outcome variables over time (e.g., improved researcher competencies, legislators' use of research)-specifically modelling change in evidence use and other outcomes of interest. Analyses of researcher and legislative staff outcomes are all powered at least at a 90% level to detect effect sizes of 0.2 or greater (power analysis conducted with Optimal Design plus Empirical Evidence). Study Component 2: Quantitative Evaluation of Observed RPC Impact on Legislative Activity To complement the self-reported quantitative assessment of change, use of research evidence in legislative activities will be observed before and following RPC participation. Data will be selected, extracted, and coded prior to quantitative analyses with indicators derived from deductive codes. Legislative data will be selected based upon the use of keywords indicating relevance to child/family. Since each bill will comprise more text than is feasible to code with traditional qualitative methods, relevant provisions will be extracted prior to coding (those containing keywords). Further, all subsection titles for each sampled bill will be reviewed for potential relevance and extracted if research evidence is used but not captured otherwise by keywords. Data on legislative activity will be coded based on a detailed codebook informed by prior findings of how research has been used by policymakers and in legislation. This coding scheme will be applied by two coders and the codebook will be revised and further specified until the intercoder reliability reaches at least .70 and 30% of the data for the first time period has been coded. Subsequently, trained coders will code individually and a lead investigator will randomly sample 10% of these to ensure consistency in coding quality. These codes will be converted into quantitative indicators of research use, which are described in outcome measures. Legislative activity will be assessed annually. Multivariate Poisson regression analyses will assess change in research use in thematic codes of (1) bills and (2) official statements between the RPC intervention and control conditions. Moderation analyses will further explore the extent to which changes in legislative activity depend on the nature of collaboration experiences, other survey data (e.g., policymakers' perceptions of the value of research), and legislator characteristics (e.g., party affiliation, years of age). Study Component 3: Ethnographic Evaluation of the RPC model The qualitative component of this study is closely linked with the quantitative work and seeks to better understand the experiences of RPC participants and processes underlying the use of research evidence in the RPC context. Using participatory and ethnographic methods, data will be collected from two sources: (1) semi-structured interviews of trial participants and RPC staff, and (2) observations of researcher trainings and researcher-staff meetings. Interview participants will be sampled from the pool of study participants randomly assigned to receive the RPC, including 11 RPC researchers and 11 RPC congressional office staff that will be interviewed prior to their involvement in the RPC, and 11 RPC researchers and 11 congressional office staff that will be interviewed following collaboration activities via the RPC. Additionally, 3 RPC staff will be interviewed prior to and following the RPC. To gain additional insight into the RPC operations and impact, observations will be conducted of trainings for researchers, formal meetings and informal interactions between researchers and congressional staff, and meetings that occur as part of any RPC collaboration. Following ethnographic best practices, "open-observations" of key meetings and training events will be conducted to enrich the interviews with more information about how attendees interact and how they respond to the RPC. Field notes and transcripts will be analysed with thematic analysis and an interpretive policy analysis that explores different rationalities and rhetoric when discussing evidence used in the collaboration.