Formative Evaluation in Implementation Research: Guideline Development

Overview

The Veterans Administration (VA) has embraced the use of implementation research as a critical way to improve care by facilitating the adoption of proven medical practices. Yet implementation science still remains rudimentary. Challenges exist in judging the success of an implementation project. Rather than relying solely on hard outcomes, an investigator must also assess how effectively an intervention is put into place at a site. This assessment comes via a formative evaluation (FE) plan. The FE plan can contain such diverse data elements as the number of fliers distributed about the intervention, how often an intervention advocate speaks to participants, the content derived from team meeting notes, and themes extracted from focus groups. The diversity of such data elements can appear foreign to investigators trained in traditional health services research. The FE plan can also lead to confusion about what to measure, when, and how to integrate the various measures into a unified whole. Complicating matters is the fact that FE is often qualitative--such as interviews, focus groups, and meeting notes which can make data difficult for investigators to interpret. Thus, there is a pressing need to provide clarity on how FE is to be used in implementation studies. This Short Term Project proposal offers a three-part approach toward specifying the best FE practices: first, a systematic review of the literature; second, qualitative interviews with VA implementation investigators; and third, a panel discussion of national experts, funded through a VA Implementation Collaborative Award. The panel discussion will outline FE guidelines based on the researchers expertise, the systematic review, and the qualitative interviews.

Objectives: The Veterans Administration (VA) has embraced the use of implementation research as a critical way to improve care by facilitating the adoption of proven medical practices. Yet implementation science still remains rudimentary. Challenges exist in judging the success of an implementation project. Rather than relying solely on hard outcomes, an investigator must also assess how effectively an intervention is put into place at a site. This assessment comes via a formative evaluation (FE) plan. The FE plan can contain such diverse data elements as the number of fliers distributed about the intervention, how often an intervention advocate speaks to participants, the content derived from team meeting notes, and themes extracted from focus groups. The diversity of such data elements can appear foreign to investigators trained in traditional health services research. The FE plan can also lead to confusion about what to measure, when, and how to integrate the various measures into a unified whole. Complicating matters is the fact that FE is often qualitative-- such as interviews, focus groups, and meeting notes which can make data difficult for investigators to interpret. Thus, there is a pressing need to provide clarity on how FE is to be used in implementation studies. This Short Term Project proposal offers a three-part approach toward specifying the best FE practices: first, a systematic review of the literature; second, qualitative interviews with VA implementation investigators; and third, a panel discussion of national experts, funded through a VA Implementation Collaborative Award. The panel discussion will outline FE guidelines based on the researchers expertise, the systematic review, and the qualitative interviews. The specific aims for this proposal include the following: Specific Aim 1: Conduct a systematic review of VA funded implementation projects to classify the specific types of FE procedures employed. Specific Aim 2: Conduct in-depth telephone interviews with investigators of VA-funded implementation projects to establish which FE methods were used and which were considered most/least successful. Specific Aim 3: Using the national experts gathered together under the umbrella of the Center for Health Equity Research and Promotions (CHERP) Implementation Research Collaborative, develop a set of guidelines outlining best FE practices in VA implementation research. Relevance to the VA, Anticipated Results, and Future Directions: This Short Term Project is designed to provide rapid results for developing the best approach toward the use of FE in VA implementation science. As such this proposal is responsive to the VA Health Services Research \& Development (HSR\&D) priority topics for both implementation and management and research methodology. The systematic review, the qualitative report, and the FE guidelines that will emerge from the project will have wide-spread applications within the VA implementation community, thereby supporting the HSR\&D research mission. The study will also serve as needed pilot data for a future Merit Review award application designed to provide the theoretical and practical approaches toward FE use within the VA.