Evaluating Implementation and Impact of PeerOnCall, a Mobile Health Approach to Peer Support for Canadian Public Safety Personnel

Overview

A prospective cohort study design will be implemented across four sectors and multiple organizations sites to evaluate implementation and impact of the apps. Pre/post impact data will be collected over a six-month implementation period to assess whether the apps lead to increased Public Safety Personnel outreach to peer support (primary outcome), and to increased mental health literacy, and decreased mental distress (secondary outcomes). Recruitment will take place at three levels; 1) organizations (including knowledge users/organization champions), 2) peer support providers, and 3) public safety personnel employed within each participating organization. Sources of implementation and impact data for the six-month trial will include: 1) app utilization data, 2) knowledge user/organization champion interviews, 3) surveys with public safety personnel, 4) focus groups with peer support providers, and 5) implementation costs. Analysis will focus on both implementation and effectiveness to address the research questions related to the impact of the apps, as well as how the process and context of implementation shapes effectiveness.

PeerOnCall was developed to support frontline PSP and includes features such as anonymous calling or messaging to a choice of peer support providers, access to self-screening tools, a personal wellness toolbox, and "tips to cope" resources. PeerOnCall Support is designed for peer support providers to provide a secure, private platform for providing support to peers who reach out via text or telephone through the PeerOnCall app. This study will explicitly scale implementation and evaluation of these apps in the context of day-to-day work within a range of PSP organizations across four different service sectors (corrections, emergency communicators, fire services and paramedic services). The implementation period for the app is six-months. Prior to the end of the six-month implementation period, the sector lead will meet with each organization to review app utilization data and to discuss options for continued usage of the app within the organization. The Consolidated Framework for Implementation Research (CFIR) will inform exploration, not only of whether the apps 'work', but also about how the apps are adopted, and the individual and organizational forces that shape adoption. The study will deploy the apps to a purposive sample of approximately 24 organizations across four sectors: 1) correctional workers; 2) emergency communicators (e.g., 911 call takers, dispatchers); 3) fire services; and 4) paramedic services. All participating organizations must have an established internal peer support service (with procedures for vetting, training, and ongoing support of the peer support providers), an organization champion who will liaise between the research team and employees, and support from leadership to participate in the study. The process of recruitment will be facilitated by sector leads and knowledge users who have extensive networks within the sectors. The number of organizations recruited across each sector will be different, recognizing differences in how services are provided, and the need for understanding breadth versus depth of implementation in key stakeholder groups. Within each sector, key knowledge users (at the sector level) and organization champions (at the organization level) will be recruited. The sector-level knowledge users will provide insights into general considerations for implementation and evaluation. In the participating organizations, one or two "champions" will be recruited who will be responsible for organization onboarding, including customizing content, securing participation of in-house peer support providers, and sharing information with co-workers via formal and informal communication channels. Peer support providers within each organization will be recruited to provide services through the app. All trained peer providers within each organization will be invited to participate. Information meetings will be held in each organization for the peer supporters regarding the PeerOnCall Support app. The orientation will ensure that the peer supporteres are aware of how to use the app and the protocols for responding to PSP who reach out for support, including those who may be in crisis. All eligible employees within each organization will be invited to participate in the research via established communication channels within the organization (e.g., recruitment posters distributed via email, social media etc.), and information meetings held in the organization in the month prior to distributing access to the PeerOnCall app. After orientation within the organization, employees will be provided with an organization code to access the PeerOnCall app once it has been downloaded from the Google Play or Apple store. Employees can choose whether to download the app and they are able to use the app without engaging in the baseline and follow-up surveys. To ensure adequate power for evaluating impact of the app, sample size estimates were based on the primary outcome of interest: the frequency of outreach to peer support through the app (count of the number of times contact was initiated by PeerOnCall users via phone and/or text). The sample size was calculated using a one-sided Poisson test since the frequency of outreach via the app can only increase from no outreach at baseline. Assuming 90% power, a significance level of 0.025 for a one-sided test, a hypothesized rate of 0.2 (20% increase in rates of outreach) and alternative hypothesis rate of 0.1 (thus a difference of 0.1), 161 participants will be required. Considering an estimated 20% drop out rate, 202 participants will be needed. Aggregate data collected will include participant adoptions rates across each organization (e.g., the number of times the app was downloaded) as well as aggregate data on how often the app was used, average durations of use, the number of times different features were accessed, and the number and duration of contacts with peer supporters. Knowledge user/Organization Champion Interviews will be conducted at baseline, three-months and six-months after app implementation begins at each participating organization to explore the context for implementation. Interview questions will be congruent with the Consolidated Framework for Implementation Research. These data will be used to identify facilitators and barriers to app use and inform interpretation of the adoption rates and outcome data both within and across participating organizations. To track immediate feedback about the peer support service, a 'pop-up' feedback survey is presented to users within the application following each peer support encounter asking for a rating of their level of distress before and after the interaction using a 10-point Visual Analogue Scale. There is also an option to provide comments about the encounter. Data from the feedback surveys will help understand the immediate impact of peer support on level of distress. To evaluate user impact over time, employees in each organization will be invited to complete baseline, three-month, and six-month follow-up surveys via a link to complete a survey. Focus groups will be scheduled at three-months and six months after the implementation period with PeerOnCall Support peer support providers to explore their experiences with providing peer support through the app. Three sets of focus groups will be conducted within each sector with six to eight participants in each group. Data regarding implementation costs within each organization will include: the organization champion's time invested on onboarding and communication, administrative time for uploading users and responding to requests, and metrics regarding peer support service provision. The cost of server hosting, troubleshooting, and regular upgrades to the apps will also be tracked across sites. Implementation within each organization will be tracked via descriptive data regarding adoption rates (proportion of employees who download \& use the app), and usage rates over time, including the number of times users reach out for peer support via phone/text, and the number of repeated calls. To evaluate effectiveness, analysis of the pre/post survey data will examine the change in scores across the dataset in dimensions of mental health literacy, levels of mental distress (standardized assessments of anxiety, depression, and Post Traumatic Stress Disorder), and frequency of outreach behaviors in seeking mental health support. Multiple hierarchical linear regression analyses will be conducted to examine differences in effectiveness based on gender, service, and job tenure. Effectiveness of the peer support service will also be evaluated by comparing change scores in level of distress before and after each peer support encounter via the in-app surveys. Identification of barriers and facilitators to adoption of the technology and peer support service will occur through qualitative content analysis of the interview transcripts with the organizational champions and focus groups with the peer supporters. The CFIR framework will guide the analysis process considering organization internal and external forces, as well as how communication strategies influence implementation, and consistency in providing the peer support service. All the data from the evaluation process will be reviewed to generate recommendations for optimizing uptake and impact of the app across different types of organizations.