ClientBot: a Conversational Agent That Supports Skills Practice and Feedback for Motivational Interviewing

Overview

This multi-stage study includes qualitative interviews, usability research, and a randomized training study of technology implementation in a real-world clinical setting. The proposed study will enhance and evaluate an artificial intelligence (AI) based, conversational agent (ClientBot) that simulates a realistic client with alcohol concerns and provides performance-based feedback to support counselor training. The research is in collaboration with PRI, which focuses on training alcohol and substance use counselors.

Stage 1. Interviews with 5 PRI trainers will assess current PRI training goals, content, and format, but also how training practices are conducted, including what types of technologies are currently used for what purposes. Trainer participants will then be introduced to the prototype ClientBot platform, including an opportunity to use it, and will be asked to describe how they might integrate the prototype into their existing training practices. Trainers will roleplay with researchers to illustrate work processes, and data collection will include interview audio recordings and field notes. Trainer participants will also be interviewed regarding their use of MI skills and which are most relevant and/or in need of training for their typical trainees. Stage 1 will also include trainees who have participated in a PRI training within the previous 3 months. Recent trainees will provide an informed perspective on the ClientBot platform relative to current PRI skills training practices, which include an MI skills focused workbook and role plays. Participant sessions will be conducted online, using an interview-based usability format. Research staff will provide a brief introduction to ClientBot as a computer-based skills training platform and then ask participants to complete specific tasks, using a 'think aloud' protocol (i.e., provide a self narration as they pursue each task). The goal of the lab-based usability analysis is to verify that the software functions as intended in a controlled setting. As part of the hour-long session, participants will have an opportunity to use 2-3 MI skills training modules. Stage 2. PRI has a robust schedule of in-person and online trainings throughout each year. For stage 2 usability study, we will recruit 20 counselors who have participated in a PRI training within the previous 3 months. Following software engineering work to implement the Phase I ClientBot tool into Lyssn's HIPAA-compliant cloud-based platform, we will conduct a single-arm, 2-week field trial among 20 recent PRI trainees to better assess user workflow, system interaction, and general usability in a community AUD setting. Participants will be encouraged to use ClientBot for 10 minutes per work day. Participants will be contacted daily via email by research staff to inquire about technical reliability (e.g., difficulties operating the tool), functional reliability (e.g., experience using ClientBot functions), and experiences integrating the system into the day's workflow. ClientBot use will be captured automatically by the system, including: which modules were used, the number of times resources were accessed and repeated, time spent viewing introductory materials, fidelity feedback provided during skills practice, and time spent in review of NLP-generated feedback. Following each practice session, participants will answer a brief set of likert questions on 0-10 scale, including: How coherent was the client's narrative and responses? How realistic was this simulated client relative to client's you have seen? How much of the client session focused on alcohol use ? How accurate was the feedback on MI skill usage? After completing all practice sessions, participants will complete the system usability scale (SUS). To gauge implementation readiness, we will also use the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). Each are brief, 5-item measures targeting core implementation outcomes with strong psychometric properties. Trainees will evaluate the perceived accuracy of the ClientBot feedback, which will also be evaluated via objective human coding. Stage 3. During Stage 3 PRI research members will identify 10-15 trainings to recruit from (assuming we will be able to recruit 50% of participants in each training, based on previous participation rates in PRI-based research). Individuals that have signed up for a PRI training will be recruited into the study approximately 6 weeks prior to the training via email invitation. Interested participants will complete informed consent online, and following consent, therapist participants will be randomized into ClientBot (CB) vs. training as usual (TAU) conditions. Participants will then be asked to complete the computer assessment of simulated patient interviews (CASPI), a computer-based, standardized assessment of MI skills that is highly correlated with objective measures of MI fidelity. We will recruit and randomize a total of 200 PRI trainees, using their ongoing online training. PRI trains approximately 1,200 individuals each year. Based on our pilot study, we anticipate a 30% recruitment rate, which would yield the required sample size of 200 in approximately 6-8 months. Research participants will be randomized to use ClientBot immediately following their PRI training, or training-as-usual (TAU), which involves no intervention immediately following training. Prior to the ClientBot intervention, participants in both arms will complete the Computer Assessment of Simulated Patient Interviews (CASPI). The CASPI presents brief client videos with prompts to use various MI skills after each video, which are recorded and then scored for MI fidelity. Participants in both arms will complete follow-up CASPI assessments after the ClientBot training period. After a 2 month washout period, the ClientBot arm will receive an additional follow-up CASPI. Both arms will then be asked to engage in standardized patient sessions to fulfill their PRI training requirements. After completion of SP sessions, participants in the TAU arm will be offered the ClientBot training intervention, followed by a final CASPI. The final months of Phase 2 / Year 2 will focus on data analyses and dissemination of findings via conference presentations and publications.