Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

Overview

This project proposes to reduce disparities in care among disadvantaged racial/ethnic minority adolescents with ADHD by improving community therapist fidelity to evidence-based behavior therapy through a technology-assisted supervision intervention. In Y01, the research team will work with stakeholders to develop the proposed supervision intervention utilizing two novel technologies: Lyssn + Care4 (LC4S). In Y02, a preliminary clinical trial (N=72) will be conducted in three community mental health agencies in Miami, FL. Adolescent participants will be randomly assigned to receive supervision from a therapist who is trained in LCS4 or provides enhanced supervision as usual(ESAU)using a permuted block randomization strategy that randomizes within site. There will also be double randomization of agency therapists to supervisors. Supervisors will deliver both conditions and investigators will test for contamination to determine the integrity of this design prior to a future R01 that measures patient outcomes. Data from therapists, adolescents and their parents, and supervisors will be collected pre-training, post-training, weekly during service delivery, at EBT completion, and at the end of the trial. The proximal intervention target is therapist fidelity to EBT and the distal targets are service delivery outcomes that include quality, quantity, and speed of delivery. Investigators will also measure indices of consumer fit: cost, acceptability, feasibility, and fidelity to supervision procedures. Sources of data will be audio recorded therapy and supervision sessions, therapist and supervisor report, and project and electronic health records. In longitudinal analyses, time will be modeled as a person-specific variable representing months since baseline. Investigators will nest adolescents within therapists for all analyses.

Y02 of this study is a small (N=72) phase 1/phase 2 clinical trial of a supervision intervention designed to improve therapist treatment fidelity and subsequent service delivery outcomes. The parallel design includes random assignment of eligible and consenting patients at three community agencies to two active supervision intervention arms (LC4S or ESAU) using a permuted block randomization strategy that accounts for agency. Participants will receive behavioral interventions from community agency staff and their service utilization will be tracked using project and agency electronic health records. Agency therapists and supervisors will also be participants in this trial. Therapists will be randomized to receive either LC4S or ESAU from their supervisor (double randomization) using a block randomization strategy that accounts for site. Supervisors will administer both supervision conditions in this trial; however, investigators will systematically assess for contamination to assess whether this design is appropriate for a future R01. To minimize bias, adolescent and parent participants will not be informed of the group to which they have been assigned. However, full masking of therapists and supervisors is not feasible in this trial because both supervisors and therapists will know whether they are participating in technology-assisted supervision activities or standard supervision activities due to the nature of these conditions. However, the primary investigator will blind therapists and supervisors to our study hypotheses and the nature of outcome measures to minimize bias in the trial. Many of these measures will be observational and objective (i.e., therapy records, therapy audio recordings), which should reduce bias stemming from self-reports. Investigators will also assess whether there are group effects on therapist accuracy of self-report. All interventions will be delivered by agency staff, who will not be required to follow intervention delivery protocols because an outcome of this study is the extent to which agency therapists follow intervention procedures with guidance provided from their supervisors. Study assessments will be administered electronically via Care4 and data collection will be oversee by study staff. Each measure of fidelity will be analyzed using a separate mixed /growth model (Duncan et al., 1999). In this design, treatment sessions (level 1) are nested within adolescents (level 2), which are nested within therapists (level 3). Each adolescent attends up to 10 sessions; each therapist treats 3 adolescents. Supervisors serve as an additional higher level, but with so few supervisors (approximately 6), this will be addressed by including dummy predictor variables representing the supervisors. The direct effect of LC4S vs. ESAU on fidelity intercept and slope will be tested for each fidelity outcome (see Table 3). With time centered at the first session, the intercept reflects initial fidelity for the ESAU condition, the group effect reflects the initial fidelity difference between ESAU and LC4S, the time effect reflects the linear change in fidelity over time for the ESAU condition, and the interaction of time and group reflects the difference between ESAU and LC4S in linear change in fidelity over time. As part of the R34, in will estimate the intraclass correlation (ICC) and design effect for the clustering effect of therapist and supervisor on outcome to determine the extent to which additional clustering will be needed in a future R01. ICC ranges from 0 to 1, with larger values reflecting a larger proportion of variance at the higher levels (here, therapist and supervisor rather than adolescent). Investigators will also test both linear and non-linear slopes to ascertain the expected shape of the LGCA in a future R01. Time to therapist MI competence, proportion of EBT delivered by 10th session, and number of sessions and days to completion of the EBT will be modeled using regression (linear, logistic, or Poisson depending on the distribution of the resulting variables) with group as a predictor. Accuracy of therapist self-report will be analyzed using polynomial regression (Laird \& LaFleur, 2013). Using the R package powerlmm, investigators have an estimate of .8 power to detect large effects (d = 0.8) representing group differences at the adolescent (level 2) level; investigators have .4 power to detect medium effects (d = .5) representing group differences at the adolescent level. Our hypotheses, however, are primarily at the level of the therapist (level 3), which has fewer units (i.e., 12 therapists per condition versus 36 adolescents per condition). As such, measures of both adolescent-level and therapist-level effects will be estimated. For power for the ICC calculation, investigators used the ICC.Sample.Size R package based on Zou (2012). With a sample size of 72 participants, 10 observations per participant, alpha equal to .05, and a two tailed test, investigators have greater than .9 power detect an ICC of 0.2. ICC values for cross-sectional data such as children within classrooms are approximately 0.2 (Hedges \& Hedberg, 2007). Investigators expect excellent precision for estimating ICC values in this study.