This SBIR Phase II proposal will fully develop and test the acceptability, feasibility, and efficacy of a novel measurement and feedback system, SMART-Wrap, tailored to Wraparound service model (WSM) for youth with serious emotional disorders (SED). SMART-Wrap will be a feasible, cost-efficient, and scalable software system to meet the pressing public health need for measurement-based care in care coordination for youth behavioral health. Results from pilot testing will determine SMART-Wrap's feasibility, usability, and efficacy in improving care quality and family outcomes, in addition to preparing the product for commercialization.
The goal of this Phase II SBIR is to develop and test a novel measurement and feedback system tailored to Wraparound Service Model (WSM), called SMART-Wrap. Aim 3 consists of a controlled efficacy study of SMART-Wrap in which 2 WSM provider organizations (WPO) will participate in a parallel group-randomized trial. Using a cascading recruitment strategy, 24 care teams will be recruited, with 12 teams randomized to the SMART-Wrap intervention versus 12 to "Services-as-Usual" (SAU), stratifying by supervisor to balance clustering effects. Randomization will be done at the care coordinator (CC) level. Each WPO supervisor (n = 12) will have equal numbers of CCs assigned to each condition (approx. 2 CCs for each supervisor), yielding 6 supervisors in each condition (4 per WPO). Care team participants must agree to use the SMART-Wrap prototype as part of their WSM over an 8-month test period and complete measures of service and implementation outcomes. CC use of standardized assessment, plan revision, and other SMART-Wrap promoted behaviors will be assessed via monthly web-based surveys. Investigators will enroll n=5 caregivers (CGs) per CC who will be the primary focus of data collection. This will yield a total recruited sample of N=120 (n=60 per group) caregivers of youth aged 8-17 with SED. Family participants will participate for 4 months total, initiating within a 4-month enrollment period. Family participants will respond to measures regarding service and youth outcomes at baseline and at 4 months. Investigators hypothesize the following for our primary outcomes: (1) compared to care teams in the control group, care teams in the SMART-Wrap group will demonstrate (a) greater use of data and feedback in service delivery; (b) greater fidelity to the Wraparound process; (c) higher self-reported teamwork, working alliance, and satisfaction with the intervention; and (d) more positive attitudes toward standardized assessment; and (2) compared to the control group, caregivers of youth receiving services from care teams using SMART-Wrap will report (a) greater goal clarity; (b) greater use of MBC strategies; (c) greater satisfaction with services and progress; (d) better fidelity to Wraparound; (e) more effective team functioning; and (f) more positive outcomes, including reduced caregiver stress and improved symptoms and functioning. Our exploratory hypotheses focus on supervision to examine whether (1) supervisors randomized to SMART-Wrap report (a) a greater proportion of supervision time spent reviewing data on family progress and case conceptualization and strategies; (b) greater alliance with the care coordinator, and (c) greater perception of CC effectiveness. Investigators will also explore (2) the validity of SMART-Wrap measurement by asking whether domains assessed via SMART-Wrap in service delivery (e.g., alliance, progress, satisfaction, functioning) are associated with data collected by the external research team via interviews and surveys. To evaluate differences on youth and CG outcomes (youth functioning on the Top Problem Assessment (TPA), youth symptoms on the Brief Problem Checklist (BPC), and caregiver strain on the Caregiver Strain Questionnaire (CSGQ), investigators will use a series of 3-level longitudinal hierarchical linear models (separate model per outcome variable) to compare differences in group rates of change and timepoint mean scores based on intervention condition. Investigators will also test for curvilinear time trends over five timepoints of data on the TPA. To evaluate implementation outcomes, investigators will conduct the following analyses. A series of 2-level cross-sectional HLMs with families nested within CCs will be run to assess CG and CC Wraparound fidelity on the Wraparound Fidelity Index, Brief Version as a function of intervention condition. Separate 3-level longitudinal HLMs will be run to test for differences, as a function of intervention condition, on CC satisfaction on the Therapist Satisfaction Inventory, attitudes toward standardized assessment on the Attitudes Towards Standardized Assessment scale, and percentage of time during supervision meetings spent (1) reviewing data on family progress and on (2) case conceptualization and strategies on the Supervision Process Questionnaire (SPQ) (with levels for timepoint, CC, and supervisor on the SPQ). Longitudinal HLMs will be run exploring the rate of change on use of data/feedback and other MFS targets on the Monthly CC Report (MCR) and Monthly CG Report (MPR), and investigators will test for curvilinear trends over 5 months. Investigators will explore differences on CG and CC alliance on the Working Alliance Inventory, satisfaction, and attendance using HLMs. Dropout is binomial; thus, investigators will use Hierarchical Generalized Modeling, log-link function to estimate odds of dropout. To evaluate exploratory analyses, investigators will conduct the following analyses. With two HLMs, investigators will predict the trajectories of youth functioning (e.g., the TPA), while accounting for time-varying changes in MCR and MPR total scores. To explore the validity of SMART-Wrap measurement, investigators will associate SMART-Wrap measures with other measures. Using Pearson r and Intraclass Correlations (ICCs), investigators will explore the relationship between timepoint-matched CC scores on youth functioning in SMART-Wrap and CG-rated scores on the TPA, CGSQ, and BPC. Investigators will also explore Pearson r correlations between ratings of progress as scored in SMART-Wrap, and HLM-calculated rate of change on the caregiver-rated TPA. Investigators will examine the relationship between parent satisfaction as recorded in SMART-Wrap and Parent Satisfaction scores. Second, investigators will study how the use of the system may be significantly related to improvement on several outcomes, including fidelity, working alliance, and youth problems and symptoms. Predictors will include CC and supervisor use of SMART-Wrap system elements, data completeness, and time in system. For these analyses, investigators will use HLMs as appropriate to the DVs (longitudinal or cross-sectional) Across analyses, investigators will model data missing at random using full maximum likelihood estimation to account for attrition bias and investigators will control for multiple comparisons using false discovery rates.
Study Type
INTERVENTIONAL
Allocation
RANDOMIZED
Purpose
TREATMENT
Masking
NONE
Enrollment
216
An SMS-based intervention to facilitate regular, repeated evaluation of intermediate outcomes through self-report assessments for youth under Wraparound service model (WSM) care.
3C Institute
Durham, North Carolina, United States
Short Message Service (SMS) Outreach - Caregiver
Caregiver Short Message Service (SMS) outreach includes 27 self-report Likert items (4 pt scale- 0= Not at all to 3= Absolutely) capturing feedback on six different themes (Therapeutic Alliance, Wraparound Fidelity, Satisfaction with Care, Making Progress, Caregiver and Family Outcomes, and Youth Outcomes). Caregivers will be asked to complete 4 items from the bank of 27, each week over the course of 16 weeks. Items will be resampled across the 4-month interval with 64 items being delivered. A mean score will be computed. SMS score means will range from 0 to 3 and lower scores indicate increased Wraparound needs.
Time frame: Weekly, ongoing for 4 months
Short Message Service (SMS) Open Rate (SMARTWrap open rate) - Caregiver
Short Message Service (SMS) open rate will be calculated from the 16 SMS's that are delivered. Rates will be computed as percentage opened (0-100%) and higher rates indicated more consistent system usage.
Time frame: ongoing, over a 4-month period
Item Completion Rate (SMARTWrap item completion rate) - Caregiver
Item completion rate will be calculated from the 64 items that are delivered. Rates will be computed as percentages (0-100%) and higher rates indicated more consistent item completion.
Time frame: ongoing, over a 4-month period
Time to Complete (SMARTWrap time) - Caregiver
Time to complete the 16 SMS delivered surveys will be calculated and reported as a mean in seconds ranging from 0 to 600, with higher scores showing longer completion times.
Time frame: ongoing, over a 4-month period
SMART-Wrap User Survey - Caregiver
The SMART-Wrap User Survey-Caregiver version is a 15-item study team developed survey to evaluate "real world" user perceptions of SMART-Wrap. Items will focus on the user experience of the Caregiver. The items will be scored using a 5-pt Likert scale (1- Strongly disagree to 5- Strongly Agree). Items will be summed and a mean computed (range 1 to 5) and higher scores indicating higher usability.
Time frame: 4-month
SMART-Wrap System Usability Scale - Caregiver
The System Usability Scale (SUS) is a widely used 10-item measure of usability considered highly sensitive and robust. Items will be completed using a 5-pt Likert scale (0- Strongly disagree to 4- Strongly Agree). The sum of the scores is multiplied by 2.5 to derive the final score. Scores range from 0-100: \<50 indicates unacceptable usability, 50-70 marginal, and \>70 indicating acceptable.
Time frame: 4-month
Monthly Caregiver Report (MPR) - Caregiver
Caregivers will report via 5-item monthly survey related to use of data in treatment, e.g., reviewed plan with care coordinator (CC), problem-solved concerns with CC, reviewed data, revised treatment goals and/or strategies, contracted by CC about concerns. Ratings will be completed using a 4-pt Likert scale (0= Not at all to 3= Absolutely). Scores will be summed and a mean computed (range 0 to 3) and higher scores indicating greater report of data used in treatment.
Time frame: Baseline, 1-month, 2-month, 3-month, 4-month
Working Alliance Inventory- Short Revised (WAI-SR) - Caregiver
The Working Alliance Inventory-Short Revised (WAI-SR) is a widely used rating scale designed to measure the working alliance between counselors and clients. Both versions have 12 5-point Likert items covering three domains: bond, goals, and tasks. As done in previous studies by the investigators, the WAI will be revised to reflect alliance between the Care Team and family. Scores range from 12 to 60, with higher scores indicating better alliance.
Time frame: 4-month
The Parent Satisfaction Scales - Caregiver
The Parent Satisfaction Scales reliably measure five dimensions of satisfaction, each with 1 Likert scale item, including: (1) access and convenience, (2) child's treatment process and relationship with providers,(3) parent and family services, (4) satisfaction with progress, and (5) global satisfaction. Respondents rate each statement using a 4-point Likert scale (1-No, Definitely Not to 4-Yes, definitely). The responses are summed, and then an average is calculated (range 1 to 4). A higher score indicates greater satisfaction with services.
Time frame: 4-month
Top Problem Assessment (TPA) - Caregiver
The Top Problem Assessment (TPA) is a consumer-focused index of the severity of the top three problems nominated by the caregiver, on a scale of 0-10. The three ratings are combined (range 0-30) and a higher score indicates higher perceived problems. TPA has good reliability and validity and sensitivity to treatment changes.
Time frame: Baseline, 1-month, 2-month, 3-month, 4-month
Brief Problem Checklist (BPC) - Caregiver
The Brief Problem Checklist (BPC) is a 12-item checklist of internalizing and, externalizing, rated on a 2-pt Likert scale (0-Not true to 2-Very true). Scores are summed (range 0-24) and higher scores indicate more problem behaviors.
Time frame: Baseline, 4-month
Caregiver Strain Questionnaire (CGSQ) - Caregiver
The Caregiver Strain Questionnaire (CGSQ) is a 21-item measure of the parent/caregiver's stress related to caregiving for a youth with emotional problems. Responses are made on a 5-point Likert Scale (0- Not at all to 4- Very much). Scores range from 0 to 84 and higher scores indicate greater caregiver strain.
Time frame: Baseline, 4-month
Wraparound Fidelity Index Brief Version WFI-EZ - Caregiver
The Wraparound Fidelity Index Brief Version (WFI-EZ) is a widely used 25-item self-report measure using a 5-pt Likert scale (0-strongly disagree to 4-strongly agree) that assesses fidelity to the Wraparound process. Research shows the WFI-EZ is reliable, valid, and discriminates against groups with differing support to WSM implementation. Scores range from 0 to 100, with higher scores indicating better satisfaction.
Time frame: 4-month
WSM Attrition and Dropout - Caregiver
Attrition and dropout will be indexed by project staff through chart review. Each family will receive a rating of 0- Dropped out of Care/ Lost to Attrition, or 1- Participated in Care for each week of the 4 month study period. Scores will range from 0 to 16 and higher scores indicating persistent treatment.
Time frame: 4-month
WSM Attendance - Caregiver
Attendance will be indexed by project staff through chart review, using a standard checklist to tally: team meetings, service encounters, therapy events, no shows and cancellations. For each attended event, 1 point will be assigned. For each, cancellation or no show, 1 point will be deducted. The range of scores will be based on participant behavior and higher scores indicating more robust program participation.
Time frame: 4-month
Short Message Service (SMS) Outreach - Youth
Youth Short Message Service (SMS) outreach includes 20 self-report Likert items (4 pt scale- 0= Not at all to 3= Absolutely) capturing feedback on six different themes (Therapeutic Alliance, Wraparound Fidelity, Satisfaction with Care, Making Progress, Caregiver and Family Outcomes, and Youth Outcomes). Youth will be asked to complete 3 items from the bank of 20, each week. Items will be resampled across the 4-month interval with 48 items being delivered. A mean score will be computed. SMS score means will range from 0 to 3 and lower scores indicate increased Wraparound needs.
Time frame: Weekly, ongoing for 4-months
WSM Attrition and Dropout - Youth
Attrition and dropout will be indexed by project staff through chart review. Each family will receive a rating of 0- Dropped out of Care/ Lost to Attrition, or 1- Participated in care for each week of the 4 month study period. Scores will range from 0 to 16 and higher scores indicating persistent treatment.
Time frame: 4-month
WSM Attendance - Youth
Attendance will be indexed by project staff through chart review, using a standard checklist to tally: team meetings, service encounters, therapy events, no shows and cancellations. For each attended event, 1 point will be assigned. For each, cancellation or no show, 1 point will be deducted. The range of scores will be based on participant behavior and higher scores indicating more robust program participation.
Time frame: 4-month
Short Message Service (SMS) Open Rate (SMARTWrap open rate) - Youth
Short Message Service (SMS) open rate will be calculated from the 16 SMS's that are delivered. Rates will be computed as percentage opened (0-100%) and higher rates indicated more consistent system usage.
Time frame: ongoing, over a 4-month period
Item Completion Rate (SMARTWrap item completion rate) - Youth
Item completion rate will be calculated from the 48 items that are delivered. Rates will be computed as percentages (0-100%) and higher rates indicated more consistent item completion.
Time frame: ongoing, over a 4-month period
Time to Complete (SMARTWrap time) - Youth
Time to complete the 16 Short Message Service (SMS) delivered item surveys will be calculated, and reported as a mean in seconds ranging from 0 to 600, with higher scores showing longer completion times.
Time frame: ongoing, over a 4-month period
System Logins (SMARTWrap login) - Care Team
The number of care team logins will be summed across the implementation period. The scores will range from 0 to an undefined number (based on user behavior) and higher values indicating more system use.
Time frame: ongoing, over a 4-month period
SMART-Wrap User Survey - Care Team
The SMART-Wrap User Survey-Care Team version is a 20-item study team developed survey to evaluate "real world" user perceptions of SMART-Wrap. Items will focus on the user experience around the Care Team dashboard/features. The items will be scored using a 5-pt Likert scale (1- Strongly disagree to 5- Strongly Agree). Items will be summed and a mean computed (range 1 to 5) and higher scores indicating higher usability.
Time frame: 4-month
SMART-Wrap System Usability Scale - Care Team
The System Usability Scale (SUS) is a widely used 10-item measure of usability considered highly sensitive and robust. Items will be completed using a 5-pt Likert scale (0- Strongly disagree to 4- Strongly Agree). The sum of the scores is multiplied by 2.5 to derive the final score. Scores range from 0-100: \<50 indicates unacceptable usability, 50-70 marginal, and \>70 indicating acceptable.
Time frame: 4-month
Current Assessment Practice Evaluation-Revised (CAPER) - Care Team
Care teams will complete the Current Assessment Practice Evaluation-Revised (CAPER), a 7-item monthly survey asking about target measurement-based care (MBC) behaviors used in their delivery of care, e.g., reviewed data (with a family or supervisor), presented data or information in a meeting, referred to an evidence-based practice. Ratings will be completed using a 4-pt Likert scale (1= None (0%), 2= Some (1-39%), 3= Half 40-60%), and 4= Most (61-100%). Scores will be summed and a mean computed (range 1 to 4) and higher scores indicating greater report of measurement-based care.
Time frame: Baseline, 1-month, 2-month, 3-month, 4-month
Working Alliance Inventory- Short Revised (WAI-SR) - Care Team
The Working Alliance Inventory (WAI) is a widely used rating scale designed to measure the working alliance between counselors and clients. Both versions have 12 5-point Likert items covering three domains: bond, goals, and tasks. As done in previous studies by the investigators, the WAI will be revised to reflect alliance between the Care Team and family. Scores range from 12 to 60, with higher scores indicating better alliance.
Time frame: 4-month
Wraparound Fidelity Index Brief Version WFI-EZ - Care Team
The Wraparound Fidelity Index Brief Version (WFI-EZ) is a widely used 25-item self-report measure using a 5-pt Likert scale (0-strongly disagree to 4-strongly agree) that assesses fidelity to the Wraparound process. Research shows the WFI-EZ is reliable, valid, and discriminates against groups with differing support to WSM implementation. Scores range from 0 to 100, with higher scores indicating better satisfaction.
Time frame: 4-month
Therapist Satisfaction Inventory (TSI) - Care Team
The Therapist Satisfaction Inventory (TSI) is a 16-item self-report measure using a 5-pt Likert scale (1- Strongly disagree to 5- Strongly agree) that encompasses therapists' affinity for the intervention being used, perceived effectiveness, capacity for individualization and flexibility, and applicability to youths they work with. Ratings are summed and a mean is computed (range 1 to 5) with higher scores indicating greater therapist satisfaction.
Time frame: 4-month
Supervision Process Questionnaire (SPQ) - Care Team
The Supervision Process Questionnaire (SPQ) measures care team ratings of supervision time spent in 8 areas: crisis assessment, administrative tasks, case management issues, case conceptualization, interventions, client relationship, supervisory relationship, and reviewing data on progress. The ratings are made on a 3pt Likert scale (1- Too little, 2- About right, and 3- Too much). Scores are summed across the 8 areas, and an average is computed (range 1 to 3). Overall scores 2 and above indicate adequate supervision.
Time frame: Baseline, 4-month
Attitudes toward Standardized Assessment Scale (ASA) - Care Team
The Attitudes toward Standardized Assessment Scale (ASA) is a 22-item measure of practitioner perceptions and attitudes about using standardized assessments in their clinical practice. Items are scored on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale , and summed to create a total score (range 22 to 110). Higher rating ratings have been associated with a greater likelihood of standardized assessment use.
Time frame: 4-month
This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.