Designing a Mobile App to Support Academic Success for Student Veterans

Overview

Compared to civilian students in higher education, student Veterans have high rates of mental health disorders (e.g., Depression: 24% vs 12.1%). As a result, Veterans with mental health disorders can be more likely to experience academic issues, such as lower enrollment rates and slower degree attainment on average. Though many student Veterans could benefit from programming embedded in supported education interventions, current in-person VA supported education treatments are often difficult to access - or not available locally - for these Veterans. In addition, there is no widely available, VA-specific online or mobile-app based resource for students. The goal of this project is to develop and evaluate a comprehensive mobile app for student Veterans with mental health disorders. VetEd will provide a resource to (1) orient student Veterans with mental health disorders to successfully transition to the role of student as defined by their self-created educational roadmap, which will include helping students acquire (2) academic skills, (3) mental health management skills, and (4) up-to-date information on psychiatric, academic, and financial resources to help them successfully meet higher-education expectations. This overall study involves three aims: 1) Developing a Veteran-centered educational support app to help student Veterans with mental health disorders to identify their perceived academic needs, app preferences, and evaluate Veteran-centered content; 2) Testing and iteratively revising the VetEd app (n =15) by assessing app software, content, human-computer interface, usability, satisfaction data, and preliminary exploration of changes in educational functioning (course activity completion, academic self-efficacy, and retention; and 3) Completing final revisions of the VetEd app for a future grant application of a larger randomized controlled trial. This pilot project is significant and innovative in three key respects: (1) it extends services based in previous, effective supported education research to address both psychiatric and academic concerns for Veterans with mental health disorders; (2) is potentially cost-effective and easy to disseminate nationally; and (3) focuses on improving Veterans' academic functioning and quality of life is substantially different than current VA mobile app offerings.

The goals of this phase are to: 1) gain feedback on the VetEd mobile app via Veteran usability testing; 2) assess Veteran interest, satisfaction, and usability of VetEd; 3) iteratively address any technological concerns, and 4) collect preliminary information on the impact of the VetEd app on academic functioning (retention, perceived course task ability, and educational self-efficacy) and community reintegration. The investigators will conduct pilot usability testing with 15 Veterans who meet eligibility criteria, conducted in three waves. Each usability testing wave will include 5 participants, based on the finding that approximately 80% of usability problems can be detected using a sample of 4 to 5 participant per wave. Recruitment. Participants will be 15 Veterans recruited from the Bedford VA medical center (VAMC), and local colleges. As noted in Phase 1, based on the rate of student Veterans seeking educational support services at the Bedford facility and the number of Veteran students in the community, the investigators should have no difficulty recruiting the proposed number of participants. Additionally, this research group has had no problem meeting prior, higher recruitment goals for the recent VITAL Supported Education (SEd) using similar recruitment strategies (n = 43). Screening. Potential participants will be pre-screened by phone to determine if basic study eligibility criteria are met. Individuals who remain eligible after the pre-screen will be invited to attend a comprehensive in-person baseline session. At the beginning of the baseline session, the investigators will provide participants with detailed information about the study and obtain written informed consent. Following informed consent, participants will complete study assessments and, with staff assistance, and download the app to their personal device. Usability testing procedures. Participants will be provided with the app and log-in information, to utilize at home over a four-week period. During that time, they will complete at least two in-app tasks within per week, including setting up an educational roadmap, tracking course assignments, learning about successful study strategies, completing a coping exercise, and finding information related to GI Bill benefits. Participants will be evaluated after app use in the following areas: (a) app usefulness: whether users can successfully complete designated tasks on the app, (b) effectiveness: the ability to accomplish assigned educational app tasks quickly and easily, (c) learnability: the ability to meet pre-determined app navigation goals (e.g. accessing and inputting information into the education roadmap), (d) satisfaction: users' feelings and opinions about the app related to their educational functioning and alignment with mental-health needs, and (e) engagement: initial and sustained interaction between a user and a mobile application. Usability measures will include two quantitative system usability scales (SUS; AES), task completion (quantitative), app log-in and use information (engagement). Perceived usefulness, engagement, and overall satisfaction will also be assessed through a post-app-testing qualitative interview (1.5 hours). After each round of testing, the investigative team will work with the programming team to improve the app. The following measures will be utilized: Mental Health Diagnostic Measure: The Structured Clinical Interview (SCID-5) is a reliable semi-structured instrument that is the gold standard for diagnosing mental health disorders. System Usability Scale (SUS): The SUS a 10-item measure, scored on a 5-point Likert scale, that assesses human-computer interaction. A SUS score above 58 is regarded as above average, and a SUS score above 80 a score above which participants are likely to recommend the product to friends. Acceptability E-Scale (AES): The AES is a 5-item validated measure of the acceptability of mobile interventions, which has a cut-off of 80% for acceptability covering overall and specific component satisfaction. Academic Functioning Outcomes: Increased academic function is the primary non-usability related goal of the VetEd app. Consequently, preliminary educational functioning information related to academic self-efficacy, preparedness for course tasks, and perceived confidence in course completion/retention will also be collected and described. In line with educational research suggesting non-grade factors can be stronger predictors of educational success and graduation, the investigators will specifically assess academic self-efficacy (Academic Self-Efficacy Scale: ASES), ability to complete course tasks (The Coping With Education Barriers Subscale: CWBS), and confidence in completing coursework and academic retention (qualitative interview). Community Reintegration: The Military to Civilian questionnaire (M2C-Q) is a 16-item intended to measure dimensions of community reintegration related to productivity (in education, work, and domestic life); social relations; community engagement; and perceived meaning in life, self-care, and leisure. Data management. VetEd will be developed and hosted on a secure server by Geisel Software, Inc. The app will collect usability and content data, but will not collect personally identifiable information. Coded usernames and passwords will be used for app log-in. When data collection is complete, Geisel Software, Inc. will transfer the coded usability data to the Bedford VAMC research staff. Data will not be shared outside the VA. Paper research files will be kept at the Bedford VAMC in a locked cabinet and use deidentified numbers. Data will be checked for inconsistencies, omissions, and errors. The research assistant will be trained in data management and confidentiality procedures, and be responsible for data entry and management. Data analysis plan. The main Phase 2 quantitative measures of acceptability and usability will be the Acceptability E-Scale and the System Usability Scale (SUS), respectively. The investigators expect that Acceptability E-Scale and SUS ratings will increase across the 3 waves of app revision, and that by the third wave the investigators will reach the benchmark of 80% on the Acceptability E-Scale and a 68 on the SUS (good acceptability). Additional data will be provided by the qualitative interviews, using rapid qualitative processes. The investigators will also assess feasibility and engagement by measuring the proportion of individuals who successfully complete the 4 week app testing, with at least 5 completed activities (of requested 8, 2 per week) or 63% of the program completed, in line with the mean number of completed sessions found in previous mobile interventions for chronic conditions. The investigators will also assess academic functioning (academic self-efficacy, course task completion, and perceived retention/course completion) and community reintegration (M2C-Q) pre- and post-app use for each student Veteran specifically to 1) characterize the sample in terms of baseline academic functioning and community reintegration, 2) describe changes in academic functioning and community reintegration, and 3) test the feasibility of these measures in anticipation of a larger, future trial.