Effects of a Large Language Model-Driven Chatbot on Reproductive Concerns After Cancer

Overview

Reproductive health has emerged as a critical yet overlooked concern among Adolescent and Young Adult (AYA) cancer survivors, given their compromised potential for biological parenthood in prime childbearing years. Large language models (LLMs) offer a promising solution to bridge onco-fertility service gaps by integrating evidence-based knowledge and therapeutic frameworks. This pilot randomized controlled trial aims to assess the feasibility and preliminary effectiveness of an LLM-driven chatbot versus electronic brochure in addressing reproductive concerns among AYA cancer survivors.

Methods/Design This is single-blind, two-arm randomized controlled trial. The study will adhere to the CONSORT 2010 checklist for pilot and feasibility trials. The subjects are people aged from 15 to 39 years with cancer diagnosis and self-reported reproductive concerns. Participants, recruitment, and randomization Simple random sampling with a fixed sample size will be employed to recruit subjects. Inclusion criteria are as follows: (1) 15-39-year-old females; (2) cancer diagnosis between ages 15-39; (3) current or prior concerns regarding fertility; (4) proficiency in Mandarin Chinese; and (5) access to a mobile device for intervention delivery. Exclusion criteria include: (1) involvement in the chatbot co-design phases; (2) inability to provide informed consent; (3) significant sensory, cognitive, or psychological impairments precluding meaningful participation; and (4) acute illness at the time of recruitment. Each interested participant will be screened by a trained research assistant to confirm their eligibility and safety to participate in this study. All participants should voluntarily agree to take part and will be asked to provide informed consent with signatures after being informed of the study purposes, procedures, risks, and benefits. For participants under 18 years old, informed consent will be obtained from their parents or legal guardian, and assent will be sought from the minor participants themselves, ensuring they understand the study's purpose and procedures. Eligible participants will be randomly assigned in a 1:1 ratio to either the experimental or the control group. Randomization will be automatically performed using a computer-generated randomization sequence embedded in the chatbot backend. The allocation sequence will be concealed from the research assistant involved in subject recruitment. Intervention group Participants assigned to the intervention group will receive daily prompts to engage with chatbot over a four-week period. The chatbot was developed by a multidisciplinary research team which consists of domain experts in engineering, oncology, reproductive health, and psychology. To mitigate LLM hallucinations for accuracy, we incorporated a knowledge base for model retrieval, including multi-turn ACT dialogues adapted from ACT tutorials, onco-fertility/gynecology/oncology question-answer pair dataset, the list of medical institutions approved to conduct human assisted reproductive technologies (ART) and relevant policies. During the subsequent follow-up phase (weeks 4-8), participants will be encouraged to interact with the chatbot as often as they wish. Control group Participants assigned to the control group will receive an electronic brochure which contains ACT psychoeducational materials, verified medical question-answer pairs, and relevant policy information. Participants will be encouraged to review the brochure daily over a four-week period to reinforce understanding and engagement with the material. During the subsequent follow-up phase (weeks 4-8), participants will be encouraged to read the brochure as often as they wish. Data collection Data will be collected at three time points: baseline (T0), immediately post-intervention at 4 weeks (T1), and follow-up at 8 weeks (T2). All outcome assessments will be administered through the study platform integrated into the chatbot backend system. Given the nature of the intervention, neither participants nor intervention providers can be blinded to group allocation. To minimize measurement bias, outcome assessments will be automatically delivered by the backend system at predefined study time points and completed by participants via self-report without researcher involvement. De-identified coded data will be used for analysis, and the data analysts will remain blinded to group allocation until database lock and data cleaning are completed. The primary outcome of this study is the feasibility of the chatbot intervention among AYA cancer survivors. Feasibility will be evaluated using engagement indicators recorded by the system, including total interaction time, number of active days, and dropout rate at T1 and T2. Secondary outcomes are intended to examine the preliminary effectiveness of the chatbot intervention in reducing reproductive concerns, depression, and decisional conflict, and in improving quality of life and psychological flexibility. These outcome measures will be measured in both groups at T0, T1, and T2 using self-reported questionnaires delivered through the chatbot system. To enhance response rates, participants who do not complete scheduled assessments will receive reminders from the research team via telephone, email, or instant messaging. In addition, at T1 and T2, the total count of dialogue exchanges between the user and the chatbot will be collected from the backend and participants will be invited to provide open-ended feedback on the intervention, including perceived advantages, disadvantages, and overall satisfaction, in order to further assess acceptability and inform future refinement of the chatbot intervention. Ethical considerations The present study was granted ethical approval from the Institutional Review Board of The Hong Kong Polytechnic University. Participants assigned to the intervention group will receive a login account linked only to a unique participant code. The chatbot system will neither require nor verify their real identities, and no directly identifiable personal information will be shared with the chatbot backend. Participants' phone numbers will be stored separately from the chatbot and linked to the participant code only if needed for participant matching in the event that a safety concern is triggered. All participants will receive information about available institutional support resources and emergency contacts. The chatbot will be prompted to detect any indications of psychological distress or crisis and address sensitive topics with empathy. If a participant expresses indications of self-harm or suicidal ideation, the system will immediately provide information for appropriate local mental health support services, including 24/7 crisis hotlines and emergency resources. In addition, a member of the research team will promptly follow up with the participant, to facilitate referral to appropriate professional support, and implement any necessary safeguarding procedures. Data processing and analysis All statistical analyses will be performed using SPSS version 29.0, NVivo version 20.0, and Python version 3.12, with statistical significance set at p \< 0.05. Descriptive statistics (mean, standard deviations, frequencies, and proportions) will be used to summarize participant characteristics and feasibility metrics. For continuous variables (e.g., interaction time, active days), independent samples t-test or Mann-Whitney U test will be used to compare means between groups as appropriate. Categorical variables (e.g., dropout rates) will be compared using Chi-square or Fisher's exact tests as appropriate. To examine the effectiveness of the intervention, a 2 (Groups: chatbot intervention vs. information control) \* 3 (Time: T0, T1, T2) repeated measures analysis of covariance (ANCOVA) will be conducted for all continuous variables (i.e., RCAC, AAQ-II, PHQ-9, DCS, and WHOQOL-BREF), with baseline characteristics included as covariates to adjust for potential confounding influences. Group will serve as the between-subject factor, and time as the within-subjects factor. Violations of sphericity will be addressed using the Greenhouse-Geisser correction as appropriate. The main effects of group and time, as well as the group \* time interaction effects will be reported. All analyses will primarily follow the intention-to-treat (ITT) principle, whereby all participants will be analyzed in the groups to which they were originally assigned. In addition, per-protocol (PP) analyses will be conducted as a secondary approach, including only participants who sufficiently adhered to the study protocol. Missing data will be addressed using maximum likelihood estimation or multiple imputations, depending on the pattern and extent of missingness. Open-ended feedback will be analyzed using content analysis to identify perceived advantages, disadvantages, and satisfaction with the chatbot.