Usability Evaluation of Gen AI-based Nutrition Chatbot for Pregnant Women

Overview

Background: Pregnancy imposes significant physical demands, with complications like gestational diabetes (GDM) and pre-eclampsia posing serious risks. Nutrition is crucial for mitigation, but accessing reliable guidance remains challenging. This study evaluates the feasibility of an AI chatbot providing nutritional guidance for managing these conditions. Methods: In a quasi-experimental design, 100 pregnant women will self-select into either the intervention group (n=50, using an AI chatbot) or control group (n=50, receiving standard care). The primary outcome is usability measured by the System Usability Scale (SUS) at 12 weeks, with an expected mean difference of ≥13 points. Secondary outcomes include technology acceptance (Technology Acceptance Model), user engagement, information accuracy, and changes in dietary knowledge/behaviors. Quantitative data will be analyzed using intention-to-treat and t-tests. Semi-structured interviews with 20 participants will explore user experiences through thematic analysis. Expected Results: The AI chatbot is anticipated to demonstrate superior usability and high user acceptance (TAM \>5.0/7), with improvements in dietary knowledge and behavior. Qualitative findings will provide insights into benefits, barriers, and engagement factors. Conclusion: This study will establish an evidence base on AI chatbot feasibility and acceptance for prenatal nutrition, informing tool optimization and future large-scale trials.

Objectives: This study primarily aims to evaluate the usability of a nutrition AI chatbot for pregnant women by comparing System Usability Scale (SUS) scores between intervention and control groups. Secondary objectives include assessing technology acceptance (Technology Acceptance Model), engagement patterns, information quality (accuracy, comprehensibility, consistency), and changes in nutritional knowledge. Design: A quasi-experimental design with two parallel groups (n=50 each) will be employed. Using self-selection, participants will choose to enroll in the intervention group (access to an AI chatbot plus routine care) or the control group (access to a standardized WeChat information service plus routine care). Routine care for all participants includes standard prenatal clinic visits and printed nutritional materials. The WeChat service for the control group will be operated by a trained research assistant using a pre-defined script during two scheduled windows daily, providing information quoted from the official nutritional leaflets. This isolates the mode of information delivery (AI versus human-facilitated messaging) as the primary variable. Participants: Inclusion criteria: pregnant women aged ≥18 years, able to consent, owning a smartphone with internet access. Exclusion criteria: enrollment in other nutrition interventions or severe mental health conditions impairing technology use. A purposive subsample of 20 participants (10 per group) will complete qualitative interviews. Sample Size: Based on an expected mean SUS score of 78 (SD=12) in the intervention group and 65 (SD=15) in the control group (Cohen's d=0.95), 23 participants per group are required for 90% power at alpha=0.05. Accounting for 50% attrition, 50 participants per group will be recruited. Propensity score matching will be applied to reduce selection bias using variables including age, gestational age, parity, education, and baseline technology use. Measurements: The primary outcome, usability, will be measured using the System Usability Scale (SUS) and the Chatbot Usability Questionnaire (CUQ) at 12 weeks. Technology acceptance will be assessed using the Technology Acceptance Model (TAM). Nutritional knowledge will be evaluated at baseline and 12 weeks using a 15-item questionnaire and the FIGO Nutrition Checklist. Information accuracy and consistency will be assessed by an expert panel rating 150 chatbot responses and repeated submission of 20 test questions. Engagement will be analyzed via application usage logs measuring adherence, intensity, and persistence. Semi-structured interviews will explore user experiences in depth. Data Analysis: Quantitative data will be analyzed using intention-to-treat principles. Primary analysis will compare mean SUS scores between groups using independent samples t-tests, with effect sizes calculated as Cohen's d. Secondary outcomes will be analyzed using similar approaches, with chi-square tests for proportions and linear mixed models for nutritional knowledge change scores. Missing data will be addressed through multiple imputation. Qualitative interview transcripts will be analyzed using thematic analysis with dual independent coding. Study Timeline: Participants will be enrolled over a 6-month period, with each participant completing a 12-week intervention and follow-up period.