This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers\' comprehension and satisfaction with the notes and ophthalmologists\' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.
Study Type
INTERVENTIONAL
Allocation
RANDOMIZED
Purpose
HEALTH_SERVICES_RESEARCH
Masking
NONE
Enrollment
851
Prospective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes.
Mayo Clinic
Rochester, Minnesota, United States
Non-Ophthalmologist Note Comprehension and Satisfaction
Assessed via survey responses evaluating understanding of the patient\'s ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary. Degree of understanding was assessed on the following Likert scale: Not at all Neutral Moderately A great deal Satisfaction was graded on the following Likert scale: Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied Preference between note types was assessed on the following Likert scale: Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal
Time frame: From enrollment to 8 weeks after enrollment
Ophthalmologist Evaluation of Summary Accuracy and Time burden
Assessed via survey responses regarding Plain Language Summary accuracy in reflecting the Standard Ophthalmology Note findings, time spent reviewing/editing, and perceived burden. Accuracy in reflecting findings was graded on the following Likert scale: Not at all Neutral A little A great deal Time spent reviewing/editing was reported using the following options: \<1 minute 1. minute 2. minutes 3. minutes 4. minutes 5. or more minutes Perceived burden was graded on the following Likert scale: Not at all Neutral A little A great deal
Time frame: Assessed at a single time point <24 hr after enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Flesch Reading Ease Scale: 0 (poor) to 100 (good)
Time frame: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Flesch-Kincaid Grade Level Scale: 0 (most difficult to read, highest grade level) to 80 (easiest to read, lowest grade level)
Time frame: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
Semantic Measure of Gobbledygook (SMOG) Index Scale: 0 (easiest to read) to 11 (most difficult to read)
Time frame: Assessed at a single time point &lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
BERTScore Measure of Linguistic Similarity Scale: -1 (least similar) to +1 (most similar)
Time frame: Assessed at a single time point &amp;lt;24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries
SBERT Cosine Similarity Measure of Linguistic Similarity Scale: -1 (least similar) to +1 (most similar)
Time frame: Assessed at a single time point &amp;lt;24 hr following enrollment
This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.