The goal of this study is to test the accuracy of Large Language Model-generated serious illness communication (SIC) summaries, the feasibility of delivering the SIC summaries, and to collect perspectives on the SIC summaries from clinicians and participants with cancer. Large Language Models (LLMs) are artificial intelligence programs that can perform various natural language processing tasks.
The study will enroll up to 70 Dana-Farber patients admitted to Brigham and Women's hospital who have an elevated mortality prediction at admission. The research study procedures include randomization to intervention or control arms on admission (3:1 intervention to control). For patients in the intervention arm, a large language model query will summarize serious illness communication documentation in their medical record screening from the prior 6 months. A summary of the documentation will be sent to the inpatient and outpatient oncology clinicians within 24 hours of admission; these clinicians will be asked to review the summaries, discuss with the patient and incorporate into care plans as appropriate. For patients with no SIC documented on day 4 of the admission, a second email will be sent. Control patients will receive usual care, which is no email sent to their oncology teams. All patients will be offered to participate in an interview on day 4 of their admission about their communication with their care teams. For patients in both arms, inpatient and outpatient oncology attendings will be sent an email survey regarding SIC for the patient and, if applicable, the utility of the SIC summaries.
Study Type
INTERVENTIONAL
Allocation
RANDOMIZED
Purpose
OTHER
Masking
NONE
Enrollment
60
A Large Language Model-based platform is integrated with the Health Vision Platform, a HIPAA-compliant, electronic health record data management system. Real-time summaries of prior SIC with be sent to clinicians caring for participants in the intervention arm. An email will be sent to the inpatient attending and responding clinician encouraging the team to review the SIC summary, discuss care preferences with the patient and ensure care is aligned with preferences and goals. If no SIC documentation is found, the email will instead encourage the outpatient oncologist to share information regarding undocumented SICs that may have occurred, as well as prompt inpatient and outpatient teams to engage the patient in SIC. For patients with no SIC documentation who are still admitted 72 hours later, the RA will send an additional email prompting SIC or asking the clinician to indicate that an SIC is not appropriate. Patients will also receive standard of care.
Brigham and Women's Hospital
Boston, Massachusetts, United States
Dana-Farber Cancer Institute
Boston, Massachusetts, United States
Serious Illness Conversation (SIC) summary accuracy
Primary outcome: We will assess SIC summary accuracy by reviewing the source documentation for each bullet point of the 5 domains of the SIC summary. The RA will read each bullet point, read the source note for the date cited in the summary and indicate with a 1 (supported) or 0 (not supported) whether the documentation supports the summarized point. If a summarized point is scored as 0, the MPIs will independently review the item and assign a 1 or 0 to indicate whether the unsupported summary could negatively impact patient care. SIC summary accuracy will be determined at the patient level as the proportion of bullet points supported by underlying documentation divided by the total number of bullet points (i.e., a score from 0 to 1). We will determine the average SIC accuracy score across all patients for the study. We will also report the percent of LLM-SIC summary statements that are unsupported and clinically significant and provide qualitative descriptions of these instances.
Time frame: 1 week
Feasibility (successful delivery of the intervention)
Feasibility is defined as the successful delivery of the summary email, and will be calculated as the number of participants with successful delivery of the intervention email divided by the total number of intervention participants, with an 80% feasibility threshold.
Time frame: 1 week
Clinician Perspectives of Clinical Utility
Clinician perspectives of clinical utility are defined as clinicians' views on the usefulness and relevance of serious illness communication (SIC) and, for intervention-arm clinicians, the LLM-generated SIC summaries for informing participant care. These perspectives will be assessed via structured REDCap surveys sent to both inpatient and outpatient clinicians, with follow-up phone interviews conducted by the research assistants for non-respondents. Clinician responses to surveys and interview questions will be summarized and stratified by intervention vs control groups. We will compare survey responses between the two study arms using Chisquared and Mann-Whitney U tests for binary and likert items, respectively. Responses to open-ended questions will be qualitatively summarized.
This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.
Time frame: 1 week
Participant Perspectives on Inpatient and Outpatient Clinician Understanding of their Care Preferences and Acceptability of SIC Summaries
Participant acceptability is defined as participant perceptions of the appropriateness and usefulness of LLM-generated summaries of serious illness conversations (SICs). It will be assessed through structured in-person interviews including the validated "Heard and Understood" questionnaire and open-ended questions regarding the acceptability of SIC summaries and participant-clinician communication. Participant responses to interview questions will be summarized and stratified by intervention vs control groups. Responses to open-ended questions will be qualitatively summarized.
Time frame: 1 week
New Documentation of SIC
New documentation of serious illness communication (SIC) during the index hospitalization is defined as any new note or entry created in the electronic health record (EHR) during the current hospital admission that reflects a discussion of the participant's values, goals, preferences, or prognosis, and meets the standardized codebook criteria for SIC.
Time frame: 1 week