Physician Diagnosis of Neurologic Cases With Large Language Models

Capital Medical University40 enrolled

Overview

This study is a prospective randomized controlled trial that aims to evaluate whether large language model (LLM) assistance improves physicians' diagnostic performance compared with conventional assistance.

The study population consists of licensed physicians who have completed at least two years of neurology training. Participants are randomly assigned in a 1:1 ratio to the LLM assistance group or the conventional assistance group. In the LLM assistance group, physicians are assisted by an in-house LLM and are allowed to use conventional resources (e.g., search engines, UpToDate, or clinical guidelines) before submitting a final diagnosis. Access to any other LLMs is not permitted. In the conventional assistance group, physicians use only conventional resources and do not have access to the in-house LLM or any other LLMs at any point during the study. The primary outcome is top-1 diagnostic accuracy. Secondary outcomes include top-3 diagnostic accuracy, the time required to complete each case, and physicians' diagnostic confidence.

Study Type

INTERVENTIONAL

Allocation

RANDOMIZED

Purpose

DIAGNOSTIC

Masking

SINGLE

Enrollment

Conditions

Neurology

Interventions

LLM assistanceOTHER

Physicians are assisted by an in-house large language model and are allowed to use conventional resources. Access to any other LLMs is not permitted.

Eligibility

Sex: ALLHealthy volunteers:

Medical Language ↔ Plain English

Inclusion Criteria: * Licensed physicians who have completed at least two years of neurology training. * Willing to participate in the study and able to provide written informed consent. Exclusion Criteria: * Physicians who have previously participated in the development of the in-house LLM or the construction of datasets used in this study. * Physicians who were involved in the clinical care of patients whose cases were used to construct the case dataset for this study. * Physicians who are known to have previously reviewed or been exposed to the specific cases used in this study.

Locations (3)

Beijing Luhe Hospital affiliated to Capital Medical University

Beijing, Beijing Municipality, China

Xuanwu Hospital Capital Medical University

Beijing, Beijing Municipality, China

Nanyang Central Hospital

Nanyang, Henan, China

Outcomes

Primary Outcomes

Top-1 diagnostic accuracy

Top-1 diagnosis is considered correct if the physician's final top-ranked diagnosis matches the reference diagnosis or a clinically similar diagnosis judged as equivalent by neurologists.

Time frame: Baseline

Secondary Outcomes

Top-3 diagnostic accuracy

Top-3 diagnosis is considered correct if the reference diagnosis or a clinically similar diagnosis judged as equivalent by neurologists appears among the physician's top three final diagnoses.

Time frame: Baseline

Time spent on diagnosis

Time participants spend per case.

Time frame: Baseline

Diagnostic confidence

Physicians' diagnostic confidence will be measured using a five-point Likert scale (1=very unsure, 5=very sure).

Time frame: Baseline

Data from ClinicalTrials.gov

This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.