Effectiveness of AI-Assisted Antibiotic Prescribing

Overview

This randomized controlled trial aims to evaluate the impact of providing LLM access to physicians on antibiotic prescribing appropriateness, using a composite outcome instrument, the Antibiotic Prescribing Appropriateness Score (APAS), that simultaneously evaluates antibiotic selection, dosing, duration, clinical reasoning, and management planning. Participants will be randomly assigned to one of two groups: the intervention group will have access to an LLM alongside conventional resources, while the control group will use conventional resources only (e.g. UpToDate, PubMed and Google Search with AI features disabled). Both groups will respond to a set of clinical vignettes covering common infectious disease scenarios requiring antibiotic prescribing decisions, with responses evaluated using an expert-validated grading rubric and independently scored by blinded raters.

Inappropriate antibiotic prescribing remains one of the most consequential drivers of antimicrobial resistance globally, with particular urgency in low- and middle-income countries, where regulatory enforcement, antibiogram coverage, and specialist infectious-disease support are limited. Recent advances in Large Language Models (LLMs) have shown promise in supporting clinical decision-making. LLMs have been explored as tools for synthesizing antibiotic recommendations and flagging spectrum mismatches. Yet LLMs may produce confident but incorrect recommendations, and their use may reduce the quality of physicians' own clinical reasoning through over-reliance. Whether this promise translates into measurably better antibiotic prescribing behavior among practising physicians, measured across the full spectrum of selection, dosing, and clinical reasoning, has not been rigorously evaluated in a randomized, controlled setting. This study evaluates whether access to an LLM affects the Antibiotic Prescribing Appropriateness Score (APAS) among licensed physicians in Pakistan. Secondary objectives include assessing whether the intervention improves antibiotic selection and spectrum-matching, whether the effect differs by physician clinical experience, and whether LLM access affects the time physicians spend per vignette. This study will be an assessor-blind, randomized controlled trial with two arms. Participants will be randomly assigned to either the intervention or control arm in a 1:1 ratio. Outcome assessors (those scoring the APAS responses) will be blinded to participants' group assignments and will evaluate vignette responses without knowledge of which arm the responding physician was in. Each participant is expected to complete 6-10 vignettes in a single, proctored, 75-minute session. The order of vignettes will be randomized independently for each participant to control for order effects. The time spent on each vignette will be automatically recorded. In the intervention arm, participants will have access to ChatGPT in addition to conventional diagnostic and reference resources (UpToDate, PubMed, Google search with AI features disabled). In the control arm, participants will use conventional resources only (UpToDate, PubMed, Google Search with AI features disabled). Both arms will evaluate the same set of clinical vignettes. Participants will be presented with clinical vignettes covering common infectious disease scenarios requiring antibiotic prescribing decisions. Vignettes will be sourced and modified based on clinically realistic cases relevant to the target population, and will follow a standardized format, including the patient's chief complaint and history of present illness, and physical examination findings. Vignettes will cover a range of infectious disease presentations encountered in family medicine, general practice, internal medicine, and emergency medicine, avoid overly rare or highly specialized conditions, and reflect a range of prescribing decisions, including empiric treatment, culture-directed de-escalation, spectrum selection, dosing decisions, and stewardship-sensitive presentations in which withholding antibiotics may be the clinically correct decision, to avoid reward bias toward prescribing. All vignettes will be developed by an independent expert panel, and each vignette will be accompanied by an expert-validated grading rubric specifying expected answers for antibiotic selection, dosing, route, and duration, clinical reasoning, and management planning. The primary measure of prescribing appropriateness will be the APAS, a four-domain rubric scored out of 10 points per vignette and expressed as a percentage. The adjusted mean difference in vignette-level APAS between the intervention and control arms is the primary outcome of our study. Responses will be independently evaluated by three qualified raters blinded to participant identity and treatment assignment. The arithmetic mean of the three raters' total scores will constitute the vignette-level APAS used in the primary analysis.