Diagnostic Accuracy of Two Large Language Models in Turkish Emergency Department Anamnesis Notes

Overview

This retrospective diagnostic accuracy study evaluates two large language models - GPT-4.1 (gpt-4.1-2025-04-14; OpenAI) and Claude Sonnet 4.6 (claude-sonnet-4-6; Anthropic) - as retrospective coding-quality instruments applied to anonymized Turkish-language emergency department anamnesis notes. The reference standard is the majority consensus of three board-certified emergency medicine specialists who independently coded each note in ICD-10, blinded to one another, to the code entered by the treating physician at case closure, and to the subsequent clinical course. Cases without chapter-level majority agreement are excluded without replacement. Both models are queried once per note with a single locked prompt at temperature 0 in stateless application programming interface calls, with no retrieval augmentation, no external tools and no extended-reasoning mode. The primary outcome is the proportion of cases in which each model's rank-1 diagnosis matches the reference standard at ICD-10 chapter level, reported with a Wilson 95% confidence interval. Registered secondary outcome measures are chapter-level Cohen's kappa between each model's rank-1 diagnosis and the reference standard; top-3 chapter accuracy for each model; and chapter-level concordance between the closure ICD-10 code and the reference standard. Additional prespecified analyses set out in the statistical analysis plan (paired between-model difference, three-character accuracy, note-length association, confidence calibration and model-to-model agreement) are reported in the primary publication. The ICD-10 code entered at case closure is characterised against the same reference standard as a description of current documentation practice; it is not a comparator, and no test of superiority or inferiority against model output is performed. The analysis plan was finalised and frozen before any accuracy computation. Reporting follows STARD-AI 2025.

STUDY DESIGN: Retrospective diagnostic accuracy study, STARD-AI 2025 reporting, single centre, cohort design. AI INDEX TESTS: (1) GPT-4.1 (model version gpt-4.1-2025-04-14; OpenAI API). (2) Claude Sonnet 4.6 (model version claude-sonnet-4-6; Anthropic API). Both accessed via the providers' developer application programming interfaces from Python. Temperature = 0. Zero-shot direct prompting with a single locked prompt version; stateless single-turn sessions with no cross-case context, no retrieval augmentation, no external tools and no extended-reasoning mode. No task-specific fine-tuning or additional training was applied; the models were used as released. MODEL INTERPRETABILITY: Interpretability analyses such as SHAP, Grad-CAM or layer-attribution visualisation are not applicable to this study. Because GPT-4.1 and Claude Sonnet 4.6 are accessed as black-box models through proprietary, closed-source commercial interfaces, internal weights, gradients and attention structures are inaccessible for post-hoc interpretability computation. REFERENCE STANDARD: Three board-certified emergency medicine specialists independently assess each anonymized note, blinded to one another, to the code entered by the treating physician, and to the subsequent clinical course. The primary diagnosis assigned by at least two of three assessors, reduced to ICD-10 chapter level, constitutes the reference standard. Cases in which all three assessors assign different chapters are excluded without replacement. No joint calibration session was held and no adjudication round was performed; each assessor coded once, according to their own clinical judgement. DATA PRIVACY: All anamnesis notes are de-identified before processing; direct patient identifiers are removed and no patient name is present in any note at any stage. Each case carries a study-specific sequential number that is not a hospital record number, and no file linking study numbers to patient identities was created or retained. Note text is transmitted to commercial application programming interfaces operated by providers established outside Turkiye; all queries are issued through the providers' developer interfaces in stateless single-turn calls, and no patient identifier is present in any submitted text. De-identified notes are stored in an encrypted, access-restricted database. Conducted in accordance with Turkish Personal Data Protection Law no. 6698. REPRODUCIBILITY OF THE INDEX TEST: The statistical analysis plan specified a test-retest assessment of within-model reproducibility. A random subset of 20 cases was drawn from the analysis set with a fixed seed recorded before the re-run, and both models were re-queried on those notes on 7 August 2026, after an interval of 3 days and 18 hours from the primary run (protocol minimum 48 hours), using the same prompt content, the same model identifiers and the same sampling parameters; the single-query-per-note statement above refers to the primary run. The prespecified measure is the proportion of cases in which the rank-1 code is identical between runs, at three-character and at ICD-10 chapter level. Two conditions differed from the primary run and are recorded in the deviation log: the byte-exact prompt file used in the primary run could not be recovered, only its SHA-256 digest having been retained, so the re-run used a prompt of identical content but unverified byte identity; and the structured-output mechanism for GPT-4.1 was JSON schema mode at re-run rather than the JSON object mode used originally, which the API rejected. The analysis is therefore reported as consistency of re-execution rather than strict prompt-identical reproducibility. This re-run is the last date of data collection and determines the study completion date. STUDY DATES: The Actual Study Start Date (1 May 2026) denotes the beginning of the retrospective encounter window from which archived notes were drawn, not the start of data collection. Ethics approval (Clinical Research Ethics Committee of Marmara University, protocol 09.2026.26-0514) was granted on 14 May 2026. Because the study is retrospective, every note analysed was already present in the hospital record system when it was retrieved; no data were generated prospectively and no patient was enrolled. STUDY FLOW: 630 consecutive eligible notes were screened and coded by all three assessors. Ten notes (cases 621-630) fell beyond the ethics-approved ceiling of 600 analysable cases and were excluded before analysis, leaving an assessment window of 620 on which inter-assessor agreement is reported. Within that window 20 notes had no chapter-level majority among the three assessors and were excluded without replacement, giving a primary analysis set of 600. A further 4 notes had no majority three-character code, giving 596 for the secondary three-character analysis. STATISTICAL ANALYSIS: Analyses are performed in Python 3.11 (pandas, statsmodels, scipy) following a statistical analysis plan finalised and frozen before any accuracy computation; selected estimates are independently recomputed in jamovi by a second investigator using a prespecified verification checklist. PATIENT AND PUBLIC INVOLVEMENT: Not applicable. This retrospective study uses existing anonymized records; there was no patient or public involvement in design or conduct. DATA SHARING: De-identified data are available from the principal investigator on reasonable request, subject to institutional approval. Eight supplementary files are provided with the primary publication: the statistical analysis plan with its deviation log; the full prompt text with its recorded SHA-256 digest; the data-preparation and analysis code; the data dictionary; the completed STARD-AI 2025 reporting checklist; the jamovi verification checklist used for independent recomputation of selected estimates; the technical specification of the two index tests; and the full chapter-level confusion matrices for both models.