AI-Assisted Chest-CT Reporting for Enhanced Speed and Quality (The DOUBLE-ACE Study)

Overview

The goal of this observational study is to learn if an AI assistant tool can help doctors who read chest CT scans (called radiologists) write their reports faster and just as well or better. Chest CT scans are common pictures taken of the inside of the chest to help with diagnosis. The main questions the study aims to answer are: (1) Does using the AI tool save radiologists time when writing their reports? (2) Are the final reports written with the AI tool's help as good as or better than reports written without it? To answer these questions, researchers will compare two time periods at several hospitals. They will look at how long it took to write reports and how good the reports were, both from a time before the AI tool was available and from a time after it was in regular use. In this study, radiologists will use the AI tool as part of their normal daily work. The tool is built into the computer system they already use to look at scans. Researchers will then measure the time and quality of the reports produced during their regular shifts.

Here we provide a summary of the study's methodological framework, including a description of the AI system under evaluation, key quality control measures, and the data analysis plan. Comprehensive details regarding the full protocol, including eligibility criteria and outcome measures, can be found in the other modules of the study protocol. 1. Background on the AI System: The study evaluates a clinically deployed AI-assisted reporting system, built upon an advanced multimodal foundation model trained on a large-scale chest CT dataset. Its performance, reliability, and generalizability have been established through rigorous validation on extensive internal and external datasets. Prior reader studies have demonstrated its clinical utility by significantly reducing reporting time through automated draft generation while maintaining or improving report quality, supporting its integration into real-world workflow for this evaluation. 2. Quality Control: To ensure objective assessment, report quality will be scored by a panel of at least two independent, blinded thoracic radiologists using a standardized rubric, with inter-rater reliability calculated. A study-specific data dictionary and Standard Operating Procedures (SOPs) for data handling and analysis will be implemented to ensure reproducibility and auditability. 3. Data Analysis Plan: A comparative statistical analysis between pre- and post-implementation groups is planned. Appropriate statistical tests (e.g., Mann-Whitney U test, mixed-effects models) will be applied based on data distribution and variable type. A sample size calculation will be conducted to ensure the study is adequately powered to detect a clinically meaningful difference in the primary efficiency endpoint. The primary analysis will be a paired comparison of outcomes (e.g., report time, quality) between the two phases for the participants who complete both. To address potential attrition (e.g., radiologist turnover during the study year) and the influence of radiologist experience, the analysis plan includes: (1) Accounting for and reporting any participant dropout between phases. (2) Conducting pre-specified subgroup or stratified analyses based on radiologist seniority (e.g., junior vs. senior) to examine its effect on the outcomes. 4. Confounding Factor Control: To minimize potential bias, the study may identify collective variables (radiologists' sex, years of relevant professional experience, etc.) that may be considered potential confounding factors according to external experts' judgements. Certain patient-related information, such as diagnosis (infection, malignancy, cardiovascular disease, etc.) and clinical scenario (e.g., inpatient, outpatient, emergency), may also be collected retrospectively, where necessary, to evaluate model performance within specific diagnostic subgroups. The study will adopt multiple possible approaches for confounding factor adjustment or analysis, which may include stratified analyses and other related statistical methods. 5. Potential Adjustments in Study Protocol or Analysis Methods: As the study may involve multiple centers, in case of ethical or administrative restrictions at certain time at specific sites, AI assistance may be temporarily suspended to approximate the scenario without AI assistance at those sites. In such cases, the corresponding results may be reported as the with-AI and without-AI phases, rather than labeling them as baseline and AI-available phases. Furthermore, radiologists who decline to provide demographic or occupational information (e.g., years of professional experience or sex)-variables that may serve as potential confounders-will be excluded from adjusted and stratified analyses that require such covariates. These approaches may need to be incorporated into confounding factor or stratified analyses, and we may update related conditions accordingly when necessary.