During this observational study, the investigators aim to assess the ability of ICU clinicians to predict the risk of impending organ failure and retrospectively compare it to the performance of previously published machine learning models. The central hypothesis of this study is that the treating physician can predict impending organ failure in adult ICU patients with similar accuracy as the best previously publishes machine learning models.
In this observational study, clinician's (physicians and nurses) assessment of the estimated imminent organ failure risk in an ICU setting are prospectively collected. Circulatory failure is investigated in the primary objective, and respiratory failure, renal failure, and mortality are investigated in secondary objectives. These assessments investigate the predictive performance and influencing factors for clinician prediction. The assessments will be collected in questionnaires and be performed by the clinicians directly involved in the patient treatment and by clinicians who are not actively responsible for the patient treatment. Furthermore, this study aims to benchmark these risk assessments made by healthcare professionals against retrospectively generated AI risk scores for the same patients and timepoints. The AI risk scores will be calculated retrospectively from a set of models from a systematic search of the current literature. The AI models that will be employed for this analysis will be identified as indicated by a systematic review protocol and must satisfy the following two criteria: they do not require any data beyond what is routinely collected during an ICU stay and may be accessed as open source. Such a comparison is vital for the understanding of the relative accuracy and reliability of AI-based predictions in the context of organ failure risk compared to human performance. The data and findings from this study are anticipated to provide evidence for the clinical utility of AI-based risk scores and pave the way for future research into the optimization of AI systems for healthcare applications.
Study Type
OBSERVATIONAL
Enrollment
499
University Hospital Inselspital, Berne
Bern, Canton of Bern, Switzerland
Clinician prediction of circulatory failure within 8 hours compared to published ML models
This outcome compares the area under the receiver operating characteristic curve (auROC) for two methods of predicting circulatory failure within 8 hours of each assessment time point: (1) ICU clinicians' risk estimates, and (2) previously published machine learning (ML) models applied retrospectively. For each assessment, we compute the auROC separately for clinicians and for the ML model for the same time points and patients. The difference in auROC (clinician minus ML) is the main measure of interest, evaluated under a non-inferiority framework with a margin of 0.025.
Time frame: Assessments are collected within the first 72 hours following admission.
Clinician prediction of respiratory failure within 24 hours compared to published ML models
This outcome compares the area under the receiver operating characteristic curve (auROC) for two methods of predicting respiratory failure within 24 hours of each assessment time point: (1) ICU clinicians' risk estimates, and (2) previously published machine learning (ML) models applied retrospectively. For each assessment, we compute the auROC separately for clinicians and for the ML model. The difference in auROCs (clinician minus ML) is the main measure of interest, evaluated using the same methodological framework as the primary outcome.
Time frame: Assessments are collected within the first 72 hours following admission.
This platform is for informational purposes only and does not constitute medical advice. Always consult a qualified healthcare professional.