Clinical Validation of Vital-PICASO for Predicting Hypoxia Within 24 Hours in General Ward Inpatients

Huinno AIM2,250 enrolled

Overview

The purpose of this retrospective study is to evaluate the clinical performance of Vital-PICASO, an artificial intelligence-based biological signal analysis software designed to predict the risk of hypoxia within 24 hours using vital-sign data from adult general ward inpatients. Electronic medical record data from patients aged 19 years or older who were admitted to a general ward at Seoul National University Hospital will be retrospectively reviewed. Hypoxia will be evaluated using two separate reference-standard criteria: oxygen saturation below 94% and administration of oxygen at 4 L/min or more. Eligible hypoxia-positive and hypoxia-negative datasets will be randomly selected. The selected vital-sign datasets will be analyzed using Vital-PICASO while the device operator is blinded to the reference-standard classification. The software-generated hypoxia risk scores will be compared with the reference-standard classifications to evaluate predictive performance. Because the study uses previously collected medical records, there is no direct participant contact and the software results will not affect patient care.

This study is a retrospective, single-center, superiority-confirmatory clinical performance study conducted at Seoul National University Hospital. Data Source and Study Population Electronic medical records will be reviewed for male and female patients aged 19 years or older who were admitted to a general ward for at least 12 hours between January 1, 2023, and May 31, 2025. Clinical data previously used in the development of Vital-PICASO will be excluded. The collected variables will include systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, body temperature, oxygen saturation, and the corresponding measurement times. Additional variables will include the occurrence, time, and location of hypoxia; oxygen administration status, flow rate, delivery method, and start time; month and year of birth; sex; height and weight when available; do-not-resuscitate status; chronic obstructive pulmonary disease status; pregnancy status; date of admission to the general ward; clinical department; and the vital-sign data collection period. Reference-Standard Groups Hypoxia will be evaluated using two separate reference-standard criteria: 1. Oxygen saturation criterion: SpO2 below 94%. 2. Oxygen-treatment criterion: administration of oxygen at 4 L/min or more. For hypoxia-positive datasets, vital-sign data collected for up to 72 hours before the first eligible hypoxia event will be extracted. If hypoxia occurs within 24 hours after admission or transfer to the general ward, all available vital-sign data recorded before the event will be collected. If a patient experiences multiple eligible hypoxia events, only the first eligible event will be included for each reference-standard criterion. A patient who meets both hypoxia criteria may contribute a dataset to both positive-group analyses. Hypoxia-negative datasets will be selected from patients who did not experience hypoxia during the selected general ward admission. Negative datasets will be randomly selected from the same general wards as the positive datasets and from a time interval comparable with the hypoxia-event reference time or from up to 72 hours after admission to the general ward. Only one eligible data interval will be selected per patient. Data Selection and Reference-Standard Establishment To reduce selection bias, candidate positive and negative datasets will be numbered and provided to an independent statistician. The statistician will use R or SAS software to randomly select the required number of datasets and generate a randomly ordered identification-code list. The initial positive or negative classification will be determined according to the predefined oxygen saturation or oxygen-treatment criterion. To improve the reliability of the study dataset, a clinician with at least 1 year of clinical experience will review the data collection reference time and relevant patient information while blinded to the initial positive or negative classification. The clinician's final determination will be used as the reference standard. Application of Vital-PICASO The medical device operator will receive vital-sign datasets identified only by screening numbers and will remain blinded to the reference-standard classification. Each dataset will be analyzed using Vital-PICASO to estimate the risk of hypoxia within 24 hours. The peak risk score and corresponding time within the analysis interval will be recorded. Performance Evaluation The software-generated risk scores will be compared with the reference-standard classifications. The two primary performance measures will be: 1. Area under the receiver operating characteristic curve for predicting hypoxia defined as SpO2 below 94%. 2. Area under the receiver operating characteristic curve for predicting hypoxia defined as oxygen administration at 4 L/min or more. Each AUROC will be compared with the prespecified comparator performance value of 0.7891. The success criterion for each primary outcome will be met if the lower bound of the 95% confidence interval for the difference between the Vital-PICASO AUROC and the comparator AUROC is greater than zero. The overall study will be considered successful if both primary outcomes meet their respective success criteria. Secondary performance measures will include area under the precision-recall curve, precision, recall, F1 score, threshold-specific performance, performance by prediction-time interval, cumulative-time performance, sensitivity, specificity, positive predictive value, and negative predictive value. Planned Analysis Datasets For the SpO2-below-94% criterion, 2,250 datasets are planned, including 1,125 positive and 1,125 negative datasets. For the oxygen-at-4-L/min-or-more criterion, 546 datasets are planned, including 273 positive and 273 negative datasets. Positive datasets may overlap between the two analyses, and the negative datasets may be used in both analyses. Because the study uses previously collected electronic medical record data, participants will not undergo any additional examination, treatment, or change in clinical care.

Eligibility

Sex: ALLMin age: 19 Years

Medical Language ↔ Plain English

Inclusion Criteria: General Criteria * Male or female patient aged 19 years or older at the time the clinical data were recorded * Electronic medical record data available between January 1, 2023, and May 31, 2025 * Admission to a general ward for at least 12 hours * Clinical data not previously used in the development of Vital-PICASO SpO2-Based Hypoxia-Positive Group * Hypoxia occurred in a general ward * Hypoxia was defined as oxygen saturation below 94% * Vital-sign data were available for up to 72 hours before the hypoxia event * If hypoxia occurred within 24 hours after admission or transfer to the general ward, all available vital-sign data recorded before the event were included * If multiple eligible hypoxia events occurred during the data collection period, only the first eligible event was included for this criterion SpO2-Based Hypoxia-Negative Group * No hypoxia occurred during the selected general ward admission * Data were randomly selected from the same general wards as the corresponding hypoxia-positive group * Vital-sign data were available from a time interval comparable with the hypoxia-event reference time or for up to 72 hours after admission to the general ward * Only one eligible data interval was included per patient Oxygen-Treatment-Based Hypoxia-Positive Group * Hypoxia occurred in a general ward * Hypoxia was defined as oxygen administration at a flow rate of 4 L/min or more * Vital-sign data were available for up to 72 hours before the initiation of oxygen administration * If oxygen administration began within 24 hours after admission or transfer to the general ward, all available vital-sign data recorded before the event were included * If multiple eligible events occurred during the data collection period, only the first eligible event was included for this criterion Oxygen-Treatment-Based Hypoxia-Negative Group * No hypoxia occurred during the selected general ward admission * Data were randomly selected from the same general wards as the corresponding hypoxia-positive group * Vital-sign data were available from a time interval comparable with the hypoxia-event reference time or for up to 72 hours after admission to the general ward * Only one eligible data interval was included per patient Exclusion Criteria: * Hypoxia occurred outside a general ward * The patient had chronic obstructive pulmonary disease * A do-not-resuscitate order was in effect at the time the clinical data were recorded * The patient was pregnant or within 3 months after delivery at the time the clinical data were recorded * The data were considered inappropriate for the study by the principal investigator

Outcomes

Primary Outcomes

Area Under the Receiver Operating Characteristic Curve for SpO2-Based Hypoxia Prediction

The area under the receiver operating characteristic curve (AUROC) is calculated by comparing the hypoxia risk scores generated by Vital-PICASO with the reference-standard classification for the SpO2-based analysis. Hypoxia-positive cases are defined as cases in which oxygen saturation below 94% occurred in the general ward. Hypoxia-negative cases are selected from patients who did not experience hypoxia during the corresponding general ward admission period. AUROC ranges from 0 to 1, with a higher value indicating better discrimination between hypoxia-positive and hypoxia-negative cases. The prespecified success criterion is met if the lower bound of the 95% confidence interval for the difference between the Vital-PICASO AUROC and the comparator AUROC of 0.7891 is greater than 0.

Time frame: Within 24 hours before the SpO2-based hypoxia event or during the corresponding reference period for negative cases

Area Under the Receiver Operating Characteristic Curve for Oxygen-Treatment-Based Hypoxia Prediction

The area under the receiver operating characteristic curve (AUROC) is calculated by comparing the hypoxia risk scores generated by Vital-PICASO with the reference-standard classification for the oxygen-treatment-based analysis. Hypoxia-positive cases are defined as cases in which oxygen was administered at a flow rate of 4 L/min or more in the general ward. Hypoxia-negative cases are selected from patients who did not experience hypoxia during the corresponding general ward admission period. AUROC ranges from 0 to 1, with a higher value indicating better discrimination between hypoxia-positive and hypoxia-negative cases. The prespecified success criterion is met if the lower bound of the 95% confidence interval for the difference between the Vital-PICASO AUROC and the comparator AUROC of 0.7891 is greater than 0.

Time frame: Within 24 hours before oxygen administration at 4 L/min or more or during the corresponding reference period for negative cases

Secondary Outcomes

Area Under the Precision-Recall Curve for SpO2-Based Hypoxia Prediction

The area under the precision-recall curve (AUPRC) is calculated using the Vital-PICASO risk scores and the reference-standard classifications for hypoxia defined as oxygen saturation below 94%. AUPRC summarizes the relationship between precision and recall across possible risk-score thresholds. A higher value indicates better predictive performance.

Time frame: Within 24 hours before the SpO2-based hypoxia event or during the corresponding reference period for negative cases

Area Under the Precision-Recall Curve for Oxygen-Treatment-Based Hypoxia Prediction

The area under the precision-recall curve (AUPRC) is calculated using the Vital-PICASO risk scores and the reference-standard classifications for hypoxia defined as oxygen administration at a flow rate of 4 L/min or more. AUPRC summarizes the relationship between precision and recall across possible risk-score thresholds. A higher value indicates better predictive performance.

Time frame: Within 24 hours before oxygen administration at 4 L/min or more or during the corresponding reference period for negative cases

Precision, Recall, and F1 Score Across Risk-Score Thresholds

Precision, recall, and F1 score are calculated across Vital-PICASO risk-score thresholds. The analyses are performed separately for SpO2-based hypoxia and oxygen-treatment-based hypoxia. Precision is the proportion of predicted positive cases that are reference-standard positive. Recall is the proportion of reference-standard positive cases that are correctly predicted as positive. The F1 score is the harmonic mean of precision and recall. Each measure ranges from 0 to 1, with a higher value indicating better classification performance.

Time frame: Within the 24-hour prediction window before the hypoxia event or corresponding reference period

Predictive Performance by Prediction-Time Interval

The predictive performance of Vital-PICASO is evaluated according to the time interval between the software prediction and the hypoxia event or corresponding reference time point. AUROC, AUPRC, precision, recall, and F1 score may be summarized by prediction-time interval. The analyses are performed separately for SpO2-based hypoxia and oxygen-treatment-based hypoxia.

Time frame: At prespecified time intervals within 24 hours before the hypoxia event or corresponding reference time point

Predictive Performance by Cumulative Prediction Window

The predictive performance of Vital-PICASO is evaluated over cumulative prediction windows before the hypoxia event or corresponding reference time point. AUROC, AUPRC, precision, recall, and F1 score may be summarized for each cumulative prediction window. The analyses are performed separately for SpO2-based hypoxia and oxygen-treatment-based hypoxia.

Time frame: Over cumulative prediction windows of up to 24 hours before the hypoxia event or corresponding reference time point

Sensitivity, Specificity, Positive Predictive Value, and Negative Predictive Value

Classification performance based on the Vital-PICASO risk scores is evaluated using sensitivity, specificity, positive predictive value, and negative predictive value. Sensitivity is the proportion of reference-standard positive cases correctly classified as positive. Specificity is the proportion of reference-standard negative cases correctly classified as negative. Positive predictive value is the proportion of predicted positive cases that are reference-standard positive. Negative predictive value is the proportion of predicted negative cases that are reference-standard negative. The analyses are performed separately for SpO2-based hypoxia and oxygen-treatment-based hypoxia.