Validation of i-ROP DL to Detect More Than Mild ROP

Overview

The purpose of the pivotal reader study is to assess the readers' accuracy in diagnosing plus disease versus no plus or pre-plus disease with or without the aid of the i-ROP DL. Ophthalmologists' performance metrics for the following modalities will be evaluated: * Standard evaluation following the standard of care process ("without i-ROP DL") * Evaluation following the standard of care process with the aid of the i-ROP DL ("with i-ROP DL") This retrospective multi-reader multi-case (MRMC) study will have an enriched sample of approximately 300 eye cases (1 study eye per subject): 60 plus cases, 120 pre-plus cases and 120 no plus cases. Enrichment is with respect to proportions of plus cases and pre-plus cases. The primary objective of this study is to evaluate whether the area under the receiver operating characteristic (ROC) curve (AUC) based on probability scores of plus disease statistically significantly non-inferior or superior with the aid of the i-ROP DL versus without the aid of the i-ROP DL. Multiple secondary endpoints are outlined in the next section.

This clinical research is a retrospective, pivotal, multi-reader, multi-case (MRMC) study with an enriched sample of approximately 300 cases which will be selected from the library of telemedicine cases collected under Telemedicine Approaches to Evaluating Acute Phase ROP (e-ROP) Study. All subjects in the library previously provided written informed consent for inclusion in the e-ROP study. As part of the consent process, subjects agreed that image data and supporting documentation could be used for future research and investigations. Only deidentified data and retinal images will be used in this study. Evaluable cases are defined as subjects who met all protocol case inclusion/exclusion criteria (see Section 6.3.3), and who have a complete set of images for examination, including all 5 recommended fields of view (posterior, nasal, temporal, Superior, and inferior., and have a reference standard diagnosis of acceptable for grding of MTM-ROP. . The quality of each image is categorized as good, adequate, poor, or missing, based on focus, clarity, and field definition. Measures to ensure subject and reader confidentiality include assignment of unique study and reader ID numbers. * Sample Size and Case Mix Enriched sample size of approximately 300 cases (subjects) with the following case mix: 60 plus cases, 120 pre-plus cases and 120 no plus cases. * Case Selection and Targeted Distribution The independent statistician will select cases for this pivotal reader study and cases will be randomly selected to provide representative distributions of race across enrolling sites, to the extent that this can be accomplished using cases in the case library. This clinical research is a retrospective, pivotal, multi-reader, multi-case (MRMC) study with an enriched sample of approximately 60 plus cases (subjects), 120 pre-plus cases and 120 no plus cases. * Reference Standard Diagnosis The reference standard diagnosis (RSD) for the 3-class ground truth (no plus, pre-plus, or plus) will be determined at the eye level by the majority classification of 5 ROP experts who will independently classify all eyes in the dataset. In the unlikely case of a 2/2/1 disagreement, the 5 experts will be asked to regrade the eye exam together until there is a majority (at least 3/5 agree). * The RSD Process The RSD experts will review all cases selected for inclusion in the reader study. The RSD experts will complete an eCRF for each case, based on eye images and available clinical information from the image acquisition site. * The Reading Process Approximately 16 ophthalmologists will independently perform two reads on all (approximately 300) cases. Each reader will read each case both without the aid of the i-ROP DL and then with the aid of the i-ROP DL. The first reading will recapitulate the current standard of care with an ophthalmologist evaluating the overall level of ROP severity without the aid of the i-ROP DL using the International Classification of ROP (ICROP) \[1\] criteria for zone, stage, and plus disease. The second reading will evaluate the effect of the i-ROP DL output on plus disease diagnosis using the same criteria. In order to minimize recall bias, the two reading visits will be separated by a memory washout period of approximately four weeks. Background Retinopathy of prematurity is a disease that develops when babies are born prematurely before the retina has fully developed \[1\]. It is a leading cause of childhood blindness worldwide, but blindness is usually avoidable with accurate diagnosis, and timely treatment of babies with treatment-requiring disease \[1\]. Each country has accepted criteria for which babies require eye examinations. In the United States, all babies born less than 31 weeks post-menstrual age (PMA) or under 1500 grams require diagnostic eye examinations starting at 31 weeks PMA or 4 weeks after birth, whichever is later \[2\]. Exams are repeated at 1-2 week intervals depending on the level of disease identified on examination. Diagnosis of ROP is made using either ophthalmoscopy (in person examination at the bedside) or using telemedicine with remote digital fundus imaging \[2\] \[3\]. Telemedicine has become an acceptable standard of care for ROP diagnostic examinations, and in most telemedicine workflows, exams are repeated weekly to minimize the risk of missing ROP. Whether exams occur in person or via telemedicine, the ophthalmologist is responsible for documenting the presence or absence of ROP using the International Classification of ROP (ICROP) guidelines, including documenting the presence and degree of zone, stage, and plus disease \[4\]. Current treatment guidelines suggest that urgent treatment (within 48 hours) should occur when babies develop "type 1" ROP, defined as Zone I stage 3 ROP (with or without plus), or any zone or stage with plus \[5\]. Because zone I stage 3 without plus is rare, these guidelines have established plus disease as the most important clinical feature determining the need for treatment in ROP. Plus disease is generally defined as dilation and tortuosity of the posterior retinal vessels in Zone I relative to a standard published photograph. However, as digital fundus imaging has become more standard, it has become clear that in practice, the diagnosis of plus disease is not always made consistently, with significant inter-observer variability. As a result, the ICROP 3rd edition, published in 2021, acknowledges that pre-plus and plus run a spectrum that is related to the overall severity of ROP in the eye \[4\] \[5\]. Because treatment is urgent when type 1 ROP is diagnosed, when borderline disease is diagnosed in patients, the ophthalmologist responsible for the care of that baby may suggest referral for in person ophthalmoscopic examination and possible treatment, to avoid treatment delay in case of logistical or transportation challenges. For that reason, it is better to diagnose severe ROP before babies progress to type 1 ROP / plus disease. The i-ROP DL system is a screening device for use by trained nurses, photographers, or technicians performing screening eye examinations using the Retcam 3 camera. The output of the device is a "refer" or "repeat" that is assigned for each patient that reflects the spectrum of vascular changes in pre-plus and plus disease \[6\] \[7\]. The output will be sent to an ophthalmologist for review if "more than mild ROP" is detected ("refer"), using the same images analyzed by the algorithm (no separate exam required). The clinician can decide further management based on a standard telemedicine workflow. In cases of "no referral needed", the baby will be scheduled for a follow-up examination in 1 week (typical for telemedicine programs). This ensures that if a baby progresses to more than mild ROP over time there are multiple opportunities to detect it and flag it for clinical review. Preliminary work suggests that babies usually reach the threshold for MTMROP 2-4 weeks prior to diagnosis of TR-ROP, making the risk of a missed case very low. The i-ROP DL system is also an assistive diagnostic device for use by the ophthalmologist performing diagnostic eye examinations using telemedicine review of digital fundus images using the Retcam 3 camera. The output of the assistive diagnostic device is a "vascular severity score" (from 1-9) that is assigned at the eye level that reflects the spectrum of vascular changes in pre-plus and plus disease \[6\] \[7\]. The vascular severity score is calculated based on the underlying probability of a classification of no plus, pre-plus or plus, and corresponds to the relative likelihood a random expert would diagnose plus disease (at the extreme end of the spectrum, 9, 100% of experts would diagnose plus disease) \[7\]. This output has two advantages that may improve the diagnosis of pre-plus and plus disease, and reduce the risk of late treatment for babies in the context of telemedicine. First, higher vascular severity is not only associated with a higher likelihood of plus disease, but also with higher stage and extent of peripheral disease \[6\] \[8\], which may or may not be visible on all the images. On telemedicine review, if the stage of disease is not visible on peripheral images, the ophthalmologist may want to consider either repeating the images or referring the baby for ophthalmoscopic examination. In this case, the "positive" output would not necessarily trigger any therapeutic intervention, but it will aid the ophthalmologist in making the diagnosis. Second, the vascular severity score will aid in the diagnosis and accuracy of plus disease by providing an objective metric, anchored to expert diagnosis of plus disease \[6\] \[7\]. Study Design Part 1: This clinical study is a retrospective, multicenter, observational study to assess the safety and effectiveness of i-ROP DL in screening for ROP in the telemedicine care setting. The study design conforms to Intent to Screen (ITS) paradigm. Part 2: This clinical research is a retrospective, pivotal, multi-reader, multi-case (MRMC) study with an enriched sample of approximately 60 plus cases (subjects), 120 pre-plus cases and 120 no plus cases. * Reference Standard Diagnosis The reference standard diagnosis (RSD) for the 3-class ground truth (no plus, pre-plus, or plus) will be determined at the eye level by the majority classification of 5 ROP experts who will independently classify all eyes in the dataset. In the unlikely case of a 2/2/1 disagreement, the 5 experts will be asked to regrade the eye exam together until there is a majority (at least 3/5 agree). * The RSD Process The RSD experts will review all cases selected for inclusion in the reader study. The RSD experts will complete an eCRF for each case, based on eye images and available clinical information from the image acquisition site. * The Reading Process Approximately 16 ophthalmologists will independently perform two reads on all (approximately 300) cases. Each reader will read each case both without the aid of the i-ROP DL and then with the aid of the i-ROP DL. The first reading will recapitulate the current standard of care with an ophthalmologist evaluating the overall level of ROP severity without the aid of the i-ROP DL using the International Classification of ROP (ICROP) \[1\] criteria for zone, stage, and plus disease. The second reading will evaluate the effect of the i-ROP DL output on plus disease diagnosis using the same criteria. In order to minimize recall bias, the two reading visits will be separated by a memory washout period of approximately four weeks. This clinical research is a retrospective, pivotal, multi-reader, multi-case (MRMC) study with an enriched sample of approximately 300 cases which will be selected from the library of telemedicine cases collected under Telemedicine Approaches to Evaluating Acute Phase ROP (e-ROP) Study. All subjects in the library previously provided written informed consent for inclusion in the e-ROP study. As part of the consent process, subjects agreed that image data and supporting documentation could be used for future research and investigations. Only deidentified data and retinal images will be used in this study. Evaluable cases are defined as subjects who met all protocol case inclusion/exclusion criteria, and who have a complete set of adequate-quality eye examination. The quality of each image is categorized as good, adequate, poor, or missing, based on focus, clarity, and field definition. Measures to ensure subject and reader confidentiality include assignment of unique study and reader ID numbers. * Sample Size and Case Mix Enriched sample size of approximately 300 cases (subjects) with the following case mix: 60 plus cases, 120 pre-plus cases and 120 no plus cases. * Case Selection and Targeted Distribution The independent statistician will select cases for this pivotal reader study and cases will be randomly selected to provide representative distributions of race across enrolling sites, to the extent that this can be accomplished using cases in the case library. Inclusion and Exclusion Criteria All cases for this pivotal validation study will meet the following eligibility inclusion and exclusion criteria. (see also Appendix A for inclusion and exclusion criteria for protocol e-ROP). Only subjects who were consented, met all inclusion criteria and none of the exclusion criteria, and completed the needed work-up to establish the reference standard diagnosis ground truth will be considered evaluable. The inclusion criteria for all cases in the study are as follows: * Eligible subjects under protocol e-ROP, defined as Infants with birth weight (BW) less than 1251 g meeting current ROP screening. * Eye cases meet the image quality requirements to establish the reference standard diagnosis of plus, pre-plus and no plus cases. * Subjects have known reference standard diagnosis ground truth. If the reference standard diagnosis process cannot create an RSD label for plus disease, the eye will not be included in the study. * Image acquired by the following device: RetCam Ophthalmic Imaging Systems 3 (Natus Medical Incorporated). The exclusion criteria for all cases in the study are as follows: * Subjects who are in violation of protocol e-ROP. * Subjects who are in violation of inclusion criteria of this protocol. Case selection and Targeted Distribution All subjects meeting inclusion and exclusion criteria from the e-ROP study will be included in this study. e-ROP study represents the natural occurrence of the plus and pre-plus disease cases in the clinical environment. There is no data enrichment within e-ROP study. Recruitment Methods This study will be conducted with board-certified ophthalmologists of varying experience levels, with at least one year of experience performing ROP examinations. These ophthalmologists will not be associated with the acquisition or review of study cases and did not participate in the RSD process. A survey questionnaire will be sent to prospective readers to determine the fraction of their professional time devoted to ROP diagnosis. Investigators will invite prospective readers through the listserv of the American Academy of Pediatric Ophthalmology and Strabismus and the Retina Society. Approximately 16 board-certified ophthalmologists, with a range of experience based on the years of experience, will be selected to read and score the study cases. ROP reading experience will be categorized into three groups: * High Experience Reader - defined as ophthalmologists who have \>10 years' experience of reading ROP images. * Medium Experience Reader - defined as ophthalmologists who have 5-10 years' experience of reading ROP images. * Low Experience Reader - defined as ophthalmologists who have 1-4 years' experience of reading ROP images. The inclusion criteria (qualification criteria) for all readers in the study are as follows: * U.S. board-eligible/board-certified and licensed Medical Doctor, Doctor of Ophthalmology. (American Board of Ophthalmology or equivalent certification) * Signed reader study agreement. * Signed informed consent. * Successful training on the study protocol and the use of the study software. Consent Process Each reader must provide written informed consent before participating in the reader study of this protocol. Because readers are located all of the world, the investigators will provide electronic consent through docusign. Procedures Involved The Reading Process Purpose The purpose of the reading process is for readers to provide the assessment categories of plus, pre-plus, and no plus, probability scores of plus and pre-plus or worse on the electronic case report forms for their interpretation of the study cases with and without the aid of the i-ROP DL. Conduct of the Reading Process Following the successful completion of the reader training, the readers will independently review and score each of the cases of the reader study case set. Clinical information about the subjects including birth weight, gestational age, and post-menstrual age at eye exam will be provided to the readers. Each reader will work at a designated reading station and will independently perform two reading sections on all cases. Each reader will read each case both as a standard reading, and a reading with the aid of the i-ROP DL. Each reader will read all cases as the standard reading during the first visit. The reader will then read the cases with the aid of the i-ROP DL during the second visit. In order to minimize memory recall bias, the two reading sections will be separated by a memory washout period of approximately four weeks. The reading station will provide preloaded cases from the server through secured web connection. The reading time allowed per case will not be limited. A lunch break will be offered during each reading session, and readers will be encouraged to take a 5-10 minute break each hour, as needed. Each reader will enter their interpretations directly into the electronic case report form. Each reader will sign an attestation form verifying the documented information. Core Reading Center The reading process will be conducted remotely at reader's home or office, as they would be in intended use case of telemedicine evaluation. Preparation of Cases for Interpretation To maintain subject confidentiality, no identifying subject data will be displayed. Unique image identifiers associated with the case ID number will be visible on each image for tracking purposes. To maintain reader confidentiality, a unique ID number will be assigned to each reader participating in the pivotal reader study for entry on the electronic case report form. Information collected during the study will be reported in such a way that it precluded identification of any image acquisition centers or readers' performance.