An Evaluation of the Effectiveness of e-IMCI Implementation in Primary Health Care Clinics in South Africa.

Overview

The research hypothesis is that sick children attending primary health care (PHC) clinics who are managed by IMCI-trained health workers (HWs) using electronic Integrated Management of Childhood Illness guidelines (e-IMCI) receive better quality of care compared to children managed by HWs using conventional paper-based IMCI (pIMCI). The aim of the study is to evaluate the effectiveness of e-IMCI to improve care for sick children under five years attending PHC clinics in one district in KwaZulu-Natal, South Africa. Objectives: 1. To assess feasibility and acceptability of eIMCI implementation in PHC clinics 2. To compare clinic-based management of sick children using e-IMCI with a gold standard IMCI assessment, and those managed using p-IMCI to a gold standard IMCI assessment 3. To determine the cost effectiveness of e-IMCI compared to p-IMCI implementation in PHC clinics Primary outcomes: * Proportion of sick children receiving all medications indicated among children managed by HWs using eIMCI and HWs using pIMCI. * Proportion of sick children with risk/high risk of Tuberculosis, HIV or HIV exposed, and/or malnutrition correctly identified among children assessed using eIMCI and children assessed using pIMCI, compared to a gold standard IMCI assessment. * Incremental cost-effectiveness of eIMCI implementation vs standard of care (pIMCI). The study will employ a prospective two-arm cluster randomized controlled trial. Sample size: a total of 30 clinics in one district will be randomly selected to participate and allocated to the intervention (eIMCI) group (n=15) and control (pIMCI) group (n=15). One IMCI trained HW will be randomly selected from each clinic to participate. Six observations will be conducted with each participating health worker Intervention HWs will receive an IMCI update and computer training based on eIMCI. Control HWs will receive a similar update using pIMCI. Both groups will receive support visits and intervention HWs will receive additional computer/IT support. Health worker knowledge will be assessed pre and post training using a self-administered questionnaire. Quality of care will be assessed in both groups using exit interviews with mothers and review of child health records. In addition, gold standard IMCI assessments will be conducted by an IMCI expert to determine correct findings. Assessment and management of the child by the IMCI expert will be compared to that of the participating HW to determine quality of care provided.

Research Hypothesis The research hypothesis is that sick children attending primary health care (PHC) clinics who are managed by IMCI-trained health workers (HWs) using electronic Integrated Management of Childhood Illness guidelines (e-IMCI) receive improved quality of care compared to children managed by HWs using conventional paper-based IMCI (pIMCI). Aim To evaluate the effectiveness of e-IMCI to improve care for sick children under five years attending PHC clinics in one district in KwaZulu-Natal (KZN), South Africa. Objectives 1. To assess the feasibility and acceptability of implementation of e-IMCI in primary health care clinics 2. To compare the clinic-based management of sick children using e-IMCI with a gold standard IMCI assessment, and those managed using p-IMCI to a gold standard IMCI assessment 3. To determine the cost effectiveness of e-IMCI compared to p-IMCI implementation in primary health care clinics Study oversight Study activities will be overseen by a eIMCI steering group. This will include representatives from KZN Department of Health, UKZN, Ilembe district and ELMA foundation. The steering committee will meet three times per year for the duration of the project to review progress of project activities and provide oversight and advice where appropriate. Research methodology The study will employ a prospective two-arm cluster randomized controlled trial (RCT) to test the hypothesis that implementation of eIMCI will effectively improve quality of care for sick children under five years attending PHC clinics. A total of 30 clinics in one district will be randomly selected to participate, with 15 clinics each randomly allocated to the intervention (eIMCI) and control (pIMCI) groups. One IMCI trained HW will be randomly selected from among all IMCI trained HWs in each clinic to participate in the study. Study site The study will be undertaken in the Ilembe District in KwaZulu-Natal (KZN), South Africa. This district was selected in partnership with the KZN Department of Health. There are four hospitals, including one regional hospital, two community health centres, and 31 primary health care (PHC) clinics. All PHC clinics are reported to have at least one IMCI trained health worker. Ilembe health district is situated in KZN, and contains one small urban area and a large deep rural area, and serves a population of approximately 663 000 people living in around 198 000 households. Ilembe district comprises 4 sub-districts namely KwaDukuza, Ndwedwe, Mandeni and Maphumulo Ilembe is characterised by high rates of poverty and unemployment, and high prevalence of Tuberculosis and HIV. Study population There are two study populations: * IMCI trained HWs * Mothers of sick children aged \< five years attending the clinic. Sampling \& recruitment PHC clinics: Simple random sampling will be used to select 30 PHC facilities to participate in the study from all 31 clinics in Ilembe district. Simple randomization will be used to assign clinics in the intervention or control groups (n=15 per group). Clinics with no IMCI trained nurses will not be eligible to participate and will be replaced. IMCI trained nurses: Simple random sampling will be used to select one IMCI trained nurse in each clinic to participate in the study from all eligible IMCI trained nurses working in the clinic. However, two IMCI trained nurses in selected clinics will participate in the intervention, with the intention that e-IMCI will be implemented in the entire clinic, rather than just selecting a single individual to participate. Mothers of children under five-years: Systematic sampling will be used to select mothers attending the clinic with children who are aged under five years to participate in the study. All mothers with sick children waiting in the queue each day will be identified. In order to reduce observer bias, sampling of children will be undertaken in a systematic way that 1) spaces out the number of children selected, and 2) is unpredictable to reduce the possibility that the observed HW can identify which child will be reviewed. We will finalise this process depending on the size of the clinic, and number of children seen daily, with the aim of observing two- three sick children each day of observation. Observations in each clinic will continue for one week (4-5 days). Sample size Sample size calculation is based on the primary outcome measure, which is the proportion of children who receive all treatments indicated and is powered to detect a difference of 20% in the proportion of children receiving all treatments in the intervention and control groups. Sample sizes of 90 in group one and 90 in group two, which were obtained by sampling 15 clusters (PHC clinics) with 6 subjects (observed children) each in the intervention group, and 15 clusters with 6 subjects each in the control group to achieve 81% power to detect a difference between the group proportions of 0.2000. The group two proportion is 0.7000. The group one proportion is assumed to be 0.7000 under the null hypothesis and 0.9000 under the alternative hypothesis. The test statistic used is the two-sided Z test (unpooled). The significance level of the test was 0.0500. Description of the intervention: The intervention will be undertaken in Ilembe District in KwaZulu-Natal. Training and support visits will be overseen and conducted by implementation staff in partnership with staff from the DoH. Development of an eIMCI/pIMCI training course: An eIMCI training course will be developed with two components 1) basic computer skills training 2) case studies to develop eIMCI skills Structured piloting of an eIMCI training course will be undertaken to ensure that participants receive enough training to reach competency. the pIMCI training will be developed with the same case studies using a paper-based approach to ensure that study participants receive exactly the same IMCI training in intervention and control clinics, so that the groups differ only in respect of how IMCI is used (computer based vs paper-based). Deployment of computers: Ahead of implementation of the eIMCI intervention computers and printers will be purchased and deployed in the child health/IMCI consulting room in all intervention clinics. This computer will be available for use of any IMCI trained health worker consulting with sick children during the intervention period. Facilitators for district based training: Training will be conducted by implementation staff assisted by local DoH trainers in Ilembe district. These trainers will be trained ahead of the district training using the course developed in the pilot. Although district DoH staff may assist with training we plan to involve as few staff members as possible to mimimise bias and contamination given that many DoH staff are routinely visiting both intervention and control clinics. Training of participating health workers (eIMCI and pIMCI): Two IMCI trained health workers will receive training in each clinic (N= 30 HWs in each arm). This will be the selected participant plus another IMCI trained health worker in the clinic (if there is one available). Participants will receive either a paper-based update of pIMCI or eIMCI training. Both computer skills and IMCI knowledge will be assessed at beginning and end of the training. Support visits (IMCI and IT) Following the training, there will be two IMCI support visits for all IMCI trained health workers (eIMCI and pIMCI). This will assist eIMCI participants to transfer their new skills to the workplace and will address any initial problems of participants. This will be a structured visit using case studies and will be mirrored for pIMCI participants using paper to ensure participants in both groups receive similar support. A cellphone based group (WhatsApp group) will be set up to provide support for IT problems to eIMCI participants if they experience any problems with the computer or the programme. If IT problem cannot be resolved over the phone an IT support specialist will visit the clinic to assist the participant. The IT support role will be conducted by external IT experts . Any study personal who are involved in the evaluation of the study will be separate from the training / intervention component. Data collection (quantitative) Preparation: Before the start of the RCT the study team will work with the Maternal Child and Womens' health directorate and with the Ilembe District Health Management Team to prepare the district for implementation of the research study. All participating clinics will be visited before the start to inform the operational manager about the study, identify IMCI trained health workers who are eligible to participate, and thereby create a sampling frame from which IMCI trained HWs will be selected. Background information will be collected about provision of child health services at each clinic and resources available at the clinic, for example equipment, staff, drugs etc including any history of recent drug stock outs at the clinic. The appropriate site for deployment of the computer will be identified. A further visit may be required for the purpose of selecting participants and placing the computer in the clinical area. Tracking of implementation of eIMCI: Data collection will commence six weeks after the start of eIMCI/pIMCI implementation in the clinic. This implementation period will allow participants to gain confidence in using eIMCI and to receive support visits. The training and data collection will be staggered to ensure a similar duration of the implementation period among all participants. The following data will be tracked in eIMCI clinics over the six-week period of implementation and for a total of six months post implementation: proportion of children seen by eIMCI trained health workers where the eIMCI system is used; number and nature of IT problems experienced by eIMCI nurses; how IT problems were addressed; number of additional visits for IT support; any stock outs of medication. Field worker selection and training: Data will be collected by a team of data collectors, comprising one expert IMCI practitioner and one field worker, in each of the intervention and control clinics. Data collection teams will collect data either in intervention or control clinics, to minimize bias there will be no cross over between the two groups. The data collection teams will receive training for two weeks on data collection processes, which will include piloting of the data collection techniques in a clinic in another district not implementing eIMCI. In addition, expert IMCI practitioners will receive any updates or refreshers in IMCI assessment that are deemed appropriate to ensure they are able to provide a competent gold standard assessment. Data collection activities: The data collection period for each participating health worker will be arranged to ensure that the selected HW is on-duty and seeing sick children. Data collection teams will be deployed in each participating clinic for one-week (four to five working days) to collect data. During the data collection period all sick children in the queue to be seen by the observed health worker will be identified, and each day a systematic sampling process will be used so that care is reviewed for 2-3 children . A minimum of six case reviews will be done for each participating health worker. The total number of sick children seen by the observed HW, and the time each child spends in the consulting room will be recorded during the period of data collection. Data will be collected using a combination of exit interviews with mothers, review of the patient held records and review of medication received or prescribed. In addition, each child will be reassessed to provide a gold-standard IMCI assessment against which the HWs management can be compared. Data sources: Health worker knowledge and skills assessment pre and post training in both groups Electronic monitoring of eIMCI use from the software Tracking tool for assessing time spent in the consulting room Exit interviews Reviews of patient held records Gold standard IMCI assessment Data Collection (qualitative) A series of qualitative data collection activities will be conducted to assess feasibility and acceptability of eIMCI implementation before, during and after eIMCI implementation in participating intervention clinics. In-depth interviews/reference groups will be conducted in IsiZulu for mothers and in English for HWs by experienced researchers. All in-depth interviews will be audio recorded. All the interviews will be transcribed and translated into English in preparation for data analysis. Reference groups: Before the implementation of eIMCI a series of four reference groups will be conducted with IMCI trained professional nurses in clinics in another district not implementing eIMCI. A group of IMCI trained nurses who routinely provide services to sick children in a given clinic will be brought together to participate in a focus group discussion (FGD). The aim of the reference groups is to explore the experiences of providing care to children in PHC clinics, how child health services are provided, and what the current challenges are with providing services to sick children and with IMCI implementation. We will then introduce and demonstrate the eIMCI programme, and nurses will be asked to comment on potential implementation of this approach. Using this approach we will be able to use experiences and responses from nurses to identify and anticipate challenges of feasibility and acceptability of implementing eIMCI. In addition we will be able to explore overall challenges related to IMCI implementation which may be equally relevant to eIMCI. If appropriate, implementation of the e-IMCI roll out may be adapted to address these concerns. In-depth interviews with mothers: In-depth interviews will be conducted with mothers managed by a health worker using eIMCI to explore their experience of having the health worker use eIMCI during the consultation. Mothers will be eligible to participate if she reports that the HW consulted the computer during the consultation. Topics to be explored will include experiences during the eIMCI consultation, and how the use of the computer was perceived by mothers. One mother will be selected for an in-depth interview (IDI) from eight randomly selected clinics during the period of data collection (n=8 participants). If necessary, in-depth interviews will be continued until no further themes emerge. In depth interviews with IMCI trained HWs implementing eIMCI: Health workers using eIMCI will be selected for in-depth interviews in six randomly selected clinics on completion of data collection. We will purposively select three eIMCI participants who used eIMCI for a high proportion of their assessments (\>80% consultations with sick children) and four participants who implemented eIMCI poorly (\<50% consultations) to identify barriers and enablers for IMCI implementation. If necessary, in-depth interviews will be continued until no new themes emerge. In addition, three further IDIs will be conducted with good implementers 6 months after completion of data collection to explore issues of sustainability and enablers and barriers to implementation in the long term. Consultation observations: In order to explore how eIMCI implementation affects the nurse- mother relationship during the consultation, a series of observations will be undertaken by an expert IMCI practitioner on the study team who is also a trained qualitative researcher. Consultations will observed among nurses using pIMCI and eIMCI (10 observations per group) and a checklist will be used to record information about nurse- mother interactions during the consultation. These observations will be conducted after the quantitative data collection is complete. Ethical considerations General All participants, both IMCI trained health workers and mothers, will provide written informed consent. Consent forms will be prepared in both the local language (isiZulu) and English, they will be read aloud to each potential participant by a trained researcher and any questions will be addressed. All participants will be informed that participation is voluntary for both community health workers (CHW) and mothers; and that mothers have a right to withdraw from the study at any time. For any mother who is unable to write, a thumbprint will be obtained. Consent forms will not be linked with the study information, and these will be kept separate from the data. All signed consent forms will be kept in a locked cupboard for a period of five years after the study is completed. During data collection If during the interview and assessment process of the child, the IMCI fieldworkers observes that the mother did not receive all the treatment the child needs, the fieldworker will ensure that the mother gets all the treatment needed for the baby according to the reassessment done by the IMCI trained fieldworker. This will be arranged in such a way to minimize the bias that could result if the HW was aware of correction being made, and to avoid undermining the HW in the eyes of the mother. Mothers of sick children who are referred to hospital will be interviewed if it does not interfere or disrupt treatment given to children; and the mother agrees to be interviewed. On completion of data collection the intervention team will work with staff in the district to continue to roll out eIMCI and ensure that control clinics will also have the opportunity to benefit from eIMCI. Computers used in the evaluation will be donated to the clinics for ongoing eIMCI implementation. To maintain anonymity and confidentiality in qualitative data, participants' identity will be protected through the use of pseudonyms or codes. Participants will be made aware of the fact that data will be used for report writing, conference presentation and thesis writing. Data analysis The quantitative data will be analysed using STATA. Qualitative data will be analysed using NVIVO software, framework analysis with pre-determined themes will be used in the process of analysing data. Cost and cost effectiveness analysis A cost and cost effectiveness analysis will be undertaken. The cost analysis will take a provider perspective and provide an estimate of incremental financial costs associated with delivering eIMCI. The provider perspective is favoured ahead of the societal perspective as the costs incurred by mothers across the two approaches to delivery are unlikely to differ significantly. The incremental cost of the eIMCI, rather than the full cost, is favoured as it will provide an estimate of the required increase in funding associated with a roll-out of eIMCI, compared to continuing only with the current approach to IMCI. A financial perspective is opted for ahead of an economic costing, as this will provide relevant information for budgeting. Start-up costs, such as computers and training, will be included, but all research costs will be excluded and only the costs associated with implementation considered. The costing for the incremental cost effectiveness analysis will also be from the provider perspective, but will differ from that used in the simple cost analysis in a number of ways. For the cost effectiveness analysis, financial and broader economic costs will be included. For example, the inclusion of resources not paid for as part of the intervention, but which are diverted to support the intervention. Moreover, the incremental costs will be measured relative to the supported pIMCI provided at the control clinics, as opposed to the standard of care. Using the data on the primary outcome measure, and the cost analysis conducted for the purpose, the incremental cost effectiveness of eIMCI will be estimated: the cost per additional sick child receiving all medications indicated. This calculation provides an estimate of the returns associated with the additional investment in eIMCI.