ePOCT+ and the medAL-suite: Development of an electronic clinical decision support algorithm and digital platform for pediatric outpatients in low- and middle-income countries
1Digital and Global Health Unit, Unisanté, Centre for Primary Care and Public Health, University of Lausanne, Lausanne, Switzerland
2Swiss Tropical and Public Health Institute, Basel, Switzerland
3Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania
4National Institute of Medical Research – Mbeya Medical Research Centre, Mbeya, United Republic of Tanzania
5Department of Information and Technology, Unisanté, Center for Primary Care and Public Health, University of Lausanne, Switzerland
6Institute of Global Health, University of Geneva, Geneva, Switzerland
7Department of Pediatrics and Child Health, Muhimbili University Health and Allied Sciences (MUHAS), Dar es Salaam, United Republic of Tanzania
8PATH, Dar es Salaam, United Republic of Tanzania
9PATH, Nairobi, Kenya
10College of Health Sciences, University of Nairobi, Nairobi, Kenya
11Department of Pediatrics, Cheikh Anta Diop University, Dakar, Senegal
12Department of Pediatrics, King George’s Medical University, Lucknow, India
13PATH, Lucknow, India
14Radboudumc, Department of Internal Medicine and Radboudumc Center for Infectious Diseases, Nijmegen, Netherlands
15University of Basel, Basel, Switzerland
16Department of Paediatrics, King Faisal Hospital, Kigali, Rwanda
17University Teaching Hospital of Kigali, Kigali, Rwanda
18School of Medicine and Pharmacy, University of Rwanda, Kigali, Rwanda
19Intelligent Global Health, Machine Learning and Optimization Laboratory, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland
20Paediatric Emergency Department, Department of Pediatrics, University Hospital Berne, Berne, Switzerland
#Corresponding author: Dr. Rainer Tan, Digital & Global Health Unit, Unisanté, Centre for Primary Care and Public Health, University of Lausanne, Lausanne, Switzerland, Email: rainer.tan@unisante.ch, Tel: +255 676 020 505ABSTRACT
Electronic clinical decision support algorithms (CDSAs) have been developed to address high childhood mortality and inappropriate antibiotic prescription by helping clinicians adhere to guidelines. Previously identified challenges of CDSAs include its limited scope, usability, and outdated clinical algorithms. To address these challenges we developed ePOCT+, a CDSA for the care of pediatric outpatients in low- and middle-income settings, and the medical algorithm suite (medAL-suite), a software for the creation and execution of CDSAs. Following the principles of digital development, we aim to describe the process and lessons learnt from the development of ePOCT+ and the medAL-suite.
In particular, this work outlines the systematic integrative development process in the design and implementation of these tools required to meet the needs of clinicians to improve uptake and quality of care. We considered the feasibility, acceptability and reliability of clinical signs and symptoms, as well as the diagnostic and prognostic performance of predictors. To assure clinical validity, and appropriateness for the country of implementation the algorithm underwent numerous reviews by clinical experts and health authorities from the implementing countries. The digitalization process involved the creation of medAL-creator, a digital platform which allows clinicians without IT skills to easily create the algorithms, and medAL-reader the mobile health (mHealth) app used by clinicians during the consultation. Extensive feasibility tests were done with feedback from end-users of multiple countries to improve the clinical algorithm and medAL-reader software.
We hope that the development framework used for developing ePOCT+ will help support the development of other CDSAs, and that the open-source medAL-suite will enable others to easily and independently implement them.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work took place within the framework of the DYNAMIC project that is funded by the Fondation Botnar, Switzerland (grant n 6278) as well as the Swiss Development Cooperation (grant n 7F-10361.01.01) received by VDA. The TIMCI project funded by UNITAID (grant n 2019-35-TIMCI) received by VDA allowed for adapting of ePOCT+ or the medAL-suite software to Senegal, Kenya and India. The funders had no role in the development of ePOCT+ or the medAL-suite software, data collection, analysis, nor did they have a role in the decision to publish, or preparation of this manuscript.
INTRODUCTION
Electronic clinical decision support algorithms (CDSAs) have been implemented in low- and middle-income countries (LMICs) in order to address excessive mortality due to poor quality of health care,[1] and antimicrobial resistance due to inappropriate antibiotic prescription.[2-4] Such tools provide guidance through every step of the outpatient consultation to ultimately suggest the diagnosis and management plan based on the inputted symptoms, signs and test results.[5] CDSAs have indeed shown that they help clinicians better adhere to guidelines,[6-8] which results in improved quality of care and, for some, more rational antibiotic prescription.[9, 10] This has, in part led the World Health Organization (WHO) and Member States to prioritize the scale-up of the development and access to digital health technologies.[11, 12]
Current CDSAs are not standardized, and concerns have been raised about their limited demographic and clinical scope,[13, 14] their usability,[14, 15] and their static and generic logic based on outdated guidelines that are unable to adapt to new evidence, evolving epidemiology, or changing resources. These challenges may contribute to variable uptake of CDSAs,[15-17] and suboptimal performance when implemented.[8, 18]
In order to address these challenges, and build on the experience of previous CDSAs by our group,[9, 10] and others,[5, 8] we developed the CDSA ePOCT+, and a supporting digital software to create and execute CDSAs, the medAL-suite. ePOCT+ is currently being implemented in several hundred health facilities within the context of implementation studies in Tanzania, Rwanda, Senegal, Kenya and India. Following the principles of digital development and guidance on CDSAs,[19-21] we aim to transparently share the rationale, strategy, and lessons learnt from this development process (figure 1).
METHODS
Scope
Compared to our previous generation CDSAs,[5, 9, 10] the target level of care (primary care health facilities), and target users (mostly nurses and non-physician clinicians) remain the same. However, the target patient population was expanded from 2 months to 5 years, to also cover young infants below 2 months, and in some countries children 5 years up to 15 years.
The expanded target population age group was enlarged to address young infants (<2 months) who are at highest risk of mortality,[22] and children aged 5-15 years whom are often neglected in international and national policies resulting in a slower decrease in mortality in LMICs compared to children under 5 years.[23] This expanded age group may help address the challenge of uptake by avoiding the need for clinicians to change tools when managing children of different age groups.
The scope of illnesses covered was also expanded in response to the frustration of clinicians using CDSAs who were not able to reach specific illnesses.[13, 15] Expanding the scope allowed for the integration of common illnesses covered by other national clinical guidelines to which clinicians are expected to adhere, and to provide more opportunity for antibiotic stewardship when providing management guidance for specific illnesses.
Three major criteria were considered when expanding the scope of illnesses: 1) Incidence of presenting symptoms and diagnoses; 2) Morbidity, mortality, and outbreak potential; and 3) Capacity to diagnose and manage specific conditions at the primary care level.
Additional conditions were identified through: 1) national guidelines; 2) fever aetiology studies; 3) national health surveys; 4) chief complaints from primary care outpatient studies; 5) clinical expert review teams from the implementation countries; 6) interviews with end user clinicians; and 6) observation of consultations at primary care health facilities (S1 Appendix). Examples of notable additions for the Tanzanian algorithm include trauma, urinary tract infection, and abdominal pain that can account for 4.3 – 21.6%,[24] 5.9 – 19.7%,[24-26] and 4.6 – 23%[10, 25] of outpatient consultations respectively.
Clinical algorithm
The target users (mostly nurses and non-physician clinicians), and setting (primary care health facilities) were important considerations when identifying the guidelines and evidence to develop the algorithm. Previously validated algorithms,[10] and the WHO Integrated Management of Childhood Illnesses (IMCI) chart booklets formed the backbone of the algorithm.[27] To support the expanded clinical scope, we turned to national guidelines to ensure adaptation to the local epidemiology, resources, and setting. For conditions not covered by these guidelines, or where guidelines were not sufficiently detailed, the addition of peer-reviewed literature and other international guidelines were required.
In order to transform narrative guidelines into Boolean decision tree logic algorithms, considerable interpretation was needed. The guiding principles for this process were derived from the properties to consider in the screening and diagnosis of a disease by Sackett and colleagues,[28] the target product profile (TPP) for CDSAs as defined by experts in the field,[20] and guidance on appropriate diagnostic and prognostic model development.[29] These include consideration of: a) the feasibility, acceptability, and reliability of clinical elements assessed at the primary care level, b) the diagnostic and prognostic value of individual and combined predictors, c) the sensitivity and specificity in relation to the severity and pre-test probability of the condition in the target population, and d) the overall clinical impression of the patient by the clinician.
a)Feasibility, acceptability, and reliability of predictors
If clinical algorithms are to be adequately utilized, the signs and symptoms used to reach a diagnosis must be feasible, acceptable and reliable when assessed by end-users. These properties were evaluated based on the results of several assessments: primarily an international Delphi study on predictors of sepsis in children,[30] a systematic review on triage tools in low-resource settings,[31] signs and symptoms included in established guidelines for primary care health care workers such as IMCI,[27] interviews with clinicians, observation of routine consultations, a Delphi survey among 30 Tanzanian health care workers (S2 Appendix), as well as subsequent feasibility tests observing clinicians using the CDSA on real and fictional cases. Notable findings from this process led to the omission of a pain score, capillary refill time, the assessment of cool peripheries, and weak and fast pulse, as they were deemed neither feasible nor reliable to be assessed at the primary care level.
b)Diagnostic and prognostic value of predictors
In the absence of validated diagnostic models for each diagnosis, we assessed individual diagnostic and prognostic factors to help guide the development of ePOCT+. Diagnostic studies derived from the population and setting of interest were preferred,[32, 33] as those developed from other settings often perform worse.[34] However, diagnostic predictors notably those predicting ‘serious bacterial infection’, often have low sensitivity, lack reference tests to confirm bacterial origin, and ignore serious infections caused by viral diseases.[35, 36] Prognostic studies are often better suited to develop clinical algorithms in order to understand which children are at risk of developing severe disease, regardless of the aetiology, to improve patient outcomes and reduce resource misallocation. [37-39] A systematic review of predictors of severe disease in febrile children presenting from the community helped identify useful clinical feature to be integrated within ePOCT+,[34] however few studies occurred at the primary care level. To address this gap we performed an exploratory analysis of clinical elements used in two CDSAs evaluated in Tanzania to predict clinical failure (S3 Appendix). This analysis found IMCI danger signs, severe general appearance, mid-upper arm circumference <12.5cm, oxygen saturation <90%, respiratory distress, and signs of anaemia and dehydration to be good predictors of clinical failure. Specific subgroup analyses on our previous generation CDSA provided further support for maintaining or modifying specific algorithm branches, particularly the inclusion of C-reactive Protein (CRP) point-of-care tests that helped safely reduce antibiotic prescription and improve confidence in management.[40, 41]
c)Sensitivity and specificity of algorithm branches in relation to severity and pre-test probability of condition
When constructing the algorithm, it was important to first identify children presenting with a severe condition, and only then use more specific branches to distinguish conditions requiring specific treatment from self-limiting illnesses requiring only supportive care (figure 2). Predictors of severe conditions need to be sufficiently sensitive to guide interventions to reduce morbidity and mortality. However if this high sensitivity comes at the cost of reduced specificity, it can result in over-referral, misallocation of limited health care resources, and excess antibiotic prescription.[37] While this concept was considered within the development of the algorithm, most predictors and models studied lacked sufficient sensitivity and specificity to appropriately meet these requirements at the primary care level, thus emphasizing the need for better predictors and models.[34, 37]
Once a severe condition has been excluded, restricting antimicrobial prescriptions can be more safely integrated given the lower risk of clinical failure. Understanding the pre-test probability (disease prevalence) of the disease guides us on the level of specificity needed for the corresponding predictors to be included in the algorithm. In the outpatient settings, few non-severe children above 2 months have a condition requiring antibiotics.[10, 26] As such, using the principles of Bayes’ theorem,[42] an algorithm for a condition of low prevalence requires a higher likelihood ratio to have a similar post-test probability than a condition with a higher prevalence. Within ePOCT+, C-Reactive Protein (CRP) test is integrated in several branches of the algorithm to increase specificity/likelihood ratio when the pre-test probability of requiring antibiotics is low. However, the pre-test probability of requiring antibiotics may increase in a child with comorbidities, and therefore a lower CRP cut-off can be used to increase sensitivity and reach the same post-test probability.
Adapting and validating the medical content
ePOCT+ was first developed for Tanzania, where the prior generation of the algorithm was validated in a randomized-controlled trial.[10] Following the expansion and adaptation of the content described above, the algorithm was internally reviewed by 13 clinicians from 6 medical institutions with good understanding of CDSAs; 5 working in Tanzania, and the other 8 with experience working in LMICs. The ePOCT+ algorithm for Rwanda, Senegal, Kenya and India were then each drafted, with rounds of internal review, by small development teams composed of clinical algorithm development specialists, and national child health experts based on country-specific objectives, guidelines, and epidemiology, using the first algorithm as a scaffold.
In each country, the ePOCT+ algorithm was reviewed by a technical panel from the ministry of health or an independent clinical expert group (usually with ministry of health representatives). The panels were asked to assess the algorithm in terms of clinical validity, feasibility in primary care, scope of illnesses, and consistency with national policy and guidelines. The process of validation varied slightly in each country according to national decision-making mechanisms, but all included written feedback, individual and group meetings.
Certain algorithm branches were highlighted for group discussion; especially those with novel content, those for which significant interpretation was required from national guidelines, and any branches with queries or comments from panel members. For the algorithms with more novel content, more formal decision processes were used.
Following the internal and external reviews, further modifications were made during the digitalization process, and feasibility tests, including feedback and review from end-users. For each major change proposed, the modification was communicated with the group to allow subsequent feedback and final approval by health authorities
Digitalization of ePOCT+ and development of the medAL-suite
We performed a landscaping review of existing CDSA software with respect to user interface, open source, data management, ease of programming and interpretation of clinical algorithms, and operability in target health facilities. Since none of the available software packages met our requirements, we developed the medAL-suite software following the requirements of the target product profile for CDSAs.[20] medAL-creator allows clinical experts to design the clinical content and logic of the algorithm, while medAL-reader is an Android based interface to execute the algorithm to end-user clinicians (figure 3). Both software were developed collaboratively between the clinicians, IT programmers, end-users via feedback from field tests, and health authorities from the implementation countries.
The World Health Organization (WHO) have recently proposed the SMART guidelines to provide guidance and structure to translate the narrative guidelines (Layer 1), to semi-structured “human readable” decision trees and digital adaptation kits (Layer 2), to computer/machine readable structured algorithms (Layer 3), to the executable form of the software (Layer 4), and finally dynamic algorithms that are trained and optimised to local data (Layer 5).[49] Each “translation” between layers is prone to interpretation and error, especially when each layer is developed by different actors and continuously adapted. To reduce error in interpretation, a major feature of medAL-creator is to allow the “computer/machine readable” structured algorithms to be “human readable”, thus merging Layers 2 and 3. medAL-creator features a “drag and drop” user interface and automatic terminology/code set enabling the clinicians with no programming knowledge to create and review the algorithm. medAL-reader is then able to automatically convert the algorithm from medAL-creator for use at point-of-care.
medAL-reader, was designed based on our previous experiences of CDSA interfaces,[7, 10] and expert guidance on successful strategies in order for the application to be intuitive to use with limited training, to align with normal workflows at primary care health facilities, and encourage user autonomy.[20, 50, 51]
Validation tests and user-experience evaluations
Validation tests were performed for each diagnosis to assure that the inputted data within medAL-creator were processed correctly into the expected output data on medAL-reader. This included automated unit and integration testing, as well as automated non-regression testing by medAL-creator, and manual verification of medication posology for all drugs according to weight and age of the patient. All issues were reviewed by a clinical and IT team to correct the problems. While such tests are encouraged by the CDSA TPP,[20] since CDSAs are not considered a “software as a medical device” by the Food and Drug Administration (FDA)[52] or European Medical Device Coordination Group,[53] these tests are not legally required.
The ePOCT+ tool underwent numerous rounds of testing in the field, from desk-based reviews focusing on user interface and analytical validation; to end-user testing evaluating user experience, acceptability, and clinical applicability; to finally assessing integrated testing in real-life conditions. All user experience feedback was reviewed by a team including both clinical and IT specialists, while all clinical content modifications were approved by both the internal and external review panels.
Ethics
Activities related to the development and piloting of ePOCT+ and the medAL-suite were done within the studies of DYNAMIC and TIMCI, for which approval was given from each country of implementation. The study protocol and related documents were approved by the institutional review boards of the Ifakara Health Institute in Tanzania (IHI/IRB/No: 11-2020 and 49-2020), the National Institute for Medical Research in Tanzania (NIMR/HQ/R.8a/Vol. IX/3486 and NIMR/HQ/R.8a/Vol. IX/3583), the National Ethics Committee of Rwanda (752/RNEC/2020), the Comité National d’Ethique pour la Recherche en Santé of Senegal (SEN20/50), the University of Nairobi Ethics and Research Committee in Kenya (UON/CHS/TIMCI/1/1), the King George’s Medical College Institutional Ethics Committee in India (103rd ECM IC/P2), the Indian Council of Medical Research (2020-9753), the cantonal ethics review board of Vaud, Switzerland (CER-VD 2020-02800 & CER-VD 2020-02799), and the WHO Ethics Review Committee (ERC.0003405 & ERC.0003406). Written informed consent was obtained from all parents or guardians of children involved in the piloting of ePOCT+ and medAL-Reader. No informed consent was obtained from health care workers involved in the development and refinement of the tools.
The exploratory analysis of predictors from the 2014 ePOCT study received approval of the study protocol and related documents by the institutional review boards of the Ifakara Health Institute and the National Institute for Medical Research in Tanzania (NIMRrHQ,R.8a,/trI’VoIl. 789), by the Ethikkommission Beider Basel in Switzerland (EKNZ UBE 15/03), and the Boston Children’s Hospital ethical review board. Written informed consent was obtained from all parents or guardians.
RESULTS
The ePOCT+ clinical algorithm and supporting evidence for each country of implementation can be found on the websites of the DYNAMIC and TIMCI studies that are implementing ePOCT+. The major features of medAL-Creator and medAL-Reader are summarized in the supplementary material (S4 Appendix), including the requirements defined by the CDSA target product profile (S5 Appendix).
The feasibility tests of ePOCT+ were conducted in 20 health facilities, leading to numerous modifications (Table 1). The improved algorithm was then piloted with over 2000 consultations before officially starting the clinical validation studies in the five countries of implementation.
DISCUSSION
ePOCT+ was derived from existing evidence and clinical validation field studies from previous generation CDSAs.[7, 9, 10] It is now being further validated in several large clinical studies. Following established development protocols, attempts were made to ensure a transparent development process, multi-stakeholder collaboration, and end-user feedback.[20, 21, 54, 55] Specifically, aligning the development process of ePOCT+ and specifications of medAL-reader to the requirements of the Target Product Profile for CDSAs was helpful to better meet the needs of end users in terms of quality, safety, performance and operational functionality.[20]. Nonetheless, there are several limitations and challenges with the development process and end-result of ePOCT+ and the medAL-suite, for which ongoing modifications and improvements will be required.
Firstly, while efforts were made to improve the performance of the algorithm, there was often a reliance on clinical guidelines which may not always be founded on the best/latest/highest quality evidence, or applicable to low resource primary care settings.[56] [57] Furthermore, they require significant interpretation to transform into algorithms. Digital Adaptation Kits (DAKs) to guide implementers in how to interpret narrative guidelines to transform into digital platforms are currently being developed by the World Health Organization and should help address this challenge in the future.[49, 58] Often supplementary evidence was needed to complement national and international guidelines. This evidence should ideally be identified through systematic reviews,[59] however those are not always feasible. Leveraging existing evidence databases as done by another CDSA may be a more feasible method to avoid biases in identifying supporting evidence.[60] Among the supporting evidence identified, there was a paucity of evidence for conditions specific to older children above 5 years, prognostic studies in the primary care setting, and diagnostic studies for conditions other than serious bacterial infection and pneumonia. Evaluating the prognostic and diagnostic value of predictors and models used in ePOCT+ during the ongoing validation studies will help to develop more efficient and better performing algorithms optimised for the target population.[49, 61]
Many modifications to ePOCT+ and medAL-reader compared to previous generation CDSAs were implemented in order to help improve uptake, addressing previously shared concerns such as limited scope, and ease of use. medAL-reader was specifically designed to follow normal healthcare worker workflows, however the introduction of other digital tools such as electronic medical records hinder these processes. As an example, it is estimated that there are over 160 digital health or health-related systems in Tanzania.[62] While efforts are currently being made to harmonize processes so that different digital systems can complement each other, rather than creating additional work, this has not yet been achieved. It is important to note, that while ePOCT+ and medAL-reader may address some challenges to uptake, there are many extrinsic and intrinsic factors that are not addressed, such as the low perceived value of following guidelines, and lack of motivation partly related to poor remuneration.[15, 63]
The digitalization process allows for increased complexity in the algorithm compared to paper guidelines. However, this complexity may limit the understanding by healthcare workers. Understanding how a diagnosis and treatment plan is reached is fundamental to clinical and patient autonomy, important for continued learning, and for fostering trust in any algorithm.[64-66] Efforts were made to present simple decision tree logic for each diagnosis. Nevertheless, the optimal method of presentation of algorithm branches to assure understanding by primary care level healthcare workers should be further explored.
CONCLUSION
ePOCT+ aims to improve clinical care of sick children in LMICs, notably by reducing unnecessary antibiotic prescription. We hope that the strong stakeholder involvement, the expanded scope of the clinical algorithm, and the novel software of the medAL-suite will result in high uptake, trust and acceptability. Widespread implementation will provide opportunity for dynamic and targeted refinements to the clinical content to improve the performance of the algorithm. We hope that the easy-to-use platform of the medAL-suite, and the framework used to develop ePOCT+ will allow health authorities and local communities to be able to take ownership of ePOCT+ or their own clinical algorithm for future adaptations and developments. Future success however, is contingent on the harmonization with national health management information systems and other digital systems.
Supporting information
Data Availability
All data produced in the present work are available upon reasonable request to the authors.
Declaration of interests
We declare no competing interests.
Funding
This work took place within the framework of the DYNAMIC project that is funded by the Fondation Botnar, Switzerland (grant n°6278) as well as the Swiss Development Cooperation (grant n°7F-10361.01.01) received by VDA. The TIMCI project funded by UNITAID (grant n°2019-35-TIMCI) received by VDA allowed for adapting of ePOCT+ or the medAL-suite software to Senegal, Kenya and India. The funders had no role in the development of ePOCT+ or the medAL-suite software, data collection, analysis, nor did they have a role in the decision to publish, or preparation of this manuscript.
Acknowledgements
Emmanuel Barchichat, Alain Fresco, and Quentin Girard from Wavemind for the IT programming of medal-creator and medal-reader software. Martin Norris, Lisa Cleveley, Dr Sabine Renggli, Ibrahim Mtabene, Peter Agrea and Dr Godfrey Kavishe for the medAL-reader tests and suggestions for improvements to both medal-reader and medal-creator. Cecile Trottet for the statistical support. The many health care workers providing feedback on the tool, patients and caretakers involved with pilot and feasibility testing. Dr Arjun Chandna and Janet Urquhart for helpful comments on the manuscript.
Supporting information captions
S1 Appendix: Prevalence of specific symptoms and diagnoses not covered in IMCI from Tanzania
S2 Appendix: Delphi survey on the reliability and feasibility of measurement of symptoms and signs
S3 Appendix: Prognostic value of predictors used in the ePOCT and ALMANACH electronic clinical decision support algorithms
S4 Appendix: Features of the medAL-creator and medAL-reader software as defined by a clinical-IT collaboration with end-user feedback
S5 Appendix: Evaluation of ePOCT+ based on the characteristics set by the target product profile for electronic clinical decision support algorithm as defined by expert consensus