Development and validation of treatment-decision algorithms for children evaluated for pulmonary tuberculosis: an individual participant data meta-analysis
1Department of Epidemiology of Microbial Diseases, Yale School of Public Health, New Haven, USA
2University of Bordeaux, Inserm UMR1219, Institut de Recherche pour le Développement (IRD) EMR 271, Bordeaux France
3Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
4Barcelona Institute for Global Health (ISGlobal), Hospital Clínic - Universitat de Barcelona, Barcelona, Spain
5National Tuberculosis and Leprosy Program, Kampala, Uganda
6Department of Global Health and Social Medicine, Harvard Medical School, Boston, USA
7University of Montpellier, Inserm, Institut de Recherche pour le Développement, TransVIHMI, Montpellier, France
8Epicentre, Mbarara, Uganda
9Department of Paediatrics, Dhaka Medical College Hospital, Dhaka, Bangladesh
10Indus Hospital & Health Network, Karachi, Pakistan
11The Aga Khan University Hospital, Karachi, Pakistan
12Centro de Investigação em Saúde de Manhiça, Maputo, Mozambique
13Instituto de Puericultura e Pediatria Martagao Gesteira. Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil
14Programme on Emerging Infections, Infectious Disease Division, icddr,b, Dhaka, Bangladesh
15Global Tuberculosis Programme, World Health Organization, Geneva, Switzerland
16US Centers for Disease Control and Prevention, Atlanta, USA
17Manhiça Health Research Centre, Maputo, Mozambique
18Barcelona Institute for Global Health, Barcelona, Spain
19Liverpool School of Tropical Medicine, Liverpool, UK
20Emory School of Medicine, Atlanta, USA
21Emory Rollins School of Public Health, Atlanta, USA
22Children’s Healthcare of Atlanta, Atlanta, USA
23Centro de Investigação em Saúde de Manhiça, CISM,e; Maputo, Mozambique
24ISGlobal, Hospital Clínic - Universitat de Barcelona, Barcelona, Spain
25Centro de Investigación Biomédica en Red de Enfermedades Infecciosas (CIBERINFEC), Barcelona, Spain
26Desmond Tutu TB Centre, Department of Paediatrics and Child Health, Faculty of Medicine and Health Sciences, Stellenbosch University, Tygerberg, South Africa
27Oxford University Clinical Research Unit, Centre for Tropical Diseases, Ho Chi Minh City, Vietnam
28Nuffield Department of Medicine, Oxford University, UK
29Socios En Salud Surcursal Perú, Lima, Perú
30Global TB Program, Baylor College of Medicine and Texas Children’s Hospital, Houston, USA
31Clinical Infectious Disease Group, German Center for Infectious Research (DZIF), Clinical TB Unit, Research Center Borstel, Borstel, Germany
32Department of Paediatrics, University of Medicine, Mandalay, Myanmar
33Department of Paediatrics, University of Medicine, Magway, Myanmar
34Mbarara University of Science and Technology, Mbarara, Uganda
35Division of Infection and Immunity, Department of Biomedical Sciences, University of Western Australia, Perth, Australia
36Department of Microbiology, Division of Basic Medical Sciences, School of Medicine, University of Global Health Equity, Kigali, Rwanda
37Faculty of Medicine, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil
38Department of Health Policy and Management, Yale School of Public Health, New Haven, USA
39Oxford Vaccine Group, Department of Paediatrics, University of Oxford, Oxford, UK
40Division of Infectious Diseases, Boston Children’s Hospital, Boston, Massachusetts
41Department of Pediatrics, Harvard Medical School, Boston, USA
42Oxford University Clinical Research Unit, Ho Chi Minh City, Vietnam
43Nuffield Department of Medicine, Centre for TropicalMedicine and Global Health, University of Oxford, Oxford, UK
44University of Health Sciences, Phnom Penh, Cambodia
45National Pediatric Hospital, Phnom Penh, Cambodia
46School of Public Health, University of Sydney, Sydney, Australia
47Directorate of Integrated Laboratory Medicine, Institute of Human Genetics, Newcastle upon Tyne NHS Foundation Trust, Newcastle upon Tyne, United Kingdom
48Department of Biostatistics, Yale School of Public Health, New Haven, USA
49Department of Paediatrics & Child Health, Red Cross Children’s Hospital, Cape Town, South Africa
50SA-MRC Unit on Child & Adolescent Health, University of Cape Town, South Africa
51The Children’s Hospital at Westmead Clinical School, Faculty of Medicine and Health, University of Sydney, Sydney, Australia
52University of Melbourne Department of Paediatrics and Murdoch Children’s Research Institute, Royal Children’s Hospital, Melbourne, Australia
53Burnet Institute, Melbourne, Australia
54Desmond Tutu Tuberculosis Centre, Department of Paediatrics and Child Health, Faculty of Medicine and Health Sciences, Stellenbosch University, Tygerberg, South Africa
55Department of Infectious Diseases, Imperial College London, London, UK
*Corresponding author Kenneth Gunasekera, 350 George Street, 3F, New Haven, CT 06510, USA, e-mail: kenneth.gunasekera@yale.eduABSTRACT
Background
Many children with pulmonary tuberculosis remain undiagnosed and untreated with related high morbidity and mortality. Diagnostic challenges in children include low bacterial burden, challenges around specimen collection, and limited access to diagnostic expertise. Algorithms that guide decisions to initiate tuberculosis treatment in resource-limited settings could help to close the persistent childhood tuberculosis treatment gap. Recent advances in childhood tuberculosis algorithm development have incorporated prediction modelling, but studies conducted to date have been small and localised, with limited generalizability.
Methods
We collated individual participant data including clinical, bacteriological, and radiologic information from prospective diagnostic studies in high-tuberculosis incidence settings enrolling children <10 years with presumptive pulmonary tuberculosis. Using this dataset, we first retrospectively evaluated the performance of several existing treatment-decision algorithms and then developed multivariable prediction models, investigating model generalisability using internal-external cross-validation. A team of experts provided input to adapt the models into a pragmatic treatment-decision algorithm with a pre-determined sensitivity threshold of 85% for use in resource-limited, primary healthcare settings.
Findings
Of 4,718 children from 13 studies from 12 countries, 1,811 (38·4%) were classified as having pulmonary tuberculosis; 541 (29·9%) bacteriologically confirmed and 1,270 (70·1%) unconfirmed. Existing treatment-decision algorithms had highly variable diagnostic performance. Our prediction model had a combined sensitivity of 86% [95% confidence interval (CI): 0·68-0·94] and specificity of 37% [95% CI: 0·15-0·66] against a composite reference standard.
Interpretation
We adopted an evidence-based approach to develop pragmatic algorithms to guide tuberculosis treatment decisions in children, irrespective of the resources locally available. This approach will empower health workers in resource-limited, primary healthcare settings to initiate tuberculosis treatment in children in order to improve access to care and reduce tuberculosis-related mortality. These algorithms have been included in the operational handbook accompanying the latest WHO guidelines on the management of tuberculosis in children and adolescents.
Funding
World Health Organization, US National Institutes of Health
RESEARCH IN CONTEXT
Evidence before the study
Treatment-decision algorithms relate information gained in the evaluation of children into an assessment of tuberculosis disease risk and empower healthcare workers to make appropriate treatment decisions. Studies in primary healthcare centres have demonstrated that use of treatment-decision algorithms can improve childhood pulmonary tuberculosis case-detection and treatment initiation in settings with high-tuberculosis incidence. To identify primary research studies on treatment-decision algorithm performance evaluation and/or development for childhood pulmonary tuberculosis, we carried out a PubMed search using the terms (‘child*’ OR ‘paediatr*’ OR ‘pediatr*’) AND (‘tuberculosis’ OR ‘TB’) AND (‘treatment-decision’ OR ‘algorithm’ OR ‘diagnos*’) to identify primary research published in any language prior to 29 June 2022.
We additionally consulted multiple experts in childhood pulmonary tuberculosis diagnosis and management, and we referred to existing, published reviews of treatment-decision algorithms. With respect to treatment-decision algorithm performance, several studies have retrospectively estimated the performance of treatment-decision algorithms in a single geographic setting; a subset of these studies have also compared the performance of multiple algorithms using data from a single geographic setting. With respect to treatment-decision algorithm development, many existing algorithms have been developed without explicit analysis of data from children with presumptive pulmonary tuberculosis, often developed from expert consensus. Gunasekera et al. used model-based approaches to analyse diagnostic evaluations data (e.g., clinical history, physical examination, chest radiograph, and results from rapid molecular and culture testing for Mycobacterium tuberculosis) collected from children with presumptive pulmonary tuberculosis in a single geographic setting to inform the development of a diagnostic algorithm while Marcy et al. and Fourie et al analysed data from multiple geographic settings. However, these studies were relatively small with limited assessment of generalisability.
Added value of this study
We collated individual participant data from 13 prospective diagnostic studies from 12 countries including 4,718 children with presumptive pulmonary tuberculosis from geographically diverse settings with a high incidence of tuberculosis in order to 1) evaluate the performance of existing treatment-decision algorithms and 2) develop multivariable logistic regression models to quantify the contribution of individual features to discriminate tuberculosis from non-tuberculosis. A panel of child tuberculosis experts provided input into performance targets and advised on how to incorporate scores derived from these models into pragmatic treatment-decision algorithms to assist in the evaluation of children presenting with presumptive pulmonary tuberculosis in primary healthcare centres.
Implications of all the available evidence
Our findings suggest that evidence-based, pragmatic treatment-decision algorithms can be developed to make sensitive and clinically appropriate decisions to treat a child with pulmonary tuberculosis. Although the specificity does not reach optimal targets for childhood tuberculosis diagnosis, pragmatic treatment-decision algorithms provide clinically relevant guidance that can empower health workers to start children on tuberculosis treatment at the primary healthcare setting and will likely contribute to reducing the case-detection gap in childhood tuberculosis. External, prospective evaluation of these novel algorithms in diverse settings is required, including assessment of their accuracy, feasibility, acceptability, impact, and cost-effectiveness. This work led to a new interim WHO recommendation to support the use of treatment-decision algorithms in the evaluation of children with presumptive tuberculosis in the 2022 updated consolidated guidelines on the management of tuberculosis in children. Two algorithms developed from this work have been included in the WHO operational handbook accompanying these guidelines.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was funded by the WHO Global Tuberculosis Programme (GTB) as well as the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the United States National Institutes of Health (NIH).
INTRODUCTION
Tuberculosis is a leading cause of mortality among children worldwide,1 accounting for ∼2·5% of the 6 million deaths in children <5 years each year.2 Modelling suggests that more than 96% of tuberculosis deaths in children and adolescents (<15 years) occurred in those not receiving tuberculosis treatment.3 The World Health Organization (WHO) estimates that fewer than 50% of the 1·1 million children <15 years who develop tuberculosis are diagnosed; the proportion is even lower among children <5 years, at about 27%.1 Thus, efforts to improve diagnosis, and thereby improve access to tuberculosis treatment, are an important opportunity to reduce tuberculosis morbidity and deaths among children.
Diagnosing pulmonary tuberculosis among children is challenging as respiratory specimens tend to be paucibacillary, resulting in a low yield of bacteriologic confirmation.4 Furthermore, collecting respiratory specimens from young children is invasive and requires resources that are generally concentrated in higher-level healthcare centres. Thus, careful symptom review, clinical examination, chest radiography, and history of Mycobacterium tuberculosis (Mtb) exposure can inform treatment decisions in clinical care. However, paediatric clinical expertise and resources to make a diagnosis are often limited at primary healthcare centres. This limits treatment access and leads to either delays in treatment initiation or no treatment initiation which are associated with worse outcomes, including mortality.5,6 Facilitating appropriate diagnostic assessment with rapid treatment initiation at healthcare settings where children initially present could contribute to reductions in tuberculosis-related morbidity and mortality.
Treatment-decision algorithms aim to standardise clinical assessment and decision-making. Algorithms relate information gained in the evaluation of children into an assessment of tuberculosis disease risk and empower healthcare workers to make appropriate treatment decisions. Adopting an algorithmic approach to treatment decision-making has been shown to improve childhood tuberculosis case detection and treatment access at primary healthcare settings.7,8 However, these algorithms were developed using consensus expert opinion rather than analysis of data.
Recent approaches for algorithm generation have used data from cross-sectional childhood tuberculosis diagnostic studies to quantify the contribution of clinical characteristics to the risk of tuberculosis disease.9-11 Evidence-based approaches are objective and offer the potential for validation; however, existing studies have been small and not generalisable. In this study, we assembled individual participant data (IPD) from children investigated for presumptive pulmonary tuberculosis. We then sought to first evaluate the performance of currently used diagnostic algorithms and then develop evidence-based treatment-decision algorithms. This work was conducted to inform the 2022 WHO guidelines for the management of tuberculosis in children and adolescents and the accompanying WHO operational handbook.12,13
METHODS
Establishment of individual participant data
We identified potential sources of IPD through responses to a public WHO Global Tuberculosis Programme Public Call for Data on the Management of Children with Tuberculosis in July 2020 and through referral from paediatric tuberculosis experts. Studies were eligible for inclusion if they 1) prospectively recruited consecutive participants <10 years (the definition of a child in the 2022 WHO guideline) attending healthcare centres in high-tuberculosis incidence countries for clinical evaluation of pulmonary tuberculosis and 2) provided a final research classification of pulmonary tuberculosis for each child. Tuberculosis was classified using the revised US National Institutes of Health (NIH) clinical case definitions of intrathoracic tuberculosis in children.14 Broadly, this defines a confirmed tuberculosis case as culture- or Xpert MTB/RIF-confirmed Mtb from respiratory specimen(s); an unconfirmed tuberculosis case as having symptoms, chest radiography findings, and/or immune tests of Mtb sensitivity suggestive of tuberculosis (including follow-up to verify or rule out tuberculosis); and an unlikely tuberculosis case as meeting criteria for neither confirmed nor unconfirmed tuberculosis. If the study did not use the NIH classification, we used the study-specific definition of unconfirmed tuberculosis and unlikely tuberculosis. Quality assessment was performed using a modified version of the Newcastle-Ottawa Scale for cohort studies.15
After identification of eligible studies, we requested IPD including details from the clinical history, physical examination, chest radiograph, and results from rapid molecular and culture testing for Mtb performed on respiratory specimens collected at study entry (Supplementary Appendix A). All data assembly and analysis were carried out using R software version 4.1.1. To account for the uncertainty associated with incomplete data, we used multilevel multiple imputation by chained equations (MICE) implemented in the MICE package to generate 100 imputed datasets (Supplementary Appendix B).16
Evaluation of existing treatment-decision algorithms
We identified existing treatment-decision algorithms and scores (henceforth referred to as algorithms) to guide the evaluation of children with presumptive pulmonary tuberculosis through consultation with members of the WHO Guideline Development Group (GDG) on the management of tuberculosis in children and adolescents. We defined a composite reference standard using the NIH definitions of confirmed and unconfirmed pulmonary tuberculosis to evaluate the performance of these algorithms. We carried out a sensitivity analysis of performance using a reference standard of confirmed pulmonary tuberculosis only (excluding children with unconfirmed tuberculosis). We used the “reitsma” function from the R package mada to pool study-level sensitivity and specificity estimates using a bivariate random effects meta-analysis (Supplementary Appendix C).17,18
Prediction model development and validation
We developed a multivariable logistic regression model to predict pulmonary tuberculosis using the composite reference standard in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis standards using the internal-external cross-validation framework.19,20 Predictors included clinical features commonly considered in the evaluation of presumptive childhood pulmonary tuberculosis in primary and secondary healthcare centres that were available in the data with <50% missingness. We also built a model without chest radiograph data to inform predictions in healthcare centres without access to radiology services.
To account for possible heterogeneity in the distribution of predictor and outcome variables, we fit the prediction model separately in each study and subsequently pooled their regression coefficients and respective standard errors. This approach was implemented in the “metapred” function of package metamisc.20,21 To account for missing data, we generated a prediction model as described above from each of the 100 imputed datasets and then used Rubin’s rules to pool the regression coefficients and standard errors to generate a final, single prediction model and compute odds ratios with 95% confidence interval (95% CI).22 We examined the c-statistic (also known as the area under the receiver operating characteristic curve) to assess the model’s ability to distinguish between children with tuberculosis and unlikely tuberculosis, and we examined the observed:expected (O:E) ratio to assess whether there were studies in which the model over- or under-predicted tuberculosis.
Algorithm development
To generate clinically and programmatically implementable algorithms, we scaled the coefficient estimates for the parameters of the final prediction models (developed from all n studies) to estimate scores for each parameter such that a combined score of >10 corresponded to classification of tuberculosis at fixed sensitivities of 90%, 85%, 80%, 75%, and 70% (Supplementary Appendix D). To estimate the sensitivity and specificity of the score in classifying tuberculosis using the composite reference standard, study-level sensitivities and specificities were pooled using the bivariate model of Reitsma et al. (implemented in the mada package) accounting for uncertainty introduced by imputation of missing data.17,18 As a sensitivity analysis, we evaluated the performance of the score against a reference standard of confirmed pulmonary tuberculosis only.
We worked with staff from the WHO Global TB Programme to identify a group of experts in childhood tuberculosis (henceforth referred to as the expert group; Supplementary Appendix E) to advise on the development of two treatment-decision algorithms from these scores.
Specifically, we sought advice on how to use this score within an algorithm intended to be used at primary healthcare centres and on selection of a performance target for the development of the score to be included within the algorithm.
Ethics
This study was approved by the Stellenbosch University Health Research Ethics Committee (Ref No. X21/02/003) and the Yale Institutional Review Board (Ref No. 2000028046). All collaborating investigators confirmed institutional ethical approval for their original data collection.
RESULTS
Data assembly
Eighteen studies were identified as having potentially appropriate data, largely sourced from diagnostic evaluation studies. The study investigators for two studies were unable to provide data in the necessary timeline and an additional three studies did not meet the inclusion criteria. From the 13 included studies carried out in 12 countries, 4,718 IPD records from children <10 years with presumptive pulmonary tuberculosis were available, of which, 541 (11·5%) were classified as having confirmed tuberculosis, 1,270 (26·9%) as having unconfirmed tuberculosis and 2,818 (59·7%) unlikely tuberculosis (Tables 1 and 2; Supplementary Appendices F and G). The data were predominantly collected at secondary or tertiary/referral healthcare centres. Although each study was required to include children with presumptive pulmonary tuberculosis, studies differed slightly with respect to inclusion criteria, variable definitions, and reference classification of tuberculosis (Supplementary Appendices H-K). All contributing studies achieved quality assessment scores of 4/5 or 5/5. (Supplementary Appendix L).
Existing treatment-decision algorithm performance
We evaluated the performance of eight existing treatment-decision algorithms. One of these algorithms was evaluated only on data from children living with HIV, and another evaluated only on data from children without HIV. Because some algorithms considered evidence that were not available in the IPD, we modified all algorithms slightly (Supplementary Appendix M). The sensitivities varied from 17% to 93% with specificities varying from 88% to 16% when evaluated against the composite reference standard (Figure 1; Supplementary Appendix N). A sensitivity analysis evaluating performance to discriminate confirmed tuberculosis from unlikely tuberculosis demonstrated marginally higher sensitivities and comparable specificities to the performance in the entire dataset (Supplementary Appendix O).
Prediction model development and validation
Odds ratios and 95% CI of the predictors included in the model are displayed in Table 3. The pooled c-statistic for the prediction model including chest x-ray features was 0·71 (95% CI: 0·66-0·76), with a summary O:E ratio of 0·90 (95% CI: 0·28-2·98). Additional internal-external cross-validation c-statistic and O:E ratio estimates are included in Supplementary Appendix P. Estimates for the model without chest x-ray features are included in Supplementary Appendix Q.
Algorithm development
The scores derived from the model prediction coefficients that correspond to classification of all tuberculosis with respective sensitivities of 90%, 85%, 80%, 75%, and 70% can be found in Supplementary Appendix R. The study-level and summary performance of these scores in classifying tuberculosis can be found in Supplementary Appendix S.
To balance the consequences of untreated tuberculosis versus the consequences of overtreatment, the expert group recommended a sensitivity threshold of 85% in classifying tuberculosis using the composite reference standard, resulting in the development of a score with a sensitivity of 0·86 (95% CI: 0·68-0·94) and a specificity of 0·37 (95% CI: 0·15-0·66) (Figure 2). An analysis of the performance in classifying confirmed tuberculosis vs. unlikely tuberculosis demonstrated a sensitivity of 0·88 (95% CI: 0·71-0·95) and specificity of 0·37 (95% CI: 0·15-0·67) (Supplementary Appendix T). Under a sensitivity threshold of 85%, the model that included only features from the baseline clinical evaluation (without chest x-ray findings) had a sensitivity of 0·84 (95% CI: 0·66-0·93) and specificity of 0·30 (95% CI: 0·13-0·56) in classifying tuberculosis (Supplementary Appendix U).
To adapt the scores into treatment-decision algorithms to be used at primary healthcare centres, the expert group recommended the following triage steps prior to classification using the score: 1) identifying children with clinical symptoms and signs requiring urgent referral to higher levels of healthcare, and 2) stratifying children by risk of mortality and progression of tuberculosis. Higher-risk children were defined by the expert group as those <2 years, severely malnourished, and/or living with HIV. These children would be evaluated using the score at the time of the initial evaluation. Children not meeting this definition would be treated for the most likely non-tuberculosis condition and complete re-evaluation in 1-2 weeks: those with persistent/worsening symptoms at follow-up would be evaluated using the score. The expert group additionally recommended to pursue, wherever available, bacteriological testing on respiratory and/or stool specimens with rapid molecular diagnostics for all children and urine lateral flow assays for HIV-positive children to align with existing WHO recommendations23.
The expert group recommendations resulted in the development of a treatment-decision algorithm (Figure 3), in which children <10 years with presumptive pulmonary tuberculosis are triaged by risk of tuberculosis-related morbidity and mortality prior to being evaluated for the presence of clinical and chest x-ray features to assign a score corresponding to tuberculosis risk. A total score of >10 results in classification of tuberculosis with a sensitivity of 85%. Known exposure to tuberculosis alone has a score >10, so this was placed above the other elements. The same parameters were used to construct the treatment-decision algorithm from the model without chest x-ray features (Supplementary Appendix V).
DISCUSSION
We assembled a large IPD dataset from nearly five thousand children from geographically diverse, high-tuberculosis incidence settings to evaluate existing treatment-decision algorithms and develop new evidence-based treatment-decision algorithms to guide evaluation of children with presumptive pulmonary tuberculosis. This algorithm-building approach uses the best currently available data to provide practical guidance to healthcare workers in primary healthcare settings to identify which clinical features, with or without chest x-ray assessment, indicate whether initiation of tuberculosis treatment is warranted. The newly developed algorithms were incorporated into the WHO operational handbook to support implementation of the new consolidated guidelines.12,13
Modelling diagnostic evaluations IPD provides a quantitative description of the strength of evidence for childhood pulmonary tuberculosis treatment-initiation decisions. Of the clinical features, only reported exposure to tuberculosis was independently sufficient to meet the threshold for treatment initiation. This was true even in the model-based score without chest x-ray features, suggesting that none of the common clinical features could independently inform highly sensitive/specific treatment decisions. While results from tuberculin skin testing were used by studies to inform classification of tuberculosis and to improve imputation of missing data, we did not include these in the algorithms given operational limitations in using tuberculin skin testing at scale in high-burden settings. Of the chest X-ray features included in the algorithm, the presence of intrathoracic lymphadenopathy and a miliary pattern, respectively, were independently sufficient to start treatment. It is worth noting that inclusion of chest x-ray features only increased the specificity of the score slightly as compared to the score developed from the model with clinical features only. Chest x-ray has additional utility in guiding childhood pulmonary tuberculosis treatment duration for severe vs. non-severe disease,24 in monitoring TB treatment response (including associated complications and sequelae) and in the diagnosis of other non-tuberculosis intrathoracic pathology.
The decision to prioritise sensitivity in our algorithm development is critical to initiate more children with tuberculosis on appropriate therapy; however, many children may be falsely treated for tuberculosis given the resulting specificity. No test or algorithm meets the WHO-target sensitivity and specificity for a confirmatory diagnostic test for childhood pulmonary tuberculosis.25 Thus, the expert group advised to develop an algorithm with a minimum sensitivity target of 85% as an acceptable balance between sensitivity and the resulting specificity. Tuberculosis treatment for drug-susceptible disease is relatively safe in children;26 especially considering the potentially severe consequences of a missed tuberculosis diagnosis. However, falsely treating for tuberculosis carries risk of delayed diagnosis of other disease, drug adverse events, and unnecessary burden on families and healthcare services. It should be noted that while these performance estimates relate to the score component of the algorithm, the overall sensitivity and specificity of the whole algorithm, including the triage steps, remains unknown and should be evaluated prospectively. As low-risk children are made to wait prior to being evaluated with the scored part of the algorithm, symptoms in some with diagnoses other than tuberculosis will resolve, likely improving specificity.
We note that the model-based scoring component of the algorithm demonstrates considerable study-level heterogeneity in sensitivity and specificity. Although this IPD is the largest of its size compiled to date, there were not enough studies to quantitatively describe the features that drive the observed heterogeneity. Given that we used data made available to WHO following a public call rather than conducting a systematic review, it is possible that some diagnostic studies may have been excluded. The inclusion of more data from existing, ongoing, and future studies, may allow meta-regression to describe study-level sources of heterogeneity. Heterogeneity may have been driven in part by varied tuberculosis prevalence in the cohorts included as well as heterogeneities in disease presentation. Given that the pre-existing treatment-decision algorithms demonstrated similar heterogeneities in performance compared to the evidence-based algorithm developed, we suggest that our approach is robust as it offers the flexibility to further interrogate the sources of heterogeneity as additional data are available. A distinct advantage of the modelling approach we used for algorithm development is the ability to revise and calibrate the model to specific settings as additional data become available.
We considered it important to evaluate existing treatment-decision algorithms and develop new algorithms using a composite standard rather than solely a microbiological standard, given the high percentage of children treated for tuberculosis without bacteriological confirmation, even in the best resourced settings, which reflects the paucibacillary nature of disease in most young children. However, this reference standard remains imperfect, and misclassification may occur.27 The underlying composition of the unconfirmed tuberculosis group may represent a heterogeneous group in which some children have tuberculosis, and some have other causes for their observed symptoms and signs. Additionally, it is possible that inclusion of unconfirmed pulmonary tuberculosis biased the estimation of the prediction model parameters, especially those used to classify the unconfirmed group. Although this is a limitation of our study, the similar performance estimates of the score developed in the primary analysis using both the composite and confirmed tuberculosis reference standards suggest that this may not be a major issue.
Given that our algorithms are intended to guide decisions to treat children in primary healthcare centres, it is a limitation that IPD was derived from primarily tertiary and referral health centres. We are not aware of studies that provide this quality of diagnostic evaluations data from presumptive childhood tuberculosis in primary healthcare centres. However, in several studies, children presenting at primary healthcare settings were directly referred for study evaluation, providing some degree of reassurance as to the generalisability of results. The pre-test probability of tuberculosis (i.e., the prevalence) is likely substantially lower among children attending primary healthcare centres and the clinical presentation may be different as compared to tertiary and referral centres from which the data were obtained. These are important given that many children with tuberculosis first present to primary healthcare centres.28 We believe that the risk-stratification and delayed entry of lower risk children with presumptive tuberculosis (who should tolerate the delay) is a practical attempt to safely raise the pre-test probability when implementing the algorithm in primary health centres. Prospective external validation of the entire algorithms will be critical to determining their accuracy, acceptability and feasibility of use at different levels of the healthcare system.
There are inherent limitations to developing a prediction model developed on data from multiple cohorts for a disease with an imperfect diagnostic gold standard. Study inclusion criteria varied, which impacts the baseline tuberculosis prevalence and applicability of the score prediction estimates. Additionally, prediction variable definitions varied among the included studies—for example, history of weight loss was variably defined as caregiver reported history of weight loss or objective weight loss and/or deviation from previous growth trajectory. This heterogeneity is also true for the study-level reference classifications, especially for unconfirmed tuberculosis. Some studies used a previous version of the NIH reference classification, which included probable and possible tuberculosis categories that we reclassified as unconfirmed tuberculosis, despite limitations using this approach.29 These may contribute to heterogeneities in estimating the association between the predictors and the outcome of tuberculosis. Finally, we note that using a prespecified prediction model, as we did, may lead to overfitting.30 Despite a reasonable summary O:E ratio for our model, the heterogeneity in study-level O:E ratio demonstrated in our internal-external cross-validation suggests that overfitting may be an issue. As more data become available, future investigation into the causes driving heterogeneity may inform more nuanced use of this algorithm within specific contexts and populations.
Pragmatic treatment-decision algorithms can lead to better detection of tuberculosis in children, with improved access to early treatment and reduced tuberculosis morbidity and mortality. Although we developed these algorithms using a thorough modelling analysis of a large high-quality IPD dataset, the disappointing specificity of the scoring component suggests that improved diagnostic tools will be necessary to meet sensitivity and specificity targets. As these diagnostic tools are identified, their data may be incorporated into treatment-decision algorithms to improve the specificity of the algorithms while maintaining high sensitivity. Additionally, decision-analytic modelling of the relative “cost” of false positive versus false negative classification of tuberculosis and prevalence of tuberculosis may provide insight to select an appropriate sensitivity threshold in future algorithm development.
Treatment-decision algorithms are now conditionally recommended by the WHO in the evaluation of children with presumptive tuberculosis, which should lead to improved diagnostic capacity at and treatment initiation at primary healthcare centres where paediatric tuberculosis expertise may be lacking. This work represents a paradigm shift in pragmatic and evidence-based approaches using advanced analytic methods to develop algorithms that draw on the best globally available data. This approach can be further improved and interrogated as additional data and diagnostic tools become available.
Supporting information
Data Availability
Data is available upon written request to the authors.
ROLE OF THE FUNDING SOURCE
This work was supported by the WHO Global Tuberculosis Programme (GTB) as well as the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the United States National Institutes of Health (NIH). It was used to inform WHO consolidated guidelines on tuberculosis management of TB in children and adolescents and the accompanying operational handbook, both published by WHO in March 2022. This research was also supported in part by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the Centers for Disease Control and Prevention (CDC).
Disclaimer
Staff from the WHO GTB as well as staff from the CDC are co-authors of this paper, offered feedback on the methodology and investigation, reviewed the manuscript, and made the decision to submit. The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions or policies of the NIH, the WHO, the CDC, and PEPFAR.
ACKNOWLEDGEMENTS
KSG was supported by the NIH the Eunice Kennedy Shriver National Institute of Child Health and Human Development [F30HD105440] as well as the Yale Medical Scientist Training Program [T32GM007205]. SA, SB and SK received funding support from USAID through Research for Decision Makers’ Activity, and the paper was prepared through the Structured Operational Research and Training Initiative (SORT IT), a global partnership led by the Special Program for Research and Training in Tropical Diseases at the World Health Organization. LMC was supported by the NIH/NIAID (K23 AI143479). RS was supported by the NIH through the Eunice Kennedy Shriver National Institute of Child Health and Human Development (K23HD072802). MMVDZ is supported by a career development grant from the EDCTP2 program supported by the European Union (TMA2019SFP-2836 TB lung-FACT2), the Fogarty International Center of the National Institutes of Health (Award Number K43TW011028) and by funding from the South African Medical Research Council. EW was supported by a scholarship for doctoral studies from the Medical Research Council of South Africa under MRC Clinician Researcher Programme, the Faculty of Medicine and Health Sciences at Stellenbosch University (Early Career Grant and Temporary Research Assistantship grant), the Harry Crossley Foundation and the South African National Research Foundation (Thuthuka programme funding for doctoral students). HJZ and MPN received funding from the NIH (R01HD058971) and the Regional Prospective Observational Research in Tuberculosis Consortium, co-funded by MRC of South Africa and NIH. TC and JLW were supported by the NIH through the Institute of Allergy and Infectious Disease [R01AI147854 and R01AI137093, respectively]. JAS was supported by a Clinician Scientist Fellowship jointly funded by the UK Medical Research Council (MRC) and the UK Department for International Development (DFID) under the MRC/DFID Concordat agreement (MR/R007942/1).
We acknowledge the patients and their caregivers for their willingness to participate in each individual study. We hope that their contribution will benefit other families affected by tuberculosis. We additionally would like to thank the following individuals who contributed to execution of this work: Albert Okumu, Andrea T. Cruz, Carlos M. Perez-Velez, Chad Heilig, Colleen Wright, Elisha Okeyo, Ha Thi Minh Ðang, Hendrik Simon Schaaf, James Orwa, Lazarus Odeny, Lesley Workman, Maria R. Jaswal, Mariaem Andres, Mark Fajans, Moe Zaw, Parisa Hariri, Patrice Ahenda, Prisca Rabuogi, Rose Abwunza, Rumana Nasrin, Sabrina Choudhury, Scott Lee, Shaikh Shumail, Shoaib Ahmed, Susan Musau, Syed Mohammad Mazidur Rahman, Walter Mchembere, Yi Kyaw, Leonardo Martinez, Vivian Cox, Bryan Vonasek, and Alexander Kay. Finally, we would like to acknowledge the members of the WHO Guideline Development Group who considered this evidence to inform the 2022 WHO consolidated guidelines on the management of tuberculosis in children and adolescents.
DATA AVAILABILITY
Data is available upon written request to the authors.