“Complex models, marginal benefits--a multi-centre development and validation study of early warning scores across 2·16 million patient admissions addressing intercurrent medical interventions”
aSection of Health Data Science & AI, Department of Public Health, University of Copenhagen, Denmark
bSection of Epidemiology, Department of Public Health, University of Copenhagen, Denmark
cDepartment of Public Health, University of Copenhagen, Denmark
dDepartment Of Intensive Care, Center For Cancer And Organ Dysfunction, Copenhagen University Hospital - Rigshospitalet
eDepartment of Clinical Medicine, University of Copenhagen, Denmark
fDepartment of Emergency Medicine, Herlev Gentofte Hospital, Denmark
gDepartment of Cardiology, Herlev Gentofte Hospital, Denmark
hDepartment of Anesthesia and Intensive Care, Copenhagen University Hospital – Bispebjerg and Frederiksberg, Copenhagen, Denmark
iDepartment Of Anesthesia, Center For Cancer And Organ Dysfunction, Copenhagen University Hospital - Rigshospitalet
jCopenhagen Health Complexity Center, Department of Public Health, University of Copenhagen, Denmark
kMRC Centre for Global Infectious Disease Analysis, Imperial College London, London, United Kingdom
*Corresponding Author Details: Alexandros Katsiferis, PhD, Section of Health Data Science & AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark, Address: Bartholinsgade 6Q, DK-1356 Copenhagen K, Denmark, Tel: +45 52724022, E-mail: alexandros.katsiferis@sund.ku.dkSummary
Background
The National Early Warning Score (NEWS) is a nationally recommended, clinically implemented system, used to prevent patient deterioration. While numerous studies have compared predictive models for clinical deterioration, large-scale evaluations of their potential clinical utility remain undetermined. Here, we compared NEWS’s clinical net benefit against simplified scoring rules and modern machine learning to determine whether simpler approaches are sufficient or if complex models provide meaningful advantages in clinical practice.
Methods
We included fifteen Danish hospitals with over 2·16 million patient admissions representing 829 610 unique patients over five years (2018 to 2023). We compared NEWS against both simpler and more complex approaches for predicting 24-hour mortality: NEWS-Light (NEWS without blood pressure and temperature), DEWS (NEWS-Light with age and sex; DEWS denotes the Demographic Early Warning Score), and a model based on eXtreme Gradient Boosting (XGB-EWS) incorporating vital signs, demographics, laboratory markers, plus medical history embeddings extracted using sentence transformers. We used propensity score weighting to mitigate intervention bias and evaluated performance using Area Under the Receiver Operating Characteristic Curve (AUC), calibration, and net benefit.
Findings
XGB-EWS achieved the highest discrimination (AUC 0·932, 95% Confidence Interval [0·929-0·936]), followed by DEWS (0·908 [0·904-0·912]), NEWS (0·902, [0·898-0·906]), and NEWS-Light (0·879, [0·873-0·885]). Decision curve analysis showed maximum net benefit differences of 1·8 additional correct mortality identifications per 10 000 patients between XGB-EWS and NEWS, and 1·7 per 10 000 between NEWS and NEWS-Light, across the evaluated risk thresholds.
Interpretation
Machine learning approaches provided marginal clinical utility improvements over traditional scoring systems, with NEWS-Light showing small performance decrements compared to full NEWS. The clinical significance of these differences must be weighed against workflow optimization benefits, suggesting healthcare systems should evaluate trade-offs between predictive performance and operational efficiency when selecting early warning approaches.
Funding
Novo Nordisk Foundation
Research in context
Evidence before this study
Early warning score systems have evolved from single vital sign monitoring to standardized multivariable scores such as the National Early Warning Score (NEWS), to even more sophisticated machine learning and deep learning frameworks utilizing electronic health records data. All these early warning scores are primarily designed to guide clinical decision-making by helping identify patients at risk of clinical deterioration within hospitals. Despite these advances, validation studies predominantly focus on statistical metrics measuring discrimination performance rather than meaningful clinical utility. On Aug 12, 2025, we searched PubMed for English-language studies published in the last 10 years, using terms including “early warning score”, “NEWS”, “NEWS2”, “machine learning”, “artificial intelligence”, “decision curve analysis”, “net benefit”, “clinical utility”, and “24-hour mortality.” While most of the published models, including the more sophisticated machine learning ones, have demonstrated better discrimination compared to traditional early warning scores, we found only one study that combined early warning score validation with clinical utility analysis for short-term clinical deterioration. Additionally, no studies have evaluated clinical utility across multiple patient subgroups while using causal inference methods to address intervention bias in a large-scale healthcare system validation.
Added value of this study
The current study represents the largest multi-center early warning score validation to date, encompassing 2·16 million patient encounters across 15 Danish hospitals and nine distinct clinical specialties over a period of five years. We used the predictimand framework with causal inference methods to address intervention bias, a critical and novel methodological advance in the assessment of early warning scores. Unlike previous research that focused primarily on discrimination metrics, we evaluated clinical utility using decision curve analysis, providing evidence that sophisticated machine learning early-warning-score approaches delivered in overall only marginal clinical utility improvements (up to 1.7 additional correctly identified mortality cases per 10 000 patients) despite better discrimination. We find that a simplified version of NEWS (NEWS without blood pressure and temperature components) achieves a comparable clinical net benefit to full NEWS. Additionally, we provide the first quantitative assessment of the healthcare resources implications in this area of research, showing that simplified approaches could potentially redirect 98·1 full-time equivalent positions annually from routine vital sign collection to direct patient care, providing evidence for healthcare administrators to reallocate clinical resources toward patient interaction and care delivery.
Implications of all the available evidence
We demonstrated that differences in the ability of the models to identify 24-hour mortality cases correctly were less than 2 per 10 000 patients. Given that often in clinical practice, a proportion of these identifications translates to lives being rescued, the real-world clinical advantage is likely even more modest. These modest gains of machine learning approaches, coupled with NEWS-Light displaying marginal performance decrements compared to the full NEWS, challenge the prevailing emphasis on algorithmic complexity over clinical value. NEWS-Light has the potential to enable workflow optimization, freeing approximately 3 minutes per patient encounter without compromising clinical effectiveness. Future research should conduct prospective cluster-randomized controlled trials comparing NEWS versus NEWS-Light implementation, while the broader prediction modeling research community should adopt utility-based evaluation frameworks to ensure that algorithmic advances translate to tangible improvements in patient care rather than statistical superiority alone.
Article notes
Competing Interest Statement
CSM and EKA have founded a spin-out company, WARD24/7 ApS, with the aim of pursuing the regulatory and commercial activities of the WARD-project (Wireless Assessment of Respiratory and circulatory Distress, a project developing a clinical support system for continuous wireless monitoring of vital signs). WARD24/7 ApS has obtained license agreement for any WARD-project software and patents. One patent has been filed: Wireless Assessment of Respiratory and circulatory Distress (WARD), EP 21184712.4 and EP 21205557.8. No other authors declared conflicts of interests.
Funding Statement
This work is funded from the Novo Nordisk Foundation via The Novo Nordisk Young Investigator Award (NNF20OC0059309). AK acknowledges funding from the Novo Nordisk Foundation via The Novo Nordisk Young Investigator Award (NNF20OC0059309). NS and MPK acknowledge funding from The Danish National Research Foundation via a chair grant (DNRF160). TVV is supported by the "Data Science Investigator - Emerging 2022" grant from the Novo Nordisk Foundation (NNF22OC0075284). AGZ and The Copenhagen Health Complexity Center is funded by TrygFonden. SB acknowledges funding from the MRC Centre for Global Infectious Disease Analysis (reference MR/X020258/1). SB acknowledges support from the Novo Nordisk Foundation via The Novo Nordisk Young Investigator Award (NNF20OC0059309). SB acknowledges the Danish National Research Foundation (DNRF160) through the chair grant. SB acknowledges support from Schmidt Sciences via the Schmidt Polymath Award (G-22-63345).
Introduction
Early warning systems are important components of modern healthcare services, providing standardized approaches to identify patients at risk of clinical deterioration.1 Following its development by the Royal College of Physicians in 2012, the National Early Warning Score (NEWS) was implemented throughout National Health Service hospitals and has been adopted or validated in healthcare systems worldwide, including its implementation in Danish hospitals.1–8 The score ranges from 0 to 20 and consists of seven vital sign parameters: respiratory rate, oxygen saturation, supplemental oxygen requirement, systolic blood pressure, pulse rate, level of consciousness, and temperature. Each parameter is assigned points based on deviation from normal values, with higher scores indicating greater risk of adverse outcomes within 24 hours. In parallel with scoring systems, such as NEWS, there has been interest in developing statistical and machine learning models that leverage broader electronic health records for predicting clinical deterioration. These approaches utilize the probabilistic output instead of a score for each patient and are based on either logistic regression or more modern machine learning architectures.1,4,7,9,10 The latter typically incorporate laboratory results, medication and comorbidity histories, and procedural data to enhance prognostication beyond that provided by vital signs alone.
Retrospective studies of Early Warning Scores (EWSs) face methodological challenges that could undermine their clinical validity. One limitation is intervention bias: when effective early warning systems prompt interventions that rescue many high-risk patients, retrospective analysis can become problematic.11–13 In practice, if high-risk patients receive effective care and survive, then their observed outcome (higher survival, that is, lower mortality) deviates from the initial baseline prediction, due to the rescuing intervention in itself. Thus, retrospective analysis may create misleading associations, potentially making high scores appear to overestimate mortality, with the latter being more prominent as healthcare systems become more responsive to EWSs.14 This represents a core misalignment between how EWSs function clinically (as pre-intervention risk stratification tools) and how they are evaluated retrospectively (based on post-intervention outcomes). The pertinent clinical question – “What would the patient’s risk trajectory be without intervening on them?” – cannot be answered using approaches that measure observed outcomes after treatments have been administered. Beyond intervention bias, the literature has methodological gaps in performance measurement. While recent guidelines, such as TRIPOD+AI, now emphasize the importance of clinical utility metrics such as net benefit, the vast majority of validation research has focused on discrimination measures such as Area Under the Receiver Operator Characteristic Curve (hereinafter referred to as AUC) that are not designed to be interpretable in a clinical setting, cannot capture real-world clinical value or misclassification costs.1,7,15–18 Moreover, existing subgroup evaluations remain limited to age- and disease-specific strata using discrimination metrics rather than clinical utility measures,4,19–21, while evidence of practical implementation value, including cost-effectiveness and feasibility, remains scarce.22
To address these methodological limitations, we conducted – what we believe is – the largest multi-centre EWS development and validation study to date, comprising 2 ·16 million individual patient encounters across 15 Danish hospitals. We employed causal prediction approaches to mitigate intervention bias and estimate the natural EWS performance in the absence of clinical interventions. Our primary aim was to evaluate the discriminative performance, calibration, and potential clinical utility of NEWS for predicting 24-hour mortality, comparing it with simplified variants and machine learning models incorporating laboratory results, previous medical procedures, and diagnoses. Secondary aims included assessing performance heterogeneity across subgroups using clinical utility metrics and estimating the healthcare resource implications of implementing simplified approaches.
Methods
Study Design & Data
We conducted a multi-center retrospective cohort study, using data from 15 hospitals (10 hospital networks) across the Capital Region and Zealand Region of Denmark, between October 16, 2018, and October 16, 2023, including a diverse range of clinical settings, specialties, and patient populations. Since every Danish citizen has a unique identification number, we directly linked their demographic information (date of birth, sex, and date of death) from the Population Statistics Register with their clinical data available in the hospitals’ records. All centers routinely collected NEWS data as part of standard clinical care during the study period. All patients were followed until death or completion of the 24-hour observation period after the initial NEWS score was recorded, whichever occurred first, with no patients lost to follow-up. The study adheres to the TRIPOD+AI guidelines.
Participants
We included all adult patients (≥18 years) admitted to the participating hospitals during the study period who had at least one NEWS assessment recorded and had hospital stays of more than 15 minutes. For individuals with multiple measurements taken during a single admission, we used the first one to establish a baseline risk before clinical interventions. Patients could contribute multiple separate hospital admissions to the study, with each admission treated as an independent encounter for analysis purposes.
Outcome
The outcome used was all-cause mortality within 24 hours of the initial NEWS recording. We selected that time horizon as it represents the timeframe for which NEWS was originally designed, is the most frequently evaluated period in EWS validation studies, and is clinically relevant for the intended use of these systems, which aim to identify patients at imminent risk of deterioration.1,14
Predictors
We extracted vital sign parameters routinely collected during clinical care: respiratory rate, oxygen saturation, supplemental oxygen, systolic and diastolic blood pressure, pulse rate, level of consciousness, and temperature. Laboratory parameters included hematological markers (hemoglobin, leukocytes, and platelets), biochemical indicators (creatinine, alanine aminotransferase, lactate dehydrogenase, albumin, and C-reactive protein), and specialized tests (arterial and venous lactate and troponin T). For patients with multiple measurements of the same marker, we calculated the median across all measurements of that specific test from the patient’s medical history prior to and including the time of the initial NEWS recording. We used the Danish Healthcare Classification System (Sundhedsvæsenets Klassifikationssystem (SKS) codes) to acquire data on previous diagnoses and medical procedures from prior hospitalizations, also including Intensive Care Unit (ICU) or respiratory support episodes. Both laboratory data and clinical history variables were only available from 2018 onwards due to data system implementation. Additional predictors included demographic variables (age, sex), hospital of admission, number of previous admissions, and temporal variables related to the NEWS recording (day of week, time of day, month). We further captured SKS codes registered within 24 hours after the initial NEWS measurement to define exposure to major interventions for subsequent causal prediction analyses.
Sample Size Considerations
We performed sample size calculations using the pmsampsize package to assess adequacy for prediction model development.23 Based on an anticipated AUC of 0·8, 100 predictor parameters for the most complex model, and an observed 24-hour mortality prevalence of 0·22%, the minimum required sample size was 289 823 admissions with 638 mortality events to achieve an events-per-parameter ratio of 6·38.
Missing Data
Median and mode imputation were respectively applied to continuous and categorical variables. Missing values in clinical history text fields (SKS descriptions) were preserved as missing indicators and handled implicitly through the text processing pipeline. For the eXtreme Gradient Boosting machine learning model (XGB-EWS), which was the only approach that incorporated laboratory parameters as predictors, the algorithm’s built-in missing value treatment handled missing laboratory values implicitly. We provide a list with missingness percentages per variable in the final dataset (Appendix, p 2).
Analytical Methods
We processed the clinical history text in Danish from previous diagnoses and medical procedures using text embeddings, which are high-dimensional numerical representations of textual data. Specifically, we used the “minishlab/potion-multilingual-128M” model, a static embedding sentence transformer distilled from a larger Transformer architecture.24,25 The inputs to that model were the concatenated structured text strings of previous diagnoses and medical procedures, with the output being 256-dimensional embeddings. Using standard Principal Component Analysis (PCA), we reduced this to 60 principal components, retaining 88.9% of the original variance. A composite embedding score was also created by averaging all principal components for use in the propensity score model, as mentioned below.
We defined 24-hour mortality risk in the absence of major interventions as the variable of interest to be predicted by the study’s models. Major clinical interventions were defined as ICU admission, respiratory and ventilatory support, surgical operations, and anesthesia administration, within 24 hours following the initial NEWS recording. We defined the study’s target population as the entire population of adult patients with a hospital stay > 15 minutes. Since some of those patients received major rescuing interventions due to a predicted high risk at baseline, we applied inverse probability weighting (IPW) in our EWS validation to correct for the intervention bias that would have distorted our results had we naively kept these patients in the evaluation (i.e. artifactual overestimation of mortality) or had we naively discarded those patients without weighting the remaining analytical sample (i.e. selection bias). IPW allowed us to weight the patients without major interventions such that they approximated the overall target population. In short, IPW allowed us to estimate the 24-hour mortality risk that would be observed without major interventions in the target population. This approach is referred to as “hypothetical strategy” in the novel predictimand framework. 12 To compute IPW, we estimated propensity scores for receiving major interventions using covariate balancing propensity scores, creating weights that made the analytical sample of patients without major interventions representative of the overall target population. The propensity score model included baseline characteristics measured at the first NEWS recording, number of previous hospitalizations, history of ICU/respiratory support, and the clinical history embedding composite score. We assessed covariate balance using standardized mean differences to evaluate whether patients without major interventions were representative of the overall hospitalized cohort. IPW validity relied on sufficient covariates for exchangeable predictive performance across intervention groups.
We developed three logistic regression models with different EWSs as predictors: NEWS, NEWS-Light (using a simplified score that excludes temperature and blood pressure parameters), and DEWS (Demographic Early Warning Score: age and sex in addition to simplified vital sign scoring), a modified version of the model proposed by Candel et al. (2023).4 We further developed a machine learning tree-based model, XGB-EWS, with all available predictors, including NEWS’s vital sign set plus diastolic blood pressure, laboratory values, history of ICU/respiratory support, clinical history embedding components, demographic variables (age, sex), and temporal features. All models incorporated the weights in both the model fitting and performance evaluation. A detailed description of the models used can be found in the Appendix (pp 3—4).
Model performance was assessed on the basis of discrimination, calibration, and clinical utility using grouped ten-fold cross-validation with hospitals as clusters to account for hospital-level effects. Discrimination was measured using the AUC, i.e., the probability that the model assigned higher risks to patients who died than to those who survived. Calibration was assessed using the Brier score, measuring the mean squared difference between predicted probabilities and observed outcomes, and through visual inspection of calibration plots. Clinical utility was evaluated using decision curve analysis, which compares the net benefit of using each model to guide clinical decisions against default strategies of treating all patients or treating no patients across different risk thresholds. Risk thresholds represent the probability cutpoints above which clinicians would initiate interventions or escalate care; for instance, a threshold of 0.01 (1%) means that patients with predicted mortality risk above 1% would receive intervention, while those below would not. We selected the risk thresholds as multiples of the weighted mortality prevalence (1×, 2×, 4×, 8×, 10×, 20×, and 30×), with subgroup analyses using subgroup-specific prevalences. The net benefit quantifies the clinical value of using a prediction model by calculating the proportion of true positives minus the proportion of false positive mortality cases weighted by the odds of the risk threshold. The latter reflects the relative importance of avoiding false positives versus identifying true positives at that specific threshold. An overview of the study’s methodology is presented in Figure 1.
Role of the funding source
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Results
During the study period from October 16, 2018, to October 16, 2023, we identified 18 877 954 EWS measurements from 2 355 905 hospital encounters across 899 100 unique patients in 10 hospital networks in Denmark. After excluding admissions with hospital stays of ≤15 minutes and restricting the analysis to the first NEWS assessment per hospitalization, the final cohort comprised 2 353 543 encounters of 898 662 individuals. We excluded those observations exposed to major 24-hour interventions (8% of the total), which resulted in a dataset of 2 161 689 hospital encounters of 829 610 individuals. We subsequently performed inverse probability weighting using the propensity score model to make this remaining sample comprising admissions without major interventions representative of the overall hospitalization cohort (Appendix, pp 5—6). The baseline characteristics of the final weighted cohort are presented in Table 1, stratified by mortality status. The weighted prevalence of 24-hour mortality was 0·22% (5 262 deaths), with the median NEWS score (Interquartile Range) being 1 (2·00), but higher (7·00 (6·00)) among patients who died within 24 hours. The weighted population comprised a majority (51·04%) of patients aged 18–65 years, with 50·36% female patients and 65·58% admitted to internal and acute medicine clinical specialties.
In terms of overall discrimination, XGB-EWS demonstrated the highest performance with an AUC of 0·932 (95% Confidence Interval 0·929–0·936), followed by DEWS at 0·908 (95% CI 0·904–0·912), NEWS at 0·902 (95% CI 0·898–0·906), and NEWS-Light at 0·879 (95% CI 0·873–0·885). The hospital-specific discriminative performance of the models (Figure 2) showed consistent patterns across the strata. We assessed model calibration using weighted deciles, comparing predicted versus observed 24-hour mortality frequencies across risk deciles. For the overall population, all models demonstrated reasonable calibration across the risk spectrum (Figure 3). When stratified by age group, a pattern of overestimation of risks was observed for the youngest group, which was balanced in those between 66 and 80 years of age, and reversed into underestimation in the oldest group for all models except the DEWS. The sex-stratified calibration analysis revealed similar calibration performance between male and female patients across all models. Hospital-specific Brier scores and detailed subgroup calibration analyses are provided in the Appendix (pp 7—9).
We conducted a decision curve analysis to evaluate the clinical utility of each model across a range of risk thresholds, the latter chosen as multiples of the weighted prevalence in the data. Table 2 shows the Net Benefit and the corresponding differences of XGB-EWS and the NEWS versions. For example, at a risk threshold of 0·0180 (8× weighted prevalence), XGB-EWS achieved a net benefit of 8·9 per 10 000 patients compared to NEWS’s 7·1 per 10 000. The maximum net benefit difference across the overall sample between XGB-EWS and NEWS, as well as NEWS and NEWS-Light across the evaluated thresholds, was 1·8 and 1·7 per 10 000 patients, respectively. We provide a full decision curve for all models and the overall population in the Appendix (p 10). Across all evaluated subgroups, the net benefit differences between XGB-EWS and NEWS were the highest in patients admitted with a NEWS score of 4 or higher at admission (maximum 17·4 per 10 000 patients), those above 80 years of age (maximum 6·2 per 10 000), and patients with circulatory system diseases (maximum 5·3 per 10 000). NEWS versus NEWS-Light comparisons showed maximum differences of 14·0 per 10 000 patients with a NEWS score of 4 or higher at admission, 5·5 per 10 000 patients with blood-related diseases, and 5·0 per 10 000 patients with circulatory system diseases. The rest of the subgroup comparisons showed differences remaining below 4·5 per 10 000 patients for both comparisons (Appendix, pp 11—18).
We conducted a full-time equivalent (FTE) analysis to estimate the potential workforce impact of implementing NEWS-Light (Figure 4). Over the 5-year study period, the 10-hospital system recorded 18 877 954 EWS assessments (this includes all assessments per admission without exclusion), corresponding to 3 775 591 annual assessments. Assuming a conservative 3-minute time saving per assessment by removing blood pressure and temperature measurements, this represented 188 779 annual hours of potential workflow optimization. Based on the Danish healthcare system standard of 37 working hours per week (1924 hours annually per FTE), this could potentially correspond to 98·1 full-time equivalent positions (188 779 ÷ 1924 = 98·1). This must be considered further conservative as we assume all hours employed are effective work hours, which is not achievable in a clinical reality.
Discussion
This large-scale validation study of 2·16 million hospital encounters represents the most comprehensive evaluation of EWSs to date, exceeding all previously reported validation studies. We found that our gradient boosting tree-based model achieved superior discriminative performance compared to traditional scoring systems, consistent with previous research.7,9,26,27 However, when evaluating potential clinical utility through decision curve analysis, an assessment rarely performed in EWS studies1, the net benefit differences were small and likely of questionable clinical significance. While the decision curve analysis showed measurable differences between models, net benefit calculations represent additional case identifications rather than lives rescued. Thus, the detection of up to 1·8 additional correct mortality identification per 10 000 patients does not directly translate to mortality reductions, as clinical outcomes depend on how much an EWS score is used in clinical decisions, the subsequent care quality, treatment availability, patient factors beyond the model’s reach, and, importantly, the severity of illness. The marginal clinical utility gains observed suggest that the additional complexity and implementation challenges of advanced models may not be solely justified when traditional scoring rules perform equivalently in terms of actionable clinical benefit across diverse hospital settings and patient populations.
The introduction of NEWS-Light, which eliminates blood pressure and temperature measurements while maintaining comparable predictive performance, is a robust and efficient alternative to improve workflow. By removing two of the most time-consuming and frequently impractical vital sign measurements, NEWS-Light may reduce workload during routine assessments. Blood pressure measurements, particularly in acutely ill patients, often require multiple attempts and can cause discomfort28, while temperature measurements may require patient repositioning and extended contact time, which could have been reallocated to other patients in need of care. The elimination of these measurements could potentially reduce each assessment episode by an estimated 3 minutes29, resulting in time savings that could be redirected toward patient interaction and care. From a patient-centered care perspective, this workflow optimization could enable healthcare providers to spend additional time listening to patient concerns, providing explanations about their condition, and addressing psychosocial needs that are often overlooked during busy clinical rounds.30 The 98·1 annual full-time equivalent positions worth of time savings identified across our hospital system reflects potential capacity that could be reallocated from routine data collection to direct patient engagement. Additionally, reducing the frequency of impractical procedures may decrease patient anxiety and improve cooperation with care protocols, particularly important for elderly patients who comprised 19% of our cohort and often experience discomfort or sleep disruptions during repeated vital sign assessments.
Our study has several limitations. The retrospective observational design limits causal inference about the effectiveness of simplified EWS, and the Danish healthcare context may limit generalizability to other healthcare systems with different organizational structures or data availability. Despite our propensity score weighting approach incorporating vital signs, demographics, medical history embeddings from natural language processing, laboratory values, and hospital-specific factors, unmeasured variables from unobserved clinical factors such as clinical intuition, real-time resource constraints, or rapid clinical changes that influence both major intervention decisions and predictive performance of EWSs may persist, potentially biasing our hypothetical risk estimates. Additionally, our definition of major clinical interventions, while clinically informed, could be refined in future studies to explore sensitivity to different criteria. While our study evaluated 24-hour mortality prediction, future research could assess performance over other clinical deterioration endpoints and longer time horizons. Most importantly, while our study demonstrates comparable predictive performance and quantifies potential efficiency gains, clustered randomized controlled trials are needed to evaluate real-world implementation effectiveness. However, our study has several strengths that advance the field. Methodologically, our study is the first to apply the predictimand framework with causal inference methods to address intervention bias in EWS validation, providing a rigorous approach for estimating performance under counterfactual conditions that previous studies have not achieved. The scale of 2·16 million encounters provides the necessary robust statistical power for detecting clinically meaningful differences across diverse patient populations and enables detailed subgroup analyses in EWS research. The multi-hospital validation across 15 hospitals in 10 hospital networks provided thorough validation of model performance, demonstrating consistent results across varied institutional settings and patient populations, strengthening confidence in our findings. Uniquely, our study provides measurable estimates of potential healthcare resource implications, quantifying 98·1 full-time equivalent positions worth of time savings, offering healthcare administrators data for implementation decisions.
In conclusion, these findings indicate that healthcare systems should carefully weigh the trade-offs between predictive performance and workflow optimization when implementing EWSs. While simplified scoring approaches showed small performance decrements, the potential for substantial time savings suggests that adoption decisions should balance these competing considerations based on local priorities and resources.
Supporting information
Data Availability
Access to the dataset supporting the conclusion of this article can only legally be accessed on acceptance from the Danish Data Protection Agency, the government body that regulates the use of Danish registry data. Interested parties can contact (PBN & KKI) for further information. We confirm that others, given a suitable project judged by the Danish Data Protection Agency, would be able to access these data in the same manner as the authors. We also confirm that the authors do not have any special access privileges that others do not have. The codebase for the analysis of the current study can be found online: https://github.com/MLGlobalHealth/Early_Warning_Scores
Declaration of interests
CSM and EKA have founded a spin-out company, WARD24/7 ApS, with the aim of pursuing the regulatory and commercial activities of the WARD-project (Wireless Assessment of Respiratory and circulatory Distress, a project developing a clinical support system for continuous wireless monitoring of vital signs). WARD24/7 ApS has obtained license agreement for any WARD-project software and patents. One patent has been filed: “Wireless Assessment of Respiratory and circulatory Distress (WARD), EP 21184712.4 and EP 21205557.8”. No other authors declared conflicts of interests.
Data sharing
Access to the dataset supporting the conclusion of this article can only legally be accessed on acceptance from the Danish Data Protection Agency, the government body that regulates the use of Danish registry data. Interested parties can contact (PBN & KKI) for further information. We confirm that others, given a suitable project judged by the Danish Data Protection Agency, would be able to access these data in the same manner as the authors. We also confirm that the authors do not have any special access privileges that others do not have. The codebase for the analysis of the current study can be found onlinehttps://github.com/MLGlobalHealth/Early_Warning_Scores
Acknowledgments
This work is funded from the Novo Nordisk Foundation via The Novo Nordisk Young Investigator Award (NNF20OC0059309). AK acknowledges funding from the Novo Nordisk Foundation via The Novo Nordisk Young Investigator Award (NNF20OC0059309). NS and MPK acknowledge funding from The Danish National Research Foundation via a chair grant (DNRF160). TVV is supported by the “Data Science Investigator - Emerging 2022” grant from the Novo Nordisk Foundation (NNF22OC0075284). AGZ and The Copenhagen Health Complexity Center is funded by TrygFonden. SB acknowledges funding from the MRC Centre for Global Infectious Disease Analysis (reference MR/X020258/1). SB acknowledges support from the Novo Nordisk Foundation via The Novo Nordisk Young Investigator Award (NNF20OC0059309). SB acknowledges the Danish National Research Foundation (DNRF160) through the chair grant. SB acknowledges support from Schmidt Sciences via the Schmidt Polymath Award (G-22-63345). We acknowledge the use of BioRender for the creation of Figure 1 (Katsiferis, A. (2025) https://BioRender.com/f1b7o4s.