Risk Prediction Modelling of 30-day all-cause mortality following percutaneous coronary intervention in an Australian population: Leveraging Machine Learning
1School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
2School of Population Health, Curtin University, Perth, WA, Australia
3Department of Cardiology, Alfred Hospital, Melbourne, VIC, Australia
4Monash-Alfred-Baker Centre for Cardiovascular Research, Monash University, Melbourne, VIC, Australia
5Department of Cardiology, Royal Melbourne Hospital, Melbourne, VIC, Australia
6Victorian Heart Institute, Monash University, Melbourne, Australia
7Artificial Intelligence and Cyber Futures Institute, Charles Sturt University, Bathurst, NSW, Australia
8Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, VIC, Australia
*Corresponding author Mohammad Rocky Khan Chowdhury PhD Felllow, Department of Epidemiology and Preventive Medicine School of Public Health and Preventive Medicine Faculty of Medicine, Nursing and Health Sciences Monash University, 553 St. Kilda Rd., Level 4, Melbourne VIC 3004, Email: rurocky84@gmail.comAbstract
Background
Pre-procedural risk prediction of 30-day all-cause mortality after percutaneous coronary intervention (PCI) aids in clinical decision-making and benchmarking hospital performance. This study aimed to identify pre-procedural factors to predict the risk of 30-day all-cause mortality post-PCI using machine learning (ML) approaches.
Methods
The study analysed 93,055 consecutive PCI procedures. Boruta feature selection method was used to identify key predictive variables. Seven ML algorithms were employed for models’ development and validation. Model performance was assessed using standard metrics for validation dataset. SHapley Additive exPlanations (SHAP) method was used to explain leading predictive variables.
Results
Among the seven ML algorithms, the Extreme Gradient Booster (XGB) had the better performance across most metrics, such as accuracy (86.7%), root mean square error (36.5%), specificity (82.5%), precision (54.0%), F1 score (52.7%), and Brier score (13.3%). The XGB model also demonstrated strong discriminatory power, achieving a receiver operating characteristics-area under the curve (ROC-AUC) of 85.5% (95% CI: 83.5%–87.4%). The XGB model identified left ventricular ejection fraction (LVEF), acute coronary syndrome (ACS), estimated glomerular filtration rate (eGFR), age, and complex lesion as the five leading factors associated with 30-day mortality post-PCI. Other factors, in order, were cardiogenic shock, body mass index (BMI), intubated out-of-hospital cardiac arrest (OHCA), lesion location, mechanical ventricular support, gender, and peripheral vascular disease (PVD).
Conclusion
The XGB algorithm was identified as the best predictive model for 30-day all-cause mortality post-PCI. It is essential to underscore the need for further validation of the model with external data to ensure its applicability to other populations.
WHAT IS ALREADY KNOWN ON THIS TOPIC
- risk-adjustment model for an Australian percutaneous coronary intervention (PCI) patient population was previously developed to predict 30-day mortality post-PCI using traditional regression model.
- knowledge, patient characteristics, and clinical practices evolve over time, requiring frequent model updates to reflect new evidence, guidelines, and interventions
WHAT THIS STUDY ADDS
- A machine learning (ML)-based preprocedural risk prediction model for 30-day mortality post-PCI was developed. The Extreme Gradient Booster (XGB) model was identified as the top performer in predicting 30-day all-cause mortality post-PCI. The model selected left ventricular ejection fraction, acute coronary syndrome, estimated glomerular filtration rate, age, and complex lesion as the top influential factors.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
- Risk prediction models aid clinical decision-making, enhance patient counselling, improve care quality, inform healthcare policies, and advance research.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Summary of Updates:
INTRODUCTION
Percutaneous coronary intervention (PCI) is one of the most widely performed medical procedures,1 and is a highly effective in treating coronary artery disease 2. Whilst PCI is extremely safe, patients remain at risk of mortality.3 In 2020-2021, approximately 48,000 PCIs were performed in Australia, with 75% being performed in males.4 It is worth noting that similar to other cardiac procedures, approximately two-thirds of deaths after PCI occur within the first 30 days following the procedure.5 In Australia, the prevalence of 30-day all-cause mortality post-PCI was around 2%.6 The 30-day mortality is influenced by various factors including patient demographic and their pre-procedural clinical status.7–10 Therefore, in order to assess pre-procedural risk of 30-day all-cause mortality, it is essential to identify and understand it’s related factors.
A risk-adjusted model for predicting 30-day all-cause mortality post-PCI can aid physicians in selecting optimal interventions tailored to individual demographics, lifestyles, patient comorbidities, and clinical presentations. Such models are also essential when comparing outcomes across individual institutions and regions. Thus, this approach may enhance intervention quality by improving risk adjustment and subsequently may reduce post-PCI mortality as well as costs through cost-effective care strategies.11 12 While existing risk-adjusted models developed by clinical quality registries have been valuable, their applicability to contemporary PCI populations may be limited due to variations in study populations, healthcare systems, registry uniformity, risk factors, and methodological approaches.7–10 Additionally, many models in the literature traditionally rely on multivariable logistic and Cox’s regression methods for outcome prediction.7–10 13
In recent years, there has been a growing interest in the use of Machine Learning (ML) methods in developing risk prediction models.14–16 Traditional methods (e.g. logistic regression or Cox regression) require more structural data, greater human input for the verification of distributional assumptions and incorporation of application knowledge in choosing the input parameters.17 Conversely, ML approaches deal with high-level non-structural big data from patient databases. These are often able to detect sophisticated data patterns with a multitude of variables that can be tested for numerous interactions and nonlinear relationships with the outcome that traditional statistical methods are sometimes struggle to explain.18–20 Further, these approaches have been successfully applied to predict patient prognoses in many public health areas, such as the risk of readmission after hospital discharge, cancer progression, and diabetic complications.21–23 Current evidence indicated that ML methods outperformed traditional regression models in population-specific mortality studies.24–26 Though the application of ML has amplified in medical and health care in Australia, its potential applications in 30-day all-cause mortality post-PCI has not been extensively explored in a contemporary Australian population. Therefore, this current study aims to identify pre-procedural factors associated with 30-day all-cause mortality post-PCI for an Australian population, find the best ML approach and compare ML’s performance metrics with the traditional logistic regression method.
METHODS
Study Population
Data used in this study were collected by the Victorian Cardiac Outcomes Registry (VCOR) comprising 93,055 consecutive PCI cases from 33 (15 public and 18 private) participating Victorian (a state in Australia) hospitals between 1 January 2013 to 31 December 2021. Patient-centred demographics, comorbidities, procedural details, in-hospital and 30-day mortality were captured by the registry, where each PCI was treated as a separate observation. PCI procedures were excluded if they were not the index admission or had missing outcome measures.27
Outcome variable
The outcome variable for this study was 30-day all-cause mortality post-PCI.
Selection of potential factors of 30-day morality
An recent systematic review identified 17 key variables as significantly associated with 30-day mortality post-PCI, however only 11 of them were available in the VCOR.28 The available variables are: age, gender, PCI indication or acute coronary syndrome (ACS), cardiogenic shock, intubated out-of-hospital cardiac arrest (OCHA), left ventricular ejection fraction (LVEF), estimated glomerular filtration rate (eGFR), mechanical ventilation, history of diabetes, vascular diseases (peripheral vascular disease (PVD), and cerebrovascular disease (CVD)). The following five variables however were not available in the VCOR registry: hypertension, single or multivessel diseases, heart failure, Thrombolysis in Myocardial Infarction (TIMI) flow and urgency of PCI. The percutaneous entry location is determined by the physician’s decision, making it inappropriate for risk prediction, and was excluded from modelling. The following variables: body mass index (BMI), chronic total occlusion (CTO), lesion complexity (ACC/AHA Lesion Classification B2/C), previous coronary artery bypass grafting (CABG) and lesion location were not among the proposed 17 variables, however they were included in the list of potential factors based on consultation with expert interventional cardiologists. Operational definitions for all of these variables were presented in Supplementary Table S1.
Handling of missing data
A higher proportion of missing values was observed for LVEF (11.3%), followed by eGFR (7.5%) (Table 1). The missing values for individual variables are all below 15%, indicating the feasibility of performing multiple imputations.29 Missing values in this study were imputed using Multiple Imputations by Chained Equations (MICE) with fully conditional specification.30
Analyses
Patients’ characteristics were reported using mean+/-standard deviation (SD) or median and percentiles for numerical data, where apply, and percentages for categorical data. The multicollinearity was assessed by the Variance Inflation Factor (VIF) (VIF>10 indicates multicollinearity) (Supplementary Figure S1).31 Further, first-degree interaction effect between clinically relevant variables were also investigated. A Chi-square test was performed to assess the association between 30-day all-cause mortality status and the independent variables. A p-value ≤0.05 or less was considered as statistical significance.
The Boruta feature selection method was used to make a short list of the plausible variables of 30-day all-cause mortality. Boruta, a wrapper-based method using the random forest classifier, is known for its consistency and lack of bias, making it superior to other variable selection techniques.32 Seven ML models were developed using the short-listed variables. The most influential variables were identified based on the best-performing models and explained using the SHapley Additive exPlanations (SHAP) method.33 All data analyses were undertaken using Stata (version 18/ StataCorp LLC), and Python (version 3.12.2) statistical software packages.
ML model development and selection of best ML model
Based on previous literature,34–36 seven ML algorithms were employed to develop predictive models, which included Decision Tree (DT), Extreme Gradient Boosting (XGB), Gradient Boosting (GB), Linear Discriminant Analysis (LDA), Logistic Regression (LR), Random Forest (RF), and Stochastic Gradient Boosting (SGB) (Supplementary Table S2).
The data was split by 70% and 30%. The models were developed in 70% of entire dataset (training dataset) and rest of 30% data was used for validation (test dataset). The class imbalance of outcome was addressed using Adaptive Synthetic (ADASYN) resampling technique. Each training model was optimized with hyperparameter tuning using 10-fold cross validation protocol (Supplementary Table S3). Finally, the training models were validated in 30% validation dataset. The best ML model was selected based on the comparison of performance metrics (accuracy, root mean square error (RMSE), sensitivity/recall, specificity, precision, F1 score, receiver operating characteristics – area under the curve (ROC-AUC) curve with 95% confidence interval (CI), precision-recall (PR) curve with 95% CI, Brier score and calibration plot) in validation dataset (Supplementary Table S4). Schematic presentation of best ML model selection was shown in Figure 1.
Health services performance
The performance of individual hospitals regarding patients’ adjusted risk of 30-day all-cause mortality is depicted in funnel plots. Funnel plots were utilized as a visual tool for comparing health services by plotting estimates of risk-adjusted 30-day all-cause mortality rates against the total number of procedures performed.37 These plots are a valuable addition to performance monitoring systems. To generate the funnel plot, adjusted risk was computed for each patient using regression coefficients derived from traditional LR analysis with the variables identified by the best ML method.
RESULTS
Baseline patient characteristics
An overview of the baseline characteristics of the 93,055 patients is presented in Table 1. The average age was 66.5 (±11.9) years and 76% were male. Around 50.5% of patients presented with ACS and 22.7% had a prior history of diabetes. CVD was present in 3.6% and PVD was present in 3.5% patients. The rate of intubated OHCA was 1.1%, and 2.2% presented with cardiogenic shock, and 2.6% of patients had severe renal impairment (eGFR <30 mL/min/1.73 m2).
The overall 30-day all-cause mortality was 2.1%. Patients who presenting with cardiac arrest had the highest mortality rates (47.5%) followed by cardiogenic shock (44.1%). Further, patients with severely reduced ejection fraction (LVEF<30%) had a mortality rate of 15.2%, while those with severe renal impairment (eGFR<30 mL/min/1.73 m2) had a mortality of 10.5%. The mortality in elderly patients (80 years and over) was 4.3% while those with ST-elevated myocardial infraction (STEMI) had a mortality rate of 6.8% (Table 1).
Potential influential factors
The Boruta feature selection method identified following eight key factors influencing 30-day all-cause mortality post-PCI: LVEF, cardiogenic shock, ACS, intubated OHCA, eGFR, mechanical ventricular support, age and complex lesion. The factors BMI, PVD, lesion location and gender were selected as moderate influential. The factors CVD, CTO, diabetes, previous PCI and previous CABG were identified as having minimal or no impact on the mortality (Figure 2).
ML model selection
Twelve influential factors, including sex, which was forcibly included due to its significance,38 were used to assess the prediction performance of each of the ML models. Among the seven ML models, the XGB model demonstrated slightly better performance across most metrics, including accuracy (86.7%), RMSE (36.5%), specificity (82.5%), precision (54.0%), F1 score (52.7%), and Brier score (13.3%) (Table 2). The XGB model demonstrated satisfactory performance across other metrics as well, which include a sensitivity/recall of 76.5%, a ROC-AUC of 85.5% (95% CI: 83.5%–87.4%), a PR score of 31.4% (95% CI: 27.1%–35.5%), and improved calibration (Table 2 and Figure 3).
For sensitivity/recall, the SGB model outperformed the others (92.1%) (Table 2). In terms of discrimination power, the ML based LR model achieved the highest ROC-AUC score of 92.6% (95% CI: 91.2% - 93.7%) compared to 85.5% (95% CI: 83.5%–87.4%) of XGB model. The calibration curve for SGB model showed better calibration compared to other models (Figure 3). Also, ML based LR model achieved the highest PR score (43.8%, 95% CI: 39.3 – 48.5) (Figure 3). Furthermore, Supplementary Table S5 and Supplementary Figure S2 showed models’ performance for the training data set.
Sensitivity analysis
In the sensitivity analysis, all models were redeveloped and their ROC-AUC scores were evaluated for validation dataset without addressing class imbalance. Additionally, ROC-AUC was assessed using a dataset with missing values removed and without addressing class imbalance. In both cases, XGB model outperformed other ML models in prediction 30-day all-cause mortality post PCI. While ROC-AUC was assessed using a dataset with missing values removed and addressing class imbalance, LR model outperformed other ML models. Also, the XGB model demonstrated satisfactory performance with ROC-AUC score of 76.6% (95% CI: 74.0% - 79.6%) (Supplementary Figure S3).
Interpretability of top factors in SHAP plot of the XBG model
The SHAP plot in the Figure 4 provides insights into the influence and direction of various variables on predicting 30-day all-cause mortality post-PCI, as depicted by the distribution of red and blue dots. The five leading factors identified by the XGB model were LVEF, ACS, eGFR, age, and complex lesion. Other factors, in order, were cardiogenic shock, BMI, intubated OHCA, lesion location, mechanical ventricular support, gender, and PVD.
In Figure 4, the variables are ranked in order of importance, with those at the top having the greatest impact on the predictions. In the XGB model, LVEF was selected as the most influential predictor followed by ACS and eGFR. The plot also showed that severely reduced ejection fraction is the most influential predictors for 30-day all-cause mortality post-PCI. Other predictors of 30-day all-cause mortality included presentation with STEMI, severe renal impairment, age 80 years or older, B2/C lesion complexity, occurrence of cardiogenic shock, overweight or obesity, intubated OHCA, lesions located in the left main or graft vessels, use of mechanical ventricular support, female gender, and the presence of PVD (Figure 4).
Health services’ performance comparison
The effectiveness of the XGB model in predicting 30-day all-cause mortality was evaluated by assessing the performance of health services using a funnel plot. In the funnel plot in Figure 5, one out of the 33 health services fell outside the 95.0% control limit when using the variables selected by the XGB model. However, none of the 33 health services exceeded the 99.8% control limit, indicating a consistent and reliable assessment of health service performance (Figure 5).
DISCUSSION
The primary aim of this study was to compare various ML models and identify an optimized risk-adjustment model for predicting 30-day all-cause mortality post-PCI. The XGB model demonstrated the best overall predictive performance and was selected as the optimal model. The model identified the key variables most strongly associated with mortality, allowing surgeons and patients to assess accurately the preoperative risk for individuals undergoing PCI.
Traditionally, the risk modelling predominately relies on traditional LR model.7–10 13 ML models offer key advantages over LR models. Although ML does not always significantly outperform traditional models in accuracy, it excels in handling large, complex datasets by automatically detecting non-linearity and interaction without requiring manual input.39 In this study, seven most commonly used ML algorithms were employed to identify pre-procedural factors associated with 30-day all-cause mortality post-PCI. The XGB model, developed using 12 key factors selected by the Boruta method, and while applied to validation data set, it outperformed other models across most metrics, including accuracy, RMSE, specificity, precision, F1 score, and Brier score. Furthermore, it demonstrated excellent discrimination (ROC-AUC of 85.5%) and strong calibration, highlighting its robustness and reliability. However, a study from Taiwan found that the DT model demonstrated excellent predictive capability for 30-day mortality post-PCI,36 while in the United States, an Adaptive Booster classifier was identified as the optimal model for predicting various cardiac intervention outcomes.35 Furthermore, in an Italian population, the RF model was identified as superior in predicting 30-day all-cause mortality post-PCI.40 The variation in identified models across studies may be due to variation in population, differences in variable definition and availability of variable in the registries/database among others.28 Despite this variation, the findings suggest that ML-based models hold significant promise in developing predictive models for 30-day all-cause mortality post-PCI due to their high performance, capacity to handle large datasets, and ability to reduce the risk of overfitting in the presence of highly imbalanced outcome.18–20
The XGB model emphasized the pivotal role of LVEF as a primary factor significantly associated with 30-day all-cause mortality post-PCI. This aligns with previous research across diverse studies, underscoring LVEF as a critical factor linked to 30-day mortality post-PCI and affirming the robust performance of the model.8 10 41 In addition to LVEF, the XGB model identified ACS, eGFR, age, and complex lesion as the most influential factors associated with 30-day all-cause mortality post-PCI. These factors represent key patient-centered clinical and procedural characteristics that significantly influence post-PCI outcomes. Understanding them is clinically valuable for enabling early risk stratification, guiding informed clinical decisions, and supporting personalized care planning for patients undergoing PCI.42–45 A previous study conducted in Taiwan using a DT model identified hyperlipidemia, hypertension, diabetes, heart failure, stroke, and chronic kidney disease as factors associated with 30-day mortality post-PCI.36 It is worth noting that studies employing ML techniques to identify factors associated with 30-day mortality post-PCI remain limited within the Australian population context. However, an Australian study using a traditional LR approach (the VCOR risk adjustment model) identified nearly identical predictors to those found by the ML-based XGB model in the current study.6 Furthermore, consistent findings across multiple studies utilizing traditional LR models have underscored the importance of factors such as age, gender, BMI, LVEF, eGFR, cardiac arrest, ACS, mechanical ventricular support, PVD, CVD, and complex lesions as significant predictors of 30-day all-cause mortality post-PCI.6 10 28The current study did not include the percutaneous entry site, a variable featured in the VCOR model, which is specific to interventional cardiologists. However, this study focused on assessing patient-centred factors and deliberately excluded treatment-related variables to avoid compromising the accuracy of individual risk estimation in the risk adjustment model.
Risk-adjusted 30-day all-cause mortality prediction is disseminated to participating hospitals in the VCOR network, allowing for health service performance benchmarking against other health services via risk-adjusted funnel plots. The XGB model’s predictive ability in evaluating 30-day all-cause mortality post-PCI demonstrated encouraging results in assessing health service performance. The XGB model’s identification of critical risk factors for 30-day all-cause mortality post-PCI suggests the possibility of integrating within the VCOR risk-adjustment process and public reporting mechanisms. It may be beneficial to repeat these analyses when data over a longer period of time is available to reassess the presence of consistent differences in prediction.
Strengths and limitations
The current study unveiled several strengths and limitations. The inclusion of a large volume of data substantially contributed in enhancing the accuracy of predicting 30-day all-cause mortality post-PCI is the key strength. Another strength of this study is the ML model’s ability to rank the most influential variables associated with mortality. The proposed set of pre-procedural variables holds promise for boosting the model’s overall performance. However, the study is not without its limitations. Firstly, the data source encompassed patients from a specific geographic region in Australia, constraining the generalizability of the findings and necessitating validation in diverse populations. Secondly, this study focused solely on patient-level risk and, therefore, included only patient-centred variables, excluding treatment choices and health system-related factors. The study only explored a subset of ML models, leaving unaddressed the performance of methods that were not evaluated herein. ML models are not able to produce p-value (significance level) and beta-coefficient while selecting influential variables. Finally, it’s worth noting that due to the limitations of the availability of all 17 factors, such as hypertension, single or multivessel diseases, heart failure, Thrombolysis in Myocardial Infarction (TIMI) flow and urgency of PCI, proposed as per earlier study,28 that there may be missed opportunities for improving overall performance metrics.
CONCLUSION
In this study, seven ML based risk prediction models for 30-day all-cause mortality post-PCI were developed and compared. The XGB model was selected as the best performing model. This model selected LVEF, ACS, eGFR, age, and complex lesion as the top influential factors. This model has the potential to assist clinicians in early identification of patients with high risk factors and benchmarking health service performance. However, there is need for further validation utilising external data to ensure the applicability of these findings to other patients’ population.
Supporting information
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Abbreviations
- ACS
- Acute coronary syndrome
- AUC
- Area under the curve
- BMI
- Body mass index
- CABG
- coronary artery bypass grafting
- CI
- Confidence interval
- CTO
- Chronic total occlusion
- CVD
- Cerebrovascular disease
- DT
- Decision Tree
- eGFR
- estimated glomerular filtration rate
- GB
- Gradient Booster
- LDA
- Linear Discriminatory Analysis
- LR
- Logistic Regression
- MICE
- Multiple Imputations by Chained Equations
- ML
- Machine Learning
- OHCA
- Out-of-hospital cardiac arrest
- PCI
- Percutaneous coronary intervention
- PVD
- Peripheral vascular disease
- RF
- Random Forest
- ROC
- Receiver operating characteristic
- SGB
- Stochastic Gradient Boosting
- SHAP
- SHapley Additive exPlanations
- STEMI
- ST-elevated myocardial infraction
- TIMI
- Thrombolysis in Myocardial Infarction
- VCOR
- Victorian cardiac outcome registry
- XGB
- Extreme Gradient Booster
Contributors
Conceptualisation: MRKC, BB, DD, DS. Methodology: BB, MRKC, MNK. Analysis: MRKC, MAM, MSI. Manuscript drafting: MRKC. Manuscript review and critical revision: BB, DS, DD, AB, CMR, SN, JL, DPC. Visualisation: MRKC, MSI. Supervision: BB, DS, DD, MNK. Project administration: DS, DD. MRKC, BB are the guarantor.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Competing interests
None declared.
Patient consent for publication
Not applicable.
Ethics approval
The VCOR was primarily approved by the ethics committee at The Alfred Hospital, Melbourne, Australia (approval number 47/12), and also approved by each participating hospital, including the use of opt-out consent. The current project received ethical approval from the Monash University Human Research Ethics Committee (MUHREC), under the project reference number 2022-35388-80724.
Data availability statement
Data are available upon reasonable request. Anonymized personal data were obtained from the Victorian Cardiac Outcome Registry (VCOR) after ethical approval and a confidentiality assessment. In accordance with Australian laws and regulations, access to personal sensitive data is restricted to researchers who meet the legal requirements for such access. For inquiries regarding data access, please contact Dr. Diem Dinh.