In this study, we developed a feature-rich neural network-based survival algorithm, PMHnet, for prediction of all-cause mortality in patients with IHD using data from 34,746 Danish patients. With the aim of providing predictions that can be used to guide treatment and care, the model was developed to operate with an index date immediately after the diagnosis-confirming coronary angiography and with a prediction horizon of five-years. The model was tested using data from 5,000 Danish patients and externally validated using data from 8,288 Icelandic patients. We found that PMHnet had excellent discrimination with tdAUCs ranging from 0.88 at six months to 0.82 at five years. Similar results were found on the external Icelandic data, which confirms that the model and its deep feature foundation generalized well to novel patients and a different healthcare setting. Evaluated on the Danish data, we found the model to be well-calibrated with predicted probabilities accurately reflecting the observed proportions, also in different risk strata.

To aid the clinical interpretation of model predictions, we used SHAP-values to highlight the most impactful features and to explain how the different features affect the prediction for the individual patient. Model explainability is important for evaluating the model output and is paramount for the clinical adaption of any ML-model³⁸, including focus on features that are clinically actionable, i.e. modifiable versus non-modifiable factors.

Compared to the GRACE2.0 score, which is widely considered the gold-standard risk-stratification tool in current clinical use for predicting mortality after acute coronary syndrome (ACS)³⁹, PMHnet had superior discrimination and calibration. However, it is important to note that there are differences between the intended patient populations for the two models. The GRACE2.0 score has been developed using a derivation cohort of patients with STEMI, n-STEMI, and unstable angina with time of initial admission as time-zero⁵. In contrast, our study used time at coronary angiography as its baseline and was applied to all patients with IHD and coronary artery pathology ranging from diffuse atheromatosis to three-vessel disease. The validity of GRACE for a cohort with such characteristics has not been established. To provide a direct comparison of the two algorithms, we used the GRACE2.0 features and re-fitted the GRACE2.0 using our training data. The re-fitted GRACE2.0 score had the same model discrimination and better model calibration than the original version (Fig. 2), but still had inferior prediction compared to PMHnet.

The above observations are in good agreement with the current literature on ML-based secondary risk-stratification models, which have found machine learning models to offer better performance than simpler, existing scores in current clinical use^{10–13, 40, 41}. A transition towards feature-rich models that utilize more of the available data can therefore be an advantage. The 584 features used in our final model are neither bespoke nor specifically collected for this decision-support application, and instead represent clinical information gathered during routine work-up, management and treatment of patients. Using models that rely on several hundred features means that the current practice of manually entering data into a webtool becomes very impractical⁴². Instead, novel risk-prediction models need to be fully integrated in the EHR systems such that data can be automatically pulled and integrated.

Among previously published machine-learning models for secondary prediction in IHD, our study has several strengths. Firstly, whereas almost all the existing literature uses binary classification, the use of survival or time-to-event models is less explored. One notable exception is the model reported by Steele et al.¹⁰ which employed random survival forests and elastic net Cox regression to predict mortality in patients (n=80,000) with a history of coronary artery disease. One of the defining characteristics of survival models is the ability to handle censored data⁴³, which in other types of models would have been left out. In addition, survival models can distinguish between “died after a week” and “died after 10 months” which e.g., would be identical in a 1-year binary classification model. To the best of our knowledge, we are the first to use neural network-based survival models in this context. Secondly, we externally validated our model using data from a different country. Most models in the literature were not suboptimal externally validated^{10, 11, 40, 41}, and among those that have been, only two used data from a different country^{12, 13}. Demonstrating that a given model can accurately predict beyond borders is crucial for generalizability. Thirdly, our model can predict all-cause mortality up to five years after the index date. Probably owing to the fact that survival models have not been used, most models in the domain exclusively operate with a prediction horizon of one or two years^{12, 13, 40, 41}. Longer prediction horizons might be necessary for long-term disease management.

Although the features we have used represent data collected from a typical clinical workflow and therefore should be generally applicable, inter-regional and inter-national differences in clinical practice may affect what data is available and when. This is for instance exemplified by differences in diagnostic work-up, or timely access to coronary angiography, but may also relate to differences in access to previous medical data. Such aspects could affect the generalizability of our included features, but with internationally accepted treatment guidelines the differences should be minimal. Possible solutions are to reduce slightly the complexity of models to better match the intersection of features available across countries/regions and/or re-fitting and possibly retraining the model each time it is deployed in a new setting. In our external validation we used data from Iceland, which in an international perspective is very similar to Denmark when comparing healthcare systems^{44, 45}. For that reason, we downscaled the model slightly to account for data availability, but re-training on Icelandic data was not deemed necessary. That being said, using the Icelandic data, we found evidence of miscalibration as the model was found to overestimate the risk for some patients and this suggests that further adjustment of the model is needed were it to be deployed in Iceland. However, since miscalibration does not affect the accurate risk-stratification of patients, we did not pursue that issue further in this study. For future studies, we note that techniques such as Platt-scaling or isotonic regression could be used to remedy calibration-issues⁴⁶.

The dynamic nature of clinical environments can lead to deterioration of model performance^{47, 48}. Advances in treatment and diagnosis mean that the baseline risk of patients with ischemic heart disease could change over time, which in turn would lead to a drift of model calibration. As an example, the logistic EuroSCORE⁴⁹, a pan-European risk-stratification model for cardiac surgery published in 2003 was since its inception gradually found to overestimate mortality⁵⁰. The EuroSCORE has since been replaced by EuroSCORE II which for the time being has remedied these issues⁵¹. This type of systematic decline of model performance has important implications for our model as well and necessitates that the model performance is continuously monitored to ensure acceptable up-to-date performance. The need for real-time monitoring of model performance is another strong argument for risk-stratification tools to be tightly integrated within EHR systems.

As a secondary analysis in this study, we tested adding a panel of polygenic risk scores to the input features in PMHnet and assessed how they might impact model performance. The 19 different genetic risk scores were included based on being related to cardiometabolic health and covered traits such as blood pressure, and total cholesterol, but also included heart failure and acute myocardial infarction. Where for example the acute myocardial infarction risk score is developed for primary risk prediction, i.e., disease development, our model is concerned with secondary risk prediction, i.e., modelling risk for those who already have the disease. It is known that CAD PRS associate with events in both primary and secondary event populations, however, these PRSs are PRSs of prevalent diseases, not mortality. Whether or not a primary risk score is useful in a secondary risk context is clear upfront, but as parts of the underlying disease process are known to be shared between the two, we hypothesized that the score could be used in our context. Focusing the analysis only on the subset of patients for which we could obtain genotypes (n=13,449, 37.4%), we found no significant difference in performance after adding the PRS scores to either model-1 (Diagnoses) and model-2 (Diagnoses + ClinicalOne). As the 585 features include the prior disease history recorded over more than 25 years, our interpretation is that a “realized” life-course disease trajectory is more informative than the germline risk that can be calculated from the genotype. Moreover, the disease trajectory also holds information on life-course exposures and may therefore also include exposure information that quite implicitly is related to genetic data only. The version of PMHnet trained on the disease history only, with and without genetics, indicates that there is little gain from the PRS tested in this case. As we did not have genetic data for the full cohort, we cannot exclude that our interpretation is affected by this aspect.

Prospective studies are needed to ascertain how feature-rich risk-stratification methods can be used to alter, guide, and hopefully improve treatment. The ability to accurately predict high-risk is useful for identifying patients that may benefit from more extensive treatment and more frequent visits at the hospital. In contrast, accurate identification of low-risk patients may potentially be used to limit work-up, extent of pharmacological treatment and follow-up content and intensity and thereby prevent potential harmful overtreatment. Striking the correct balance between over- and undertreatment can contribute to the advancement of precision medicine, and here a well-calibrated and highly discriminative risk-prediction model can serve as an important tool.

In routine cardiology, a multitude of diagnostic, prognostic, and treatment-related scores are applied. However, all the presently applied scores are based on a very limited number of features. The present findings indicate a significantly added value of applying far more features and of introducing machine-learning. Thus, precision treatments in cardiology may benefit from using more features and machine-learning to replace the present scores.