Does the logistic EuroSCORE offer an advantage over the additive model
http://www.100md.com
《血管的通路杂志》
For the Providence Health System Cardiovascular Study Group, Providence Health System, Portland, Oregon, USA
Abstract
There are two versions of the EuroSCORE cardiac surgery risk model for mortality: the logistic regression which provides a predicted probability of death, and an additive score, a simplified approximation to the logistic probability, which can be easily calculated without a computing aid. A recent comparison of these two models' performance concluded that the logistic probability did not offer a distinct advantage over the additive score, and that the additive score was just as accurate as the logistic probability, even in high-risk patients. This conflicts with the conclusion of our previous study of the same issue. Thus, we did further analyses of our original data, following the approach of these authors. The additive and logistic EuroSCORE models were retrospectively applied to predict operative mortality in 23,463 cardiac surgery patients operated from 1997–2004 at Providence Health System hospitals. Both the logistic and additive risk predictions are higher than the observed mortality, and were calibrated downward to reconcile this difference. The observed-to-predicted ratio is relatively constant for the logistic model, but it varies with the risk score for the additive model. Thus, the logistic model can be recalibrated to fit the actual outcome well, but the recalibrated additive model cannot. The (recalibrated) logistic model is more accurate for different patient groups, including the high-risk patients, and should be preferred for clinical use.
Key Words: Logistic model; Additive model; EuroSCORE; Cardiac surgery; Mortality
1. Introduction
The EuroSCORE cardiac surgery risk model for mortality comes in two versions: a probability of death derived from a logistic regression model [1], and an additive score, a simplified approximation to the logistic probability [2], which can be calculated by hand, without any computing aid. A recent comparison of these two models performance by Shanmugam et al. concluded that ‘the logistic EuroSCORE did not offer a distinct advantage over the additive model and the logistic EuroSCORE was not more accurate even in high-risk patients’ [3]. Our recent study of this same topic came to the opposite conclusion: that the additive score is inferior to the logistic probability, and that it underestimates the probability of death for high-risk patients [4].
Our original approach was to consider the logistic result as the true probability and to show that the additive score was a poor approximation of it. Shanmugam et al. used another approach, to consider the observed mortality as the truth and to compare both EuroSCORE models to it. We used the ideas generated by this study to perform further analyses to support our previous conclusions, that the logistic probability is superior to the additive score and should be used whenever possible. Three critical aspects of their analyses were re-visited: (1) The logistic EuroSCORE over-predicted the mortality in their patient population (5.2% expected, 2.95% observed), yet it was not recalibrated to adjust for this baseline difference; (2) The discrepancy between the observed (O) mortality and the expected (E) mortality predicted by the model was measured by their difference (O-E), rather than by their ratio (O/E), so that high-risk patients would tend to have larger differences; (3) the patient groupings used for the analyses were defined by the additive scores rather than the logistic probabilities.
2. Materials and methods
The additive and logistic EuroSCORE models were applied to predict operative mortality for 23,463 patients who underwent cardiac surgery from 1997 to 2004 in the Providence Health System hospitals [4]. There were 69% CABG surgeries, 15% valve, 13% CABG plus valve and 3% other cardiac procedures, with a total of 852 operative deaths, or 3.6% mortality. Both EuroSCORE methods over-predicted the mortality: the logistic probability predicted 1944.5 deaths (the sum of the individual predictions), or 8.3% mortality, and the additive score predicted 1261.7 deaths, or 5.4% mortality.
To make the overall predictions aligning with the observed mortality, we calibrated the EurosSCORE predictions. The recommended way is by performing a new logistic regression with the logit of the probability as the only risk factor [5]. Since this approach would not work with the additive score, a simpler calculation was used to calibrate both the logistic and additive probabilities: the calibrated probability was set equal to the predicted probability times the ratio of total observed deaths to predicted deaths. Thus we multiplied each logistic probability by 3.6/8.3=0.438 and each additive score by 3.6/5.4=0.675. Statistical analyses were performed with S-PLUS version 6.1 (Insightful Corporation, Seattle, WA).
3. Results
The patients were grouped by their logistic probabilities into 10 equal sized subgroups. Fig. 1 (upper panel) shows the observed and the (uncalibrated) EuroSCORE mortality by subgroup. The logistic probabilitiy is higher than the observed mortality in all subgroups. The additive score is higher than the observed mortality except in the highest risk subgroup. The O/E ratio for the logistic model is relatively stable for all subgroups, but the O/E ratio for the additive model is not (Fig. 1, lower panel). The stability of the O/E ratio for the logistic probability implies that a recalibration will produce a good fit to the observed mortality.
Fig. 2 shows the same analyses using the recalibrated EuroSCORE. The upper panel shows that the recalibrated logistic mortality is closer to the observed than the additive score. The lower panel compares the O/E ratios for both EuroSCORE models; the O/E ratio of the recalibrated logistic probability is closer to 1 than the additive score.
4. Discussion
Shanmugam and colleagues compared the additive and logistic EuroSCOREs by presenting the O-E mortality for different risk groups [3]. They concluded that the additive EuroSCORE is a simple and easily applied system of risk assessment and that the logistic EuroSCORE does not offer a distinct advantage over the additive model, even in high-risk patients.
We respectfully disagree. In the present study, the O/E ratios are approximately constant for the logistic model but not so for the additive model (Fig. 1). Although the discrimination of the EuroSCORE, as measured by the ROC statistic, is excellent, as shown in our data and that of many others, it appears to consistently overestimate the mortality risk [3,6]. When the discrimination is good but the calibration is not, the model can be made more accurate by recalibration (although the opposite is not true) [7]. After recalibration, the logistic model fits the actual outcome better than the additive model (Fig. 2). This is a supernumerary analysis to confirm that the logistic model performs better than the additive model, and that the additive model underestimates the risk for high-risk patients, as shown previously using other statistical approaches [4]. Also, it appears from the data and figures presented in Shanmugam's paper [3], that using this same analysis on their data would lead to the same conclusion.
5. Conclusions
The logistic model has a relatively constant O/E ratio across risk groups, while the additive model does not. Thus, the recalibrated logistic model fits the actual outcome well, but the recalibrated additive model does not. The logistic model is more accurate for different patient groups and should be the primary choice for clinical use and group comparisons.
Acknowledgements
The following hospitals provided patient data: Providence Anchorage Medical Center, Anchorage, Alaska; Providence Everett Medical Center, Everett, Washington; Providence St. Peter Hospital, Olympia, Washington; Providence Yakima Medical Center, Yakima, Washington; Providence Portland Medical Center, Portland, Oregon; Providence St. Vincent Medical Center, Portland, Oregon; Providence St. Joseph Medical Center, Burbank, California; Providence Holy Cross Medical Center, Mission Hills, California; Providence Little Company of Mary Hospital, Torrance, California.
References
Roques F, Michel P, Goldstone AR, Nashef SA. The logistic EuroSCORE. Eur Heart J 2003; 24:881–882.
Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999; 16:9–13.
Shanmugam G, West M, Berg G. Additive and logistic EuroSCORE performance in high risk patients. Interact CardioVasc Thorac Surg 2005; 4:299–303.
Jin R, Grunkemeier GL. Additive vs. logistic risk models for cardiac surgery mortality. Eur J Cardiothorac Surg 2005; 28:240–243.
Peterson ED, DeLong ER, Muhlbaier LH, Rosen AB, Buell HE, Kiefe CI, Kresowik TF. Challenges in comparing risk-adjusted bypass surgery mortality results: results from the Cooperative Cardiovascular Project. J Am Coll Cardiol 2000; 36:2174–2184.
Yap CH, Mohajeri M, Ihle BU, Wilson AC, Goyal S, Yii M. Validation of EuroSCORE model in an Australian patient population. ANZ J Surg 2005; 75:508–512.
Harrell FE Jr, Lee KL, Califf RM, Pryor DB, Rosati RA. Regression modelling strategies for improved prognostic prediction. Stat Med 1984; 3:143–152.(Ruyun Jin and Gary L. Gru)
Abstract
There are two versions of the EuroSCORE cardiac surgery risk model for mortality: the logistic regression which provides a predicted probability of death, and an additive score, a simplified approximation to the logistic probability, which can be easily calculated without a computing aid. A recent comparison of these two models' performance concluded that the logistic probability did not offer a distinct advantage over the additive score, and that the additive score was just as accurate as the logistic probability, even in high-risk patients. This conflicts with the conclusion of our previous study of the same issue. Thus, we did further analyses of our original data, following the approach of these authors. The additive and logistic EuroSCORE models were retrospectively applied to predict operative mortality in 23,463 cardiac surgery patients operated from 1997–2004 at Providence Health System hospitals. Both the logistic and additive risk predictions are higher than the observed mortality, and were calibrated downward to reconcile this difference. The observed-to-predicted ratio is relatively constant for the logistic model, but it varies with the risk score for the additive model. Thus, the logistic model can be recalibrated to fit the actual outcome well, but the recalibrated additive model cannot. The (recalibrated) logistic model is more accurate for different patient groups, including the high-risk patients, and should be preferred for clinical use.
Key Words: Logistic model; Additive model; EuroSCORE; Cardiac surgery; Mortality
1. Introduction
The EuroSCORE cardiac surgery risk model for mortality comes in two versions: a probability of death derived from a logistic regression model [1], and an additive score, a simplified approximation to the logistic probability [2], which can be calculated by hand, without any computing aid. A recent comparison of these two models performance by Shanmugam et al. concluded that ‘the logistic EuroSCORE did not offer a distinct advantage over the additive model and the logistic EuroSCORE was not more accurate even in high-risk patients’ [3]. Our recent study of this same topic came to the opposite conclusion: that the additive score is inferior to the logistic probability, and that it underestimates the probability of death for high-risk patients [4].
Our original approach was to consider the logistic result as the true probability and to show that the additive score was a poor approximation of it. Shanmugam et al. used another approach, to consider the observed mortality as the truth and to compare both EuroSCORE models to it. We used the ideas generated by this study to perform further analyses to support our previous conclusions, that the logistic probability is superior to the additive score and should be used whenever possible. Three critical aspects of their analyses were re-visited: (1) The logistic EuroSCORE over-predicted the mortality in their patient population (5.2% expected, 2.95% observed), yet it was not recalibrated to adjust for this baseline difference; (2) The discrepancy between the observed (O) mortality and the expected (E) mortality predicted by the model was measured by their difference (O-E), rather than by their ratio (O/E), so that high-risk patients would tend to have larger differences; (3) the patient groupings used for the analyses were defined by the additive scores rather than the logistic probabilities.
2. Materials and methods
The additive and logistic EuroSCORE models were applied to predict operative mortality for 23,463 patients who underwent cardiac surgery from 1997 to 2004 in the Providence Health System hospitals [4]. There were 69% CABG surgeries, 15% valve, 13% CABG plus valve and 3% other cardiac procedures, with a total of 852 operative deaths, or 3.6% mortality. Both EuroSCORE methods over-predicted the mortality: the logistic probability predicted 1944.5 deaths (the sum of the individual predictions), or 8.3% mortality, and the additive score predicted 1261.7 deaths, or 5.4% mortality.
To make the overall predictions aligning with the observed mortality, we calibrated the EurosSCORE predictions. The recommended way is by performing a new logistic regression with the logit of the probability as the only risk factor [5]. Since this approach would not work with the additive score, a simpler calculation was used to calibrate both the logistic and additive probabilities: the calibrated probability was set equal to the predicted probability times the ratio of total observed deaths to predicted deaths. Thus we multiplied each logistic probability by 3.6/8.3=0.438 and each additive score by 3.6/5.4=0.675. Statistical analyses were performed with S-PLUS version 6.1 (Insightful Corporation, Seattle, WA).
3. Results
The patients were grouped by their logistic probabilities into 10 equal sized subgroups. Fig. 1 (upper panel) shows the observed and the (uncalibrated) EuroSCORE mortality by subgroup. The logistic probabilitiy is higher than the observed mortality in all subgroups. The additive score is higher than the observed mortality except in the highest risk subgroup. The O/E ratio for the logistic model is relatively stable for all subgroups, but the O/E ratio for the additive model is not (Fig. 1, lower panel). The stability of the O/E ratio for the logistic probability implies that a recalibration will produce a good fit to the observed mortality.
Fig. 2 shows the same analyses using the recalibrated EuroSCORE. The upper panel shows that the recalibrated logistic mortality is closer to the observed than the additive score. The lower panel compares the O/E ratios for both EuroSCORE models; the O/E ratio of the recalibrated logistic probability is closer to 1 than the additive score.
4. Discussion
Shanmugam and colleagues compared the additive and logistic EuroSCOREs by presenting the O-E mortality for different risk groups [3]. They concluded that the additive EuroSCORE is a simple and easily applied system of risk assessment and that the logistic EuroSCORE does not offer a distinct advantage over the additive model, even in high-risk patients.
We respectfully disagree. In the present study, the O/E ratios are approximately constant for the logistic model but not so for the additive model (Fig. 1). Although the discrimination of the EuroSCORE, as measured by the ROC statistic, is excellent, as shown in our data and that of many others, it appears to consistently overestimate the mortality risk [3,6]. When the discrimination is good but the calibration is not, the model can be made more accurate by recalibration (although the opposite is not true) [7]. After recalibration, the logistic model fits the actual outcome better than the additive model (Fig. 2). This is a supernumerary analysis to confirm that the logistic model performs better than the additive model, and that the additive model underestimates the risk for high-risk patients, as shown previously using other statistical approaches [4]. Also, it appears from the data and figures presented in Shanmugam's paper [3], that using this same analysis on their data would lead to the same conclusion.
5. Conclusions
The logistic model has a relatively constant O/E ratio across risk groups, while the additive model does not. Thus, the recalibrated logistic model fits the actual outcome well, but the recalibrated additive model does not. The logistic model is more accurate for different patient groups and should be the primary choice for clinical use and group comparisons.
Acknowledgements
The following hospitals provided patient data: Providence Anchorage Medical Center, Anchorage, Alaska; Providence Everett Medical Center, Everett, Washington; Providence St. Peter Hospital, Olympia, Washington; Providence Yakima Medical Center, Yakima, Washington; Providence Portland Medical Center, Portland, Oregon; Providence St. Vincent Medical Center, Portland, Oregon; Providence St. Joseph Medical Center, Burbank, California; Providence Holy Cross Medical Center, Mission Hills, California; Providence Little Company of Mary Hospital, Torrance, California.
References
Roques F, Michel P, Goldstone AR, Nashef SA. The logistic EuroSCORE. Eur Heart J 2003; 24:881–882.
Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999; 16:9–13.
Shanmugam G, West M, Berg G. Additive and logistic EuroSCORE performance in high risk patients. Interact CardioVasc Thorac Surg 2005; 4:299–303.
Jin R, Grunkemeier GL. Additive vs. logistic risk models for cardiac surgery mortality. Eur J Cardiothorac Surg 2005; 28:240–243.
Peterson ED, DeLong ER, Muhlbaier LH, Rosen AB, Buell HE, Kiefe CI, Kresowik TF. Challenges in comparing risk-adjusted bypass surgery mortality results: results from the Cooperative Cardiovascular Project. J Am Coll Cardiol 2000; 36:2174–2184.
Yap CH, Mohajeri M, Ihle BU, Wilson AC, Goyal S, Yii M. Validation of EuroSCORE model in an Australian patient population. ANZ J Surg 2005; 75:508–512.
Harrell FE Jr, Lee KL, Califf RM, Pryor DB, Rosati RA. Regression modelling strategies for improved prognostic prediction. Stat Med 1984; 3:143–152.(Ruyun Jin and Gary L. Gru)