Immune Signatures in Follicular Lymphoma
http://www.100md.com
《新英格兰医药杂志》
To the Editor: The recent article by Dave et al. (Nov. 18 issue)1 is an important study correlating gene expression with survival in follicular lymphoma. My colleagues and I have reexamined their findings.
Analysis with the use of standard tools such as Significance Analysis of Microarrays (SAM) suggests that there is little or no predictive power in these data. With help from the authors and after considerable effort, we reconstructed their analysis. Dave et al. divided their data at random into equal-sized training and test sets, applied a detailed model-fitting recipe to the training set, and finally evaluated the model's performance in predicting survival in the test set.
Their recipe gives a significant P value when used on the test set, but the result is extremely fragile. When the training and test sets are swapped, and the recipe is applied, their model does not emerge. In fact, no model is found that is significantly associated with survival when applied to the test set. As another example of fragility, if one makes small changes to some of the reasonable but arbitrary choices in their recipe, nothing significant emerges. Thus, our reanalysis sheds serious doubt on the reproducibility of the authors' biologic findings. (Full details may be found at www-stat.stanford.edu/~tibs/FL/report.)
Robert Tibshirani, Ph.D.
Stanford University
Stanford, CA 94305
tibs@stanford.edu
References
Dave SS, Wright G, Tan B, et al. Prediction of survival in follicular lymphoma based on molecular features of tumor-infiltrating immune cells. N Engl J Med 2004;351:2159-2169.
To the Editor: Dave et al. suggest that survival in follicular lymphoma correlates with two gene signatures, which they termed immune-response 1 and immune-response 2. Because most of the genes in these signatures were expressed more highly in CD19– cells, the authors conclude incorrectly that survival is based on features of the nonmalignant cells in the tumors. It must be noted that many of the genes were expressed at only slightly higher levels in CD19– cells. We estimate that the CD19– cells contributed more than 50 percent of the messenger RNA for 33 of the 43 immune-response 1 genes, but only 2 of the 24 immune-response 2 genes. When each signature was tested separately, survival in the test set correlated only with the immune-response 2 signature (Tibshirani R: personal communication). Thus, it is unwarranted to conclude that survival correlates with features in the nonmalignant cells, as claimed in the title of the paper.
Wan-Jen Hong, B.S.
Roger Warnke, M.D.
Gilbert Chu, M.D., Ph.D.
Stanford University School of Medicine
Stanford, CA 94305
chu@stanford.edu
The authors reply: Nothing in Tibshirani's analysis calls into dispute the fact that we discovered and validated a strong association between gene expression in follicular lymphoma and overall survival. Our validation method is the accepted standard for supervised analyses of microarray data to create a survival predictor.1 Our predictor was highly associated with survival in the independent test set (P<0.001). Furthermore, we randomly selected half of our cases 2000 times and computed the association of our model with survival in each half-set (using model coefficients derived from the original training set). The median P value for these samplings was less than 0.001, and no half-set sampling of our data had a P value greater than 0.011 (Figure 1).
Figure 1. Results of Samplings of Half-Sets and Association between Gene Expression and Survival.
Our predictor could not have been discovered with the SAM method, which relies solely on univariate associations with survival. Rather, our predictor derives its strength from the synergistic combination of two gene-expression signatures in a multivariate model. Tibshirani confuses the ability of our method to discover a survival association with the fact that we actually found one that validated the association. When he exchanged the training set for the test set, he was unable to rediscover our gene-expression predictor because some genes in our predictor fell below the P value threshold for association with survival in the test set. This does not negate the fact that our model is highly associated with survival in the test set.
Hong et al. have made three errors. First, in our sorted subpopulations, the CD19– fraction contained, on average, 12.6 percent contamination with follicular-lymphoma cells, not 25 percent, as they claim. To believe that the higher expression of the immune-response signatures in the CD19– fraction is due to this 12.6 percent contamination requires that the lymphoma cells in the CD19– fraction have expression of the immune-response signatures that was more than eight times as high as that in their counterparts in the CD19+ fraction. Second, Hong et al. incorrectly discount the immune-response 1 signature, which contributes significantly to the survival model in the test set (P<0.001). Third, many of the immune-response signature genes are selectively expressed in T cells, monocytes, or dendritic cells, or in more than one of these, but not in B cells, making the contention of Hong et al. even more implausible.
Louis M. Staudt, M.D., Ph.D.
George Wright, Ph.D.
Sandeep Dave, M.D.
National Cancer Institute
Bethesda, MD 20892
lstaudt@mail.nih.gov
Analysis with the use of standard tools such as Significance Analysis of Microarrays (SAM) suggests that there is little or no predictive power in these data. With help from the authors and after considerable effort, we reconstructed their analysis. Dave et al. divided their data at random into equal-sized training and test sets, applied a detailed model-fitting recipe to the training set, and finally evaluated the model's performance in predicting survival in the test set.
Their recipe gives a significant P value when used on the test set, but the result is extremely fragile. When the training and test sets are swapped, and the recipe is applied, their model does not emerge. In fact, no model is found that is significantly associated with survival when applied to the test set. As another example of fragility, if one makes small changes to some of the reasonable but arbitrary choices in their recipe, nothing significant emerges. Thus, our reanalysis sheds serious doubt on the reproducibility of the authors' biologic findings. (Full details may be found at www-stat.stanford.edu/~tibs/FL/report.)
Robert Tibshirani, Ph.D.
Stanford University
Stanford, CA 94305
tibs@stanford.edu
References
Dave SS, Wright G, Tan B, et al. Prediction of survival in follicular lymphoma based on molecular features of tumor-infiltrating immune cells. N Engl J Med 2004;351:2159-2169.
To the Editor: Dave et al. suggest that survival in follicular lymphoma correlates with two gene signatures, which they termed immune-response 1 and immune-response 2. Because most of the genes in these signatures were expressed more highly in CD19– cells, the authors conclude incorrectly that survival is based on features of the nonmalignant cells in the tumors. It must be noted that many of the genes were expressed at only slightly higher levels in CD19– cells. We estimate that the CD19– cells contributed more than 50 percent of the messenger RNA for 33 of the 43 immune-response 1 genes, but only 2 of the 24 immune-response 2 genes. When each signature was tested separately, survival in the test set correlated only with the immune-response 2 signature (Tibshirani R: personal communication). Thus, it is unwarranted to conclude that survival correlates with features in the nonmalignant cells, as claimed in the title of the paper.
Wan-Jen Hong, B.S.
Roger Warnke, M.D.
Gilbert Chu, M.D., Ph.D.
Stanford University School of Medicine
Stanford, CA 94305
chu@stanford.edu
The authors reply: Nothing in Tibshirani's analysis calls into dispute the fact that we discovered and validated a strong association between gene expression in follicular lymphoma and overall survival. Our validation method is the accepted standard for supervised analyses of microarray data to create a survival predictor.1 Our predictor was highly associated with survival in the independent test set (P<0.001). Furthermore, we randomly selected half of our cases 2000 times and computed the association of our model with survival in each half-set (using model coefficients derived from the original training set). The median P value for these samplings was less than 0.001, and no half-set sampling of our data had a P value greater than 0.011 (Figure 1).
Figure 1. Results of Samplings of Half-Sets and Association between Gene Expression and Survival.
Our predictor could not have been discovered with the SAM method, which relies solely on univariate associations with survival. Rather, our predictor derives its strength from the synergistic combination of two gene-expression signatures in a multivariate model. Tibshirani confuses the ability of our method to discover a survival association with the fact that we actually found one that validated the association. When he exchanged the training set for the test set, he was unable to rediscover our gene-expression predictor because some genes in our predictor fell below the P value threshold for association with survival in the test set. This does not negate the fact that our model is highly associated with survival in the test set.
Hong et al. have made three errors. First, in our sorted subpopulations, the CD19– fraction contained, on average, 12.6 percent contamination with follicular-lymphoma cells, not 25 percent, as they claim. To believe that the higher expression of the immune-response signatures in the CD19– fraction is due to this 12.6 percent contamination requires that the lymphoma cells in the CD19– fraction have expression of the immune-response signatures that was more than eight times as high as that in their counterparts in the CD19+ fraction. Second, Hong et al. incorrectly discount the immune-response 1 signature, which contributes significantly to the survival model in the test set (P<0.001). Third, many of the immune-response signature genes are selectively expressed in T cells, monocytes, or dendritic cells, or in more than one of these, but not in B cells, making the contention of Hong et al. even more implausible.
Louis M. Staudt, M.D., Ph.D.
George Wright, Ph.D.
Sandeep Dave, M.D.
National Cancer Institute
Bethesda, MD 20892
lstaudt@mail.nih.gov