Systematic review of publication bias in studies on publication bias
1 Institut für Allgemeinmedizin, Universittsklinikum Hamburg-Eppendorf, Martinistrasse 52, 20246 Hamburg, Germany
Introduction
Publication bias is a well known phenomenon in clinical literature,1 2 in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading.3 Selective underreporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data.1 We investigated whether there is preferential publication of positive papers on publication bias.
, 百拇医药
Methods and results
We identified studies that assessed the impact of publication bias in Medline (January 1993 to October 2003) using the search terms "publication bias", "citation bias", "language bias", location bias", "reference bias", or "multiple publication bias". We also searched the references of a Cochrane review on publication bias.4 We restricted the search to publications that primarily investigated publication bias and whose acceptance therefore might have depended on whether they had found publication bias or not. We retrieved 265 references. Of these, we chose 148 for full examination. Their bibliographies yielded 26 additional papers. We excluded 148 studies because they gave no original data. All remaining 26 were included in the analysis (see bmj.com).
, 百拇医药
We used a funnel plot to evaluate reports for publication bias. In a funnel plot the effect size is plotted versus a measure of its precision, such as sample size. With increasing sample size, random variations of the effect are smaller. Thus, data from several studies are expected to be symmetrically distributed in a funnel shaped area of the plot if no publication bias is present. Conversely, an asymmetrical funnel plot indicates a biased study sample.5
We plotted effect size versus sample size (figure). The effect is the ratio of the odds of a positive study being published to the odds of a negative study. We transformed reported relative risks into odds ratios. We did not transform hazard ratios. The vertical line indicates no publication bias; 23 of 26 studies report preferential publication of positive results.
, 百拇医药
Funnel plot of 26 reports on publication bias, with reported effect as dependent variable
The median reported odds ratio is 2.3, indicating preferred publication of positive results. The sloping line results from a regression with the reported effect as dependent variable. Its slope does not differ significantly from zero (P = 0.13)—that is, the asymmetry of the data is not statistically significant.
Comment
, 百拇医药
We found no evidence of publication bias in reports on publication bias. But, with just 26 studies, the power to detect asymmetry in a funnel plot was low.5 Furthermore, the definition of the terms "positive" and "significant" is non-uniform and sometimes rather arbitrary in the studies reinvestigated here. For example, Dickersin (see bmj.com) used the definition "studies reported to have statistically significant findings were combined with those reported to have findings of great importance. Together they are referred to as `significant' and are contrasted with the remainder, which are referred to as `not significant.'"
, 百拇医药
Most data on publication bias were recorded retro-spectively and lack prospective registration, as does the present analysis. Prospective and registered studies on publication bias are needed.
What is already known on this topic
Studies estimated to have publication bias seem more likely to be published at all, earlier, and in journals with higher impact factors; as a consequence effects are often overestimated
, 百拇医药
What this study adds
These findings do not indicate publication bias in reports on publication bias
References for the 26 studies are on bmj.com
This article was posted on bmj.com on 3 June 2005: http://bmj.com/cgi/doi/10.1136/bmj.38478.497164.F7
Contributors: Both authors developed the design of this review, did the literature search, and analysed and interpreted the trials. The article was jointly written by both authors. H-HD is guarantor.
, http://www.100md.com
Competing interests: None declared. Funding: Unna Stiftung, Düsseldorf, Germany. Ethical approval: Not needed.
References
Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263: 1405-8.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-72.
Egger M, Smith GD. Misleading meta-analysis. BMJ 1995;310: 752-4.
Scherer RW, Langenberg P. Full publication of results initially presented in abstracts. Cochrane Database Methodol Rev 2000;(4): MR000005.
Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315: 629-34., 百拇医药(Hans-Hermann Dubben, Hans)
Introduction
Publication bias is a well known phenomenon in clinical literature,1 2 in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading.3 Selective underreporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data.1 We investigated whether there is preferential publication of positive papers on publication bias.
, 百拇医药
Methods and results
We identified studies that assessed the impact of publication bias in Medline (January 1993 to October 2003) using the search terms "publication bias", "citation bias", "language bias", location bias", "reference bias", or "multiple publication bias". We also searched the references of a Cochrane review on publication bias.4 We restricted the search to publications that primarily investigated publication bias and whose acceptance therefore might have depended on whether they had found publication bias or not. We retrieved 265 references. Of these, we chose 148 for full examination. Their bibliographies yielded 26 additional papers. We excluded 148 studies because they gave no original data. All remaining 26 were included in the analysis (see bmj.com).
, 百拇医药
We used a funnel plot to evaluate reports for publication bias. In a funnel plot the effect size is plotted versus a measure of its precision, such as sample size. With increasing sample size, random variations of the effect are smaller. Thus, data from several studies are expected to be symmetrically distributed in a funnel shaped area of the plot if no publication bias is present. Conversely, an asymmetrical funnel plot indicates a biased study sample.5
We plotted effect size versus sample size (figure). The effect is the ratio of the odds of a positive study being published to the odds of a negative study. We transformed reported relative risks into odds ratios. We did not transform hazard ratios. The vertical line indicates no publication bias; 23 of 26 studies report preferential publication of positive results.
, 百拇医药
Funnel plot of 26 reports on publication bias, with reported effect as dependent variable
The median reported odds ratio is 2.3, indicating preferred publication of positive results. The sloping line results from a regression with the reported effect as dependent variable. Its slope does not differ significantly from zero (P = 0.13)—that is, the asymmetry of the data is not statistically significant.
Comment
, 百拇医药
We found no evidence of publication bias in reports on publication bias. But, with just 26 studies, the power to detect asymmetry in a funnel plot was low.5 Furthermore, the definition of the terms "positive" and "significant" is non-uniform and sometimes rather arbitrary in the studies reinvestigated here. For example, Dickersin (see bmj.com) used the definition "studies reported to have statistically significant findings were combined with those reported to have findings of great importance. Together they are referred to as `significant' and are contrasted with the remainder, which are referred to as `not significant.'"
, 百拇医药
Most data on publication bias were recorded retro-spectively and lack prospective registration, as does the present analysis. Prospective and registered studies on publication bias are needed.
What is already known on this topic
Studies estimated to have publication bias seem more likely to be published at all, earlier, and in journals with higher impact factors; as a consequence effects are often overestimated
, 百拇医药
What this study adds
These findings do not indicate publication bias in reports on publication bias
References for the 26 studies are on bmj.com
This article was posted on bmj.com on 3 June 2005: http://bmj.com/cgi/doi/10.1136/bmj.38478.497164.F7
Contributors: Both authors developed the design of this review, did the literature search, and analysed and interpreted the trials. The article was jointly written by both authors. H-HD is guarantor.
, http://www.100md.com
Competing interests: None declared. Funding: Unna Stiftung, Düsseldorf, Germany. Ethical approval: Not needed.
References
Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263: 1405-8.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-72.
Egger M, Smith GD. Misleading meta-analysis. BMJ 1995;310: 752-4.
Scherer RW, Langenberg P. Full publication of results initially presented in abstracts. Cochrane Database Methodol Rev 2000;(4): MR000005.
Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315: 629-34., 百拇医药(Hans-Hermann Dubben, Hans)