Comparison of descriptions of allocation concealment in trial protocol
http://www.100md.com
《英国医生杂志》
1 Nordic Cochrane Centre, H:S Rigshospitalet, DK-2100 Copenhagen, Denmark, 2 Department of Medicine, University of Toronto, 3 Cancer Research UK Medical Statistics Group, Centre for Statistics in Medicine, Oxford OX3 7LF
Correspondence to: J Pildal jpl@dadlnet.dk
Objectives To compare how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and to estimate how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.
Design Cohort study of 102 sets of trial protocols and corresponding publications.
Setting Protocols of randomised trials approved by the scientific and ethical committees for Copenhagen and Frederiksberg, 1994 and 1995.
Main outcome measures Frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols, and the proportion of protocols where methods were reported to be adequate but descriptions were unclear in the trial publications.
Results 96 of the 102 trials had unclear allocation concealment according to the trial publication. According to the protocols, 15 of these 96 trials had adequate allocation concealment (16%, 95% confidence interval 9% to 24%), 80 had unclear concealment (83%, 74% to 90%), and one had inadequate concealment. When retrospectively defined loose criteria for concealment were applied, 83 of the 102 trial publications had unclear concealment. According to their protocol, 33 of these 83 trials had adequate allocation concealment (40%, 29% to 51%), 49 had unclear concealment (59%, 48% to 70%), and one had inadequate concealment.
Conclusions Most randomised clinical trials have unclear allocation concealment on the basis of the trial publication alone. Most of these trials also have unclear allocation concealment according to their protocol.
Selection bias occurs in randomised clinical trials if patients with a better prognosis are preferentially allocated to one of the treatment arms. The results of the trial will then to some degree reflect this difference in prognosis rather than just a difference in the effects of the compared treatments.
The purpose of randomisation is to avoid selection bias, as patients with known and unknown differences in prognosis will tend to be equally distributed between the treatment groups. To ensure true randomisation, however, the random allocation sequence should not only be sequentially and irreversibly administered but should also be concealed to the individuals in charge of enrolment and treatment allocation. Otherwise, knowledge of the upcoming allocation will permit selective assignment of patients by manipulation of either the sequence of treatments to be allocated or the sequence of patients to be enrolled.1
Surveys have shown that 44% to 93% of publications of randomised controlled trials lack a clear description of allocation concealment.2 Empirical studies have shown that publications of trials in which allocation concealment is unclear or inadequate are associated with, on average, a 20-30% exaggeration of the treatment effect (measured as a ratio of odds ratios) compared with trials of the same interventions with adequate concealment.3-9 Generation of a truly random sequence is an interrelated issue for which there is suggestive empirical evidence of an associated inflation of the treatment effect if the trial publication does not document adequate procedures.4-6 Both issues are of major concern because the effect of many treatments is less than these average biases. We compared how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and we estimated how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.
Methods
Our cohort consisted of all published randomised trials (apart from trials in dentistry) whose protocols were approved by the scientific and ethical committees for Copenhagen and Frederiksberg in 1994 and 1995. We identified trial publications by contacting the principal investigators and by searching PubMed, Embase, and the Cochrane central register of controlled trials (final search in May 2003; median publication year 1999, range 1995-2003).10 In total, 102 protocols were published in 122 trial reports. When there was more than one publication of a trial, we examined all publications for information on allocation concealment.
Outcomes and data extraction
Our outcome measures were frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols; the proportion of protocols where methods were reported to be adequate when the trial publications gave unclear descriptions; the type and frequency of methods used for allocation concealment; and the prevalence of other trial characteristics that might undermine concealment (for example, stating the block size in the protocol).
Two authors extracted data from the first half of the published reports and the second half of the protocols while another pair of authors extracted data from the rest. Disagreements were resolved within each pair, thus avoiding assessment of both trial publication and protocol for the same trial.
Assessment of adequacy of allocation concealment
We considered the following methods for allocation concealment as adequate:3-6 11-13 central randomisation; numbered coded vehicles; opaque, sealed, and sequentially numbered envelopes; and other methods containing convincing means of concealment. Inadequate methods concerned open or predictable sequences of allocation (for example, alternation), date of birth, case record number or similar, and open tables of random numbers. We categorised studies as unclear that did not fall into one of these categories or that provided no information.
To ensure consistency and transparency and to capture how strict application compared with loose application of our criteria might influence our results, we operationalised our interpretations of authors' descriptions of allocation concealment (see table on bmj.com). The strict criteria are those recommended for Cochrane reviews,11 except for an elaboration on central randomisation, as specified in the table on bmj.com.12 13 The loose criteria, which we defined retrospectively, comprised the most liberal criteria used in any of the previous empirical studies of bias associated with unclear or inadequate allocation concealment.3-9 For instance, in a study by Schulz et al4 envelopes had to be opaque, sealed, and sequentially numbered to qualify as adequate concealment, whereas in a study by Kjaergard et al6 use of sealed envelopes without further details qualified as adequate (see bmj.com for examples of how the criteria were applied on our sample).
Assessment of adequacy of sequence generation
Adequate methods of sequence generation included computer generated random numbers, tables of random numbers, or drawing lots or envelopes. Inadequate methods could be related to prognosis such as date of birth or year of admission. Unclear methods were methods not falling into one of these two categories or where the methods were not described.
Statistical analysis
We calculated 95% confidence intervals using the exact binomial method in Stata version 8.
Results
Allocation concealment
Using the strict criteria, 96 of the 102 trials (94%, 95% confidence interval 88% to 98%) had unclear allocation concealment according to their publications. According to their protocols, 15 of these 96 trials (16%, 9% to 24%) had adequate allocation concealment and one had inadequate concealment, whereas most (80 of 96; 83%, 74% to 90%) had unclear concealment (table 1).
Table 1 Adequacy of allocation concealment as described in pairs of protocols and corresponding trial publications in 102 trials according to strict and loose criteria. Values are numbers
Using the loose criteria, 83 of the 102 trials had unclear allocation concealment (81%, 72% to 88%). According to the protocols, 33 of these 83 publications (40%, 29% to 51%) had adequate allocation concealment, one had inadequate concealment, and 49 (59%; 48% to 70%) had unclear concealment (see table 1).
According to the strict criteria, 20 of the 102 studies (five publications and 19 protocols; see table 1) described adequate allocation concealment. When the loose criteria were applied, however, 51 studies (18 publications and 45 protocols) described adequate concealment.
Sequence generation
Eighty one of the 102 trial publications gave no information on how the allocation sequence was generated; 16 of these 81 trials (20%; 12% to 30%) described adequate sequence generation in the protocol. No protocols or trial publications reported inadequate methods of sequence generation.
Methods used for allocation concealment
Table 2 lists the methods used to achieve allocation concealment. Numbered coded vehicles was the most frequently applied method according to the protocols (26 of 102) but had the lowest rate of appearance in the trial publications (three of 26). None of the 17 trials using central randomisation fulfilled the strict criteria, as none described concealment of the randomisation sequence from the central staff, only four described irreversibility of the treatment assignment, and none described that prognostic data irrelevant to stratification must not be revealed to the central office (in three trials such data were positively requested). In 39 of the 102 trials neither the protocols nor the publications provided any information on attempts to conceal the allocation. In four trials, the protocol and the publication gave conflicting information on which method was used.
Table 2 Methods for allocation concealment in pairs of protocols and corresponding trial publications. Values are numbers
Trial characteristics that might weaken an otherwise adequate allocation concealment regimen
Block randomisation
In 14 trials, block randomisation could partly have compromised allocation concealment because the block size was explicitly stated in the protocol. This is problematic since a known block size enables qualified guesswork to predict upcoming allocations towards the end of the block. This can weaken allocation concealment even in multicentre studies if they are stratified per centre and in double blind studies if the blinding becomes compromised—for example, because of adverse effects.
Tasks that should not be carried out by the same party
The preparation of envelopes for concealment was described in the passive tense in nine of the 13 studies using the envelope method for allocation concealment (see table 2). Thus it is unknown whether the same person prepared the envelopes, enrolled the patients, and administered the envelopes, particularly as seven of the nine studies were single centre studies. An example of lack of separation of functions for central randomisation was when the same party had information on the prognosis of the next patient to be enrolled and was involved in concealing the sequence and in administering it (see bmj.com).
Code envelopes
In 42 of the 55 double blind studies, a security system for emergency code breaking was described in the protocol but mentioned in only one publication. Overall, 90% (38 of 42) of these protocols specified that envelopes or a similar system would be present at the clinical location. Deciphering the contents of such envelopes, for instance by holding them against strong light, might have revealed the allocation for the next patient; yet only one of the 38 protocols (3%) described the envelopes as opaque. Although such code envelopes are a theoretical threat to the allocation concealment, it is unknown whether their presence on the clinical location is associated with exaggerated effect estimates. Consequently, our criteria for assessment of allocation concealment by the means of envelopes did not include assessment of code envelopes.(Julie Pildal, research fellow1, An-Wen C)
Correspondence to: J Pildal jpl@dadlnet.dk
Objectives To compare how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and to estimate how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.
Design Cohort study of 102 sets of trial protocols and corresponding publications.
Setting Protocols of randomised trials approved by the scientific and ethical committees for Copenhagen and Frederiksberg, 1994 and 1995.
Main outcome measures Frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols, and the proportion of protocols where methods were reported to be adequate but descriptions were unclear in the trial publications.
Results 96 of the 102 trials had unclear allocation concealment according to the trial publication. According to the protocols, 15 of these 96 trials had adequate allocation concealment (16%, 95% confidence interval 9% to 24%), 80 had unclear concealment (83%, 74% to 90%), and one had inadequate concealment. When retrospectively defined loose criteria for concealment were applied, 83 of the 102 trial publications had unclear concealment. According to their protocol, 33 of these 83 trials had adequate allocation concealment (40%, 29% to 51%), 49 had unclear concealment (59%, 48% to 70%), and one had inadequate concealment.
Conclusions Most randomised clinical trials have unclear allocation concealment on the basis of the trial publication alone. Most of these trials also have unclear allocation concealment according to their protocol.
Selection bias occurs in randomised clinical trials if patients with a better prognosis are preferentially allocated to one of the treatment arms. The results of the trial will then to some degree reflect this difference in prognosis rather than just a difference in the effects of the compared treatments.
The purpose of randomisation is to avoid selection bias, as patients with known and unknown differences in prognosis will tend to be equally distributed between the treatment groups. To ensure true randomisation, however, the random allocation sequence should not only be sequentially and irreversibly administered but should also be concealed to the individuals in charge of enrolment and treatment allocation. Otherwise, knowledge of the upcoming allocation will permit selective assignment of patients by manipulation of either the sequence of treatments to be allocated or the sequence of patients to be enrolled.1
Surveys have shown that 44% to 93% of publications of randomised controlled trials lack a clear description of allocation concealment.2 Empirical studies have shown that publications of trials in which allocation concealment is unclear or inadequate are associated with, on average, a 20-30% exaggeration of the treatment effect (measured as a ratio of odds ratios) compared with trials of the same interventions with adequate concealment.3-9 Generation of a truly random sequence is an interrelated issue for which there is suggestive empirical evidence of an associated inflation of the treatment effect if the trial publication does not document adequate procedures.4-6 Both issues are of major concern because the effect of many treatments is less than these average biases. We compared how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and we estimated how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.
Methods
Our cohort consisted of all published randomised trials (apart from trials in dentistry) whose protocols were approved by the scientific and ethical committees for Copenhagen and Frederiksberg in 1994 and 1995. We identified trial publications by contacting the principal investigators and by searching PubMed, Embase, and the Cochrane central register of controlled trials (final search in May 2003; median publication year 1999, range 1995-2003).10 In total, 102 protocols were published in 122 trial reports. When there was more than one publication of a trial, we examined all publications for information on allocation concealment.
Outcomes and data extraction
Our outcome measures were frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols; the proportion of protocols where methods were reported to be adequate when the trial publications gave unclear descriptions; the type and frequency of methods used for allocation concealment; and the prevalence of other trial characteristics that might undermine concealment (for example, stating the block size in the protocol).
Two authors extracted data from the first half of the published reports and the second half of the protocols while another pair of authors extracted data from the rest. Disagreements were resolved within each pair, thus avoiding assessment of both trial publication and protocol for the same trial.
Assessment of adequacy of allocation concealment
We considered the following methods for allocation concealment as adequate:3-6 11-13 central randomisation; numbered coded vehicles; opaque, sealed, and sequentially numbered envelopes; and other methods containing convincing means of concealment. Inadequate methods concerned open or predictable sequences of allocation (for example, alternation), date of birth, case record number or similar, and open tables of random numbers. We categorised studies as unclear that did not fall into one of these categories or that provided no information.
To ensure consistency and transparency and to capture how strict application compared with loose application of our criteria might influence our results, we operationalised our interpretations of authors' descriptions of allocation concealment (see table on bmj.com). The strict criteria are those recommended for Cochrane reviews,11 except for an elaboration on central randomisation, as specified in the table on bmj.com.12 13 The loose criteria, which we defined retrospectively, comprised the most liberal criteria used in any of the previous empirical studies of bias associated with unclear or inadequate allocation concealment.3-9 For instance, in a study by Schulz et al4 envelopes had to be opaque, sealed, and sequentially numbered to qualify as adequate concealment, whereas in a study by Kjaergard et al6 use of sealed envelopes without further details qualified as adequate (see bmj.com for examples of how the criteria were applied on our sample).
Assessment of adequacy of sequence generation
Adequate methods of sequence generation included computer generated random numbers, tables of random numbers, or drawing lots or envelopes. Inadequate methods could be related to prognosis such as date of birth or year of admission. Unclear methods were methods not falling into one of these two categories or where the methods were not described.
Statistical analysis
We calculated 95% confidence intervals using the exact binomial method in Stata version 8.
Results
Allocation concealment
Using the strict criteria, 96 of the 102 trials (94%, 95% confidence interval 88% to 98%) had unclear allocation concealment according to their publications. According to their protocols, 15 of these 96 trials (16%, 9% to 24%) had adequate allocation concealment and one had inadequate concealment, whereas most (80 of 96; 83%, 74% to 90%) had unclear concealment (table 1).
Table 1 Adequacy of allocation concealment as described in pairs of protocols and corresponding trial publications in 102 trials according to strict and loose criteria. Values are numbers
Using the loose criteria, 83 of the 102 trials had unclear allocation concealment (81%, 72% to 88%). According to the protocols, 33 of these 83 publications (40%, 29% to 51%) had adequate allocation concealment, one had inadequate concealment, and 49 (59%; 48% to 70%) had unclear concealment (see table 1).
According to the strict criteria, 20 of the 102 studies (five publications and 19 protocols; see table 1) described adequate allocation concealment. When the loose criteria were applied, however, 51 studies (18 publications and 45 protocols) described adequate concealment.
Sequence generation
Eighty one of the 102 trial publications gave no information on how the allocation sequence was generated; 16 of these 81 trials (20%; 12% to 30%) described adequate sequence generation in the protocol. No protocols or trial publications reported inadequate methods of sequence generation.
Methods used for allocation concealment
Table 2 lists the methods used to achieve allocation concealment. Numbered coded vehicles was the most frequently applied method according to the protocols (26 of 102) but had the lowest rate of appearance in the trial publications (three of 26). None of the 17 trials using central randomisation fulfilled the strict criteria, as none described concealment of the randomisation sequence from the central staff, only four described irreversibility of the treatment assignment, and none described that prognostic data irrelevant to stratification must not be revealed to the central office (in three trials such data were positively requested). In 39 of the 102 trials neither the protocols nor the publications provided any information on attempts to conceal the allocation. In four trials, the protocol and the publication gave conflicting information on which method was used.
Table 2 Methods for allocation concealment in pairs of protocols and corresponding trial publications. Values are numbers
Trial characteristics that might weaken an otherwise adequate allocation concealment regimen
Block randomisation
In 14 trials, block randomisation could partly have compromised allocation concealment because the block size was explicitly stated in the protocol. This is problematic since a known block size enables qualified guesswork to predict upcoming allocations towards the end of the block. This can weaken allocation concealment even in multicentre studies if they are stratified per centre and in double blind studies if the blinding becomes compromised—for example, because of adverse effects.
Tasks that should not be carried out by the same party
The preparation of envelopes for concealment was described in the passive tense in nine of the 13 studies using the envelope method for allocation concealment (see table 2). Thus it is unknown whether the same person prepared the envelopes, enrolled the patients, and administered the envelopes, particularly as seven of the nine studies were single centre studies. An example of lack of separation of functions for central randomisation was when the same party had information on the prognosis of the next patient to be enrolled and was involved in concealing the sequence and in administering it (see bmj.com).
Code envelopes
In 42 of the 55 double blind studies, a security system for emergency code breaking was described in the protocol but mentioned in only one publication. Overall, 90% (38 of 42) of these protocols specified that envelopes or a similar system would be present at the clinical location. Deciphering the contents of such envelopes, for instance by holding them against strong light, might have revealed the allocation for the next patient; yet only one of the 38 protocols (3%) described the envelopes as opaque. Although such code envelopes are a theoretical threat to the allocation concealment, it is unknown whether their presence on the clinical location is associated with exaggerated effect estimates. Consequently, our criteria for assessment of allocation concealment by the means of envelopes did not include assessment of code envelopes.(Julie Pildal, research fellow1, An-Wen C)