Model for Assessment of Proficiency of Human Immunodeficiency Virus Type 1 Sequencing-Based Genotypic Antiretroviral Assays
http://www.100md.com
微生物临床杂志 2005年第8期
Department of Immunology/Microbiology, Rush Medical College, Chicago, Illinois
New England Research Institute, Inc. Watertown, Massachusetts
Pediatrics, University of Medicine and Dentistry of New Jersey, New Jersey Medical School, Newark, New Jersey
ABSTRACT
Use of sequencing-based genotyping as a diagnostic assay for human immunodeficiency virus (HIV) antiretroviral resistance is increasing. Periodic evaluation of the proficiency of laboratories performing this assay should be established. It is important to identify components of the assay that influence the generation of reliable sequencing data and that should and can be monitored. A model was developed to determine what parameters were reasonable and feasible for assessing the performance of genotyping assays. Ten laboratories using the genotyping platform, HIV-1 Genotyping System (HGS) v. 1 and software versions 1.1 or 2.0, participated in two rounds of testing. For each round, each group was sent a panel consisting of three clinical samples to sequence in real time. Six months later, seven laboratories using the TRUGENE HIV-1 Genotyping Kit participated in a separate round, working with both panels at the same time. Analysis of the data showed that one main indicator of genotyping proficiency was achievement of 98% sequence homology of a sample tested to a group consensus sequence for that sample. A second was concordant identification of codons at sites identified with resistance mutations in the sample, although scoring of these criteria is still undetermined from this study. These criteria are applicable to all sequence-based genotyping platforms and have been used as a baseline for assessing the performance of genotyping for the determination of antiretroviral resistance in our ongoing proficiency program.
INTRODUCTION
Genotyping of the human immunodeficiency virus type 1 (HIV-1) protease (PR) and reverse transcriptase (RT) genes by population sequencing to assess antiretroviral resistance is a standard diagnostic assay to aid the clinical management of HIV-infected individuals (3). Advisory groups have recommended HIV resistance genotyping to assist in analysis of certain clinical care settings (e.g., beginning therapy for AIDS-associated retrovirus [ARV]-nave individuals; choosing new regimens for those failing ARV therapy) (4) These assays comprise multiple sequential components and require expertise in both the physical performance of the assays and the subjective evaluation of the generated data (Fig. 1). The genotyping assays that are currently Food and Drug Administration (FDA)-approved, ViroSeq HIV-1 Genotyping System v. 2.0 (Celera Diagnostics, Alameda, CA) and TRUGENE HIV-1 Genotyping Kit (Bayer HealthCare LLC, Berkeley, CA), provide positive and negative run controls that are to be amplified and sequenced with each batch of clinical samples. The positive controls are basically homogeneous in composition, as they are, respectively, either diluted cultured HIV-1 strain 8E5 or RNA transcripts of HIV-1 derived from a plasmid bearing an appropriate insert. Because HIV-1 quasispecies circulate in the infected individual (1), clinical samples contain heterogeneous virus populations. As a result, mixed nucleotides (mixtures) at individual nucleotide positions may be evident in sequence analysis and may indicate viral variants of increasing or declining resistance. The presence of mixtures may also result from sequence artifacts generated through suboptimal performance of some technical portion of the assay. For example, inadequate sample preparation, reverse transcription, PCR amplification, binding of the sequencing primer to the sequencing template, purification of PCR or sequencing products, and electrophoresis through gel or polymer matrices may each contribute to data of questionable quality (7). At times, the nucleotide mixtures need to be examined and resolved by visual evaluation of the electropherograms by the person performing the assay. In these instances, the sequence editor has to assess whether the mixtures observed are genuine or artifacts. This process is called editing (6). Given the technical and subjective complexity of the assays, there is a need for testing the reproducibility of the assays and the proficiency and consistency of the operators by use of samples more similar to real clinical specimens than the run controls provided.
The members of the Pediatric Virology Core Laboratories of the Pediatric AIDS Clinical Trials Group (PACTG) performed genotyping on two panels of plasma samples to identify factors that might contribute to interlaboratory variability in the performance of sequenced-based genotypic assays. Participants used either the HIV-1 Genotyping System (HGS) v. 1 or TRUGENE HIV-1 Genotyping Assay, both of which included platform-associated software. Information from the study was used as a model for evaluation of performance of sequenced-based genotypic assays and to begin to formulate criteria for assessing the overall quality of sequence data generated. Several of these criteria are presented here.
MATERIALS AND METHODS
Sequencing assay. The research-use-only (RUO) HIV-1 Genotyping System (HGS) v. 1 accompanied by software versions 1.1 (panel 1) and 2.0 (panel 2) (Applied Biosystems, Foster City, CA) or the TRUGENE HIV-1 Genotyping Assay (TRUGENE) accompanied by software v. 3.0.1(Visible Genetics, Suwanee GA) was used by the participating laboratories. These systems were the predecessors to the current FDA-approved ViroSeq HIV-1 Genotyping System v. 2.0 (Celera Diagnostics, Alameda, CA) and TRUGENE HIV-1 Genotyping Kit (Bayer HealthCare LLC, Berkeley, CA). To obtain the kit, personnel from each laboratory were trained to perform the assay and edit the resulting data prior to receipt of certification samples from each company. Editing guidelines were taught during formal training provided by the company.
Plasma samples and panels. Two panels each containing 1 ml of plasma from each of three HIV-1-infected individuals were distributed to 10 laboratories of the PACTG Sequencing Working Group. These samples were collected, characterized, and sequenced by the Virology Quality Assurance (VQA) Laboratory (5) prior to distribution. Panel 1 samples (Table 1) were from donors untreated with antiretroviral drugs for at least 1 year and were used to determine whether the HGS laboratories were performing the sequencing portion of the assay consistently. Panel 2 plasma donors demonstrated increased viral loads while on therapy, which may indicate therapeutic failure. Their plasma samples were used to see whether identification of mutations and editing occurred consistently across laboratories (6). The panels were distributed approximately 6 months apart to those laboratories in which the HGS kit was used. The laboratories using the TRUGENE assay (total, n = 7; PACTG, n = 5; Visible Genetics, Inc.; University of Colorado, n = 2) received both panels at the same time 6 months after panel 2 was sent to the HGS group. Distribution of the panels to the respective groups was made according to the availability of (i) the RUO research and clinical configurations of the respective kits for use and (ii) a critical mass of trained and certified laboratories able to participate to generate enough data to analyze. The HGS laboratories were given the same version of the kit configuration and the then-current software to use for each panel. The TRUGENE groups used the same version of reagents for their panels but were provided either the research or clinical version of the software by the manufacturer. For this study, both groups of laboratories were instructed to use their respective kits and software as they were trained to do by the companies. Plasma was collected by protocols approved by the Institutional Review Board at Rush University Medical Center.
Data collection and analysis. Within 4 weeks after receipt of the panels, laboratories personnel submitted sequence data as text files in FASTA format electronically to Frontier Science & Technology Research Foundation, Amherst, NY, for data collection and transcription. The transcribed data were sent to New England Research Institute, Watertown, MA, and the VQA Laboratory for analysis of each sample. The laboratories that used the HGS kit also sent the VQA Laboratory a copy of the final edited "project" that contained the embedded electropherograms of the individual sequences used to form the consensus sequences for further examination, if needed. TRUGENE users sent only the text files of the edited consensus sequences, because the version of the software system in use when the panels were sequenced differed among laboratories. Also, electropherograms associated with the edited TRUGENE cases were not readily accessible to off-site work stations, a characteristic of the platform operating system, Openstep, at the time.
Both HGS genotyping software versions 1.1 and 2.0 and TRUGENE software v. 3.0.1 allow editing to be tracked. Bases identified by each kit's software, which aligns bidirectional overlapping data from sequencing primers to form a consensus sequence, are automatically recorded in upper case. The individual nucleotides in the consensus sequence that are changed manually are recorded in lowercase characters. The case designation is preserved in the sequence text file generated by the software, thus documenting bases that were manually edited.
For each platform, the group consensus sequence (GCS) was generated by alignment with Align Plus (Scientific Educational Software, Durham, North Carolina) at a 60% threshold. Homologies to the GCS were determined in a second alignment using the GCS as the reference strain. Sequence data from each platform were analyzed separately partly due to differences in length and portion of the protease (PR) and reverse transcriptase (RT) gene sequence generated by the kits, software design and editing guidelines associated with each assay, and overall protocol design used to produce sequence data. HGS data were analyzed for the entire PR gene (99 codons) and the RT gene for codons 1 to 320; TRUGENE data covered codons 4 to 99 of the PR gene and codons 37 to 247 of the RT gene. Any codon for any laboratory that contained a nucleotide in lowercase characters was classified as edited. The number of edited codons for each sample for each laboratory was tabulated. Data were examined for (i) concordance of submitted nucleotides to the GCS for that sample; (ii) frequency of editing; and (iii) sites on the entire template that were edited.
Mutations associated with antiretroviral resistance were identified for each laboratory for the samples in panel 2 only. These mutations were recognized by algorithms incorporated into the software version that was available when the assay was performed.
RESULTS
Data from both panels were obtained from 9 of 10 laboratories in which HGS was used, while data from panel 1 were only obtained from the 10th laboratory. A total of 114 separate gene sequences, 57 from RT and 57 from PR, were analyzed. Six of seven laboratories in which the TRUGENE kit was used submitted data for all six samples in both panels. The seventh laboratory submitted two samples each for panel 1 and 2. A total of 80 sequences, 40 from PR and 40 from RT, were analyzed.
Homology to group consensus sequence versus editing. Each group's data were evaluated to determine how closely their individually edited consensus sequences matched the total group consensus sequence for each gene across the entire backbone. Agreement among the HGS laboratories was high for both the PR and the RT genes. Of 114 individual consensus sequences submitted, only one displayed < 98% homology to the GCS. Concordance to the group consensus sequence was 97.3 to 100.0% for PR (297 bases) (Fig. 2A) and 98.0 to 100.0% for RT (960 bases) (Fig. 2B). The percentage of codons in each sequence edited by laboratory personnel to form a consensus sequence ranged between 2.0 and 87.9% and between 4.7 and 63.6% for the protease and RT genes, respectively. Homology was negatively correlated with edit rate for PR (Spearman rank correlation, r = –0.36; P = 0.0061) but not for RT (Spearman rank correlation, r = –0.21; P = 0.13), although the trends in the two plots look similar. However, even at the highest editing rates, homology to the GCS remained high. Only 11 (5 PR, 6 RT) of 114 sequences had <99% homology to the GCS and only 1 of the 11 (97.3%) had <98% homology to the GCS. Also, no association between experience in performance of the assay and amount of editing was observed (data not shown).
The laboratories using the TRUGENE assay showed similar trends. For the protease gene (Fig. 3A), 34 of 40 submitted sequences were concordant to the GCS at >98.0%. Editing of the protease gene ranged from 0.0 to 28.1%. The clustering of concordance was tighter for the RT gene (Fig. 3B) where 34 of 40 sequences matched the GCS at >98.5%. RT gene editing ranged between 0.4 and 13.3%. Again, no clear-cut correlation between the amount of editing and final homology was apparent.
Identification of mutations associated with antiretroviral resistance. The ability to identify mutations associated with antiretroviral resistance was used as another measure of genotyping performance. During prescreening analysis a total of 36 mutations that were associated with antiretroviral resistance were identified across the samples in panel 2 (Table 1). Among the nine laboratories in which the HGS platform was used for this panel, there were five discrepancies in the codons with mutations. Table 2 shows the reported composition of these codons and the editing decisions (lowercase letters) made to identify the composition of the reported codon. There were four instances in which the codon from a single laboratory differed from the codons from the other eight groups. Three of these four discrepancies involved a codon that contained a mixed base that included the resistance mutation. It was interesting to see that the identical codon sequences were from the eight laboratories without discrepancies, even though editing patterns differed among laboratories. For sample 02RG03 RT, four groups identified a mixed base (Y) in the codon AYA while the other four identified the codon as ATA. All eight codons were unedited. For the laboratories in which the TRUGENE assay was used, there were three discrepant identifications of mutations associated with antiretroviral resistance for the panel 2 samples. The discrepancies in the sequences from the TRUEGENE and HGS assays occurred at different codons. The combinations of nucleotides reported for these codons were similar in pattern to those described in Table 2 (data not shown). The results indicate that discrepancies in data could be generated during earlier steps in the assay before editing and may appear unambiguous during the proofreading stage, precluding the need for editing.
Using the HGS group data in Table 2, for nine laboratories working with 36 mutations, the overall failure to detect a mutation was low (1.5%). However, to measure the proficiency of an individual laboratory, it might be more appropriate to examine the number of discordant identifications made for each set of 36 mutations associated with the panel. Using the examples in Table 2, discordant identifications were made twice. For 02RG01 L24I (PR), eight of nine groups identified only the wild-type (WT) codon and one of nine reported the mutation. For 02RG02 T215Y (RT), eight of nine groups reported the mutation and one of nine groups reported a wild-type codon. The frequency of disagreement of these two individual laboratories with the rest of the group for the entire panel would be 1 of 36 or 2.8%. Each codon reported for a site of interest that was discrepant from the codon reported by the rest of group could be flagged and considered in the assessment of laboratory proficiency.
Discordant nucleotide identification along the template. The ability to accurately report the identity of each nucleotide at each position along the template could be considered in the assessment and evaluation of genotyping proficiency. This issue was also described by Sayer et al. (8) in their analysis of, predominantly, in-house assays and by Shafer et al., who examined replicate testing by two laboratories also using in-house assays (9). In our testing, all laboratories used the same sequencing platform and the group consensus sequence for each sample was generated by alignment using a 60% threshold. This means that the same nucleotide in the GCS was identified by 60% of the laboratories. Figure 4 shows three examples of discordant identification of a nucleotide by individual groups by use of the HGS platform at positions 532, 534, and 846. In 8 of 10 sequences, the base at position 532 was identified as "r." In the other two sequences, from laboratories 3 and 4, the nucleotide was identified as "g." The "r" is interpreted by International Union of Biochemistry (IUB) code to be a mixture composed of bases "a" and "g." The converse situation is shown for position 534. Here, 60% of the laboratories identified the base to be a pure base "a," which is recorded in the group consensus sequence, and 40% of the groups designated the base to be "r." In the final example at position 846, 9 of 10 laboratories identified the base as a "c," but 1 group reported a "t." In this example, no mixed bases were identified for that position.
The issue to be resolved is how to define an accurate answer or whether this can even be accomplished. Using data for nucleotide position 532 and 534 as an example, since the "r" is composed of "a"and "g," accuracy can be construed in several ways. For example, a narrow assessment of the situations described above could be that the nucleotide designated in the GCS is correct and all deviations are wrong. A broader view could be that if there is a mixture reported by any of the labs ("r") which is comprised of any of the pure nucleotides identified ("a" or "g"), then results reporting "r" or "a" or "g" are all potentially equally accurate. The broadest view might be that each discrepant base reported is accurate if the reporting can be verified by visual inspection of the edited electropherograms or text files. The situation represented for nucleotide position 846 is more straightforward. Here, 9 of 10 groups identified and reported the base as "c." Only 1 group reported a different base "t." Since no mixtures were identified and reported by any group, the discrepant laboratory would be considered to have reported an incorrect nucleotide.
DISCUSSION
This study identified two criteria that could be useful for the initial assessment of performance of sequence-based genotyping assays when participants are using the same sequencing platform. These criteria are (i) the overall sequence homology of each laboratory's consensus sequence to a group consensus sequence and (ii) the identification of mutations associated with antiretroviral resistance. A high level of concordance of the sequence from a laboratory to the group consensus sequence suggests that laboratory personnel were capably performing the technical and subjective portions of the assay. Concordant identification of mutations can provide an additional evaluation of the consistency of performance of the subjective component of the assay, i.e., editing, among laboratories if mixtures are present. Our data showed that individual sequence homologies to the group consensus sequence were generally 98%. Since only 1 of 114 HGS and 12 of 80 TRUGENE sequence data sets were <98% homologous to the respective GCS, the data suggest that a cutoff value for assessment of proficiency should be a minimum of 98% homology and could possibly be set higher. Until more data are obtained and analyzed to refine the recommendation, the suggestion of 98% homology would incorporate the more stringent interpretation of mixtures discussed above for Fig. 4. This would mean that if a mixed base is represented in the GCS, then data returned by a laboratory should show a mixed base in the same position to be considered homologous. Sequence homology to a group consensus sequence was shown to be slightly lower by Sayer et al. (8), but these results may be influenced by the high proportion of in-house assays in the group evaluated (six of nine laboratories). Although some samples in our study displayed <98% concordance to the GCS, it is possible that the discrepancies were generated in part by the use of RUO kits where less-stringent quality control of kit reagents was in place. It is anticipated that the number of samples that should reach at least >98% concordance to the GCS would increase using FDA-approved kits and software for analysis.
Another indicator that might contribute to assessing proficiency is the generation of discordant data along the template sequence (8, 9). By examining the patterns of editing reflected by the case designation of each base reported, potential difficulty in technical performance may be identified. For instance, large regions of sequence requiring editing may reflect generation of poor sequence data. Areas of discordance in sequence reporting may point out differences in editing strategy (6) that can affect the identification of mutations associated with antiretroviral resistance.
It is more difficult to set forth criteria to evaluate the identification of mutations associated with antiretroviral resistance. Our data suggest that concordant identification of the presence or absence of codons representing antiretroviral resistance at a site of interest was high but was not always 100%. In the examples described in Table 2, in one instance, one laboratory reported the presence of a resistant mutation when the rest of the group identified a WT codon; in the other instance, one laboratory reported a WT codon when the rest identified a resistance mutation. More data, perhaps using replicate samples in panels, need to be analyzed to determine how best to evaluate proficiency in identification of antiretroviral resistance mutations.
Editing performance was generally not found to be useful in proficiency testing evaluation. The data in Table 2 showed that different editing strategies might be used while still employing the guidelines provided during company-sponsored training. The variation in strategy might be reflective of the quality of sequence data generated from the technical components of the assay. However, the data shown in Fig. 2 and 3 indicate that the laboratories submitted data that were highly concordant with the rest of the group despite editing practices. The data suggest that generation of good sequence data can occur although the editing process may not be absolutely uniform among laboratories.
As stated earlier, both FDA-approved genotyping systems provide positive and negative controls and instruction to use them with each batch of samples to be sequenced. The kit controls are useful for determining whether the protocol, reagents, and instrumentation are in good working order and whether individuals performing the assays are able to generate good-quality sequences. Depending on the control used, different steps of the protocol can be monitored routinely during assay performance, which contributes to the laboratory's success in proficiency testing. However, as also mentioned, these defined controls do not allow the complexities of the assay to be monitored in a way similar to proficiency testing.
Our experience with these panels and pre-FDA-approved kits suggests that analysis of proficiency of individual laboratories should be grouped by platform to allow for differences in the configurations of the assays (2). In this way the performance by the laboratories can be evaluated within platform fairly and will not be influenced by the differences in kit configuration and software interpretations or by the length of sequence data generated. Our data also suggest that the use of clinical samples to test proficiency may provide an accurate evaluation of the complexities involved in performance of genotyping assays.
A proficiency program for approximately 40 participant laboratories using primarily the TRUGENE and ViroSeq platforms was initiated using several of the criteria described in this study. Seven panels of clinical samples have been distributed over the past 3 years. Cumulative data from these panels, especially those of replicate samples, have reinforced and strengthened the use of these criteria as an effective means of monitoring the performance of genotyping assays and learning more about the factors that can contribute to the variability of the assay.
ACKNOWLEDGMENTS
This work was supported by Public Health Service contract NO1-AI-85354 from the National Institute of Allergy and Infectious Diseases.
We thank Applied Biosystems for providing the HIV-1 Genotyping System version 1 and software versions 1.1 and 2.0.
Members of the Pediatric AIDS Clinical Trials Group Sequencing Working Group include Grace Aldrovandi, University of Alabama at Birmingham, Don Brambilla, New England Research Institute, Clark Brown, Applied Biosystems, Susan Eshleman, Johns Hopkins Medical Institutions, Susan Fiscus, University of North Carolina, Lisa Frenkel, University of Washington, Hasnah Hamdan, Nichols Institute, Stephen Hart, Frontier Science and Technology Research Foundation, Diana Huang, Rush Medical College, Andrea Kovacs, University of Southern California, Paul Krogstad, University of California at Los Angeles, Phillip LaRussa, Columbia University, Paul Palumbo, University of Medicine and Dentistry of New Jersey, Walter Scott, University of Miami, Stephen Spector, University of California at San Diego, John Sullivan, University of Massachusetts, Adriana Weinberg, University of Colorado Health Sciences Center, and Yu Qi Zhao, Northwestern University. Participants outside of the PACTG group who contributed sequence data include Daniel Kuritzkes (University of Colorado; now at Harvard University) and Lynne Hough (Applied Sciences, Norcross, GA).
REFERENCES
Coffin, J. M. 1995. HIV population dynamics in vivo: implications for genetic variation, pathogenesis, and therapy. Science 267:483-489.
Erali, M., S. Page, L. G. Reimer, and D. R. Hillyard. 2001. Human immunodeficiency virus type 1 drug resistance testing: a comparison of three sequence-based methods. J. Clin. Microbiol. 39:2157-2165.
Hanna, G. J., and R. T. D'Aquila. 2001. Clinical use of genotypic and phenotypic drug resistance testing to monitor antiretroviral chemotherapy. Clin. Infect. Dis. 32:774-782.
Hirsch, M. S., F. Brun-Vezinet, B. Clotet, B. Conway, D. R. Kuritzkes, R. T. D'Aquila, L. M. Demeter, S. M. Hammer, V. A. Johnson, C. Loveday, J. W. Mellors, D. M. Jacobsen, and D. D. Richman. 2003. Antiretroviral drug resistance testing in adults infected with human immunodeficiency virus type 1: 2003 recommendations of an International AIDS Society-USA Panel. Clin. Infect. Dis. 37:113-128.
Hollinger, F. B., J. W. Bremer, L. E. Myers, J. W. Gold, L. McQuay, and The NIH/NIAID/DAIDS/ACTG Virology Laboratories. 1992. Standardization of sensitive human immunodeficiency virus coculture procedures and establishment of a multicenter quality assurance program for the AIDS Clinical Trials Group. J. Clin. Microbiol. 30:1787-1794.
Huang, D. D., S. H. Eshleman, D. J. Brambilla, P. E. Palumbo, and J. W. Bremer. 2003. Evaluation of the editing process in human immunodeficiency virus type 1 genotyping. J. Clin. Microbiol. 41:3265-3272.
Perkin-Elmer Corporation. 1995. Comparative PCR sequencing: a guide to sequencing-based mutation detection. Applied Biosystems Division, Perkin-Elmer Corporation, Foster City, CA.
Sayer, D. C., S. Land, L. Gizzarelli, M. French, G. Hales, S. Emery, F. T. Christiansen, and E. M. Dax. 2003. Quality assessment program for genotypic antiretroviral testing improves detection of drug resistance mutations. J. Clin. Microbiol. 41:227-236.
Shafer, R. W., K. Hertogs, A. R. Zolopa, A. Warford, S. Bloor, B. J. Betts, T. C. Merigan, R. Harrigan, and B. A. Larder. 2001. High degree of interlaboratory reproducibility of human immunodeficiency virus type 1 protease and reverse transcriptase sequencing of plasma samples from heavily treated patients. J. Clin. Microbiol. 39:1522-1529.(Diana D. Huang, James W. )
New England Research Institute, Inc. Watertown, Massachusetts
Pediatrics, University of Medicine and Dentistry of New Jersey, New Jersey Medical School, Newark, New Jersey
ABSTRACT
Use of sequencing-based genotyping as a diagnostic assay for human immunodeficiency virus (HIV) antiretroviral resistance is increasing. Periodic evaluation of the proficiency of laboratories performing this assay should be established. It is important to identify components of the assay that influence the generation of reliable sequencing data and that should and can be monitored. A model was developed to determine what parameters were reasonable and feasible for assessing the performance of genotyping assays. Ten laboratories using the genotyping platform, HIV-1 Genotyping System (HGS) v. 1 and software versions 1.1 or 2.0, participated in two rounds of testing. For each round, each group was sent a panel consisting of three clinical samples to sequence in real time. Six months later, seven laboratories using the TRUGENE HIV-1 Genotyping Kit participated in a separate round, working with both panels at the same time. Analysis of the data showed that one main indicator of genotyping proficiency was achievement of 98% sequence homology of a sample tested to a group consensus sequence for that sample. A second was concordant identification of codons at sites identified with resistance mutations in the sample, although scoring of these criteria is still undetermined from this study. These criteria are applicable to all sequence-based genotyping platforms and have been used as a baseline for assessing the performance of genotyping for the determination of antiretroviral resistance in our ongoing proficiency program.
INTRODUCTION
Genotyping of the human immunodeficiency virus type 1 (HIV-1) protease (PR) and reverse transcriptase (RT) genes by population sequencing to assess antiretroviral resistance is a standard diagnostic assay to aid the clinical management of HIV-infected individuals (3). Advisory groups have recommended HIV resistance genotyping to assist in analysis of certain clinical care settings (e.g., beginning therapy for AIDS-associated retrovirus [ARV]-nave individuals; choosing new regimens for those failing ARV therapy) (4) These assays comprise multiple sequential components and require expertise in both the physical performance of the assays and the subjective evaluation of the generated data (Fig. 1). The genotyping assays that are currently Food and Drug Administration (FDA)-approved, ViroSeq HIV-1 Genotyping System v. 2.0 (Celera Diagnostics, Alameda, CA) and TRUGENE HIV-1 Genotyping Kit (Bayer HealthCare LLC, Berkeley, CA), provide positive and negative run controls that are to be amplified and sequenced with each batch of clinical samples. The positive controls are basically homogeneous in composition, as they are, respectively, either diluted cultured HIV-1 strain 8E5 or RNA transcripts of HIV-1 derived from a plasmid bearing an appropriate insert. Because HIV-1 quasispecies circulate in the infected individual (1), clinical samples contain heterogeneous virus populations. As a result, mixed nucleotides (mixtures) at individual nucleotide positions may be evident in sequence analysis and may indicate viral variants of increasing or declining resistance. The presence of mixtures may also result from sequence artifacts generated through suboptimal performance of some technical portion of the assay. For example, inadequate sample preparation, reverse transcription, PCR amplification, binding of the sequencing primer to the sequencing template, purification of PCR or sequencing products, and electrophoresis through gel or polymer matrices may each contribute to data of questionable quality (7). At times, the nucleotide mixtures need to be examined and resolved by visual evaluation of the electropherograms by the person performing the assay. In these instances, the sequence editor has to assess whether the mixtures observed are genuine or artifacts. This process is called editing (6). Given the technical and subjective complexity of the assays, there is a need for testing the reproducibility of the assays and the proficiency and consistency of the operators by use of samples more similar to real clinical specimens than the run controls provided.
The members of the Pediatric Virology Core Laboratories of the Pediatric AIDS Clinical Trials Group (PACTG) performed genotyping on two panels of plasma samples to identify factors that might contribute to interlaboratory variability in the performance of sequenced-based genotypic assays. Participants used either the HIV-1 Genotyping System (HGS) v. 1 or TRUGENE HIV-1 Genotyping Assay, both of which included platform-associated software. Information from the study was used as a model for evaluation of performance of sequenced-based genotypic assays and to begin to formulate criteria for assessing the overall quality of sequence data generated. Several of these criteria are presented here.
MATERIALS AND METHODS
Sequencing assay. The research-use-only (RUO) HIV-1 Genotyping System (HGS) v. 1 accompanied by software versions 1.1 (panel 1) and 2.0 (panel 2) (Applied Biosystems, Foster City, CA) or the TRUGENE HIV-1 Genotyping Assay (TRUGENE) accompanied by software v. 3.0.1(Visible Genetics, Suwanee GA) was used by the participating laboratories. These systems were the predecessors to the current FDA-approved ViroSeq HIV-1 Genotyping System v. 2.0 (Celera Diagnostics, Alameda, CA) and TRUGENE HIV-1 Genotyping Kit (Bayer HealthCare LLC, Berkeley, CA). To obtain the kit, personnel from each laboratory were trained to perform the assay and edit the resulting data prior to receipt of certification samples from each company. Editing guidelines were taught during formal training provided by the company.
Plasma samples and panels. Two panels each containing 1 ml of plasma from each of three HIV-1-infected individuals were distributed to 10 laboratories of the PACTG Sequencing Working Group. These samples were collected, characterized, and sequenced by the Virology Quality Assurance (VQA) Laboratory (5) prior to distribution. Panel 1 samples (Table 1) were from donors untreated with antiretroviral drugs for at least 1 year and were used to determine whether the HGS laboratories were performing the sequencing portion of the assay consistently. Panel 2 plasma donors demonstrated increased viral loads while on therapy, which may indicate therapeutic failure. Their plasma samples were used to see whether identification of mutations and editing occurred consistently across laboratories (6). The panels were distributed approximately 6 months apart to those laboratories in which the HGS kit was used. The laboratories using the TRUGENE assay (total, n = 7; PACTG, n = 5; Visible Genetics, Inc.; University of Colorado, n = 2) received both panels at the same time 6 months after panel 2 was sent to the HGS group. Distribution of the panels to the respective groups was made according to the availability of (i) the RUO research and clinical configurations of the respective kits for use and (ii) a critical mass of trained and certified laboratories able to participate to generate enough data to analyze. The HGS laboratories were given the same version of the kit configuration and the then-current software to use for each panel. The TRUGENE groups used the same version of reagents for their panels but were provided either the research or clinical version of the software by the manufacturer. For this study, both groups of laboratories were instructed to use their respective kits and software as they were trained to do by the companies. Plasma was collected by protocols approved by the Institutional Review Board at Rush University Medical Center.
Data collection and analysis. Within 4 weeks after receipt of the panels, laboratories personnel submitted sequence data as text files in FASTA format electronically to Frontier Science & Technology Research Foundation, Amherst, NY, for data collection and transcription. The transcribed data were sent to New England Research Institute, Watertown, MA, and the VQA Laboratory for analysis of each sample. The laboratories that used the HGS kit also sent the VQA Laboratory a copy of the final edited "project" that contained the embedded electropherograms of the individual sequences used to form the consensus sequences for further examination, if needed. TRUGENE users sent only the text files of the edited consensus sequences, because the version of the software system in use when the panels were sequenced differed among laboratories. Also, electropherograms associated with the edited TRUGENE cases were not readily accessible to off-site work stations, a characteristic of the platform operating system, Openstep, at the time.
Both HGS genotyping software versions 1.1 and 2.0 and TRUGENE software v. 3.0.1 allow editing to be tracked. Bases identified by each kit's software, which aligns bidirectional overlapping data from sequencing primers to form a consensus sequence, are automatically recorded in upper case. The individual nucleotides in the consensus sequence that are changed manually are recorded in lowercase characters. The case designation is preserved in the sequence text file generated by the software, thus documenting bases that were manually edited.
For each platform, the group consensus sequence (GCS) was generated by alignment with Align Plus (Scientific Educational Software, Durham, North Carolina) at a 60% threshold. Homologies to the GCS were determined in a second alignment using the GCS as the reference strain. Sequence data from each platform were analyzed separately partly due to differences in length and portion of the protease (PR) and reverse transcriptase (RT) gene sequence generated by the kits, software design and editing guidelines associated with each assay, and overall protocol design used to produce sequence data. HGS data were analyzed for the entire PR gene (99 codons) and the RT gene for codons 1 to 320; TRUGENE data covered codons 4 to 99 of the PR gene and codons 37 to 247 of the RT gene. Any codon for any laboratory that contained a nucleotide in lowercase characters was classified as edited. The number of edited codons for each sample for each laboratory was tabulated. Data were examined for (i) concordance of submitted nucleotides to the GCS for that sample; (ii) frequency of editing; and (iii) sites on the entire template that were edited.
Mutations associated with antiretroviral resistance were identified for each laboratory for the samples in panel 2 only. These mutations were recognized by algorithms incorporated into the software version that was available when the assay was performed.
RESULTS
Data from both panels were obtained from 9 of 10 laboratories in which HGS was used, while data from panel 1 were only obtained from the 10th laboratory. A total of 114 separate gene sequences, 57 from RT and 57 from PR, were analyzed. Six of seven laboratories in which the TRUGENE kit was used submitted data for all six samples in both panels. The seventh laboratory submitted two samples each for panel 1 and 2. A total of 80 sequences, 40 from PR and 40 from RT, were analyzed.
Homology to group consensus sequence versus editing. Each group's data were evaluated to determine how closely their individually edited consensus sequences matched the total group consensus sequence for each gene across the entire backbone. Agreement among the HGS laboratories was high for both the PR and the RT genes. Of 114 individual consensus sequences submitted, only one displayed < 98% homology to the GCS. Concordance to the group consensus sequence was 97.3 to 100.0% for PR (297 bases) (Fig. 2A) and 98.0 to 100.0% for RT (960 bases) (Fig. 2B). The percentage of codons in each sequence edited by laboratory personnel to form a consensus sequence ranged between 2.0 and 87.9% and between 4.7 and 63.6% for the protease and RT genes, respectively. Homology was negatively correlated with edit rate for PR (Spearman rank correlation, r = –0.36; P = 0.0061) but not for RT (Spearman rank correlation, r = –0.21; P = 0.13), although the trends in the two plots look similar. However, even at the highest editing rates, homology to the GCS remained high. Only 11 (5 PR, 6 RT) of 114 sequences had <99% homology to the GCS and only 1 of the 11 (97.3%) had <98% homology to the GCS. Also, no association between experience in performance of the assay and amount of editing was observed (data not shown).
The laboratories using the TRUGENE assay showed similar trends. For the protease gene (Fig. 3A), 34 of 40 submitted sequences were concordant to the GCS at >98.0%. Editing of the protease gene ranged from 0.0 to 28.1%. The clustering of concordance was tighter for the RT gene (Fig. 3B) where 34 of 40 sequences matched the GCS at >98.5%. RT gene editing ranged between 0.4 and 13.3%. Again, no clear-cut correlation between the amount of editing and final homology was apparent.
Identification of mutations associated with antiretroviral resistance. The ability to identify mutations associated with antiretroviral resistance was used as another measure of genotyping performance. During prescreening analysis a total of 36 mutations that were associated with antiretroviral resistance were identified across the samples in panel 2 (Table 1). Among the nine laboratories in which the HGS platform was used for this panel, there were five discrepancies in the codons with mutations. Table 2 shows the reported composition of these codons and the editing decisions (lowercase letters) made to identify the composition of the reported codon. There were four instances in which the codon from a single laboratory differed from the codons from the other eight groups. Three of these four discrepancies involved a codon that contained a mixed base that included the resistance mutation. It was interesting to see that the identical codon sequences were from the eight laboratories without discrepancies, even though editing patterns differed among laboratories. For sample 02RG03 RT, four groups identified a mixed base (Y) in the codon AYA while the other four identified the codon as ATA. All eight codons were unedited. For the laboratories in which the TRUGENE assay was used, there were three discrepant identifications of mutations associated with antiretroviral resistance for the panel 2 samples. The discrepancies in the sequences from the TRUEGENE and HGS assays occurred at different codons. The combinations of nucleotides reported for these codons were similar in pattern to those described in Table 2 (data not shown). The results indicate that discrepancies in data could be generated during earlier steps in the assay before editing and may appear unambiguous during the proofreading stage, precluding the need for editing.
Using the HGS group data in Table 2, for nine laboratories working with 36 mutations, the overall failure to detect a mutation was low (1.5%). However, to measure the proficiency of an individual laboratory, it might be more appropriate to examine the number of discordant identifications made for each set of 36 mutations associated with the panel. Using the examples in Table 2, discordant identifications were made twice. For 02RG01 L24I (PR), eight of nine groups identified only the wild-type (WT) codon and one of nine reported the mutation. For 02RG02 T215Y (RT), eight of nine groups reported the mutation and one of nine groups reported a wild-type codon. The frequency of disagreement of these two individual laboratories with the rest of the group for the entire panel would be 1 of 36 or 2.8%. Each codon reported for a site of interest that was discrepant from the codon reported by the rest of group could be flagged and considered in the assessment of laboratory proficiency.
Discordant nucleotide identification along the template. The ability to accurately report the identity of each nucleotide at each position along the template could be considered in the assessment and evaluation of genotyping proficiency. This issue was also described by Sayer et al. (8) in their analysis of, predominantly, in-house assays and by Shafer et al., who examined replicate testing by two laboratories also using in-house assays (9). In our testing, all laboratories used the same sequencing platform and the group consensus sequence for each sample was generated by alignment using a 60% threshold. This means that the same nucleotide in the GCS was identified by 60% of the laboratories. Figure 4 shows three examples of discordant identification of a nucleotide by individual groups by use of the HGS platform at positions 532, 534, and 846. In 8 of 10 sequences, the base at position 532 was identified as "r." In the other two sequences, from laboratories 3 and 4, the nucleotide was identified as "g." The "r" is interpreted by International Union of Biochemistry (IUB) code to be a mixture composed of bases "a" and "g." The converse situation is shown for position 534. Here, 60% of the laboratories identified the base to be a pure base "a," which is recorded in the group consensus sequence, and 40% of the groups designated the base to be "r." In the final example at position 846, 9 of 10 laboratories identified the base as a "c," but 1 group reported a "t." In this example, no mixed bases were identified for that position.
The issue to be resolved is how to define an accurate answer or whether this can even be accomplished. Using data for nucleotide position 532 and 534 as an example, since the "r" is composed of "a"and "g," accuracy can be construed in several ways. For example, a narrow assessment of the situations described above could be that the nucleotide designated in the GCS is correct and all deviations are wrong. A broader view could be that if there is a mixture reported by any of the labs ("r") which is comprised of any of the pure nucleotides identified ("a" or "g"), then results reporting "r" or "a" or "g" are all potentially equally accurate. The broadest view might be that each discrepant base reported is accurate if the reporting can be verified by visual inspection of the edited electropherograms or text files. The situation represented for nucleotide position 846 is more straightforward. Here, 9 of 10 groups identified and reported the base as "c." Only 1 group reported a different base "t." Since no mixtures were identified and reported by any group, the discrepant laboratory would be considered to have reported an incorrect nucleotide.
DISCUSSION
This study identified two criteria that could be useful for the initial assessment of performance of sequence-based genotyping assays when participants are using the same sequencing platform. These criteria are (i) the overall sequence homology of each laboratory's consensus sequence to a group consensus sequence and (ii) the identification of mutations associated with antiretroviral resistance. A high level of concordance of the sequence from a laboratory to the group consensus sequence suggests that laboratory personnel were capably performing the technical and subjective portions of the assay. Concordant identification of mutations can provide an additional evaluation of the consistency of performance of the subjective component of the assay, i.e., editing, among laboratories if mixtures are present. Our data showed that individual sequence homologies to the group consensus sequence were generally 98%. Since only 1 of 114 HGS and 12 of 80 TRUGENE sequence data sets were <98% homologous to the respective GCS, the data suggest that a cutoff value for assessment of proficiency should be a minimum of 98% homology and could possibly be set higher. Until more data are obtained and analyzed to refine the recommendation, the suggestion of 98% homology would incorporate the more stringent interpretation of mixtures discussed above for Fig. 4. This would mean that if a mixed base is represented in the GCS, then data returned by a laboratory should show a mixed base in the same position to be considered homologous. Sequence homology to a group consensus sequence was shown to be slightly lower by Sayer et al. (8), but these results may be influenced by the high proportion of in-house assays in the group evaluated (six of nine laboratories). Although some samples in our study displayed <98% concordance to the GCS, it is possible that the discrepancies were generated in part by the use of RUO kits where less-stringent quality control of kit reagents was in place. It is anticipated that the number of samples that should reach at least >98% concordance to the GCS would increase using FDA-approved kits and software for analysis.
Another indicator that might contribute to assessing proficiency is the generation of discordant data along the template sequence (8, 9). By examining the patterns of editing reflected by the case designation of each base reported, potential difficulty in technical performance may be identified. For instance, large regions of sequence requiring editing may reflect generation of poor sequence data. Areas of discordance in sequence reporting may point out differences in editing strategy (6) that can affect the identification of mutations associated with antiretroviral resistance.
It is more difficult to set forth criteria to evaluate the identification of mutations associated with antiretroviral resistance. Our data suggest that concordant identification of the presence or absence of codons representing antiretroviral resistance at a site of interest was high but was not always 100%. In the examples described in Table 2, in one instance, one laboratory reported the presence of a resistant mutation when the rest of the group identified a WT codon; in the other instance, one laboratory reported a WT codon when the rest identified a resistance mutation. More data, perhaps using replicate samples in panels, need to be analyzed to determine how best to evaluate proficiency in identification of antiretroviral resistance mutations.
Editing performance was generally not found to be useful in proficiency testing evaluation. The data in Table 2 showed that different editing strategies might be used while still employing the guidelines provided during company-sponsored training. The variation in strategy might be reflective of the quality of sequence data generated from the technical components of the assay. However, the data shown in Fig. 2 and 3 indicate that the laboratories submitted data that were highly concordant with the rest of the group despite editing practices. The data suggest that generation of good sequence data can occur although the editing process may not be absolutely uniform among laboratories.
As stated earlier, both FDA-approved genotyping systems provide positive and negative controls and instruction to use them with each batch of samples to be sequenced. The kit controls are useful for determining whether the protocol, reagents, and instrumentation are in good working order and whether individuals performing the assays are able to generate good-quality sequences. Depending on the control used, different steps of the protocol can be monitored routinely during assay performance, which contributes to the laboratory's success in proficiency testing. However, as also mentioned, these defined controls do not allow the complexities of the assay to be monitored in a way similar to proficiency testing.
Our experience with these panels and pre-FDA-approved kits suggests that analysis of proficiency of individual laboratories should be grouped by platform to allow for differences in the configurations of the assays (2). In this way the performance by the laboratories can be evaluated within platform fairly and will not be influenced by the differences in kit configuration and software interpretations or by the length of sequence data generated. Our data also suggest that the use of clinical samples to test proficiency may provide an accurate evaluation of the complexities involved in performance of genotyping assays.
A proficiency program for approximately 40 participant laboratories using primarily the TRUGENE and ViroSeq platforms was initiated using several of the criteria described in this study. Seven panels of clinical samples have been distributed over the past 3 years. Cumulative data from these panels, especially those of replicate samples, have reinforced and strengthened the use of these criteria as an effective means of monitoring the performance of genotyping assays and learning more about the factors that can contribute to the variability of the assay.
ACKNOWLEDGMENTS
This work was supported by Public Health Service contract NO1-AI-85354 from the National Institute of Allergy and Infectious Diseases.
We thank Applied Biosystems for providing the HIV-1 Genotyping System version 1 and software versions 1.1 and 2.0.
Members of the Pediatric AIDS Clinical Trials Group Sequencing Working Group include Grace Aldrovandi, University of Alabama at Birmingham, Don Brambilla, New England Research Institute, Clark Brown, Applied Biosystems, Susan Eshleman, Johns Hopkins Medical Institutions, Susan Fiscus, University of North Carolina, Lisa Frenkel, University of Washington, Hasnah Hamdan, Nichols Institute, Stephen Hart, Frontier Science and Technology Research Foundation, Diana Huang, Rush Medical College, Andrea Kovacs, University of Southern California, Paul Krogstad, University of California at Los Angeles, Phillip LaRussa, Columbia University, Paul Palumbo, University of Medicine and Dentistry of New Jersey, Walter Scott, University of Miami, Stephen Spector, University of California at San Diego, John Sullivan, University of Massachusetts, Adriana Weinberg, University of Colorado Health Sciences Center, and Yu Qi Zhao, Northwestern University. Participants outside of the PACTG group who contributed sequence data include Daniel Kuritzkes (University of Colorado; now at Harvard University) and Lynne Hough (Applied Sciences, Norcross, GA).
REFERENCES
Coffin, J. M. 1995. HIV population dynamics in vivo: implications for genetic variation, pathogenesis, and therapy. Science 267:483-489.
Erali, M., S. Page, L. G. Reimer, and D. R. Hillyard. 2001. Human immunodeficiency virus type 1 drug resistance testing: a comparison of three sequence-based methods. J. Clin. Microbiol. 39:2157-2165.
Hanna, G. J., and R. T. D'Aquila. 2001. Clinical use of genotypic and phenotypic drug resistance testing to monitor antiretroviral chemotherapy. Clin. Infect. Dis. 32:774-782.
Hirsch, M. S., F. Brun-Vezinet, B. Clotet, B. Conway, D. R. Kuritzkes, R. T. D'Aquila, L. M. Demeter, S. M. Hammer, V. A. Johnson, C. Loveday, J. W. Mellors, D. M. Jacobsen, and D. D. Richman. 2003. Antiretroviral drug resistance testing in adults infected with human immunodeficiency virus type 1: 2003 recommendations of an International AIDS Society-USA Panel. Clin. Infect. Dis. 37:113-128.
Hollinger, F. B., J. W. Bremer, L. E. Myers, J. W. Gold, L. McQuay, and The NIH/NIAID/DAIDS/ACTG Virology Laboratories. 1992. Standardization of sensitive human immunodeficiency virus coculture procedures and establishment of a multicenter quality assurance program for the AIDS Clinical Trials Group. J. Clin. Microbiol. 30:1787-1794.
Huang, D. D., S. H. Eshleman, D. J. Brambilla, P. E. Palumbo, and J. W. Bremer. 2003. Evaluation of the editing process in human immunodeficiency virus type 1 genotyping. J. Clin. Microbiol. 41:3265-3272.
Perkin-Elmer Corporation. 1995. Comparative PCR sequencing: a guide to sequencing-based mutation detection. Applied Biosystems Division, Perkin-Elmer Corporation, Foster City, CA.
Sayer, D. C., S. Land, L. Gizzarelli, M. French, G. Hales, S. Emery, F. T. Christiansen, and E. M. Dax. 2003. Quality assessment program for genotypic antiretroviral testing improves detection of drug resistance mutations. J. Clin. Microbiol. 41:227-236.
Shafer, R. W., K. Hertogs, A. R. Zolopa, A. Warford, S. Bloor, B. J. Betts, T. C. Merigan, R. Harrigan, and B. A. Larder. 2001. High degree of interlaboratory reproducibility of human immunodeficiency virus type 1 protease and reverse transcriptase sequencing of plasma samples from heavily treated patients. J. Clin. Microbiol. 39:1522-1529.(Diana D. Huang, James W. )