A rapid assessment of community-wide HIV/STI intervention in China
http://www.100md.com
《性传输传染杂志》
Background/objectives: The key to HIV/STI control is community-wide intervention (CWI) which depends heavily on continuous monitoring and evaluation. Unfortunately, comprehensive CWI assessment methodology and reports are generally lacking. This study developed, applied, and evaluated a rapid tool for assessing CWI in China.
Methods: A total of 120 county level respondents in charge of county-wide responses to HIV/STI thoughout China were selected randomly and surveyed using a structured inventory consisting of three tiers of indicators developed via consensus group techniques. The respondents were asked to rate each of the indicators against a five grade (1–5) scale. 30 pairs of the same staff from within Anhui Province were surveyed to gauge inter-rater reliability.
Results: Response rate for the nationwide survey was 85% and for inter-rater reliability survey, 90%. Correlation coefficients between the inter-rater ratings ranged from 0.68 to 0.95. The overall average rating of CWI in China was 2.85. Average ratings for the six first tier indicators, organisation and policy development, goals and objectives setting, project and action planning, resource exploitation, project and task implementation, and CWI evaluation were 2.87, 2.83, 2.67, 2.77, 3.26, and 2.71 respectively. Ratings derived for the 24 second tier indicators ranged from 2.1 to 3.86; while for the 96 third tier indicators, 1.90 to 4.40
Conclusions: The instrument developed proved to be reliable, useful, and easily applicable in common communities. Application of it in China revealed that a large gap exists between desired and actual CWI, and areas meriting particular attention include policy and incentives development, intervention planning and evaluation, and fund raising and utilisation.
Abbreviations: CDC, centre for disease control; CWI, community-wide intervention
Keywords: HIV/STI; community intervention; China
It is becoming clear that effective control of human immunodeficiency virus and sexually transmitted infections (HIV/STI) depends heavily on community-wide intervention (CWI), since the epidemic dynamic is deeply embedded in individual desires, social and cultural relationships, as well as environmental and economic processes.1–3 The essence of CWI against HIV/STI is to maximise outcomes by mobilising the whole community and using comprehensive approaches targeting different groups and factors via different means and channels in a synergetic manner.1,4,5 To reach this goal, we must continuously monitor and evaluate intervention activities and impacts.6 Although there is a substantial literature on evaluation of research trials or demonstration programmes against HIV/STI,7–11 reports on routine responses to the epidemic in general communities at large from a holistic perspective (or CWI) are generally lacking.
Evaluation of CWI can be performed differently, ranging from simply asking a few broad questions to conducting exhaustive investigations involving multiple methods, multiple targets, multiple funding sources, multiple perspectives, multiple paradigms, multiple roles, and multiple solutions to multiple problems.12,13 While Rehle and colleagues’ handbook provides useful guidance and tools in implementing,14 an exhaustive approach to CWI evaluation is seldom feasible or necessary. What are needed mostly are practical approaches that require minimum resources and expertise and are, therefore, more appropriate for use by resource poor communities.
China is witnessing a rapid increase in HIV/STI cases.15,16 Whether can China reverse this trend depends largely on rigorous and coordinated responses of the communities that make up the country. Unfortunately, there has been no systematic evaluation of the status quo of CWI in China. Meanwhile, anecdotal accounts are disturbing: inadequate leadership limits multisector involvement and concordant intervention; conflict of interests has prevented many proved programmes since, for example, syndromic management of STDs could cause a service provider to lose up to two thirds of his revenue for the same number of STD patients served; programmes suffer from missing links between job performance and reward; and ignorance of planning and evaluation results in contradictory efforts and futile programmes, etc. This study aims at developing a rapid yet comprehensive evaluation method and using it to assess current CWI in China.
Inventory development
We implemented three steps in developing an instrument for CWI assessment. Firstly, a draft was produced after extensive publication review. Then, a nominal group technique was applied three times to refine the draft involving a panel of seven experts on CWI against HIV/STI, health services management, health education and behaviour modification, epidemiology, and health statistics.17 Finally, the inventory generated was piloted and revision made according to feedback solicited. Underlying the instrument development was a hierarchical framework that divided CWI tasks into distinctive and equally important subareas level by level until a balance had been reached between manageability and specificity of the divisions. As a result, the final instrument consists of three tiers of indicators. The six first tier indicators represent overall components (A–F), including organisation and policy development, goal and objectives setting, project and action planning, resource exploitation, project or task implementation, and intervention monitoring and evaluation. Similarly, the 24 second tier and the 96 third tier indicators are components of the first and second tier indicators, respectively. All the 96 third tier indicators were phrased as succinct statements, each describes a standard for a specific aspect of CWI (see appendix on STI website, www.stijournal.com). Each statement has a five degree rating scale (see table 1). At the end of the inventory, there is a structured question about the respondent’s knowledge about the actual status inquired about. Completion of the inventory requires about 30 minutes.
Data collection
A "community" in this study was defined as a county or county level city (hereafter referred to as county). In China, counties are administrative subunits below provinces or cities, defined by a whole range of factors including population size, geographic boundaries, and sociocultural traditions. They are the most meaningful units to launch a CWI against HIV/STI and have actually been assigned the task to do so. For this study, 120 counties were selected randomly from the list of over 2000 counties that make up mainland China using random number table. One staff member from each centre for disease control (CDC) of the sampled counties was selected as the respondent. The selection was aimed at identifying the best informant of local CWI from the CDCs. Thus, preset criteria were used to prioritise relevant staff members (2–6 people per CDC) including years in charge of local CWI, professional title, and years of relevant education and training. The one member with top ranking was sent, in April 2003, the questionnaire developed above by mail and asked to rate actual interventions in their local counties according to stated standards.
A covering letter was also sent to the respondents along with the questionnaire, which clearly states that: (a) the survey is purely for research purposes and it is of enough importance to report as objectively as possible; (b) data collected will be kept in a safe place with total confidentiality; (c) only aggregate statistics will be published, no county specific data or comparisons will be disclosed, and the research team assures no harm to any respondent and his or her organisation or county as a result of the survey. In addition, each respondent was sent an addressed and stamped envelope to facilitate the return of the completed questionnaire. Telephone recalls were made to those who failed to respond within 1 month.
In order to gauge inter-rater reliability, 30 pairs of staff (two members from the same CDC) in Anhui Province were selected and surveyed using the same method.
Data process and analysis
SPSS 10.0 was used to process and analyse the data. Ratings for the third tier indicators by a specific respondent were derived directly by translating the responses of "" "" "" "" and "" into numbers 1, 2, 3, 4, and 5 respectively, while the ratings for the first and second tier indicators were calculated as averages of the components (that is, the second or third tier indicators). Similarly, the overall rating of CWI was generated by adding all the six first tier ratings and then dividing the sum by 6. Means and standard deviations of the ratings against all the indicators were calculated. Bivariate correlation analysis was performed to measure inter-rater reliability. Comparisons of means between ratings of different indicators were also made when necessary using the "one way ANOVA" procedure.
From the 30 pairs of staff members in Anhui Province surveyed for inter-rater reliability, 27 pairs of valid questionnaires were returned; from the nationwide survey, 102 valid questionnaires were collected, adding up to a response rate of 85%. Correlation coefficients between the ratings of the same indicators given by the paired staff members were high, ranging from 0.68 to 0.95. Means and standard deviations of all the ratings derived from the nationwide survey are given in table 2, while figure 1 depicts all the ratings on a set of radar diagrams. The overall average rating was 2.85. This is quite low since the maximum theoretical rating for any given indicator is 5. Average ratings for the six first tier indicators—organisation and policy development, goals and objectives setting, project and action planning, resource exploitation, project and task implementation, and CWI evaluation— were 2.87, 2.83, 2.67, 2.77, 3.26, and 2.71 respectively. The difference between ratings of most indicators is statistically significant (p<0.05).
More specifically, ratings concerning the first component of CWI, organisation and policy development, indicate that relatively greater progress had been made in leadership and management organisation development (A1a: 3.60; A1c: 3.61) and responsibility definition (A2a: 3.46; A2c: 3.31; A2d: 3.77). At first glance, these are encouraging signs, for management structure has an important role in CWI.4 However, the low ratings of the mechanisms for bringing the structure into play (A3a-d: 1.95–2.92) suggest that these organisational and responsibilities developments may be more symbolic than functional. This point is also supported by the significantly lower rating on the establishment of intervention network (which is much more difficult) than that of steward organisations (A1d: 2.85 v A1a: 3.60 and A1c: 3.61; p<0.05), for it indicates that organisations were developed upon easiness to build rather than real need.
Another point worth noting is that the respondents gave rather low ratings to CWI related policy development (A4a-d: 2.33–3.19). This implies again that the steward structures were not functioning well and is consistent with other reports that HIV/STI was not adequately recognised by local governments, that the infected individuals were often stigmatised and denied care, schooling and jobs, and that there were various policy barriers to proved interventions like methadone substitution, needle and syringe exchange for injecting drug users, and condom promotion among sex workers.18,19
The second component, goal and objective setting, and the third, project and action plan, are all integral parts of a broader subject, CWI planning. Most indicators concerning these aspects were rated lower than 3.00, which coincide with a very popular doggerel in China that translates as "planning and planning, paper work for wall hanging," meaning plans were developed not for guiding interventions but for displays. Thus, it becomes quite plain why limited resources were invested in planning (B1c: 1.97); why stakeholders were seldom involved in the process (B2a: 2.79; C1b: 2.77); why practical guidelines and protocols were generally lacking (C2a-d: 2.78–3.06; C3a: 2.69) and other proved principles hardly observed; and why, production and dissemination of documents gained the highest score (B4a: 3.50). As a result, it is logical to see low ratings on objectives and project feasibility and efficacy (B3c: 2.84; B3d: 2.89) and synergetic efforts (C3d: 2.18).
The ratings on the fourth component, resource development, indicate that performance on information exploitation (D4a-d: 3.13–3.86) was relatively better than that on others. This may be due partly to China’s long history of mandatory but formative reporting of infectious diseases and the fact that publications had become a primary criterion in determining promotion of health staff. One thing worth mentioning in particular here is that although these ratings are relatively high, they are still far below the standard set by the experts and much work needs to be done in assuring data quality and use of information.20,21 Another area where certain efforts seem had been made is training of health workers in relation to HIV/STI (D1a: 3.75). This reflects China’s recent strategy for the epidemics that emphasises "total health staff training." However, related ratings such as service quality (E3c: 3.06), project design (C3: 2.42), and reports by others lead to a impression that either the training was of limited efficacy or the trained were not fully used or both.22–25 The remaining ratings regarding this component are rather discouraging, of which fund and material acquirement and utilisation appear to be most problematic. In other words, the respondents believed that fund and materials needed for CWI were neither available (D2: 2.11) nor used wisely (D3: 2.44). The results also indicate that available resources within health systems were better mobilised than those outside health systems (D1a: 3.75 v D1b-d: 2.76–2.96; D3a-b: 2.64–2.91 v D3c: 2.02; p<0.05).
This is perhaps another sign of lack of multisector CWI.
Turning to the fifth component, project and task implementation, it seems that great efforts had been invested in preventing blood borne transmission (E4a: 4.45). This is understandable because a substantial proportion of HIV infections in China were caused by paid plasma collection which has brought worldwide concern.15 Even so, the respondents were still quite cautious about blood safety (E4b: 4.40 or 15% from total safety). Comparatively, the respondents perceived that much less effort was spent in safeguarding other invasive clinical procedures (E4c: 3.17) and hence they were more pessimistic about the safety of these procedures (E4d: 3.41). This is significant since transmission through invasive procedures has been well documented and overuse of these procedures is common in China.26–28 The respondents also thought that various measures had been used to educate the general public (E2a: 4.14) and population coverage by these initiatives was rated 3.71 (E2b). These should certainly be viewed as positive developments. But the low ratings on knowledge (E2c: 3.07) and behaviour changes (E2d: 2.44) may imply limited effects of these education activities. The main reason underlying this lack of efficacy may be poor performance on planning and designing or framing of education activities mentioned above and evaluation to be discussed below. Another problem to be inferred from the ratings (E3a: 2.78, E3b: 2.79) is low accessibility and availability of care and help for the infected. Given the rapid increase in the number of HIV/STI patients in China and the tremendous suffering and consequences of lack of treatment and help, this problem merits ample attention. In addition, discrepancies (p<0.01) seem to exist between the intervention efforts targeted at specific risk groups (E1a-b: 3.35 and 2.71) and the general public (E2a-b: 4.14 and 3.71). The contribution to this phenomenon may come from two sides. On the one hand, risk groups are difficult to reach, and on the other, there may not be enough momentum, as suggested by ratings on functioning mechanisms, from the service provider side to tackle the difficulties; instead they focused primarily on interventions requiring minimum effort but which were of great "face" value.
Regarding the sixth component, intervention evaluation, low ratings similar to CWI planning mentioned above are seen again here, suggesting that evaluation in the respondents’ counties fell far short of optimal. Perhaps the most discouraging findings are the extremely low ratings on utilisation of evaluation results (F4a-d: 2.15–2.54). When there is a real lack of motivation to use evaluation to draw lessons to improve future interventions, it is hard to expect rigorous evaluation planning (F1: 2.82), data collection (F2: 2.95) and analysis (F3: 2.72). Bearing this in mind, it is also not surprising that the relatively better performance on routine reporting (F2b: 3.40) may due more to outside forces rather than being driven by the evaluation itself. In fact, the indicator of routine reporting included in CWI evaluation collapses substantially with HIV/STI registering and reporting (D4b: 3.86), which is a mandatory task stipulated by Chinese state legislation.
In our study, primary data were obtained through subjective ratings. Thus, one major concern may be that the method is prone to biases—for example, participants may have had incentives to report more favourably for political reasons or because of job security concerns. These could be kept to a minimum given the variety of measures taken in ensuring data quality including, as described in the data collection section, careful design of the questionnaire, selection of informed respondents, communication of study purposes, and assurance of confidentiality. Perhaps of even greater concern is whether a single individual can accurately rate CWI since the instrument consists of a variety of indicators ranging from governmental and managerial matters (for example, A1a: CWI steering group has been set up) to quite specific issues applicable to non-governmental organisations (for example, F1d: indicators specified by the evaluation plan are complete, rational, reliable, meaningful, and measurable). Therefore, it is reasonable to think that a single governmental employee may not know to what degree a given indicator has been met without performing extensive background research and completion of the questionnaire may turn out to be an educated guess on the part of the respondent. Again, this problem may not be as serious as it seems. This is because (a) Chinese CDC staff members in charge of overall HIV/STI control are quite different from those in western countries; they are selected from the most experienced local HIV/STI intervention professionals and are responsible for and knowledgeable about both technical and managerial matters of CWI; (b) all NGOs in China are not real NGOs but "governmental NGOs" and NGO based HIV/STI interventions are subject to CDC supervision; therefore, it is typical for a county CWI manager to have visited most of the related organisations within the county every year; (c) all the participants selected in our study had been given the responsibility for overall HIV/STI intervention for over 3 years and reported that they know the actual status referred to in the questionnaire well (27%) or very well (73%). The high inter-rater correlation coefficients also indicate that the ratings are quite reliable. In addition, as indicated in the results section, the findings derived from the ratings seem to be quite consistent with relevant existing research.
As mentioned in the introduction section, while there are numerous reports documenting various evaluations of singular intervention programmes, comprehensive assessment of CWI against HIV/STI epidemic is generally lacking. The approach provided by our study is noteworthy in several senses. Firstly, it helps in coming up with an important lack of methods for HIV/STI intervention evaluation. Secondly, its utilisation requires minimum resources and the method, therefore, is applicable routinely by resource poor communities. Thirdly, the hierarchical structure of the instrument greatly facilitates grasping of evaluation results: the overall index gives a general impression and the first to the third tier indicators provide information about more and more refined subareas helped by proper charting (for example, fig 1), it becomes extremely easy to understand the general situation as well as to identify or locate specific good or bad aspects in a community or communities of concern. Fourthly, and perhaps most importantly, it provides a ready means for translating findings into meaningful actions since the third tier indicators are in fact designed as desirable activities or standards of CWI. For example, the indicator A3b was rated extremely low (1.95) and thus indicates a problem here. By referring to the statement against A3b in the questionnaire, one can easily find that this problem could be corrected by asking all the related leaders and staff to make formal commitment to their intervention duties.
For this reason, the instrument can also be used as a checklist for implementing CWI. Finally, the instrument is especially useful for relevant CDC staff to conduct self assessment of CWI during which the raters are free of confidentiality, and other concerns.
Application of the method revealed important findings concerning CWI in China. Generally speaking, there exists a large gap between actual CWI and standards established by the experts, and responses to HIV/STI are not developing in a synergetic manner with undue efforts being focused on some areas while others are neglected. Major problems underlining these phenomena include lack of political will and policy support, low or inappropriate incentives (for example, overemphasis on "face" value), inadequate intervention planning, poor supervision and evaluation, shortage of funds, and inefficient fund utilisation and ineffectiveness. These problems interact with each other and may form a vicious cycle in which inadequate political and policy support affects input to and incentives of CWI; low or bad incentives prevent rigorous planning, implementation, and evaluation which in turn result in ineffectiveness and inefficiency; and finally low effectiveness and efficiency reinforce political negligence. To break through this cycle, therefore, comprehensive measures are needed to target all these problems simultaneously, with specific emphases on building a strong political will and effective incentives.
In fact, this paper documents the first part of our study that aims at developing two complementary tools—rapid and specific CWI assessment instruments. What we are going to do next is to develop a more detailed inventory by further dividing the third tier indicators into fourth, fifth, and even sixth tier indicators. It will be designed to help communities to conduct in-depth investigations into specific CWI interest areas identified by the rapid method. Indicators of the specific instrument will be segregated into different groups that are applicable to different levels and organisations and its application will involve requesting different respondents to complete the portion specific to their own levels or organisations so that more detailed and reliable data could be obtained though at a higher price.
ACKNOWLEDGEMENTS
This study was supported partly by the World AIDS Foundation, grant WAF 268(01–083) and partly by the Foreign Loan Office, Ministry of Public Health, China. The authors wish to thank Professor Roger Detels at US UCLA and Professor Zuyou Wu and Dr Keming Rou at China CDC for their helpful comments, suggestions, and revision of the manuscript.
CONTRIBUTORS
DBW participated in conceptualising the study, directed implementation of the study, and wrote this manuscript; XJZ was the principal investigator of the WAF grant, participated in development of the study, and made revisions to the manuscript; HBZ and BS had an important role in designing the inventory and data collection; CYZ participated inventory designing and performed data analysis.
FOOTNOTES
Human participant protection: The protocol for this study was approved by the review board of Anhui Ethics Association.
UNAIDS. Comfort and hope: six case studies on mobilizing family and community care for and by people with HIV/AIDS. UNAIDS, 1999. Available at www.unaids.org/html/pub/Publications/IRC-pub01/KC099-Comfort&Hope_en_pdf.htm.
UNAIDS. Community mobilization and AIDS. UNAIDS technical update, April 1997. Available at www.unaids.org./html/pub/Publications/IRC-pub03/CommMob-TV_EN_pdf.thm.
Wu Z, Detels R, Zhang J, et al. Community-based trial to prevent drug use among youths in Yunnan, China. Am J Public Health 2002;92:1952–7.
Wang DB, Zhang XJ, Ye DQ, et al. A systematic framework to HIV/STDs intervention. Chinese J Dis Control 2002;1:59–62.
Van PE. Shared responsibility between all affected by HIV/AIDS: "Comprehensive care across a continuum". AIDS Asia 1995;2:10–15.
Milstein R, Wetterhal S, Christenson G, et al. Framework for program evaluation in public health. US CDC September 1999;48 (RR11) :1–40.
Grassly N, Garnett G, Schwartlander B, et al. The effectiveness of HIV prevention and the epidemiological context. Bull World Health Organ 2001;79:1121–32.
Cowan M, Langhaug F, Mashungupa P, et al. School based HIV prevention in Zimbabwe: feasibility and acceptability of evaluation trials using biological outcomes. AIDS 2002;16:1673–8.
Chen HT. Designing and conducting participatory outcome evaluation of community-based organizations’ HIV prevention programs. AIDS Educ Prev 2002;14 (Suppl A) :18–26.
Stephenson JM. Evaluation of behavioural interventions in HIV/STI prevention. Sex Transm Infect 1999;75:69–71.
Pinkerton SD, Holtgrave DR, Leviton LC, et al. Model-based evaluation of HIV prevention interventions. Eval Rev 1998;22:155–74.
Mertens E, Carael M. Evaluation of HIV/STD prevention, care and support: an update on WHO’s approaches. AIDS Educ Prev 1997;9:133–45.
Rugg D, Buehler J, Renaud M, et al. Evaluating HIV prevention: a framework for national, state, and local levels. Am J Evaluation 1999;20:35–56.
Rehle T, Saidel T, Mills S, et al. Evaluating programs for HIV/AIDS prevention and care in developing countries: a handbook for program managers and decision makers. Family Health Int 2001.
UNAIDS/WHO. China epidemiological fact sheets on HIV/AIDS and sexually transmitted infections, 2002 update. UNAIDS/WHO Epidemiological Fact-sheet, UNAIDS/WHO; 2002. Available at www.unaids.org/html/pub/Publications/Fact-sheets01/China_EN_pdf.htm.
National Intelligence Council. The next wave of HIV/AIDS: Nigeria, Ethiopia, Russia, India, and China. National Intelligence Council, 2002. Available at www.odci.gov/nic.
Delbecq AL, VandeVen AH, Gustafson DH. Group techniques for program planning, a guide to nominal group technique and Delphi processes. Scott Foreman 1975.
Meng XD, Qiou BH, Huang L. Strengthening multi-sector cooperation and enhancing comprehensive intervention competence against HIV/AIDS . Chin J STD/AIDS Prev Treatment 2001; (suppl) :2–4.
Qiou RZ. Effective and ethical approaches to HIV/AIDS containment . Chin J STD/AIDS Prev Treatment 2001; (suppl) :6–11.
Yu J, Liang GJ. Survey of fail-to-reported cases with China national HIV/STI surveillance system . Diseases Surveillance 2000;15:88–90.
Jiao JX, He YF, Liang PF. Quality of STD and infectious disease report card completion . Occupation and Health 2001;17:91.
Shen XW, Hao CG. Problems with and suggestions to county level Anti-epidemic stations in controlling HIV/STI . Chin J STD/AIDS Prev Treatment 2001; (suppl) :13–14.
Wang QQ, Yang P. Survey of application of STD syndromic management by clinicians. Chinese J STD/AIDS Prev Treatment 2001;7:145–7.
Shu XH, Yang P, Zhong MY, et al. Survey of clinicians’ knowledge about HIV/STI in Nanjing City . Chin J Health Education 2001;17:325–9.
Ye LX, Duan QH, Wei SP, et al. HIV/STI diagnosis and treatment quality survey in Wuhan City . Chinese J STD/AIDS Prev Treatment 1999;5:277–9.
Hou Q, Yang YH. Problems with HIV infections and suggestions. The People’s Liberation Army Nursing J 2002;19:45–47.
Du ZZ. Control of HIV/AIDS in Hospitals. Chin J Hosp Infect 2001;11:300.
Gissuelquist DP. Estimating HIV-1 transmission efficiency through unsafe medical injections. Int J STD AIDS 2002;13:152–9.
1 Anhui Medical University, Hefei, China
2 Anhui Provincial Center for Disease Control, Hefei, China(D B Wang, X J Zhang, H B )
Methods: A total of 120 county level respondents in charge of county-wide responses to HIV/STI thoughout China were selected randomly and surveyed using a structured inventory consisting of three tiers of indicators developed via consensus group techniques. The respondents were asked to rate each of the indicators against a five grade (1–5) scale. 30 pairs of the same staff from within Anhui Province were surveyed to gauge inter-rater reliability.
Results: Response rate for the nationwide survey was 85% and for inter-rater reliability survey, 90%. Correlation coefficients between the inter-rater ratings ranged from 0.68 to 0.95. The overall average rating of CWI in China was 2.85. Average ratings for the six first tier indicators, organisation and policy development, goals and objectives setting, project and action planning, resource exploitation, project and task implementation, and CWI evaluation were 2.87, 2.83, 2.67, 2.77, 3.26, and 2.71 respectively. Ratings derived for the 24 second tier indicators ranged from 2.1 to 3.86; while for the 96 third tier indicators, 1.90 to 4.40
Conclusions: The instrument developed proved to be reliable, useful, and easily applicable in common communities. Application of it in China revealed that a large gap exists between desired and actual CWI, and areas meriting particular attention include policy and incentives development, intervention planning and evaluation, and fund raising and utilisation.
Abbreviations: CDC, centre for disease control; CWI, community-wide intervention
Keywords: HIV/STI; community intervention; China
It is becoming clear that effective control of human immunodeficiency virus and sexually transmitted infections (HIV/STI) depends heavily on community-wide intervention (CWI), since the epidemic dynamic is deeply embedded in individual desires, social and cultural relationships, as well as environmental and economic processes.1–3 The essence of CWI against HIV/STI is to maximise outcomes by mobilising the whole community and using comprehensive approaches targeting different groups and factors via different means and channels in a synergetic manner.1,4,5 To reach this goal, we must continuously monitor and evaluate intervention activities and impacts.6 Although there is a substantial literature on evaluation of research trials or demonstration programmes against HIV/STI,7–11 reports on routine responses to the epidemic in general communities at large from a holistic perspective (or CWI) are generally lacking.
Evaluation of CWI can be performed differently, ranging from simply asking a few broad questions to conducting exhaustive investigations involving multiple methods, multiple targets, multiple funding sources, multiple perspectives, multiple paradigms, multiple roles, and multiple solutions to multiple problems.12,13 While Rehle and colleagues’ handbook provides useful guidance and tools in implementing,14 an exhaustive approach to CWI evaluation is seldom feasible or necessary. What are needed mostly are practical approaches that require minimum resources and expertise and are, therefore, more appropriate for use by resource poor communities.
China is witnessing a rapid increase in HIV/STI cases.15,16 Whether can China reverse this trend depends largely on rigorous and coordinated responses of the communities that make up the country. Unfortunately, there has been no systematic evaluation of the status quo of CWI in China. Meanwhile, anecdotal accounts are disturbing: inadequate leadership limits multisector involvement and concordant intervention; conflict of interests has prevented many proved programmes since, for example, syndromic management of STDs could cause a service provider to lose up to two thirds of his revenue for the same number of STD patients served; programmes suffer from missing links between job performance and reward; and ignorance of planning and evaluation results in contradictory efforts and futile programmes, etc. This study aims at developing a rapid yet comprehensive evaluation method and using it to assess current CWI in China.
Inventory development
We implemented three steps in developing an instrument for CWI assessment. Firstly, a draft was produced after extensive publication review. Then, a nominal group technique was applied three times to refine the draft involving a panel of seven experts on CWI against HIV/STI, health services management, health education and behaviour modification, epidemiology, and health statistics.17 Finally, the inventory generated was piloted and revision made according to feedback solicited. Underlying the instrument development was a hierarchical framework that divided CWI tasks into distinctive and equally important subareas level by level until a balance had been reached between manageability and specificity of the divisions. As a result, the final instrument consists of three tiers of indicators. The six first tier indicators represent overall components (A–F), including organisation and policy development, goal and objectives setting, project and action planning, resource exploitation, project or task implementation, and intervention monitoring and evaluation. Similarly, the 24 second tier and the 96 third tier indicators are components of the first and second tier indicators, respectively. All the 96 third tier indicators were phrased as succinct statements, each describes a standard for a specific aspect of CWI (see appendix on STI website, www.stijournal.com). Each statement has a five degree rating scale (see table 1). At the end of the inventory, there is a structured question about the respondent’s knowledge about the actual status inquired about. Completion of the inventory requires about 30 minutes.
Data collection
A "community" in this study was defined as a county or county level city (hereafter referred to as county). In China, counties are administrative subunits below provinces or cities, defined by a whole range of factors including population size, geographic boundaries, and sociocultural traditions. They are the most meaningful units to launch a CWI against HIV/STI and have actually been assigned the task to do so. For this study, 120 counties were selected randomly from the list of over 2000 counties that make up mainland China using random number table. One staff member from each centre for disease control (CDC) of the sampled counties was selected as the respondent. The selection was aimed at identifying the best informant of local CWI from the CDCs. Thus, preset criteria were used to prioritise relevant staff members (2–6 people per CDC) including years in charge of local CWI, professional title, and years of relevant education and training. The one member with top ranking was sent, in April 2003, the questionnaire developed above by mail and asked to rate actual interventions in their local counties according to stated standards.
A covering letter was also sent to the respondents along with the questionnaire, which clearly states that: (a) the survey is purely for research purposes and it is of enough importance to report as objectively as possible; (b) data collected will be kept in a safe place with total confidentiality; (c) only aggregate statistics will be published, no county specific data or comparisons will be disclosed, and the research team assures no harm to any respondent and his or her organisation or county as a result of the survey. In addition, each respondent was sent an addressed and stamped envelope to facilitate the return of the completed questionnaire. Telephone recalls were made to those who failed to respond within 1 month.
In order to gauge inter-rater reliability, 30 pairs of staff (two members from the same CDC) in Anhui Province were selected and surveyed using the same method.
Data process and analysis
SPSS 10.0 was used to process and analyse the data. Ratings for the third tier indicators by a specific respondent were derived directly by translating the responses of "" "" "" "" and "" into numbers 1, 2, 3, 4, and 5 respectively, while the ratings for the first and second tier indicators were calculated as averages of the components (that is, the second or third tier indicators). Similarly, the overall rating of CWI was generated by adding all the six first tier ratings and then dividing the sum by 6. Means and standard deviations of the ratings against all the indicators were calculated. Bivariate correlation analysis was performed to measure inter-rater reliability. Comparisons of means between ratings of different indicators were also made when necessary using the "one way ANOVA" procedure.
From the 30 pairs of staff members in Anhui Province surveyed for inter-rater reliability, 27 pairs of valid questionnaires were returned; from the nationwide survey, 102 valid questionnaires were collected, adding up to a response rate of 85%. Correlation coefficients between the ratings of the same indicators given by the paired staff members were high, ranging from 0.68 to 0.95. Means and standard deviations of all the ratings derived from the nationwide survey are given in table 2, while figure 1 depicts all the ratings on a set of radar diagrams. The overall average rating was 2.85. This is quite low since the maximum theoretical rating for any given indicator is 5. Average ratings for the six first tier indicators—organisation and policy development, goals and objectives setting, project and action planning, resource exploitation, project and task implementation, and CWI evaluation— were 2.87, 2.83, 2.67, 2.77, 3.26, and 2.71 respectively. The difference between ratings of most indicators is statistically significant (p<0.05).
More specifically, ratings concerning the first component of CWI, organisation and policy development, indicate that relatively greater progress had been made in leadership and management organisation development (A1a: 3.60; A1c: 3.61) and responsibility definition (A2a: 3.46; A2c: 3.31; A2d: 3.77). At first glance, these are encouraging signs, for management structure has an important role in CWI.4 However, the low ratings of the mechanisms for bringing the structure into play (A3a-d: 1.95–2.92) suggest that these organisational and responsibilities developments may be more symbolic than functional. This point is also supported by the significantly lower rating on the establishment of intervention network (which is much more difficult) than that of steward organisations (A1d: 2.85 v A1a: 3.60 and A1c: 3.61; p<0.05), for it indicates that organisations were developed upon easiness to build rather than real need.
Another point worth noting is that the respondents gave rather low ratings to CWI related policy development (A4a-d: 2.33–3.19). This implies again that the steward structures were not functioning well and is consistent with other reports that HIV/STI was not adequately recognised by local governments, that the infected individuals were often stigmatised and denied care, schooling and jobs, and that there were various policy barriers to proved interventions like methadone substitution, needle and syringe exchange for injecting drug users, and condom promotion among sex workers.18,19
The second component, goal and objective setting, and the third, project and action plan, are all integral parts of a broader subject, CWI planning. Most indicators concerning these aspects were rated lower than 3.00, which coincide with a very popular doggerel in China that translates as "planning and planning, paper work for wall hanging," meaning plans were developed not for guiding interventions but for displays. Thus, it becomes quite plain why limited resources were invested in planning (B1c: 1.97); why stakeholders were seldom involved in the process (B2a: 2.79; C1b: 2.77); why practical guidelines and protocols were generally lacking (C2a-d: 2.78–3.06; C3a: 2.69) and other proved principles hardly observed; and why, production and dissemination of documents gained the highest score (B4a: 3.50). As a result, it is logical to see low ratings on objectives and project feasibility and efficacy (B3c: 2.84; B3d: 2.89) and synergetic efforts (C3d: 2.18).
The ratings on the fourth component, resource development, indicate that performance on information exploitation (D4a-d: 3.13–3.86) was relatively better than that on others. This may be due partly to China’s long history of mandatory but formative reporting of infectious diseases and the fact that publications had become a primary criterion in determining promotion of health staff. One thing worth mentioning in particular here is that although these ratings are relatively high, they are still far below the standard set by the experts and much work needs to be done in assuring data quality and use of information.20,21 Another area where certain efforts seem had been made is training of health workers in relation to HIV/STI (D1a: 3.75). This reflects China’s recent strategy for the epidemics that emphasises "total health staff training." However, related ratings such as service quality (E3c: 3.06), project design (C3: 2.42), and reports by others lead to a impression that either the training was of limited efficacy or the trained were not fully used or both.22–25 The remaining ratings regarding this component are rather discouraging, of which fund and material acquirement and utilisation appear to be most problematic. In other words, the respondents believed that fund and materials needed for CWI were neither available (D2: 2.11) nor used wisely (D3: 2.44). The results also indicate that available resources within health systems were better mobilised than those outside health systems (D1a: 3.75 v D1b-d: 2.76–2.96; D3a-b: 2.64–2.91 v D3c: 2.02; p<0.05).
This is perhaps another sign of lack of multisector CWI.
Turning to the fifth component, project and task implementation, it seems that great efforts had been invested in preventing blood borne transmission (E4a: 4.45). This is understandable because a substantial proportion of HIV infections in China were caused by paid plasma collection which has brought worldwide concern.15 Even so, the respondents were still quite cautious about blood safety (E4b: 4.40 or 15% from total safety). Comparatively, the respondents perceived that much less effort was spent in safeguarding other invasive clinical procedures (E4c: 3.17) and hence they were more pessimistic about the safety of these procedures (E4d: 3.41). This is significant since transmission through invasive procedures has been well documented and overuse of these procedures is common in China.26–28 The respondents also thought that various measures had been used to educate the general public (E2a: 4.14) and population coverage by these initiatives was rated 3.71 (E2b). These should certainly be viewed as positive developments. But the low ratings on knowledge (E2c: 3.07) and behaviour changes (E2d: 2.44) may imply limited effects of these education activities. The main reason underlying this lack of efficacy may be poor performance on planning and designing or framing of education activities mentioned above and evaluation to be discussed below. Another problem to be inferred from the ratings (E3a: 2.78, E3b: 2.79) is low accessibility and availability of care and help for the infected. Given the rapid increase in the number of HIV/STI patients in China and the tremendous suffering and consequences of lack of treatment and help, this problem merits ample attention. In addition, discrepancies (p<0.01) seem to exist between the intervention efforts targeted at specific risk groups (E1a-b: 3.35 and 2.71) and the general public (E2a-b: 4.14 and 3.71). The contribution to this phenomenon may come from two sides. On the one hand, risk groups are difficult to reach, and on the other, there may not be enough momentum, as suggested by ratings on functioning mechanisms, from the service provider side to tackle the difficulties; instead they focused primarily on interventions requiring minimum effort but which were of great "face" value.
Regarding the sixth component, intervention evaluation, low ratings similar to CWI planning mentioned above are seen again here, suggesting that evaluation in the respondents’ counties fell far short of optimal. Perhaps the most discouraging findings are the extremely low ratings on utilisation of evaluation results (F4a-d: 2.15–2.54). When there is a real lack of motivation to use evaluation to draw lessons to improve future interventions, it is hard to expect rigorous evaluation planning (F1: 2.82), data collection (F2: 2.95) and analysis (F3: 2.72). Bearing this in mind, it is also not surprising that the relatively better performance on routine reporting (F2b: 3.40) may due more to outside forces rather than being driven by the evaluation itself. In fact, the indicator of routine reporting included in CWI evaluation collapses substantially with HIV/STI registering and reporting (D4b: 3.86), which is a mandatory task stipulated by Chinese state legislation.
In our study, primary data were obtained through subjective ratings. Thus, one major concern may be that the method is prone to biases—for example, participants may have had incentives to report more favourably for political reasons or because of job security concerns. These could be kept to a minimum given the variety of measures taken in ensuring data quality including, as described in the data collection section, careful design of the questionnaire, selection of informed respondents, communication of study purposes, and assurance of confidentiality. Perhaps of even greater concern is whether a single individual can accurately rate CWI since the instrument consists of a variety of indicators ranging from governmental and managerial matters (for example, A1a: CWI steering group has been set up) to quite specific issues applicable to non-governmental organisations (for example, F1d: indicators specified by the evaluation plan are complete, rational, reliable, meaningful, and measurable). Therefore, it is reasonable to think that a single governmental employee may not know to what degree a given indicator has been met without performing extensive background research and completion of the questionnaire may turn out to be an educated guess on the part of the respondent. Again, this problem may not be as serious as it seems. This is because (a) Chinese CDC staff members in charge of overall HIV/STI control are quite different from those in western countries; they are selected from the most experienced local HIV/STI intervention professionals and are responsible for and knowledgeable about both technical and managerial matters of CWI; (b) all NGOs in China are not real NGOs but "governmental NGOs" and NGO based HIV/STI interventions are subject to CDC supervision; therefore, it is typical for a county CWI manager to have visited most of the related organisations within the county every year; (c) all the participants selected in our study had been given the responsibility for overall HIV/STI intervention for over 3 years and reported that they know the actual status referred to in the questionnaire well (27%) or very well (73%). The high inter-rater correlation coefficients also indicate that the ratings are quite reliable. In addition, as indicated in the results section, the findings derived from the ratings seem to be quite consistent with relevant existing research.
As mentioned in the introduction section, while there are numerous reports documenting various evaluations of singular intervention programmes, comprehensive assessment of CWI against HIV/STI epidemic is generally lacking. The approach provided by our study is noteworthy in several senses. Firstly, it helps in coming up with an important lack of methods for HIV/STI intervention evaluation. Secondly, its utilisation requires minimum resources and the method, therefore, is applicable routinely by resource poor communities. Thirdly, the hierarchical structure of the instrument greatly facilitates grasping of evaluation results: the overall index gives a general impression and the first to the third tier indicators provide information about more and more refined subareas helped by proper charting (for example, fig 1), it becomes extremely easy to understand the general situation as well as to identify or locate specific good or bad aspects in a community or communities of concern. Fourthly, and perhaps most importantly, it provides a ready means for translating findings into meaningful actions since the third tier indicators are in fact designed as desirable activities or standards of CWI. For example, the indicator A3b was rated extremely low (1.95) and thus indicates a problem here. By referring to the statement against A3b in the questionnaire, one can easily find that this problem could be corrected by asking all the related leaders and staff to make formal commitment to their intervention duties.
For this reason, the instrument can also be used as a checklist for implementing CWI. Finally, the instrument is especially useful for relevant CDC staff to conduct self assessment of CWI during which the raters are free of confidentiality, and other concerns.
Application of the method revealed important findings concerning CWI in China. Generally speaking, there exists a large gap between actual CWI and standards established by the experts, and responses to HIV/STI are not developing in a synergetic manner with undue efforts being focused on some areas while others are neglected. Major problems underlining these phenomena include lack of political will and policy support, low or inappropriate incentives (for example, overemphasis on "face" value), inadequate intervention planning, poor supervision and evaluation, shortage of funds, and inefficient fund utilisation and ineffectiveness. These problems interact with each other and may form a vicious cycle in which inadequate political and policy support affects input to and incentives of CWI; low or bad incentives prevent rigorous planning, implementation, and evaluation which in turn result in ineffectiveness and inefficiency; and finally low effectiveness and efficiency reinforce political negligence. To break through this cycle, therefore, comprehensive measures are needed to target all these problems simultaneously, with specific emphases on building a strong political will and effective incentives.
In fact, this paper documents the first part of our study that aims at developing two complementary tools—rapid and specific CWI assessment instruments. What we are going to do next is to develop a more detailed inventory by further dividing the third tier indicators into fourth, fifth, and even sixth tier indicators. It will be designed to help communities to conduct in-depth investigations into specific CWI interest areas identified by the rapid method. Indicators of the specific instrument will be segregated into different groups that are applicable to different levels and organisations and its application will involve requesting different respondents to complete the portion specific to their own levels or organisations so that more detailed and reliable data could be obtained though at a higher price.
ACKNOWLEDGEMENTS
This study was supported partly by the World AIDS Foundation, grant WAF 268(01–083) and partly by the Foreign Loan Office, Ministry of Public Health, China. The authors wish to thank Professor Roger Detels at US UCLA and Professor Zuyou Wu and Dr Keming Rou at China CDC for their helpful comments, suggestions, and revision of the manuscript.
CONTRIBUTORS
DBW participated in conceptualising the study, directed implementation of the study, and wrote this manuscript; XJZ was the principal investigator of the WAF grant, participated in development of the study, and made revisions to the manuscript; HBZ and BS had an important role in designing the inventory and data collection; CYZ participated inventory designing and performed data analysis.
FOOTNOTES
Human participant protection: The protocol for this study was approved by the review board of Anhui Ethics Association.
UNAIDS. Comfort and hope: six case studies on mobilizing family and community care for and by people with HIV/AIDS. UNAIDS, 1999. Available at www.unaids.org/html/pub/Publications/IRC-pub01/KC099-Comfort&Hope_en_pdf.htm.
UNAIDS. Community mobilization and AIDS. UNAIDS technical update, April 1997. Available at www.unaids.org./html/pub/Publications/IRC-pub03/CommMob-TV_EN_pdf.thm.
Wu Z, Detels R, Zhang J, et al. Community-based trial to prevent drug use among youths in Yunnan, China. Am J Public Health 2002;92:1952–7.
Wang DB, Zhang XJ, Ye DQ, et al. A systematic framework to HIV/STDs intervention. Chinese J Dis Control 2002;1:59–62.
Van PE. Shared responsibility between all affected by HIV/AIDS: "Comprehensive care across a continuum". AIDS Asia 1995;2:10–15.
Milstein R, Wetterhal S, Christenson G, et al. Framework for program evaluation in public health. US CDC September 1999;48 (RR11) :1–40.
Grassly N, Garnett G, Schwartlander B, et al. The effectiveness of HIV prevention and the epidemiological context. Bull World Health Organ 2001;79:1121–32.
Cowan M, Langhaug F, Mashungupa P, et al. School based HIV prevention in Zimbabwe: feasibility and acceptability of evaluation trials using biological outcomes. AIDS 2002;16:1673–8.
Chen HT. Designing and conducting participatory outcome evaluation of community-based organizations’ HIV prevention programs. AIDS Educ Prev 2002;14 (Suppl A) :18–26.
Stephenson JM. Evaluation of behavioural interventions in HIV/STI prevention. Sex Transm Infect 1999;75:69–71.
Pinkerton SD, Holtgrave DR, Leviton LC, et al. Model-based evaluation of HIV prevention interventions. Eval Rev 1998;22:155–74.
Mertens E, Carael M. Evaluation of HIV/STD prevention, care and support: an update on WHO’s approaches. AIDS Educ Prev 1997;9:133–45.
Rugg D, Buehler J, Renaud M, et al. Evaluating HIV prevention: a framework for national, state, and local levels. Am J Evaluation 1999;20:35–56.
Rehle T, Saidel T, Mills S, et al. Evaluating programs for HIV/AIDS prevention and care in developing countries: a handbook for program managers and decision makers. Family Health Int 2001.
UNAIDS/WHO. China epidemiological fact sheets on HIV/AIDS and sexually transmitted infections, 2002 update. UNAIDS/WHO Epidemiological Fact-sheet, UNAIDS/WHO; 2002. Available at www.unaids.org/html/pub/Publications/Fact-sheets01/China_EN_pdf.htm.
National Intelligence Council. The next wave of HIV/AIDS: Nigeria, Ethiopia, Russia, India, and China. National Intelligence Council, 2002. Available at www.odci.gov/nic.
Delbecq AL, VandeVen AH, Gustafson DH. Group techniques for program planning, a guide to nominal group technique and Delphi processes. Scott Foreman 1975.
Meng XD, Qiou BH, Huang L. Strengthening multi-sector cooperation and enhancing comprehensive intervention competence against HIV/AIDS . Chin J STD/AIDS Prev Treatment 2001; (suppl) :2–4.
Qiou RZ. Effective and ethical approaches to HIV/AIDS containment . Chin J STD/AIDS Prev Treatment 2001; (suppl) :6–11.
Yu J, Liang GJ. Survey of fail-to-reported cases with China national HIV/STI surveillance system . Diseases Surveillance 2000;15:88–90.
Jiao JX, He YF, Liang PF. Quality of STD and infectious disease report card completion . Occupation and Health 2001;17:91.
Shen XW, Hao CG. Problems with and suggestions to county level Anti-epidemic stations in controlling HIV/STI . Chin J STD/AIDS Prev Treatment 2001; (suppl) :13–14.
Wang QQ, Yang P. Survey of application of STD syndromic management by clinicians. Chinese J STD/AIDS Prev Treatment 2001;7:145–7.
Shu XH, Yang P, Zhong MY, et al. Survey of clinicians’ knowledge about HIV/STI in Nanjing City . Chin J Health Education 2001;17:325–9.
Ye LX, Duan QH, Wei SP, et al. HIV/STI diagnosis and treatment quality survey in Wuhan City . Chinese J STD/AIDS Prev Treatment 1999;5:277–9.
Hou Q, Yang YH. Problems with HIV infections and suggestions. The People’s Liberation Army Nursing J 2002;19:45–47.
Du ZZ. Control of HIV/AIDS in Hospitals. Chin J Hosp Infect 2001;11:300.
Gissuelquist DP. Estimating HIV-1 transmission efficiency through unsafe medical injections. Int J STD AIDS 2002;13:152–9.
1 Anhui Medical University, Hefei, China
2 Anhui Provincial Center for Disease Control, Hefei, China(D B Wang, X J Zhang, H B )