Chair (Stream 1): Philippa Middleton (Australasian Cochrane Centre)
Chair (Stream 2): Brendan Kearney (Chair, Australasian Cochrane Centre Steering Committee)
1948 and all that. How far back should we look for RCTs and CCTs?
Author: Mike Clarke
Background: The MRC Streptomycin trial which was published in the British Medical Journal in 1948 has gained a reputation as the first randomised controlled trial. However, although it was a well conducted and described study, it was not the earliest to use a random or quasi-random process to allocate patients to different treatments.
Objective: To demonstrate the need to look back earlier than 1948 if as complete a set as possible of randomised controlled trials (RCTs) and controlled clinical trials (CCTs) is to be obtained. Materials: Sporadic examples of randomised and quasi-randomised trials from before the streptomycin trial are often cited, and re-cited, in discussions on the history of controlled trials. Among these are a study of pulmonary tuberculosis which was published in 1931 and used the flip of a coin to allocate patient groups to sanocrysin or placebo. There were also a number of trials that used quasi-random methods, such as alternate allocation, around this time. A random sample of BMJ and Lancet volumes from the 1930s and 1940s were searched to investigate the relative rarity of such examples.
Results: Approximately 1-3 reports of RCTs and CCTs were found in each sixmonth volume. These were predominantly CCTs, such as two trials that were printed back-to-back in the 11 December 1937 issue of the BMJ. Some pre-1948 RCTs have also been found, including two on breast-feeding in the Lancet in the second half of 1947.
Conclusions: Although there are many fewer reports of RCTs and CCTs in the literature of half a century ago in comparison to now - perhaps a few per year in the BMJ or Lancet compared to about 100 in each today - such reports do exist. Thus, the determining factor in how far back to look should be the yield of relevant studies in the journal being searched, rather than a cut-off date derived from a trial which was neither the first to be randomised, nor the first to be controlled.
Using the capture-recapture technique in systematic literature searches
Authors: C Mark Airey, Pat Spoor, Cathy Bennett, Rhys Williams
Background: In common with other groups within the Cochrane Collaboration, the Diabetes Review Group is developing a specialised register of randomised controlled trials (RCTs) compiled by searching electronic databases and through systematic hand searching of the literature. The effectiveness of these combined methods in identifying all RCTs, and thus the completeness of the register, can be tested using a technique which examines the degree of overlap between these methods of ascertainment and uses a simple formula to estimate the size of the true population of trials.
Objective: To estimate the true number of RCTs reported in a given body of literature using the "capture-recapture" method and estimate the proportion missed by the combination of MEDLINE and hand searching. Methodology: MEDLINE was searched to identify RCTs published over a 10 year period in the journal Diabetic Medicine. The journal was also hand searched for RCTs. The size of the "total population" of publications dealing with RCTs was calculated using the "capture-recapture" formula and the number of RCTs unidentified by either method was calculated.
Results: 115 RCTs were identified by both methods, 35 by the MEDLINE search alone, and 8 only by handsearching. An estimated 2.4 RCTs were missed by both.
Conclusions: The use of "capture-recapture" methodology demonstrates the effectiveness of the Diabetes Groups search strategy and provides an estimate of the completeness of its specialised register of RCTs.
Publication of clinical studies in German general health care journals
Authors: Gerd Antes, Alric Rüther
Background: A necessary starting point for unbiased reviews is identification of all the potentially relevant RCTs, published and unpublished. The search for RCTs is organized and done by review groups as a crucial step in the preparation of a review. A large group of journals which may be overlooked by specific review groups consists of the general health care journals. For this reason a project (funded by the EU program Biomed1) was launched to organize the handsearch of these journals in Europe. After receiving considerable support by the German Ministry of Health the originally planned work could be extended and allowed to analyse the results of the handsearch.
Objectives: Present and examine the results of handsearching German language general health care journals. Consider characteristics and quality of published studies and of articles.
Methods: General health care journals were handsearched by students and scientists. For the quality assessment different scoring systems were applied.
Results: The handsearching produced a large amount of identified articles eligible for inclusion into the database of RCTs. Quantity and quality of articles and studies show big changes over time, in particular a decrease of published studies during the last years.
Conclusions: The German general health care journals proved to be a rich source of controlled and randomized clinical trials. This indicates that German specialised journals should also be handsearched by review groups.
US Patent documents on the Internet: an unexplored source of unpublished clinical trials.
Author: Jacqueline C Wootton
Background: The US patent literature is a valuable source of information and research data relevant to the Cochrane Collaboration. Many patent publications contain detailed historical reviews, case reports, clinical trials, pilot studies and basic biological research of relevance to meta-analytical studies.
Objectives: To describe:
Results: Examples of searches and unpublished clinical trials will be demonstrated.
Conclusions: Much of the scientific, efficacy and safety data necessary to obtain a patent is unpublished elsewhere. The US Patents literature is a potential valuable resource for the Cochrane Collaboration, as yet not fully explored.
How many redundant publications does it take to screw in a light bulb?
Authors: Patricia Huston, David Moher
Background: Although those who conduct systematic reviews are adept at identifying weakness in trials and have been successful in promoting more rigorous research design they have been surprisingly blasé about shoddy reporting practices such as redundant publication.
Objective: To expose and denounce a nonobvious form of redundant publication that occurs when researchers from one or more centre(s) participating in a multicentre trial publish their results before the complete trial is reported.
Method: A case study of two multicentre trials done on the new antipsychotic agent, risperidone.
Results: Over a dozen publications or reports describing these two trials in part and in full were found, often with no reference to the larger study and with different authors noted for each version.
Discussion: This form of redundant publication that we have dubbed "disaggregation" is a breach of trust implicit to scientific publishing and can result in inadvertent double-counting of data and an overestimation of the consistency of research results.
Conclusion: Denouncing practices such as disaggregation is not simply encouraging "publication etiquette": it is the promotion of ethical, inclusive and transparent research. Researchers, reviewers, editors and an informed readership all have a role to play in preventing the scientific literature from being used as a covert form of advertising.
Evaluation of the efficacy of a prospective clinical trials register: the ISTH register
Authors: Joseph Berki, Nadine Bossard, Jean-Pierre Boissel, Margaret Haugh, The French Cochrane Centre, France
Background: The ISTH (International Society on Thrombosis and Haemostatis) prospective multicentre clinical trials register was first published in 1975, and is the oldest register of this type in existence. The efficacy of this register has never been assessed.
Objective: To assess the efficacy of a prospective clinical trials register from 1974 to 1989.
Methods: The evaluation was limited to trials assessing thrombolytic therapy in acute myocardial infarction. Since there is a lapse between the planing phase and publication of a clinical trial, the literature handsearching and electronic searching was done from 1974 to 1993. The INDEX MEDICUS, and MEDLINE were used for direct searching, and indirect searching was performed using the references cited in trial reports identified. The efficacy of the register was defined as the number of trials registered as a percentage of the published trials identified.
Results: The INDEX MEDICUS search identified 113 multicentre clinical trials assessing thrombolytic therapy in acute myocardial infarction, of which only 43 (38%) had been registered in the ISTH register. The MEDLINE search identified 78 trials of which 32 (41%) had been registered in the ISTH register. The indirect searching identified 90 trials of which 34 (38%) had been registered. The combination of the three methods identified 156 trials of which 58 (37%) had been registered. 14 trials registered in the ISTH register were not found by any search technique. The 11 investigators responsible for these trials were contacted by letter and we found out that two trials had never started. Information for 3 other trials was unclear, the investigators for 3 other trials had moved (letters returned) and no news was obtained about the other letters.
Conclusions: The ISTH register is far from exhaustive. Many explanations exist for this, one important one being the fact that registration is on a voluntary-basis, and while many investigators are happy to supply information about planned and on-going clinical trials, this is not true for all. This situation will only improve if registration becomes mandatory before clinical trials can be initiated.
Indexing the Cochrane Systematic Reviews
Authors: Michael Zacks, William Hersh
Background: The optimal approach to indexing the Cochrane Collaboration reviews to facilitate retrieval is unknown. Current indexing vocabularies may inadequately cover the reviews' concepts. Furthermore, lists of terms may not capture their context.
Objectives: To determine whether current indexing vocabularies cover the reviews' content and whether their context can be represented by an indexing schema of semantic relationships.
Methods: All Systematic Reviews in Issue 2 of the Cochrane Database were included in the study. Each review was indexed by concepts such as diseases, therapies, tests, agents, outcomes or processes of care. A list of relationships between concepts was developed, and one or more term-relationship-term tuples were assigned. When possible, exact MeSH terms were used. If no exact MeSH term existed, exact matches from other standardized vocabularies were sought. If unavailable, a similar or less specific MeSH term was sought. If neither existed, a similar or less specific non-MeSH term was chosen.
Results: Of the 285 articles reviewed, 6 (2%) contained insufficient information for indexing. 70% of the indexed articles could be represented by one term-relationship-term tuple, while the remainder required more. Of the 608 indexing terms, 57% had exact, 6% similar, and 31% less specific MeSH matches. 4% had exact, 0% similar, and 2% less specific non-MeSH matches.
Conclusions: An indexing schema of semantic relationships using standardized medical vocabularies represented nearly all of the current reviews. While only 57% of concepts had exact MeSH matches, more general terms existed for nearly all the rest. Future investigation is required to determine the significance of insufficiently detailed indexing terms and whether the semantic relationship approach permits effective searching.
Meta-analyses from different sources of information
Authors: Alessandro Liberati, Roberto D'Amico, Valter Torri, Angelo Tinazzi, Cinzia Leonetti, Silvia Pifferi
Background: Individual patient data metaanalysis has been described as the gold standard for systematic reviews of the effects of health care.
Objectives: To compare results of metaanalyses based on three different sources of information: a) aggregate data extracted from published reports (ADP); b) aggregate data supplied by study investigators (ADS); c) individual patient data supplied by study investigators (IPD).
Methods: RCTs on Selective Decontamination of the Digestive Tract (SDD) - a form of antibiotic prophylaxis for patients in intensive care units - have been searched and identified starting 1990. A total of 25 RCTs, totalling 4310 patients, was identified. For this analysis 17/25 RCTs for which all the three sources were available have been used to independently calculate the effect of SDD on the odds of developing Respiratory Tract Infection (RTI) and mortality, using the fixed effect model.
Results: The three meta-analyses are summarized below for RTI (panel A) and mortality (panel 13).
(panel A) Source Events/Pts OR 95%CI ADP 702/3142 0.45 0.38-0.53 ADS 694/3377 0.41 0.35-0.49 IPD 638/3263 0.41 0.34-0.49 (panel B) Source Events/Pts OR 95%CI ADP 762/3142 0.87 0.74-1.03 ADS 975/3564 0.92 0.80-1.08 IPD 829/3357 0.89 0.76-1.05
Conclusions: Despite its advantages, the amount of resources required by IPM and its dependence on time and willingness of investigators to collaborate should not be overlooked. While exclusions after randomization emerged as a limiting factor to the use of meta-analyses based on ADP, ADS can be considered a valid and cheaper alternative to IPD, at least when categorical data are used and time dependent censoring is not relevant.
A comparison of effect estimates from meta-analysis of published data and effect estimates from pooled patient data
Authors: Karen Steinberg, Jay Smith, Donna Stroup,
Background: Because of diminishing resources for new studies that could take years to complete, and the large number of interventions and treatments that need to be evaluated, quantitative methods for combining data will be needed to fully utilize results from existing observational studies. Quantitative methods for combining existing research results include reviews that combine original individual patient data and reviews that combine published and unpublished summary results.
Objectives: To determine the relative merits of two quantitative methods used to estimate summary effects of observational studies by comparing results from a published pooled analysis to those from an original meta-analysis.
Methods: We compared a published pooled analysis that included an analysis of the relationship of oral contraceptive (OC) use and risk for epithelial ovarian cancer to an original meta-analysis of the studies used in the published pooled analysis. We calculated a dose-response slope from a linear model for the effects of duration of OC use on the log relative risk of ovarian cancer, again weighting each risk estimate for a given duration by the inverse of the variance of that risk. We tested for heterogeneity and performed subgroup analyses. We calculated estimates of the cost of both procedures.
Results: We found excellent quantitative agreement between the summary effect from the meta-analysis and the pooled analysis. A protective effect was apparent for studies with hospital control subjects (OR = 0.62 [0.39-0.99]) and for studies with community control subjects (OR = 0.55 [0.42-0.70]). A planning grant ($23,000) and 2 year analytical grant ($235,000) combined to show a cost of $259,300 for the pooled analysis, approximately 5 times that of the meta-analysis.
Conclusions: We conclude that pooled analysis is preferred under the following circumstances: 1) when relationships among outcomes, exposures, and confounders that are not reported by the original investigators are of interest, 2) the original summary parameters differ and cannot be combined, or 3) when the issue being investigated is new and only a few studies are available. We conclude that meta-analysis is appropriate in the following circumstances:
In public health and epidemiology, there may be fewer opportunities to apply pooled analysis than to apply meta-analysis because data from original studies for use in pooled analysis may be accessible only to limited numbers of research groups.
Including literature based data in individual patient data meta-analyses
Authors: Laurence Collette, Stefan Suciu, Luc Bijnens and Richard Sylvester.
Background: Individual patient data meta-analyses are nowadays the golden standard for conducting meta-analyses. When individual patient data cannot be retrieved for a trial, omitting the trial from the analysis should be avoided.
Objectives: This paper investigates the possibility of including in an individual patient data meta-analysis the results of trials for which only published results are available.
Methods: Assuming the sample size of each treatment arm (N1, N2), the observed number of events per treatment arm (O1, O2) and the value of the logrank test statistic [LR=(O1-E1)˛/ Var(O1-E1)], or even only its associated p-value are available in the trial's publication, the following estimates of the quantity Var(O1-E1) are considered:
1) (O1+O2)/4
2) (O1+O2)N1N2/(N1+N2)˛
3) O1O2/(O1+O2)
The estimated value of |O1-E1| is SqrRoot(LR x V) where V is one of the above variance estimates.
Results: Based on simulations using these approximate estimates of the variance and the logrank statistic for each trial, it has been found that reliable estimates of the common hazard ratio and of the overall logrank p-value are obtained. These estimates of the common hazard ratio and of the overall logrank chi-square test statistic have the same limitations as the individual patient data estimates of the same quantities with regards to the proportional hazards assumption, to differences in follow-up between the treatment arms and to the presence of heterogeneous treatment effects.
Conclusions: When other biases can be considered to be negligible, this method provides a reliable way of introducing literature based data in an individual patient data meta-analysis. Although these additional data contribute to the estimate of the treatment effect, they cannot be used in more refined analyses, for instance computation of overall Kaplan-Meier curve and subgroup analyses.
Specifics problems encountered in a meta-analysis of homeopathic medicine.
Authors: Michel Cucherat, Jean-Pierre Boissel, Margaret Haugh, on behalf the HMRAG group.
Background: The homeopathic medicine (defined here as the use of high dilutions of active principles) is widely used in several European countries. The mechanism of action remains unclear and the reality of the efficacy of these treatments is controversial. On the initiative of the European Parliament a quantitative systematic overview of the evidence on the clinical efficacy of the homeopathic medicine has been undertaken.
Objective: To assess the efficacy of homeopathy versus placebo or no treatment.
Methods: The randomised clinical trials testing homeopathic medicine versus placebo or no treatment were identified through several ways: electronic searching on bibliographic databases such as Medline, Embase; handsearching in homeopathic journals; contacts with experts and pharmaceutical firms. Reports in all language were considered. Reading and data extraction was performed by several readers using a specifically designed summary form. Each paper was read twice by two independent readers.
The statistical analysis for this meta-analysis raised very specific problems. The question addressed is broad and all trials, irrespective of the disease and the treatment must be taken into consideration. Combining the p-values is a method which can take into account the diversity found and can be used to compute an overall indicator of the efficacy. However in many trials the primary outcome was not defined, and this might have led to a potential bias because a significant result could be due to chance alone after a large number of comparisons.
Finally several meta-analysis were performed grouping the trials by their methodological quality, with the trials with a clearly defined primary outcome classed as the highest quality.
Results: Conculsions:
EPIMETA: Software for Meta Analyses of Epidemiologic Studies
Authors: David Williamson and Donna Stroup, Centers for Disease Control and Prevention (CDC), United States
Background: Meta analysis is a systematic, quantitative approach consisting of statistical methods which combine results from independent studies for synthesizing conclusions in medical and other areas of research. Its prevalence and acceptance in the scientific literature has increased dramatically during the past decade, especially in clinical trials for evaluating therapeutic effectiveness and making subsequent clinical recommendations. Recently there has been increased need and interest in applying meta analysis methods to epidemiologic studies and other public health research. To date, there have been few commercially available statistical software packages which perform meta analysis. Rather, most individuals have developed their own computer code or modified existing softwares for their research needs.
Objectives: Here we describe EPIMETA, a new statistical software codeveloped by CDC for application to epidemiologic research data. A companion manual, including an overview of meta analysis methods and a tutorial for using EPIMETA, is also described.
Discussion: EPIMETA implements a systematic approach comprised of quantitative methods to analyze epidemiologic and other public health data. It focuses on statistical procedures to estimate weighted averages for parameters of interest, such as relative risks or odds ratios. The analysis procedure performs "within" study and "among" studies analyses, and includes options for transformations, selection of fixed or random effects model, and inclusion of fixed or estimated intercept. The software designates outlier studies and provides a series of diagnostic graphics portraying results and characterizing contributions of individual studies.
The program is DOS-based, but features a Windows-like interface to facilitate data entry, file manipulation, and subgroup analysis. EPIMETA was designed to integrate with EpiInfo, a CDC-developed data management and analysis software familiar to many public health officials. EPIMETA is written in C++ and requires an IBM-compatible computer, 386 processor, DOS 3.1, 600k RAM, 2Mb hard disk space, and EpiInfo installed on the system.
Selecting methods to display graphically the results of systematic reviews: where is the evidence?
Authors: Alejandro R Jadad; Parminder Raina; Deborah J. Cook; Stephen D. Walter, Vicki Torrance, Paul D. Krueger, Larry W. Chambers
Background: Graphical displays are used to facilitate the description, exploration, interpretation and communication of the quantitative information contained in systematic reviews. As with any other tool, however, graphics can be misused, leading to inappropriate interpretation of data. It would be ideal to select graphical methods for the display of systematic reviews based on empirical evidence.
Objectives: To conduct a systematic search of the literature to identify reports of methods to display graphically the results of systematic reviews.
Methods: A systematic search of MEDLINE (1966 to January 1996), CINHAL (1982 to December 1995), HEALTH (I 975 to December 1995), PsychINFO (I 967 to February 1996) was conducted and supplemented with a manual search of 9 textbooks on systematic reviews or data display, search of reference lists, and electronic mail contact with the members of the Cochrane Statistical Methods Group and the Epidemio-L Internet discussion list. The archives of the latter were also searched.
Results: Twelve methods were identified: stem-and-leaf displays, funnel plots, the "confidence interval" method (the one used in most Cochrane Reviews), plots of the difference O-E against variance, the "odd man out", radial plots of odds ratios, confidence interval function, box plots, "criteria based meta-analysis" displays, reciprocal scales, the "sloping line" and "analysis of proximities" methods. We did not find studies evaluating the usefulness of any of these methods.
Conclusions: Despite the widespread use of graphics to represent the results of systematic reviews, we did not identify any empirical evidence to a) support the selection of any particular method, or b) justify the use of graphics to represent the results of systematic reviews. Studies analyzing the effectiveness of these methods on data interpretation and outcomes are required.
Optimal graphical display of the results of meta-analyses of individual patient data
Authors: Luc Bijnens, Laurence Collette, Albert Ivanov, Guido Hoctin Boes, and Richard Sylvester
Background: In order to make the results of meta-analyses interpretable by a wide audience the summary plot has to be as self explanatory as possible. The graphical displays summarizing the results of meta-analyses are not always easy to interpret.
Objectives: The objective of this paper is to present some way of simplifying the plots that summarize the results of meta-analyses.
Methods: The following modifications of the forest plot (graphical summary) are suggested: 1. deletion of the (O-E) and Var(O-E) columns; 2. addition of the survival benefit; 3. systematic use of the log scale; 4. use of 95% confidence intervals for the hazard ratio.
Results and conclusions:
Australian Aboriginal Children:- the Randomised Controlled Trial and Special Populations
Authors: Peter S. Morris, Amanda J. Leach, Phillip Baker and John D. Mathews. Menzies School of Health Research, Darwin.
Background: Improving the health of Aboriginal children is regarded as a national priority. The identification of effective medical therapies will assist with improving health outcomes, and could allow Aboriginal people to have greater control over the process.
Objective: To identify and assess the RCT's, (randomised controlled trials), involving Australian Aboriginal children, or populations living in similar circumstances.
Methods: Electronic searching of Medline (1966-95), the Australasian Medical Index (1970-95), and hand searching of The Aboriginal and Torres Strait Islander Health Information Bulletin and "Aboriginal Health: an annotated Bibliography", were performed. Experts in the area of Aboriginal child health were consulted to try and locate any other studies not listed.
Results: 4 RCT's involving Australian Aboriginal children were found. 2 other studies were controlled but not randomised. 5 of the 6 studies were listed on Medline, and the 6th study was on the Australasian Medical Index. This represents a very small number of studies both in terms of Australian research, and within the Aboriginal Health field.
Identification of RCT's involving similar populations was more problematic and relied on inefficient searching strategies. While several hundred studies were found, it was often difficult to ascertain their clinical relevance. Few studies from developing countries assessed outcomes other than death or severe illness.
Conclusion: RCT's involving Aboriginal children are rare. The generalisability of findings from other populations is not clear. These facts may be contributing to the lack of effective therapies being provided to Aboriginal communities.
A more systematic approach to research involving Aboriginal people is desirable, and enterprises like the Cochrane Collaboration could have an enormous impact. However, to realise its full potential, the issue of equality, (Cochrane's third "E"), should be reconsidered.
Copyright © The Cochrane Collaboration 1999
This page was last updated on 23 March, 1998
Comments for improvement or correction are welcome.
georg.koch@cochrane.de