How can we spread the gospel if we cannot write it down? - more room for qualitative reviews
Authors: Willem J.J. Assendelft, Bart W. Koes, Lex M. Bouter
Background: The Cochrane Collaboration aims at the writing and dissemination of systematic reviews for all clinical areas. The methodology for conducting systematic reviews, as described in the Toolkit and required for entering into the Review Manager (RevMan), has mainly been developed in clinical areas where large, well-performed randomized clinical trials (RCTs) exist. RCTs in these areas generally have uniform patient categories, outcome measures and follow-up moments, and are therefore eligible for statistical pooling.
Method: For the Cochrane review "Spinal manipulation for low back pain" a protocol was written according to the guidelines in the Toolkit. We already knew that the RCTs on this topic varied considerably in size, type of patients, methodological quality and outcome measurement. To enable a sensible pooling we planned to extract data on: patients (4 different characteristics), outcomes (4 different types) and timing of follow-up (short-term, intermediate and long-term). The quality assessment list contained 8 validity items, instead of the single item "concealment of allocation" which is currently available in RevMan.
The protocol has now been accepted, and as a pilot we applied the protocol to the 8 chiropractic RCTs available on this subject.
Results: We tried to pool the 8 chiropractic studies. Most outcome measures, however, in combination with the various follow-up moments, were simply not covered by enough RCTs to perform a pooling which was clinically sensible. This was mainly caused by insufficient reporting of essential details and a substantial heterogeneity in the outcome measures and follow-up timing. Therefore, we were only able discuss this subset of studies on a systematic "qualitative" study-to-study basis. A major constraint was the low adherence on many of the validity items.
Discussion: To our opinion a uniform, quantitative, presentation of systematic reviews, as presently required by the Cochrane Collaboration, will not apply to all topics. To enable inclusion of systematic "qualitative" reviews of methodologically "underdeveloped" subjects like ours, RevMan should have more manuscript processing options. In addition, the assessment of the validity of the included RCTs should receive more attention.
Using evidence to define policy and influence practice-a case study of the Northern Sydney Fractured Neck of Femur (FNOF) Project
Authors: Chamberlain A, March L, Holt D,Cumming R, Cameron I, Finnegan T, Kurrle S, Skinner J.
Background: Approximately 1000 FNOFs are treated in Northern Sydney Health Service (NSAHS) hospitals each year. Although the incidence of FNOF does not appear to be increasing in NSAHS, numbers are. FNOF is a condition with the potential for streamlining inpatient and rehabilitation services. NSAHS was successful in gaining funding for two years under the N.S.W. Health Department's Health Outcomes Programme Grants scheme.
Objectives: To develop best practice guidelines for the prevention, treatment and rehabilitation of FNOF. Implement and monitor a best practice critical care pathway for FNOF treatment and rehabilitation.
Methods: Using Cochrane Collaboration systematic review methods, 7 volunteer reviewers undertook a systematic review of English language RCTs and meta-analyses using Medline and key source journals during 1995. A draft report outlining the results of both the review and a retrospective audit of current practice, was circulated to NSAHS orthopaedic nursing, surgical staff and allied health professionals for comment.
The guidelines were then incorporated into a critical care pathway for the treatment and rehabilitation phases of FNOF in two NSAHS hospitals. The adoption and maintenance of the guidelines was monitored via a prospective audit.
Results: Key areas of service provision requiring change were identified by the systematic review and retrospective audit. Several strategies were employed to raise awareness and promote adoption of the best practice guidelines. This paper will present a case study of the project, highlighting successes and failures in strengthening the link between evidence, policy and practice in health care.
A systematic review and meta-analysis of psychological and pharmacological treatments for depression
Authors: Rachel Churchill, Simon Wessely, Glyn Lewis
Background: Depression affects about 20-30% of the population in the UK. About 90% of cases are managed in primary care using both physical and psychological treatments. Although the first line treatment for depression is still primarily the prescription of tricyclic antidepressants, psychological treatments are becoming increasingly available in primary care settings.
The simultaneous use of medication and psychological treatments is controversial. Some believe that psychological treatments undermine the effect of antidepressants whilst advocates of psychological therapies have strong theoretical reasons for preventing patients from simultaneously receiving medication, despite the proven efficacy of antidepressants. Nevertheless, the possibility remains that the outcome is better for those treated with a combination of antidepressants and psychological treatments than it is for those treated with either one alone.
Objectives: To conduct a systematic review and meta-analysis of all randomised controlled trials comparing the effects of psychological and pharmacological treatments, alone and in combination, for depression.
Methods: Searches to be completed by July 1996. Electronic databases being searched: MEDLINE, Science Scisearch, Social Scisearch, PsycLit, EMBASE, PAHO and Information on Scientific and Technical Proceedings. Handsearching: Behaviour Research and Therapy, Cognitive Therapy and Research, International Clinical Psychopharmacology, Journal of Clinical Psychopharmacology. Bibliographies, "grey literature", book sections, theses and dissertations are also being identified and searched.
Results: A total of 23 reports have been identified pertaining to 10 trials conducted between 1974 and 1992. The quality of the trials is generally poor. Only one report identified so far describes the method of randomisation used. Intention to treat analyses have not been employed. Attrition rates, reasons for drop-out and point at which drop-out occurred are not specified. Only 3 trials have been followed up beyond the initial treatment period. Generalizability of the trials is questionable, with the majority of the trials having been conducted in the US, predominantly involving patients in secondary care settings. The final results will be ready for presentation at the Cochrane Colloquium in October 1996.
Searching meeting abstracts: a comparison between an experienced handsearcher and an indexed, electronic database
Authors: Mike Clarke, Liz Greaves
Background: Meeting abstracts are one way to identify randomised controlled trials (RCTs). Occasionally, these abstracts are included in an indexed bibliographic database. This is so for the annual meeting of the American Society of Clinical Oncology (ASCO) which is in Cancerlit.
Objectives: To compare a handsearch of the 1,471 abstracts from the 1992 ASCO meeting with an electronic search of an indexed set of these, as means for identifying RCTs.
Methods: An experienced handsearcher read each of the abstracts and an electronic search of Cancerlit was performed.
Results: The handsearcher found 208 abstracts reporting the results of RCTs, 26 which might be RCTs, and 42 that mentioned current or future RCTs. An electronic search for abstracts tagged with the publication type "randomised controlled trial" identified 179 abstracts. Adding "double blind", "phase III" and the character-string "random" found 119 more. Combining the hand and electronic searching yields 285 abstracts that were, or might be, RCTs. Handsearching found 276 (97%) and the electronic search found 262 (92%) of these. Both found nearly all the probable RCTs, but only 4 of the 26 possible RCTs were found electronically. Some of the latter would be found by adding words such as "placebo" to the search; but others, such as reports mentioning at least two treatments but with an inadequate description of how the groups were formed, probably could not have been found electronically.
Conclusions: A combination of hand and electronic searching will yield the highest proportion of RCTs. However, if there are limits on time and resources, it might be preferable to handsearch proceedings which are not available in an indexed database and to do an electronic search alone for those that are. The success of both processes could be improved if abstracts reporting RCTs were more clearly identified.
Identifying reports of randomized controlled trials (RCTs) in the Lancet: the contribution of the Cochrane Collaboration
Authors: Mike Clarke, Steven McDonald, Carol Lefebvre
Background: Since 1993, the Cochrane Collaboration has contributed towards improving the retrieval of RCTs from MEDLINE in two important ways; by devising a highly sensitive search strategy, and by identifying reports of RCTs for retagging by the US National Library of Medicine (NLM). In 1994, following the reading of over 100,000 titles and abstracts from MEDLINE, 20,000 reports of RCTs were submitted to the NLM for inclusion in MEDLINE in 1995. MEDLINE is also being enhanced as a result of journal handsearching throughout the Collaboration. The Lancet was handsearched during 1995, and the retagged records were made available in MEDLINE in 1996.
Objective: A case study, using reports published in the Lancet in 1992, to identify the number of reports of RCTs correctly tagged by NLM, and the number of additional reports identified by the electronic search and the handsearch.
Methods: Reports of RCTs from the Lancet in 1992 were classified according to whether they had been tagged originally by NLM, retagged as a result of the search strategy, or retagged as a result of hand-searching.
Results: A search of MEDLINE in 1994 would have identified 100 Lancet records tagged as RCTs. An additional 7 records were retagged by 1995 as a result of the electronic search, and a further 18 were retagged by 1996 as a result of the handsearch. Of the 100 reports originally tagged as RCTs by NLM, 8 were not reports of RCTs, and therefore should not have been tagged as such.
Conclusions: Even for a major journal such as the Lancet, 22% of RCTs may be missed by relying on NLM indexing alone. Further investigations will aim to identify whether there are systematic differences between the reports indexed as RCTs as a result of the three processes discussed above.
The role of systematic reviews in meta-analyses: Selective serotonin reuptake inhibitors and tricyclics for the treatment of depression
Authors: Rachel Churchill, Matthew Hotopf, Rebecca Hardy, Glyn Lewis, Charles Normand
Background: Meta-analyses can provide a clear presentation of individual and pooled trial data, but they are limited by the quality of the original RCTs. 105 RCTs and four meta-analyses have failed to provide a clear answer to the question of whether selective serotonin reuptake inhibitors (SSRIs) or tricyclic/heterocyclic antidepressants should be used as first line treatment for depression in primary care settings. Systematic reviews, by illustrating the limitations of trials, can provide information of greater value.
Objectives: To conduct a systematic review and meta-analysis of trials comparing SSRIs and tricyclic antidepressants for the treatment of depression in primary care.
Methods: All randomised controlled trials reported in previous meta-analyses comparing SSRIs and tricyclic antidepressants were used. In addition, a Medline and Index Medicus search was performed. Two journals were also handsearched: International Clinical Psychopharmacology and Acta Psychiatrica Scandinavica.
Results: The key findings of the systematic review are that the majority of trials are small, fail to conduct intention to treat analyses, are based in secondary care where only a minority of patients are treated, use observer rated assessments of depressive symptoms which are open to observer bias, and fail to give economic evaluations. A meta-analysis using drop outs from treatment showed that overall the SSRIs had a modest advantage over tricyclics and heterocyclics (Risk Ratio 0.90; (95% CI: 0.86-0.97)). As hypothesised, this effect was strongest when older ricyclics were used as the comparison group, probably due to their more prominent side effects (RR 0.88; 95% CI: 0.82-0.95). When compared with newer tricyclics or heterocyclics no significant advantage for the SSRIs could be found (RR=0.92; (95% CI: 0.82-1.04) for new tricyclics, and RR 1.02; (95% CI: 0.83-1.25) for heterocyclics).
Conclusions: The poor quality of many trials and these still equivocal results, based on drop out not clinical recovery, indicate a need for a large RCT based in primary care, and using a newer tricyclic as the comparison drug.
Implementation of Cochrane Pregnancy and Childbirth Reviews: an Italian project
Authors: Angelo Penna, Fabio Parazzini, Anna Calabro, Gianni Barro, Roberto Grilli, Cristina Sideli, Giovanni Filardo, Alessandro Liberati
Background: The transfer of evidence based information into clinical practice is an important research issue. Variation in the patterns of monitoring of pregnancy and in treatments for threatened abortion and preterm delivery is commonplace.
Objectives: Promoting - through two distinct projects - the use of evidence-based practice guidelines based on information derived from CPCD (Cochrane Pregnancy Childbirth Database).
Methods: The first project involves a multidisciplinary panel of experts asked to determine the appropriateness of different tests currently used in pregnancy care on the basis of CPCD data. The second foresees an audit of care delivered to 800 women admitted - over a six months period in the 15 hospitals within the region - for threatened abortion and preterm delivery. Results will be compared to information available in CPCD and discussed with obstetricians and midwifes in charge to identify area for improvement.
Results: The usefulness of CPCD and the acceptability of the overall strategy will be discussed together with the results of the audit in terms of correspondence of care provided to recommendations derived from the database.
Conclusions: Studies of this sort can provide an important test of the usefulness of systematic reviews on the effects of health care interventions such as those currently being produced by the Cochrane Collaboration.
Collaboration to identify controlled trials in general health care journals in Europe
Authors: Steven McDonald and Carol Lefebvre on behalf of the BIOMED General Health Care Journals Project Group
Background: As part of the European Union BIOMED Programme, six partners (Denmark, France, Germany, Italy, Netherlands and UK) are collaborating in a Project, co-ordinated by the UK Cochrane Centre, to identify reports of controlled trials, by handsearching European general health care journals. Once identified these reports are forwarded to the US National Library of Medicine (NLM) where they are retagged in MEDLINE. From April 1996 these reports have also been made available in the The Cochrane Controlled Trials Register (CCTR), published as part of The Cochrane Library.
Objectives: To ensure that by 1998 reports of trials published in European general health care journals since 1948 are accessible through MEDLINE and CCTR.
Methods: General health care journals published in Europe have been identified and ranked according to their likely yield of reports of trials. Each partner in the Project has recruited and trained handsearchers to identify these reports. Once identified, the MEDLINE record for each report is downloaded and forwarded to the NLM, through the Baltimore Cochrane Center, for retagging with the appropriate Publication Type terms. For those reports of trials not on MEDLINE, records will be forwarded to the NLM for inclusion in an ancillary database currently under development. All records will also be included in CCTR.
Results: Over thirty journals are being/have been handsearched, and over 8000 reports of trials have been identified. This project has resulted in 4000 reports, which were not tagged as trials, being forwarded to the NLM during 1995. These have been identifiable in MEDLINE as reports of trials since January 1996. In addition to these 4000 reports, 2000 pre-1966 reports are now available on CCTR.
Discussion: This Project has already contributed importantly to the Cochrane Collaboration, and should have provided substantial help to those preparing systematic reviews. Issues surrounding quality control of the handsearching, and effective means of data transfer will be discussed.
Searching to identify meta-analyses: it aint easy
Authors: McKibbon KA, Jadad AR, Moher M, Jones AL, Cook DJ, Tugwell P, Moher D
Background: Our team was funded to provide empirical evidence on several issues related to the quality assessment of reports of randomized controlled trials used in meta-analyses. To provide some useful reliability and validity to these questions we felt it was important to survey the literature as an efficient first step.
Objective: To complete an electronic search of the medical literature to identify published meta-analyses. The search was aimed at maximising recall (sensitivity) while retaining a high level of precision.
Methods: We developed search strategies to identify articles, purporting to be meta-analyses, on Medline and Embase. The search was translated, with the available controlled vocabulary, so that it could be executable in the two systems. The Medline search was completed (January 1, 1996-December 30, 1995) using the OVID search engine. The Embase search was completed January 1, 1980 to November 30, 1995) using the windows version of Silverplatter (winspirs). The search strategies included a mixture of subject indexing and free text terms.
Results: Our search strategies identified 1464 articles in Medline and 3159 articles in Embase. An initial analysis identified 425 articles from Medline that specify meta-analysis in the title. Further analysis indicates that 28% of the unspecified articles are in fact reports of meta-analysis. Additional results will be presented as will sensitivity comparisons of meta-analysis identified by both databases.
Conclusions: Authors of meta-analyses should be encouraged to specify this in the title of the report.
Consumer Information on Prostate Cancer Screening
Authors: Dorothy Reading, Ron Borland, Val Clarke, David Hill, Graham Giles
Background: The rate of screening for prostate cancer has been rising significantly in recent years. There is a great deal of ignorance. misunderstanding, confusion and conflicting claims In relation to prostate cancer, but the magnitude of problems in the community arising from this confusion has not been quantified. The issues are mainly focused on whether, who how, and when to screen for prostate cancer, and an the appropriate management of prostate disease. Men who simply follow professional advice could be managed in vastly different ways. depending upon which doctor they consult. A good consumer education program is required. Given the different recommendations of professional groups, it is important that consumers understnd the issues and make Informed decisions about screening or, if symptomatic, the way their condition is managed.
Objectives & Methods: The Objectives of Stage 1 are to ascertain the current state of knowledge in the areas of screening for prostate Cancer and information provision to consumers, to obtain baseline data in relation to the level of demand for prostate cancer screening, to ascertain if general practitioners will support and participate in the project and to identify other relevant initiatives being undertaken in this am. These objectives will be achieved by undertaking a literature review, a survey of general practitioners and an activities scan.
The objectives of Stage 2 are to produce a valid tested guide for consumers and a costed Proposal for wide dissemination of this guide. This will be achieved by undertaking a study of consumer knowledge, attitudes and experience relevant to prostate cancer followed by the development and the pre-testing of a consumer guide, The consumer study will involve, two samples of men aged 50-75 - a representative sample and a sample of male doctors, as a bench mark for estimating achievable levels of public understanding of the complex issues relating to prostate disease.
Timeline: The study will be completed by the end of June, 1997. At the conference a progress report will be given on achievements to date.
Psoriatic arthritis: A quantitative overview of therapeutic options
Authors: G Jones, M Crotty, B Shea, P Brooks and the PSA meta-analysis study group.
Abstract: Objective: To use the technique of meta-analysis to undertake a systematic review of published and unpublished randomized controlled trials of pharmacological agents to determine their relative efficacy and toxicity in the treatment of psoriatic arthritis.
Methods: Literature search using search terms of psoriasis, arthritis and controlled/randomised trial. The need for uniform data presentation necessitated correspondence with a number of original authors. The main outcome measure was the change in pooled disease index with component variables derived from OMERACT.
Results: Nineteen randomised trials were identified of which twelve were included in the quantitative analysis with data from 792 subjects. Although all agents were better than placebo, parenteral high dose methotrexate and salazopyrin were the only two agents that achieved statistical significance. In all trials the placebo group improved over baseline (pooled improvement 0.43 DI units, 95% CI 0.28-0.59). There was insufficient data to examine toxicity.
Conclusions: Parenteral high dose methotrexate and salazopyrin are the only two agents with well demonstrated published efficacy in psoriatic arthritis. The magnitude of the effect seen with etretinate, oral low dose methotrexate, azathiaprine and perhaps colchicine suggests that they may be effective but that further multicentre clinical trials are required to establish their efficacy. Furthermore, the magnitude of the improvement observed in the placebo group strongly suggests that uncontrolled trials should not be used to guide management decisions in this condition.
A trend analysis and search strategies for the identification of meta-analyses in MEDLINE
Authors: Frederic M. Wolf, Carol Lefebvre, William C. McGaghie, University of Michigan, UK Cochrane Centre, Northwestern University
Abstract:.Objectives: TO estimate
1) the growth of meta-analysis as an analytic tool for reviewing health care research findings
2) the unique and combined contributions of key word and free-text retrieval to identify a sensitive strategy for retrieving meta-analyses in the medical literature.
Methods: Meta-analysis citations were retrieved yearly over a 19 year time period and subjected to it statistical trend analysis to characterized the nature of publication trends over time. The MEDLINE database was searched from 1976 through 1994 inclusive using two independent inlet-faces. SilverPlatter and OVID. We used free-text searches of titles and abstracts and "meta-analysis" as a MeSH Heading (introduced by NLM in 1989) and as a Publication Type (introduced by NLM in 1993). Results: A combined search strategy yielded a total of 3063 citations, but at least 10% of these citations would have been missed by not using a multiple search strategy. There has been exponential growth in the number of articles published (R7 = -97, 1)<.0001), from one in 1977 to over 100 in 1988 to over 300 in 1990 and over 600 in 1994.
Conclusions: Tile, number of citations of reports of or about meta-analysis doubled ettch year between 1987 and 1989, and tben increased each year by over 25% between 19X9 and 1992. The introduction of "meta-analysis" as a Publication Type in MEDIINE will facilitate the distinction between identifying meta-analyses per se and articles about meta-analysis. The reason that any search may under- or over estimate the true number of meta-analyses will be discussed.
Sensitivity, specificity, and predictive value of using abstracts to identify RCTs of asthma educational interventions in MEDLINE and EMBASE
Authors: Fredric M. Wolf, Cyril M. Grum, Noreen M. Clark. University of Michigan, Ann Arbor, Mi. USA.
Background: Searches of electronic databases can miss some true. RCTs that are identified by hand Searching journals. However, all citations retrieved from an optimally sensitive search are not neccesarily true reports of RCTS. One practical question is the degree to which individial citations may accurately be classified as true RCTs (on the basis of the database's abstract alone.
0bjectives: To ascertain the degree. to which
1) individual MEDLINE (Index Medicus) and EMBASE, (Excerpta Medica) citations May accurately be, classified as true RCTs on the basis of the abstract and
2) a detailed reading of full text articles, particularly methods sections, is neccssary.
Methods: A total of 108 citations were retrieved in electronic searches of MEDLINE and EMBASE databases from 1980- 1993 based on the standard Cochrane Collaboration search strategy for RCTs in combination with "and asthma or wheez*" and "and education". Each citation was classified on the basis of the abstract alone by two independent raters as 1) a true RCT or 2) not a true RCT or too ambiguous to classify without additional information (Kappa=0.83). Citations were then classified on life basis of the full text article, with disagreements resolved by consensus, i.e., so that there was 100% agreement.
A 2 x 2 table was constructed in order to assess the accuracy of RCT classification by abstracts alone, using full text classification as the "gold standard" for comparitive purposes.
Results/Conclusions: The sensitivity (accuracy) of abstract classification of RCTs was 84%, the specificity (precision) 100%, the positive predictive value 100%, and the negative predictive values 85%. Classifying abstracts in a manner designed to be maximally specific and predictive of actual RCTs in the literature, as expected, results in some RCTs being missed that could only be identified on the basis of reading the full text of the article.
Approximately 16% of true RCTs would be missed by abstract classification alone.
This page was last updated on 18 November 1996
Comments for improvement or correction are welcome.
georg.koch@cochrane.de