Chair (Stream 1): Cindy Farquhar (Cochrane Menstrual Disorders Review Group)
Chair (Stream 2): Monica Fischer (Cochrane Collaboration Steering Group and Nordic Cochrane Centre)
So what the hell is an odds ratio? Can we make the results of reviews more easily understood?
Authors: Jon Deeks, Gordon Dooley, Douglas Altman, Dave Sackett (Centre for Statistics in Medicine, Centre for Evidence-Based Medicine, Update Software, Oxford)
Background: Systematic reviews in CDSR and elsewhere have traditionally used odds ratios (OR) to analyze and express treatment effects when outcomes are measured as event rates. Interpreting the OR as an approximation to the relative risk (RR) will lead to exaggeration of the treatment effect unless event rates are very low. Statistical methods exist (and are now available in CDSR) for combining relative risks. We have investigated the effects of using these methods in the reviews in CDSR as compared with using methods based on the odds ratio.
Objectives: To examine the effect of analyzing and reporting the results of meta-analyses as relative risks rather than odds ratios, noting apparent and real exaggerations of treatment benefit and increases in between study heterogeneity.
Methods: Systematic reviews contained within CDSR were analyzed using Mantel-Haenszel methods for combining odds ratios and relative risks. Apparent exaggerations in treatment effects were assessed by comparing OR and RR common effect estimates. Real exaggerations were estimated by comparing numbers-needed-to-treat calculated separately for both methods. Heterogeneity statistics were also calculated for both methods, together with a summary of the distribution of control group event rates in the contributing clinical trials.
Results: There are 1507 presentations of syntheses of dichotomous outcomes in the CDSR. We report on the apparent and real exaggerations of treatment benefit in these reviews comparing odds ratio and relative risk analyses, and relate our findings to the distribution of control group event rates.
Implications: We will produce recommendations on the suitability of using RRs rather than ORs as summaries of effect size for dichotomous data, and note circumstances when this approach is likely to introduce bias and excessive heterogeneity.
Weight of evidence and reliability of meta-analysis
Authors: Kameshwar Prasad, Geetha Menon All India Institute of Medical Sciences, New Delhi, India
Background: Effect size obtained from a meta-analysis has been used to classify the level of evidence available for the use of a health care intervention. However, a given effect size obtained from a meta-analysis of only small randomized trials would have lower weight of evidence and hence be less reliable than that obtained from a meta-analysis of large (with small) randomised trials. There is a need to develop methods to assess the reliability of an effect size from a meta-analysis according to the weight of evidence.
Methods: We used log-likelihood ratio as measure of `weight of evidence' expressed in support units (su). Log-likelihood ratio of `one' was taken as one support unit. Support units for point-estimates were calculated against the null-hypothesis for 69 meta-analyses of randomised trials of antiplatelet therapy in different class of patients for different end-points published by Anti-platelet Trialists' Collaboration. The meta-analyses were classified into three groups: highly reliable (more than 10 su), reliable (five to 10 su) and unreliable (less than five su).
Results: Support unit scores for the meta-analyses ranged from 0.01 to 92.29 (mean 15.45). 30 meta-analyses scored more than 10 support units, seven scored between 5 and 10 and 32 scored less than five support units. On comparison with the collaborators' reliance on the evidence, these categories had good correspondence.
Conclusions: We have developed a method to assess the reliability of a meta-analysis based on the weight of evidence. The method will allow us to determine the robustness of conclusions from a systematic review. More experience with the application of the method will help define its utility.
An efficient search strategy identifying primary papers on diagnostic test evaluations.
Authors: Walter Deville, Dick Bezemer, Lex Bouter.
Background: Different search strategies for diagnosis are suggested during the past years. The predictive value of these strategies for the identification of publications about diagnostic test evaluations remains however quite low. An efficient search strategy - subject independent - was developed by screening 9 journals on family medicine available in Medline from 1992 through 1995.
The 'gold standard' were the Haynes searches for 1991 (the sensitive one and the specific one combined), published elsewhere. The search resulted in 306 abstracts, wherefrom 193 of primary papers. Abstracts were classified in clearly diagnostic field were identified, as were textwords also relevant for this field. With a logistic regression analysis a predictive model was developed of a combination of MESH terms only and one including textwords. The last one had a sensitivity of 95% and a specificity of 89% using only three terms: exploding sensitivity and specificity, and the textwords specificity and likelihood ration. This strategy will further be validated in another set of journals.
Identifying diagnostic studies relevant for primary health care.
Authors: Trudy van der Weijden, Geert-Jan Dinant, Nico van Duijn, Joris IJzermans, Frank Buntinx MEDION group, University of Limburg (the Netherlands) & University of Leuven (Belgium).
Background: In the scope of reviewing diagnostic studies, the first important step is collecting data in a systematic and reproducible way.
Objective: To examine the sensitivity and precision of MESH searching in comparison with a combination of MESH and freetext searching for the identification of diagnostic studies in Medline.
Methods: Two medical tests were choosen to address the question; the ability of the erythrocyte sedimentation rate (ESR) to discriminate between 'pathology' and 'no pathology', and the diagnostic accuracy of the dipstick test for urinary tract infection. The results of each Medline search were compared with a gold standard set of studies: a literature database on ESR collected continuously during many years by one of the authors (GJD), and the result of all available search methods on dipstick.
Outcome measures: Sensitivity: the proportion of the total number of known diagnostic studies identified by the search.
Precision: the proportion of publications retrieved by Medline that were actually diagnostic studies.
Results: With the combined search high sensitivity could be reached: 0.91 and 0.98 for ESR and dipstick respectively. The precision was 0.10 for ESR and 0.67 for dipstick. By restricting the search to the primary care setting sensitivity droppped to 0.10 (ESR) and 0.00 for (dipstick).
Conclusion: Combining freetext and MESH searching, without restriction to the setting, is the most valuable strategy in systematic searching for diagnostic studies. The precision seems dependent on the extent of the indication area of the test.
Diagnostic value of digital rectal examination (DRE) for detection of prostate cancer: The influence of methodological characteristics on outcome
Authors: Henrica CW de Vet, Arjen Hogendam, Bart Liedekerken, Trudy van der Weijden, Frank Buntinx. MEDION group, University of Limburg (the Netherlands) & University of Leuven (Belgium)
Background: For clinicians it is important to know the accuracy of diagnostic tests. Systematic reviews on the diagnostic value of specific tests are rare. We performed such a review for DRE and prostate cancer, with emphasis on the methodological characteristics of the studies.
Objectives: To examine the influence of methodological characteristics on the outcome of diagnostic studies.
Methods: We collected studies on DRE and prostatic cancer, published between 1983 and 1996. We searched in MEDLINE and FAMILY using a broad set of MESH terms, and checked the references of the retrieved papers. Inclusion criteria were: study population included both cases and noncases and language was English, French, German, Dutch, Spanish or Italian. Methodological quality of the papers was assessed using the critera forwarded by the Cochrane Methods Group on Meta-analysis of diagnostic and screening tests.
Results and conclusions: We found 47 eligible studies. Pooled sensitivity and specificity and positive predictive value will be presented for different settings. The influence of the following methodological characteristics will be assessed: valid reference standard, independent measurement of diagnostic test and reference standard, avoidance of verification bias, withdrawals.
Results will be discussed against the background of improvement of methodological quality of diagnostic studies.
Use of continuous outcome measures in systematic reviews
Authors: Frederic M Wolf, Jon Deeks, Gordon Dooley, Douglas C Altman
Background: While mortality and morbidity are typically measured on binary scales (e.g., alive-dead; better-worse) in medical research and thus odds or relative risk ratios are pooled in systematic reviews/meta-analyses, there are many outcomes that are measured on continuous, interval scales. When the identical scale of measurement is used acoss studies (e.g., length of stay, bilirubin, G11bAI, DBP), then a pooled weighted mean difference can be calculated. When different scales are used to assess a conceptually similar outcome from different studies (e.g.. quality of life, pain, pulmonary function as peak flow, FEV, or %FEV), then a standardized mean difference is used to convert the outcomes to one comparable common metric for pooling and comparison purposes.
Objectives: To describe the frequency and appropriateness of use of continuous outcomes in systematic reviews, and the types of problems to which they are applied.
Methods: Descriptive methods (simple counts and percentages) were used to examine systematic reviewss in CDSR.
Results: The complete set of 98 reviews in issue 3 of CDSR (Cochrane Library, BMJ Publishing, April 1996) were examined. These reviews contained a total of 1671 analyses or comparisons, or about 17 outcomes per review. Of the total set of 1671, 164 (about 10%) were outcomes measured on continuous scales, while the remainder were binary outcomes.
Conclusions: A content analysis of the systematic reviews in which these 164 continuous outcomes appear is underway. The number and percentage of analyses in which a pooled weighted mean difference was appropriately used will be determined, as well as the frequency with which a pooled standardized mean difference was used or ought to have been used. Anomalies and possible errors will be tabulated (eg, extremely skewed data, questionable measures of variability). Some indication of the scope, use and quality of analyses of continuous outcomes should be possible to serve as a guide for future work.
Prevention of fungal morbidity and mortality in neutropenic patients
Authors: Helle Krogh Johansen, Peter C Gotzsche. The Nordic Cochrane Centre, Denmark
Background: Fungal infection is believed to be an important cause of morbidity and mortality in neutropenic patients. The diagnosis is difficult and the effect of treatment is uncertain.
Objective: To study whether antifungals are effective in patients with haematologic malignancies and neutropenia.
Methods: Randomised trials comparing antifungals vs no prevention or placebo were collected. A modified Cochrane search strategy was used. Data were extracted independently by two observers. Mortality was the primary outcome.
Results: Thirty studies were identified. Data were available in 24; amphotericin B was used in 7, fluconazole in 5, ketoconazole in 8, other antifungals in 4. The randomisation method was not fully described in any study. The median followup period was 56 days (not available in 11 studies). No significant difference was found in mortality (odds ratio 0.91, 95% CI 0.73-1.13, 197/1118 (17-6%) patients died in the treatment groups vs 212/1130 (18.8%) in the untreated groups; number of deaths was not available in 8 studies).
Number of systemic fungal infections (odds ratio 0.48, 0.29-0.78), number of colonized patients (odds ratio 0.40, 0.26-0.63), and use of additional antifungal therapy (odds ratio 0-69, 0.52-0.90) were lower in the intervention groups.
Conclusion: Antifungals have no effect on mortality, only on surrogate outcomes in patients with haematologic malignancies and neutropenia. Prevention with these drugs seems unwarranted but larger trials are needed.
Management of frequent attenders in primary care
Authors: David Gill, Martin Dawes, Chris Bass, Julia Cartwright, Laurence Mynors-Wallis, Richard Mayou and Mike Sharpe.
Background: Demand for health care is rising everywhere, and resources are limited. Primary care has been no exception, with consequent morale and recruitment problems among GPs. It is beyond GPs to influence mass consulting behaviour, though public education programmes to encourage responsible use of GP services have been tried. However, surveys indicate that a small minority of frequent attenders (e.g. 2%) is responsible for 10% or more of total workload. Frequent attenders could thus be possible subjects for specific management, both to improve their health and to optimise resource use.
Objectives: to review systematically randomised controlled trials on managing frequent attenders.
Methods: we searched for relevant trials on Medline, Psyclit and Embase, also consulting reference lists, reviews, experts, the Science Citation Index and the Cochrane Primary Care database.
Results: we found no trials of management strategies for unselected frequent attenders. We found four small trials relating to selected subgroups: two of psychiatric consultation and advice for those with distress / medically unexplained symptoms (one positive, one negative), one of group GP discussion treatment for frequently attending mothers with small children (positive), and one of a letter to frequent non-attenders (those who make frequent appointments and then do not arrive) (positive).
Conclusions: Frequent attenders are an important minority in primary care, who may have unmet health needs, and major effects on resource use and staff morale. There is a lack of published trials on managing this important group. Two small trials indicated simple strategies of a GP letter and a GP discussion group might be worth further evaluation.
Global input into Review Groups - an example
Authors: Karla Soares, Jair Mari, Clive Adams
Background: Many Review Groups within the Collaboration are dealing with problems of global importance. Over 77% of the world population live in low/middle income countries. Sheer weight of numbers may dictate that most people with, for example, chronic airways disease, diabetes, pre-eclampsia or schizophrenia, are living in these countries. Compared with higher income countries, lower income nations have scarce resources for provision of care and so there is an urgent need to identify effective - and cost efficient - interventions.
Objective: To describe the experience of a single reviewer (KS), from Brazil, working within a Cochrane Collaborative Review Group.
Methods: A case report.
Results: KS, funded by a Brazilian post graduate grant (CAPES), has worked in the UK, with the Cochrane Schizophrenia Group. She is in the latter stages of a PhD focusing on tardive dyskinesia (a movement disorder often seen in those with schizophrenia). The thesis includes several systematic reviews also published on Cochrane Database of Systematic Reviews (CDSR). KS had a background in epidemiological research but was new to methodical literature searching and systematic reviewing. During this time KS was supervised and trained by JM (Brazil) and CA (UK). The Schizophrenia Group, KS and those with tardive dyskinesia have benefited from this collaboration.
Conclusions: This experience provides a practical example of how people from different health care cultures can work together. If the reviews within the CDSR are not to be impoverished by a one-sided perspective, global collaboration is crucial. Systematic reviews are a powerful, relatively inexpensive, way by which people from lower income countries can help evaluate the health care provision for everyone - not just a fraction (23%) of the worlds' population.
Development of a sensitive search strategy for reports of randomized controlled trials in EMBASE.
Authors: Carol Lefebvre and Steven McDonald
Background: EMBASE, published by Elsevier in the Netherlands, is the principal European general health care database. Started in 1974, EMBASE currently indexes over 3000 journals, of which about 1000 are not indexed in MEDLINE. It is especially strong in pharmacology, and is likely to be the most important source of RCTs after MEDLINE. It is widely recognized, however, that the quality of methodological indexing in EMBASE was until recently relatively poor, and it is therefore difficult to locate reports of RCTs. In 1993, Carol Lefebvre, in consultation with others, developed a sensitive search strategy for RCTs in MEDLINE. Over the past two years, efforts within the Cochrane Collaboration to enhance MEDLINE (1975-93) using this search strategy have resulted in over 40,000 reports of trials being identified and subsequently retagged by the US National Library of Medicine for MEDLINE.
Objective: To devise and evaluate a sensitive search strategy for EMBASE.
Methods: Search terms (both free-text and from the controlled vocabulary) are being identified and tested for sensitivity and precision. The strategy will be developed, based on these data.
Results: The results will be presented in the form of tables of sensitivity and precision data, together with the final search strategies for the main EMBASE providers on CD-ROM (eg OVID and SilverPlatter) and online (eg DataStar/Dialog and BIDS).
Conclusions: As Collaborative Review Groups attempt to build up their trials registers, they are increasingly looking to sources such as EMBASE to supplement the trials identified through MEDLINE, or by handsearching specialized journals. The sensitive search strategy should facilitate this activity, and should be of benefit to those developing specialized registers, and to those preparing systematic reviews.
Registry-based Observational Studies of the Management and Outcome of Cancer: A New Way to Look At Old Problems
Authors: William MacKillop and Patti Groome. The Radiation Oncology Research Unit, Dept of Oncology, Queen's University, KRCC and KGH Kingston, Ontario, Canada
Abstract: The system of knowledge on which the practice of oncology is based was strengthened when randomized clinical trials (RCTs) replaced institution-based observational studies as our method for testing hypotheses about the effectiveness of treatment, but, there remains a void between the world of knowledge and the world of practice. Registry-Based Observational Studies (RBOs) can help to bridge this gap. Population-based cancer registries were originally established to study the epidemiology of cancer. RB0s exploit the existence of cancer registries to study the epidemiology of cancer treatment. Since RBOs deal with unselected cohorts of cases, they are free of the problems of referral bias and selection bias which contaminate institution-based observational studies. RBOs and RCTs are complementary, rather than competing, methodologies. RCTs are primarily designed to tell us how to manage individual patients, while RBOs are primarily designed to tell us how to manage the health care system. RBOs have the potential to tell us about
Antenatal corticosteroids to prevent adverse perinatal outcomes in the presence of preterm prelabour rupture of the membranes (pPROM). Meta-analyses including unpublished data.
Authors: Arne Ohlsson, Janet Lacy. University of Toronto Maternal, Infant and Reproductive Health Research Unit, Toronto, Ontario, Canada.
Background: Systematic reviews of the effect of antenatal corticosteroids on perinatal outcomes in the presence of pPROM have indicated a need for more precise estimates of the effect size.
Objectives: By obtaining additional data from published studies of this intervention in pPROM populations, and thus increasing the statistical power, we intended to confirm/refute the possible benefits of antenatal corticosteroids in reducing adverse perinatal outcomes.
Methods: Medline, Embase, Reference Update, and Cochrane Pregnancy & Childbirth Database were searched. The quality of the studies was assessed using the Jadad method. We contacted 5 primary authors and received unpublished information from two studies, both of high quality (HQ). This included the original data set on tape from one large study. Data was abstracted by two researchers using preset definitions. Odds ratios (OR) and 95% confidence intervals (CI) were calculated using Revman, for all studies (AS) and for HQ studies.
Results: Previous published studies provide information on at least one outcome in 779 infants. We obtained data on an additional 395 mothers (405 infants) from two HQ studies. Inconsistent ascertain ment of intraventricular haemorrhage prevents the use of meta-analytic techniques to assess the effect of antenatal corticosteroids on this important outcome.
The summary ORs and CIs were for: RDS (AS) 0.55 (0.38-0.79) RDS (HQ) 0.67 (0.30-1.48) Neonatal infection (AS) 1.16 (0.58-2.30) Neonatal infection (HQ) 0.97 (0.15-6.19) Neonatal death (AS) 0.69 (0.32-1.49) Neonatal death (HQ) 0.97 (0.15-6.19) Endometritis (AS) 1.65 (0.91-2.98) Endometritis (HQ) 1.21 (0.50-2.91) Chorioamnionits (AS) 0.88 (0.57-1.36) Chorioamnionitis (HQ) 0.88 (0.45-1.71) Cesarean section (AS) 1.04 (0.73-1.49) Cesarean section (HQ) 0.73 (0.38-1.41)
Conclusions: After obtaining unpublished data, we conclude that the benefit (risks) of antenatal corticosteroids in the presence of pPROM, although suggesting a reduction in RDS, remains an enigma.
Copyright © The Cochrane Collaboration 1999
This page was last updated on 23 March, 1998
Comments for improvement or correction are welcome.
georg.koch@cochrane.de