Assessing the Quality of Primary Studies: Empirical Study of Meta-Analyses Published in Leading Journals

Tallon D., Juni P., Egger M. MRC Health Services Research Collaboration, University of Bristol, UK

Introduction:
Objectives: To examine the methods used for assessing the quality of component studies in meta-analyses of randomised controlled trials published in English-language medical journals.
Methods: A handsearch 1993-1997 of four general medicine journals (Ann Intern Med, BMJ, Lancet, JAMA) and four specialist journals (Am J Cardiol, Cancer, Circulation, Obstet Gynot) identified 135 meta-analyses. We used a standardised questionnaire to extract relevant information.
Results: Authors of 47 (34.8%) meta-analyses assessed some dimension of quality of primary studies, with little evidence of a trend in the frequency of quality assessment over time. Beneficial and statistically significant results for primary outcomes were reported in about 70%, independent of whether the quality of primary studies was assessed or not.Meta-analyses sponsored by the pharmaceutical industry were less likely (2 out of 18, 11.1 %) to have assessed the quality of component studies than meta-analyses with other or unknown sources of funding (44 out of 117 37.6%, p=0.03). Conversely, articles which referred to the Cochrane Collaboration were more likely to include an assessment of study quality (11 out of 20, 55%) than articles that did not (35 out of 115, 30.4%, p=0.03). A total of 33 different scales and checklists were used. For 26 meta-analyses (55.3%) authors developed their own instrument. Study quality was incorporated in the analysis in 26 articles (55.3%). The most common method of incorporating quality data was to exclude trials deemed to be of poor quality (table).

Instrument used for quality assessment
Checklist	20 (42.6%)
Scale	29 (61.7%)
Methodological domain covered
Concealment of treatment allocation	31 (66.0%)
Blinding of outcome assessment	29 (61.7%)
Withdrawals and dropouts	27 (57.4%0
Use of quality data in analysis
Any use of quality data	26 (55.3%)
Exclusion of trials of low quality	15 (31.9%)
Analysis stratified by quality	4 (8.5%)
Trials weighted by quality	3 (6.4%)
Quality included in regression analysis	3 (6.4%)

Discussion: Our findings indicate that there is widespread uncertainty about how best to assess the quality of primary studies. Methodological research is needed to clarify this important issue.

Rome 1999 PA17