Do we need comprehensive literature searches? A study Of publication and language bias in meta-analyses of controlled trials.

Bartlett C., Egger M., JüniP., Sterne J.A.C. MRC Health Services Research Collaboration, Department of Social Medicine, University of Bristol, UK

Comprehensive literature searches are widely recommended to minimise bias in meta-analysis. Such searches are, however, time-consuming and costly. We studied the impact of unpublished and non-English language trials on effect estimates in a sample of meta-analyses. Methods: We searched for meta-analyses in journals and the Cochrane Database of Systematic Reviews that combined (5 trials, had binary outcomes and employed comprehensive literature searches. We identified 159 meta-analyses, 60 of which included trials from grey literature (such as proceedings, theses, books, unpublished data) and 50 of which included trials published in non-English languages. We used the analytical method of the original meta-analysis to assess the impact of excluding unpublished or non-English language trials. Results: Where present, grey literature contributed an average of 22% of the trials and 18% of the weight. Treatment effect estimates in unpublished trials were on average 7% (95% CI -2% to 18%) less beneficial than in the published trials. We calculated the change in combined treatment effect estimates when grey trials were excluded. In 41 (68%) meta-analyses the changes were <5%. In the 19 meta-analyses in which the change in estimate was (5%, 9 showed increased and 8 showed decreased benefit. Among published trials those published in non-English language journals contributed an average 21% of the trials and 17% of the weight. Treatment effect estimates were on average 16% (3% to 26%) more beneficial in non-English trials. In 29 meta-analyses (58%) the changes were <5% when excluding non-English trials. In the 21 in which the change in effect estimate was (5% 5 showed increased and 16 showed decreased benefit. Conclusions: Exclusion of grey literature tended to increase estimated treatment benefits but the variability between meta-analyses means that the effect of ignoring grey literature is unpredictable. In general, exclusion of non-English literature decreased estimated treatment benefits, although again there was substantial between-meta-analysis heterogeneity. This is in contrast to previous studies which suggested that in conventional medicine exclusion of non-English language trials tends to increase estimated treatment benefits.

Cape Town O6