Publication bias and meta-analysis: A practical example

Burdett S. and Stewart L.A. Meta-analysis Group, Clinical Trials Unit, London, UK

There is strong evidence of publication bias, whereby the results of positive trials are more likely to be published than negative or inconclusive ones. Considerable time and effort is needed to locate and obtain data from unpublished trials. However excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions. Objectives To explore and quantify the impact of trials either unpublished, or published only as abstracts, on the results of all completed individual patient data (IPD) meta-analyses coordinated by our group (5 meta-analysis projects, 12 therapeutic comparisons). Methods For each of the comparisons within the IPD meta-analysis projects, results were calculated for (A) fully published trials only, (B) published trials + abstracts, and (C) published trials + abstracts + unpublished trials. All analyses were calculated from IPD. There is therefore no difference in type of data available between trials and no confounding factors in terms of excluded patients or duration of follow up. Results Published trials only Published trials + abstracts Published trials + abstracts + unpublished trials HR (95% CI) Trials Patients HR (95% CI) Trials Patients HR (95% CI) Trials Patients OVARIAN 1 0.92 (0.85-1.00) 15 2899 0.94 (0.86-1.01) 17 3030 0.94 (0.87-1.01) 19 3146 2 0.89 (0.76-1.04) 7 787 0.90 (0.79-1.02) 9 1120 0.92 (0.82-1.04) 11 1318 3 0.90 (0.80-1.02) 6 1224 - - - 0.91 (0.81-1.02) 8 1408 4 0.87 (0.73-1.03) 4 729 - - - 0.89 (0.76-1.04) 7 925 5 1.10 (0.92-1.31) 6 807 1.05 (0.92-1.19) 9 1723 1.06 (0.94-1.19) 11 2061 BLADDER 0.81 (0.63-1.05) 3 198 0.89 (0.73-1.10) 4 357 0.91 (0.75-1.10) 5 479 NSCLCO 1 0.87 (0.73-1.05) 4 908 0.88 (0.76-1.01) 10 1729 0.88 (0.77-1.00) 12 2312 2 0.95 (0.72-1.25) 2 244 0.95 (0.79-1.14) 4 560 0.94 (0.79-1.11) 6 668 3 0.92 (0.83-1.01) 13 1873 0.91 (0.83-1.00) 14 1954 0.92 (0.85-1.00) 18 2368 4 0.76 (0.66-0.87) 9 964 - - - - - - SOFT TISSUE 0.88 (0.76-1.02) 13 1515 - - - 0.89 (0.76-1.03) 14 1544 PORT 1.13 (0.95-1.34) 6 1294 - - - 1.21 (1.05-1.39) 9 2128 Discussion Analyses based only on trials that were fully published at the time the IPD meta-analyses were conducted, tend to give more favourable results. The addition of data from trials presented only as abstracts or those that were unpublished give less encouraging results and in most cases move the estimated treatment effect towards a null result. In the PORT meta-analysis the less favourable, unpublished results move the result further away from a null effect and increase the estimated harm of treatment. Although the differences in effect size are relatively modest in these examples, the trend is worrying. All systematic reviews should at least identify unpublished trials and where possible, obtain data from them. Ovarian 1 - single non-platinum agent vs non-platinum combination NSCLCO 1 - Surgery vs surgery + chemotherapy Ovarian 2 - single non-platinum agent vs platinum combination NSCLCO 2 - Surgery + radiotherapy vs surgery + radiotherapy + chemotherapy Ovarian 3 - Addition of platinum to a regimen NSCLCO 3 - Radical radiotherapy vs radical radiotherapy + chemotherapy Ovarian 4 - Single agent platinum versus platinum combination NSCLCO 4 - Supportive care vs supportive care + chemotherapy Ovarian 5 - Cisplatin vs carboplatin

Cape Town O11