Methods Groups Newsletter   Volume 4 June 2000

Julian Higgins (on behalf of the First Contact Steering Group)

Lack of suitable data from published reports is an almost universal problem for Cochrane reviewers. Many of us are familiar with obstacles such as unclear methods of allocation concealment, poor information about dropouts and missing standard deviations, among others. Equally problematic is the decision we have to take about whether time and money are well invested in pursuing the missing information. We surveyed Cochrane Review Groups to find out what advice they give to their reviewers. The typical advice was to write a simple letter of request and we wonder if this can be improved upon.

To answer this question we have developed a randomised trial. ‘First Contact’ will compare a simple letter with an intensive approach that uses methods, which appear to work based on evidence from randomised trials within population surveys. Details of the protocol are available on our Internet site (see below). We encourage any Cochrane reviewer uncertain about the best way to make first contact with an investigator to join the trial. Randomisation will take place at the UK Cochrane Centre, and current results from a Bayesian analysis of the accumulating results will be available on-line to inform reviewers both within and outside First Contact.

For more details keep an eye on: www.mrc-bsu.cam.ac.uk/firstcontact or contact: julian.higgins@mrc-bsu.cam.ac.uk .

Tierney JF, Clarke M, Stewart LA. Is there bias in the publication of individual patient data (IPD) meta-analyses? International Journal of Technology Assessment in Health Care 2000; 16:1-11.

There is increasing empirical evidence for the existence of bias in the publication of primary clinical research, with statistically significant results being published more readily, more quickly and in higher impact journals. Meta-analysis of individual patient data (IPD) may represent a gold standard of clinical research, giving the best possible summary of current evidence for a particular question, but publication of these may also be subject to bias. Our study aimed to explore which factors might be associated with publication of IPD meta-analyses, and to identify potential sources of bias.

The responsible investigator of each known IPD meta-analysis in cancer was surveyed by means of a questionnaire to obtain descriptive characteristics of their meta-analysis, the nature of its results and details of its publication history. Published meta-analyses were identified through searches of MEDLINE and CancerLit, and unpublished meta-analyses were identified through contact with all IPD meta-analysts who were known to be working in cancer. A questionnaire was developed based on previous surveys of factors that affect publication of clinical studies, and of authors’ criteria for selecting journals.

Forty-four potentially eligible IPD meta-analyses in cancer were identified, and data were available for 38 of these. There was no clear evidence that unpublished meta-analyses differed from published meta-analyses in the median number of trials, patients, comparisons or endpoints. Also, there was no good evidence that overall publication status of meta-analyses in cancer is dependent on the statistical or clinical significance of the results. However, those meta-analyses with non-significant results did seem to take longer to publish and were published in lower impact journals compared to those with more striking results. Based on these current data, there seems to be no strong association between the results of IPD meta-analyses in cancer and their publication. The more reliable estimates of effect, greater power to detect an effect, and the highly collaborative and international nature of these projects, may ensure that publication arises, irrespective of the results.

Bernadette Kelly and Ann Wales

The Cochrane Health Libraries Group prize is awarded annually in recognition of the use of research evidence to improve patient care. Our prize winning submission in 1999 showed how two distinct types of research evidence had been necessary to achieve significant improvements in prophylaxis of venous thromboembolism at Glasgow Royal Infirmary (UK).

The 1996 National Confidential Enquiry into Perioperative Deaths highlighted the need for improved prophylaxis of deep vein thrombosis (DVT).¹ This was confirmed by a hospital audit which revealed that only a quarter of all medical patients and half of all surgical patients received prophylaxis. The Scottish Intercollegiate Guidelines Network (SIGN) guideline "Prophylaxis of Venous Thromboembolism" was invaluable in addressing this deficiency in patient care. It synthesized the best available research evidence (from controlled trials and meta-analyses) and made recommendations for practice. Using the evidence identified by SIGN, local guidelines were also developed. Unfortunately, initial attempts at introducing the guidelines (using dissemination of information and traditional continuing medical education methods) resulted in no significant increase in the use of prophylaxis for venous thromboembolism.

The failure of these methods highlighted the need to identify evidence on effective implementation strategies. A key systematic review was identified from The Cochrane Library - "Implementing Clinical Practice Guidelines: can guidelines be used to improve clinical practice?".² This synthesized and evaluated the extensive, diverse literature on the question, and made recommendations which were practical as well as being rooted in best evidence. It recommended active professional participation in the educational intervention, which should take place close to the point of clinical decision-making and, ideally, should be integrated into the process of health care delivery. The introduction of a Thrombosis Prevention Co-ordinator for a three-month period, with a multi-faceted role involving educating and prompting clinical staff, achieved this integration of the educational process with clinical decision making. This implementation strategy was highly effective. Approximately three-quarters of medical patients and almost all surgical patients received prophylaxis and these high levels of prophylaxis have been maintained.

The practicalities of incorporating the guidelines confirmed that an evidence-based approach to implementation is as important as initial formulation of the evidence-based recommendations for practice. Our experience reflected several of the conclusions that can be found in the research literature - in particular, the need to base implementation strategies on an understanding of local circumstances and barriers to change. The need to persevere with alternative and multi-faceted implementation strategies was a real-life demonstration of the practical point emphasized by Feder et al that successful guideline implementation is likely to require "time, enthusiasm and resources".³

The effective use of clinical guidelines assumes a new importance in the clinical governance setting. In this context, our practical experience underlines the importance of information sources based on identification, evaluation and synthesis of research evidence (e.g. the SIGN guidelines and The Cochrane Library), to both the production and implementation of recommendations for practice.

References

1. Campling E, Devlin H, Hoile R, Lunn, J. The Report of the National Confidential Enquiry into Perioperative Deaths 1991-92. London: NCEPOD, 1993.

2. NHS Centre for Reviews and Dissemination. Implementing clinical practice guidelines: can guidelines be used to improve clinical practice? Effective Health Care 1994; 8:1-12.

3. Feder G, Eccles M, Grol R, Griffiths C, Grimshaw J. Using clinical guidelines. BMJ 1999; 318:728-730.

Jesse Berlin

Investigating modifiers of treatment effect using meta-analysis: how important is sex and how can we be sure?

Understanding to whom the results of studies of treatments apply is one of the major challenges faced by policy makers. Can the results of those studies be generalized to subjects or types of subjects who did not participate in the original research? Data on this issue are almost non-existent. Despite this, clinicians and regulatory bodies make decisions every day about the applicability of trial results. Some take a conservative approach and assume that the effect seen in the trial applies to everyone eligible for the trial and no one who would have been ineligible. Others engage in multiple subgroup analysis, and extrapolations to groups beyond those included in the trial. There are problems with either approach, but the most fundamental issue is the lack of empirical work to support these, or indeed any other approach.

A group of enthusiasts, including a number of people within the Cochrane Collaboration, are proposing to develop an international collaboration of investigators to answer two questions. Firstly, how often is the effect of treatment modified by subject characteristics? The initial approach to this question will focus on sex and age as effect modifiers, but will also consider clinical factors specific to particular therapies and diseases. Secondly, how often do patient level and group-level meta-analyses of the same subgroup question lead to the same conclusions?

This project will be conducted in a collaborative way, through the establishment of the Empirical Studies of Trials that Estimate Effect Modification (ESTEEM) Collaborative Group. The hope is to gather data from 20-40 individual patient data (IPD) meta-analyses to address the above two questions. Patient-level analyses would be conducted either centrally or by the investigators who performed the original IPD meta-analysis, depending on the preference of those investigators. Group-level analyses would all be conducted centrally.

A number of people have already been contacted about this proposed project. We are hoping that this article will stimulate further interest. A preliminary meeting of interested parties will be held immediately following the Symposium on Systematic Reviews to be held in Oxford in early July 2000. A second meeting will be held during the Cochrane Colloquium in Cape Town in October 2000. Those who wish to become involved should contact Jesse Berlin by email (jberlin@cceb.upenn.edu). We are particularly in need of people who are able to provide data to help answer one or both of the above questions.

Chris Cates

Acute ear infections are the commonest childhood infection to be treated with antibiotics in the community in a variety of countries (including the USA, UK and Norway). The arguments for and against the use of antibiotics have raged for the past decade, but in spite of this a 1998 audit in our practice in Hertfordshire (UK) showed that around 50% of all antibiotics used in children were given for this condition.

Two reviews were published in the BMJ in mid 1997 on this subject. The first was a Cochrane review, which pooled the results of all identified randomised controlled trials comparing antibiotic with placebo for children with acute otitis media.¹ The second did not attempt to pool results and made no secret of the authors' desire to dissuade readers from using antibiotics for this condition.² It did, however, present useful background data on the varied approach to antibiotic use in different countries and pointed out that in the Netherlands, where antibiotics are usually withheld for 3 days, there is much less of a problem with bacteria resistance.

The Cochrane review provided quantitative data in relation to the impact of antibiotic use, and the outcomes were well focussed on issues that matter to patients.¹ Pain is the immediate problem and the review showed no discernible impact of antibiotics when measured at 24 hours. In contrast the antibiotics did make a small but statistically significant difference at 2 to 7 days. The most striking result, however, was that by this stage 85% of the children were pain free, meaning that 23 children need to be treated for one extra to benefit. On the down side, the antibiotics increased the risk of rashes, diarrhoea and vomiting and for every 12 children treated, one extra child suffered in this way.

Within our practice the first step was for the practice team to discuss how we should respond to these findings. As I had some previous experience of successfully increasing the uptake of Tetanus immunisations by giving out leaflets to patients,³ I offered to write a handout for parents summarising the evidence from the Cochrane review.

Having prepared the handout we gave it to parents of children with acute otitis media who are not unduly ill so they had access to the evidence when they need it, and we explained why we are no longer routinely giving antibiotics. In effect we were asking the parents to carry out an "N of 1" trial on their child who had previously got better using antibiotics and was now being tried with pain killers alone. We checked that the parents were giving their child the full dose of painkillers.

In order to give the parents confidence to try the new approach we also gave them a prescription for an antibiotic so that they could take it to a pharmacist if the child did not get better as anticipated. This reflected the dilemma that we shared with the parents: we cannot tell for sure which one of the 23 children (calculated from the results of the Review) will benefit from the antibiotics.

Allowing the doctors in our practice to set their own level for deciding when to give antibiotics enabled each of them to adopt the new policy at the rate they felt comfortable with, and the deferred prescription also gave freedom of choice to the parents. A small minority went straight to a pharmacist and started the antibiotic course at once, but those who waited usually did not need to use the prescription at all.

The resultant drop in antibiotic prescribing in our practice was compared to a local control practice and to national prescribing data. The results were published last year. We achieved a 20% reduction in total antibiotic use in children compared to national figures of 4%.⁴

Even the parents are talking to their friends about the changes. One of the most encouraging things of all is that, in the last year of data collection, not only has our antibiotic use dropped further but the control practice has followed suit and fallen to the same level.

There has been interest in the handout from all over the world and we have been awarded NHS Beacon status as a result of this project. Further information about the project is available on www.cates.cwc.net.

References

1. Del Mar C, Glasziou P, Hayem M. Are antibiotics indicated as initial treatment for children with acute otitis media? A meta-analysis. BMJ 1997; 314:1526-1529.

2. Froom J, Culpepper L, Jacobs M, DeMelker RA, Green LA, van Buchem L, et al. Antimicrobials for acute otitis media? A review from the International Primary Care Network. BMJ 1997; 315:98-102.

3. Cates C. A handout about tetanus immunisation: influence on immunisation rate in general practice. BMJ 1990; 300:789-790.

4. Cates C. An evidence based approach to reducing antibiotic use in children with acute otitis media: controlled before and after study. BMJ 1999; 318:715-716.

Individual summaries of currently proposed Cochrane Methodology Reviews prepared by the Empirical Methodological Studies Methods Group.

The aim of this review is to summarise comparisons of randomised clinical trials versus non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres. In 1998, the review concluded that failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are. The size of these distortions can be as large as, or larger than, the size of the real effects.

It has long been recognised that completed research is frequently left unpublished. Failure to publish is not only inappropriate scientific conduct, it also influences the information available for interpretation by the scientific community. If research is left randomly unpublished, there is less information available, but that information would be unbiased. The tendency of investigators to submit manuscripts (and of editors and reviewers to accept them) based on the strength and direction of the research findings, is termed publication bias.

The aim of this review is to assess the evidence for publication bias with respect to trials of health care and update a review that was first published in 1997. This previous review concluded that failure to publish is not a random event and that there is evidence of publication bias.

An update of this annotated bibliography is underway by David Moher’s review group, who are looking at ways of assessing the quality of randomised controlled trials (RCTs). In 1995, the review contained information on 25 scales (with 1 to 170 points) and 9 checklists (with 4 to 57 items). The primary aim of these scales and checklists was to indicate the quality of the methodology in RCTs and to indicate if the results of trials were likely to be biased or free of bias. Cochrane reviewers, who must work hard to assess the quality of trials included in reviews, should obtain much useful information from this update.

This review combines data from the reviewer’s study of the subsequent publication of the results of randomised trials initially presented as abstracts at national ophthalmology meetings with data from similar studies, to determine the rate at which abstracts go on to be published in full, and the association between selected study characteristics and full publication. In 1994, the conclusions were that approximately half of all trials initially presented in abstract form are subsequently published as full-length reports. Most of those that will be published appear in full within 2 years. Full publication may be associated with "significant" results and the number of participants in the trials.

Meta-analyses based on individual patient data (IPD MA), in which data on all participants in all relevant randomised trials are centrally collected and re-analysed, have been proposed as the gold standard for systematic reviews. Compared to meta-analyses based on published aggregated data, or aggregate data collected from the trialists, IPD MA allow more statistically powerful analyses of, for example, the time to particular outcomes, different patient subgroups and complex outcomes. The aim of this review is to identify key reasons for the differences between meta-analyses based on IPD and meta-analyses based on trialists’ aggregate data (i.e. data collected from responsible trialists that included as much follow-up as possible on all randomised participants in all relevant trials), and aggregate data extracted from published reports.

Concern about the amount of time spent writing and reviewing grant applications has been expressed by both researchers and funding bodies. The process of grant giving relies heavily on peer review for the assessment of the quality of applications but there appears to be little evidence of the effects of these procedures. A number of criticisms about peer review of grant giving have focused on the reliability of the process and the existence of a number of biases. There is documented evidence of gender bias, and other biases might be related to age, institution, cronyism, discipline, etc. In spite of these concerns and limitations of the peer review of grant applications, little has been done to address aspects such as equity, effectiveness and efficiency of the process.

The aims of this review are to estimate the effects of the processes involved in the peer review of grant giving and to identify the existing evidence on effects of gender, institution and other possible biases in the peer review of grant applications.

Learned societies and journal editors usually base their publication decisions on the views of researchers working on topics related to the study that has been submitted for publication. This system for deciding which research is presented at meetings or published in journals is known as peer review. The use of peer review is assumed to raise the quality of the end-product and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested. This review aims to assess the effects of processes undertaken as part of editorial peer review of original research studies submitted for the publication in biomedical journals.

The contents of peer-reviewed journals should be accurate and complete and should present research findings in a responsible and comprehensible way. Journals try to improve the accessibility and accuracy of information by specifying the format and style of papers that are acceptable and by performing checks on accepted material. Most journals also make an active attempt to "improve" the presentation of papers and ensure that they conform to "house style" through the process of copy editing.

Despite the time and resources devoted to layout and technical editing, most journals do not present any evidence about the effects of their design (e.g. type face, column width) or their house style (e.g. use of abbreviations, presentation of numbers) on legibility, readability or comprehension; nor do they measure the effects of technical editing on the ability of readers to draw conclusions from papers. The aim of this review is to assess the effects of technical editing of papers accepted for publication in science journals on the paper’s accuracy, consistency, completeness, legibility, readability and comprehensibility.

This systematic review will examine differences in the clinical care and outcomes of similar patients entered in randomised controlled trials (RCTs) and patients who were eligible but not randomised. As patients are not randomly assigned to be inside or outside RCTs (but this is determined rather by interactions of personal, professional and societal factors), the review will only be able to speculate about the putative determinants of why seemingly similar patients fare differently inside than outside RCTs.

RCTs are the gold standard for the assessment of the efficacy and the effectiveness of healthcare interventions. "Outcomes research", using large electronic databases of routinely collected data are being used and promoted increasingly as a substitute for RCTs. Because outcome research relies on observational data that is usually collected retrospectively, it may be at high risk of bias. It is, however, potentially more widely applicable and easier to conduct than RCTs. The aim of this review is to systematically summarise comparisons of the results of RCTs and outcomes studies.