The four speakers in each of the daily plenary sessions addressed the theme for that day.
Chair: Chris Silagy
Andy Oxman
Editor of the Cochrane Collaboration Handbook
A brief history of the Cochrane Database of Systematic Reviews (CDSR)
At the Cochrane Colloquium in 1995 Dave Sackett described the Cochrane Collaboration as an airplane that took off while it is still being built. In this 15 minute presentation I will examine the contents of the Cochrane Database of Systematic Reviews in an historical context: taking into account where the journey began, where we are relative to our destination, and the uncertainties that arise from travelling in a vehicle that is still under construction.
Clive Adams
Co-ordinating Editor, Cochrane Schizophrenia Group
The Cochrane Schizophrenia Group (CSG) will be wearing something old, something new, something borrowed, and something blue. Electroconvulsant therapy (ECT) is an old treatment for schizophrenia but, in global terms, is still a widely used intervention that has just been evaluated in a systematic review within the CSG module. Little changed between the '50s and the last 1980s in the drug treatment of those with schizophrenia until the use of clozapine (something new) for people with illnesses that were resistant to drug treatment. This new drug is also subject to a systematic review and this is being undertaken by reviewers from Syria, Finland and Russia. For our first review the CSG borrowed a journal-published review of family intervention for those with schizophrenia. This review has evolved within the Cochrane Library and provides a good example of the importance of regular maintenance of a systematic review. Lastly, a review of community care policy (case management) - now a statutory obligation within the UK - will be displayed. The surprising results of this review have caused the 'phones to ring in Whitehall (something blue!)
Hazel Fraser
Administrator, Cochrane Stroke Group
Experiences of the Stroke Group
Clinical practice should, where possible, be based on comprehensive, unbiased, and up-to-date systematic reviews of primary research. While these reviews sometimes show clear evidence of a positive, negative, or no treatment effect, more often than not they highlight areas of uncertainty where further research is needed, and also help to improve the quality of such research.
Members of the Stroke Group have produced systematic reviews which (i) have directly changed clinical practice within the UK (Stroke Units); (ii) have directly influenced research (Anticoagulants in acute stroke); (iii) have shown no evidence to support the wide use of some interventions (Glycerol in acute stroke, Hemodilution in acute stroke); and (iv) show the necessity for further research where there is an indication that benefit may exist (Secondary prevention of stroke in patients with nonrheumatic atrial fibrillation - anticoagulants vs antiplatlets).
The experiences of the Stroke Group show that, whatever the conclusion of a systematic review, there is always something to be learned from it, and while it may pose more questions than it answers, at the very least, it is a useful tool for pointing the way towards more effective research.
Marc Keirse
Editor, Cochrane Pregnancy and Childbirth Group
The Pregnancy and Childbirth Group
When a discipline, as is the case for maternity care, has been through the entire exercise of discovering as many RCTs as possible and systematically reviewing them as well as possible, the scope for year to year advances is limited. Most of us probably die with 90% of the wisdom acquired in early life intact. There is no reason to expect that any field of medicine would be different from those who practise it.
Major contributions and innovations in the health care of such a discipline therefore tend to come from five sources:
The Pregnancy and Childbirth Group has witnessed and will report on all five of these contributions. In doing so it acknowledges that most of its efforts are still directed toward adapting its widely acclaimed achievements in maternity care to a changing module that may service the needs of patients, consumers, and health care workers alike.
Chair: Andy Oxman
Les Irwig
Department of Public Health & Community Medicine
University of Sydney
RCTs and other forms of evidence
Cochrane's original challenge was to organise "a critical summary........ of all relevant randomised controlled trials". In the preface to the Cochrane Collaboration Handbook, a bracketed phrase has been added: "(and, when RCTs are not available, reviews of the most reliable evidence from other sources)". In similar vein, the aims of the Collaboration are expressed in the handbook as including systematic reviews of randomised controlled trials of the effects of health care "and of other evidence when appropriate". These quotes raise several questions. Should Cochrane reviews ever be based on non-randomised evidence? And when is other evidence appropriate to help collaborative groups draw "conclusions about how their new results should be applied in health care and, as it says in the recent advert for the Cochrane Library provide "the information you need to make the best health choices"?
Based on a simple decision analytic model (eg Glasziou and Irwig, BMJ 1995; 311: 1356-9) the information we need includes 2 important components:
Effects of health care. The benefits and harms of an intervention are best determined by randomised trials. Extreme caution is required before accepting any other evidence about the effects of health care. Where other evidence is considered, criteria for quality assessment are especially important, for example those developed by the Cochrane Collaboration on Effective Professional Practice Collaborative Review Group. In addition, the effects of study type and quality on measures of effect should be assessed during analysis.
Applicability. Where the magnitude of benefits and harms are known from systematic review of randomised trials, their applicability to individuals or groups of people requires assessment of expected event rates for all relevant outcomes in the absence of the intervention. For this purpose, which includes issues of prognosis and diagnosis, observational studies are indeed the appropriate design.
The question for the Collaboration to consider is not simply 'which evidence?' The questions are:
When, if ever, should non-randomised evidence be systematically reviewed to determine the effects of an intervention?
Should the Collaboration extend its activities into the area of applicability, for which observational studies of prognosis and diagnosis are a critical component? If so, when should that be done?
Jeanne Daly
La Trobe University
Systematic Reviews in Social Context
The randomised trial is a beautifully systematic methodological device. However, problems can arise if we pay too little attention to the research context within which trials are conducted and the social context within which the results of systematic reviews are to be implemented.
In any important field of health research there is usually a considerable body of knowledge which needs to be reviewed in a systematic manner. Included here may be trials of various degrees of rigour but also research which uses other research methods, including qualitative methods. In this presentation I shall argue that this full body knowledge needs to be included in reviews. Qualitative methods provide not only a method of research but may also be applied to develop a rigorous approach to the systematic categorisation and analysis of a range of studies which use different methodological approaches.
Finally, knowledge gained from systematic review in a field does not stand alone. These reviews need to be applied to the everyday context in which people experience health problems so that the reviews inform researchers and community alike.
Brian Haynes
Director, Canadian Cochrane Centre
The issue of which evidence should be reviewed by the Cochrane Collaboration has no right answer! It is easy to support in good conscience, a "hard line" position of only randomised controlled trials, on the several grounds-
First, as almost all would agree, randomised controlled trials are the least biased form of assessment of health care interventions. Second, there are many randomised trials that have yet to be surnmarised. Third, until justice is done by these strongest-of-evidence studies, it does not make sense (and is possibly unethical) to consider other forms of evidence. Thus, one could conclude that randomised trials must be summarised first; when done, we can move on to other types of investigations.
It is also easy to support a contrary view. In some fields, the RCTs have already been summarised adequately to know what the current best treatments are and, at the least, we should avoid becoming compulsive, especially if that means foregoing systematic review of evidence concerning who has the disorder of interest (diagnosis) and who is most likely to respond to treatment (prognosis), particularly when this information is essential in the ethical allocation and application of limited resources. Further, for specific fields, such as public health and occupational risk and injury, randomised trials are unfeasible for some interventions and environments, yet quasi-experimental evidence is abundant and may provide the best evidence that will ever be available. Further, there are practitioners in these fields who agree with the principles of the Cochrane Collaboration, and who are willing and able to review such evidence. Should these people form a separate group from Cochrane (as has been suggested to be necessary for occupational injury, for example)? Or should they be invited to join methods groups to develop the criteria for reviewing such evidence (as for diagnosis), then granted "membership" to form review groups?
But then, does the Cochrane Collaboration have the organizational capacity to sustain enough review groups to summarise all RCTs?
It seems to me that the question comes down to one of organization, resources, and will to proceed, rather than 'yes' or 'no' on the principles of classes of evidence. Of course we should summarise the strongest available evidence about the detection, prevention and management of health problems, whether this evidence is from RCTs or not.
Can we do so? I think we should work towards this goal, but suspect that we could lose the whole enterprise by moving too quickly in this direction, without a clear and reasonable plan. We should encourage separate leadership to develop such a plan. If there is good leadership, we should proceed to the next steps.
Hilda Bastian
Co-ordinator, Consumer Network
What should go into a Cochrane Review?
What attracted a lot of consumers (and indeed others) to the Cochrane Collaboration wasn't the vision of collecting every randomised controlled trial (RCT) under the sun: it was the broader concept of being able to make decisions "in the light of the best available evidence". And it wasn't just the potential of a database of meta-analyses: it was the example of the Collaboration's predecessors in print, Effective Care in Pregnancy and Childbirth and its offspring Guide. Those books were always about much more than just RCTs, important as the trials were.
In at least some contrast, the Collaboration has focused almost entirely on the RCT component. We are often warned about the dangers of stepping outside this boundary. However, we also need to consider what dangers there might be in maintaining too exclusive a focus on randomised controlled trials.
Cochrane reviews should include the least biased evidence, and they need to be systematic. For many important issues, this will not mean RCTs. For example, RCTs, when we as patients are not "blind" to the treatment we are getting, are not always the best and least biased form of evidence on how we feel. Can the preparation of systematic reviews and consideration of the results ever be adequate if they are based on nothing else then what is revealed by the typical Cochrane search strategy? Or if the other bits and pieces that are used to understand the overall picture have been gathered with little or no attempt at being systematic?
People often ask if we can afford to extend the Collaboration beyond the RCT. We also need to consider whether we can afford not to do so.
Chair: Margaret Haugh
Iain Chalmers
Director, UK Cochrane Centre
"Some introductory observations"
The now classic papers published by Cynthia Mulrow and Andy Oxman in the late 1980s heralded an increasing acknowledgment that the science of research synthesis which had been developed by psychologists during the 19870s should be brought to bear within the field of health care. The potentially lethal consequences of reviews which failed to respect scientific principles shown in the analysis published by Elliott Antman, Joe Lau, Bruce Kupelnick, Fred Mosteller and Tom Chalmers in 1992 illustrated the potential magnitude of the human costs of continued failure to get to grips with the agenda that Archie Cochrane outlined in 1979.
Those contributing to the Cochrane Collaboration - and many others - are now beginning to respond to these challenges. Presentations by others within this plenary session will describe how two of the underlying principles of the Collaboration - avoiding bias, and keeping up to date - are being pursued. In my introductory remarks I will take up three themes which I think are relevant to the quality of Cochrane reviews: the social challenge facing collaborative review groups (CRGs); the key role of specialised trials registers in servicing the members of CRGs; and the existing and potential advantages of electronic publishing.
Lisa Bero
Co-Director, San Francisco Cochrane Centre
Electronic peer review and updating of Cochrane reviews
Electronic publication can change the process of peer review traditionally used by print journals (Chalmers, 1991). The electronic publication of Cochrane reviews offers several opportunities for improved quality compared to print publication: it allows for a potentially larger pool of peer referees; it allows for the modification and updating of the review that was originally published; and it allows the authors of a review more than one or two chances to respond to criticism (Bero and Rennie, 1995).
In my plenary remarks, I will describe the mechanisms being developed by the San Francisco Cochrane Centre to take advantage of the opportunities offered by electronic publication. We are developing mechanisms for users of the database to send criticism electronically while reading the database; for reviewers to respond to criticism; for cataloguing and displaying criticisms and responses in the database; and for monitoring the impact on quality of the criticism management system. The writing of new software for the electronic criticism management system has involved consultation with experts on electronic communication, users of the database, and reviewers. As the goal of the criticism management system is to improve the quality of the reviews, we continue to welcome input about how to solicit, respond to and incorporate constructive criticism of Cochrane reviews.
Kay Dickersin
Director, Baltimore Cochrane Center
Completeness of Evidence
The practice of evidence-based medicine relies on valid and reliable reviews. While much improvement has, been made toward educating the users and creators of reviews regarding the importance of a systematic approach, relatively little progress has been made toward ensuring completeness of reviews. Although we know that results from up to half of studies undertaken are never published, we have not created a system for identifying all trials undertaken. Efforts are underway around the world to establish such registers of trials, yet progress has been very slow. In addition, complete identification of the published literature remains problematic.
The Cochrane Collaboration is working toward comprehensive identification of all initiated trials and the creation of registers of trials. Over 700 collaborators are searching the published literature from 1948 forward and to date have identified and retagged over 50,000 randomized and Controlled trials on MEDLINE. Trials reports not on MEDLINE will be tagged in a similar manner and added to an ancillary d database. The Cochrane Controlled Trials Register is a composite register of MEDLINE-indexed trials and trials not on the MEDLINE database. The Trials Registers Development Group, convened in July 1996, is coordinating efforts to build user friendly, accessible registers of all trials.
Edward Dickinson
Co-ordinator, Cochrane Field - Health Care of Older People
Managing outside input
This presentation will consider what can be learnt from successful organisations in relation to high quality products and services. The context is the unparalleled change in the way that services are designed and delivered - thriving organisations have learnt to focus on quality. The biggest switch has been from a "product" focus to a "market" focus - understanding the needs of customers is the key to success.
Some worry that this approach may compromise the technical aspects of products or services which become the "whim" of the customer. On the contrary, customers assess the quality of services according to a number of dimensions and amongst these there is usually technical excellence - customers often expect reliability and safety as a given.
Managing outside input is concerned with the interfaces between the operating core (the steel mill, the car production line, the banking machine room, the software compiler), the rest of the organisation and the outside world. In practice, several models might be, useful; for example. the Gap Service Quality Model is an accessible tool for designing high quality services; also the Value Chain helps us understand the linkages which are the key to successful differentiation.
Strategically, implementing these ideas requires vision, top level commitment and excellent supportive management.
Chair: Cornelius Baker
Jeremy Grimshaw
Department of Public Health
Programme Director, Health Services Research Unit
Aberdeen, UK
Clinical Guidelines, systematic reviews and effective professional practice
Clinical guidelines and systematic reviews are commonly used methods of identifying effective treatments. However, there are philosophical differences between guidelines and systematic reviews. The presentation will consider the similarities, differences and relationship between guidelines and systematic reviews, and the importance of active implementation of guidelines and systematic revews to promote effective professional practice.
David Henry
Faculty of Medicine and Health Sciences
The University of Newcastle
Using the data in subsidation decisions regarding pharmaceutical products in Australia
Drug subsidation programs around the world have come under increasing pressure because of the high costs of modern pharmaceuticals. In Australia, subsidation of medication use in the community is a federal government function, and since 1993 applications to list new products on the Pharmaceutical Benefits Scheme have had to compare the cost-effectiveness of the new product with drugs already in use for the relevant indications. Since 1995 the Australian cost-effectiveness guidelines have stated a preference for therapeutic claims to be based on the results of systematic reviews of randomised trials. Sponsors are encouraged to perform initial trial-based economic analyses, and to use trial-based estimates of treatment effect in subsequent economic modelling. This approach has been criticised as setting 'unreasonable........... evidentiary demands'.
The operation of this program has provided unique experience of the use of evidence from trials and systematic reviews in policy-making. The following observations can be made:
In general, drug regulatory systems have relied on evaluation of single trials that are regarded as 'pivotal', and have not viewed evidence as a continuum. The new requirement in Australia for the conduct of systematic reviews has highlighted weaknesses in the methodological skills of personnel working in government and industry. This has clear implications for training.
In summarising the data for use in policy-making, a single measure of effect size derived from a conventional meta-analysis is sub-optimal. This has implications for the presentation of data in the CDSR. To develop valid approaches to the application of data it is necessary to investigate and explain the range of effect sizes that are seen in a collection of trials. Another consideration is whether absolute or relative measures of benefit and risk should be incorporated into the economic models.
Although the Australian approach to cost-effectiveness assessment uses a hierarchical approach to the classification of data, with a clear stated preference for unconfounded data from 'head to head' randomised trials, it is often necessary to base decisions on poor quality evidence - eg single arm (uncontrolled) phase II studies of new anti-cancer drugs. Despite the availability of adequate resources some groups of new drugs are not being subjected to adequate testing in phase I and phase II studies.
The conduct of economic analysis frequently exposes the inadequacy of clinical 'outcomes' that are used in clinical trials. There is an urgent need for trials to capture experiences that are important to patients.
Varying cost-effectiveness 'thresholds' may have to be used in different settings to allow for the perceived importance of providing therapy to particular groups of subjects. This is important when there is no alternative treatment available and the disease is disabling. In making such judgements it is helpful to have access to the explicit preferences of the community.
Mark Ragg
Independent Medical Journalist
Australia
The Cochrane and the media
The general public believes journalists are about as trustworthy as used car salesmen. Professionals have an even lower opinion, rating us down with politicians. Yet we are an important conduit for health information not just for the public, but for other health professionals. So use us.
Specialist health journalists have a tremendous enthusiasm for evidence-based medicine. Partly this is because we like research-based findings, and partly it is because we realised, even earlier than many doctors, how much medicine was based on guesswork, anecdote and staring longingly at the ceiling, waiting for inspiration.
It can be very confusing, time-consuming and expensive to use the media through the appropriate channels of institutional public relations officers and companies, launches, press conferences and press releases. Fortunately, technological advances have provided a solution to the problems faced by doctors in getting in touch with the media. This wonderful technology will be described, and its uses explained.
Andrew Penn
Associate Professor, Division of Neurology
Head of Research and Development
Synapse Publishing
Using and obtaining the evidence in synchrony: Disease Guidance Systems
StrokeNet is an on-line "Disease Guidance System" designed to help optimize the management of stroke patients, irrespective of the clinical setting. It is built using a publishing system that allows for update of content over the Internet and also allows the flow of outcome information back as part of clinical trials or registries. The intent therefore is twofold: to disseminate information on best practice and to collect information that helps refine our knowledge of what best practice is.
StrokeNet helps the physician perform various tasks, the most important of which is generating sets of patient orders. In so doing it enables a second task, the generation of a patient summary. Lastly it allows the tracking of patient outcomes for personal professional or institutional purposes or within the context of registries and clinical trials. Content is maintained by an international editorial board and is planned for implementation in a number of countries. A long term goal of the system is to provide a substantiated model of best practice upon which to base institutional and regional planning for stroke.
StrokeNet is a project within the G7 Global Health Application initiative and the Canadian Network of Centers of Excellence, HEALNet, focusing on evidence-based decision making in healthcare.
There are a series of other Disease Guidance Systems presently being constructed.
This page was last updated on 15 November
Comments for improvement or correction are welcome.
georg.koch@cochrane.de