Methods Groups Newsletter Volume 4 June 2000

The inaugural meeting of the Campbell Collaboration was held in Philadelphia, USA, on 24-25 February 2000. Patterned after the Cochrane Collaboration and championed by many of the same people, the Campbell Collaboration is an emerging international organisation that aims to help policy makers and practitioners make informed decisions in the area of public policy by preparing, maintaining, and promoting access to systematic reviews of studies on the effects of social interventions and educational practices.

The Campbell Collaboration was named after the American psychologist and methodologist, Donald Campbell, who drew attention to the need for society to assess more rigorously the effects of social and educational experiments. These experiments relate to schooling, delinquency and criminal justice, mental health, welfare, housing, and employment.

Over 80 people from North America and Europe attended the meeting. The meeting was a follow-up to earlier organisational meetings that took place in July 1999 in London and in December in Stockholm. In addition to general sessions, the meeting began the process of developing Campbell Review Groups. Attendees interested in education, crime and justice, and social welfare, met in breakout groups to define their scope and start building an organisational infrastructure. Review Groups in other areas are expected to emerge in coming years, including in the area of nursing. Breakout groups also met to discuss the organisation’s needs concerning primary research and systematic review methods, and software and dissemination.

The beginning of a database of trials in public policy areas similar to The Cochrane Controlled Trials Register already exists. Called SPECTR, the database is available on CD-ROM from Update Software (www.update-software.com).

Much time was spent at the February meeting in examining ways in which the Cochrane and Campbell Collaborations could co-operate so as to share scarce resources and avoid duplication. This issue was especially salient to the Methods Groups because of the considerable overlap in methods used by medical and behavioural scientists. The Campbell Methods Group established a working committee of four members who will be joined by a similar group from the Cochrane Collaboration to look at ways to integrate activities, where appropriate.

A Steering Committee for the Campbell Collaboration was elected and will be chaired by Robert Boruch, from the University of Pennsylvania Graduate School of Education. Additional information on the Campbell Collaboration can be obtained by visiting our Internet site at www.learning.gse.upenn.edu/campbell/intro.html, or by emailing Robert Boruch at robertb@gse.upenn.edu.

There is considerable interest in the contribution that the findings from qualitative research have to make to improving the quality and relevance of systematic reviews of the effectiveness of health care. It has been argued, for example, that inclusion of such findings can influence the criteria by which effectiveness is judged and hence the data to be extracted from trials; provide a research-based context for interpreting inconclusive trial results; and contribute to understanding of heterogeneity in outcomes. Others have argued that process evaluations, which usually involve but are not confined to qualitative methods, should be included in systematic reviews where possible. There is also a role for qualitative research to contribute to methodological developments associated with systematic reviews in other ways. Finally, there is increasing interest in the development of methods for the systematic review of qualitative research itself (or studies based on a plurality of research methods) to answer questions other than those concerned with effectiveness.

However, the intrepid travellers who are exploring the scope for including studies using (but not necessarily confined to) qualitative methods within a systematic review have highlighted important methodological issues that need to be resolved before significant progress will be made. These include questions concerning the search strategy to be adopted, the ways in which quality is to be appraised, and how findings from studies using different qualitative methods can be synthesised. A separate but related question is whether, and how, qualitative research findings could be incorporated into Cochrane, or similar systematic, reviews. There have been useful developments on all these fronts. Strategies to search for studies using qualitative methods and/or other methods such as process evaluations, have been developed at the Social Sciences Research Unit at the Institute of Education in London, at the Centre for Reviews and Dissemination at the University of York, and at the Health Care Practice Research and Development Unit at Salford University. Elsewhere, the University of Rochester has developed a relevant search filter and the New Zealand Health Technology Assessment Clearing House’s search protocol is also useful.

Similarly, there is a rapidly expanding pool of appraisal tools for qualitative research to choose from. However, whilst these do have many common elements there are also many important differences in emphasis and approach. Some of them are also proving difficult to apply in practice. There is clearly an urgent need to develop a consensus around what would constitute a ‘good enough’ appraisal tool for qualitative and/or multi-method studies for use within the Cochrane and Campbell Collaborations. Once these issues have been resolved, there remains the equally complex question of how data synthesis should be approached in relation to qualitative studies and/or process evaluations. There are at least two issues here: whether and how data from different types of qualitative studies can be synthesised, and how the meta-analysis of qualitative and quantitative data might be achieved. Work in the latter area is already under way (for example, by Lilford at the University of Birmingham and by Jones and colleagues at the University of Leicester). In addition, the UK Health Technology Assessment programme will soon commission research aiming to develop methods for the appraisal and review of qualitative studies.

The possible Cochrane Qualitative Methods Group has been collecting information on these and other methodological issues and full details will shortly be available on the Cochrane Qualitative Methods Group Internet site: www.salford.ac.uk/iphrp/cochrane. The embryonic Intervention Process Methods Group of the Campbell Collaboration will have similar interests. The possibility of combining these two groups is being considered. Jennie Popay is currently convening both groups and is keen that people who are working and/or interested in these issues should contact her (J.M.Popay@salford.ac.uk). Many people have already made contact and to them apologies for what some might feel is slow progress in moving forward. A research fellow is soon to be appointed who will be able to devote more time to the work - so look for more rapid progress in the future. In the meantime if you do contact Jennie and you haven’t already sent a brief description of any related work you are involved in, please do so.

Systematic reviews within the Cochrane Collaboration are normally based on experimental studies, mainly randomised trials, as these are considered the best research designs to ensure unbiased results and, when they are well conducted, to control for unknown and unmeasured confounders.

There are, however, several conditions in which an experimental design can be inappropriate, inadequate or even impossible. These conditions are especially prevalent in most fields of alcohol and, particularly, drug treatment research. It is for this reason that the editorial group of the Cochrane Drugs and Alcohol Group encourages the inclusion of observational studies of higher quality within their reviews. This is done in the context of a sensitivity analysis comparing the results with those from randomised trials, and is carried out using explicit rules developed by the Group.

We report here on the approach we have followed for three Cochrane reviews. The first review performed by our Group focuses on the use of Naltrexone in preventing relapse in opioid addicts after detoxification. Eleven studies met the inclusion criteria. The meta-analysis was performed with the available randomised trials. The results were then compared with those of other controlled, but non-randomised studies, to investigate whether they led to different results. The answer was negative and the conclusions were unchanged. This review is now to be updated.

The other experience concerns two proposed reviews, both of which were published on Issue 2, 2000 of The Cochrane Library. The first using Buphrenorphine for the management of opioid withdrawal, and the second using opioid antagonist and adrenergic agonist for the management of opioid withdrawal. In both cases no randomised trials have been found and we know of only a few controlled observational studies. Other studies such as prospective single group studies and case series will be discussed. However, there has been discussion within the Group about the appropriateness of this approach and how best to evaluate the available evidence.

The Group is seeking advice and would like to discuss solutions to this problem. Marica Ferri, the Review Group Co-ordinator, would welcome ideas from people who are working and/or interested in these issues. She can be contacted at dacochrane@asplazio.it.

Methods Groups Modules on The Cochrane Library
Sally Hopewell

This article describes changes to the format of Methods Groups’ modules on The Cochrane Library. The changes are designed to make it clearer and easier for people to obtain information from the modules. Currently some sections of the modules appear long and confusing, and it is sometimes unclear what information should be under which headings. The following changes were discussed and agreed at the Methods Groups Convenors’ meeting at the Colloquium in Rome last year, and will be implemented when a new version of the software used to prepare Methods Groups’ modules is avaliable. In the meantime Methods Groups are encouraged to ensure that their modules contain the information suggested here.

New format for the modules of registered Methods Groups:

Title of the Cochrane Methods Group

Contact details for the Group
Including address, telephone and fax numbers, email and internet address, if applicable.

Date of last module update

Date of registration

Key words

What’s new
To describe any important changes to the module since it was last updated.

Summary
To include a summary of the key points from the module.

Background
To include a brief synthesis of the underlying issues that provide the motivation and rationale for the Group’s existence.

Scope
To highlight the principal aims and objectives of the Group. This should include a description of how the Group will support Cochrane Review Groups (for example, by providing advice to editors, suggesting co-reviewers, or refereeing protocols and reviews).

Topics
To indicate the specific areas of interest and research for the Group.

Contributors
The contributors should be listed alphabetically by name. The list should show the member’s name and their country and should be updated regularly, perhaps through an annual survey of members’ interests (as is done by the Individual Patient Data Methods Group).

Diary
To include dates of past, present and future key events, meetings and workshops – probably arranged as a chronological list.

Reports (Research / Reports)
At present the content of this section varies widely between Methods Groups. It might be most useful if it included short, dated accounts of research and reports produced by the Group. This section should also include an outline of any proposed research to be carried out by the Group, along with expected start and finish dates.

References

Acknowledgements

New format for the information on Possible Methods Groups

Title of the Possible Cochrane Methods Group

Contact details for the Group
Including address, telephone and fax numbers, email and internet address, if applicable.

Summary
To include a brief summary outlining the possible Group’s main aims and objectives. This section should also include information about the group’s progress towards registration.

The Cochrane Collaboration Steering Group meeting was held on 5 and 6 March 2000 in San Antonio. This is a summary of some of the items from the minutes of the meeting which are of particular relevance to Methods Groups. A full copy of the minutes is available on: www.cochrane.de/cochrane/5-6032000.htm.

Business Plan (item 9)

Fenella Rouse has been commissioned to prepare a Business Plan for the Cochrane Collaboration. A working group will be formed to advise Fenella. A draft of the Plan will be sent to entities for their feedback following preliminary approval by the Steering Group by July. The plan will then be revised as needed and finalised following discussion during the Colloquium in Cape Town.

CEO/Business Manager (item 13)

The possibility of employing an executive officer or business manager was discussed, and entity representatives were asked to consult with their constituencies for their views on this position. The following points were made during the Steering Group discussion: two sets of skills are needed, one related to a central operational role, and the other related to fund-raising and external relations, and it may not be possible to find one person with both sets of skills; it may be possible to appoint someone with the necessary operational skills and delegate responsibilities for fund-raising and external relations; the person who is appointed should fully appreciate and support the ethos of the Collaboration; an important element of the ethos is that this is a highly collaborative organisation; and the job description should reflect the collaborative, non-hierarchical nature of the Collaboration. The job description will be developed in parallel with the Business Plan.

Cochrane-Campbell Methods Groups (item 17)

The Campbell Collaboration held its inaugural meeting in Philadelphia in February 2000. A process needs to be put in place to move forward with the issues in the discussion paper on joint Methods Groups. A working party has been established to do this within the Campbell Collaboration and a similar 4-member working party needs to be created within the Cochrane Collaboration. At the Campbell meeting, Empirical Methodological Studies, Non-randomised Studies, and Process Evaluation were identified as three Methods Groups that might potentially become joint between the two Collaborations quite soon.

Cape Town Colloquium (item 31.2)

There should be a meeting between Cochrane Review Groups and Methods Groups.

Monitoring of Methods Groups (items 21.6 and 27.2.2)

In future, it will be pre-specified that material gathered from Methods Groups during the annual monitoring process would be shared with their reference Cochrane Centre. Each Group should provide a brief statement of the specific ways in which they are able to help members of the Collaboration. In common with the monitoring and evaluation checklists for other entities, a question about the number of active participants from developing countries will be added to the monitoring checklist for Methods Groups.

Quality Advisory Group (item 10.1)

A new advisory group (concerned with the quality of Cochrane reviews) has been established. The Group is co-convened by Philippa Middleton and Pim Assendelft.

Cochrane Reviewers' Handbook (items 8.1, 18 and 19.5)

A revised version of the Handbook will be released in 2000 to coincide with the release of RevMan 4.1. Philippa Middleton agreed to join the Handbook Advisory Group. Guidelines will be added to the Handbook regarding the conversion into Cochrane Reviews of existing reviews using individual patient data, and the change in policy on the approval of synopses for Cochrane reviews.

Quality Advisory Group
Pim Assendelft and Philippa Middleton (Co-convenors)

The Quality Advisory Group was established by the Cochrane Collaboration Steering Group in February 2000. We are now forming the Group and are proposing our initial priorities for co-ordination and action.

The remit of the Quality Advisory Group is to co-ordinate activities aimed at improving the quality of Cochrane reviews. It will try to identify and follow up issues of quality that are relevant to Cochrane reviews. The Quality Advisory Group will also provide advice on the development of standards and tools for assessing the quality of Cochrane reviews. Given the large amount of potential work in this area, the Group will also help to identify priority areas for quality related activities. Both methodological and technical (copy) editing aspects of improving the quality of Cochrane reviews will be addressed.

The Quality Advisory Group will be accountable to the Cochrane Collaboration Steering Group through the Executive Group. The co-convenors (Pim Assendelft and Philippa Middleton) were nominated by the Chair of the Steering Group, and approved by the full Steering Group. Group members, to be nominated by the co-convenors of the Group, and approved by the Executive of the Steering Group, will include members of the Software Development Group, Handbook Group, Criticism Management Group, representatives from Methods Groups, Centres, Update Software, Fields, Consumer Network, two Steering Group members, a co-ordinating editor, a Review Group editor (not a co-ordinating editor), a Review Group Co-ordinator, a reviewer, and additional members appointed on the basis of their demonstrated interest and activity in the area. The Group plan to have yearly meetings at Cochrane Colloquia. There are plans for an email discussion list and, possibly, additional email discussion lists to deal with particular projects.

Recent work to identify additional records for the Cochrane Methodology Register
Mike Clarke

This article describes a recent project at the UK Cochrane Centre, which used a small grant from the UK National Health Service Research and Development Programme to enhance the content of the Cochrane Methodology Register (CMR). The broad intention for the CMR is to include empirical studies and reports of methods relevant to systematic reviews and evaluations of health care or social interventions. Unfortunately, it has proved extremely difficult to identify relevant articles. Instead, the CMR has grown from the bibliographic collections of individuals; ad hoc searching of electronic databases such as MEDLINE and EMBASE; and prospective searching of selected journals (such as the BMJ and JAMA). The recent project took a more systematic approach and involved the handsearching of two journals that were known to contain a large number of reports of methodological studies (Controlled Clinical Trials and Statistics in Medicine), and the development of a series of search strategies for MEDLINE.

Controlled Clinical Trials was searched from its first appearance in 1980 to the end of 1999. This represents a total of 100 issues and in addition to full articles and correspondence, the searching included the abstracts for the annual meeting of the Society of Clinical Trials. These abstracts proved an extremely rich source of relevant material, none of which are currently indexed in MEDLINE or any other database. The searching of the journal identified 395 methodological reports, less than 10% of which were already in the CMR. Statistics in Medicine was also searched from its appearance, in 1982, to the end of 1999. This journal is published more frequently than Controlled Clinical Trials, so a total of 276 issues were searched. 292 methodological articles were identified and, again, only about 10% of these were already in the CMR.

A series of search strategies were developed for MEDLINE, which were deliberately designed to be sensitive rather than specific. It was expected that a large proportion of the records retrieved would not be relevant to the CMR but that a reasonable number of articles that were relevant would be retrieved. These search strategies retrieved 23,546 MEDLINE records. The titles and abstracts of all of these were checked, and 977 reports that might relate to studies of relevant methodological issues have been identified, 36 of which were duplicates from the handsearching of Controlled Clinical Trials and Statistics in Medicine.

All the records that have been newly identified (1628) will be transferred to the CMR in time for inclusion in Issue 4, 2000 of The Cochrane Library. This will more than double the number of records in the CMR (currently 1312).

In addition to identifying the methodological articles, the handsearching of Controlled Clinical Trials and Statistics in Medicine and the assessment of the MEDLINE records also sought reports of definite or possible randomised and quasi-randomised trials. The handsearching identified 485 reports, of which 192 do not have a MEDLINE record and, of those that do, 156 had not been indexed with the Publication Type RANDOMIZED CONTROLLED TRIAL or CONTROLLED CLINICAL TRIAL. The MEDLINE part of the project identified 100 records that were not already indexed with either of these Publication Type terms. The 448 records that are not already correctly indexed in MEDLINE will be passed to the New England Cochrane Center, Providence Office for inclusion in the Cochrane Controlled Trials Register and for re-tagging in MEDLINE.

Thanks are due to Teresa Clarke, Anne Lusher and Inekke Loke in Oxford for identifying and processing these records, and to Richard Lilford at the University of Birmingham for making the grant available.

How can Methods Groups best interact with Cochrane Review Groups
Sally Hopewell and Andrea Nelson

This article aims to give suggestions and to generate discussion on how Methods Groups and Review Groups can work together better to improve the quality of systematic reviews and to identify the best methods for assembling, appraising and synthesising information.

One suggestion is an Internet site containing contact details, areas of expertise and a section on "questions we are willing to answer" for each Methods Group. Some Methods Groups already have their own Internet sites, for example the Prospective Meta-analysis and the Non-randomised Studies Methods Groups. The possibility of combining specific information from these and other sites into a single source of reference could be explored.

Many questions asked by Review Groups and reviewers may well have been asked previously by others, for example how to choose between using odds ratios, relative risks or risk differences in a Cochrane review. The provision of a consensus answer to questions such as these would be helpful, and the inclusion of a "Frequently Asked Questions" section on the Internet site could help to address some of these issues.

The revision of Methods Groups’ modules on The Cochrane Library (which is planned when the ModMan software is next updated) also aims to make it clearer and easier for people to obtain information on individual Methods Groups (see article on page 5). The new modules will encourage Methods Groups to define their role and scope within the Collaboration more clearly and to identify areas of specific expertise and interest.

Another suggestion is for a meeting between Methods Groups and Review Groups at Cochrane Colloquia. However, given the huge demands on time at these events, the agenda for such a meeting will need to be clearly defined. A possible format would involve presentations from individual Methods Groups highlighting their areas of work and discussing how they are able to help Review Groups and reviewers. This could be followed by a more open discussion, and a question and answer session between people from Review Groups and Methods Groups.

Strategies also need to be developed to encourage Cochrane entities to seek the advice of Methods Groups. Alongside this, each Methods Group should consider developing standardised methods of dealing with requests from other entities, which will include the implementation of a plan of action following such requests. Suggestions and ideas on how best to take these issues forward would be most welcome: please contact one of the authors or Mike Clarke (Convenor of the Methods Groups).

The new PubMed
Kathi Canese

PubMed (www.ncbi.nlm.nih.gov/entrez/query.fcgi) is an Internet-based retrieval service of the National Center for Biotechnology Information (NCBI), a division of the US National Library of Medicine (NLM), which provides free access to MEDLINE and other related databases. It offers a simple search interface, specialised clinical searches with built-in quality filters based on research methods, a related articles function, links to outside Internet sites through LinkOut, document ordering, and a citation matching tool for creating links to PubMed.

A new version of PubMed was released in September 1999. This version includes a Features Bar with links for Limits, Preview/Index, History, and Clipboard. The Limitsfeature includes pull-down menus by Publication Type, Language, Subsets, Ages, Human or Animal, Gender, and Publication Date. Preview/Index provides two different functions. Index displays an alphabetic list of terms within a search field (e.g. MeSH Terms, Author Names) and allows users to refine a search from these fields. Preview lists the number of results before displaying the citations. The History feature stores each query as a numbered search statement that can be used in new strategies. The Clipboard is a holding place that allows users to view or temporarily save selected citations from one search or several searches so that these can be printed, saved to a file, or ordered.

LinkOut is a new feature in PubMed whereby URLs are provided by third parties that point back to their Internet sites and resources from specific records such as journal articles, nucleotide or protein sequences, or protein structures. For example, the recently added MEDLINE records for Cochrane reviews contain links to the relevant abstract on the Update Software Internet site. Publishers, aggregators, libraries, biological databases, sequence centres, and other Internet resources can use LinkOut to provide links to their Internet sites.

The new PubMed Books was developed to provide users interested in learning more about an unfamiliar topic with links to reference books for more information. When clicked the link redisplays the citation and abstract with terms as links to sections of the book.

An additional change includes displaying all authors on the PubMed summary format rather than only the first.

Soon Cubby Stored Searches will be available to store and update search strategies. Cubby Stored Searches will only retrieve citations added to PubMed since the last time a user updated their search. Another Cubby service, LinkOut Preferences, will allow users to customise their LinkOut display, so that they will only see links of interest to them.

Additional information:

PubMed Overview: www.ncbi.nlm.nih.gov/PubMed)

PubMed Help: www.ncbi.nlm.nih.gov/entrez/query/static/help/pmhelp.html

LinkOut Overview:  www.ncbi.nlm.nih.gov/entrez/query/static/linkoutoverview.html

Books Overview: www.ncbi.nlm.nih.gov:80/books/mboc/bookshelp/bookover.html