Examining reference lists to find relevant studies for systematic reviews

Systematic reviews are summaries of the information that is available on one topic. The most common way to find information for a systematic review is to search electronic literature databases. To increase the chances of finding important information, researchers can also search the tables of contents of journals, and they can contact experts or organizations for more information on the topic of the review. Another way to find more information is to check through the reference lists of relevant studies to see if these references include reports of other studies that might be eligible for the review. It is important to determine whether or not checking reference lists is a good use of time and resources when conducting systematic reviews.

We found 12 studies that explored whether or not checking reference lists was useful for systematic reviews. These studies reported a range of results, from identifying only a few additional studies (2.5%: 2 of 79 included studies) to identifying many additional studies (42.7%: 111 of 260 included studies) through checking reference lists. Unfortunately, none of the studies looked at how much time or money were spent on the process of checking reference lists, and it was suggested this would be almost impossible to estimate.

Unfortunately our findings are based on weak information. The data do suggest that in situations where researchers may have difficulty locating information, checking through the reference lists may be an important way to reduce the risk of missing relevant information.

Authors' conclusions: 

There is some evidence to support the use of checking reference lists for locating studies in systematic reviews. However, this evidence is derived from weak study designs. In situations where the identification of all relevant studies through handsearching and database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. The challenge, therefore, is for review authors to recognize those situations.

Read the full abstract...
Background: 

Checking reference lists to identify relevant studies for systematic reviews is frequently recommended by systematic review manuals and is often undertaken by review authors. To date, no systematic review has explicitly examined the effectiveness of checking reference lists as a method to supplement electronic searching.

Objectives: 

To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.

Search strategy: 

We searched the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science abstracts (LISA) (1969 to July 2008) and MEDLINE (1966 to July 2008). We contacted experts in systematic review methods and examined reference lists of articles.

Selection criteria: 

Studies of any design which examined checking reference lists as a search method for systematic reviews in any area. The primary outcome was the additional yield of relevant studies (i.e. studies not found through any other search methodologies); other outcomes were publication types identified and data pertaining to the costs (e.g. cost-effectiveness, cost-efficiency) of checking reference lists.

Data collection and analysis: 

We summarized data descriptively.

Main results: 

We included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias. The additional yield (calculated by dividing the additional 'unique' yield identified by checking reference lists by the total number of studies found to be eligible within the study) of relevant studies identified through checking reference lists ranged from 2.5% to 42.7%. Only two studies reported yield information by publication type (dissertations and systematic reviews). No cost data were reported although one study commented that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper, and for that particular study costs were not specifically estimated.

Health topics: