Is there value in making author contact when determining risk of bias in a Cochrane Systematic Review?

posted on: 2013-07-23 05:56

Helen Atherton is a Research Fellow in the Department of Primary Care Health Sciences at the University of Oxford, and a Cochrane author with the Consumers and Communication Review Group. She is a health services researcher and her main research interest is in eHealth, particularly the use of information and communitcations technology (ICT) in healthcare delivery. In this post she discusses the value of author contact in assessing risk of bias of studies included in Cochrane Reviews.

photo of blog author helen athertonWhen conducting a Cochrane Review, attempts to assess risk of bias are often restricted by incomplete reporting of study design, e.g. not reporting on allocation concealment. When information is missing we end up rating domains as ‘unclear’, which is unhelpful for review authors, editors and readers. One approach is to contact the authors of the included studies and ask them for the missing information on study conduct. The Cochrane Handbook for Systematic Reviews of Interventions advises that though this is an option, contacting authors of trial reports may lead to overly positive answers, so a careful approach should be taken in doing so. Overall, little evidence is available about how well author contact works for assessing risk of bias.

We completed three Cochrane Reviews in 2012, all evaluating the effects of using email in healthcare settings. In each review the majority of included studies had not provided adequate descriptions of key aspects of study design, such as method of randomisation. A communication method such as email is a complex intervention, and testing it in the context of a trial presents challenges. Aside from the complexity posed by variation of format, use of email can take different forms, and is used differently across different settings and populations. Additionally, with complex interventions, the write-up of studies becomes more challenging: context is influential in how a complex intervention works, so authors often need to convey detailed contextual information in a short journal article format. Information may be left out, by accident or design.

Owing to these factors, reviews of complex interventions look very different to reviews of, for example, devices or drug treatments. The complexity was reflected in our reviews by the differences observed between the included studies. Faced with included studies that looked so different from each other, and realising that the review would present a narrative synthesis rather than a meta-analysis, we felt that getting a good assessment of risk of bias seemed vital in allowing us to appraise the studies clearly that we did have, so we decided to contact the authors. We figured that it was worth a go in trying to learn more about the studies and what their findings might mean.  

We sent each author a standardised letter, using a template provided by our Review Group (Consumers and Communication), together with a list of the type of risk of bias on which we wanted to obtain information. Bearing in mind that some studies had been published some time ago, we also provided a brief definition of the type (commonly known as the domain).

Contacting authors worked well for our reviews. We were able to obtain contact with all the authors, via email, with the exception of one author from a study published in 1995, whom we managed to contact via letter. Review One (Email for clinical communication between patients/caregivers and healthcare professionals) included nine studies, each featuring unclear domains. Before author contact, 34 of 54 domains were unclear; after author contact only three of 54 domains remained unclear. Review Two (Email for the provision of information on disease prevention and health promotion) included six studies; five had unclear domains. Author contact reduced the number of unclear domains by 50%, from sixteen to eight. Review Three (Email for clinical communication between healthcare professionals) included only one study, and author contact changed the number of unclear domains from two to one. The risk that study investigators would offer overly positive answers did not hold true for our reviews. We actually considered that the information they provided made us more inclined to rate the risk of bias as ‘high’ rather than ‘low’. A full breakdown of the risk of bias assessments and their changes before and after correspondence will be reported in full in an oral presentation at the Cochrane Colloquium in Québec.

These findings may reflect the nature of the studies. Trials of interventions such as email are a relatively new development: authors tend to be enthusiastic clinicians, rather than drug companies or established research teams. The fact that our response rate was so high also indicates that much of the information we needed was already available directly to the study investigators. Maybe the motivations for conducting the research are different - or perhaps there is less riding on the results?

Our findings did change our perception of the included studies. Where before we had been uncertain about how reliable they were, the risk of bias information was used along with other factors (choice and measurement of outcomes, implementation of interventions) in deeming the findings of the included studies largely unreliable. In view of this, is author contact something that should be advised for all reviews of complex interventions? It is possible that the value of author contact lies in the nature of the intervention under review. Most interventions under review are not as complex as email, so this approach may not offer as much in terms of context.

The narrative approach to the review meant that we did not have to use risk of bias assessment in deciding whether or not to exclude studies. For reviews with meta-analysis, the risk of bias information may be used in a different way: to determine which studies have been included in the analysis, or to determine whether an analysis has been conducted. Where analysis is not possible, the risk of bias information becomes useful in a different way, providing material with which to synthesise in a narrative way.

How might Cochrane guidance help authors to address these issues? Making author contact for risk of bias assessment optional does not lead to consistency across Cochrane Reviews, as it is left to the discretion of the review author. If there is potential for author contact to have an impact on the conclusions of a review it should be mandatory, and consistent methods should be developed. Otherwise, authors may be introducing bias in deciding not to do this - though this is arguably easy to say when reviews have a low number of included studies. Authors of reviews with included studies in double figures and above may disagree. Maybe there would have to be a cut-off? More than one unclear domain in 50% of your included studies? Contact authors. Dealing with a complex intervention? Contact authors. Where should the line be drawn? Other things to bear in mind are restrictions due to time and funding. Many authors are already preparing reviews as extra workload beyond their day jobs: is there capacity for another stage in what is already an arduous process?

Amidst the fog of reviews that were complicated and presented a challenge in how to present the data, author contact allowed for some clarity on study conduct. It was valuable. How valuable it could be as part of the wider reviewing process is currently a known unknown that should be addressed.

Helen Atherton (helen.atherton@phc.ox.ac.uk)

Comments

Re: Is there value in making author contact when determining ...

I agree that contacting authors clarifies the risk of bias process (I was involved in a review of parenting programmes for conduct problems). Details from authors give a fuller description of what happened and helped us make a better judgement on RoB. We also found a high level of response from authors - only one author (out of 13) did not respond to us; also most gave very detailed responses. So the value of contacting authors is not just restricted to e-based interventions.

Re: Is there value in making author contact when determining ...

I read Helen's post with interest as I am currently completing two Cochrane reviews in which I have had to contact almost every author to verify or complete the Risk of Bias assessment. I too have been concerned about whether their e-mail responses that "yes, they did that" warrants the same rating as a statement to the same extent that has been peer reviewed and published. To overcome this, I have noted in the Risk of Bias table when a positive rating of "low" is based on correspondence with the author rather than reported in the original paper and usually quote the e-mail response.

Since we too are unable to pool the data due to methodological and clinical heterogeneity, being able to use the risk of bias information to help in the synthesis of the evidence and ensure its accuracy and completeness was important to us.

Marie Westby, PT, PhD

Re: Is there value in making author contact when determining ...

This is great- fantastic bit of work. Interestingly it contrasts with the other bit of empirical evidence i know about (Vale et al bmj 2013) who compared RoB in trials with information obtained during IPD analyses- they found all the unclears were actually low risk of bias? Perhaps generalising from these studies is tricky? 

It would be good to get a better handle on this as it has really important implications for GRADE. Asking authors seems like a logical way of being more informed though..

Post new comment

The content of this field is kept private and will not be shown publicly.
If you do NOT enter a subject, the first few words of your response will be used.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <p> <br> <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>

More information about formatting options

Updated on: July 23, 2013, 5:56

The Cochrane Official Blog is curated and maintained by the Cochrane Web Team. To submit items for publication to the blog, please email techsupport@cochrane.org.

The Cochrane Blog presents commentary and personal opinion on topics of interest from a range of contributors to the work of The Cochrane Collaboration. Opinions posted on the Cochrane Blog are those of the individual contributors and do not necessarily reflect the views or policies of The Cochrane Collaboration.