Key messages
• Training peer reviewers may have little or no effect on the quality of peer review.
• Larger, well-designed studies are needed to give better estimates of the effect.
What is a peer reviewer?
A peer reviewer is a person who evaluates the research work done by another person. The peer reviewer is usually a researcher with skills similar to those needed to conduct the research they evaluate.
What is peer review used for?
Both funders and publishers of research can be uncertain if a research project or report is of good quality. Many use peer reviewers to evaluate the quality of a project or report.
How could peer review quality be improved by training reviewers?
Training peer reviewers might make them better at identifying strengths and weaknesses in the research they assess.
What did we want to find out?
We wanted to find out if training peer reviewers increased the quality of their work.
What did we do?
We searched for studies that looked at training for peer reviewers compared with no training, different types of training, or standard journal or funder practice. We extracted information and summarized the results of all the relevant studies. We rated our confidence in the evidence based on factors such as study methods and sizes.
What did we find?
We found 10 studies that involved a total of 1213 units of analysis (722 reviewers and 491 manuscripts). Eight studies included only journal peer reviewers. The remaining two studies included grant peer reviewers.
Main results
Emails reminding peer reviewers to check items of reporting checklists, compared with standard journal practice, probably have little or no effect on the completeness of reporting (evidence from 2 studies with 421 manuscripts).
Reviewer training, compared with standard journal practice, may slightly improve peer reviewer ability to detect errors (evidence from 1 study with 418 reviewers). The reviewers who received training identified 3.25 out of 9 errors on average, whereas the reviewers who received no training identified 2.7 out of 9 errors on average.
Reviewer training, compared with standard journal practice, may have little or no effect on stakeholders' assessment of review quality (evidence from 6 studies with 616 reviewers and 60 manuscripts).
We are unsure about the effect of a video course, compared with no video course, on agreement between reviewers (evidence from 1 study with 75 reviewers).
Structured individual feedback on scoring, compared with general information on scoring, may have little or no effect on the change in agreement between reviewers (evidence from 1 study with 41 reviewers).
What are the limitations of the evidence?
We have little confidence in most of the evidence because most studies lacked important information and included too few reviewers. Additionally, it is unclear whether the studies measured peer review quality in a valid and reliable way.
How up to date is this evidence?
The evidence is up to date to April 2022.
Evidence from 10 RCTs suggests that training peer reviewers may lead to little or no improvement in the quality of peer review. There is a need for studies with more participants and a broader spectrum of valid and reliable outcome measures. Studies evaluating stakeholders' assessments of the quality of peer review should ensure that these instruments have sufficient levels of validity and reliability.
Funders and scientific journals use peer review to decide which projects to fund or articles to publish. Reviewer training is an intervention to improve the quality of peer review. However, studies on the effects of such training yield inconsistent results, and there are no up-to-date systematic reviews addressing this question.
To evaluate the effect of peer reviewer training on the quality of grant and journal peer review.
We used standard, extensive Cochrane search methods. The latest search date was 27 April 2022.
We included randomized controlled trials (RCTs; including cluster-RCTs) that evaluated peer review with training interventions versus usual processes, no training interventions, or other interventions to improve the quality of peer review.
We used standard Cochrane methods. Our primary outcomes were 1. completeness of reporting and 2. peer review detection of errors. Our secondary outcomes were 1. bibliometric scores, 2. stakeholders' assessment of peer review quality, 3. inter-reviewer agreement, 4. process-centred outcomes, 5. peer reviewer satisfaction, and 6. completion rate and speed of funded projects. We used the first version of the Cochrane risk of bias tool to assess the risk of bias, and we used GRADE to assess the certainty of evidence.
We included 10 RCTs with a total of 1213 units of analysis. The unit of analysis was the individual reviewer in seven studies (722 reviewers in total), and the reviewed manuscript in three studies (491 manuscripts in total).
In eight RCTs, participants were journal peer reviewers. In two studies, the participants were grant peer reviewers. The training interventions can be broadly divided into dialogue-based interventions (interactive workshop, face-to-face training, mentoring) and one-way communication (written information, video course, checklist, written feedback). Most studies were small.
We found moderate-certainty evidence that emails reminding peer reviewers to check items of reporting checklists, compared with standard journal practice, have little or no effect on the completeness of reporting, measured as the proportion of items (from 0.00 to 1.00) that were adequately reported (mean difference (MD) 0.02, 95% confidence interval (CI) −0.02 to 0.06; 2 RCTs, 421 manuscripts).
There was low-certainty evidence that reviewer training, compared with standard journal practice, slightly improves peer reviewer ability to detect errors (MD 0.55, 95% CI 0.20 to 0.90; 1 RCT, 418 reviewers).
We found low-certainty evidence that reviewer training, compared with standard journal practice, has little or no effect on stakeholders' assessment of review quality in journal peer review (standardized mean difference (SMD) 0.13 standard deviations (SDs), 95% CI −0.07 to 0.33; 1 RCT, 418 reviewers), or change in stakeholders' assessment of review quality in journal peer review (SMD −0.15 SDs, 95% CI −0.39 to 0.10; 5 RCTs, 258 reviewers).
We found very low-certainty evidence that a video course, compared with no video course, has little or no effect on inter-reviewer agreement in grant peer review (MD 0.14 points, 95% CI −0.07 to 0.35; 1 RCT, 75 reviewers).
There was low-certainty evidence that structured individual feedback on scoring, compared with general information on scoring, has little or no effect on the change in inter-reviewer agreement in grant peer review (MD 0.18 points, 95% CI −0.14 to 0.50; 1 RCT, 41 reviewers, low-certainty evidence).