What is Screen4Me?
Screen4Me is a service designed to help review teams complete the screening of search results much more quickly, without compromising on quality. It consists of three components: Known Assessments, the RCT Classifier, and Cochrane Crowd.
- Known Assessments are records that have been through Cochrane’s citizens science platform, Cochrane Crowd, and have received a final classification of either describing a randomised controlled trial (RCT) or not.
- The RCT Classifier is a machine-learning routine that assigns records a 'likelihood' score, based on whether the machine thinks the record is describing an RCT or not.
- Cochrane Crowd is Cochrane’s crowdsourcing platform, where contributors from around the world help to identify randomised trials and other types of healthcare-related research.
Please note: Screen4Me is currently only available for Cochrane intervention reviews that intend to include evidence from randomised controlled trials only.
How does Screen4Me work?
Once the review team has conducted the searches, the de-duplicated search results are loaded into the Cochrane Register of Studies. Once imported, the Screen4Me workflow can be applied.
The first part of the workflow will identify any already known RCTs; the second stage will remove very obvious non-RCT records from the set via machine learning; and the final stage sends the remaining records to be assessed by volunteers via Cochrane Crowd. At the end of the process, the author team should be left with a much smaller batch of records to assess.
Screen4Me projects take on average two weeks to complete. The search results set is expected to reduce by 50-70%.
How do I access it?
There are two main ways to access Screen4Me:
- Contact the Cochrane information specialist you are working with. Many Cochrane information specialists have been trained to run the Screen4Me service. They will do this using the Cochrane Register of Studies application.
- If you are not working with a Cochrane information specialist or they do not have access to Screen4Me, contact Cochrane Support to arrange the Screen4Me service for you.
Report Screen4Me in Cochrane protocols and reviews
If you have plan to use Screen4Me in your review, or if you have used it, you need to report this clearly in your methods. See below for reporting guidance.
Screen4Me reporting guidance
Additional references
When reporting use of Screen4Me in your Cochrane protocol or review, add these references to your additional references in RevMan first, so that you can link to them when inserting the template text below.
Noel-Storr AH, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, Wisniewski S, McDonald S, Murano M, Glanville J, Foxlee R, Beecher D, Ware J, Thomas J. An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. J Clin Epidemiol. 2021 May;133:130-139. https//doi.org/10.1016/j.jclinepi.2021.01.006. Epub 2021 Jan 18. PMID: 33476769.
Thomas J, McDonald S; Noel-Storr AH, Shemilt I, Elliott J, Mavergames C, Marshall I. Machine learning reduces workload with minimal risk of missing studies: development and evaluation of an RCT classifier for Cochrane Reviews. Journal of Clinical Epidemiology 2020; https://doi.org/10.1016/j.jclinepi.2020.11.003
Noel-Storr AH, Dooley G, Wisniewski S, Glanville J, Thomas J, Cox S, Featherstone R, Foxlee R, Cochrane Centralised Search Service showed high sensitivity identifying randomised controlled trials: A retrospective analysis. Journal of Clinical Epidemiology 2020; 127: 142-150 https://doi.org/10.1016/j.jclinepi.2020.08.008
Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide. Res Synth Methods. 2018;9(4):602-614
Protocols
RevMan section: Selection of studies
Add the following template text at the start of this section if you plan to use the Screen4Me workflow:
"To support the identification of RCTs, we will use Cochrane’s Screen4Me workflow developed by Cochrane for initial screening of the search results in the evidence synthesis process. Screen4Me will be used according to the user guide, and pre-set parameters and accessed via the Cochrane Register of Studies tool. Outputs from Screen4Me are justified for use in our synthesis because both components of the tool (the RCT Classifier and the Cochrane Crowd) have been validated individually for use on health-related searches for randomized controlled trials (Thomas 2020; Noel-Storr 2021). A detailed description of the methodology, including parameters and validation procedures, is available from the Screen4Me page on Cochrane.org."
Add the following template text at the start of this section if you plan to use the RCT Classifier only:
"For the identification of RCTs, we will use Cochrane’s RCT Classifier developed by Cochrane for initial screening of the search results in the evidence synthesis process. The RCT Classifier, accessed via [insert the tool used: Covidence; the Cochrane Register of Studies; EPPI Reviewer], will be used according to the user guide, and pre-set parameters. Outputs from the RCT Classifier are justified for use in our synthesis because the RCT classifier has been validated for use on health-related searches for randomized controlled trials (Thomas 2020). A detailed description of the building, calibration and validation of the RCT Classifier can be found in Thomas 2020."
Reviews
RevMan section: Selection of studies
Add the following template text at the start of this section if you plan to use the Screen4Me workflow:
"To support the identification of RCTs, we used Cochrane’s Screen4Me workflow developed by Cochrane for initial screening of the search results in the evidence synthesis process. Screen4Me was used according to the user guide, and pre-set parameters and accessed via the Cochrane Register of Studies tool. Outputs from Screen4Me are justified for use in our synthesis because both components of the tool (the RCT Classifier and the Cochrane Crowd) have been validated individually for use on health-related searches for randomized controlled trials (Thomas 2020; Noel-Storr 2021). A detailed description of the methodology, including parameters and validation procedures, is available from the Screen4Me page on Cochrane.org."
Add the following template text at the start of this section if you plan to use the RCT Classifier only:
"For the identification of RCTs, we used Cochrane’s RCT Classifier developed by Cochrane for initial screening of the search results in the evidence synthesis process. The RCT Classifier, accessed via [insert the tool used: Covidence; the Cochrane Register of Studies; EPPI Reviewer], will be used according to the user guide, and pre-set parameters. Outputs from the RCT Classifier are justified for use in our synthesis because the RCT classifier has been validated for use on health-related searches for randomized controlled trials (Thomas 2020). A detailed description of the building, calibration and validation of the RCT Classifier can be found in Thomas 2020."
RevMan section: Results of the search
Add the following template text at the start of this section:
The search identified a total of [insert number] results. In assessing the studies we used Cochrane’s Screen4Me workflow to help identify potential reports of randomized trials. The results of the Screen4Me assessment process can be seen in Figure x. We then assessed the remaining [insert number] records left in after Screen4Me. [Author team then continues to describe their screening process e.g. assessing remaining records based on a title and abstract assessment.]
RevMan section: Figures
1. Screen4Me summary diagram
We recommend that you export the Screen4Me summary diagram from your Screen4Me project and import it into the Figures sections of the review. You should call it: Screen4Me summary diagram. Here is an example. Screen4Me produces a flow diagram per search so if you use Screen4Me multiple times for a single review, you will have generated multiple Screen4Me flow diagrams.
2. PRISMA flow diagram example
In the example below, the key information about what was excluded at the Screen4Me stage of the screening process is detailed in the Screening exclusion box.
RevMan section: Acknowledgements
If you use the Cochrane Crowd component of Screen4Me, you must acknowledge by name those Crowd contributors who screen 250 or more records for your Screen4Me review/project. You can see who should get acknowledgment within the Screen4Me project summary screen. Here is an example of Acknowledgments section:
We would like to acknowledge and thank the following people for their help in assessing the search results for this review via Cochrane’s Screen4Me workflow: [insert names].
Updates
Some users will make use of Screen4Me for updates of reviews where Screen4Me has not been used previously. Here is an example of how to reflect the use of Screen4Me in these circumstances:
RevMan section: Figures - PRISMA flow diagram example
Multiple uses of Screen4Me for an individual review
Some users will make use of Screen4Me more than once for a specific review (for example for a living systematic review). If Screen4Me is used in this way, we recommend you can now export a table of Screen4Me results related to a specific review from the Cochrane Register of Studies. Each row in the table reflects use of Screen4Me for each search update or top-up search. It is not mandatory to publish this table, but it is a nice way to show the results for each individual search where Screen4Me has been used. Below, we give an example of the summary table and an example of how to reflect the use of Screen4Me in the study flow diagram
Screen4Me summary table
Below, is an example of the new summary table feature available in Screen4Me. You can export this table from the Cochrane Register of Studies tool (or ask for it to be exported by the information specialist who is running your Screen4Me workflow for you. Each row on the table presents a search that has been run for a single review. The column shows the results of each component part of Screen4Me: the Prior (or Known) Assessments, the RCT Classifier, and the Cochrane Crowd.]
RevMan section: Figures - PRISMA flow diagram example
Communicating with Cochrane Crowd
If you have used Cochrane Crowd, we would like to alert Crowd contributors when reviews they have contributed to are published. Please contact us when your review is published. The Cochrane Crowd team will send an alert to the Crowd contributors and flag the review as published on the Cochrane Crowd platform.
