In this webinar, part of the Artificial Intelligence (AI) methods in evidence synthesis series, the presenters covered:-
- Why do we all need to embrace responsible AI?
- What are the recommendations for responsible AI?
- What changes to evidence synthesis processes and governance will help authors and others use AI responsibly?
- How will the joint AI Methods Group between Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence (CEE) support this?
The session was aimed at evidence synthesists, methodologists, AI developers, and those from organizations, funders or publishers involved in evidence synthesis.
Understanding expectations for evidence synthesis when using AI compared to current best practice
As mentioned in the webinar, the evidence synthesis field needs community-based best practices to facilitate the use of automation and AI in evidence synthesis. One area of uncertainty is how 'correct' should evidence synthesis be. We do not currently have consensus on how correct it should be with current best practice (i.e., humans only), how this changes if we add AI, what an acceptable impact of errors might be, and whether this changes for different types of evidence synthesis. The Wellcome-funded DESTinY consortium, of which Cochrane is a partner, has launched a survey to better understand community expectations, which will inform future work and how the next generation of evidence synthesis tools driven by AI are built and evaluated. We welcome feedback from anyone interested in this topic via this survey. Open until 2 July 2025 and takes approximately 35 minutes.
New & updated Responsible AI use in Evidence Synthesis (RAISE) recommendations & guidance
As discussed in the webinar, RAISE is now a three-paper collection available on OSF, including:-
- RAISE 1 with recommendations for practice for the main roles in the evidence synthesis ecosystem to enhance collaboration & communication for the transparent & reliable use of AI in evidence synthesis.
- RAISE 2 with guidance on building & evaluating AI evidence synthesis tools, which focuses on determining if an AI tool does what it claims to do to an acceptable standard, including how to build and validate AI tools, conduct evaluations to build a cumulative evidence base, including performance metrics to consider, & report evaluations.
- RAISE 3 with guidance on selecting & using AI evidence synthesis tools, which focuses on understanding whether an AI tool can be used for a specific evidence synthesis, including how to assess, select & use an AI tool, including ethical, legal and regulatory considerations, & the current state of AI tools for evidence synthesis.
This is a joint initiative with individuals from across 30+ organizations, including Cochrane.
Presenter Bios
Dr. Anna Noel-Storr has worked for Cochrane since 2008. A highly experienced health information specialist and methodological researcher with a particular interest in developing and evaluating new methods, processes and workflows for study identification within evidence synthesis production. Anna led the development of the Cochrane Crowd citizen science platform and currently leads on Cochrane’s Evidence Pipeline initiative. In 2019 she was the recipient of the Chris Silagy Award for outstanding contribution to Cochrane. During the pandemic, she played a leading role in the set up and implementation of the Cochrane COVID-19 Study Register, a highly curated, up-to-date repository of studies related to COVID-19.
Ella Flemyng is Head of Editorial Policy and Research Integrity in Cochrane. She leads work to define and deliver on policies and standards, so Cochrane evidence continues to be trusted and valued. This includes ensuring AI is used responsibly in Cochrane reviews. Ella worked in open access, STEM publishing before joining Cochrane in 2019.
Prof. James Thomas is Principal Investigator of the Evidence Reviews Facility for the Department of Health and Social Care, England, which is a large programme of policy-relevant systematic reviews with accompanying methodological development to increase their relevance and applicability for decision-making. He co-led the Cochrane ‘Project Transform’ which is implementing novel technologies and processes (including machine learning and crowdsourcing) to improve the efficiency of systematic reviews, and is co-investigator on a major Collaborative Award from Wellcome, the Human Behaviour-Change Project, led by Susan Michie (UCL), which is developing technologies and methodologies to organise, synthesise and present the literature in behavioural science.
Prof. Gerald Gartlehner is a professor for evidence-based medicine and clinical epidemiology and the co-director of Cochrane Austria. He is also a co-convenor of the Cochrane Rapid Review Methods Groups. With over two decades of experience in conducting evidence synthesis, Gartlehner has recently focused on evaluating the performance of LLMs
for semi-automating various steps in the evidence synthesis processes.
Prof. Joerg Meerpohl is director of the Institute for Evidence in Medicine at the Medical Center & Faculty of Medicine, University of Freiburg, Germany and founding director of the Freiburg GRADE Center, which was established 2013. He is a board-qualified pediatrician and pediatric hematologist and oncologist by training. For more than a decade Prof. Meerpohl has been an active member of the GRADE working group and has worked as GRADE methodology advisor with several WHO expert panels and the Robert Koch Institute in Germany amongst others. Professor Meerpohl is also the Director of Cochrane Germany. From 2015 to 2018, and since 2024 he has been an elected member of the Cochrane Governing Board and currently also serves on the GRADE Guidance Group. His main research interests include systematic review methodology, transparency in research, and guideline methodology. Professor Meerpohl has published more than 300 PubMed listed articles.
Biljana Macura is the Co-Editor-in-Chief of Environmental Evidence – the journal of the Collaboration for Environmental Evidence – and a Senior Research Fellow at the Stockholm Environment Institute. She is particularly interested in evidence-informed decision-making, knowledge co-production and stakeholder engagement, policy needs assessment, and the use of different forms of evidence to investigate complex environmental and development challenges. Biljana also works on improving and developing methodologies for evidence synthesis, including the responsible use of AI technologies in this domain.
