Artificial intelligence (AI) tools are increasingly being used in evidence synthesis. But how can we build and evaluate these tools to ensure they meet the needs and expectations of the evidence synthesis community? This session, part of the series: Being responsible when using AI for evidence synthesis, will highlight:
- The different classes of AI technology and the implications for the principles of evidence synthesis when used
- The key phases involved in building an AI tool
- Important considerations when evaluating AI performance, including performance metrics and the reporting of evaluation methods and results
- How the Responsible use of AI in evidence SynthEsis (RAISE) recommendations and guidance can help guide the development and evaluation of AI tools (https://osf.io/fwaud/overview)
The session will also explore considerations for building and evaluating large language model (LLM)-based tools, including prompt engineering. These concepts are relevant both for developers creating large-scale tools and platforms and for review teams designing local prompts to support their own evidence synthesis workflows.
It is open to everyone. Methodologists, AI tool developers, evidence synthesists and Cochrane authors, as well as anyone else interested in understanding the expectations for building and evaluating responsible AI use in evidence synthesis, such as funders, editors and publishers, trainers, or organisations or groups that produce evidence synthesis.
Presenter bios
Nadia Soliman is a Senior Research Fellow at the Evidence for Policy and Practice Information Centre (EPPI Centre) University College London, and Co-Director of the EPPI Centre AI Lab. Her research interests focus on the development, evaluation, and responsible use of automation tools and methods for evidence synthesis and systematic reviewing. In her role as Co-Director of the AI Lab, she supports the coordination of methodological innovation, capacity building, and collaborations to advance the use of AI in evidence synthesis while promoting rigorous and trustworthy approaches. She has a background in pharmacology and experience conducting systematic reviews and meta-analyses to narrow the translation gap between preclinical and clinical research of chronic pain.
Dr Promise M Nduku is a Research Fellow at the African Synthesis Centre for Climate Change, Environment and Development (ASCEND) at the University of Cape Town, where he contributes to AI-enabled living evidence synthesis on climate change, adaptation, climate risk and climate-related health. He is also Co-Founder and Director of the Evidence Synthesis and Learning Lab at Elevate Evidence Hub, where he leads the production of rigorous, policy-relevant evidence and supports organisations in strengthening institutional systems for evidence use. He has collaborated with a wide range of local and international organisations to produce evidence products that inform policy and practice. He is a member of the Technical Advisory Group of the Campbell Collaboration and the Methodological Advisory Group of the European Centre for Disease Prevention and Control.
Ailbhe Finnerty Mutlu is a Research Fellow at the Evidence for Policy and Practice Information Centre (EPPI Centre) University College London, and Associate Director of the EPPI Centre AI Lab. Her research focuses on the development and evaluation of tools and methods for automating evidence synthesis, with a particular interest in understanding their performance, reliability, and appropriate use in practice. Her background is in the social sciences, and she has worked extensively across interdisciplinary teams and projects. Her work aims to bridge the gap between social and computer sciences by bringing together perspectives from both disciplines. In particular, she focuses on translating the methodological and practical needs of evidence synthesis into the development and evaluation of appropriate computational and AI-based approaches.
