In an era defined by an abundance of information, and new ways to find it, Cochrane’s Chief Executive Officer, Karla Soares-Weiser argues that systematic reviews remain an essential tool to help individuals, healthcare professionals and policymakers make decisions. However, the challenge is no longer simply producing reliable evidence but maintaining the trust that makes Cochrane’s evidence useful.
How information seeking is changing
In the early days of the internet, people started to turn to ‘Dr Google’ for health information and advice. Search engines transformed how people sought answers to their health questions, reshaped relationships between patients and healthcare professionals, and provided a new starting point for those looking to answer policy and practice questions. We are now seeing the same level of change, but this time, people are turning to generative AI chatbots to ask their health questions.
Illustrating the scale of this shift, a UK study that polled 2,000 people found that one in seven people are using AI chatbots for health advice instead of seeing their GP. Another study attempted to characterize the intents and topics behind the health queries asked in Microsoft Co-Pilot, finding that nearly one in five conversations involves personal symptom assessment or discussion of a condition.
The challenges with AI
AI chatbots have the potential to make information more accessible; people can ask a question naturally and receive an immediate response in their own language, which is particularly valuable for non- native English speakers. However, while generative AI models can support people seeking information, they also introduce significant challenges, particularly in high-stakes fields like health.
The large language models (LLMs) that drive chatbots are based on prediction, rather than genuine understanding. They generate text by predicting the most likely sequence of words, based on patterns in their training data, which is a biased subset of human knowledge. While the latest chatbots do retrieve information from the internet, this is not done in a systematic way, meaning trustworthy information can be lost in favour of misinformation. What’s more, LLMs tendency towards sycophancy can lead them to invent facts that they cannot find, known as hallucinations. There have been reports of invented details like nonexistent papers, fabricated DOIs and fictitious journals. A study published in BMJ Open found that half the responses to medical information questions from five popular chatbots were problematic.
Not only do these models produce inaccurate information, but they also produce convincing and friendly responses. This blurring of the lines between human and machine voices makes detecting hallucinations challenging. In fact, researchers from MIT (USA) have found that AI models use more confident language when hallucinating than when presenting facts. Models were 34% more likely to use phrases like "definitely," "certainly," and "without a doubt" when generating incorrect information.
Bias presents a further challenge. Because AI systems are trained on data created by humans, the systems exploit and amplify the human biases that are embedded in the data. For example, the data may underrepresent certain demographic groups, or it may reflect biases inherent in current societal structures, such as racism and sexism. Learning from these flawed data inevitably leads to biased outputs. Moreover, it has been shown that when AI systems amplify biases, these are further internalized by humans, creating a feedback loop.
So, not only is there more information available than ever before, it is easier to access but more likely to cause harm. AI can generate information in seconds, but it may hallucinate information to fill gaps, amplify harmful bias and not present the certainty of information accurately.
There are indications that people adopting AI assistants is not the same as people trusting them. A YouGov report indicated that only 31% of people trust information provided by AI assistants, compared with 77% of people trusting information provided by search engines. But familiarity reduces this uncertainty, in people who use AI assistants regularly, the level of trust rises to 49%.
In health, the consequences of inaccuracies can be serious. Not only do systematic reviews still have a place in helping people navigate an increasingly complex information landscape, the trust that people have in this evidence is crucial.
Why should we value systematic reviews?
Systematic reviews are the most reliable way to find out if a particular intervention or test is effective. They bring together all relevant research on a specific question and use rigorous, transparent methods to produce an objective summary of the evidence. The methods are specifically designed to reduce the risk of selective reporting and researcher bias. This provides a much better foundation for decision-making than relying on information from a single study presented by an AI assistant.
At Cochrane, we follow a particularly rigorous process to produce our reviews. We have very strict editorial policies and methodological standards, including requirements for our authors regarding disclosure of interests and acknowledgement of technologies used. In addition, we do not accept commercial or conflicted funding. This helps ensure that our evidence remains independent and trustworthy, and that it is not influenced by commercial or financial interests.
Not only do systematic reviews produce a reliable, unbiased outcome; any uncertainties are always reported. We will never claim something is “definite” if it isn’t. Cochrane reviews aim to provide a balanced summary of the potential benefits and harms of interventions and our reviews always include an indication of how confident you can be of the findings.
Finally, instead of making up information to fill the gaps, systematic reviews let us know what we don’t know. By identifying current and ongoing studies, they can also highlight where there are specific gaps in knowledge or evidence is lacking, and where more research might be needed.
The future
At Cochrane, we are already seeing changes in how people access information. Across the world, fewer users are clicking through to full website content, instead relying on summaries generated by search engines, algorithms, and AI assistants.
Health is one of the most high-stakes domains in which people interact with AI. We know that tens of thousands of people are now accessing Cochrane plain language summaries via AI-assisted platforms. However, many users do not click through to the original source. This raises important questions about how our content is used, interpreted and attributed, and how we ensure its integrity outside our own platforms.
For over 30 years, Cochrane has built a reputation of trust, independence and methodological rigour. As AI becomes more embedded in how information is accessed and shared, it is our responsibility to uphold our standards and preserve the trust people have in our evidence.
It is not just how our reviews are shared; it is ensuring the information has meaning and impact when it reaches someone who might use it. Rigorous methods and research integrity matter more than ever. Transparency, independence, accountability, reproducibility, and openness about uncertainty and are not administrative requirements. They are the foundations of trustworthy evidence.
