Medicine’s over-generalisation problem — and how AI might make things worse

In medicine, there’s a well-known maxim: never say more than your data allows. It’s one of the first lessons learned by clinicians and researchers.

Journal editors expect it. Reviewers demand it. And medical researchers mostly comply. They hedge, qualify and narrow their claims — often at the cost of clarity. Take this conclusion, written to mirror the style of a typical clinical trial report:

“In a randomized trial of 498 European patients with relapsed or refractory multiple myeloma, the treatment increased median progression free survival by 4.6 months, with grade three to four adverse events in 60 per cent of patients and modest improvements in quality-of-life scores, though the findings may not generalize to older or less fit populations.”

It’s medical writing at its most exacting — and exhausting. Precise, but not exactly easy to take in.

Unsurprisingly, then, those careful conclusions often get streamlined into something cleaner and more confident. The above example might be simplified into something like: “The treatment improves survival and quality of life.” “The drug has acceptable toxicity.” “Patients with multiple myeloma benefit from the new treatment.” Clear, concise — but often beyond what the data justify.

Philosophers call these kinds of statements generics — generalisations without explicit quantifiers. Statements like “the treatment is effective” or “the drug is safe” sound authoritative, but they don’t say: For whom? How many? Compared to what? Under what conditions?

Generalisations in medical research

In previous work in the ethics of health communication, we highlighted how generics in medical research tend to erase nuance, transforming narrow, population-specific findings into sweeping claims that readers might misapply to all patients.

In a systematic review of over 500 studies from top medical journals, we found more than half made generalizations beyond the populations studied. More than 80 per cent of those were generics, and fewer than 10 per cent offered any justification for these broad claims.

Researchers’ tendency to over-generalize may reflect a deeper cognitive bias. Faced with complexity and limited attention, humans naturally gravitate toward simpler, broader claims — even when they stretch beyond what the data support. In fact, the very drive to explain the data, to tell a coherent story, can lead even careful researchers to overgeneralize.

Artificial intelligence (AI) now threatens to significantly exacerbate this problem. In our latest research, we tested 10 widely used large language models (LLMs) — including ChatGPT, DeepSeek, LLaMA and Claude — on their ability to summarise abstracts and articles from top medical journals.

Even when prompted for accuracy, most models routinely removed qualifiers, oversimplified findings and repackaged researchers’ carefully contextualised claims as broader statements.

AI-generated summaries

Analyzing nearly 5,000 LLM-generated summaries, we found rates of such over-generalizations as high as 73 per cent for some models. Very often, they converted non-generic claims into generics, for example, shifting from “the treatment was effective in this study,” to simply “the treatment is effective,” which misrepresented the study’s true scope.

Strikingly, when we compared LLM-generated summaries to ones written by human experts, chatbots were nearly five times more likely to produce broad generalisations. But perhaps most concerning was that newer models — including ChatGPT-4o and DeepSeek — tended to generalise more, not less.

What explains these findings? LLMs trained on overgeneralized scientific texts may inherit human biases from the input. Through reinforcement learning from human feedback, they may also start favouring confident, broad conclusions over careful, contextualised claims, because users often prefer concise, assertive responses.

The resulting miscommunication risks are high, because researchers, clinicians and students increasingly use LLMs to summarise scientific articles.

In a recent global survey of nearly 5,000 researchers, almost half reported already using AI in their research — and 58 per cent believed AI currently does a better job summarising literature than humans. Some claim that LLMs can outperform medical experts in clinical text summarisation.

Our study casts doubt on that optimism. Over-generalizations produced by these tools have the potential to distort scientific understanding on a large scale. This is especially worrisome in high-stakes fields like medicine, where nuances in population, effect size and uncertainty really matter.

Precision matters

So what can be done? For human authors, clearer guidelines and editorial policies that address both how data are reported and how findings are described can reduce over-generalizations in medical writing. Also, researchers using LLMs for summarization should favour models like Claude — the most accurate LLM in our study — and remain aware that even well-intentioned accuracy prompts can backfire.

AI developers, in turn, could build prompts into their LLMs that encourage more cautious language when summarising research. Lastly, our study’s methodology can help benchmark LLMs’ overgeneralization tendency before deploying them in real-world contexts.

In medical research, precision matters — not only in how we collect and analyse data, but also in how we communicate it. Our research reveals a shared tendency in both humans and machines to overgeneralize — to say more than what the data allows.

Tackling this tendency means holding both natural and artificial intelligence to higher standards: scrutinising not only how researchers communicate results, but how we train the tools increasingly shaping that communication. In medicine, careful language is imperative to ensure the right treatments reach the right patients, backed by evidence that actually applies.