AI benchmarks reward confident guesses over admitted uncertainty

A September 2025 paper argued that language-model training and widely used benchmarks reward guessing over acknowledging uncertainty. The problem becomes harder to detect when a model presents a false statement in fluent, confident language without signaling what it does not know.

The risk is not confined to factual trivia. A 2026 audit of five public chatbots found that 49.6 percent of their answers to health questions were somewhat or highly problematic, while Taiwan's health ministry has classified incorrect generative-AI output as a patient-safety risk for medical institutions.

Hallucination describes a plausible falsehood

An AI hallucination is a generated statement that appears plausible but is false. The term covers invented facts and citations, unsupported claims and answers that conflict with available evidence. Fluency is part of the risk because grammar and specificity can make an unsupported answer look verified.

The September 4, 2025, paper by Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang argues that hallucinations emerge from statistical errors during pretraining and persist because common evaluations reward guessing. The analysis does not depend on a single chatbot or transformer design, and it does not claim that every model produces the same errors at the same rate.

The authors demonstrated the behavior by asking an open-source model for Kalai's birthday and telling it to answer only if it knew. Across three attempts, the model supplied three different dates, all wrong. The example shows that an instruction to withhold an uncertain answer does not necessarily overcome the model's learned incentive to provide one.

Pretraining creates errors before a chatbot answers a question

During pretraining, a language model learns a probability distribution from large collections of text. It does not store a verified ledger of every statement and then consult that ledger before generating a response. For facts with little or no learnable pattern, many plausible strings may fit the surrounding language even though only one is correct.

The 2025 paper models this as a classification problem: deciding whether a candidate output is valid can itself be difficult, and generation requires making that distinction across possible responses. Its statistical lower bounds apply even under an assumption of error-free training data. Real training collections can add ambiguity, contradictions and outdated material, but those are not required for the paper's core mechanism.

This does not mean that every error is unavoidable after training. The authors distinguish the behavior of a pretrained base model from a finished system that can abstain, retrieve information or use other controls. Their narrower conclusion is that the usual pretraining objective creates predictable error pressure, while later design choices determine how much of that pressure reaches users.

Binary benchmarks make guessing pay

Many evaluations give one point for a correct answer and zero for either a wrong answer or an admission of uncertainty. Under that rule, abstaining cannot improve the score, while guessing has some chance of earning a point. A model optimized for leaderboard performance can therefore appear stronger by attempting questions it cannot answer reliably.

The authors proposed adding explicit confidence targets to existing evaluations: a correct answer would earn one point, an abstention would earn zero, and an incorrect answer would incur a penalty tied to the stated threshold. At a 0.75 confidence target, for example, an error would lose two points; at 0.9, it would lose nine. The proposal is intended to make withholding an answer rational when estimated confidence falls below the threshold.

The reform remains a proposal, not evidence that major benchmark operators have changed their scoring. It also addresses one incentive rather than every source of false output. Retrieval can supply better evidence, and post-training can teach abstention, but either control can still fail when the source is poor or the system applies it incorrectly.

Health-chatbot audit found problems in half of responses

A BMJ Open study published in April 2026 tested free versions of ChatGPT, Gemini, Meta AI, Grok and DeepSeek with 50 health prompts per model and rated 49.6 percent of the 250 responses as problematic. Thirty percent were somewhat problematic and 19.6 percent were highly problematic. The questions covered cancer, vaccines, stem cells, nutrition and athletic performance and were designed to pressure the systems toward misinformation or contraindicated advice.

Two subject specialists rated answers in each category using predefined criteria. The study found no statistically significant difference in overall response quality among the five chatbots, although Grok produced more highly problematic responses than expected under the study's comparison. The tested versions were available between 2022 and 2024, and the prompts were run in February 2025, so the results do not establish the performance of later releases.

Citations did not supply a dependable shortcut for judging the answers. Across 25 prompts that requested 10 references from each model, the chatbots returned 1,013 of the requested 1,250 references. Median completeness was 40 percent, and no chatbot produced a fully complete and accurate reference list for any prompt under the study's method.

Taiwan guidance treats hallucination as an institutional risk

Taiwan's Ministry of Health and Welfare, the national authority responsible for health policy, issued guidance on May 29, 2026, for hospitals and clinics preparing to introduce or already using generative AI. The document is administrative guidance rather than a binding rule, and it does not govern consumer chatbots outside regulated medical settings.

The guidance lists plausible but incorrect output as one of six risk categories and says medical institutions should tell patients and families how a generative-AI system is involved, what it does and where its limits lie. When a system interacts directly with the public, the institution should disclose that AI is operating it and warn that its output may contain hallucinations or other errors. The document also leaves final confirmation and responsibility with qualified medical personnel when AI is used in clinical judgment, patient communication or medical records.

This is a national example, not a standard for all English-speaking markets. Its value for international readers lies in the control structure: identify the system's role, disclose its limits, monitor output and preserve a named human decision-maker. Legal duties and enforcement differ by country.

Controls need to address both evidence and incentives

A source link is useful only when it resolves to a real document and supports the statement attached to it. Systems that retrieve from a controlled collection can narrow the material available for an answer, but retrieval does not prove that the model interpreted the document correctly. Human review remains a separate control in settings where an error can affect health, legal rights or finances.

Evaluation design determines what developers optimize. If benchmarks reward every attempted answer and do not distinguish an honest abstention from an error, reported accuracy can favor overconfident behavior. Confidence-aware scoring can change that incentive, but its effect must be measured after adoption rather than inferred from the proposal alone.

The evidence supports a limited conclusion: hallucination is not adequately described as a rare software malfunction. It arises from how generative models learn and how their outputs are rewarded, while its practical harm depends on the safeguards around a particular use. A fluent answer, even one with citations, still requires verification against the underlying source.