A 2024 systematic review found that explainable artificial intelligence increased clinicians' trust in five of 10 empirical studies, while two studies reported both increases and decreases. The findings challenge the assumption that adding an explanation automatically makes a medical AI system more trustworthy.

The review examined studies in which practicing clinicians used machine-learning systems and trust or a related measure was recorded as an outcome. Two studies found no effect, and one did not reach a conclusion, leaving a small and mixed evidence base.

Explainability changes what a user can inspect

A black-box model produces an output through internal operations that its user cannot readily interpret. Explainable AI adds information intended to show which inputs, features or relationships influenced that output. The explanation may help a clinician question a result, but it is not the same as proof that the result is correct.

The review found that clinicians responded more favorably when explanations appeared understandable and relevant to their work. Other participants ignored explanations that seemed redundant, slowed decisions or did not fit the clinical context. Some users also misunderstood the information or used it mainly to confirm an opinion they already held.

The authors identified a deeper limitation: the validity of the explanations used in the included studies had not been critically evaluated. An explanation can simplify a model's behavior or appear persuasive without accurately representing the model or established medical knowledge. Trust therefore needs calibration against evidence, not maximization.

A diagram contrasts an opaque AI output with a system that displays evidence and an inspectable decision path

A drug search shows an inspectable human-machine workflow

A 2021 paper from researchers at BenevolentAI described one way to make computational drug searching open to expert scrutiny. The team combined machine-assisted literature extraction, a biomedical knowledge graph and repeated queries directed by scientists while searching for existing drugs that might act on biological processes involved in COVID-19.

The researchers reported that their visual analytics workflow connected information about drugs, genes, proteins, mechanisms and pathways and led them to select the rheumatoid arthritis drug baricitinib as a candidate. The graph drew on dozens of biomedical databases and scientific literature processed by machine-learning tools.

Users did not receive a single unexplained ranking. They built visual graph patterns, reviewed the returned relationships and revised their questions over several iterations. The paper described this process as an exchange between the expert and the system, with researchers able to inspect what the graph contained and change direction when a result was not biologically useful.

The workflow also exposed where human judgment entered the search. Scientists chose the mechanisms to investigate, assessed candidate drugs and added their own knowledge to the system's representation. The authors acknowledged that user bias could steer the work toward familiar areas and said experimental validation remained the safeguard for predictions.

A diagram shows medical literature, drug data and protein targets feeding an AI knowledge graph that produces candidates for expert review

The case does not prove that explanation drove the result

The selection of baricitinib does not isolate which part of the discovery workflow mattered. The process combined data integration, machine learning, visual tools, pharmacological assessment and expert decisions. It cannot show that the same candidate would have been missed by a less interpretable system.

The evidence also came from researchers with a direct institutional interest in the technology. All six authors disclosed that they were employed by BenevolentAI. The paper documents the company's workflow in detail, but independent comparisons would be needed to determine how it performs against other drug-repurposing methods.

Health governance requires more than a readable output

The World Health Organization's 2021 guidance says humans should remain in control of health decisions and lists transparency, explainability and intelligibility among six principles for AI in health. It also calls for defined use cases, safety and accuracy requirements, accountability mechanisms, inclusive design and continuing assessment after deployment.

Those requirements address different failure modes. Documentation can show how a system was built and where it should be used; an explanation can help a user examine a particular output; validation tests performance in the intended population; and monitoring can detect problems after deployment. None substitutes for the others.

The 10-study review found no uniform relationship between an explanation and clinician trust. It also did not test whether changes in trust improved patient outcomes. The available evidence supports designing explanations around clinical tasks and checking whether users understand them, while keeping performance, population limits and human accountability visible.

What remains unverified

Published evidence does not support a global estimate of how many hospitals routinely use explainable medical AI. APPI News could not find comparable adoption data across countries at the time of writing. Regulatory status also depends on the specific product, intended use and country, so the research examples in this report should not be read as evidence that a medical AI system is authorized for clinical use.

Explainability can make parts of a decision process easier to question, but a readable explanation may still be incomplete or misleading. For medical AI, the stronger test is whether the explanation is faithful, the system has been validated for its intended setting, and trained professionals retain responsibility for decisions.