Medical AI failures require a traceable chain of responsibility

The International Medical Device Regulators Forum (IMDRF) finalized 10 good machine learning practice principles for artificial intelligence (AI)-enabled medical devices in January 2025. The framework connects intended use, representative data, independent testing, human-AI interaction and monitoring across the product lifecycle.

The World Health Organization (WHO) says humans should remain in control of medical decisions, trained people should use AI under appropriate conditions, and affected people should have routes to question decisions and seek redress. Neither the WHO guidance nor the IMDRF principles assigns legal fault to a hospital, clinician or supplier in an individual incident.

Responsibility starts with a declared use

A hospital cannot assign review duties until it states what a system is meant to do. The deployment record needs to identify the intended users, patient population, inputs, outputs, care setting and decision that the output may influence. It also needs exclusions, fallback steps and a named authority who can stop use.

That scope separates a tool that drafts administrative text from software that influences a clinical decision. The same model can carry different risks when its users, population or place in the workflow changes. Medical-device classification, authorization and professional duties must be checked in every country where the system will operate.

Four control records locate a failure

A practical responsibility map separates the health care organization, clinical users, the manufacturer or supplier, and the data and information-governance team. Each position controls different evidence before deployment and after an error.

Control position	Record needed before deployment	Evidence needed after an incident
Health care organization	Approved use, local risk decision, workflow test, escalation route and stop authority	Release approval, incident timeline, restriction or suspension decision and corrective action
Clinical user	Role, training, output-review step, known limits and fallback procedure	Output reviewed, action taken, override or correction and relevant clinical record
Manufacturer or supplier	Intended use, data description, performance evidence, version identity and change process	Known issue, version history, investigation, update notice and remediation record
Data and information governance	Data sources, access rules, retention period, interface mapping and audit design	Inputs, outputs, access events, model version, timestamps and reviewer identity

This map is an APPI News synthesis of the cited governance and technical documents, not a legal allocation of liability. Its purpose is to identify which records can show where a control failed and who had authority to act.

Data and workflow evidence must describe the same system

IMDRF calls for evaluation data that represent the intended population, environment and measurement inputs, training and test sets that are appropriately independent, and external validation proportionate to risk. A result from another patient mix, hospital system or device cannot establish performance after those conditions change.

The same principles call for testing the human-AI team in the intended clinical workflow rather than testing the model alone. That assessment can examine whether users understand the output and its limits, whether the interface encourages overreliance, and how the system behaves during foreseeable misuse. A benchmark score does not answer those questions.

For a system that influences care, the local workflow can identify which trained professional reviews the output, what evidence appears on screen and what happens when the system produces no result. A separate APPI News framework organizes deployment evidence into intended use, performance, interoperability, human control and monitoring. Those controls need to refer to the same product version and clinical purpose.

FHIR can carry provenance but cannot create accountability

Health Level Seven International (HL7) develops Fast Healthcare Interoperability Resources (FHIR), a standard for electronic health-information exchange. FHIR's Provenance resource can identify the activity, entities and agents involved in creating, revising, deleting or signing a resource version, while AuditEvent records activity as system events occur.

Those structures can connect an AI output to source records and the systems or people that acted on it. FHIR does not decide which events a hospital must retain, who may access them, how long logs remain available or whether a model was clinically valid. Those controls belong to the implementation and the rules that apply to it.

An audit design for medical AI can record input identifiers, the model and software version, the output time, the user who reviewed it and the action that followed. It also needs a way to preserve corrections and overrides without erasing the earlier event. A trace that omits the deployed version cannot show whether an error belonged to the evaluated model or a later release.

Updates need separate regulatory and hospital records

Final August 2025 guidance from the US Food and Drug Administration (FDA) says a predetermined change control plan should describe planned AI-device modifications, the methods used to develop, validate and implement them, and an assessment of their effects. The FDA reviews such plans within US 510(k), De Novo and premarket approval pathways.

A modification covered by an FDA-authorized plan can proceed without a separate US marketing submission for every change when the authorized method is followed. That arrangement does not apply automatically outside the United States, authorize changes beyond the plan or decide who reviews an output in a hospital. Regulatory status and change requirements vary by country.

APPI News has separately examined how planned medical AI changes need a description, a validation and implementation plan, and an impact assessment. A hospital still needs its own release record showing which version arrived, which interfaces changed, which local tests passed and who authorized activation.

Monitoring must name a stop decision

IMDRF says deployed models should undergo risk-based monitoring in real-world use and that controls should address retraining risks such as overfitting, unintended bias and performance degradation. A monitoring file can define the baseline version, measures, review schedule, alert thresholds and investigation owner.

An incident record should connect a signal to the decision that followed. The operating policy can name who may narrow use, suspend the system, restore a prior version or approve a return to service. Without those permissions, a dashboard can show deterioration while leaving no one able to act.

WHO also calls for continuous and transparent assessment during actual use. Monitoring therefore belongs to the responsibility chain rather than to a technical dashboard alone. The record needs to show whether a warning led to review, restriction, remediation or no action.

Six records make the chain auditable

A procurement and deployment file can combine six records before routine use begins. The list below is an APPI News synthesis, not an IMDRF certification program or a legal test in any country.

1. Intended use and exclusions: The record identifies the users, population, inputs, outputs, setting, decision influenced, prohibited uses and fallback.
2. Data and evaluation: It describes data sources, patient and site coverage, equipment, training-test separation, reference standards, subgroup results and unresolved limits.
3. Human workflow: It tests training, output interpretation, review authority, overrides, missing results and reasonably foreseeable misuse in the clinical setting.
4. Traceability: It links inputs, outputs, access events, model and software versions, timestamps, reviewers, actions and retention rules.
5. Change control: It covers update notices, acceptance tests, version locking, rollback and the regulatory status of each release in every market where it will be used.
6. Incident response: It names alert thresholds, investigators, reporting routes, suspension authority, corrective action and the conditions for returning to service.

The records need to agree with one another. A contract that permits an unannounced model update conflicts with a validation report tied to a fixed version, while a monitoring plan without a stop authority leaves an operational gap.

The framework preserves evidence, not a verdict

WHO and IMDRF provide governance principles, while HL7 provides technical record structures and the US FDA guidance addresses specified US device changes. None of those documents determines liability in an individual case. APPI News did not verify medical AI liability rules in every country, so this article does not identify which party would be legally responsible after a particular incident.

A connected record can still show the intended use, data source, deployed version, output, reviewer, later action and authority to suspend the system. It does not settle legal fault, but it preserves the facts that a hospital, regulator or court may need to examine.