Six controls health organizations need before deploying medical AI

Health organizations testing generative artificial intelligence need to settle how they will protect data, reconstruct decisions and stop unsafe outputs before deployment. International guidance does not supply one universal technical design, but it does identify controls that can be tested before a system reaches staff or patients.

The six controls below cover audit logs, de-identification, data boundaries, traceability, human authority and credential security. They apply whether an organization builds its own model interface or buys a service, although the legal and clinical requirements still depend on the country and intended use.

Audit logs should preserve evidence without copying secrets

An audit record can become a second store of sensitive information if it copies prompts, clinical text, access tokens or credentials. The exposure may be wider than in the source system because support teams, monitoring services and outside vendors often receive access to operational logs.

OWASP says sensitive personal data, including health information, and secrets such as access tokens, passwords and encryption keys should usually be removed, masked, sanitized, hashed or encrypted rather than recorded directly in logs. It also says logs must be protected against unauthorized access, modification and deletion.

A safer audit design defines the minimum fields needed to reconstruct an event, such as a request identifier, timestamp, policy version, model version, outcome and reviewer action. If the original content must remain available for an investigation, it can stay in a separately controlled system rather than being duplicated into a broadly accessible log stream.

De-identification needs a documented re-identification test

Removing names does not necessarily make a record anonymous. Dates, rare conditions, locations and other indirect identifiers can distinguish a person when combined with outside information. A project therefore needs to state whether it is anonymizing records or merely replacing direct identifiers with pseudonyms.

The UK Information Commissioner's Office says pseudonymized data remains personal data for a party that can reconnect it to a person with separately held information. Its guidance tells organizations to define their goal, assess likely attackers, protect mapping information and document their choices and risk assessment.

The care.data program in England illustrates why technical assurances must be paired with clear governance and public communication. An archived NHS England page says the program planned a notice to 22 million households explaining data use and the right to object, and records that the program closed in 2016. That record does not prove that every later health-data initiative will fail, but it shows that public notice and objection mechanisms are part of deployment design rather than an afterthought.

Data boundaries should be explicit and independently tested

Keeping source documents or a retrieval index inside an organization's own environment can reduce the material sent to an external model service. It is not a complete privacy control. Retrieved passages, user prompts and generated answers can still disclose information, while administrators or compromised accounts may reach the local store.

A deployment review should map every transfer instead of relying on labels such as “private cloud” or “de-identified.” The map should cover source ingestion, index creation, retrieval, prompt construction, model processing, output storage, telemetry, backups and support access. Each boundary needs a retention rule, an access owner and a test showing what leaves it.

Outputs need provenance and repeatable evaluation

Generated text can sound authoritative while being false. The US National Institute of Standards and Technology's 2024 generative AI profile identifies confidently presented erroneous content as confabulation and describes particular risks when people rely on it in health care.

The NIST profile recommends checking sources and citations in generated content, documenting how human oversight is applied and tracking changes that affect content provenance. Retrieval from approved material can narrow the evidence available to a model, but a source link is useful only if it supports the specific statement beside it. Evaluators should test unsupported claims, incorrect citations, omitted qualifications and answers produced when the source set contains no answer.

Repeatability also depends on configuration records. An organization should be able to identify the model, system prompt, retrieval collection, policy set and evaluation version used for a past output. Reproducing the exact wording may be impossible with a nondeterministic model, but the organization can still reconstruct the inputs, controls and decision path.

People must retain authority over consequential decisions

A status dashboard can help a trained reviewer find a failed check, missing source or low-confidence result. It should not turn a collection of automated checks into an unexplained approval when the output affects care, eligibility or access to services.

World Health Organization guidance says humans should remain in control of health systems and medical decisions, and that people affected by algorithm-based decisions need mechanisms to question them and seek redress. Human review is effective only when the reviewer has enough information, time and authority to reject an output. The escalation path and final decision owner should be assigned before launch.

The WHO's 2023 regulatory considerations call for documentation across the product lifecycle and attention to intended use, human interventions, training models and cybersecurity threats. Those controls also make it easier to identify when a system is being used outside the conditions in which it was assessed.

Credentials and monitoring need their own release checks

A system designed to protect health data can still expose an application programming interface key through an error message, build artifact or support bundle. Secret scanning should cover the finished deployment package and runtime output, not only the source repository. Suspected exposure should trigger revocation or rotation through a rehearsed incident process.

Post-deployment monitoring should track failed safeguards, unusual data flows, reviewer overrides, complaints and changes in output quality. These records should feed an incident-response process and a clear decision on when to restrict, roll back or retire the system.

Procurement should demand evidence for each control

Compliance claims are difficult to evaluate when vendors provide only architecture diagrams or broad assurances. Buyers can ask for a data-flow inventory, logging fields, deletion procedures, evaluation results, change records, incident responsibilities and evidence that reviewers can override the system. Contract terms should identify subcontractors and the countries where data is processed.

No control eliminates every failure mode. The practical goal is to make claims testable, limit the data exposed by any one failure and keep responsibility with an identifiable organization and trained people. APPI News found no independent measurements for the product design and embedding-model comparison in the source report, so this analysis does not repeat them.

Frequently asked questions

Does keeping a retrieval index on premises make cloud AI private?
No. It can reduce the source material transferred to a provider, but prompts, retrieved passages, outputs, telemetry and support access still need separate controls and testing.

Is pseudonymized health data anonymous?
Not necessarily. If an organization can reconnect the data to a person using a key or other information, the records remain identifiable to that organization and require protection.

Can citations stop a model from producing false claims?
No. Retrieval and citations can support verification, but evaluators must check whether each cited source exists and supports the associated statement.

What evidence should be retained for a past AI decision?
The record should identify the inputs or controlled pointers to them, policy and model versions, retrieval collection, automated results, reviewer action and final decision owner without duplicating unnecessary health data or credentials.