Study shows LLMs leak personal data in clinical notes even with privacy instructions

Sensitive context is not made safe by placing it in a single prompt: research shows LLMs can leak irrelevant personal data even with privacy instructions. Separate retrieval, privacy sanitisation and generation as distinct layers, give the model only a minimal, sanitised payload, and place filtering, access control and verification outside the model.

A peer-reviewed study in Frontiers in Digital Health on ambient clinical documentation with LLMs, published on 31 March 2026, found that the language models tested can, when drafting clinical notes, include information about third parties that does not belong in the note, even when they appear to recognise privacy rules correctly. The researchers also found that separating generation and privacy editing, combined with structured output and deterministic post-processing, produced the greatest reductions among the tested approaches, but none eliminated leakage entirely.

The study used synthetically enriched transcripts and model tests. The reported figures are therefore not universal leakage rates for production environments. But the direction of the evidence is relevant for decision-makers: anyone giving sensitive context to a model cannot assume that an instruction in the prompt keeps the information within bounds.

What did the research into privacy leaks in AI-generated clinical notes show?

The study showed that LLMs can include irrelevant personal information about third parties in clinical notes, even when they have been given privacy instructions. Separating generation and privacy processing could reduce the leakage further; structured output and deterministic post-processing could add to that reduction. None of the measures examined eliminated the leakage completely. The test used synthetic data, so the exact percentages do not map one-to-one onto practice.

The mechanism matters more than the numbers. A model that correctly paraphrases the instruction "do not mention information about third parties" does not reliably act on it. In a generative model, recognising a rule and complying with it are not the same process. That is why the study supports treating privacy enforcement as a separate processing step rather than relying only on another sentence in the prompt.

Why does a privacy instruction in the prompt alone offer insufficient protection?

Because the prompt asks the model to follow a rule while the raw sensitive source is already in its context, privacy instructions alone do not reliably prevent reproduction. In our analysis, the protection should sit before the model (what goes in) and after the model (what comes out), rather than only in the model itself. An instruction changes the probability, not the possibility.

A preprint on arXiv about privacy-preserving RAG via multi-agent semantic rewriting describes that separation concretely: privacy extraction, semantic analysis and reconstruction take place separately, after which the generation component receives only a sanitised context. The original text is used solely for retrieval indexing. The authors report less exposure under the grey-box attacks examined, but acknowledge residual leakage under adaptive and multi-query attacks and give no absolute guarantee.

Our assessment: that last caveat is the heart of the open question. How these architectures perform under targeted, repeated attacks in a real production environment has not been demonstrated. An organisation adopting such a pattern buys risk reduction within a specified threat model, not proof of safety.

What does a layered architecture for secure context injection look like?

The core is that the raw sensitive source stays behind a retrieval and privacy layer, while the generation model receives only a minimal, structured and sanitised payload. Four layers, each with its own task, make that controllable.

• Source classification and purpose binding. First determine which passages are necessary and distinguish patient data, trade secrets, quasi-identifiers and ordinary context. The de-identification guidance from the Office for Civil Rights shows why that is needed: removing names does not automatically suffice when quasi-identifiers make re-identification possible.
• Separated retrieval. Use sensitive sources to find the relevant records, but give the generator only a sanitised minimum.
• Privacy-preserving transformation. Remove explicit identifiers, generalise interwoven quasi-identifiers and treat vector stores, prompt logs and evaluation datasets as sensitive artefacts. An architecture proposal on arXiv about clinical AI deployment patterns emphasizes that those stores and outputs can create additional copies of protected health information that require controls.
• Independent control. Let output filters, access policy, leak detection and human approval operate outside the model, so that a fault in the generation is not the last line of defence.

This approach translates directly to trade secrets. Keep only the facts the task needs, hold the link to real names, contract parties or internal identifiers in a separately controlled system, and prevent raw prompts, retrieval context and model output from being logged by default. See also our broader treatment of AI privacy and GDPR and of treating external content as untrusted data.

What does this mean for directors, lawyers and CISOs working with medical records or trade secrets?

Our analysis: because the Frontiers study shows that a privacy instruction in the prompt reduces but does not prevent leakage, any organisation that gives a complete record as context to an external model runs a real leak risk; the CISO should therefore now determine which systems feed raw sensitive text directly to the model and move that flow behind a retrieval and sanitisation layer. Because the RAG preprint acknowledges residual leakage under adaptive attacks, a contract with an AI supplier should include a clause on logging minimisation, retention periods for prompt logs and the right to independent leak tests. Because vector stores, evaluation datasets and model output can create additional copies of sensitive information, the lawyer should, as part of the organisation’s own privacy and compliance review, check whether its documentation accounts for these derived data flows. This is an editorial recommendation, not a conclusion that privacy-by-design documentation under GDPR Article 25 specifically requires the cited documentation. Because none of the measures examined offered complete protection, the director should record who approves an AI-drafted note or summary before it enters a record. These are recommendations from our analysis, not prescriptions formulated by the cited studies themselves.

In brief, for anyone making a decision now:

1. Map which systems feed raw sensitive text directly to a model and redesign that flow with separated retrieval.
2. Treat embeddings, logs and evaluation datasets as sensitive artefacts in the record and DPIA.
3. Place filtering, access control and leak detection outside the model, not as a prompt instruction.
4. Anchor human approval and traceability before AI output ends up in a record.

For the broader separation of tasks, see also our analysis of establishing the data route per processing step.

Sources: The article draws on a peer-reviewed study in Frontiers in Digital Health, two arXiv preprints on privacy-preserving RAG and HIPAA deployment patterns, and the de-identification guidance from the Office for Civil Rights.