AI Privacy Is Not a Post-Processing Problem

What CLIOPATRA reveals about where the missing boundary actually sits

There is a slightly uncomfortable possibility beginning to emerge in enterprise AI governance.

Not that organisations are ignoring AI privacy. In many cases they are doing the opposite. They are writing policies, setting up approval boards, configuring model gateways, reviewing vendor terms, running DPIAs, limiting retention, drafting acceptable-use language, and asking security teams to examine prompt logs.

All of that is sensible. Some of it is overdue.

But I think there is a more awkward question underneath it: what if a large part of the enterprise AI privacy effort is happening too late?

The instinctive model for governance is still quite linear. Data goes into a system. The system processes it. Then controls examine the output, review the logs, remove the obvious identifiers, and decide whether anything unsafe has happened. That model made sense when the objects of control were databases, documents, emails, endpoints, and network flows. It becomes less comfortable when the system is a language model, because the model does not merely store or transmit data. It interprets it, embeds it, summarises it, clusters it, and converts it into intermediate representations that may no longer look like the original data but may still carry the same sensitive meaning.

That distinction matters. In AI systems, privacy leakage is not always a file leaving the building. It may be a pattern inferred from many interactions. It may be an attribute reconstructed from a cluster. It may be a sensitive detail that survives redaction because it was never a direct identifier in the first place. It may even be a governance dashboard, built to protect users, becoming a new surface through which their private information can be inferred.

That, I think, is why the CLIOPATRA paper is worth taking seriously. A link to the original paper on Arxiv is here

CLIOPATRA, formally "Extracting Private Information from LLM Insights," is not another generic warning that people might paste secrets into chatbots. We have had enough of those, and most security teams already understand that problem. The paper is more interesting because it examines something subtler: whether privacy-preserving LLM insight systems can themselves be induced to reveal private information.

The researchers focus on systems like Anthropic's Clio, which was presented as a way of generating privacy-preserving insights into real-world AI use. Clio's own description is careful and thoughtful. It distils conversations into abstracted clusters, with privacy protections such as anonymisation, aggregation, and higher-level analysis. In broad terms, that is exactly the capability many enterprises will want. A CISO wants to know how employees are using AI. A governance team wants to know whether users are putting regulated data into models. A safety team wants to identify misuse patterns. Nobody wants to read raw private conversations one by one.

So the insight layer appears to solve a real problem. It lets an organisation see patterns without exposing the individual.

CLIOPATRA challenges that assumption.

The attack is elegant, in the unpleasant way that good security research often is. The adversary does not ask the system to reveal a secret. Instead they insert carefully crafted malicious chats into the dataset being analysed. Those chats are designed to cluster with a target user's chat, influence the summarisation, trigger the exposure of sensitive information, and slip past the privacy auditor. In the medical scenario tested, the attacker knows only limited public information about the target, such as age, gender, and one symptom. Yet the attack extracts the medical history from up to 65% of target chats, and with nearly 100% precision in the configurations where it commits to a guess.

The paper is cautious in scope. It uses synthetic medical target chats mixed into a larger dataset. It is not claiming that every enterprise AI insight system is currently leaking in precisely this way. Nor should anyone read it as a simple indictment of Anthropic; the authors are explicit that Clio was chosen partly because it is unusually well documented, which is the opposite of a reason to single it out. The better reading is not "Clio bad." It is "this class of architecture deserves closer scrutiny."

The really important point is that the failure is not the failure of one obvious control. It is the interaction of several controls that each look reasonable in isolation.

PII redaction sounds reasonable. Clustering sounds reasonable. Aggregation sounds reasonable. Summarisation sounds reasonable. LLM-based privacy auditing sounds reasonable. And yet the researchers show that an adaptive adversary can design inputs that exploit the seams between those layers.

Two of their findings are worth sitting with, because they are the ones a CISO probably has not heard before.

The first is about redaction. The extractor model is explicitly instructed to strip personal information from each chat before anything downstream happens. It does not reliably do so. Across the model families tested, age survived in up to 44% of facets, gender in up to 63%, and symptoms in up to 92%. So the redaction layer is, at the same time, too lossy to leave the data fully useful and too leaky to leave it safe. That is an awkward combination, and it is not a tuning problem you optimise away; it is what happens when you ask a model to reason about what is sensitive at the same moment you ask it to reason about everything else.

The second finding is sharper still, and I suspect it is the one that should make technical governance teams pause. The system's last line of defence is an LLM auditor that scores each cluster summary from 1 (least private) to 5 (most private), with the cluster dropped if it scores too low. In the runs where the attack successfully extracted a disease, the auditor flagged essentially nothing. More than half of the successful-attack clusters were rated a perfect 5 out of 5. Swapping in a more capable auditor, Claude Sonnet 4.5, caught no more than the weaker one did. The auditor justified its high scores by noting the absence of explicit identifiers, and the fact that age, gender, and a symptom are individually unremarkable.

Which is true, and entirely beside the point. The whole attack is built out of individually unremarkable facts that become identifying in combination. An LLM checking another LLM's output for leakage is the same failure mode, one layer up. You have not added an independent control; you have added a correlated one, and called it defence in depth.

This is where I think we have become too comfortable. We tend to assume that weak privacy controls can be stacked into strong privacy architecture. In some domains that holds, but only when the controls have genuinely independent failure modes. In LLM-based systems the layers are usually not independent. They all rely, in different ways, on a model's capacity to understand what is sensitive, what is instruction, what is context, what is safe to reveal, and what is merely statistically descriptive.

That is a great deal to ask of a model. It is also where CLIOPATRA meets the broader prompt-injection debate. The UK's National Cyber Security Centre has made the point that current large language models do not enforce a robust internal boundary between instructions and data inside a prompt. The consequence is very practical. If an AI system reads untrusted content, that content may not stay passive. It may become instruction-like. It may alter what is retrieved, summarised, prioritised, suppressed, or disclosed. OWASP puts prompt injection and sensitive-information disclosure among the core risks for LLM applications; NIST's generative AI profile flags indirect prompt injection and data poisoning in similar terms. The point is not that every model is catastrophically unsafe. The OWASP top 10 for LLMs is here.

The point is that the old separation between "content" and "control" is much weaker in AI systems than most enterprise architectures were built to assume.

This is where I think the phrase semantic boundary earns its place.

We already have a mature language for certain boundaries. TLS protects data in transit. IAM controls who can access what. DLP tries to detect regulated content leaving approved channels. CASB and SSE give security teams visibility across cloud services. CNAPP secures cloud workloads. SIEMs collect evidence after the fact. None of these is obsolete. But none of them quite describes the thing enterprises now need between sensitive organisational data and AI reasoning systems.

A semantic boundary is a different kind of control. Its concern is not where the packet goes, or who clicked submit, or whether the final answer contains a national insurance number. Its concern is what meaningful content is allowed into the AI system in the first place.

That sounds abstract. In practice it is concrete, and CLIOPATRA gives us a clean way to make it concrete.

Take the paper's own worked example. The target chat reads, roughly: a 55-year-old woman with bone pain, pelvic weakness, functional disability, and a history of vitamin D deficiency. The attacker knows the age, the gender, and one symptom, and reconstructs the rest from the cluster summary. The whole attack hinges on a small set of true facts surviving into the system and clustering together.

Now consider what the traditional answers do to that chat.

Blocking stops it. It is sometimes necessary, especially for clearly prohibited data or uncontrolled external tools, but it is not an adoption strategy. If every useful prompt is stopped, people route around the control: an unapproved model, a personal account, a pasted excerpt, a screenshot, a "temporary" workaround that becomes permanent. Security teams know this pattern. Controls that destroy productivity tend to produce less security, not more.

Redaction strips the woman's age, gender, history, and symptoms. The model can still produce fluent output, but it is no longer reasoning about a real clinical picture; in legal, medical, financial, and HR contexts, that loss of fidelity makes the answer useless or, worse, plausibly wrong. And, as the paper shows, the redaction does not even hold; the age and gender leak anyway.

Hashing and tokenisation turn the patient into TOKEN_4571. A model cannot reason clinically about a token. The more human-readable context you strip, the less useful the model becomes, and the more you are back to the redaction trade-off in a different costume.

Context-preserving pseudonymisation takes a different route, and it is the one place where there is a specific answer to this specific attack. The 55-year-old woman becomes a different but equally coherent person: a plausible surrogate name, a realistic but non-real demographic profile where the clinical relationships still hold. The model receives language it can reason over. The raw sensitive values never cross the boundary. And the attacker's regex on "55-year-old female, bone pain" now lands on a coherent-but-wrong individual. You have not removed the signal, which is what breaks the model; you have substituted a different, internally consistent signal. The cluster still forms, the summary still reads naturally, the reconstruction still happens, and it reconstructs a fiction.

This is the kind of architectural territory products such as AliasPath are exploring. The interesting point is not the product name. It is the pattern: govern the data path into the model, not merely the model's answer on the way out. If the original real values rehydrate, they do so for authorised users inside the boundary, under logging and access control, while every downstream artefact, every facet, embedding, summary, cluster, and dashboard, carries the surrogate.

There is a caveat I want to put on the table rather than smooth over, because a careful reader will spot it otherwise. Clio is an analytics system. It sits downstream, ingesting raw chats to produce insight. A semantic boundary on the inference path does not, by itself, defend Clio, because Clio is not on that path. What it does is more upstream than that: it changes what counts as "raw truth" in the corpus at all. If the sensitive medical detail had entered the workflow as a surrogate to begin with, the analytics layer would have been clustering and summarising fiction, and the breach would have exfiltrated fiction. The defence is not "bolt a boundary onto Clio." It is "stop assuming the second-order system is safe to feed with first-order truth."

Because that assumption is the quiet problem here. Many enterprises treat AI monitoring as a safe, second-order activity. The model interaction happens; then another system analyses usage and produces insight. But if that second-order system ingests raw interactions, it inherits the sensitivity of everything it analyses, and it may become a more attractive target than the original system, because it concentrates cross-user prompts, topics, classifications, and summaries in one place.

The governance layer becomes the breach surface. CLIOPATRA is, among other things, a demonstration of exactly that.

None of this means abandoning AI usage analytics. Enterprises need it; without it they cannot understand adoption, misuse, policy violations, data exposure, or emerging risk. It means the analytics has to be privacy-preserving by design, not privacy-labelled by intention, and that the material it reasons over should already have crossed a semantic boundary before it arrives.

There is also a private-model trap worth naming, because it is the most common rebuttal from large institutions. "We run our own models inside our own environment, so this is solved."

Sometimes that materially reduces risk; keeping data in a controlled environment is genuinely different from sending it to a public consumer chatbot. But it does not dissolve the semantic problem. Private models still have prompts, logs, embeddings, retrieval stores, agents, tools, and administrative users. They still process untrusted content. They still generate intermediate artefacts. A private model may even increase internal amplification: if permissions are messy, it surfaces information faster than the access model was designed to tolerate; if retrieval is broad, it makes dormant oversharing newly actionable. The issue was never only model location. It is data movement and data meaning.

This is why I suspect "AI data trust layer" is a more durable idea than the current vocabulary of "AI DLP." DLP is part of the story, but the term carries baggage: detection of known patterns, prevention of exfiltration. The AI problem is wider. It includes semantic transformation, purpose-aware handling, context preservation, governed rehydration, prompt-injection resilience, and evidence over derived representations, not just original ones.

A serious version of that layer has to do several things that pull against each other. It has to detect sensitivity that is contextual, not just formatted; a merger target or a disciplinary process is sensitive in ways no regex for passport numbers will catch. It has to transform content while keeping enough structure for the model to stay useful, which means consistency across a session, a document, and a whole matter; if "Sarah Malik" becomes "Rebecca Khan" in one paragraph and "Emma Wilson" in the next, the reasoning breaks. It has to treat rehydration as the serious part, because pseudonymisation is not anonymisation. The ICO and EDPB are both clear that pseudonymised data remains personal data where it can be reattributed; the governance of that reattribution key is therefore central, not incidental. A link to their work is here, and it has to produce evidence, because AI governance is drifting from "do we have a policy?" to "can we prove what happened?", which is a meaningful shift.

It also has to assume hostile input, which is the part CLIOPATRA insists on. The threat is not only the careless employee oversharing. It is adversarial text arriving through customer messages, web pages, documents, RAG sources, support tickets, repositories, third-party tools; text engineered to poison clusters, bend summaries, and induce disclosure later. The security question is not only "what did the user type?" It is "what untrusted content is the system about to treat as meaningful?"

I should be honest about the limits, because presenting pseudonymisation as a cure would be exactly the kind of heuristic overconfidence the paper is warning against. Poor detection misses sensitive content. Poor transformation destroys utility. Poor rehydration governance creates a fresh insider-risk problem. Deterministic surrogates can still leak patterns to an attacker with enough auxiliary information. Some workflows genuinely need raw data inside a controlled environment. And any control with too much friction pushes users straight back into shadow AI. The category has to be treated as engineering, not theatre.

The defensible version of the argument is narrower than "a semantic boundary eliminates AI privacy risk." It is this: without some form of semantic boundary, many organisations will be left relying on after-the-fact controls over systems that have already consumed the sensitive truth. They will be asking output filters, privacy auditors, dashboards, and policy documents to solve a problem that began several steps earlier. CLIOPATRA is useful precisely because it makes that timing visible. It shows leakage emerging not from the final answer but from the machinery of insight itself; that an attacker may not need access to a target's data if they can shape how that data is clustered and summarised; and that "privacy-preserving" is not a property you obtain by chaining a few plausible heuristics and asking another model to check the result.

The more uncomfortable lesson is that AI privacy may be less like document security and more like contamination control. Once raw sensitive meaning enters the workflow, it can move, diffuse, transform, and reappear in places that no longer resemble the source. By then the question is not whether the answer contains a name. It is whether the system has already learned, encoded, grouped, or exposed something it should never have been allowed to process in that form.

Which leaves enterprises with a fairly practical question.

Not whether they have an AI policy. Most will. Not whether they have a model gateway. Many will. Not whether the vendor says prompts are not used for training; that may matter, but it is not the whole issue.

The harder question is whether they can govern the sensitive meaning entering AI workflows before it becomes part of the machinery.

Because if they cannot, much of AI governance ends up auditing the consequences of a boundary that was never really there.

Rob Westmacott, Founder, AliasPath™ Opinions are my own.