The Prompt Gap: The things we think and do not say about AI prompt security

AI prompt security; prompt gap; AI governance gap

Somewhere in your organisation, right now, someone is about to paste something into a chatbot.

It is 10:47pm. They are not the employee in the training video; they are not the satirical figure who emails their credentials to a Nigerian prince. They are a competent, senior, mostly sober professional with a deadline in the morning and a tab open to whichever model they privately prefer. They will paste a document, they will get a fast and useful answer, they will close the laptop, and they will go to bed. Nothing visible will happen. The logs will not light up. The DLP tool will not scream. The AI steering committee will not convene.

And yet, arguably, quite a lot has happened.

Most enterprises have spent the last eighteen months building what might be called the upstream half of AI governance. There are policies, and DPIAs, and vendor questionnaires, and approved model lists, and mandatory training (everyone has done the training; everyone did it on their phone during the Tuesday sales call). There is a steering committee that meets on alternate Thursdays. There is a slide in the board pack, rendered in the corporate typeface, with green ticks next to most of the boxes.

Then, at 10:47pm, a prompt leaves a keyboard, and between the policy and the paste there is nothing.

This is, I think, the defining unsolved problem of enterprise AI adoption, and it does not have a clean name, which is part of the reason no one is solving it properly; you cannot buy a product category that has not been named, and budgets cannot be allocated to a gap. So let me propose one.

I want to call it the Prompt Gap.

The Prompt Gap is the space between the governance programme upstream (policy, approved vendors, training, documentation) and the risk event downstream (a prompt, containing real data, arriving at a model and being processed). Your governance lives on the left. Your risk lives on the right. In between sits a gap where none of your controls actually operate, and this gap is structural, not operational, which means no amount of policy refinement or training frequency will close it.

To see why it is structural rather than operational, it helps to look at three ordinary paste events across three different industries, each of which illustrates a slightly different flavour of the same underlying problem.

I. The banker

It is the Sunday before a Monday morning diligence call. A VP at a mid-market investment bank is working through a confidential information memorandum for a live sell-side mandate: forty-seven pages, a confusing management presentation appendix, synergy tables that do not reconcile with the standalone financials. She needs a concise summary of the commercial narrative before 8am, and she needs to flag any inconsistencies the buyer's team might raise on the call.

She pastes the CIM into a frontier model and asks for a three-page summary with a table of discrepancies. She gets one. It is, I think, a genuinely useful piece of work.

What has also happened, in addition to the useful summary, is that the name of the target, the names of several of its largest customers, a financial forecast that constitutes material non-public information, and the commercial terms of a live M&A process have all been transmitted to an inference layer outside the firm's perimeter. The firm's Chinese walls, which were carefully designed in the physical era and then ported with some difficulty to the digital era, have now been breached at the prompt layer; the model does not know there are supposed to be walls.

The firm has a policy about this. The policy says do not do this. The VP has done the training.

Everyone has done the training.

II. The clinician

A consultant at a large teaching hospital is writing a discharge summary for a complex patient: multiple comorbidities, a non-standard treatment pathway, several letters from community teams that need synthesising into a single paragraph for the GP. She has sixteen minutes before her next patient.

She pastes the letters into a chatbot, having first deleted the patient's name from the top. She knows, roughly, that she is supposed to de-identify before doing this, and she has de-identified, and she is quietly proud of herself for de-identifying, because most of her colleagues do not bother.

What she has not done, because no training module has ever explained this to her, is remove the quasi-identifiers. The date of birth is still there. The postcode is still there. The unusual combination of conditions is still there, along with the name of the specialist clinic she was referred to in 2022, which treats roughly eleven patients a year with that profile. The model does not need the name; the pattern is the identifier. The clause, as it were, is the fingerprint.¹

Her hospital has a policy. It says de-identify. She has de-identified. Everyone involved feels, broadly, that they have done their job.

III. The consultant

A senior Partner at a large professional services firm is working on an engagement for Client A, a consumer goods company, while her colleague two desks away is working on a separate engagement for Client B, who competes with Client A in three product categories. The firm has an internal AI assistant, deployed through the official enterprise tenant of a major model provider, connected via a retrieval-augmented pipeline to the firm's document management system so that colleagues can ask questions and get grounded answers from prior work.

She asks the assistant a question about pricing strategy in one of the overlapping categories. The system retrieves relevant context from the vector store, including, because the vector store does not know about client walls, a passage from the Client B engagement that her colleague was working on last Tuesday. The response she receives is grounded, in part, in competitor confidential information. She does not know this. The model does not know this. The firm's engagement letters, which promise both clients strict confidentiality, do not know this either.

The firm has a policy. The policy talks about access controls. The access controls, which are very good, govern who can see which documents in the document management system. They do not govern what the model ingests during retrieval, because retrieval is not, technically, viewing.

Why the Prompt Gap exists

The Prompt Gap exists because every piece of enterprise risk technology we have inherited from the last two decades was built for a different kind of risk event.

In the perimeter era, the bad thing happened somewhere you could watch. A file left your network; an endpoint connected to an external drive; an email went to the wrong address; a login happened from an unusual country. Your tools looked at the thing leaving and, with varying degrees of sophistication, decided whether to stop it, allow it, or log it for later unpleasantness. This worked reasonably well for a long time, not because the tools were especially clever but because the risk surface was observable.

The risk surface of generative AI is the inference call itself. It is not a perimeter event, because the perimeter was crossed the moment the enterprise decided (correctly, I think) that its people needed to use AI. It is not a data-at-rest event, because the data in question is transient and lives, very briefly, in the body of a prompt. It is a paste-and-enter event; it happens in a microsecond; the data it exposes is whatever happens to be salient to a tired professional at 10:47pm; and by the time it is exposed, it has already been processed.

The upstream controls cannot see this event, because they live in PDFs. The downstream controls cannot change this event, because they observe rather than transform. The model-layer controls cannot govern this event, because the model's job is to read the prompt, and by the time the model is reading the prompt, the data has already left.

So there is a gap, and the gap is where the risk actually lives.

Why the obvious solutions do not close it

Three things get pointed at when the Prompt Gap comes up in enterprise conversations, and none of them close it.

Data loss prevention tools are, broadly, block-or-allow instruments built for exfiltration in the perimeter era. When pointed at AI prompts they do what they know how to do; they block.

Blocking breaks the workflow. Broken workflows get routed around, usually via a personal phone, which is a surface you cannot see at all. A DLP tool configured aggressively against AI prompts does not eliminate the risk; it moves it, and the place it moves it to is worse than where it started.

Monitoring and logging tools are after-the-fact. They tell you, sometimes hours later and sometimes days later, what was sent and to whom. This is useful for forensics and unhelpful for prevention; the prompt has already been processed, the model has already generated, the regulator has, in the scenarios where one is involved, already been given the material they need.

There is a whole genre of AI governance product that is essentially a dashboard showing you the last month of your exposure. It is a dashboard. It is not a control. It is not effective AI prompt security.

Policy and training are the most honest failures in the list, because everyone in the room knows they are a hope. Policy is a document describing what people should do if they were different people; training is the same document, with a video. Both assume that at 10:47pm, with a deadline at 9am, the VP will remember the module she clicked through in March. Sometimes she will. Often she will not. The distribution of outcomes is not the distribution you want to brief a board on.

The ownership vacuum

One reason the Prompt Gap persists, beyond the structural reasons, is that no one in the modern enterprise is quite sure whose problem it is.

The CISO looks at it and sees a data question, which feels like a DPO problem. The DPO looks at it and sees an exfiltration question, which feels like a CISO problem. The General Counsel assumes that IT has bought a product for this, because IT buys products for things. IT has not bought a product for this, because the product category has not been clearly named, but IT has sent an email, and the email went out, and the email said please be careful, and everyone agreed to be careful.

The result is a kind of ownership vacuum in which each function assumes another function is closing the gap, and the gap remains open, and the prompts keep going.

What the answer actually has to look like

A solution to the Prompt Gap, if one existed, would have a particular shape. It would have to operate at the prompt layer itself, between the keyboard and the model, which is the only place where the risk actually lives. It would have to transform rather than block, because blocking breaks workflows and broken workflows get routed around. It would have to preserve semantic meaning, because a model given meaningless input produces meaningless output, and an AI programme that degrades output quality does not survive its first quarterly review. It would have to be reversible, because the answer the model produces has to come back to the user in a form they can actually use, which means the transformation has to be undone on the way out.

It would have to work the same way whether the person pasting is a banker, a clinician, or a consultant, because the Prompt Gap is not a vertical problem. It is a structural one, and structural problems respond, in my experience, only to structural answers.

And it would have to exist in a place that none of the current controls occupy, which is: architecturally, between the human and the inference call, transforming the data in flight.

I notice, writing that paragraph, that I have described the shape of a solution without naming a product. This is on purpose. The point of this essay is not to sell anything; the point is to give a name to a problem that has been sitting, unspoken, in a lot of people's heads, so that the next procurement conversation can at least start from the right question.

The question worth asking about AI prompt security

When your board, sometime in the next four quarters, asks what sits architecturally between your people's keyboards and the models they are pasting into, what is the thing you point at?

Not the policy. Not the training. Not the dashboard that shows last month's exposure. Not the DLP tool that breaks the workflow badly enough that everyone uses their phone.

The thing. The layer. The control that fires at 10:47pm, on a Sunday, when the VP has a deadline in the morning and the model is waiting.

If the honest answer is nothing, that is worth knowing, and it is worth knowing now, and it is worth knowing before the first regulator, board, or plaintiff's lawyer gets there first.

Footnotes

1. The clause-as-fingerprint problem generalises well beyond healthcare; in any document-intensive profession, the structure and phrasing of a passage is often as identifying as the named parties within it, which is why naive redaction approaches tend to underperform in practice and sometimes quite badly.