PII Detectors Are Not Confidentiality Detectors

The question a privacy gate actually has to answer

Most “AI privacy” setups put a PII detector in front of the cloud model: find names, e-mail addresses and card numbers, mask them, send the rest. That deals with personal data. It does not answer the question a company actually cares about: may this text leave the building at all?

A negotiated price for one customer, a trading position, a draft board announcement, a legal dispute in progress: none of these needs a single name or e-mail address to be confidential. And an ordinary “lunch at 12?” mail is full of names but harmless.

We built a benchmark to measure how far apart the two questions are, on real company mail rather than synthetic examples. The short answer: very far.

Data: real mail, not generated examples

• Corpus: the public Enron e-mail corpus (May 2015 release), US energy-trading mail from 2000 to 2002. After cleaning, 3,690 e-mails were labelled and split by mailbox, so no custodian’s mail is in both training and test.
• Test set: 800 e-mails from 57 mailboxes. A calibration set of 300 e-mails (used only to pick thresholds) and a reserve of 890 were kept separate.
• Masking: e-mail addresses and phone numbers were replaced by placeholders ([EMAIL_1], [PHONE_1]) in every item before any system saw it; person names were not. In a 100-item audit every e-mail address was masked, there were no false redactions, and 4 internal phone extensions were missed.
• Context pairs: 129 public SEC documents, each shown twice with a one-line context: once “draft, before publication” (must stay local) and once “published” (may go to the cloud). The text is otherwise identical.

What “confidential” means here

Two annotators from different model families, Codex (GPT-6) and Claude Opus, labelled every item independently and blind, with the same written guidelines. The situation in the guidelines: an employee wants to paste this text into an external cloud AI assistant with no special confidentiality contract; judge it as of the moment it was written, from the organisation’s point of view.

Each item gets one of four levels:

Two rules matter most for this article. Incidental names and contact details alone do not make a text confidential. And numbers alone are not confidential: a public price list is public, a negotiated price is not.

The two annotators agree on the route for 84.5% of test e-mails (Cohen’s κ 0.69). Codex is stricter: it calls 56.1% of test mail local, Opus 44.1%. Rather than pick one, every result is reported under four label definitions (union, Codex, Opus, intersection). Under the union label used in the headline (local if either annotator says so), 463 of the 800 test e-mails are confidential. Under the intersection, 339. On the context pairs both annotators followed the context line in all 129 cases.

These are silver labels from two models, not human gold. We come back to this below.

Systems and the metric

We scored 26 systems in five families: PII and secret detectors (Presidio 2.2.366 with spaCy, the PasteGuard detector, a simple “contains contact details” rule), trained classifiers (TF-IDF with logistic regression, e5, bge, XLM-R; trained on Enron training mail, on synthetic data, or both), hosted zero-shot LLMs (Claude Haiku 5.5, DeepSeek V4.1 Flash, MiMo V2.6 Flash, through zero-data-retention endpoints or a subscription, masked text only), a small local zero-shot LLM (Ministral 3B), and an open local decision model (basal-omni-1.0, Apache-2.0, run with llama.cpp).

The metric is what a gate operator feels: leak = share of confidential mail routed to the cloud, measured at a fixed level of over-blocking (share of harmless mail kept local). Each system’s threshold was set on the calibration set as the lowest one that keeps at most 20% of harmless mail local.

Results: PII detection ranks confidentiality at chance

Test set, 800 e-mails, union label, 95% bootstrap intervals over whole mailboxes:

The bottom of the table is the point of the title. An AUROC of 0.5 is a coin flip. Used as a confidentiality signal, Presidio’s standard “highest entity score” ranks confidential mail at 0.506. Nearly every e-mail contains a person’s name, so to keep 80% of harmless mail flowing the threshold has to sit above almost every score, and all confidential mail goes to the cloud. Combining every span into one score (noisy-OR) helps a little and still sends 61% out. PasteGuard and a contact-detail rule do no better.

That is not a bug in those tools. They answer a different question correctly. Business confidentiality lives in what the text says: a volume, a deal ticket, an opinion about a regulator’s next move. Personal data is mostly incidental.

Context separates the families sharply. On the 129 SEC pairs the only difference is the line “draft before publication” vs “published”. Every hosted LLM and basal routed both versions of every pair correctly (100%). Ministral 3B got 81%. Every trained classifier got close to 0%: it reads vocabulary, so both versions get the same score.

Synthetic training data did not transfer. The same classifiers trained only on synthetic mail reached AUROC 0.58 to 0.63 on real Enron test mail and leaked 62 to 69% of confidential mail. If your confidentiality model was trained on generated examples, test it on your own mail before trusting it.

The pre-registered verdict: inconclusive

The benchmark was designed to decide one thing: does the local model beat the best cheap trained classifier by a clear margin? Scope, systems, decision rule and labels were written down and frozen (43 files with sha256 hashes) before any system ran, after eight rounds of review of the plan.

The frozen rule compares basal with each classifier family’s best entry at matched over-blocking (each classifier re-thresholded to basal’s over-blocking on the test items). “Continue” needed two things under all four label definitions: the lower bound of the leak difference above zero against every entry, and basal leaking at most 5% of the mail both annotators call confidential.

Under the union label basal leaks 11.2 points less than the bge comparator [+4.6, +20.6]. Under all four definitions it leaks less than every classifier family’s best entry. But against e5 the lower bound is 0.0 (union) and −0.3 points (Opus labels), and under the intersection label basal still leaks 6.5% of the mail both annotators call confidential, above the 5% screen. The rule’s output: “neither rule holds with the same reason under every label definition”, so inconclusive.

The plan allowed a second look on the reserve data, provided the model gave the same scores as in the first look (tolerance 0.001). It did not. Identical requests to the same llama.cpp server gave scores up to 0.06 apart, and up to 0.15 when re-scoring calibration mail. Loosening the tolerance after seeing the first result would have bent the rule, so we did not. The deviation is written up as such, and the reserve was scored only descriptively. It agrees with the test set (basal 13.6%, Haiku 11.5%, DeepSeek 17.2%, e5 21.5% leak, on 1,690 e-mails).

Run-to-run variation turned out to be a finding of its own. Re-scoring the same 300 calibration e-mails, basal kept 99.3% of its routing decisions. The hosted LLMs kept 86% (Haiku) to 95%. Classical pipelines were bit-for-bit stable. A gate built on a generative model will route some borderline texts differently on a second try.

Weak spots

• Labels come from models. κ 0.69 between two model families is decent, not gold. Human labels on a subset are planned and not yet in. 26.8% of union-confidential test mail is confidential for only one annotator, and basal leaks 40% of those borderline items.
• Family overlap. Claude Haiku shares a family with the Opus annotator, so part of its lead may be label agreement. DeepSeek is the cleaner independent comparison, and at matched over-blocking it is level with basal (−0.4 points [−2.5, +2.1]).
• Operating points differ. The 20% threshold is chosen on calibration data, so realised over-blocking on test varies (MiMo 12%, basal 22%). The matched comparison corrects for that; the table above does not.
• Intervals are approximate. In a coverage simulation of the frozen rule some bounds missed slightly more often than nominal. No interval here should be read as an exact 95% interval.
• One domain, one era, one language. Early-2000s US energy trading. Other industries and languages may look different.

What this means if you run an AI privacy gate

Masking personal data is necessary and cheap. Treat it as one layer, not as the answer to “may this leave?“. On real company mail, PII-based scoring tells you almost nothing about business confidentiality. Models that read context do much better, but even the best still sent 12 to 17% of confidential mail out at a 20% over-blocking budget, and they do not give the same answer twice. Measure the gate on your own text, with your own definition of confidential, before relying on it.

SafeLLM builds custom PII and data-egress protection for AI and LLM pipelines. If you need this measured or built for your data, get in touch.