I Asked 26 AI Systems Which E-mails May Leave the Company

Originally published on chrystianschutz.com.

Every company that lets its people use a cloud AI has the same problem: some of what they paste into it should never leave the building. A draft of the quarterly results. A trading position. A negotiation. The usual fix is a gateway that sits in front of the AI and decides, text by text, may this go out, or must it stay on our own machines?

I wanted to know which kind of system makes that call best. So I took real company mail, labelled it before running anything, and asked 26 systems the same question: basal, an open decision model made in Poland that runs on your own GPU, LLMs from three vendors, fine-tuned classifiers, and the PII detectors that most AI gateways ship with today.

The short version

• A small Polish model was as good as the cloud LLMs. basal omni max, an open-weights decision model by Polish researcher Remek Kinas, let 15.6% of confidential e-mails out. At the same over-blocking, DeepSeek V4.1 Flash let out 15.1% and Claude Haiku 5.5 11.9%; both differences are inside the noise. The difference that matters: basal runs on your own GPU, while the LLMs have to receive the e-mail before they can tell you it should not have been sent.
• Every classifier trained on this exact mail leaked more. The best, a fine-tuned e5, let out 21.6%, six points more than basal.
• PII detectors are not confidentiality detectors. Presidio let out 60%, PasteGuard 75%, a “block anything with a name or a contact” rule 82%. Confidential business mail is confidential because of what it says, not because it contains a phone number.
• Context is the dividing line. Given the same document twice, once as an unpublished board draft and once as a published press release, basal and the LLMs got all 129 pairs right. The trained classifiers got almost none, because the words are identical.
• The honest verdict is “promising, not proven”. The rule I fixed before the run asked basal to beat every baseline under every label definition. It beat all of them on the point estimate, but against e5 the lower bound touched zero. So the pre-registered verdict is inconclusive.

How much confidential mail gets out?

There are two ways to get this decision wrong. A leak is a confidential e-mail sent to the cloud. Over-blocking is a harmless e-mail kept local, which costs you the better cloud model for no reason. Any system can reach zero leaks by blocking everything, so the only fair comparison is leaks at the same over-blocking. That is what the slider in the chart at the top does.

basal’s threshold was fixed in advance on separate calibration mail, aiming for at most 20% over-blocking; on the test mail it landed at 22%. At that point the ranking is: Claude Haiku, DeepSeek and basal close together, then basal’s small sibling (mini), then the fine-tuned classifiers, MiMo and the 3B Ministral, and then, far behind, everything that looks for personal data.

The data

The Enron corpus is the mailboxes of an energy company that collapsed in 2001, released during the investigation. It is the closest thing there is to a public sample of real corporate mail: deals, prices, trading positions, legal matters, lunch plans, fantasy football.

• 800 test e-mails from 57 mailboxes. Training, calibration and test are split by mailbox, so no system sees the same person’s mail in training and in test.
• Labels came first. Two annotators from different model families, OpenAI’s Codex and Claude Opus, labelled every e-mail blind with the same written guidelines, and the labels were frozen before any system ran. They agree on the route 84% of the time (κ 0.68). Codex is stricter, so the main label calls an e-mail confidential if either annotator does: 463 of the 800.
• Masking first. E-mail addresses and phone numbers are masked before any system sees the text, as a real gateway would do. Names are left in.
• The plan was pre-registered. The metric, the threshold rule, the bootstrap and the decision rule were written down and frozen (43 files with checksums) before the first scored run, after eight rounds of review by Codex. The first seven ended with “do not freeze yet”.

Who is level with basal, and who is not

This is the comparison the rule was built on: each system re-thresholded to basal’s over-blocking on the same e-mails, with intervals from a bootstrap over whole mailboxes. Claude Haiku is 3.7 points better and DeepSeek 0.4 points better, and both intervals include zero. Everything else leaks more, from 6 points for e5 to 67 points for the contact rule.

Two caveats on the LLMs. Claude Haiku comes from the same family as one of the two annotators, so part of its lead may be agreement with its relative; DeepSeek is the cleaner comparison. And the hosted LLMs only see the text because it was sent to them, which is exactly what the gateway is supposed to decide.

I also scored 890 more e-mails from 36 other mailboxes, outside the frozen rule. The order held: Claude Haiku 11.5%, basal 13.6%, DeepSeek 17.2%, e5 21.5%.

Same words, different secret

This is the test that separates the families most sharply. I took 129 public filings from SEC EDGAR and showed each one twice, with a single context line on top: “internal draft prepared for the management board, not yet published” or “published via SEC EDGAR, publicly available”. The draft must stay local; the published version may go out.

A classifier that learned which words appear in confidential mail sees the same words twice and gives the same answer twice: it gets one of the two wrong every time. basal, Claude Haiku, DeepSeek and MiMo read the context line and got all 129 pairs right. So did basal mini, the small version that answers in about a third of a second.

The wording of the question matters. With a short, plain question, basal max got 88% of the pairs right. With a question that spells out what counts as confidential (prices, deal terms, unreleased results), it got 100%. On 211 Polish public-procurement notices built the same way, the plain question kept only 10% of the drafts local; the detailed one kept all of them.

Ask twice, get a different answer

The pre-registered plan had a second look on more data, and it required basal to reproduce its first-round scores to three decimal places. It could not, and neither could any other generative model. Identical requests to basal moved its score by up to 0.15. That sounds alarming, but almost all of the movement is on mail that sits near the threshold: basal sends the same e-mail the same way 99.3% of the time. Claude Haiku does so 86% of the time, one e-mail in seven. Classical pipelines are bit-for-bit stable, and also leak more.

I did not change the rule after seeing this, so the frozen verdict stays on the first look. For anyone building a gateway, the lesson is practical: a generative model will route a few borderline texts differently on a retry, so log the decision, not just the input.

What about other decision models?

basal is one of a new kind of model, and one of the few built in Poland. You do not chat with it: you give it a question, a set of answers with criteria, and a text, and it returns a probability for each answer in a single forward pass. Before the Enron round I ran a smaller benchmark on synthetic Polish and English texts, with Jev 1.13, a hosted decision model on OpenRouter, alongside. On the Polish set basal was ahead of Jev by 5 points of macro-F1 (interval 1.7 to 9.3).

Synthetic data is kind to everyone, and it can be actively misleading: classifiers trained on the synthetic texts and then tested on real Enron mail fell to an AUROC of about 0.6, barely better than chance. That is why the real-mail round exists.

What this means if you run an AI gateway

1. Do not call a PII detector a confidentiality filter. On business mail it ranks e-mails little better than chance (AUROC 0.51 to 0.64, where 0.5 is a coin).
2. A local decision model is a real option. It is level with good hosted LLMs on this task, and the text never leaves your machines to be judged.
3. Write the policy into the question. The same model went from 10% to 100% on Polish drafts when the question named what counts as confidential.
4. Expect some wobble. Log decisions, and treat a score near the threshold as “keep local”.

Limits

• The labels come from two AI annotators, not from Enron’s lawyers. I am labelling a sample by hand as a check.
• Enron is one company, 25 years ago, in English. The Polish part only tests context lines on public documents.
• The verdict is inconclusive by its own rule. A decisive v2 needs a deterministic basal server and a fresh test sample.

The full method, every table and the pre-registration are in the paper: Must This Text Stay In-House? (PDF).

basal is a Polish model, built and released openly by Remek Kinas under Apache-2.0: code on GitHub, weights on Hugging Face (omni max GGUF, omni mini), server on PyPI. I ran basal-omni-1.0-max as a Q8 GGUF in llama.cpp and the mini version in bf16. An earlier write-up of the PII-detector side of this benchmark is on the SafeLLM blog.

Need this for your company?

If your team ships LLM features and someone has asked “what exactly goes to the model?”, we can help: PII masking and a confidentiality decision step on your stack (not only Apache APISIX), measured on a labelled sample of your own data, with the logs and documentation the AI Act and GDPR call for. See how we work or book a free call.