· Research · 8 min read
Polish PII Detection Benchmark: PESEL, NIP, REGON and Inflected Names
Six open-source PII detectors on 1,022 Polish test documents: who finds PESEL, NIP, REGON, ID card and bank account numbers, who handles names in all seven grammatical cases, and who masks order numbers that only look like identifiers. Method, numbers with confidence intervals, failures and limits.
Why Polish PII is its own problem
If you put a PII filter in front of an LLM and your users write in Polish, the default tooling was not built for your text. Two things make Polish different.
Identifiers. Polish documents are full of national numbers: PESEL (personal ID, 11 digits), NIP (tax ID, 10 digits), REGON (business register, 9 or 14 digits), ID card numbers (three letters and six digits), passport numbers and 26-digit NRB bank accounts. Most have a checksum, so a good detector can tell a real PESEL from a random 11-digit order number. A detector that only looks at the shape of the number cannot.
Names change form. A Polish name has up to seven grammatical cases: Jan Kowalski, Jana Kowalskiego, Janowi Kowalskiemu, z Janem Kowalskim, o Janie Kowalskim. A detector trained on English, or a pseudonymiser that keys on the exact string, sees five different people.
We wanted numbers, not impressions. So we built a small benchmark that anyone can rerun on a laptop. Everything ran locally. No LLM API was used to build the data or to score it.
Method
Test data
- Synthetic documents, 1,294 in total: short emails, support tickets, CRM notes, invoices, HR notes and chat messages in Polish. 1,022 form the test split that is reported here. 272 are a held-back dev split, assigned by hash.
- Identifiers come from a fresh implementation of the published checksum rules, written for this benchmark and sharing no code with any detector. Unit tests check it against published examples (published PESEL and IBAN examples and the NIP/REGON numbers of public bodies) and against two other implementations: python-stdnum and Faker’s
pl_PLprovider. - Names come from hand-written declension tables with 46 first names, 14 noun surnames and rule-declined -ski/-cki surnames, in all seven cases, with and without a title (pan, pani). About a quarter of the names (128 of 500 in the test split) appear ASCII-folded (Wojcik instead of Wójcik), as people often type.
- Hard negatives: 1,080 numbers that look like identifiers but are not: 11-digit order numbers, 10-digit customer numbers, KRS court register numbers, EAN-13 barcodes, parcel numbers, invoice numbers, dates, amounts and postcodes.
- A small real slice: 13 snippets from public pages of government bodies, universities and one company, with 26 NIP/REGON values labelled from the page’s own text, plus 19 other numbers on the same pages as negatives. No private persons appear in the benchmark.
The data was generated with a recorded seed, and its sha256 was recorded before any detector ran. Two independent generations produced byte-identical files. A 50-item manual spot check caught two template bugs, which were fixed before the freeze.
Detectors
| Name | What it is |
|---|---|
regex_naive | Digit-shape regexes for each identifier, no checksums, no names. The baseline. |
presidio_default_en | Microsoft Presidio 2.2.366, out of the box (English NER and recognizers). |
presidio_pl_upstream | Presidio with the spaCy pl_core_news_lg model and upstream’s Polish recognizers (PESEL, plus email, phone, IBAN). |
pii_presidio | The pii-presidio 0.1.0 recognizer pack from PyPI for pl, set up as its README shows (no name recognizer). |
presidio_pl_asw | The same spaCy setup plus our own presidio-pl recognizers (checksum-validated, context-scored). Our own code: read its numbers with that in mind. |
gliner_pii_edge | knowledgator/gliner-pii-edge-v1.0, a zero-shot GLiNER model, threshold 0.5. |
All Presidio variants use a score threshold of 0.4. All latency figures are on CPU (4 cores of a Ryzen 7 8700F) after a warm-up.
Scoring
For each gold span we ask whether a detector found it (a prediction of the same broad group overlaps it), whether it masked it fully, and whether it gave the exact type. “Identifier” groups PESEL, NIP, REGON, ID card, passport and bank account. Every rate comes with its count and a 95% Wilson interval. F1 intervals are document-level bootstrap.
Results
Recall (“found”) on the 1,022-document test split:
| Detector | Person | PESEL | NIP | REGON | ID card | Passport | Bank | Phone | |
|---|---|---|---|---|---|---|---|---|---|
| regex_naive | 0.00 | 1.00 | 0.83 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| presidio_default_en | 0.70 | 0.00 | 0.04 | 0.04 | 0.00 | 0.05 | 0.56 | 1.00 | 0.35 |
| presidio_pl_upstream | 1.00 | 1.00 | 0.00 | 0.01 | 0.00 | 0.00 | 0.57 | 1.00 | 0.98 |
| pii_presidio | 0.00 | 1.00 | 0.61 | 1.00 | 0.52 | 1.00 | 0.56 | 1.00 | 0.86 |
| presidio_pl_asw | 1.00 | 1.00 | 0.73 | 0.74 | 1.00 | 0.65 | 1.00 | 1.00 | 0.98 |
| gliner_pii_edge | 0.68 | 0.82 | 0.48 | 0.85 | 0.85 | 0.79 | 0.76 | 0.97 | 0.74 |
Test positives per type: 500 names, 241 PESEL, 228 NIP, 239 REGON, 200 ID cards, 152 passports, 245 bank accounts, 160 emails, 236 phones.
Precision and the cost of false alarms:
| Detector | F1 identifiers [95% CI] | F1 person | Hard negatives masked as an ID | ms per 1k chars |
|---|---|---|---|---|
| regex_naive | 0.861 [0.849, 0.875] | 0 | 33.9% | 0.3 |
| presidio_default_en | 0.218 | 0.286 | 2.1% | 66 |
| presidio_pl_upstream | 0.450 | 0.903 [0.886, 0.923] | 1.7% | 90 |
| pii_presidio | 0.863 | 0 | 2.2% | 84 |
| presidio_pl_asw | 0.926 [0.915, 0.937] | 0.903 | 0.0% (0/845) | 83 |
| gliner_pii_edge | 0.695 | 0.443 | 52.5% | 348 |
What the numbers say:
- Default Presidio does not cover Polish. On Polish text it finds 70% of names and almost no national identifiers. Switching to the Polish spaCy model fixes names (99.8% found on our templates) but upstream Presidio still has no NIP, REGON, ID card or passport recognizer.
- A regex finds everything and too much. The naive baseline has the highest recall, but it masked every 10-digit customer number, every KRS number and every 11-digit order number it saw. It also read 34% of phone numbers as identifiers. Checksums matter: 35 of our 300 synthetic phone numbers happen to pass the REGON checksum, and 9 of 120 KRS numbers pass the NIP checksum. So even a checksum alone is not enough without context.
- Checksums plus context words give the best precision, at a recall cost. Our
presidio_pl_aswlabelled none of the 845 identifier-like hard negatives as an ID and read 1 of 236 phone numbers as one. The price: it keeps bare numbers below the threshold when there is no context word nearby. NIP recall is 0.89 with a word like NIP nearby and 0.43 without. REGON drops from 1.00 to 0.35, and passport from 1.00 to 0.00. If your text is mostly bare numbers, this setting is wrong for you. - GLiNER (zero-shot) is not ready for Polish identifiers.
gliner-pii-edgefinds a fair share of everything, but it labelled 53% of hard negatives as identifiers, rarely gets the type right (NIP typed recall 0.02) and was the slowest on CPU. It was weakest on names in the instrumental (0.31) and locative (0.53) cases.
Names in seven cases
With the Polish spaCy model, name recall was 1.00 in six cases and 0.99 in the locative, and 0.99 on ASCII-folded names. With the English default, recall fell from 0.80 in the nominative to 0.58 in the dative. Our templates are simple and use common names, so treat the spaCy figure as an upper bound. Real text with rare surnames, nicknames, lowercase names or surnames that are also ordinary words (Kowal, Sikora) will do worse.
The real slice
On 26 real NIP/REGON values from public pages, presidio_pl_asw and the regex both found 11/13 NIP and 12/13 REGON. pii_presidio found 8/13 and 12/13. Upstream Polish Presidio found none. The misses were real-world formats: a NIP written as 526 26 05 735 and a REGON as 36 92 67 361. With 26 items this is a sanity check, not a measurement.
Reversible pseudonymisation
Masking is half the job. If you replace names before calling an LLM and restore them afterwards, the same person must get the same placeholder in every case. We tested 300 documents that each mention one person three times in different cases.
- Mask → unmask restored the text exactly in 300/300 documents.
- Keying placeholders on the exact string gave all three mentions one placeholder in only 2 of 300 documents. To the LLM, Kowalski, Kowalskiego and Kowalskim are three people.
- Keying on the spaCy lemma raised that to 152 of 300. With two people per document, it reached 55 of 300, and no two people were merged. The lemmatiser itself makes mistakes: it turned Wójcikiem into Wójcikie.
If your workflow depends on the model reasoning about who did what, inflection-aware pseudonymisation is the gap to close. Off-the-shelf lemmatisation does not close it.
What was not tested
- OpenAI Privacy Filter (
openai/privacy-filter) loaded on our GPU, but the benchmark run ended without writing output, and we did not rerun it in time. No number is reported for it. - GLiNER multi-PII (
urchade/gliner_multi_pii-v1) failed to load because a tokenizer dependency was missing. It was fixed but not rerun. - No LLM-based detectors, no commercial APIs, and no addresses, dates of birth or free-text health data.
How to reproduce
The benchmark is a folder of plain Python. It contains generators and validators written from the published rules (specs.py), declension tables (names.py), the dataset builder with a fixed seed, the frozen synthetic_v1.jsonl with its sha256, the public slice, one runner per detector, a stdlib-only scorer, the round-trip test and 20 checksum unit tests. All library versions are pinned in the README. On 4 CPU cores each Presidio variant scores the whole set in under a minute after model load, and GLiNER in about two minutes. The folder is not on a public repository yet. Get in touch and we will send it.
Limits
- The text is synthetic and built from templates. Detectors that rely on sentence structure may score differently on real emails and tickets.
- We wrote both the benchmark and one of the detectors (
presidio_pl_asw). The data shares no code with that detector, and the test split was frozen before any run, but we know how that detector works. That is why its weak spots are listed above and not hidden. - Each identifier type has 150-250 positives, so intervals are ±1-7 points. Small differences between detectors are noise.
- The ID card check rule was confirmed only against a specimen number, not an official worked example.
- Thresholds were not tuned per detector. Every Presidio variant used 0.4 and GLiNER used 0.5.
Takeaway
For Polish text, a privacy filter needs three things that default tools lack: Polish name recognition, checksum-validated national identifiers, and context, so that order numbers stay unmasked. Even with all three, two problems remain open. Bare numbers without a context word are a recall/precision trade-off you must choose deliberately, and inflected names break naive reversible pseudonymisation.
SafeLLM builds custom PII protection for Polish and other Central and Eastern European languages. If you need it tuned to your own documents, get in touch.
