feat(analyzer): add German PII recognizers (DE_*)#1909
Conversation
Adds 9 new predefined recognizers for German personally identifiable information, all disabled by default (enabled: false) and scoped to supported_language: de. Recognizers with checksum validation: - DE_TAX_ID: Steueridentifikationsnummer (§§ 139a-e AO) ISO 7064 Mod 11,10 checksum (Bundeszentralamt für Steuern) - DE_SOCIAL_SECURITY: Rentenversicherungsnummer / RVNR (§ 147 SGB VI) Deutsche Rentenversicherung Bund checksum algorithm - DE_HEALTH_INSURANCE: Krankenversicherungsnummer KVNR (§ 290 SGB V) GKV-Spitzenverband checksum; DSGVO Art. 9 (health data) Pattern-based recognizers: - DE_TAX_NUMBER: Steuernummer (§ 139a AO) – ELSTER 13-digit and state-specific slash-separated formats - DE_PASSPORT: Reisepassnummer (PassG § 4, ICAO Doc 9303) - DE_ID_CARD: Personalausweisnummer (PAuswG) – nPA and legacy format - DE_KFZ: KFZ-Kennzeichen (FZV § 8, ECJ C-582/14) - DE_HANDELSREGISTER: Handelsregisternummer HRA/HRB (§§ 9, 14 HGB) - DE_PLZ: Postleitzahl (DSGVO Art. 4 Nr. 1) – very low base confidence (0.05), context words required for actionable results Also updates: - predefined_recognizers/__init__.py: imports and __all__ - conf/default_recognizers.yaml: all 9 recognizers registered - docs/supported_entities.md: new Germany section - CHANGELOG.md: entry under [unreleased] 133 new tests pass (9 test files, one per recognizer). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Added .gitattributes to force LF for *.sh files. CRLF endings in entrypoint.sh caused 'no such file or directory' when running the presidio-analyzer Docker container on Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Enabled all 9 DE_* recognizers in default_recognizers.yaml and added 'de' to supported_languages so they are loaded and visible to clients (e.g. LiteLLM) without requiring ad-hoc configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The registry and engine supported_languages must match. Missing 'de' in default_analyzer.yaml caused a startup crash. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…aults Reverts the deployment-specific changes made for local testing: - German recognizers back to enabled=false (upstream contribution convention) - supported_languages back to [en] only in both config files Users who want German PII detection should enable the DE_* recognizers explicitly in their own deployment configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- D205: add blank line between summary and description (de_health_insurance_recognizer)
- E501: shorten long lines in de_health_insurance_recognizer, de_kfz_recognizer,
de_social_security_recognizer, de_tax_id_recognizer
- D214/D406: fix Examples section formatting in de_kfz_recognizer (auto-fixed)
All 133 tests still pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@microsoft-github-policy-service agree |
|
I know there's #1830 , but sadly it seems inactive. |
…tterns Enable German (de) as a supported language alongside English: - default_analyzer.yaml: add 'de' to supported_languages - default_recognizers.yaml: add 'de' to supported_languages, enable all nine German PII recognizers (DeTaxIdRecognizer, DeTaxNumberRecognizer, DePassportRecognizer, DeIdCardRecognizer, DeSocialSecurityRecognizer, DeHealthInsuranceRecognizer, DeKfzRecognizer, DeHandelsregisterRecognizer, DePlzRecognizer) - spacy_en_de.yaml: add dedicated en+de NLP config (en_core_web_lg + de_core_news_md) for deployments that only need English and German Fix DeKfzRecognizer: - Replace \b word-boundary anchors with (?<![\w-]) / (?!\w) lookarounds to prevent false matches starting mid-plate after a hyphen separator (e.g. 'M-AB 123' was matching as 'AB 123' instead of 'M-AB 123') - Add two missing patterns for the common Bindestrich+Leerzeichen format '[District]-[Letters] [Digits]' (e.g. M-AB 123, HH-XY 999) Fix DeTaxNumberRecognizer: - Replace \b anchors with (?<!\w) / (?!\w) lookarounds on slash-format patterns to prevent boundary bleed into surrounding words - Raise Bayern/BW (3/3/5) pattern confidence from 0.3 to 0.4 NOTE: After this PR is merged a corresponding update to the LiteLLM Presidio guardrail configuration is needed to map the newly active German entity types (DE_KFZ, DE_TAX_ID, DE_TAX_NUMBER, DE_ID_CARD, DE_PASSPORT, DE_SOCIAL_SECURITY, DE_HEALTH_INSURANCE, DE_HANDELSREGISTER, DE_PLZ) to the desired masking actions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove 'de' from supported_languages in default_analyzer.yaml and default_recognizers.yaml so defaults remain ["en"] as tests expect - Add enabled: false to all German recognizers in default_recognizers.yaml (opt-in instead of opt-out, consistent with other disabled recognizers) - Move Germany imports to correct alphabetical position in __init__.py (after Finland, before India) to fix ruff I001 import-ordering error Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new set of Germany-specific, pattern-based PII recognizers to Presidio Analyzer (entity types prefixed DE_*), along with configuration wiring, documentation updates, and unit tests.
Changes:
- Introduces 9 new
PatternRecognizerimplementations underpredefined_recognizers/country_specific/germany/(some with checksum validation viavalidate_result()). - Exposes the new recognizers via
presidio_analyzer.predefined_recognizersand registers them (disabled by default) indefault_recognizers.yaml. - Adds dedicated pytest coverage for each recognizer and documents the new entity types in
docs/supported_entities.md, plus a changelog entry.
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/init.py | Adds Germany-specific recognizers package. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_tax_id_recognizer.py | Adds DE tax id recognizer + checksum validation. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_tax_number_recognizer.py | Adds DE tax number recognizer patterns + context. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_passport_recognizer.py | Adds DE passport recognizer patterns + context. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_id_card_recognizer.py | Adds DE ID card recognizer patterns + context. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_social_security_recognizer.py | Adds DE social security recognizer + checksum validation. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_health_insurance_recognizer.py | Adds DE health insurance (KVNR) recognizer + checksum validation. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_kfz_recognizer.py | Adds DE vehicle registration plate recognizer patterns + context. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_handelsregister_recognizer.py | Adds DE commercial register (HRA/HRB) recognizer patterns + context. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany/de_plz_recognizer.py | Adds DE postal code recognizer with low base score + context. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/init.py | Exports Germany recognizers and adds them to __all__. |
| presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml | Registers Germany recognizers (disabled by default). |
| presidio-analyzer/presidio_analyzer/conf/spacy_en_de.yaml | Adds a spaCy config supporting both EN and DE models. |
| presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml | Normalizes YAML formatting (no functional change apparent). |
| presidio-analyzer/tests/test_de_tax_id_recognizer.py | Tests DE tax id recognizer including checksum validation. |
| presidio-analyzer/tests/test_de_tax_number_recognizer.py | Tests DE tax number recognizer patterns. |
| presidio-analyzer/tests/test_de_passport_recognizer.py | Tests DE passport recognizer patterns. |
| presidio-analyzer/tests/test_de_id_card_recognizer.py | Tests DE ID card recognizer patterns. |
| presidio-analyzer/tests/test_de_social_security_recognizer.py | Tests DE social security recognizer including checksum validation. |
| presidio-analyzer/tests/test_de_health_insurance_recognizer.py | Tests DE health insurance recognizer including checksum validation. |
| presidio-analyzer/tests/test_de_kfz_recognizer.py | Tests DE vehicle registration plate recognizer patterns. |
| presidio-analyzer/tests/test_de_handelsregister_recognizer.py | Tests DE commercial register recognizer patterns. |
| presidio-analyzer/tests/test_de_plz_recognizer.py | Tests DE postal code recognizer behavior/position/score range. |
| docs/supported_entities.md | Documents the new DE_* entity types and detection methods. |
| CHANGELOG.md | Adds an unreleased entry describing the new German recognizers. |
| .gitattributes | Enforces LF line endings for *.sh scripts. |
You can also share your feedback on Copilot code review. Take the survey.
…izers - de_id_card_recognizer: fix docstring example T220001292 (10 chars) -> T22000129 (9 chars) - de_plz_recognizer: tighten regex to exclude boundary values 01000 and 99999 via negative lookahead; update docstring accordingly - test_de_plz_recognizer: add negative test cases for 01000 and 99999 - de_social_security_recognizer: fix day regex [4-7]\d (40-79) -> 5[1-9]|[67]\d so supplemental range is correctly 51-81 per spec, not 41-81 - de_tax_number_recognizer: remove undocumented 10-digit plain format from docstring (no corresponding pattern exists) - de_passport_recognizer: remove unused _ICAO_CHARS constant; fix docstring example F204004812 (10 chars) -> F20400481 (9 chars) - de_health_insurance_recognizer: fix docstring example A123456780 (invalid checksum) -> A123456787 (valid checksum) - de_tax_id_recognizer: remove invalid example 02476291358 (leading zero); remove unimplemented digit-frequency constraint from docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MvdB
left a comment
There was a problem hiding this comment.
Implemented suggested changes.
…IN recognizers
Adds four new German PII recognizers with tests:
- DE_LANR (Lebenslange Arztnummer): 9-digit physician number with KBV
check digit validation (weights [4,9,2,10,5,3], cross-sum, mod 10).
Legal basis: § 75 Abs. 7 SGB V.
- DE_BSNR (Betriebsstättennummer): 9-digit practice/site-of-care number,
no public checksum, relies on context words for high-confidence matches.
Legal basis: § 75 Abs. 7 SGB V.
- DE_VAT_ID (Umsatzsteuer-Identifikationsnummer): fixed-prefix pattern
"DE" + 9 digits per § 27a UStG / BZSt format.
- DE_FUEHRERSCHEIN (Führerscheinnummer): post-2013 EU-harmonized format
[A-Z]{2}\d{8}[A-Z0-9] (11 chars) per FeV Anlage 8 / EU Directive
2006/126/EC. No checksum (KBA algorithm not published). Pre-2013
card formats are explicitly out of scope.
Also moves spacy_en_de.yaml from presidio_analyzer/conf/ to
docs/recipes/german-language-support/ per reviewer feedback, and
reverts the docker-compose.yml NLP_CONF_FILE build arg.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rewrites the README to follow the template.md structure: Overview, Quick Start, Approach (with entity/checksum table), Results (TBD pending formal evaluation), Key Findings, and Tips sections. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@SharonHart The telematik_id in #1830 was not implemented because its format seems too flexible and lacks a strict, publicly verifiable validation scheme leading to high risk of false positives. |
SharonHart
left a comment
There was a problem hiding this comment.
Amazing contribution, thanks!
…#1909) * feat(analyzer): add German PII recognizers (DE_*) Adds 9 new predefined recognizers for German personally identifiable information, all disabled by default (enabled: false) and scoped to supported_language: de. Recognizers with checksum validation: - DE_TAX_ID: Steueridentifikationsnummer (§§ 139a-e AO) ISO 7064 Mod 11,10 checksum (Bundeszentralamt für Steuern) - DE_SOCIAL_SECURITY: Rentenversicherungsnummer / RVNR (§ 147 SGB VI) Deutsche Rentenversicherung Bund checksum algorithm - DE_HEALTH_INSURANCE: Krankenversicherungsnummer KVNR (§ 290 SGB V) GKV-Spitzenverband checksum; DSGVO Art. 9 (health data) Pattern-based recognizers: - DE_TAX_NUMBER: Steuernummer (§ 139a AO) – ELSTER 13-digit and state-specific slash-separated formats - DE_PASSPORT: Reisepassnummer (PassG § 4, ICAO Doc 9303) - DE_ID_CARD: Personalausweisnummer (PAuswG) – nPA and legacy format - DE_KFZ: KFZ-Kennzeichen (FZV § 8, ECJ C-582/14) - DE_HANDELSREGISTER: Handelsregisternummer HRA/HRB (§§ 9, 14 HGB) - DE_PLZ: Postleitzahl (DSGVO Art. 4 Nr. 1) – very low base confidence (0.05), context words required for actionable results Also updates: - predefined_recognizers/__init__.py: imports and __all__ - conf/default_recognizers.yaml: all 9 recognizers registered - docs/supported_entities.md: new Germany section - CHANGELOG.md: entry under [unreleased] 133 new tests pass (9 test files, one per recognizer). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: enforce LF line endings for shell scripts in containers Added .gitattributes to force LF for *.sh files. CRLF endings in entrypoint.sh caused 'no such file or directory' when running the presidio-analyzer Docker container on Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(config): enable German recognizers and add 'de' language support Enabled all 9 DE_* recognizers in default_recognizers.yaml and added 'de' to supported_languages so they are loaded and visible to clients (e.g. LiteLLM) without requiring ad-hoc configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(config): add 'de' to analyzer engine supported_languages The registry and engine supported_languages must match. Missing 'de' in default_analyzer.yaml caused a startup crash. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(config): restore default enabled=false and single-language defaults Reverts the deployment-specific changes made for local testing: - German recognizers back to enabled=false (upstream contribution convention) - supported_languages back to [en] only in both config files Users who want German PII detection should enable the DE_* recognizers explicitly in their own deployment configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix ruff linting issues in German recognizers - D205: add blank line between summary and description (de_health_insurance_recognizer) - E501: shorten long lines in de_health_insurance_recognizer, de_kfz_recognizer, de_social_security_recognizer, de_tax_id_recognizer - D214/D406: fix Examples section formatting in de_kfz_recognizer (auto-fixed) All 133 tests still pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(config): enable German language support and fix DE recognizer patterns Enable German (de) as a supported language alongside English: - default_analyzer.yaml: add 'de' to supported_languages - default_recognizers.yaml: add 'de' to supported_languages, enable all nine German PII recognizers (DeTaxIdRecognizer, DeTaxNumberRecognizer, DePassportRecognizer, DeIdCardRecognizer, DeSocialSecurityRecognizer, DeHealthInsuranceRecognizer, DeKfzRecognizer, DeHandelsregisterRecognizer, DePlzRecognizer) - spacy_en_de.yaml: add dedicated en+de NLP config (en_core_web_lg + de_core_news_md) for deployments that only need English and German Fix DeKfzRecognizer: - Replace \b word-boundary anchors with (?<![\w-]) / (?!\w) lookarounds to prevent false matches starting mid-plate after a hyphen separator (e.g. 'M-AB 123' was matching as 'AB 123' instead of 'M-AB 123') - Add two missing patterns for the common Bindestrich+Leerzeichen format '[District]-[Letters] [Digits]' (e.g. M-AB 123, HH-XY 999) Fix DeTaxNumberRecognizer: - Replace \b anchors with (?<!\w) / (?!\w) lookarounds on slash-format patterns to prevent boundary bleed into surrounding words - Raise Bayern/BW (3/3/5) pattern confidence from 0.3 to 0.4 NOTE: After this PR is merged a corresponding update to the LiteLLM Presidio guardrail configuration is needed to map the newly active German entity types (DE_KFZ, DE_TAX_ID, DE_TAX_NUMBER, DE_ID_CARD, DE_PASSPORT, DE_SOCIAL_SECURITY, DE_HEALTH_INSURANCE, DE_HANDELSREGISTER, DE_PLZ) to the desired masking actions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(config): restore English-only defaults and fix import ordering - Remove 'de' from supported_languages in default_analyzer.yaml and default_recognizers.yaml so defaults remain ["en"] as tests expect - Add enabled: false to all German recognizers in default_recognizers.yaml (opt-in instead of opt-out, consistent with other disabled recognizers) - Move Germany imports to correct alphabetical position in __init__.py (after Finland, before India) to fix ruff I001 import-ordering error Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(de-recognizers): address Copilot review feedback on German recognizers - de_id_card_recognizer: fix docstring example T220001292 (10 chars) -> T22000129 (9 chars) - de_plz_recognizer: tighten regex to exclude boundary values 01000 and 99999 via negative lookahead; update docstring accordingly - test_de_plz_recognizer: add negative test cases for 01000 and 99999 - de_social_security_recognizer: fix day regex [4-7]\d (40-79) -> 5[1-9]|[67]\d so supplemental range is correctly 51-81 per spec, not 41-81 - de_tax_number_recognizer: remove undocumented 10-digit plain format from docstring (no corresponding pattern exists) - de_passport_recognizer: remove unused _ICAO_CHARS constant; fix docstring example F204004812 (10 chars) -> F20400481 (9 chars) - de_health_insurance_recognizer: fix docstring example A123456780 (invalid checksum) -> A123456787 (valid checksum) - de_tax_id_recognizer: remove invalid example 02476291358 (leading zero); remove unimplemented digit-frequency constraint from docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(de-recognizers): add DE_LANR, DE_BSNR, DE_VAT_ID, DE_FUEHRERSCHEIN recognizers Adds four new German PII recognizers with tests: - DE_LANR (Lebenslange Arztnummer): 9-digit physician number with KBV check digit validation (weights [4,9,2,10,5,3], cross-sum, mod 10). Legal basis: § 75 Abs. 7 SGB V. - DE_BSNR (Betriebsstättennummer): 9-digit practice/site-of-care number, no public checksum, relies on context words for high-confidence matches. Legal basis: § 75 Abs. 7 SGB V. - DE_VAT_ID (Umsatzsteuer-Identifikationsnummer): fixed-prefix pattern "DE" + 9 digits per § 27a UStG / BZSt format. - DE_FUEHRERSCHEIN (Führerscheinnummer): post-2013 EU-harmonized format [A-Z]{2}\d{8}[A-Z0-9] (11 chars) per FeV Anlage 8 / EU Directive 2006/126/EC. No checksum (KBA algorithm not published). Pre-2013 card formats are explicitly out of scope. Also moves spacy_en_de.yaml from presidio_analyzer/conf/ to docs/recipes/german-language-support/ per reviewer feedback, and reverts the docker-compose.yml NLP_CONF_FILE build arg. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(recipes): align german-language-support README with recipe template Rewrites the README to follow the template.md structure: Overview, Quick Start, Approach (with entity/checksum table), Results (TBD pending formal evaluation), Key Findings, and Tips sections. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Summary
Adds 9 new pattern-based recognizers for German-language PII, covering the most relevant entity types under DSGVO/BDSG:
DE_TAX_IDDE_TAX_NUMBERDE_PASSPORTDE_ID_CARDDE_SOCIAL_SECURITYDE_HEALTH_INSURANCEDE_KFZDE_HANDELSREGISTERDE_PLZDesign decisions
PatternRecognizerpattern and are placed inpredefined_recognizers/country_specific/germany/enabled: false) following the project convention for country-specific recognizersDE_TAX_ID,DE_SOCIAL_SECURITY,DE_HEALTH_INSURANCE) implementvalidate_result()returningTrue/Falseto promote/suppress matches — notNone, to ensure invalid numbers are filtered out rather than kept at base scoreDE_PLZhas intentionally low base confidence (0.05) due to high false-positive risk; documented in docstringDE_TAX_NUMBERincludes both ELSTER 13-digit and regional slash-format patterns with graduated confidence scoresglobal_regex_flags: 26(IGNORECASE | MULTILINE | DOTALL) consistent with the rest of PresidioTest plan
presidio-analyzer/tests/test_de_*.py)validate_result()tests for checksum-based recognizerspython -m pytest presidio-analyzer/tests/test_de_*.pyruff checkpasses with no issuesdefault_recognizers.yamlentries follow the existing country-specific patterndocs/supported_entities.mdupdated with Germany sectionCHANGELOG.mdentry added under[unreleased]🤖 Generated with Claude Code