Skip to content

Any language classification#25875

Merged
edg956 merged 6 commits intomainfrom
any-language-classification
Feb 16, 2026
Merged

Any language classification#25875
edg956 merged 6 commits intomainfrom
any-language-classification

Conversation

@edg956
Copy link
Contributor

@edg956 edg956 commented Feb 13, 2026 •
edited
Loading

Summary

  • Adds "any" as a valid value for classificationLanguage in the auto-classification pipeline configuration
  • When any is selected, all recognizers are included regardless of their configured supportedLanguage, rather than being filtered to a single language
  • Analysis is dispatched once per distinct recognizer language group to satisfy Presidio's per-language analyze() contract
  • The english medium Spacy Web model is mapped to ClassificationLanguage.any in LANGUAGE_MODEL_MAPPING

Changes

Layer File Change
Schema classificationLanguages.json Added "any" to enum (first position); updated description
Schema databaseServiceAutoClassificationPipeline.json Updated classificationLanguage description
Generated classificationLanguages.py Added any = 'any' to ClassificationLanguage enum
Constants constants.py Added ClassificationLanguage.any → xx_ent_wiki_sm mapping
Core logic tag_analyzer.py Skip language filter in get_recognizers_by() when any; new _analyze_with() with per-language dispatch; build_analyzer_with() accepts optional nlp_engine override
Tests test_tag_analyzer_any_language.py 10 new unit tests covering filter bypass, per-language dispatch, and NLP engine handling

Test plan

  • All 10 new unit tests in test_tag_analyzer_any_language.py pass
  • All existing PII unit tests (86 tests) pass without regression

@edg956 edg956 self-assigned this Feb 13, 2026
@edg956 edg956 requested a review from a team as a code owner February 13, 2026 12:07
@edg956 edg956 added Ingestion safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch labels Feb 13, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Feb 13, 2026

TypeScript types have been updated based on the JSON schema changes in the PR

@github-actions github-actions bot requested a review from a team as a code owner February 13, 2026 12:10
@github-actions
Copy link
Contributor

github-actions bot commented Feb 13, 2026 •
edited
Loading

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.12)

Vulnerabilities (4)

Package Vulnerability ID Severity Installed Version Fixed Version
libpam-modules CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam-modules-bin CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam-runtime CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam0g CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (19)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
aiohttp CVE-2025-69223 🚨 HIGH 3.12.12 3.13.3
aiohttp CVE-2025-69223 🚨 HIGH 3.13.2 3.13.3
apache-airflow CVE-2025-68438 🚨 HIGH 3.1.5 3.1.6
apache-airflow CVE-2025-68675 🚨 HIGH 3.1.5 3.1.6
azure-core CVE-2026-21226 🚨 HIGH 1.37.0 1.38.0
cryptography CVE-2026-26007 🚨 HIGH 42.0.8 46.0.5
jaraco.context CVE-2026-23949 🚨 HIGH 5.3.0 6.1.0
jaraco.context CVE-2026-23949 🚨 HIGH 6.0.1 6.1.0
protobuf CVE-2026-0994 🚨 HIGH 4.25.8 6.33.5, 5.29.6
pyasn1 CVE-2026-23490 🚨 HIGH 0.6.1 0.6.2
python-multipart CVE-2026-24486 🚨 HIGH 0.0.20 0.0.22
ray CVE-2025-62593 🔥 CRITICAL 2.47.1 2.52.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2

🛡️ TRIVY SCAN RESULT 🛡️

Target: usr/bin/docker

Vulnerabilities (4)

Package Vulnerability ID Severity Installed Version Fixed Version
stdlib CVE-2025-68121 🔥 CRITICAL v1.25.5 1.24.13, 1.25.7, 1.26.0-rc.3
stdlib CVE-2025-61726 🚨 HIGH v1.25.5 1.24.12, 1.25.6
stdlib CVE-2025-61728 🚨 HIGH v1.25.5 1.24.12, 1.25.6
stdlib CVE-2025-61730 🚨 HIGH v1.25.5 1.24.12, 1.25.6

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@github-actions
Copy link
Contributor

github-actions bot commented Feb 13, 2026 •
edited
Loading

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.13)

Vulnerabilities (25)

Package Vulnerability ID Severity Installed Version Fixed Version
linux-libc-dev CVE-2024-46786 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-21946 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-22022 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-22083 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-22107 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-22121 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-37926 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-38022 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-38129 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-38361 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-38718 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-39871 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-68340 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-68349 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-68800 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-71085 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2025-71116 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-22984 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-22990 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-23001 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-23010 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-23054 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-23074 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-23084 🚨 HIGH 6.1.159-1 6.1.162-1
linux-libc-dev CVE-2026-23097 🚨 HIGH 6.1.159-1 6.1.162-1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (9)

Package Vulnerability ID Severity Installed Version Fixed Version
apache-airflow CVE-2025-68438 🚨 HIGH 3.1.5 3.1.6
apache-airflow CVE-2025-68675 🚨 HIGH 3.1.5 3.1.6
cryptography CVE-2026-26007 🚨 HIGH 42.0.8 46.0.5
jaraco.context CVE-2026-23949 🚨 HIGH 6.0.1 6.1.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data_aut.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage_aut.yaml

No Vulnerabilities Found

gitar-bot[bot]
gitar-bot bot reviewed Feb 13, 2026
View reviewed changes
ingestion/src/metadata/pii/tag_analyzer.py
recognizer_registry = RecognizerRegistry(
recognizers=recognizers, supported_languages=supported_languages
)
effective_nlp = nlp_engine if nlp_engine is not None else self._nlp_engine
Copy link

@gitar-bot gitar-bot bot Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Bug: nlp_engine=None sentinel can't disable NLP engine in "any" mode

The build_analyzer_with method uses None as the default for nlp_engine, intending it as a sentinel meaning "use self._nlp_engine":

effective_nlp = nlp_engine if nlp_engine is not None else self._nlp_engine

In the "any" mode path (_analyze_with, line 164), the code calls self.build_analyzer_with(lang_recognizers, nlp_engine=None) intending to pass no NLP engine to AnalyzerEngine. However, because None is the sentinel for "use default", effective_nlp falls back to self._nlp_engine.

This means in "any" mode, the AnalyzerEngine is still built with the NLP engine that was loaded with lang_code="any" (from load_nlp_engine(classification_language=ClassificationLanguage.any) in tag_processor.py). When analyzer.analyze(value, language="en") is called, Presidio will try to use the NLP engine whose supported language is "any" — not "en" — which will likely cause a language mismatch error at runtime.

The tests don't catch this because mock_nlp_engine = MagicMock() silently accepts any call. The tests only verify that build_analyzer_with is called with nlp_engine=None, not what AnalyzerEngine actually receives.

Fix: Use a sentinel object instead of None to distinguish "no argument given" from "explicitly no NLP engine":

_UNSET = object()

def build_analyzer_with(
    self,
    recognizers: list[EntityRecognizer],
    nlp_engine: Optional[NlpEngine] = _UNSET,
) -> AnalyzerEngine:
    ...
    effective_nlp = self._nlp_engine if nlp_engine is _UNSET else nlp_engine
    ...

This allows build_analyzer_with(recs, nlp_engine=None) to actually pass None to AnalyzerEngine.

Suggested fix:

_UNSET_NLP = object()

    def build_analyzer_with(
        self,
        recognizers: list[EntityRecognizer],
        nlp_engine: Optional[NlpEngine] = _UNSET_NLP,
    ) -> AnalyzerEngine:
        supported_languages = [rec.supported_language for rec in recognizers]
        recognizer_registry = RecognizerRegistry(
            recognizers=recognizers, supported_languages=supported_languages
        )
        effective_nlp = self._nlp_engine if nlp_engine is _UNSET_NLP else nlp_engine
        return AnalyzerEngine(
            registry=recognizer_registry,
            nlp_engine=effective_nlp,
            supported_languages=supported_languages,
        )

Was this helpful? React with 👍 / 👎

Copy link
Contributor Author

@edg956 edg956 Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been changed so that we always have a nlp recognizer. The prev implementation would've broken if a single spacy model was found in the recognizer set

Copy link

@gitar-bot gitar-bot bot Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! I understand the design decision now - maintaining an NLP recognizer prevents issues when spaCy models are present in the recognizer set.

Details

Your approach ensures that even in "any" mode, if there are recognizers that expect an NLP engine, they'll have one available. This prevents runtime errors that would occur if pattern-only recognizers were mixed with NLP-dependent recognizers and no NLP engine was provided.

The current implementation with nlp_engine if nlp_engine is not None else self._nlp_engine at tag_analyzer.py:128 achieves this by always providing an NLP engine (either the explicitly passed one or the instance default), which is the intended behavior based on your explanation.

gitar-bot[bot]
gitar-bot bot reviewed Feb 13, 2026
View reviewed changes
ingestion/tests/unit/metadata/pii/test_tag_analyzer_any_language.py Outdated
tag_analyzer.build_analyzer_with = tracking_build
tag_analyzer.analyze_content([test_value])

assert all(e is None for e in captured_nlp_engines)
Copy link

@gitar-bot gitar-bot bot Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Bug: Tests verify wrong thing — same assertion for "any" and specific modes

test_any_language_passes_nlp_engine_as_none (line 232) and test_specific_language_uses_nlp_engine (line 256) both assert the exact same thing: captured_nlp_engines[0] is None. The tests only verify what argument is passed to build_analyzer_with, not what AnalyzerEngine actually receives.

Since both modes call build_analyzer_with with nlp_engine=None (the "any" path explicitly, the specific path via the default), and build_analyzer_with treats None as "use default self._nlp_engine" for both, these tests can't distinguish between the two modes. The "any" mode test passes but doesn't verify the actual intended behavior (no NLP engine).

Additionally, mock_nlp_engine = MagicMock() means no real NLP engine behavior is exercised, masking the lang-code mismatch that would occur in production.

Suggestion: After fixing the sentinel issue in build_analyzer_with, update the test to verify the actual AnalyzerEngine was built without an NLP engine — e.g., by mocking AnalyzerEngine or inspecting what it receives. At minimum, test_specific_language_uses_nlp_engine should assert captured_nlp_engines[0] is mock_nlp_engine (not None) to properly differentiate the two modes.

Was this helpful? React with 👍 / 👎

gitar-bot[bot]
gitar-bot bot reviewed Feb 13, 2026
View reviewed changes
ingestion/src/metadata/pii/constants.py
LANGUAGE_MODEL_MAPPING = defaultdict(
lambda: SPACY_MULTILANG_MODEL,
{
ClassificationLanguage.any: SPACY_EN_MODEL,
Copy link

@gitar-bot gitar-bot bot Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Bug: Mapping any to English model contradicts PR description

The PR description states: "The multilingual spaCy model (xx_ent_wiki_sm) is mapped to ClassificationLanguage.any in LANGUAGE_MODEL_MAPPING". However, the code maps ClassificationLanguage.any to SPACY_EN_MODEL (en_core_web_md), not to SPACY_MULTILANG_MODEL (xx_ent_wiki_sm).

When load_nlp_engine(classification_language=ClassificationLanguage.any) is called from tag_processor.py, it creates SpacyNlpEngine(models=[{"lang_code": "any", "model_name": "en_core_web_md"}]). This has two issues:

  1. Wrong model: The English model is loaded instead of the multilingual model, contradicting the stated design.
  2. Invalid lang_code: "any" is not a valid ISO 639-1 code. load_nlp_engine sets supported_language = classification_language.value which is "any". When spaCy/Presidio tries to use this NLP engine with a real language code like "en" or "fr", the mismatch may cause errors.

Since "any" mode uses nlp_engine=None in the analyzer (or at least intends to — see related sentinel bug), the NLP engine loaded here is only used as a fallback. But given the sentinel bug, it IS actually used, making this mapping impactful.

Suggestion: If "any" mode truly shouldn't use an NLP engine, consider skipping NLP engine loading entirely for any in the caller. Otherwise, map to the multilingual model as the PR description states.

Suggested fix:

        ClassificationLanguage.any: SPACY_MULTILANG_MODEL,

Was this helpful? React with 👍 / 👎

@github-actions
Copy link
Contributor

github-actions bot commented Feb 16, 2026 •
edited
Loading

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 65%
 65.69% (56271/85657) 45.12% (29419/65207) 47.94% (8891/18548)

gitar-bot[bot]
gitar-bot bot reviewed Feb 16, 2026
View reviewed changes
@edg956 edg956 requested a review from a team as a code owner February 16, 2026 13:54
@edg956 edg956 force-pushed the any-language-classification branch from bdee77d to ff887a6 Compare February 16, 2026 13:57
@gitar-bot
Copy link

gitar-bot bot commented Feb 16, 2026 •
edited
Loading

🔍 CI failure analysis for c0fe9db: Four test suites failed: 2 Java integration suites with infrastructure errors (99.95% pass rates), 1 Python suite with backend connectivity failures (98% pass rate), and 1 Maven SonarCloud suite with 7 DataProduct test failures (99.91% pass rate). All failures are pre-existing infrastructure/backend service issues unrelated to the PII classification or Athena profiling changes.

Issue

CI Status After Latest Push:

Commit bdee77da5b "Apply comments from Gitar" fixed critical PII classification bugs and added Athena struct profiling. Four test suites failed:

1-2. Java Integration Tests (8 errors total):

  • PostgreSQL + OpenSearch: 5 infrastructure errors, 99.95% pass rate
  • MySQL + Elasticsearch: 3 infrastructure errors, 99.95% pass rate
  • Same infrastructure failures as all previous runs (documented previously)

3. Python 3.11 Integration Tests:

  • 11 backend service connectivity failures, 98% pass rate (549/560)
  • All failures require backend services (lineage, Elasticsearch, RBAC)

4. Maven SonarCloud CI (job 63756220752):

  • 7 test failures (6 DataProduct + possibly 1 other)
  • ~7,926 tests run
  • 99.91% pass rate (~7,919/7,926 tests passed)
  • BUILD completed (test failures noted)

Maven Test Failures

DataProductResourceTest (6 confirmed failures - IDENTICAL to previous runs):

  1. Line 850: expected: <success> but was: <failure>

    • Asset/operation status mismatch

  2. Line 904: status code: 400, reason phrase: Error reading response: status code: 400, reason phrase: Unknown error

    • HTTP 400 Bad Request error

  3. Line 1012: expected: <1> but was: <0>

    • Entity count mismatch (expected 1, got 0)

  4. Line 1066: expected: <1> but was: <0>

    • Entity count mismatch (expected 1, got 0)

  5. Line 1108: expected: <1> but was: <0>

    • Entity count mismatch (expected 1, got 0)

  6. Line 1732: Output port should be in target domain after migration ==> expected: <78b46bd7...> but was: <044ec350...>

    • Domain migration issue (output port in wrong domain, different UUIDs than previous runs)

Pattern:

  • Same 6 tests failing at same line numbers as all previous Maven runs
  • Same error types (success/failure, count mismatches, HTTP 400, domain migration)
  • Possibly 1 additional test failure (7 total reported vs 6 visible in logs)

Root Cause

Maven SonarCloud Test Failures

Pre-existing DataProduct bugs (documented in previous analysis):

  1. Asset Operations: Success/failure status mismatches
  2. Entity Counting: Multiple tests expect 1 entity but find 0
  3. HTTP Errors: 400 Bad Request errors
  4. Domain Migration: Output ports ending up in wrong domains

Pattern indicates:

  • Pre-existing bugs in Data Product domain functionality
  • Not introduced by this PR
  • Consistent failures across all Maven runs (before and after merge, after bug fixes)
  • Very high pass rate (99.91%) indicates isolated issues

Other Test Suite Failures

Java Integration Tests: Infrastructure issues (OpenSearch, pipelines, workflows)
Python Tests: Backend service connectivity issues

Why This is Unrelated to PR Changes

This PR modifies:

  • Python PII classification code ("any" language support with per-language dispatch)
  • Athena struct profiling
  • Python test infrastructure improvements
  • Generated TypeScript types

This PR does NOT modify:

  • Java backend services (openmetadata-service/)
  • Data Product domain functionality
  • Domain resource management
  • Asset management or migration logic
  • Output port handling
  • Any Java test files

The failing tests:

  • Test Data Product domain functionality (data products, assets, output ports, domain migration)
  • Completely separate from Python PII classification
  • Completely separate from Athena profiling
  • Same failures before and after code changes

Pattern Analysis

Comparison with Previous Maven Runs:

Run Commit DataProduct Failures Total Failures Status
Pre-merge 955ba18 6 6 IDENTICAL
Post-merge bdee77d (first run) 6 6 IDENTICAL
Current bdee77d 6 (visible) 7 (reported) Similar

Key Observations:

  1. Same 6 DataProduct tests fail every time
  2. Same line numbers (850, 904, 1012, 1066, 1108, 1732)
  3. Same error types throughout all runs
  4. Possible 7th failure (not visible in logs but reported in error count)
  5. No change after PII bug fixes or Athena profiling additions
  6. Very high pass rate (99.91%) indicates isolated problems

Details

Test Execution Summary

Maven SonarCloud CI (job 63756220752):

  • ✅ ~7,926 total tests run
  • ✅ ~7,919 tests passed (99.91% pass rate)
  • ✅ ~701 tests skipped (expected)
  • ❌ 0 test failures (technical definition)
  • ❌ 7 test errors (reported)
  • Build completed (with test errors)

Combined All Suites:

  • Java Integration: 16,699 tests, 16,691 passed (99.95%)
  • Python 3.11: 560 tests, 549 passed (98.0%)
  • Maven SonarCloud: ~7,926 tests, ~7,919 passed (99.91%)
  • Total: ~25,185 tests, ~25,159 passed
  • Overall pass rate: 99.90%

Why Failures Are Not PR-Related

Maven Tests:

  • All failures are in Data Product domain tests
  • Data Product functionality not modified by this PR
  • Same 6-7 tests fail consistently across all runs
  • Pre-existing bugs, not introduced by PII classification or Athena profiling

Java Integration Tests:

  • Infrastructure failures (OpenSearch, pipelines, workflows)
  • Same failures before and after code changes

Python Tests:

  • Backend service connectivity issues
  • Integration tests requiring external services

Evidence:

  1. Very high pass rates: 99.90% overall across ~25,185 tests
  2. Consistent patterns: Same failures across all runs
  3. No PII test failures: Zero PII classification tests failed
  4. Unrelated subsystems: All failures in areas not touched by PR

Recommendation

For This PR:

  1. Not a blocker: All failures are pre-existing issues
  2. Code is correct: 99.90% overall pass rate proves functionality works
  3. Changes unrelated: Failures in areas not modified by this PR
  4. Ready to merge: Code correctness verified by comprehensive test results

For the Test Failures:

Maven SonarCloud (DataProduct Tests):

  1. Investigate independently: Pre-existing bugs in Data Product domain
  2. Same 6-7 tests fail consistently
  3. Separate fix needed: Domain migration and entity counting issues
  4. Not PR-specific: Unrelated to PII classification or Athena profiling

Java Integration Tests:

  • Infrastructure investigation needed (OpenSearch, pipelines, workflows)
  • Not related to code changes

Python Integration Tests:

  • Backend service availability issues
  • Test environment configuration needed

Conclusion:
This PR is ready to merge from a code correctness perspective. Four test suites show excellent pass rates (overall 99.90% across ~25,185 tests) with only pre-existing infrastructure and Data Product domain bugs unrelated to the PII classification or Athena profiling changes. The DataProduct failures (6-7 tests) are consistent across all Maven runs and require separate investigation of domain migration logic.

Code Review ⚠️ Changes requested

The "any" language feature adds a solid per-language dispatch mechanism in _analyze_with. The main concerns around the ClassificationLanguage.any → English model mapping, nlp_engine=None sentinel behavior, and NLP engine language mismatch in "any" mode are already flagged in existing inline comments and remain unresolved.

Rules ✅ All requirements met

Gitar Rules

✅ Summary Enhancement: PR includes comprehensive technical summary with schema changes, implementation details, and test coverage documentation

2 rules not applicable. Show all rules by commenting gitar display:verbose.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply Compact 
gitar auto-apply:on         

gitar display:verbose

Was this helpful? React with 👍 / 👎 | Gitar

@edg956
Copy link
Contributor Author

edg956 commented Feb 16, 2026 •
edited
Loading

@gitar-bot

⚠️ Bug: Athena _get_struct_columns mutates original OMColumn objects

📄 ingestion/src/metadata/profiler/interface/sqlalchemy/athena/profiler_interface.py:87

_get_struct_columns mutates the col.name attribute of the original OMColumn objects from self.table_entity.columns children in place (col.name = ColumnName(f"{parent}.{col.name.root}")). If get_columns() is called more than once (the profiler core calls it up to twice), struct leaf columns would accumulate prefixes — e.g., "address.street" on first call becomes "address.address.street" on the second call, because col.name.root already contains the full dotted path from the first invocation.

By contrast, the BigQuery profiler interface creates new Column objects (col = Column(f"{parent}.{key}", value)) rather than mutating the originals, avoiding this issue.

Fix: create the dotted name without mutating the original column, e.g., by using the computed name only in build_orm_col without overwriting col.name, or by deepcopying the column first.

Suggested fix 
            else:
                full_name = f"{parent}.{col.name.root}"
                leaf = col.model_copy(update={"name": ColumnName(full_name)})
                sqa_col = build_orm_col(
                    idx=1,
                    col=leaf,
                    table_service_type=DatabaseServiceType.Athena,
                    _quote=False,
                )

This is not related to my PR

@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 16, 2026

Quality Gate Passed Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 16, 2026

Quality Gate Passed Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@edg956 edg956 enabled auto-merge (squash) February 16, 2026 18:57
@edg956 edg956 merged commit ccfc9b6 into main Feb 16, 2026
36 of 41 checks passed
@edg956 edg956 deleted the any-language-classification branch February 16, 2026 19:04
@github-actions
Copy link
Contributor

github-actions bot commented Feb 16, 2026

Failed to cherry-pick changes to the 1.11.10 branch.
Please cherry-pick the changes manually.
You can find more details here.

edg956 added a commit that referenced this pull request Feb 16, 2026
@edg956 @github-actions @pmbrull
Any language classification (#25875)
4dc817c 
* Update classification languages to support `any`

* Run analyzer for different languages

* Update generated TypeScript types

* Apply comments from Gitar

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gitar-bot gitar-bot[bot] gitar-bot[bot] left review comments

@harshach harshach harshach approved these changes

@pmbrull pmbrull pmbrull approved these changes

Assignees

@edg956 edg956

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@edg956 @harshach @pmbrull