zh context words are not being picked up

[ChirangaL]

Hi Everyone,
Im trying to get the context words in foreign languages to work. Below code that uses Spacy zh is not picking up the context words. Is there a way to use chinese context words to enhance score?

from presidio_analyzer import PatternRecognizer, AnalyzerEngine, RecognizerRegistry

# Create configuration containing engine name and models
configuration = {
    "nlp_engine_name": "spacy",
    "models": [{"lang_code": "zh", "model_name": "zh_core_web_lg"}],
}

# Create NLP engine based on configuration file
provider = NlpEngineProvider(nlp_configuration=configuration)
nlp_engine_with_spanish = provider.create_engine()

context_aware_enhancer = LemmaContextAwareEnhancer(
    context_similarity_factor=0.45, min_score_with_context_similarity=0.4
)

# Define the recognizer with the defined pattern and context words
chinese_recognizer = PatternRecognizer(
   supported_entity="CHINESE_TEXT",
   patterns=[Pattern(
            name="HK_PASSPORT",
            regex="[a-zA-Z]{1,2}\d{6}(?:\([\dA]\)|[\dA])",
            score=0.3,
        )],
   context=["护照"],
    supported_language="zh"
)

registry = RecognizerRegistry()
registry.add_recognizer(chinese_recognizer)
analyzer = AnalyzerEngine(registry=registry, supported_languages=["zh"], nlp_engine=nlp_engine_with_spanish,
                          context_aware_enhancer=context_aware_enhancer,
                          default_score_threshold=0.4,)

# Test
results = analyzer.analyze(text="护照AB123456(A)", language="zh")
print("Result:")
print(results)

[omri374]

Hi, thanks for posting this. The context enhancement mechanism works on tokens. It could be that spaCy tokenizes 护照AB123456(A) as one token? I'm not a native speaker and never used the zh model so I'm not sure.

[omri374]

You can pass return_decision_process=True to the .analyze method to see if context was used, and if yes, how.

[ChirangaL]

Hi @omri374,

I used below code to compare the Spanish and Chinese together. What I noticed is

1. Tokenization is happening
2. Lemma is not being picked up.

Any thoughts?

from presidio_analyzer import PatternRecognizer, AnalyzerEngine, RecognizerRegistry

# Create configuration containing engine name and models
configuration = {
    "nlp_engine_name": "spacy",
    "models": [ {"lang_code": "en", "model_name": "en_core_web_lg"},
               {"lang_code": "zh", "model_name": "zh_core_web_lg"},
               {"lang_code": "es", "model_name": "es_core_news_md"}]
}

# Create NLP engine based on configuration file
provider = NlpEngineProvider(nlp_configuration=configuration)
nlp_engine_with_spanish = provider.create_engine()

context_aware_enhancer = LemmaContextAwareEnhancer(
    context_similarity_factor=0.45, min_score_with_context_similarity=0
)

# Define the recognizer with the defined pattern and context words
chinese_recognizer = PatternRecognizer(
   supported_entity="CHINESE_TEXT",
   patterns=[Pattern(
            name="HK_PASSPORT",
            regex="[a-zA-Z]{1,2}\d{6}(?:\([\dA]\)|[\dA])",
            score=0.3,
        )],
   context=[
                "香港特別行政區護照",
                "香港護照",
                "護照號碼",
                "旅行文件",
                "香港旅行文件",
                "英國國民(海外)護照",
                "特區護照",
                "文件號碼",
                "護照身份證明",
                "官方旅行文件",
                '旅行',
                '文件'
            ],
    supported_language="zh"
)

spanish_recognizer = PatternRecognizer(
   supported_entity="CHINESE_TEXT",
   patterns=[Pattern(
            name="HK_PASSPORT",
            regex="[a-zA-Z]{1,2}\d{6}(?:\([\dA]\)|[\dA])",
            score=0.3,
        )],
   context=["pasaporte"],
    supported_language="es"
)

registry = RecognizerRegistry()
registry.add_recognizer(chinese_recognizer)
registry.add_recognizer(spanish_recognizer)

analyzer = AnalyzerEngine(registry=registry, supported_languages=["zh","es", "en"], nlp_engine=nlp_engine_with_spanish,
                          context_aware_enhancer=context_aware_enhancer,
                          default_score_threshold=0.4,
                          log_decision_process=True)

# Test
analyzer.analyze(text="旅行文件AB123456(A)", language="zh", return_decision_process=True)
analyzer.analyze(text="pasaporte AB123456(A)", language="es", return_decision_process=True)

Chinese

[2024-01-03 11:22:30,434][decision_process][INFO][None][nlp artifacts:{"entities": [], "tokens": ["\u65c5\u884c", "\u6587\u4ef6", "AB", "123456", "(", "A)"], "lemmas": ["", "", "", "", "", ""], "tokens_indices": [0, 2, 4, 6, 12, 13], "keywords": ["", "", "", "", "", ""], "scores": []}]

[2024-01-03 11:22:30,435][decision_process][INFO][None][["{'entity_type': 'CHINESE_TEXT', 'start': 4, 'end': 15, 'score': 0.3, 'analysis_explanation': {'recognizer': 'PatternRecognizer', 'pattern_name': 'HK_PASSPORT', 'pattern': '[a-zA-Z]{1,2}\\d{6}(?:\\([\\dA]\\)|[\\dA])', 'original_score': 0.3, 'score': 0.3, 'textual_explanation': None, 'score_context_improvement': 0, 'supportive_context_word': '', 'validation_result': None, 'regex_flags': regex.I|M|S}, 'recognition_metadata': {'recognizer_name': 'PatternRecognizer', 'recognizer_identifier': 'PatternRecognizer_13008803920'}}"]]

Spanish

[2024-01-03 11:22:30,439][decision_process][INFO][None][nlp artifacts:{"entities": [], "tokens": ["pasaporte", "AB123456(A", ")"], "lemmas": ["pasaporte", "AB123456(A", ")"], "tokens_indices": [0, 10, 20], "keywords": ["pasaporte", "ab123456(a"], "scores": []}]

[2024-01-03 11:22:30,439][decision_process][INFO][None][["{'entity_type': 'CHINESE_TEXT', 'start': 10, 'end': 21, 'score': 0.75, 'analysis_explanation': {'recognizer': 'PatternRecognizer', 'pattern_name': 'HK_PASSPORT', 'pattern': '[a-zA-Z]{1,2}\\d{6}(?:\\([\\dA]\\)|[\\dA])', 'original_score': 0.3, 'score': 0.75, 'textual_explanation': None, 'score_context_improvement': 0.45, 'supportive_context_word': 'pasaporte', 'validation_result': None, 'regex_flags': regex.I|M|S}, 'recognition_metadata': {'recognizer_name': 'PatternRecognizer', 'recognizer_identifier': 'PatternRecognizer_12126176720'}}"]]

[ChirangaL]

Ok. I think I found the answer. Spacy just doesnt support lemmatization for chinese language.
explosion/spaCy#10386

I have updated the code to use Stanza and it is doing lemmatization properly.

configuration = {
    "nlp_engine_name": "stanza",
    "models": [ {"lang_code": "en", "model_name": "en"},
               {"lang_code": "zh", "model_name": "zh"},
               {"lang_code": "es", "model_name": "es"}]
}

[omri374]

Yes, looks like lemmas are missing. Another option would be to tweak the ContextAwareEnhancer to use tokens instead of lemmas.