Displacy not recognising custom entities

[vrunm]

I am trying to write a custom ner program. I am able to extract those entities properly.The displacy is not able to display them.

import spacy
from spacy import displacy
import re
from IPython.display import HTML

# Load the spaCy English model
nlp = spacy.load("en_core_web_sm")

# Process the cleaned text
doc = nlp(cleaned_text)

# Define the custom NER patterns
ner_patterns = [
    {"label": "BUN", "pattern": [{"SHAPE": "Xxxxx"}, {"LOWER": "bun"}]},
    {"label": "SerumUricAcid", "pattern": [{"SHAPE": "Xxxxx"}, {"LOWER": "serum"}, {"LOWER": "uric"}, {"LOWER": "acid"}]},
    {"label": "SerumPotassium", "pattern": [{"SHAPE": "Xxxxx"}, {"LOWER": "serum"}, {"LOWER": "potassium"}]}
]

# Add the custom NER patterns to the entity ruler
ruler = nlp.add_pipe("entity_ruler", after='ner')
ruler.add_patterns(ner_patterns)

# Extracted values
extracted_values = {}

# Process the tokens to extract values
for token in doc:
    if token.text in ["BUN", "Serum", "Uric", "Acid", "Potassium"]:
        label = token.text
        next_token = token.nbor()
        if next_token.like_num:
            value = next_token.text
            extracted_values[label] = value

# Print the extracted values
print("Extracted values:")
for label, value in extracted_values.items():
    print(label + ":", value)

# Prepare entity highlighting with linear gradients
colors = {
    "BUN": "linear-gradient(90deg, #aa9cfc, #fc9ce7)",
    "SerumUricAcid": "linear-gradient(90deg, #ffabab, #ffd8a8)",
    "SerumPotassium": "linear-gradient(90deg, #a8e6cf, #dcedc1)"
}
options = {"ents": ["BUN", "SerumUricAcid", "SerumPotassium"], "colors": colors}

# Render the displaCy visualization with highlighting
html = displacy.render(doc, style="ent", options=options)

# Display the HTML
HTML(html)

In the output I am getting the extracted values just fine. The displacy is working just fine with normal entities.Its just not able to recognise custom labels. It prints the text again with no highlight whatsoever.

Cleaned text being used:
BIOCHEMISTRY KIDNEY FUNCTION TEST (KFT) TEST VALUE UNIT REFERENCE BUN 10.27 mg/dl 7.9 - 20 Serum Urea 22 mg/dl 13 - 40 Serum Creatinine H 0.9 mg/dl 0.5 - 0.8 Serum Calcium 9.0 mg/dl 8.8 - 10.6 Serum Potassium 3.9 mmol/L 3.5 - 5.1 Serum Sodium L 132 mmol/L 136 - 146 Serum Uric Acid 5 mg/dl 2.6 - 6 Urea / Creatinine Ratio 24.44 BUN / Creatinine Ratio 11.41 ~~~ End of report ~~~ LABSMART SAMPLE REPORT Patient Name: Mrs. Dummy Registered on: 09/08/2022 11:35 AM 1001 Age / Sex: 34 YRS / F Collected on: 09/08/2022 Referred By: Dr. Self Received on: 12/08/2022 Reg. no. / UHID: 1001 / Reported on: 09/08/2022 11:35 AM Investigations: Kidney Function Test (KFT) Page 1 of 1 Mr. Sachin Sharma DMLT, Lab Incharge Dr. A. K. Asthana MBBS, MD Pathologist

[shadeMe]

Hi, thanks for providing the code - I'm able to reproduce this locally. We'll investigate and get back to you once we know more.

[adrianeboyd]

I had another look at this and it looks your entity_ruler component isn't annotating any entities, either because the patterns are incorrect or because of conflicts with the entities from ner. You can see the underlying annotation with displacy if you leave out the custom options.

To see just the entity_ruler annotation, try:

nlp = spacy.load("en_core_web_sm", disable=["ner"])

...

displacy.serve(doc, style="ent")

(I don't think the SHAPE patterns are matching like you're expecting.)

If you want to combine ner and entity_ruler, then you might need to use overwrite_ents=True to let entity_ruler overwrite existing spans if there are overlaps, or put the entity_ruler component first.