Joint NER and Relation Extraction model misses few relationships while predicting!

[karndeepsingh]

Hello,
I have trained NER and Relation model on my own dataset. I got the dataset annotated with my team manually and the model is able to predict Entities perfectly but it misses on few relationships between the entities. Following images show the NER and Relation output and you can see it is not predicting relations between every entity please help me to understand how this can be improved.

NER Output:

Relation Output:

[karndeepsingh]

Adding on to the above points here is one more example where I am unable to get relations between entites for "Lender" and "Party Name" (near to Lender) in the paragraph. While Borrower's relationship with other entities are getting extracted. Any help to understand this problem. Thanks

[svlandeg]

Hi,

Any ML model will always make mistakes - including false positives as well as false negatives - so it's not unexpected that you won't get perfect results.

That said, if you're looking to improve recall mainly, you could have a look at setting the threshold value of the relation_extractor: https://github.com/explosion/projects/blob/v3/tutorials/rel_component/scripts/rel_pipe.py#L45

[karndeepsingh]

@svlandeg Thanks for your answer. Following things I need to know:

1. Just for clarity,Do we need to add threshold value in rel_pipe.py file under RelationExtraction class.
2. Also I would like to know that I should change the threshold and retrain the model or just changing the threshold in rel_pipe.py file will work while inferencing using the already trained model on different threshold.

[karndeepsingh]

@svlandeg I ran evaluate.py file to evaluate my model on different threshold and I found 0.4 threshold is working good then, how I can set this threshold while inferencing?
I am loading my Relationship model in following way and after loading the model I am passing the text to the nlp model for inferencing. But I am not sure how to add threshold here to improve recall of the model. Please, help me to understand.

import random
import typer
from pathlib import Path
import spacy
import pandas as pd
from spacy import displacy
from spacy.tokens import DocBin, Doc
from spacy.training.example import Example
from rel_pipe import make_relation_extractor, score_relations
from rel_model import create_relation_model, create_classification_layer, create_instances, create_tensors
# We load the relation extraction (REL) model
nlp2= spacy.load("./training/model-best",)
nlp2.add_pipe('sentencizer')

[svlandeg]

If you don't want to retrain your model, you can make a copy of the model-best directory and change the threshold values manually in these places:

• file config.cfg, setting components.relation_extractor.threshold
• file cfg in submap relation_extractor

Then when you load the copied, adjusted model back in, you can double check the threshold by printing

nlp.get_pipe("relation_extractor").threshold

[karndeepsingh]

@svlandeg Thank you so much for this informative answer. I will definitely try and check the performance after changing this threshold.