low precision/recall on training a custom NER model for new entity using spacy

[euricocovas]

Hi

I am training a custom NER model for a new entity using spacy and python. In this case I have (50,000)+ examples which describe companies. I am after extracting a new type of entity from those descriptions. I trained a NER model with the new entity label, using 30 hand annotated examples and/or auto-annotated examples using the new entity from the wikidata details of those companies. I tested on 3 hand annotated examples. So, 30 examples for training set, 3 for test set. Average length of the examples is 2000+ chars/300+ words. The results are not great, although I am still impressed it worked straight out of the box ... I have a few questions maybe you could be so kind to help or suggest improvements.

1. The results, as a function of the number of examples:
   

The results with wikidata are not good, and this is because wikidata data is not very good/related to the company descriptions I have. That is fine. But the hand annotated results don't reach those 80% or so accuracies I see in some papers... Hand annotating these examples is costly, how far do you think I have to go to get those 80%?

2. The results seem insensitive to dropout rate, not much changes, is this expected?

3. The results seem to saturate after number iterations = 20 or so, is this expected?

4. The results seem insensitive to batch size, not much changes, is this expected?

5. I always start with a empty "en" model ner_model = spacy.blank('en'). I tried transfer learning, i.e., start with en_core_web_sm/en_core_web_md/en_core_web_lg and add my new entity label to existing entity label, in the hope that the other entities helped to filter out false positives, but no real improvement, is this expected? Is there a way to control the "Catastrophic Forgetting"? I used a special type of regularization "L2-SP" successfully once on a paper of mine that I found in a paper on neural networks: https://arxiv.org/pdf/1802.01483.pdf. However, I do not see a way to control regularization with a customized function for spacy underlying neural network... Has anyway done any L1 or L2 weight regularization approach for the underlying neural network?

6. I also tried the en_core_web_trf transformer model, but the loss function seems to go down so slowly that I gave up even with a GPU...

7. Is there a way to do customized loss functions? I am thinking that my problem is not that I want to exactly find the new entity in all its occurrences in each example, my ideal loss function would ignore more than 1 occurrence. As I am after entity extraction, it is irrelevant if the NER find one or more occurrences of the new entity inside the example, all I care is the first occurrence...

Many thanks for any suggestions,

Eurico

[svlandeg]

Hi!

30 examples for training set, 3 for test set.

I think these numbers are too low to draw any meaningful conclusions - especially the 3 instances in the test set. Having only 3 examples make it very difficult to measure fine-grained changed of whether the model is improving with more data, more iterations, different batch size, etc.

Further, the 30 training examples also seems very few. I realise annotating by hand is costly, but the ML algorithms need to have sufficient data to be able to generalize a bit. It's difficult to say upfront just how much data - this really depends on the amount of ambiguity and variation in your data.

A video you might find interesting is this one by @ines: https://www.youtube.com/watch?v=59BKHO_xBPA. It details the usage of Prodigy (our annotation tool) and how to use match patterns to quickly & efficiently bootstrap a model and then iteratively improve upon it. This kind of strategy might help you annotate some more data more quickly.

[euricocovas]

Many thanks for your answer. You are right the size is very small. We are trying to extend and to use annotation tools, many thanks again. Great tool Prodigy btw!