Not getting predictions for more than half of the entities after training spacy NER model #12576
-
|
I have around 20 entities (sometimes more) that I am trying to put in a model for training all having a good sample size of around 1000 samples. The spacy NER model gets trained on the data but when I do predictions, I only get predictions for half of my entities sometimes for 10 or 12 sometimes for 8 entities only. following is the configuration I use. Please suggest what changes might help me to train the model better. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 9 replies
-
|
Hi @ad-straive, please provide a sample of your training data, run |
Beta Was this translation helpful? Give feedback.
-
|
Also, I recommend collecting the distribution of entity labels in your train/dev/test sets. It's possible that you don't have enough samples for some of your labels. |
Beta Was this translation helpful? Give feedback.
-
|
Please find my inline replies:
I meant number of entity_labels that I am trying to fit in a single model is 20 or, sometimes it is 30 different entity_labels in a single model. In total have around 100 different entity_labels that I am trying to put in 4 different models. For convinience let's say 25 labels each in 4 models for training. But when I have these many (25) labels in one model I am not able to get prediction for around half the labels after training with sufficient data (each label having around 2000 samples).
The above evaluation result was from the model trained using all the data. When I reduce the data to 200 samples per label for model training, I get predictions for more set of labels. Sharing evaluation results below from both the models. Evaluation on model trained using around 2000 samples each: Evaluation on model trained using 200 samples each: It seems more like a memory issue because the model gets trained on all entity_labels when number of samples for each label is less. |
Beta Was this translation helpful? Give feedback.
-
|
Below are the model training logs for both the cases: (if not necessary let me know I will delete)
|
Beta Was this translation helpful? Give feedback.
-
|
I have tried few other variations like decreasing the number of entitiy_labels(el) in a model, that helps but even those model have difficulty predicting all labels. Like I had tried training a 10 entity_labels model using 2000 samples each in that case I got predictions for only 7 el. But the same 10 el model with 1000 sample each gave prediction for 9 el. Based on my experimentations I could only figure out that there needs to be a balance between the number of entity_labels and number of samples/per entity_labels. If one increases I need to decrease the other one to train a proper model. So my question would be - |
Beta Was this translation helpful? Give feedback.
-
As mentioned before - please provide distribution of entity labels in your train/dev/test sets. Also, why and how are you using four different models?
If you don't run into out-of-memory issues, don't worry about your GPU memory. |
Beta Was this translation helpful? Give feedback.
Hi @ad-straive, please provide a sample of your training data, run
spacy benchmark accuracyon your data and share the evaluation results.