NER models give different results when I add the Date label for my data

[Buddhikailabs]

I have trained 2 models with 10 labels and 11 labels.
10 labels

11 labels

In the first model precision, recall and F1 score of each entity is fine. But when I add a Date label for the dataset precision , recall and F1 has decreased significantly. I using google colab free version for training the model.

[polm]

Please don't post screenshots of text, copy and paste text as text.

Can you give more information about your training? With just the info above there's not really any way to help.

Does each model have the same data and same config, except for the extra labels, or are there other differences? What's the config look like? How about the output of debug data?

[Buddhikailabs]

Yes. Only new label is added to the data set. And size of data is slightly changed

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = "/content/drive/MyDrive/SpacyModelTrain/V3_normal_encoding/train.spacy"
dev = "/content/drive/MyDrive/SpacyModelTrain/V3_normal_encoding/validation.spacy"

[system]
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000

[components]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH", "SHAPE"]
rows = [5000, 2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[initialize]
vectors = "en_core_web_lg"

Debug data of 10 label model

============================ Data file validation ============================
tcmalloc: large alloc 1643593728 bytes == 0x55da142f0000 @  0x7fa7844c92a4 0x7fa772a278bd 0x7fa772a26a9d 0x7fa772a23275 0x7fa772a23a1e 0x55d9d07f51cd 0x55d9d08e7b3d 0x55d9d0869458 0x55d9d086402f 0x55d9d07f6aba 0x55d9d0865cd4 0x7fa5664b21c8 0x7fa5664b416e 0x7fa5664baca7 0x7fa5664bdf32 0x55d9d07f5b32 0x55d9d08691dd 0x55d9d07f69da 0x55d9d0864eae 0x7fa5664b21c8 0x7fa5664b416e 0x7fa5664ba980 0x55d9d07f5424 0x55d9d07f5120 0x55d9d0869b80 0x55d9d086466e 0x55d9d07f6aba 0x55d9d0865cd4 0x55d9d086466e 0x55d9d07f6aba 0x55d9d0865108
tcmalloc: large alloc 1662509056 bytes == 0x55daa8428000 @  0x7fa7844c92a4 0x7fa772a24fd3 0x7fa772a26a9d 0x7fa772a23275 0x7fa772a23a1e 0x55d9d07f51cd 0x55d9d08e7b3d 0x55d9d0869458 0x55d9d086402f 0x55d9d07f6aba 0x55d9d0865cd4 0x55d9d07f69da 0x55d9d08692c0 0x7fa5664b21c8 0x7fa5664b416e 0x7fa5664ba980 0x55d9d07f5424 0x55d9d07f5120 0x55d9d0869b80 0x55d9d086466e 0x55d9d07f6aba 0x55d9d0865cd4 0x55d9d086466e 0x55d9d07f6aba 0x55d9d0865108 0x55d9d086466e 0x55d9d07f6aba 0x55d9d0865cd4 0x55d9d086402f 0x55d9d0735e2b 0x55d9d0866633
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, ner
1749 training docs
194 evaluation docs
✔ No overlap between training and evaluation data
⚠ Low number of examples to train a new pipeline (1749)

============================== Vocab & Vectors ==============================
ℹ 118877 total word(s) in the data (16596 unique)
ℹ 684830 vectors (684830 unique keys, 300 dimensions)
⚠ 15336 words in training data without vectors (13%)

========================== Named Entity Recognition ==========================
ℹ 10 label(s)
0 missing value(s) (tokens with '-' label)
✘ 1 invalid whitespace entity spans
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities crossing sentence boundaries
Entity spans consisting of or starting/ending with whitespace characters are
considered invalid.

================================== Summary ==================================
✔ 6 checks passed
⚠ 2 warnings
✘ 1 error

Debug data of new model with 11 labels(Date label is the newly added label)

============================ Data file validation ============================
tcmalloc: large alloc 1643593728 bytes == 0x558d41c60000 @  0x7faca65262a4 0x7fac9b0a18bd 0x7fac9b0a0a9d 0x7fac9b09d275 0x7fac9b09da1e 0x558cfc7e51cd 0x558cfc8d7b3d 0x558cfc859458 0x558cfc85402f 0x558cfc7e6aba 0x558cfc855cd4 0x7fab1c99c1c8 0x7fab1c99e16e 0x7fab1c9a4ca7 0x7fab1c9a7f32 0x558cfc7e5b32 0x558cfc8591dd 0x558cfc7e69da 0x558cfc854eae 0x7fab1c99c1c8 0x7fab1c99e16e 0x7fab1c9a4980 0x558cfc7e5424 0x558cfc7e5120 0x558cfc859b80 0x558cfc85466e 0x558cfc7e6aba 0x558cfc855cd4 0x558cfc85466e 0x558cfc7e6aba 0x558cfc855108
tcmalloc: large alloc 1662509056 bytes == 0x558dd5d98000 @  0x7faca65262a4 0x7fac9b09efd3 0x7fac9b0a0a9d 0x7fac9b09d275 0x7fac9b09da1e 0x558cfc7e51cd 0x558cfc8d7b3d 0x558cfc859458 0x558cfc85402f 0x558cfc7e6aba 0x558cfc855cd4 0x558cfc7e69da 0x558cfc8592c0 0x7fab1c99c1c8 0x7fab1c99e16e 0x7fab1c9a4980 0x558cfc7e5424 0x558cfc7e5120 0x558cfc859b80 0x558cfc85466e 0x558cfc7e6aba 0x558cfc855cd4 0x558cfc85466e 0x558cfc7e6aba 0x558cfc855108 0x558cfc85466e 0x558cfc7e6aba 0x558cfc855cd4 0x558cfc85402f 0x558cfc725e2b 0x558cfc856633
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, ner
1800 training docs
218 evaluation docs
✔ No overlap between training and evaluation data
⚠ Low number of examples to train a new pipeline (1800)

============================== Vocab & Vectors ==============================
ℹ 121623 total word(s) in the data (16643 unique)
ℹ 684830 vectors (684830 unique keys, 300 dimensions)
⚠ 15544 words in training data without vectors (13%)

========================== Named Entity Recognition ==========================
ℹ 11 label(s)
0 missing value(s) (tokens with '-' label)
✘ 1 invalid whitespace entity spans
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities crossing sentence boundaries
Entity spans consisting of or starting/ending with whitespace characters are
considered invalid.

================================== Summary ==================================
✔ 6 checks passed
⚠ 2 warnings
✘ 1 error

[polm]

Hm, nothing in the output you gave is jumping out at me as wrong.

From your description, it sounds like maybe you have a base dataset with your ten labels annotated, and then you have 100 or so extra examples where you've only annotated dates. Is that the case? If so, what might be happening is the model could be un-learning other labels or getting inconsistent information. For example, if you have a sentence like John Smith's birthday is January 1 and you've marked "January 1" as a DATE but you didn't label "John Smith" as a PERSON, that could confuse the model and cause problems.

Does that accurately describe your extra data?

If that doesn't, I'd need to think more about what the problem could be. Would it be possible to share your data or part of it?