Implement EntityRuler upstream of NER component for training NER model

[kliu-qt]

Is there a way to have an EntityRuler in front of a NER component during training so it provides some information for NER to learn from?

I tried including a EntityRuler in the config file and set it as annotating component. But I'm getting this error below:

Aborting and saving the final best model. Encountered exception: ValueError("[E024] Could not find an optimal move to supervise the parser. Usually, this means that the model can't be updated in a way that's valid and satisfies the correct annotations specified in the GoldParse.

My config file looks like this:

[paths]
train = ""
dev = ""
vectors = "en_core_web_lg"
init_tok2vec = "en_core_web_lg"

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","entity_ruler","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.entity_ruler]
source = "path_to_ruler"

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 5000
max_epochs = 0
max_steps = 0
eval_frequency = 200
frozen_components = ["entity_ruler"]
annotating_components = ["entity_ruler"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[polm]

The EntityRuler is designed to be used with the NER component, including putting the ruler first so that the NER can predict around it - see this section in the docs.

I am not sure if this is an issue with annotating_components or with your data. Can you train a model using your data if you remove the EntityRuler from annotating_components?

[kliu-qt]

I can train with my data by removing EntityRuler from annotating_components.

[adrianeboyd]

Is the entity ruler adding entity labels that aren't in your training data otherwise?

[kliu-qt]

No, the EntityRuler have the same labels as the ones in my labeled corpus.

[adrianeboyd]

I think the problem might be that the labels aren't initialized correctly for ner? I would try the following:

• generate a small sample .spacy corpus using the entity ruler to annotate all entity types, being sure to include all possible entity types as single-word and multi-word entities if possible
• use spacy init labels to generate the ner labels from this corpus (you can provide the same file as train and dev here, since dev isn't used for this)
• add the labels to the ner initialize block in the config (as shown in the sidebar: https://spacy.io/usage/training#initialization-labels)

[kliu-qt]

btw this is what the config file looks like now in case you want to see.

Thanks for helping

train = "temp.spacy"
dev = "temp.spacy"
vectors = "en_core_web_lg"
init_tok2vec = "en_core_web_lg"

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","entity_ruler","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.entity_ruler]
source = "path_to_entityRuler"

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 5000
max_epochs = 0
max_steps = 0
eval_frequency = 200
frozen_components = ["entity_ruler"]
annotating_components = ["entity_ruler"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.ner]

[initialize.components.ner.labels]
@readers = "spacy.read_labels.v1"
path = "path_to_ner.json"

[initialize.tokenizer]```

[darrkj]

What do you mean by hints? It seems odd to have the entity ruler set as an annotating_component as I thought that meant the training would use what is created by that component to train and in this case it is creating the same annotation the ner is predicting. The documentation states that having the entity ruler prior to ner will force ner to respect the entities that originate from the entity ruler, so ner wont create a new entity if there are tokens that are part of an entity that entity ruler created. The only way I have been able to get this to work though is by not actually using the entity_ruler as the component but instead adding a custom named component that wraps the entity ruler, some of that is discussed here #9776.

[polm]

Your Markdown formatting is off, you need to use three backticks to start and end code blocks. See the docs on that.

I'm still not sure why you'd get an error. If you can put an example we can run to reproduce the issue on Github somewhere, maybe with fake data similar to your own, we can debug that for you - if all your labels are pre-declared, or present in the NER annotations, then it should work.

About giving hints to the NER model, can you describe that in more detail? While the NER model can learn to annotate around existing labels, it's not a very frequently used approach, so more details would help us understand what you're doing better. A concrete example (like a sample sentence) would be great.

[kliu-qt]

@polm
Regarding giving hints to the NER, here's an example.

In the sentences "I bought some nestle chocolate" and "I'm gonna nestle in for the night" the entityRuler may tag both, and a ML model that hasn't seen nestle as a form of Nestlé before may not tag either. The hope is by highlighting the possibility of 'nestle' in such a sentence being a brand, the ML model might be able to recognize it more reliably than just working from part-of-speech. The point is to see whether such benefit manifests, and it may well not.

[polm]

Thanks for the example, that's very helpful for understanding what you're doing.

In training, each Example has a reference (gold) and predicted Doc associated with it. Normally the model tries to make the "predicted" Doc match the gold Doc, and updates its weights ("learns") based on any failures it makes.

If you use annotating_components, that means that modifications are made to the predicted Doc. So in your case you may update the predicted Doc to be equal to the gold Doc before the NER component sees it. In that case it won't have anything to do and won't be able to learn much. And since the NER component doesn't change existing annotations, it won't update "nestle in bed" to remove the annotation, and can't learn from that either.

So I'm not sure that what you are trying to do would work. (That said, I still don't think it should result in an error, so if you have example code we'd still be glad to see it.)

However, there are other ways of doing what you want to do. You might want to look into Weak Supervision (for example with skweak) or Data Augmentation (for example with nlpaug).