Help with training a NER model that returns confidence score for each entity

[akdanwlf]

Hello Everyone,

I would appreciate it if anyone can help me with the following two things:

• Validate the config file for the task I am doing
• Suggest how should I generate the labels JSON file to be used to initialize the spancat component.

Task
I need to build a ner model that provides a confidence score with each entity it finds.

Approach
I am trying to train a spacy pipeline containing tok2vec, spancat, and ner. I have a custom labelled IOB format training and validation data which I converted to spacy binary format using python -m spacy convert ./train.iob ./ -n 10.

I added the required configuration to include tok2vec, span-cat and ner, also updated span-cat to use tok2vec listener, like suggested here #9601 (reply in thread). But when I run the debug command python -m spacy debug data config.cfg to validate, I get the following error.

ValueError: [E143] Labels for component 'spancat' not initialized. This can be fixed by calling add_label, or by providing a representative batch of examples to the component's initialize method.

So I am trying to add the initialize.component.spancat and initialize.component.spancat.labels to the config file, but I couldn't find a source on how to generate the spancat.json file.

[initialize.components.spancat]

[initialize.components.spancat.labels]
@readers = "spacy.read_labels.v1"
path = "../data/spancat.json"

My config.cfg file looks like this:

[paths]
train = "../data/train.spacy"
dev = "../data/val-data.spacy"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","spancat","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = "en_core_web_lg"
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
before_init = null
after_init = null

[initialize.lookups]
@misc = "spacy.LookupsDataLoader.v1"
lang = ${nlp.lang}
tables = ["lexeme_norm"]

[initialize.components]

[initialize.tokenizer]

Using Python v3.8 and spaCy v3.1.4

[polm]

Sorry you're having trouble with this, you might find it helpful to look at the spancat example project for reference.

So I am trying to add the initialize.component.spancat and initialize.component.spancat.labels to the config file, but I couldn't find a source on how to generate the spancat.json file.

The file is just a JSON list of labels, like ["red", "green", "blue"]. However normally you don't need the file unless you're using a streaming corpus or something. It could possibly be an issue if your annotations aren't being picked up for some reason.

Are you sure you need an NER component in addition to a spancat component? In the linked issue the user just asked if it was possible to include both in the same pipeline, which it is, but typically I would expect you to only need one or the other.

[akdanwlf]

Grateful for the help. I am able to complete my task. I would like to help anyone else with a similar problem, if you guys suggest I can contribute towards any documentation regarding this?

[polm]

Thanks for the offer! What kinds of changes to the docs did you have in mind / what would have helped you here?

[steve-solun]

@polm I think that adding a colab or a notebook with a simple example of labeled entities + spacy blank model + relevant config file showing a working example how to train a spancat model will be super helpful. I am investing so much time to gather the bits and bytes to make it work, without any luck. See my questions here and on stackoverflow, let's make a simple working example from A to Z.

[rmitsch]

@steve-solun Are you aware of the projects tutorials on NER? If so and you still think these are insufficient, could you let me know how do you think we could improve those?

[steve-solun]

@rmitsch yes, I have checked all the docs and most of the projects are 1+ years old and don't contain a simple readable real world A-Z example of taking annotated data and training a spancat model.
I will be happy if you would share the tutorial in a colab and show people how to use it(I am not the first and not the last). It should be constantly updated and add more and more examples. It will populate the methods and make them much more efficient. You will get so much good valuable feedback.