Unable to train Spancat model, accuracy stuck at 0.0

[us]

Losses decrease to 0 within 2000-3000 epochs, f, p, r scores do not go higher than 0 (image 1).

Then I tried a different datasets and got a result like this in image2. But when I tried the old dataset again with the same config, I stayed at 0 again, then I tried a different dataset and I was left at 0 again. I tried different different sizes for ngram_range_suggester but I couldn't find what is wrong!

image 1

image2

How to reproduce the behaviour

Here is the preprocess and config files that I used for training.

preprocess.py for creating .spacy files.

def main(
        input_path: Path = typer.Argument(..., exists=True, dir_okay=False),
        output_path: Path = typer.Argument(..., dir_okay=False),
):
    nlp = spacy.blank("en")
    doc_bin = DocBin()
    for eg in srsly.read_jsonl(input_path):
        doc = nlp.make_doc(eg["text"])
        doc.spans['sc'] = []
        for s in eg.get("spans", []):
            span = doc.char_span(s["start"], s["end"], label=s["label"])
            if span:
                doc.spans["sc"].append(span)
        if doc.spans:
            doc_bin.add(doc)
    doc_bin.to_disk(output_path)
    print(f"Processed {len(doc_bin)} documents: {output_path.name}")

Here is my config.cfg

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","spancat"]
batch_size = 100
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_range_suggester.v1"
min_size = 1
max_size = 30

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 128
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
gold_preproc = false
limit = 0
augmenter = null
max_length = 0

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
gold_preproc = false
limit = 0
augmenter = null
max_length = 0

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null
annotating_components = []

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = true
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.0
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Info about spaCy

• spaCy version: 3.4.1
• Platform: macOS-12.3.1-x86_64-i386-64bit
• Python version: 3.9.13

[polm]

Sorry this isn't working for you. Flat zero scores in training usually indicate an issue with training data rather than a config problem. Your preprocessing script looks OK, but maybe there's an issue with your annotations and no spans are being created?

You can try using spacy debug data to check your data, or log spans in your preprocess script to be sure they're getting created. How many non-empty annotations do you have?

[us]

I checked data several times, it looks good.

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, spancat
8120 training docs
928 evaluation docs
✔ No overlap between training and evaluation data

============================== Vocab & Vectors ==============================
ℹ 977009 total word(s) in the data (22663 unique)
ℹ No word vectors present in the package

============================ Span Categorization ============================

Spans Key   Labels
---------   --------------------
sc          {'LABEL_1'}

ℹ Span characteristics for spans_key 'sc'
ℹ SD = Span Distinctiveness, BD = Boundary Distinctiveness

Span Type          Length   SD     BD
----------------   ------   ----   ----
LABEL_1   9.32     2.49   1.95
----------------   ------   ----   ----
Wgt. Average       9.32     2.49   1.95

ℹ Over 90% of spans have lengths of 1 -- 16 (min=1, max=34). The most
common span lengths are: 5 (2.41%), 6 (1.94%), 7 (13.6%), 8 (18.74%), 9
(17.06%), 10 (13.49%), 11 (8.82%), 12 (5.51%), 13 (3.62%), 14 (1.73%), 15
(2.2%), 16 (1.78%). If you are using the n-gram suggester, note that omitting
infrequent n-gram lengths can greatly improve speed and memory usage.
✔ Spans are distinct from the rest of the corpus
✔ Boundary tokens are distinct from the rest of the corpus
✔ Good amount of examples for all labels
✔ Examples without ocurrences available for all labels

================================== Summary ==================================
✔ 7 checks passed

[polm]

Hm, that's weird, in that case there's nothing obviously wrong, so maybe it's something in the details...

It looks like you've changed some of the default values in the config, could you try setting them back to their original values? Specifically these values (note these are in separate sections, and I'm just listing them for together for compactness):

L2 = 0.01 # is 0.0 in your config
discard_oversize = fase # is true in your config

One other thing is that your pipeline is configured to include vectors, but there are no vectors in your config file. You should set include_static_vectors to false or include some vectors, such as by specifying en_core_web_lg for paths.vectors.

[polm]

Talking with the team, it sounds like there's another possibility - it seems that there may be a bug where the spancat can fail silently if it runs out of memory. Can you try using a very small batch size (like 10) and reducing the size of your ngrams (so more like 1-5) to see if the model is able to learn?

[darrkj]

I have gotten this error in a few scenarios. One thing I have noticed recently though is the complexity of the task, simple things were less likely to generate while predictions of more complex spans made it more likely. I then annotated more training data and retrained, getting the same repeated zero. However if I package the results and pip install it I can see that it is able to make valid predictions on out of sample documents.

[goonhoon]

Hi @darrkj, apologies for reviving this thread but could you explain what 'packaging the results' mean? I am facing the same issue and think that the dataset being too large could be an issue.

Thanks!

[kormilitzin]

@darrkj I have been experiencing a very similar behaviour, i.e., complex annotations resulted in zeros in training but the trained model was able to predict meaningfully. I am wondering if there is a common patter and something to do with annotations / spans?

Are there any metrics that can indicate this strange behaviour?

[polm]

It's possible to have zero scores in training but actually train a model if your logging is configured incorrectly. For example, in the config of the original poster here's the section that controls logged scores:

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

Note the "sc" in the scores here - that depends on the spans key. If you change the key but don't change the score weights, the actual scores will be stored under different names, so they won't show up in output.

This was not the case with the OP, where we can see the spans key and score weights match.

[darrkj]

This was sorta the issue I had. I went back and checked a few things and it was these values. The issue I had was not the name thought it was the initial value, I had each of them set to null to start. I made the change like they are there to be 1, 0 and 0 and then retrained and the values are no longer all zero.

[kormilitzin]

Thanks @darrkj. Could you give a simple example of you changed the initial values? So my understanding is that you set the initial values as:

I made the change like they are there to be 1, 0 and 0

similar to @polm's answer in above:

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

am I correct?