Simple Spancat model fails to train with Spanruler annotations

[Kau832]

I was reading through the example SpanRuler project as well as ljvmiranda921's blog post on SpanRuler but decided to train my sample model using the spanruler annotations rather than adding the ruler to a pretrained model:

I annotated a sample dataset using the below preprocessing script and validated the data in debug data. I split the training data into training and dev folders, adjust my ngram suggester to cover the relevant span lengths from debug data and began my training. However, all the training scorer metrics are 0.00. I have spent about 20 hours on this in this week alone and cannot figure out why this is happening.

Preprocessing, data debug, training evaluation, sample displacy render and config.cfg all below. Thanks for any help!

Preprocessing to save annotations to doc.spans["ruler"]:

import spacy
from spacy.tokens import DocBin

nlp = spacy.load("en_core_web_lg")
ruler = nlp.add_pipe("span_ruler", name="ruler")

patterns = [{"label": "Effective Date", "pattern": [
    {"ENT_TYPE": "DATE", "OP": "{4,}"},
    {"TEXT": '('},
    {"TEXT": 'the'},
    {"TEXT": '"'},
    {"TEXT": 'Effective'},
    {"TEXT": 'Date'},
    {"TEXT": '"'},
    {"TEXT": ')'}
]}]

ruler.add_patterns(patterns)

for i in range(100, 510):

    text_open = open(f"inputfiles/ ({i}).txt", "r", encoding='utf8')
    text = text_open.read()
    doc = nlp(text)

    db = DocBin()
    for span in doc.spans["ruler"]:
            print(span.text, span.label_)

    text_open.close()


    db.add(doc)
    db.to_disk(f"./train/{i}.spacy")

Pre-training displacy render:

Data debug:

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, spancat
410 training docs
99 evaluation docs
✔ No overlap between training and evaluation data
⚠ Low number of examples to train a new pipeline (410)

============================== Vocab & Vectors ==============================
ℹ 3850658 total word(s) in the data (47285 unique)
ℹ No word vectors present in the package

============================ Span Categorization ============================

Spans Key   Labels
---------   ------------------
ruler       {'Effective Date'}

ℹ Span characteristics for spans_key 'ruler'
ℹ SD = Span Distinctiveness, BD = Boundary Distinctiveness

Span Type        Length     SD     BD     N
--------------   ------   ----   ----   ---
Effective Date    11.79   2.79   2.35   122
--------------   ------   ----   ----   ---
Wgt. Average      11.79   2.79   2.35     -

ℹ Over 90% of spans have lengths of 1 -- 14 (min=11, max=15). The most
common span lengths are: 11 (59.02%), 12 (13.93%), 13 (13.11%), 14 (11.48%). If
you are using the n-gram suggester, note that omitting infrequent n-gram lengths
can greatly improve speed and memory usage.
✔ Spans are distinct from the rest of the corpus
✔ Boundary tokens are distinct from the rest of the corpus
✔ Good amount of examples for all labels
✔ Examples without ocurrences available for all labels

================================== Summary ==================================
✔ 7 checks passed
⚠ 1 warning

Training evaluation:

ℹ Saving to output directory: models
ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

=========================== Initializing pipeline ===========================
[2023-04-06 21:43:43,681] [INFO] Set up nlp object from config
[2023-04-06 21:43:43,690] [INFO] Pipeline: ['tok2vec', 'spancat']
[2023-04-06 21:43:43,692] [INFO] Created vocabulary
[2023-04-06 21:43:43,693] [INFO] Finished initializing nlp object
[2023-04-06 21:44:23,781] [INFO] Initialized pipeline components: ['tok2vec', 'spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'spancat']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_RULER_F  SPANS_RULER_P  SPANS_RULER_R  SCORE
---  ------  ------------  ------------  -------------  -------------  -------------  ------
  0       0         84.11        665.71           0.00           0.00           0.00    0.00
  0     100          2.08         83.03           0.00           0.00           0.00    0.00
  0     200          0.00         35.00           0.00           0.00           0.00    0.00
  0     300          0.00         40.00           0.00           0.00           0.00    0.00
  0     400          0.00         13.00           0.00           0.00           0.00    0.00
  1     500          0.00         40.00           0.00           0.00           0.00    0.00
  1     600          0.00         21.00           0.00           0.00           0.00    0.00
  1     700          0.00         31.00           0.00           0.00           0.00    0.00
  1     800          0.00         25.00           0.00           0.00           0.00    0.00
  2     900          0.00         23.00           0.00           0.00           0.00    0.00
  2    1000          0.00         22.00           0.00           0.00           0.00    0.00
  2    1100          0.00         39.00           0.00           0.00           0.00    0.00
  2    1200          0.00         28.00           0.00           0.00           0.00    0.00
  3    1300          0.00         40.00           0.00           0.00           0.00    0.00
  3    1400          0.00         26.00           0.00           0.00           0.00    0.00
  3    1500          0.00         39.00           0.00           0.00           0.00    0.00
  3    1600          0.00         23.00           0.00           0.00           0.00    0.00
✔ Saved pipeline to output directory
models\model-last

config.cfg:

[paths]
train = "train"
dev = "dev"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","spancat"]
batch_size = 10
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "ruler"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.suggester]
@misc = "spacy.ngram_suggester.v1"
sizes = [11, 12, 13, 14]

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"


[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 100
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = null
spans_sc_p = null
spans_sc_r = null
spans_ruler_f = 1.0
spans_ruler_p = 0.0
spans_ruler_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[Kau832]

Bump - tried a different series of ngrams in the suggester (just to make sure they aren't recognised as different lengths) but nothing as of yet to make this work.

[svlandeg]

Hi! It'll take a little more time than usual for us to have a look into your question as the spaCy team was collectively on a meetup this week, sorry! I'll try to get to this early next week.

[Kau832]

Sure! No worries and take your time.

[svlandeg]

Hey! So I looked at your code, output and config file in detail and nothing jumps out at me. It's certainly weird though.

Can you share the exact commands you ran for spacy debug data and spacy train? Also, can you run debug data on the dev data just to double check that the spans are in fact in there?

One other suggestion is to run the training and evaluate on the training dataset. So you'd have both paths.train and paths.dev refer to the same directory - you can try this with either both pointing to "train" or both to "dev". The point of that experiment is to see whether the algorithms can at least overfit on the data they see in training and replicate that.

[Kau832]

Hi, thanks for getting back to me!

I tried training with dev and train folders both having the same date but the issue persists. I train using the python -m spacy train config.cfg --output ./models command and debug using python -m spacy debug data config.cfg.

[adrianeboyd]

Hi, are you still having trouble with this?