Issues resuming training on a stopped pipeline

[edgar-mh]

Hey team,

I'm trying to train a custom NER transformer model on Google Colab, but due to it's uptime and resources limitations, it typically fails around epoch 3~4. I've come across this discussion and tried to implement it, but I'm clearly missing something, as I'm getting the following stack trace:

ℹ Saving to output directory: /content/drive/My
Drive/Models/NER/ner_gpu_mh_resume
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-09-14 14:10:12,193] [INFO] Set up nlp object from config
INFO:spacy:Set up nlp object from config
[2022-09-14 14:10:12,202] [DEBUG] Loading corpus from path: /content/drive/My Drive/Models/NER/data/dev
DEBUG:spacy:Loading corpus from path: /content/drive/My Drive/Models/NER/data/dev
[2022-09-14 14:10:12,203] [DEBUG] Loading corpus from path: /content/drive/My Drive/Models/NER/data/train
DEBUG:spacy:Loading corpus from path: /content/drive/My Drive/Models/NER/data/train
[2022-09-14 14:10:12,203] [INFO] Pipeline: ['transformer', 'ner']
INFO:spacy:Pipeline: ['transformer', 'ner']
[2022-09-14 14:10:12,203] [INFO] Resuming training for: ['ner', 'transformer']
INFO:spacy:Resuming training for: ['ner', 'transformer']
[2022-09-14 14:10:12,210] [DEBUG] Loading lookups from spacy-lookups-data: ['lexeme_norm']
DEBUG:spacy:Loading lookups from spacy-lookups-data: ['lexeme_norm']
[2022-09-14 14:10:12,525] [INFO] Added vocab lookups: lexeme_norm
INFO:spacy:Added vocab lookups: lexeme_norm
[2022-09-14 14:10:12,526] [INFO] Created vocabulary
INFO:spacy:Created vocabulary
[2022-09-14 14:10:12,527] [INFO] Finished initializing nlp object
INFO:spacy:Finished initializing nlp object
[2022-09-14 14:10:12,527] [INFO] Initialized pipeline components: []
INFO:spacy:Initialized pipeline components: []
✔ Initialized pipeline

============================= Training pipeline =============================
[2022-09-14 14:10:12,537] [DEBUG] Loading corpus from path: /content/drive/My Drive/Models/NER/data/dev
DEBUG:spacy:Loading corpus from path: /content/drive/My Drive/Models/NER/data/dev
[2022-09-14 14:10:12,538] [DEBUG] Loading corpus from path: /content/drive/My Drive/Models/NER/data/train
DEBUG:spacy:Loading corpus from path: /content/drive/My Drive/Models/NER/data/train
[2022-09-14 14:10:12,566] [DEBUG] Removed existing output directory: /content/drive/My Drive/Models/NER/ner_gpu_mh_resume/model-last
DEBUG:spacy:Removed existing output directory: /content/drive/My Drive/Models/NER/ner_gpu_mh_resume/model-last
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
⚠ Aborting and saving the final best model. Encountered exception:
ValueError()
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 209, in train_while_improving
    annotates=annotating_components,
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1164, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])  # type: ignore
  File "spacy/pipeline/transition_parser.pyx", line 405, in spacy.pipeline.transition_parser.Parser.update
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 309, in begin_update
    return self._func(self, X, is_train=True)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/tb_framework.py", line 38, in forward
    has_upper=model.attrs["has_upper"],
  File "spacy/ml/parser_model.pyx", line 213, in spacy.ml.parser_model.ParserStepModel.__init__
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/usr/local/lib/python3.7/dist-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/usr/local/lib/python3.7/dist-packages/thinc/layers/chain.py", line 55, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/usr/local/lib/python3.7/dist-packages/spacy_transformers/layers/listener.py", line 61, in forward
    model.verify_inputs(docs)
  File "/usr/local/lib/python3.7/dist-packages/spacy_transformers/layers/listener.py", line 50, in verify_inputs
    raise ValueError
ValueError

As per the discussion mentioned, I've created a new config.cfg based on the initial one, where I'm sourcing the ner and transformer components from last-model:

[paths]
train = "/content/drive/My Drive/Models/NER/data/train"
dev = "/content/drive/My Drive/Models/NER/data/dev"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
source = "/content/drive/My Drive/Models/NER/ner_gpu_mh/model-last/"
component = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.transformer]
source = "/content/drive/My Drive/Models/NER/ner_gpu_mh/model-last/"
component = "ner"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
annotating_components = []
before_to_disk = null
frozen_components = []

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
before_init = null
after_init = null

[initialize.components]

[initialize.lookups]
@misc = "spacy.LookupsDataLoader.v1"
lang = ${nlp.lang}
tables = ["lexeme_norm"]

[initialize.tokenizer]

I've also ran the spacy debug data config.cfg -V command to find out if I had any obvious errors, but couldn't find any:

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: transformer, ner
Components from other pipelines: ner, transformer
87960 training docs
38973 evaluation docs
⚠ 10754 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 2147845 total word(s) in the data (16732 unique)
10 most common words: ',' (113021), '.' (85422), 'and' (80227), 'to' (56306),
'a' (47167), 'the' (42987), 'of' (34109), 'with' (29833), 'for' (27238), '-'
(26573)
ℹ No word vectors present in the package

========================== Named Entity Recognition ==========================
ℹ 21 label(s)
0 missing value(s) (tokens with '-' label)
Labels in train data: 'ORG' (64136), 'SKILL' (33335), 'PERSON' (25455), 'DATE'
(18678), 'INDUSTRY' (18562), 'CARDINAL' (12470), 'ROLE' (8635), 'PERCENT'
(7827), 'MONEY' (6633), 'GPE' (3384), 'TIME' (2776), 'PRODUCT' (2386), 'ORDINAL'
(1622), 'NORP' (596), 'WORK_OF_ART' (593), 'LOC' (538), 'EVENT' (208),
'QUANTITY' (119), 'FAC' (112), 'LANGUAGE' (51), 'LAW' (29)
⚠ Low number of examples for label 'LAW' (29)
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
✔ No entities crossing sentence boundaries
To train a new entity type, your data should include at least 50 instances of
the new label

================================== Summary ==================================
✔ 5 checks passed
⚠ 2 warnings

Does anyone have a clue to what is going on?

Environment
spaCy version: 3.4.1
Location: /usr/local/lib/python3.7/dist-packages/spacy
Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.13

Disclaimer

• I'm new to spacy and NLP in general, so I might have done some newbie mistake somewhere.
• Additionally, my train data annotations were basically built by adding an EntityRuler on top of a RoBERTa model and not manually (hence the number of tokens with labels different from SKILL, ROLE and INDUSTRY). I ensured I wasn't passing noise to the EntityRuler patterns, though.

[polm]

It looks like you're loading an NER component as a Transformer?

[components.transformer]
source = "/content/drive/My Drive/Models/NER/ner_gpu_mh/model-last/"
component = "ner" # Should be "transformer"

Besides that, note that you shouldn't provide extra arguments when sourcing components - they'll be ignored in favor of the settings on the sourced component.

[edgar-mh]

Oh, what a newbie mistake! Thanks for pointing that out, I was at my wit's end.
I'll give it a try, comment/remove all extra arguments for the sourced components and I'll let you know the result.

Thanks polm!

[edgar-mh]

Pipeline training started successfully.
Thank you so much for your help!

[polm]

No worries, there's a lot of settings and it's easy to make a small mistake like that.