Errors in spaCy train (spacy-nightly)

[yw2903]

I encounter the following error when running command:

python -m spacy train /home/weiy/core/NER_v2/config.cfg --paths.train /home/weiy/prodigy_to_spacy/NER_clause1123.spacy --output /home/weiy/NER_models/NER_clause1123

My config.cfg file is like this:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v1"

So what's a corpus and what does it do in training spacy model?

Your Environment

• Python Version Used: 3.7
• spaCy Version Used: spacy3.0.0rc2
• Environment Information: Azure Rstudio

[svlandeg]

A corpus is a dataset containing gold-standard annotations to train (and evaluate) a ML model. Typically, a Corpus registered function is used, as seems to be the case in your config (as I can deduce from the error message).

It looks like your config probably contains a section like this:

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
...

This means that you'll need to define the variable dev from the section [paths]. You can keep this at null and override on the CLI, exactly as you've done for the variable train. So you'll need something like this:

python -m spacy train config.cfg --paths.train NER_dataset_train.spacy --paths.dev NER_dataset_dev.spacy --output output_folder

If you don't have a separate dev set, you can reuse the training set. Just be aware that it'll be more difficult to spot overfitting. Overfitting generally occurs when the ML model learns to replicate the training dataset "too well", and starts to lose its generalization capability. You can typically see this during training when the training loss keeps improving (i.e. decreasing) while the dev performance starts getting worse. You won't be able to see that if you don't have a separate dev set. But technically, the training will still run properly.

[yw2903]

Thanks. I add --path.dev to the command and the above issue is resolved. However, I encounter another error:

File "/home/weiy/.venvSpacy/lib/python3.7/site-packages/spacy/pipeline/tok2vec.py", line 225, in initialize
    assert doc_sample, Errors.E923.format(name=self.name)
AssertionError: [E923] It looks like there is no proper sample data to initialize the Model of component 'tok2vec'. This is likely a bug in spaCy, so feel free to open an issue: https://github.com/explosion/spaCy/issues

[svlandeg]

Can you provide the full config file?

[yw2903]

This is my config file for CPU preferred:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v1"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 2000
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null

[initialize.components]

[initialize.tokenizer]

This is my config file for GPU preferred and the error is as following:

train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "roberta-base"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 500
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null

[initialize.components]

[initialize.tokenizer]

[yw2903]

Also, I would like to train on en_core_web_lg model, instead of a blank en model, where can I add that to the config file? Thanks.

[svlandeg]

Hi! As I told you on the Issue tracker, we converted this issue to a thread on the discussion board, as this is more of a place where the community can come together, help eachother, discuss best practices, and so on :-)

Anyway I selected my answer to your first question as the correct answer as that solved your original problem. Going forward, with this discussion board and being able to select "correct answers", it might make sense to open new threads for new issues, as that also helps others to understand quickly what the current problem is and whether they can help.

With respect to your second question: I copy-pasted your entire config file and ran it with the command

python -m spacy train config_6472.cfg --paths.train=".\train" --paths.dev=".\dev"

and this works fine on my end:

ℹ Using CPU

=========================== Initializing pipeline ===========================
Set up nlp object from config
Pipeline: ['tok2vec', 'ner']
Created vocabulary
Finished initializing nlp object
Initialized pipeline components: ['tok2vec', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00     36.77    0.00    0.00    0.00    0.00
  0     200       6989.62   1381.27   17.49   19.05   16.16    0.17

I assume the problem is with your data. Are you sure there is valid data in your .spacy files? Can you try reading them in with the docbin class and see what's there?

    nlp = English()
    doc_bin = DocBin().from_disk(your_train_loc)
    docs = list(doc_bin.get_docs(nlp.vocab))
    entities = 0
    for doc in docs:
        entities += len(doc.ents)
    print(f"TRAIN docs: {len(docs)} with {entities} entities")

    doc_bin = DocBin().from_disk(your_dev_loc)
    docs = list(doc_bin.get_docs(nlp.vocab))
    entities = 0
    for doc in docs:
        entities += len(doc.ents)
    print(f"DEV docs: {len(docs)} with {entities} entities")

[mandar-avhad]

AssertionError: [E923] It looks like there is no proper sample data to initialize the Model of component 'tok2vec'. To check your input data paths and annotation, run: python -m spacy debug data config.cfg

I'm getting this error while running the train command.

And when I run the debug command, I'm getting an error as follows :
ValueError: [E913] Corpus path can't be None. Maybe you forgot to define it in your config.cfg or override it on the CLI?

Please help.

[blizaga]

same case

[svlandeg]

These errors point to the fact that the training data wasn't found during training or debugging. Did you provide a path to the training data in the config file or on the command line, as described here?