Issue with debug model for ner_demo_update project

[lzmodel]

How to reproduce the behaviour

1. Executed spacy project run create-config for ner_demo_update project
2. Run spacy debug model configs/config.cfg ner -P0 -P1 --gpu-id 0
   ========================== STEP 0 - before training ==========================
   ...
   ℹ Layer 66: model ID 175: 'maxout'
   ℹ Layer 67: model ID 176: 'layernorm'

✘ Could not initialize the model: you'll have to provide a valid
'train_corpus' argument in the config file.

3. Executed spacy project run train - successfully
4. Edited config.cfg file by adding params like in command line --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --training.eval_frequency 10 --training.max_steps 500 --training.patience 50 and run spacy debug model - error remains the same.

I tried to edit this config.cfg with my data and by deleting some components from pipeline, but i can not know exactly if these error mine or from original config.

Your Environment

• spaCy version: 3.4.1
• Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.35
• Python version: 3.9.13
• Pipelines: en_core_web_sm (3.4.1), en_core_web_lg (3.4.1), en_core_web_trf (3.4.1)
• ** Environment Information:** conda env

[danieldk]

I can't reproduce this issue. From a fresh checkout of the project:

$ spacy project run create-config
$ spacy debug model configs/config.cfg ner -P0 -P1 --gpu-id 0
[...]
ℹ Layer 66: model ID 175: 'maxout'
ℹ Layer 67: model ID 176: 'layernorm'
✔ Succesfully ended analysis - model looks good.

What changes did you make to the configuration? What do you have in the paths section? Could you also check if debug data works for the original project configuration?

[lzmodel]

I just did the following:

1. deleted the ner_demo_update project completely
2. reinstalled the project
3. spacy project run create-config
4. run python -m spacy debug model configs/config.cfg ner -P0 -P1 --gpu-id 0
   Result is the same:

ℹ Layer 65: model ID 168: 'layernorm'
ℹ Layer 66: model ID 175: 'maxout'
ℹ Layer 67: model ID 176: 'layernorm'

✘ Could not initialize the model: you'll have to provide a valid
'train_corpus' argument in the config file.

[adrianeboyd]

I think you're just missing the overrides for the corpus paths in your command (something like --paths.train train.spacy --paths.dev dev.spacy).

[lzmodel]

@adrianeboyd How to check paths correctly?

1. I edited config.cfg by adding

train = "./corpus/train.spacy"
dev = "./corpus/dev.spacy"

2. Run spacy debug data ./configs/config.cfg - OK.
3. Run spacy debug model configs/config.cfg ner -P0 -P1 --gpu-id 0 - OK
4. I changed the directory to my data, that i used for training model from scratch many times (data exist exactly) - now i have another error:

...
File "spacy/pipeline/tagger.pyx", line 297, in spacy.pipeline.tagger.Tagger.initialize
  File "spacy/pipeline/tagger.pyx", line 322, in spacy.pipeline.tagger.Tagger.add_label
  File "spacy/pipeline/trainable_pipe.pyx", line 224, in spacy.pipeline.trainable_pipe.TrainablePipe._allow_extra_label
ValueError: [E922] Component 'tagger' has been initialized with an output dimension of 50 - cannot add any more labels.

The Path to the data is correct. To check it I tried to corrupt it and got another message "FileNotFoundError: [Errno 2] No such file or directory".

[rmitsch]

Moving this to discussions, as this seems to be a usage-related question rather than an issue with spaCy per se.

[rmitsch]

@lzmodel Please provide your complete config, it'll help us reproducing your issue.

[lzmodel]

It's my own data, labeled with Label Studio, exported and converted into spacy format. With another config I can debug it:

spacy debug data ./train/cfg_spc/cpu10v23-lg-tok.cfg

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, ner
758 training docs
187 evaluation docs
⚠ 1 training examples also in evaluation data
⚠ Low number of examples to train a new pipeline (758)

============================== Vocab & Vectors ==============================
ℹ 395493 total word(s) in the data (11877 unique)
ℹ 514157 vectors (514157 unique keys, 300 dimensions)
⚠ 4665 words in training data without vectors (1%)

========================== Named Entity Recognition ==========================
ℹ 10 label(s)
0 missing value(s) (tokens with '-' label)
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
✔ No entities crossing sentence boundaries

================================== Summary ==================================
✔ 6 checks passed
⚠ 3 warnings

[rmitsch]

With another config I can debug it: ...

You're saying "another config". Previously you mentioned that debugging worked for you:

3. Run spacy debug model configs/config.cfg ner -P0 -P1 --gpu-id 0 - OK ...

So is the debug command working for you with the config you posted above? Also, can you share the data you're using (at least a minimal version thereof)?

[lzmodel]

Sorry, @rmitsch Raphael, I missed and did not write what I changed yet, because I considered it not significant - the size of the model. In the file project.yml, I read that

for en_core_web_trf or other trf pipeline, use spacy-transformers>=1.1.1.

So, if you specify pipeline: "en_core_web_lg" or pipeline: "en_core_web_trf" in project.yml, run create config, than spacy debug model gives an error

✘ Could not initialize the model: you'll have to provide a valid
'train_corpus' argument in the config file

I checked it with data from the project ner_demo_update

train = "./corpus/train.spacy"
dev = "./corpus/dev.spacy"

Answers to your questions:

1. So is the debug command working for you with the config you posted above?

Config posted above contains updating of en_core_web_lg model. It does not work. In paragraph 3, I wrote about a completely different model config for training from scratch.

2. Also, can you share the data you're using (at least a minimal version thereof)?

Yes, I can. But as you can see above I found the shortest way to reproduce error in debugging this model

[rmitsch]

Let me rehash this, because I'm not sure I follow. So if you

1. clone the ner_demo_update project,
2. use the config you posted here, but with the original data (i. e. the train and dev paths are pointing to the original files, not your self-labeled ones),
3. specify en_core_web_lg or en_core_web_trf (but not e. g. en_core_web_sm) in your project.yml,
4. and finally run spacy debug model,
   then you run into

python -m spacy debug model configs/config.cfg ner -P0 -P1 --gpu-id 0

Is this correct?

[lzmodel]

Sorry Raphael @rmitsch, it's my fought with the corpus paths. Thank you for your time.