Keep getting a `ValueError: Out shape is mismatched` #11208

thomazmoon · 2022-07-26T04:30:10Z

thomazmoon
Jul 26, 2022

I'm trying to train a new ner model using google colab's gpu. I'm not really sure what happened because it was working ~2 weeks ago, but now whenever I try to train, I keep getting the ValueError: Out shape is mismatched.

Steps i'm taking

1. I first get my labeled data into the doc format and save the train and dev files in a new folder.

How I'm adding the text and annotations

  # load a new spacy model
  nlp = spacy.blank("en")
  nlp.add_pipe("ner")

  # incase we have bad labels i can see which ones and go back and fix it later
  bad_labels = []

  # create a DocBin object
  db_train = DocBin()
  db_test = DocBin()

  # adding in train_set to doc
  train_skip = 0
  validation_skip = 0
  for text_train, annot in X_train:
    doc = nlp.make_doc(text_train) # create doc object from text
    ents = []
    for start, end, label in annot["entities"]: # add character indexes
      span = doc.char_span(start, end, label=label) #, alignment_mode="contract"
      if span is None:
        train_skip +=1
        pass
      else:
        ents.append(span)
    try:
      # label the text with the ents
      doc.ents = ents 
      db_train.add(doc)
    except ValueError:                  # when there are overlapping labels
      print(f"Sentence skipped")
      train_skip += 1
      bad_labels.append( (text_train, annot) )
      pass
      

  # adding in validation to doc
  for text_val, annot in X_val:
    doc = nlp.make_doc(text_val) # create doc object from text
    ents = []
    for start, end, label in annot["entities"]: # add character indexes
        span = doc.char_span(start, end, label=label) #, alignment_mode="contract"
        if span is None:
          validation_skip +=1
          pass
        else:
          ents.append(span)
    try:
      # label the text with the ents
      doc.ents = ents 
      db_test.add(doc)
    except ValueError:                  # when there are overlapping labels
      print(f"Sentence skipped")
      validation_skip += 1
      bad_labels.append( (text_val, annot) )
      pass
  db_train.to_disk(file_path + "/train.spacy") # save the docbin object
  db_test.to_disk(file_path + "/dev.spacy") # save the docbin object

2. Then I make a new config which seems to give a good output

os.chdir(f"/content/drive/Shareddrives/Data/NER/sent_exploration/spacy_models/{folder_name}/")

!python -m spacy init config config.cfg --lang en --pipeline "transformer, ner" --gpu --optimize accuracy --force

Output from initializing a config

ℹ Generated config template specific for your use case
- Language: en
- Pipeline: ner
- Optimize for: accuracy
- Hardware: GPU
- Transformer: roberta-base
✔ Auto-filled config with all values
✔ Saved config
config.cfg
You can now add your data and train your pipeline:
python -m spacy train config.cfg --paths.train ./train.spacy --paths.dev ./dev.spacy

3. Finally I call the training CLI

!python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./dev.spacy --gpu-id 0

But I keep getting this error

✔ Created output directory: output
ℹ Saving to output directory: output
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-07-26 03:30:19,411] [INFO] Set up nlp object from config
[2022-07-26 03:30:19,430] [INFO] Pipeline: ['transformer', 'ner']
[2022-07-26 03:30:19,436] [INFO] Created vocabulary
[2022-07-26 03:30:19,441] [INFO] Finished initializing nlp object
Downloading: 100% 481/481 [00:00<00:00, 452kB/s]
Downloading: 100% 878k/878k [00:00<00:00, 8.01MB/s]
Downloading: 100% 446k/446k [00:00<00:00, 6.08MB/s]
Downloading: 100% 1.29M/1.29M [00:00<00:00, 10.9MB/s]
Downloading: 100% 478M/478M [00:08<00:00, 58.2MB/s]
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 72, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/initialize.py", line 84, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1317, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "spacy/pipeline/transition_parser.pyx", line 575, in spacy.pipeline.transition_parser.Parser.initialize
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/tb_framework.py", line 47, in init
    lower.initialize()
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/_precomputable_affine.py", line 150, in init
    acts1 = predict(ids, tokvecs)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/_precomputable_affine.py", line 131, in predict
    hiddens = model.predict(tokvecs[:-1])  # (nW, f, o, p)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/_precomputable_affine.py", line 29, in forward
    Yf[0] = model.get_param("pad")
  File "cupy/_core/core.pyx", line 1409, in cupy._core.core.ndarray.__setitem__
  File "cupy/_core/_routines_indexing.pyx", line 54, in cupy._core._routines_indexing._ndarray_setitem
  File "cupy/_core/_routines_indexing.pyx", line 959, in cupy._core._routines_indexing._scatter_op
  File "cupy/_core/_kernel.pyx", line 1161, in cupy._core._kernel.ufunc.__call__
  File "cupy/_core/_kernel.pyx", line 594, in cupy._core._kernel._get_out_args
ValueError: Out shape is mismatched

debug data

I've tried running the debug data command via
!python -m spacy debug data config.cfg --paths.train ./train.spacy --paths.dev ./dev.spacy but I don't see anything that would throw the Out shape is mismatched error

`debug output`

* **I will be annotating and adding more in later, but I wanted to make sure things will work so far.**

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable
`⚠ Low number of examples to train a new pipeline (108)`.    
  
=============================== Training stats ===============================
Language: en
Training pipeline: transformer, ner
108 training docs
20 evaluation docs
✔ No overlap between training and evaluation data
⚠ Low number of examples to train a new pipeline (108)   

============================== Vocab & Vectors ==============================
ℹ 846 total word(s) in the data (539 unique)
ℹ No word vectors present in the package   

========================== Named Entity Recognition ==========================
ℹ 32 label(s)
0 missing value(s) (tokens with '-' label)
⚠ Low number of examples for label 'STREET' (6)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
✔ No entities crossing sentence boundaries  

================================== Summary ==================================
✔ 6 checks passed
⚠ 32 warnings
 * The warnings are "⚠ Low number of examples for label"

What i've tried so far

I've already checked that cupy-cuda are the correct version for the CUDA colab is running.
made sure the labels are not overlapping and don't go past the length of the text
tested a previously working annotated file that is throwing the same issues now
downloaded the config directly from spacy and ran !python -m spacy init fill-config base_config.cfg config.cfg instead of step 2

My `spacy.info()`

location: '/usr/local/lib/python3.7/dist-packages/spacy',
pipelines: {'en_core_web_sm': '3.4.0'},
platform: 'Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic',
python_version: '3.7.13',
spacy_version: '3.4.0'

Any sort of help would be greatly appreciated.

Answered by svlandeg

Jul 26, 2022

Thanks again for the report! 3.4.1 is now available which should hopefully fix things. Let us know if you do still run into problems though!

View full answer

thomazmoon · 2022-07-26T06:02:40Z

thomazmoon
Jul 26, 2022
Author

⭐️ Sort of a fix ⭐️

Downgrading to spacy 3.3 seemed to do the trick and get everything working again.

You can just run !pip install -U spacy==3.3

packages that got downgraded

spacy: 3.4 --> 3.3
typing-extensions: 4.1.1 --> 3.10.0.2
pydantic: 1.9.1 --> 1.8.2
thinc: 8.1.0 --> 8.0.17

I'm assuming there either:

some sort of dependency error with the new update and colab?
I did look at What's new in spaCy 3.4 when I saw colab was 3.4 but I didn't really see anything that would be causing the error since I was already calling init config with version 3.4.
Maybe Doc.has_vector now matching changed something in the way I was organizing things?

Either way, I just hope this post might help anyone else who might have been stuck in the same position I was. :)

2 replies

adrianeboyd Jul 26, 2022

As note, we are working on a fix for this in #11194, which should be in the next patch release (v3.4.1).

thomazmoon Jul 26, 2022
Author

Thank you for the update!
Very quick fix 😁

svlandeg · 2022-07-26T13:15:26Z

svlandeg
Jul 26, 2022

Thanks again for the report! 3.4.1 is now available which should hopefully fix things. Let us know if you do still run into problems though!

1 reply

thomazmoon Jul 26, 2022
Author

Awesome! Seems to work now!
Thank you!

NishantJalasutram · 2022-07-26T16:48:51Z

NishantJalasutram
Jul 26, 2022

I am facing the same issue. I was able to train custom NER models till last night. From today, I am unable to. I am doing this Google Colab Pro+.

Please help.

2 replies

thomazmoon Jul 26, 2022
Author

@NishantJalasutram
Check your spaCy version. I usually just run !pip list and scroll down to spacy.
You can also use !pip show spacy if you prefer to just see the spacy package.

My Colab still has Version: 3.4.0 by default whenever I connect to a new runtime or my notebook is disconnected.
I'm assuming yours does too so you'll probably have to upgrade it if your version is also 3.4 or 3.4.0:

You can do it quickly by

Running !pip install -U spacy==3.4.1 in a cell
Restart your time and you should be good to go

if it still isn't working

You might have to update your config with the simple CLI spacy gives here.
python -m spacy init fill-config config-v3.3.cfg config-v3.4.cfg
or you can just downgrade your spacy to the version 3.3 via: !pip install -U spacy==3.3

NishantJalasutram Jul 26, 2022

Thanks.. It is working now.. Yayy!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Keep getting a `ValueError: Out shape is mismatched` #11208

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Keep getting a ValueError: Out shape is mismatched #11208

Uh oh!

thomazmoon Jul 26, 2022

Steps i'm taking

1. I first get my labeled data into the doc format and save the train and dev files in a new folder.

2. Then I make a new config which seems to give a good output

3. Finally I call the training CLI

debug data

What i've tried so far

My spacy.info()

Replies: 3 comments · 5 replies

Uh oh!

thomazmoon Jul 26, 2022 Author

⭐️ Sort of a fix ⭐️

Uh oh!

adrianeboyd Jul 26, 2022

Uh oh!

thomazmoon Jul 26, 2022 Author

Uh oh!

svlandeg Jul 26, 2022

Uh oh!

thomazmoon Jul 26, 2022 Author

Uh oh!

NishantJalasutram Jul 26, 2022

Uh oh!

Uh oh!

thomazmoon Jul 26, 2022 Author

Uh oh!

NishantJalasutram Jul 26, 2022

Keep getting a `ValueError: Out shape is mismatched` #11208

thomazmoon
Jul 26, 2022

My `spacy.info()`

Replies: 3 comments 5 replies

thomazmoon
Jul 26, 2022
Author

thomazmoon Jul 26, 2022
Author

svlandeg
Jul 26, 2022

thomazmoon Jul 26, 2022
Author

NishantJalasutram
Jul 26, 2022

thomazmoon Jul 26, 2022
Author