Jointly training rel_component with textcat_multilabel raises ValueError during backpropagation?

[shaunabanana]

Hi! I'm trying to train the demo rel_component pipeline jointly with textcat_multilabel, but can't seem to get it to work. Specifically, it raises a ValueError during the textcat back propagation. Any pointers to why this is happening or how to fix it would be greatly appreciated!

Here's how I can minimally reproduce this error:

1. Clone the rel_component project
2. Modify scripts/parse_data.py to add some cats data (I tried with both random generated data and custom labeled data):
   For example, after line 109, add the code below to generate some random cats data:

doc.cats = {
    "a": random(),
    "b": random(),
    "c": random(),
}

3. Add textcat_multilabel component in configs/rel_trf.cfg, as generated by the quick start widget:

[nlp]
pipeline = ["transformer", "textcat_multilabel", "relation_extractor"]
...

[components.textcat_multilabel]
factory = "textcat_multilabel"

[components.textcat_multilabel.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat_multilabel.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.textcat_multilabel.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.textcat_multilabel.model.linear_model]
@architectures = "spacy.TextCatBOW.v1"
exclusive_classes = false
ngram_size = 1
no_output_layer = false

4. Run python -m spacy project run train_gpu as usual, and get the following:

================================= train_gpu =================================
Running command: /home/user/.virtualenvs/spacy-experiments/bin/python -m spacy train configs/rel_trf.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py --gpu-id 0
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-03-27 07:35:56,871] [INFO] Set up nlp object from config
[2021-03-27 07:35:56,881] [INFO] Pipeline: ['transformer', 'textcat_multilabel', 'relation_extractor']
[2021-03-27 07:35:56,885] [INFO] Created vocabulary
[2021-03-27 07:35:56,885] [INFO] Finished initializing nlp object
[2021-03-27 07:36:13,472] [INFO] Initialized pipeline components: ['transformer', 'textcat_multilabel', 'relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'textcat_multilabel',
'relation_extractor']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS TEXTC...  LOSS RELAT...  CATS_SCORE  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  -------------  -------------  ----------  -----------  -----------  -----------  ------
⚠ Aborting and saving the final best model. Encountered exception:
TypeError("'NoneType' object is not iterable")
Traceback (most recent call last):
  File "/home/user/.anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/.anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/cli/train.py", line 59, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/training/loop.py", line 114, in train
    raise e
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/training/loop.py", line 98, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/training/loop.py", line 194, in train_while_improving
    nlp.update(
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/language.py", line 1107, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/spacy/pipeline/textcat.py", line 205, in update
    bp_scores(d_scores)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/thinc/layers/chain.py", line 60, in backprop
    dX = callback(dY)
  File "/home/user/.virtualenvs/spacy-experiments/lib/python3.8/site-packages/thinc/layers/concatenate.py", line 68, in backprop
    dX += bwd(dY)
TypeError: 'NoneType' object is not iterable

Environment

• Operating System: Ubuntu 18.04.3 LTS
• Python Version Used: 3.8.5
• spaCy Version Used: 3.0.5
• Environment Information:

[svlandeg]

Thanks for the report, I can reproduce this and am looking into this.

For now: it looks as if it does work correctly if you switch the order of the textcat_multilabel and the relation_extractor in the pipeline (which is already weird by itself)...

[shaunabanana]

Thank you for the tip! I've tried switching the order, and it is working fine now. Indeed it is quite weird...

[svlandeg]

FYI: this has been addressed by #8012 and will be available in the next release (3.1). Thanks for your patience!

[Lolologist]

@svlandeg I am using 3.1 and running textcat then ner doesn't work, but ner then textcat does.

[svlandeg]

Hi @Lolologist, could you open a new discussion with a reproducible code snippet & config, and the full error stack trace? Thanks!

[Lolologist]

@svlandeg never actually followed up on this, sorry. We're seeing the error in 3.2 as well; I can get a config file but don't know about the training data used to actually train the model (proprietary :/ )

I'll see what I can do!