Using a customized NER model in the RE model

[FelipeSBarros]

Hey there!

First of all thanks for this amazing framework!!

I am woking on a project where we are training two pipelines: NER (actually, we are improving an already existing model adding new tags) and a Relation Extraction model, on different spaCy projects.

The first (NER improvement) project were built quite straight-forward, using the spaCy project quick start.

Looking on spaCy tutorial about RE, specially in this video version, Sophie makes it clear that the example is not considering the NER process in the pipeline, but only, tok2vec and the relation_extraction being created, suggesting that in a "reallystic scenarious" we should add NER.

So, as I have a customized NER trained specifically for my project I would like to ensure using it in the context of the RE model.

Considering that the tok2vec is essential to the relation_extraction by its use in the create_instance_tensor, I kept it as a component and just added a a ner to the pipeline with the above block:

[components.ner]
source="my_NER_enhancement"

By the way, the my_NER_enhacement, is the already trained model on a different spacy project, intalled from tar.gz.
So, my questions are:

1. Does it make sense?
2. By doing so I am getting the following warning and error:
   ⚠️ Sourced component 'ner' may not work as expected: source vectors are not identical to current pipeline vectors. warnings.warn(Warnings.W113.format(name=sourced_component))
   🚫 ValueError: Shape mismatch for blis.gemm: (15, 96), (256, 64)
   See complete trackback:

 python -m spacy project run train_cpu ./spacy_tests/RE/ --force 
[2022-07-25 20:01:48,106] [INFO] Set up nlp object from config
[2022-07-25 20:01:48,122] [INFO] Pipeline: ['tok2vec', 'ner', 'relation_extractor']
[2022-07-25 20:01:48,122] [INFO] Resuming training for: ['ner']
[2022-07-25 20:01:48,130] [INFO] Created vocabulary
/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/initialize.py:138: UserWarning: [W113] Sourced component 'ner' may not work as expected: source vectors are not identical to current pipeline vectors.
  warnings.warn(Warnings.W113.format(name=sourced_component))
[2022-07-25 20:01:48,186] [INFO] Finished initializing nlp object
[2022-07-25 20:01:48,757] [INFO] Initialized pipeline components: ['tok2vec', 'relation_extractor']
Traceback (most recent call last):
  File "/home/felipe/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/felipe/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/language.py", line 1156, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])  # type: ignore
  File "spacy/pipeline/transition_parser.pyx", line 398, in spacy.pipeline.transition_parser.Parser.update
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/thinc/model.py", line 309, in begin_update
    return self._func(self, X, is_train=True)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/ml/tb_framework.py", line 33, in forward
    step_model = ParserStepModel(
  File "spacy/ml/parser_model.pyx", line 216, in spacy.ml.parser_model.ParserStepModel.__init__
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/thinc/layers/linear.py", line 35, in forward
    Y = model.ops.gemm(X, W, trans2=True)
  File "thinc/backends/numpy_ops.pyx", line 93, in thinc.backends.numpy_ops.NumpyOps.gemm
  File "blis/py.pyx", line 74, in blis.py.gemm
ValueError: Shape mismatch for blis.gemm: (15, 96), (256, 64)

Thanks in advance

[FelipeSBarros]

Perhaps there is something to do with freezing ner component?
Or even replacing listeners?:

To automatically replace a shared token-to-vector listener with an independent copy of the token-to-vector layer, you can use the replace_listeners setting of a sourced component, pointing to the listener layer(s) in the config.
(source: https://spacy.io/usage/training/#config-components)

[components.tok2vec]
# factory = "tok2vec"
source="pt_NER_enhancement"
replace_listeners = ["model.tok2vec"]

[polm]

It makes sense to have your own NER model when creating a real relex pipeline.

Sourcing the NER component is also fine. However, like the error says, the vectors need to match. The issue in this case is probably that you don't have the same word vectors in the NER pipeline and the pipeline you're sourcing them in. You can specify vectors in [initialize] using the name of your original pipeline.

If that doesn't fix it, it would help to have the full config for both of your pipelines.

[polm]

To be clear, your config was not related to this error. Regarding your question:

Looking to the double ner exemple, I understand that it is not necessary to add tok2vec in the pipeline, as the model has its own trainned tok2vec. Right?

If you are training a new component that requires a tok2vec, you will need to have a tok2vec. That tok2vec can be in the pipeline (this is the "listeners pattern") or it can be declared directly in the component that uses it. In your config you have a Tok2VecListener for the rel component, so you should have a tok2vec in the pipeline (and you do).

Considering it how could I point the relation_extractor 's mehotd create_instance that uses rok2vec to the already trained one from ner model? Does it make sense?

I understand what you mean. You want to re-use the tok2vec you trained for the NER model. You can do this, but you have to freeze it - if you don't freeze it, then the NER model will stop working. On the other hand, freezing it will likely make the RE model not work as well. Still, it could be worth a try.

Your config for the tok2vec is almost correct since you use source = instead of factory. There are a few things you can fix:

• remove replace_listeners - the tok2vec doesn't listen to itself.
• remove the [component.tok2vec] blocks further down - they are ignored with a source component
• add the tok2vec to the list of frozen components

[polm]

Let me also link in the sourcing docs for reference...

https://spacy.io/usage/training#config-custom

[FelipeSBarros]

Hey, @polm ! Thansk a lot!
Yesterday I did a kind of a debug and found this problem.

How did you get your copy of the project, and when did you make it? The fix was added November last year, and I would expect any copies made after that to use the updated version.

I was using this repo as reference. It was shared by my advisor. I have opened an issue so they know that it has changed.

There are a few things you can fix:
remove replace_listeners - the tok2vec doesn't listen to itself.

Ok, Done. My doubt now is: as I am sourcing a NER model, should I add component=tok2vec or it is not necessary?

remove the [component.tok2vec] blocks further down - they are ignored with a source component

Ok, But my components.relation_extractor.model.create_instance_tensor.tok2vec needs the tok2vec's parameter width. How could I point it out?

[components.relation_extractor.model.create_instance_tensor.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

add the tok2vec to the list of frozen components
Done. Should it be in annotating_components also, right? Mainly the NER model as we will need its annotations to the relation extractor mdoel. Or I am misunderstanding something?

The problem I am having now seems to be with de vectors parameters, which seem not being working:

⚠ Aborting and saving the final best model. Encountered exception:
ValueError('Shape mismatch for blis.gemm: (15, 96), (256, 64)')

I will talk to my advisor to see if we can solve this problem together.

Once again, thanks for your help!

[polm]

Hey, I notice you forked my copy of the projects repo - you should use the official projects repo as a source, not my fork.

Ok, Done. My doubt now is: as I am sourcing a NER model, should I add component=tok2vec or it is not necessary?

The component argument is only required if the name of the component you are sourcing is different than it was in the original pipeline. In your case there is no need for it.

Ok, But my components.relation_extractor.model.create_instance_tensor.tok2vec needs the tok2vec's parameter width. How could I point it out?

In this case I would just hard code it.

Done. Should it be in annotating_components also, right? Mainly the NER model as we will need its annotations to the relation extractor mdoel. Or I am misunderstanding something?

Yes, it should also be in annotating_components. annotating_components should be used whenever the output of the component is required during training.

The problem I am having now seems to be with de vectors parameters, which seem not being working:

Not sure what the cause of that would be, it looks like the kind of mismatch that would be caused by changing word vectors or the source tok2vec... It would help if you shared the entire stack trace.

[FelipeSBarros]

Hi, @polm . Thansk again for your attention.

Not sure what the cause of that would be, it looks like the kind of mismatch that would be caused by changing word vectors or the source tok2vec... It would help if you shared the entire stack trace.
Here is the entire traceback:

python -m spacy project run train_cpu ./spacy_tests/RE/ --force

================================= train_cpu =================================
Running command: /media/felipe/DATA/Repos/agenda_transparente/.venv/bin/python -m spacy train configs/rel_tok2vec.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy --nlp.lang pt -c ./scripts/custom_functions.py
ℹ Saving to output directory: training
ℹ Using CPU

=========================== Initializing pipeline ===========================
[2022-08-01 17:41:41,890] [INFO] Set up nlp object from config
[2022-08-01 17:41:41,899] [INFO] Pipeline: ['tok2vec', 'ner', 'relation_extractor']
[2022-08-01 17:41:41,904] [INFO] Created vocabulary
[2022-08-01 17:41:51,722] [INFO] Added vectors: pt_core_news_lg_custom
[2022-08-01 17:41:53,214] [INFO] Finished initializing nlp object
[2022-08-01 17:41:53,858] [INFO] Initialized pipeline components: ['relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner', 'relation_extractor']
ℹ Frozen components: ['tok2vec', 'ner']
ℹ Set annotations on update for: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS RELAT...  ENTS_F  ENTS_P  ENTS_R  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           0.92   99.02   98.05  100.00         1.62        57.25         3.15    0.51
⚠ Aborting and saving the final best model. Encountered exception:
ValueError('operands could not be broadcast together with shapes (1156,2)
(1092,2) ')
Traceback (most recent call last):
  File "/home/felipe/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/felipe/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/language.py", line 1156, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])  # type: ignore
  File "/media/felipe/DATA/Repos/agenda_transparente/spacy_tests/RE/scripts/rel_pipe.py", line 133, in update
    loss, gradient = self.get_loss(examples, predictions)
  File "/media/felipe/DATA/Repos/agenda_transparente/spacy_tests/RE/scripts/rel_pipe.py", line 146, in get_loss
    gradient = scores - truths
ValueError: operands could not be broadcast together with shapes (1156,2) (1092,2) 

I will be trying a few variations, but without much enthusiasm. I will let you know I a succeed.

[svlandeg]

Sophie makes it clear that the example is not considering the NER process in the pipeline, but only, tok2vec and the relation_extraction being created, suggesting that in a "reallystic scenarious" we should add NER.

May I ask what "reallystic scenarious" refers to? Is there a typo in the video's subtitles?

[FelipeSBarros]

Sorry @svlandeg ! The typos are my fault:

Thanks for the tutorials!

[svlandeg]

Ha ok, no worries, I was just curious 🙈

[FelipeSBarros]

@polm I am wrinting as a new topic because I have tried a new approach: merging all process in one projec (tokenizer, NER and RE).
It seems that the training process went fine, although the warning "ℹ Could not determine any instances in doc." appeared very often.
But once I tries to build the bet-model as package, I got the following error:

ValueError: [E002] Can't find factory for 'relation_extractor' for language Portuguese (pt). This usually happens when spaCy calls `nlp.create_pipe` with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator `@Language.component` (for function components) or `@Language.factory` (for class components).

Here is the entire trace back:

 python -m spacy project run package ./spacy_tests/NER_enhancement/ --force

================================== package ==================================
Running command: /media/felipe/DATA/Repos/agenda_transparente/.venv/bin/python -m spacy package training/model-best packages --name NER_enhancement --version 0.0.0 --code ../pt_tokenizer/pt_tokenizer.py --force
ℹ Building package artifacts: sdist
✔ Including 1 Python module(s) with custom code
Traceback (most recent call last):
  File "/home/felipe/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/felipe/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/package.py", line 49, in package_cli
    package(
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/package.py", line 105, in package
    meta = get_meta(input_dir, meta)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/cli/package.py", line 300, in get_meta
    nlp = util.load_model_from_path(Path(model_path))
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/util.py", line 488, in load_model_from_path
    nlp = load_model_from_config(config, vocab=vocab, disable=disable, exclude=exclude)
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/util.py", line 525, in load_model_from_config
    nlp = lang_cls.from_config(
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/language.py", line 1782, in from_config
    nlp.add_pipe(
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/language.py", line 792, in add_pipe
    pipe_component = self.create_pipe(
  File "/media/felipe/DATA/Repos/agenda_transparente/.venv/lib/python3.9/site-packages/spacy/language.py", line 655, in create_pipe
    raise ValueError(err)
ValueError: [E002] Can't find factory for 'relation_extractor' for language Portuguese (pt). This usually happens when spaCy calls `nlp.create_pipe` with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator `@Language.component` (for function components) or `@Language.factory` (for class components).

Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, doc_cleaner, parser, beam_parser, entity_linker, ner, beam_ner, entity_ruler, lemmatizer, tagger, morphologizer, senter, sentencizer, textcat, spancat, textcat_multilabel

Here is the package's project.yml block:

  - name: package
    help: "Package the trained model as a pip package"
    script:
      - "python -m spacy package training/model-best packages --name ${vars.name} --version ${vars.version} --code ../pt_tokenizer/pt_tokenizer.py --force"
    deps:
      - "training/model-best"
    outputs_no_cache:
      - "packages/${vars.lang}_${vars.name}-${vars.version}/dist/${vars.lang}_${vars.name}-${vars.version}.tar.gz"

[FelipeSBarros]

Hey, @polm . Just to let you know. We could solve the problem and train our pipelines in the same project. If I can I share later the problem and solutions.
Thanks for your help!

[polm]

Glad you figured it out! I'd love to hear the details.