Setting listener from sourced transformer

[hnalla]

Hello,

I'm trying to train a text categorization model. I would like to train it using the transformer from en_core_web_trf since this pipeline also contains other components that I find useful for my current project.

I'm currently using this config, but I'm not sure this is the right way to implement it. (referenced from here: Originally posted by @polm in #11187 (comment))

Is this the right way to implement this? My intuition is telling me that I should somehow reference the "sourced" component instead of the the generic spacy-transformers.TransformerListener.v1 listener, but I'm a newbie to spacy and I could be wrong!

Any help is appreciated!

The components section in config.cfg looks like this:

[nlp]
lang = "en"
pipeline = ["transformer","textcat_multilabel"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}


[components]

[components.textcat_multilabel]
factory = "textcat_multilabel"
threshold = 0.5

[components.textcat_multilabel.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat_multilabel.model.linear_model]
@architectures = "spacy.TextCatBOW.v2"
exclusive_classes = false
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat_multilabel.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
source = "en_core_web_trf"
component = "transformer"

[polm]

If you want to use a custom component in addition to the pretrained pipelines, I would recommend you train your component in isolation, and then source the components you want from the pretrained pipeline afterwards, whether in code or using spacy assemble.

The one downside of this approach is you'll need two copies of the Transformer (/tok2vec), which will take up more disk and memory. But the alternative is training your model with a frozen Transformer, which will limit the accuracy you can achieve. (Also freezing Transformers isn't straightforward at the moment - you can't use frozen_components, you have to set grad_factor = 0.)

[hnalla]

Thanks @polm for the quick response!
I've decided to train in isolation like you suggested. But, I have a trivial problem now.
I want to rename the transformer (to transformer_textcat) that is being trained in isolation since I have to use both the en_core_web_trf's transformer and this custom transformer.

How would I do that?
I'm getting this error when I use the config below :

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/cli/train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/training/loop.py", line 112, in train
    log_step(info if is_best_checkpoint is not None else None)
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/training/loggers.py", line 61, in log_step
    losses = [
  File "/home/custom_user/projects/dev/hnalla/client/src/python/alerts/.venv/lib/python3.9/site-packages/spacy/training/loggers.py", line 62, in <listcomp>
    "{0:.2f}".format(float(info["losses"][pipe_name]))
KeyError: 'transformer_textcat'

[nlp]
lang = "en"
pipeline = ["transformer_textcat","textcat_multilabel"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat_multilabel]
factory = "textcat_multilabel"
threshold = 0.5

[components.textcat_multilabel.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat_multilabel.model.linear_model]
@architectures = "spacy.TextCatBOW.v2"
exclusive_classes = false
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat_multilabel.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer_textcat]
source = "en_core_web_trf"
component = "transformer"

[hnalla]

I was confusing myself.

transformer config needs to look like this:

[components.transformer]
source = "en_core_web_trf"
component = "transformer"

and the code to python

def load_pipeline(source_pipeline_path: str) -> Language:
    pipeline = spacy.load("en_core_web_trf")
    # add other components
    source_pipeline = spacy.load(source_pipeline_path)
    pipeline.add_pipe(
        "transformer",
        name="transformer_textcat",
        before="custom_component_1",
        source=source_pipeline,
    )
    pipeline.add_pipe(
        "textcat_multilabel", after="transformer_textcat", source=source_pipeline
    )
    return pipeline

[polm]

Sounds like you got it working? Glad you figured it out if so, but if you have any more issues just let us know!