Textcat with transformer listener

[AleksandrTulenkov]

Hi! I'm trying to create fully independent transformer textcat component using en_trf transformer to fine-tune and replace listener, somebody can help/share with config how I can handlle that?
My aprouch was:

[paths]
train = "corpus/train.spacy"
dev = "corpus/dev.spacy"
raw = null
init_tok2vec = null
vectors = null

[system]
seed = 0
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["transformer","textcat"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[components]

[components.textcat]
factory = "textcat_multilabel"
threshold = 0.5


[components.textcat.model]
@architectures = "spacy.TextCatCNN.v1"
exclusive_classes = false
nO = null

[components.textcat.model.tok2vec]
replace_listeners = ["model.transformer"]


[components.transformer]
source = "en_core_web_trf"

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
gold_preproc = ${corpora.train.gold_preproc}
max_length = ${corpora.train.max_length}
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths:train}
gold_preproc = false
max_length = 500
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 10
max_steps = 20000
eval_frequency = 200
frozen_components = ["transformer"]
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
eps = 0.00000001
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null
cats_score = 1
cats_macro_f = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Got:
✘ Config validation error
textcat_multilabel.model -> tok2vec instance of Model expected

{'@architectures': 'spacy.TextCatCNN.v1', 'exclusive_classes': False, 'nO': None, 'tok2vec': {'replace_listeners': ['model.transformer']}}

[polm]

When using a listener you should use a Listener architecture in the config. For an example, in the quickstart widget you can choose textcat/GPU/accuracy and you'll see a section like this:

[components.textcat_multilabel.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

The replace_listeners argument in the config is designed to be used when sourcing components, not when initializing a listener.

[AleksandrTulenkov]

Well, I wanted to take source transformer (en_trf):
[components.transformer]
source = "en_core_web_trf"
copy embedings from it, and fine tune during textcat training.
And it works as mentioned in quickstart, but I also wanted to make it independent (I guess with replace listeners feature), like mentioned here:
https://spacy.io/usage/embeddings-transformers/#embedding-layers-config

In general my pipeline should looks like:

When I'm trying to do like this:

[paths]
train = "corpus/train.spacy"
dev = "corpus/dev.spacy"
raw = null
init_tok2vec = null
vectors = null

[system]
seed = 0
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["transformer","textcat"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[components]

[components.textcat]
factory = "textcat_multilabel"
threshold = 0.5


[components.textcat.model]
@architectures = "spacy.TextCatCNN.v1"
exclusive_classes = false
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "transformer"

[components.transformer]
source = "en_core_web_trf"
replace_listeners = ["model.transformer"]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
gold_preproc = ${corpora.train.gold_preproc}
max_length = ${corpora.train.max_length}
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths:train}
gold_preproc = false
max_length = 500
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 10
max_steps = 20000
eval_frequency = 200
frozen_components = ["transformer"]
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
eps = 0.00000001
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null
cats_score = 1
cats_macro_f = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

I've got:

[2022-07-24 21:06:34,320] [WARNING] [W086] Component 'textcat' will be (re)trained, but it needs the component 'transformer' which is frozen. You can either freeze both, or neither of the two. If you're sourcing the component from an existing pipeline, you can use the `replace_listeners` setting in the config block to replace its token-to-vector listener with a copy and make it independent. For example, `replace_listeners = ["model.tok2vec"]` See the documentation for details: https://spacy.io/usage/training#config-components-listeners
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'textcat']
ℹ Frozen components: ['transformer']
ℹ Initial learn rate: 0.0
E    #       LOSS TEXTCAT  CATS_SCORE  SCORE
---  ------  ------------  ----------  ------
⚠ Aborting and saving the final best model. Encountered exception: ValueError()
Traceback (most recent call last):
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\cli\_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\typer\main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\cli\train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\training\loop.py", line 122, in train
    raise e
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\training\loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\training\loop.py", line 203, in train_while_improving
    nlp.update(
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\language.py", line 1123, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy\pipeline\textcat.py", line 203, in update
    scores, bp_scores = self.model.begin_update([eg.predicted for eg in examples])
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\model.py", line 309, in begin_update
    return self._func(self, X, is_train=True)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\thinc\model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy_transformers\layers\listener.py", line 58, in forward
    model.verify_inputs(docs)
  File "C:\Users\aatul\Anaconda3\envs\nlp_core\lib\site-packages\spacy_transformers\layers\listener.py", line 47, in verify_inputs
    raise ValueError
ValueError

[polm]

Please format your config / code / exceptions as Markdown code blocks to make them easier to read, see the Github Markdown guide. Also please paste the whole stack trace / error output.

[AleksandrTulenkov]

Sorry, done

[polm]

Ah OK, thank you for the edits, it makes what you're trying to do more clear.

I assume that you have trained the textcat separately and want to combine it with your NER component. I also assume that you do not want to train both at the same time - instead you train the textcat, then train the NER.

If that's the case I would suggest not putting the textcat in the training file. If you include it in training it will run while training the NER, but you aren't using the annotations in training, you just need them afterwards. So it'll make training slow for no reason.

Instead you should add it in code or using spacy assemble. You can see an example of both of those in the Double NER example project, for example see here for an example of replace_listeners in a config. Note that in the config both final components are sourced.

[AleksandrTulenkov]

Thanks, but I think my second atempt wasn't clear again.

1. I already have some trained NER components and I wanted to train and add few new textcat compponents.
2. Right now its looks like: transformer -> ner -> ner ... (with listeners)
3. I can do like that: transformer -> ner -> ner ... (with listeners)-> textcat_transformer ->textcat(with textcat_transformer listener)
4. But whats I really want: transformer -> ner -> ner ... (with listeners)-> textcat(independent)

And spacy has apportunity to buid independent components, right? or I missed something

[polm]

spaCy allows you to build independent components. There is a limitation that all components in a pipeline have to depend on the same word vectors, but I don't think that affects your case.

In the double NER example, the final pipeline is like [ [ner (independent)] [ner (independent)] ]. I guess what you want is [transformer, ner, ner, [textcat (independent)] ], where only the textcat is independent, right?

In that case what I would recommend you do is:

1. Train your textcat separately
2. Add the textcat to your NER training config as a sourced component and disable it
3. Train your NER pipeline
4. Modify the config to enable the textcat before loading it

I believe you can also do this basically the same way as the double NER project - basically, source all the components, and only call replace_listeners for the textcat - but I would need to check how it works when you include multiple components using the same feature source.