✘ Config validation error - instance of model expected | with independent transformers

[Larsdegroot]

Hi all!

I'm trying to create a pipeline for relation extraction. I believe that having seperate transformer / tok2vec layers for the ner and relation extraction component will increase accuracy at the cost of efficiency, but for my use case efficiency is not that important.

In my config i've defined that the ner and parser use a transformer listener, and the relation extractor defines its own transformer.
however when running spacy train configs/config.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py --gpu-id 0

I get the following error:

================================= train_gpu =================================
Running command: /home/ldegroot/anaconda3/envs/re/bin/python3 -m spacy train configs/config.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py --gpu-id 0
/home/ldegroot/anaconda3/envs/re/lib/python3.10/site-packages/cupy/_environment.py:437: UserWarning: 
--------------------------------------------------------------------------------

  CuPy may not function correctly because multiple CuPy packages are installed
  in your environment:

    cupy, cupy-cuda111

  Follow these steps to resolve this issue:

    1. For all packages listed above, run the following command to remove all
       existing CuPy installations:

         $ pip uninstall <package_name>

      If you previously installed CuPy via conda, also run the following:

         $ conda uninstall cupy

    2. Install the appropriate CuPy package.
       Refer to the Installation Guide for detailed instructions.

         https://docs.cupy.dev/en/stable/install.html

--------------------------------------------------------------------------------

  warnings.warn(f'''
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
✘ Config validation error
relation_extractor.model.create_instance_tensor -> tok2vec   instance of Model expected

{'@architectures': 'rel_instance_tensor.v1', 'get_instances': {'@misc': 'rel_instance_generator.v1'}, 'pooling': {'@layers': 'reduce_mean.v1'}, 'tok2vec': {'factory': 'transformer', 'max_batch_items': 4096, 'set_extra_annotations': {'@annotation_setters': 'spacy-transformers.null_annotation_setter.v1'}, 'pooling': {'@layers': 'reduce_mean.v1'}, 'model': {'@architectures': 'spacy-transformers.TransformerModel.v3', 'name': 'roberta-base', 'get_spans': {'@span_getters': 'spacy-transformers.strided_spans.v1', 'window': 128, 'stride': 96}, 'tokenizer_config': {'use_fast': True}}}}

---

I've narrowd the problem down to having something to do with the independent transformers because when i run the same training command with a config that uses a listener for all components this issue occurs. (however no config validation error)

---

Entire config:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
seed = 342
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["transformer","parser","ner", "relation_extractor"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.relation_extractor]
factory = "relation_extractor"
threshold = 0.5

[components.relation_extractor.model]
@architectures = "rel_model.v1"

[components.relation_extractor.model.create_instance_tensor]
@architectures = "rel_instance_tensor.v1"

[components.relation_extractor.model.create_instance_tensor.tok2vec]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.relation_extractor.model.create_instance_tensor.tok2vec.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
tokenizer_config = {"use_fast": true}

[components.relation_extractor.model.create_instance_tensor.tok2vec.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.relation_extractor.model.create_instance_tensor.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.get_instances]
@misc = "rel_instance_generator.v1"

[components.relation_extractor.model.classification_layer]
@architectures = "rel_classification_layer.v1"
nI = null
nO = null

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.parser]
factory = "parser"
learn_tokens = false
min_action_freq = 30
moves = null
scorer = {"@scorers":"spacy.parser_scorer.v1"}
update_with_oracle_cut_size = 100

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = false
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = ["parser"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
dep_uas = 0.25
dep_las = 0.25
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = 0.0
ents_f = 0.5
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[polm]

Before looking into what's going wrong here - while having a tok2vec and Transformer in the same pipeline at training time should technically work, it won't provide any benefit, and it may be easier to have one pipeline per token source (tok2vec/Transformer) and use spacy assemble to put them together after training. In particular I would expect the different models to have different rates of convergence, and training them together would make that harder to see.

[Larsdegroot]

I'm sorry i'm seeing that i wrote it to be a bit confusing. What i mean was that i have either a 2 independent transformers or 2 independent tok2vec layers in the pipeline. (so not mixing transformer and tok2vec)

[polm]

That's actually worse because it's easier to accidentally listen to the wrong thing.

If your feature sources aren't actually in the same backprop graph, which seems to be your goal here, then it should be easier to train them separately.

[polm]

Sorry for the delayed followup on this. It looks like the first issue is that the create_instance_tensor.tok2vec entry in the config is similar to [component.transformer.model], not [component.transformer] - so you should specify an architecture, not factory. However if you do that it doesn't work due to an error about FullTransformerBatch not being iterable. I think this may be a limitation of the REL component.

What you can do in this case is use a listener in the REL component and have two Transformers in your pipeline. If you do that you'll have to give the Transformers different names (like transformer and transformer_rel), and you'll need to change the upstream setting in your listeners to the name of the Transformer you want to listen to. That's not a setup we test extensively, but it should work.