pretraining to init_tok2vec (textcat) - ValueError: Cannot deserialize model: mismatched structure

[barebra]

I'm currently pretraining with 170.000 PubMed-abstracts (component: "textcat", layer: "tok2vec"). Epoch-loss is still decreasing and I just wanted to run a first pilot-experiment (train/evaluate) with an intermediate model (model75.bin). I learned from this forum that the config for training has to resemble the one of pretraining, so I copy/pasted the [pretraining]-section from the config_pretrain.cfg to the config.cfg and started spacy train.

Anyway, there seems to be a problem with mismatched structure (see title). Can you help me? Do you need more information (error log, config.cfg)?

[rmitsch]

Hi @barebra! Yes, please provide your stack trace/error log and your config file.

[barebra]

Hi rmitsch,

thank you for helping!

Here comes the error log from 1 of the parallel engines (im running spacy from Jupyter notebook, using ipyparallel on 10 engines):

[3:apply]:

ValueError Traceback (most recent call last)
File :1

File ~/.venv/lib/python3.9/site-packages/ipyparallel/client/remotefunction.py:148, in (f, *sequences)
144 return self.view.apply(self.func, *args, **kwargs)
147 if sys.version_info[0] >= 3:
--> 148 _map = lambda f, *sequences: list(map(f, *sequences))
149 else:
150 _map = map

File /tmp/ipykernel_878771/822473947.py:73, in classify(dict_key)

File ~/.venv/lib/python3.9/site-packages/spacy/cli/train.py:72, in train(config_path, output_path, use_gpu, overrides)
70 msg.divider("Initializing pipeline")
71 with show_validation_error(config_path, hint_fill=False):
---> 72 nlp = init_nlp(config, use_gpu=use_gpu)
73 msg.good("Initialized pipeline")
74 msg.divider("Training pipeline")

File ~/.venv/lib/python3.9/site-packages/spacy/training/initialize.py:84, in init_nlp(config, use_gpu)
80 nlp.initialize(
81 lambda: islice(train_corpus(nlp), sample_size), sgd=optimizer
82 )
83 else:
---> 84 nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
85 logger.info(f"Initialized pipeline components: {nlp.pipe_names}")
86 # Detect components with listeners that are not frozen consistently

File ~/.venv/lib/python3.9/site-packages/spacy/language.py:1327, in Language.initialize(self, get_examples, sgd)
1325 if pretrain_cfg:
1326 P = registry.resolve(pretrain_cfg, schema=ConfigSchemaPretrain)
-> 1327 init_tok2vec(self, P, I)
1328 self._link_components()
1329 self._optimizer = sgd

File ~/.venv/lib/python3.9/site-packages/spacy/training/initialize.py:193, in init_tok2vec(nlp, pretrain_config, init_config)
191 if weights_data is not None:
192 layer = get_tok2vec_ref(nlp, P)
--> 193 layer.from_bytes(weights_data)
194 logger.info(f"Loaded pretrained weights from {init_tok2vec}")
195 return True

File ~/.venv/lib/python3.9/site-packages/thinc/model.py:619, in Model.from_bytes(self, bytes_data)
617 msg = srsly.msgpack_loads(bytes_data)
618 msg = convert_recursive(is_xp_array, self.ops.asarray, msg)
--> 619 return self.from_dict(msg)

File ~/.venv/lib/python3.9/site-packages/thinc/model.py:636, in Model.from_dict(self, msg)
634 nodes = list(self.walk())
635 if len(msg["nodes"]) != len(nodes):
--> 636 raise ValueError("Cannot deserialize model: mismatched structure")
637 for i, node in enumerate(nodes):
638 info = msg["nodes"][i]

ValueError: Cannot deserialize model: mismatched structure

... 6 more exceptions ...

Ah, okay! Maybe this error arises because of parallelization? I'm running 10 different mixtures (train, dev, test) of my dataset in parallel.

[barebra]

And here comes the configuration for training:

[paths]
train = null
dev = null
vectors = "en_core_sci_lg"
init_tok2vec = "pretrain_vectors/model75.bin"

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","textcat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat]
factory = "textcat"
scorer = {"@scorers":"spacy.textcat_scorer.v1"}
threshold = 0.5

[components.textcat.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat.model.linear_model]
@architectures = "spacy.TextCatBOW.v2"
exclusive_classes = true
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@Loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
cats_score = 1.0
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null

[pretraining]
max_epochs = 1000
dropout = 0.2
n_save_every = null
n_save_epoch = null
component = "textcat"
layer = "tok2vec"
corpus = "corpora.pretrain"

[pretraining.batcher]
@batchers = "spacy.batch_by_words.v1"
size = 3000
discard_oversize = false
tolerance = 0.2
get_length = null

[pretraining.objective]
@architectures = "spacy.PretrainVectors.v1"
maxout_pieces = 3
hidden_size = 300
loss = "cosine"

[pretraining.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[rmitsch]

Ah, okay! Maybe this error arises because of parallelization? I'm running 10 different mixtures (train, dev, test) of my dataset in parallel.

I'm not sure I understand the parallelization here - are you training 10 models on a different data split each? What happens when you train without parallelization?

My assumption is rather that pretraining.objective.hidden_size doesn't match up.

[barebra]

Okay! I trained without parallelization and the Value error reoccured (see below).
Regarding the "pretraining.objective.hidden_size": I followed instructions on https://spacy.io/usage/embeddings-transformers#pretraining-objectives.

---

ValueError Traceback (most recent call last)
Cell In[22], line 3
1 for item in tencv_data_dict.keys():
2 tencv_data = tencv_data_dict[item]
----> 3 classify(tencv_data)
4 #
5 metrics = pd.read_json("./metrics.json")

Cell In[20], line 50, in classify(tencv_data)
48 #
49 from spacy.cli.train import train
---> 50 train("./config.cfg", "./textcat_model", overrides={"paths.train": "./train.spacy", "paths.dev": "./dev.spacy"})
51 #
52 from spacy.cli.evaluate import evaluate

File ~/.venv/lib/python3.9/site-packages/spacy/cli/train.py:72, in train(config_path, output_path, use_gpu, overrides)
70 msg.divider("Initializing pipeline")
71 with show_validation_error(config_path, hint_fill=False):
---> 72 nlp = init_nlp(config, use_gpu=use_gpu)
73 msg.good("Initialized pipeline")
74 msg.divider("Training pipeline")

File ~/.venv/lib/python3.9/site-packages/spacy/training/initialize.py:84, in init_nlp(config, use_gpu)
80 nlp.initialize(
81 lambda: islice(train_corpus(nlp), sample_size), sgd=optimizer
82 )
83 else:
---> 84 nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
85 logger.info(f"Initialized pipeline components: {nlp.pipe_names}")
86 # Detect components with listeners that are not frozen consistently

File ~/.venv/lib/python3.9/site-packages/spacy/language.py:1327, in Language.initialize(self, get_examples, sgd)
1325 if pretrain_cfg:
1326 P = registry.resolve(pretrain_cfg, schema=ConfigSchemaPretrain)
-> 1327 init_tok2vec(self, P, I)
1328 self._link_components()
1329 self._optimizer = sgd

File ~/.venv/lib/python3.9/site-packages/spacy/training/initialize.py:193, in init_tok2vec(nlp, pretrain_config, init_config)
191 if weights_data is not None:
192 layer = get_tok2vec_ref(nlp, P)
--> 193 layer.from_bytes(weights_data)
194 logger.info(f"Loaded pretrained weights from {init_tok2vec}")
195 return True

File ~/.venv/lib/python3.9/site-packages/thinc/model.py:619, in Model.from_bytes(self, bytes_data)
617 msg = srsly.msgpack_loads(bytes_data)
618 msg = convert_recursive(is_xp_array, self.ops.asarray, msg)
--> 619 return self.from_dict(msg)

File ~/.venv/lib/python3.9/site-packages/thinc/model.py:636, in Model.from_dict(self, msg)
634 nodes = list(self.walk())
635 if len(msg["nodes"]) != len(nodes):
--> 636 raise ValueError("Cannot deserialize model: mismatched structure")
637 for i, node in enumerate(nodes):
638 info = msg["nodes"][i]

ValueError: Cannot deserialize model: mismatched structure

[barebra]

I started pretraining again with a pretraining.objective.hidden_size of 200, since I came to know that word vectors from en_core_sci_lg only have a dimensionality of 200 (not 300 like en_core_web_lg). Anyway, there is no improvement regarding the ValueError when I use a model.bin from pretraining for training.

[svlandeg]

Hi @barebra,

Sorry that this has been confusing. Just to double check - have you been using the exact same config file for both your pretraining and training steps? You really should only need one file, in which the pretraining section will be ignored in the spacy train step.

[barebra]

Hi Sofie,   yes, I.m using the config.cfg - that was created from spaCy during pretraining (based upon the config_pretrain.cfg) and is stored in the "pretraining" directory together with the model.bins and the log.json - for training. Yesterday I tried a "positive control" for the pretraining2training task, and it worked: I used the word vectors from spaCy's en_core_web_lg (300 dimensions) in combination with default settings of pretraining (e.g. spacy.PretrainCharacters.v1, hidden_size=300 etc.). The word vectors I would like to use are from en_core_sci_lg (Allen AI, 200 dimensions). So, I assume I need to change hidden_size to 200 as well? Correct? But, using the "positive control", I will also change certain parameters, e.g. pretraining.component and pretraining.layer, trying to reproduce the error.

[svlandeg]

What might be convenient for you is to have a look at this spaCy project: https://github.com/explosion/projects/tree/v3/benchmarks/pretraining_morphologizer_oscar. It contains a few different ways of pretraining, you'll be most interested to run this one:

spacy project run training_vector

which has 3 subcommando's: pretrain_vector, train_vector and evaluate_vector. You can see how these are defined in the project.yml and you can see the config file used.

First, try and run this out-of-the box and verify that it works. Then you can start making the changes you need, and see where/if it breaks down. If you create a personal Github repo with your copy of this, then it'll be easier for us to follow along and to see what's going on.