Identical models being trained even with different parameters

[shaikh58]

Hi! First of all want to thank the Spacy team for being available and helpful when users ask questions! I'm currently facing an issue that I'd like some help with.

I'm training an NER model using 1.5k Prodigy-annotated data points. I'm running a Spacy project, with tok2vec and NER as the components. I have experimented with multiple parameters, all of them leading to what seems like the exact same model being trained. (The NER f-scores for each entity and the overall f-score are identical each time).

In config.cfg, I've varied:

• [initialize] -> en_core_web_sm/en_core_web_lg/null
• components.tok2vec -> source from en_core_web_sm/en_core_web_lg, and factory (to train the component)
• I've even tried pretraining for 10 epochs with the annotated data, then initialized training with the .bin weights file

In all these experiments, I get the exact same F-score on the evaluation set. I create a new directory for each experiment, remove project.lock and the training directories, and rerun spacy project run all.

I might be missing something obvious here, but would really appreciate some help as to why this is happening! A brief explanation of the difference between the [initialize] block and tok2vec initialization would also be very helpful, as I haven't yet been able to find an explanation I fully understood.

Thanks,
Mustafa

[mbrunecky]

I am afraid that more info is needed.
Specifically, the whole config.cfg, plus perhaps a bit more about your data - how many documents/sentences/words and NER labels is that?
My experience may not apply to you, but in my domain a dataset of 5000 documents each having at least 3k bytes and 2 NER entity labels is barely enough to get f-scores over 0.7.

[shaikh58]

thanks for your response! I've copied in the config.cfg into a reply in polm's thread below. Would be great if you could take a look and see if you can see anything too. The data is 1500 annotated product descriptions (e.g. "this desk is made out of wood and is rectangular...."), each of which are up to 1300 characters long. The issue is that the scores are identical for each of the labels in all my experiments, pretrain or not, initialize with en_core_web_xx or not. Again, I might be missing something obvious here

[polm]

There are a variety of things that could be going wrong here, but just to check one thing - is include_static_vectors set to True? If not then the vectors in your model won't be used as features in tok2vec and so whatever settings you use there won't affect the model.

If it is set to True then do please provide your config and more info.

[shaikh58]

Hi polm, thanks for your response! Below are the contents of my config file (apologies for just copying it directly in here, it was giving me trouble uploading even as .txt.), and yes include_static_vectors is set to true. The data is 1500 annotated product descriptions (e.g. "this desk is made out of wood and is rectangular...."), each of which are up to 1300 characters long. I generated the config file by using Spacy's suggested base_config.cfg, then running init fill config as part of the pipeline. I then manually entered the path to pretrained vectors in ini_tok2vec.

Here's a screenshot of the model training output, thought it might be useful:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = "/home/......./model322.bin"
raw_text = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.pretrain]
@readers = "spacy.JsonlCorpus.v1"
path = ${paths.raw_text}
min_length = 5
max_length = 500
limit = 0

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]
max_epochs = 10
dropout = 0.2
n_save_every = null
component = "tok2vec"
layer = ""
corpus = "corpora.pretrain"

[pretraining.batcher]
@batchers = "spacy.batch_by_words.v1"
size = 3000
discard_oversize = false
tolerance = 0.2
get_length = null

[pretraining.objective]
@architectures = "spacy.PretrainCharacters.v1"
maxout_pieces = 3
hidden_size = 300
n_characters = 4

[pretraining.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[shaikh58]

hi, following up on this. any help is greatly appreciated!

[polm]

About your parameters...

[initialize] -> en_core_web_sm/en_core_web_lg/null

Where are you setting those in the initialize block? Vectors? The small model has no vectors so it won't do anything and will be the same as null. The last four entries aren't really relevant for what you're trying to do, so there's not really any reason to tweak them (though see the docs if you haven't yet).

[shaikh58]

Yes I’m setting the vectors parameter in the [initialize] block. I’ve tried en_core_web_md/lg and also none, which made no difference to the final model. This could be explained by the fact that my data has little overlap with the standard pipelines. However what I don’t understand is how pretraining tok2vec also has no impact on the model whatsoever. I made sure to include the bin file as the tok2vec init, and the pipeline seems to be loading the pretrained weights. Any ideas there?

[polm]

Sorry we haven't been able to help more with this. Just to be absolutely sure we weren't missing anything obvious, I used a sample project - textcat_goemotions - and tried changing it to use the vectors from the large english model, instead of the default of no vectors. For testing I trained for just 2000 iterations (<2 epochs), and while the loss and overall scores barely changed, the per-label scores were all slightly different. So it's not like the functionality doesn't work...

One possibility is that there's nothing wrong with your config, but your data has easy and hard cases, and regardless of the configuration the model is able to learn the easy cases right away but not the hard ones. One thing that supports this is that your training seems to end quite quickly due to early stopping, with your best model reached after only 1400 iterations.

Maybe you could run spacy debug data on your training and dev data and see if anything comes up?

[shaikh58]

Hi Polm, you raise an interesting point regarding the dataset. It did seem strange that Spacy itself could have an issue, or that my config wasn’t working. I will try this exercise with a completely different dataset and check to see if that changes things. We do also have a small dataset with many similar entries and this could be the cause. It’s also been very helpful to confirm my understanding of my configs, so thanks for all your help