Mismatched IDs Error and nlp.rehearse

[nashcaps2255]

Have a textcat multilabel model which I am trying to update with nlp.rehearse to alleviate the catastrophic forgetting problem.

nlp = spacy.load('my_model')

examples = []
for line in file_:
   text, label = line.split("|")
   doc = nlp(text)
   gold_dict = {"cats": {label: float(1)}}
   gold_dict = Example.from_dict(doc, gold_dict)
   examples.append(example)


optimizer = nlp.resume_training()
nlp.rehearse(examples, sgd = optimizer) 

Results in......

ValueError: [E953] Mismatched IDs received by the Tok2Vec listener: 179568814531392983158587824 vs. 2172509679243279887229

[polm]

Sorry to hear you ran into that error, I don't think we've heard of that coming up in rehearsal before.

Could you provide your training config so we have more information on your pipeline architecture?

Also note that while rehearsal will help avoid catastrophic forgetting, the surest way to avoid is to train from scratch, so that may be a good idea if it's an option.

[nashcaps2255]

[paths]
train = null
dev = null
vectors = "en_core_web_lg"
init_tok2vec = null

[system]
gpu_allocator = null 
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec", "textcat_multilabel"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat_multilabel]
factory = "textcat_multilabel"
scorer = {"@scorers":"spacy.textcat_multilabel_scorer.v1"}
threshold = 0.5

[components.textcat_multilabel.model]
@architectures = "spacy.TextCatEnsemble.v2"
n0 = null

[components.textcat_multilabel.model.linear_model]
@architectures = "spacy.TextCatBOW.v2"
exclusive_classes = false
ngram_size = 1
no_output_layer = false 
n0 = null

[components.textcat_multilabel.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = 64
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 64
attrs = ["ORTH", "SHAPE"]
rows = [5000, 2500]
include_static_vectors = true 

[components.tok2vec.model.encode]
@architectures = "spacy.MishWindowEncoder.v2"
width = 64
depth = 4
window_size = 1

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false 
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false 
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 300000
max_epochs = -1
max_steps = 100000
eval_frequency = 1000
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = 0.2 

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false 

[training.optimizers]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true 
L2 = 0.01
grad_clip = 1.0
use_averages = false 
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
cats_score = 1.0
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null 
cats_macro_auc_per_type = null 

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null 
before_init = null 
after_init = null

[initialize.components]

[initalize.tokenizer]

[thomashacker]

Hey, sorry for the late reply!

Currently, using nlp.rehearse with components that use Tok2VecListeners don't work. The only current fix is to use inline tok2vec components.

We're looking into this issue #12044 and are working on a fix in this PR #12045.