Huggingface tokenizer provides incorrect model_max_length

[ANenashev]

How to reproduce the behaviour

I'm using nlpaueb/legal-bert-base-uncased transformer model.

[paths]
train = train.spacy
dev = eval.spacy
vectors = null
init_tok2vec = null

[system]
seed = 0
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["sentencizer","transformer","dev_bert_clauses_classifier_ref"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[nlp.tokenizer]
@tokenizers = "spacy.Tokenizer.v1"

[components]

[components.sentencizer]
factory = "sentencizer"
punct_chars = null

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "nlpaueb/legal-bert-base-uncased"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.sent_spans.v1"

[components.dev_bert_clauses_classifier_ref]
factory = "dev_bert_clauses_classifier_ref"
labels_limit = 4

[components.dev_bert_clauses_classifier_ref.model]
@architectures = "dev_clauses_classifier_model.v1"

[components.dev_bert_clauses_classifier_ref.model.create_clauses_tensors]
@architectures = "dev_clause_tensor.v1"

[components.dev_bert_clauses_classifier_ref.model.create_clauses_tensors.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 0.0

[components.dev_bert_clauses_classifier_ref.model.create_clauses_tensors.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.dev_bert_clauses_classifier_ref.model.create_clauses_tensors.get_clauses]
@span_getters = "spacy-transformers.doc_spans.v1"

[components.dev_bert_clauses_classifier_ref.model.classifier_model]
@architectures = "dev_lstm_classifier_model.v1"
embeddings_dim = 768
rnn_hidden_dim = 100
nO = 24
bidirectional = True

[corpora]

[corpora.dev]
@readers = "dev_CorpusWSents.v1"
path = ${paths.dev}
gold_preproc = false
max_length = 0
limit = 0
augmenter = null

[corpora.train]
@readers = "dev_CorpusWSents.v1"
path = ${paths.train}
gold_preproc = false
max_length = 0
limit = 0
augmenter = null

[training]
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 3
patience = 0
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
get_length = null
size = 500
buffer = 256

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
cats_score = 1.0
sent_p = 0.0
sent_r = 0.0
sent_f = 0.0

[pretraining]

[initialize]

[initialize_components]

It's tokenizer model_max_length equal to 1000000000000000019884624838656 instead of 512. This causes error on sentences longer than 512 wordpiece tokens because truncate_oversize_splits doesn't work.

ℹ Pipeline: ['sentencizer', 'transformer',
'dev_bert_clauses_classifier_ref']
ℹ Initial learn rate: 0.001
E    #       LOSS TRANS...  LOSS DEV_B...  SENTS_F  SENTS_P  SENTS_R  CATS_SCORE  SCORE 
---  ------  -------------  -------------  -------  -------  -------  ----------  ------
⚠ Aborting and saving the final best model. Encountered exception:
RuntimeError('The size of tensor a (588) must match the size of tensor b (512)
at non-singleton dimension 1')
Traceback (most recent call last):
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/cli/train.py", line 59, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/training/loop.py", line 114, in train
    raise e
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/training/loop.py", line 98, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/training/loop.py", line 212, in train_while_improving
    score, other_scores = evaluate()
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/training/loop.py", line 264, in evaluate
    scores = nlp.evaluate(dev_corpus(nlp))
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/language.py", line 1354, in evaluate
    examples,
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy/util.py", line 1478, in _pipe
    yield from proc.pipe(docs, **kwargs)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy_transformers/pipeline_component.py", line 202, in pipe
    self.set_annotations(subbatch, self.predict(subbatch))
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy_transformers/pipeline_component.py", line 218, in predict
    activations = self.model.predict(docs)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/thinc/model.py", line 312, in predict
    return self._func(self, X, is_train=False)[0]
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/spacy_transformers/layers/transformer_model.py", line 142, in forward
    tensors, bp_tensors = transformer(wordpieces, is_train)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/thinc/model.py", line 288, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/thinc/layers/pytorchwrapper.py", line 80, in forward
    Ytorch, torch_backprop = model.shims[0](Xtorch, is_train)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/thinc/shims/pytorch.py", line 29, in __call__
    return self.predict(inputs), lambda a: ...
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/thinc/shims/pytorch.py", line 38, in predict
    outputs = self._model(*inputs.args, **inputs.kwargs)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/transformers/modeling_bert.py", line 838, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/nenashevas/.local/share/virtualenvs/bert-clauses-classifier-t7CyBQ2n/lib/python3.7/site-packages/transformers/modeling_bert.py", line 201, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (588) must match the size of tensor b (512) at non-singleton dimension 1
python-BaseException

How can I set correct model_max_length value for HF tokenizer or for truncate_oversize_splits function?

Your Environment

Info about spaCy

• spaCy version: 3.0.3
• Platform: Linux-5.4.0-58-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.4

[adrianeboyd]

There are some models where this setting is just missing from the config and some where transformers used to have an internal default that's since been removed, so the model authors aren't really at fault (some albert models, I think?), but in both cases the model_max_length setting default is their internal VERY_LARGE_INTEGER (I am not kidding, this is the actual name).

The solution is to save a local copy of the model with the updated setting:

from transformers import AutoTokenizer, AutoModel
name = "nlpaueb/legal-bert-base-uncased"
local_path = "/path/to/legal-bert-base-uncased"
model = AutoModel.from_pretrained(name)
tokenizer = AutoTokenizer.from_pretrained(name)

# add the setting (note that you can modify tokenizer.model_max_length on the fly, 
# but frustratingly this change isn't saved as part of the saved config)
tokenizer.init_kwargs["model_max_length"] = 512

# save
tokenizer.save_pretrained(local_path)
model.save_pretrained(local_path)

Then for components.transformer.model.name in the spacy config, you provide the path to the local model directory rather than the name from their model repo.

[adrianeboyd]

Update: a much simpler solution is to set this in [components.transformer.model.tokenizer_config]:

[components.transformer.model.tokenizer_config]
use_fast = true
model_max_length = 512