Using methods in spacy config file

[noobistz]

Sometimes there is a need to add a special token to the tokenizer, for instance, when using gpt2 in spacy config:

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "gpt2"

[components.transformer.model.tokenizer_config]
use_fast = true
pad_token = '[PAD]'

However, when the pad_token is added, the length of the embedding changes. Adding model.resize_token_embeddings(len(tokenizer)) in the next line of code after instantiating the model will rectify the issue.

With spacy config file, can model.resize_token_embeddings(len(tokenizer)) be added any where so that we can still use cli to train a gpt2 model?

[adrianeboyd]

I haven't tested this, but in general I think the right place to do this would be in the after_init callback.

I haven't done much training/testing with gpt2 models, but I can see that in our tests we just use "<|endoftext|>" as the pad token, which is already in the vocab.

[noobistz]

thanks, using "<|endoftext|>" does indeed work