Can't use textcat with pre-trained model

[JoEinw]

I wrote the following code to train a BERT-Model for Text Classification.
When I use a blank Model everything works fine, but when using a pre-trained model i get this error:

...lib\site-packages\thinc\model.py", line 175, in get_dim
raise ValueError(err)
ValueError: Cannot get dimension 'nO' for model 'sparse_linear': value unset

initialize the model

nlp = spacy.load("en_core_web_trf")                                     # pre-trained model
#nlp=spacy.blank("en")                                                         #blank model

# add the text classification pipe
textcat = nlp.add_pipe("textcat")
textcat.add_label("non-functional")
textcat.add_label("functional")

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']
with nlp.disable_pipes(*other_pipes):                               # only train textcat
    #optimizer = nlp.begin_training()                                #blank model
    optimizer = nlp.create_optimizer()                               #pre-trained model

[danieldk]

Thanks for your question!

If you want to train a model for text classification specifically, it is better to start from a fresh, pretrained transformer model. en_core_web_trf is a spaCy pipeline that uses a transformer that is finetuned on several syntax-based tasks and NER. Training a textcat pipe on top of this transformer would most likely give you worse results than finetuning a pretrained transformer from scratch. An additional problem is that when you update the transformer like this, it will 'forget' the other tasks that it is trained on, and the existing components in the en_core_web_trf pipe will usually perform badly.

You can start with a fresh pretrained transformer by generating a configuration for a textcat pipeline using the quickstart.

---

To give some more background on the error that you get --- the code example is not complete enough to reproduce the error, but it looks like the Thinc model is not properly initialized. The pipe needs to be initialized using the initialize method, which should be provided with several Examples to allow the Thinc model to infer the output dimension.

[JoEinw]

Thanks for the answer! The Quickstart Guide is what i would have tried next, but this looks a bit confusing for me now, are there any examples how i can train a pretrained transformer?

[danieldk]

Could you give more information on which step of the quickstart you do get stuck?

We also have example projects for training text categorization models:

https://github.com/explosion/projects/tree/v3/tutorials/textcat_docs_issues
https://github.com/explosion/projects/tree/v3/tutorials/textcat_goemotions
https://github.com/explosion/projects/tree/v3/pipelines/textcat_demo
https://github.com/explosion/projects/tree/v3/pipelines/textcat_multilabel_demo

Even though they don't use transformers, you could swap out the tok2vec component by a transformer once you have a working training setup.