Fine-Tune BERT Transformer with spaCy 3

[CarlaMPR]

Hello! I'm trying to train a custom NER model by fine-tuning a transformer model with spaCy 3. In the base. config.cfg file:
[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
tokenizer_config = {"use_fast": true}

what models can I use on "name" instead of roberta-base? For example, can I change to this one https://huggingface.co/dmis-lab/biobert-v1.1 ?

Thank you!!

[rmitsch]

Hi @CarlaMPR, see the docs on transformers in spaCy:

It supports all models that are available via the HuggingFace transformers library.

biobert should work as well. If you run into any problems, let us know here.

[CarlaMPR]

Thank You! So, as biobert doesn't appear on this page, https://huggingface.co/docs/transformers/index#supported-models, does it mean I can't use it?

[rmitsch]

As a rule of thumb: as long as the model can be loaded with AutoModel.from_pretrained(), it should work (we can't guarantee that though). So go ahead and give it a shot.

I recommend to have a look at the spaCy + HuggingFaceFAQ for other potential useful info.

[CarlaMPR]

Thank you!