Pretrain with transformers

[thiagoms-tc]

I'm trying to pretrain a textcat model using transformers(lucas-leme/FinBERT-PT-BR) from hugginface as a base to latter do the fine tuning with proper annotations, but I can't generate a configuration file for the pretraining of this model that works, is that possible?

[vinbo8]

What exactly do you mean by "pretrain" in this context? Are you trying to pretrain a transformer model from scratch, or do you mean you want to fine-tune the model's weights on an annotated downstream task?

[thiagoms-tc]

I'm thinking of taking some texts without any annotations and using this spatial command (python -m spacy [pre-training]) to increase the model's vocabulary and only after that fit the model with the labeled data.

[vinbo8]

If you intend to continue pre-training your models on unannotated data, I would suggest using transformers instead - we unfortunately do not support this at the moment.

[thiagoms-tc]

Thank you.