BERT and spacy - is it possible to use BERT masked language model from spacy? #7542
-
|
I know spacy integrates BERT for generating word \ document embeddings. (using BERT as an external model is of course possible, but the integration seems non trivial - for example handling the different tokenization methods) |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hey, sorry for the delayed reply on this. Short answer is no, Natural Language Generation type tasks are considered out of scope for spaCy. You could do something using it as a pipeline component. The English Transformer model is already based on RoBERTa, you can look at the source of spacy-transformers for more details on how the difference in tokenization is handled. |
Beta Was this translation helpful? Give feedback.
Hey, sorry for the delayed reply on this. Short answer is no, Natural Language Generation type tasks are considered out of scope for spaCy.
You could do something using it as a pipeline component. The English Transformer model is already based on RoBERTa, you can look at the source of spacy-transformers for more details on how the difference in tokenization is handled.