Custom tok2vec/word vectors with pretrained morphologizer and lemmatizer

[Pandalei97]

Hi ! I have a question about the custom pipelines.

In our project, we have our own word vectors to initialize a tok2vec layer. For the lemmatization, we are thinking about taking the morphologizer and lemmatizer from a spacy pretrained pipeline. Does it make sense to get directly the two components from other pipelines ? Especially that the morphologizer will listen to a tok2vec layer which is different from which it was trained.

[polm]

The morphologizer should not be used with a tok2vec it wasn't trained with. You can package it with the original tok2vec it was trained with or retrain it - the former is probably easier.

I believe all the lemmatizers are rule based so it should be fine to bring them in from another pipeline.