Adding lemmatizer to trained NER model

[spatiebalk]

I am training a NER model that also needs to have a lemmatizer. This lemmatizer I initialized with the source model en_core_web_sm as that is fine and does not need to be trained in my use case. However, after training, the NER goes fine, but the lemmatizer gives a different result than the en_core_web_sm model itself.

I was wondering whether something is wrong in my config file (here added as .txt but used as .cfg) and whether I can fix this without having to retrain the entire NER model?

I see there is a lemmatizer file in my model: trained_model/lemmatizer/lookups/lookups.bin. so is it maybe possible to replace this with the file used by en_core_web_sm? If so, where could I find this file?

config_NER_model_txt.txt

[thomashacker]

Hello,
you also need to source the tok2vec layer from the pre-trained model.

[components.tok2vec]
source = "en_core_web_sm"

You'll also need to add the tok2vec component to the pipeline variable in the [nlp] section
pipeline = ["senter", "transformer", "tok2vec", "parser", "tagger", "attribute_ruler", "entity_ruler", "ner", "lemmatizer"]

[thomashacker]

I'll convert your issue into a discussion since it's not an "issue" 😄