NER training with custom wordvectors

[Gitclop]

I have custom wordvectors in spacy model, that i want to use for NER training. The spacy model is saved as folder-structure, the wordvectors are stored in its vocab.

How do i load the model, or the vectors in the training config?

Current config:

[paths]
train = null
dev = null
vectors = "de_core_news_lg"
[system]
gpu_allocator = null

[nlp]
lang = "de"
pipeline = ["tok2vec","ner"]
batch_size = 1000

[components]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [5000, 1000, 2500, 2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[initialize]
vectors = ${paths.vectors}

[Gitclop]

Followup question since i mix up model/component/pipeline.
I've trained spacy models for senter- and NER. I also trained wordvectors with gensim word2vec.
How should i combine all those models (or components?)
Right now i just add them to my pipeline:

def setup(self, f):

        nlp = spacy.load('de_core_news_lg', exclude=['senter'])
        senter_source = spacy.load(self.path+'Senter/Senterv1')
        
        nlp.add_pipe('senter', source=senter_source, before='parser')

        nlp.vocab.vectors = self.addWordVectors(f).vectors

        path = self.path+'Spacy-Model'
        nlp.to_disk(path=path)
        
        return nlp

def addWordVectors(self, f):
    path = self.path
    Vectorfile = f
    gensim_model = Word2Vec.load(self.path_to_w2v+Vectorfile)
    vocab = Vocab()
    # Hinzufügen von Wörtern und Vektoren aus dem gensim-Modell
    for word in gensim_model.wv.index_to_key:
        vector = gensim_model.wv.get_vector(word)
        vocab.set_vector(word, vector)
    return vocab

So should i add my ner-model like the senter-model with add_pipe()?
When the pipeline is done, i save it to disk

nlp.to_disk(path=path)

and reload it with

nlp = spacy.blank('de')
nlp = nlp.from_disk("/path")

So in short: I train different models for different components.
Then when i instantiate the nlp object i use a standard language model, exclude the components that i have my own models for and add those components (or models?) manualy to the nlp object and finaly save the nlp object.

Is that the way to go, or should i export the whole model so i can use it like

nlp = spacy.load('my_custom_model')

Because this way i could easily use it in my traing config.

[Gitclop]

Found the solution. Had to remake the base.config, added the full path to my Model to vectors = remade the full config and bingo! Its training with my own vectors now.

[svlandeg]

Thanks for reporting back! Happy to hear you were able to resolve your issues.