Update 'ner' tok2vec embeddings during training

[nikita-galileo]

Based on my understanding of the 'ner' component, tok2vec (spacy.HashEmbedCNN.v2) outputs embeddings per token that are contextually aware. I've been playing around with the embeddings gotten from looking at the tokvecs in the ParserStepModel and it doesn't seem like the embeddings are updating with the model training. Happy to provide more info, currently I'm training my model with the following loop:

  optimizer = nlp.initialize(lambda: training_examples)

  training_losses = []
  with nlp.disable_pipes(*[pipe for pipe in nlp.pipe_names if pipe != 'ner']):
      for itn in range(n_iter):
          batches = minibatch(training_examples, minibatch_size)

          for batch in tqdm(batches):
              training_loss = nlp.update(
                  batch,
                  drop=0.5,
                  sgd=optimizer)

              training_losses.append(training_loss["ner"])

and I have some code to extract the tokvecs from the ParserStepModels, which after analysis do not seem clustered around the labels the model is predicting

[adrianeboyd]

It's hard to know for sure without more info about the pipeline config and your custom code.

If your ner has an internal tok2vec model (vs. listening to separate tok2vec component), then its weights should be updated in this loop.

If you have a separate tok2vec component (which is the pipeline design you'd get from the quickstart or init config), then it won't be updated because you've disabled it in this context with nlp.disable_pipes.

In general, we strongly recommend using spacy train instead of a custom training loop like this. If you want to do it programmatically, spacy train is just a CLI wrapper around training methods that you can call directly if you wish.