How to initialize tok2vec Transformer with a custom spacy ner model?

[laith07]

I have some trouble with the initialization of a tok2vec Transformer with a custom spacy ner model. How do I use tok2vec properly before the ner step starts in the pipeline?

Init

   nlp = spacy.load("./output_training_11.11")
    ner = nlp.get_pipe("ner")
    config = {"model": DEFAULT_TOK2VEC_MODEL}
    nlp.add_pipe("tok2vec",  before='ner')`

Training:

with nlp.disable_pipes(*other_pipes):

  optimizer = nlp.begin_training()

  for i in range(100):
    random.shuffle(TRAIN_DATA)
    for text, annotation in TRAIN_DATA:
        doc = nlp.make_doc(text)
        #print(annotation)
        try:
            example = Example.from_dict(doc, annotation)
            #print(example)
        except ValueError as e:
            print(e)
            print("Text: " , doc[0:50])
            print("------")
        nlp.update([example], 
                   sgd=optimizer, drop=dropout)



nlp("random text")


ValueError: [E109] Component 'tok2vec' could not be run. Did you forget to call `initialize()`?

[polm]

I'm a little confused at what you're trying to do here.

nlp = spacy.load("./output_training_11.11")
ner = nlp.get_pipe("ner") # it looks like you're not using the ner you created here?
config = {"model": DEFAULT_TOK2VEC_MODEL} # where are you using this, and what is DEFAULT_TOK2VEC_MODEL?
nlp.add_pipe("tok2vec",  before='ner')`

You don't want a tok2vec and a transformer component in the same pipeline - they replace each other. (This can be different if you use replace_listeners, but that doesn't look like what you're doing here.)

Also I see you're using a custom training loop, but note we don't recommend that in v3 - it's much easier to avoid problems using the training config. Maybe taking a look at the spaCy course would help.

[laith07]

Thanks for the replay.

How can I replace the transformer with the tok2vec? I want to start an experiment and use tok2vec instead of the transformer.

And about the training I looked up the code from the book "Mastering spaCy" (Chapter 7), but thanks for the advice.

And is there a way to train a model with span instead of docs? I am getting the same error as this issue #5727 described. Some text (from user) are without proper whitespace like
text = "Susanist gut." entities = [(0, 4, "PERSON")]

the proper sentence would be ->
text = "Susan ist gut." entities = [(0, 4, "PERSON")]

Do you have a suggestion how to solve this issue?

[polm]

Ah OK, thanks for the clarification. If you want to switch a tok2vec in for a transformer you need to modify the pipeline section of your config and replace the transformer related blocks with a tok2vec block. You may also need to modify things like Listeners.

You can get an idea of the complete differences by going to the quickstart and comparing CPU and GPU configs. There are a lot of details, so it might be easier to generate a config from scratch and modify it to include your customizations.

And is there a way to train a model with span instead of docs?

No. Also I don't think that the issue with your data is spans vs docs.

The problem is that the model can only give one label to each token - it can't label half a token, and there's no reasonable way for it to know that "Susanist" should be two words. A simple thing you can do with data like that is decide whether you would rather have "Susanist" labelled as PERSON or not labelled at all. When you create training data, you can use char_span's alignment_mode argument to enforce your policy.

[laith07]

Thanks @polm