More than one NER component in the pipeline

[Pandalei97]

Hi there ! I have a question about having more than one NER component in the pipeline.

Suppose that we have 2 NER models, for the sentence "It's Trump Tower", the first ner will recognize Trump as a name, and the seconde ner will recognize Trump Tower as a place.

If we add both the two ner components in the pipeline, I see that currently it only keeps Trump Tower, which overlaps Trump. I wonder if it's Spacy who manages it or it's a coincidence.

If it's spacy who does the job, can I know more details about how it manage different overlapping cases ?
For example, for the text ABC, if ner1 recognize AB and ner2 recognize BC, will it keep the two entities ?

Thanks a lot !

[svlandeg]

Hi!

Typically, I think the first NER will annotate what it can, and the second NER won't overwrite/change any of those predictions. In that sense, I'm a little puzzled by this sentence:

If we add both the two ner components in the pipeline, I see that currently it only keeps Trump Tower, which overlaps Trump.

NER components typically preserve existing annotations - so I would have expected that you get "Trump" from the first NER, and perhaps nothing, or just "Tower", from the second. Are you 100% sure that the first would have correctly recognized "Trump" if you run it in a pipeline by itself? Perhaps in this particular case, "Trump" simply wasn't recognized as an entity by the first NER, leaving all tokens to the second NER to annotate. To verify - can you share some example code & output?

For example, for the text ABC, if ner1 recognize AB and ner2 recognize BC, will it keep the two entities ?

Assuming these are three different tokens, then I'd expect ner1 to recognize AB, but after that ner2 won't be able to recover BC, because tokens can't be in different named entities at the same time.

Hope that helps - but happy to discuss further if you can provide more details/code!

[svlandeg]

I don't think there's really a buit-in mechanism to accomodate for that specific use-case, but I think it could be relatively clean with a few very light-weight components in between, as you describe.

Another thing that might be useful for you at this point, is the new doc.spans that was introduced since 3.0. You can keep the annotations per NER there, and do some sort of consolidation step at the end of your pipeline and write the final result back to doc.ents?

[svlandeg]

Wait, actually, the EntityRecognizer inherits from the Parser and the latter has a mechanism self.postprocesses which is a list of hooks that is called on each doc after setting the annotations as usual. If you define such a custom "postprocessing" hook, you could probably do the copying of the entities there. I don't think any of this is documented, but you can find the relevant code here: https://github.com/explosion/spaCy/blob/master/spacy/pipeline/transition_parser.pyx

Alternatively, you could create a subclass of EntityRecognizer and redefine set_annotations all together.

[Pandalei97]

Thanks a lot, that's really helpful ! I will try to use self.postprocesses.

By the way, actually I use

ner = spacy.load(ner_path, vocab=nlp.vocab)
ner.replace_listeners("tok2vec", "ner", ["model.tok2vec"])

to load the ner model. How can it load it by using EntityRecognizer in Spacy3 ?

I tried

config = Config().from_disk(ner_path+ "/config.cfg")
ner = EntityRecognizer(nlp.vocab, config=config)

but I got a error while initializing EntityRecognizer.

  File "spacy\pipeline\transition_parser.pyx", line 37, in spacy.pipeline.transition_parser.Parser.__init__
TypeError: __init__() takes at least 2 positional arguments (1 given)

It seems that we have a lot of parameters to pass, but in Spacy2 we only needed to do

ner = EntityRecognizer(nlp.vocab)
ner.from_disk(ner_path)

to load the model.

Do you have any idea ?

[svlandeg]

Why exactly do you need to work through the constructor of EntityRecognizer?

In spaCy v3, all of these internals have changed quite a bit. The factory ner is defined in https://github.com/explosion/spaCy/blob/master/spacy/pipeline/ner.pyx and there you'll see that it passes onwards a DEFAULT_NER_MODEL and other arguments. In v3, everything is much more explicit, so the NER constructor needs to be fed a Machine learning model and all of the parameters. The NER factory then provides a default set of reasonable parameter values & model implementation, that is suitable for most use-cases. But if you want to call the constructor directly, you'll have to feed all of this yourself, too.

[Pandalei97]

Okay !!! I confused the the NER model and the NER pipe ! Thanks a lot for your answer, it's all clear for me now. 😁😁