If training a customer NER model, is there a meaningful difference between using SpanCategorizer vs. "regular" NER?

[Lolologist]

I want to train one or more custom entities, and also get scores for those entities. I know that a/the current way to do that is by using SpanCategorizer, and that SpanCategorizer's suggesters (https://spacy.io/api/spancategorizer#suggesters) are different from however NER finds its candidate spans. Also, the documentation seems to at least heavily hint that SpanCategorizer is intended for arbitrary spans that aren't really entities per se.

Is there anything fundamentally different in how they work that would make it a bad idea/needing careful attention to train traditional entities via SpanCategorizer?

[polm]

See this comment. Basically the SpanCat is more general / gets to make fewer assumptions, so it can be less accurate. Depending on application the difference in accuracy with an NER component may be negligible or it may be more significant.

[Lolologist]

Thanks very much for the reply! Semi-unrelated question; if I write a different span suggester (probably one based on dependency parses in case you're looking for "phrases") is that something that you all would be interested in incorporating?

[svlandeg]

Hi @Lolologist, I don't think we want to have a bunch of different suggesters in the library, as often people will still want to be able to tweak some things or implement their own. But if you have some useful implementations that you feel are generic enough, it might make sense to publish them in the spaCy universe?