Labelling entities in the training data for NER

[shrinidhin]

Hi. So I am training a custom NER model in spacy for identifying custom entities in my data. Consider the following text:
Example 1 : Nifty Metal, IT, FMCG,Financial Services and PSU Bank indices
Now I want the entity labels to be recognized as follows:

Nifty Metal                    INDEX
Nifty IT                       INDEX
Nifty FMCG                     INDEX
Nifty Financial Services       INDEX
Nifty PSU Bank                 INDEX

Example 2: NIFTY SMALLCAP 100 AND MIDCAP 100

To be recognized and labelled as:

NIFTY SMALLCAP 100  INDEX
NIFTY MIDCAP 100    INDEX

I am having trouble understanding how to tag this data using BILOU tagging approach?Should I instead use an entity ruler and add it as a pattern?

[polm]

NER is only for contiguous spans of text. So you can't get output like you want using IOB labels or the NER model (or matcher) alone.

What you can do is annotate the whole span of interest (Nifty ... indices) and then use post-processing to manually convert that string to the strings you want. The grammatical structure in your example is known as "coordination" and is pretty complicated, but if you focus on a few kinds of examples you want to deal with you may be able to handle it using dependency information.

[shrinidhin]

My input is complete sentences. They are texts from news articles to be precise. I save them into .iob files and tag each token with my custom entity label for training. The other non entity ones are usually tagged 'O'.
I used spacy's tokenizer because I wanted to avoid the brackets or punctuation. If the brackets or hyphens are separate tokens, Will they be counted as delimiters which I will have to eliminate while preparing training data?

[polm]

If I were generating .iob files I would do something like this:

import spacy

nlp = spacy.blank("en") # just need a tokenizer

for tok in nlp("this is a text"):
    print(tok)

This way each token will be on its own line, and then you can add a label after it. It is important that you do not delete any tokens here.

I'm not sure what you mean by "counted as delimiters" - brackets are tokens, just like other punctuation or normal words, and they still need IOB tags.

[shrinidhin]

Thanks!This is very helpful.The problem with having punctuation tagged separately, is that the tokeniser also splits words like It's as It and 's. So wouldn't that result in a poorly trained entity recognizer?

[polm]

Sorry, I didn't see your question since it was edited in.

In IOB training data, every token needs a label, though for some tokens that label is O ("other"). For a word that is an entity in parentheses, you might have labels like this:

(    O
Paul     B-PERSON
)     O

As long as the tokenizer you use to make the training data and the tokenizer used when training the model are the same, it doesn't matter how the tokenizer works.

I don't really understand what It's has to do with anything, sorry. If you have something like Paul's then that would be two tokens and only Paul would be an entity normally, for example.

[shrinidhin]

Got it. This discussion has been very helpful. Thank you very much!