Unable to set entity information for token 2 which is included in more than one span in entities, blocked, missing or outside

[martinjack]

Hi. When trying train model NER, i get error (Unable to set entity information for token 2 which is included in more than one span in entities, blocked, missing or outside.).
Raw string: obl. Kiev, m. Kiev, RUSANІVS''KA NABER., 4/1, kv. 90

How to make work train, when exists duplicate name?

[polm]

You can have duplicate names - that is, more than one instance of the same word in a document. But you can't have two labels on the same token, like you can't say that "John" in "His name is John Smith" is both a PERSON and LOCATION, or that "John" and "John Smith" are separate entities.

If you data has conflicting information like that, it's up to you to decide how to resolve it. Usually you'll have to discard some labels using rules you decide.

What is the actual code/data that gave you this error?

[harshchavhan]

Hi @polm is there any way to train spancat model, having labels with overlapping spans ?!

[polm]

Yes, if you save the annotations you have in spans then it's OK for them to be overlapping for a spancat. For a guide to spancat see the blogpost.

However, if you are surprised to have overlapping spans, you should look at your data and figure out why that's the case.

[NaelsonDouglas]

Hi @polm is there any way to train spancat model, having labels with overlapping spans ?!

Check this out

https://cees-roele.medium.com/detecting-toxic-spans-with-spacy-c5533786bbf8

[umayerr]

Hi @polm I am working with a set of chemical data which has nested annotations. I annotated both span and overlapping span with the same label. For example, "steroid" and "steroid hormones" in the sentence "... for the hydroxyls on ring D of such steroid hormones." are annotated with the a same label ENTITY. But I am getting the same error. Is this because of the same reason? If so, we have to deal nested annotation with spancat approach, is that right? Thank you!

[adrianeboyd]

Have a look at this project, which compares ner and spancat for biomedical data: https://github.com/explosion/projects/tree/v3/experimental/ner_spancat_compare

For more specific questions, please open a new thread.

[martinjack]

Hi, on step doc_to_bin getting error

def get_doc_bin(training_data,nlp):
    # the DocBin will store the example documents
    db = DocBin()
    for text, annotations in training_data:
        doc = nlp(text.lower()) #Construct a Doc object)
        ents = []
        items = []
        for start, end, label in annotations:
            span = doc.char_span(start, end, label=label)
            if span is not None:
                ents.append(span)

        doc.ents = ents
        db.add(doc)
    return db

Entity list:
[("obl. Kiev, m. Kiev, RUSANІVS''KA NABER., 4/1, kv. 90", [(0, 4, 'RegionType'), (5, 9, 'Region'), (11, 13, 'LocalityType'), (5, 9, 'Locality'), (33, 39, 'StreetType'), (20, 32, 'Street'), (41, 44, 'HouseNumber'), (46, 49, 'ApartmentType'), (50, 52, 'Apartment')])]

Entity list after docbin:
[obl., kiev, m., kiev, naber., rusanіvs''ka, 4/1, kv., 90]

My config file.
config.txt

My raw data
raw.csv

[ljvmiranda921]

Hmm, yeah, it seems like you're setting two entities (Locality and Region) in the span index 5 to 9. As mentioned, you need to decide which entity should be assigned to Kiev.

[martinjack]

Hmm, i try later use rules

If you data has conflicting information like that, it's up to you to decide how to resolve it. Usually you'll have to discard some labels using rules you decide.

[martinjack]

Thanks, @ljvmiranda921, @polm your help.

After creation entity_ruler, is all work

[vijetadeshpande]

I am facing the same issue in a nested NER problem that I am working on. Is there any way we can put multiple entity-tags for a single token?

e.g,
token in the text is: 'glioma'
tags can be: 'tumor', 'brain-tumor', 'astrocytoma'

Something like it.

[ljvmiranda921]

You might want to take a look at the Doc.spans property and using SpanCategorization as your baseline. We do have an upcoming spaCy tutorial project (still In review) that demonstrates how to do it in a nested NER dataset (GENIA).

[vijetadeshpande]

Thanks for sharing, I'll check it out!