ValueError: [E199] Unable to merge 0-length span at doc[23:23].-- Problem with Spacy Custom Spacy Components

[connersdavis]

How to reproduce the behaviour

I have written a custom spacy NER function seen below, which I add my pipeline. Though it seems to work well in most instances, I have been debugging a few cases in my code-- when I came across this weird bug. While debugging my own code, I instantiated a new spacy model-- without adding my custom component.

-- nlp_spacy = spacy.load('en_core_web_sm', disable = ['ner'])

(...)

then I run the 'problem' text on the model

-- doc = nlp_spacy(': the noted that in late january he began to have a return of bloating and looser stools, but not diarrhea')

then use this doc to debug my custom component.

When I run my custom component once, I receive the above error 'ValueError: [E199] Unable to merge 0-length span at doc[23:23].-' and nothing is returned.

If I run the custom component again, and everything seems to work fine-- no error, etc...

@Language.component("Multi_Word_NER")

def Multi_Word_NER(doc):

diarrhea_bag_of_words =  ['(diarrhea)', 
            '(water[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))',
            '(liquid[a-zA-Z]*[ ]{1,}(stoo[a-zA-Z]*l|bm[a-zA-Z]*|bowel[a-zA-Z]*[ ]{1,}movement[a-zA-Z]*))', 
            '(loose[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))', 
            '(frequent[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))', 
            '(semi[ ]{1,}form[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))',
            '(unform[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))', 
            '(poor[a-zA-Z]*[ ]{1,}form[a-zA-Z]*(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))'
               ]

diarrhea_negation =  ['(?:[ ])(form[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))']
                   
diarrhea_implied = ['(constipa[a-zA-Z]*)', 
                  '(normal[a-zA-Z]*[ ]{1,}(stool[a-zA-Z]*|bm[a-zA-Z]*|bowel[ ]{1,}movement[a-zA-Z]*))']

bag_of_words_gen = ['(bm[a-zA-Z]*)', '(bowel[ ]{1,}movement[a-zA-Z]*)', '(stool[a-zA-Z]*)']

positive_indicators = [r'loose[a-zA-Z]*', r'water[a-zA-Z]*', r'frequent[a-zA-Z]*', r'liquid[a-zA-Z]*', 'semi[ ]{1,}form[a-zA-Z]*',  r'unform[a-zA-Z]*']

adj_negation = [r'^form[a-zA-Z]*']

count = 0
#original_ents = list(doc.ents)

for ent_category, Label in zip([diarrhea_bag_of_words, diarrhea_negation, bag_of_words_gen, diarrhea_implied], ['Diarrhea', 'Diarrhea_neg', 'Diarrhea_gen', 'Diarrhea_imp']):
    
    original_ents = list(doc.ents)

    mwt_ents = []
    indexes = []
    
    Label_gen = []
    
    
    
    for regex in ent_category:
        for match in re.finditer(regex, doc.text, re.IGNORECASE):
           
            
            
            if Label == 'Diarrhea_gen':
                #print('here', [word.text for word in doc])
                
                pos_result = check_word(positive_indicators, [word.text for word in doc])
                
                neg_result = check_word(adj_negation, [word.text for word in doc])
             
                
                
                if pos_result[0] == True:
                    Label_gen.append('Diarrhea')
                    
                elif neg_result[0] == True:
                    
                   
                    Label_gen.append('Diarrhea_neg')
                else:
                    continue
                    
        
            start, end = match.span(1)
            
            span = doc.char_span(start, end, alignment_mode = 'expand')
            
            
            
            if span is not None:
                mwt_ents.append((span.start, span.end, span.text))
              
                token_locations = (span.start, span.end)
                indexes.append(token_locations)
              
    
    
    mwt_ents.sort(key=lambda x : x[1])

    indexes.sort(key=lambda x : x[1])
    
    
    if len(mwt_ents) != 0 and Label == 'Diarrhea_gen':
        
        for ent, Label_g in zip(mwt_ents, Label_gen):
            
            start, end, name = ent
            per_ent = Span(doc, start, end, label = Label_g)
            original_ents.append(per_ent)
            
        filtered = filter_spans(original_ents)
        doc.ents = filtered
        
        for (start, end), Label_g in zip(indexes, Label_gen):
            start -= count
            end -= count
            with doc.retokenize() as retokenizer:
                #try:
                    
                attrs = {'POS': 'NOUN', 'ENT_TYPE': Label_g}
                retokenizer.merge(doc[start:end], attrs = attrs)
            #except:
                    #pass
            
    
    elif len(mwt_ents) != 0:
        for ent in mwt_ents:
            start, end, name = ent
            per_ent = Span(doc, start, end, label = Label)
            original_ents.append(per_ent)


        filtered = filter_spans(original_ents)
        doc.ents = filtered
        
        for start, end in indexes:
            print(start, end)
            with doc.retokenize() as retokenizer:
                #try:
                attrs = {'POS': 'NOUN', 'ENT_TYPE': Label}
                retokenizer.merge(doc[start:end], attrs = attrs)
                #except:
                    #pass
            
    
    else:
        continue

return doc 

What would be the cause of this? I don't understand how this could be an issue on my end. For now, as a work around, it seems that if I nest the retokenize section of the function in try statements, this issue doesn't occur. Attached are two pictures, the results after I run the first, then the second.

Your Environment

• Operating System: Microsoft Windows Server Version 1607 (OS Build 14393.5291)
• Python Version Used: '3.6.8'
• spaCy Version Used: 3.0.7
• Environment Information: Jupyter Labs

[polm]

Like the error states, you can't merge a span of zero length into a new token - it's unclear what that should do, besides nothing. Here's a shorter example that gives the same error:

import spacy

nlp = spacy.blank("en")

doc = nlp("fish cat")
with doc.retokenize() as retok:
    retok.merge(doc[0:0])

I am not exactly sure how you are getting zero-length spans in your code, but you can check spans before merging to see if they have zero length and skip them.

Also, to make it easier for us to help you, if you'd like to share mode code, please read the Github Markdown guide and don't share screenshots of code or errors, paste them as text.

[connersdavis]

Thanks for your help and suggestions on markdown. I fixed one error and now a new one presents. -- ValueError: [E093] token.ent_iob values make invalid sequence: I without B but|O on|O bad|O days|O has 12|O 15|I bowel|O movements|O and|O generally|O- I have isolated the problem down to the section of my code that deals with retokenizing each entity, so that each entity is one token. Here is the code:

count = 0
 for start, end in indexes:
        if end - start != 0:
            start -= count
            end -= count

            with doc.retokenize() as retokenizer:
                attrs = {'POS': 'NOUN', 'ENT_TYPE': 'range'}
                retokenizer.merge(doc[start:end], attrs = attrs)
                count += end - start 

        else:
            pass

As I said, this section seems to be the issue, if I comment it out-- everything seems to work fine, but without the entities being single tokens. Each set of indexes corresponds to the token locations of each entity. Right now, I am accounting for the doc token shrinkage by keeping track of the count, then subtracting it from the start and end positions. Not sure how to proceed-- wondering if you had any ideas.

[polm]

It's hard to tell what's going on here without knowing what indexes are, and without the original formatting on the error output. Could you use print or something to record the values for the inputs that are giving you the errors and give a short, self-contained example we could run to check what you're doing?

Looking at has 12|O 15|I bowel|O, looks like you have an I (inside) tag without a B (beginning) tag, which is not valid. I am not sure how you got that, as the retokenizer knows how to merge entity tags to keep them valid.

Also, just to be clear, it looks like you're trying to merge entities into one token. If that's the case there's a builtin function you can use called merge_entities. If what you're trying to do is not exactly merging entities, then the definition of that function may be a useful reference.

[connersdavis]

you know what... I just realized the issue is that I was attempting to retokenize ents that already were a single-token long. As soon as I filtered out entities that were one-token already, this error went away. Thank you for your help @polm!

[polm]

Glad you figured it out!