Spacy Misaligned Entities

[ambuje]

Hi All,

This is regarding:-

"UserWarning: [W030] Some entities could not be aligned in the text "はい . シマムラ ヒロユキ 様 、 本日 は ありがとう ござい ます ." with entities "[(5, 13, 'PERSON')]". Use spacy.gold.biluo_tags_from_offsets(nlp.make_doc(text), entities) to check the alignment. Misaligned entities (with BILUO tag '-') will be ignored during training."

Is it possible to get all the misaligned data points in a form of a list? So that one can go through each text and correct this warning.

Thank you,

[polm]

What code are you using that gives that error, and what spaCy version are you using? It looks like you're using v2.

There's no single function that shows every example of misaligned text, but you can just check the output for "-" tags to see if it's misaligned.

In v3 there's some more versatile ways of handling this, though if you're having alignment issues you need to look at why. In this case it looks like your annotation is シマムラ ヒロユ, which is missing the final character, so maybe you have inclusive character offsets - if that's the case you can fix it by adding one to your final offset for everything. But your text also contains spaces after every word, which is weird for Japanese, so maybe those were added automatically and are causing the issue.

[ambuje]

Yup I am using spacy v2.

Thank you for the reply. I was able to check if the text is misaligned.

[ambuje]

Hi All,

I have written a function that can help you to know whether the data point is misaligned.

import pandas as pd
def misaligned_data(data):
    from collections import Counter
    l=[]
    f_t=[]
    ent=[]
    for i in data:
        txt=i[0]
        e=i[1]['entities']
        f_t.append(txt)
        ent.append(e)
        f=spacy.gold.biluo_tags_from_offsets(nlp.make_doc(txt),e)
        if '-' in f:
            l.append(0)
        else:
            l.append(1)
    dt=pd.DataFrame(
    {'Text': f_t,
     'Entities': ent,
     'Label': l
    })
    print('Distribution',Counter(l))
    return dt  

This function will return a data frame with "Text, Entities, and Label" columns. The Label column will have 0 and 1 as a value. 0 will indicate that the text is misaligned.
From this function, one can get the misaligned data points.

Input: Spacy training format
Example: -
[('Customer:09234054-2378ですはい宅井ですはいそうです年間にすると支',
{'entities': [(26, 28, 'PERSON')]}),
('はい . 石川 様 、 注文 番号 が こちら システム の 方 で 確認 が 取れ まし た . ただ です ね 、 こちら の ご 購入 、 保証 延長 サービス の 対象 外 に なっ て しまう ん です .',
{'entities': [(5, 7, 'PERSON')]})]

Thank you,