Document length incorrect? Can't create span.

[grinay]

I'm creating test data and faces into the issue with doc.char_span method, I start dig deeper and found interesting thing.
Environment details:
Macos Monterey 12.4 MacBook Pro (13-inch, M1, 2020)
spaCy version 3.4.1
Platform macOS-12.4-arm64-arm-64bit
Python version 3.9.13

import spacy
from spacy.tokens import Span

data=('1616 Chapala St, Ste 2', [(0, 22, 'ADDRESS')])

nlp = spacy.blank("en");

doc = nlp(data[0]);
anno=data[1][0]

new_ent = Span(doc, anno[0], anno[1], label=anno[2])
print(new_ent)

Lead to next error

IndexError Traceback (most recent call last)
Input In [282], in <cell line: 11>()
8 doc = nlp(data[0]);
9 anno=data[1][0]
---> 11 new_ent = Span(doc, anno[0], anno[1], label=anno[2])
12 print(new_ent)

File ~/miniforge3/envs/mlp/lib/python3.9/site-packages/spacy/tokens/span.pyx:101, in spacy.tokens.span.Span.cinit()

IndexError: [E035] Error creating span with start 0 and end 22 for Doc of length 6.

[polm]

When you create a span using the Span(...) constructor, the indices are token indices, not character indices like in Doc.char_span. This is why the Doc in your example is reported as having length 6.

[grinay]

@polm , thanks for the fast answer, yes my fault here.
Can you help to find in this scenario, that exactly where I getting error at first place.

import spacy
from spacy.tokens import Span
sentence="""
_**Able here PM tough.**_ ```worry Enter college accept admit small. He seek owner
1616 Chapala St, Ste 2.wife food light degree man. ``` ![Which how bill tree morning do list cause.](https://picsum.photos/331 "Say PM same at simply. Particularly hot maybe address. Government fund somebody month hour score next. Condition community really race modern measure. Floor impact late.") _**Training use affect individual effect particular.**_ **common** |Week onto quickly sea.|Head six air management.|Ground protect treat fight figure indicate.| |----------------------|------------------------|-------------------------------------------| |Participant beautiful director health.|Part operation myself.|Why tend cell standard difficult cut school but.| |Step keep service win.|Push consider question.|Student test garden bank memory why energy.| |Rule hand she fact show.|Action head nor for.|Hope consumer free often know join.| |Money high do.|Mention someone minute.|Be country senior myself special.| |Each someone product I guess.|View walk white design same military.|Hope perhaps out agent represent manager.| |-----------------------------|-------------------------------------|-----------------------------------------| |Ago feeling big activity while radio season.|Half any call walk officer method.|Top blue lay any job business real last.| |Piece feel argue necessary manage cup better.|Throughout best education picture behind system according.|Floor prove by human court information.| |Strategy success sea visit fight attack indeed.|A popular church material.|Rather cold represent participant cell.| |What evening drop box way.|Look similar majority popular animal.|Around run part so up.| > Media whole collection fund paper. _**Organization painting name according two daughter film about.**_ Half fill society health.
"""

data=(sentence, [(83, 105, 'ADDRESS')])

nlp = spacy.blank("en");

newdoc = nlp(data[0]);
anno=data[1][0]
print(sentence[anno[0]:anno[1]])
print(newdoc.text[anno[0]:anno[1]])

span = newdoc.char_span(anno[0], anno[1], label=anno[2])
print(span)
doc.ents = [span]
print(doc.ents)

it outputs address correct from first two print, but fails on creating span and output None
1616 Chapala St, Ste

1616 Chapala St, Ste
None

leads to :

TypeError Traceback (most recent call last)
Input In [339], in <cell line: 19>()
17 span = newdoc.char_span(anno[0], anno[1], label=anno[2])
18 print(span)
---> 19 doc.ents = [span]
20 print(doc.ents)

File ~/miniforge3/envs/mlp/lib/python3.9/site-packages/spacy/tokens/doc.pyx:757, in spacy.tokens.doc.Doc.ents.set()

File ~/miniforge3/envs/mlp/lib/python3.9/site-packages/spacy/tokens/doc.pyx:1926, in spacy.tokens.doc.get_entity_info()

TypeError: object of type 'NoneType' has no len()

[grinay]

Ok seems I understand, so my address should be wrapped with spaces, to order it align exactly into one token. otherwise it considered text like address:My streee etc... address:My considered as one token and can't be extracted to span.

[polm]

You can extract a single token as a span. If you try to extract something with doc.char_span that doesn't align to a token, it will be None, though you can modify it to expand or contract, see the alignment_mode option.

The issue with your annotation is that you include a space at the end. The newline at the start also shouldn't really be in the entity.

Here are some examples that might help make it clear:

import spacy

nlp = spacy.blank("en")
doc = nlp("hot dog")

# This is a whole token so it's fine
span1 = doc.char_span(0, 3)
print(span1)  # => "hot"

# This is part of a token, so it's None
span2 = doc.char_span(0, 2)
print(span2)  # => None

# This is half a token, but it expands, so it's fine
span3 = doc.char_span(0, 2, alignment_mode="expand")
print(span3)  # => "hot"

# This starts with a space, so it will return None
# if you use "contract" it will give the token
span4 = doc.char_span(3, 7)
print(span4)  # => None