Can't create char_span if length of char 1

[Artem-Haholkin-deepsee]

When I'm creating Span from a doc I want to create a span of length 1. But it returns None instead of span

  model = spacy.load(en_core_web_trf,
                   disable=[
                       'tok2vec',
                       'tagger',
                       'morphologizer',
                       'parser',
                       'attribute_ruler',
                       'lemmatizer'
                   ])

  ruler = model.add_pipe("entity_ruler", before="ner")
  ruler.add_patterns([
          {"label": "pattern", "pattern": [{"LOWER": c}]}
          for c in list_of_ents
      ] )
doc = nlp(text)
span = doc.char_span(start, end)

As I understand something wrong with tokens/chars
When I do doc[start:end] it throws an error not enough tokens to unpack. But I expect it won't be tokens but chars
According to documentation getting one item gives a tokens. But i think it should work another way for char_span

Environment

OS: MacOS Monterey 12.4
Python: Anaconda 3.8.12
Spacy: 3.0.6

[rmitsch]

Hi @Artem-Haholkin-deepsee, please provide a complete, minimal example so we can reproduce your issue.

[Artem-Haholkin-deepsee]

@rmitsch
List of tokens looks this way
[..., 'n=1', ...]. There are about ~175 tokens in a document
So I'm trying to extract '1'. Char index is 503.
And I want to to make a span with this char span = doc.char_span(503, 504, label=397) and getting nothing

[rmitsch]

By default, Doc.char_span() strictly snaps to token boundaries and returns None if the specified indices don't align with identified tokens (see documentation). In your case "1" is likely not the entire token, hence None is returned. You can change this behavior by choosing another alignment_mode. Example:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
print(doc.char_span(0, 4))  # "This" 
print(doc.char_span(0, 3))  # None
print(doc.char_span(0, 3, alignment_mode="contract"))  # None
print(doc.char_span(0, 3, alignment_mode="expand"))  # "This"
print(doc.text[0:3])  # "Thi"

Note that even other alignment modes considers Token boundaries. If you just want the characters without consideration for tokens, I'd recommend the last option (slicing on doc.text). This depends on your use-case though.

[Artem-Haholkin-deepsee]

As I understood there is no way to get span from one char inside one token
The last option isn't good for me because slicing one char doc.text[0:1] returns token, not char
@rmitsch Thanks for answering.

[rmitsch]

You could also build a Span object yourself. What do you want to do with the 1-char span?

[Edit] Moving this to discussions 🙂