'spacy.tokens.token.Token' object has no attribute 'offset'

[searchivarius]

First of all, many thanks for the wonderful library.

This has been superuseful! What is a bit upsetting, though, that there is no universal support for start, end offsets among different span types. There is an explicit value of the start in the NER, but not in the token (one can apparently compute these from token_with_ws though). Furthermore, the token has something called offset, but it is not a character offset and does not seem to be written to the write attribute in the class variable: The description of the issue is below.

How to reproduce the behaviour

Documentation claims the token has an attribute offset, but I am pretty sure this is incorrect.

import spacy
nlp=spacy.load('en_core_web_sm')
doc=nlp('this is a test')
doc[1].offset

This will fail, but looking [at the code] the value of offset (https://github.com/explosion/spaCy/blob/master/spacy/tokens/token.pyx#L99) is written into attribute i instead, so the following works correctly:

doc[1].i

It is very easy to fix, just save additionally (or instead) the value of offset to the attribute offset.

Your Environment

• Operating System: Linux
• Python Version Used: 3.6
• spaCy Version Used: I tried several versions 2.2, 2.3, 3.0, this seems to work like this everywhere.
• Environment Information: I believe it shouldn't work anywhere.

[polm]

Sorry this has been confusing. Tokens have token offsets in the i attribute and character offsets in the idx attribute; the token index is only referred to as offset (variable name) in the docs when it's an argument to the Token constructor. There is no guarantee that constructor arguments will have the same name when used as attributes.

You can find the list of attributes on the Token at the bottom of the page you linked to, which includes i and idx.

If that doesn't answer your question or I've misunderstood you let me know.

[searchivarius]

Hi @polm thank you very much for clarifying, I guess I haven't read docs very carefully. idx does work without a problem.