Can the white space in Chinese segmentation be retained?

[lingvisa]

For example:

贺峻霖 心动

Although the doc.text keep the white space, the token list doesn't. Instead it uses the 2nd token's (心动) start position to indicate that there is a white space preceding it (start+1) . In Chinese social media, especially Weibo, users often use a white space as a delimiter to conceptually isolate words, which can be a good clue for word boundary, as shown in the example above. After the white space is removed, how to conveniently check there is a space after the token "峻霖"? Is it possible that white spaces are retained as a normal token in Chinese language?

[polm]

spaCy doesn't remove spaces - for ASCII spaces, the first one after a word can be recovered using the token.whitespace_ attribute. For other kinds of spaces (like full-width spaces), or multiple spaces in a row, they are preserved as tokens. You asked about this before in #8879.

[lingvisa]

I understand now. For the example above, the segmentation would be:
`[贺 峻霖 space 心动]'

So bool(token[1].whitespace_) = True and everything else will be False.

That's good enough and thanks!

[lingvisa]

One more question on this: For '我们是' (We are), the idx of 是 is 2, and '我们 是', the idx of 是 is 3 due to the white space. Is this intended?

[polm]

Yes, that's intentional - idx is the index in the original string. For any token, if text is the input to nlp, token.text[0] == text[token.idx].