StringStore : Adding strings with a "." impacts across all stringstores

[vimal-mathew]

How to reproduce the behaviour

x1 = spacy.strings.StringStore()
x1.add("root.abc.com")
x2 = spacy.strings.StringStore()
"root" in x2 # returns True
"root" in x1 # returns True

Info about spaCy

Running on Amazon Linux EC2 instance. Also replicated same issue on a Macbook Pro (anaconda)

• spaCy version: 2.2.3
• Platform: Darwin-19.2.0-x86_64-i386-64bit
• Python version: 3.7.3

[adrianeboyd]

There are some hard-coded symbols that are always accessible through the StringStore even though they aren't actually saved in the StringStore:

import spacy
x1 = spacy.strings.StringStore()
assert len(x1) == 0
assert ("IS_ALPHA" in x1) == True

The full list is here: https://github.com/explosion/spaCy/blob/4890db63399d24f088ff6978aa157a0e4672e2eb/spacy/symbols.pxd

This is clearly confusing and hopefully at some point it will be possible to treat all of them the same way as any other string and avoid this hard-coded enum, but for now that's why you see root in every StringStore.

The StringStore doesn't tokenize or analyze root.abc.com in any way, so you should be able to see:

assert ("root.abc.com" in x1) == True
assert ("root.abc.com" in x2) == False

Whether root or abc is in the StringStore is completely unrelated to whether you added root.abc.com.

[vimal-mathew]

Thanks for the explanation