Square brackets are still not tokenized properly

[neongreen]

How to reproduce the behaviour

I found #1144, which says it's fixed, but the same problem persists.

>>> import spacy
>>> nlp = spacy.blank("en")
>>> print('  '.join([w.text for w in nlp('descent.[citation needed]fault')]))
descent.[citation  needed]fault

I expected [ and ] to be tokenized independently.

Your Environment

• spaCy version: 3.3.1
• Platform: macOS-12.6-arm64-arm-64bit
• Python version: 3.10.8
• Pipelines: en_core_web_trf (3.3.0)

[adrianeboyd]

#1144 says that you can configure this however you want through the tokenizer settings. This is the updated URL: https://spacy.io/usage/linguistic-features#native-tokenizers

It's definitely possible to split on [ and ] as infixes by modifying the tokenizer settings. These are the closest examples in the docs: https://spacy.io/usage/linguistic-features#native-tokenizer-additions

[neongreen]

The existing prefix and suffix lists already include \[ and \], so I'm not sure how exactly the tokenizer settings could be modified to make it work.

FWIW in my own pipeline I ended up just giving up (after spending about an hour on it) and resorted to a retokenizer. The link is here, in case somebody else stumbles upon the same problem. oversett/oversett@c64e425#diff-451f5ba017c5f8d2306bd266927b94dfba0ec249e9371236ecb133b71d220bc6R11-R22

[adrianeboyd]

You'd need to add [ and ] to the infixes (not prefixes or suffixes) for this to work.

[neongreen]

Ok alright, guess I misunderstood what prefixes/suffixes are.