Custom behavior of HYPH, the "-" symbol

[antonpibm]

Hello.

I try to analyze a text where the hyphen ("-") symbol has special meanings. I would like to achieve to specific behaviors, preferably with Spacy's built in pipes and also preferably without having to use string.replace function.

1. When a hyphen appears with leading and trailing spaces mark it with a POS (or tag or dep) other than "HYPH" e.g. "ADP" or "AUX", example "Our best employee - Bob"
2. When a hyphen has no leading and no trailing space, combine all the words together into a single token, if not possible set the hyphens to other POS than "HYPH"

I've tried "add_special_case" for (1) with setting " - " to ORTH: " - " but it had no effect, as it looks like "-" is represented as a token after the tokenizer, and it looks like there is no way to differentiate on whether it has spaces or not, I don't know which other pipe could be relevant.
I've tried attribute_ruler and merge_subtokens for (2) but couldn't achieve what I was hoping for.

Any idea on how to achieve this?

[adrianeboyd]

The feature for whether there's a following space is called SPACY:

matcher.add("TRAILING_SPACE", [[{"SPACY": True}]])

add_special_case will only work in v2 not v3, if you're setting anything other than ORTH and NORM (in v3 there should be an error if you try, so I don't think it will fail silently). In v3, you can use an attribute ruler to set annotation or a custom component if you want to retokenize in addition.

[antonpibm]

Thx a lot! That worked like a charm, eventually using text "-" and SPACY for (1) was enough, so I split the occurrences of "-" with:

{
"patterns": [[{"TEXT": "-", "SPACY": True}]], "attrs": {"POS": "ADP", "DEP": "prep"}
},
{
"patterns": [[{"TEXT": "-", "SPACY": False}]], "attrs": {"DEP": "compound"}
},

then as you've suggested merge them:

from spacy import Language
@Language.component("combine_dash")
def set_combine_dash(doc):
start = -1
with doc.retokenize() as retokenizer:
for token in doc[1:-1]:
if token.text == "-" and token.dep_ == "compound" and start == -1:
start = token.i - 1
elif start != -1 and token.text != "-" and doc[token.i + 1].text != "-": # Handle sequence of dashes
retokenizer.merge(doc[start:token.i+1])
start = -1
if start != -1:
retokenizer.merge(doc[start:])
return doc

sorry for bad indentation