Rule-based matching should be able to specify beginning/end of sentence

[jl-ai]

Feature description

I would like to match token sequences if and only if the occur at the end (or more rarely) the beginning of a sentence. (This would let me match headings like "Risk factors:\n" but avoid non-heading sentences like "here are the risk factors: financial challenges, history of domestic violence.")

Ideally it would look like this:

pattern = [{"LOWER": "risk"}, {"LOWER": "factors"}, {"TEXT": ":"}, {"END_OF_SENT": True}]
matcher.add("RiskFactorsHeading", None, pattern)

I tried to get clever with regex'es and do something like this:

pattern = [{"LOWER": "risk"}, {"LOWER": "factors"}, {"TEXT": ":"}, {"TEXT": {"REGEX": "."}, "OP": "!"}]
matcher.add("RiskFactorsHeading", None, pattern)

but that didn't seem to work.

Could the feature be a custom component or spaCy plugin?

Not as far as I can tell.

[adrianeboyd]

Currently the beginning of the sentence is supported as {"IS_SENT_START": True}. There was a bug in v2.2.3 that has been fixed in v2.2.4 for this feature.

There is a PR (#4697) for a similar SENT_END feature, which we need to have another look at, since I think it was 99% finished.

[jl-ai]

Ah, that's great news. Thanks. (I hadn't seen this on the docs or in the demo.) That's great news.

[delzac]

@adrianeboyd I can get {"IS_SENT_START": True} to work and that's great. But i get an exception for {"IS_SENT_END": True}. Has IS_SENT_END been implemented or am i missing something?

[delzac]

from #5375, it seems like IS_SENT_END has been removed from the Matcher schema. Anyone has any suggest on how one might be able to match for end of sentence?

[adrianeboyd]

No, because of how the information is saved internally for the doc, the Matcher doesn't have access to IS_SENT_END. Just using built-in token features, you'd have to look for whether the following token is IS_SENT_START.

Alternatively, if speed is less of an issue, you could potentially copy IS_SENT_END to a custom token extension and use that in your pattern.

[delzac]

Got it, thanks for the reply!