Extending the Sentencizer | Custom rules

[kapilok2014]

Hi SpaCy Experts,

We have tested and compared the default sentencizer (parser), senter and SentenceRecognizer.
We don't want to split on Abbreviations (e.g. Govt. Inc. etc.) or bullet points [ a. b). etc. ]
Our splitting logic is mostly rule-based (abbrev. lookup OR regex) and we think the model-based senter will be too much effort to customize.

Please advise how we can extend the rule-based Sentencizer with the custom logic mentioned above.

Thanks
Kapil

[polm]

If you already have working logic that includes rules, instead of using the Sentencizer you can create a small custom component that assigns is_sent_start to all tokens in a Doc. The sentencizer is only for very simple punctuation based tokenization.

We don't want to split on Abbreviations (e.g. Govt. Inc. etc.) or bullet points [ a. b). etc. ]

Note that if you're concerned about cases like that, you usually want a statistical model to handle ambiguous cases like "He works for Stuff Inc. I don't.", where an abbreviation is also an end of a sentence.