Spacy 3 Sentence Segmentation

[Phat-Loc]

I want to fine tune the Sentence Segmentation functionality for my use case. Namely I want to handle cases where "XYZ Inc. " is treated as a sentence end but it really is just and abbreviation of Incorporated. What is the recommended way of doing this? Are there any tools in either Displacy or Prodigy that can visualize or annotate this.

1. Add a custom function? If so where in the pipeline e.g. before parse?
2. Train the SentenceRecognizer?
3. Train the dependencyparser? If so how are there any examples of fine tuning an already existing model such as en_core_web_sm?

Thanks!

[polm]

I think your best option at the moment is trying to train the SentenceRecognizer.

Using a custom function to set is_sent_start = False on abbreviations you're sure aren't final sentence words is also an option, but currently there is a bug with setting only some of the is_sent_start flags (#7716), so that might not work as expected.

Training the dependency parser would require a lot of hard-to-build data, so while it would work I imagine getting the data would be a problem. We don't have any examples of fine-tuning the dependency parser partly because our primary training data can't be shared publicly due to licensing issues, partly because the data is hard to build, and partly because the dependency parser rarely needs fine-tuning.

There's nothing specific for sentence splitting in displaCy or Prodigy, though in Prodigy you can just mark the first token in a sentence to use as training data.

[Phat-Loc]

Thanks! I will give it a try.

[adrianeboyd]

Just a note: the bug in #7716 doesn't affect the SentenceRecognizer itself.

If you want to fine-tune the senter in en_core_web_sm, you can use a script like this to set up the config:

https://github.com/explosion/projects/blob/9d5fce5f95ddf5f35c3370b2074b25e995525f51/pipelines/ner_demo_update/scripts/create_config.py

This was written with ner in mind. For senter, you'd want to enable the senter and disable the parser at the beginning of the script so that the eval you're seeing during training is just for the senter. See: https://spacy.io/models#design-modify

If you're a current prodigy user, you can request access to the nightly version that has new sent.correct and sent.teach recipes for creating training data for the senter component: https://support.prodi.gy/t/prodigy-nightly-spacy-v3-support-ui-for-overlapping-spans-improved-feeds-more/3861