Custom sentence recognize (senter)

[nopperguin]

How to write training data for and train a custom sentence recognizer?
All other pipeline components should not be impacted.

[adrianeboyd]

See some suggestions:

• Sentence segmentation performance #8532 (reply in thread)
• Re-Training SentenceRecognizer #6926

The senter is updated independently during training by default, but when you run the full pipeline, there can be interactions between the senter and other components that set sentence boundaries, in particular the parser.

[nopperguin]

Thanks @adrianeboyd. I tried suggested methods and I found they all need to train the senter from scratch. Is there any way I can just update the senter component in the pipeline? If so, how to write training data for this?

Below is the way I am trying, But sentence does not split at all. Am I going the correct way? Do I need to add more examples? Or this is not the way to do this.

`import spacy
from spacy.tokens import Doc, DocBin
from spacy.training import Example

nlp=spacy.load("en_core_web_sm")

examples = []
doc = Doc(nlp.vocab, words=["ping", "the", "node","and", "reboot"])
gold_dict = {
"sent_starts":[1,-1,-1,1,-1]}
examples.append(Example.from_dict(doc, gold_dict))
doc = Doc(nlp.vocab, words=["ping", "node","N1","and", "reboot"])
gold_dict = {
"sent_starts":[1,-1,-1,1,-1]}
examples.append(Example.from_dict(doc, gold_dict))
doc = Doc(nlp.vocab, words=["ping", "and", "reboot"])
gold_dict = {
"sent_starts":[1,1,-1]}
examples.append(Example.from_dict(doc, gold_dict))
doc = Doc(nlp.vocab, words=["ping", "node","and", "reboot"])
gold_dict = {
"sent_starts":[1,-1,1,-1]}
examples.append(Example.from_dict(doc, gold_dict))
nlp.update(examples)
new_doc=nlp("ping node and reboot")

for s in new_doc.sents:
print(s)`

Thanks in advance

[adrianeboyd]

Extending the senter from en_core_web_sm is a bit trickier than it looks at first because it's disabled by default in en_core_web_sm. As a result, it's not being updated during training or applied with nlp(text) in your code above.

I would recommend creating a separate pipeline just with the senter component for training:

nlp = spacy.blank("en")
nlp.add_pipe("senter", source=spacy.load("en_core_web_sm"))
nlp.to_disk("/path/to/en_senter")

You can update this pipeline with nlp.update to test it out, but for training scenarios with more than a handful of examples, you'll get better results with spacy train, which handles shuffling, batching, and a number of other training details for you. You can create .spacy files for training using the suggestions for Doc.from_docs in the linked threads above. The training data is pretty easy to create (run the existing senter to print out one sentence per line and correct the line breaks by hand with a text editor) and I'd recommend having at least 500+ examples to start.

Create a config:

spacy init config -p senter senter.cfg

Edit the config to source the existing senter rather than training from scratch:

[components.senter]
source = "/path/to/en_senter"

# remove the rest of the senter config!

[corpora]

Train:

spacy train senter.cfg --paths.train train.spacy --paths.dev dev.spacy -o /path/to/en_modified_senter

Then you can replace the senter in en_core_web_sm with it:

nlp = spacy.load("en_core_web_sm")
nlp.remove_pipe("senter")
nlp.add_pipe("senter", first=True, source=spacy.load("/path/to/en_modified_senter"))
nlp.to_disk("/path/to/en_core_web_sm_modified_senter")

You want to be sure the senter runs before the parser or the parser will set all the sentence boundaries and the senter won't do anything.