question regarding splitting corpus

[fschwiet]

First, thanks for putting this corpus together. I was hoping to ask a question.

I want to run some code against the corpus, but its problematic to load the entire files into memory as individual strings and run the code (using NLP tools, they tend to run out of memory). So I want to split it up without causing in splits at sentence boundaries. So I am reading small chunks via a text stream, splitting on a text pattern.

Is it safe to split on newlines (\n)? Or could that be splitting sentences? Or to paraphrase, are any sentences broken up across lines in the files?

Sentence boundaries can be difficult to detect, but it seems like this corpus uses a single line per paragraph (like example A) but it does not seem the corpus splits sentences to fit text in a page (like in B). I was hoping you could confirm.

A:

Sentence boundaries can be difficult to detect, but it seems like this corpus uses a single line per paragraph (like example A) but it does not seem the corpus splits sentences to fit text in a page (like in B).

B:

Sentence boundaries can be difficult to detect, but 
it seems like this corpus uses a single line per paragraph 
(like example A) but it does not seem the corpus splits 
sentences to fit text in a page (like in B).