Best Practice for Large Documents

[morankyle]

Currently have a Spacy pipeline which includes various built-in pipes such as ner, parser, etc, as well a custom component for extracting SVOs and SpacyTextBlob.

The model runs into issues with very large documents, I've read Spacy needs 1GB per 100,000. Is there any best practice for splitting up documents before sending them thru the pipeline in? Or is it possible to somehow split the doc into spans and send those thru the pipeline individually?

[polm]

If you're running into memory errors due to large individual docs, usually you should be able to split them into paragraphs without a loss in performance.

This is mentioned in the same comment that suggests 1GB per 100,000 characters, actually.

[morankyle]

Awesome thank you! Actually came across that comment in a different Stack Over Flow thread so that was helpful!