Performance with nlp.pipe

[Uzay-G]

Hello,
I'm trying to write a pipeline to process documents of the following scale: upper hundreds of documents, total ~1.5M words.

I've been using nlp.pipe and disabling the language features I don't need.

However, it seems now that when I process a large batch with nlp.pipe simply too much memory is used and the Jupyter kernel (I'm running this in a Jupyter notebook) dies.

When I process the documents individually with nlp(doc) it works, albeit much slower.

Is there a way I could fix this? Maybe instead of calling nlp.pipe on all my documents, batch them into smaller collections (like 15 docs), and then process those with nlp.pipe?

[polm]

Have you tried passing a batch_size to nlp.pipe?

Not sure what environment you're developing in, but note that Jupyter kernels often set a memory limit lower than the system memory, so you might want to look into adjusting that.

upper hundreds of documents, total ~1.5M words.

Does that mean each document is around 10,000 words? That's pretty long - you might find it easier to work with documents if you slice them into paragraphs or other sub-units.

[Uzay-G]

Thanks for the advice! I had learned about nlp.pipe through (https://spacy.io/usage/processing-pipelines) and hadn't noticed the batch_size parameter. Thanks a bunch :)

I'll also look into splitting documents further if need be.