Is there a limit on the number of documents spaCy can handle?

[echatzikyriakidis]

Is there a limit on the number of documents spaCy can handle?

[svlandeg]

In theory, no, though ofcourse any script you execute will always be bound by the physical constraints of the machine it's running on. Any particular reason why you're asking?

If you plan on processing many texts, we definitely advise using nlp.pipe, which returns a generator of Doc objects. See also "Tips for efficient processing".

[echatzikyriakidis]

@svlandeg

I am asking this just to understand better my limitations with spaCy.

So, nlp.pipe is a way to parallelize multiple nlp(doc) calls? What if I use a different library for this? For example, I have my data in a dataframe and I use pandarallel. I parallelize nlp(doc) in different cores.

Is there any other advantage of using nlp.pipe?