Would it be faster to process entire corpus?

[the1gofer]

I have about 1600 documents that I'm wanting to process with Spacy, but it takes ages to generate the doc for that many strings. Would it be faster to combine them all into one large text then slice them up somehow down the road?

[polm]

If you're running into speed issues with lots of documents we usually recommend using nlp.pipe. From the docs:

texts = ["One document.", "...", "Lots of documents"]
for doc in nlp.pipe(texts, batch_size=50):
    assert doc.has_annotation("DEP")

How large are your individual documents?

[polm]

OK, that should be fine. Give nlp.pipe a try and let us know if it doesn't work.

[the1gofer]

is there any way to identify them as they are processed? Let's say I load 100 documents, and then I want to compare one to every other document, and 2 to every other document, and so on. When I get done I'll know that some of them are more similar than others, but it'll be difficult to know which ones specifically. If I have id's for each one, can I tie them together some so when they come out the other end, I know document one and document fifteen are the same?

is that what the context in the (doc, context) is? I don't see much about what it is specifically.

Edit: that is what that does. Figured it out.

[polm]

I'm a little confused about what you're asking and what you figured out, but just in case, here's an answer:

To add IDs to docs, you can set a custom attribute. That will let you track an individual doc through the pipeline.

"Context" usually refers to the tokens around another token, like the sentence that contains it, which is important for many kinds of predictions. Where did you see (doc, context)?

For finding most similar docs, unless you have a very small number of docs, you generally want to use some kind of approximate nearest neighbor search rather than doing a comparison with every other doc. Annoy is an easy to use library with that functionality. (100 is small enough that you probably don't need to use Annoy though.)

[the1gofer]

From n.p.pipe:

• as_tuples | If set to True, inputs should be a sequence of (text, context) tuples. Output will then be a sequence of (doc, context) tuples.

when I passed a (text, id), I got back (doc, id). Unless I'm miss-understanding what's going on here.

[polm]

Ah, OK, I didn't realize that's where that came from. Yes, you've understood correctly, context there is just passing through whatever was in the input with the output.