Textcat on large documents - best practice for performance

[NixBiks]

Hi,

I have a textcat task on large documents where speed/performance is important. As the Explosion team point out in the docs, the textcat can struggle on large documents so it might be a better idea to split the document up into smaller paragraphs and do some aggregation. My concern is that it's slower to create Doc for each paragraph. I see the following possibilities

1. Create a Doc for each paragraph and run the textcat on each doc.
2. Have a single Doc and run a trained SpanCatetegorizer on each paragraph using a custom suggester function.
3. Something else...

I'd like to get some feedback on what is generally recommended if that's possible.

[pmbaumgartner]

Hey @mr-bjerre - typically with longer documents, I take the approach of breaking things down to the sentence level and classifying there, then aggregating those predictions to the whole document level. The assumption there is that typically always at least one sentence that indicates how a document should be classified. You could also use the predicted probabilities of each sentence to do something more complicated when labeling the entire document.

I should not in general it's hard to give general recommendations for something like this, it can depend heavily on the content of your documents. Do you have any more details about your problem that you can share?

[NixBiks]

Hey Peter,

Thanks for your comments. So if you would classify each sentence - would you use the SpanCategorizer or use a TextCategorizer for that part?

To me it isn’t clear if SpanCategorizer is meant for those type of tasks

[pmbaumgartner]

Without knowing more about the type of text, I'd go with the TextCategorizer. The SpanCategorizer is still experimental and when annotating data it can be hard to determine what the boundaries are for the relevant spans. With single sentences its easier to make a labeling decision.