How to use Cython to speed up pipeline of model inference

[vsocrates]

I have a pipeline of 3 transformer-based textcat components. I followed the recommendations here and am using pipe after disabling all components except my textcat component, but inference is exceedingly slow (especially when the whole pipeline is Dockerized). I am wondering how I might use Cython to speed this up.

At the following link, it seems like the Cython API isn't available until after you run text through the documents.

Is there another way to improve speed for multiple transformer-based models pipelined in sequence? Thanks!

[adrianeboyd]

I suspect that cython isn't going to help much here, since most of the time is spent in the transformer component.

Have you evaluated the performance of textcat with word vectors (along the lines of spacy init config -o accuracy -p textcat) rather than transformers? Or evaluated your task with something like fastText?

For text classification, you can often get competitive performance with simpler and faster approaches. I can find several recent papers on this topic, just to pick one relatively recent one: https://aclanthology.org/2023.acl-short.160.pdf

Especially if you're only doing text classification and not using any other token-level tasks like parsing or NER, I'd definitely recommend evaluating some of the (fast) options outside spacy. If you'd still like to have a Doc in the end, you can still set Doc.cats easily from the output of an external component.