textcat only more accurate than transformer on classification

[fracav]

I have trained the spacy pipeline for text classification several times, always getting better results using transformer.
I'm working on a long text classification use case and it happens to me that I can't train transformer properly.

My use case considers 5 classes currently. I'm starting from a minimal test considering 60 examples per category (10 for the test).
Using the exact same data I get sooooo different results:

The textcat only pipeline quickly manages to obtain an excellent score (100%):

============================= Training pipeline =============================
[2022-02-01 17:52:22,037] [DEBUG] Loading corpus from path: test.spacy
[2022-02-01 17:52:22,038] [DEBUG] Loading corpus from path: train.spacy

E # LOSS TEXTCAT CATS_SCORE SCORE

---

0 0 0.75 4.17 0.04
0 200 36.25 93.65 0.94
0 400 2.64 96.43 0.96
1 600 0.03 100.00 1.00
1 800 0.00 100.00 1.00
1 1000 0.01 100.00 1.00
2 1200 0.00 100.00 1.00
2 1400 0.00 100.00 1.00
2 1600 0.00 100.00 1.00
3 1800 0.00 100.00 1.00
3 2000 0.00 100.00 1.00
4 2200 0.00 100.00 1.00

Now trying the tranformer based pipeline I get the following results:

============================= Training pipeline =============================
[2022-02-01 17:36:16,362] [DEBUG] Loading corpus from path: test.spacy
[2022-02-01 17:36:16,363] [DEBUG] Loading corpus from path: train.spacy

E # LOSS TRANS... LOSS TEXTCAT CATS_SCORE SCORE

---

tcmalloc: large alloc 1189552128 bytes == 0x560d6ec90000 @ 0x7f326a2c22a4 0x7f325881dfd3 0x7f325881fa9d 0x7f325881c275 0x7f325881ca1e 0x560c84b8c2ed 0x560c84c7de1d 0x560c84bffe99 0x560c84bfa9ee 0x560c84b8dbda 0x560c84bfc737 0x560c84b8dafa 0x560c84bffd00 0x560c84b8dafa 0x560c84bfbc0d 0x560c84bfa9ee 0x560c84b8dbda 0x560c84bfb915 0x560c84b8dafa 0x560c84bfbc0d 0x560c84b8dafa 0x560c84bfbc0d 0x560c84b8dafa 0x560c84bfbc0d 0x560c84bfaced 0x560c84b8dbda 0x560c84bfb915 0x560c84b8dafa 0x560c84bffd00 0x560c84bfaced 0x560c84b8dbda
0 0 0.00 0.75 0.00 0.00
1 200 0.00 152.27 0.00 0.00
2 400 0.05 147.84 16.67 0.17
3 600 0.00 168.22 17.74 0.18
4 800 0.00 169.59 17.74 0.18

Could this misbehavior be caused by the long texts? Do you have any suggestions about it?

[polm]

If the plain textcat is working well it sounds like you should just use it. While Transformers are usually more accurate, there's no guarantee that's always the case.

How long is your long text? The Transformer has several features to help with long text, but in practice it's often challenging to get good results. In this case since the non-Transformer is doing so well maybe you have very sparse but strong features, like specific words that determine the label.

Also note that Transformers are data hungry, and if you only have 50 examples per class it might be unable to learn a useful model, or it may just not be able to modify the weights meaningfully. So if you want to improve the performance of your Transformers model, the first thing I would do is get more data.

[fracav]

thanks for the reply polm,

for the first tests the textcat seems the best solution.

Some texts are quite long, even 300 words. it makes sense to me what you say about sparse features. I have a lot more data available to label. Let's see who will perform better with more data.