Textcat model has extremely low score on predictions - too little sample (19 categories, 5400 rows).

[goonhoon]

Hi everyone,

I work with 5400 rows of text and labels to train a model to recognise which type of a contractual provision the text refers to. The number of categories is 19, each with the following number of texts to train on. They are usually one sentence to one shorter paragraph long).

509	DOCUMENT NAME
509	PARTIES 
471	DATE 
392	EFFECTIVE DATE 
415	TERM 
227	RENEWAL 
115	RENEWAL NOTICE 
433	GOVERNING LAW 
255	NON-COMPETE 
76	EXCLUSIVITY 
141	NON-SOLICIT 
200	CONVENIENCE 
139	CHANGE OF CONTROL 
383	ASSIGNMENT 
435	LICENSE 
434	LIABILITY 
92	LIQUIDATED DAMAGES 
102	WARRANTY 
185	INSURANCE

With certain simpler text samples, the score is pretty high, such as here:

BARLEY SUPPLY AGREEMENT


{'DOCUMENT NAME': 0.8416135311126709, 'PARTIES': 0.010904299095273018, 'DATE': 0.020291084423661232, 'EFFECTIVE DATE': 0.018535619601607323, 'TERM': 0.00805568415671587 [...]

Whereas with some more complex clause the prediction does not even make it to the top 5 (albeit this one is nearly identical to the ones it was trained on):

Neither party may assign any of its rights or obligations under this Agreement without the prior written consent of the other except that: (1) Vapotherm may assign this Agreement or [...]


{'DOCUMENT NAME': 2.390841136179489e-10, 'PARTIES': 2.8532981533047064e-10, 'DATE': 5.917693557400128e-10, 'EFFECTIVE DATE': 3.7701397559430916e-07, 'TERM': 4.451022959983675e-06 [...]

The training evaluation when creating a new clean model:

ℹ Pipeline: ['textcat']
ℹ Initial learn rate: 0.001
E    #       LOSS TEXTCAT  CATS_SCORE  SCORE 
---  ------  ------------  ----------  ------
  0       0          0.05        1.70    0.02
  0     200          7.85       39.23    0.39
  0     400          5.48       49.62    0.50
  0     600          5.50       54.69    0.55
  0     800          4.97       57.10    0.57
  0    1000          3.84       60.80    0.61
  0    1200          4.47       59.97    0.60
  1    1400          4.03       61.68    0.62
  1    1600          2.99       62.10    0.62
  1    1800          3.00       61.73    0.62
  2    2000          2.85       61.46    0.61
  2    2200          2.39       64.21    0.64
  3    2400          2.85       65.45    0.65
  3    2600          2.29       64.70    0.65
  4    2800          2.49       63.48    0.63
  5    3000          2.13       63.67    0.64
  5    3200          1.95       62.98    0.63
  6    3400          1.86       62.83    0.63
  7    3600          2.11       63.74    0.64
  7    3800          2.05       63.54    0.64
  8    4000          1.79       63.05    0.63
✔ Saved pipeline to output directory
contracat\model-last

I apologise for the wall of data above. My question is: is this a common occurence for a dataset of this size? I understand that it's far to small for it to work properly, but I would expect more accurate guesses, especially given how structured legalese is.

Many thanks!

[goonhoon]

Oh wow, I thought the evaluations are ordered but they are not, hence why I missed the predictions actually being correct. Apologies for the useless topic then!

Is there a way to order them from largest to lowest score?

[adrianeboyd]

Not built-in, but doc.cats is a standard python dictionary that you can modify however you'd like.