"Overflow encountered in exp" while using textcat_multilabel

[Pandalei97]

Hi ! When I train the textcal model using textcat_multilabel, I often get a RuntimeWarning in the sigmoid function. The training continues but the score is being extremely bad. Can you solve this ?

Message example:

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'textcat_multilabel']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS TEXTC...  CATS_SCORE  SCORE
---  ------  ------------  -------------  ----------  ------
  0       0          0.00           0.00       65.45    0.65
{path_to_venv}\lib\site-packages\thinc\backends\ops.py:575: RuntimeWarning: overflow encountered in exp
  return cast(FloatsType, 1.0 / (1.0 + self.xp.exp(-X)))
  0     200          0.14           3.49       60.59    0.61
  1     400          0.16           3.02       68.35    0.68

Summary on data:

============================ Data file validation ============================
✔ Corpus is loadable
✔ Pipeline can be initialized with data

=============================== Training stats ===============================
Language: fr
Training pipeline: tok2vec, textcat_multilabel
2207 training docs
1226 evaluation docs
⚠ 12 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 31398 total word(s) in the data (6188 unique)
10 most common words: '.' (1431), 'de' (1338), ',' (934), ':' (753), 'd'' (563),
'la' (482), 'et' (383), 'en' (382), '-' (362), 'pas' (307)
ℹ 947364 vectors (947364 unique keys, 300 dimensions)
⚠ 1333 words in training data without vectors (0.04%)
10 most common words without vectors: ' ' (162), ' ' (72), '  ' (29),
...(Some other words)

====================== Text Classification (Multilabel) ======================
ℹ Text Classification: 1 label(s)
Labels: 'NEGATION'
Labels in train data: 'NEGATION' (911.0)

================================== Summary ==================================
✔ 2 checks passed
⚠ 2 warnings

Some additional environment informations:
Spacy : 3.0.6
Thinc : 8.0.3
Python : 3.6.5

Thanks in advance for your response ! :)

[polm]

So you're using textcat_multilabel but you only have one label. Do you expect to only have one label?

If you have a category and you want to label whether documents are in it or not, you should use normal (non-multilabel) text cat, and label documents as positive or negative (so you have two labels). Some frameworks or libraries will handle that differently, but spaCy requires that to avoid creating a special case for binary classification.

Additionally, the label NEGATION is a little weird. I wouldn't typically expect that to apply to a single document - it's more something that would be applicable to a span in a document or something. Can you give some examples of your data and what you're trying to do?

I realize these sounds like high-level questions in response to a low-level error, but the textcat models are designed to choose from multiple labels, so having only one won't work and seeing math-related issues isn't terribly surprising.

[Pandalei97]

Thanks for your advice ! We have measured that when we were in Spacy2.0. The result was not so satisfying, so we decided to train the textcat model separately. I will try to see if it's better to train them together in Spacy 3.0. :)

[arc12]

@adrianeboyd re "I've seen the same warning in the demo textcat pipelines, but never got a chance to look into the details."

I think this might be worthwhile, as the issue seems real to me (and the answer to the OP got side-tracked into some good practice matters).

I've experienced the issue as reported (also with plain textcat, not just _multilabel). I have 63 categories which are singly assigned to almost 10000 documents. There are almost equal numbers of documents in each category.

At first, I just ignored the warning, as the CAT_SCORE is good. HOWEVER: when I look at the accuracy for each category separately, I find that some fail completely. For these categories, using "dev" documents for the "problem category" gives prediction scores of zero. Visual inspection of the document text makes me think there are sufficient keywords for the classifier to learn. Indeed, I am wondering whether there are some keywords which are directly encoding the category (or nearly so). That seems a plausible cause of overflow. It might also explain why the issue is not widely reported, yet appears in textcat demo cases.

Unfortunately, I do not understand the inner workings of thinc sufficiently to work out how to proceed. Also, I'm not in a position to share my data.

I should add that a similar warning is given when running textcat.predict()

thinc\backends\ops.py:572: RuntimeWarning: overflow encountered in exp
  return cast(FloatsType, 1.0 / (1.0 + self.xp.exp(-X)))

[Pandalei97]

@polm @adrianeboyd Should this be reported as an issue or labelled as a bug ? For our data, we are forced to use TextCatCNN since there is no such error fot this architecture. But as you mentioned in your documentation, it's less accurate than TextCatEnsemble, and we do have worse scores than those of our textcat models trained in Spacy2.0.

[Malet]

I encountered this bug as a result of badly labelled data - some of the labels were set to -1

[adrianeboyd]

We should do something to avoid this warning, but it really looks like it's a warning and not an immediate error. Whether it's slightly under or over the overflow, the returned result is going to be 0.0, so I don't immediately see how this would cause the problems described above. There could obviously be other problems in the model design, but I think this warning might be a red herring.

While some of the demo projects do have really minimal training data with just a handful of sentences, the demo textcat cases have more realistic data that shouldn't lead to unusual overfitting.

It looks like there isn't enough validation to catch data issues like a label set to -1, so I could see that leading to weird behavior during training.