Training with TextCatCNN.v2 does not improve

[DarrelYee-SiA]

I'm trying to train a textcat model using the TextCatCNN.v2 architecture, but I find that the score does not improve over iterations, even if the textcat loss converges. Below is the training log:

=========================== Initializing pipeline ===========================
[2021-10-27 17:30:36,538] [INFO] Set up nlp object from config
[2021-10-27 17:30:36,549] [DEBUG] Loading corpus from path: test_200.spacy
[2021-10-27 17:30:36,550] [INFO] Pipeline: ['tok2vec', 'textcat']
[2021-10-27 17:30:36,551] [DEBUG] Due to streamed train corpus, using only first 100 examples for initialization. If necessary, provide all labels in [initialize]. More info: https://spacy.io/api/cli#init_labels
[2021-10-27 17:30:36,557] [INFO] Created vocabulary
[2021-10-27 17:30:36,559] [INFO] Finished initializing nlp object
[2021-10-27 17:30:36,709] [INFO] Initialized pipeline components: ['tok2vec', 'textcat']
✔ Initialized pipeline

============================= Training pipeline =============================
[2021-10-27 17:30:36,727] [DEBUG] Loading corpus from path: test_200.spacy
[2021-10-27 17:30:36,734] [DEBUG] Removed existing output directory: output\model-best
[2021-10-27 17:30:36,741] [DEBUG] Removed existing output directory: output\model-last
ℹ Pipeline: ['tok2vec', 'textcat']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS TEXTCAT  CATS_SCORE  SCORE
---  ------  ------------  ------------  ----------  ------
  0       0          0.00          0.01        0.00    0.00
  0     100          0.00          0.79        0.00    0.00
  0     200          0.00          0.63        0.00    0.00
  0     300          0.00          0.53        0.00    0.00
  0     400          0.00          0.42        0.00    0.00
  0     500          0.00          0.34        0.00    0.00
  0     600          0.00          0.27        0.00    0.00
  0     700          0.00          0.22        0.00    0.00
  0     800          0.00          0.18        0.00    0.00
  0     900          0.00          0.14        0.00    0.00
  0    1000          0.00          0.12        0.00    0.00
  0    1100          0.00          0.09        0.00    0.00
  0    1200          0.00          0.08        0.00    0.00
  0    1300          0.00          0.06        0.00    0.00
  0    1400          0.00          0.05        0.00    0.00

I'm not sure why this is happening because when using TextCatBOW.v2, training accuracy seems to improve as expected:

=========================== Initializing pipeline ===========================
[2021-10-27 17:28:54,060] [INFO] Set up nlp object from config
[2021-10-27 17:28:54,073] [DEBUG] Loading corpus from path: test_200.spacy
[2021-10-27 17:28:54,075] [INFO] Pipeline: ['tok2vec', 'textcat']
[2021-10-27 17:28:54,075] [DEBUG] Due to streamed train corpus, using only first 100 examples for initialization. If necessary, provide all labels in [initialize]. More info: https://spacy.io/api/cli#init_labels
[2021-10-27 17:28:54,081] [INFO] Created vocabulary
[2021-10-27 17:28:54,082] [INFO] Finished initializing nlp object
[2021-10-27 17:28:54,191] [INFO] Initialized pipeline components: ['tok2vec', 'textcat']
✔ Initialized pipeline

============================= Training pipeline =============================
[2021-10-27 17:28:54,208] [DEBUG] Loading corpus from path: test_200.spacy
ℹ Pipeline: ['tok2vec', 'textcat']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS TEXTCAT  CATS_SCORE  SCORE
---  ------  ------------  ------------  ----------  ------
  0       0          0.00          0.01        0.00    0.00
  0     100          0.00          0.54       73.41    0.73
  0     200          0.00          0.20       86.64    0.87
  0     300          0.00          0.10       90.75    0.91
  0     400          0.00          0.05       92.44    0.92
  0     500          0.00          0.03       92.76    0.93
  0     600          0.00          0.02       93.44    0.93
  0     700          0.00          0.01       94.09    0.94
  0     800          0.00          0.01       94.09    0.94
  0     900          0.00          0.01       94.09    0.94

I suspect some settings may be wrong in tok2vec or TextCatCNN.v2, but i'm not sure what. I've attached my config file, a sample of my training data, the static eval binary, my custom data loader function, and the class labels json into a single zip file here spacy cat model.zip. I ran using spacy train config.cfg --output output/. --verbose --code custom_loader.py

If you could help me figure out what's wrong I would be grateful.

[polm]

So investigating this took a while because you have several customizations that aren't relevant and there are some issues with your code.

1. The filename given for the json with the labels is wrong.
2. The custom loader has an infinite loop. Remove the while True (if there is a reason for it it's not clear).
3. Your HashEmbed is customized in an odd way. In particular the "ID" field is currently unused, and while it's not the core of your problem it won't help you.
4. The way your tok2vec is specified in the textcat block is not correct. If you use a listener, look at the quickstart output for examples of how to specify that. You can also look at the textcat goemotions project for a non-listener example.

4 is one of the main issues that was causing you trouble. The BOW model doesn't use tok2vec, so it wasn't clear your tok2vec was misconfigured. But when you tried to use it with the CNN, the tok2vec and the textcat aren't talking to each other, which is one problem.

However even with that resolved your model won't learn. The issue is with the training data loader - when you load in examples, you are setting the value for the true category to 1, and not setting the other categories at all. This works with the BOW classifier but not with the CNN classifier.

I'm not sure if this is a bug on our end or not, but for your purposes you can fix this by setting values for all categories.

Fixed loader:

import spacy
import pickle
from spacy.training import Example
import spacy


labels = ["Women Clothes,Fabric,Others", "Beauty,Skincare,Lip Treatment,Others", "Home & Living,Dinnerware,Water Bottles & Accessories"]
@spacy.registry.readers("corpus_reader_train")
def stream_data(source:str):
    def generate_stream(nlp):
        with open(source, 'rb') as file:
            data_train = pickle.load(file)

        for title, label in data_train:
            doc = nlp.make_doc(title)

            cats = {}
            for ll in labels:
                cats[ll] = 1 if ll == label else 0
            yield Example.from_dict(doc, {"cats": cats})

    return generate_stream

Working config:

[paths]
train = train_binary_200.pkl
dev = test_binary_200.spacy
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["textcat"]
batch_size = 500
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat]
factory = "textcat"
threshold = 0.5

[components.textcat.model]
@architectures = "spacy.TextCatCNN.v2"
exclusive_classes = true
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.textcat.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.textcat.model.tok2vec.encode.width}
attrs = ["NORM","ORTH","LOWER","PREFIX","SUFFIX","SHAPE","ID"]
rows = [2000, 1000, 1000, 1000, 1000, 1000, 1000]
include_static_vectors = true

[components.textcat.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
gold_preproc = false
max_length = 0
limit = 0

[corpora.train]
@readers = "corpus_reader_train"
source = ${paths.train}

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 2000
max_epochs = -1
max_steps = 20000
eval_frequency = 100
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
cats_score = 1.0
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.textcat]

[initialize.components.textcat.labels]
@readers = "spacy.read_labels.v1"
path = "textcat_binary_200.json"
require = true

[initialize.tokenizer]

[DarrelYee-SiA]

Thanks for looking into this, I've gotten it to work. I figured that the generated Example objects might require all possible labels under cats, but the documentation implied that spacy will figure out the labels through the first 100 datapoints. Also, not having all labels worked with TextCatBOW.v2 but not TextCatCNN.v2, which added to the confusion. If the intention is for all labels to be included in Example, maybe this can be clearer in the docs, and all architectures should ideally respond to training in the same way.

Regarding 2), I added the infinite loop since I intended to halt training manually and assumed training would undesirably halt once the loader reached the end of the static corpus file, if without an infinite loop. What's the behaviour once the loader reaches the end of the file, will it start a new epoch automatically?

[polm]

If the intention is for all labels to be included in Example, maybe this can be clearer in the docs, and all architectures should ideally respond to training in the same way.

I am actually not clear on whether that's intended behavior or not, but it's definitely wrong that the models behave differently. We will probably update it to work without all labels.

What's the behaviour once the loader reaches the end of the file, will it start a new epoch automatically?

An "epoch" is normally defined as one pass through the training data. It is normal for training to have many epochs as it iterates over the data repeatedly and gets better at predicting it.

You should not use a streaming corpus unless your training data is too large to fit into memory.