Train only improve loss and auc

[info2000]

I'm training a textcat model with spacy cli train
I did this before and worked well, but with a new train/eval dataset files, created exactly equal than before, isn't training

I tried to change batcher type from padded to words, seed, eval interval, score_weights, but no changes

someone could help me about where could be the issue?

train item sample: doc.cats and doc.doc

`{'Arts AND Entertainment': 0, 'Autos AND Vehicles': 0, 'Beauty AND Fitness': 1, 'Books AND Literature': 0, 'Business AND Industrial': 0, 'Computers AND Electronics': 0, 'Finance': 0, 'Food AND Drink': 0, 'Games': 0, 'Health': 0, 'Hobbies AND Leisure': 0, 'Home AND Garden': 0, 'Internet AND Telecom': 0, 'Jobs AND Education': 0, 'Law AND Government': 0, 'Online Communities': 0, 'People AND Society': 0, 'Pets AND Animals': 0, 'Real Estate': 0, 'Reference': 0, 'Science': 0, 'Shopping': 0, 'Sports': 0, 'Travel': 0} 

Estos son los tres coches eléctricos presentados por Foxconn Model C, Model E y Model T, así son los coches eléctricos fabricados y desarrollados por Foxconn Hace tan solo unos días, ya os adelantamos en Somos Eléctricos las primeras imágenes de uno de los coches eléctricos que Foxconn iba a presentar hoy, pero lo que no sabíamos es que en total iba a ver tres propuestas. El gigante taiwanés, conocido principalmente por ser el fabricante de dispositivos tan conocidos como el iPhone y el iPad de Apple, ahora se atreve con el mercado de la automoción, aprovechando esta oportunidad que está surgiendo con la transformación automotriz hacia una era 100% elé`

**eval item sample:**
`{'Arts AND Entertainment': 0, 'Autos AND Vehicles': 0, 'Beauty AND Fitness': 0, 'Books AND Literature': 0, 'Business AND Industrial': 0, 'Computers AND Electronics': 0, 'Finance': 0, 'Food AND Drink': 0, 'Games': 0, 'Health': 0, 'Hobbies AND Leisure': 0, 'Home AND Garden': 0, 'Internet AND Telecom': 0, 'Jobs AND Education': 0, 'Law AND Government': 0, 'Online Communities': 0, 'People AND Society': 1, 'Pets AND Animals': 0, 'Real Estate': 0, 'Reference': 0, 'Science': 0, 'Shopping': 0, 'Sports': 0, 'Travel': 0} 

Horóscopo del 10 de diciembre de 2021 y del fin de semana Horóscopo del 10 de diciembre de 2021 y del fin de semanaActualizado Viernes, 10 diciembre 2021 - 08:06La actriz Emmanuelle Chriqui cumple 44 años.ARIESEste fin de semana irradiarás entusiasmo y vitalidad, y no dudarás en mostrar tu lado más atrevido y valiente para conseguir eso que llevas tiempo planeando. Te sonreirá la suerte. TAUROConfía más en ti y evita comentar tus proyectos hasta que no estén totalmente consolidados. Este fin de semana te sentará de maravilla hacer una pequeña escapada para cambiar de aires. GÉMINISParece que la vida te lleva por caminos insospechados y te arrebata lo`

first steps output:

`E    #       LOSS TRANS...  LOSS TEXTCAT  CATS_SCORE  CATS_MICRO_P  CATS_MACRO_P  SCORE 
---  ------  -------------  ------------  ----------  ------------  ------------  ------
  0       0           0.00          0.04        0.00          0.00          0.00    0.00
  0     103           0.00          4.11        0.00          0.00          0.00    0.00
  0     206           0.00          4.06        0.00          0.00          0.00    0.00
  0     309           0.00          4.02        0.00          0.00          0.00    0.00
  0     412           0.00          3.95        0.00          0.00          0.00    0.00
  0     515           0.00          3.99        0.00          0.00          0.00    0.00
  0     618           0.00          3.98        0.00          0.00          0.00    0.00
Waiting for W&B process to finish... (success).`

`
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 15

[nlp]
lang = "es"
pipeline = ["transformer","textcat"]
batch_size = 8
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat]
factory = "textcat"
scorer = {"@scorers":"spacy.textcat_scorer.v1"}
threshold = 0.5

[components.textcat.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat.model.linear_model]
@architectures = "spacy.TextCatBOW.v1"
exclusive_classes = true
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "dccuchile/bert-base-spanish-wwm-cased"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 25000
max_epochs = -1
max_steps = 0
eval_frequency = 103
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = false
size = 128
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.WandbLogger.v3"
project_name = "cat1Exclusive"
remove_config_values = ["paths.dev", "corpora.train.path", "corpora.dev.path"]


[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 1000
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
cats_score = null
cats_score_desc = null
cats_micro_p = 0.5
cats_micro_r = null
cats_micro_f = null
cats_macro_p = 0.5
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null


[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]
`

[polm]

If you're getting flat zero values that usually points to a data problem. What's the output of spacy debug data with your dataset?

[info2000]

@polm this is the debug output,
is the same with the working datasets than with this failing

tcmalloc: large alloc 1073750016 bytes == 0x4ef72000 @  0x7f83d63922a4 0x592b76 0x4286f3 0x5bb021 0x5bb176 0x5936cc 0x548c51 0x51566f 0x593dd7 0x511e2c 0x593dd7 0x511e2c 0x598ef4 0x511f68 0x598ef4 0x511f68 0x598ef4 0x7f81ddeb38a6 0x4bc4ab 0x7f81ddeb1dc3 0x593784 0x515244 0x549e0e 0x4bca8a 0x59c019 0x595ef6 0x5134a6 0x549e0e 0x593fce 0x511e2c 0x549e0e
Some weights of the model checkpoint at dccuchile/bert-base-spanish-wwm-cased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertModel were not initialized from the model checkpoint at dccuchile/bert-base-spanish-wwm-cased and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.


✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: es
Training pipeline: transformer, textcat
74099 training docs
1308 evaluation docs
⚠ 58 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 9218054 total word(s) in the data (241271 unique)
10 most common words: 'de' (602219), ',' (408143), 'la' (283379), 'en' (236065),
'.' (217706), 'el' (208993), 'que' (204316), 'y' (177785), 'a' (155704), 'los'
(110924)
ℹ No word vectors present in the package

================== Text Classification (Exclusive Classes) ==================
ℹ Text Classification: 24 label(s)
Labels: 'Games', 'Food AND Drink', 'Health', 'Sports', 'Computers AND
Electronics', 'Beauty AND Fitness', 'Internet AND Telecom', 'Arts AND
Entertainment', 'Books AND Literature', 'People AND Society', 'Business AND
Industrial', 'Law AND Government', 'Pets AND Animals', 'Science', 'Home AND
Garden', 'Finance', 'Autos AND Vehicles', 'Hobbies AND Leisure', 'Reference',
'Shopping', 'Travel', 'Jobs AND Education', 'Real Estate', 'Online Communities'

================================== Summary ==================================
✔ 2 checks passed
⚠ 1 warning

[polm]

OK, that output looks fine.

I tried to change batcher type from padded to words, seed, eval interval, score_weights, but no changes

I would undo all your changes (use the default settings) and then change the eval_interval to something like 1000 (or higher). The eval interval doesn't affect the actual process of training, but 1. evaluation is relatively slow 2. You have 74k training documents. I wouldn't expect significant progress every 100 iterations (800 docs, with batch size of 8).

Also, just to be sure there are no issues with your data, I would recommend training a model without Transformers first - it should train very quickly.