Multilabel textcat with transformers: Hyperparameter Tuning

[hp52]

Hi,
I'm training a multilabel textclassifier with transformers. I want to do some hyperparameter tuning. I'm wondering which parameters I should consider, I was thinking about

components.textcat_multilabel.model.linear_model.ngram_size
training.dropout
training.optimizer.learn_rate.initial_rate

Does that make sense and are there any additional parameters that I should consider?

My config file looks like this:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "de"
pipeline = ["transformer","textcat_multilabel"]
batch_size = 64
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat_multilabel]
factory = "textcat_multilabel"
threshold = 0.5

[components.textcat_multilabel.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat_multilabel.model.linear_model]
@architectures = "spacy.TextCatBOW.v1"
exclusive_classes = false
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat_multilabel.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "bert-base-german-cased"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
#max_length = 0
max_length = 500
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 300
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
#patience = 1600
patience = 0
#max_epochs = 0
#max_epochs = 20
max_epochs = -1
#max_steps = 20000
max_steps = 0
#eval_frequency = 200
eval_frequency = 400
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
cats_score_desc = null
cats_micro_p = 0.0
cats_micro_r = 0.0
cats_micro_f = 0.0
cats_macro_p = 0.0
cats_macro_r = 0.0
cats_macro_f = 0.0
cats_macro_auc = 0.0
cats_f_per_type = null
cats_macro_auc_per_type = null
cats_score = 1.0

[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.textcat_multilabel]

[initialize.components.textcat_multilabel.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/textcat_multilabel.json"

[initialize.tokenizer]

Thanks for your help.

[svlandeg]

Yep, I think that's a good start. In general, I think the two most important sections are training.batcher and training.optimizer (including the learn_rate).

Have fun experimenting!

[hp52]

Thanks for your reply. So, I could possibly also play with training.batcher.size for example. I'm a bit confused about the batch size. What is the difference between training.batcher.size and nlp.batch_size?

[svlandeg]

Good question! And it's often a bit tricky to find this type of information in the docs, because you have to know where to look.

For the standard nlp section of the config, you can find more information here: https://spacy.io/api/data-formats#config-nlp:

batch_size: Default batch size for Language.pipe and Language.evaluate

So this is really only relevant when making predictions - it's how many docs will be processed together.

The section on training is here: https://spacy.io/api/data-formats#config-training
But then it becomes tricky, because the training.batcher refers to a specific function, and each such "batcher" function can have different arguments. One thing you could do, is try out different batching functions (though they might not all work for transformer-based architectures). An overview of the built-in batcher functions in spaCy is here: https://spacy.io/api/top-level#batchers

Your config has

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

so then we can find more details at the docs specifically for spacy.batch_by_padded.v1: https://spacy.io/api/top-level#batch_by_padded where you will find:

size: The largest padded size to batch sequences into. Can also be a block referencing a schedule, e.g. compounding.

And if you want to learn more about schedules, you'd go onwards to https://thinc.ai/docs/api-schedules#compounding and down the rabbit hole you go ;-)

This is just to say that really there is a lot of flexibilty and a lot of options, and next to the built-in ones, you can even write your own functions. If you end up doing a hyperparameter search on some of these options, we'd definitely be interested in hearing the results! While two use-cases are never the same, it's always interesting to hear what works in some cases, and what doesn't.

[hp52]

Thank your for the explanations, very helpful.