Which hyperparameters are recommended for tuning on a custom SpaCy NER model?

[ag-din]

Hi everyone,

I want to perform a hyperparameter tuning of a custom spaCy NER model. I understand that the hyperparameters and their values are specified in the config.cfg file.

I have 2 datasets:
A) one dataset has approx. ~60k texts of approx. 500 characters each, where entities (1 word or a set of 2 or 3 words) have been identified with one of 3 possible labels.
B) Other dataset has approx. ~60k texts of approx. 500 characters each, where entities (1 word or a set of 2 or 3 words) have been identified with only one label.

I have reviewed this documentation on hyperparameters in spaCy:

1. FAQ: Guide to understanding hyperparameters in spaCy
2. Fine tuning Vs. Training Spacy NER
3. What parameters are optimized during ner model training?
4. Hyperparameter tuning of custom components

I read in 1) (posted in 2022) that " “If you want to do hyperparameter tuning in spaCy, keep in mind that the right parameters to tune are always directly linked to the kind of data you're working with. ”"
In 3) (posted in 2020): "the only ones really worth tuning on individual problems are the batch size and the dropout rate. Tuning the other hyper-parameters will generally result in less than 1% improvement in accuracy, so we haven't made that a prominent workflow".

However, I have not found any documentation (more up to date) on which hyperparameters are critical and worthwhile for tuning a NER model. Any recommendations?

🤔

Thank you very much.

[polm]

I'm not sure we can give you much better advice than the hyperparameter tuning FAQ, especially if we don't know anything about your data. You should look at the architecture docs for NER and try modifying the parameters there based on how you think they might affect your problem. As mentioned in the older post, for NER we generally don't tweak that many parameters.

[ag-din]

Hi @polm. Thank you for your comment. I have one dataset with approx. ~60k texts of approx. 500 characters each, where entities (1 word or a set of 2 or 3 words) have been identified with one of 3 possible labels. Currently, I'm only tuning nlp.batch_size (500, 1000, 2000), training.dropout (0.1, 0.3, 0.5), training.optimizer.learn_rate (0.001 to 0.01, range in log scaling) and training.optimizer.@optimizers ("Adam.v1", "RAdam.v1"), do you think I should include any more or remove any of the above? I appreciate your time.

[polm]

Thanks for the information about your data, but that's not enough information to really help - it's really important to actually see data before making predictions about hyperparameters, and even then usually the best you can do is give an educated guess. Like it says in the FAQ, keep in mind that hyperparameter tuning is often the last thing you should do, and that the important thing is to just pick some parameters and try them.