-
Notifications
You must be signed in to change notification settings - Fork 170
Open
Description
From my understanding, the workflow suggested in the README is the following (supposing we want to perform hyperparameter optimization):
- Split data in train-val-test sets
- For each hyperparameter configuration, train model on train set and evaluate it on val set
- Choose model with lowest val loss: retrain it on train+val set
- Calibrate model with temperature scaling using the val set
- Evaluate final model on test set
But why should we calibrate the model only on the validation set? For example, in scikit-learn calibration example they calibrate the classifier on the train set.
My idea would be the following:
- Split data in train-val-test sets
- For each hyperparameter configuration, train and calibrate model on train set, then evaluate on val set
- Choose model with lowest val loss: retrain it and calibrate it on train+val set
- Evaluate final model on test set
Is there something I'm missing?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels