Skip to content

Why calibration parameter T is learned on validation set and not training set? #37

@rgraziosi-fbk

Description

@rgraziosi-fbk

From my understanding, the workflow suggested in the README is the following (supposing we want to perform hyperparameter optimization):

  1. Split data in train-val-test sets
  2. For each hyperparameter configuration, train model on train set and evaluate it on val set
  3. Choose model with lowest val loss: retrain it on train+val set
  4. Calibrate model with temperature scaling using the val set
  5. Evaluate final model on test set

But why should we calibrate the model only on the validation set? For example, in scikit-learn calibration example they calibrate the classifier on the train set.

My idea would be the following:

  1. Split data in train-val-test sets
  2. For each hyperparameter configuration, train and calibrate model on train set, then evaluate on val set
  3. Choose model with lowest val loss: retrain it and calibrate it on train+val set
  4. Evaluate final model on test set

Is there something I'm missing?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions