Feature Request: Expected calibration error

[vijay-jaisankar]

I would like to raise a PR that implements the ECE metric in this file.

More specifically, the function proposed will take a list of softmax outputs, the y_true labels, and the number of bins. It will then calculate ECE as follows:

source

If this sounds like a good idea, please assign me to the same.

Thanks,
Vijay

[paxcema]

Hey @vijay-jaisankar, love this idea!

Just for reference, one of the pending bits of work from our end is to port into this repo some logic from Lightwood. In particular, we have an ECE implementation here (along with MCE and two other scores). I would recommend that you use it as reference for your contribution.

[vijay-jaisankar]

Hello @paxcema , thanks for the info, will check it out while making the PR!