Tradeoff analysis results not consistent with fairness diagonsis #5
Description
In fairness diagnosis, F1-score is 0.956+-0.003 and it is selected as the perf_metric_name. However, in the tradeoff analysis the results are calculated as:
Performance versus Fairness Trade-Off
Single Threshold
Privileged/Unprivileged Threshold 0.700
Best f1_score* 0.108
Separated Thresholds
Privileged Threshold 0.300
Unprivileged Threshold 0.300
Best f1_score* 0.120
Separated Thresholds under Neutral Fairness (0.001)
Privileged Threshold 0.300
Unprivileged Threshold 0.409
Best f1_score* 0.117
*estimated by approximation, subject to the resolution of mesh grid
Please provide more explanation in the document and help to resolve this issue. Thanks.