Label match F score computation with the predicted and ground truth are all zeros

[phoxis]

Run the following

y <- create_subset (toyml, rows = 1:4);
y$dataset[1,y$labels$index] <- c (0, 0, 0, 0, 0);
y1 <- y$dataset[,y$labels$index];
multilabel_evaluate (y, y1);

y and y has identical label assignment. But the macro-F1 and other scores are not 1. Probably because the label y3 is all zeros, and therefore it translates to 0. This should be 1 in such a case.

[rivolli]

This is a troubling issue, and I don't know how to solve it.
The F1 formula is: tp / (tp + 0.5 * (fp+fn))
If tp is 0 the F1 is 0, considering you have an "empty" label, tp is not possible to be different of 0.

What do you suggest, should I use a different F1 formulation for such cases?

[rivolli]

An alternative for such cases is compute the measures manually:

> x <- multilabel_evaluate (y, y1, labels = TRUE)

> x$labels
   accuracy AUC balacc        F1 precision recall TP TN FP FN
y1     0.75   1    0.5 0.8571429      0.75      1  3  0  1  0
y2     1.00   1    1.0 1.0000000      1.00      1  3  1  0  0
y3     1.00  NA    NaN 0.0000000      0.00      0  0  4  0  0
y4     1.00   1    1.0 1.0000000      1.00      1  2  2  0  0
y5     1.00   1    1.0 1.0000000      1.00      1  1  3  0  0

[phoxis]

I have previously followed as it is implemented in MULAN library. I used the mldr library in R before as well where there was a bug, and I adapted the F-Score from them and used it as follows.

mldr_MacroFMeasure <- function(trueLabels, bipartition) {
macroPrecision <- colSums(trueLabels & bipartition) / colSums(bipartition)
macroRecall <- colSums(trueLabels & bipartition) / colSums(trueLabels)

macroRecall[is.nan (macroRecall)] <- 1;
macroPrecision[is.nan (macroPrecision)] <- 1;
temp <- (macroPrecision * macroRecall * 2 / (macroPrecision + macroRecall))
temp[is.nan(temp)] <- 0;
return ( mean (temp));
}

This was a quick fix, but essentially, in case the F-Score is NaN then I consider it 1. It is a quick and dirty trick, but I think when all the targets are zeros and the predictions are all zeros, the F-Score should be 1?
A way can be a check if the target is all zeros, then the labels can be flipped (all zeros to ones) to continue with the computation?