AreaUnderCurve works with non binary labels but should probably print a warning for the user

[camilodlt]

The documentations correctly says that the AreaUnderCurve will calculate the auc metric for the binary case : It is expected that ŷ be a vector of distributions over the binary set of unique elements of y.

However, the function can still work with more than 2 categoricals levels. Looking at the code, the positive class is always assumed to be the last one positive_class = CategoricalArrays.levels(first(ŷ))|> last. So, when passing labels UnivariateFinite{Multiclass{N}}, with N > 2, the value returned is the AUC for the N vs all other labels.

This is not a bug as the code is doing what the documentation says it will, but I'm wondering if the way in which it silently works (but wrong) on the multi class case can be improved (maybe with a warning, or an assertion error?)

Ideally, StatisticalMeasures could implement the macro auc for that case, which is the average of individual AUC scores.

[ablaom]

Thanks for pitching in here, @camilodlt.

I would certainly support adding a multiclass version of AUC. What I think makes sense, however, is to make a general wrapper that turns a binary measure into a 1 vs many metric, as that's a common construction, no? There could be options for aggregation, such as micro, versus macro, versus returning a dictionary, like we have for MulticlassFScore. This code addition would live at StatisticalMeasuresBase.jl. Then, to make it discoverable, we add a new MulticlassAUC metric here, implemented using the wrapper.

What do you think?

This is not a bug as the code is doing what the documentation says it will, but I'm wondering if the way in which it silently works (but wrong) on the multi class case can be improved (maybe with a warning, or an assertion error?)

My first thought is that adding these checks may slow the metric down, although I suppose if we are only talking about CategoricalArrays then we can get the unique levels fast. I wouldn't think users generally expect something called AUC to just work with multiclass data. It's really a binary thing and only becomes multiclass with the 1 vs many contrivance. I don't remember exactly, but I don't think we perform this kind of check more generally.

[camilodlt]

Hello thanks for the reply.

I wouldn't think users generally expect something called AUC to just work with multiclass data. It's really a binary thing and only becomes multiclass with the 1 vs many contrivance

I agree that having a MulticlassAUC and a AUC is a good design choice. I still think that in the binary case, the user should only be allowed to use the AUC function and in the multi class case the MulticlassAUC as this forces the user to choose an aggregation metric, or go with the default which is normally "macro".

In Scikit learn, the auc entry point works for binary and multi class data, however the user must change the multi_class parameter from "raise" (which raises an error if data is multiclass) to "ovr" or "ovo". So a similar choice is forced by design.

For StatisticalMeasures, I think a good starting point can be to implement "ovr". When the user passes multiclass predictions and labels, MulticlassAUC will calculate the different auc scores, then aggregate or return a dict.

My first thought is that adding these checks may slow the metric down, although I suppose if we are only talking about CategoricalArrays then we can get the unique levels fast

Exactly, I think that quickly checking the levels at the beginning is fast and by caching that we could point to the correct metric. This again to avoid having the AUC function working with binary and multi class data while only on the binary case the result will be correct.

Just as an example :

julia> l = categorical(collect(1:3))

julia> labels = categorical([rand(1:3) for i in 1:100]);

julia> preds = [UnivariateFinite(l, softmax(rand(3))) for i in 1:100];

julia> preds[1]
           UnivariateFinite{Multiclass{3}}
     ┌                                        ┐
   1 ┤■■■■■■■■■■■■■■■■■■■■■■ 0.340432
   2 ┤■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.462206
   3 ┤■■■■■■■■■■■■■ 0.197362
     └                                        ┘

julia> StatisticalMeasures.AreaUnderCurve()(preds, labels)
0.46132971506105835

AUC runs fine but the metric is not correct, it only consider the AUC for the class 3 vs all.

This would be a macro average :

using MLJ
using StatisticalMeasures

function auc_one_vs_all(yhat_prob, gt, class_name)
    scores = pdf.(yhat_prob, class_name)
    return StatisticalMeasures.Functions.auc(scores, gt, class_name)
end
function multiclass_auc(prob_preds::Vector{<:UF}, gt::Vector{Int}) where {UF <: UnivariateFinite}
    @assert length(prob_preds) == length(gt)
    classes = levels(first(prob_preds))
    aucs = [auc_one_vs_all(prob_preds, gt, c) for c in classes]
    macro_auc = mean(aucs)
    return macro_auc, aucs
end

julia> multiclass_auc(preds,  Int.(labels.refs))
(0.5309980789085267, [0.5692835692835693, 0.5623809523809524, 0.46132971506105835])

And we can see that 0.46132971506105835 (the 3 vs all) is what AreaUnderCurve returned. (That function I did is just a proof of concept not a proposition to implement it like that)

[ablaom]

Thanks for the elaboration and the sk-learn point of reference. But what say you about my basic proposition that one-vs-many should be first be added as a wrapper one_vs_many_measure(measure; options...) for general measures measure (at StatisticalMeasuresBase.jl) and then MulticlassAUC(; options...) is implemented as one_vs_many_measure(auc; options...). Here options would include aggregation (e.g. MacroAvg())?

[camilodlt]

I think that's a good idea in case other measures also benefit from it 👍

[ablaom]

Okay, I think we're on the same page, then. Any chance you would be interested in this first step, adding the one_versus_rest_measure(measure) wrapper? It would require some work familiarising yourself with that code base, in particular regarding the measure traits, so you know how these should be forwarded to the wrapped measures. I can provide support and a reasonably prompt review.

The API should match what we already have for MulticlassFScore:

MulticlassFScore()
# MulticlassFScore(
#   beta = 1.0,
#   average = MacroAvg(),
#   return_type = LittleDict,
#   levels = nothing,
#   perm = nothing,
#   rev = nothing,
#   checks = true)

using Random
y = rand("abc", 100);
yhat = shuffle(y);
class_weights = Dict('a'=>1, 'b'=>5, 'c'=>10);

MulticlassFScore()(yhat, y)
# 0.2843915343915344

MulticlassFScore()(yhat, y, class_weights)
# 1.6917989417989416

MulticlassFScore(average=NoAvg())(y, yhat)
# LittleDict{Char, Float64, Tuple{Char, Char, Char}, Tuple{Float64, Float64, Float64}} with 3 entries:
#   'a' => 0.214286
#   'b' => 0.305556
#   'c' => 0.333333

MulticlassFScore(levels=['c', 'a', 'b'], average=NoAvg(), return_type=Vector)(yhat, y)
# 3-element Vector{Float64}:
#  0.3333333333333333
#  0.21428571428571427
#  0.3055555555555556

The options for average are: NoAvg(), MicroAvg() and MacroAvg(). Specifying
class_weights is dissallowed for the MicroAvg() option. (We should probably also allow
:none, :micro and :macro.)

Perhaps another would be WeightedMacroAvg() where passing class_weights is dissallowed
and we instead use the distribution of classes in y to carry out the weighting.

The options levels (or rev, perm) only have an effect except when average=NoAvg(),
in which case the only effect is on the order of the result, a Vector or LittleDict.

The wrapper one_versus_rest_measure(measure, options...) would then have options average, levels, rev, and perm.

One annoyance is that StatisticalMeauresBase.jl does not own MacroAvg, etc. Rather StatisticalMeasures does. But I'd be happy to fix that.

After looking at MulticlassFScore code, I don't propose we refactor that. But the exising implementation might be useful in testing the wrapper.